SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} - \mathbb {E} _ {x _ {0} \sim p _ {d a t a} (x _ {0})} \ln p _ {\theta} (x _ {0}) = - \mathbb {E} _ {x _ {0} \sim p _ {d a t a} (x _ {0})} \ln \mathbb {E} _ {p _ {\theta} (z _ {t _ {1: T}})} [ p (x _ {0} | z _ {t _ {1: T}}) ] \\ = - \mathbb {E} _ {x _ {0} \sim p _ {\text {d a t a}} (x _ {0})} \ln \mathbb {E} _ {q \left(z _ {t _ {1: T}} \mid x _ {0}\right)} \left[ \frac {p \left(x _ {0} \mid z _ {t _ {1 : T}}\right) p _ {\theta} \left(z _ {t _ {1 : T}}\right)}{q \left(z _ {t _ {1 : T}} \mid x _ {0}\right)} \right] \\ \leq - \operatorname {E L B O} = - \mathbb {E} _ {x _ {0} \sim p _ {\text {d a t a}} (x _ {0})} \mathbb {E} _ {q \left(z _ {t _ {1: T}} \mid x _ {0}\right)} \left[ \ln \frac {p \left(x _ {0} \mid z _ {t _ {1 : T}}\right) p _ {\theta} \left(z _ {t _ {1 : T}}\right)}{q \left(z _ {t _ {1 : T}} \mid x _ {0}\right)} \right] \\ = \mathbb {E} _ {x _ {0} \sim p _ {d a t a} (x _ {0})} \left[ - \mathbb {E} _ {q \left(z _ {t _ {1}} \mid x _ {0}\right)} \ln p \left(x _ {0} \mid z _ {t _ {1}}\right) + \mathrm {K L} [ q \left(z _ {t _ {T}} \mid x _ {0}\right) | | p _ {p r i o r} \left(z _ {t _ {T}}\right) ] \right. \\ + \sum_ {j = 2} ^ {T} \mathbb {E} _ {z _ {t _ {j}} \sim q (z _ {t _ {j}} \mid x _ {0})} \mathbb {E} _ {z _ {t _ {j - 1}} \sim q (z _ {t _ {j - 1}} \mid z _ {t _ {j}}, x _ {0})} \ln \left. \frac {q (z _ {t _ {j - 1}} \mid z _ {t _ {j}} , x _ {0})}{p _ {\theta} (z _ {t _ {j - 1}} \mid z _ {t _ {j}})} \right]. \\ \end{array}
$$
Thus, when we disregard its first term, AS-KLUB is equivalent to -ELBO without its first two terms. Since these three terms typically do not affect optimization, optimizing the generator  $f_{\theta}$  with AS-KLUB is equivalent to using -ELBO.
C Derivation of (Weighted) ELBOs of Gaussian Diffusion from KLUB
Let us denote  $\alpha_0 = 1$  and  $z_0 = x_0$ . Following the definition in Ho et al. [23] and Song et al. [58], we define a Gaussian diffusion-based generative model as
$$
\begin{array}{l} z _ {1} \sim \mathcal {N} (\sqrt {\alpha_ {1}} x _ {0}, 1 - \alpha_ {1}), \\ z _ {2} \sim \mathcal {N} \left(\sqrt {\frac {\alpha_ {2}}{\alpha_ {1}}} z _ {1}, 1 - \frac {\alpha_ {2}}{\alpha_ {1}}\right), \\ \end{array}
$$
$$
z _ {t} \sim \mathcal {N} \left(\sqrt {\frac {\alpha_ {t}}{\alpha_ {t - 1}}} z _ {t - 1}, 1 - \frac {\alpha_ {t}}{\alpha_ {t - 1}}\right),
$$
$$
z _ {T} \sim \mathcal {N} \left(\sqrt {\frac {\alpha_ {T}}{\alpha_ {T - 1}}} z _ {T - 1}, 1 - \frac {\alpha_ {T}}{\alpha_ {T - 1}}\right),
$$
where the diffusion scheduling parameters  $\alpha_{t}$  in this paper is related to  $\beta_{t}$  in Ho et al. [23] as
$$
\beta_ {t} := 1 - \frac {\alpha_ {t}}{\alpha_ {t - 1}}.
$$
The same as the derivations in Ho et al. [23], we can express the forward marginal distribution at time  $t$  as
$$
z _ {t} \sim \mathcal {N} (\sqrt {\alpha_ {t}} x _ {0}, 1 - \alpha_ {t}).
$$
Assuming  $\mathbb{E}[x_0] = 0$  and  $\mathrm{var}[x_0] = 1$ , the signal-to-noise ratio at time  $t$  is defined as
$$
\mathrm {S N R} _ {t} = \mathbb {E} _ {x _ {0}} \left[ \left(\frac {\mathbb {E} [ z _ {t} | x _ {0} ]}{\mathrm {s t d} [ z _ {t} | x _ {0} ]}\right) ^ {2} \right] = \frac {\alpha_ {t}}{1 - \alpha_ {t}} \mathbb {E} [ x _ {0} ^ {2} ] = \frac {\alpha_ {t}}{1 - \alpha_ {t}}.
$$
Since  $\sqrt{\frac{\alpha_{t-1}}{\alpha_t}} x_t \sim \mathcal{N}(z_{t-1}, \frac{\alpha_{t-1}}{\alpha_t} - 1)$  and  $z_{t-1} \sim \mathcal{N}(\sqrt{\alpha_{t-1}} z_0, 1 - \alpha_{t-1})$ , using the conjugacy of the Gaussian distributions with respect to their mean, we can express the conditional posterior as  $q(z_{t-1} | x_0, z_t)$
$$
\begin{array}{l} = \mathcal {N} \left(\left(\frac {1}{1 - \alpha_ {t - 1}} + \frac {\alpha_ {t}}{\alpha_ {t - 1} - \alpha_ {t}}\right) ^ {- 1} \left(\frac {\sqrt {\alpha_ {t - 1}} x _ {0}}{1 - \alpha_ {t - 1}} + \frac {\sqrt {\alpha_ {t - 1} \alpha_ {t}} x _ {t}}{\alpha_ {t - 1} - \alpha_ {t}}\right), \left(\frac {1}{1 - \alpha_ {t - 1}} + \frac {\alpha_ {t}}{\alpha_ {t - 1} - \alpha_ {t}}\right) ^ {- 1}\right) \\ = \mathcal {N} \left(\frac {\sqrt {\alpha_ {t - 1}}}{1 - \alpha_ {t}} (1 - \frac {\alpha_ {t}}{\alpha_ {t - 1}}) x _ {0} + \frac {1 - \alpha_ {t - 1}}{1 - \alpha_ {t}} \sqrt {\frac {\alpha_ {t}}{\alpha_ {t - 1}}} z _ {t}, \quad \frac {1 - \alpha_ {t - 1}}{1 - \alpha_ {t}} (1 - \frac {\alpha_ {t}}{\alpha_ {t - 1}})\right). \\ \end{array}
$$
The forward Gaussian diffusion chain
$$
q \left(z _ {1: T} \mid x _ {0}\right) = \prod_ {t = 1} ^ {T} p \left(z _ {t} \mid z _ {t - 1}\right)
$$
can be equivalently sampled in reverse order as
$$
q (z _ {1: T} \mid x _ {0}) = q (z _ {T} \mid x _ {0}) \prod_ {t = 2} ^ {T} q (z _ {t - 1} \mid z _ {t}, x _ {0}).
$$
As this reverse chain is non-causal and non-Markovian, we need to approximate it with a Markov chain during inference, which is expressed as
$$
p (z _ {1: T}) = p _ {p r i o r} (z _ {T}) \prod_ {t = 2} ^ {T} p _ {\theta} (z _ {t - 1} \mid z _ {t}).
$$
The usual strategy [58, 35, 72] is to utilize the conditional posterior to define
$$
p _ {\theta} \left(z _ {t - 1} \mid z _ {t}\right) = q \left(z _ {t - 1} \mid z _ {t}, \hat {x} _ {0} = f _ {\theta} \left(z _ {t}, t\right)\right).
$$
In what follows, we redefine the time  $t$  as a continuous variable between 0 and 1. We let  $t_{1:T}$  be a set of  $T$  discrete time points within that interval, with  $0 \leq t_1 < t_2 < \ldots < t_T \leq 1$ . The corrupted data observations over these  $T$  time points are defined as  $z_{t_{1:T}}$ .
C.1 Optimization of Gaussian Diffusion via Negative ELBO
Viewing  $p_{\theta}(x_0 \mid z_{1:T})$  as the decoder,  $p_{\theta}(z_{1:T})$  as the prior, and  $q(z_{1:T} \mid x_0)$  as the inference network, we can optimize the generator parameter via the negative ELBO as
$$
\begin{array}{l} - \mathbb {E} _ {x _ {0}} \ln p _ {\theta} (x _ {0}) \leq - \mathrm {E L B O} = \mathbb {E} _ {x _ {0}} \mathbb {E} _ {q (z _ {t _ {1: T}} \mid x _ {0})} \left[ - \ln \frac {p (x _ {0} \mid z _ {t _ {1 : T}}) p (z _ {t _ {1 : T}})}{q (z _ {t _ {1 : T}} \mid x _ {0})} \right] \\ = \mathbb {E} _ {x _ {0}} \mathbb {E} _ {q (z _ {t _ {1}} | x _ {0})} \ln p (x _ {0} | z _ {t _ {1}}) + \mathbb {E} _ {x _ {0}} \mathrm {K L} [ q (z _ {t _ {T}} | x _ {0}) | | p _ {p r i o r} (z _ {t _ {T}}) ] + \sum_ {j = 2} ^ {T} \mathbb {E} _ {x _ {0}} \mathbb {E} _ {q (z _ {t _ {j}} | x _ {0})} [ L (t _ {j - 1}, z _ {t _ {j}}, x _ {0}) ], \\ \end{array}
$$
where the first two terms are often ignored and the focus is placed on the remaining  $T - 2$  terms, defined as
$$
\begin{array}{l} L (s, z _ {t}, x _ {0}) = \mathrm {K L} \left(q \left(z _ {s} \mid z _ {t}, x _ {0}\right) \mid \mid q \left(z _ {s} \mid z _ {t}, \hat {x} _ {0} = f _ {\theta} \left(z _ {t}, t\right)\right)\right) \\ = \frac {1}{2 \frac {1 - \alpha_ {s}}{1 - \alpha_ {t}} \left(1 - \frac {\alpha_ {t}}{\alpha_ {s}}\right)} \left(\frac {\sqrt {\alpha_ {s}}}{1 - \alpha_ {t}} \left(1 - \frac {\alpha_ {t}}{\alpha_ {s}}\right)\right) ^ {2} \| x _ {0} - f _ {\theta} (z _ {t}, t) \| _ {2} ^ {2} \\ = \frac {1}{2} \left(\frac {\alpha_ {s}}{1 - \alpha_ {s}} - \frac {\alpha_ {t}}{1 - \alpha_ {t}}\right) \| x _ {0} - f _ {\theta} \left(z _ {t}, t\right) \| _ {2} ^ {2}, \tag {27} \\ \end{array}
$$
where  $0 < s < t < 1$ . We choose  $s = \max(t - 1 / T, 0)$  during training.
Since  $z_{t} = \sqrt{\alpha_{t}} x_{0} + \sqrt{1 - \alpha_{t}}\epsilon_{t}$ ,  $\epsilon_t\sim \mathcal{N}(0,1)$ , it is true that
$$
x _ {0} = \frac {z _ {t} - \sqrt {1 - \alpha_ {t}} \epsilon_ {t}}{\sqrt {\alpha_ {t}}}.
$$
Instead of directly predicting  $x_0$  given  $x_{t}$ , we can equivalently predict  $\hat{\epsilon}_t = \epsilon_\theta (z_t,t)$  given  $z_{t}$  and let
$$
\hat {x} _ {0} = f _ {\theta} (z _ {t}, t) = \frac {z _ {t} - \sqrt {1 - \alpha_ {t}} \epsilon_ {\theta} (z _ {t} , t)}{\sqrt {\alpha_ {t}}}.
$$
Thus (27) can also be written as
$$
\begin{array}{l} L (s, z _ {t}, x _ {0}) = \frac {1}{2} \left(\frac {\alpha_ {s}}{1 - \alpha_ {s}} - \frac {\alpha_ {t}}{1 - \alpha_ {t}}\right) \| x _ {0} - f _ {\theta} (z _ {t}, t) \| _ {2} ^ {2} \\ = \frac {1}{2} \left(\mathrm {S N R} _ {s} - \mathrm {S N R} _ {t}\right) \| x _ {0} - f _ {\theta} \left(z _ {t}, t\right) \| _ {2} ^ {2} \\ = \frac {1}{2} \left(\frac {\alpha_ {s}}{1 - \alpha_ {s}} - \frac {\alpha_ {t}}{1 - \alpha_ {t}}\right) \frac {1 - \alpha_ {t}}{\alpha_ {t}} \| \epsilon_ {t} - \epsilon_ {\theta} (z _ {t}, t) \| _ {2} ^ {2} \\ = \frac {1}{2} \left(\frac {\mathrm {S N R} _ {s}}{\mathrm {S N R} _ {t}} - 1\right) \| \epsilon_ {t} - \epsilon_ {\theta} \left(z _ {t}, t\right) \| _ {2} ^ {2}, \tag {28} \\ \end{array}
$$
which agrees with Equations 13 and 14 in Kingma et al. [35].
C.2 Optimization of Gaussian Diffusion via KLUB Conditional
Following the definition of KLUB (Conditional), for Gaussian diffusion we have
$$
\operatorname {K L U B} \left(s, z _ {t}, x _ {0}\right) = \operatorname {K L} \left(q \left(z _ {s} \mid \hat {x} _ {0} = f _ {\theta} \left(z _ {t}, t\right), z _ {t}\right) \mid \mid q \left(z _ {s} \mid x _ {0}, z _ {t}\right)\right)
$$
For two Gaussian distributions  $q_{1}$  and  $q_{2}$  that have the same variance, we have  $\mathrm{KL}(q_1||q_2) = \mathrm{KL}(q_2||q_1)$  and hence
$$
\begin{array}{l} \operatorname {K L U B} \left(s, z _ {t}, x _ {0}\right) = \operatorname {K L} \left(q \left(z _ {s} \mid \hat {x} _ {0} = f _ {\theta} \left(z _ {t}, t\right), z _ {t}\right) \mid \mid q \left(z _ {s} \mid x _ {0}, z _ {t}\right)\right) \\ = \operatorname {K L} \left(q \left(z _ {s} \mid x _ {0}, z _ {t}\right) \mid \mid q \left(z _ {s} \mid \hat {x} _ {0} = f _ {\theta} \left(z _ {t}, t\right), z _ {t}\right)\right) \\ = L (s, z _ {t}, x _ {0}). \\ \end{array}
$$
Therefore, over the same set of time points  $t_{1:T}$ , optimizing via KLUB conditional is identical to optimizing via -ELBO. As analyzed before, optimizing via KLUB conditional (i.e., -ELBO) may not be able to directly counteract the error accumulated over the course of diffusion, and hence could lead to slow convergence.
C.3 Optimization of Gaussian Diffusion via KLUB Marginal and SNR-weighted Negative ELBO
Following the definition of KLUB (Marginal), for Gaussian diffusion we have
$$
\begin{array}{l} \operatorname {K L U B} \left(z _ {t}, x _ {0}\right) = \operatorname {K L} \left(\mathcal {N} \left(\sqrt {\alpha_ {t}} x _ {0}, 1 - \alpha_ {t}\right) \mid \mid \mathcal {N} \left(\sqrt {\alpha_ {t}} f \left(z _ {t}, t\right), 1 - \alpha_ {t}\right) \right. \\ = \frac {\alpha_ {t}}{1 - \alpha_ {t}} \| x _ {0} - f (z _ {t}, t) \| _ {2} ^ {2} \\ = \operatorname {S N R} _ {t} \| x _ {0} - f (z _ {t}, t) \| _ {2} ^ {2} \\ = \| \epsilon_ {0} - \epsilon_ {\theta} (z _ {t}, t) \| _ {2} ^ {2}. \\ \end{array}
$$
It is surprising to find out that the KLUB Marginal is identical to the SNR weighted -ELBO first introduced in Ho et al. [23] and further discussed in Hang et al. [20].
C.4 Optimization of Gaussian Diffusion via KLUB
Following beta diffusion, we can also combine two KLUBs as
$$
\begin{array}{l} \omega \mathrm {K L U B} (s, z _ {t}, x _ {0}) + (1 - \omega) \mathrm {K L U B} (z _ {t}, x _ {0}) \\ = \left[ \frac {\omega}{2} \left(\frac {\alpha_ {s}}{1 - \alpha_ {s}} \frac {1 - \alpha_ {t}}{\alpha_ {t}} - 1\right) + (1 - \omega) \right] \| \epsilon_ {0} - \epsilon_ {\theta} (z _ {t}, t) \| _ {2} ^ {2}. \\ \end{array}
$$
Since  $\frac{\alpha_s}{1 - \alpha_s}\frac{1 - \alpha_t}{\alpha_t}$  is in general close to 1 when  $s$  is not too far from  $t$ , when  $\omega$  is not too close to 1, a combination of these two KLUBs does not result in an algorithm that clearly differs from the use of an SNR-weighted negative ELBO.
D Illustration of Forward and Reverse Beta Diffusion
Illustration of Forward Beta Diffusion. We first visualize the beta forward diffusion process by displaying a true image  $x_0$  and its noise-corrupted versions over the course of the forward diffusion process. Specifically, we display the noise corrupted and masked image at time  $t = 0, 0.05, 0.1, \ldots, 1$  as
$$
\tilde {z} _ {t} = \max  \left(\min  \left(\frac {1}{S _ {c a l e}} \left(\frac {z _ {t}}{\alpha_ {t}} - S _ {h i f t}\right), 1\right), 0\right), z _ {t} \sim \mathrm {B e t a} (\eta \alpha_ {t} x _ {0}, \eta (1 - \alpha_ {t} x _ {0})).
$$
It is clear from Figure 1 that with multiplicative beta-distributed noises, the image becomes both noisier and sparser as time increases, and eventually becomes almost completely dark. Thus the forward process of beta diffusion can be considered as simultaneously noisy and masking the pixels. This clearly differs beta diffusion from Gaussian diffusion, whose forward diffusion gradually applies additive random noise and eventually ends at a Gaussian random noise. In addition, the reverse diffusion process of Gaussian diffusion can be considered a denoising process, whereas that of beta diffusion is simultaneously demasking and denoising the data, as illustrated below.
Illustration of Reverse Beta Diffusion. We further illustrate the reverse process of beta diffusion by displaying
$$
\tilde {z} _ {t _ {j - 1}} = \max  \left(\min  \left(\frac {1}{S _ {c a l e}} \left(\frac {z _ {t _ {j - 1}}}{\alpha_ {t _ {j - 1}}} - S _ {h i f t}\right), 1\right), 0\right),
$$
where  $z_{t_{j-1}} = z_{t_j} + (1 - z_{t_j}) p_{(t_{j-1} \leftarrow t_j)}$  is iteratively computed as in Algorithm 2. We also display
$$
\hat {x} _ {0} = f _ {\theta} \left(z _ {t _ {j}}, t _ {j}\right) \approx \mathbb {E} \left[ x _ {0} \mid z _ {t _ {j}} \right],
$$
where the approximation would become more and more accurate when  $\theta$  approaches its theoretical optimal  $\theta^{*}$ , under which we have  $f_{\theta^{*}}(z_{t_j},t_j) = \mathbb{E}[x_0|z_{t_j}]$ .
As shown in Figure 2, starting from a random image drawn from
$$
z _ {t _ {J}} \sim \operatorname {B e t a} (\eta \alpha_ {t _ {J}} \hat {x} _ {0}, \eta (1 - \alpha_ {t _ {J}} \hat {x} _ {0})), \hat {x} _ {0} = \mathbb {E} [ x _ {0} ],
$$
most of whose pixels would be completely dark, beta diffusion gradually demasks and denoises the image towards a clean image through multiplicative transforms, as shown in Algorithm 2.
Statistical distance between the true and generated data distributions
Figure 5: Comparison of the statistical distances between the true and generated data distributions over the course of training. The blue, green, and orange curves are for "Gauss ELBO," "Beta ELBO," and "Beta KLUB," respectively. From the left to right are the plots for Wasserstein-1 distance, Jensen-Shannon divergence, and Hellinger distance, respectively.
Figure 6: Analogy to Figure 5 for comparing "KLUB Conditional," "KLUB Marginal," and "KLUB."
E Additional Results for Synthetic Data
Quantitative Evaluation Metrics. We consider Wasserstein-1, Jensen-Shannon divergence (JSD), and Hellinger distance as three complimentary evaluation metrics. Wasserstein-1 has a high tolerance of misalignment between the supports of the true data density and these of the generated data. In other words, it is not sensitive to both misaligning the modes of the true density and these of the generated density and placing density in the regions where there are no data. By contrast, both JSD and Hellinger can well reflect the misalignments between the supports of high data density regions.
Wasserstein-1: We monitor the performance of different algorithms during training by generating 100k data points using the trained model and drawing 100k data points from the data distribution, and computing the Wasserstein-1 distance between their empirical distributions, which can be done by sorting them and taking the mean of their element-wise absolute differences. If the true distribution is discrete in that it takes values uniformly at random from a discrete set  $\mathcal{D}$ , then we compute the Wasserstein-1 distance by sorting a set of 10k generated samples from small to large and calculating the mean of their absolute element-wise differences with a non-decreasing vector of the same size, which consists of an equal number of copies for each unique value in  $\mathcal{D}$ .
JSD and Hellinger distance: We discretize the 100k generated data into 100 equal-sized bins between 0 and 1 and compute the frequency of the data in each bin, which provides an empirical probability mass function (PMF). We then compute both the JSD and Hellinger distance between the empirical PMF of the generated data and the empirical (true) PMF of the true (discrete) data.
Additional Results. The observations in Figure 3 are further confirmed by Figure 5, which shows that while Gaussian diffusion has good performance measured by the Wasserstein-1 distance, it is considerably worse than beta diffusion in aligning the supports between the true and generated data distributions. Within beta diffusion, either KLUB or its argument-swapped version leads to good performance measured by both the JSD and Hellinger distance, suggesting their excellent ability to concentrate their generations around the true data supports. However, the proposed KLUB has a much better recovery of the true underlying density in comparison to its argument-swapped version, as confirmed by examining the first subplot in Figure 5 and comparing the second and third subplots in Figure 3.
Figure 7: Analogy to Figure 3 for the mixture of continuous range-bounded data and point masses.
Figure 8: Analogy to Figure 5 for the mixture of continuous range-bounded data and point masses.
Figure 9: Analogy to Figure 8 for comparing "KLUB Conditional," "KLUB Marginal," and "KLUB."
We further conduct an ablation study between KLUB and its two variants: "KLUB Conditional" and "KLUB Marginal," corresponding to  $\omega = 1$  and  $\omega = 0$ , respectively, in the loss given by (18). The results suggest "KLUB Conditional" and "KLUB Marginal" have comparable performance, which is not that surprising considering that in theory, they both share the same optimal solution given by (14). More closely examining the plots suggests that "KLUB Marginal" converges faster than "KLUB Conditional" in aligning the generation with the data supports, but eventually delivers comparable performance, and combining them leads to the "KLUB" that has a good overall performance.
We note we have conducted additional experiments by varying model parameters and adopting different diffusion schedules. Our observation is that while each algorithm could do well by carefully tuning these parameters, "Beta KLUB" is the least sensitive to parameter variations and is consistently better than or comparable to the other methods across various combinations of model parameters. While the toy data is valuable for showcasing the unique properties of beta diffusion, the performance of "Beta KLUB" is not that sensitive to model parameters, and the observed patterns may be disrupted by image-specific settings. Hence, their utility in tuning beta diffusion for image generation, where data dimension and model size/architecture are much larger and more complex, is limited.
E.1 Mixture of Continuous Range-bounded Distributions and Point Masses
We consider an equal mixture of three range-bounded distributions and two unit point masses, including a Uniform distribution between 0.1 and 0.2, a Beta(1,5) distribution first scaled by 0.1 and then shifted by 0.3, a unit point mass at  $x_0 = 0.5$ , a Beta(0.5,0.5) distribution first scaled by 0.1 and
then shifted by 0.6, and a unit point mass at  $x_0 = 0.8$ . A random sample is generated as
$$
x _ {0} = y _ {k}, k \sim \mathrm {D i s c r e t e} (\{1, 2, 3, 4, 5 \}), y _ {1} \sim \mathrm {U n i f} (0. 1, 0. 2), y _ {2} \sim \frac {1}{0 . 1} \mathrm {B e t a} (\frac {y _ {2} - 0 . 3}{0 . 1}; 1, 5),
$$
$$
y _ {3} = 0. 5, y _ {4} \sim \frac {1}{0 . 1} \operatorname {B e t a} \left(\frac {y _ {4} - 0 . 6}{0 . 1}; 0. 5, 0. 5\right), y _ {5} = 0. 9. \tag {29}
$$
We follow the same experimental protocol in Section 4.1. The results are shown in Figures 7 and 8, from which we again observe that "Gauss ELBO" does a poor job in aligning its generation with the true data supports and places a proportion of its generation to form smooth transitions between the boundaries of high-density regions; "Beta ELBO" well captures the data supports and density shapes, but has a tendency to overestimate the density of smaller-valued data; whereas "Beta KLUB" very well aligns its generated data with the true data supports and captures the overall density shapes and the sharp transitions between these five range-bounded density regions, largely avoiding placing generated data into zero-density regions.
F Preconditioning of Beta Diffusion for Image Generation
It is often a good practice to precondition the input of a neural network. For beta diffusion, most of our computation is operated in the logit space and hence we will consider how to precondition  $\mathrm{logit}(z_t)$  before feeding it as the input to the generator  $f_{\theta}$ . Specifically, since we have
$$
z _ {t} \sim \operatorname {B e t a} \left(\eta \alpha_ {t} x _ {0}, \eta \left(1 - \alpha_ {t} x _ {0}\right)\right),
$$
we can draw the logit of  $z_{t}$  as
$$
\operatorname {l o g i t} \left(z _ {t}\right) = \ln u _ {t} - \ln v _ {t}, u _ {t} \sim \operatorname {G a m m a} \left(\eta \alpha_ {t} x _ {0}, 1\right), v _ {t} \sim \operatorname {G a m m a} \left(\eta \left(1 - \alpha_ {t} x _ {0}\right), 1\right).
$$
Assume  $x_0 \sim \mathrm{Unif}[x_{\min}, x_{\max}]$ , where  $x_{\min} = S_{shift}$  and  $x_{\max} = S_{\text{scale}} + S_{shift}$ , we have
$$
\begin{array}{l} \mathbb {E} [ \operatorname {l o g i t} (z _ {t}) ] = \mathbb {E} [ \ln u _ {t} ] - \mathbb {E} [ \ln v _ {t} ] \\ = \mathbb {E} _ {x _ {0}} [ \psi (\eta \alpha_ {t} x _ {0}) ] - \mathbb {E} _ {x _ {0}} [ \psi (\eta (1 - \alpha_ {t} x _ {0})) ], \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {x _ {0}} [ \psi (\eta \alpha_ {t} x _ {0}) ] = \frac {1}{\eta \alpha_ {t} (x _ {\max} - x _ {\min})} \int_ {\eta \alpha_ {t} x _ {\min}} ^ {\eta \alpha_ {t} x _ {\max}} d \ln \Gamma (z) \\ = \frac {1}{\eta \alpha_ {t} \left(x _ {\max } - x _ {\min }\right)} \left[ \ln \Gamma \left(\eta \alpha_ {t} x _ {\max }\right) - \ln \Gamma \left(\eta \alpha_ {t} x _ {\min }\right) \right], \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {x _ {0}} [ \psi (\eta (1 - \alpha_ {t} x _ {0})) ] = \frac {1}{\eta \alpha_ {t} (x _ {\max } - x _ {\min })} \int_ {\eta (1 - \alpha_ {t} x _ {\max })} ^ {\eta (1 - \alpha_ {t} x _ {\min })} d \ln \Gamma (z) \\ = \frac {1}{\eta \alpha_ {t} \left(x _ {\operatorname* {m a x}} - x _ {\operatorname* {m i n}}\right)} \left[ \ln \Gamma \left(\eta \left(1 - \alpha_ {t} x _ {\operatorname* {m i n}}\right)\right) - \ln \Gamma \left(\eta \left(1 - \alpha_ {t} x _ {\operatorname* {m a x}}\right)\right) \right]. \\ \end{array}
$$
We further estimate the variance of  $\mathrm{logit}(z_t)$  as
$$
\begin{array}{l} \operatorname {v a r} \left[ \log \mathrm {i t} \left(z _ {t}\right) \right] = \operatorname {v a r} \left[ \ln u _ {t} \right] + \operatorname {v a r} \left[ \ln v _ {t} \right] \\ = \mathbb {E} _ {x _ {0}} [ \psi^ {(1)} (\eta \alpha_ {t} x _ {0}) ] + \mathbb {E} _ {x _ {0}} [ \psi^ {(1)} (\eta (1 - \alpha_ {t} x _ {0})) ] + \operatorname {v a r} _ {x _ {0}} [ \psi (\eta \alpha_ {t} x _ {0}) ] + \operatorname {v a r} _ {x _ {0}} [ \psi (\eta (1 - \alpha_ {t} x _ {0})) ] \\ \end{array}
$$
$$
\mathbb {E} _ {x _ {0}} [ \psi^ {(1)} (\eta \alpha_ {t} x _ {0}) ] = \frac {1}{\eta \alpha_ {t} (x _ {\max} - x _ {\min})} [ \psi (\eta \alpha_ {t} x _ {\max}) - \psi (\eta \alpha_ {t} x _ {\min}) ],
$$
$$
\mathbb {E} _ {x _ {0}} [ \psi^ {(1)} (\eta (1 - \alpha_ {t} x _ {0})) ] = \frac {1}{\eta \alpha_ {t} (x _ {\max} - x _ {\min})} [ \psi (\eta (1 - \alpha_ {t} x _ {\min})) - \psi (\eta (1 - \alpha_ {t} x _ {\max})) ],
$$
$$
\begin{array}{l} \operatorname {v a r} _ {x _ {0}} \left[ \psi (\eta \alpha_ {t} x _ {0}) \right] = \mathbb {E} _ {x _ {0}} \left[ \psi^ {2} \left(\eta \alpha_ {t} x _ {0}\right) - \left(\mathbb {E} _ {x _ {0}} \left[ \psi (\eta \alpha_ {t} x _ {0}) \right]\right) ^ {2} \right. \\ \approx \max  \left(\frac {1}{1 0 0} \sum_ {i = 0} ^ {1 0 0} \frac {\psi^ {2} \left(\eta \alpha_ {t} \left(x _ {\operatorname* {m i n}} + \frac {i}{1 0 0} (x _ {\operatorname* {m a x}} - x _ {\operatorname* {m i n}})\right)\right)}{2 \mathbf {1} (i = 0) + \mathbf {1} (i = 1 0 0)} - (\mathbb {E} _ {x _ {0}} [ \psi (\eta \alpha_ {t} x _ {0}) ]) ^ {2}, 0\right), \\ \end{array}
$$