SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To build the look-up table, we need a robust way to estimate the mutual information. The target mutual information  $I_{\mathcal{N}}^{\mathrm{ss}}(x_0; x_t)$  can be computed analytically when the data  $q(x_0)$  follows a Gaussian distribution. When the data follows an arbitrary distribution, it can be approximated with a Gaussian mixture and the mutual information can be calculated using numerical integration. Estimating the mutual information for arbitrary noisy distributions is more difficult. We find that the Kraskov estimator (Kriskov et al., 2004) works well when the mutual information is high ( $I > 2$ ). When the mutual information is lower, we build a different estimator using DSIVI bounds (Molchanov et al., 2019).
$$
I ^ {\mathrm {S S}} (x _ {0}; x _ {\nu}) = D _ {K L} \left(q ^ {\mathrm {S S}} (x _ {0}, x _ {t}) \parallel q ^ {\mathrm {S S}} (x _ {0}) q ^ {\mathrm {S S}} (x _ {t})\right) = \mathcal {H} [ x _ {t} ] - \mathcal {H} [ x _ {t} | x _ {0} ] \tag {143}
$$
This conditional entropy is available in closed form for many distributions in the exponential family. Since the marginal distribution  $q^{\mathrm{ss}}(x_0,x_t) = \int q^{\mathrm{ss}}(x_t|x_0)q(x_0)dx_0$  is a semi-implicit distribution (Yin & Zhou, 2018), we can use the DSIVI sandwich (Molchanov et al., 2019) to obtain an upper and lower bound on the marginal entropy  $\mathcal{H}[x_t]$ :
$$
\mathcal {H} [ x _ {t} ] = - \mathbb {E} _ {q ^ {\mathrm {s s}}} \log q ^ {\mathrm {s s}} (x _ {t}) = - \mathbb {E} _ {q ^ {\mathrm {s s}}} \log \int q ^ {\mathrm {s s}} (x _ {t} | x _ {0}) q (x _ {0}) d x _ {0} \tag {144}
$$
$$
\mathcal {H} \left[ x _ {t} \right] \geq - \mathbb {E} _ {x _ {0} ^ {0: K} \sim q \left(x _ {0}\right)} \mathbb {E} _ {x _ {t} \sim q \left(x _ {t} \mid x _ {0}\right) \mid_ {x _ {0} = x _ {0} ^ {0}}} \log \frac {1}{K + 1} \sum_ {k = 0} ^ {K} q ^ {\mathrm {s s}} \left(x _ {t} \mid x _ {0} ^ {k}\right) \tag {145}
$$
$$
\mathcal {H} \left[ x _ {t} \right] \leq - \mathbb {E} _ {x _ {0} ^ {0: K} \sim q \left(x _ {0}\right)} \mathbb {E} _ {x _ {t} \sim q \left(x _ {t} \mid x _ {0}\right) \mid_ {x _ {0} = x _ {0} ^ {0}}} \log \frac {1}{K} \sum_ {k = 1} ^ {K} q ^ {\mathrm {S S}} \left(x _ {t} \mid x _ {0} ^ {k}\right) \tag {146}
$$
These bounds are asymptotically exact and can be estimated using Monte Carlo. We use  $K = 1000$  when the mutual information is high  $(\frac{1}{2} \leq I < 2)$ ,  $K = 100$  when the mutual information is lower  $(0.002 \leq I < \frac{1}{2})$ , and estimate the expectations using  $M = 10^{8}K^{-1}$  samples for each timestamp. For values  $I > 2$  we use the Kraskov estimator with  $M = 10^{5}$  samples and  $k = 10$  neighbors. For values  $I < 0.002$  we fit an exponential curve  $i(t) = e^{at + b}$  to interpolate between the noisy SIVI estimates, obtained with  $K = 50$  and  $M = 10^{5}$ .
For evaluating the mutual information  $I(x_0;G_t)$  between the clean data and the tail statistics, we use the Kraskov estimator with  $k = 10$  and  $M = 10^5$ .
The mutual information look-up table for the Beta star-shaped diffusion, as well as the used estimations of the mutual information, are presented in Figure 11. The resulting schedule for the beta diffusion is presented in Figure 12, and the comparison of the mutual information schedules for the tail statistics between Beta SS-DDPM and the referenced Gaussian SS-DDPM is presented in Figure 13.
For Categorical SS-DDPM we estimate the mutual information using Monte Carlo. We then choose the noising schedule to match the cosine schedule used by Hoogeboom et al. (2021) using a similar technique.
H Normalizing the tail statistics
We illustrate different strategies for normalizing the tail statistics in Figure 14. Normalizing by the sum of coefficients is not enough, therefore we resort to matching the mean and the variance empirically. We refer to this trick as time-dependent tail normalization.
Also, proper normalization allows us to visualize the tail statistics by projecting them back into the original domain using  $\mathcal{T}^{-1}(\tilde{G}_t)$ . The effect of normalization is illustrated in Figure 15.
Figure 11: The mutual information look-up table for Beta star-shaped diffusion.
Figure 12: The schedule for Beta star-shaped diffusion.
Figure 13: Mutual information between clean data and the tail statistics for beta star-shaped diffusion.
I Synthetic data
We compare the performance of DDPM and SS-DDPM on synthetic tasks with exotic data domains. In these tasks, we train and sample from DDPM and SS-DDPM using  $T = 64$  steps. We use an MLP with 3 hidden layers of size 512, swish activations and residual connections through hidden layers (He et al., 2015). We use sinusoidal positional time embeddings (Vaswani et al., 2017) of size 32 and concatenate them with the normalized tail statistics  $\tilde{G}_t$ . We add a mapping to the corresponding domain on top of the network. During training, we use gradient clipping and EMA weights to improve stability. All models on synthetic data were trained for 350k iterations with batch size 128. In all our experiments with SS-DDPM on synthetic data we use time-dependent tail normalization. We use DDPM with linear schedule and  $L_{simple}$  or  $L_{vlb}$  as a loss function. We choose between  $L_{simple}$  and  $L_{vlb}$  objective based on the KL divergence between the data distribution and the model distribution  $D_{KL}(q(x_0)\parallel p_\theta (x_0))$ . To make an honest comparison, we precompute normalization statistics for DDPM in the same way that we do in time-dependent tail normalization. In DDPM, a neural network makes predictions using  $x_{t}$  as an input, so we precompute normalization statistics for  $x_{t}$  and fix them during the training and sampling stages.
Probabilistic simplex We evaluate Dirichlet SS-DDPM on a synthetic problem of generating objects on a three-dimensional probabilistic simplex. We use a mixture of three Dirichlet distributions with different parameters as training
(a) No normalization  $\tilde{G}_t = G_t$
(b) Normalized by the sum of coefficients  $\tilde{G}_t = \frac{G_t}{\sum_{s=t}^{T} a_s}$
(c) Matched the mean and variance  $\tilde{G}_t = (G_t - \mathbb{E}G_t)\frac{\mathrm{std}(\mathcal{T}(x_0))}{\mathrm{std}(G_t)} +\mathbb{E}\mathcal{T}(x_0)$
Figure 14: Example trajectories of the normalized tail statistics with different normalization strategies. The trajectories are coming from a single dimension of Beta SS-DDPM.
Figure 15: Visualizing the tail statistics for Beta SS-DDPM with different normalization by mapping the normalized tail statistics  $\tilde{G}_t$  into the data domain using  $\mathcal{T}^{-1}(\tilde{G}_t)$ . Top row: normalized by the sum of coefficients,  $\tilde{G}_t = \frac{G_t}{\sum_{s=t}^{T} a_s}$ . Bottom row: matched the mean and variance,  $\tilde{G}_t = (G_t - \mathbb{E}G_t) \frac{\mathrm{std}(\mathcal{T}(x_0))}{\mathrm{std}(G_t)} + \mathbb{E}\mathcal{T}(x_0)$
data. The Dirichlet SS-DDPM forward process is illustrated in Figure 8. To map the predictions to the domain, we put the Softmax function on the top of the MLP. We optimize Dirichlet SS-DDPM on the VLB objective without any modifications using Adam with a learning rate of 0.0004. The DDPM was trained on  $L_{vlb}$  using Adam with a learning rate of 0.0002. An illustration of the samples is presented in Table 4.
Table 4: Generating objects from three-dimensional probabilistic simplex.
Original	DDPM	Dirichlet SS-DDPM
Symmetric positive definite matrices We evaluate Wishart SS-DDPM on a synthetic problem of generating symmetric positive definite matrices of size  $2 \times 2$ . We use a mixture of three Wishart distributions with different parameters as training data. The Wishart SS-DDPM forward processes are illustrated in Figure 10. For the case of symmetric positive definite matrices  $V$ , MLP predicts a lower triangular factor  $L_{\theta}$  from the Cholesky decomposition  $V_{\theta} = L_{\theta}L_{\theta}^{\mathrm{T}}$ . For stability of sampling from the Wishart distribution and estimation of the loss function, we add a scalar matrix  $10^{-4}\mathbf{I}$  to the predicted symmetric positive definite matrix  $V_{\theta}$ . In Wishart SS-DDPM we use an analogue of  $L_{simple}$  as a loss function. To improve the training stability, we divide each KL term corresponding to a timestamp  $t$  by  $n_t$ , the de-facto concentration parameter
from the used noising schedule (see Table 6). Wishart SS-DDPM was trained using Adam with a learning rate of 0.0004. The DDPM was trained on  $L_{simple}$  using Adam with a learning rate of 0.0004. Since positive definite  $2 \times 2$  matrices can be interpreted as ellipses, we visualize the samples by drawing the corresponding ellipses. An illustration of the samples is presented in Table 5.
Table 5: Generating symmetric positive definite  $2 \times 2$  matrices.
Original	DDPM	Wishart SS-DDPM
J Geodesic data
We evaluate von Mises-Fisher SS-DDPM on the problem of restoring the distribution on the sphere from empirical data of fires on the Earth's surface (EOSDIS, 2020). We illustrate points on the sphere using a 2D projection of the Earth's map. We train and sample from von Mises-Fisher SS-DDPM using  $T = 100$  steps. The forward process is illustrated in Figure 9. We use the same MLP architecture as described in Appendix I and also use time-dependent tail normalization. To map the prediction onto the unit sphere, we normalize the three-dimensional output of the MLP. We optimize  $L_{vlb}$  using the AdamW optimizer with a learning rate of 0.0002 and exponential decay with  $\gamma = 0.999997$ . The model is trained for 2,000,000 iterations with batch size 100. For inference, we also use EMA weights with a decay of 0.9999.
K Discrete data
We evaluate Categorical SS-DDPM on the task of unconditional character-level text generation on the text8 dataset. We evaluate our model on sequences of length 256. Using the property of Categorical SS-DDPM, that we can directly compute mutual information between  $x_0$  and  $G_t$  (see Appendix F.4), we match the noising schedule of  $G_t$  to the noising schedule of  $x_t$  by Austin et al. (2021). We optimize Categorical SS-DDPM on the VLB objective. Following Austin et al. (2021), we use the default T5 encoder architecture with 12 layers, 12 heads, mlp dim 3072 and qkv dim 768. We add positional embeddings to the sequence of tail statistics  $G_t$  and also add the sinusoidal positional time embedding to the beginning. We use Adam (Kingma & Ba, 2015) with learning rate  $5 \times 10^{-4}$  with a 10,000-step learning rate warmup, but instead of inverse sqrt decay, we use exponential decay with  $\gamma = 0.999995$ . We use a standard 90,000,000/5,000,000/500,000 train-test-validation split and train neural network for 512 epochs (time costs when using 3 NVIDIA A100 GPUs: training took approx. 112 hours and estimating NLL on the test set took approx. 2.5 hours). Since the Softmax function does not break the sufficiency of tail statistic in Categorical SS-DDPMs (see Appendix F.4), we can use the Softmax function as an efficient input normalization for the neural network. Categorical SS-DDPM also provides us with a convenient way to define  $\log p(x_0|G_1)$ , and we make use of it to estimate the NLL score (see Appendix F.4).
L Image data
In our experiments with Beta SS-DDPM, we use the  $\mathrm{NCSN}++$  neural network architecture and train strategy by Song et al. (2020b) (time costs when using 4 NVIDIA 1080 GPUs: training took approx. 96 hours, sampling of 50,000 images took approx. 10 hours). We put a sigmoid function on the top of  $\mathrm{NCSN}++$  to map the predictions to the data domain. We use time-dependent tail normalization and train Beta SS-DDPM with  $T = 1000$ . We matched the noise schedule in Beta SS-DDPM to the cosine noise schedule in Gaussian DDPM (Nichol & Dhariwal, 2021) in terms of mutual information, as described in Appendix G.
M Changing discretization
When sampling from DDPMs, we can skip some timestamps to trade off computations for the quality of the generated samples. For example, we can generate  $x_{t_1}$  from  $x_{t_2}$  in one step, without generating the intermediate variables  $x_{t_1 + 1:t_2 - 1}$ :
$$
\tilde {p} _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {t _ {1}} \mid x _ {t _ {2}}\right) = q ^ {\mathrm {D D P M}} \left(x _ {t _ {1}} \mid x _ {t _ {2}}, x _ {0}\right) | _ {x _ {0} = x _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {t _ {2}}, t _ {2}\right)} = \tag {147}
$$
$$
\begin{array}{l} = \mathcal {N} \left(x _ {t _ {1}}; \frac {\sqrt {\overline {{\alpha}} _ {t _ {1}} ^ {\mathrm {D D P M}}} \left(1 - \frac {\overline {{\alpha}} _ {t _ {2}} ^ {\mathrm {D D P M}}}{\overline {{\alpha}} _ {t _ {1}} ^ {\mathrm {D D P M}}}\right)}{1 - \overline {{\alpha}} _ {t _ {2}} ^ {\mathrm {D D P M}}} x _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {t _ {2}}, t _ {2}\right) + \frac {\sqrt {\frac {\overline {{\alpha}} _ {t _ {2}} ^ {\mathrm {D D P M}}}{\overline {{\alpha}} _ {t _ {1}} ^ {\mathrm {D D P M}}} \left(1 - \overline {{\alpha}} _ {t _ {1}} ^ {\mathrm {D D P M}}\right)}{1 - \overline {{\alpha}} _ {t _ {2}} ^ {\mathrm {D D P M}}} x _ {t _ {2}}, \frac {1 - \overline {{\alpha}} _ {t _ {1}} ^ {\mathrm {D D P M}}}{1 - \overline {{\alpha}} _ {t _ {2}} ^ {\mathrm {D D P M}}} \left(1 - \frac {\overline {{\alpha}} _ {t _ {2}} ^ {\mathrm {D D P M}}}{\overline {{\alpha}} _ {t _ {1}} ^ {\mathrm {D D P M}}}\right)\right) = (148) \\ = \mathcal {N} \left(x _ {t _ {1}}; \frac {\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}} - \bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}\right)} x _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {t _ {2}}, t _ {2}\right) + \frac {\sqrt {\bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}\right)}{\sqrt {\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}\right)} x _ {t _ {2}}, \frac {(1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}) \left(\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}} - \bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}\right)}{(1 - \bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}) \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}}\right) (149) \\ \end{array}
$$
In the general case of SS-DDPM, we can't just skip the variables. If we skip the variables, the corresponding tail statistics will become atypical and the generative process will fail. To keep the tail statistics adequate, we can sample all the intermediate variables  $x_{t}$  but do it in a way that doesn't use additional function evaluations:
$$
G _ {t _ {1}} = \sum_ {s = t _ {1}} ^ {t _ {2} - 1} A _ {s} ^ {\top} \mathcal {T} \left(x _ {s}\right) + G _ {t _ {2}}, \text {w h e r e} \tag {150}
$$
$$
x _ {s} \sim q \left(x _ {s} \mid x _ {0}\right) \mid_ {x _ {0} = x _ {\theta} ^ {\mathrm {S S}} \left(G _ {t _ {2}}, t _ {2}\right)} \tag {151}
$$
In case of Gaussian SS-DDPM this trick is equivalent to skipping the variables in DDPM:
$$
G _ {t _ {1}} = \frac {1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}}} \sum_ {s = t _ {1}} ^ {t _ {2} - 1} \frac {\sqrt {\bar {\alpha} _ {s} ^ {\mathrm {S S}}} x _ {s}}{1 - \bar {\alpha} _ {s} ^ {\mathrm {S S}}} + \frac {1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}}} \frac {\sqrt {\bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}}}{1 - \alpha_ {t _ {2}} ^ {\mathrm {D D P M}}} G _ {t _ {2}} = \tag {152}
$$
$$
\begin{array}{l} = \frac {1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}}} \sum_ {s = t _ {1}} ^ {t _ {2} - 1} \frac {\bar {\alpha} _ {s} ^ {\mathrm {S S}}}{1 - \bar {\alpha} _ {s} ^ {\mathrm {S S}}} x _ {\theta} ^ {\mathrm {S S}} \left(G _ {t _ {2}}, t _ {2}\right) + \frac {1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}}} \sum_ {s = t _ {1}} ^ {t _ {2} - 1} \sqrt {\frac {\bar {\alpha} _ {s} ^ {\mathrm {S S}}}{1 - \bar {\alpha} _ {s} ^ {\mathrm {S S}}}} \epsilon_ {s} + \frac {\sqrt {\bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}\right)}{\sqrt {\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}\right)} G _ {t _ {2}} (153) \\ \mathbb {E} G _ {t _ {1}} = \frac {1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}}} \left(\frac {\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}} - \frac {\bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}}\right) x _ {\theta} ^ {\mathrm {S S}} \left(G _ {t _ {2}}, t _ {2}\right) + \frac {\sqrt {\bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}\right)}{\sqrt {\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}\right)} G _ {t _ {2}} = (154) \\ = \frac {\left(1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}\right) \left(\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}} - \bar {\alpha} _ {t _ {2}}\right)}{\sqrt {\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}\right) \left(1 - \bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}\right)} x _ {\theta} ^ {\mathrm {S S}} \left(G _ {t _ {2}}, t _ {2}\right) + \frac {\sqrt {\bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}\right)}{\sqrt {\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}\right)} G _ {t _ {2}} = (155) \\ = \frac {\left(\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}} - \bar {\alpha} _ {t _ {2}}\right)}{\sqrt {\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}\right)} x _ {\theta} ^ {\mathrm {S S}} \left(G _ {t _ {2}}, t _ {2}\right) + \frac {\sqrt {\bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}\right)}{\sqrt {\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}\right)} G _ {t _ {2}} (156) \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {D} G _ {t _ {1}} = \frac {\left(1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}\right) ^ {2}}{\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}} \sum_ {s = t _ {1}} ^ {t _ {2} - 1} \frac {\bar {\alpha} _ {s} ^ {\mathrm {S S}}}{1 - \bar {\alpha} _ {s} ^ {\mathrm {S S}}} = \frac {\left(1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}\right) ^ {2}}{\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}} \frac {\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}} - \bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}}{\left(1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}\right) \left(1 - \bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}\right)} = (157) \\ = \frac {\left(1 - \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}\right) \left(\bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}} - \bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}\right)}{\left(1 - \bar {\alpha} _ {t _ {2}} ^ {\mathrm {D D P M}}\right) \bar {\alpha} _ {t _ {1}} ^ {\mathrm {D D P M}}} (158) \\ \end{array}
$$
In general case, this trick can be formalized as the following approximation to the reverse process:
$$
p _ {\theta} ^ {\mathrm {S S}} \left(x _ {t _ {1}: t _ {2}} \mid G _ {t _ {2}}\right) = \prod_ {t = t _ {1}} ^ {t _ {2}} q ^ {\mathrm {S S}} \left(x _ {t} \mid x _ {0}\right) | _ {x _ {0} = x _ {\theta} \left(\mathcal {G} _ {t} \left(x _ {t: T}\right), t\right)} \approx \prod_ {t = t _ {1}} ^ {t _ {2}} q ^ {\mathrm {S S}} \left(x _ {t} \mid x _ {0}\right) | _ {x _ {0} = x _ {\theta} \left(\mathcal {G} _ {t _ {2}} \left(x _ {t _ {2}: T}\right), t _ {2}\right)} \tag {159}
$$
Table 6: Examples of SS-DDPM in different families. There are many different ways to parameterize the distributions and the corresponding schedules. Further details are discussed in Appendix F.1-F.8.
DISTRIBUTION
qSS(xt|xt)	NOISING SCHEDULE	TAIL STATISTIC
Gt(xt:T)	KL DIVERGENCE
DKL(qSSxt|xt||pSSxt|xt+1:T))
GAUSSIAN
N(xt;√αtxt,1-αt)xt∈R	1←t→0 αt→T 0	1-α′s ∑S=t T/√α′s x̂s, where α′t = ∑S=t T/1+∑S=t T/1-αs	α′t(xθ-x0)2/2(1-αt)
BETA
Beta(xt;αt(x0),βt(x0))xt∈[0,1]	αt(x0)=1+νtxt
βt(x0)=1+νt(1-x0)	∑Tt=tsνs log xs/1-xs	log Beta(αt(xθ),βt(xθ))+Beta(αt(x0),βt(x0))+νt(x0-x0)(ψ(αt(x0))-ψ(βt(x0)))
+∞←t→0 νt→T 0
DIRICHLET
Dir(xt;αt1(x0),...,αtK(x0))xt∈[0,1]K
∑i=1Kxtk=1	αkt(x0)=1+νtxtk+∞←t→0 νt→T 0	∑Tt=tsνs log xs	∑K[k=1] [log Γ(αkt(xθ))/Γ(αkt(x0))+νt(x0-k-αk)ψ(αkt(x0))]
CATEGORICAL
Cat(xt;pt(x0))xt∈{0,1}D
∑i=1Dxti=1	pt(x0)=x0QtI←t→0 Qt→T QT	∑Tt=ts log (Qsxt)	∑D[i=1](pt(x0))i log (pt(x0))i/(pt(x0))i
VON MISES
vM(xt;xt0,kt)xt∈[−π,π]	+∞←t→0 κt→T 0	∑Tt=ts κs (cos xs/sin xs)	κtI1(τt)/I0(τt)(1-cos(x0-x0))
VON MISES-FISHER
vMF(xt;xt0,kt)xt∈[−1,1]K
||xt||=1	+∞←t→0 κt→T 0	∑Tt=ts κs xs	κtIK/2(τt)/IK/2-1(τt)xt(0-x0)
GAMMA
Γ(xt;αt,βt(x0))xt∈(0,+∞)	βt(x0)=αt(ξt+(1-ξt)x0-1)	∑Tt=ts αs(1-ξs)xs	αt [log βt(x0)/βt(x0)+βt(x0)/βt(x0)-1]
+∞←t→0 αt→T αT0←t→0ξt→T 1
WISHART
W(Xt;nt,Vt(X0))xt∈RP×pxt>0	μt(X0)=ξtI+(1-ξt)X0-1	∑Tt=ts n_s(1-ξ_s)X_s	-nt/2 [log |Vt-1(X_θ)Vt(X_0)| -tr (Vt-1(X_θ)Vt(X_0))+p]
Vt(X0)=nt-1μt-1(X0)
+∞←t→0 nt→T n_T0←t→0ξt→T 1