SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 1. Assume random variables  $x$  and  $y$  follow a joint distribution  $p(x, y) = p(x)p(y|x)$ , where  $p(y|x)$  is given by  $f(y, h(x))$ . Then  $p(y|x) = p(y|z)|_{z=h(x)}$ , where the variable  $z$  is defined as  $z = h(x)$ .
Proof. We can write down the joint distribution over all variables as follows:
$$
p (x, y, z) = p (x) f (y, h (x)) \delta (z - h (x)). \tag {40}
$$
By integrating  $x$  out, we get
$$
\begin{array}{l} p (y, z) = \int p (x, y, z) d x = \int p (x) f (y, h (x)) \delta (z - h (x)) d x = \int p (x) f (y, z) \delta (z - h (x)) d x = (41) \\ = f (y, z) \int p (x) \delta (z - h (x)) d x = f (y, z) p (z) (42) \\ \end{array}
$$
$$
\left. p (y | z) \right| _ {z = h (x)} = \frac {p (y , z)}{p (z)} \Bigg | _ {z = h (x)} = f (y, z) \bigg | _ {z = h (x)} = f (y, h (x)) = p (y | x) \tag {43}
$$
$$
q ^ {\mathrm {S S}} \left(x _ {t} \mid x _ {0}\right) = h _ {t} \left(x _ {t}\right) \exp \left\{\eta_ {t} \left(x _ {0}\right) ^ {\top} \mathcal {T} \left(x _ {t}\right) - \Omega_ {t} \left(x _ {0}\right) \right\}, \tag {13}
$$
$$
q ^ {\mathrm {s s}} \left(x _ {t - 1} \mid x _ {t: T}\right) = \int q ^ {\mathrm {s s}} \left(x _ {t - 1} \mid x _ {0}\right) q ^ {\mathrm {s s}} \left(x _ {0} \mid x _ {t: T}\right) d x _ {0} \tag {44}
$$
$$
q ^ {\mathrm {s s}} (x _ {0} | x _ {t: T}) = \frac {q (x _ {0}) \prod_ {s = t} ^ {T} q ^ {\mathrm {s s}} (x _ {s} | x _ {0})}{q ^ {\mathrm {s s}} (x _ {t : T})} = \frac {q (x _ {0})}{q ^ {\mathrm {s s}} (x _ {t : T})} \left(\prod_ {s = t} ^ {T} h _ {s} (x _ {s})\right) \exp \left\{\sum_ {s = t} ^ {T} \left(\eta_ {s} (x _ {0}) ^ {\top} \mathcal {T} (x _ {s}) - \Omega_ {s} (x _ {0})\right) \right\} = \tag {45}
$$
$$
\begin{array}{l} = \frac {q \left(x _ {0}\right)}{q ^ {\mathrm {S S}} \left(x _ {t : T}\right)} \left(\prod_ {s = t} ^ {T} h _ {s} \left(x _ {s}\right)\right) \exp \left\{\sum_ {s = t} ^ {T} \left(\left(A _ {s} f \left(x _ {0}\right) + b _ {s}\right) ^ {\top} \mathcal {T} \left(x _ {s}\right) - \Omega_ {s} \left(x _ {0}\right)\right) \right\} = (46) \\ = \frac {q \left(x _ {0}\right)}{q ^ {\mathrm {S S}} \left(x _ {t : T}\right)} \left(\prod_ {s = t} ^ {T} h _ {s} \left(x _ {s}\right)\right) \exp \left\{f \left(x _ {0}\right) ^ {\top} \sum_ {s = t} ^ {T} A _ {s} ^ {\top} \mathcal {T} \left(x _ {s}\right) + \sum_ {s = t} ^ {T} \left(b _ {s} ^ {\top} \mathcal {T} \left(x _ {s}\right) - \Omega_ {s} \left(x _ {0}\right)\right) \right\} = (47) \\ = \frac {q \left(x _ {0}\right)}{q ^ {\mathrm {S S}} \left(x _ {t : T}\right)} \left(\prod_ {s = t} ^ {T} h _ {s} \left(x _ {s}\right)\right) \exp \left\{f \left(x _ {0}\right) ^ {\top} G _ {t} + \sum_ {s = t} ^ {T} \left(b _ {s} ^ {\top} \mathcal {T} \left(x _ {s}\right) - \Omega_ {s} \left(x _ {0}\right)\right) \right\} = \left[ \begin{array}{l} a \\ b e n o r m a l i z e d \end{array} \right] = (48) \\ = \frac {q \left(x _ {0}\right) \exp \left\{f \left(x _ {0}\right) ^ {\top} G _ {t} - \sum_ {s = t} ^ {T} \Omega_ {s} \left(x _ {0}\right) \right\}}{\int q \left(x _ {0}\right) \exp \left\{f \left(x _ {0}\right) ^ {\top} G _ {t} - \sum_ {s = t} ^ {T} \Omega_ {s} \left(x _ {0}\right) \right\} d x _ {0}} = \left[ \text {L e m m a 1 f o r} G _ {t} \text {a s} z, \right] = q ^ {\mathrm {S S}} \left(x _ {0} \mid G _ {t}\right) (49) \\ \end{array}
$$
$$
q ^ {\mathrm {s s}} \left(x _ {t - 1} \mid x _ {t: T}\right) = \int q ^ {\mathrm {s s}} \left(x _ {t - 1} \mid x _ {0}\right) q ^ {\mathrm {s s}} \left(x _ {0} \mid x _ {t: T}\right) d x _ {0} = \int q ^ {\mathrm {s s}} \left(x _ {t - 1} \mid x _ {0}\right) q ^ {\mathrm {s s}} \left(x _ {0} \mid G _ {t}\right) d x _ {0} = q ^ {\mathrm {s s}} \left(x _ {t - 1} \mid G _ {t}\right) \tag {50}
$$
D Gaussian SS-DDPM is equivalent to Gaussian DDPM
$$
\frac {\overline {{\alpha}} _ {t} ^ {\mathrm {S S}}}{1 - \overline {{\alpha}} _ {t} ^ {\mathrm {S S}}} = \frac {\overline {{\alpha}} _ {t} ^ {\mathrm {D D P M}}}{1 - \overline {{\alpha}} _ {t} ^ {\mathrm {D D P M}}} - \frac {\overline {{\alpha}} _ {t + 1} ^ {\mathrm {D D P M}}}{1 - \overline {{\alpha}} _ {t + 1} ^ {\mathrm {D D P M}}}. \tag {24}
$$
Proof. We start by listing the necessary definitions and expressions used in the Gaussian DDPM model (Ho et al., 2020).
$$
q ^ {\mathrm {D D P M}} \left(x _ {0: T}\right) = q \left(x _ {0}\right) \prod_ {i = 1} ^ {T} q ^ {\mathrm {D D P M}} \left(x _ {t} \mid x _ {t - 1}\right) \tag {51}
$$
$$
q ^ {\mathrm {D D P M}} \left(x _ {t} \mid x _ {t - 1}\right) = \mathcal {N} \left(x _ {t}; \sqrt {1 - \beta_ {t}} x _ {t - 1}, \beta_ {t} \mathbf {I}\right) \tag {52}
$$
$$
\alpha_ {s} ^ {\mathrm {D D P M}} = 1 - \beta_ {s} ^ {\mathrm {D D P M}} \tag {53}
$$
$$
\bar {\alpha} _ {t} ^ {\mathrm {D D P M}} = \prod_ {s = 1} ^ {t} \alpha_ {s} ^ {\mathrm {D D P M}} \tag {54}
$$
$$
q ^ {\mathrm {D D P M}} \left(x _ {t} \mid x _ {0}\right) = \mathcal {N} \left(x _ {t}; \sqrt {\bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} x _ {0}, \left(1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}\right) \mathbf {I}\right) \tag {55}
$$
$$
\tilde {\mu} _ {t} \left(x _ {t}, x _ {0}\right) = \frac {\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}} \beta_ {t}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} x _ {0} + \frac {\sqrt {\alpha_ {t} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}\right)}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} x _ {t} \tag {56}
$$
$$
\tilde {\beta} _ {t} ^ {\mathrm {D D P M}} = \frac {1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} \beta_ {t} \tag {57}
$$
$$
q ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t}, x _ {0}\right) = \mathcal {N} \left(x _ {t - 1}; \tilde {\mu} _ {t} \left(x _ {t}, x _ {0}\right), \tilde {\beta} _ {t} ^ {\mathrm {D D P M}} \mathbf {I}\right) \tag {58}
$$
$$
p _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {0: T}\right) = q ^ {\mathrm {D D P M}} \left(x _ {T}\right) \prod_ {i = 1} ^ {T} p _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t}\right) \tag {59}
$$
$$
p _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t}\right) = \mathcal {N} \left(x _ {t - 1}; \tilde {\mu} _ {t} \left(x _ {t}, x _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {t}, t\right)\right), \tilde {\beta} _ {t} ^ {\mathrm {D D P M}} \mathbf {I}\right) \tag {60}
$$
In this notation, the variational lower bound for DDPM can be written as follows (here we omit the term corresponding to  $x_{T}$  because it's either zero or a small constant):
$$
\begin{array}{l} \mathcal {L} ^ {\mathrm {D D P M}} (\theta) = \mathbb {E} _ {q ^ {\mathrm {D D P M}} \left(x _ {0: T}\right)} \left[ \log p _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {0} \mid x _ {1}\right) - \sum_ {t = 2} ^ {T} D _ {K L} \left(q ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t}, x _ {0}\right) \| p _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t}\right)\right) \right] = (61) \\ = \mathbb {E} _ {q ^ {\mathrm {D D P M}} \left(x _ {0: T}\right)} \left[ \log p _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {0} \mid x _ {1}\right) - \sum_ {t = 2} ^ {T} D _ {K L} \left(q ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t}, x _ {0}\right) \| q ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t}, x _ {0}\right) | _ {x _ {0} = x _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {t}, t\right)}\right) \right] (62) \\ \end{array}
$$
$$
D _ {K L} \left(q ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t}, x _ {0}\right) \| q ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t}, x _ {0}\right) | _ {x _ {0} = x _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {t}, t\right)}\right) = \frac {\left(\frac {\sqrt {\alpha_ {t - 1} ^ {\mathrm {D D P M}} \beta_ {t} ^ {\mathrm {D D P M}}}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} \left(x _ {0} - x _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {t} , t\right)\right)\right) ^ {2}}{2 \tilde {\beta} _ {t} ^ {\mathrm {D D P M}}} \tag {63}
$$
For convenience, we additionally define  $\overline{\alpha}_{T + 1}^{\mathrm{DDPM}} = 0$ . Since all the involved distributions are typically isotropic, we consider a one-dimensional case without the loss of generality.
First, we show that the forward processes  $q^{\mathrm{SS}}(x_0,G_{1:T})$  and  $q^{\mathrm{DDPM}}(x_{0:T})$  are equivalent.
$$
G _ {t} = \frac {1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}} \sum_ {s = t} ^ {T} \frac {\sqrt {\bar {\alpha} _ {s} ^ {\mathrm {S S}}} x _ {s}}{1 - \bar {\alpha} _ {s} ^ {\mathrm {S S}}} \tag {64}
$$
Since  $G_{t}$  is a linear combination of independent (given  $x_0$ ) Gaussian random variables, its distribution (given  $x_0$ ) is also Gaussian. All the terms conveniently cancel out, and we recover the same form as the forward process of DDPM.
$$
\begin{array}{l} q ^ {\mathrm {S S}} \left(G _ {t} \mid x _ {0}\right) = \mathcal {N} \left(G _ {t}; \frac {1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}} \sum_ {s = t} ^ {T} \frac {\bar {\alpha} _ {s} ^ {\mathrm {S S}} x _ {0}}{1 - \bar {\alpha} _ {s} ^ {\mathrm {S S}}}; \left(\frac {1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}}\right) ^ {2} \sum_ {s = t} ^ {T} \left(\frac {\bar {\alpha} _ {s} ^ {\mathrm {S S}}}{(1 - \bar {\alpha} _ {s} ^ {\mathrm {S S}}) ^ {2}} (1 - \bar {\alpha} _ {s} ^ {\mathrm {S S}})\right)\right) = (65) \\ = \mathcal {N} \left(G _ {t}; \frac {1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}} \frac {\bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} x _ {0}; \left(\frac {1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}}\right) ^ {2} \frac {\bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}\right) = (66) \\ = \mathcal {N} \left(G _ {t}; \sqrt {\bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} x _ {0}; \left(1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}\right)\right) = q ^ {\mathrm {D D P M}} \left(x _ {t} \mid x _ {0}\right) | _ {x _ {t} = G _ {t}} (67) \\ \end{array}
$$
Since both processes share the same  $q(x_0)$ , and have the same Markovian structure, matching these marginals sufficiently shows that the processes are equivalent:
$$
q ^ {\mathrm {S S}} \left(x _ {0}, G _ {1: T}\right) = q ^ {\mathrm {D D P M}} \left(x _ {0: T}\right) | _ {x _ {1: T} = G _ {1: T}} \tag {68}
$$
Consequently, the posteriors are matching too:
$$
q ^ {\mathrm {S S}} \left(G _ {t - 1} \mid G _ {t}, x _ {0}\right) = q ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t}, x _ {0}\right) | _ {x _ {t - 1, t} = G _ {t - 1, t}} \tag {69}
$$
Next, we show that the reverse processes  $p_{\theta}^{\mathrm{SS}}(G_{t - 1}|G_t)$  and  $p_{\theta}^{\mathrm{DDPM}}(x_{t - 1}|x_t)$  are the same. Since  $G_{t}$  in SS-DDPM is the same as  $x_{t}$  in DDPM, we can assume the predictive models  $x_{\theta}^{\mathrm{SS}}(G_t,t)$  and  $x_{\theta}^{\mathrm{DDPM}}(x_t,t)$  to coincide at  $x_{t} = G_{t}$ .
$$
\begin{array}{l} G _ {t - 1} = \frac {1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}} \sum_ {s = t - 1} ^ {T} \frac {\sqrt {\bar {\alpha} _ {s} ^ {\mathrm {S S}}} x _ {s}}{1 - \bar {\alpha} _ {s} ^ {\mathrm {S S}}} = \frac {1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}} \frac {\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {S S}}} x _ {t - 1}}{1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {S S}}} + \frac {1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}} \frac {\sqrt {\bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} G _ {t} = (70) \\ = \frac {1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}} \sum_ {s = t - 1} ^ {T} \frac {\sqrt {\bar {\alpha} _ {s} ^ {\mathrm {S S}}} x _ {s}}{1 - \bar {\alpha} _ {s} ^ {\mathrm {S S}}} = \frac {1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}} \frac {\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {S S}}} x _ {t - 1}}{1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {S S}}} + \frac {\sqrt {\alpha_ {t} ^ {\mathrm {D D P M}} (1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}})}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} G _ {t}, (71) \\ \end{array}
$$
where  $x_{t-1} \sim p_{\theta}^{\mathrm{ss}}(x_{t-1}|G_t) = q^{\mathrm{ss}}(x_{t-1}|x_0)|_{x_0 = x_\theta(G_t,t)}$ . Therefore,  $p_{\theta}^{\mathrm{ss}}(G_{t-1}|G_t)$  is also a Gaussian distribution. Let's take care of the mean first:
$$
\mathbb {E} _ {p _ {\theta} ^ {\mathrm {S S}} \left(G _ {t - 1} \mid G _ {t}\right)} G _ {t - 1} = \frac {1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}} \frac {\bar {\alpha} _ {t - 1} ^ {\mathrm {S S}}}{1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {S S}}} x _ {\theta} \left(G _ {t}, t\right) + \frac {\sqrt {\alpha_ {t} ^ {\mathrm {D D P M}} \left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}\right)}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} G _ {t} = \tag {72}
$$
$$
\begin{array}{l} = \frac {1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}} \left(\frac {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}} - \frac {\bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}\right) x _ {\theta} \left(G _ {t}, t\right) + \frac {\sqrt {\alpha_ {t} ^ {\mathrm {D D P M}} \left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}\right)}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} G _ {t} = (73) \\ = \left(\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}} - \frac {\left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}\right) \sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}} \alpha_ {t} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}\right) x _ {\theta} \left(G _ {t}, t\right) + \frac {\sqrt {\alpha_ {t} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}\right)}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} G _ {t} = (74) \\ = \left(\frac {1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}} - \left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}\right) \alpha_ {t} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}\right) \sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}} x _ {\theta} \left(G _ {t}, t\right) + \frac {\sqrt {\alpha_ {t} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}\right)}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} G _ {t} = (75) \\ = \left(\frac {1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}} - \alpha_ {t} ^ {\mathrm {D D P M}} + \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}\right) \sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}} x _ {\theta} \left(G _ {t}, t\right) + \frac {\sqrt {\alpha_ {t} ^ {\mathrm {D D P M}} \left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}\right)}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} G _ {t} = (76) \\ = \frac {\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}} \beta_ {t}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} x _ {\theta} \left(G _ {t}, t\right) + \frac {\sqrt {\alpha_ {t} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}\right)}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} G _ {t} (77) \\ \end{array}
$$
Now, let's derive the variance:
$$
\begin{array}{l} \mathbb {D} _ {p _ {\theta} ^ {\mathrm {S S}} \left(G _ {t - 1} \mid G _ {t}\right)} G _ {t - 1} = \left(\frac {1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}} \frac {\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {S S}}}}{1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {S S}}}\right) ^ {2} \left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {S S}}\right) = (78) \\ = \left(\frac {1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}{\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}} \sqrt {\frac {\bar {\alpha} _ {t - 1} ^ {\mathrm {S S}}}{1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {S S}}}}\right) ^ {2} = \frac {(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}) ^ {2}}{\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}} \frac {\bar {\alpha} _ {t - 1} ^ {\mathrm {S S}}}{1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {S S}}} = \frac {(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}) ^ {2}}{\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}} \left(\frac {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}} - \frac {\bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}\right) = (79) \\ = \left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}\right) \left(1 - \frac {\left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}\right) \alpha_ {t} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}}\right) = \frac {\left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}\right) \left(1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}} - \left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}\right) \alpha_ {t} ^ {\mathrm {D D P M}}\right)}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} = (80) \\ = \frac {1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}} - \alpha_ {t} ^ {\mathrm {D D P M}} + \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}\right) = \frac {1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} \beta_ {t} ^ {\mathrm {D D P M}} = \tilde {\beta} _ {t} ^ {\mathrm {D D P M}} (81) \\ \end{array}
$$
$$
p _ {\theta} ^ {\mathrm {S S}} \left(G _ {t - 1} \mid G _ {t}\right) = \mathcal {N} \left(G _ {t - 1}; \frac {\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}} \beta_ {t}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} x _ {\theta} \left(G _ {t}, t\right) + \frac {\sqrt {\alpha_ {t} ^ {\mathrm {D D P M}}} \left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}\right)}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} G _ {t}, \tilde {\beta} _ {t} ^ {\mathrm {D D P M}}\right) = p _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t}\right) | _ {x _ {t - 1, t} = G _ {t - 1, t}} \tag {82}
$$
Finally, we show that the variational lower bounds are the same.
$$
\begin{array}{l} \mathcal {L} ^ {\mathrm {s s}} (\theta) = \mathbb {E} _ {q ^ {\mathrm {s s}} (x _ {0: T})} \left[ \log p _ {\theta} ^ {\mathrm {s s}} \left(x _ {0} \mid G _ {1}\right) - \sum_ {t = 2} ^ {T} D _ {K L} \left(q ^ {\mathrm {s s}} \left(x _ {t - 1} \mid x _ {0}\right) \| p _ {\theta} ^ {\mathrm {s s}} \left(x _ {t - 1} \mid G _ {t}\right)\right) \right] = (83) \\ = \mathbb {E} _ {q ^ {\mathrm {S S}} \left(x _ {0: T}\right)} \left[ \log p _ {\theta} ^ {\mathrm {S S}} \left(x _ {0} \mid G _ {1}\right) - \sum_ {t = 2} ^ {T} D _ {K L} \left(q ^ {\mathrm {S S}} \left(x _ {t - 1} \mid x _ {0}\right) \| q ^ {\mathrm {S S}} \left(x _ {t - 1} \mid x _ {0}\right) | _ {x _ {0} = x _ {\theta} \left(G _ {t}, t\right)}\right) \right] (84) \\ \end{array}
$$
Since  $G_{1}$  from the star-shaped model is the same as  $x_{1}$  from the DDPM model, the first term  $\log p_{\theta}^{\mathrm{SS}}(x_0|G_1)$  coincides with  $\log p_{\theta}^{\mathrm{DDPM}}(x_0|x_1)|_{x_1 = G_1}$ .
$$
\begin{array}{l} D _ {K L} \left(q ^ {\mathrm {s s}} \left(x _ {t - 1} \mid x _ {0}\right) \| q ^ {\mathrm {s s}} \left(x _ {t - 1} \mid x _ {0}\right) | _ {x _ {0} = x _ {\theta} \left(G _ {t}, t\right)}\right) = \frac {\left(\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {s s}}}\right) \left(x _ {0} - x _ {\theta} \left(G _ {t} , t\right)\right) ^ {2}}{2 \left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {s s}}\right)} = (85) \\ = \frac {\overline {{\alpha}} _ {t - 1} ^ {\mathrm {S S}}}{2 \left(1 - \overline {{\alpha}} _ {t - 1} ^ {\mathrm {S S}}\right)} \left(x _ {0} - x _ {\theta} \left(G _ {t}, t\right)\right) ^ {2} = \frac {1}{2} \left(\frac {\overline {{\alpha}} _ {t - 1} ^ {\mathrm {D D P M}}}{1 - \overline {{\alpha}} _ {t - 1} ^ {\mathrm {D D P M}}} - \frac {\overline {{\alpha}} _ {t} ^ {\mathrm {D D P M}}}{1 - \overline {{\alpha}} _ {t} ^ {\mathrm {D D P M}}}\right) \left(x _ {0} - x _ {\theta} \left(G _ {t}, t\right)\right) ^ {2} = (86) \\ = \frac {1}{2} \frac {\overline {{\alpha}} _ {t - 1} ^ {\mathrm {D D P M}} - \overline {{\alpha}} _ {t} ^ {\mathrm {D D P M}}}{(1 - \overline {{\alpha}} _ {t - 1} ^ {\mathrm {D D P M}}) (1 - \overline {{\alpha}} _ {t} ^ {\mathrm {D D P M}})} (x _ {0} - x _ {\theta} (G _ {t}, t)) ^ {2} = \frac {1}{2} \frac {\overline {{\alpha}} _ {t - 1} ^ {\mathrm {D D P M}} \beta_ {t} ^ {\mathrm {D D P M}}}{(1 - \overline {{\alpha}} _ {t - 1} ^ {\mathrm {D D P M}}) (1 - \overline {{\alpha}} _ {t} ^ {\mathrm {D D P M}})} (x _ {0} - x _ {\theta} (G _ {t}, t)) ^ {2} = (87) \\ = \frac {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}} \beta_ {t} ^ {\mathrm {D D P M}} \left(x _ {0} - x _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {t} , t\right)\right) ^ {2}}{2 \left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}\right) \left(1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}\right)} = \frac {\frac {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}} \left(\beta_ {t} ^ {\mathrm {D D P M}}\right) ^ {2}}{\left(1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}\right) ^ {2}} \left(x _ {0} - x _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {t} , t\right)\right) ^ {2}}{2 \frac {1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} \beta_ {t} ^ {\mathrm {D D P M}}} = (88) \\ = \frac {\left(\frac {\sqrt {\bar {\alpha} _ {t - 1} ^ {\mathrm {D D P M}}} \beta_ {t} ^ {\mathrm {D D P M}}}{1 - \bar {\alpha} _ {t} ^ {\mathrm {D D P M}}} \left(x _ {0} - x _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {t} , t\right)\right)\right) ^ {2}}{2 \tilde {\beta} _ {t} ^ {\mathrm {D D P M}}} = D _ {K L} \left(q ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t}, x _ {0}\right) \| q ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t}, x _ {0}\right) | _ {x _ {0} = x _ {\theta} ^ {\mathrm {D D P M}} \left(x _ {t}, t\right)}\right) (89) \\ \end{array}
$$
Therefore, we finally obtain  $\mathcal{L}^{\mathrm{SS}}(\theta) = \mathcal{L}^{\mathrm{DDPM}}(\theta)$ .
E Duality between star-shaped and Markovian diffusion: general case
We assume that  $\mathcal{T}(x_t)$  and all matrices  $A_{t}$  (except possibly  $A_{T}$ ) are invertible. Then, there is a bijection between the set of tail statistics  $G_{t:T}$  and the tail  $x_{t:T}$ .
First, we show that the variables  $G_{1:T}$  form a Markov chain.
$$
\begin{array}{l} q \left(G _ {t - 1} \mid G _ {t: T}\right) = q \left(G _ {t - 1} \mid x _ {t: T}\right) = \int q \left(G _ {t - 1} \mid x _ {0}, x _ {t: T}\right) q \left(x _ {0} \mid x _ {t: T}\right) d x _ {0} = (90) \\ = \int q \left(G _ {t - 1} \mid G _ {t}, x _ {0}\right) q \left(x _ {0} \mid G _ {t}\right) d x _ {0} = q \left(G _ {t - 1} \mid G _ {t}\right) (91) \\ \end{array}
$$
$$
q \left(x _ {0}, G _ {t: T}\right) = q \left(x _ {0} \mid G _ {1: T}\right) \prod_ {t = 2} ^ {T} q \left(G _ {t - 1} \mid G _ {t: T}\right) = q \left(x _ {0} \mid G _ {1}\right) \prod_ {t = 2} ^ {T} q \left(G _ {t - 1} \mid G _ {t}\right) = q \left(x _ {0}\right) q \left(G _ {1} \mid x _ {0}\right) \prod_ {t = 2} ^ {T} q \left(G _ {t} \mid G _ {t - 1}\right) \tag {92}
$$
The last equation holds since the reverse of a Markov chain is also a Markov chain.
This means that a star-shaped diffusion process on  $x_{1:T}$  implicitly defines some Markovian diffusion process on the tail statistics  $G_{1:T}$ . Due to the definition of  $G_{t-1} = G_t + A_{t-1}^{\mathsf{T}}\mathcal{T}(x_{t-1})$ , we can write down the following factorization of that process:
$$
q \left(x _ {0}, G _ {t: T}\right) = q \left(x _ {0}\right) q \left(G _ {T}\right) \prod_ {t = 2} ^ {T} q \left(G _ {t - 1} \mid G _ {t}, x _ {0}\right) \tag {93}
$$
The posteriors  $q(G_{t-1}|G_t, x_0)$  can then be computed by a change of variables: