SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {\partial^ {2} Q _ {N _ {c}} \left(\theta_ {0}\right)}{\partial \theta_ {i} \partial \theta_ {j} ^ {T}} = 2 \left[ \frac {1}{N _ {c}} \sum_ {i = 1} ^ {N _ {c}} \frac {\partial g \left(x _ {i} , \theta_ {0}\right)}{\partial \theta_ {i}} \right] ^ {T} W _ {N _ {c}} \left[ \frac {1}{N _ {c}} \sum_ {i = 1} ^ {N _ {c}} \frac {\partial g \left(x _ {i} , \theta_ {0}\right)}{\partial \theta_ {j}} \right] \tag {19} \\ + 2 \left[ \frac {1}{N _ {c}} \sum_ {i = 1} ^ {N _ {c}} \frac {\partial^ {2} g (x _ {i} , \theta_ {0})}{\partial \theta_ {i} \theta_ {j}} \right] ^ {T} W _ {N _ {c}} \left[ \frac {1}{N _ {c}} \sum_ {i = 1} ^ {N _ {c}} g (x _ {i}, \theta_ {0}) \right] \xrightarrow {p} 2 \Gamma_ {0, i} ^ {T} W \Gamma_ {0, j}. \\ \end{array}
$$
Therefore, it is easy to prove that  $\theta_{GMM} - \theta_0 \stackrel{p}{\to} 0$ , and uses law of large numbers we could obtain,
$$
\sqrt {T} \frac {\partial Q _ {T} \left(\theta_ {0}\right)}{\partial \theta_ {m}} = 2 \left[ \underbrace {\frac {1}{M} \sum_ {i = 1} ^ {M} \frac {\partial g \left(x _ {i} , \theta_ {0}\right)}{\partial \theta_ {m}}} _ {\stackrel {{P}} {{\rightarrow}} \mathbb {E} \left(\frac {\partial g \left(X , \theta_ {0}\right)}{\partial \theta_ {m}}\right) = \Gamma_ {0, m}} \right] ^ {T} \underbrace {W _ {M}} _ {\stackrel {{P}} {{\rightarrow}} W} \left[ \underbrace {\frac {1}{\sqrt {T}} \sum_ {i = 1} ^ {M} g \left(x _ {i} , \theta_ {0}\right)} _ {\stackrel {{d}} {{\rightarrow}} \mathcal {N} \left(0, \mathbb {E} \left(g (X, \theta_ {0}) g (X, \theta_ {0}) ^ {T}\right)\right)} \right] \xrightarrow {d} 2 \Gamma_ {0, m} ^ {T} W \mathcal {N} \left(0, \Phi_ {0}\right), \tag {20}
$$
therefore, we have
$$
\begin{array}{l} \sqrt {T} \left(\theta_ {G M M} - \theta_ {0}\right) \approx - \left(\frac {\partial^ {2} Q _ {T} \left(\theta_ {0}\right)}{\partial \theta \partial \theta^ {T}}\right) ^ {- 1} \sqrt {T} \frac {\partial Q _ {T} \left(\theta_ {0}\right)}{\partial \theta} \tag {21} \\ \stackrel {d} {\rightarrow} \mathcal {N} (0, (\Gamma_ {0} ^ {T} W \Gamma_ {0}) ^ {- 1} \Gamma_ {0} ^ {T} W \Phi_ {0} W \Gamma_ {0} (\Gamma_ {0} ^ {T} W \Gamma_ {0}) ^ {- 1}). \\ \end{array}
$$
When the number of parameters  $d$  equals the number of moment conditions  $N$ ,  $\Gamma_0$  becomes a (nonsingular) square matrix, and therefore,
$$
\begin{array}{l} \sqrt {T} \left(\theta_ {G M M} - \theta_ {0}\right) \xrightarrow {d} \mathcal {N} \left(0, \left(\Gamma_ {0} ^ {T} W \Gamma_ {0}\right) ^ {- 1} \Gamma_ {0} ^ {T} W \Phi_ {0} W \Gamma_ {0} \left(\Gamma_ {0} ^ {T} W \Gamma_ {0}\right) ^ {- 1}\right) \tag {22} \\ = \mathcal {N} (0, \Gamma_ {0} ^ {- 1} W ^ {- 1} (\Gamma_ {0} ^ {T}) ^ {- 1} \Gamma_ {0} ^ {T} W \Phi_ {0} W \Gamma_ {0} \Gamma_ {0} ^ {- 1} W ^ {- 1} (\Gamma_ {0} ^ {T}) ^ {- 1}) \\ = \mathcal {N} (0, \Gamma_ {0} ^ {- 1} \Phi_ {0} (\Gamma_ {0} ^ {T}) ^ {- 1}), \\ \end{array}
$$
which means that the foregoing observation suggests that the selection of  $W_{N_c}$  has no bearing on the asymptotic variance of the GMM estimator. Consequently, it implies that regardless of the specific method employed to determine  $W_{N_c}$ , provided the moment estimates are asymptotically consistent,  $W_{N_c}$  serves as the optimal weight matrix, even when dealing with small samples.
The moments utilized for fitting via the generalized method of moments in each step are computed based on the noise network's first  $n$ -th order. To ensure adherence to the aforementioned proposition, it is necessary to assume that the  $n$ -th order of the noise network converges with probability to  $\mathbb{E}_{q(x_0|x_t)}[\mathrm{diag}(\epsilon \otimes^{n - 1}\epsilon)]$ , details in Appendix B. Consequently, the  $n$ -th order moments derived from the noise networks converge with probability to the true moments. Therefore, any choice of weight matrix is optimal.
A.3 Non-Gaussian distribution of transition kernel within large discretization steps
we can apply Bayes' rule to the posterior distribution  $q(x_{t}|x_{t + \Delta t})$  as follows:
$$
\begin{array}{l} q \left(x _ {t} \mid x _ {t + \Delta t}\right) = \frac {q \left(x _ {t + \Delta t} \mid x _ {t}\right) q \left(x _ {t}\right)}{q \left(x _ {t + \Delta t}\right)} = q \left(x _ {t + \Delta t} \mid x _ {t}\right) \exp \left(\log \left(q \left(x _ {t}\right)\right) - \log \left(q \left(x _ {t + \Delta t}\right)\right)\right) \tag {23} \\ \propto \exp \left(- \frac {\left\| x _ {t + \Delta t} - x _ {t} - f _ {t} (x _ {t}) \Delta t \right\| ^ {2}}{2 g _ {t} ^ {2} \Delta t} + \log p (x _ {t}) - \log (x _ {t + \Delta t})\right), \\ \end{array}
$$
where  $\Delta t$  is the step size,  $q(x_{t})$  is the marginal distribution of  $x_{t}$ . When  $x_{t + \Delta t}$  and  $x_{t}$  are close enough, using Taylor expansion for  $\log p(x_{t + \Delta t})$ , we could obtain:
$$
\log p \left(x _ {t + \Delta t}\right) \approx \log p \left(x _ {t}\right) + \left(x _ {t + \Delta t} - x _ {t}\right) \nabla_ {x _ {t}} \log p \left(x _ {t}\right) + \Delta t \frac {\partial}{\partial t} \log p \left(x _ {t}\right), \tag {24}
$$
$$
q \left(x _ {t} \mid x _ {t + \Delta t}\right) \propto \exp \left(- \frac {\left\| x _ {t + \Delta t} - x _ {t} - \left[ f _ {t} \left(x _ {t}\right) - g _ {t} ^ {2} \nabla_ {x _ {t}} \log p \left(x _ {t}\right) \right] \Delta t \right\| ^ {2}}{2 g _ {t} ^ {2} \Delta t} + O (\Delta t)\right). \tag {25}
$$
By ignoring the higher-order terms, the reverse transition kernel will be Gaussian distribution. However, as  $\Delta t$  increases, the higher-order terms in the Taylor expansion cannot be disregarded, which causes the reverse transition kernel to deviate from a Gaussian distribution.
Empirically, from Fig. 2 in our paper, we observe that as  $T$  decreases, the reverse transition kernel increasingly deviates from a Gaussian distribution. For more details, please refer to Appendix D.2.
B Calculation of the first order moment and higher order moments
Suppose the forward process is a Gaussian distribution same with the Eq. (1) as  $q(x_{t}|x_{s}) = N(x_{t}|a(t)x_{t - 1},\sigma (t)I)$ .
And let  $1 \geq t > s \geq 0$  always satisfy,  $q(x_{t}|x_{s}) = N(x_{t}|a_{t|s}x_{s},\beta_{t|s}I)$ , where  $a_{t|s} = a_t / a_s$  and  $\beta_{t|s} = \sigma_t^2 - a_{t|s}^2\sigma_s^2$ ,  $\sigma_s = \sqrt{1 - a(s)},\sigma_t = \sqrt{1 - a(t)}$ . It's easy to prove that the distribution  $q(x_{t}|x_{0})$ ,  $q(x_{s}|x_{t},x_{0})$  are also a Gaussian distribution [21]. Therefore, the mean of  $x_{s}$  under the measure  $q(x_{s}|x_{t})$  would be
$$
\begin{array}{l} \mathbb {E} _ {q \left(x _ {s} \mid x _ {t}\right)} \left[ x _ {s} \right] = \mathbb {E} _ {q \left(x _ {0} \mid x _ {t}\right)} E _ {q \left(x _ {s} \mid x _ {t}, x _ {0}\right)} \left[ x _ {s} \right] \tag {26} \\ = \mathbb {E} _ {q \left(x _ {0} \mid x _ {t}\right)} \left[ \frac {1}{a _ {t \mid s}} \left(x _ {t} - \frac {\beta_ {t \mid s}}{\sigma_ {t}} \epsilon_ {t}\right) \right] \\ = \frac {1}{a _ {t | s}} \left(x _ {t} - \frac {\beta_ {t | s}}{\sigma_ {t}} \mathbb {E} _ {q \left(x _ {0} \mid x _ {t}\right)} \left[ \epsilon_ {t} \right]\right). \\ \end{array}
$$
And for the second order central moment  $\mathrm{Cov}_{q(x_s|x_t)}[x_s]$ , we use the total variance theorem, refer to [2], and similar with [2], we only consider the diagonal covariance.
$$
\begin{array}{l} \operatorname {C o v} _ {q \left(x _ {s} \mid x _ {t}\right)} [ x _ {s} ] = \mathbb {E} _ {q \left(x _ {0} \mid x _ {t}\right)} \operatorname {C o v} _ {q \left(x _ {s} \mid x _ {t}, x _ {0}\right)} [ x _ {s} ] + \operatorname {C o v} _ {q \left(x _ {0} \mid x _ {t}\right)} \mathbb {E} _ {q \left(x _ {s} \mid x _ {t}, x _ {0}\right)} [ x _ {s} ] \tag {27} \\ = \lambda_ {t} ^ {2} I + \operatorname {C o v} _ {q (x _ {0} | x _ {t})} \tilde {\mu} \left(x _ {n}, \mathbb {E} _ {q (x _ {0} | x _ {n})} [ x _ {0} ]\right) \\ = \lambda_ {t} ^ {2} I + \frac {a _ {s} \beta_ {t | s} ^ {2}}{\sigma_ {t} ^ {4}} \operatorname {C o v} _ {q \left(x _ {0} \mid x _ {t}\right)} [ x _ {0} ] \\ = \lambda_ {t} ^ {2} I + \frac {a _ {s | 0} \beta_ {t | s} ^ {2}}{\sigma_ {t} ^ {4}} \frac {\sigma_ {t} ^ {2}}{a _ {t | 0}} \mathrm {C o v} _ {q (x _ {0} | x _ {t})} [ \epsilon_ {t} ] \\ = \lambda_ {t} ^ {2} I + \frac {\beta_ {t | s} ^ {2}}{\sigma_ {t} ^ {2} a _ {t | s}} \left(\mathbb {E} _ {q (x _ {0} | x _ {t})} [ \epsilon_ {t} \odot \epsilon_ {t} ] - \mathbb {E} _ {q (x _ {0} | x _ {t})} [ \epsilon_ {t} ] \odot \mathbb {E} _ {q (x _ {0} | x _ {t})} [ \epsilon_ {t} ]\right), \\ \end{array}
$$
since the higher-order moments are diagonal matrix, we use  $\mathrm{diag}(M)$  to represent the diagonal elements that have the same dimensions as the first-order moments, such as  $\mathrm{diag}(x_s \otimes x_s) = \operatorname{Cov}_{q(x_s|x_t)}[x_s]$  and  $\mathrm{diag}(x_s \otimes x_s \otimes x_s) = \hat{M}_3$  have the same dimensions as  $x_s$
Moreover, for the diagonal elements of the third-order moments, we have  $\mathbb{E}_{q(x_s|x_t)}[\mathrm{diag}(x_s\otimes x_s\otimes x_s)] = \mathbb{E}_{q(x_0|x_t)}\mathbb{E}_{q(x_s|x_t,x_0)}[x_s\odot x_s\odot x_s]$ , we could use the fact that  $\mathbb{E}_{q(x_s|x_t,x_0)}[(x_s - \mu (x_t,x_0))\odot (x_s - \mu (x_t,x_0))\odot (x_s - \mu (x_t,x_0))] = 0$ , therefore,
$$
\begin{array}{l} \hat {M} _ {3} = \mathbb {E} _ {q \left(x _ {s} \mid x _ {t}\right)} \left[ \operatorname {d i a g} \left(x _ {s} \otimes x _ {s} \otimes x _ {s}\right) \right] = \mathbb {E} _ {q \left(x _ {0} \mid x _ {t}\right)} \mathbb {E} _ {q \left(x _ {s} \mid x _ {t}, x _ {0}\right)} \left[ \operatorname {d i a g} \left(x _ {s} \otimes x _ {s} \otimes x _ {s}\right) \right] \tag {28} \\ = \mathbb {E} _ {q \left(x _ {0} \mid x _ {t}\right)} \mathbb {E} _ {q \left(x _ {s} \mid x _ {t}, x _ {0}\right)} [ 3 \operatorname {d i a g} \left(x _ {t} \otimes x _ {t} \otimes \mu \left(x _ {t}, x _ {0}\right)\right) \\ \left. \left. - 3 \operatorname {d i a g} \left(x _ {t} \otimes \mu \left(x _ {t}, x _ {0}\right) \otimes \mu \left(x _ {t}, x _ {0}\right)\right) + \operatorname {d i a g} \left(\mu \left(x _ {t}, x _ {0}\right) \otimes^ {2} \mu \left(x _ {t}, x _ {0}\right)\right) \right] \right. \\ = \underbrace {\left[ \left(\frac {a _ {t | s} \sigma_ {s} ^ {2}}{\sigma_ {t} ^ {2}}\right) ^ {3} \operatorname {d i a g} \left(x _ {t} \otimes x _ {t} \otimes x _ {t}\right) + 3 \lambda_ {t} ^ {2} \frac {a _ {t | s} \sigma_ {s} ^ {2}}{\sigma_ {t} ^ {2}} x _ {t} \right]} _ {\text {C o n s t a n t t e r m}} \\ + \underbrace {\left[ \frac {3 a _ {t | s} ^ {2} \sigma_ {s} ^ {4} a _ {s | 0} ^ {2} \beta_ {t | s} ^ {2}}{\sigma_ {t} ^ {8}} (\operatorname {d i a g} (x _ {t} \otimes x _ {t})) + \frac {a _ {s | 0} \beta_ {t | s}}{\sigma_ {t} ^ {2}} I \right] \odot \mathbb {E} _ {q (x _ {0} | x _ {t})} [ x _ {0} ]} _ {\text {L i n e a r t e r m i n} x _ {0}} \\ + \underbrace {3 \frac {a _ {t | s} \sigma_ {s} ^ {2}}{\sigma_ {t} ^ {2}} (\frac {a _ {s | 0} \beta_ {t | s}}{\sigma_ {t} ^ {2}}) ^ {2} x _ {t} \odot \mathbb {E} _ {q (x _ {0} | x _ {t})} [ \mathrm {d i a g} (x _ {0} \otimes x _ {0}) ]} _ {\text {Q u a d r a t i c t e r m i n} x _ {0}} + \underbrace {(\frac {a _ {s | 0} \beta_ {t | s}}{\sigma_ {t} ^ {2}}) ^ {3} \mathbb {E} _ {q (x _ {0} | x _ {t})} [ \mathrm {d i a g} (x _ {0} \otimes x _ {0} \otimes x _ {0}) ]} _ {\text {C u b i c t e r m i n} x _ {0}}, \\ \end{array}
$$
where the third equation is derived from the decomposition of higher-order moments of Gaussian distribution, the fourth equation is obtained by splitting It is noted to highlight that in our study, we only consider the diagonal higher-order moments in our method for computational efficiency since estimating full higher-order moments results in escalated output dimensions (e.g., quadratic growth for covariance and cubic for the third-order moments) and thus requires substantial computational demand and therefore all outer products can be transformed into corresponding element multiplications and we have:
$$
\begin{array}{l} \mathbb {E} _ {q \left(x _ {0} \mid x _ {t}\right)} [ \operatorname {d i a g} \left(x _ {0} \otimes^ {2} x _ {0}\right) ] = \frac {1}{\alpha^ {\frac {3}{2}} (t)} \mathbb {E} _ {q \left(x _ {0} \mid x _ {t}\right)} [ \operatorname {d i a g} \left(\left(x _ {t} - \sigma (t) \epsilon\right) \otimes^ {2} \left(x _ {t} - \sigma (t) \epsilon\right)\right) ] \tag {29} \\ = \frac {1}{\alpha^ {\frac {3}{2}} (t)} \mathbb {E} _ {q \left(x _ {0} \mid x _ {t}\right)} [ \operatorname {d i a g} \left(x _ {t} \otimes^ {2} x _ {t} - 3 \sigma (t) \left(x _ {t} \otimes x _ {t}\right) \otimes e \right. \\ \left. + 3 \sigma^ {2} (t) x _ {t} \otimes (\epsilon \otimes \epsilon) - \sigma^ {3} (t) (\epsilon \otimes^ {2} \epsilon)) \right] \\ = \frac {1}{\alpha^ {\frac {3}{2}} (t)} \left[ \mathbb {E} _ {q \left(x _ {0} \mid x _ {t}\right)} \left[ x _ {t} \odot^ {2} x _ {t} \right] - 3 \sigma (t) \left(x _ {t} \odot x _ {t}\right) \odot \mathbb {E} _ {q \left(x _ {0} \mid x _ {t}\right)} [ \epsilon ] + \right. \\ + 3 \sigma^ {2} (t) x _ {t} \mathbb {E} _ {q \left(x _ {0} \mid x _ {t}\right)} [ \epsilon \odot \epsilon ] - \sigma^ {3} (t) [ \epsilon \odot^ {2} \epsilon ] ], \\ \end{array}
$$
Therefore, when we need to calculate the third-order moment, we only need to obtain  $\mathbb{E}_{q(x_0|x_t)}[\epsilon_t],\mathbb{E}_{q(x_0|x_t)}[\epsilon_t\odot \epsilon_t]$  and  $\mathbb{E}_{q(x_0|x_t)}[\epsilon_t\odot \epsilon_t\odot \epsilon_t]$ . Similarly, when we need to calculate
the  $n$ -order moment, we will use  $\mathbb{E}_{q(x_0|x_t)}[\epsilon_t], \ldots, \mathbb{E}_{q(x_0|x_t)}[\epsilon_t \odot^{n-1}\epsilon_t]$ . Bao et al. [2] put forward using a sharing network and using the MSE loss to estimate the network to obtain the above information about different orders of noise.
The function  $h(f_{[1]}^3, f_{[2]}^3, f_{[3]}^3)$  in Algo. 2 is defined as  $h(f_{[1]}^3(x_t, t), f_{[2]}^3(x_t, t), f_{[3]}^3(x_t, t)) = M_1(f_{[1]}^3(x_t, t)), M_2(f_{[2]}^3(x_t, t)), M_3(f_{[3]}^3(x_t, t))$ , where  $M_1(f_{[1]}^3(x_t, t)) = \mathbb{E}_{q(x_s|x_t)}[x_s]$  in Eq. (26),  $M_2(f_{[2]}^3(x_t, t)) = \operatorname{Cov}_{q(x_s|x_t)}[x_s]$  in Eq. (27) and  $M_3(f_{[3]}^3(x_t, t)) = \hat{M}_3 = \mathbb{E}_{q(x_s|x_t)}[\operatorname{diag}(x_s \otimes x_s \otimes x_s)]$  in Eq. (28)
B.1 Objective function for Gaussian mixture transition kernel with two components
Recall that the general objective function to fit a Gaussian mixture transition kernel in each sampling step via GMM is shown in Eq. (30).
$$
\min  _ {\theta} Q \left(\theta , M _ {1}, \dots , M _ {N}\right) = \min  _ {\theta} \left[ \frac {1}{N _ {c}} \sum_ {i = 1} ^ {N _ {c}} g \left(x _ {i}, \theta\right) \right] ^ {T} W \left[ \frac {1}{N _ {c}} \sum_ {i = 1} ^ {N _ {c}} g \left(x _ {i}, \theta\right) \right], \tag {30}
$$
where  $A^T$  denotes the transpose matrix of matrix A. In our paper, we propose to use the first three moments to fit a Gaussian mixture transition kernel  $p(x_s|x_t) = \frac{1}{3}\mathcal{N}(\mu_t^{(1)},\sigma_t^2) + \frac{2}{3}\mathcal{N}(\mu_t^{(2)},\sigma_t^2)$  in each sampling step from  $x_{t}$  to  $x_{s}$ . Therefore, the final objective function as follows:
$$
\min  _ {\mu_ {t} ^ {(1)}, \mu_ {t} ^ {(2)}, \sigma_ {t} ^ {2}} [ \binom {M _ {1} - (\frac {1}{3} \mu_ {t} ^ {(1)} + \frac {2}{3} \mu_ {t} ^ {(2)})} {M _ {2} - (\frac {1}{3} [ (\mu_ {t} ^ {(1)}) ^ {2} + \sigma_ {t} ^ {2} ] + \frac {2}{3} [ (\mu_ {t} ^ {(2)}) ^ {2} + \sigma_ {t} ^ {2}) ])} ] ^ {T} I \tag {31}
$$
$$
\left. \begin{array}{c} M _ {1} - (\frac {1}{3} \mu_ {t} ^ {(1)} + \frac {2}{3} \mu_ {t} ^ {(2)}) \\ M _ {2} - (\frac {1}{3} [ (\mu_ {t} ^ {(1)}) ^ {2} + \sigma_ {t} ^ {2} ] + \frac {2}{3} [ (\mu_ {t} ^ {(2)}) ^ {2} + \sigma_ {t} ^ {2}) ]) \\ M _ {3} - (\frac {1}{3} [ (\mu_ {t} ^ {(1)}) ^ {3} + 3 \mu_ {t} ^ {(1)} \sigma_ {t} ^ {2} ] + \frac {2}{3} [ (\mu_ {t} ^ {(2)}) ^ {3} + 3 \mu_ {t} ^ {(2)} \sigma_ {t} ^ {2} ]), \end{array} \right) \end{array}
$$
where to simplify the analysis, we use the scalar form of parameters  $\mu_t^{(1)},\mu_t^{(2)},\sigma_t^2$  as representation.
C Modeling reverse transition kernel via exponential family
Analysis in Sec. 3.1 figures out that modeling reverse transition kernel via Gaussian distribution is no longer sufficient in fast sampling scenarios. In addition to directly proposing the use of a Gaussian Mixture for modeling, we also analyze in principle whether there are potentially more suitable distributions i.e., the feasibility of using them for modeling.
We would turn back to analyzing the original objective function of DPMs to find a suitable distribution. The forward process  $q(x_{t}|x_{s}) = N(x_{t}|a_{t|s}x_{s},\beta_{t|s}I)$ , consistent with the definition in Appendix B. DPMs' goal is to optimize the modeled reverse process parameters to maximize the variational bound  $L$  in Ho et al. [14]. And the ELBO in Ho et al. [14] can be re-written to the following formula:
$$
L = D _ {\mathrm {K L}} \left(q \left(x _ {T}\right) \mid \mid p \left(x _ {T}\right)\right) + \mathbb {E} _ {q} \left[ \sum_ {t \geq 1} D _ {\mathrm {K L}} \left(q \left(x _ {s} \mid x _ {t}\right) \mid \mid p \left(x _ {s} \mid x _ {t}\right)\right) \right] + H \left(x _ {0}\right), \tag {32}
$$
where  $q_{t} \doteq q(x_{t})$  is the true distribution and  $p_{t} \doteq p(x_{t})$  is the modeled distribution, and the minimum problem could be transformed into a sub-problem, proved in Bao et al. [3]:
$$
\min  _ {\{\theta \}} L \Leftrightarrow \min   ^ {T}} D _ {\mathrm {K L}} \left(q \left(x _ {s} \mid x _ {t}\right) \mid \mid p _ {\theta_ {s \mid t}} \left(x _ {s} \mid x _ {t}\right)\right). \tag {33}
$$
We have no additional information besides when the reverse transition kernel is not Gaussian. But Lemma C.3 proves that when the reverse transition kernel  $p_{\theta_{s|t}}(x_s|x_t)$  is exponential family  $p_{\theta_t}(x_s|x_t) = p(x_t,\theta_{s|t}) = h(x_t)\exp \Big(\theta_{s|t}^T t(x_t) - \alpha (\theta_{s|t})\Big)$ , solving the sub-problem Eq. (33) equals to solve the following equations, which is to match moments between the modeled distribution and true distribution:
$$
\mathbb {E} _ {q \left(x _ {s} \mid x _ {t}\right)} [ t (x _ {s}) ] = \mathbb {E} _ {p \left(x _ {t}, \theta_ {s \mid t}\right)} [ t (x _ {s}) ]. \tag {34}
$$
When  $t(x) = (x,..,x^n)^T$ , solving Eq.(34) equals to match the moments of true distribution and modeled distribution.
Meanwhile, Gaussian distribution belongs to the exponential family with  $t(x) = (x,x^2)^T$  and  $\theta_t = (\frac{\mu_t}{\sigma_t^2},\frac{-1}{2\sigma_t^2})^T$ , details in Lemma C.2. Therefore, when modeling the reverse transition kernel as Gaussian distribution, the optimal parameters are that make its first two moments equal to the true first two moments of the real reverse transition kernel  $q(x_{s}|x_{t})$ , which is consistent with the results in Bao et al. [3] and Bao et al. [2].
The aforementioned discussion serves as a motivation to acquire higher-order moments and identify a corresponding exponential family, which surpasses the Gaussian distribution in terms of complexity. However, proposition C.1 shows that finding such exponential family distribution with higher-order moments is impossible.
Proposition C.1 (Infeasibility of exponential family with higher-order moments.). Given the first  $n$ -th order moments, It's non-trivial to find an exponential family distribution for  $\min D_{\mathrm{KL}}(q||p)$  when  $n$  is odd. And it's hard to solve  $\min D_{\mathrm{KL}}(q||p)$  when  $n$  is even.
C.1 Proof of Proposition C.1
Lemma C.2. (Gaussian Distribution belongs to Exponential Family). Gaussian distribution  $p(x) = \frac{1}{\sqrt{2\pi}\sigma} \exp \left(-\frac{(x - \mu)^2}{2\sigma^2}\right)$  is exponential family with  $t(x) = (x,x^2)^T$  and  $\theta = (\frac{\mu}{\sigma^2}, - \frac{1}{2\sigma^2})^T$
Proof. For simplicity, we only prove one-dimensional Gaussian distribution. We could obtain:
$$
\begin{array}{l} p (x) = \frac {1}{\sqrt {2 \pi} \sigma} \exp \left(- \frac {(x - \mu) ^ {2}}{2 \sigma^ {2}}\right) \tag {35} \\ = \frac {1}{\sqrt {2 \pi \sigma^ {2}}} \exp \left(- \frac {1}{2 \sigma^ {2}} \left(x ^ {2} - 2 \mu x + \mu^ {2}\right)\right) \\ = \exp \left(\log \left(2 \pi \sigma^ {2}\right) ^ {- 1 / 2}\right) \exp \left(- \frac {1}{2 \sigma^ {2}} \left(x ^ {2} - 2 \mu x\right) - \frac {\mu^ {2}}{\sigma^ {2}}\right) \\ = \exp \left(\log \left(2 \pi \sigma^ {2}\right) ^ {- 1 / 2}\right) \exp \left(- \frac {1}{2 \sigma^ {2}} (- 2 \mu \quad 1) (x \quad x ^ {2}) ^ {T} - \frac {\mu^ {2}}{\sigma^ {2}}\right) \\ = \exp \bigg ((\frac {\mu}{\sigma^ {2}} - \frac {- 1}{2 \sigma^ {2}}) (x x ^ {2}) ^ {T} - (\frac {\mu^ {2}}{2 \sigma^ {2}} + \frac {1}{2} \log (2 \pi \sigma^ {2})) \bigg), \\ \end{array}
$$
where  $\theta = \left(\frac{\mu}{\sigma^2},\frac{-1}{2\sigma^2}\right)^T$  and  $t(x) = (x,x^{2})^{T}$
Lemma C.3. (The Solution for Exponential Family in Minimizing the KL Divergence). Suppose that  $p(x)$  belongs to exponential family  $p(x, \theta) = h(x) \exp \big(\theta^T t(x) - \alpha(\theta)\big)$ , and the solution for minimizing the  $E_q[\log p]$  is  $E_q[t(x)] = E_{p(x, \theta)}[t(x)]$ .
Proof. An exponential family  $p(x, \eta) = h(x) \exp \bigl( \eta^T t(x) - \alpha(\eta) \bigr) \propto f(x, \eta) = h(x) \exp \bigl( \eta^T t(x) \bigr)$  with log-partition  $\alpha(\eta)$ . And we could obtain its first order condition on  $E_q[\log p]$  as:
$$
\nabla_ {\eta} \log f (x, \eta) = \nabla_ {\eta} \left(\log h (x) + \eta^ {T} t (x)\right) = t (x) \tag {36}
$$
$$
\begin{array}{l} \nabla_ {\eta} \alpha (\eta) = \nabla_ {\eta} \log \left(\int f (x, \eta) d x\right) = \frac {\int \nabla_ {\eta} f (x , \eta) d x}{\int f (x , \eta) d x} \tag {37} \\ = e ^ {- \alpha (\eta)} \int t (x) f (x, \eta) d x = \int t (x) p (x, \eta) d x = \mathbb {E} _ {p (x, \eta)} [ t (x) ] \\ \end{array}
$$
In order to minimize the  $D_{\mathrm{KL}}(q||p) = \int q\log (q / p) = -\mathbb{E}_q[\log p]$ , we have:
$$
\begin{array}{l} \mathbb {E} _ {q} [ \log p ] = \int d q \log (h (x)) + \int d q (\eta^ {T} t (x) - \alpha (\eta) \\ \Longrightarrow \frac {\partial}{\partial \eta} \mathbb {E} _ {q} [ \log p ] = \int d q [ \frac {\partial}{\partial \eta} (\eta^ {T} x - \alpha (\eta)) ] = 0 \\ \Longrightarrow \int d q (x - \mathbb {E} _ {p (x, \eta)} [ t (x) ]) = \mathbb {E} _ {q} [ t (x) ] - \mathbb {E} _ {p (x, \eta)} [ t (x) ] = 0 \\ \Longrightarrow \mathbb {E} _ {q} [ t (x) ] = \mathbb {E} _ {p (x, \eta)} [ t (x) ] \\ \end{array}
$$
For the second-order condition, we have the following:
$$
\begin{array}{l} \frac {\partial^ {2}}{\partial \eta^ {2}} \alpha (\eta) = \frac {\partial}{\partial \eta} \int d p (x, \eta) t (x) \tag {38} \\ = \int \frac {\partial}{\partial \eta} h (x) \exp \left(\eta^ {T} t (x) - \alpha (\eta)\right) t (x) d x \\ = \int h (x) t (x) \frac {\partial}{\partial \eta} \exp \left(\eta^ {T} t (x) - \alpha (\eta)\right) d x \\ = \int h (x) t (x) \exp \left(\eta^ {T} t (x) - \alpha (\eta)\right) d x (t (x) - \mathbb {E} _ {p} [ t (x) ]) \\ = \int p (x, \eta) d x \left(t ^ {2} (x) - t (x) \mathbb {E} _ {p} [ t (x) ]\right) \\ = \mathbb {E} _ {p (x, \eta)} [ t ^ {2} (x) ] - \mathbb {E} _ {p (x, \eta)} [ t (x) ] ^ {2} = C o v _ {p (x, \eta)} [ t (x) ] \geq 0 \\ \end{array}
$$
Therefore, the second-order condition for the cross entropy would be:
$$
\begin{array}{l} \frac {\partial^ {2}}{\partial \eta^ {2}} \mathbb {E} _ {q} [ \log p ] = \frac {\partial}{\partial \eta} \left(\mathbb {E} _ {q} [ t (x) ] - \mathbb {E} _ {p (x, \eta)} [ t (x) ]\right) \tag {39} \\ = - \int \frac {\partial}{\partial \eta} p (x, \eta) t (x) d x \\ = - \frac {\partial^ {2}}{\partial \eta^ {2}} \alpha (\eta) = - C o v _ {p (x, \eta)} [ t (x) ] \leq 0 \\ \end{array}
$$
When we assume that the reverse process is Gaussian, the solution to Eq. (33) equals to match the moment of true distribution and modeled distribution  $\mu = E_q[x], \Sigma = Cov_q[x]$ .
Lemma C.4. (Infeasibility of the exponential family with higher-order moments). Suppose given the first  $N$ -th order moments  $M_{i}$ ,  $i = 1, \dots, N$  and modeled  $p$  as an exponential family. It is nontrivial to solve the minimum problem  $E_{q}[\log p]$  when  $N$  is odd and it's difficult to solve when  $N$  is even.
Proof. While given the mean, covariance, and skewness of the data distribution, assume that we could find an exponential family that minimizes the KL divergence, so that the distribution would satisfy the following form: