SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hanzhong Guo\*1,3, Cheng Lu4, Fan Bao4, Tianyu Pang2, Shuicheng Yan2, Chao Du\*2, Chongxuan Li\*1,3
$^{1}$ Gaoling School of Artificial Intelligence, Renmin University of China  $^{2}$ Sea AI Lab, Singapore
$^{3}$ Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China  $^{4}$ Tsinghua University
{allanguo, tianyupang, yansc, duchao}@sea.com;
lucheng.1c15@gmail.com; bf19@mails.tsinghua.edu.cn;
chongxuanli@ruc.edu.cn
Recently, diffusion models have achieved great success in generative tasks. Sampling from diffusion models is equivalent to solving the reverse diffusion stochastic differential equations (SDEs) or the corresponding probability flow ordinary differential equations (ODEs). In comparison, SDE-based solvers can generate samples of higher quality and are suited for image translation tasks like stroke-based synthesis. During inference, however, existing SDE-based solvers are severely constrained by the efficiency-effectiveness dilemma. Our investigation suggests that this is because the Gaussian assumption in the reverse transition kernel is frequently violated (even in the case of simple mixture data) given a limited number of discretization steps. To overcome this limitation, we introduce a novel class of SDE-based solvers called Gaussian Mixture Solvers (GMS) for diffusion models. Our solver estimates the first three-order moments and optimizes the parameters of a Gaussian mixture transition kernel using generalized methods of moments in each step during sampling. Empirically, our solver outperforms numerous SDE-based solvers in terms of sample quality in image generation and stroke-based synthesis in various diffusion models, which validates the motivation and effectiveness of GMS. Our code is available at https://github.com/Guohanzhong/GMS.
In recent years, deep generative models and especially (score-based) diffusion models [35, 14, 37, 18] have made remarkable progress in various domains, including image generation [15, 7], audio generation [22, 32], video generation [16], 3D object generation [31], multi-modal generation [42, 5, 12, 43], and several downstream tasks such as image translation [28, 45] and image restoration [19].
Sampling from diffusion models can be interpreted as solving the reverse-time diffusion stochastic differential equations (SDEs) or their corresponding probability flow ordinary differential equations (ODEs) [37]. SDE-based and ODE-based solvers of diffusion models have very different properties and application scenarios. In particular, SDE-based solvers usually perform better when given a sufficient number of discretization steps [18]. Indeed, a recent empirical study [24] suggests that SDE-based solvers can potentially generate high-fidelity samples with realistic details and intricate semantic coherence from pre-trained large-scale text-to-image diffusion models [34]. Besides, SDE-based solvers are preferable in many downstream tasks such as stroke-based synthesis [27], image translation [45], and image manipulation [20].
*Work done during an internship at Sea AI Lab. †Correspondence to Chao Du and Chongxuan Li.
(a) #Step=5
(b) #Step=10
Figure 1: Sampling on a mixture of Gaussian. GMS (ours) and SN-DDPM excel in fitting the true distribution when the transition kernel is Gaussian, with a sufficient number of sampling steps (c). However, GMS outperforms SN-DDPM [2] when sampling steps are limited and the reverse transition kernel deviates from Gaussian (a-b).
(c) #Step=20
Despite their wide applications, SDE-based solvers face a significant trade-off between efficiency and effectiveness during sampling, since an insufficient number of steps lead to larger discretization errors. To this end, Bao et al. [3] estimate the optimal variance of the Gaussian transition kernel in the reverse process instead of using a handcraft variance to reduce the discretization errors. Additionally, Bao et al. [2] explore the optimal diagonal variance when dealing with an imperfect noise network and demonstrate state-of-the-art (SOTA) performance with a few steps among other SDE-based solvers. Notably, these solvers all assume that the transition kernel in the reverse process is Gaussian.
In this paper, we systematically examine the assumption of the Gaussian transition kernel and reveal that it can be easily violated under a limited number of discretization steps even in the case of simple mixture data. To this end, we propose a new type of SDE-based solver called Gaussian Mixture Solver (GMS), which employs a more expressive Gaussian mixture transition kernel in the reverse process for better approximation given a limited number of steps (see visualization results on toy data in Fig. 1). In particular, we first learn a noise prediction network with multiple heads that estimate the high-order moments of the true reverse transition kernel respectively. For sampling, we fit a Gaussian mixture transition kernel in each step via the generalized methods of the moments [11] using the predicted high-order moments of the true reverse process.
To evaluate GMS, we compare it against a variety of baselines [14, 2, 17, 36] in terms of several widely used metrics (particularly sample quality measured by FID) in the tasks of generation and stroke-based synthesis on multiple datasets. Our results show that GMS outperforms state-of-the-art SDE-based solvers [14, 2, 17] in terms of sample quality with the limited number of discretization steps (e.g.,  $< 100$ ). For instance, GMS improves the FID by 4.44 over the SOTA SDE-based solver [2] given 10 steps on CIFAR10. Furthermore, We evaluate GMS on a stroke-based synthesis task. The findings consistently reveal that GMS achieves higher levels of realism than all aforementioned SDE-based solvers as well as the widely adopted ODE-based solver DDIM [36] while maintaining comparable computation budgets and faithfulness scores (measured by  $L_{2}$  distance). Such empirical findings validate the motivation and effectiveness of GMS.
In this section, we provide a brief overview of the (score-based) diffusion models, representative SDE-based solvers for diffusion models, and applications of such solvers.
Diffusion models gradually perturb data with a forward diffusion process and then learn to reverse such process to recover the data distribution. Formally, let  $x_0 \in \mathbb{R}^n$  be a random variable with unknown data distribution  $q(x_0)$ . Diffusion models define the forward process  $\{x_t\}_{t \in [0,1]}$  indexed by time  $t$ , which perturbs the data by adding Gaussian noise to  $x_0$  with
$$
q \left(x _ {t} \mid x _ {0}\right) = \mathcal {N} \left(x _ {t} \mid a (t) x _ {0}, \sigma^ {2} (t) I\right). \tag {1}
$$
In general, the function  $a(t)$  and  $\sigma(t)$  are selected so that the logarithmic signal-to-noise ratio  $\log \frac{a^2(t)}{\sigma^2(t)}$  decreases monotonically with time  $t$ , causing the data to diffuse towards random Gaussian noise [21]. Furthermore, it has been demonstrated by Kingma et al. [21] that the following SDE shares an identical transition distribution  $q_{t|0}(x_t|x_0)$  with Eq. (1):
$$
\mathrm {d} x _ {t} = f (t) x _ {t} \mathrm {d} t + g (t) \mathrm {d} \omega , \quad x _ {0} \sim q \left(x _ {0}\right), \tag {2}
$$
where  $\omega \in \mathbb{R}^n$  is a standard Wiener process and
$$
f (t) = \frac {\mathrm {d} \log a (t)}{\mathrm {d} t}, \quad g ^ {2} (t) = \frac {\mathrm {d} \sigma^ {2} (t)}{\mathrm {d} t} - 2 \sigma^ {2} (t) \frac {\mathrm {d} \log a (t)}{\mathrm {d} t}. \tag {3}
$$
Let  $q(x_{t})$  be the marginal distribution of the above SDE at time  $t$ . Its reversal process can be described by a corresponding continuous SDE which recovers the data distribution [37]:
$$
\mathrm {d} x = \left[ f (t) x _ {t} - g ^ {2} (t) \nabla_ {x _ {t}} \log q (x _ {t}) \right] \mathrm {d} t + g (t) \mathrm {d} \bar {\omega}, \quad x _ {1} \sim q (x _ {1}), \tag {4}
$$
where  $\bar{\omega} \in \mathbb{R}^n$  is a reverse-time standard Wiener process. The only unknown term in Eq. (4) is the score function  $\nabla_{x_t}\log q(x_t)$ . To estimate it, existing works [14, 37, 18] train a noise network  $\epsilon_{\theta}(x_t,t)$  to obtain a scaled score function  $\sigma(t)\nabla_{x_t}\log q(x_t)$  using denoising score matching [38]:
$$
\mathcal {L} (\theta) = \int_ {0} ^ {1} w (t) \mathbb {E} _ {q \left(x _ {0}\right)} \mathbb {E} _ {q (\epsilon)} [ \| \epsilon_ {\theta} \left(x _ {t}, t\right) - \epsilon \| _ {2} ^ {2} ] \mathrm {d} t, \tag {5}
$$
where  $w(t)$  is a weighting function,  $q(\epsilon)$  is standard Gaussian distribution and  $x_{t} \sim q(x_{t}|x_{0})$  follows Eq. (1). The optimal solution of the optimization objective Eq. (5) is  $\epsilon_{\theta}(x_t,t) = -\sigma (t)\nabla_{x_t}\log q(x_t)$
Hence, samples can be obtained by initiating the process with a standard Gaussian variable  $x_{1}$ , then substituting  $\nabla_{x_t}\log q(x_t)$  with  $-\frac{\epsilon_{\theta}(x_t,t)}{\sigma(t)}$  and discretizing reverse SDE Eq. (4) from  $t = 1$  to  $t = 0$  to generate  $x_0$ .
2.2 SDE-based solvers for diffusion models
The primary objective of SDE-based solvers lies in decreasing discretization error and therefore minimizing function evaluations required for convergence during the process of discretizing Eq. (4). Discretizing the reverse SDE in Eq. (4) is equivalent to sample from a Markov chain  $p(x_{0:1}) = p(x_1)\prod_{t_{i - 1},t_i\in S_t}p(x_{t_{i - 1}}|x_{t_i})$  with its trajectory  $S_{t} = [0,t_{1},t_{2},\dots,t_{i},\dots,1]$ . Song et al. [37] proves that the conventional ancestral sampling technique used in the DPMs [14] that models  $p(x_{t_{i - 1}}|x_{t_i})$  as a Gaussian distribution, can be perceived as a first-order solver for the reverse SDE in Eq. (4). Bao et al. [3] finds that the optimal variance of  $p(x_{t_{i - 1}}|x_{t_i})\sim \mathcal{N}(x_{t_{i - 1}}|\mu_{t_{i - 1}|t_i},\Sigma_{t_{i - 1}|t_i}(x_{t_i}))$  is
$$
\Sigma_ {t _ {i - 1} | t _ {i}} ^ {*} \left(x _ {t _ {i}}\right) = \lambda_ {t _ {i}} ^ {2} + \gamma_ {t _ {i}} ^ {2} \frac {\sigma^ {2} \left(t _ {i}\right)}{\alpha \left(t _ {i}\right)} \left(1 - \mathbb {E} _ {q \left(x _ {t _ {i}}\right)} \left[ \frac {1}{d} \left\| \mathbb {E} _ {q \left(x _ {0} | x _ {t _ {i}}\right)} \left[ \epsilon_ {\theta} \left(x _ {t _ {i}}, t _ {i}\right) \right] \right\| _ {2} ^ {2} \right]\right), \tag {6}
$$
where,  $\gamma_{t_i} = \sqrt{\alpha(t_{i - 1})} -\sqrt{\sigma^2(t_{i - 1}) - \lambda_{t_i}^2}\sqrt{\frac{\alpha(t_i)}{\sigma^2(t_i)}}$ $\lambda_{t_i}^2$  is the variance of  $q(x_{t_{i - 1}}|x_{t_i},x_0)$ . AnalyticDPM [3] offers a significant reduction in discretization error during sampling and achieves faster convergence with fewer steps. Moreover, SN-DDPM [2] employs a Gaussian transition kernel with an optimal diagonal covariance instead of an isotropic covariance. This approach yields improved sample quality and likelihood compared to other SDE-based solvers within a limited number of steps.
2.3 Applications of SDE-based solvers for stroke-based synthesis
The stroke-based image synthesis is a representative downstream task suitable for SDE-based solvers. It involves the user providing a full-resolution image  $x^{(g)}$  through the manipulation of RGB pixels, referred to as the guided image. The guided image  $x^{(g)}$  possibly contains three levels of guidance: a high-level guide consisting of coarse colored strokes, a mid-level guide comprising colored strokes on a real image, and a low-level guide containing image patches from a target image.
SDEdit [27] solves the task by first starting from the guided image  $x^{(g)}$  and adding Gaussian noise to disturb the guided images to  $x_{t}$  and  $q(x^{(g)}(t_0)|x^{(g)})\sim \mathcal{N}(x_{t_0}|a(t_0)x^{(g)},\sigma^2 (t)I)$  same with Eq. (1). Subsequently, it solves the corresponding reverse stochastic differential equation (SDE) up to  $t = 0$  to generate the synthesized image  $x(0)$  discretizing Eq. (4). Apart from the discretization steps taken by the SDE-based solver, the key hyper-parameter for SDEdit is  $t_0$ , the time step from which we begin the image synthesis procedure in the reverse SDE.
(a) logarithm mean of  $L_{2}$ -norm
(b) logarithm median of  $L_{2}$ -norm
Figure 2: Empirical evidence of suboptimality of Gaussian kernel on CIFAR10. (a) and (b) plot the logarithm of the image-wise mean and median of  $L_{2}$ -norm between Gaussian-assumed third-order moments and estimated third-order moments. Clearly, as the number of sampling steps decreases, the disparity between the following two third-order moments increases, denoting that the true transition kernel diverges further from the Gaussian distribution. See Appendix D.2 for more details.
3 Gaussian mixture solvers for diffusion models
In this section, we first show through both theoretical and empirical evidence, that the true reverse transition kernel can significantly diverge from Gaussian distributions as assumed in previous SOTA SDE-based solvers [3, 2], indicating that the reverse transition can be improved further by employing more flexible distributions (see Sec. 3.1). This motivates us to propose a novel class of SDE-based solvers, dubbed Gaussian Mixture Solvers, which determines a Gaussian mixture distribution via the generalized methods of the moments [11] using higher-order moments information for the true reverse transition (see Sec. 3.2). The higher-order moments are estimated by a noise prediction network with multiple heads on training data, as detailed in Sec. 3.3.
3.1 Suboptimality of Gaussian distributions for reverse transition kernels
As described in Sec. 2, existing state-of-the-art SDE-based solvers for DPMs [3, 2] approximate the reverse transition  $q(x_{t_{i-1}} | x_{t_i})$  using Gaussian distributions. Such approximations work well when the number of discretization steps in these solvers is large (e.g., 1000 steps). However, for smaller discretization steps (such as when faster sampling is required), the validity of this Gaussian assumption will be largely broken. We demonstrate this theoretically and empirically below.
First, observe that  $q(x_{s}|x_{t})^{1}$  can be expressed as  $q(x_{s}|x_{t}) = \int q(x_{s}|x_{t},x_{0})q(x_{0}|x_{t})\mathrm{d}x_{0}$ , which is non-Gaussian for a general data distribution  $q(x_0)$ . For instance, for a mixture of Gaussian or a mixture of Dirac  $q(x_0)$ , we can prove that the conditional distributions  $q(x_{s}|x_{t})$  in the reverse process are non-Gaussian, as characterized by Proposition 3.1, proven in Appendix A.1.
Proposition 3.1 (Mixture Data Have non-Gaussian Reverse Kernel). Assume  $q(x_0)$  is a mixture of Dirac or a mixture of Gaussian distribution and the forward process is defined in Eq. (1). The reverse transition kernel  $q(x_s|x_t)$ ,  $s < t$  is a Gaussian mixture instead of a Gaussian.
Empirically, we do not know the distributions of the real data (e.g., high-dimensional images) and cannot obtain an explicit form of  $q(x_{s}|x_{t})$ . However, even in such cases, it is easy to validate that  $q(x_{s}|x_{t})$  are non-Gaussian. In particular, note that the third-order moment of one Gaussian distribution  $(M_3^{(G)})$  can be represented by its first-order moment  $(M_1)$  and its second-order moment  $(M_2)^2$ , which motivates us to check whether the first three order moments of  $q(x_{s}|x_{t})$  satisfy the relationship induced by the Gaussian assumption. We perform an experiment on CIFAR-10 and estimate the first three orders of moments  $\hat{M}_1, \hat{M}_2, \hat{M}_3$  for  $q(x_{s}|x_{t})$  from data by high-order noise networks (see details in Sec. D.2). As shown in Fig. 2, we plot the mean and median of  $l_2$ -norm
To enhance the clarity of exposition, we introduce the notation  $s \doteq t_{i-1}$  and  $t \doteq t_i$  to denote two adjacent time steps in the trajectory. Consequently, we refer to  $q(x_s|x_t)$  as the reverse transition kernel in this context.
2The scalar case is  $M_3^{(G)} = M_1^3 + 3M_1(M_2 - M_1^2)$ . See Appendix D.2 for the vector case.
between the estimated third-order moment  $\hat{M}_3$  and third-order moment  $M_3^{(G)}$  calculated under the Gaussian assumption given the different number of steps at different time steps  $t$ . In particular, when the number of steps is large (i.e.,  $\# \mathrm{Step} = 1000$ ), the difference between the two calculation methods is small. As the number of steps decreases, the  $l_{2}$ -norm increases, indicating that the true reverse distribution  $q(x_s|x_t)$  is non-Gaussian. With the time step closer to  $t = 0$ , the  $l_{2}$ -norm increases too.
Both the theoretical and empirical results motivate us to weaken the Gaussian assumption in SDE-based solvers for better performance, especially when the step size is large.
3.2 Sampling with Gaussian mixture transition kernel
There are extensive choices of  $p(x_s|x_t)$  such that it is more powerful and potentially fits  $q(x_s|x_t)$  better than a Gaussian. In this paper, we choose a simple mixture of the Gaussian model as follows:
$$
p \left(x _ {s} \mid x _ {t}\right) = \sum_ {i = 1} ^ {M} w _ {i} \mathcal {N} \left(x _ {s} \mid \mu_ {i} \left(x _ {t}\right), \Sigma_ {i} \left(x _ {t}\right)\right), \quad \sum_ {i = 1} ^ {M} w _ {i} = 1, \tag {7}
$$
where  $w_{i}$  is a scalar and  $\mu_i(x_t)$  and  $\Sigma_i(x_t)$  are vectors. The reasons for choosing a Gaussian mixture model are three-fold. First, a Gaussian mixture model is multi-modal (e.g., see Proposition 3.1), potentially leading to a better performance than Gaussian with few steps. Second, when the number of steps is large and the Gaussian is nearly optimal [35, 3], a designed Gaussian mixture model such as our proposed kernel in Eq. (9) can degenerate to a Gaussian when the mean of two components are same, making the performance unchanged. Third, a Gaussian mixture model is relatively easy to sample. For the sake of completeness, we also discuss other distribution families in Appendix C.
Traditionally, the EM algorithm is employed for estimating the Gaussian mixture [26]. However, it is nontrivial to apply EM here because we need to learn the reverse transition kernel in Eq. (7) for all time step pairs  $(s,t)$  by individual EM processes where we need to sample multiple  $x_{s}^{3}$  given a  $x_{t}$  to estimate the parameters in the Gaussian mixture indexed by  $(s,t)$ . This is time-consuming, especially in a high-dimensional space (e.g., natural images) and we present an efficient approach.
For improved flexibility in sampling and training, diffusion models introduce the parameterization of the noise network  $\epsilon(x_{t}, t)$  [14] or the data prediction network  $x_{0}(x_{t}, t)$  [18]. With such a network, the moments under the  $q(x_{s} | x_{t})$  measure can be decomposed into moments under the  $q(x_{0} | x_{t})$  measure so that sampling any  $x_{t}$  to  $x_{s}$  requires only a network whose inputs are  $x_{t}$  and  $t$ , such as the decomposition shown in Eq. (10). In previous studies, Gaussian transition kernel was utilized, allowing for the distribution to be directly determined after obtaining the estimated first-order moment and handcrafted second-order moment [14] or estimated first-order and second-order moment [3]. In contrast, such a feature is not available for the Gaussian mixtures transition kernel in our paper.
Here we present the method to determine the Gaussian mixture given a set of moments and we will discuss how to obtain the moments by the parameterization of the noise network in Sec. 3.3. Assume the length of the estimated moments set is  $N$  and the number of parameters in the Gaussian mixture is  $d$ . We adopt a popular and theoretically sound method called the generalized method of moments (GMM) [11] to learn the parameters by:
$$
\min  _ {\theta} Q \left(\theta , M _ {1}, \dots , M _ {N}\right) = \min  _ {\theta} \left[ \frac {1}{N _ {c}} \sum_ {i = 1} ^ {N _ {c}} g \left(x _ {i}, \theta\right) \right] ^ {T} W \left[ \frac {1}{N _ {c}} \sum_ {i = 1} ^ {N _ {c}} g \left(x _ {i}, \theta\right) \right], \tag {8}
$$
where  $\theta \in \mathbb{R}^{d\times 1}$  includes all parameters (e.g., the mean of each component) in the Gaussian mixture defined in Eq. (7). For instance,  $d = 2M*D_{\mathrm{data}} + M - 1$  in Eq. (7), where  $D_{\mathrm{data}}$  represents the number of dimensions of the data and  $\theta$  contains  $M$  mean vectors with  $D_{\mathrm{data}}$  dimensions,  $M$  variance vectors of with  $D_{\mathrm{data}}$  dimensions (considering only the diagonal elements), and  $M - 1$  weight coefficients which are scalar. The component of  $g(x_i,\theta)\in \mathbb{R}^{N\times 1}$  is defined by  $g_{n}(x_{i},\theta) = M_{n}(x_{i}) - M_{n}^{(GM)}(\theta)$  where  $M_{n}(x_{i})$  is the  $n$ -th order empirical moments stated in Sec. 3.3 and  $M_n^{(GM)}(\theta)$  is the  $n$ -th order moments of Gaussian mixture under  $\theta$ , and  $W$  is a weighted matrix,  $N_{c}$  is the number of samples.
Theoretically, the parameter  $\hat{\theta}_{\mathrm{GMM}}$  obtained by GMM in Eq. (8) consistently converges to the potential optimal parameters  $\theta^{*}$  for Gaussian mixture models given the moments' condition because  $\sqrt{N_c} (\hat{\theta}_{\mathrm{GMM}} - \theta^*)\xrightarrow{d}\mathcal{N}(0,\mathbb{V}(\theta_{GMM}))$ , as stated in Theorem 3.1 in Hansen [11].
<sup>3</sup> Note that  $x_{s}$  serves as the "training sample" in the corresponding EM process.
Hence, we can employ GMM to determine a Gaussian mixture transition kernel after estimating the moments of the transition kernel. To strike a balance between computational tractability and expressive power, we specifically focus on the first three-order moments in this work and define a Gaussian mixture transition kernel with two components shown in Eq. (9) whose vectors  $\mu_t^{(1)},\mu_t^{(2)}$  and  $\sigma_t^2$  are parameters to be optimized. This degree of simplification is acceptable in terms of its impact. Intuitively, such a selection has the potential to encompass exponential modes throughout the entire trajectory. Empirically, we consistently observe that utilizing a bimodal Gaussian mixture yields favorable outcomes across all experimental configurations.
$$
p \left(x _ {s} \mid x _ {t}\right) = \frac {1}{3} \mathcal {N} \left(\mu_ {t} ^ {(1)}, \sigma_ {t} ^ {2}\right) + \frac {2}{3} \mathcal {N} \left(\mu_ {t} ^ {(2)}, \sigma_ {t} ^ {2}\right), \tag {9}
$$
meanwhile, under the simplification in our paper, the number of parameters  $d = 3 * D_{\mathrm{data}}$  in our Gaussian transition kernel is equal to the number of moments condition  $N = 3 * D_{\mathrm{data}}$ . According to proposition 3.2, under the selection of arbitrary weighted weights, the asymptotic mean (asymptotically consistent) and asymptotic variance of the estimator remain consistent, proof in Appendix A.2. Hence, any choice of weighted weights is optimal. To further streamline optimization, we set  $W = I$ .
Proposition 3.2 (Any weighted matrix is optimal for  $d = N$ ). Assume the number of parameters  $d$  equals the number of moments condition  $N$ , and  $\epsilon_{\theta}^{n}(x_{t}, t) \stackrel{p}{\to} \mathbb{E}_{q(x_{0}|x_{t})}[\text{diag}(\epsilon \otimes^{n-1}\epsilon)]$  (where  $\otimes^n$  denotes  $n$ -fold outers product) which denotes  $n$ -th order noise network converging in probability. The asymptotic variance  $\mathbb{V}(\theta_{GMM})$  and the convergence speed of GMM remain the same no matter which weighted matrix is adopted in Eq. (8). Namely, any weighted matrix is optimal.
What's more, we provide a detailed discussion on the selection of parameters such as the choice of different parameters to optimize and the different choices of weight  $w_{i}$  in the Gaussian mixture in Appendix E.1. Combining with Sec. 4.1, our empirical findings illustrate the efficacy of the GMS across various benchmarks via using the Gaussian transition kernel in Eq. (9) fitted by the objective function shown in Eq. (31) in Appendix B.1 via the ADAN [41] as optimization method. Details regarding the parameters for this optimizer are provided in Appendix E.5.
3.3 Estimating high-order moments for non-Gaussian reverse process