SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Sec. 3.2, we have explored the methodology for determining a Gaussian mixture transition kernel given a set of moments  $M_1,\ldots ,M_n$ . In the subsequent section, we will present an approach for estimating these moments utilizing noise networks and elucidate the process of network learning.
Given the forward process described by Eq. (1), it can be inferred that both  $q(x_{t}|x_{s})$  and  $q(x_{s}|x_{t},x_{0})$  follow Gaussian distributions. Specifically,  $q(x_{t}|x_{s})\sim \mathcal{N}(x_{t}|a_{t|s}x_{s},\sigma_{t|s}^{2}I)$  [21], where  $a_{t|s} = \frac{\alpha(t)}{\alpha(s)}$  and  $\sigma_{t|s}^2 = \sigma^2 (t) - a_{t|s}^2\sigma^2 (s)$ . Consequently, we can deduce that  $\mathbb{E}_{q(x_s|x_t)}[x_s\otimes^n x_s] = \mathbb{E}_{q(x_0|x_t)q(x_s|x_t,x_0)}[x_s\otimes^n x_s]$ , where  $\otimes^n$  denotes  $n$ -fold outer product. Thus, we can first utilize the Gaussian property of  $q(x_{s}|x_{t},x_{0})$  and employ an explicit formula to calculate the moments under the measure of  $q(x_{s}|x_{t},x_{0})$ . What's more, we only consider the diagonal elements of higher-order moments for computational efficiency, similar to Bao et al. [2] since estimating full higher-order moments results in escalated output dimensions (e.g., quadratic growth for covariance and cubic for the third-order moments) and thus requires substantial computational demands. The expression of diagonal elements of the third-order moment of the reverse transition kernel can be derived as:
$$
\begin{array}{l} \hat {M} _ {3} = \mathbb {E} _ {q \left(x _ {s} \mid x _ {t}\right)} \left[ \operatorname {d i a g} \left(x _ {s} \otimes x _ {s} \otimes x _ {s}\right) \right] = \mathbb {E} _ {q \left(x _ {0} \mid x _ {t}\right)} \mathbb {E} _ {q \left(x _ {s} \mid x _ {t}, x _ {0}\right)} \left[ \operatorname {d i a g} \left(x _ {s} \otimes x _ {s} \otimes x _ {s}\right) \right] = \tag {10} \\ \underbrace {\left[ \left(\frac {a _ {t | s} \sigma_ {s} ^ {2}}{\sigma_ {t} ^ {2}}\right) ^ {3} \operatorname {d i a g} \left(x _ {t} \otimes x _ {t} \otimes x _ {t}\right) + 3 \lambda_ {t} ^ {2} \frac {a _ {t | s} \sigma_ {s} ^ {2}}{\sigma_ {t} ^ {2}} x _ {t} \right]} _ {\text {C o n s t a n t t e r m}} \\ + \underbrace {\left[ \frac {3 a _ {t | s} ^ {2} \sigma_ {s} ^ {4} a _ {s | 0} ^ {2} \beta_ {t | s} ^ {2}}{\sigma_ {t} ^ {8}} (\operatorname {d i a g} (x _ {t} \otimes x _ {t})) + \frac {a _ {s | 0} \beta_ {t | s}}{\sigma_ {t} ^ {2}} I \right] \odot \mathbb {E} _ {q \left(x _ {0} \mid x _ {t}\right)} [ x _ {0} ]} _ {\text {L i n e a r t e r m i n} x _ {0}} \\ + \underbrace {3 \frac {a _ {t | s} \sigma_ {s} ^ {2}}{\sigma_ {t} ^ {2}} (\frac {a _ {s | 0} \beta_ {t | s}}{\sigma_ {t} ^ {2}}) ^ {2} x _ {t} \odot \mathbb {E} _ {q (x _ {0} | x _ {t})} [ \mathrm {d i a g} (x _ {0} \otimes x _ {0}) ]} _ {\text {Q u a d r a t i c t e r m i n} x _ {0}} + \underbrace {\left(\frac {a _ {s | 0} \beta_ {t | s}}{\sigma_ {t} ^ {2}}\right) ^ {3} \mathbb {E} _ {q (x _ {0} | x _ {t})} [ \mathrm {d i a g} (x _ {0} \otimes x _ {0} \otimes x _ {0}) ]} _ {\text {C u b i c t e r m i n} x _ {0}}, \\ \end{array}
$$
Algorithm 1 Learning of the high order noise network
1: Input: The  $n$ -th order noise network  $\epsilon_{\theta, \phi_n}^n(x_t, t)$  and its tunable parameter  $\phi_n$   
2: repeat  
3:  $x_0 \sim q(x_0)$   
4:  $t \sim \text{Uniform}([1, \dots, T])$   
5:  $\epsilon_t \sim \mathcal{N}(0, I)$   
6:  $x_t = \alpha(t)x_0 + \sigma(t)\epsilon_t$   
7: Take a gradient step on  $\phi_n^{(t)} \gets \nabla_{\phi_n}\left\| \epsilon_t^n - \epsilon_{\theta, \phi_n}^n(x_t, t) \right\|_2^2$   
8: until converged  
Algorithm 2 Sampling via GMS with the first three order moments  
1: Input: The assembled noise network  $f^3(x_t, t)$  in Eq. (13)  
2:  $x_T \sim \mathcal{N}(0, I)$   
3: for  $t = T$  to 1 do  
4:  $z_t \sim \mathcal{N}(0, I)$   
5:  $\hat{M}_1, \hat{M}_2, \hat{M}_3 = h(f_{[1]}^3(x_t, t), f_{[2]}^3(x_t, t), f_{[3]}^3(x_t, t))$  in Appendix B  
6:  $\pi_k^*, \mu_k^*, \sigma_k^* = \min_{\pi_k, \mu_k, \sigma_k} Q(\pi_k, \mu_k, \sigma_k, \hat{M}_1, \hat{M}_2, \hat{M}_3)$  in Sec. 3.2  
7:  $i \sim \pi_k^*, x_{t-1} = \mu_i^* + \sigma_i^* z_t$   
8: end for  
9: return  $x_0$
where  $\otimes$  is the outer product  $\odot$  is the Hadamard product. Additionally,  $\lambda_t^2$  corresponds to the variance of  $q(x_s|x_t,x_0)$ , and further elaboration on this matter can be found in Appendix B.
In order to compute the  $n$ -th order moment  $\hat{M}_n$ , as exemplified by the third-order moment in Eq. (10), it is necessary to evaluate the expectations  $\mathbb{E}_{q(x_0|x_t)}[x_0], \ldots, \mathbb{E}_{q(x_0|x_t)}[\mathrm{diag}(x_0 \otimes^{n-1} x_0)]$ . Furthermore, by expressing  $x_0$  as  $x_0 = \frac{1}{\sqrt{\alpha(t)}} (x_t - \sigma(t)\epsilon)$ , we can decompose  $\hat{M}_n$  into a combination of terms  $\mathbb{E}_{q(x_0|x_t)}[\epsilon], \ldots, \mathbb{E}_{q(x_0|x_t)}[(\epsilon \odot^{n-1}\epsilon)]$ , which are learned by neural networks. The decomposition of third-order moments  $\hat{M}_3$ , as outlined in Eq. (29), is provided to illustrate this concept. Therefore in training, we learn several neural networks  $\{\epsilon_\theta^n\}_{n=1}^N$  by training on the following objective functions:
$$
\min  _ {\left\{\epsilon_ {\theta} ^ {n} \right\} _ {n = 1} ^ {N}} \mathbb {E} _ {t} \mathbb {E} _ {q (x _ {0}) q (\epsilon)} \| \epsilon^ {n} - \epsilon_ {\theta} ^ {n} (x _ {t}, t) \| _ {2} ^ {2}, \tag {11}
$$
where  $\epsilon \sim \mathcal{N}(0,I)$ ,  $x_{t} = \alpha (t)x_{0} + \sigma (t)\epsilon$ , and  $\epsilon^n$  denotes  $\epsilon \odot^{n - 1}\epsilon$ . After training, we can use  $\{\epsilon_{\theta}^n\}_{n = 1}^N$  to replace  $\{\mathbb{E}_{q(x_0|x_t)}[\epsilon \odot^{n - 1}\epsilon]\}_{n = 1}^N$  and estimate the moments  $\{\hat{M}_n\}_{n = 1}^N$  of reverse transition kernel.
However, in the present scenario, it is necessary to infer the network a minimum of  $n$  times in order to make a single step of GMS. To mitigate the high cost of the aforementioned overhead in sampling, we adopt the two-stage learning approach proposed by Bao et al. [2]. Specifically, in the first stage, we optimize the noise network  $\epsilon_{\theta}(x_t,t)$  by minimizing the expression  $\min \mathbb{E}_t\mathbb{E}_{q(x_0)q(\epsilon)}\| \epsilon -\epsilon_{\theta}(x_t,t)\| _2^2$  or by utilizing a pre-trained noise network as proposed by [14, 37]. In the second stage, we utilize the optimized network as the backbone and keep its parameters  $\theta$  fixed, while adding additional heads to generate the  $n$ -th order noise network  $\epsilon_{\theta ,\phi_n}^n (x_t,t)$ .
$$
\epsilon_ {\theta , \phi_ {n}} ^ {n} (x _ {t}, t) = \mathrm {N N} (\epsilon_ {\theta} (x _ {t}, t), \phi_ {n}), \tag {12}
$$
where NN is the extra head, which is a small network such as convolution layers or small attention block, parameterized by  $\phi_{n}$ , details in Appendix E.3. We present the second stage learning procedure in Algorithm 1. Upon training all the heads, we can readily concatenate the outputs of different heads, as the backbone of the higher-order noise network is shared. By doing so, we obtain the assembled noise network  $f^{N}(x_{t},t)$ . When estimating the  $k$ -th order moments, it suffices to extract only the first  $k$  components of the assembled noise network.
$$
f ^ {N} \left(x _ {t}, t\right) = \operatorname {c o n c a t} \left(\underbrace {\left[ \epsilon_ {\theta} \left(x _ {t} , t\right) , \epsilon_ {\theta , \phi_ {2}} ^ {2} \left(x _ {t} , t\right) , . . . , \epsilon_ {\theta , \phi_ {k}} ^ {k} \left(x _ {t} , t\right) \right.} _ {\text {R e q u i r e d f o r e s t i m a t i n g} \hat {M} _ {k}}, \dots , \epsilon_ {\theta , \phi_ {N}} ^ {N} \left(x _ {t}, t\right) ]\right), \tag {13}
$$
Table 1: Comparison with competitive SDE-based solvers w.r.t. FID score  $\downarrow$  on CIFAR10 and ImageNet  ${64} \times  {64}$  . our GMS outperforms existing SDE-based solvers in most cases. SN-DDPM denotes Extended AnalyticDPM from Bao et al. [2].
# TIMESTEPS K	CIFAR10 (LS)
10	20	25	40	50	100	200	1000
DDPM, βt	43.14	25.28	21.63	15.24	15.21	10.94	8.23	5.11
DDPM, βt	233.41	168.22	125.05	82.31	66.28	31.36	12.96	3.04
SN-DDPM	21.87	8.32	6.91	4.99	4.58	3.74	3.34	3.71
GMS (OURS)	17.43	7.18	5.96	4.52	4.16	3.26	3.01	2.76
# TIMESTEPS K	CIFAR10 (CS)	IMAGENET 64 × 64
10	25	50	100	200	1000	25	50	100	200	400	4000
DDPM, βt	34.76	16.18	11.11	8.38	6.66	4.92	29.21	21.71	19.12	17.81	17.48	16.55
DDPM, βt	205.31	84.71	37.35	14.81	5.74	3.34	170.28	83.86	45.04	28.39	21.38	16.38
SN-DDPM	16.33	6.05	4.19	3.83	3.72	4.08	27.58	20.74	18.04	16.72	16.37	16.22
GMS (OURS)	13.80	5.48	4.00	3.46	3.34	4.23	26.50	20.13	17.29	16.60	15.98	15.79
To this end, we outline the GMS sampling process in Algorithm 2, where  $f_{[2]}^3 (x_t,t)$  represents  $\mathrm{concat}([\epsilon_{\theta}(x_t,t),\epsilon_{\theta ,\phi_2}^2 (x_t,t)])$ . In comparison to existing methods with the same network structure, we report the additional memory cost of the assembled noise network in Appendix E.6.
In this section, we first illustrate that GMS exhibits superior sample quality compared to existing SDE-based solvers when using both linear and cosine noise schedules [14, 30]. Additionally, we evaluate various solvers in stroke-based image synthesis (i.e., SDEdit) and demonstrate that GMS surpasses other SDE-based solvers, as well as the widely used ODE-based solver DDIM [36].
4.1 Sample quality on image data
In this section, we conduct a quantitative comparison of sample quality using the widely adopted FID score [13]. Specifically, we evaluate multiple SDE-based solvers, including a comparison with DDPM [14] and Extended AnalyticDPM [2] (referred to as SN-DDPM) using the even trajectory.
As shown in Tab. 1, GMS demonstrates superior performance compared to DDPM and SN-DDPM under the same number of steps in CIFAR10 and ImageNet  $64 \times 64$ . Specifically, GMS achieves a remarkable 4.44 improvement in FID given 10 steps on CIFAR10. Appendix E.7 illustrates in more detail the improvement of GMS when the number of sampling steps is limited. Meanwhile, we conduct GMS in ImageNet  $256 \times 256$  via adopting the U-ViT-Huge [4] backbone as the noise network in Appendix E.8. Furthermore, taking into account the additional time required by GMS, our method still exhibits improved performance, as detailed in Appendix E.9. For integrity, we provide a comparison with other SDE-based solvers based on continuous time diffusion such as Gotta Go Fast [17], EDM [18] and SEED [10] in Appendix E.10 and shows that GMS largely outperforms other SDE-based solvers when the number of steps is less than 100. In Appendix G, we provide generated samples from GMS.
4.2 Stroke-based image synthesis based on SDEdit [27]
Evaluation metrics. We evaluate the editing results based on realism and faithfulness similar with Meng et al. [27]. To quantify the realism of sample images, we use FID between the generated images and the target realistic image dataset. To quantify faithfulness, we report the  $L_{2}$  distance summed over all pixels between the stroke images and the edited output images.
SDEdit [27] applies noise to the stroke image  $x^g$  at time step  $t_0$  using  $\mathcal{N}(x_{t_0}^g|\alpha (t)x^g,\sigma^2 (t)I)$  and discretize the reverse SDE in Eq. (4) for sampling. Fig. 9 demonstrates the significant impact of  $t_0$  on the realism of sampled images. As  $t_0$  increases, the similarity to real images decreases. we choose
(a) Trade-off between realism and faithfulness
(b)  $t_0 = 0.3T$
Figure 3: Result among different solvers in SDEdit.  $t_0$  denotes the time step of the start of reverse. (a): The points on each line represent the same  $t_0$  and the number of sampling steps. We select  $t_0 = [300, 400, 500]$ , number of steps  $= [50, 100]$ . (b): When  $t_0 = 300$ , the effect of DDIM diminishes more prominently with the increase in the number of steps.
(c)  $t_0 = 0.4T$
(d)  $t_0 = 0.5T$
the range from  $t_0 = 0.3T$  to  $t_0 = 0.5T$  ( $T = 1000$  in our experiments) for our further experiments since sampled images closely resemble the real images in this range.
Fig. 3(a) illustrates that when using the same  $t_0$  and the same number of steps, edited output images from GMS have lower faithfulness but with higher realism. This phenomenon is likely attributed to the Gaussian noise introduced by the SDE-based solver during each sampling step. This noise causes the sampling to deviate from the original image (resulting in low faithfulness) but enables the solver to transition from the stroke domain to the real image domain. Fig. 3(b) to Fig. 3(d) further demonstrates this phenomenon to a certain extent. The realism of the sampled images generated by the SDE-based solver escalates with an increase in the number of sampling steps. Conversely, the realism of the sampled images produced by the ODE-based solver diminishes due to the absence of noise, which prevents the ODE-based solver from transitioning from the stroke domain to the real image domain. Additionally, in the SDEdit task, GMS exhibits superior performance compared to SN-DDDPM [2] in terms of sample computation cost. Fig. 4 shows the samples using DDIM and GMS when  $t_0 = 400$  and the number of steps is 40.
Faster solvers. In addition to SDE-based solvers, there are works dedicated to improving the efficiency of ODE-based solvers [25, 23, 8]. Some approaches use explicit reverse transition kernels, such as those based on generative adversarial networks proposed by Xiao et al. [40] and Wang et al. [39]. Gao et al. [9] employ an energy function to model the reverse transition kernel. Zhang and Chen [44] use a flow model for the transition kernel.
Non-Gaussian diffusion. Apart from diffusion, some literature suggests using non-Gaussian forward processes, which consequently involve non-Gaussian reverse processes. Bansal et al. [1] introduce a generalized noise operator that incorporates noise. Nachmani et al. [29] incorporate Gaussian mixture or Gamma noise into the forward process. While these works replace both the forward and reverse processes with non-Gaussian distributions, our approach aims to identify a suitable combination of non-Gaussian distributions to model the reverse process.
This paper presents a novel Gaussian mixture solver (GMS) for diffusion models. GMS relaxes the Gaussian reverse kernel assumption to reduce discretization errors and improves the sample quality under the same sampling steps. Experimental results show that GMS outperforms existing SDE-based solvers, achieving a remarkable 4.44 improvement in FID compared to the state-of-the-art SDE-based solver proposed by Bao et al. [2] given 10 steps. Furthermore, due to the presence of noise, SDE-based solvers prove more suitable for stroke-based synthesis tasks and GMS still outperforms state-of-the-art SDE-based solvers.
(a) Real Images
(b) Stroke
(c) GMS
Figure 4: SDEdit samples of GMS and DDIM. Due to the presence of noise, SDE-based solvers, such as GMS Ours, generate images with more details.
(d) DDIM
Limitations and broader impacts. While GMS enhances sample quality and potentially accelerates inference speed compared to existing SDE-based solvers, employing GMS still fall short of real-time applicability. Like other generative models, diffusion models can generate problematic fake content, and the use of GMS may amplify these undesirable effects.
[1] Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.09392, 2022.
[2] Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. arXiv preprint arXiv:2206.07309, 2022.
[3] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201.06503, 2022.
[43] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
[44] Qinsheng Zhang and Yongxin Chen. Diffusion normalizing flow. Advances in Neural Information Processing Systems, 34:16280-16291, 2021.
[45] Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. arXiv preprint arXiv:2207.06635, 2022.
When  $q(x_0)$  is a mixture of Dirac distribution which means that  $q(x_0) = \sum_{i=1}^{M} w_i \delta(x - x_i)$ ,  $\sum_{i=1}^{M} w_i = 1$ , which has total  $M$  components, and when the forward process  $q(x_t | x_0)$  is a Gaussian distribution as Eq. (1), the reverse process  $q(x_s | x_t)$  would be:
$$
\begin{array}{l} q (x _ {s} | x _ {t}) = \int q (x _ {s} | x _ {t}, x _ {0}) q (x _ {0} | x _ {t}) d x _ {0} = \int q (x _ {s} | x _ {t}, x _ {0}) q (x _ {0}) q (x _ {t} | x _ {0}) / q (x _ {t}) d x _ {0} \\ = 1 / q (x _ {t}) \int q (x _ {s} | x _ {t}, x _ {0}) q (x _ {0}) q (x _ {t} | x _ {0}) d x _ {0} = 1 / q (x _ {t}) \sum_ {i = 1} ^ {M} w _ {i} q (x _ {s} | x _ {t}, x _ {0} ^ {i}) q (x _ {t} | x _ {0} ^ {i}) \\ \end{array}
$$
According to the definition of forward process, the distribution  $q(x_{t}|x_{0}^{i}) = \mathcal{N}(x_{t}|\sqrt{\bar{a}_{t}} x_{0}^{i},(1 - \bar{a}_{t})I)$ . Due to the Markov property of forward process, when  $t > s$ , we have  $q(x_{s},x_{t}|x_{0})q(x_{s}|x_{0})q(x_{t}|x_{s})$ . The term  $q(x_{s}|x_{t},x_{0})$  would be viewed as a Bayesian posterior resulting from a prior  $q(x_{s}|x_{0})$ , updated with a likelihood term  $q(x_{t}|x_{s})$ . And therefore
$$
q (x _ {s} | x _ {t}, x _ {0} ^ {i}) = N (x _ {s} | \mu_ {q} (x _ {t}, x _ {0}), \Sigma_ {q} (x _ {t}, x _ {0}) I) = \mathcal {N} (x _ {s} | \frac {a _ {t | s} \sigma_ {s} ^ {2}}{\sigma_ {t} ^ {2}} x _ {t} + \frac {a _ {s} \sigma_ {t | s} ^ {2}}{\sigma_ {t} ^ {2}} x _ {0}, \frac {\sigma_ {s} ^ {2} \sigma_ {t | s} ^ {2}}{\sigma_ {t} ^ {2}} I)
$$
It is easy to prove that, the distribution  $q(x_{s}|x_{t})$  is a mixture of Gaussian distribution:
$$
\begin{array}{l} q (x _ {s} | x _ {t}) \propto \sum_ {i = 1} ^ {M} w _ {i} q (x _ {s} | x _ {t}, x _ {0} ^ {i}) q (x _ {t} | x _ {0} ^ {i}) \\ = \sum_ {i = 1} ^ {M} w _ {i} \mathcal {N} \left(x _ {t} \mid \sqrt {\bar {a} _ {t}} x _ {0} ^ {i}, (1 - \bar {a} _ {t}) I\right) * \mathcal {N} \left(x _ {s} \mid \mu_ {q} \left(x _ {t}, x _ {0}\right), \Sigma_ {q} \left(x _ {t}, x _ {0}\right)\right) \\ \end{array}
$$
When  $t$  is large,  $s$  is small,  $\sigma_{t|s}^{2}$  would be large, meaning that the influence of  $x_0^i$  would be large.
Secondly, when  $q(x_0)$  is a mixture of Gaussian distribution which means that  $q(x_0) = \sum_{i=1}^{M} w_i \mathcal{N}(x_0^i | \mu_i, \Sigma_i), \sum_{i=1}^{M} w_i = 1$ . For simplicity of analysis, we assume that this distribution is a one-dimensional distribution or that the covariance matrix is a high-dimensional Gaussian distribution with a diagonal matrix  $\Sigma_i = \mathrm{diag}_i(\sigma^2)$ . Similar to the situation above, for each dimension in the reverse process:
$$
\begin{array}{l} q (x _ {s} | x _ {t}) = 1 / q (x _ {t}) \int q (x _ {s} | x _ {t}, x _ {0}) q (x _ {0}) q (x _ {t} | x _ {0}) d x _ {0} \\ = \sum_ {i = 1} ^ {M} w _ {i} / q (x _ {t}) \int q (x _ {s} | x _ {t}, x _ {0}) \mathcal {N} (x _ {0} | \mu_ {i}, \Sigma_ {i}) q (x _ {t} | x _ {0}) d x _ {0} \\ = \sum_ {i = 1} ^ {M} w _ {i} / q (x _ {t}) \int \frac {1}{\sqrt {2 \pi} \sigma_ {q}} e ^ {- \frac {(x _ {0} ^ {i} - \mu_ {q}) ^ {2}}{\sigma_ {q} ^ {2}}} \frac {1}{\sqrt {2 \pi} \sigma_ {i}} e ^ {- \frac {(x _ {0} ^ {i} - \mu_ {i}) ^ {2}}{\sigma_ {i} ^ {2}}} \frac {1}{\sqrt {2 \pi} \sqrt {1 - \bar {a} _ {t}}} e ^ {- \frac {(x _ {t} - \sqrt {\bar {a} _ {t} x _ {0}}) ^ {2}}{1 - \bar {a} _ {t}}} d x _ {0} \\ = \sum_ {i = 1} ^ {M} w _ {i} / q (x _ {t}) \int Z _ {i} \frac {1}{\sqrt {2 \pi} \sigma_ {x}} e ^ {- \frac {(x _ {0} ^ {i} - \mu_ {x}) ^ {2}}{\sigma_ {x} ^ {2}}} e ^ {- \frac {(x _ {s} - \mu (x _ {t})) ^ {2}}{\sigma (x _ {t})}} d x _ {0} \\ \end{array}
$$
And  $q(x_{s}|x_{t})$  could be a Gaussian mixture which has  $M$  component.
Recall that our objective function is to find optimal parameters:
$$
\hat {\theta} = \underset {\theta} {\operatorname {a r g m i n}} [ \underbrace {Q _ {N _ {c}} (\theta)} _ {1 \times 1} ] = \underset {\theta} {\operatorname {a r g m i n}} [ \underbrace {g _ {N _ {c}} (\theta) ^ {T} W _ {N _ {c}} g _ {N _ {c}} (\theta)} _ {1 \times N} ] \tag {14}
$$
where  $g_{M}(\theta)$  is the moment conditions talked about in Sec. 3.2,  $W$  is the weighted Matrix,  $N_{c}$  is the sample size. When solving such problems using optimizers, which are equivalent to the one we used when selecting  $\theta_{GMM}$  such that  $\frac{\partial Q_T(\theta_{GMM})}{\partial\theta} = 0$ , and its first derivative:
$$
\underbrace {\frac {\partial Q _ {N _ {c}} (\theta)}{\partial \theta}} _ {d \times 1} = \left(\frac {\frac {\partial Q _ {N _ {c}} (\theta)}{\partial \theta_ {1}}}{\frac {\partial Q _ {N _ {c}} (\theta)}{\partial \theta_ {2}}} \right. \left. \frac {\partial Q _ {N _ {c}} (\theta)}{\partial \theta_ {3}}\right), \frac {\partial Q _ {N _ {c}} (\theta)}{\partial \theta_ {m}} = 2 \underbrace {\left[ \frac {1}{N _ {c}} \sum_ {i = 1} ^ {N _ {c}} \frac {\partial g (x _ {i} , \theta)}{\partial \theta_ {m}} \right] ^ {T}} _ {1 \times N} \underbrace {W _ {N _ {c}}} _ {N \times N} \underbrace {\left[ \frac {1}{N _ {c}} \sum_ {i = 1} ^ {M} g (x _ {i} , \theta) \right]} _ {N \times 1} \tag {15}
$$
its second derivative (the Hessian matrix):
$$
\underbrace {\frac {\partial^ {2} Q _ {N _ {c}} (\theta)}{\partial \theta^ {2}}} _ {d \times d} = \left( \begin{array}{c c c c} \frac {\partial^ {2} Q _ {N _ {c}} (\theta)}{\partial \theta_ {1} \theta_ {1}} & \frac {\partial^ {2} Q _ {N _ {c}} (\theta)}{\partial \theta_ {1} \theta_ {2}} & \dots & \frac {\partial^ {2} Q _ {N _ {c}} (\theta)}{\partial \theta_ {1} \theta_ {d}} \\ \frac {\partial^ {2} Q _ {N _ {c}} (\theta)}{\partial \theta_ {2} \theta_ {1}} & \dots & \dots & \dots \\ \dots & \dots & \dots & \dots \\ \dots & \dots & \dots & \frac {\partial^ {2} Q _ {N _ {c}} (\theta)}{\partial \theta_ {d} \theta_ {d}} \end{array} \right) \tag {16}
$$
$$
\begin{array}{l} , \frac {\partial^ {2} Q _ {N _ {c}} (\theta)}{\partial \theta_ {i} \theta_ {j}} = 2 [ \frac {1}{M} \sum_ {i = 1} ^ {N _ {c}} \frac {\partial g (x _ {i} , \theta)}{\partial \theta_ {i}} ] ^ {T} W _ {N _ {c}} [ \frac {1}{N _ {c}} \sum_ {i = 1} ^ {N _ {c}} \frac {\partial g (x _ {i} , \theta)}{\partial \theta_ {j}} ] \\ + 2 \left[ \frac {1}{N _ {c}} \sum_ {i = 1} ^ {N _ {c}} \frac {\partial^ {2} g (x _ {i} , \theta)}{\partial \theta_ {i} \theta_ {j}} \right] W _ {N _ {c}} \left[ \frac {1}{N _ {c}} \sum_ {i = 1} ^ {N _ {c}} g (x _ {i}, \theta) \right]. \\ \end{array}
$$
By Taylor's expansion of the gradient around optimal parameters  $\theta_0$ , we have:
$$
\frac {\partial Q _ {N _ {c}} \left(\theta_ {G M M}\right)}{\partial \theta} - \frac {\partial Q _ {N _ {c}} \left(\theta_ {0}\right)}{\partial \theta} \approx \frac {\partial^ {2} Q _ {N _ {c}} \left(\theta_ {0}\right)}{\partial \theta \partial \theta^ {T}} \left(\theta_ {G M M} - \theta\right) \tag {17}
$$
$$
\longmapsto (\theta_ {G M M} - \theta) \approx - (\frac {\partial^ {2} Q _ {N _ {c}} (\theta_ {0})}{\partial \theta \partial \theta^ {T}}) ^ {- 1} \frac {\partial Q _ {N _ {c}} (\theta_ {0})}{\partial \theta}.
$$
Consider one element of the gradient vector  $\frac{\partial Q_{N_c}(\theta_0)}{\partial\theta_m}$
$$
\frac {\partial Q _ {N _ {c}} \left(\theta_ {0}\right)}{\partial \theta_ {m}} = 2 \left[ \underbrace {\frac {1}{M} \sum_ {i = 1} ^ {N _ {c}} \frac {\partial g \left(x _ {i} , \theta_ {0}\right)}{\partial \theta_ {m}}} _ {\stackrel {P} {\rightarrow} \mathbb {E} \left(\frac {\partial g \left(x _ {i} , \theta_ {0}\right)}{\partial \theta_ {m}}\right) = \Gamma_ {0 , m}} \right] ^ {T} \underbrace {W _ {N _ {c}}} _ {\stackrel {P} {\rightarrow} W} \underbrace {\frac {1}{N _ {c}} \sum_ {i = 1} ^ {N _ {c}} g \left(x _ {i} , \theta_ {0}\right)} _ {\stackrel {P} {\rightarrow} \mathbb {E} [ g \left(x _ {i} , \theta_ {0}\right) ] = 0}. \tag {18}
$$
Consider one element of the Hessian matrix  $\frac{\partial^2Q_{N_c}(\theta_0)}{\partial\theta_i\partial\theta_j^T}$