SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Junyu Zhang
Central South University zhangjunyu@csu.edu.cn
Daochang Liu
The University of Sydney daochang.liu@sydney.edu.au
Shichao Zhang
Central South University zhangsc@csu.edu.cn
Chang Xu*
The University of Sydney c.xu@sydney.edu.au
The past few years have witnessed great success in the use of diffusion models (DMs) to generate high-fidelity images with the help of stochastic differential equations (SDEs). However, discretization error is an inevitable limitation when utilizing numerical solvers to solve SDEs. To address this limitation, we provide a theoretical analysis demonstrating that an appropriate combination of the contrastive loss and score matching serves as an upper bound of the KL divergence between the true data distribution and the model distribution. To obtain this bound, we utilize a contrastive loss to construct a contrastive sampling chain to fine-tuning the pre-trained DM. In this manner, our method reduces the discretization error and thus yields a smaller gap between the true data distribution and our model distribution. Moreover, the presented method can be applied to fine-tuning various pre-trained DMs, both with or without fast sampling algorithms, contributing to better sample quality or slightly faster sampling speeds. To validate the efficacy of our method, we conduct comprehensive experiments. For example, on CIFAR10, when applied to a pre-trained EDM, our method improves the FID from 2.04 to 1.88 with 35 neural function evaluations (NFEs), and reduces NFEs from 35 to 25 to achieve the same 2.04 FID. The code is available at Contrastive-Sampling.
Diffusion models (DMs) [57, 22] have emerged as powerful generative models, breaking records in image generation [67, 29, 26, 32], and finding rapid applications in other domains such as video generation [23], 3D point cloud generation [46], text-to-image generation [52], speech synthesis [7, 6], inverse problems [62, 30], and lossless compression [36]. While score-based generative models (SGMs) [57, 58] and denoising diffusion probabilistic models (DDPMs) [47, 22] are two branches of DMs, a certain parameterization reveals an equivalence between them [45]. In a seminal work, Song et al. [61] generalized DMs through the lens of stochastic differential equations (SDEs). Specifically, for any given stochastic diffusion process that progressively diffuses a data point into random noise with a continuum noise schedule, a DM learns to remove the added noise with a reverse-time SDE [1]. For SGMs, SDEs utilize deep neural networks to match the gradient of the log probability density with respect to data at each noise scale, which is dubbed denoising score matching [25, 68, 59]. In this manner, DMs enable not only exact likelihood computation [61] like variational auto-encoders (VAE) [37] but also higher sample quality [19] than widely-used generative adversarial networks (GANs) [3].
In practice, directly solving the reverse-time SDE to obtain an image is intractable. Instead, numerical solvers [69, 2, 44] are utilized to discretize the problem by constructing a discrete sampling chain with many sampling steps. Concretely, numerical solvers decompose the intractable integration of an SDE into multiple integration intervals [31] and solve them iteratively. However, due to the intractability of the integration in high dimensions, numerical solvers are unable to obtain the exact solution for the integration of each interval. Instead, they provide approximate solutions, which introduce a discretization error. This error represents the discrepancy between the approximate solution and the exact solution. It is important to note that any discretization scheme used by numerical solvers introduces discretization errors [42] in each integration interval. Consequently, there is a gap between the intermediate data distribution and its corresponding model distribution at each discretization step [75]. The accumulation of these discretization errors results in a larger gap between the true data distribution and its model distribution. It is worth noting that the estimation error, resulting from score matching or noise prediction during training, can also contribute to the gap between these two distributions [75, 17]. However, in this paper, we solely concentrate on the discretization error. From an integration point of view, the discretization error gradually decreases as the step size of the interval decreases. However, it cannot be completely avoided because the step size cannot be reduced to infinitesimal values.
To alleviate this issue, more discretization steps can be used to make each approximate solution of the discretization step closer to the exact solution, but this significantly increases the computational cost during sampling. For example, DDPM [22] requires 1000 steps to produce an image, with each step requiring the evaluation of the neural network once, which is substantially slower than GANs [28, 55]. While recent works have made it possible to achieve high-quality images with significantly reduced sampling steps [29, 32, 70], they still encounter the issue of discretization errors. On the other hand, some fast sampling DMs [27, 65, 72, 53] speed up sampling process for SDEs or ordinary differential equations (ODEs) [61], but suffer from a sharp decline in image quality since the discretization error grows larger. Hence, discretization errors have an severe impact on DMs.
More recently, several studies have been conducted to enhance numerical solvers which aim to reduce the discretization error. DPM-Solver [44] utilizes the semi-linear structure to avoid the corresponding discretization error by analytically computing the linear part of the solutions. By comparison, Liu et al. [42] combine high-order methods with Denoising Diffusion Implicit Model (DDIM) [56] to solve the ODE and achieve further acceleration. However, the accuracy of this approximation is not theoretically justified and may suffer from significant discretization error if the step size is large [75]. To fill the theoretical gap, Zhang and Chen [75] propose to utilize an exponential integrator to remedy the discretization error, achieving much better sample quality with smaller step numbers compared to previous fast sampling approaches. Orthogonal to this direction, our focus lies in minimizing the discretization error by optimizing the upper bound of the Kullback-Leibler (KL) divergence between the true sampling chain and a simulated chain at each time step.
The objective of our work is to establish a contrastive sampling chain to fine-tune any pre-trained DMs so as to reducing the discretization error, contributing to a small gap between data distribution and model distribution. Our method is motivated by the observation that minimizing the KL divergence between the true sampling chain and a simulated chain at each corresponding time step effectively reduces the discretization error. This reduction in discretization error directly translates into an improvement in sample quality. To achieve this, we propose minimizing a contrastive loss [11, 9, 21] that effectively reduces the gap between intermediate data distribution and the model data distribution at each time step. Specifically, by selecting instances on the sampling chain of a pre-trained DM for a same image as positive pairs, and choosing negative instances from other training images, a contrastive loss function is formed. To keep the pre-trained model stable in its generative ability, in practice, we combine the contrastive loss with the original generative loss using dynamic weighting schedules during the fine-tuning process. Moreover, the contrastive loss is optimized with backpropagation through time (BPTT) to spread the gradients on the whole sampling chain. In this manner, our method reduces the gap between true data distribution and model distribution, which improves sample quality without increasing sampling time overhead. Comprehensive experiments validate that our method can improve generation quality for various pre-trained models when using the same neural function evaluations (NFEs), or require less NFEs to achieve the same generation quality.
In a nutshell, our work makes the following contributions: 1) We demonstrate that the discretization error results in a gap between each intermediate data distribution and its corresponding model data distribution. 2) We analyze that an appropriate combination of contrastive loss and score
matching serves as an upper bound for the KL divergence between the data distribution and the model distribution. 3) We propose a contrastive sampling chain to fine-tune a pre-trained DM with the assistance of our derived upper bound. 4) We present dynamic weighting schedules and BPTT as optimization techniques for our method.
As mentioned previously, SGMs and DDPMs have been considered almost equivalent when a certain parameterization is applied. For the sake of simplicity, we solely focus on utilizing denoising score matching under SDEs for further investigation in this paper. Below, we present a comprehensive review of the entire DMs process with the lens of SGMs.
Forward noising diffusion: The forward diffusion of a DM for  $D$ -dimensional data is a linear diffusion described by the stochastic differential equation (SDE) [54]
$$
d x = \boldsymbol {F} _ {t} x d t + \boldsymbol {G} _ {t} d \omega , \tag {1}
$$
where  $\pmb{F}_t \in \mathbb{R}^{D \times D}$  denotes the linear drift coefficient,  $\pmb{G}_t \in \mathbb{R}^{D \times D}$  denotes the diffusion coefficient, and  $\omega$  is a standard Wiener process. The diffusion Eq. (1) is initiated at the training data and simulated over a fixed time schedule  $[0, T]$ . Denote by  $p_t(x_t)$  the marginal distribution of  $x_t$  and by  $p_{0t}(x_t \mid x_0)$  the conditional distribution from  $x_0$  to  $x_t$ , then  $p_0(x_0)$  represents the underlying distribution of the training data. The simulated stochastic process is represented by  $\{x_t^{\mathrm{SDE}}\}_{t \in [0, T]}$ , where  $p_T(x_T)$  is a prior  $\pi(x_T)$  which is easy to sample from, like Gaussian distribution. The parameters  $\pmb{F}_t$  and  $\pmb{G}_t$  are chosen such that the conditional marginal distribution  $p_{0t}(x_t \mid x_0)$  is a simple Gaussian transition kernel, denoted as  $\mathcal{N}(\mu_t x_0, \Sigma_t)$ . Three popular SDEs in DMs are summarized by Song et al. [61], which are variance preserving SDE (VP SDE), variance exploding SDE (VE SDE) and sub-variance preserving (subVP SDE). Ideally, we enable to diffuse any data distribution to a prior distribution  $\pi(x_T)$  with one of those three SDEs.
Backward denoising diffusion: Under mild assumptions [61], the forward diffusion Eq. (1) is associated with a reverse-time diffusion process
$$
d x = \left[ \boldsymbol {F} _ {t} x - \boldsymbol {G} _ {t} \boldsymbol {G} _ {t} ^ {T} \nabla \log p _ {t} (x) \right] d t + \boldsymbol {G} _ {t} d \omega , \tag {2}
$$
where  $\omega$  denotes a standard Wiener process in the reverse-time direction,  $\nabla \log p_t(x)$  denotes the gradient of the log probability density with respect to data at each time step  $t$ . In general, with a known prior distribution  $\pi (x_{T})$ , one can model the data distribution  $p_0(x_0)$  with Eq. (2) as  $F_{t}$  and  $G_{t}$  are fixed according to the forward SDEs. However, to solve Eq. (2), one needs to match the score function  $\nabla \log p_t(x)$ , which is not accessible.
Training: The basic idea of DMs is to use a time-dependent score matching network  $s_{\theta}(x_t, t)$  to approximate the score  $\nabla \log p_t(x)$ . This is achieved by score matching techniques [25, 68, 59] where the score network  $s_{\theta}$  is trained by minimizing the denoising score matching loss
$$
\mathcal {J} _ {\mathrm {S M}} (\theta ; \omega (t)) = \mathbb {E} _ {t \sim U [ 0, T ]} \mathbb {E} _ {p (x _ {0}) p _ {0 t} (x _ {t} | x _ {0})} \left[ \omega (t) \| \nabla \log p _ {0 t} (x _ {t} | x _ {0}) - s _ {\theta} (x _ {t}, t) \| _ {2} ^ {2} \right]. \tag {3}
$$
Here  $\nabla \log p_{0t}(x_t|x_0)$  has a closed form expression as  $p_{0t}(x_t|x_0)$  is a simple Gaussian distribution which represents the discretized form of a given SDE,  $\omega (t)$  denotes a time-dependent weighting function. This loss can be evaluated using empirical samples by Monte Carlo methods and thus standard stochastic optimization algorithms can be used for training.
Sampling: Once the score network  $s_{\theta}(x_t,t) \approx \nabla \log p_t(x)$  is matched for almost all  $x \in \mathbb{R}^D$  and  $t \sim U[0,T]$ , it can be used to generate new samples by solving the backward SDE Eq. (2) with  $\nabla \log p_t(x)$  replaced by  $s_{\theta}(x_t,t)$ . It turns out there are infinitely many diffusion processes one can use. In this work, to show the scalability of our method, we consider a general expression of SDEs
$$
d x ^ {\mathrm {S D E}} = \left[ \boldsymbol {F} _ {t} x ^ {\mathrm {S D E}} - \frac {1 + \lambda^ {2}}{2} \boldsymbol {G} _ {t} \boldsymbol {G} _ {t} ^ {T} s _ {\theta} \left(x ^ {\mathrm {S D E}}, t\right) \right] d t + \lambda \boldsymbol {G} _ {t} d \omega , \tag {4}
$$
parameterized by  $\lambda > 0$ . When  $\lambda = 0$ , Eq. (4) reduces to an ordinary differential equation (ODE) known as the probability flow ODE [8]. The reverse-time diffusion Eq. (2) with an approximated score is a special case of Eq. (4) with  $\lambda = 1$ .
3 Discretization Error Analysis
To generate a new image, one can sample  $x_{T}$  from a standard distribution  $\pi(x_{T})$  and solve Eq. (4) to obtain an image  $x_{0}^{\mathrm{SDE}}$ . However, in practice, exact solutions are not attainable as it is intractable to solve Eq. (4) directly. To remedy this, one needs to discretize Eq. (4) over time to get an approximated solution, which leads to a discretization error. For brevity, we next investigate the discretization error of solving the probability flow ODE  $(\lambda = 0)$
$$
\frac {d x}{d t} = \boldsymbol {F} _ {t} x - \frac {1}{2} \boldsymbol {G} _ {t} \boldsymbol {G} _ {t} ^ {T} s _ {\theta} \left(x _ {t} ^ {\text {S D E}}, t\right), \tag {5}
$$
and  $x_{t}^{\mathrm{SDE}}$  represents the discretization samples solved by Eq. (4) with a numerical solver. The exact solution to this ODE is
$$
x _ {t} ^ {\mathrm {S D E}} = \Psi (t, T) x _ {T} ^ {\mathrm {S D E}} + \int_ {T} ^ {t} \Psi (t, \tau) \left[ - \frac {1}{2} \boldsymbol {G} _ {\tau} \boldsymbol {G} _ {\tau} ^ {T} s _ {\theta} \left(x _ {\tau} ^ {\mathrm {S D E}}, \tau\right) \right] d \tau , \tag {6}
$$
where  $\Psi(t,T)$  satisfying  $\frac{\partial\Psi(t,T)}{\partial(t)} = F_T\Psi(t,T)$ ,  $\Psi(t,T) = I$  is known as the transition matrix from time  $T$  to  $t$  associated with  $F_{\tau}$ . There exist many numerical solvers for Eq. (5) associated with different discretization schemes to approximate Eq. (6). As the discretization step size goes to infinitesimal, the solutions obtained from all these methods converge to that of Eq. (5).
However, the performances of these methods can be dramatically different when the step size is large. On the other hand, to achieve fast sampling in DMs, one needs to approximately solve Eq. (5) with a small number of discretization steps, and thus large step size. Concretely, the discretization of Eq. (4) equals to build a discretization sampling chain, which is  $\{x_{t}^{\mathrm{SDE}}\}_{t\in [0,T]}$ , and iteratively convert it to a new image  $x_0^{\mathrm{SDE}}$  with randomly initialize a sample from  $x_{T}$ , where  $T$  is the total sampling steps. Following the philosophy of discretization sampling, a small number of discretization steps is equivalent to small sampling steps  $T$  and a large step size  $\Delta t$  from  $x_{T}$  to  $x_{t}$ . When Euler method applied to Eq. (5), the discretization form can be expressed as
$$
x _ {t} ^ {\mathrm {S D E}} = x _ {T} ^ {\mathrm {S D E}} - \left[ \boldsymbol {F} _ {T} x _ {T} ^ {\mathrm {S D E}} - \frac {1}{2} \boldsymbol {G} _ {T} \boldsymbol {G} _ {T} ^ {T} s _ {\theta} (x _ {T} ^ {\mathrm {S D E}}, T) \right] \Delta t. \tag {7}
$$
From the integral point of view, samples  $x_0^{\mathrm{SDE}}$  obtained from Eq. (7) equals the exact solutions of Eq. (5) if and only if the step size between  $x_T^{\mathrm{SDE}}$  and  $x_t^{\mathrm{SDE}}$  goes to infinitesimal. In practice, it's impossible to achieve it, especially the fast sampling demand in DMs that requires small  $T$  even one-step sampling [63], which means  $x_0^{\mathrm{SDE}}$  is an approximate solution. Hence, the discretization error is essentially a gap between the approximate solution and the exact solution
$$
\begin{array}{l} \Delta (x _ {t} ^ {\mathrm {S D E}}, x _ {T} ^ {\mathrm {S D E}}) = \frac {1}{2} \left\| \int_ {T} ^ {t} \Psi (t, \tau) \left[ \boldsymbol {G} _ {\tau} \boldsymbol {G} _ {\tau} ^ {T} s _ {\theta} (x _ {\tau} ^ {\mathrm {S D E}}, \tau) \right] d \tau \right. \\ \left. - \left[ 2 * \boldsymbol {F} _ {T} x _ {T} ^ {\text {S D E}} - \boldsymbol {G} _ {T} \boldsymbol {G} _ {T} ^ {T} s _ {\theta} \left(x _ {T} ^ {\text {S D E}}, T\right) \right] \Delta t \right\| _ {2} ^ {2}. \\ \end{array}
$$
In light of this, the presence of  $\Delta (x_{t}^{\mathrm{SDE}},x_{T}^{\mathrm{SDE}})$  creates a gap between  $p_t(x_t)$  and  $p_t^{\mathrm{SDE}}(x_t^{\mathrm{SDE}})$  since each sample  $x_{t}^{\mathrm{SDE}}$  only approximates the exact solution  $x_{t}$ . Consequently, one enables to optimize the KL divergence between  $p_t(x_t)$  and  $p_t^{\mathrm{SDE}}(x_t^{\mathrm{SDE}})$  to minimize this gap, denoted as  $D_{\mathrm{KL}}\left(p_t\| p_t^{\mathrm{SDE}}\right)$ . By minimizing this KL term, one can effectively reduce the discretization error. This is because each  $p_t^{\mathrm{SDE}}(x_t^{\mathrm{SDE}})$  from the simulated chain will closely approximate  $p_t(x_t)$  from the true sampling chain.
It is well-known that maximizing the log-likelihood of a probabilistic model is equivalent to minimizing the KL divergence from the data distribution to the model distribution [60, 43]. Similarly, in order to improve the log-likelihood of DMs, we can optimize the KL divergence  $D_{\mathrm{KL}}(p_0 \| p_0^{\mathrm{SDE}})$  between the true distribution  $p_0(x_0)$  and its corresponding model distribution  $p_0^{\mathrm{SDE}}(x_0^{\mathrm{SDE}})$ .
In what follows, we demonstrate that with a appropriate combination of  $D_{\mathrm{KL}}(p_t \| p_t^{\mathrm{SDE}})$  and score matching losses  $\mathcal{I}_{\mathrm{SM}}(\theta; g(t)^2)$ , they actually become an upper bound on  $D_{\mathrm{KL}}(p_0 \| p_0^{\mathrm{SDE}})$  [60]. Notably, the weighting function  $\omega(t)$  in Eq. (3) is replaced by  $g(t)^2$  which is the diffusion coefficient of a SDE in Eq. (1).
Theorem 1. Let  $p_0(x_0)$  be the true data distribution,  $\pi(x_T)$  be a known prior distribution. Suppose  $\{x_t\}_{t \in [0,T]}$  is a stochastic process defined by the SDE in Eq. (1) with  $x_0 \sim p_0(x_0)$ , where the marginal distribution of  $x_t$  is denoted as  $p_t(x_t)$ . By comparison,  $\{x_t^{\mathrm{SDE}}\}_{t \in [0,T]}$  is another stochastic process obtained by the reverse-SDE in Eq. (4) from a pre-trained DM with  $x_T^{\mathrm{SDE}} \sim p_T(x_T)$  and  $x_0^{\mathrm{SDE}} \sim p_0^{\mathrm{SDE}}(x_0^{\mathrm{SDE}})$ , where the marginal distribution of  $x_t^{\mathrm{SDE}}$  is denoted as  $p_t^{\mathrm{SDE}}(x_t^{\mathrm{SDE}})$ . Under some regularity conditions detailed in Appendix B, we have
$$
D _ {\mathrm {K L}} \left(p _ {0} \| p _ {0} ^ {\mathrm {S D E}}\right) \leq D _ {\mathrm {K L}} \left(p _ {t} \| p _ {t} ^ {\mathrm {S D E}}\right) + \mathcal {J} _ {\mathrm {S M}} (\theta ; g (t) ^ {2}). \tag {9}
$$
Sketch of proof. Let  $\pmb{\mu}$  and  $\pmb{\nu}$  denote the path measures [40, 38] of SDEs in Eq. (1) and Eq. (4) with  $\lambda = 1$ . Conceptually,  $\pmb{\mu}$  is the joint distribution of the forward diffusion process  $\{x_{t}\}_{t\in [0,T]}$  given by Eq. (1) and  $\pmb{\nu}$  represents another joint distribution of the process  $x_0^{\mathrm{SDE}}\sim p_0^{\mathrm{SDE}}(x_0^{\mathrm{SDE}})$  simulated by Eq. (4). Since we can marginalize  $\pmb{\mu}$  and  $\pmb{\nu}$  to obtain distributions  $p_0(x_0)$  and  $p_0^{\mathrm{SDE}}(x_0^{\mathrm{SDE}})$ , the data processing inequality [60] gives  $D_{\mathrm{KL}}\left(p_0\| p_0^{\mathrm{SDE}}\right)\leq D_{KL}(\pmb {\mu}_t\| \pmb {\nu}_t)$ , where  $\pmb {\mu}_t$  and  $\pmb {\nu}_t$  are subset of  $\pmb{\mu}$  and  $\pmb{\nu}$  respectively. Intuitively,  $D_{\mathrm{KL}}\left(p_0\| p_0^{\mathrm{SDE}}\right)$  is a subset of the path measure between  $\pmb{\mu}$  and  $\pmb{\nu}$  which accumulate all the KL divergence  $D_{\mathrm{KL}}\left(p_t\| p_t^{\mathrm{SDE}}\right)_{t\in [0,T]}$  [40]. Hence,  $D_{\mathrm{KL}}\left(p_0\| p_0^{\mathrm{SDE}}\right)\leq$ $D_{KL}(\pmb {\mu}_t\| \pmb {\nu}_t) = \sum_i^t D_{\mathrm{KL}}\left(p_i\| p_i^{\mathrm{SDE}}\right), t\sim [0,T]$ . From the chain rule for the KL divergence [40], we also have  $D_{KL}(\pmb {\mu}_t\| \pmb {\nu}_t) = D_{\mathrm{KL}}\left(p_t\| p_t^{\mathrm{SDE}}\right) + \mathbb{E}_{p_i(\mathbf{z})}\left[D_{\mathrm{KL}}\left(\pmb {\mu}(\cdot \mid x_i = \mathbf{z})\| \pmb {\nu}\left(\cdot \mid x_i^{\mathrm{SDE}} = \mathbf{z}\right)\right)\right]$  with  $i\in [0,t]$ , where the KL divergence in the final term can be computed by applying the Girsanov theorem [48] to Eq. (4) and the reverse-time SDE of Eq. (1). Combining above analysis completes the proof, detailed see in Appendix B.
Though we demonstrate that the combination of score matching losses  $\mathcal{I}_{\mathrm{SM}}(\theta;g(\cdot)^2)$  and a KL term  $D_{\mathrm{KL}}(p_t\| p_t^{\mathrm{SDE}})$  is an upper bound of  $D_{\mathrm{KL}}(p_0\| p_0^{\mathrm{SDE}})$ , it is intractable to optimize Eq. (9) due to the unknown function form of  $p_t$ . To circumvent this problem, we combined with the mutual information (MI) theory [50] that MI between  $p_t(x_t)$  and  $p_t^{\mathrm{SDE}}(x_t^{\mathrm{SDE}})$  can be expressed as  $I(p_t^{\mathrm{SDE}}(x_t^{\mathrm{SDE}}),p_t(x_t))\leq D_{KL}(p_t^{\mathrm{SDE}}\parallel p_t)$ . Moreover, when applied the Jensen's inequality to this term, we enable to obtain an upper bound of  $D_{KL}(p_t\parallel p_t^{\mathrm{SDE}})$  by a InfoNCE loss:  $I_{\mathrm{InfoNCE}}(x_t^{\mathrm{SDE}},x_j^{\mathrm{SDE}})\geq D_{KL}(p_t\parallel p_t^{\mathrm{SDE}})$  [49], where  $j\in [0,T]$  and  $j\neq t$  (detailed in Appendix B). Hence, we obtain a new upper bound of the KL divergence of  $D_{\mathrm{KL}}(p_0\| p_0^{\mathrm{SDE}})$  when applies this term to Eq. (9), which is
$$
D _ {\mathrm {K L}} \left(p _ {0} \| p _ {0} ^ {\mathrm {S D E}}\right) \leq \Gamma \left(x _ {j} ^ {\mathrm {S D E}}, x _ {t} ^ {\mathrm {S D E}}; \beta (t)\right) + \mathcal {I} _ {\mathrm {S M}} \left(\theta ; g (t) ^ {2}\right), \tag {10}
$$
where  $\Gamma(x_{t}^{\mathrm{SDE}}, x_{j}^{\mathrm{SDE}}; \beta(t)) = \beta(t) * I_{\mathrm{InfoNCE}}(x_{t}^{\mathrm{SDE}}, x_{j}^{\mathrm{SDE}})$  is a scaling of  $I_{\mathrm{InfoNCE}}$  with the weighting function  $\beta(t)$ . Hence, it is obvious that the combination of score matching losses  $\mathcal{J}_{\mathrm{SM}}$  and a contrastive loss  $I_{\mathrm{InfoNCE}}(x_{t}^{\mathrm{SDE}}, x_{j}^{\mathrm{SDE}})$  in Eq. (10) is an upper bound of the KL divergence between data distribution  $p_{0}(x_{0})$  and the model distribution  $p_{0}^{\mathrm{SDE}}(x_{0}^{\mathrm{SDE}})$ .