SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tianyu Pang $^{\dagger 1}$ , Cheng Lu $^{2}$ , Chao Du $^{1}$ , Min Lin $^{1}$ , Shuicheng Yan $^{1}$ , Zhijie Deng $^{\dagger 3}$
<sup>1</sup>Sea AI Lab, Singapore
$^{2}$ Department of Computer Science, Tsinghua University
$^{3}$ Qing Yuan Research Institute, Shanghai Jiao Tong University
{tianyupang, duchao, linmin, yansc}@sea.com;
lucheng.lc15@gmail.com; zhijied@sjtu.edu.cn
Recently, diffusion probabilistic models (DPMs) have achieved promising results in diverse generative tasks. A typical DPM framework includes a forward process that gradually diffuses the data distribution and a reverse process that recovers the data distribution from time-dependent data scores. In this work, we observe that the stochastic reverse process of data scores is a martingale, from which concentration bounds and the optional stopping theorem for data scores can be derived. Then, we discover a simple way for calibrating an arbitrary pretrained DPM, with which the score matching loss can be reduced and the lower bounds of model likelihood can consequently be increased. We provide general calibration guidelines under various model parametrizations. Our calibration method is performed only once and the resulting models can be used repeatedly for sampling. We conduct experiments on multiple datasets to empirically validate our proposal. Our code is available at https://github.com/thudzj/Calibrated-DPMs.
In the past few years, denoising diffusion probabilistic modeling [17, 40] and score-based Langevin dynamics [42, 43] have demonstrated appealing results on generating images. Later, Song et al. [46] unify these two generative learning mechanisms through stochastic/ordinary differential equations (SDEs/ODEs). In the following we refer to this unified model family as diffusion probabilistic models (DPMs). The emerging success of DPMs has attracted broad interest in downstream applications, including image generation [10, 22, 48], shape generation [4], video generation [18, 19], superresolution [35], speech synthesis [5], graph generation [51], textual inversion [13, 34], improving adversarial robustness [50], and text-to-image large models [32, 33], just to name a few.
A typical framework of DPMs involves a forward process gradually diffusing the data distribution  $q_{0}(x_{0})$  towards a noise distribution  $q_{T}(x_{T})$ . The transition probability for  $t \in [0, T]$  is a conditional Gaussian distribution  $q_{0t}(x_t|x_0) = \mathcal{N}(x_t|\alpha_t x_0, \sigma_t^2 \mathbf{I})$ , where  $\alpha_{t}, \sigma_{t} \in \mathbb{R}^{+}$ . Song et al. [46] show that there exist reverse SDE/ODE processes starting from  $q_{T}(x_{T})$  and sharing the same marginal distributions  $q_{t}(x_{t})$  as the forward process. The only unknown term in the reverse processes is the data score  $\nabla_{x_t} \log q_t(x_t)$ , which can be approximated by a time-dependent score model  $s_\theta^t(x_t)$  (or with other model parametrizations).  $s_\theta^t(x_t)$  is typically learned via score matching (SM) [20].
In this work, we observe that the stochastic process of the scaled data score  $\alpha_{t}\nabla_{x_{t}}\log q_{t}(x_{t})$  is a martingale w.r.t. the reverse-time process of  $x_{t}$  from  $T$  to 0, where the timestep  $t$  can be either continuous or discrete. Along the reverse-time sampling path, this martingale property leads to concentration bounds for scaled data scores. Moreover, a martingale satisfies the optional stopping theorem that the expected value at a stopping time is equal to its initial expected value.
†Corresponding authors.
Based on the martingale property of data scores, for any  $t \in [0,T]$  and any pretrained score model  $s_{\theta}^{t}(x_{t})$  (or with other model parametrizations), we can calibrate the model by subtracting its expectation over  $q_{t}(x_{t})$ , i.e.,  $\mathbb{E}_{q_t(x_t)}[s_\theta^t (x_t)]$ . We formally demonstrate that the calibrated score model  $s_{\theta}^{t}(x_{t}) - \mathbb{E}_{q_{t}(x_{t})}[s_{\theta}^{t}(x_{t})]$  achieves lower values of SM objectives. By the connections between SM objectives and model likelihood of the SDE process [23, 45] or the ODE process [28], the calibrated score model has higher evidence lower bounds. Similar conclusions also hold for the conditional case, in which we calibrate a conditional score model  $s_{\theta}^{t}(x_{t},y)$  by subtracting its conditional expectation  $\mathbb{E}_{q_t(x_t|y)}[s_\theta^t (x_t,y)]$ .
In practice,  $\mathbb{E}_{q_t(x_t)}[s_\theta^t (x_t)]$  or  $\mathbb{E}_{q_t(x_t|y)}[s_\theta^t (x_t,y)]$  can be approximated using noisy training data when the score model has been pretrained. We can also utilize an auxiliary shallow model to estimate these expectations dynamically during pretraining. When we do not have access to training data, we could calculate the expectations using data generated from  $s_\theta^t (x_t)$  or  $s_\theta^t (x_t,y)$ . In experiments, we evaluate our calibration tricks on the CIFAR-10 [25] and CelebA  $64\times 64$  [27] datasets, reporting the FID scores [16]. We also provide insightful visualization results on the AFHQv2 [7], FFHQ [21] and ImageNet [9] at  $64\times 64$  resolution.
2 Diffusion probabilistic models
In this section, we briefly review the notations and training paradigms used in diffusion probabilistic models (DPMs). While recent works develop DPMs based on general corruptions [2, 8], we mainly focus on conventional Gaussian-based DPMs.
2.1 Forward and reverse processes
We consider a  $k$ -dimensional random variable  $x \in \mathbb{R}^k$  and define a forward diffusion process on  $x$  as  $\{x_t\}_{t \in [0, T]}$  with  $T > 0$ , which satisfies  $\forall t \in [0, T]$ ,
$$
x _ {0} \sim q _ {0} (x _ {0}), \quad q _ {0 t} \left(x _ {t} \mid x _ {0}\right) = \mathcal {N} \left(x _ {t} \mid \alpha_ {t} x _ {0}, \sigma_ {t} ^ {2} \mathbf {I}\right). \tag {1}
$$
Here  $q_{0}(x_{0})$  is the data distribution;  $\alpha_{t}$  and  $\sigma_{t}$  are two positive real-valued functions that are differentiable w.r.t.  $t$  with bounded derivatives. Let  $q_{t}(x_{t}) = \int q_{0t}(x_{t}|x_{0})q_{0}(x_{0})dx_{0}$  be the marginal distribution of  $x_{t}$ . The schedules of  $\alpha_{t},\sigma_{t}^{2}$  need to ensure that  $q_{T}(x_{T})\approx \mathcal{N}(x_{T}|0,\widetilde{\sigma}^{2}\mathbf{I})$  for some  $\widetilde{\sigma}$ . Kingma et al. [23] prove that there exists a stochastic differential equation (SDE) satisfying the forward transition distribution in Eq. (1), and this SDE can be written as
$$
d x _ {t} = f (t) x _ {t} d t + g (t) d \omega_ {t}, \tag {2}
$$
where  $\omega_{t} \in \mathbb{R}^{k}$  is the standard Wiener process,  $f(t) = \frac{d\log\alpha_{t}}{dt}$ , and  $g(t)^2 = \frac{d\sigma_t^2}{dt} - 2\frac{d\log\alpha_t}{dt}\sigma_t^2$ . Song et al. [46] demonstrate that the forward SDE in Eq. (2) corresponds to a reverse SDE constructed as
$$
d x _ {t} = \left[ f (t) x _ {t} - g (t) ^ {2} \nabla_ {x _ {t}} \log q _ {t} \left(x _ {t}\right) \right] d t + g (t) d \bar {\omega} _ {t}, \tag {3}
$$
where  $\overline{\omega}_t\in \mathbb{R}^k$  is the standard Wiener process in reverse time. Starting from  $q_{T}(x_{T})$ , the marginal distribution of the reverse SDE process is also  $q_{t}(x_{t})$  for  $t\in [0,T]$ . There also exists a deterministic process described by an ordinary differential equation (ODE) as
$$
\frac {d x _ {t}}{d t} = f (t) x _ {t} - \frac {1}{2} g (t) ^ {2} \nabla_ {x _ {t}} \log q _ {t} \left(x _ {t}\right), \tag {4}
$$
which starts from  $q_{T}(x_{T})$  and shares the same marginal distribution  $q_{t}(x_{t})$  as the reverse SDE in Eq. (3). Moreover, let  $q_{0t}(x_0|x_t) = \frac{q_{0t}(x_t|x_0)q_0(x_0)}{q_t(x_t)}$  and by Tweedie's formula [12], we know that  $\alpha_{t}\mathbb{E}_{q_{0t}(x_{0}|x_{t})}[x_{0}] = x_{t} + \sigma_{t}^{2}\nabla_{x_{t}}\log q_{t}(x_{t}).$
2.2 Training paradigm of DPMs
To estimate the data score  $\nabla_{x_t}\log q_t(x_t)$  at timestep  $t$ , a score-based model  $s_\theta^t (x_t)$  [46] with shared parameters  $\theta$  is trained to minimize the score matching (SM) objective [20] as
$$
\mathcal {J} _ {\mathrm {S M}} ^ {t} (\theta) \triangleq \frac {1}{2} \mathbb {E} _ {q _ {t} \left(x _ {t}\right)} \left[ \| \boldsymbol {s} _ {\theta} ^ {t} \left(x _ {t}\right) - \nabla_ {x _ {t}} \log q _ {t} \left(x _ {t}\right) \| _ {2} ^ {2} \right]. \tag {5}
$$
To eliminate the intractable computation of  $\nabla_{x_t}\log q_t(x_t)$ , denoising score matching (DSM) [49] transforms  $\mathcal{J}_{\mathrm{SM}}^t (\theta)$  into  $\mathcal{J}_{\mathrm{DSM}}^t (\theta)\triangleq \frac{1}{2}\mathbb{E}_{q_0(x_0),q(\epsilon)}\left[\left\| s_\theta^t (x_t) + \frac{\epsilon}{\sigma_t}\right\| _2^2\right]$ , where  $x_{t} = \alpha_{t}x_{0} + \sigma_{t}\epsilon$  and
$q(\epsilon) = \mathcal{N}(\epsilon |0,\mathbf{I})$  is a standard Gaussian distribution. Under mild boundary conditions, we know  $\mathcal{J}_{\mathrm{SM}}^t (\theta)$  and  $\mathcal{J}_{\mathrm{DSM}}^t (\theta)$  is equivalent up to a constant, i.e.,  $\mathcal{J}_{\mathrm{SM}}^t (\theta) = \mathcal{J}_{\mathrm{DSM}}^t (\theta) + C^t$  and  $C^t$  is a constant independent of the model parameters  $\theta$ . Other SM variants [31, 44] are also applicable here. The total SM objective for training is a weighted sum of  $\mathcal{J}_{\mathrm{SM}}^t (\theta)$  across  $t\in [0,T]$ , defined as  $\mathcal{J}_{\mathrm{SM}}(\theta ;\lambda (t))\triangleq \int_0^T\lambda (t)\mathcal{J}_{\mathrm{SM}}^t (\theta)dt$ , where  $\lambda (t)$  is a positive weighting function. Similarly, the total DSM objective is  $\mathcal{J}_{\mathrm{DSM}}(\theta ;\lambda (t))\triangleq \int_0^T\lambda (t)\mathcal{J}_{\mathrm{DSM}}^t (\theta)dt$ . The training objectives under other model parametrizations such as noise prediction  $\pmb{\epsilon}_{\theta}^{t}(x_{t})$  [17, 33], data prediction  $\pmb{x}_{\theta}^{t}(x_{t})$  [23, 32], and velocity prediction  $\pmb{v}_{\theta}^{t}(x_{t})$  [18, 38] are recapped in Appendix B.1.
2.3 Likelihood of DPMs
Suppose that the reverse processes start from a tractable prior  $p_T(x_T) = \mathcal{N}(x_T|0, \widetilde{\sigma}^2\mathbf{I})$ . We can approximate the reverse-time SDE process by substituting  $\nabla_{x_t}\log q_t(x_t)$  with  $\boldsymbol{s}_{\theta}^{t}(x_{t})$  in Eq. (3) as  $dx_{t} = [f(t)x_{t} - g(t)^{2}s_{\theta}^{t}(x_{t})]dt + g(t)d\overline{\omega}_{t}$ , which induces the marginal distribution  $p_t^{\mathrm{SDE}}(x_t;\theta)$  for  $t \in [0,T]$ . In particular, at  $t = 0$ , the KL divergence between  $q_0(x_0)$  and  $p_0^{\mathrm{SDE}}(x_0;\theta)$  can be bounded by the total SM objective  $\mathcal{J}_{\mathrm{SM}}(\theta;g(t)^2)$  with the weighing function of  $g(t)^2$ , as stated below:
Lemma 1. (Proof in Song et al. [45]) Let  $q_{t}(x_{t})$  be constructed from the forward process in Eq. (2). Then under regularity conditions, we have  $\mathcal{D}_{KL}\left(q_0\| p_0^{SDE}(\theta)\right)\leq \mathcal{J}_{SM}(\theta ;g(t)^2) + \mathcal{D}_{KL}\left(q_T\| p_T\right)$
Here  $\mathcal{D}_{\mathrm{KL}}(q_T\| p_T)$  is the prior loss independent of  $\theta$ . Similarly, we approximate the reverse-time ODE process by substituting  $\nabla_{x_t}\log q_t(x_t)$  with  $s_\theta^t (x_t)$  in Eq. (4) as  $\frac{dx_t}{dt} = f(t)x_t - \frac{1}{2} g(t)^2 s_\theta^t (x_t)$ , which induces the marginal distribution  $p_t^{\mathrm{ODE}}(x_t;\theta)$  for  $t\in [0,T]$ . By the instantaneous change of variables formula [6], we have  $\frac{\log p_t^{\mathrm{ODE}}(x_t;\theta)}{dt} = -\mathbf{tr}\left(\nabla_{x_t}\left(f(t)x_t - \frac{1}{2} g(t)^2 s_\theta^t (x_t)\right)\right)$ , where  $\mathbf{tr}(\cdot)$  denotes the trace of a matrix. Integrating change in  $\log p_t^{\mathrm{ODE}}(x_t;\theta)$  from  $t = 0$  to  $T$  can give the value of  $\log p_T(x_T) - \log p_0^{\mathrm{ODE}}(x_0;\theta)$ , but requires tracking the path from  $x_0$  to  $x_T$ . On the other hand, at  $t = 0$ , the KL divergence between  $q_{0}(x_{0})$  and  $p_0^{\mathrm{ODE}}(x_0;\theta)$  can be decomposed:
Lemma 2. (Proof in Lu et al. [28]) Let  $q_{t}(x_{t})$  be constructed from the forward process in Eq. (2). Then under regularity conditions, we have  $\mathcal{D}_{KL}\left(q_0\| p_0^{ODE}(\theta)\right) = \mathcal{J}_{SM}(\theta ;g(t)^2) + \mathcal{D}_{KL}\left(q_T\| p_T\right) + \mathcal{J}_{Diff}(\theta)$ , where the term  $\mathcal{I}_{Diff}(\theta)$  measures the difference between  $s_\theta^t (x_t)$  and  $\nabla_{x_t}\log p_t^{ODE}(x_t;\theta)$ .
Directly computing  $\mathcal{I}_{\mathrm{Diff}}(\theta)$  is intractable due to the term  $\nabla_{x_t}\log p_t^{\mathrm{ODE}}(x_t;\theta)$ , nevertheless, we could bound  $\mathcal{I}_{\mathrm{Diff}}(\theta)$  via bounding high-order SM objectives [28].
3 Calibrating pretrained DPMs
In this section we begin with deriving the relationship between data scores at different timesteps, which leads us to a straightforward method for calibrating any pretrained DPMs. We investigate further how the dataset bias of finite samples prevents empirical learning from achieving calibration.
3.1 The stochastic process of data score
According to Kingma et al. [23], the form of the forward process in Eq. (1) can be generalized to any two timesteps  $0 \leq s < t \leq T$ . Then, the transition probability from  $x_{s}$  to  $x_{t}$  is written as  $q_{st}(x_t|x_s) = \mathcal{N}\left(x_t\big|\alpha_{t|s}x_s,\sigma_{t|s}^2\mathbf{I}\right)$ , where  $\alpha_{t|s} = \frac{\alpha_t}{\alpha_s}$  and  $\sigma_{t|s}^2 = \sigma_t^2 -\alpha_{t|s}^2\sigma_s^2$ . Here the marginal distribution satisfies  $q_{t}(x_{t}) = \int q_{st}(x_{t}|x_{s})q_{s}(x_{s})dx_{s}$ . We can generally derive the connection between data scores  $\nabla_{x_t}\log q_t(x_t)$  and  $\nabla_{x_s}\log q_s(x_s)$  as stated below:
Theorem 1. (Proof in Appendix A.1) Let  $q_{t}(x_{t})$  be constructed from the forward process in Eq. (2). Then under some regularity conditions, we have  $\forall 0 \leq s < t \leq T$ ,
$$
\alpha_ {t} \nabla_ {x _ {t}} \log q _ {t} \left(x _ {t}\right) = \mathbb {E} _ {q _ {s t} \left(x _ {s} \mid x _ {t}\right)} \left[ \alpha_ {s} \nabla_ {x _ {s}} \log q _ {s} \left(x _ {s}\right) \right], \tag {6}
$$
where  $q_{st}(x_s|x_t) = \frac{q_{st}(x_t|x_s)q_s(x_s)}{q_t(x_t)}$  is the transition probability from  $x_t$  to  $x_s$ .
Theorem 1 indicates that the stochastic process of  $\alpha_{t}\nabla_{x_{t}}\log q_{t}(x_{t})$  is a martingale w.r.t. the reverse-time process of  $x_{t}$  from timestep  $T$  to 0. From the optional stopping theorem [14], the expected
value of a martingale at a stopping time is equal to its initial expected value  $\mathbb{E}_{q_0(x_0)}[\nabla_{x_0}\log q_0(x_0)]$ . It is known that, under a mild boundary condition on  $q_0(x_0)$ , there is  $\mathbb{E}_{q_0(x_0)}[\nabla_{x_0}\log q_0(x_0)] = 0$  (proof is recapped in Appendix A.2). Consequently, as to the stochastic process, the martingale property results in  $\mathbb{E}_{q_t(x_t)}[\nabla_{x_t}\log q_t(x_t)] = 0$  for  $\forall t\in [0,T]$ . Moreover, the martingale property of the (scaled) data score  $\alpha_{t}\nabla_{x_{t}}\log q_{t}(x_{t})$  leads to concentration bounds using Azuma's inequality and Doob's martingale inequality as derived in Appendix A.3. Although we do not use these concentration bounds further in this paper, there are other concurrent works that use roughly similar concentration bounds in diffusion models, such as proving consistency [47] or justifying trajectory retrieval [52].
3.2 A simple calibration trick
Given a pretrained model  $\pmb{s}_{\theta}^{t}(x_{t})$  in practice, there is usually  $\mathbb{E}_{q_t(x_t)}[s_\theta^t (x_t)]\neq 0$ , despite the fact that the expect data score is zero as  $\mathbb{E}_{q_t(x_t)}[\nabla_{x_t}\log q_t(x_t)] = 0$ . This motivates us to calibrate  $s_{\theta}^{t}(x_{t})$  to  $s_{\theta}^{t}(x_{t}) - \eta_{t}$ , where  $\eta_t$  is a time-dependent calibration term that is independent of any particular input  $x_{t}$ . The calibrated SM objective is written as follows:
$$
\begin{array}{l} \mathcal {J} _ {\mathrm {S M}} ^ {t} (\theta , \eta_ {t}) \triangleq \frac {1}{2} \mathbb {E} _ {q _ {t} (x _ {t})} \left[ \| \boldsymbol {s} _ {\theta} ^ {t} (x _ {t}) - \eta_ {t} - \nabla_ {x _ {t}} \log q _ {t} (x _ {t}) \| _ {2} ^ {2} \right] \tag {7} \\ = \mathcal {J} _ {\mathrm {S M}} ^ {t} (\theta) - \mathbb {E} _ {q _ {t} (x _ {t})} \left[ s _ {\theta} ^ {t} (x _ {t}) \right] ^ {\top} \eta_ {t} + \frac {1}{2} \| \eta_ {t} \| _ {2} ^ {2}, \\ \end{array}
$$
where the second equation holds after the results of  $\mathbb{E}_{q_t(x_t)}[\nabla_{x_t}\log q_t(x_t)] = 0$ , and there is  $\mathcal{I}_{\mathrm{SM}}^t (\theta ,0) = \mathcal{I}_{\mathrm{SM}}^t (\theta)$  specifically when  $\eta_t = 0$ . Note that the orange part in Eq. (7) is a quadratic function w.r.t.  $\eta_t$ . We look for the optimal  $\eta_t^* = \arg \min_{\eta_t}\mathcal{I}_{\mathrm{SM}}^t (\theta ,\eta_t)$  that minimizes the calibrated SM objective, from which we can derive
$$
\eta_ {t} ^ {*} = \mathbb {E} _ {q _ {t} \left(x _ {t}\right)} \left[ \boldsymbol {s} _ {\theta} ^ {t} \left(x _ {t}\right) \right]. \tag {8}
$$
After taking  $\eta_t^*$  into  $\mathcal{J}_{\mathrm{SM}}^t (\theta ,\eta_t)$ , we have
$$
\mathcal {J} _ {\mathrm {S M}} ^ {t} \left(\theta , \eta_ {t} ^ {*}\right) = \mathcal {J} _ {\mathrm {S M}} ^ {t} (\theta) - \frac {1}{2} \left\| \mathbb {E} _ {q _ {t} \left(x _ {t}\right)} \left[ \boldsymbol {s} _ {\theta} ^ {t} \left(x _ {t}\right) \right] \right\| _ {2} ^ {2}. \tag {9}
$$
Since there is  $\mathcal{J}_{\mathrm{SM}}^t (\theta) = \mathcal{J}_{\mathrm{DSM}}^t (\theta) + C^t$ , we have  $\mathcal{J}_{\mathrm{DSM}}^t (\theta ,\eta_t^*) = \mathcal{J}_{\mathrm{DSM}}^t (\theta) - \frac{1}{2}\left\| \mathbb{E}_{q_t(x_t)}[s_\theta^t (x_t)]\right\| _2^2$  for the DSM objective. Similar calibration tricks are also valid under other model parametrizations and SM variants, as formally described in Appendix B.2.
Remark. For any pretrained score model  $s_{\theta}^{t}(x_{t})$ , we can calibrate it into  $s_{\theta}^{t}(x_{t}) - \mathbb{E}_{q_{t}(x_{t})}[s_{\theta}^{t}(x_{t})]$  which reduces the SM/DSM objectives at timestep  $t$  by  $\frac{1}{2}\left\| \mathbb{E}_{q_{t}(x_{t})}[s_{\theta}^{t}(x_{t})]\right\|_{2}^{2}$ . The expectation of the calibrated score model is always zero, i.e.,  $\mathbb{E}_{q_t(x_t)}[s_\theta^t (x_t) - \mathbb{E}_{q_t(x_t)}[s_\theta^t (x_t)] = 0$  holds for any  $\theta$  which is consistent with  $\mathbb{E}_{q_t(x_t)}[\nabla_{x_t}\log q_t(x_t)] = 0$  satisfied by data scores.
Calibration preserves conservativeness. A theoretical flaw of score-based modeling is that  $s_{\theta}^{t}(x_{t})$  may not correspond to a probability distribution. To solve this issue, Salimans and Ho [37] develop an energy-based model design, which utilizes the power of score-based modeling and simultaneously makes sure that  $s_{\theta}^{t}(x_{t})$  is conservative, i.e., there exists a probability distribution  $p_{\theta}^{t}(x_{t})$  such that  $\forall x_{t} \in \mathbb{R}^{k}$ , we have  $s_{\theta}^{t}(x_{t}) = \nabla_{x_{t}} \log p_{\theta}^{t}(x_{t})$ . In this case, after we calibrate  $s_{\theta}^{t}(x_{t})$  by subtracting  $\eta_{t}$ , there is  $s_{\theta}^{t}(x_{t}) - \eta_{t} = \nabla_{x_{t}} \log \left( \frac{p_{\theta}^{t}(x_{t})}{\exp(x_{t}^{\top}\eta_{t})Z_{t}(\theta)} \right)$ , where  $Z_{t}(\theta) = \int p_{\theta}^{t}(x_{t}) \exp(-x_{t}^{\top}\eta_{t}) dx_{t}$  represents the normalization factor. Intuitively, subtracting by  $\eta_{t}$  corresponds to a shift in the vector space, so if  $s_{\theta}^{t}(x_{t})$  is conservative, its calibrated version  $s_{\theta}^{t}(x_{t}) - \eta_{t}$  is also conservative.
Conditional cases. As to the conditional DPMs, we usually employ a conditional model  $s_{\theta}^{t}(x_{t},y)$ , where  $y \in \mathcal{V}$  is the conditional context (e.g., class label or text prompt). To learn the conditional data score  $\nabla_{x_t}\log q_t(x_t|y) = \nabla_{x_t}\log q_t(x_t,y)$ , we minimize the SM objective defined as  $\mathcal{J}_{\mathrm{SM}}^t (\theta)\triangleq \frac{1}{2}\mathbb{E}_{q_t(x_t,y)}\left[\| s_\theta^t (x_t,y) - \nabla_{x_t}\log q_t(x_t,y)\| _2^2\right]$ . Similar to the conclusion of  $\mathbb{E}_{q_t(x_t)}[\nabla_{x_t}\log q_t(x_t)] = 0$ , there is  $\mathbb{E}_{q_t(x_t|y)}[\nabla_{x_t}\log q_t(x_t|y)] = 0$ . To calibrate  $s_{\theta}^{t}(x_{t},y)$ , we use the conditional term  $\eta_t(y)$  and the calibrated SM objective is formulated as
$$
\begin{array}{l} \mathcal {I} _ {\mathrm {S M}} ^ {t} (\theta , \eta_ {t} (y)) \triangleq \frac {1}{2} \mathbb {E} _ {q _ {t} \left(x _ {t}, y\right)} \left[ \| \boldsymbol {s} _ {\theta} ^ {t} \left(x _ {t}, y\right) - \eta_ {t} (y) - \nabla_ {x _ {t}} \log q _ {t} \left(x _ {t}, y\right) \| _ {2} ^ {2} \right] \tag {10} \\ = \mathcal {J} _ {\mathrm {S M}} ^ {t} (\theta) - \mathbb {E} _ {q _ {t} (x _ {t}, y)} \left[ s _ {\theta} ^ {t} (x _ {t}, y) ^ {\top} \eta_ {t} (y) + \frac {1}{2} \| \eta_ {t} (y) \| _ {2} ^ {2} \right], \\ \end{array}
$$
and for any  $y \in \mathcal{V}$ , the optimal  $\eta_t^*(y)$  is given by  $\eta_t^*(y) = \mathbb{E}_{q_t(x_t|y)}[s_\theta^t(x_t,y)]$ . We highlight the conditional context  $y$  in contrast to the unconditional form in Eq. (7). After taking  $\eta_t^*(y)$  into  $\mathcal{J}_{\mathrm{SM}}^t(\theta, \eta_t(y))$ , we have  $\mathcal{J}_{\mathrm{SM}}^t(\theta, \eta_t^*(y)) = \mathcal{J}_{\mathrm{SM}}^t(\theta) - \frac{1}{2} \mathbb{E}_{q_t(y)}\left[\|\mathbb{E}_{q_t(x_t|y)}[s_\theta^t(x_t,y)]\|_2^2\right]$ . This conditional calibration form can naturally generalize to other model parametrizations and SM variants.
3.3 Likelihood of calibrated DPMs
Now we discuss the effects of calibration on model likelihood. Following the notations in Section 2.3, we use  $p_0^{\mathrm{SDE}}(\theta ,\eta_t)$  and  $p_0^{\mathrm{ODE}}(\theta ,\eta_t)$  to denote the distributions induced by the reverse-time SDE and ODE processes, respectively, where  $\nabla_{x_t}\log q_t(x_t)$  is substituted with  $s_\theta^t (x_t) - \eta_t$
Likelihood of  $p_0^{\mathrm{SDE}}(\theta, \eta_t)$ . Let  $\mathcal{J}_{\mathrm{SM}}(\theta, \eta_t; g(t)^2) \triangleq \int_0^T g(t)^2 \mathcal{J}_{\mathrm{SM}}^t(\theta, \eta_t) dt$  be the total SM objective after the score model is calibrated by  $\eta_t$ , then according to Lemma 1, we have  $\mathcal{D}_{\mathrm{KL}}(q_0 \| p_0^{\mathrm{SDE}}(\theta, \eta_t)) \leq \mathcal{J}_{\mathrm{SM}}(\theta, \eta_t; g(t)^2) + \mathcal{D}_{\mathrm{KL}}(q_T \| p_T)$ . From the result in Eq. (9), there is
$$
\mathcal {J} _ {\mathrm {S M}} \left(\theta , \eta_ {t} ^ {*}; g (t) ^ {2}\right) = \mathcal {J} _ {\mathrm {S M}} \left(\theta ; g (t) ^ {2}\right) - \frac {1}{2} \int_ {0} ^ {T} g (t) ^ {2} \left\| \mathbb {E} _ {q _ {t} \left(x _ {t}\right)} \left[ \boldsymbol {s} _ {\theta} ^ {t} \left(x _ {t}\right) \right] \right\| _ {2} ^ {2} d t. \tag {11}
$$
Therefore, the likelihood  $\mathcal{D}_{\mathrm{KL}}(q_0\| p_0^{\mathrm{SDE}}(\theta ,\eta_t^*))$  after calibration has a lower upper bound of  $\mathcal{J}_{\mathrm{SM}}(\theta ,\eta_t^*;g(t)^2) + \mathcal{D}_{\mathrm{KL}}(q_T\| p_T)$ , compared to the bound of  $\mathcal{J}_{\mathrm{SM}}(\theta ;g(t)^2) + \mathcal{D}_{\mathrm{KL}}(q_T\| p_T)$  for the original  $\mathcal{D}_{\mathrm{KL}}(q_0\| p_0^{\mathrm{SDE}}(\theta))$ . However, we need to clarify that  $\mathcal{D}_{\mathrm{KL}}(q_0\| p_0^{\mathrm{SDE}}(\theta ,\eta_t^*))$  may not necessarily smaller than  $\mathcal{D}_{\mathrm{KL}}(q_0\| p_0^{\mathrm{SDE}}(\theta))$ , since we can only compare their upper bounds.
Likelihood of  $p_0^{\mathrm{ODE}}(\theta, \eta_t)$ . Note that in Lemma 2, there is a term  $\mathcal{J}_{\mathrm{Diff}}(\theta)$ , which is usually small in practice since  $s_{\theta}^{t}(x_{t})$  and  $\nabla_{x_t}\log p_t^{\mathrm{ODE}}(x_t;\theta)$  are close. Thus, we have
$$
\mathcal {D} _ {\mathrm {K L}} \left(q _ {0} \| p _ {0} ^ {\mathrm {O D E}} (\theta , \eta_ {t})\right) \approx \mathcal {J} _ {\mathrm {S M}} (\theta , \eta_ {t}; g (t) ^ {2}) + \mathcal {D} _ {\mathrm {K L}} \left(q _ {T} \| p _ {T}\right),
$$