SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Limitations. While our calibration method provably improves the model's likelihood, it does not necessarily yield a lower FID score, as previously discussed [45]. Besides, for text-to-image generation, post-training computation of  $\mathbb{E}_{q_t(x_t|y)}[s_\theta^t (x_t,y)]$  becomes infeasible due to the exponentially large number of conditions  $y$ , necessitating dynamic recording with multimodal modules.
Zhijie Deng was supported by Natural Science Foundation of Shanghai (No. 23ZR1428700) and the Key Research and Development Program of Shandong Province, China (No. 2023CXGC010112).
[1] Kazuoki Azuma. Weighted sums of certain dependent random variables. Tohoku Mathematical Journal, Second Series, 19(3):357-367, 1967.
[2] Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.09392, 2022.
[3] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In International Conference on Learning Representations (ICLR), 2022.
[4] Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In European Conference on Computer Vision (ECCV), 2020.
[5] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations (ICLR), 2021.
[6] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[7] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In IEEE International Conference on Computer Vision (CVPR), 2020.
[8] Giannis Daras, Maurizio Delbrazio, Hossein Talebi, Alexandros G Dimakis, and Peyman Milanfar. Soft diffusion: Score matching for general corruptions. arXiv preprint arXiv:2209.05442, 2022.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[10] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[11] Joseph L Doob and Joseph L Doob. Stochastic processes, volume 7. Wiley New York, 1953.
[12] Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106(496):1602-1614, 2011.
[13] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022.
[14] Geoffrey Grimmett and David Stirzaker. Probability and random processes. Oxford university press, 2001.
[15] Xiangming Gu, Chao Du, Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang. On memorization in diffusion models. arXiv preprint arXiv:2310.02664, 2023.
[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS), pages 6626-6637, 2017.
[36] Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[37] Tim Salimans and Jonathan Ho. Should ebms model the energy or the score? In Energy Based Models Workshop-ICLR, 2021.
[38] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations (ICLR), 2022.
[39] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
[40] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning (ICML), pages 2256-2265. PMLR, 2015.
[41] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations (ICLR), 2021.
[42] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems (NeurIPS), pages 11895-11907, 2019.
[43] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[44] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Conference on Uncertainty in Artificial Intelligence (UAI), 2019.
[45] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[46] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021.
[47] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023.
[48] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[49] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-1674, 2011.
[50] Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In International Conference on Machine Learning (ICML), 2023.
[51] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. In International Conference on Learning Representations (ICLR), 2022.
[52] Kexun Zhang, Xianjun Yang, William Yang Wang, and Lei Li. Redi: Efficient learning-free diffusion inference via trajectory retrieval. arXiv preprint arXiv:2302.02285, 2023.
A Detailed derivations
In this section, we provide detailed derivations for the Theorem and equations shown in the main text. We follow the regularization assumptions listed in Song et al. [45].
Proof. For any two timesteps  $0 \leq s < t \leq T$ , i.e., the transition probability from  $x_{s}$  to  $x_{t}$  is written as  $q_{st}(x_t|x_s) = \mathcal{N}\left(x_t\big|\alpha_{t|s}x_s,\sigma_{t|s}^2\mathbf{I}\right)$ , where  $\alpha_{t|s} = \frac{\alpha_t}{\alpha_s}$  and  $\sigma_{t|s}^2 = \sigma_t^2 -\alpha_{t|s}^2\sigma_s^2$ . The marginal distribution  $q_{t}(x_{t}) = \int q_{st}(x_{t}|x_{s})q_{s}(x_{s})dx_{s}$  and we have
$$
\begin{array}{l} \nabla_ {x _ {t}} \log q _ {t} (x _ {t}) = \frac {1}{\alpha_ {t | s}} \nabla_ {\alpha_ {t | s} ^ {- 1} x _ {t}} \log \left(\frac {1}{\alpha_ {t | s} ^ {k}} \mathbb {E} _ {\mathcal {N} \left(x _ {s} \mid \alpha_ {t | s} ^ {- 1} x _ {t}, \alpha_ {t | s} ^ {- 2} \sigma_ {t | s} ^ {2} \mathbf {I}\right)} \left[ q _ {s} (x _ {s}) \right]\right) \\ = \frac {1}{\alpha_ {t | s}} \nabla_ {\alpha_ {t | s} ^ {- 1} x _ {t}} \log \left(\mathbb {E} _ {\mathcal {N} \left(\eta \mid 0, \alpha_ {t | s} ^ {- 2} \sigma_ {t | s} ^ {2} \mathbf {I}\right)} \left[ q _ {s} \left(\alpha_ {t | s} ^ {- 1} x _ {t} + \eta\right) \right]\right) \\ = \frac {\mathbb {E} _ {\mathcal {N} \left(\eta \mid 0 , \alpha_ {t | s} ^ {- 2} \sigma_ {t | s} ^ {2} \mathbf {I}\right)} \left[ \nabla_ {\alpha_ {t | s} ^ {- 1} x _ {t}} q _ {s} \left(\alpha_ {t | s} ^ {- 1} x _ {t} + \eta\right) \right]}{\alpha_ {t | s} \mathbb {E} _ {\mathcal {N} \left(\eta \mid 0 , \alpha_ {t | s} ^ {- 2} \sigma_ {t | s} ^ {2} \mathbf {I}\right)} \left[ q _ {s} \left(\alpha_ {t | s} ^ {- 1} x _ {t} + \eta\right) \right]} \\ = \frac {\mathbb {E} _ {\mathcal {N} \left(\eta \mid 0 , \alpha_ {t | s} ^ {- 2} \sigma_ {t | s} ^ {2} \mathbf {I}\right)} \left[ q _ {s} \left(\alpha_ {t | s} ^ {- 1} x _ {t} + \eta\right) \nabla_ {\alpha_ {t | s} ^ {- 1} x _ {t} + \eta} \log q _ {s} \left(\alpha_ {t | s} ^ {- 1} x _ {t} + \eta\right) \right]}{\alpha_ {t | s} \mathbb {E} _ {\mathcal {N} \left(\eta \mid 0 , \alpha_ {t | s} ^ {- 2} \sigma_ {t | s} ^ {2} \mathbf {I}\right)} \left[ q _ {s} \left(\alpha_ {t | s} ^ {- 1} x _ {t} + \eta\right) \right]} \tag {15} \\ = \frac {\mathbb {E} _ {\mathcal {N} \left(x _ {s} \mid \alpha_ {t | s} ^ {- 1} x _ {t} , \alpha_ {t | s} ^ {- 2} \sigma_ {t | s} ^ {2} \mathbf {I}\right)} \left[ q _ {s} \left(x _ {s}\right) \nabla_ {x _ {s}} \log q _ {s} \left(x _ {s}\right) \right]}{\alpha_ {t | s} \mathbb {E} _ {\mathcal {N} \left(x _ {s} \mid \alpha_ {t | s} ^ {- 1} x _ {t} , \alpha_ {t | s} ^ {- 2} \sigma_ {t | s} ^ {2} \mathbf {I}\right)} \left[ q _ {s} \left(x _ {s}\right) \right]} \\ = \frac {\int \mathcal {N} \left(x _ {t} \mid \alpha_ {t | s} x _ {s} , \sigma_ {t | s} ^ {2} \mathbf {I}\right) q _ {s} \left(x _ {s}\right) \nabla_ {x _ {s}} \log q _ {s} \left(x _ {s}\right) d x _ {s}}{\alpha_ {t | s} \int \mathcal {N} \left(x _ {t} \mid \alpha_ {t | s} x _ {s} , \sigma_ {t | s} ^ {2} \mathbf {I}\right) q _ {s} \left(x _ {s}\right) d x _ {s}} \\ = \frac {1}{\alpha_ {t | s}} \mathbb {E} _ {q _ {s t} (x _ {s} | x _ {t})} \left[ \nabla_ {x _ {s}} \log q _ {s} (x _ {s}) \right]. \\ \end{array}
$$
Note that when the transition probability  $q_{st}(x_t|x_s)$  corresponds to a well-defined forward process, there is  $\alpha_t > 0$  for  $\forall t \in [0,T]$ , and thus we achieve  $\alpha_t\nabla_{x_t}\log q_t(x_t) = \mathbb{E}_{q_{st}(x_s|x_t)}[\alpha_s\nabla_{x_s}\log q_s(x_s)]$ .
A.2 Proof of  $\mathbb{E}_{q_0(x_0)}[\nabla_{x_0}\log q_0(x_0)] = 0$
Proof. The input variable  $x \in \mathbb{R}^k$  and  $q_0(x_0) \in \mathcal{C}^2$ , where  $\mathcal{C}^2$  denotes the family of functions with continuous second-order derivatives. We use  $x^i$  denote the  $i$ -th element of  $x$ , then we can derive the expectation
$$
\begin{array}{l} \mathbb {E} _ {q _ {0} (x _ {0})} \left[ \frac {\partial}{\partial x _ {0} ^ {i}} \log q _ {0} (x _ {0}) \right] = \int \dots \int q _ {0} (x _ {0}) \frac {\partial}{\partial x _ {0} ^ {i}} \log q _ {0} (x _ {0}) d x _ {0} ^ {1} d x _ {0} ^ {2} \dots d x _ {0} ^ {k} \\ = \int \dots \int \frac {\partial}{\partial x _ {0} ^ {i}} q _ {0} \left(x _ {0}\right) d x _ {0} ^ {1} d x _ {0} ^ {2} \dots d x _ {0} ^ {k} \tag {16} \\ = \int \frac {\partial}{\partial x _ {0} ^ {i}} \left(\int q _ {0} (x _ {0} ^ {i}, x _ {0} ^ {\backslash i}) d x _ {0} ^ {\backslash i}\right) d x _ {0} ^ {i} \\ = \int \frac {d}{d x _ {0} ^ {i}} q _ {0} \left(x _ {0} ^ {i}\right) d x _ {0} ^ {i} = 0, \\ \end{array}
$$
This continuously differentiable assumption can be satisfied by adding a small Gaussian noise (e.g., with variance of 0.0001) on the original data distribution, as done in Song and Ermon [42].
where  $x_0^{\setminus i}$  denotes all the  $k - 1$  elements in  $x_0$  except for the  $i$ -th one. The last equation holds under the boundary condition that  $\lim_{x_0^i \to \infty} q_0(x_0^i) = 0$  hold for any  $i \in [K]$ . Thus, we achieve the conclusion that  $\mathbb{E}_{q_0(x_0)}[\nabla_{x_0} \log q_0(x_0)] = 0$ .
A.3 Concentration bounds
We describe concentration bounds [11, 1] of the martingale  $\alpha_{t}\nabla_{x_{t}}\log q_{t}(x_{t})$
Azuma's inequality. For discrete reverse timestep  $t = T, T - 1, \dots, 0$ , Assuming that there exist constants  $0 < c_{1}, c_{2}, \dots, < \infty$  such that for the  $i$ -th element of  $x$ ,
$$
A _ {t} \leq \frac {\partial}{\partial x _ {t - 1} ^ {i}} \alpha_ {t - 1} \log q _ {t - 1} \left(x _ {t - 1}\right) - \frac {\partial}{\partial x _ {t} ^ {i}} \alpha_ {t} \log q _ {t} \left(x _ {t}\right) \leq B _ {t} \text {a n d} B _ {t} - A _ {t} \leq c _ {t} \tag {17}
$$
almost surely. Then  $\forall \epsilon > 0$ , the probability (note that  $\alpha_0 = 1$ )
$$
P \left(\left| \frac {\partial}{\partial x _ {0} ^ {i}} \log q _ {0} \left(x _ {0}\right) - \frac {\partial}{\partial x _ {T} ^ {i}} \alpha_ {T} \log q _ {T} \left(x _ {T}\right) \right| \geq \epsilon\right) \leq 2 \exp \left(- \frac {2 \epsilon^ {2}}{\sum_ {t = 1} ^ {T} c _ {t} ^ {2}}\right). \tag {18}
$$
Specially, considering that  $q_{T}(x_{T}) \approx \mathcal{N}(x_{T}|0,\widetilde{\sigma}^{2}\mathbf{I})$ , there is  $\frac{\partial}{\partial x_T^i}\log q_T(x_T) \approx -\frac{x_T^i}{\widetilde{\sigma}^2}$ . Thus, we can approximately obtain
$$
P \left(\left| \frac {\partial}{\partial x _ {0} ^ {i}} \log q _ {0} \left(x _ {0}\right) + \frac {\alpha_ {T} x _ {T} ^ {i}}{\widetilde {\sigma} ^ {2}} \right| \geq \epsilon\right) \leq 2 \exp \left(- \frac {2 \epsilon^ {2}}{\sum_ {t = 1} ^ {T} c _ {t} ^ {2}}\right). \tag {19}
$$
Doob's inequality. For continuous reverse timestep  $t$  from  $T$  to 0, if the sample paths of the martingale are almost surely right-continuous, then for the  $i$ -th element of  $x$  we have (note that  $\alpha_0 = 1$ )
$$
P \left(\sup  _ {0 \leq t \leq T} \frac {\partial}{\partial x _ {t} ^ {i}} \alpha_ {t} \log q _ {t} \left(x _ {t}\right) \geq C\right) \leq \frac {\mathbb {E} _ {q _ {0} \left(x _ {0}\right)} \left[ \max  \left(\frac {\partial}{\partial x _ {0} ^ {i}} \log q _ {0} \left(x _ {0}\right) , 0\right) \right]}{C}. \tag {20}
$$
A.4 High-order SM objectives
Lu et al. [28] show that the KL divergence  $\mathcal{D}_{\mathrm{KL}}\left(q_0\| p_0^{\mathrm{ODE}}(\theta)\right)$  can be bounded as
$$
\mathcal {D} _ {\mathrm {K L}} \left(q _ {0} \| p _ {0} ^ {\mathrm {O D E}} (\theta)\right) \leq \mathcal {D} _ {\mathrm {K L}} \left(q _ {T} \| p _ {T}\right) + \sqrt {\mathcal {J} _ {\mathrm {S M}} (\theta ; g (t) ^ {2})} \cdot \sqrt {\mathcal {J} _ {\text {F i s h e r}} (\theta)}, \tag {21}
$$
where  $\mathcal{J}_{\mathrm{Fisher}}(\theta)$  is a weighted sum of Fisher divergence between  $q_{t}(x_{t})$  and  $p_t^{\mathrm{ODE}}(\theta)$  as
$$
\mathcal {J} _ {\text {F i s h e r}} (\theta) = \frac {1}{2} \int_ {0} ^ {T} g (t) ^ {2} D _ {F} \left(q _ {t} \| p _ {t} ^ {\mathrm {O D E}} (\theta)\right) d t. \tag {22}
$$
Moreover, Lu et al. [28] prove that if  $\forall t\in [0,T]$  and  $\forall x_{t}\in \mathbb{R}^{k}$ , there exist a constant  $C_F$  such that the spectral norm of Hessian matrix  $\| \nabla_{x_t}^2\log p_t^{\mathrm{ODE}}(x_t;\theta)\| _2\leq C_F$ , and there exist  $\delta_1,\delta_2,\delta_3 > 0$  such that
$$
\left\| \boldsymbol {s} _ {\theta} ^ {t} \left(x _ {t}\right) - \nabla_ {x _ {t}} \log q _ {t} \left(x _ {t}\right) \right\| _ {2} \leq \delta_ {1},
$$
$$
\left\| \nabla_ {x _ {t}} \mathbf {s} _ {\theta} ^ {t} \left(x _ {t}\right) - \nabla_ {x _ {t}} ^ {2} \log q _ {t} \left(x _ {t}\right) \right\| _ {F} \leq \delta_ {2}, \tag {23}
$$
$$
\left\| \nabla_ {x _ {t}} \mathbf {t r} \left(\nabla_ {x _ {t}} \boldsymbol {s} _ {\theta} ^ {t} (x _ {t})\right) - \nabla_ {x _ {t}} \mathbf {t r} \left(\nabla_ {x _ {t}} ^ {2} \log q _ {t} (x _ {t})\right) \right\| _ {2} \leq \delta_ {3},
$$
where  $\| \cdot \| _F$  is the Frobenius norm of matrix. Then there exist a function  $U(t;\delta_1,\delta_2,\delta_3,q)$  that independent of  $\theta$  and strictly increasing (if  $g(t)\neq 0$ ) w.r.t.  $\delta_{1},\delta_{2}$ , and  $\delta_3$ , respectively, such that the Fisher divergence can be bounded as  $D_F(q_t\| p_t^{\mathrm{ODE}}(\theta))\leq U(t;\delta_1,\delta_2,\delta_3,q)$ .
The case after calibration. When we impose the calibration term  $\eta_t^* = \mathbb{E}_{q_t(x_t)}[s_t^t (x_t)]$  to get the score model  $s_{\theta}^{t}(x_{t}) - \eta_{t}^{*}$ , there is  $\nabla_{x_t}\eta_t^* = 0$  and thus  $\nabla_{x_t}(s_\theta^t (x_t) - \eta_t^*) = \nabla_{x_t}s_\theta^t (x_t)$ . Then we have
$$
\left\| \boldsymbol {s} _ {\theta} ^ {t} \left(x _ {t}\right) - \eta_ {t} ^ {*} - \nabla_ {x _ {t}} \log q _ {t} \left(x _ {t}\right) \right\| _ {2} \leq \delta_ {1} ^ {\prime} \leq \delta_ {1},
$$
$$
\left\| \nabla_ {x _ {t}} \left(\boldsymbol {s} _ {\theta} ^ {t} \left(x _ {t}\right) - \eta_ {t} ^ {*}\right) - \nabla_ {x _ {t}} ^ {2} \log q _ {t} \left(x _ {t}\right) \right\| _ {F} \leq \delta_ {2}, \tag {24}
$$
$$
\left\| \nabla_ {x _ {t}} \mathbf {t r} \left(\nabla_ {x _ {t}} \left(s _ {\theta} ^ {t} (x _ {t}) - \eta_ {t} ^ {*}\right)\right) - \nabla_ {x _ {t}} \mathbf {t r} \left(\nabla_ {x _ {t}} ^ {2} \log q _ {t} (x _ {t})\right) \right\| _ {2} \leq \delta_ {3}.
$$
From these, we know that the Fisher divergence  $D_F(q_t\| p_t^{\mathrm{ODE}}(\theta ,\eta_t^*)) \leq U(t;\delta_1',\delta_2,\delta_3,q) \leq U(t;\delta_1,\delta_2,\delta_3,q)$ , namely,  $D_F(q_t\| p_t^{\mathrm{ODE}}(\theta ,\eta_t^*))$  has a lower upper bound compared to  $D_F(q_t\| p_t^{\mathrm{ODE}}(\theta))$ . Consequently, we can get lower upper bounds for both  $\mathcal{J}_{\mathrm{Fisher}}(\theta ,\eta_t^*)$  and  $\mathcal{D}_{\mathrm{KL}}(q_0\| p_0^{\mathrm{ODE}}(\theta ,\eta_t^ {*}))$ , compared to  $\mathcal{J}_{\mathrm{Fisher}}(\theta)$  and  $\mathcal{D}_{\mathrm{KL}}(q_0\| p_0^{\mathrm{ODE}}(\theta))$ , respectively.
B Model parametrization
This section introduces different parametrizations used in diffusion models and provides their calibrated instantiations.
Along the research routine of diffusion models, different model parametrizations have been used, including score prediction  $s_{\theta}^{t}(x_{t})$  [42, 46], noise prediction  $\pmb{\epsilon}_{\theta}^{t}(x_{t})$  [17, 33], data prediction  $\pmb{x}_{\theta}^{t}(x_{t})$  [23, 32], and velocity prediction  $\pmb{v}_{\theta}^{t}(x_{t})$  [38, 18]. Taking the DSM objective as the training loss, its instantiation at timestep  $t \in [0,T]$  is written as
$$
\mathcal {J} _ {\mathrm {D S M}} ^ {t} (\theta) = \left\{ \begin{array}{l l} \frac {1}{2} \mathbb {E} _ {q _ {0} (x _ {0}), q (\epsilon)} \left[ \| \boldsymbol {s} _ {\theta} ^ {t} (x _ {t}) + \frac {\epsilon}{\sigma_ {t}} \| _ {2} ^ {2} \right], & \text {s c o r e p r e d i c t i o n}; \\ \frac {\alpha_ {t} ^ {2}}{2 \sigma_ {t} ^ {4}} \mathbb {E} _ {q _ {0} (x _ {0}), q (\epsilon)} \left[ \| \boldsymbol {x} _ {\theta} ^ {t} (x _ {t}) - x _ {0} \| _ {2} ^ {2} \right], & \text {d a t a p r e d i c t i o n}; \\ \frac {1}{2 \sigma_ {t} ^ {2}} \mathbb {E} _ {q _ {0} (x _ {0}), q (\epsilon)} \left[ \| \boldsymbol {\epsilon} _ {\theta} ^ {t} (x _ {t}) - \epsilon \| _ {2} ^ {2} \right], & \text {n o i s e p r e d i c t i o n}; \\ \frac {\alpha_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}} \mathbb {E} _ {q _ {0} (x _ {0}), q (\epsilon)} \left[ \| \boldsymbol {v} _ {\theta} ^ {t} (x _ {t}) - (\alpha_ {t} \epsilon - \sigma_ {t} x _ {0}) \| _ {2} ^ {2} \right], & \text {v e l o c i t y p r e d i c t i o n}. \end{array} \right. \tag {25}
$$
B.2 Calibrated instantiation
Under different model parametrizations, we can derive the optimal calibration terms  $\eta_t^*$  that minimizing  $\mathcal{J}_{\mathrm{DSM}}^t (\theta ,\eta_t)$  as
$$
\eta_ {t} ^ {*} = \left\{ \begin{array}{l l} \mathbb {E} _ {q _ {t} \left(x _ {t}\right)} \left[ \boldsymbol {s} _ {\theta} ^ {t} \left(x _ {t}\right) \right], & \text {s c o r e p r e d i c t i o n}; \\ \mathbb {E} _ {q _ {t} \left(x _ {t}\right)} \left[ \boldsymbol {x} _ {\theta} ^ {t} \left(x _ {t}\right) \right] - \mathbb {E} _ {q _ {0} \left(x _ {0}\right)} \left[ x _ {0} \right], & \text {d a t a p r e d i c t i o n}; \\ \mathbb {E} _ {q _ {t} \left(x _ {t}\right)} \left[ \boldsymbol {\epsilon} _ {\theta} ^ {t} \left(x _ {t}\right) \right], & \text {n o i s e p r e d i c t i o n}; \\ \mathbb {E} _ {q _ {t} \left(x _ {t}\right)} \left[ \boldsymbol {v} _ {\theta} ^ {t} \left(x _ {t}\right) \right] + \sigma_ {t} \mathbb {E} _ {q _ {0} \left(x _ {0}\right)} \left[ x _ {0} \right], & \text {v e l o c i t y p r e d i c t i o n}. \end{array} \right. \tag {26}
$$
Taking  $\eta_t^*$  into  $\mathcal{J}_{\mathrm{DSM}}^t (\theta ,\eta_t)$  we can obtain the gap
$$
\mathcal {J} _ {\mathrm {D S M}} ^ {t} (\theta) - \mathcal {J} _ {\mathrm {D S M}} ^ {t} (\theta , \eta_ {t} ^ {*}) = \left\{ \begin{array}{l l} \frac {1}{2} \| \mathbb {E} _ {q _ {t} \left(x _ {t}\right)} \left[ \boldsymbol {s} _ {\theta} ^ {t} \left(x _ {t}\right) \right] \| _ {2} ^ {2}, & \text {s c o r e p r e d i c t i o n}; \\ \frac {\alpha_ {t} ^ {2}}{2 \sigma_ {t} ^ {4}} \| \mathbb {E} _ {q _ {t} \left(x _ {t}\right)} \left[ \boldsymbol {x} _ {\theta} ^ {t} \left(x _ {t}\right) \right] - \mathbb {E} _ {q _ {0} \left(x _ {0}\right)} \left[ x _ {0} \right] \| _ {2} ^ {2}, & \text {d a t a p r e d i c t i o n}; \\ \frac {1}{2 \sigma_ {t} ^ {2}} \| \mathbb {E} _ {q _ {t} \left(x _ {t}\right)} \left[ \boldsymbol {\epsilon} _ {\theta} ^ {t} \left(x _ {t}\right) \right] \| _ {2} ^ {2}, & \text {n o i s e p r e d i c t i o n}; \\ \frac {\alpha_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}} \| \mathbb {E} _ {q _ {t} \left(x _ {t}\right)} \left[ \boldsymbol {v} _ {\theta} ^ {t} \left(x _ {t}\right) \right] + \sigma_ {t} \mathbb {E} _ {q _ {0} \left(x _ {0}\right)} \left[ x _ {0} \right] \| _ {2} ^ {2}, & \text {v e l o c i t y p r e d i c t i o n}. \end{array} \right. \tag {27}
$$
C Visualization of the generations
We further show generated images in Figure 5 to double confirm the efficacy of our calibration method. Our calibration could help to reduce ambiguous generations on both CIFAR-10 and CelebA.
(a) CIFAR-10, w/ calibration
(b) CIFAR-10, w/o calibration
(c) CelebA, w/ calibration
Figure 5: Unconditional generation results on CIFAR-10 and CelebA using models from [17] and [41] respectively. The number of sampling steps is 20 based on the results in Tables 1 and 2.
(d) CelebA, w/o calibration