SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and  $\mathcal{D}_{\mathrm{KL}}\left(q_0\| p_0^{\mathrm{ODE}}(\theta ,\eta_t^*)\right)$  approximately achieves its lowest value. Lu et al. [28] show that  $\mathcal{D}_{\mathrm{KL}}\left(q_0\| p_0^{\mathrm{ODE}}(\theta)\right)$  can be further bounded by high-order SM objectives (as detailed in Appendix A.4), which depend on  $\nabla_{x_t}\pmb{s}_\theta^t (x_t)$  and  $\nabla_{x_t}\mathbf{tr}\left(\nabla_{x_t}\pmb{s}_\theta^t (x_t)\right)$ . Since the calibration term  $\eta_{t}$  is independent of  $x_{t}$ , i.e.,  $\nabla_{x_t}\eta_t = 0$ , it does not affect the values of high-order SM objectives, and achieves a lower upper bound due to the lower value of the first-order SM objective.
3.4 Empirical learning fails to achieve  $\mathbb{E}_{q_t(x_t)}[s_\theta^t (x_t)] = 0$
A question that naturally arises is whether better architectures or learning algorithms for DPMs (e.g., EDMs [22]) could empirically achieve  $\mathbb{E}_{q_t(x_t)}[s_\theta^t (x_t)] = 0$  without calibration? The answer may be negative, since in practice we only have access to a finite dataset sampled from  $q_{0}(x_{0})$ . More specifically, assuming that we have a training dataset  $\mathbb{D}\triangleq \{x_0^n\}_{n = 1}^N$  where  $x_0^n\sim q_0(x_0)$  and defining the kernel density distribution induced by  $\mathbb{D}$  as  $q_{t}(x_{t};\mathbb{D})\propto \sum_{n = 1}^{N}\mathcal{N}\left(\frac{x_{t} - \alpha_{t}x_{0}^{n}}{\sigma_{t}} |\mathbf{0},\mathbf{I}\right)$ . When the quantity of training data approaches infinity, we have  $\lim_{N\to \infty}q_t(x_t;\mathbb{D}) = q_t(x_t)$  holds for  $\forall t\in [0,T]$ . Then the empirical DSM objective trained on  $\mathbb{D}$  is written as
$$
\mathcal {J} _ {\mathrm {D S M}} ^ {t} (\theta ; \mathbb {D}) \triangleq \frac {1}{2 N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {q (\epsilon)} \left[ \left\| s _ {\theta} ^ {t} \left(\alpha_ {t} x _ {0} ^ {n} + \sigma_ {t} \epsilon\right) + \frac {\epsilon}{\sigma_ {t}} \right\| _ {2} ^ {2} \right], \tag {12}
$$
and it is easy to show that the optimal solution for minimizing  $\mathcal{J}_{\mathrm{DSM}}^t (\theta ;\mathbb{D})$  satisfies (assuming  $s_\theta^t$  has universal model capacity)  $s_{\theta}^{t}(x_{t}) = \nabla_{x_{t}}\log q_{t}(x_{t};\mathbb{D})$ . Given a finite dataset  $\mathbb{D}$ , there is
$$
\mathbb {E} _ {q _ {t} \left(x _ {t}; \mathbb {D}\right)} \left[ \nabla_ {x _ {t}} \log q _ {t} \left(x _ {t}; \mathbb {D}\right) \right] = 0, \text {b u t t y p i c a l l} \mathbb {E} _ {q _ {t} \left(x _ {t}\right)} \left[ \nabla_ {x _ {t}} \log q _ {t} \left(x _ {t}; \mathbb {D}\right) \right] \neq 0, \tag {13}
$$
indicating that even if the score model is learned to be optimal, there is still  $\mathbb{E}_{q_t(x_t)}[s_\theta^t (x_t)]\neq 0$  Thus, the mis-calibration of DPMs is partially due to the dataset bias, i.e., during training we can only access a finite dataset  $\mathbb{D}$  sampled from  $q_{0}(x_{0})$
Furthermore, when trained on a finite dataset in practice, the learned model will not converge to the optimal solution [15], so there is typically  $\pmb{s}_{\theta}^{t}(x_{t}) \neq \nabla_{x_{t}} \log q_{t}(x_{t};\mathbb{D})$  and  $\mathbb{E}_{q_t(x_t;\mathbb{D})}[s_\theta^t (x_t)] \neq 0$ . After calibration, we can at least guarantee that  $\mathbb{E}_{q_t(x_t;\mathbb{D})}\left[s_\theta^t (x_t) - \mathbb{E}_{q_t(x_t;\mathbb{D})}[s_\theta^t (x_t)]\right] = 0$  always holds on any finite dataset  $\mathbb{D}$ . In Figure 3, we demonstrate that even state-of-the-art EDMs still have non-zero and semantic  $\mathbb{E}_{q_t(x_t)}[s_\theta^t (x_t)]$ , which emphasises the significance of calibrating DPMs.
Figure 1: Time-dependent values of  $\frac{1}{2}\|\mathbb{E}_{q_t(x_t)}[\pmb{\epsilon}_\theta^t(x_t)]\|_2^2$  (the first row) and  $\frac{g(t)^2}{2\sigma_t^2}\|\mathbb{E}_{q_t(x_t)}[\pmb{\epsilon}_\theta^t(x_t)]\|_2^2$  (the second row) calculated on different datasets. The models on CIFAR-10 and CelebA are trained on discrete timesteps ( $t = 0,1,\dots,1000$ ), while those on AFHQv2, FFHQ, and ImageNet are trained on continuous timesteps ( $t \in [0,1]$ ). We convert data prediction  $\pmb{x}_\theta^t(x_t)$  into noise prediction  $\pmb{\epsilon}_\theta^t(x_t)$  based on  $\pmb{\epsilon}_\theta^t(x_t) = (x_t - \alpha_t\pmb{x}_\theta^t(x_t)) / \sigma_t$ . The y-axis is clamped into  $[0,500]$ .
3.5 Amortized computation of  $\mathbb{E}_{q_t(x_t)}[s_\theta^t (x_t)]$
By default, we are able to calculate and restore the value of  $\mathbb{E}_{q_t(x_t)}[s_\theta^t (x_t)]$  for a pretrained model  $s_{\theta}^{t}(x_{t})$ , where the selection of timestep  $t$  is determined by the inference algorithm, and the expectation over  $q_{t}(x_{t})$  can be approximated by Monte Carlo sampling from a noisy training set. When we do not have access to training data, we can approximate the expectation using data generated from  $p_t^{\mathrm{ODE}}(x_t;\theta)$  or  $p_t^{\mathrm{SDE}}(x_t;\theta)$ . Since we only need to calculate  $\mathbb{E}_{q_t(x_t)}[s_\theta^t (x_t)]$  once, the raised computational overhead is amortized as the number of generated samples increases.
Dynamically recording. In the preceding context, we focus primarily on post-training computing of  $\mathbb{E}_{q_t(x_t)}[s_\theta^t (x_t)]$ . An alternative strategy would be to dynamically record  $\mathbb{E}_{q_t(x_t)}[s_\theta^t (x_t)]$  during the pretraining phase of  $s_\theta^t (x_t)$ . Specifically, we could construct an auxiliary shallow network  $h_{\phi}(t)$  parameterized by  $\phi$ , whose input is the timestep  $t$ . We define the expected mean squared error as
$$
\mathcal {J} _ {\operatorname {C a l}} ^ {t} (\phi) \triangleq \mathbb {E} _ {q _ {t} \left(x _ {t}\right)} \left[ \| h _ {\phi} (t) - \boldsymbol {s} _ {\theta} ^ {t} \left(x _ {t}\right) ^ {\dagger} \| _ {2} ^ {2} \right], \tag {14}
$$
where the superscript  $\dagger$  denotes the stopping gradient and  $\phi^{*}$  is the optimal solution of minimizing  $\mathcal{I}_{\mathrm{Cal}}^t (\phi)$  w.r.t.  $\phi$ , satisfying  $h_{\phi^*}(t) = \eta_t^* = \mathbb{E}_{q_t(x_t)}[s_\theta^t (x_t)]$  (assuming sufficient model capacity). The total training objective can therefore be expressed as  $\mathcal{J}_{\mathrm{SM}}(\theta ;\lambda (t)) + \int_0^T\beta_t\cdot \mathcal{J}_{\mathrm{Cal}}^t (\phi)$ , where  $\beta_{t}$  is a time-dependent trade-off coefficient for  $t\in [0,T]$ .
In this section, we demonstrate that sample quality and model likelihood can be both improved by calibrating DPMs. Instead of establishing a new state-of-the-art, the purpose of our empirical studies is to testify the efficacy of our calibration technique as a simple way to repair DPMs.
4.1 Sample quality
Setup. We apply post-training calibration to discrete-time models trained on CIFAR-10 [25] and CelebA [27], which apply parametrization of noise prediction  $\pmb{\epsilon}_{\theta}^{t}(x_{t})$ . In the sampling phase, we employ DPM-Solver [29], an ODE-based sampler that achieves a promising balance between sample efficiency and image quality. Because our calibration directly acts on model scores, it is also compatible with other ODE/SDE-based samplers [3, 26], while we only focus on DPM-Solver cases in this paper. In accordance with the recommendation, we set the end time of DPM-Solver to  $10^{-3}$  when the number of sampling steps is less than 15, and to  $10^{-4}$  otherwise. Additional details can be
Table 1: Comparison on sample quality measured by FID  $\downarrow$  with varying NFE on CIFAR-10. Experiments are conducted using a linear noise schedule on the discrete-time model from [17]. We consider three variants of DPM-Solver with different orders. The results with  $\dagger$  mean the actual NFE is order  $\times  \left\lfloor  \frac{\mathrm{{NFE}}}{\text{order }}\right\rfloor$  which is smaller than the given NFE, following the setting in [29].
Noise prediction	DPM-Solver	Number of evaluations (NFE)
10	15	20	25	30	35	40
εθt(xt)	1-order	20.49	12.47	9.72	7.89	6.84	6.22	5.75
2-order	7.35	†4.52	4.14	†3.92	3.74	†3.71	3.68
3-order	†23.96	4.61	†3.89	†3.73	3.65	†3.65	†3.60
εθt(xt) - Eqxt(xt) [εθt(xt)]	1-order	19.31	11.77	8.86	7.35	6.28	5.76	5.36
2-order	6.76	†4.36	4.03	†3.66	3.54	†3.44	3.48
3-order	†53.50	4.22	†3.32	†3.33	3.35	†3.32	†3.31
Table 2: Comparison on sample quality measured by FID  $\downarrow$  with varying NFE on CelebA  ${64} \times  {64}$  . Experiments are conducted using a linear noise schedule on the discrete-time model from [41]. The settings of DPM-Solver are the same as on CIFAR-10.
Noise prediction	DPM-Solver	Number of evaluations (NFE)
10	15	20	25	30	35	40
εθt(xt)	1-order	16.74	11.85	7.93	6.67	5.90	5.38	5.01
2-order	4.32	†3.98	2.94	†2.88	2.88	†2.88	2.84
3-order	†11.92	3.91	†2.84	†2.76	2.82	†2.81	†2.85
εθt(xt) - Eq(txt) [εθt(xt)]	1-order	16.13	11.29	7.09	6.06	5.28	4.87	4.39
2-order	4.42	†3.94	2.61	†2.66	2.54	†2.52	2.49
3-order	†35.47	3.62	†2.33	†2.43	2.40	†2.43	†2.49
found in Lu et al. [29]. By default, we employ the FID score [16] to quantify the sample quality using 50,000 samples. Typically, a lower FID indicates a higher sample quality. In addition, in Table 3, we evaluate using other metrics such as sFID [30], IS [39], and Precision/Recall [36].
Computing  $\mathbb{E}_{q_t(x_t)}[\epsilon_\theta^t (x_t)]$ . To estimate the expectation over  $q_{t}(x_{t})$ , we construct  $x_{t} = \alpha_{t}x_{0} + \sigma_{t}\epsilon$  where  $x_0\sim q_0(x_0)$  is sampled from the training set and  $\epsilon \sim \mathcal{N}(\epsilon |0,\mathbf{I})$  is sampled from a standard Gaussian distribution. The selection of timestep  $t$  depends on the sampling schedule of DPM-Solver. The computed values of  $\mathbb{E}_{q_t(x_t)}[\epsilon_\theta^t (x_t)]$  are restored in a dictionary and warped into the output layers of DPMs, allowing existing inference pipelines to be reused.
We first calibrate the model trained by Ho et al. [17] on the CIFAR-10 dataset and compare it to the original one for sampling with DPM-Solvers. We conduct a systematical study with varying NFE (i.e., number of function evaluations) and solver order. The results are presented in Tables 1 and 3. After calibrating the model, the sample quality is consistently enhanced, which demonstrates the significance of doing so and the efficacy of our method. We highlight the significant improvement in sample quality  $(4.61\rightarrow 4.22$  when using 15 NFE and 3-order DPM-Solver;  $3.89\rightarrow 3.32$  when using 20 NFE and 3-order DPM-Solver). After model calibration, the number of steps required to achieve convergence for a 3-order DPM-Solver is reduced from  $\geq 30$  to 20, making our method a new option for expediting the sampling of DPMs. In addition, as a point of comparison, the 3-order DPM-Solver with 1,000 NFE can only yield an FID score of 3.45 when using the original model, which, along with the results in Table 1, indicates that model calibration helps to improve the convergence of sampling.
Then, we conduct experiments with the discrete-time model trained on the CelebA 64x64 dataset by Song et al. [41]. The corresponding sample quality comparison is shown in Table 2. Clearly, model calibration brings significant gains (3.91→3.62 when using 15 NFE and 3-order DPM-Solver; 2.84→2.33 when using 20 NFE and 3-order DPM-Solver) that are consistent with those on the CIFAR-10 dataset. This demonstrates the prevalence of the mis-calibration issue in existing DPMs and the efficacy of our correction. We still observe that model calibration improves convergence of sampling, and as shown in Figure 2, our calibration could help to reduce ambiguous generations. More generated images are displayed in Appendix C.
Table 3: Comparison on sample quality measured by different metrics, including FID ↓, sFID ↓, inception score (IS) ↑, precision ↑ and recall ↑ with varying NFE on CIFAR-10. We use Base to denote the baseline  $\epsilon_{\theta}^{t}(x_{t})$  and Ours to denote calibrated score  $\epsilon_{\theta}^{t}(x_{t}) - \mathbb{E}_{q_{t}(x_{t})}\left[\epsilon_{\theta}^{t}(x_{t})\right]$ . The sampler is DPM-Solver with different orders. Note that FID is computed by the PyTorch checkpoint of Inception-v3, while sFID/IS/Precision/Recall are computed by the Tensorflow checkpoint of Inception-v3 following github.com/kynkaat/improved-precision-and-recall-metric.
Method	Number of evaluations (NFE)
20	25	30
FID	sFID	IS	Pre.	Rec.	FID	sFID	IS	Pre.	Rec.	FID	sFID
1-ord.	9.72	6.03	8.49	0.641	0.542	7.89	5.45	8.68	0.644	0.556	6.84	5.12	8.76
Base	2.44	4.36	9.15	0.654	0.590	3.92	4.22	9.17	0.657	0.591	3.74	4.18	9.20
3-ord.	3.89	4.18	9.29	0.652	0.597	3.73	4.15	9.21	0.657	0.595	3.65	4.12	9.22
1-ord.	8.86	6.01	8.56	0.649	0.544	7.35	5.42	8.76	0.653	0.560	6.28	5.09	8.84
Ours	4.03	4.31	9.17	0.661	0.592	3.66	4.20	9.20	0.664	0.594	3.54	4.14	9.23
3-ord.	3.32	4.14	9.38	0.657	0.603	3.33	4.11	9.28	0.665	0.597	3.35	4.08	9.27
w/o calibration (baseline)
Figure 2: Selected images on CIFAR-10 (generated with NFE = 20 using 3-order DPM-Solver) demonstrating that our calibration could reduce ambiguous generations, such as generations that resemble both horse and dog. However, we must emphasize that not all generated images have a visually discernible difference before and after calibration.
w/ calibration (ours)
4.2 Model likelihood
As described in Section 3.3, calibration contributes to reducing the SM objective, thereby decreasing the upper bound of the KL divergence between model distribution at timestep  $t = 0$  (either  $p_0^{\mathrm{SDE}}(\theta ,\eta_t^*)$  or  $p_0^{\mathrm{ODE}}(\theta ,\eta_t^*)$ ) and data distribution  $q_{0}$ . Consequently, it aids in raising the lower bound of model likelihood. In this subsection, we examine such effects by evaluating the aforementioned DPMs on the CIFAR-10 and CelebA datasets. We also conduct experiments with continuous-time models trained by Karras et al. [22] on AFHQv2  $64\times 64$  [7], FFHQ  $64\times 64$  [21], and ImageNet  $64\times 64$  [9] datasets considering their top performance. These models apply parametrization of data prediction  $\pmb{x}_{\theta}^{t}(x_{t})$ , and for consistency, we convert it to align with  $\pmb{\epsilon}_{\theta}^{t}(x_{t})$  based on the relationship  $\pmb{\epsilon}_{\theta}^{t}(x_{t}) = (x_{t} - \alpha_{t}\pmb{x}_{\theta}^{t}(x_{t})) / \sigma_{t}$ , as detailed in Kingma et al. [23] and Appendix B.2.
Given that we employ noise prediction models in practice, we first estimate  $\frac{1}{2}\|\mathbb{E}_{q_t(x_t)}[\pmb{\epsilon}_\theta^t(x_t)]\|_2^2$  at timestep  $t \in [0,T]$ , which reflects the decrement on the SM objective at  $t$  according to Eq. (9) (up to a scaling factor of  $1/\sigma_t^2$ ). We approximate the expectation using Monte Carlo (MC) estimation with training data points. The results are displayed in the first row of Figure 1. Notably, the value of  $\frac{1}{2}\|\mathbb{E}_{q_t(x_t)}[\pmb{\epsilon}_\theta^t(x_t)]\|_2^2$  varies significantly along with timestep  $t$ : it decreases relative to  $t$  for CelebA but increases in all other cases (except for  $t \in [0.4,1.0]$  on ImageNet  $64 \times 64$ ). Ideally, there should be  $\frac{1}{2}\|\mathbb{E}_{q_t(x_t)}[\nabla_{x_t}\log q_t(x_t)]\|_2^2 = 0$  at any  $t$ . Such inconsistency reveals that mis-calibration issues exist in general, although the phenomenon may vary across datasets and training mechanisms.
Then, we quantify the gain of model calibration on increasing the lower bound of model likelihood, which is  $\frac{1}{2}\int_{0}^{T}g(t)^{2}\left\| \mathbb{E}_{q_{t}(x_{t})}\left[s_{\theta}^{t}(x_{t})\right]\right\|_{2}^{2}dt$  according to Eq. (11). We first rewrite it with the model parametrization of noise prediction  $\pmb{\epsilon}_{\theta}^{t}(x_{t})$ , and it can be straightforwardly demonstrated that it equals  $\int_0^T\frac{g(t)^2}{2\sigma_t^2}\| \mathbb{E}_{q_t(x_t)}\left[\pmb {\epsilon}_\theta^t (x_t)\right]\| _2^2$ . Therefore, we calculate the value of  $\frac{g(t)^2}{2\sigma_t^2}\| \mathbb{E}_{q_t(x_t)}\left[\pmb {\epsilon}_\theta^t (x_t)\right]\| _2^2$  using MC
Figure 3: Visualization of the expected predicted noises with increasing  $t$ . For each dataset, the first row displays  $\mathbb{E}_{q_t(x_t)}[\epsilon_\theta^t (x_t)]$  (after normalization) and the second row highlights the top-10% pixels that  $\mathbb{E}_{q_t(x_t)}[\epsilon_\theta^t (x_t)]$  has high values. The DPM on CelebA is a discrete-time model with 1000 timesteps [41] and that on FFHQ is a continuous-time one [22].
estimation and report the results in the second row of Figure 1. The integral is represented by the area under the curve (i.e., the gain of model calibration on the lower bound of model likelihood). Various datasets and model architectures exhibit non-trivial gains, as observed. In addition, we notice that the DPMs trained by Karras et al. [22] show patterns distinct from those of DDPM [17] and DDIM [41], indicating that different DPM training mechanisms may result in different mis-calibration effects.
Visualizing  $\mathbb{E}_{q_t(x_t)}[\epsilon_\theta^t (x_t)]$ . To better understand the inductive bias learned by DPMs, we visualize the expected predicted noises  $\mathbb{E}_{q_t(x_t)}[\epsilon_\theta^t (x_t)]$  for timestep from 0 to  $T$ , as seen in Figure 3. For each dataset, the first row normalizes the values of  $\mathbb{E}_{q_t(x_t)}[\epsilon_\theta^t (x_t)]$  into [0, 255]; the second row calculates pixel-wise norm (across RGB channels) and highlights the top-10% locations with the highest norm. As we can observe, on facial datasets like CelebA and FFHQ, there are obvious facial patterns inside  $\mathbb{E}_{q_t(x_t)}[\epsilon_\theta^t (x_t)]$ , while on other datasets like CIFAR-10, ImageNet, as well as the animal face dataset AFHQv2, the patterns inside  $\mathbb{E}_{q_t(x_t)}[\epsilon_\theta^t (x_t)]$  are more like random noises. Besides, the facial patterns in Figure 3 are more significant when  $t$  is smaller, and become blurry when  $t$  is close to  $T$ . This phenomenon may be attributed to the bias of finite training data, which is detrimental to generalization during sampling and justifies the importance of calibration as described in Section 3.4.
We conduct ablation studies focusing on the estimation methods of  $\mathbb{E}_{q_t(x_t)}[\epsilon_\theta^t (x_t)]$
Estimating  $\mathbb{E}_{q_t(x_t)}[\pmb{\epsilon}_\theta^t (x_t)]$  with partial training data. In the post-training calibration setting, our primary algorithmic change is to subtract the calibration term  $\mathbb{E}_{q_t(x_t)}[\pmb{\epsilon}_\theta^t (x_t)]$  from the pretrained DPMs' output. In the aforementioned studies, the expectation in  $\mathbb{E}_{q_t(x_t)}[\pmb{\epsilon}_\theta^t (x_t)]$  (or its variant of other model parametrizations) is approximated with MC estimation using all training images. However, there may be situations where training data are (partially) inaccessible. To evaluate the effectiveness of our method under these cases, we examine the number of training images used to estimate the calibration term on CIFAR-10. To determine the quality of the estimated calibration term, we sample from the calibrated models using a 3-order DPM-Solver running for 20 steps and evaluate the corresponding FID score. The results are listed in the left part of Table 4. As observed, we need to use the majority of training images (at least  $\geq 20,000$ ) to estimate the calibration term. We deduce that this is because the CIFAR-10 images are rich in diversity, necessitating a non-trivial number of training images to cover the various modes and produce a nearly unbiased calibration term.
Estimating  $\mathbb{E}_{q_t(x_t)}[\epsilon_\theta^t (x_t)]$  with generated data. In the most extreme case where we do not have access to any training data (e.g., due to privacy concerns), we could still estimate the expectation over  $q_{t}(x_{t})$  with data generated from  $p_0^{\mathrm{ODE}}(x_0;\theta)$  or  $p_0^{\mathrm{SDE}}(x_0;\theta)$ . Specifically, under the hypothesis that  $p_0^{\mathrm{ODE}}(x_0;\theta)\approx q_0(x_0)$  (DPM-Solver is an ODE-based sampler), we first generate  $\widetilde{x}_0\sim p_0^{\mathrm{ODE}}(x_0;\theta)$
Table 4: Sample quality varies w.r.t. the number of training images (left part) and generated images (right part) used to estimate the calibration term on CIFAR-10. In the generated data case, the images used to estimate the calibration term  $\mathbb{E}_{q_t(x_t)}[\epsilon_\theta^t (x_t)]$  is crafted with 50 sampling steps by a 3-order DPM-Solver.
Training data	Generated data
# of samples	FID ↓	# of samples	FID ↓
500	55.38	2,000	8.80
1,000	18.72	5,000	4.53
2,000	8.05	10,000	3.78
5,000	4.31	20,000	3.31
10,000	3.47	50,000	3.46
20,000	3.25	100,000	3.47
50,000	3.32	200,000	3.46
Figure 4: Dynamically recording  $\mathbb{E}_{q_t(x_t)}[\epsilon_\theta^t (x_t)]$ . During training, the mean square error between the ground truth and the outputs of a shallow network for recording the calibration terms rapidly decreases, across different timesteps  $t$ .
and construct  $\widetilde{x}_t = \alpha_t\widetilde{x}_0 + \sigma_t\epsilon$ , where  $\widetilde{x}_t \sim p_t^{\mathrm{ODE}}(x_t; \theta)$ . Then, the expectation over  $q_t(x_t)$  could be approximated by the expectation over  $p_t^{\mathrm{ODE}}(x_t; \theta)$ .
Empirically, on the CIFAR-10 dataset, we adopt a 3-order DPM-Solver to generate a set of samples from the pretrained model of Ho et al. [17], using a relatively large number of sampling steps (e.g., 50 steps). This set of generated data is used to calculate the calibration term  $\mathbb{E}_{q_t(x_t)}[\epsilon_\theta^t (x_t)]$ . Then, we obtain the calibrated model  $\pmb{\epsilon}_{\theta}^{t}(x_{t}) - \mathbb{E}_{q_{t}(x_{t})}\left[\pmb{\epsilon}_{\theta}^{t}(x_{t})\right]$  and craft new images based on a 3-order 20-step DPM-Solver. In the right part of Table 4, we present the results of an empirical investigation into how the number of generated images influences the quality of model calibration.
Using the same sampling setting, we also provide two reference points: 1) the originally mis-calibrated model can reach the FID score of 3.89, and 2) the model calibrated with training data can reach the FID score of 3.32. Comparing these results reveals that the DPM calibrated with a large number of high-quality generations can achieve comparable FID scores to those calibrated with training samples (see the result of using 20,000 generated images). Additionally, it appears that using more generations is not advantageous. This may be because the generations from DPMs, despite being known to cover diverse modes, still exhibit semantic redundancy and deviate slightly from the data distribution.
Dynamical recording. We simulate the proposed dynamical recording technique. Specifically, we use a 3-layer MLP of width 512 to parameterize the aforementioned network  $h_{\phi}(t)$  and train it with an Adam optimizer [24] to approximate the expected predicted noises  $\mathbb{E}_{q_t(x_t)}[\pmb{\epsilon}_\theta^t (x_t)]$ , where  $\pmb{\epsilon}_{\theta}^{t}(x_{t})$  comes from the pretrained noise prediction model on CIFAR-10 [17]. The training of  $h_\phi (t)$  runs for 1,000 epochs. Meanwhile, using the training data, we compute the expected predicted noises with MC estimation and treat them as the ground truth. In Figure 4, we compare them to the outputs of  $h_{\phi}(t)$  and visualize the disparity measured by mean square error. As demonstrated, as the number of training epochs increases, the network  $h_{\phi}(t)$  quickly converges and can form a relatively reliable approximation to the ground truth. Dynamic recording has a distinct advantage of being able to be performed during the training of DPMs to enable immediate generation. We clarify that better timestep embedding techniques and NN architectures can improve approximation quality even further.
We propose a straightforward method for calibrating any pretrained DPM that can provably reduce the values of SM objectives and, as a result, induce higher values of lower bounds for model likelihood. We demonstrate that the mis-calibration of DPMs may be inherent due to the dataset bias and/or sub-optimally learned model scores. Our findings also provide a potentially new metric for assessing a diffusion model by its degree of "uncalibration", namely, how far the learned scores deviate from the essential properties (e.g., the expected data scores should be zero).