On Calibrating Diffusion Probabilistic Models
Authors: Tianyu Pang
Cheng Lu
Chao Du
Min Lin
Shuicheng YAN
Zhijie Deng
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion Probabilistic Models
Model Calibration
TL;DR: We propose a straightforward method for calibrating diffusion probabilistic models that reduces the values of SM objectives and increases model likelihood lower bounds.

Abstract: Recently, diffusion probabilistic models (DPMs) have achieved promising results in diverse generative tasks. A typical DPM framework includes a forward process that gradually diffuses the data distribution and a reverse process that recovers the data distribution from time-dependent data scores. In this work, we observe that the stochastic reverse process of data scores is a martingale, from which concentration bounds and the optional stopping theorem for data scores can be derived. Then, we discover a simple way for calibrating an arbitrary pretrained DPM, with which the score matching loss can be reduced and the lower bounds of model likelihood can consequently be increased. We provide general calibration guidelines under various model parametrizations. Our calibration method is performed only once and the resulting models can be used repeatedly for sampling. We conduct experiments on multiple datasets to empirically validate our proposal. Our code is available at https://github.com/thudzj/Calibrated-DPMs.

On Calibrating Diffusion Probabilistic Models
Tianyu Pang",, Cheng Lu², Chao Du', Min Shuicheng Yan', Zhijie Deng
Seaa AI Lab, Singapore
2 Department of Computer Science, Tsinghua University
3 Qing Yuan Research Institute, Shanghai Jiao Tong University
(tianyupang, duchao, limmin, (anac)esee.ccoom com; lucheng. 1c150gmai1..... com; zhijiedde tu. edu.
Abstract
Recently, diffusion probabilistic models (DPMs) have achieved promising results in diverse generative tasks. A typical DPM framework includes forward process that gradually diffuses the data distribution and a reverse process that recovers the data distribution from time-dependent data scores. In this work, we observe that the stochastic reverse process of data scores is martingale, from which concentration bounds and the optional stopping theorem for data scores can be derived. Then, we discover simple way for calibrating an arbitrary pretrained DPM, with which the score matching loss can be reduced and the lower bounds of model likelihood can consequently be increased. We provide general calibration guidelines under various model parametrizations. Our calibration method is performed only once and the resulting models can be used repeatedly for sampling. We conduct experiments on multiple datasets to empirically validate our proposal. Our code is available https://github.com wttpg//iwwbooottuui/aaiaiiiiaaea Catttudii//iibbattd-DPSs..
Introduction
In the past few years, denoising diffusion probabilistic modeling [17, and score-based Langevin dynamics [42, 43] have demonstrated appealing results on generating images. Later, Song et al. [46] unify these two generative learning mechanisms through stochastic/ordinary differential equations (SDEs/ODEs). In the following we refer to this unified model family as diffusion probabilistic models (DPMs). The emerging success of DPMs has attracted broad interest in downstream applications, including image generation [10, 22, 48], shape generation [4], video generation [18, 19]. super- resolution [35], speech synthesis [5], graph generation [51]. textual inversion [13, 34], improving adversarial robustness [50], and text-to-image large models [32, 33J, just to name few.
A typical framework of DPMs involves forrwadd process gradually diffusing the data distribution (z())) towards noise distribution (xT)) The transition probability for E [0, T'I is conditional Gaussian distribution gor (2x (ro) (2; (0,ato, Tf), where 02, E RR Song et al. [46] show that there exist reverse SDE/ODE, processes starting from (r(TT) and sharing the same marginal distributions ge (xx)) as the forward process. The only unknown term in the reverse processes is the data score V., log qe (Iz), which can be approximated by a time-dependent score model 8) (z;) (or with other model parametrizations), (z,) is typically learned via score matching (SM) [20].
In this work, we observe that the stochastic process of the scaled data score as Va, log (xx.) is martingale w.r.t. the reverse-time process of x; from T to O, where the timestep can be either continuous or discrete. Along the reverse-time sampling path, this martingale property leads to concentration bounds for scaled data scores. Moreover, a martingale satisfies the optional stopping theorem that the expected value at a stopping time is equal to initial expected value.
Corresponning authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Based on the martingale property of data scores, for any t E [0.T] and any pretrained score model (x²) (or with other model parametrizations), we can calibrate the model by subtracting its expectation over qe (2xx, i.e., (..) (=x)]] We formally demonstrate that the calibrated score model (5(2)) (x.) (xx)) achieves lower values of SM objectives. By the connections between SM objectives and model likelihood of the SDE process [23, 45] or the ODE process [28], the calibrated score model has higher evidence lower bounds. Similar conclusions also hold for the conditional case, which we calibrate a conditional score model (xe, by subtracting its
conditional expectation (xslp) (z).
In practice, E (s;) [xt)] or (g(((r))v)) (1,, y)] can be approximated using noisy training data when the score model has been pretrained. We can also utilize an auxiliary shallow model to estimate these expectations dynamically during pretraining. When we do not have access to training data, we could calculate the expectations using data generated from s (2x) or (3r,. y). In experiments, we evaluate our calibration tricks on the CIFAR-10 [25] and CelebA 64 64 [27] datasets, reporting the FID scores 6]. We also provide insightful visualization results on the AFHQv2 [7], FFHQ [21] and ImageNet [9] at 64 x 64 resolution.
2 Diffusion probabilistic models
In this section, we briefly review the notations and training paradigms used in diffusion probabilistic models (DPMs). While recent works develop DPMs based on general corruptions [2, 8], we mainly focus conventional Gaussian-basse -base DPMs.
2.1 Forward and reverse processes
We consider k-dimennional random variable x E IR and define forward diffusion process on æ as {#t (e(oT)) with T > 0, which satisfies Vt E [),
(zo), (nr(Fz/ = N(x 0,,00 o;
Here (ro) is the data distribution; ag and oe are two positive real-valued functions that are differentiable w.r.t. t with bounded derivatives. Let (x(x)) = f goi (z (ooo)=/)0o(a (roo)) )dzo be the marginal distribution of zz. The schedules of a₁, of need to ensure that (r(IT) se [(rrT(,, 00,0000 I²1) I) for some
5. Kingma et al. [23] prove that there exists a stochastic differential equation (SDE) satisfying the forward transition distribution in Eq. (1), and this SDE can be written as
dx; f dt) + g(t)dwr,
where w, E R is the standard Wiener process, (t) d and g()) doy dt
2 o log di of Song de
di
et al. [46] demonstrate that the forward SDE in Eq. (2) corresponds to reverse SDE constructed as dr. [f))); 2 log (xx)) dt + g(t)
where W. E R is the standard Wiener process in reverse time. Starting from or (#rr) the marginal distribution of the reverse SDE process is also qe (xz) for t E [0, 7]. There also exists a deterministic process described by ordinary differential equation (ODE)
dx.
dt f((),; 2 (1) log (xx).
which starts from qr (zT) and shares the same marginal distribution (xx) as the reverse SDE in Eq. (3). Moreover, let 4or (x,) = [o((a]( and by Tweedie's formula [12], we know that ar goo
g((x,)
[mo] = It + VfV v, log (zz).
2.2 Training paradigm of DPMs
To estimate the data score V.., log qu (are) at timestep a score-based model s (xxx) [46] with shared parameters 0 is trained to minimize the score matching (SM) objective [20] as
(0) A E (x.) (z)) V log qi (zx)
(5)
To eliminate the intractable computation of V.,, log ga (Iz). denoising score matching (DSM) [49]
transforms (0) into (0) A 1 (go
5 (xx) + - where It = a; + are and

---

= is standard Gaussian distribution. Under mild boundary conditions, we know (0) and Josm(() (0) equivalent up to constant, i... (0) Jos (0) + C* and C* is constant independent of the model parameters 0. Other SM variants [31, 44] are also applicable here. The total SM objective for training is a weighted sum of JSM (0) across [o, T.. defined as (0; A(()) So SM))TM (O)d1, where X(1) is a positive weighting function. Similarly, the total DSM objective is JDSM A(t) & So TisM (f)dt. The training objectives under other model parametrizations such as noise prediction (z4) [17, 333, data prediction (xx) [23, and velocity prediction (x+) [18, 38] are recapped in Appendix B.I.
2.3 Likelihood of DPMs
Suppose that the reverse processes start from a tractable prior pr( - (x = 022 We can approximate the reverse-time SDE process by substituting Va, log (e with s/ (1re) in Eq. (3) as dx. [f(()== g()) s/ (==)] dt + g(t) which induces the marginal distribution pt SDD (1>; 0) for t E [0, T" In particular, t 0, the KL divergence between (x(o) and ph
yyDR/ (Ir;; 6) can be bounded
by the total SM objective Jsm(0; (t))² with the weighing function of g(t)?, ", as stated below:
Lemma 1. (Proof in Song et al. [45]) Let (xx) be constructed from the forward process in Eq. (2).
Then under regularity conditions, we have DxL (go (0ll S Jsu(O; g(t)²) + DxL (grller).
Here DRL. (yr lPP)) is the prior loss independent of 0. Similarly, we approximate the reverse-time ODE process by substituting aa log (z(ez) with (xx) ) in Eq. (4) as dz, = f(t)z, s((t))(((z)) which induces the marginal distribution p 8) for t [0, T.. By the instantaneous change 21(())²/((<)))
of variables formula [6], we have log (2.:0)
tr (Vz, (J(()x,
where tr(-)
di
denotes the trace of a matrix. Integrating change in log ( 0) from t = 0 to T' can give the p (mo; f). but requires tracking the path from mo to TT- On the other
value of log (xrr)) - log log hand, at L = 0, the KL divergence between go and po po 0) can be decomposed:
Lemma 2. (Proof in Lu et al. [28]) Let (e(x)) be constructed from the forward process in Eq. (2). Then under regularity conditions, we have DEL (qullPPPE(0)) (0)) Ism g()) + DxL (qr (DD((0, where the term (D) measures the difference between (xx) and Vz. log p (x O).
Directly computing (0) is intractable due to the term Va, logpt (xx;; ()) nevertheless, we could bound JoHt (0) via bounding high-order SM objectives [28].
3 Calibrating pretrained DPMs
In this section we begin with deriving the relationship between data scores at different timesteps. which leads us to a straightforward method for calibrating any pretrained DPMs. We investigate further how the dataset bias of finite samples prevents empirical learning from achieving calibration.
3.1 The stochastic process of data score
According to Kingma et al. [23], the form of the forward process in Eq. (1) can be generalized to any two timesteps 0 < s (=)()))))) = . (ze Cy of where agss = as and o²,- = of - a?io?? Here the marginal distribution satisfies (x²) = (1.(xell))q
< t < T. Then, the transition probability from x, to is written as
We can generally derive the connection between data scores Va, log (z)1) and Vz, (xx) stated below:
Theorem 1. (Proof in Appendix 1) Let (zx) be constructed from the forward process in Eq. (2).
Then under some regularity conditions, we have vO S <T
Or Va, log (x(e) (m)) Ja,, V., log qu (x+)).
(6)
where (xL(2)))))
Ruelate lx, (9s(t)) is the transition probability from Xa to x.. ((()))
Theorem indicates that the stochastic process o Vz, log q (I:) is a martingale w.r.t. the reverse- time process of De from timestep T to O. From the optional stopping theorem [14], the expected
3

---

value of a martingale at stopping time is equal to its initial expected value (oo( IV zo log go (To) It is known that, under a mild boundary condition on go (2o). there is E (zu() [V z0 log qo (o(z))] = 0 (proof is recapped in Appendix A.2). Consequently, as the stochastic process, the martingale property results in E (mx)) 1V,, log qe (T+)] = O for Vt E [0, 2'.. Moreover, the martingale property of the (scaled) data score or Va, log qt (xx) leads to concentration bounds using Azuma's inequality and Doob's 's martingale inequality as derived in Appendix A.3. Although we do not use these concentration bounds further in this paper, there are other concurrent works that use roughly similar concentration bounds in diffusion models, such as proving consistency [47] or justifying trajectory retrieval [52].
3.2 A simple calibration trick
Given a pretrained model si (az)) in practice, there is usually (m.) [x))) # O, despite the fact that the expect data score is zero as (o.) n.,, log (x(xx)] = This motivates us calibrate (x,)) to s/ (1ze) T], where 7/t is a time-dependent calibration term that is independent of any particular input 24.- The calibrated SM objective is written as follows:
1
JSM (0, nt)
(es(())) (11e) T) Vz, log (1+))
2
JSM (0) (xx) (1a) T Th + 11
where the second equation holds after the results of (.x) VZ,, log (=x)) = 0, and there is JSM (f, 0)- JSM (0) specifically when 7). = O. Note that the orange part in Eq. (7) is quadratic function w.r.t. 7/-- We look for the optimal vyū = arg min,, (0, ne) that minimizes the calibrated SM objective, from which we can derive
Th
= Eg [x]()))]
(8)
After taking oi into JSM nt), ; we have
Jsa (0, nt) [S)
[s] 1 Bitter
1
(x:))]]
(9)
2
Since there is (0) = (0) + CH, we have JsSM (0, n))) TSM (0) 2 E (m.) [
for the DSM objective. Similar calibration tricks are also valid under other model parametrizations and SM variants, as formally described in Appendix B.2.
Remark. For any pretrained score model s (zx), we can calibrate it into (fxx)) - (x)) [s] (xt)).
which reduces the SM//DMM objectives timestep by 1 [E (z.) [s) The expectation of the calibrated score model is always zero, i.e., Bg.( [s] (z:)) - ((.) (xx)] = holds for any 0, which is consistent with (x.) ,V,, log qi (z+)) satisfied by data scores.
Calibration preserves conservativeness. A theoretical flaw of basre- modeling is that (1ra) may not correspond to a probability distribution. To solve this issue, Salimans and Ho [37] develop an energy-based model design, which utilizes the power of score-based modeling and simultaneously makes sure that s (xx)) is conservative, i.e., there exists probability distribution (1x)) such that Vx, E RR* we have s/ (It) = Va, log logr' (at). In this case, after we calibrate s/ (2+e) by subtracting
T/, there is (xx) 17p = Va, log exp(x,To n. (o) where (0) = f p& (z:) exp (-x, m)) dr.
((.)))
represents the normalization factor. Intuitively, subtracting by q/e corresponds
shift in the vector space, so if s (xx)) is conservative, its calibrated version (xx) - JJt is also conservative.
Conditional cases. As to the conditional DPMs, we usually employ conditional model (2,, v). where E y' is the conditional context (e.g., class label or text prompt). To learn the con- ditional data score Va, log ge (= = V., a, log (e v), we minimize the SM objective de- fined (0) A (, (Fea)) [[)s]
Vz, log qe (22, 3]]] Similar to the conclusion of Exge [V,, log ge (are) = O, there is (ostt)) [ log qe (mely)) = 0. To calibrate si (Ix, y), use the conditional term ne (v) and the calibrated SM objective is formulated as
(8, (x()) A 1
2 F (as.))) [1ls) v) (e(v) Vz, log a (F2, 12
(10)
= = (0)-Bgi(rr)) (6( v) (v) + 2 (x)

---

and for any y E V, the optimal n/ (b) is given by ni (3) = (mile) [2) v)]. We highlight the conditional context in contrast to the unconditional form in Eq. (7). After taking ni (v) into (0, (z)),, we have [S (f)) = (f) - (w) (o (z) w)ll This conditional calibration form can naturally generalize other model parametrizations and SM variants.
3.3 Likelihood of calibrated DPMs
Now we discuss the effects of calibration on model likelihood. Following the notations in Section 2.3, we use P (0, .T.) and (0, mh) to denote the distributions induced by the reverse-time SDE and ODE processes, respectively, where Va, log (14) is substituted with s)(x)) T)-.
Likelihood of (0, nt). Let [f, ne (()))) = Sn' g(1)² [M, me dt be the total SM objective after the score model is calibrated by nu then according to Lemma 1, we have DEL (gollpd (0, (.)m)) - TSM (0, nh5 (t)))) + DKL (er (Pr)). From the result in Eq. (9), there is
/T (() |B [s)( 1 ! Js, ni: (()))) Jsm Jsm(O:g(t))) 2
(11)
Therefore, the likelihood DRL. (golledDE (0, (()) after calibration has a lower upper bound of (0, n g{)) + DKL (qrlppr), compared to the bound of g + DRL (qrlprr) for the original DRL (40 (llg) However, we need to clarify that DRL (follr (Ee (()) may not necessarily smaller than DKi. (o))) since we can only compare their upper bounds.
Likelihood of Po (0, nt). Note that in Lemma 2, there is term (0), which is usually small in practice since (xx) and V., log PEDE (Iz; 8) are close. Thus, we have
DRL. Alollo (0,, (0, s Jsm(0,, mhi (())(()))))))))))))()))))) + DKI. (sT(leT).
and DKL (gollpg DollpoDE(0 (0, nt)) approximately achieves its lowest value. Lu et al. [28] show that DKL (gollrd DDE (0)) can be further bounded by high-order SM objectives (as detailed in Appendix A.4), which depend on Va, (1rz) and V., tr (Vz, s/ (zr)). Since the calibration term 7/ is independent of tr i.e. V,, M/ = O, it does not affect the values of high-order SM objectives, and achieves a lower upper bound due to the lower value of the first-order SM objective.
3.4 Empirical learning fails to achieve (x. (s) = 0
question that naturally arises is whether better architectures or learning algorithms for DPMs (e.g., EDMs [22]) could empirically achieve (..) [xt)] = O without calibration? The answer may be negative, since in practice we only have access to a finite dataset sampled from go(x)) More specifically, assuming that we have training dataset ID &
where xb ~ (o())) and defining the kernel density distribution induced by ID as q (2.; D) & ENEW 24-0:4ã (==-ar00 I
When the quantity of training data approaches infinity, we have lim 9e (e( D) = qe (e())) holds for Vt E T). Then the empirical DSM objective trained on D is written
N
JBSM (0; DD)
A
1 2N
Σ
By(e)
8 (ara" + oxe) +
(12)
and it is easy to show that the optimal solution for minimizing (is DD) satisfies (assuming s/ has universal model capacity) (xx) = V., log (1;; D). Given a finite dataset D, there is
E (x;() [V,, log (zy; [D)] = 0, but typically Eg((se) [V, log (= ID)] + O,
(13)
indicating that even the score model is learned to be optimal, there is still (xx) [s] (xx)) ) * O.
Thus, the mis-calibration of DPMs is partially due the dataset bias, i.e. during training we can only access a finite dataset D sampled from qo(2)))
Furthermore, when trained on finite dataset in practice, the learned model will not converge to the optimal solution [15], so there is typically s (x,) # Vz, log (m,; D) and (x,,,) [s] (2x) * O.
After calibration, we can at least guarantee that (.y (x,) - E (z, [s] (xx))] = 0 always holds on any finite dataset D. In Figure we demonstrate that even statc-of-the-ar EDMs still have non-zero and semantic E,,(..) (xx)), which emphasises the significance of calibrating DPMs.
5

---

CIFAR-10
CelebA 64x64
AFRIQ22 64x64
FFHQ 64x64
ImageNet 64xc64
(1000)
0.12
R Edlt vill
1.000
0.80
0.05
0.50
8,06
0.000
0.06
0025
0.055
0.20
240
500
1000
200
at3
GCD
s00
1000
0.0
0.2
0.4
0.5
1.0
(U0
0.2
0.4
08
0.5
10
0.0
0.2
0.6
0.6
0.5
1.
Timasipp
Timsdep
Thrrespp
Thresdap
Timeslop
400
100
600
300
300
300
200
100
100
100
300 Transoop
0.6
100
"Iznodop
Tinostep
Threstop
Figure 1: TTieeddepeddett values of 1 g...)) (=x()) 1I (the first row) and 9(1)²
(xx) [+] (at)) 1
2n
(the second row) calculated different datasets. The models CIFAR-10 and CelebA is trained on discrete timesteps (t = 0, 1, .......................... 1000), while those on AFHQv2, FFHQ. and ImageNet are trained on continuous timesteps
(xz) [0, 1)). We convert data prediction (x²))) into noise prediction eţ (1r) based on eg (zy) = (2)- o The y-axis is clamped into [0, 500].
3.5 Amortized computation of (xa) (xx)]
By default, we are able to calculate and restore the value of (x,( [2,)) for pretrained model % (axx, where the selection of timestep / is determined by the inference algorithm, and the expectation over (e(xx.) can be approximated by Monte Carlo sampling from noisy training set. When we do not have access to training data, from (xx; 0)) or pt SSDE (xx;;) Since we only need to calculate (g [ (2,) once, the raised computational overhead is amortized as the number of generated samples increases.
approximate the expectation using data generated
Dynamically recording. the preceding context, we focus primarily on post- post-training computing of (mr)) [s (xx)) An alternative strategy would be to dynamically record (x, [s) (zx)] during the pretraining phase of sf(x).. Specifically, we could construct an auxiliary shallow network (t) parameterized by d, whose input the timestep t. We define the expected mean squared error as
(o) A Eq (m:) (1) s}(x)))
(14)
where the superscript denotes the stopping gradient and d² is the optimal solution of minimizing (+) w.r.t. o, satisfying has (t)
E (wx) [=x()] (assuming sufficient model capacity).
The total training objective can therefore be expressed as [O: A(t)) + So Be Tea(o) where 3e is a time-dependent trade-off coefficient for / E [0,7].
Experiments
In this section, we demonstrate that sample quality and model likelihood can be both improved by calibrating DPMs. Instead of establishing new state-of-the-arr the purpose of our empirical studies to testify the efficacy our calibration technique as a simple way to repair DPMs.
4.1 Sample quality
Setup. We apply post-training calibration to discrete-time models trained on CIFAR-10 10 [25] and CelebA [27], which apply parametrization of noise prediction c/ ). In the sampling phase, we employ DPM-Solver [29], an ODE-based based sampler that achieves promising balance between sample efficiency and image quality. Because our calibration directly acts on model scores, is also compatible with other ODE/SDE- samplers [3, 26], while we only focus on DPM-Solver cases in this paper. accordance with the recommendation, we set the end time of DPM-Solver to 10-3 when the number of sampling steps is less than 15, and to 10-1 otherwise. Additional details can be

---

Table 1: Comparison sample quality measured by FID + with varying NFE CIFAR-10.
Experiments are conducted using a linear noise schedule on the discrete-time model from [17]. We consider three variants of DPM-Solver with different orders. The results with i mean the actual NFE is order x
NFF which is smaller than the given NFE, following the setting in [29].
order
Number of evaluations (NFE)
Noise prediction
DPM-Solver
10
15
20
25
30
35
40
1-order
20.49
12.47
9.72
7.89
6.84
6.22
5.75
(12e)
2-order
7.35
14.52
4.14
13.92
3.74
13.71
3.68
3-order
12396
4.61
13889
13.73
3.65
13.65
13.60
1-order
19.31
11.77
8.86
7.35
6.28
5.76
5.36
(14) - Byc((e) (x+(
2-order
6.76
14.36
4.03
13.66
3.54
13.44
3,48
3-order
53.50
4.22
1..22
33.33
3.35
13.32
1..1
Table 2: Comparison on sample quality measured by FID 4 with varying NFE on CelebA 64x64.
Experiments are conducted using a linear noise schedule on the discrete- model from [41]. The settings of DPM-Solver are the same as on CIFAR-1O.
Number of evaluations (NFE)
Noise prediction
DPM-Solver 10
15
20
25
30
35
40
1-order
16.74
11.85
7.93
6.67
5.90
5.38
5,01
(2x)
2-order
4.32
13.98
2.94
12.88
2.88
12.88
2.84
3-order
$11.92
3.91
t2.84
12.76
2.82
12.81
t2.5
1-order
16.13
11.29
7.09
6.06
5.28
4.87
4.39
(24) Bae(or) (xx))
2-order
4.42
f3.94
2.61
12.66
2.54
12.52
2.49
3-order
135.47
3.62
12.33
12.43
2.40
12.43
12.49
found in Lu et al. [29]. By default, employ the FID score [16] quantify the sample quality using 50,000 samples. Typically, a lower FID indicates a higher sample quality. In addition, Table 3, we evaluate using other metries such sFID [30], IS [39], and Precision/Recall [36].
Computing (x.) (xe)). To estimate the expectation over qe (xxe), we construct 2, a, + oiC. where 20 (00)) is sampled from the training set and (c I) is sampled from standard Gaussian distribution. The selection of timestep t depends on the sampling schedule of DPM-Solver. The computed values of (x.) [ [2 are restored in dictionary and warped into the output layers of DPMs, allowing existing inference pipelines to be reused.
We first calibrate the model trained by Ho et al. [17] on the CIFAR- 10 dataset and compare it to the original one for sampling with DPM-Solvers. We conduct a systematical study with varying NFE (i.e.. number of function evaluations) and solver order. The results are presented in Tables 1 and 3. After calibrating the model, the sample quality is consistently enhanced, which demonstrates the significance of doing and the efficacy of method. We highlight the significant improvement in sample quality (4.61 - when using 15 NFE and 3-order DPM-Solver; 3.89-3.332 when using 20 NFE and 3-order DPM-Solver). After model calibration, the number of steps required achieve convergence for a 3-order DPM-Solver is reduced from 230 to 20, making our method a new option for expediting the sampling of DPMs. In addition, as a point of comparison, the 3-order DPM-Solver with 1,000 NFE can only yield an FID score of 3.45 when using the original model, which, along with the results in Table indicates that model calibration helps to improve the convergence of sampling.
Then, we conduct experiments with the discrete-time time model trained on the CelebA 64x64 dataset by Song al. [41]. The corresponding sample quality comparison is shown in Table 2. Clearly, model calibration brings significant gains (3.91 -3.62 when using 15 NFE and 3-order DPM-Solver; 2.83 when using 20 NFE and 3-order DPM-Solver) that are consistent with those the CIFAR-I0 dataset. This demonstrates the prevalence of the mis-calibration issue in existing DPMs and the efficacy of our correction. We still observe that model calibration improves convergence of sampling, and as shown in Figure 2, our calibration could help to reduce ambiguous generations. More generated images are displayed in Appendix C.
7

---

Table 3: Comparison on sample quality measured by different metrics, including FID sFID inception score (IS) t. precision t and recall f with varying NFE on CIFAR-10. We use Base to denote the baseline (22)) and Ours to denote calibrated score (xa) (ox) ) (x))) The sampler is DPM-Solver with different orders. Note that FID is computed by the PyTorch checkpoint of Inception-v3, while SFIDD/SS/Precision/Ree Inception-v3 following
are computed by the Tensorflow checkpoint of
Number of evaluations (NFE)
Method
20 IS
25
30
FID sFID
Pre.
Rec.
FID sFID
IS
Pre.
Rec.
FID
sFID
IS
Pre.
Rec.
1-ord.
99722 6.03 8.49 0.641 0.542 7.89 5.45 8.68
0.644 0.556 6.84 5.12 8.76 0,650 0.565
Base 2-ord.
4.14 4.36 9.15 0.654 0.590 3.92
4.22 9.17 0.657 0.591
3.74
4.18 9.20 0.658 0.591
3-ord.
3.89 4.18 9.29 0.652 0.597
3.73
4.15 9.21 0.657 0.595
3.65
4.12 9.22 0.658 0.593
1-ord.
8.86 6.01 8.56 0.649 0.544 7.35 5.42
8.76 0.653 0.560
6.28 5.09
8.84 0.653 0.568
Ours 2-ord.
4.03 4.31 9.17 0.661 0.592
3.66
4.20 9.20 0.664 0.594
3.54
4.14 9.23 0.662 0.599
3-ord.
3.32 4.14 9.38 0.657 0.603
3.33
4.11 9.28 0.665 0.597
3.35
4.08
9.27 0.662 0.600
w/o calibration (baseline)
w/ calibration (ours)
Figure 2: Selected images on CIFAR-10 (generated with NFE 20 using 3-order DPM-Solver) demonstrating that our calibration could reduce ambiguous generations, such as generations that resemble both horse and dog. However, we must emphasize that not all generated images have visually discernible difference before and after calibration.
4.2 Model likelihood
As described in Section 3.3, calibration contributes to reducing the SM objective, thereby decreasing the upper bound of the KI. divergence between model distribution at timestep & = 0 (either (iDDE(0, nf or pODE (0, ni)) )) and data distribution Consequently, it aids in raising the lower bound of model likelihood. In this subsection, we examine such effects by evaluating the aforementioned DPMs on the CIFAR-10 10 and CelebA datasets. We also conduct experiments with continuous- time models trained by Karras et al. [22] on AFHQv2 64 x 64 [7], FFHQ 64 x 64 [21], and ImageNet 64 [9] datasets considering their top performance. These models apply parametrization of data prediction ai (art), and for consistency, we convert to align with (a;) based on the relationship c/ (are) =
(I2a o (2)))/ as detailed in Kingma et al. [23] and Appendix B.2.
Given that we employ noise prediction models in practice, we first estimate (() (c)
||3 at
timestep t [o. 77. which reflects the decrement on the SM objective at according to Eq. (9) (up to scaling factor of 1/o?). We approximate the expectation using Monte Carlo (MC) estimation with training data points. The results are displayed in the first row of Figure 1. Notably, the value of (e.)). (x:)) 112 varies significantly along with timestep t: it decreases relative to & for CelebA but increases in all other cases (except for i E (0.4, 1.0] on ImageNet 64 64), Ideally, there should be 1 (( [Vz, log ge (zx)) 1/ - at any l. Such inconsistency reveals that mis-calibration issues exist in general, although the phenomenon may vary across datasets and training mechanisms.
Then, we quantify the gain of model calibration on increasing the lower bound of model likelihood. which is 1 5_66 (()² [| (xx) [s] (<x))]] 2 dt according to Eq. (11). We first rewrite it with the model parametrization of noise prediction cl( and it can be straightforwardly demonstrated that it equals
Jo a(t) 20 ([x). (c)( [|.. Therefore, we calculate the value of g(t)²
207 [[ (wx) (c) [l using MC
8

---

CelebA 64x64
FFHQ FFHQ 64x64
Timestep
Figure 3: Visualization of the expected predicted noises with increasing t. For each dataset, the first row displays (ix) (2x)] (after normalization) and the second row highlights the top-10% pixels that (x,) [e] (xx) has high values. The DPM on CelebA is a discrete-time model with 1000 timesteps [41] and that on FFHQ is continuous-timg one [22].
estimation and report the results in the second row of Figure 1. The integral is represented by the area under the curve (i.e., the gain of model calibration on the lower bound of model likelihood). Various datasets and model architectures exhibit non-trivial gains, observed. In addition, we notice that the DPMs trained by Karras et al. [22] show patterns distinct from those of DDPM [17] and DDIM [41], indicating that different DPM training mechanisms may result in different mis cii-caaibratiin effects.
Visualizing Egg (z,) [ (z. To better understand the inductive bias learned by DPMs, we visualize the expected predicted noises (xx) [c] (3+)) for timestep from 0 to as seen in Figure 3. For each dataset, the first row normalizes the values of Eg (z,) [<) (2t) into 255]: the second row calculates pixel-wise norm (across RGB channels) and highlights the top-10% locations with the highest norm. As we can observe, on facial datasets like CelebA and FFHQ, there are obvious facial patterns inside (x,( [E] (ar,). while on other datasets like CIFAR-IO, ImageNet, as well the animal face dataset AFHQv2, the patterns inside (x.) 1c (zx) are more like random noises. Besides, the facial patterns in Figure 3 are more sigmificant when t is smaller, and become blurry when is close to T. This phenomenon may be attributed to the bias of finite training data, which is detrimental to generalization during sampling and justifies the importance of calibration as described in Section 3.4.
4.3 Ablation studies
We conduct ablation studies focusing on the estimation methods of (xx) (x:)).
Estimating (ux) (zx)] with partial training data. In the post-training calibration setting, our primary algorithmic change is to subtract the calibration term (x,) [=]( from the pretrained DPMs' output. In the aforementioned studies, the expectation in (x.) [ (xx)] (or its variant of other model parametrizations) is approximated with MC estimation using all training images. However, there may be situations where training data (partially) inaccessible. To evaluate the effectiveness of our method under these cases, we examine the number of training images used to estimate the calibration term on CIFAR-10. To determine the quality of the estimated calibration term, we sample from the calibrated models using a 3-order DPM-Solver running for 20 steps and evaluate the corresponding FID score. The results listed in the left part of Table 4. As observed, we need to use the majority of training images (at least 2 20,000) to estimate the calibration term. We deduce that this is because the CIFAR- 10 images are rich in diversity, necessitating non-trivial number of training images to cover the various modes and produce a nearly unbiased calibration term.
Estimating (x.( (x:)) with generated data. In the most extreme case where do not have access to any training data (e.g., due to privacy concerns), we could still estimate the expectation over qo (222) with data generated from po ODD 0)) or po ( 0). Specifically, under the hypothesis that ODE (zo; 0)
Po (DE(2o; 0) se go (zo) (DPM-Solver is ODE- sampler), first generate To Pu (

---

Table 4: Sample quality varies w.r.t. the ber of training images (left part) and generated images (right part) used to estimate the cali- bration term on CIFAR-10. In the generated data case, the images used to estimate the cali- bration term (xx) (xx)) is crafted with 50 sampling steps by a 3-order DPM-Solver.
Figure 4: Dynamically recording (x.) [ (z))) During training, the mean square error between the ground truth and the outputs of shallow net- work for recording the calibration terms rapidly decreases, across different timesteps t.
0.8
t=0
Training data # of samples
Generated data samples
Mean Square Bror
0.7
250
t=500
FID 4
FID +
0.6
t=750
500
55.38
2,000
8.80
999
0.5
1,000
18.72
5,000
4.53
2,000
8.05
10,000
3.78
0.4
5,000
4.31
20,000
3.31
10,000
3.47
50,000
3.46
0.3
20,000
3.25
100,000
3.47
0.2
50,000
3.32
200,000
3.46
0
200
400
800
800
1000
Training Epoch
and construct Te og + d2C, where T, pt (1 0). Then, the expectation over q (24) could be approximated by the expectation over p; pi (zz;
Empirically, the CIFAR-10 dataset, from the pretrained model of Ho et al. [17], using a relatively large number of sampling steps (e.g.. 50 steps). This set of generated data is used to calculate the calibration term (x.)) (e] (ar Then,
adopt 3-order DPM- to generate set of samples
we obtain the calibrated model e (zz) E (x.)) [ (2+)] and craft new images based on 3-order 20-step DPM-Solver. In the right part of Table 4, we present the results of an empirical investigation into how the number of generated images influences the quality of model calibration.
Using the same sampling setting, we also provide two reference points: 1) the originally calibbabdddted model can reach the FID score of 3.89, and 2) the model calibrated with training data reach the FID score of 3.32. Comparing these results reveals that the DPM calibrated with a large number of high-quality generations can achieve comparable FID scores those calibrated with training samples (see the result of using 20,000 generated images). Additionally, it appears that using more generations is not advantageous. This may be because the generations from DPMs, despite being known cover diverse modes, still exhibit semantic redundancy and deviate slightly from the data distribution.
Dynamical recording. We simulate the proposed dynamical recording technique. Specifically, we use a 3-layer MLP of width 512 to parameterize the aforementioned network (a(()) and train it with an Adam optimizer [24] to upproximate the expected predicted noises (x.) (x))], where (xx)) comes from the pretrained noise prediction model on CIFAR- 10 [17]. The training of ha(() runs for 1,000 epochs. Meanwhile, using the training data, we compute the expected predicted noises with MC estimation and treat them as the ground truth. In Figure 4, we compare them to the outputs of (t) and visualize the disparity measured by mean square error. As demonstrated, as the number of training epochs increases, the network (t) quickly converges and can form relatively reliable approximation to the ground truth. Dynamic recording has a distinct advantage of being able to be performed during the training of DPMs to enable immediate generation. We clarify that better timestep embedding techniques and NN architectures can improve approximation quality even further.
5 Discussion
We propose a straightforward method for calibrating any pretrained DPM that can provably reduce the values of SM objectives and, result, induce higher values of lower bounds for model likelihood. We demonstrate that the mis-calibration of DPMs may be inherent due to the dataset bias and/or sub-optimally learned model scores. Our findings also provide potentially new metric for assessing a diffusion model by its degree of "uncalibration". namely. how far the learned scores deviate from the essential properties (e.g., the expected data scores should be zero).
Limitations. While our calibration method provably improves the model's likelihood, does not necessarily yield lower FID score, previously discussed [45]. Besides, for text-to-image genera- tion, post-training computation of (w. (z),, v)] becomes infeasible due to the exponentially large number of conditions y. necessitating dynamic recording with multimodal modules.
10

---

Acknowledgements
Zhijie Deng was supported by Natural Science Foundation of Shanghai (No. 23ZR1428700 and the Key Research and Development Program of Shandong Province, China (No. 2023CXGC010112)
References
[1] Kazuoki Azuma. Weighted sums of certain dependent random variables. Tohoku Mathematical Journal, Second Series, 19(3):357-377, 1967.
[2] Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.09392, 2022.
[3] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In International Conference on Learning Representationn (ICLR). 2022.
[4] Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In European Conference on Computer Vision (ECCV), 2020.
[5] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations (ICLR), 2021.
[6] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[7] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wor Ha. Stargan v2: Diverse image synthesis for multiple domains. In IEEE International Conference Computer Vision (CVPR), 2020.
[8] Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G Dimakis, and Peyman Milan- far. Soft diffusion: Score matching for general corruptions. arXiv preprint arXiv:2209.08 05442, 2022.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[10] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[11] Joseph L Doob and Joseph L Doob. Stochastic processes, volume 7. Wiley New York, 1953.
[12] Bradley Efron. Tweedie's 's formula and selection bias. Journal of the American Statistical Association, 106(496):602-16114 2011.
[13] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01e. 016118, 2022.
[14] Geoffrey Grimmett and David Stirzaker. Probability and random processes. Oxford university press, 2001.
[15] Xiangming Gu, Chao Du, Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang. On memorization in diffusion models. arXiv preprint arXiv:2310.02664, 2023.
[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS), pages 6626-6637, 2017.
11

---

[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[18] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.0230: 2022.
[19] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprini arXiv:2204. 2234045800 2022.
[20] Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research (JMLR), 6(Apr))695-709, 2005.
[21] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE International Conference on Computer Vision (CVPR), 2019.
[22] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems (NeurlPS), 2022.
[23] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[25] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[26] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations (ICLR), 2022.
[27] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In International Conference on Computer Vision (ICCV), 2015.
[28] Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for score-based diffusion odes by high order denoising score matching, In International Conference on Machine Learning (ICML), 2022.
[29] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[30] Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint arXiv:2103.. 103334,03 2021.
[31] Tianyu Pang, Kun Xu, Chongxuan Li, Yang Song, Stefano Ermon, and Jun Zhu. Efficient learning of generative models via finte-differrece score matching. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[32] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 22X 2022.
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[34] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2008. 12242, 2022.
[35] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022.
12

---

[36] Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assess- ing generative models via precision and recall. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[37] Tim Salimans and Jonathan Ho. Should ebms model the energy or the score? In Energy Based Models Workshop-ICLR, 2021.
[38] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations (ICLR), 2022.
[39] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
[40] Jascha Sohl- Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning (ICML), pages
PMLR, 2015.
[41] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Representations (ICLR), 2021.
[42] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems (NeurIPS), pages 11855- 1 11907, 2019.
[43] Yang Song and Stefano Ermon. Improved techniques for training score- bassed generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[44] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. Conference on Uncertainty in Artificial Intelligence (UAI), 2019.
[45] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[46] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021.
[47] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01466) 2023.
[48] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[49] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-1674 2011.
[50] Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In International Conference on Machine Learning (ICML), 2023.
[51] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: geometric diffusion model for molecular conformation generation. In International Conference on Learning Representations (ICLR), 2022.
[52] Kexun Zhang, Xianjun Yang, William Yang Wang, and Lei Li. Redi: Efficient learning diffusion inference via trajectory retrieval. arXiv preprint arXiv:2302.02285. 2023.
13

---

A Detailed derivations
In this section, we provide detailed derivations for the Theorem and equations shown in the main text. We follow the regularization assumptions listed in Song et al. [45].
A.1 Proof of Theorem
Proof. For any two timesteps 0 S < i.e., the transition probability from x, 2y is written
as (x) = .V (1x (11sss 02/01) )). where Odje= = and of of -aq og. The marginal distribution (x(x)) (xr (x,,) and we have
V ax log (((e))
log
E
les (2s())))
R4
f/s
log
E
((
( (2. + m)
0t/s
n2
IF
1 qo (e It + m)
oejo (1/00000
2s (0 + n)
E
9o (0 2e + v aplletn log qu 1))
(15)
Co (0 / + m)
EE (1 (1o( v., log do (x,))
as/alE V (ollattaa ( (xx))
JN (r aas of I) ao (x.x))a log q (xx)ddr
atls ʃ N 2e c4 I o7 90 ())
Ey Vz, log 4o.))))
Note that when the transition probability gec(2))=)) corresponds to well-defined forward process, there is a₁ > 0 for Vt E [O, and thus we achieve a Vz. log (z,) (q,((,,):)) (a, Vv, log (xxo).
A.2 Proof of E ( Van log 9o (zo) =
Proof. The input variable æ E R* and qo(=a) E C², where C2 denotes the family of functions with continuous second-order derivatives.' We use xf denote the i-th element of x, then we can derive the expectation
i dxi
= {
(xo)
log qo (2o)
go (zo) (0(zz)
Dzh log go (2o)dx ... day
(o( ... dat
axd
(16)
/
90 (2) m) X yan
day
dxb
/
d dag
10( 0.
'This continuously differentiable assumption can be satisfied by adding small Gaussian noise (e.g., with variance of 0.0001) the original data distribution, as done in Song and Ermon [42].
14

---

where 26 denotes all the k: - 1 elements in zо except for the i-th one. The last equation holds under the boundary condition that lim (0(z)) = hold for any E [K]. Thus, we achieve the conclusion that (xo( [vo log 2o (x = O.
Vá
A.3 Concentration bounds
We describe concentration bounds [11, 1] of the martingale or V log (x())
Azuma's inequality. For discrete reverse timestep 1 T ", Assuming that there exist constants 0 < C1, C2, < oo such that for the i-th element of x,
0
A, S
1 log 9t-1 (
a log de (xx)) & B, and B, A Sc at
(17)
δxi-1
drk
almost surely. Then Ve > 0, the probability (note that on = 1)
((
① or log qT (2T) 26 < 2exp
2c²
P
log 40 (2o)
(18)
dxy
Σ1=
c
Specially, considering that @r (IT) N (x010,, I), there is approximately obtain
a Bay.
log (IT)
Thus,
P(12rg Dul log 20 (2o) +
aT
2c
2τ) S 2exp
(19)
Σ
Doob's inequality. For continuous reverse timestep t from to O, the sample paths of the martingale are almost surely right-continuous, then for the i-th element of we have (note that 1)
a OSEET dr,
log de (zt) 2 c)
E Eyu(Ro)
max
azg
log 2o (20),
sup P
(20)
C
A.4 High-order SM objectives
Lu et [28] show that the KI. divergence DKL. (40 11n QDE (0)) can bounded as
DRL. (4o IPo ODE (0)) S DKI. (uTlleT) + / Jsm(0;g(1y) v TFisher (0),
(21)
where Frisher (0) is weighted sum of Fisher divergence between (xxx) and pt ((DE(()
Trisher (0) = L. (" g(t) DE (gellp,
ODE (0)) dt.
(22)
Moreover, Lu et al. [28] prove that if VI E [0, T and Ve, E R there exist constant Cr such that the spectral norm of Hessian matrix 2 such that
2, log ODE (x 0ll S CF, and there exist 61, 82. 83 >
(xx) Vz, log (1:)ll2 S 8,
V s/ (x,) - v2, log qe (x()))) $ 82, VV tr (v,, sx()) - V tr (V2, log (x())) ll2 S 63.
(23)
where -| is the Frobenius norm of matrix. Then there exist function (tt d1, 82, 8z, q) that independent and strictly increasing (if g(t) * 0) w.r.. 8, 82, and d3, respectively, such that the Fisher divergence can be bounded as DF (gellp.
S SU(t; 82, 82, g).
The after calibration. When impose the calibration term of Fyy,(.)) [s) get the score model sy (2xe) nf there is V., = 0 and thus V., (s)( m.) & (x.). Then have
$/ (x+) ni- V., log ( S 5 < 8n. V., (s/(2z)) nz²) v2 log qe ( 82.
(24)
V tr zv,, (8/(x)) ni)) V., tr (vă, (ogg((()) ||2 S 83.
15

---

From these, we know that the Fisher divergence DF (0. I 0i, 82, 03.4) & U(t d1, d3, q), namely, De (ge (0, has lower upper bound compared to Dr (4: (l))) Consequently,
can get lower upper bounds for both Frisher mb and DEL (gollnd DollpoDE( (0, nt))). compared (f) and DxL (gollz80((0) respectively.
B Model parametrization
This section introduces different parametrizations used in diffusion models and provides their cali- brated instantiations.
B.1
Preliminary
Along the research routine of diffusion models, different model parametrizations have been used, including score prediction (zx) [42, 46], noise prediction (2,,) [17, 33], data prediction 2y (z²) [23, 32], and velocity prediction v (2+) [38, 18J. Taking the DSM objective as the training
loss, its instantiation at timestep t E [0, T") is written as
E gx(e))
(1x) + -
score prediction;
2
o2 E (o(())))((()
(11=)(()) - No mullā]
data prediction;
JDS (0)
207
(25)
2 E (()))
(x,) e
noise prediction;
of E (z(o)-n((e)
(xx)) (age - o velocity prediction.
207
B.2 Calibrated instantiation Under different model parametrizations, we can derive the optimal calibration terms ne that minimiz- ing Jsw (0,7) m)) as
(x.,) (xx)).
score prediction;
(xx)
(ac (are] - Exo(zo) [oo].
data prediction;
(26)
E (x.))
(xx)))
noise prediction;
E (sr,) [ug (2x,)] or (o))) [xa].
velocity prediction.
Taking or into Josm (0, me) we can obtain the gap
(xx)) (xxe) 1|2.
score prediction;
|E (1ra))
(1ra)
[2g]] (xr+) (xo [wo] ll
data prediction;
(hs TosM(0, (1))))))
20
(27)
F (w.)
[ (x:)) 112.
noise prediction;
20
[1 (xx)] + or Bgg [To]
velocity prediction.
20
C Visualization of the generations
further show generated images in Figure to double confirm the efficacy our calibration method. Our calibration could help to reduce ambiguous generations on both CIFAR-10 and CelebA.
(a) CIFAR-IO, Figure 5: Unconditional generation results on CIFAR-10 and CelebA using models from [17] and [41] respectively. The number of sampling steps is 20 based the results in Tables and 2.
(b) 10, calibration
(c) CelehA,
calibration
16


Reviews:
[{"review_id": "7ztJ86i5GQ", "reviewer": "Reviewer_2qSH", "summary": "The paper makes an observation regarding the reverse process of diffusion probabilistic model, noting that the data score term  is a martingale with respect to this process. A key contribution of the paper is a theorem to this effect and the associated proof. Leveraging this observation, the authors propose a calibration technique: one can calibrate a pretrained diffusion model at any time step by subtracting its expectation. Experimental results in the paper demonstrate that the calibrated score model achieves lower values of score matching objectives. Furthermore, the paper provides evidence that the calibrated score model yields higher evidence lower bounds. Lastly, the paper explains that similar conclusions hold true for the conditional score model.", "questions": "Q1. Can the authors provide a clearer commentary about the claimed subjective quality improvement. The generated images in Appendix C are extremely small. There is almost no discussion beyond “Our calibration could help to reduce ambiguous generations” – this is almost impossible for a reader to verify with the way the images are presented and it is not clear exactly what is meant. \n\nQ2. Could the authors comment on how the method performs for other performance metrics for generative models?\n\nQ3. What is the computational and memory cost (particulary in comparison to the original model)? i.e., does it increase the computational overhead by 30 percent? Is the memory cost an additional 20 percent? \n", "limitations": "The paper provides a very brief discussion of the limitations. This is definitely useful, and the authors acknowledge some of the key limitations. On the other hand, it would strengthen the paper if the supplementary material contained a more in-depth discussion. In particular, additional discussion concerning overhead and whether the proposed method translates to genuine practical benefits would be welcome. The section could discuss the challenge of evaluating generative models and suggest potential research avenues fhat might help establish whether the calibration improvement translates to more applied settings.  ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "S1. The paper makes a novel and significant observation concerning the martingale nature of the data score term in probabilistic diffusion models with respect to the reverse process. \n\nS2. The paper is clearly written, well-organized, and provides a key theorem and proof that are presented in a readily-followed fashion. \n\nS3.  The innovative technique for calibration is soundly based on theoretically-derived principles, is relatively simple to compute, and achieves an experimental performance improvement. The method is extended to the conditional setting. \n\nS4. The paper provides visualizations of the expected predicted noises and associated discussion that helps the reader understand the inductive bias. The additional experimental analysis is a welcome addition (sensitivity to the number of training samples, performance using generated samples, dynamic recording performance). \n", "weaknesses": "W1. Although the proposed method achieves an improvement in terms of model likelihood, the experiments are less convincing that there is a meaningful practical improvement in terms of generated samples. There is an improvement in FID, but the sample images are presented in the appendix in such a way that it is almost impossible to discern a difference between the calibrated and uncalibrated models.\n\nW2. The reported experiments only assess sample quality using FID. Multiple papers have demonstrated how this metric can be misleading in some circumstances and have proposed alternatives, e.g., precision and recall metrics, that can be used in conjunction with FID to provide a clearer and more complete assessment of the capabilities of a generative model. \n\nW3. There is very little discussion of computational and memory requirements for the calibration. The paper could benefit from a much clearer discussion of the costs that are incurred in order to obtain the calibration benefit.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review and suggestions, we have uploaded a rebuttal PDF.\n\n***W1 & Q1: Providing a clearer commentary on the improvement of image quality***\n\nIn **Figure A** of the rebuttal PDF, we provide examples demonstrating that our calibration could reduce ambiguous generations, such as eliminating generations that resemble both horse and dog. Intuitively, uncalibrated scores will contain redundant information from all classes and lead to the generation of ambiguous features.\n\n***W2 & Q2: Generative performance under other metrics***\n\nIn **Table A** of the rebuttal PDF, we assess sample quality using FID and other performance metrics including sFID, inception score (IS), precision and recall. As can be seen, our calibration consistently improves sample quality under different metrics, and we will present full results in the revision.\n\n***W3 & Q3: Computational and memory costs***\n\nWe provide two methods for implementing calibration in our work: post-training computation and dynamical recording.\n\nFor post-training computation, we compute the score mean for each inference timestep (e.g., each timestep utilized by DPM-Solver) and restore them for reuse. Therefore, the amortized computational cost for calibration is precisely $\\\\frac{\\\\mathcal{M}}{\\\\mathcal{N}}$, where $\\\\mathcal{M}$ is the number of samples used to calculate each score mean, and $\\\\mathcal{N}$ is the number of new samples generated during inference. If we use $\\\\mathcal{M}=20,000$ samples to calculate each score mean as ablated in Table 3, and we generate $\\\\mathcal{N}=100,000$ new samples during inference, the amortized computational cost will be $20\\\\%$; if we generate more samples, such as $\\\\mathcal{N}=1,000,000$, the amortized computational cost will be $2\\\\%$. As to the memory cost, since we first calculate score means and then restore them as detached tensors, they consume negligible inference memory during generation.\n\nFor dynamical recording, both the extra computational and memory costs are less than $1\\\\%$, because we use a shallow MLP for recording, which is relatively lightweight in comparison to diffusion models. In the revision, we will provide more detailed analyses of computational and memory costs.\n\n***Limitations***\n\nThank you for the instructive suggestions. Dynamical recording could significantly reduce the computational and memory overheads of calibration, while a calibrated diffusion model could potentially benefit downstream applications such as image editing/customization, semi-supervised learning, and improving adversarial robustness. In addition to model likelihood and quality metrics, we can evaluate a diffusion model by its degree of `uncalibration’, namely, how far the learned scores deviate from the essential properties (e.g., the expected data scores should be zero). In the revision, we will have a more in-depth discussion."}, {"review_id": "DfPqXuEIrh", "reviewer": "Reviewer_pqYH", "summary": "This paper presents a simple way to calibrate an arbitrary pretrained diffusion probabilistic model (DPM), which can reduce the score matching loss and increase the lower bounds of model likelihood. The authors observe that the stochastic reverse process of data scores is a martingale, from which concentration bounds and the optional stopping theorem for data scores can be derived. The proposed calibration method is easy to follow and can be performed only once, resulting in models that can be used repeatedly for sampling. The premise is that DPMs are by default uncalibrated (which they show experimentally), and that calibration can result in improved generation. This again is showed experimentally using the FID metric.\n\n—-\nRaised score from 4 to 6 post-rebuttal. On the whole it seems like the method can have some positive impacts - mostly in terms of model likelihood (with subsequent downstream benefits), and sometimes in terms of the quality of generated images.", "questions": "In the supplementary material, some example generated images are given with/without calibration. I zoomed right in, but I struggled to see any difference between pairs of images - is there anything that demonstrates that calibrating improves matters?", "limitations": "Yes, the authors have partially addressed the limitations in that they acknowledged that the model likelihood comes down but not necessarily FID, although they don't answer the question of whether it improves generation quality! Also there is a scaling issue with the method that is identified.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- clear and concise presentation of the proposed calibration method\n- empirical validation of the method on multiple datasets,\n- derivation of concentration bounds and the optional stopping theorem for data scores", "weaknesses": "This biggest weakness is solely relying on model likelihood and the FID score for measuring generative performance. FID (Heusel et al., 2017) calculates the Wasserstein-2 (a.k.a Fréchet) distance between multivariate Gaussians fitted to the embedding space of the Inception-v3 network of generated and real image. A major drawback with FID is its high bias. The sample size to calculate FID has to be large enough (usually above 50K). Smaller sample sizes can lead to over-estimation of the actual FID. This can clearly bee seen in Table 3. \n\nIt's not a surprise that the model likelihood improves (since this is directly linked to calibration), but the link between model likelihood and generation quality is not at all clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review and suggestions, we have uploaded a rebuttal PDF.\n\n***W1: Solely relying on model likelihood and the FID score for measuring generative performance***\n\nFirst, we need to clarify that Table 3 is an ablation study on *the number of samples used to estimate the calibration term*, rather than the number of samples used to calculate the FID score. As stated in Section 4.1 (Lines 217-218), we use 50K samples to calculate the FID score for all experiments in our paper.\n\nIn **Table A** of the rebuttal PDF, we assess sample quality using FID and other performance metrics including sFID, inception score (IS), precision and recall. As can be seen, our calibration consistently improves sample quality under different metrics, and we will present full results in the revision.\n\n***W2: The link between model likelihood and generation quality is not at all clear***\n\nThere have been several works discussing the link between model likelihood and generation quality in diffusion models [1,2,3,4,5], and their observations are that there is usually a trade-off between model likelihood and generation quality (i.e., improving model likelihood may degrade generation quality, and vice versa). In contrast, our calibration could improve both model likelihood and generation quality at the same time.\n\n***Q1: Is there anything that demonstrates that calibrating improves matters?***\n\nIn addition to the quantitative improvements in model likelihood and various generative metrics (FID, sFID, IS, precision and recall), we show in **Figure A** of the rebuttal PDF how our calibration can reduce ambiguous generations, such as eliminating generations that resemble both horse and dog. Intuitively, uncalibrated scores will contain redundant information from all classes and lead to the generation of ambiguous features.\n\n\n***References:*** \\\n[1] Kingma et al. Variational Diffusion Models. NeurIPS 2021 \\\n[2] Song et al. Maximum Likelihood Training of Score-Based Diffusion Models. NeurIPS 2021 \\\n[3] Huang et al. A Variational Perspective on Diffusion-Based Generative Models and Score Matching. NeurIPS 2021 \\\n[4] Vahdat et al. Score-based Generative Modeling in Latent Space. NeurIPS 2021 \\\n[5] Lu et al. Maximum Likelihood Training for Score-Based Diffusion Odes by High Order Denoising Score Matching. ICML 2022"}, {"review_id": "SgAypUsgFE", "reviewer": "Reviewer_LmXp", "summary": "This paper introduces a general calibration technique for diffusion probabilistic models (DPMs). The authors derive a time-dependent calibration term that is independent of any particular input under different model parameterizations. This calibration term can be computed in advance and repeatedly used for sampling. The experimental results demonstrate the effectiveness of the proposed calibration method with a noticeable reduction of FIDs when taking 15-40 NFEs.", "questions": "What are the FID results of AFHQv2 64×64, FFHQ 64×64, and ImageNet 64×64? Following my concerns on stability, in the main paper, the authors state that they have conducted experiments on these datasets based on pre-trained models considering their top performance. However, I cannot find these numbers in the main paper and in the appendix. Why do you choose to only present CIFAR-10 and CelebA 64x64 in the main paper? I believe that an improvement on the ImageNet 64×64 is more convincing. I suggest the authors keep the FIDs of all these datasets in the main paper and compare them to the original DPMs.", "limitations": "The impact of this paper may be limited by the lack of comparison against other existing methods. I tend to categorize this paper into research that improves pre-trained DPMs generally at inference time. Along this direction of research, there are several relevant works, which I believe the author could easily compare to, i.e., (not trainable) PNDM [1],  Analytic-DPM [2], and (trainable) BDDM [3].\n\n[1] Liu, Luping, et al. \"Pseudo Numerical Methods for Diffusion Models on Manifolds.\"\n[2] Bao, Fan, et al. \"Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models.\"\n[3] Lam, Max WY, et al. \"Bilateral Denoising Diffusion Models.\"\n", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The proposed calibration method in this paper is novel, theoretically sound, and practically useful in that it in principle can be applied to all kinds of DPMs to improve sampling with little computational overhead. The authors provide a performance analysis with varying numbers of samples for approximating the calibration term using the MC method. The FID results suggest that 20K of MC samples are sufficient to estimate the calibration term and reach the best quality. In this sense, the pre-sampling computation seems not to be too expensive. Overall, this approach appears to be general and easy to implement. Therefore, I believe this paper is significant and worth acceptance in NeurIPS.", "weaknesses": "Although the calibration term is not $x_t$-dependent, it depends on time ($t$) and the model parameters ($\\theta$). I am therefore concerned about the stability of the proposed approach. From Figure 1, we can basically confirm that the range of calibration terms drastically varies over different data sets. In this sense, if we apply the proposed approach to a new data set, there are at least three dynamic factors -- $t$, $\\theta$, and data distribution $q_0(x_0)$ that could affect the performance of this calibration method. In regard to this, I suggest the author conduct more validation experiments to prove the stability and consistency of the proposed method over different time steps, models, and datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review and suggestions, we have uploaded a rebuttal PDF.\n\n***W1 & Q1: The stability of the proposed approach and FID results of AFHQv2 64×64, FFHQ 64×64, and ImageNet 64×64***\n\nIndeed, the gain of our calibration trick is proportional to the degree of `uncalibration’ for a diffusion model, or how far the uncalibrated learned scores deviate from the essential properties (e.g., the expected data scores should be zero). We will conduct more validation experiments to demonstrate the stability and consistency of our calibration.\n\n**Table B** of the rebuttal PDF contains the FID results for AFHQv2 64×64, FFHQ 64×64, and ImageNet 64×64. To reduce computational burden, we did not run full FID experiments on these datasets in the original main paper. In the revision, we will provide full FID results on AFHQv2 64×64, FFHQ 64×64, and ImageNet 64×64.\n\n***Limitations***\n\nThank you for your constructive suggestions. Our calibration is conceptually compatible with these advanced samplers/solvers [1,2,3], because we focus on improving model score estimation and these methods focus on how to efficiently utilize model scores for better/faster sampling. In the revision, we will conduct empirical studies to ensure that our calibration is compatible with these methods.\n\n***References:*** \\\n[1] Liu et al. Pseudo Numerical Methods for Diffusion Models on Manifolds. ICLR 2022 \\\n[2] Bao et al. Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models. ICLR 2022 \\\n[3] Lam et al. Bilateral Denoising Diffusion Models. NeurIPS 2021"}, {"review_id": "pUkZcGyk05", "reviewer": "Reviewer_hZuU", "summary": "The paper proposes a simple procedure to improve the calibration of pre-trained diffusion models. \n\nDiffusion models have achieved strong practical performance, but its score estimation is often seen as a black box. The paper sheds some light on the issue, by detecting the inherent lack of calibration of the current state-of-the-art models, and proposing a simple procedure that reliably drives up the lower bound for model likelihood. Empirical evaluations showcase the efficacy of the method.", "questions": "None. ", "limitations": "I believe that the authors can make a few improvements in its presentation of empirical results. I understand that page limit prohibits the addition of more content, but I believe that some plots deserve to be in the main body of text. \n- Figure 1 showcases the potential lack of calibration for pre-trained DPMs, but does not show the expectation of predicted noise _after_ calibration. I believe that one can show the effectiveness of calibration by comparing the 2 images and discovering the calibrated model has a much lower expected value of the predicted noise. \n- The paper could benefit from also presenting some generated images as it shows any improvements in the most straightforward way. Currently there is a figure in the appendix about how calibration \"reduce ambiguous generations\", but there are too many examples to see a measurable difference. I believe a smaller plot highlighting the said ambiguous generations deserves a spot in the main text. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper discusses the problem of calibration in diffusion models, a question long overdue in this area of research: while diffusion models have achieved eye-catching generative results, little is known about whether the score function is a loyal depiction of the ground truth, or even a gradient field in itself. \n\nThe paper gives theoretically sound justifications of by framing the evolution of the score function as a martingale, and proposes a simple yet empirically quite effective way that adjust pre-trained diffusion models closer to the underlying score function. The overall writing and presentation of the paper is sound.", "weaknesses": "I think Section 3.4 is unsatisfactory, as I think the proper conclusion should be that the uncalibrated DPM training objective do not inherently minimize the mean of the expected predicted noises. My 2 more specific comments are listed below. \n\n- The logic between the explanations in 3.4 and Figure 2 seems confounding. I have understood the main points made in the paper after several reads, so I think the paper could adopt an easier to follow explanation. I believe the author's main point is (i) the optimal score attainable via diffusion modeling is capped by finite training data, so while the optimal score $\\nabla \\log q_t(x_t;\\mathbb{D})$ has expectation zero with respect to the marginal distribution conditioned on data, but it does not likely have expectation zero when given the entirety of $q_t$; and (ii) state-of-the-art diffusion models do not inherently minimize $\\lVert\\mathbb{E}_{q_t} s_\\theta^t \\rVert$. I believe those two points are quite separate but lines 188-191 present it as one singular point, which seems as if the paper treats the diffusion modeling as the gold standard of learning the scores, and that even the true score carries a bias in a finite-data setting. \n- I admit that the finite data bias can present a problem, but it is not what the calibration trick is for. Frankly speaking, the calibration trick does not manufacture new training data just because it is more calibrated, but only pushes the score estimate _towards_ the optimal score estimate attainable in a finite data regime, i.e., $\\nabla \\log q_t(x_t;\\mathbb{D})$. It is with this argument that I believe the calibration trick will attempt to modify the score estimate even when presented with the ground truth $\\nabla \\log q_t(x_t)$. Therefore, I believe the main point is not about dataset bias, but that common score matching objectives do not take into consideration the minimization of $\\lVert\\mathbb{E}_{q_t} s_\\theta^t \\rVert$. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review and suggestions, we have uploaded a rebuttal PDF.\n\n***W1: About Section 3.4***\n\nThank you for your insightful analyses, which are greatly helpful to us. Actually, we conducted preliminary trials on incorporating a loss of $\\\\|\\\\mathbb{E}\\_{q\\_{t}(x\\_{t})}[\\\\boldsymbol{\\\\epsilon}\\^{t}\\_{\\\\theta}(x\\_{t})]\\\\|\\^{2}$ during training. However, since the expectation operator $\\\\mathbb{E}\\_{q\\_{t}(x\\_{t})}$ is inside the norm operator $\\\\|\\\\cdot\\\\|$ (the norm operator is convex), we can only obtain a biased estimation of $\\\\|\\\\mathbb{E}\\_{q\\_{t}(x\\_{t})}[\\\\boldsymbol{\\\\epsilon}\\^{t}\\_{\\\\theta}(x\\_{t})]\\\\|$ during training using samples in a mini-batch, and the bias could be large for small mini-batch sizes. In contrast, our calibration using either post-training computation or dynamical recording can exploit more data samples to obtain asymptotically unbiased estimates of $\\\\mathbb{E}\\_{q\\_{t}(x\\_{t})}[\\\\boldsymbol{\\\\epsilon}\\^{t}\\_{\\\\theta}(x\\_{t})]$ (as stated in Line 204 for dynamical recording). According to your suggestions, we will better clarify the claims of Section 3.4 and Figure 2 in the revision.\n\n\n***Limitations***\n\nAs to Figure 1, the expectation of predicted noise after calibration is zero as $\\\\|\\\\mathbb{E}\\_{q\\_{t}(x\\_{t})}[\\\\boldsymbol{\\\\epsilon}\\^{t}\\_{\\\\theta}(x\\_{t})-\\\\mathbb{E}\\_{q\\_{t}(x\\_{t})}[\\\\boldsymbol{\\\\epsilon}\\^{t}\\_{\\\\theta}(x\\_{t})]]\\\\|=0$ for noise prediction and $\\\\|\\\\mathbb{E}\\_{q\\_{t}(x\\_{t})}[\\\\boldsymbol{s}\\^{t}\\_{\\\\theta}(x\\_{t})-\\\\mathbb{E}\\_{q\\_{t}(x\\_{t})}[\\\\boldsymbol{s}\\^{t}\\_{\\\\theta}(x\\_{t})]]\\\\|=0$ for score prediction. Therefore, as indicated by Eq. (11), the entire area under the curves in Figure 1 is counted as the likelihood improvements led by our calibration.\n\n\nIn **Figure A** of the rebuttal PDF, we provide examples demonstrating that our calibration could reduce ambiguous generations, such as eliminating generations that resemble both horse and dog. Intuitively, uncalibrated scores will contain redundant information from all classes and lead to the generation of ambiguous features."}, {"review_id": "A56O8J40Xq", "reviewer": "Reviewer_DT4C", "summary": "This paper investigates calibrating diffusion models. They notice that the expected data score equals zero. Since models not usually do not learn this correctly, they introduce a calibration term to subtract from a learned score. The new objective now includes the expected model score and performs better compared to the uncalibrated models.", "questions": "- Why is FID going up with the number of samples in Table 3 in both columns?\n\n- Can you incorporate this loss somehow during training so that the final model is calibrated?\n\n- Does this hold for any data distribution?\n\n- Could using 20 and 50 diffusion steps for generation be too little?", "limitations": "Limitations are addressed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and the method is presented clearly. The theoretical results are convincing. The main appeal of the method is its simplicity. The solver part of the results is solid.", "weaknesses": "Since the focus is on post-processing, the method is best suited for discrete time diffusion (like DDPM). Otherwise, it is unclear how the expected values are saved for continuous t. Line 221 mentions the selection of t depends on the sampling schedule of the solver, but the solver then cannot be adaptive. The alternative of learning the network as in eq. 14 would solve this but seems overly complicated. As can be seen from Fig 1 the lines are simple so fitting some simpler model would make more sense.\n\nThe results are showing the number of solver evaluations is lower and the likelihood is better. Likelihood is evaluated in a weird way. The only thing that is shown is that the expected learned score is not zero for \"uncalibrated\" models which is somewhat tautological. In this case, some other relevant metric like FID would be better.\n\nThere is no empirical demonstration that the diffusion models are uncalibrated, in a similar way that confidence-accuracy diagrams show this for classification. One could learn some simple synthetic dataset from a known distribution.\n\nIt is not clear why this would be a relevant task, as presented. I assume guidance would further uncalibrate the models but it is used for better samples. Perhaps the image domain is not the best choice here.\n\nEq. 13 is presented as a fact, whereas it seems this is to be demonstrated.\n\nThe main issue is the limited motivation for improving generative models on likelihood and a bad choice of experimental setup. Something other than images could solve both of these issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review and suggestions, we have uploaded a rebuttal PDF.\n\n***W1: It is unclear how the expected values are saved for continuous t and adaptive sampling schedules***\n\nIn our work, we present two methods for implementing calibration: post-training computation and dynamical recording.\n\nPost-training computation can be applied to both discrete and continuous timestep $t$ for non-adaptive sampling schedules. For adaptive sampling schedules, directly applying post-training computation may necessitate additional tricks such as interpolation.\n\nDynamical recording can thus be used for both discrete/continuous timesteps and non-adaptive/adaptive sampling schedules. The learning framework in Eq. (14) is simply a regression between the recording network and the score outputs, which can be easily implemented with a few lines of code. Furthermore, because we use a shallow MLP for recording (described in Lines 312-322), which is relatively lightweight in comparison to diffusion models, the extra computational and memory costs of dynamical recording are less than $1\\\\%$.\n\n***W2: Likelihood evaluation and showing results on other relevant metrics like FID would be better***\n\nAs indicated by Eq. (11), the entire area under the curves (i.e., integral w.r.t. timestep $t$) in Figure 1 is counted as the likelihood improvements led by our calibration, for both SDE and ODE solvers.\n\nIn **Table A** of the rebuttal PDF, we assess sample quality using FID and other relevant metrics including sFID, inception score (IS), precision and recall. As can be seen, our calibration consistently improves sample quality under different metrics, and we will present full results in the revision.\n\n***W3: No empirical demonstration that diffusion models are uncalibrated, in a similar way as confidence-accuracy diagrams***\n\nAlthough we use the same term, there is no direct relationship between the confidence-accuracy calibration studied in discriminative learning and the generative calibration studied in our paper. Intuitively, we observe some essential properties that should hold for any data scores (e.g., the expected data scores should be zero) and define a diffusion model as `uncalibrated’ if its learned model scores deviate from these essential properties. The mechanism of our calibration, such as Eq. (7), is more similar to variance reduction techniques (though not the same; we are more like DSM-loss reduction), which exploit existing observations to obtain better (score) estimators.\n\n***W4: The main issue is the limited motivation for improving generative models on likelihood***\n\nImproving model likelihood has long been a primary goal in generative learning; for example, generative models such as variational autoencoders (VAEs), energy-based models (EBMs), normalizing flows, and autoregressive models are all trained by maximizing likelihood.\n\nImproving model likelihood has also been well-motivated and widely studied in diffusion models [1,2,3,4,5]. Model likelihood is a principled metric that indicates how well a model distribution is learned towards the data distribution (under KL divergence) and can be practically applied for data compression and density estimation.\n\n***Q1: Why is FID going up with the number of samples in Table 3 in both columns?***\n\nThis phenomenon may be related to ODE solvers. It is found that using more neural function evaluations (NFE) for ODE solvers may increase the FID score [6,7], implying that an overly accurate estimation of the score may not necessarily decrease the FID score. As a result, in Table 3, using more training/generated data to obtain a more accurate estimate of the score mean does not result in a lower FID score. We will conduct ablation studies to further investigate this phenomenon in the revision.\n\n***Q2: Can you incorporate this loss during training so that the final model is calibrated?***\n\nYes, we could intuitively incorporate a loss of, say, $\\\\|\\\\mathbb{E}\\_{q\\_{t}(x\\_{t})}[\\\\boldsymbol{\\\\epsilon}\\^{t}\\_{\\\\theta}(x\\_{t})]\\\\|$ during training. However, since the expectation operator $\\\\mathbb{E}\\_{q\\_{t}(x\\_{t})}$ is inside the norm operator $\\\\|\\\\cdot\\\\|$ (the norm operator is convex), a potential concern is that we can only obtain a biased estimation of $\\\\|\\\\mathbb{E}\\_{q\\_{t}(x\\_{t})}[\\\\boldsymbol{\\\\epsilon}\\^{t}\\_{\\\\theta}(x\\_{t})]\\\\|$ during training using samples in a mini-batch, and the bias could be large for small mini-batch sizes. Post-training computation and dynamical recording, on the other hand, can use more data samples to obtain asymptotically unbiased estimates.\n\n***Q3: Does this hold for any data distribution?***\n\nYes, our conclusions apply to any data distribution considered in the diffusion model literature (satisfying mild regularity conditions such as $q\\_{0}(x_{0})\\\\rightarrow 0$ when $x\\_{0}\\\\rightarrow \\\\infty$, as assumed by default in [8, 9]).\n\n\n***Q4: Could using 20 and 50 diffusion steps for generation be too little?***\n\nExisting ODE solvers can generate high-quality images with $20 \\\\sim 50$ (or even $10 \\\\sim 20$) diffusion steps [6,7].\n\n\n***References:*** \\\n[1] Kingma et al. Variational Diffusion Models. NeurIPS 2021 \\\n[2] Song et al. Maximum Likelihood Training of Score-Based Diffusion Models. NeurIPS 2021 \\\n[3] Huang et al. A Variational Perspective on Diffusion-Based Generative Models and Score Matching. NeurIPS 2021 \\\n[4] Vahdat et al. Score-based Generative Modeling in Latent Space. NeurIPS 2021 \\\n[5] Lu et al. Maximum Likelihood Training for Score-Based Diffusion Odes by High Order Denoising Score Matching. ICML 2022 \\\n[6] Lu et al. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. NeurIPS 2022 \\\n[7] Karras et al. Elucidating the Design Space of Diffusion-Based Generative Models. NeurIPS 2022 \\\n[8] Ho et al. Denoising Diffusion Probabilistic Models. NeurIPS 2020 \\\n[9] Song et al. Score-Based Generative Modeling through Stochastic Differential Equations. ICLR 2021"}]

Author rebuttal:
We thank all reviewers for their constructive feedback, and we have responded to each reviewer individually. We have also uploaded a rebuttal PDF that includes:

- **Table A**: Assessing sample quality using FID and other performance metrics including sFID, inception score (IS), precision and recall.

- **Table B**: FID results for AFHQv2 64×64, FFHQ 64×64, and ImageNet 64×64.

- **Figure A**: Selected examples demonstrating that our calibration could reduce ambiguous generations.

Meta-review: The authors propose a novel calibration technique wherein a pretrained diffusion model using a time-dependent term to enforce zero-mean of the score function. 
This insight stems from the fact that it holds for the true score function of the corresponding diffusion process.

All reviewers and I agree that this papers makes a good contribution: the proposed correction term is easy to implement and the authors succeeds in justifying its use. In addition, method's advantages are well-supported through numerical experiments.

As you proceed to prepare the final version of your work for submission, kindly take into account the feedback provided by the reviewers in order to enhance the quality and clarity of your paper.
Decision: Accept (poster)