Stable Target Field for Reduced Variance Score Estimation in Diffusion Models
Authors: Yilun Xu
Shangyuan Tong
Tommi S. Jaakkola
Venue: ICLR 2023 poster (2023)
Keywords: generative model
score-based models
diffusion models
variance reduction

Abstract: Diffusion models generate samples by reversing a fixed forward diffusion process. Despite already providing impressive empirical results, these diffusion models algorithms can be further improved by reducing the variance of the training targets in their denoising score-matching objective. We argue that the source of such variance lies in the handling of intermediate noise-variance scales, where multiple modes in the data affect the direction of reverse paths. We propose to remedy the problem by incorporating a reference batch which we use to calculate weighted conditional scores as more stable training targets. We show that the procedure indeed helps in the challenging intermediate regime by reducing (the trace of) the covariance of training targets. The new stable targets can be seen as trading bias for reduced variance, where the bias vanishes with increasing reference batch size. Empirically, we show that the new objective improves the image quality, stability, and training speed of various popular diffusion models across datasets with both general ODE and SDE solvers. When used in combination with EDM, our method yields a current SOTA FID of 1.90 with 35 network evaluations on the unconditional CIFAR-10 generation task. The code is available at https://github.com/Newbeeer/stf

Published as a conference paper at ICLR 2023
STABLE TARGET FIELD FOR REDUCED VARIANCE SCORE ESTIMATION IN DIFFUSION MODELS
Yilun Xu", Shangyuan Tong, Tommi Jaakkola
Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology yixuêmit edu;; (sytong, tommi 10 eecsail mit...m.t..mm edu
ABSTRACT
Diffusion models generate samples by reversing fixed forward diffusion process. Despite already providing impressive empirical results, these diffusion models algorithms can be further improved by reducing the variance of the training targets in their denoising score-matching objective. We argue that the source of such variance lies in the handling of intermediate noise-variance scales, where multiple modes in the data affect the direction of reverse paths. We propose to remedy the problem by incorporating a reference batch which we use to calculate weighted conditional scores more stable training targets. We show that the procedure indeed helps in the challenging intermediate regime by reducing (the trace of) the covariance of training targets. The new stable targets can be seen as trading bias for reduced variance, where the bias vanishes with increasing reference batch size. Empirically, we show that the new objective improves the image quality, stability, and training speed of various popular diffusion models across datasets with both general ODE and SDE solvers. When used in combination with EDM (Karras et al., 2022), our method yields a current SOTA FID of 1.90 with 35 network evaluations on the unconditional CIFAR-10 generation task. The code is available at wwt.t::t//tt.. // thu.. www/Newweeer//eeeeeeeeeeeee
INTRODUCTION
Diffusion models Sooh--Dickttein et al., 2015; Song & Ermon, 2019; Ho al., 2020) have recently achieved impressive results on a wide spectrum of generative tasks, such as image generation (Nichol al., 2022; Song et al., 2021b), 3D point cloud generation (Luo & Hu, 2021) and molecular conformer generation (Shi et al., 2021: Xu et al., 2022a). These models can be subsumed under a unified framework in the form of Itō stochastic differential equations (SDE) (Song et al., 2021b). The models learn dime-dependent score fields via cctre-matching (Hyvärinen & Dayan, 2005), which then guides the reverse SDE during generative sampling. Popular instances of diffusion models include variance-exploding (VE) and variance- prreeeriggg (VP) SDE (Song et al., 2021b). Building these formulations, EDM (Karras et al., 2022) provides the best performance to date.
We argue that, despite achieving impressive empirical results, the current training scheme of diffusion models can be further improved. In particular, the variance of training targets in the denoising score- matching (DSM) objective can be large and lead to suboptimal performance. To better understand the origin of this instability, decompose the score field into three regimes. Our analysis shows that the phenomenon arises primarily in the intermediate regime, which is characterized by multiple modes or data points exerting comparable influences on the scores. In other words, in this regime, the sources of the noisy examples generated in the course of the forward process become ambiguous. We illustrate the problem in Figure 1(a), where each stochastic update of the score model is based on disparate targets.
We propose generalized version of the denoising mctrh-ng objective, termed the Stable Target Field (STF) objective. The idea is to include an additional reference batch of examples that are used to calculate weighted conditional scores targets. We apply self-normalized importance sampling to aggregate the contribution of each example in the reference batch. Although this process can substantially reduce the variance of training targets (Figure 1(b)), especially in the intermediate regime,
"Equal Contribution.

---

Published as a conference paper at ICLR 2023
(a) DSM
(b) STF
Figure 1: Illustration differences between the DSM objective and our proposed STF objective. The "destroyed" images (in blue box) are close to each other while their sources (in red box) are not. Although the true score in expectation is the weighted average of vi, the individual training updates of the DSM objective have high variance, which our STF objective reduces significantly by including a large reference batch (yellow box).
of Experimentally, we show that our STF objective achieves new state-of-the-art performance on CIFAR- 10 unconditional generation when incorporated into EDM (Karras et al., 2022). The resulting FID score (Heusel et al., 2017) is 1.90 with 35 network evaluations. STF also improves the FID/Inception scores for other variants of score-based models, i.e. VE and VP SDEs (Song et al., 2021b), in most cases. In addition, enhances the stability of converged score-based models on CIFAR-10 and CelebA 642 across random seeds, and helps avoid generating noisy images in VE. STF accelerates the training of score-based models (3.6x speed-up for VE on CIFAR-10) while obtaining comparable or better FID scores. To the best of our knowledge, STF is the first technique to accelerate the training process of diffusion models. We further demonstrate the performance gain with increasing reference batch size, highlighting the negative effect of large variance.
it does introduce some bias. However, show that the bias together with the the STF training targets shrinks to zero as we increase the size of the reference batch.
Our contributions are summarized as follows: (1) We detail the instability of the current diffusion mod- els training objective in principled and quantitative manner, characterizing region in the forward process, termed the intermediate phase, where the score-learning targets are most variable (Section 3). (2) We propose generalized score-matching objective, stable target field, which provides more stable training targets (Section 4). (3) We analyze the behavior of the new objective and prove that is asymptotically unbiased and reduces the rrace-of-covariance of the training targets by a factor pertaining to the reference batch size in the intermediate phase under mild conditions (Section 5). (4) We illustrate the theoretical arguments empirically and show that the proposed STF objective improves the performance, stability, and training speed of core-bbsed methods. In particular, it achieves the current state-of-the-arr FID score the CIFAR-10 benchmark when combined with EDM (Section 6).
BACKGROUND ON DIFFUSION MODELS
In diffusion models, the forward process! is an SDE with learned parameter, in the form of: dx = f(x, t)dt + g()ddw,
where x E Rd with x(0) Po being the data distribution, f E [0, 1]. f: Rd x [0, 1] - Rd g: [] + IR, and w Rdd is the standard Wiener process. gradually transforms the data distribution to known prior as time goes from 0 to 1. Sampling of diffusion models is done via corresponding reverse-time SDE (Anderson, 1982):
dx [f(x - log (x)) dī g())dw,
where denotes time traveling backward from to 0. Song et al. (2021b) proposes a probability flow ODE that induces the same marginal distribution px(x) as the SDE: dx
For simplicity, we focus the version where the diffusion coefficient g(f) is independent of x(t).
2

---

Published as a conference paper at ICLR 2023
[[(x. t) V log (x df. Both formulations progressively recover Po from the prior p1. We estimate the score of the transformed data distribution at time 1, Vx log (x), via a neural network, sy (x, t). Specifically, the training objective a weighted sum of the denoising score-mas more-matching (Vin- cent, 2011):
min Ee (1) X0)Bx-
[x]s)(((1),)) Vx(e)
(I)
where is the distribution for time variable, e.g., (0, 1] for VE/VP (Song et al., 2021b) and
log-normal distribution for EDM Karras et al. (2022), and X(t) = of is the positive weighting function to keep the time-dependent loss at the same magnitude (Song et al., 2021b), and Pijo (x)(xx() the transition kernel denoting the conditional distribution of x(!) given x². Specifically, diffusion models "destroy" data according to a diffusion process utilizing Gaussian transition kernels, which result in Paio (x())(x) = (u,, of I). Recent works (Xu et al., 2022b; Rissanen et al., 2022) have also extended the underlying principle from the diffusion process to more general physical processes where the training objective is not necessarily score- score-related.
3 UNDERSTANDING THE TRAINING TARGET IN SCORE OBJECTIVE
The vanilla denoising score-matching objective at time is:
(OsMa(,..) t) px)(x) Ep (x(())))) [x)s)(((t). V (())) log (x(t)(x)1B 12).
(2)
where the network is trained fit the individual targets Vx(e) log Pooo(((())) (x())xx (x(1), the "influence" exerted by clean data x(). We can swap the order of the sampling process by first sampling x(t) from pa and then x from Poje( (-(x)))). Thus, Sy has a closed form minimizer:
SisM (x((), Epo (x)) V xxe) log (x = V xx1) log (xg))
(3)
The score field is a conditional expectation of Vx(() log Prjo (x())x) with respect the posterior distribution Poje- In practice, a Monte Carlo estimate of this target can have high variance (Owen, 2013; Elvira & Martino, 2021). In particular, when multiple modes of the data distribution have comparable influences on x(). Po P (-Ix(1) is multi-mode distribution, as also observed in Xiao al. (2022). Thus the targets Vx(1) log (x(o(x(()) (x) vary considerably across different x and this can strongly affect the estimated score at (x(t), t), resulting in slower convergence and worse performance in practical stochastic gradient optimization (Wang et al., 2013).
To quantitatively characterize the variations of individual targets at different time, we propose metric the average ofce-oo-covariance training targets at time t:
(t) (x(()()) Tr (TvO (x)x(x))())) (V x(() log (xg)))((((t))x)))
= Ep.((x()) Epg (xlx(1))
|| Vx(t) log (xg)(((x(()xx) - Vx(0) log (x(1))
We use V() to define three successive phases relating the behavior of training tar- gets, As shown in Figure 2(a), the three phases partition the score field into near, intermediate, and far regimes (Phase I~3 respectively). Intu- itively, (L) peaks in the intermediate phase (Phase 2). where multiple distant modes in the data distribution have comparable influences on the same noisy perturbations, resulting in stable targets. In Phase 1, the posterior Poje concentrates around one single mode, thus low variation. In Phase the targets remain similar across modes since lim,-1 Prjo (x())x) x for commonly used transition kernels.
Pheaee:
0.8
0.6
Two Oraaniia
0.4
CIAAS-3 13
0.2
0.0
0.2
(a) ODE Sampling
(b) VoSM(f) versus f
Figure 2: (a): Illustration of the three phases in a two-mode distribution. (b): Estimated (t) for two distributions. We normalize the maximum value to for illustration purposes.
We validate this argument empirically in Figure 2(b), which shows the estimated (DSM(() for mixture of two Gaussians as well as a subset of CIFAR-10 dataset (Krizhevsky et al., 2009) for more
We omit "(0)" from x0) when there ambiguity.
3

---

Published as a conference paper at ICLR 2023
realistic setting. Here we use VE SDE, Le., p()o(x(t))))= =N LoNxxx,, ( (2 for some om and
оM (Song et al., 2021b). (t) exhibits similar phase behavior across in both toy and realistic cases. Moreover, (() reaches its maximum value in the intermediate phase, demonstrating the large variations of individual targets. We defer more details to Appendix C.
4 TREATING SCORE AS A FIELD
The vamilla denoising score matching approach (Equation 3) can be viewed as a Monte Carlo estimator, ie.. Vx(1) log Pe (x()) po x()) Vx(t) log Prjo (x) z +E Vx(() Pogo Prjo(x()) (x(() (x)) where x; is sampled from Poje(-Ix(t) and n = 1. The variance of a Monte Carlo estimator is proportional to so we propose to use a larger batch (n) to counter the high variance problem described in Section 3. Since sampling directly from the posterior Poje is not practical, we first apply importance sampling with the proposal distribution po- Specifically, we sample a large reference batch B3 = { Pe and get the following approximation:
(x())x)(x(t)) xx(1) log logpe)(((())xx). (x,)
Vx(e) (x
The importance weights can be rewritten as Po ( (x)xx())/pooxx p))(((()))))ppxxx(((() (x(t)) However, this basic importance sampling estimator has two issues. The weights now involve an unknown normalization factor (x((t)) and the ratio between the prior and posterior distribution can be large in high dimensional spaces. To remedy these problems, we appeal to seff-normalization techniques (Hesterberg, 1995) to further stabilize the training targets:
(x)(((()
V xx(e) (x
Vx(1) log 14)1)oo(
(x))))(((()))x)) (x,)))
(5)
We term this new training target in Equation Stable Target Field (STF). practice, we sample the reference batch B,. = }= from rā and obtain x(1) by applying the transition kernel to the "first" training data x,. Taken together, the new STF objective becomes:
(0 t) E (xx) ~g Ex(t)=p (((x)))
"
Pejo Pejo (x(())xX)) x+)
(xx(( Σ
Vx(0 log (x()) (x+)
(6)
Cj=1 (oioPo(()))))) (t)))((((()))}) (x)))x))
A=1
When n 1, STF reduces to the vanilla denoising cotre-matching (Equation 2). When n > 1, STF incorporates reference batch to stabilize training targets. Intuitively, the new weighted target assigns larger weights to clean data with higher influence on x(t), ie.,, higher transition probability (x))x
Similar to our analysis in Section 3, we can again swap the sampling process in Equation 6 so that, for perturbation x(t), we sample the reference batch B, = (xx ) from Poje (-Ixx()))p where the first element involves the posterior, and the rest follow the data distribution. Thus, the minimizer of the new objective (Equation 6) is (derivation can be found in Appendix B.1)
(xjo(x(t) xx) E, (x)o(((())
( (1x())) (x:) 3
V Vx(1) log Pajo (x))(((t)))x))
Note that although STF significantly reduces the variance, introduces bias: the minimizer is no longer the true score. Nevertheless, in Section 5, we show that the bias converges to 0 as c0, while reducing the crace-o-ccovariacee of the training targets by factor of n when Poje RJ po- further instantiate the STF objective (Equation 6) with transition kernels in the form of Pejo (x)))x() xx= = [xxx I),, which includes EDM (Karras et al., 2022), VP (through reparameteriza- tion) and VE (Song al., 2021b);
x ||x(t) [|x(()-xx:ll 20
exp
Ex,
(xx ( Hip so (x())))
Pop ExI
(Xt X
x())
1= E, exp
[x((t) xx, x, 20
4

---

Published as a conference paper at ICLR 2023
To aggregate the time-dependent STF objective over t. sample the time variable t from the training distribution y and apply the weighting function X()). Together, the final training objective for STF (t) (A(t) t)]. We summarize the training process in Algorithm 1. The small batch size [B] is the same as the normal batch size in the vanilla training process. We defer specific use cases of STF objectives combined with various popular diffusion models to Appendix A.
Algorithm 1 Learning the stable target field
Input: Training iteration T. Initial model Sg, dataset D, learning rate for ( -= 1...7 7 do
Sample large reference batch B5, from D, and subsample small batch B = [xi) from Br, Uniformly sample the time {4:]-1 ~ qe (1)18)
Obtain the batch of perturbed samples {x, (t:)) by applying the transition kernel Pajo B Calculate the stable target field of B,, for all x, (4:):
(t.
V8, ( (4:) = ExEB, Zye8, (x (1)))) (x.(A)) log ((x(())))
Calculate the loss: (())
[B] E X(4)) (x,(),,, - VB. (x,(1))11
Update the model parameter: 0 = 0 ()(( end for return
ANALYSIS
In this section, we analyze the theoretical properties of approach. In particular, we show that the new minimizer (x()) (t),,) Then, we show that the proposed STF reduces the trace-of-covariang of training targets propositional the reference batch size in the intermediate phase, with mild conditions (Section 5.2).
(Equation 7) converges to the true score asymptotically (Section 5.
5.1 ASYMPTOTIC BEHAVIOR
Although in general (x()) t) & V x((r) lxg)) the bias shrinks toward 0 with a increasing In the following theorem we show that the minimizer of STF objective at (x(1), (), ie., S51P (x(e(x(1),0) (). asymptotically normal when
oc.
Theorem 1. Suppose VI E [0, 1},0 < <a < co, then
Vx (ogr( 3 N NO.0 Cov Cor(Vx(oP1((x(())) (x()()))
(S (x(),,)
We defer the proof to Appendix B.2. The theorem states that, for commonly used transition kernels, mean normal, and larger reference batch size (n) will lead to smaller asymptotic variance. As can be seen in Equation 8, when - oo, STTE (x), 1) highly concentrates around the true score Vx(() log (x())).
Sin (x(t), t) V xx) p. converges
5.2
TRACE OF COVARIANCE
We now highlight the small variations of the training targets in the STF objective compared to the DSM. As done Section study the trace-of-covariance of training targets in STF:
Tr (Cor Poje (-x(()))))))
Pe Peo (x(+))x)) (()((((())xx))
Vx(e) log (og)))o(x(())x) xxx) )
VlTF(() E (x(e))
Σ
In the following theorem we compare VSTF with VISM- particular, we can upper bound VsiF(t) by Theorem 2. Suppose Vt E [0, 11,0 az < co, then
1 Vstr(t) (t) n Voosu(t) + v3d
of V VBpi (po(x) ll (o)x(x(x((t)) ) + +0(12)
5

---

Published as a conference paper at ICLR 2023
where Dy is an oddivegennce with (v) = (1/y- - 1)²
(y < 1.5) Further, when n > d and
8y/27 - 1/3 (y 2 1.5)
(x)x(x(x(t)) se pu(x) for all x(t). Vsre(t) 2
Vosu(t) n-1
We defer the proof to Appendix B.3. The second term that involves f-divergence D, is necessary to capture how the coefficients, iee, Pejo (x))(x(()(xx)) [x+)/ Pejo used to calculate the weighted score target, vary across different samples x x(t). This term decreases monotonically as function of t. In Phase Poja (t) differs substantially from Px(x) and the divergence term Dy dominates. In contrast to the upper bound, both (t) and (DSM(() (t) have minimal variance small values of t since the training target is always dominated by one x. The theorem has more relevance in Phase 2, where the divergence term decreases to value comparable to VDSM (t). In this phase, we empirically observe that the ratio of the two terms in the upper bound ranges from 10 100. Thus, when we use a large reference batch size (in thousands), the theorem implies that STF offers a considerably lower variance (by factor of 10 more) relative to the DSM objective. In Phase 3, the second term vanishes to 0, as pt Pejo with large o, for commonly used transition kernels. As a result, STF of the training targets by at least n - 1 times in the far field.
reduces the average muce-of-covariang
Together, we demonstrate that the STF targets have diminishing bias (Theorem 1) and much more stable during training (Theorem 2). These properties make the STF objective more favorable for diffusion models training with stochastic gradient optimization.
6 EXPERIMENTS
In this section, first empirically validate our theoretical analysis in Section 5, especially for variance reduction in the intermediate phase (Section 6.1). Next, we show that the STF objective improves various diffusion models image generation tasks in terms of image quality (Section 6.2). In particular, STF achieves state-of-the-ard performance on top of EDM. In addition, we demonstrate that STF accelerates the training of diffusion models (Section 6.3), and improves the convergence speed and final performance with increasing reference batch size (Section 6.3).
6.1 VARIANCE REDUCTION IN THE INTERMEDIATE PIIAS
CIFAR-10-4096
Two Gaassim
CIFAR: 10-4066 CIFAR-I0-4066
18
0.8
Voueff
0.8
Dutl
0.8
10
100
0.4
0.8
Veuelt)
n 100
Venit))
04
0.4
0.2
0.2
2
0.0
Phon
Phos
Shos
0.0
0.0
0.2
0.4
0.8
0.8
1.0
0.0
0.2
0.6
0.6
0.8
1.0
0.0
0.2
0.8
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
(a)
(b)
(c)
(d)
Figure 3: (a, b): VISM (f) and D(t) versus f. We normalize the maximum values to for illustration purposes. (c, d): VSTF(() with varying reference batch size n.
The proposed Algorithm I utilizes a large reference batch to calculate the stable target field instead of the individual target. In addition the theoretical analysis in Section 5, we provide further empirical study to characterize the intermediate phase and verify the variance reduction effects by STE. Apart from (L). we also quantify the average divergence between the posterior Poje x()) and the data
distribution Po time (introduced in Theorem 2): D(t)=
(x(t) [Dy (Poje(x (t)) || po [x)))] Intuitively, the number of high-density modes in Poje (- x()) grows D(t) decreases. To investigate their behaviors, we construct two synthetic datasets: (1) a 64-dimensional mixture of two Gaussian components (Two Gaussians), and (2) a subset of 1024 images of CIFAR- 10 (CIFAR-10-4096),
Figure 3(a) and Figure 3(b) show the behaviors of VI and D(L) Two Gaussian and CIFAR- 10-4096. In both settings, Viss(()) reaches its peak in the intermediate phase (Phase 2). while D(t) gradually decreases over time. These results agree with our theoretical understanding from Section 3. In Phase 2 and 3, several modes of the data distribution have noticeable influences on the scores, but only in Phase 2 are the influences much more distinct, leading to high variations of the individual target V Vx(0) log P:jo (1)(((()) (x) Poje (-Ix

---

Published as a conference paper at ICLR 2023
Figure 3(c) and Figure 3(d) further show the relationship between (t) and the reference batch size n. Recall that when n = 1, STF degenerates to individual target and (I) = VIDSM We observe that (t decreases when enlarging
In particular, the predicted relation VtTe(t) S VDSM( (L) - 1) in Theorem 2 holds for the two Gaussian datasets where D, is small. On the high dimensional dataset CIFAR-10-4096 the stable target field can still greatly reduce the training target variance with large reference batch sizes
6.2 IMAGE GENERATION
Table I: CIFAR-I0 10 sample quality (FID, Inception) and number of function evaluation (NFE).
Methods
Inception
FID +
NFE +
StyleGAN2-ADA (Karras et al., 2020)
9.83
2.92
DDPM (Ho et al., 2020)
9.46
3.17
1000
NCSNx2 (Song & Ermon, 2020)
8,40
10.87
1161
PFGM (Xu et al., 2022b)
9.68
2.48
104
VE (Song et al., 2021b)
DSM RK45
9.27
8.90
264
STF (ours) RK45
9.52 f
5.51 +
200
DSM PC
9.68
2.75
2000
STF (ours) PC
9.86 f
2.66 4
2000
VP (Song et al., 2021b)
DSM DDIM
9.20
5.16
100
STF (ours) DDIM
9.28
5.06 $
100
DSM RK45
9.46
2.90
140
STF (ours) RK45
9.43 |
2.99 f
140
EDM (Karras al., 2022)
DSM - Heun, NCSN+
9.82
1.98
35
STF (ours) Hcun, NCSN++ DSM - Heun, DDPM+
9.93 1
1.90 4
35
9.78
1.97
35
STF (ours) Heun, DDPM+
9.79 f
1.92 J
35
We demonstrate the effectiveness of the new objective on image generation tasks. We consider CIFAR-10 (Krizhevsky al., 2009) and CelebA 64 64 (Yang al., 2015) datasets. We set the reference batch size n to 4096 (CIFAR-10) and 1024 (CelebA 642). We choose the current state-of- the-art score-based method EDM (Karras et al., 2022) as the baseline, and replace the DSM objective with our STF objective during training. We also apply STF to two other popular diffusion models, VE/VP SDEs (Song et al., 2021b). For fair comparison, we directly adopt the architectures and the typer--arameters in Karras al. (2018) and Song al. (2021b) for EDM and VE/VP respectively. In particular, we use the improved NCSN+ models (Karras et al., 2022) in the EDM scheme. To highlight the stability issue, we train three models with different seeds for VE CIFAR- 10. We provide more experimental details Appendix D.I.
Numerical Solver. The reverse-time ODE and SDE in scored-based models are compatible with any perproe solvers. We use the adaptive solver RK45 method (Dormand & Prince, 1980; Song et al., 202 lb) (RK45) for VE/VP and the popular DDIM solver (Song et al., 202 for VP. We adopt Heun's s 2nd order method (Heun) and the time discretization proposed by Karras et al. (2022) for EDM. For SDEs, we apply the predictor-correctn (PC) sampler used in (Song et al., 2021b).
We denote the methods in objective-sample format, i.e., A-B, where A E (DSM, STF} and B E (RK45, PC, DDIM, Heun). We defer more details to Appendix D.2.
Results. For quantitative evaluation of the generated samples, we report the FID scores (Heusel et al., 2017) (lower is better) and Inception (Salimans et al., 2016) (higher is better). We measure the sampling speed by the average NFE (number of function evaluations). We also include the results of several popular generative models (Karras et al., 2020; Ho et al., 2020; Song & Ermon, 2019; Xu et al., 2022b) for reference.
Table and Table report the sample quality and the sampling speed on unconditional gener- ation of CIFAR-10 10 and CelebA 642. Our main findings are: (1) STF achieves new state-of- the-art FID scores for unconditional generation CIFAR-10 benchmark. As shown in Ta-
7

---

Published as a conference paper at ICLR 2023
ble 1, The STF objective obtains FID of 1.90 when incorporated with the EDM scheme. To the best of our knowledge, this is the lowest FID score on the unconditional CIFAR-IO gener- ation task. In addition, the STF objective consistently improves the EDM across the two ar- chitectures. (2) The STF objective improves the performance of different diffusion mod- els. We observe that the STF objective improves the FID/Inception scores of VE/VP/EDM on CIFAR- 10, for most ODE and SDE samplers. STF consistently provides performance gains for VE across datasets. Remarkably, our objective achieves much better sample quality using ODE samplers for VE. with an FID score gain of 3.39 on CIFAR- 10. and 2.22 on Celeba 64". For VP, STF provides better results the popular DDIM sampler, while suffering from a slight performance drop when using the RK45 sampler. (3) The STF objective stabilizes the converged VE model with the RK45 sampler. In Appendix E. we report the standard deviations of performance metrics for converged models with different seeds CIFAR-10 10 with VE. We observe that models trained with the STF objective give more consistent results, with smaller standard deviation of used metrics.
Table 2: FID and NFE on CelebA 64
Methods/FFEs
FID J
NFE
CelebA 64 RK45
VE (DSM)
7.56
260
VE
5.34
266
We further provide generated samples in Appendix F. One inter- esting observation is that when using the RK45 sampler for VE on CIFAR-10. 10, the generated samples from the STF objective do not contain noisy images, unlike the vanilla DSM objective.
CelebA 64 PC
VE (DSM)
9.13
2000
VE (STF)
8.28
2000
6.3
ACCELERATING TRAINING OF DIFFUSION MODELS
CIFAR-10
Celcha 64
175
DSM
50
DSM
STF
STF
150
45
BD Store
125
40
100
FiD
35
20
40
80
1000
120
140
Wall-clock training time (h)
Wall-clock training time (h)
(a) CIFAR-10
(b) CelebA 64 x 64
Figure 4: FID and generated samples throughout training on (a) CIFAR-10 and (b) CelebA 642.
The muriance-rdduction techniques in neural network training can help find better optima and achieve faster convergence rate (Wang et al., 2013; Defazio et al., 2014; Johnson & Zhang, 2013). In Figure 4, we demonstrate the FID scores every 50k iterations during the course of training. Since our goal is to investigate relative performance during the training process, and because the FID scores computed on 1 samples are strongly correlated with the full FID scores on 50k sample (Song & Ermon, 2020), we report FID scores on IK samples for faster evaluations. We apply ODE samplers for FID evaluation, and measure the training time on two NVIDIA A100 GPUs. For a fair comparison, we report the average FID scores of models trained by the DSM and STF objective on VE versus the wall-clock training time (h).
The STF objective achieves better FID scores with the same training time, although the calculation of the target field by the reference batch introduces slight overhead (Al- gorithm I). In Figure 4(a), we show that the STF objective drastically accelerates the training of diffusion models on CIFAR-10. The STF objective achieves comparable FID scores with 3.6x less training time (25h versus 90h).
CIFAR-10
DSM (n=1)
70
(n=128)
STF (n=512)
FID Score
80
STF (n=4096)
50
For CelebA 642 datasets, the training time improvement is less significant than on CIFAR- 10. Our hypothesis is that the STF objective is more effective when there are multiple well-separated modes in data distribution, e.g., the ten classes in CIFAR-10, 10, where the DSM objective suffer from relatively larger variations in the intermediate phase. In addition, the converged models have better final performance when pairing with the STF both datasets.
40
40
60
80
call-clock training time
FID scores in the training course with varying reference batch size.
Figure
8

---

Published as a conference paper at ICLR 2023
6.4 EFFECTS OF THE REFERENCE BATCH SIZE
According to our theory (Theorem 2), the upper bound of the trace-of-covariance of the STF target decreases proportionally to the reference batch size. Here study the effects of the reference batch size (n) on model performances during training. The FID scores are evaluated on 1 samples using the RK45 sampler. As shown in Figure 5, models converge faster and produce better samples when increasing n. It suggests that smaller variations of the training targets indeed speed up training and improve the final performances of diffusion models.
RELATED WORK
Different phases of diffusion models. The idea of diffusion models having different phases has been explored in prior works though the motivations and definitions vary (Karras et al., 2022; Choi et al., 2022). Karras et al. (2022) argues that the training targets are difficult and unnecessary to learn in the very near field (small t in our Phase 1). whereas the training targets are always dissimilar to the true targets in the intermediate and far field (our Phase 2 and Phase 3). As a result, their solution is sampling / with a log-normal distribution to emphasize the relevant region (relatively large in our Phase 1). In contrast, we focus on reducing large training target variance in the intermediate and far field, and propose STF to better estimate the true target (ef. Karras et al. (2022)). Choi et al. (2022) identifies a key region where the model learns perceptually rich contents, and determines the training weights A(t) based on the signal-to-notl ratio (SNR) at different t. As SNR is monotonically decreasing over time, the resulting up- up--eeigted region does not match our Phase 2 characterization. In general, our proposed STF method reduces the training target variance in the intermediate field and is complementary to previous improvements of diffusion models.
Importance sampling. The technique of importance sampling has been widely adopted in machine learning community, such as debiasing generative models (Grover et al., 2019), counterfactual learning (Swaminathan & Joachims, 2015) and reinforcement learning (Metelli et al., 2018). Prior works using importance sampling to improve generative model training include reweighted wake- sleep (RWS) (Bornschein & Bengio, 2014) and importance weighted autoencoders (IWAE) (Burda al., 2015). RWS views the original wake-sleep algorithm (Hinton et al., 1995) as importance sampling with one latent variable, and proposes to sample multiple latents to obtain gradient estimates with lower bias and variance. IWAE utilizes importance sampling with multiple latents to achieve greater flexibility of encoder training and tighter log- log-liieldood lower bound compared to the standard variational autoencoder (Kingma & Welling, 2013; Rezende et al., 2014).
Variance reduction for Fisher divergence. One popular approach to score-matching is minimize the Fisher divergence between true and predicted scores (Hyvärinen & Dayan, 2005). Wang et al. (2020) links the Fisher divergence to denoising score- score-matchee matching (Vincent, 2011) and studies the large variance problem (in ((1/a)) of the Fisher divergence when + O. They utilize a control variate to reduce the variance. However, this is typically not a concern for current diffusion models as the time-dependent objective can be viewed as multiplying the Fisher divergence by A(l) of. resulting in a variance objective even when f - O.
8 CONCLUSION
We identify large target variance as a significant training issue affecting diffusion models. We define three phases with distinet behaviors, and show that the high-varianee targets appear in the intermediate phase. As a remedy, we present a generalized score-matching objective, Stable Target Field (STF), whose formulation is analogous to the self nelm-nomml importance sampling via a large reference batch. Albeit longer an unbiased estimator, our proposed objective asymptotically unbiased and reduces the trace-of-com ance-of-covariance of the training targets, which we demonstrate theoretically and empirically. We show the effectiveness of our method image generation tasks, and show that STF improves the performance, stability, and training speed over various state-of-the-an diffusion models. Future directions include a principled study on the effect of different reference batch sampling procedures.
-- so we expect that training diffusion models with reference batch of more samples in the neighborhood of x (the sample from which x(t) is perturbed) would lead to an even better estimation of the score field. Moreover, the three-phase analysis can effectively capture the behaviors of other physics-inspires generative models, such as PFGM (Xu et al., 2022b) or the more advanced PFGM++ (Xu al., 2023). Therefore, we anticipate that STF can enhance the performance and stability of these models further.
Our presented approach is uniformly sampling from the whole dataset ( [x: [x:]² ~ T

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We are grateful to Benson Chen for reviewing an early draft this paper. We would like to thank Hao He and the anonymous reviewers for their valuable feedback. YX and TJ acknowledge support from MIT-DSTA Singapore collaboration, from NSF Expeditions grant (award 1918839) "Understanding the World Through Code", and from MIT-IBM Grand Challenge project. ST and TJ also acknowledge support from the ML for Pharmaceutical Discovery and Synthesis Consortium (MLPDS).
REFERENCES
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-3222 1982.
Jörg Bornschein and Yoshua Bengio. Reweighted wake-sleep. arXiv preprint arXiv:1406.275/ 2014.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint /rXiv:/509.00 (00.00090000000 /0, 2015.
Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11472-11481 2022.
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems, 27, 2014.
R. Dormand and J. Prince. family of embedded runge-kutta formulae. Journal of Computational and Applied Mathematics, 6:19- 26, 1980.
Víctor Elvira and Luca Martino. Advances in importance sampling. Wiley StatsRef: Statistics Reference Online, 2021.
Aditya Grover, Jiaming Song, Ashish Kapoor, Kenneth Tran, Alekh Agarwal, Fric J Horvitz, and Stefano Ermon. Bias correction of learned generative models using likelihood-fire importance weighting. Advances in neural information processing systems, 32, 2019.
Tim Hesterberg. Weighted average importance sampling and defensive mixture distributions. Techno- metrics, 37:185-194, 1995.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NIPS, 2017.
Geoffrey Hinton, Peter Dayan, Brendan Frey, and Radford M Neal. The "wake-sleep" algorithm for unsupervised neural networks. Science, (5885)))::1188--8 158-1161, 1995.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Aapo Hyvärinen and Peter Dayan, Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems, 26, 2013.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. ArXiv, abs/1710.10196. 2018.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. ArXiv, 0aos/2006006766, 2020.
Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL Dttps://cpenrevieee net forum? d Fu
10

---

Published as a conference paper at ICLR 2023
Diederik P Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2836-2844, 2021.
Alberto Maria Metelli, Matteo Papini, Francesco Faccio, and Marcello Restelli. Policy optimization via importance sampling. In NeurIPS, 2018.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
Art B. Owen. Monte Carlo theory, methods and examples. 2013.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference machine learning, pp. 1278-1286. PMLR, 2014.
Severi Rissanen, Markus Heinonen, and A. Solin. Generative modelling with inverse heat dissipation. ArXiv, abs/2206. 13397, 2022.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. ArXiv, abs/ 100d//666003498, 2016.
Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular
conformation generation. In ICML, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2265.2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ArXiv, abs/2010.02502, 202
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. ArXiv, abs/2006.09011. 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 202 URL https: http:///w/////ee/eeeeeeeee net fforrm?? id==xTIG12RRHS
Adith Swaminathan and Thorsten Joachims. The self normalized estimator for counterfactual learning. In NIPS, 2015.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Compula- tion, 23: 1111666116744 2011.
Chong Wang. X. Chen, Alex Smola, and E. Xing. Variance reduction for stochastic gradient optimization. In NIPS, 2013.
Ziyu Wang, Shuyu Cheng, Li Yueru, Jun Zhu, and Bo Zhang. A wasserstein minimum velocity approach to learning unnormalized models. In International Conference on Artificial Intelligence and Statistics, pp. 3728-3732 3738. PMLR, 2020.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with demoising diffusion GANs. In International Conference on Learning Representations, 2022. URL https:://ppense net forum? dd=-:mi00-jppppp----cm

---

Published as a conference paper at ICLR 2023
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: geometric diffusion model for molecular conformation generation. ArXiv, abs/2203.00997 2022a.
Yilun Xu, Ziming Liu, Max Tegmark, and Tommi Jaakkola, Poisson flow generative models. arXiv preprint arXiv:2209. 11178, 2022b.
Yilun Xu, Ziming Liu, Yonglong Tian, Shangyuan Tong, Max Tegmark, and T. Jaakkola. Pfgm++: inwpeessiiiiiiiiiiee generative models. ArXiv, http2332204266. 2023.
Unlocking the potential of physics-inspired
Shuo Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. From facial parts responses to face detection: A deep learning approach. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 3676-3684, 2015.
12


Reviews:
[{"review_id": "_cGmo6hcmy1", "reviewer": "Reviewer_B7Es", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Score-based generative models are learned via denoising score matching at many time steps during a diffusion process. This paper identifies an important problem with such loss function---estimating this loss via simple monte carlo could have high variance in the middle stage of the diffusion. The authors then propose to use self-normalized importance sampling to reduce the variance, which leads to a new objective function called stable target field. Empirical results show that this new objective consistently improves the performance of score-based generative models. ", "summary_of_the_review": "I find this paper identify an important problem, propose a nice solution, and is sufficiently practical to have large impact. Therefore, I recommend acceptance. ", "strength_and_weaknesses": "## Strengths\n\n* I find the paper very interesting, address a relevant problem in score-based model training, and potentially have high impact. Simply observing the variance problem in the middle stage of training is nontrivial and requires a deep understanding of the denoising score matching idea and the training of score-based models.\n\n* Moreover, the proposed fix is simple and very practical. Both the theory and empirical results suggest the modified loss has lower variance.\n\n## Weaknesses\n\nI do not see much weakness except a few minor points below:\n\n* It would be nice to give a derivation of eq. (7) in appendix.\n* It would be nice to see if there a connection between the modified denoising score matching loss and an importance-weighted version of the ELBO. See, e.g., \n\nBornschein, J., & Bengio, Y. (2014). Reweighted wake-sleep. arXiv preprint arXiv:1406.2751.\nBurda, Y., Grosse, R., & Salakhutdinov, R. (2015). Importance weighted autoencoders. arXiv preprint arXiv:1509.00519.\n\n* since this paper focus on reducing variance of score matching loss, the following paper that proposes a control variate for DSM could be relevant\n\nWang, Z., Cheng, S., Yueru, L., Zhu, J., & Zhang, B. (2020, June). A wasserstein minimum velocity approach to learning unnormalized models. In International Conference on Artificial Intelligence and Statistics (pp. 3728-3738). PMLR.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T0SK1qMjw4H", "reviewer": "Reviewer_M5mL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyzes the denoising score matching objective used to train diffusion models, which have recently become the dominating class of deep generative models. The paper argues that at intermediate perturbation levels the denoising score matching objective is particularly noisy and \"unstable\", because one perturbed data sample could correspond to many different original clean data samples (the paper calls this phase 2; at the same time, we aren't yet in the regime where every data point is almost entirely destroyed and the ground truth score is effectively almost the same standard normal score for all data points -- phase 3 in the language of the paper). Standard denoising score matching uses only a single data sample among the plausible ones to estimate the score in each iteration. This effectively leads to high variance training (although unbiased). The paper therefore proposes to include a reference batch of additional samples to calculate a more stable score estimate. This corresponds to a form of importance sampling, introducing an unknown normalization factor. This problem is circumvented with self-normalized importance sampling, which, however, introduces bias. Ultimately, the paper proposes \"Stable Target Field\" (STF), a biased but lower variance score estimator for training score-based generative models.\n\nExperimentally, the paper shows superior performance of \"Variance-Exploding\" SDE models when trained with STF compared to standard denoising score matching training. Also, training tends to converge quicker. These advantages primarily manifest when using ODE samplers to generate data.", "summary_of_the_review": "In summary, I think the main idea and the proposed method of the paper, the STF for training diffusion models, is interesting, well-motivated, and also novel. I do not have any concerns with respect to the technical and methodological novelty.\n\nHowever, my concern is that the method has been primarily validated on experiments using a standard VESDE scheme, together with ODE solvers for sampling. This setup is known to be challenging and in practice, almost all diffusion models actually rely on a different setup, either the VPSDE or the recent EDM scheme. It isn't clear whether there is any benefit when applied to these methods. If there isn't, then the new method is not particularly important or significant. Consequently, I cannot recommend the paper for publication at this point. I would recommend the authors to run experiments with different SDEs and setups and demonstrate the benefit of STF more broadly.", "strength_and_weaknesses": "**Strengths:**\n\n- The idea to use an additional reference batch to create a more stable and lower variance target score estimate for denoising score matching is novel. It is an interesting, well-motivated and smart idea.\n\n- I like how the paper derives the STF approach and constructs its main method. The analysis of the original denoising score matching objective, followed by the introduction of the reference batch leveraging an importance sampling framework, and then resorting to self-normalized importance sampling to avoid the intractable normalization factor, is elegant.\n\n- The work includes explicit technical analyses on the bias and variance of the proposed STF objective.\n\n- The variance reduction analyses (Figure 3) in the different phases are quite interesting and insightful.\n\n**Weaknesses:**\n\n- It is nice that the paper provides some analytical analyses on the bias and variance of the STF approach. However, how tight is the bound on the STF variance in Theorem 2? I agree that when $p_{0|t}(x|x(t))\\approx p_0(x)$, then the bound is meaningful. But this only really happens in \"phase 3\" (essentially, phase 3 is defined by $p_{0|t}(x|x(t))\\approx p_0(x)$). But the variance reduction also works well in the more relevant \"phase 2\", as empirically shown. But in this relevant regime, where $D_f$ is not yet 0, the bound is very hard to interpret and provides little insight.\n\n- The paper states that Variance Exploding SDEs (VESDE) are the state-of-the-art SDEs used for training score-based generative diffusion models. It is correct that Song et al. [1] achieved strong results with the VESDE when combined with predictor-corrector SDE sampling. However, almost the entire literature on diffusion models uses Variance Preserving SDEs (VPSDE). Moreover, it has recently been shown how to subsume all different common diffusion model SDEs under one umbrella in Karras et al. [2]. This paper then shows how a VESDE-based model can be improved significantly simply by using appropriate model and network parametrizations and preconditionings (their \"EDM\" scheme). Consequently, I would consider this EDM the state-of-the-art approach. Moreover, the main question is, does the STF approach also help when applied to more modern and widely used diffusion models, this is, (a) VPSDE and (b) Karras et al.'s EDM approach? The paper focuses on the vanilla VESDE with ODE-based sampling, even though it is well-known that the vanilla VESDE does not play well with ODE solvers and isn't widely used like that in the literature. In fact, already when sampling stochastically with predictor-corrector sampling, the benefit by STF is very marginal and almost not significant (Table 1, PC sampler (SDE)). Also note that the Karras et al. paper is not cited or discussed at all.\n\n- Note that Karras et al. [2] also discuss different perturbation regimes (see paragraph \"Loss weighting and sampling\" in their Section 5), similar to the different phases discussed in this work. The fact that there are effectively different phases during SGM training is known. This should be acknowledged.\n\n- In Section 3, the paper points to the multimodality of the distribution $p_{0|t}(x|x(t))$. Note that this is extensively discussed also in Xiao et al. [3]. It would make sense to acknowledge that work.\n\n- Regarding \"The STF objective stabilizes the model performance at convergence.\": I think this is overclaiming. The model only shows a significantly lower variance on the evaluation metric for the setup VESDE + RK45 (already when using the SDE framework with a PC sampler, the variance isn't much lower; Table 1). To verify this claim more broadly, again, experiments on other diffusion models should be run (VPSDE, EDM).\n\n- The paper mentions the computational overhead induced by the reference batch. How small or big is this overhead actually? My understanding is that it is small, because the samples from the reference batch are only used to estimate the target score, but we do not have to call the expensive neural network on them. However, what about memory? Some more details would be helpful here. Note that this comment did not influence my rating significantly.\n\n[1] Song et al., Score-Based Generative Modeling through Stochastic Differential Equations, 2021.\n\n[2] Karras et al., Elucidating the Design Space of Diffusion-Based Generative Models, 2022.\n\n[3] Xiao et al., Tackling the Generative Learning Trilemma with Denoising Diffusion GANs, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RZNmV6Jvvs", "reviewer": "Reviewer_Qtdt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to reduce the sample variance for\ntraining score-matching models. The method is based on using multiple\nreference samples and weighted importance sampling to compute more\nstable targets. Small-scale experiments show improvement when using\nthe proposed scheme.\n", "summary_of_the_review": "The paper proposes an interesting idea for improving score-matching training. On the flip side the paper presents limited\nexperimental evidence in favor of the proposed approach.\n", "strength_and_weaknesses": "**Strengths**\n- I found the paper interesting to read, with the teorethical\n  part well  explained.\n- The paper demonstrates in simple examples the high target variance\n  of vanilla score-matching, and how it is reduced when using the\n  proposed approach.\n- The approach improves the sampling of the baseline methods, without\n  incurring in extra training time. In the case of CIFAR-10, it\n  significantly improves the training efficiency.\n\n**Weaknesses**\n\n- The empirical evidence in favor of the method is rather weak, with\n  experiments showing some improvements only for the small CIFAR-10\n  benchmark.\n- In Figures 6 and 7, FID is computed on only 1K samples, yet the\n  Inception features are typically of higher dimension, so the Gaussian\n  approximation for the FID would be ill-conditioned. I wonder if the\n  authors can comment on that (Curves show scores of ~40, while ~2 is\n  reported in Table 1.)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster