High Precision Score-based Diffusion Models
Authors: Dongjun Kim
Seungjae Shin
Kyungwoo Song
Wanmo Kang
Il-chul Moon
Venue: ICLR 2022 Submitted (2022)
Keywords: Diffusion Model
Score-based Model

Abstract: Recent advances in diffusion models bring the state-of-the art performance on image generation tasks. However, the image generation is still an arduous task in high resolution, both theoretically and practically. From the theory side, the difficulty arises in estimating the high precision diffusion because the data score goes to $\infty$ as $t \rightarrow 0$ of the diffusion time. This paper resolves this difficulty by improving the previous diffusion models from three aspects. First, we propose an alternative parameterization for such unbounded data score, which theoretically enables the unbounded score estimation. Second, we provide a practical soft truncation method (ST-trick) to handle the extreme variation of the score scales. Third, we design a reciprocal variance exploding stochastic differential equation (RVESDE) to enable the sampling at the high precision of $t$. These three improvements are applicable to the variations of both NCSN and DDPM, and our improved versions are named as HNCSN and HDDPM, respectively. The experiments show that the improvements result in the state-of-the-art performances in the high resolution image generation, i.e. CelebA-HQ. Also, our ablation study empirically illustrates that all of alternative parameterization, ST-trick, and RVESDE contributes to the performance enhancement.

Under review as a conference paper at ICLR 2022
HIGH PRECISION SCORE-BASED DIFFUSION MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent advances in diffusion models bring the state-of-the art performance on image generation tasks. However, image generation is still an arduous task in high resolution, both theoretically and practically. From the theory side, the difficulty arises in estimating the high precision diffusion because the data score goes to oo as + 0 of the diffusion time. This paper resolves this difficulty by improving the previous diffusion models from three aspects. First, we propose an alternative parameterization for such an unbounded data score, which theoretically enables the unbounded score estimation. Second, we provide a practical Soft Truncation trick (ST-trick) to handle the extreme variation of the score scales. Third, we design Reciprocal Variance Exploding Stochastic Differential Equation (RVESDE) enable the sampling at the high precision of t. These three improvements are applicable to the variations of both NCSN and DDPM, and our improved versions are named as HNCSN and HDDPM, respectively. The experiments show that the improvements result in the state-of-the-art performances in the high resolution image generation, i.c. CelebA-HQ. Also, our ablation study empirically illustrates that all of 1) alternative parameterization, 2) ST-trick, and 3) RVESDE contributes to the performance enhancement.
INTRODUCTION
Recent advances in the generative models enable creating of highly realistic images. One direction of such modeling is likelihood-free models (Karras et al. 2019 based on the minimax training. The other direction likelihood- bassdd models, including VAE Vahdat and Kautz (2020), autoregressive models (Parmar et al., 2018), and flow models (Grcié et al, Diffusion models (Ho et al.) 2020) are one of the most successful likelihood baased models where the generative process modeled by the reverse diffusion process. The success of diffusion models achieves the state-of-the-ard performance in image generation (Dhariwal and Nichol 2021 Song et al. 2020).
Despite of such success, the score-based diffusion models still struggle in generating realistic images with high resolution (Jolicoeur- Martineau et al. 202T). Rather than improving the network architecture, this paper focuses mostly on the theoretic and the optimization aspects of the diffusson model to improve the sample quality. We first observe that the diffusion time acts in distinctive ways on the sample generation: the diffusion process with the front diffusion time (f Rs 0) influences the local sample fidelity, and the end diffusion time (t R T) shapes the sample global structure. However, the unbounded data score the front diffusion time results in the training instability, and this instability yields the diffusion model with poor estimation limiting the local sample fidelity, which prevents the advancement in the high resolution because of its locality. Hence, this paper constructs a diffusion model capable of training both front time and end time, effectively.
We observe that the current score parametrization is incapable of estimating the data score that blows up as + O, which requires a new theory-driven parametrization for the unbounded score network to enable the successful score estimation near f O. This parametrization is universally applicable for any diffusion models including NCSN (Song and Ermon| 2019) and DDPM (Ho et al.. 2020). and we call this parametrization as High precision NCSN (HNCSN) and High precision DDPM (HDDPM), respectively. Second, we propose a practical optimization method (ST-trick) to improve the score estimation at the end of the diffusion time (t zs T). This ST-trick provides a distinctive way to optimize the diffusion model parameters, and we argue that this trick successfully trains the score network at the end diffusion time by its nature, which is likely to be ignored due to the minor contribution to the diffusion loss near L R T in the previous studies. Like the new parametrization,

---

Under review as a conference paper at ICLR 2022
ST-trick fits to any diffusion model, including NCSN/HNCSN and DDPM/HDDPM, Finally, suggest a new SDE (RVESDE) that 1) mitigates the theoretic dilemma of VESDE that arises as + 0 and 2) gives additional performance boost by reducing the Monte-Cart variance for the diffusion loss. We perform the ablation study on our three improvements, and we find that using 1) the new parametrization, 2) ST-trick, and 3) RVESDE all together improves both density estimation and sample fidelity on benchmark datasets. Although we proceed our arguments based on the continuous diffusion models, we emphasize that both new parametrization and ST-trick are applicable for discrete diffusion models, such as ADM (Dhariwal and Nichol 2021 and CDM (Ho et al. 2021). as well.
PRELIMINARY
The diffusion model trains the network by minimizing the Negative Evidence Lower BOund (NELBO)
log (x S C(O;) =g²) {' ())),
(NELBO)
Ep.txa)
(1)
where
(0) Ex, [|)soxX,,) Vx, log pt (x:) Ex,,x, |1s5o t) Vx. log Poe (x, (x) [x))]
up a constant. Here, À is the weighting function for the diffusion loss of L. (0) at each diffusion time, Pe(x;) ) the marginal distribution of the diffusion process of [x, ) and pue (xe (xo) is the transition probability defined by N (xx: ((()xo, (())), where μ(t) and o (1) are determined by the governing diffusion process, Previous works suggest couple of diffusion processes: 1) VESDE dx: = g(l) dw,
with gt)) Omin (Laxx) )&, /2 2 log amin ), and 2) VPSDE dx, 3()(( dt + BB( dw, with
3(t) 3min + (Bmax 8min )t. With these diffusion schemes, VESDE has ptt)
with (1(1)= = omii [[ ([maz) j2t - 1]. and VPSDE has ju(t) = e à Jo ds ds with o² = 1 - e Já ds
We observe that the transformation of yt = es Já 8(s) ds x, reduces VPSDE to VESDE of dy: = et So [(s) /3tt) dw, by the Ito's lemma OKsendal 2013). so VPSDE and VESDE are indeed equivalent representations of the linear diffusion. Given their equivalence under the suggested transformation, we build our model based on VESDE as default diffusion otherwise stated.
2.1 EFFECT OF DIFFUSION TIME ON DIFFUSION MODEL
Density Estimation In practice, previous diffusion models truncate the diffusion time from [0, 7 to (e, 7"] to stabilize the training. Figure emphasizes the loss contribution the NELBO at t -> 0, which signifies lowering in the practice. Moreover, Figure 2 2shows the cumulative loss of Figure integrated over (a, maz by changing the integrating variable from / to in Eq. 2
T
Figure I: Loss contribution to NELBO by diffusion time.
and the high precision (i.e., a Ri 0 or 0) has significant portion on the NELBO.
mmnn
[(0:g²) 2
vL (0)
Therefore, lowering min (c) guarantees the tighter NELBO given its large portion at = c. Empirically, NCSN++ (Song et al. 2020) performs 3.45 for NELBO trained with Junin. 10 on CIFAR- 10, but this NELBO is tightened up to 3.04 by lowering Tmin to 10-3 in our experiments.
lead (be)
Figure 2: NELBO by diffu- sion level.
Sample Generation While we discussed the value of the loss at t-+0, O. both fron and end diffusion time contribute the sample quality in two distinctive ways.
Diffusion Time Horizon
End Time (L + T): The diffusion model creates a fake sample by reverting the diffusion process with the score estimation. Figure shows two contrastive image genera- tions: the realistic image at the first row, and the unrealistic image at the second row with curly hair on its forehead. This unwanted hair is synthesized the early steps of the reverse diffusion, which correspond to the end time of the forward diffusion. Therefore, Figure implies that the overall sample quality determined on the range of the end diffusion time.
Good
Not Good
Figure 3: Generative process.
2

---

Under review as a conference paper at ICLR 2022
Front Time (t 0) The sample generation in Figure 4 shows that the local fidelity heav- ily depends upon the denoising process at the front of the diffusion time. This intuitive re- sult is grounded by theory: observe that the random variable |x; x xoll//2 (t) follows the xr-distribution of degree d, where is the data dimension. If the score network exactly esti- mates the data score, the generated image time would follow x,, and the average distance between
Figure 4: Generated images from CelebA-HQ
256 x 256. Image quality (NIQE (Mittal et al.
2012 improves as the denoising proceeds.
the real sample (Xo) with the fake sample (x,) becomes (x. (ep (x, (xa) []] X, xo l o m d.
Therefore, even if the score network perfectly estimates the data score, this leads that minimizing min acts in a crucial way for the pixel-wise precision.
From these observations, we identified that the front diffusion time critical in generating the high resolution image for the pixel-wise details, and we also acknowledge that the end diffusion time contributes to the overall image structure. Now, our question becomes whether the existing models can learn the data score at both front and end diffusion time.
2.2 PROBLEMS OF TRAINING THE DIFFUSION MODEL WITH HIGH PRECISION
Unbounded Score Figure presents that the data score is unbounded as the diffusion level, (t). diminishes. This requires the score parametrization to handle this unboundedness.. For instance, the parametrization of so (x,, (t)) yields the failure of the score esti- mation at t 10-4 on the toy example in Figure
Difision leed (eg)
Figure 5: Unbounded score.
Theoretically, Lemma [ states that the score estimation with so (x.,,) (DDPM) or se (x (NCSN) fails if the data score is unbounded. See Appendix for the proof.
{s Rd [0, + Rd, is locally Lipschitz). Suppose a continuous vector field defined subset U a compact manifold M (i.e, v U C M - Rd) ) is unbounded, then there exists no s E H. such that lim, s(x, t) = v(x) a.e. on U.
Lemma 1. Let Hju,1
The score network, So. is an element in HO.T for any neural architecture as long as we put in the network, Lemma indicates that the vanilla score network cannot estimate unbounded data score.
Figure 6: Score estimation of NCSN and HNCSN toy dataset.
Training Instability Training the diffusion time on the
front range is advantageous on the creation of high-quality samples, However, training the diffusion model with lowered amin induces the numerical instability, which eventually harms the overall sample quality.
To analyze the logic of the training instability, Figure 7 illustrates the Monte-Carlo samples of the diffusion loss, which blows up as - 0 (or - 0). This leads that the diffusion loss is dominated by the diffusion time with the front range in practice, and the loss at the end diffusion time is barely counted in the gradient signal. This imbalanced loss brings the immatured score estimation at the end diffusion time, which ruins the overall sample shape.
Figure 7: Imbalanced Monte- Carlo losses.
Limitation of Diffusion Process Continuous VESDE has its dif-
fusion variance be o² (t) o² main (Immsx) )2r - -1] As introduced in Song and Ermon 2020), VESDE has been suggested in its orig- inal discrete version based on the geometric property for its smooth transition of the distributional shift. Mathematically, the variance is geometric di log o² (t) is a constant, but the continuous VESDE breaks this geometrie progression as in Figure X
Difisio tine
To make it geometric, Soone et al. (020) approximates the VESDE diffusion process by the transition probability of Poo (x, xo)
Figure 8: VESDE violates the geometric progression.
3

---

Under review as a conference paper at ICLR 2022
N [(xx; Xo, amin (2 II so this approximate diffusion process satisfies the geometric progres sion. However, this approximation leads that is not converging to xo in distribution because min (omnx)) )at min > 0 as 0.
Proposition 1. There is no SDE that has the stochastic process (x.) defined by transition
probability Poo(x) (x)))x)) N (xx; Xo, o min
(Iuux)) I), the solution.
Omen
Proposition indicates that we approximate the diffusion process by Por (xx(xx)) N (xr; Xo, o min @men I then the reverse diffusion cannot be modeled by a reverse SDE, There-
fore, VESDE loses its theoretic ground on the continuation of the discrete Markov chain.
HIGH PRECISION SCORE-BASED DIFFUSION MODELS
Given the three problems of unbounded score, training instability, and limited diffusion process, this section improves the diffusion model in three corresponding aspects. Section 3.1 introduces the theory-driven new parametrization, called HNCSN and HDDPM, that enables the successful score estimation for the diffusion model with small min- Section 3.2 proposes the practical technique, ST-trick, which could estimate the score network the end of the diffusion time without being dominated by the front diffusion. Section 3.3 suggests the new SDE, RVESDE, which develops the VESDE suitable for small mmin-
3.1 PARAMETRIZATION MODEL FOR UNBOUNDED SCORE
As presented in Lemma the score estimation of baseline models (NCSN/DDPM) fails because the locally Lipschitz with respect to the time argument does not hold anymore at t = 0. From this, we make the time argument extendable to the infinity t = O. To proceed, let us define n : H IR as function that satisfies 7)(1) - oo or -00 as + 0. Then, the parametrization of so sa,,,, (()) would successfully estimate the data score by Proposition
{s Rd xx) - Rd s is locally Lipschitz). Suppose a continuous vector field v defined on a d-dimensional open subset U of a compact manifold M is unbounded, and the projection of each axis is locally integrable. Then, there exists H(,,s)) such that
Proposition 2.
Lei H(
lim s(x, 11) vxx) a.e. on U.
Table 1: Instantiation of the unbounded parametrization from Proposition pendix B for the choice of C1, C22 Ou-
Proposition 2 becomes the foundation of the un- bounded parametrization, and the introduction of 7 is the key to enable the estimation of the unbounded data score. Therefore, we intro- duce the new parametrization for the high pre- cision score network as so (x(,,, ntt)), rather than so (x, f) (DDPM) or so (x,, a (f)) (NCSN).
I See Ap-
Model
v)(t)
(()) c +
i 2 da if <
HNCSN
v(t))
HDDPM
T)(t) =
20 de (
Table presents example parametrizations following Proposition 2 We suggest HNCSN with (t) as the mixture of log (t) at the end diffusion time and o[e] at the front diffusion time, in which the
inverse function is beneficial on estimating the rapidly blowing data score by orders of 2. Note that
Song and Ermon 2020) proposed the score parametrization of (a(x.) and Misicocur-Maatincau et al.
(
(2200) showed that the optimal score function becomes s" (xx, t) = Ex [xo]
o(1)* where
(1)
2 I). Therefore, we suggest HNCSN to put a(t) in place of log a (t) at the front time.
Before choose optimal 77 for DDPM, observe that DDPM with the importance sampling has (1)
(P(()
as the importance weight, and this weight forces the sampled diffusion time highly concentrated on t = c. Since small difference between two diffusion times near f1 t2 R c results in large deviation of data scores in their scales, DDPM cannot approximate this extreme scale deviation with the vanilla parametrization of Sp (x,, () by Proposition
We propose HDDPM with r7(t) =
(t) (2(
dt as the antiderivative of the importance weight. For
instance, HDDPM with VPSDE has its antiderivative
21) dt log (1 -e (s)) o=(()
4

---

Under review as a conference paper at ICLR 2022
Jo B(s) ds. This antiderivative diverges to -00 0 for both VPSDE and subVPSDE, so it satisfies the condition of Proposition
The choice of such (tt for HDDPM helps the enhanced score esti- mation near t R because the input of 7t(t) is distributed uniformly when draw samples from the importance weight. Concretely, when the sampling distribution on the diffusion time is given by g'tt)
100%M
p(t) x oa(Ej* the 17distributtion from the importance sampling be- comes p(n) x 1, which is depicted in Figure 9 Therefore, this uniform distribution on (t() in HDDPM guarantees the high preci- sion to the score estimation.
Moodded (
Figure 9: Cumulative density function of t and r).
3.2 SOFT TRUNCATION TRICK FOR PRACTICAL SCORE ESTIMATION
While the unbounded parametrization suggested in Section 3.1 provides the theoretic guarantee the score estimation, this section proposes practical optimization method for the high precision score estimation. In order to successfully train the score network without sacrificing the global fidelity, we detour the training instability by introducing the Soft Truncation (ST) trick that is taking distinctive diffusion truncation for every mini-batch. In formula,
(sr(O; X, = Er(r) (C(;))
where P(T) is the prior distribution for the diffusion truncation. In every mini-batch update, we optimize C(0; X, with the sampled r F(r) from the prior. Figure iillstrates four cases: blue dots with 7 e, green dots with T = o (0.01), etc. With ST-trick, mini mini-batch with high truncation of T 0 (1) (purple) focuses on the score estimation at large t, and mini-batches with medium a- 10.1) (green) and o -0000) (red) focus on the middle region of (c. 7). In consequence, ST-trick trains the diffusion model by partitioning its diffusion time for each mini-batch update.
Table 2: Example pairs of Ap).
The ST-trick with P(r) (T) unconditionally selects e for every batch update, so is equiva- lent with the original optimization of L(O; X, e). Although is free to choose F. we find that the reciprocal functions work best in practice. Ob- serving that # (r.r(())/ - (T) as k 0o where Z is the normalizing constant, the recipro- cal functions are connected to the original optimization. We enumerate few examples of k = 2 and k in Table 2 for RVESDE and VPSDE, respectively. Note that ST-trick is independent the choice on the weighting function À for the batch-wise optimization loss, A,)
k
A
X(t)
(,)))
A(8)
RVE
A(8)
2
+ (1mm(()))))
(1- ()A())
A(t)
(,(r)
A(t)
A(1)
1
(1j2((())
(1 log, (0)(()
Theoretic Analysis of ST-trick The ST-loss reduces the original diffusion loss to (0;
C(O; Ap. e) with Xp()) := (J²² (P(T() dr) in expectation by the Fubini theorem Stein and Shakarchi, 2009). While the ST-loss in average is identical to the original diffusion loss, the major difference lies on that 1) L(0; Xp, c) optimizes the diffusion loss in its averaged form and 2) LST(0: ;0;AA X, IP) with ST-trick optimizes the diffusion loss with the Monte-Carlo sampled truncation, т.
To inspect how soft truncation affects to the optimization, let us assume X(t) = g²()). Then, we observe that the theoretic analysis previously suggested in Song et al. (2021) is directly extendable any E (0, by simply switching the positive truncation time, T, in place of the zero time:
(x=) log (x+)] < L(0
where pe,r is the marginal probability distribution at time of the reverse (generative) process that starts from the prior distribution and flows backwards in time, and p, the marginal probability density at time 7 of the forward diffusion process. Then, the above inequality is equivalent to
D (p-lloo S L(0: T),
after omitting constants independent for the above two inequalities. This KL bound means that ST-trick simply ignores the time range beneath and optimizes the proxy of the KL divergence between the forward and the reverse diffusion at time т. On the other hand, no such interpretation can be applied to the diffusion loss without ST-trick, L(O; Ap,
5

---

Under review as a conference paper at ICLR 2022
3.3 RECIPROCAL VARIANCE EXPLODING SDE FOR SMALL TRUNCATION
This section introduces RVESDE whose variance, (1), is geometric by its nature. RVESDE is defined as the form of dx, =g(() dw, with
g(t) { mmax Omax
2e log ( "minn
(omins
if > 0, i = NF
The transition probability of RVESDE becomes (xr(Xx(Xo) = N (xx:; X0, maa
(amin 2
Recall that VESDE assumes g(1) = Omin Ommnn
2 2lgg (fmaa) and Po: (xx Xo)
N ((x;XXo Xo; minn [( (Tmax) ) I] As presented in Figure RVESDE attains the geometric property the expense of having reciprocated time, 1/t. We could successfully model the forward diffusion process with RVESDE without any approximation, and this exact modeling of the forward process enables applying the stochastic calculus to compute the log-likelihood or to generate new samples.
Omen
It is common to apply the importance sampling (Song et al., 2021 in order to minimize the Monte- Carlo variance on the diffusion loss as illustrated in Figure (c). RVESDE is advantageous on VE/VPSDEs that the importance sampling is easy to implement: the diffusion loss with RVESDE becomes
T g (0) clog [-
[(0: g2 2
mmaz
o2 (
Tmin
by substituting the integrating variable into the reciprocated time. Therefore, the importance sampling for RVESDE is equivalent to drawing the diffusion time uniformly on a reciprocated interval, [ 2.
EXPERIMENTS
This section empirically studies our suggestions on benchmark datasets. including CIFAR-10 10 (Krzzhevsky et al. 2009). 10 (Coates et al. CelebA
I (Liu et al. al. 2015)) ) 64 x 64, CelebA-HO KKrras et al. 2011 7) 256 x 256, LSUN (Yu al. 2015) Bedroom/Church 256 x 256, and FFHQ (Karras et al. 2019). We compute bits-per-dim, proportional to NL.., by applying the Instantaneous Change of Variable Chen et al. ) Song et al. (2020) to the probability flow (Song et al. 2020) with the uniform dequantization (Theis et al. 2015; Hoogeboom et al. 2020). rather than the vari- ational dequantization (Ho et al. 2019) in order to sidestep the need of training an auxiliary flow network. Also, we use the FID introduced in Parmar et al. (202I) for computing the FID (Heusel et al. (2117) score with the predictor-corrector sampler (Song et al. 2020). Training and evaluation details are available in Appendix B
4.1 QUANTITATIVE RESULTS
Table 3 compares HNCSN++ (RVE) with ST-trick against the current best generative models, and performances. On CIFAR-10, we observe that our model improves NCSN++ (VE) in NLL and IS at the expense of sacrificing FID slightly. In particular, our model surpasses the previous best IS performance SSyyleGAN2-ADA++Tummn (Karras) et al. 2020) by performing 10.11 11 on 100. HNCSN+ trained on CelebA is the hest performer in terms of the FID performance, but CR-NVAE (Sinha and Dieng 2021) outperforms our model in NLL. However, we emphasize that HDDPM++ in Table O with ST-trick (1.52) outperforms CR- NVAE (1.86) in NLL by far. In particular, HNCSN++ with high- high-dimensional CelebA-HO 256 x 256 performs the state-of-the-an the GAN models (PGGAN (Karras et al.) 2017)). Finally, HNCSN+ on STL-10 dataset largely outperforms the baselines by improving the state-of-the-art FID from 15.17 (Park and Kim! 2021) to 7.71, and IS from 11.01 (Park and Kim 202T) to 13.43.
Table shows that our model establishes the state-of-the
in FID out of the baselines in the diffusion models (NCSN++) and
In Table when computing the Inception Score (IS) (Salimans al. 2016). there is minor discrepancy between likelihood-based models and likelihood-free models. IS-50k in diffusion models
'The existence and uniqueness of solution for RVESDE is guaranteed by Theorem 5.2.1 in Oksendal 2001
We downsize the dataset from 96 x 96 48 48 following Jiang al. 2021) Park and Kim 202T

---

Under review as a conference paper at ICLR 2022
Table Performance comparisons on benchmark datasets. The boldfaced numbers present the best performance, and the underlined numbers present the second-best performance.
CITARIO
CelehA
CelebA-III IIQ
STL--10
Model
64
256 256
NLL (.)
FID (1)
15 (1)
NLL
FID
FID
FID
IS
(RVE)
1,93
1.92
7.71
Likelihoud- bused Models
CR-NVAE (Sinha Diena,
1.86
LSGM (FID) Valldia et di 2024
3.43
2.10
DenseFlow-244 Flow-l4--11 74-10 (Crece
2.98
1,99
Gamma Distribution DDIM (Nachmani et al
2021
2.92
VDM Kingma al. 2021
2.65
NCSN++ coMt. (dscp, VEJ et al 2020
2.20
9.89
2.39
3.95
7.23
DDPM- cont. (doep, sub-
Nowht MD 2010
2.99
2.41
9.57
ScorcFlow
norm. flow)
2021
Improved DDPM (Iampii) Netiol and Diarwal 2021
3.37
2.90
Likeliheedd-ree Models
StyleGAN2-AD BAAANN--AA-TiRI ADA-TTuning Karms el 2020
2.92
10.02
Stylefomerr (Park and Kim,
2.82
9.94
3.66
15:17
11.01
PGGAN (17 2017
8.03
TransGAN mang m 2024
9.26
9.02
5.01
18.28
10.43
(Song et al. 2020: Ho et al. 2020) computes the score with 50k generated images once, while IS-5k in GAN models (Karras et al. 2020; Park and Kim, 2021) computes the score 10 trials with independently generated 5k samples and report the performance the average of scores. We report IS-50k following Song et al. 2220), but we note that IS-5k performs almost identical to IS-50k by performing 10.07 CIFAR- 10 and 13.34 on STL-110 in terms of IS-5k.
One particular observation in Table the IS performance (13.34) of STL-10 that exceeds the number of classes (10) of the dataset. We follow Park and Kim (2021), the current state-of-the-ar model on STL-10, to train STL-10 with 105k images by aggregating the labeled (5k) and the unlabeled (100k) images Tiang al. 202T). T). Although the labeled images have 10 classes, the unlabeled images sampled from a broader distribution of images, so the unlabeled dataset contains the other types of animals such bears or rabbits, whose classes are different from the labeled classes. Therefore, well-trained model with 105k images would perform IS that exceeds the number of labeled classes
4.2 ABLATION STUDY
Throughout the ablation study, we train the model on CelebA default otherwise stated.
High Precision Parametrization Table 4 presents the model performances after 0.5M of training iterations. First, NCSN+ with lowered miin effectively reduces the performances on both NLL and FID, as anticipated in Section 2 Second, the new parametrization of HNCSN gives slight performance gain compared to the original parametrization of NCSN. This perfor- mance gain comes from the better estimation for the data score, evidenced by Figure 13 (a). Third, the new parametrization HDDPM largely improves the sample quality at the expense of slight NLL degradation.
Table Ablation study for the new parametrization.
Model
aminn
NLL
Celeba
NCSN-+ (VE)
10
2.39
3.95
NCSN-+ (VE)
10-2 8
1.96
HNESN++ (VF)
10-3
1.97
3.54
IICCNN+
10-3
1,97
DDPM- (S)
10-3
1.53
HDDPM++ (VP.18)
10-3
1.64
4.65
Soft Truncation Figure 10 compares the diffusion model with/without ST-trick in terms of the sample fidelity by training iteration. To compare in fair setting, we train the model in pair of comparative settings: first, we train the model with vanilla diffu- sion loss of L(;; Xp, c) with (p( (1 (t); and second, we train according the same loss with ST-trick of A()) (L) and P(T) x 3.2 At the initial stage of training. the sample fidelity with ST-trick worse than that without ST-trick because the model with ST-trick learns the data score of small diffusion slower. However, ST-trick effectively updates the score network as training proceeds, and Figure shows that the model with ST-trick surpasses the model without ST-trick after the training.
Traiing Seaaiee
Figure 10: ST-trick improves the sample generation.
7

---

Under review as a conference paper at ICLR 2022
Table 5: Ablation study for ST-trick trained on HNCSN+ (RVE).
Table 5 presents the performances after 0.5M of training steps. It compares the diffusion models with an identical weighting function, (1 $)9², with/without ST-trrik,, and it shows that ST-trick improves NLL from 1.98 to 1.93 and FID from 3.18 to 1.92, where 1.92 beats the current state-of-the-art FID (2.92) by far.
P(T)
Opúiication Loss
ST
Celehi
NLL
(x(r)
2(0.s².0 2(0:
1.97
6.6
1.88
3.18
Cor
1,93
1.92
Table T compares our ST-trick with the vanilla optimization of the hard truncation on weighting functions of X(t)
Table 6: Ablation study for ST-trick.
(t) and X(t) = (t). Table O shows that ST-trick is effective on both sample generation and density estimation. In particular, for any model settings, we conclude that ST- trick is a highly effective method that significantly enhances the sample quality without any of model modification.
Model
A
ST
NI.
CelebA
FID
NCSN+ (VE)
a²
x
2.39
3.95
FINCSN+ (RVE)
x²
/
x
1,97
193
3.36
RVESDE Table 4 shows the efficacy of RVESDE on HNCSN++. Applying RVESDE improves the FID score while preserving the NLL performance. compared to
DDPM4 (VP)
/
x
1.79
1.75
2.88
HDDPM-+ (VP)
o²
/
x
1.78
1.73
2.22
3.23
VESDE on HNCSN++. To amalyze the performance, recall
2 that Proposition implies that the inequality of NELBO in does not hold anymore, the NLL performance of VESDE loses its theoretic ground on bounding the log-likehn log-likelihood. However, our RVESDE allows the inequality of NELBO, so the NLL performance in Table is theoretically grounded. In addition, the experiments in Table 6 shows that RVESDE harmonizes with ST-trick better than VESDE.
T
DDPMH (VP,
g²
/
x
1.52
1.53
4.50
5.31
HDDPM- (VP.
8²
x
1.64
4,65
1.52
Table 7: Ablation study for amin trained on HNCSN+ (RVE).
Optimal Precision Table 7 finds that lowering min too small might harm the sample quality due to the training instability. We find that min = 10-3 is a sweet spot for the optimal sample quality. This tendency maintains even when we apply the training stabilization method, such ST-trick.
CIFAR-10
mmnn
NLL
PID-100
10 3
2.96
7.04
The reasoning of such opposite trend could be partially explained by how the data is saved. When we save the data, it is 8-bit quantized in order to downsize the file size, so it is enough to downscale Omin less than 1/256 to fit the model distribution to the data distribution, which is approximately 0.004. Therefore, lowering min too small incurs severe numerical instability without gaining additional performance enhancement.
4
2.97
8.17
10-3
2.97
8.20
10-3
5.09
10
2.97
5.54
5
2.96
Empirical result in Figure indicates that Gmin of 10-3 3 is indeed enough to create high precision images 256 x 256 dataset. Figure Ti illustrates the histogram of the image-by- image quality metric, evaluated by Natural Image Quality Evaluator (NIQE) (Mittal et al., 2012). with 5k samples. Figure II implies that the denoised images up to the diffusion levels of 0.1 or 0.01 are not satisfactory in terms of this mmae-by--mme discrepancy of the mage-to-imaage quality metric explains the gain in FID when we lower amin in Table where FID is distributional metric for sample quality.
nigh-dimensional FFHQ
Geswrated =0
Ganarded 0.22)
metric. In fact, this
Aenus Image Quality Enkkator IN DE)
Figure 11: Optimal preci- sion on FFHQ 256 x 256.
High Dimensions datasets, we train HNCSN++ (RVE) with ST-trick for miin 10-3
To clarify the modeling effect on high dimensional
Jown 30
MCSA
on datasets of 256 x 256 resolutions. After the training of 3.3M
iterations, we draw 5k samples by stopping the reverse process at
1101 2 /10-3 When we denoise up shows that our model largely improves the sample quality: HNCSN+
10 Table 8
Natural Image Quality Evaluatio (NIQE)
Figure 12: Image-by-image
outperforms by reporting 13.52 and 14.20 in FID-5k compared the
quality histogram on CelebA-HQ 256 x 256.
NCSN++ of 16.42 and 29. on FFHQ 256 x 256 and LSUN Church 256 x 256, respectively. Such huge discrepancy on high-dimensional
datasets is attributed to the poorly synthesized images from NCSN++, which is evidenced by the right
8

---

Under review as a conference paper at ICLR 2022
MedioCiato Varanee
VESDE:
RVESDE:
(a) Estimated Score
Loss Density
Loss Variance
Qualitative experimental results for (a) score estimation, (b) Monte-Cart loss density of ST-trick, (c) Monte-Carlo loss variance without ST-trick.
Figure 13:
Figure 14: Generated images from FFHQ (Ist column) / FFHQ 256 (2,3th columns) / CelebAHQ (4,5th columns) LSUN Bedroom (6,7th columns) / LSUN Church (8,9th columns)
heavy tail in Figure 12 Also, as we denoise further from 10 10-2 to 10 the sample quality improves
from 13.52 to 10.35 FFHQ 256 256 and 14.20 to 13.75 on LSUN Church 256 256.
Table 8: Ablation study for denoising effect on high- dimensional datasets.
4.3 QUALITATIVE RESULTS
This section presents the qualitative experimental results for the proposed methods. Figure 13 (a) compares the score estimation for HNCSN+ and NCSN++ CelebA. HNCSN++ succeeds in estimating the unbounded score, whereas NCSN+ shows poor estimation the front diffusion time (or small diffusion level). Figure [3 (b) compares the diffusion model with/without ST-trick terms of the Monte-Carlo loss density. As expected, the density is bimodal for the model without ST-trick, but it turns out that this heavy tail is largely improved by ST-trick. Figure 13 (c) illustrates that the Monte- Carlo loss variance reduced with RVESDE. compared to VESDE. We attribute this variance reduction to the exact modeling of the diffusion process.
FFHQ
Model
RID-5k
FID-SL
HINSSK+
10-1
60.56
48.06
10
13.52
14.20
10
10.35
13,75
10- 2
Figure 14 shows the generated images from various benchmark datasets. We find that our model succeeds generating realistic images on high-dimensional datasets with resolution greater or equal to 256 x 256. See Appendix for the full image generation results.
5 CONCLUSION
This paper tackles the problem of exploding data score with three points: 1) we provide a theory- driven parametrization for the high precision estimation; 2) we introduce a practical optimization method of the diffusion model parameters; and 3) we suggest a new type of SDE that enables to apply the stochastic calculus small diffusion level. Applying the three improvements to NCSN/DDPM, we presented the high precision version of NCSN and DDPM, which excels in generating the high resolution images.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Potential risk from this work is the negative usage of the deep generative models, such as deep- fake images. Our contribution enables the creation of the high resolution virtual images, and acknowledge that there is any chance of misuse for malicious purposes.
7 REPRODUCIBILITY STATEMENT
Complete proofs of the theoretic analyses in the main paper are provided in Appendix[A] The code for the experiments would be released the reviewers at the discussion phase through anonymous repository for the reproducibility. Most of our code is built based on the released code of the previous work (Song al. 2020) including the network structure and hyperparameters, as explained in Appendix
REFERENCES
Anokhin. I., Demochkin. K., Khakhulin. T., Sterkin, G., Lempitsky. V., and Korzhenkov, D. (2020). Image generators with ceaiitionnall-indeeedeeee pixel synthesis. arXiv preprint 13Xiv:2011.13775.
Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. (2018). Neural ordinary differential equations. arXiv preprint arXiv: vrXiv:/806.07366
Coates, A., Ng, A., and Lee, H. (2011). An analysis of single-layer networks in unsupervised feature learning. Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-20 223. JMLR Workshop and Conference Proceedings.
Dhariwal, P. and Nichol, A. (2021). Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2105.05233.
Evans, L. C. (1998). Partial differential equations. Graduate studies in mathematics, 19(2).
Greić, M., Grubišić, I., and Šegvić, S. (2021). Densely connected normalizing flows. arXiv preprint 1rXiv:210600402
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017). Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30.
Ho, J., Chen, X., Srinivas, A., Duan, Y., and Abbeel, P. (2019). Flow++: Improving flow-based
generative models with variational dequantization architecture design. International Conference Machine Leurning, pages 2722-2730. PMLR.
Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.1 1239.
Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T. (2021). Cascaded diffusion models for high fidelity image generation. arXiv preprint arXiv:2116. 15282.
Hoogeboom, E., Cohen, T. S., and Tomezak, J. M. (2020). Learning discrete distributions by dequantization. arXiv preprint arXiv:2001.11235.
Jiang, Y., Chang, S., and Wang, Z. (2021). Transgan: Two transformers can make one strong gan. arXiv preprint arXiv:2102.0707. 07.07744.
Jolicoeur-Martineau, A., Li, K., Piché-Taillefer R., Kachman, T., and Mitliagkas, I. (2021). Gotta go fast when generating data with score-based models. arXiv preprint a00i 140000
Jolicoeur-Matinnaau A., Tiché-Taillffer R.. Combes. R. T. d.. and Mitliagkas, I. (2020). Adversarial score matching and improved sampling for image generation. arXiv preprint arXiv:2009.05.4
Karras. T., Aila, T., Laine, S., and Lehtinen, J. (2017). Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 1710. /0166.
10

---

Under review as a conference paper at ICLR 2022
Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., and Aila, T. (2020). Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.06676
Karras, T., Laine, S., and Aila, T. (2019). A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVP Conference on Computer Vision and Pultern Recognition, pages 4401-4410.
Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6900.
Kingma, D. P., Salimans, T., Poole, B., and Ho, J. (2021). Variational diffusion models. arXiv preprint arXiv:2107.006300
Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.
Lin, C. H., Chang, C.-C., Chen, Y-S., Juan, D.-C., Wei, W., and Chen, H.-T. (2019). Coco-gan: Generation by parts via conditional coordinating. Proceedings of the IEEF/CVF International Conference Computer Vision, pages 4512-4521.
Lin, J., Zhang, R., Ganz, F., Han, S., and Zhu, J.-Y. (2021). Anycost gans for interactive image synthesis and editing, arXiv preprint arXiv:2103.03243.
Liu, Z., Luo, P., Wang, X., and Tang, X. (2015). Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738.
Mittal, A., Soundararajan, R., and Bovik, A. C. (2012). Making a "completely blind" image quality analyzer. IEEE Signal processing letters, 20(3):209-22 -12.
Nachmani, E., Roman, R. S., and Wolf, L. (2021). Non gaussian denoising diffusion models. arXiv preprint wxxv:2106.07582
Nichol, A. and Dhariwal, P. (2021). Improved denoising diffusion probabilistic models. arXiv preprint
Oksendal, B. (2013). Stochastic differential equations: an introduction with applications. Springer Science & Business Media.
Park, J. and Kim, Y. (2021). Styleformer: Transformer based generative adversarial networks with style vector. arXiv preprint arXiv:2106.07023,
Parmar, G., Zhang, R., and Zhu, J.-Y. (2021). On buggy resizing libraries and surprising subtleties in fid calculation. arXiv preprini 1r4iv:2 1/222.
Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N., Ku, A., and Tran, D. (2018). Image transformer. In International Conference on Machine Learning, pages 4055-4064. PMLR.
Salimans, T., Goodfellow, Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016). Improved techniques for training gans. Advances in neural information processing systems, 29:2234-2242.
Schonfeld, E., Schiele, B., and Khoreva, A. (2020). A u-net based discriminator for generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8207-8216.
Sinha, S. and Dieng, A. B. (2021). Consistency regularization for variational auto-encoders. arXiv preprint arXiv:2 14859.
Skorokhodov, L., Ignatyev, S., and Elhoseiny, M. (2020), Adversarial generation of continuous images. arXiv preprint arXiv:2011.1200
Song, Y., Durkan, C., Murray, I., and Ermon, S. (2021). Maximum likelihood training of score-based diffusion models. arXiv e-prints, pages arXiv-2101.
Song. Y. and Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. arXiv preprint arXiv: 190700000000000
11

---

Under review as a conference paper at ICLR 2022
Song. Y. and Ermon, S. (2020). Improved techniques for training score- score-based generative models. arXiv preprint arXiv:2006.09011
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. (2020). Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13445
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1985 1958.
Stein, E. M. and Shakarchi, R. (2009). Real analysis: measure theory, integration, and Hilbert spaces. Princeton University Press.
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2816-2826.
Theis, L.,, Oord, A. v. d., and Bethge, M. (2015). A note on the evaluation of generative models. arXiv preprint arXiv: 1.011444
Vahdat, A. and Kautz, J. (2020). Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03899
Vahdat, A., Kreis, K., and Kautz, J. (2021). Score-based generative modeling in latent space. arXiv preprint arXiv:2106.054)
Welling, M. and Teh, (2011). Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-II), pages 681 681-688. Citeseer.
Yang. C., Shen, Y., Xu, Y., and Zhou, B. (2021). Data- efficient instance generation from instance discrimination. arXiv preprint
Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., and Xiao, (2015). Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 1rX.v:1506.03365.
12

---

Under review as a conference paper at ICLR 2022
A PROOFS OF LEMMA I PROPOSITION 1 AND PROPOSITION 2
Lemma 1. Let Ho. = (s Rd x [0, + TR, s is locally Lipschitz). Suppose a continuous vector field v defined on subset U of a compact manifold M (i.e., v U C CM- + R is unbounded, then there exists no s H( such that l t( v(x) a.e. on U.
Proof. Since U an open subset of compact manifold M, |X] xx I diam(M) for all
X2 X1,X2 E U. Also, if 12 E [1, 14 - [2| is bounded. Hence, the local Lipschitzness of s implies that there exists a positive K > such that ((1x,,4) - 5(x2, S K [[]xx x2ll + [t1 - (2))
for any X1,
U and f1, [0, 2 Therefore, for any s Hjo.Tj. there exists C' > 0 such that
[|s(x, || < C' for all x E U and t E [0, 7], which leads no that satisfies s(x,t) - exx) a.e. on U as O.
Proposition 2. Let H(),o0) (s: Rd x (1) Rdd s is locally Lipschitz). Suppose a continuous vector field v defined on a d-dimensional open subset U of a compact manifold M is unbounded, and the projection of v on each axis is locally integrable. Then, there exists E HiI,so) such that y s(x,1) 11) vxx) a.e. on U.
Proof. Let h be a standard mollifier function. If (x( = =- "x(/t),,, then U/ == he * converges to o a.e. on as f + (Theorem 7-(ii) of Appendix C in Evans (1998). Therefore, if we define s(x, 17) == 11/1(x] (x) on the domain of 11/n and s(x, 1) == 0 elsewhere, then s(x, n7) = v1/n (x) -
(xx a.e. on U as + co.
Now, to show that s x 17) is locally Lipschitz, let M x [n. ग be a compact subset of R" x [1, oo). From [|((x), 77) - s 72)1l = || (x1) U1/02 (x2) II S I| (xx) - vi/n (x2) + ||/1/m (x2)
V1/n. (x2))l if there exists Ki K2 > such that |/// (x1) - U1/m (x2) S K1| X X2 and
||11/m (x1) - /12m2 (x1) I K2
72l for all X2 M and 72 E [n. 7). then s(x, n) =
11/x(x) is Lipschitz on M x [n, 7).
First, since ///1 is infinitely differentiable its domain (Theorem 7-(i) of Appendix C in Evans (1988) and 17 [y7 7), there exists K₁ > 0 such that [|01/,(x1] / (x2) < Kg [x] x |
Second, the mollifier satisfies the uniform convergence on any compact subset of U (Theorem
(iii) of Appendix C in Evans 1998)), which leads that ||0/// (x) 11/m (x)]l < K2 m
ma
K2 Im nal
K3 m
v2l for some K2, K3 > Therefore, s becomes an element of Hp,
B IMPLEMENTATION DETAILS
Since the major contribution of this paper is developing the diffusion model in perspectives of theory and optimization method, we use the same model architecture introduced in Song et al.) (2020).
B.1 EXPERIMENTAL DETAILS
Training Throughout the experiments, we train our model with a learning rate of 0.0002, warmup of 5000 iterations, gradient clipping by 1. On the experiments for HNCSN+ with miin = 10 we train the diffusion model by truncating miin to 10 at the initial stage of training, and after the saturation in terms of the FID score, we begin to train the score network with the whole range of [amén, maaz] We find this way of training works best in practice for the training stability. We follow the convention of Song et al. (202)) to choose Tmaz. Also, for DDPM-style model training, we use 3mi = 0.1 and 9maz = 20. For all experiments, we set = 1 and = 1.
We stabilize the training by the Exponential Moving Average (EMA) (ong et al. 2020) with the rate of 0.999 on NCSN++/HNCSN++ and 0.9999 on DDPM++/HDDPM++ training by applying the dropout (Srivastava al., 2014) with 0.1 rate. For the optimizer, we use the Adam optimizer (Kingma and Ba 2014)) with (B1, B2) = (0.9, 0.999).
addition, we stabilize the
On datasets of resolution 32 32, memory. On STL-10 with resolution 48 x 48, we use the batch size of 192, and on datasets of resolution 64 x 64, we experiment with 128 batch size. The batch size for the datasets of resolution
the batch size of 128, which consumes about 48Gb GPU
13

---

Under review as a conference paper at ICLR 2022
256 256 is 40, which takes nearly 120Gb of GPU memory. On the dataset of 1024 1024 resolution, we use the batch size 16, which takes around 120Gb of GPU memory. We use five NVIDIA RTX- 3090 GPU machines train the model exceeding 48Gb, and GPU machines to train the model that consumes less than 48Gb.
a pair of NVIDIA RTX-3090
Evaluation We apply the EMA with rate of 0.999 DDPM++/DDDMH+ For the density estimation, we obtain the NLL performance by the In- stantaneous Change of Variable (Song al. 2020 Chen al. 2018). For the sampling, apply the Predictor-Correcto (PC) algorithm introduced in Song et al. (020). We set the signal-to-noise ratio as 0.16 on 32 x 32 datasets, 0.17 on 48 x 48 and 64 x 64 datasets, 0.075 on 256x256 sized datasets, and 0.15 on 1024 x 1024. On datasets less than 256 x 256 resolution, we iterate 1,000 steps for the PC sampler, while we apply 2,000 steps on the other high-dimensional datasets. Throughout the experiments, we use the reverse diffusion (Song et al. (2020) for the predictor algorithm and the annealed Langevin dynamies (Welling and Teh 2011) for the corrector algorithm.
NCSN+
and 0.9999
We compute the FID score (Song et al. 2020) based on the modified Inception VI network using the tensorflow-gan package for CIFAR-10 dataset, and we use the clean-FID (Parmar et al. 2021) based on the Inception V3 network (Szegedy et al. (016)) for the remaining datasets. We note that FID computed by Parmar et al. 2021 reports a higher FID score compared to the original FID calculation
B.2 CHOICE OF n1(
HNCSN As described in Section 3.1 HNCSN assumes the score parametrization of se (t() with rtt) being the mixture of log (t) and a(z) Despite of the clear advantage log (t) that the distribution of p(o) is uniform in VE/RVESDES, the drastic scale variation at z 0 prevents the
enhanced score estimation. We empirically find that the reciprocal function of a(t) works best on the
estimation at the front diffusion time, and we combine these two functions to create continuously differentiable 7)(t) by
log (t) (8) +
if o 2 ( 0o:
(t)
C2
where C1, O2 and dg are the hyperparameters. By acknowledging the parametrization of log at)) intro- duced in Song et al. (2020). we choose do as 0.01. Also, to satisfy the continuously differentiability of (t()), the choice of hyperparameters ci and 62 satisfies system of equations with degree so Cj and C2 are fully determined with this system of equations.
HDDPM Looking the antiderivative o (t) dt (-(
((e) ds) + 3(s log o (t) + + ((s) ds, the unbounded function reduces to r7()) 2 log (t)
log (1
C (s) ds. Therefore, HDDPM resembles the temporal parametrization of log (t) (Song et al. 2020) at R 0 because the second term, f ((s) 4) ds, is nearly zero at f. Rv O.
C ADDITIONAL EXPERIMENTAL RESULTS
Figure 15 shows how images are created from the trained model on 256-dimensional datasets, and Figures from 16 to 2T present randomly generated samples of the trained model. Also, Tables 9 and [0] present the additional experimental results in terms FID on LSUN Bedroom and FFHQ 256 256.
*See
Tecttt/s///tthhb..coo///aaa////r for the detailed experimental results.
14

---

Under review as a conference paper at ICLR 2022
Figure 15: Image generation by denoising via predictor-corrector sampler.
Table 9: LSUN Bedroom
Table 10: FFHQ
Bedroom 256 256
FFHQ
Model
Model
256
256
FID
UDM (RVE) + ST
4.57
UDM (RVE) + ST
5.54
Likelihood- Models
Likelihood free Models
ADM Dharwwll and Nichol
2021
1.90
InsGen Yang al. 2021
3.31
DDPM el al. 2020)
4.90
Anycost GANN L.m
2021
3.35
L.kellhood-free Models
StyleGAN2 ADAHEU Kaaras al. 2020
3.62
CIPS Anokhin el al. (2020)
4.38
StyleGAN Karras al. 2019
2.65
INR-GAN--DISSoooooaa Skorokhoddy el al. 2020
4.95
U-Net GAN Schoofeld GANNSoonnllll at
BigGAN/Schonteld Schonteld 2020
2020
11.48
COCO-GAN Lin et al. 2019
6.95
15

---

Under review as a conference paper at ICLR 2022
Figure 16: Random samples on CIFAR1O.
16

---

Under review as a conference paper at ICLR 2022
Figure 17: Random samples on LSUN Bedroom.
17

---

Under review as a conference paper at ICLR 2022
Figure 18: Random samples on LSUN Church.
18

---

Under review as a conference paper at ICLR 2022
Figure 19: Random samples on FFHQ 256.
19

---

Under review as a conference paper at ICLR 2022
UJ
Figure 20: Random samples on CelebA AH 256.
20

---

Under review as a conference paper at ICLR 2022
Figure 21: Random samples on FFHQ 1024
21


Reviews:
[{"review_id": "rEwC9z2Urhu", "reviewer": "Reviewer_ekSh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. For the figures (e.g., figure1,2,5), it would be better to explain in detail how the plots are generated and their settings. For example, in figure 5, it would be important to mention what data distribution it is (is it the distribution in figure 6), are the $y$ axis averaged over one single batch? How many data samples are considered? The result in figure 5 depends a lot on the underlying distribution: if the data distribution is $N(0,I)$,  the $y$ axis (data score) scale might not be that ill-conditioned even when $\\sigma$ is small? In many cases, losses with values close to infinity happen at datapoints in low-density regions where scores are not well defined. Thus, a sufficient amount of noise is needed to ensure that the estimated scores are well defined. Thus, in some sense $\\sigma_{\\text{min}}$ might need to be (strictly) non-zero when the data scores are not well defined.\n2. In figure 5, the x, y labels could be improved. Currently the x, y labels do not seem to depend on each other.\n3. In figure 6, it seems that the proposed method is still not able to capture the data score well on the toy dataset (although it is slightly better than NCSN++).\n4. If we consider $L_t=\\mathbb{E}[||s_\\theta(x_t,t)-\\frac{x-x_t}{\\sigma^2}||_2^2]$, and expanding the terms. When $\\sigma\\to 0$, the entries in the term $(\\frac{x-x_t}{\\sigma^2})^2$ go to infinity, which could cause $L_t$ to blow up when $\\sigma$ is small. When $\\sigma\\to0$, the training variance can also be large (https://arxiv.org/abs/2002.07501), which causes instability issues. In some sense, tuning the weighting function $g(t)$ might not be able to completely solve the issue that the estimated scores with denoising score matching (DSM) are inaccurate when $\\sigma\\to0$. To see this, one could use a model conditioned on a single noise scale $\\sigma$ (e.g., 1e-3), and plot the accuracy of the estimated score trained with DSM. It seems that the proposed weighting could improve the joint training of conditional models with shared parameters, but might still not be able to solve the large estimation error issue when $\\sigma\\to0$ (see figure 5). Related to this direction, this paper (https://arxiv.org/abs/2002.07501) proposes a variance reduction technique for DSM with Taylor expansion. This approach often improves the stability and accuracy of training DSM when $\\sigma$ is close to zero.\n5. The empirical results show that the proposed method allows training the model with a smaller $\\sigma_{\\text{min}}$ (which could correlate with sample quality) than baseline SDE models. The proposed method also has better sample qualities than current SDE models. \n\n", "summary_of_the_paper": "Diffusion models have achieved great performance in many applications. However, learning a good diffusion model can be challenging since the data score can go to infinity when $t$ goes to zero. To address this issue, this paper proposes an alternative parameterization for unbounded data scores and provides a practical trick (ST-trick) to handle the variation of the scales of score values. It proposes Reciprocal Variance Exploding Stochastic Differential Equation to sample from the model. The proposed method can be applied to existing NCSN and DDPM models. Empirically, it achieves strong performance on high-resolution image generation.", "summary_of_the_review": "This is a very interesting work that studies the effect of how the sampling steps at different $t$ affect the performance of score-based diffusion models. It also proposes a method to train SDE-based models with smaller $\\sigma_\\text{min}$ so that samples corresponding to $\\sigma_\\text{min}$ would ideally have better local fidelity. Empirical results show that the proposed method can improve the sample quality when applied to existing SDE-based models. However, the experimental setup is not very clear and I also have some concerns about the problem formulation (see Main Review). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "d-u1GOCPBYc", "reviewer": "Reviewer_e3pu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- Overall, the results presented in Table 3 seem quite good. On this note, the improvement on CelebA (64x64) from FID 3.95 of NCSN++ [1] to FID 1.92 (!!) seems really good. I don't assume that the number is wrong, however, I find it suspicious that no samples on this dataset are presented and that the number improved from 2.78 (I am aware of the preprint version of this paper) to 1.92, whereas the NLL number remained the same. I encourage the authors to share checkpoint + minimal FID testing script with the reviewers so that this number can be confirmed.\n\n- (iii) In my opinion, the ST trick is well-motivated and the high-level idea of focusing some batches on smaller time steps and some batches on larger time steps seems intuitive. \n\n- (iv) The authors propose an SDE which leads to simple importance sampling. \n\n- I appreciate that the authors tested their models on several datasets.\n\nWeaknesses:\n\n- The paper uses the probability flow formulation from [1] to compute NLL and the PC sampler from [1] to compute FID and IS. Using different samplers, these are two distinct (!!) models and values of NLL and FID/IS should not be put in the same row as is done in Table 3. This is misleading. I encourage the authors to dedicate one row to FID/IS with PC sample and one row to NLL/FID/IS with ODE sampler. Same comment applies to values reported from other papers in Table 3 and to other Tables in the paper.\n\n- When I was reading the paper (in particular Section 2), I was under the impression that the authors would train their models using likelihood weighting [3] (see, for example, $\\lambda = g^2$ in Eq. 1). However, the way Section 4 is written (\"On CIFAR-10, we observe that our model improves NCSN++ (VE) in NLL and IS at the expense of sacrificing FID slightly) it seems that the authors used the re-weighted $\\lambda$ from [1] for some models. To the best of my knowledge, there is no comment in either main paper or appendix which weighting function $\\lambda$ is actually used in the results of tables 3/4/5/7. On the other hand, two different weighting functions are used for Table 6. I encourage the authors to clarify.\n\n- (i) In my opinion, proposing to train models for lower $\\varepsilon$ is not so much a contribution but rather just a hyperparameter search. In the paper's defense, NLL & FID do improve for NCSN++ (table 4) when setting $\\sigma_{\\mathrm{min}}$ from $0.01$ to $0.001$. Decreasing  $\\sigma_{\\mathrm{min}}$ even further results in insignificant improvements of NLL and worsens FID significantly (Table 6). I think it would have been better to just say that $\\sigma_{\\mathrm{min}}$ is a hyperparameter and defer the discussion around Figure 4 to the appendix.\n\n- (ii) I disagree with the paper on the significance of this \"problem\". Generally, score-based generative models are parameterized in such a way that $s_\\theta(x, t) = \\alpha_\\theta(x, t) / r(t)$ (see for example [1]) where $\\alpha_\\theta$ is a neural network and $r(t) \\to 0$ as $t \\to 0$. See for example the implementation (https://github.com/yangsong/score_sde_pytorch/blob/1618ddea340f3e4a2ed7852a0694a809775cf8d0/models/utils.py#L129) of [1] which the reviewed paper is based on. So to the best of my knowledge, this parameterization takes care of the \"problem\". Furthermore, the \"performance gain\" using the parameterization with $\\eta(t)$ in Table 4 (compare row 2 and 3) seems insignificant. Is Figure 13 (a) based on the toy example from Figure 6? If so, I am not sure that the result directly translates to high-dimensional image generation.\n\n- Figure 10: It seems that the FID scores are computed using only 1k generated samples? From my personal experience in computing FID scores, 1000 samples are not enough to make any conclusions (variance is too high). I encourage the authors to repeat the experiment with at least 10k samples.\n\n- (iv) Single ablation (row 3 vs row 4 in Table 4) does not seem enough evidence for justifying RVESDE as an important contribution. I think the authors should have done more experiments, even if only on toy datasets. Furthermore, in my opinion, the main benefit of RVESDE is that importance sampling is simple, however, there is no result shown for RVESDE + importance sampling?!\n\n- The paper is somewhat hard to read and unclear in some claims; see quote from summary.\n\nSuggestions:\n\n- I think the \"soft truncation\" trick is a cool idea and might solve an important problem. In my opinion, the paper could have been a lot better if this one promising direction would have been studied more in detail rather than studying four problems/solutions.\n\nReferences:\n\n[3] Song et al. Maximum Likelihood Training of Score-Based Diffusion Models. arXiv:2101.09258v4.", "summary_of_the_paper": "The paper outlines four problems and proposes a solution for each problem:\n\n(i) Generally, score-based generative models (for example [1]) do not compute the loss on the entire time interval $[0,1]$ but rather only on $[\\varepsilon, 1]$, for some $\\varepsilon > 0$. The paper claims that choosing $\\varepsilon$ too large results in poor details for image generation (see Figure 4), for example, the details (pores) of a human nose.\n\n(ii) The paper shows that the the squared Euclidean norm of the data score (for an unspecified data distribution) empirically diverges as $t \\to 0$.  The paper then presents a lemma that states that no neural network can learn an unbounded function as long as it is conditioned on time $t$ or variance $\\sigma$. The paper then proposes to condition on $\\eta(t)$ where $\\eta$ is constructed such that either $\\eta(t) \\to \\infty$ or $\\eta(t) \\to -\\infty$ as $t \\to 0$.\n\n(iii) The paper shows that Monte Carlo samples of the likelihood loss have a large variety of scales: values are very large for small times $t$ (or equivalently small noise levels) and very small for large times $t$ (or equivalently large noise levels). Due to the large variety of scales, the authors claim that \"the diffusion loss is dominated by the end diffusion time with the front range in practice, and the loss at the end diffusion time is barely counted in the gradient signal. This imbalanced loss brings the immatured score estimation at the end diffusion time, which ruins the overall sample shape.\" (I am not entirely clear what this quote means.) The paper then proposes a so-called \"soft-truncation\" (ST) trick. The high-level intuition of the ST trick is described as follows: instead of sampling time points $t$ uniformly over the time range $[\\varepsilon, 1]$ in each batch; in each batch it samples a $\\tau \\sim \\mathbb{P}(\\tau)$ and then samples $t$ uniformly over the time range $[\\tau, 1]$. The authors claim that this has the effect that some batches focus on large time steps (providing a good gradient signal for large noise levels) and some batches focus on small time steps (providing a good gradient signal for small noise levels).  The original setup with fixed $\\varepsilon$ can be recovered by setting $\\mathbb{P}(\\tau)= \\delta(\\tau = \\varepsilon)$. The authors then propose a particular one-parameter ($k$) distribution $\\mathbb{P}$ and show different instantiations of $k$. The family is designed in such a way that $\\mathbb{P}(\\tau) \\to \\delta(\\tau = \\varepsilon)$ as $k \\to \\infty$.\n\n(iv)  The authors show that the continuous VESDE [1],  proposed  as a continuous extension of  [2],  is not actually “geometric” as apparently desired by [2]. The authors then propose the \"Reciprocal Variance Exploding\" SDE (RVESDE) which can be shown to be \"geometric\". The authors then claim that importance sampling is simple for the RVESDE.\n\nThe paper tests the proposed solutions on CIFAR10 (32x32), CelebA (64x64), CelebA-HQ (256-256), and STL-10 (48x48).\n\nReferences:\n\n[1] Song et al. Denoising Diffusion Implicit Models. ICLR 2021.\n\n[2] Song & Ermon. Improved Techniques for Training Score-Based Generative Models. NeurIPS 2020.", "summary_of_the_review": "The paper outlines four problems and one fix for each problem. In my opinion, three fixes are either only incremental or not sufficiently tested. The \"soft truncation\" trick, on the other hand, seems like a good idea, however, this contribution is buried under the three other problemx/fixes and not enough space/thought is devoted to it. In my opinion, the weaknesses of the paper clearly outperform its strengths.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WyaqTLeJMVt", "reviewer": "Reviewer_Veok", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First of all, I would like to applaud the authors for pinpointing the practical issues of training score-based diffusion models and for making an effort in fixing these problems. Below are the strengths of the paper\n- The three issues identified in this paper are, to the best of my knowledge, quite relevant in training diffusion models. \n- The experiments are well thought-out: the ablation study is quite thorough. \n\nBelow are some weaknesses of the paper:\n1. Presentation needs more work: the paper contains a lot of imprecise languages and somewhat misleading statements and reasonings. See points (b,c,e,k,l)\n2. Questionable technical assumption: The paper hinges heavily on the assumption that the score would blow up (to infinity) as $t\\rightarrow0$ (such as Lemma 1, see point (g)). It is not very obvious why this is a reasonable assumption, especially when uniform or variational dequantization is usually performed in practice, even when we consider empirical data distributions that are a mixture of dirac point masses. Perhaps the authors can provide some more explanation in the rebuttal. \n3. Uncertainty of the experiments not presented. Are all experiments with a single seed? \n4. A lot of grammatical mistakes and typos. \n\nMore detailed comments and questions:\na) Intro: I don’t quite understand the logic behind the references in the very first paragraph in the intro. I understand some of them are perhaps state-of-the-arts or perhaps quite new, but I would suggest including a more canonical / standard reference for each category. \n\nb) In section 2, the authors claimed that VESDE and VPSDE are equivalent up to rescaling (and time change). Thus they build their model based on VESDE. But the two SDEs induce very different diffusion, and due to the way neural nets are initialized, this could result in a pretty drastic difference in practice. I’d suggest making this point clearer since in practice training with VESDE does lead to a pretty different performance than VPSDE (as observed in Song et al 2020). \n\nc) Eq (1): the RHS is missing some terms (see Song et al 2021 thm3 or Huang et al 2021 eq (25)). I notice there’s a short note after the score matching loss equality “up to a constant”, but it’s not clear if this applies to eq (1), which can be confusing. Also, I’d suggest perhaps adding the refs there. \n\nd) after eq (2): what is $\\sigma(\\epsilon)$?\n\ne) Page 3 first paragraph: It says “if the score network exactly estimates the data score, the generated image at time $t$ would follow $x_t$”. This statement feels a bit weird and seems incorrect: score-based diffusion models are not about exactly inverting the perturbing diffusion process. It’s more of a distributional property that the dynamic can be reverted, not a pointwise property.  \n\nf) Is Fig 4 a generated sample? Or is it simply a real data point but perturbed with different levels of noise?\n\ng) Lemma 1: the subset U should be open? Also, why is the assumption on an unbounded vector field a reasonable assumption here? \n\nh) Figure 5: How is this curve estimated or computed? Is it for a specific dataset? Is it the score of an empirical distribution? \n\ni) Training instability: from figure 7, I can tell that the loss might have high magnitude and high variance for small sigma. It’s not directly obvious that this will cause training instability. Perhaps a training curve (of avg NLL) will better reflect it? Also, what does different color / different $\\tau$ mean? \n\nj) Proposition 1: The VESDE proposed in Song et al. has this problem since a Gaussian noise with variance $\\sigma_\\min^2$ is always included. That is, they are learning to recover the distribution of $p_\\text{date}$ convolved with $N(0, \\sigma_\\min^2). Not sure it is fair to say VESDE loses its theoretic ground. Also, the same issue does not seem to apply to VPSDE (even though the authors claimed the two are basically equivalent). \n\nk) Section 3.1: Not sure why there is a negative sign in $\\eta(t)$ for HNCSN. The optimal score function $s^*$ is a function of $x_t, t$, why is that equal to a random variable $z$? Importance sampling is introduced in the following paragraph without any introduction / reference. $\\eta(t)$ for HDDPM is defined as a definite integral? The differential uses the same variable as the argument of $\\eta$, which is confusing. What is the $\\eta$-distribution in the last paragraph? Please make this paragraph more precise, I don’t understand why this uniformized representation guarantees high precision, what is high precision exactly?  Also, won’t $\\eta$ go to infinity as $t$ goes to 0? How come one ends up having a uniform distribution in this unbounded space. \n\nl) Section 3.2: What is the third argument of $\\mathcal{L}(\\cdot; \\cdot, \\cdot)$ in the ST-trick? I got the feeling that this is performing random truncation of the perturbing SDE, i.e. the lower integration time ($\\epsilon$?) is randomized? Overall the presentation of ST needs more work, I could barely understand what is actually being done here.\n\nm) If the lower integration time is randomized, this still will not result in an unbiased training loss for the lower bound on likelihood right? Then what is the advantage of ST compared to importance sampling proposed in Song et al 2021 and Huang et al 2021?\n\nn) Table 3 is a bit inconclusive, and the improvement of HNCSN++ (RVE) + ST seems a bit marginal. I find it hard to judge the result, what’s the take-home message?\n\no) In the High Precision Parameterization paragraph of ablation, it says “..with lowered $\\sigma_\\min effectively reduces the performances”, should it be “improves” instead? The performance gain from the new parameterization is in terms of what? FID?\n\np) It is unclear for the argument against VESDE using non-trivial $\\sigma_\\min$, the authors actually tried to sample with $\\sigma\\rightarrow 0$ after training. This is regarding the paragraph *Optimal Precision* in the ablation. Lowering $\\sigma_\\min$ will cause numerical instability during training I suppose, but at test time technically one can still extrapolate to smaller $\\sigma$. Also, it’s unclear why the 8-bit representation has anything to do with said instability. Isn’t the data uniformly dequantized?  \n", "summary_of_the_paper": "The paper identifies three problems with score-based diffusion models: 1) unbounded score, 2) training stability, and 3) bias in training using VESDE. The authors then propose the corresponding solution to each problem, with some theoretical motivation, which is then empirically tested. \n", "summary_of_the_review": "The paper presents three practical problems and solutions to training / parameterizing score-based diffusion models. Overall the direction is quite relevant, but the presentation of the work is subpar (I’ve pointed at specific sections/paragraphs for future improvement in detailed comments), which makes me doubt it would benefit the ICLR audience *in its current form*. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "KtgwEXAInqK", "reviewer": "Reviewer_QU8e", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think paper makes a tangible contribution to the diffusion model literature.  The paper is, however, not very well written. It is quite technical, which makes it challenging to read. But the English is also difficult to understand in places.  I would encourage the authors to address the writing to make it more accessible.  I would further recommend adding more citations throughout, especially in Section 2.\n\nThe fact that the score function is unbounded and causes problems for optimization is not particularly surprising.  What is less clear, perhaps, is whether different parameterizations for learning diffusion models suffer from the same root problem.  For example, one might suppose that the epsilon parameterization introduced in the DDPM paper by Ho et al. would not suffer from the same problem. I think this paper would be much stronger if they can claim that other existing parameterizations like that used by Ho et al all suffer from the problem addressed in this paper. This issue should be discussed at length.\n\nMotivation:\nThe impact of the problems identified close to the beginning and the end of the diffusion process (regarding Figures 3 and 4), are not particularly compelling.  The fact that the hair is synthesized in the wrong region of the image in the early stages of the reverse diffusion process seems to be a relatively weak argument for the problem the authors have identified.  Similarly, regarding Figure 4, it is not entirely clear that the problems that occur as a function of sigma_min being too large are all local, nor is it clear that reducing sigma_min from 10^-2 to 10^-3 has a major impact on image quality in this case.  I am not trying to disagree with the claim that the score function becomes unbounded, and this leads to challenges in the optimization.  Rather, I think the issue is with the impact of the problems raised on the generative model.\n\nThe motivation for the ST trick, which appears to be useful, is also not well explained.  More intuitions for it would be very helpful.  That said, this ST-trick appears to be very useful.\n\nExperiments:\nThe results provided in Table 3 are limited to relatively small images, and in the case of CelebA and CelebA-HQ to datasets with relatively little diversity.  Results are ImageNet 256x256, or even ImageNet 64x64 would also be nice to see, and would also allow comparison with recent SOTA methods like ADM and CDM.\n\nConcerning the quantitative measures of performance, the Appendix states that FID is computed using the InceptionV1 network.  Are the FID numbers with which they compare also computed using InceptiveV1?\n\nThe issue of different parameterizations of the loss in diffusion models is also relevant to the experimental work. One important properties of the experiments is the different parameterizations used by different authors in optimizing the diffusion models.  Again, I am thinking of the epsilon parameterization which may not suffer from the same problems as the score function per se. Which methods in Table 3, for example, use which parameterizations? Directly comparing different parameterizations with all other things held constant might be useful for the authors to do directly.\n\nFinally, the paper finds that improved results in training diffusion models occur as sigma_min reduces from 10^-2 to 10^-3, but for values lower than that performance degrades.  This may also be related to conventional quantization levels in images for which there are 256 resolvable values in each color channel at each pixel.  This suggests that adding noise any smaller than 10^-3 would be sufficiently small relative to quantization that there is little signal for denoising at noise levels lower than 10^-3.  To what extent does the lowest effective sigma_min depends on pixel quantization.  For example, with 12 bit medical images, should sigma_min go even lower to further improve performance of the model?", "summary_of_the_paper": "This paper considers two problems associated with optimizing diffusion models, namely, training instability due to unbounded denoising score functions when the level of noise is very small (at the later stages of the reverse diffusion process), and problems at the earlier stages of the reverse diffusion process where coarse-scale artifacts are sometimes introduced as the corresponding terms of the objective have relatively little influence on the training loss. These problems are addressed through a new parameterization that is better suited to extremely small noise levels, and a soft truncation (ST) trick that better enables the optimization with sufficient influence given to the entire range of noise levels.  The paper has strong technical contributions and very good experimental work, showing improved quantitative measures of performance, namely NLL, FID, and IS.", "summary_of_the_review": "I think this is an interesting paper.  It is technical and I did not go through all the mathematical details carefully.  However, I think the results are potentially useful and lasting. The paper itself can be cleaned up. The ST-trick appears to be quite useful, but I am less convinced about the significance of the problem with the loss parameterization as discussed above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "84wf22tPf4r", "reviewer": "Reviewer_sL3N", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**novelty & significance**\n\nThe paper studies a popular model -- the SDE-base score-matching model -- and identifies an important problem. To the best of my knowledge, this exact problem -- issues related to t close to 0 -- has not been studied before in the exact form as is presented in the paper. In this regard, this paper is somewhat novel and its topic suits the venue. \n\n**technical correctness**\n\nI'm quite concerned about this aspect, mostly in terms of the theory and how the theory motivates the narrative of the solutions. \n\n- First, the issue of exploding data scores is only justified through a toy experiment (Figure 5), whose description is quite lacking. For instance, the y-axis in this figure shows that the quantity is $\\| \\nabla x \\log p_t(x_t) \\|$. This quantity, however, is a random variable. I am thus confused about what is exactly being shown in the figure -- is it the average data score, is it the max data score over a constrained subset of x-space, or something else? In general, how do you even obtain this data score exactly? Is this a toy example where you know the ground truth data generation model and its density? That's even more surprising, since most densities we use on a day-to-day basis has finite data scores, and thus would suggest a wild discontinuity of the score at time t=0. This seems unlikely to happen if you use smooth forward SDE models. \n- Lemma 1 doesn't seem quite right to me. A locally Lipschitz function could be unbounded on a non-compact subset. For instance, consider the function s(x, t) = 1/x defined on (0, 1) x [0, 1]. At t=0, this function could still be unbounded (take x->0). At the same time, it's easy to show that the function is locally Lipschitz, since the derivative is always finite at any input location you pick within the domain (though not bounded anymore). \n- The text on the top of page 3 (section about Front Time) seems weird to me as well. The authors have motivated the discussion with the pair of random variables x_0, and x_t where x_t is sampled conditioned on x_0. With this, the authors argue that the L2 distance between a generated sample at small t (denoted x_t' here to distinguish between x_t) and data x_0 can't be close. In this latter conclusion x_0 and x_t' are independent, whereas in the motivating justification, x_0 and x_t follows a Markov chain. The explanation therefore seems like a non-explanation to me, since the assumptions aren't right. \n\n**presentation**\n\nWhile I can sense that authors are trying to do a good job in this respect, much can still be improved. I find the sections 2.2 and 3 particularly confusing. There are two causes to this confusion:\n  - The context of the toy experiment isn't explained carefully. Also, are plots 5, 7, 8 all generated based on this toy experiment? How different quantities in these plots are computed are also left undescribed. This makes evaluating correctness difficult. (recall the issue about data score in the first bullet for the \"correctness\" section of this review). \n  - More motivation and context could be given to certain parts of the main text. \n\n**experiments**\n\nThis part seems somewhat insufficient. For instance, authors didn't verify whether the exploding data score issue occurs on natural image data like CIFAR, assuming that Figure 5 is on the toy experiment. Again, I don't quite understand what the y-axis is, since I don't see random fluctuation along the vertical direction of the plot as I expect, since the quantity is a random variable. Ablation studies about the soft-truncation trick and selecting sigma_min seems interesting. ", "summary_of_the_paper": "This paper studies the behaviour of training SDE-based score-matching generative models for time t close to 0. The paper identifies two issues with existing forward SDEs for training such models: 1) the score of the marginal distribution could be large, and 2) the expected squared loss could be large, draining the loss contribution at other times. The paper introduces several remedies to fix this issue.  \n\n", "summary_of_the_review": "The paper studies an interesting problem and attempts to present corresponding solutions. The presentation could be improved. Authors should also try to address the issue regarding the correctness of Lemma 1. I am willing to change my mind if the authors could present a convincing argument of why it's true (how my counter argument is wrong), or a fix to the theory. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes the behavior of score function as $t \rightarrow$ 0 and proposes simple approaches to mitigate issues around estimating an unbounded data score. The reviewers have acknowledged the importance of the problem and its relevance to current efforts on denoising diffusion models. However, they have also raised serious concerns regarding the clarity of the presentation and missing information across different sections. Additionally, the unbounded data score is only shown on a toy example, and it is not clear if it exists in real-world datasets. Finally, e3pu pointed out that in the commonly used score parameterization from Song et al. the score parameterization can in fact grow to infinity when needed. Given these criticisms and without a response from the authors, we don't believe that the paper is ready for publication at ICLR.
Decision: Reject