Soft Diffusion: Score Matching For General Corruptions
Authors: Giannis Daras
Mauricio Delbracio
Hossein Talebi
Alex Dimakis
Peyman Milanfar
Venue: Submitted to ICLR 2023 (2023)
Keywords: diffusion
score-based models
generative models

Abstract: We define a broader family of corruption processes that generalizes previously known diffusion models. To reverse these general diffusions, we propose a new objective called Soft Score Matching that provably learns the score function for any linear corruption process and yields state of the art results for CelebA. Soft Score Matching incorporates the degradation process in the network. 
Our new loss trains the model to predict a clean image, that after corruption, matches the diffused observation.
We show that our objective learns the gradient of the likelihood under suitable regularity conditions for a family of corruption processes. 
We further develop a principled way to select the corruption levels for general diffusion processes and a novel sampling method that we call Momentum Sampler. 
We show experimentally that our framework works for general linear corruption processes, such as Gaussian blur and masking.
We achieve state-of-the-art FID score $1.85$ on CelebA-64, outperforming all previous linear diffusion models. 
We also show significant computational benefits compared to vanilla denoising diffusion.

Under review as a conference paper at ICLR 2023
SOFT DIFFUSION SCORE MATCHING FOR GENERAL CORRUPTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
We define broader family of corruption processes that generalizes previously known diffusion models. To reverse these general diffusions, we propose a new objective called Soft Score Matching that provably learns the score function for any linear corruption process and yields state of the art results for CelebA. Soft Score Matching incorporates the degradation process in the network. Our new loss trains the model to predict a clean image, that after corruption, matches the diffused observation. We show that our objective learns the gradient of the likeli- hood under suitable regularity conditions for family of corruption processes. We further develop principled way to select the corruption levels for general diffu- sion processes and novel sampling method that we call Momentum Sampler. We show experimentally that our framework works for general linear corruption pro- cesses, such Gaussian blur and masking. We achieve state-of-the art FID score 1.85 on CelebA-64, outperforming all previous linear diffusion models. We also show significant computational benefits compared vanilla denoising diffusion.
INTRODUCTION
2020 Song et al. 2021b) and Denoising Diffusion Probabilistic Models (DDPMs) (Sohl- Dickstein et al. 2015 Ho al. 2020 Song et al. 2021a) are two powerful classes of generative models that produce samples by inverting diffusion process. These two classes have been unified under single framework (Song et al. 2021b) and are widely
Score-based models (Song & Ermon 2019
known as diffusion models. Diffusion modeling has found great success in wide range of appli-
cations (Croitoru et al. 2022 Yang et al. (2022,, including image (Saharia et al.] 2002a Ramesh
ct al.
2022; Rombach et al. 2022 Dhariwal & Nichol, 2021). audio (Kong et al. 2021] Richter
ct al.
2022,
Serrà et al. 2022). video generation (Ho et al. | [0022b). as well as solving inverse prob-
lems
Daras al.
2022 Kadkhodaie & Simoncelli] 2021 Kawar et al., 2022 2021 Jalal et al.,
2021
Saharia et al.
2022b Laumont et al., 2022) Whang et al. 2022 Chung et al. 2022).
Karras et al. (2022) analyze the design space of diffusion models. The authors identify three stages:
i) the noise scheduling, ii) the network parametrization (each one leads to a different loss function), iii) the sampling algorithm. We argue that there is one important step: choosing how to corrupt. Typically, the diffusion is additive noise of different magnitudes (and sometimes input rescalings). There have been a few recent attempts to use different corruptions (Deasy al. 202 Hoogeboom
ct al. 2022a b Avrahami et al. 2022, Nachmani et al.) 2021 Johnson et al.) 2021; et al. [022]
Ye et al. 2022). but the results are usually inferior to diffusion with additive noise. Also, common framework on how to properly design general corruption processes is missing.
We present such principled framework for learning to invert general class of corruption processes. We propose a new objective called Soft Score Matching that provably learns the score for any regular linear corruption process. Soft Score Matching incorporates the filtering process in the network and trains the model to predict clean image that after corruption matches the diffused observation.
Our theoretical results show that Soft Score Matching learns the score (i.e. likelihood gradients) for corruption processes that satisfy regularity condition that we identify: the diffusion must transform any image into any other image with nonzero likelihood. Using our method and Gaussian Blur paired with little noise as the diffusion mechanism, for linear diffusion models. We also show that our corruption process leads to generative models that are faster compared vanilla Gaussian denoising diffusion.
achieve state-of- oottee----h-arrr the FID CelebA (FID 1.85)

---

Under review as a conference paper at ICLR 2023
Blur + Noise Forward Process
Masking Noise
Reverse Process
Reverse Process
Uncurated Samples (oose)
Top two rows: Demonstration of our generalized diffusion method. Instead of corrupting by only adding noise, propose framework to provably learn the function reverse any linear diffusion (left: blur and noise, right: masking and noise). Our (blur and noise) models achieve state-of-the-art FID score 1.85 for linear diffusion models on CelebA- 64. Uncurated samples shown in the last three rows.
Figure 1:
Our contributions: a) We propose a learning objective that: i) provably learns the score for a wide family of regular diffusion processes and ii) enables learning under limited randomness in the diffusion. b) We present a principled way to select the intermediate distributions. Our method minimizes the Wasserstein distance along the path from the initial the final distribution. c) We propose novel sampling method that we call Momentum Sampler: our sampler uses convex combination of corruptions at different diffusion levels and inspired by momentum methods in optimization, d) We train models on CelebA and C10.R-10. Our trained models on CelebA achieve a new sfate-of-thh-aat FID score of 1.85 for linear diffusion models while being significantly faster compared to models trained with vanilla Gaussian denoising diffusion.
2 BACKGROUND
Diffusion models are generative models that produce samples by inverting corruption process. The corruption level is typically indexed by time t, with 0 corresponding to clean and 1 to fully corrupted images. The diffusion process can be discrete or continuous. The two general classes
of diffusion models are Score-Base Based Models (Song & Ermon 2019 2020 Song et al.) 2021b) and Denoising Diffusion Probabilistic Models (DDPMs) (Sohl-Dicksteil et al. 2015 Ho et al. 2020)).
The typical diffusion in score-based modeling is additive noise of increasing magnitude. The pertur- bation kernel at time l is: q: (x,, (n)) = =N(xx = Σ = of f). where atp ~ is clean image. Score models are trained with the Denoising Score Matching (DSM) objective:
min (1))) W E( mr: (m) (mm [ V æ. log qu (x (=.[=)))1?
(1)
where We scales differently the weights of the inner objectives. If we train for each noise level I independently, given enough data and model capacity, the network is guaranteed to recover the gradient of the log likelihood (Vincent 201 known as the score function. In other words, the model sø (zx(t) is trained such that: (t) sharing and conditioning on time f to learn all the scores. Once the model trained, we start from sample of the final distribution, Φ1, and then use the learned score to gradually denoise it (Song & Ermon 2019 2020). The final variance of is selected to be very large such that the distribution Q approximately Gaussian, i.e. the signal to noise ratio tends to 0.
Vz, log go (.x). In practice, we parameter
DDPMs corrupt by rescaling the input images and by adding noise. The corruption can be modelled with Markov chain with perturbation kernel qi (æt [x,- Ar) N( l vT F.x At, Σ
2

---

Under review as a conference paper at ICLR 2023
Br Typically, 31 which is derived by minimizing an evidence lower bound (ELBO) (Ho et al. 2020).
and hence Q1
(O,
I). DDPMs are also trained with the DSM objective
In their seminal work. Song et al. (b021b) observe that the diffusions of both Score Based models and DDPMs can be expressed as solutions of Stochastic Differential Equations (SDEs) of the form:
dæ = (ax, t)dt + g())dw,
where is the standard Wiener process. Particularly, Score- Score-Based models use: f(x, 0, g(t)
V at and DDPMs use: f(x, t) = - 3,,, g(t) = v As explained earlier, for Score-Based models we need large noise at the end for the final distribution approximate normal distribu- tion. Hence, the corresponding SDE named Variance Exploding (VE) SDE (Song et al. 202Tb). DDPMs usually have final distribution of unit variance and hence their SDE is known as the Variance Preserving (VP) SDE (Song et al., 202 [Snn et al. (2021b) also propose another SDE with bounded variance, the subVP-SDE. that experimentally yields better likelihoods.
For both Score-Based models and DDPMs, Eq. is known as the Forward SDE. This SDE is reversible Anderson ( (982) and the Reverse SDE is given below:
dx (xx,
(t) log 2 (<x) dt g(t)du;
where is the reverse time standard Wiener process. Typically, V log qi (x) is approximated by (0()))t) and samples are generated by solving the Reverse SDE (Song et al. 2021b).
3 METHOD
Our framework for training diffusion models with more general corruptions includes three compo- nents: i) the training objective, ii) the sampling, iii) the scheduling of the corruption mechanism.
3.1
TRAINING OBJECTIVE
We study corruption processes of the form:
3; C,ay 8c71:
(4)
where C. IR" + R is deterministic linear operator, n) is a Wiener process, and is non- negative scalar controlling the noise level time t, and 2o ~ (z)).. We further denote with of the variance of the noise at level and we assume that is non-decreasing function of f. Unless stated otherwise, we assume that time is continuous and runs from that at t 0, have Co xnx and σu 0, i.c. assume that recovering zo from Φ; is harder as gets larger (i.e., entropy of qe qo xx) increases with t). Eq. 44) defines general class of diffusion processes, that includes (as special cases) the VE, VP and subVP SDEs used in [Song et al. (202T)). Our diffusion the sum of a deterministic linear corruption of æo and stochastic part that progressively adds noise. For any corruption process of this family, we are interested in learning the scores, i.c. Vz, log ge (x(e)) for all t.
to 1. Additionally, we assume corresponds to natural images. We also
For the vanilla Gaussian denoising diffusion, the celebrated result of Vincent (201 shows that we only need access to the gradient of the conditional log-likelihood, Va. log ge (Zty (0)) in order learn the score, Vx, log (xz). By revisiting the proof of Vincent (2011). Ip. we find that this is actually true for a wide set of corruption processes, as long as some mild technical conditions are satisfied. In fact, the following general Theorem holds:
Theorem 3.1. Let 9o be two distributions TR. Assume that all conditional distributions, qe (o, 2o)) are fully supported and differentiable in R², Let:
(0) Enn-ag
[1150(ar.)) - Van log ge (2:)))1)
(5)
J2(0)
2 B (=) (mn (m))
Vn, log qe [=0)]
(6)
Then, there is universal constant C (that does not depend on 0) such that: J (0) Jz(0) C.
3

---

Under review as a conference paper at ICLR 2023
Theorem 3.1 implies that minimizing the second function is equivalent to minimizing the first one. The second function is nothing else than the DSM objective. Our main observation is that noise is not always necessary for learning the score using the DSM objective. A necessary condition is that the corruption process gives non-zero probability to all æ; for any image zo- This is easily achieved by adding Gaussian noise, but this is not the only option. The proof of this Theorem is deferred in the Appendix and it is following the calculations of Vincent (2011).
Network parametrization. For the class of diffusion processes given by Eq. qo (x (x)) N (z); μ Ci E of I) and hence the objective becomes:
(4) we have that:
C,a0 2i of
L(t)= = 2
s0(2))t)
mo)
As shown, the objective of the model is to predict the (normalized) difference, Ciaro 2t, which is actually the noise, V)-- We argue that even though this objective is theoretically grounded, in many cases, it would not work in practice because we would need infinite samples actually learn the vector-field Væ, log qr (2t) in a way that would allow sampling.
Assume that the corruption process is blurring (at different levels) paired with additive noise of small magnitude. The objective written in Eq. (7) learns the distributions of blurry (and slightly noisy images) by just removing noise. Hence, practice we might only learn these distributions locally (around the blurry images) and hence we might not be able to reduce the blurriness. This point might be better understood after we present our Sampling Method in Section 3.2
To account for this problem, we propose a network reparametrization which leverages that we know the linear corruption mechanism, C.. Specifically, we propose the following parametrization:
se sx(2](t) = C,ho (z, t) Zt.
(8)
of
Crucially, the network incorporates the corruption process. The loss becomes:
L(t)= E (0 (2,
[|Ca(ho(ax)lt) =u)]l²]
(9)
of
When C2 is blurring matrix, this loss function is the MSE between the blurred prediction of he and the blurred clean image. Finally, as observed in previous works (Song & Ermon, 2019:] Ho et al.)
2020 Karras et al. 2022), the optimization landscape becomes smoother when we are predicting the residual, instead of the clean image directly. This corresponds to the additional reparametrizationn:
ha [t) (x.(()) + wt.
(10)
which leads to the final form of our loss function:
T(t)= (a (o0a)))) (w, (aw) of
[IC (00())))) r.)113]
(11)
where r, is the residual with respect to the clean image, i.e. rt 20 24, Following prior work, we use a single network conditioned on time & that is optimized for all L(). Hence, the total loss is:
== 11
(o) (ata) ||C, (2)(())) r2)l
(12)
where the weights are usually chosen to be 1 or 1/0? Karras et al.
2022 Kingma et al. 2021)
We call our training objective Soft Score Matching. The name is inspired from "soft filtering" a term used in photography to denote image filter that removes fine details (e.g., blur, fading, etc). As in the Denoising Score Matching, the network is essentially trained to predict the residual to the clean image, but in our case, the loss is in the filtered space. When there is filtering matrix, i...
C. = I, we recover the DSM objective used in (Song & Ermon 2019 2020 Song et al. 2021b).
Comparison with objectives used in other works. Previous (Anonymous 2022) or coneur- rent (Bansal et al.] 2022 Rissanen et al., 2022, Hoogeboom & Salimans, [002)] works that con- sider different degradations than Gaussian Diffusion, use the heuristic objective of predicting the clean image, i.e. they minimize: | t) rell. This is actually an upper-bound
our loss, - r-l Since the spectral norm, ||Ce]]. is fixed, one
Le. ||C, (0o rl & ||C, can optimize for the upper-bound by minimizing (x,)( rıll. Instead, Soft Score Matching optimizes directly for learning the score. Experimentally. Soft Score Matching outperforms (ours FII:: 1. 1.85, theirs: 5.91), under the exact same setting, this simple baseline (see Experiments).
4

---

Under review as a conference paper at ICLR 2023
3.2 SAMPLING
Naive Sampler. Once the model is trained, we need a way generate samples. The simplest idea is to use recursively our model, Φg(xx((), to get estimates of the clean image. Z0. To move from corruption level t to cor- ruption level t At. we feed the image to the model to get an estimate of the clean image, Do, and then corrupt back to level t - At. This idea is shown in Algorithm |
Algorithm I Naive Sampler
Require: pl, 10: :C ot, At In ~ pi (æ)
for l = to 0 with step Àl do
Lu Φa(we (1)
7. NO.
2 Ci-srão + 01-2:7
end for return Jo
Momentum Sampler. As we will show in the Exper- iments section, the naive sampler we presented above leads to generated images that lack diversity. We propose an simple, yet novel, alternative method for sampling from the general linear diffusion model presented in Eq. (4). Our method is inspired by the continuous formulation of diffusion models that is introduced in [Song et al. (222lb)
The first step is to find a Markovian stochastic process that is "close" to the non-Markovian corrup- tion process of Eq. 4 Consider the following SDE:
dæ = C,E(zo|z,,dtt + d(o dw,
(13)
dt
where w is the standard Wiener process. This special case of the Itô SDE: dæ f (a. t) +
g(t)dw, that appears in Song et al. (2021b), i.e. f(x,t) C, mx) and g(t) = V di
d(o?)
Crucially, E[ does not depend previous values of æc, and our SDE is indeed Ito SDE.
To build some intuition, it is useful to think of the toy setting where the dataset contains one image, R" Under this setting, the Markovian corruption described by the SDE of Eq. (13) has the same marginal distributions as the process of Eq. (4) As we explain in the Appendix (Section D in the general case, the SDE (13) introduces an approximation error with respect to the corruption process of Eq. 4 the former uses the conditional expectation, E[z0//z]] while the latter corrupts directly æy- Nevertheless, the approximation error for the considered corruptions seems to be small, given the experimental success of the derived sampler. Intuitively, this is because for low corruption levels, the conditional expectation and Bg are close, while for high corruption levels, the distance contracted by the multiplication with the corruption matrix C2 (see also Appendix, Section D
Eq. 13) describes reversible diffusion process (Anderson 1982). The reverse also an Ito SDE:
d(o?), di
d(o?) di
dæ. C.
V.. log qe (xx) dt +
dw.
(14)
where is standard Wiener process when time flows backwards from = to In practice, to solve Eq. (14). we discretize the SDE (i.e., apply Fuler-Maruyama, and approximate the function derivatives with finite differences). The Euler- Maruyaaa discretization is given below:
Ft-A: - F. = (C,-at E(j)o/ - (0ī- - V (a::) + Vo of- (15)
where 77 (O, I). In this update Equation, there are two unknowns: the conditional expecta- tion, E| æz]. and the score function, V log gr (x;) (xr). We show that these are actually connected,
through the Tweedie's (Efron 2011 Robbins 1956 Stein 1981) formula. Specifically, it holds that:
C [. = æ; + of V log gr (x:e..
(16)
The proof is given for completeness in the Appendix, Lemma A.1 use our model that provably learns the score according to Theorem
To estimate Va, log gi (axz) we Putting everything together:
Az; Dt-At 2t
(17)
= (Cr, A: Ct) (0o( + - ميمم (c.
(Do(aa)lt)) + t
=x + Vo o-
Our sampler is summarized in Algorithm 2 Essentially, there is one update for deblurring and one for denoising. At the core of this update equation, is the prediction of the clean image, dgg Once
5

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Momentum Sampler
Require: D1.00.Co a,, At #1 (a))
for 1 to with step At do
So Φa(x;(t) + 3t
Coarse prediction the clean image.
gi + C.to
> Coarse prediction of filtered image at t.
n (00.))
é, Yt
Estimate of noise at > Filtered image at with noise at t At.
(o 2e 0/-
E: + Vot o.-2.7
2t-At f xt
ÿ1 + Ciao
P Coarse prediction of filtered image at f At.
3i-A1 Zi-At (ğ:-21- -9))
D Filtered image at At with noise f At.
end for return 370
the clean image is predicted, we blur it back to two different corruption levels, t and At. The deblurring gradient is the residual between the blurred images at levels - At and f. Interestingly. the denoising update is the same the one used in typical score-based models (that only use additive noise). In fact, if there is no blur (C, = I), our sampler becomes exactly the sampler used for the Variance Exploding (VE) SDE in Song et al. (2021b).
We call our sampler Momentum Sampler because we can think of it as a generalization of the update of the Naive Sampler, where there is momentum term. To understand this better, we look at the setting where there is no noise. Then, the update rule of the Momentum Sampler is:
(18) As seen, the first term is what the Naive Sampler would use to update the image level and the second term is what the Naive Sampler would use to update the image at level t At. If these two directions are aligned, then the gradient Ax, small. Hence, there is notion of momentum, analogous to how the term is used in classical optimization. In the Appendix, Section B.2 we also present DDIM-type (Song al. 2021a) sampler for which the momentum term also appears.
Ar, Ct- C,
Probability Flow Momentum Sampler. The update rule of our sampler was derived using the discretization of the backward SDE of our corruption, given in Eq. (14). Similarly to Song et al. (002b), we can also consider the Ordinary Differential Equation (ODE) associated with this SDE:
1 d(o)) Va, log (zx)) dt.
dæ: C,E(ao
(19)
2 dt
Again, we approximate C,E[ [r). Va, log (xx;) with trained network and get deter- ministic version the Momentum Sampler, which we call Probability Flow Momentum Sampler, as in Song et al. (2021b). We detail our derivations in the Appendix, Section B.I
3.3 SCHEDULING
The last piece of our framework is how to choose the corruption levels, i.c. the scheduling of the diffusion. For example, for Gaussian Blur, the scheduling decides how much time is spent in the diffusion in the very blurry, somewhat blurry and almost no blurry regimes. We provide a principled way of choosing the corruption levels for arbitrary corruption processes.
Let Do the distribution of real images and D, be a known distribution that we know how to sample from, e.g. Normal Distribution. In the design phase of score-based modeling, the goal is choose a set of intermediate distributions, } that smoothly transform images from Do to samples from the distribution Dj. Let θ {θ1, 02 ..., ..... be the space of diffusion parameters, i.c. each A corresponds to a distribution Dø. In the case of blur for example, 6; controls how much we blur the image. Let also M : X x X - IR be a metric that measures distances between distributions, e.g. M might be the Wasserstein Distance of distributions with support X'.
We construct a weighted graph G, with the nodes being the distributions and the weights given by:
w (Do, Do,) = [M [M (De.. Do,), M (Do., Da,) S e, 0o, otherwise.
(20)

---

Under review as a conference paper at ICLR 2023
For fixed e, we choose the distributions that minimize the cost of the path between Do and Di- The parameter e expresses the power of the best neural network we can train to reverse one step of the diffusion. If = oo, then for any metric M, the shortest path is to go directly from Do to D1- However, it is impossible to denoise in one step a super noisy image. Hence, we need to go through many intermediate steps, which is forced by setting of the candidate distributions we are getting closer to finding the geodesic between Do and Dj. However, the computational cost of the method increases since we need to estimate all the pairwise distances (D Do,)) ). In practice, we use relatively small number of candidate distributions, e.g. T 256 and once the path is found, we do linear interpolation to extend the continuous case.
smaller value. As we increase the number
EXPERIMENTS
We evaluate method in CelebA-64 and CIFAR-10. We show that by just changing the corrup- tion mechanism (and using our framework for scheduling, learning and sampling) we significantly improve the FID score and reduce the sampling time. We the architecture and the training hyperparameters from Song et al. 202Tbb (full details can be found in the Appendix).
Gaussian Blur as our primary corruption mechanism. To illus- trate that our method works more generally, we also show results with masking which is discussed separately later. Consistent with the description of our method, our deterministic corruptions are also paired with additive low magnitude noise. This is required by our theoretical results, otherwise the conditional log-likeling likelihood, log g (e (0) would be undefined. We also find the addition of noise beneficial in practice (see Appendix F.I.I for ablation studies on the role of noise).
For most of our experiments,
Scheduling. We use the standard geometric scheduling for the noise (Sog & Ermon 2019
2020
Song et al. 202Tb) and use the methodology described in Section 3.3 to select the blur levels. We use the Wasserstein distance as the metric M to measure how close are the different distributions. To clearly illustrate the switch a different corruption, our diffusion has an initial stage that increases the noise (with no blur) and then we fix the noise (to small value, e.g. = 0.1) and change (using our scheduling framework) the amount of blur. Our diffusion spends less than 20% of the total time
in the initial stage that increases the noise. We ablate those choices in Section F.] of the Appendix.
One important property of our framework is that the scheduling is dataset specific. Intuitively, the way we corrupt depends on the nature of the data we are modelling. The interested reader can find the found schedulings for each dataset in Figure 6 of the Appendix.
Results. We train our networks on CelebA-64 and CIFAR-10 10 using the found schedulings and the training objective of Eq. (12. We start by showing uncurated samples of our models in Figure 2
Figure 2: Uncurated samples from our trained models CIFAR-10 (left) and CclebA (right).
The generated images have high diversity and fidelity in both datasets. We compare the FID (Heusel et al. 2017) obtained by our method on CelebA with many natural basclines that use any of the VE, VP PP SDEs. Specifically, we compare against DDPM (Ho et al. (020)) that uses the VP SDE, DDIM (Song et al. 2021a) that uses the same model but with different sampler, DDPM++ (Kim) ct al., 2022b) that is the state-of-the-art model for VP-SDE and the NCSN++ models (Song et al. b02 trained with the VE and VubDDsS For a fair comparison, we only use reported num- bers in published papers for the baselines and we do not rerun them ourselves. Our model achieves state-of-the-da FID score, 1.85, in CelebA, outperforming all the other methods. The results are summarized in Table I. For CIFAR- 10, we obtain FID score 3.86 with our Probability Flow Mo- mentum Sampler and 3.91 with our Momentum Sampler. We summarize the results in Table Our
7

---

Under review as a conference paper at ICLR 2023
best FID is competitive with similar samplers to similar methods, e.g. with NCSN+ (VE SDE) with Reverse SDE sampling. However, it is behind other state-of-the-an models such as LSGM (Vahdat et al. 202T and DDPM (Ho et al., 2020). We believe that this performance gap can be decreased in the future by further research in the area of diffusion models with general corruptions.
Our method is superior in sampling time, for both CIFAR-10 and CelebA. Figure 3 shows how FID changes based on the Number of Function Evaluations (NFEs). Our method requires significant less steps to achieve the same or better quality than NCSN++ (VE SDE) (Song et al. 2021b), using the same architecture and training hyperparameters (FID values taken from (Ma et al. 2022
Model
FID
512
DDPM (VP SDE)
(Ho et al.
2020
3.26
DDIM (VP SDE) (Song et al. 202
3.51
DDPM+ (VP SDE) Kim ct al. 20226)
1.90
0 4
NCSN++ (subVP-SDE) (Song el al.
202 lb)
3.95
NCSN++ (VE SDE) (Song et al. 2021b)
3.25
Ours (VE SDE Blur)
1.85
200
400
6000
600
1930
Number of Function Evalkations (NFEs)
Table 1: FID results on CelebA-64.
Figure 3: FID versus NFEs (CelebA-64).
Ablation Study for Sampling. For all the results presented so far, we used the Momentum Sampler that we introduced in Section 3.2 and Algorithm 2 In this section, we ablate the choice of the sampler. Specifically, we compare with the intuitive Naive Sampler described in Algorithm I We show that the choice of the sampler has a dramatic effect in the quality and the diversity of the generated images. Results are shown in Figure 4 The images from the Naive Sampler seem repetitive and lack details. This is reflected in the poor FID score. The Momentum Sampler leads to images with greater variety and detail, dramatically improving FID from 27.82 to 1.85.
(a) Naive Sampler (uncurated). FID: 27.82.
(b) Momentum Sampler (uncurated). FID; 1.85.
gives repetitive images that lack details. Momentum Sampler 4b) dramatically improves the sampling quality and the FID score.
Figure 4: Effect of sampling method on the quality of the samples. Naive Sampler 4a
Training Objective. Concurrent works (Hoogehoom & Salimans 2022 Bansal al. 2022 that also consider different corruption processes have used as their loss a simple objective: given an input image, they predict the residual to the clean. As explained in Section [3.]] this is equivalent to optimizing for an upper-bound of the Soft Score Matching objective (which is minimizing the error to the score-function, see Theorem 3.T). We show that if we use our exact pipeline (model, scheduling, sampler, training hyperparameters and so on) and we replace our Soft Score matching Loss with the loss introduced in [Bansal et al. (2022), FID score increases from 1.85 5.91. This experiment shows the effectiveness of the Soft Score Matching objective.
Ablation Studies for Scheduling. We perform extensive ablations on the scheduling to understand the role of noise in the framework and whether our scheme outperforms other natural baselines. The results are detailed in the Appendix, Section F. Our main findings are: i) the Momentum Sampler works even if noise and blur are changing simultaneously (i.e. for schedulings with non-fixed noise), ii) lowering the maximum value of noise leads to important performance degradation - for very small noise the method completely fails and iii) our found scheduling outperforms significantly (baseline FID: 8.35, ours: 1.85) a natural baseline that sets blur parameters such that MSE between the corrupted and the clean image decays in the same rate for Gaussian Denoising and Soft Diffusion.
8

---

Under review as a conference paper at ICLR 2023
Masking Diffusion Models. To show the generality of our framework, we also train models with (discrete) masking diffusion paired with noise. Figure 1 (top 2 rows) shows the forward and the (learned) reverse process for blur on the left and masking the right. We train the model with our Soft Score Matching objective on CelebA. Unconditional samples from the model trained with masking can be found in Figure [3 of the Appendix. In Figure [ we show the predictions of our two trained models (blur and masking) for the conditional mean, [[2o| at different times of the diffusion. Soft Score Matching trains the model make prediction that matches the real images in the filtered space. Hence, given masked images the masking model is incentived to predict right only the observed noisy region. As diffusion time t becomes smaller (cleaner images), the observed region grows and the model predicts bigger windows. Although it is interesting that we can train Masking Diffusion models, there are several limitations compared to Blur Diffusion (and even Gaussian Diffusion). We observe that these models are very slow to sample from: with 1024 sampling steps, FID is 30.92 while with 4096, FID improves to 12.37.
5 RELATED WORK
We showed that by just changing the corruption mechanism, we observe important computational benefits. Reducing the number of function evaluations for sample generation with diffusion models
is an active area of research. Policoeur- Martinnea et al. (202T); Liu et al. (2022); Song et al. (021
propose more efficient samplers for diffusion models. Rombach et al. (2022); Daras al. (2022)
train diffusion models on low-dimensional latent spaces. Xiao et al. (2022) combine diffusion mod-
els with Generative Adversarial Networks (GANs) to allow for bigger denoising steps. Ryu & Ye (2022); Ho et al. (2022a) generate high-resolution images progressively, from coarse to fine quality. Salimans & Ho (2022) train progressively a student network that mimics the teacher diffusion model with fewer sampling steps. We note that all these works are orthogonal to ours and therefore can be used combination with our framework for even faster sampling.
On scheduling, there is closely related work by Bao et al. (2022). The authors find a closed form solution (w.r.t. the score function) for the variance of the reverse SDE for Gaussian Diffusion. Then, they select noise scheduling that minimizes the KL along the path from the initial to the final distribution. In our work, we use Wasserstein distances and consider general corruption processes for which it is unclear whether such closed form solution exists. Instead, we estimate the distances in a data-driven way. This allows us to schedule arbitrary diffusion processes in a principled way.
There is significant recent (Anonymous 2022) and concurrent work (Rissanen al. 2022, Bansal
et al. 2022 Hoogeboom & Salimans| 2022) that proposes diffusion with other degradations. These works have significant differences since they use different loss functions and sampling methods. Soft Score Matching experimentally outperforms (under the exact same setting) the loss functions used in the concurrent works (ours FID: 1.85, theirs: 5.91). Our (blur) models obtain state-of-the-art FID for linear diffusion models on CelebA (FID 1.85). For CIFARIO. 10, we outperform (FID: 3.86) Gaussian diffusion with Variance Exploding SDE (Sogg et al.) 2021b). Hoogeboom & Salimans (2022) also use blurring (but with Variance Preserving SDE) to further push the CIFAR- 10 performance to FID 3. 3.17. These advancements show that there are multiple diffusions with promising potential.
CONCLUSIONS AND FUTURE WORK
We presented a framework to train and sample from diffusion models that reverse general linear cor- ruptions. We showed that by changing the corruption process, we can get significant sample quality improvements and computational benefits. This work opens several future research directions. For example, it is possible to optimize or learn the corruption process for solving a specific type of inverse problem. It is also worth exploring if mixing different corruptions (blur, noise, masking, etc.) improves performance. Our work could also be extended to the non-linear case, leveraging the techniques introduced in (Rombach et al. 2022) Kim et al.) 2022al Wang et al.] (2022). Finally, is important to understand the role of noise, from both theoretical and practical standpoint.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326 1982.
Anonymous. Diffusion models in space and time via the discretized heat equation. In Submitted to ICLR Workshop on Deep Generative Models for Highly Structured Data, 2022. URL https:
/ /oopenzeveew.oo
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18208-18218, 2022.
Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie Li, Hamid Kazemi, Furong Huang, Micah Gold- blum, Jonas Geiping, and Tom Goldstein. Cold Diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.0990 2022.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models. arXiv preprint arXiv:2201.06503, 2022.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint arXiv:2206.00 00941, 2022.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion Models in Vision: A Survey. arXiv preprint arXiv:2209.04747 2022.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems. volume 26. Curran Associates, Inc., 2013. URL btttps: htwp:occeeddnggs..eei....... neurips cc/paperr2213/fille/ af2ld0c97db2ed (tt.:///...2eeee1111222221333333
pdf
Marco Cuturi, Laetitia Panaxaaaax,,,sos, Yingtao Tian, Charlotte Bunne, Geoff Davis, and Olivier Teboul. Optimal Transport Tools (OTT): JAX Toolbox for all things Wasserstein. arXiv preprint arXiv:2201. 12324, 2022.
Giannis Daras, Yuval Dagan, Alexandros G Dimakis, and Constantinos Daskalakis. Score-guided intermediate layer optimization: Fast langevin mixing for inverse problem. arXiv preprint arXiv:2206.09100 2022.
Jacob Deasy, Nikola Simidjievski, and Pietro Liò. Heavy-tailed denoising score matching. arXiv preprint arXiv:21 2021.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-885 8794, 2021.
John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics. 6(1):19-26, 1980.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Associa- lion, 106(496):6022-161) 2011.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-68511 2020.
10

---

Under review as a conference paper at ICLR 2023
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23: 2721,20224
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. AXXv::2204033558 2022b.
Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. arXiv preprint arXiv:2209005557, 2022.
Emiel Hoogeboom, Alexey Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In International Conference on Learning Rep- resentations, 2022a.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant dif- fusion for molecule generation in 3d. In International Conference Machine Learning, pp- 8867-8887. PMLR, 2022b.
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advances Neural Information Processing Systems, 34:1 10111499888100 14944, 2021.
Daniel D Johnson, Jacob Austin, Rianne van den Berg, and Daniel Tarlow. Beyond in-place corrup- tion: Insertion and deletion in denoising probabilistic models. arXiv preprini urXiv:2107.07675. 2021.
Alexia
Ke Li, Rémi Piché-Taillefe Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105. 14080, 2021.
Zahra Kadkhodaie and Eero Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in a denoiser. Advances in Neural Information Processing Systems, 34:13242- 13254, 2021.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint Ttf::22000003664 2022.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochas- tically. Advances in Neural Information Processing Systems, 34:21757-21769, 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Advances in Neural Information Processing Systems, 2022.
Dongjun Kim, Byeonghu Na, Se Jung Kwon, Dongsoo Lee, Wanmo Kang, and I1-Chul Moon. Maxi- mum likelihood training of implicit nonlinear diffusion model. In Advances in Neural Information Processing Systems, 2022a.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and II-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. In International Conference on Machine Learning, pp. 11201-11222 12288 PMLR, 2022b.
Diederík Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances in neural information processing systems, 34:21696-21707, 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
Rémi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. Bayesian imaging using plug & play priors: when langevin meets tweedie. SIAM Journal on Imaging Sciences, 15(2):701-737, 2022.
Sangyun Lee, Hyungjin Chung, Jachyeon Kim, and Jong Chul Ye. Progressive deblurring of diffu- sion models for coarse-to-fine image synthesis. arXiv preprint arXiv:2207. 2022.
11

---

Under review as a conference paper at ICLR 2023
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models manifolds. arXiv preprint arXiv:2202.0977 2022.
Hengyuan Ma, Li Zhang, Xiatian Zhu, Jingfeng Zhang, and Jianfeng Feng. Accelerating score- based generative models for high-resolution image synthesis. arXiv preprint arXiv:2206.04400 040229 2022.
Dimitra Maoutsa, Sebastian Reich, and Manfred Opper. Interacting particle solutions of fokker- planck equations through gradient-log-densty estimation. Entropy, 22(8):802, 2020.
Eliya Nachmani, Robin San Roman, and Lior Wolf. Denoising diffusion gamma models. arXiv preprint arXiv:2//0.05945 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusson probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A. Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. 2022.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 1.Xiv:2204.06122 2022.
Julius Richter, Simon Welker, Jean-Marie Lemercier, Bunlong Lay, and Timo Gerkmann. Speech enhancement and dereverberation with diffusion-basee generative models. arXiv preprint arXiv:2208. 05830, 2022.
Severi Rissanen, Markus Heinonen, and Amo Solin. Generative modelling with inverse heat dissi- pation. arXiv preprint arXiv:2206./333 133997 2022.
Robbins. An empirical bayes approach statistics. Proc. 3rd Berkeley Symp. Math. Statist. Probab., 1956, volume 1, pp. 157-163, 1956.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEF/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Dohoon Ryu and Jong Chul Ye. Pyramidal denoising diffusion probabilistic models. arXiv preprint 2:Xiv:2208.018e0 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.1 1487, 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image supoutiesolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022b.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffussin models. In International Conference on Learning Representations, 2022. URL/Itttps://operrviii..
Joan Serrà, Santiago Pascual, Jordi Pons, R Oguz Araz, and Davide Scaini. Universal speech en- hancement with score-based diffusion. arXiv preprint arXiv:2206.0300 03066, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 202 la.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution.
Advances in Neural Information Processing Systems, 32, 2019.
12

---

Under review as a conference paper at ICLR 2023
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438-1244 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021b.
Charles M Stein. Estimation of the mean of multivariate normal distribution. The annals of Statistics, pp. 1135-1151, 1981.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score- generative modeling in latent space. In Neural Information Processing Systems (NeurIPS), 2021.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compu- tation, 23(7):6661-167 2011.
Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion- gan: Training gans with diffusion. arXiv preprint arX2022206.02262, 2022. URL https://
a arxiv. mrxiv.org/abs/22000
02262
Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Pey- man Milanfar. Deblurring via stochastic refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16293-16303, 2022.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference on Learning Representations, 2022.
Ling Yang, Zhilong Zhang, Shenda Hong. Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Ming-Hsuan Yang, and Bin Cui. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.00796 2022.
Mao Ye. Lemeng Wu, and Qiang Liu. arXiv:2209.01170, 2022.
First hitting diffusion models.
arXiv preprint
A APPENDIX
A.1 PROOFS
Theorem 3.1 Let qu be distributions in R". Assume that all the conditional distributions, ge (x, (o). are supported and differentiable in R". Let:
(1) = Egenng [1100(a.)= - Va, log ge (2x)))1
(21)
2
J2(0) = am ) (o)). (2n
(1l5) Ve, log ge [x)
(22)
Then, there is universal constant C (that does not depend 0) such that: Jı (+) J2( C.
The proof of this Theorem is following the calculations of Vincent (20IT). We observe that long as the technical conditions listed are satisfied, the proof holds independent of the corruption type. We provide the proof below for the shake of completeness.
Proof of Theorem
3.1
Ji
F Eurrrq
[|10( 280 T Va, log (are) + I/, log (xx)
(23)
E Bnrwn [1180(#=))13]] Enzra (3a()))) Vmn (x:)) + Ci.
(24)
13

---

Under review as a conference paper at ICLR 2023
Similarly,
J2(0) = 2 Bes [[]s(())]1]] - (1x mn) [sp(<)=)* Ve, log (=(2 + O2.
(25)
It suffices to show that:
Eerry (z)()))VV Va, log qe (zx)] = E
[so(x,)** Vx, log (2)l2000)) (26)
We start with the second term.
E ((0 (200
[s00 Vzn
ar log qe (ar [20)]
20 (1)))))(()) Vay mz log ge (at (Dg | daoo
(27)
$ (zx) (00) Vz, log qe (20)) dxydæy
(28)
ax
T (Ixz)
90
V 2e (x amo) dx
(29)
2r (x, 200)
o (xx) (00 (00) V (2: (20 dæ;dæo
(30)
m
$ (z,) (go 9 dændæ,
(31)
mn
$ (at))
go( V ( (dzo dx.
(32)
a:
T. (x1)
(L
Va, Ve,
(90(zu) (x) dæy dx
(33)
J00
Vv. (/r To (0(2))) 20 ) dx
o (Iz)
(34)
m2
[ sg (x:) V ge (x,
(35)
(5 (x,) V n. log (x) (x,
(36)
[o) (xx) Vz, log qe
(37)
Lemma A.1 (Tweedie's ' formula). Consider the corruption process: T; C + o.m where ~ NO, I)- Denote with qe (x,)) the density of æ. and assume that log qu (xx)) is differentiable everywhere. Then, for the Minimum Mean- Mean-Square Estimation (MMSE) of an given æt, it holds that:
C,Elaola..] xz + oi as log qt (are..
(38)
Proof.
(((x,) ( (
(39)
(x)e) = foo(20)) Van (ms mo daz
(40)
(00) V v.. log ( xo )
(41)
14

---

Under review as a conference paper at ICLR 2023
We know that: qi (z) (a0) N Cioo, E = Hence,
oi
Vx, = /10(200) (1(r)(())))))))) Cuay 3t dar
(42)
1
20 (12 Is faolzo)al (ar (xo )do
C
(43)
1
C /10(000 ( )andao - amg((are)
(44)
0,
V n, (1))))
1
(C,b xz)
(45)
(2)2)
01
V
log
(C,B(ao) - -xt)
(46)
o?
B DETERMINISTIC SAMPLERS
B. PROBABILITY FLOW ODE
In the main text, we derived our Momentum Sampler by analyzing the Backward SDE associated with our corruption process. Inspired by the works of Song et al. (2021b); Maoutsa (2020),
we also consider deterministic sampling that is derived by looking at the ODE that describes our diffusion. Particularly, the ODE:
dæ; t) 2 (t) a. log q (x:) dt.
(47)
has the same marginal distributions (Anderson 1982
Maoutsa al. 2020 Song
202 Chen
et al. 2018) with the Backward SDE:
dæ. [f 1) 9² (l) lo:g di + g())dw.
(48)
For our case, Eq. becomes:
dæ: C.E[o//a 1 dd(o?)
m. log (xs)) de.
(49)
2 dt
The first-order discretization this ODE given below:
(oī- of) - Vx, log qi (xt).
T:-At - x2 = (Ct-a: - C.)EEooo [x]]
(50)
2
We estimate C, 2o x. and Va, log (xx)) with our neural network and we get the Neural
ODE
(Chen et al. 2018):
Ar; 1t-Ar D2
(51)
oi- of
of (Ce
=C - Ct) (o((x))) + xt
coe de xz (t) + xz
x.)
which is the update rule of our Probability Flow ODE Momentum Sampler.
We note that our simple discretization is not the only way to solve the ODE of Eq. more sophisticated, e.g. see Dormand & Prince (1980).
We can use
B.2 DDIM
A popular sampling scheme the DDIM method, introduced in Song et al. (2021a). The idea is to the network to predict d from a; and then use the forward model to move from to to at- At-
15

---

Under review as a conference paper at ICLR 2023
This is same with the Naive Sampler we introduced in the main paper but the difference in DDIM is that part of the stochasticity replaced by "simulated" noise. The main trick to simulate the noise is observe that once know 2o and 22 can once again the forward model to estimate the noise. We can use the same idea for Soft Diffusion sampling. Specifically, we propose the following DDIM-type sampler:
2t C,ho(ze,t) +h2n.
h2 - CC- t)) + +v oo
(52)
20
en (0) simulated noise
This Equation is very similar to Equation 12, page 5 in the DDIM (Song et al, 202 paper. The parameter controls the amount of stochasticity, i.e. how much of the noise is simulated. For k = 0, we get a deterministic sampler. Experiments with the deterministic DDIM-type sampler can be found in Section F3
Figure 5: Conditional means E p/o) xr.] predictions of our blur/masking models, at different diffusion times.
C SCHEDULINGS
We use our framework to select the blur levels in unsupervised way. For solving the opti- mal transport problem, we use the Sinkhorn Distances by Cuturi (2013) that has been extensively used for image experiments. We the implementation of this paper from the software package ott-jax (Cuturi et al. 2022). We experimented with both using the whole dataset and using slices. As expected, dataset slices lead to higher estimation errors, but we observe that the found scheduling doesn't 't change much, i.e. the estimation error increases approximately uniformly for all the pairs for reasonably sized dataset slices. For the schedules found in the paper, used slices of 10% of the dataset. We start with 256 different blur levels and we tune such that the shortest path contains 32 distributions. We then use linear interpolation extend the continuous case. Full experimental details can be found in the Appendix. These choices seem to work well in practice, but further optimization could made in future work.
Figure 6 shows the found schedulings for the CelebA and the CIFAR-10 datasets. Notice that the scheduling is slightly different between the two datasets - the diffusion depends on nature data we are trying to model. We underline that the parameters for the blur are selected without any supervision, by solving the optimization problem we defined in Section 3.3
0.10
CelebA (64x64)
CIFAR10
0.08
GaussianBlure (lussianBlurio)
50
Noisela)
30
0.04
0.02
CelebA (64x64)
10
0.2
0.4
8.0
0.2
0.8
diffusion time
diffusion time
(a) Noise Scheduling
(b) Blur Scheduling CelebA
Figure 6: Diffusion scheduling for CelebA-64 and CIFAR-10. The blur corruption levels lected without supervision to minimize the sum of the Wasserstein distances between consecutive distributions. Notice that the scheduling slightly different between the two datasets the diffusion depends on the nature of the data we are trying to model. The support of the Gaussian blur kernel was set to x 65 and 161 161 for CIFAR-10 10 and CelebA-64 datasets respectively.
16

---

Under review as a conference paper at ICLR 2023
D TRAINING DETAILS
Hyperparameters. For our trainings, we use Adam optimizer with learning rate 2e -4, 31 0.9, 82 0.999, c le 8. We additionally use gradient clipping for gradient norms bigger than 1. For the learning rate scheduling, we use 5000 steps of linear warmup. We use batch size 128 and we train for
2M iterations (based on observed FID performance).
Blur parameters. For the blurring operator, we use Gaussian blur with fixed kernel size and we vary the variance of the kemel. For Celeb/--64, we keep the kernel half size fixed to 80 and we vary the standard deviation from 0.01 to 23. For CIFAR-IO, we keep the kernel half size fixed to 32 and we vary the standard deviation from 0.01 to 18. For both datasets, we implement blur with peroo-pdding. We chose the final blur level such that the final distribution is easy to sample from. In both cases, the final distribution becomes noise top of (almost) single color.
Final distribution For the blurring models, the final distribution is the distribution of very blurry images with additive Gaussian noise (we first blur, then add noise). At the limit, each blurry image becomes constant image having single color (i.c., the average color of the image). Hence, to sample from this final distribution, we first have to sample a single color (from the distribution of average colors in the dataset), generate constant image having that color, and then add little noise to the constant image. The distribution of average colors for all the considered datasets is very simple and we model it with a Gaussian distribution. Specifically, we fit three-dimensional gaussian distribution (one dimension for each color channel, diagonal covariance) to the average color of the dataset. To begin the inference process, we sample one color from this distribution, we create an image where every pixel has this color and then we add Gaussian noise.
Architecture. We use the architecture of Song et al. (2021b) without any changes.
Training Objective For all our experiments, we scale the loss at level t with w(t) = 1/// as in Song & Ermon 2019 2020 Song et al. (2021b).
Compute and Training Time We train our models on 16 v2-TPUs. Our blur models on CelebA had an average speed of 6 iterations per second. For CIFAR- 10, the average speed was 11 iterations per second We note that there is no overhead over the NCSN+ paper other than projecting to the measurements space, which can be done very efficiently for both blur and masking.
Evaluation We keep one checkpoint every 10000 steps and we keep the best model among the kept checkpoints based on the obtained FID score. We use 50000 samples to evaluate the FID, as is typically done in prior work. Regarding the number of steps, we selected the best FID among models evaluated with steps ranging from 200 to 1000 with step size of 80 (10 experiments total).
The Momentum Sampler seems to have a U-shaped performance plot, i.e. there is a sweet spot in the number of function evaluations that gives the lowest FID score. We observed this in both CelebA and CIFAR-10.
However, this not unique to our sampler- the general belief that sample quality improves with the number of steps has been called into question by recent works. Specifically, Karras et. al. do an extensive evaluation of the role of NFES for different samplers and they find that many stochastic samplers behave similarly to our Momentum Sampler. For example, we refer the interested reader to Figure 4, page 8 of the paper Elucidating the Design Space of Diffusion- -ased Generative Models that clearly shows that for many stochastic samplers performance slightly deteriorates after a certain threshold of function evaluations and the optimum is at some intermediate point. We emphasize, however, that the deterioration plers and therefore selecting the correct number NFES is important for our method. Consistent with the observations by Karras al. (2022) (e.g. Figure 2, page 4), for deterministic samplers the performance flattens (DDIM) or deteriorates slightly (Probability Flow Momentum Sampler) in the regime of very high Number of Function Evaluations (NFEs).
observed in CelebA is bigger compared to other stochastic sam-
We also want to note that selecting the optimal number of function evaluations for reporting FID is not uncommon in the literature. It is done in both the landmark papers Elucidating the Design Space
17

---

Under review as a conference paper at ICLR 2023
of Diffusion- Beafusioo-BBased Generative Models (Karras et al. 2022) and srore-Based Generative Modeling through Stochastic Differential Equations (Song et al. 2021b).
E LIMITATIONS AND THINGS THAT DID NOT WORK
Our method has several limitations. First, requires the diffusion operator to be known. This is not always the case, e.g. see Peebles et al. (2022) where diffusion is applied to checkpoints of different models. Another limitation is that our framework does not offer any guidance on which diffusion operators actually more or less useful for learning the data distribution. Particularly, we already showed that blurring is a much more powerful diffusion mechanism than masking, in the sense that it leads to better FID) scores and faster generations. It is yet be whether blurring is going to be outperformed by some other corruption method. Our method also only concerns linear diffusions (however, the extension to non-linear is relatively straightforward).
On the theoretical side, our method has also some shortcomings. First, it only intuitively explains why the reparametrization to the Denoising Score Matching is needed. Second, since our method is based on the Denoising Score Matching. it only works when the conditional log-likelihood is defined everywhere. There are distributions for which such condition is not satisfied, but still, can be learned (to some extent) with heuristic methods (Bansal et al.) 2022).
Another limitation of our work is the derivation of the Momentum Sampler and its sampling distri- bution. Consider the forward process:
dæ;
Co [ ] + g(t)dw
(53)
This process is described by an Ito SDE, i.e. an SDE of the form:
dæ, - f( f)dt g(t)dw,
(54)
where f(x,,)) t) C.Emoopo [0)]
The sampling process we write in the main body of the paper (that leads to the Momentum Sam- pler) is exactly the reverse of this forward process, where Exo [atg| is approximated by the neural network. Hence, the Fokker-Planch equations hold and we are guaranteed that (as long as the approximation of the conditional expectation and the approximation the score function are accurate), we sampling from the correct distribution (Song al. 2021b).
During training, we cannot use this forward process because we do not have the conditional expec- tation. Instead, replace the conditional expectation of æy with 2y itself. The mismatch between ægand the conditional expectation of &n leads to an additional approximation error in the learning of the score. For small the conditional expectation will be very close 26 and hence this approxima- tion eиror is small. For large values of f, the distance between the conditional expectation and æo is bigger. However, we are multiplying with the corruption matrix C,, which is removing information as grows. Therefore the distance of the corrupted conditional expectation and the corrupted o is also expected to be small. Our training process learns the correct score for the corruption process applied directly to Zp. Our sampler, however, assumes that we have learned the score using the conditional expectation of wo. We intuitively expect that these two are not far away (for all () as we explained. However, precisely characterizing this mismatch remains open.
On the practical side, we believe that our objective, Soft Score Matching, sometimes leads to slower sampling compared to the simpler objective of predicting the clean image. For example, for masking, since the model only penalized in the observed region, there is no incentive in expanding this region. Hence, to achieve smooth transition between different masking levels we need to many steps.
Experimentally, tried using framework with even less stochasticity, but it did not work, e.g. see 7 would be interesting to understand better what is causing the failure and also what is the proper amount of randomness required each diffusion step.
18

---

Under review as a conference paper at ICLR 2023
(a) Images generated with model trained with (b) Images generated with model trained with
Fmax = 0.025
mmax = 0.05
Figure 7: Ablation study for the magnitude of noise.
F ADDITIONAL RESULTS
F.1 SCHEDULING ABLATIONS
F.1.1 ABLATION STUDIES FOR NOISE
In the experiments of the main paper, our diffusion involves an initial stage where only noise is added. Then noise fixed and the images are getting corrupted by the deterministic operator (e.g- blur or masking). In this section, we show two ablations regarding the noise.
Magnitude of noise. In this ablation study, we still keep the noise fixed for a significant part the diffusion, but ablate the magnitude of the noise. Specifically, we attempt study to what extent noise is needed in order to learn to reverse corruption processes with Soft Score Matching, We train two additional models CelebA-64 where attempt decrease the maximum noise lower value. The corruptions for both models involves an initial stage where noise grows geometrically rate from the initial value (0.01) the maximum value. Then, all the models keep the noise fixed their maximum value for the rest of the diffusion. We use the following maximum values: i) o maax = 0.1 (model used in the paper), ii) max 0.05, and iii) m 0.025. Unconditional samples from the two ablation models are shown in Figure 7 As shown, both models fail to produce realistic samples. The quality of samples deteriorates significantly the ablation models are significantly worse than the ones produced by our state-of-the-art model (see Figure 2 (right)). We want to underline that this not conclusive study. It might be the case that with different hyperparameters one can make Soft Score Matching work with lower values of noise. For example, we might need to tune the weights w(() since for the ablations (and the state-of-the-an model), we use wtt) 1/oî (as in Song & Ernon 2019 2020); Song et al. (2021b)) which might be causing instabilities for low values of noise (Nichol & Dhariwal 2021).
the noise decreases the samples from
Noise changing throughout the diffusion. We also train model where noise and blur are chang- ing simultaneously throughout the diffusion. This is a sanity check to verify that our framework (learning and sampling) still works when the model needs deblur and denoise the same time. For the noise scheduling, we simply use geometric scheduling from 0.1 to 0.01. We keep the blur parameters the same with the state-of-the-arr model. i.c. we use the blur parameters shown in Fig- ure (6) This model achieves competitive FID score, 4.31. This score could probably be further improved (by jointly selecting the blur and the noise scheduling with our framework), but this beyond the scope of this ablation.
F.1 ABLATION STUDY FOR BL.R
For all our experiments so far, we chose the blur corruption levels based on the scheduling method we described in Section 3.3 We show the benefits of our approach by comparing to natural base- line for selecting the diffusion levels. For this natural baseline, we use the scheduling of Variance Exploding (VE) as guidance. Specifically, we choose the blur parameters such that the MSE between the corrupted image and the clean image decays with the same rate for the Gaussian Denoising Dif- fusion and our Soft Diffusion (blur and low magnitude noise). Formally, let {q/ He be the (noisy) distributions used in Song et al. (2021b) for the Variance Exploding (VE) SDE and let { H-u the blurry (and noisy) distributions we want select. At level choose the blur parameters such
19

---

Under review as a conference paper at ICLR 2023
that:
E ans ( (an, (ar (o (1na)
2t xx||²]]
[ ax, (ar (000))))
2o 2;|l?]
(55)
Ex Jouen (#1 ao go (20))
21
(r (2) æe (0))
21 21113]
We retrain CelebA using this natural baseline method for selecting the diffusion parameters. For a fair comparison, we keep the architecture and all the hyperparameters the same and we only ablate the scheduling of the blur. We measure FID for both the trained model with the baseline scheduling and we observe it increases from 1.85 to 8.35. Apart from this large deterioration in performance, the baseline model obtains its best FID score after 2000 steps, while with our scheduling we only need 280 steps to obtain the best FID. This experiment shows that the choice of scheduling is really important for the model performance but for also the computational requirements of the sampling.
F.2 CIFAR-10. 10: ADDITIONAL RESULTS
We report how FID is changing based the Number of Function Evaluations for CIFAR-10 with Momentum Sampling in Figure The baseline model we are comparing against is NCSN++ trained with VE SDE, since model also uses a VE SDE. For the baseline, sampler with Maruya-aau discretization (similar to our Momentum Sampler). The numbers taken directly from Ma et al. (2022).
the Reverse SDE
CIFARIO
nesn++
ours
16
RD
100 110
120
130
140
150
160
Number of Function Evaluations (NFEs)
Figure 8: Demonstration of how FID changes based on Number of Function Evaluations (NFEs) for CIFAR-10 for our (blur) model on CIFAR-10 with momentum sampling. Our model offers significant performance benefits for low number of function evaluations.
We also show visual samples from our blurring model, NCSN+ (VE SDE) and DDPM++ (VP SDE) in Figure 9 For all the models we fix the NFEs 200 and show samples generated with the Euler-Maruyama discretization of the associated reverse ODE. As shown, the baseline models generate samples with artifacts for low NFEs while our model leads images of superior visual quality. We note that different samplers can be used to accelerate sampling for all models, e.g. samplers from Karras al. (2022) or DDIM-type samplers (Song et al. (021a), as we show in Section B.2
We also report best FID performance for different samplers for ours and competing methods in Table 2. Our best FID is competitive with similar samplers to similar methods, e.g. with NCSN+ trained with VE SDE and using the Reverse SDE sampler. However, is significantly behind the state-of- the-art LSGM (Vahdat et al., 2021)) model. We believe that this performance gap can be decreased in the future by further research in the area of diffusion models with general corruptions.
F.3 SAMPLING ABLATIONS
We perform several ablations regarding the sampling method, additional the results mentioned in the paper.
DDIM We experiment with the deterministic DDIM-type sampler and report our results in the Table CelebA-64. Our DDIM-type sampler very effective when the number of function eval- uations is very low it achieves FID 5.08 with only 50 steps while in comparison, the Momentum
20

---

Under review as a conference paper at ICLR 2023
(a) Ours (VE SDE)
(h) NCSN+ (VE SDE)
(c) DDPM+ (VP SDE)
Figure 9: Visual comparison of samples from model and baselines for 200 NFEs. For all the models, we are using the Huler-Maruyama discretization of the associated reverse ODE to generate the samples. As shown, model leads to superior visual quality.
Method
FID
Ours (VE SDE) Naive Sampler
40.07
Ours (VE SDE) Momentum Sampler
3.91
Ours (VE SDE) Probability Flow Momentum Sampler
3.86
NCSN++ (VE SDE) Reverse SDE
4.79
NCSN+ (VE SDE) Probability Flow
10.54
DDPM (VP SDE)
3.17
LSGM (Vahdat et al. 2021
2.10
Table 2: FID Results on CIFAR-10 for different samplers. For each of our samplers, we report the best result obtained among 10 runs with different NFEs, ranging from 200 to 1000 with step size of 80. The results for competing methods are taken directly from the papers.
Sampler, achieves FID 30.31 with 50 steps. The efficiency of DDIM is consistent with what has been observed in (Song et al. 2021a Karras al., 2022). DDIM's performance is also getting worse as we increase the number of steps. Instead, the performance of the Momentum Sampler, as observed in Figure 3 has a U-shape: there an intermediate optimal number of steps achieve the best FID score. We believe this could be related to the fact that Momentum Sampler is not deterministic. Specifically, in Karras al. 2022) it is observed that for deterministic samplers per- formance usually flattens after point (e.g. see Figure 2, page 4) while for some stochastic samplers performance goes again up after a certain number of function evaluations (e.g. see Figure 4, page 8).
NFEs
FID
25
10.54
50
5.08
100
3.40
200
2.92
300
2.83
600
2.80
1000
2.79
Table 3: Results of DDIM (Song et al. 2021a) sampler for CelebA-64 dataset using our blurring model. The DDIM sampler is particularly effective for low number of function evaluations and maintains its performance as we increase the number of steps.
Probability Flow Momentum Sampler For CelebA-64 we reported results only for the (stochas- tic) Momentum Sampler in the main body of the paper (see Figure 3). We report here results for the deterministic counterpart of this sampler, Probability Flow Momentum Sampler, summarized in Table 4 The deterministic version of the Momentum Sampler seems to be performing slightly
21

---

Under review as a conference paper at ICLR 2023
worse than the stochastic version FID jumps from 1.85 to 2.14. However, this sampler, similar to the DDIM-type sampler, maintains more performance as we increase the number of steps. This is an important advantage for the deterministic sampling methods since practitioners need to put less effort into tuning the NFEs to get the best result.
NFE
FID
50
13.77
100
3.27
200
2.14
300
2.21
600
2.59
1000
2.49
Table 4: Results of the Probability Flow Momentum Sampler for Celeba-64 for our blurring models.
Predictor-Correcte Samplers Finally, we perform experiments with Predictor-Corrector sam- plers, as proposed in [Song et al. (2021b). The idea that we are alternating at each diffusion step between two different samplers. We experiment with DDIM Predictor and a Probability Flow Mo- mentum Sampler corrector. The results are summarized Table The Predictor- Cortectrrr sampler maintains
of the benefits of both samplers. Namely, for low number of function evaluations has better performance than the Probability Flow Momentum Sampler (benefit coming from the DDIM sampler) and for higher number of function evaluations performance is better than the DDIM sampler (benefit coming from the Probability Flow Momentum Sampler). There is a spot, at 100 NFBs, where the Predictor-Correctu sampler better than both the Predictor and the Corrector: We encourage future research in identifying even better pairs of Correctoss that might outperform both the Predictor and the Corrector in some regime.
NFE
FID
25
10.56
50
5.21
100
3.21
200
2.86
300
2.71
600
2.73
1000
2.73
Table 5: Results of a Predictor- (Song et al. 2021b) sampler for Celeba- 64 for our blurring
models. The Predictor the DDIM-type sampler and the corrector our Probability Flow Momentum Sampler.
F.4 NEAREST SAMPLES IN TRAINING DATA
To verify that model does simply memorize the training dataset, present generated images from our model and their nearest neighbor (L2 pixel distance) from the dataset. The results are shown in Figure 10
22

---

Under review as a conference paper at ICLR 2023
(a) Generated images.
(b) Nearest neighbors from dataset.
Figure 10: Generated images and nearest neighbors (1.2 pixel distance) from the training dataset. As shown, the model produces new samples and does not simply memorize the training dataset.
F.5 UNCURATED SAMPLES
Figures I1 and 12
show more uncurated samples from our trained models on CelebA and CIFAR-1O.
23

---

Under review as a conference paper at ICLR 2023
Figure 11: More uncurated samples from our blur model trained CelebA. These samples obtained using our Momentum Sampler with 280 NFEs.
24

---

Under review as a conference paper at ICLR 2023
Figure 12: More uncurated samples from our blur model trained CIFAR-10. These samples
obtained using the Probabilistic Flow Momentum Sampler with 200 NFEs.
25

---

Under review as a conference paper at ICLR 2023
Figure 13:
Uncurated samples from masking model trained on CelebA. These samples
obtained using the Probabilistic Flow Momentum Sampler with 1024 NFEs.
26

---

Under review as a conference paper at ICLR 2023
G IMPLEMENTATION OF DEGRADATION OPERATORS
Blurring. The blurring operator is implemented as a convolution with Gaussian kernel. The Gaussian kernel is truncated to have fixed support size (161 x 161 pixels), and normal- ized to have area one. The standard deviation of the Gaussian kernel sigma (param- eter controlling the strength of the blur) computed using the optimal transport optimization for the scheduling. The obtained schedule for the parameter s sigmabblus in CIFARIO can be accessed from the following anonymous url: attps:://rrivee google..goo......oooooooo....... www.olle..oo/..... com file/d/ 1 192kdo 19qq EGCm EGCm7KY520Z --005=502000000000-g---0
def hetprate__aaussaaaeeeeeeeeeeeeee (sigma blur, hali_size v ap.arange (-half_size, half_size 1)
x, y np (p.meshgriddv,
k = np. ep. (- (x**2 + y*+2)
blurx++))))
k np. sum
return k
Masking. The masking operator is implemented using centered binary mask that sets to zero some percentage of pixels in the image. The scheduling with the masking operators can be accessed from the following anonymous url: httpss///riveeeoogl.ee.. (12271/wwwwwaa007770000D1U1looo2
Noise. The noise degradation implemented by adding to the blurred or masked image white Gaussian noise having standard deviation sigma_noise. The scheduling of the noise be accessed from the following anonymous url: https://driss gom.ge.com/f..l..l.....
170wF TTO.://DDrraaaeeeeeWWWWeiiiiilar
27


Reviews:
[{"review_id": "nHfsnlJYn96", "reviewer": "Reviewer_x7zn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper extends the original diffusion models to linear corruptions, which include Gaussian blur and masking. Motivated by the SDE form of the backward process, the paper proposes the so-called \"momentum sampler\".\nThe noise/corruption scheduling are selected based on some Wasserstein matric. Empirically, they improve the FID score from previous SOTA (1.90) to 1.85 on CelebA 64x64.", "summary_of_the_review": "The delivery of the paper should be improved. The experimental sections should be improved. ", "strength_and_weaknesses": "\nStrength:\n\n(1) Build a framework for diffusion with general linear corruptions.\n\n(2) Corruption scheduling using the Wasserstein metric seems interesting.\n\n(3) Momentum sampler seems interesting.\n\n\n\nWeakness:\n\n(1) The authors should provide specific forms of $C_t$ for Gaussian blur and masking.\n\n(2) The scheduling of the diffusion is unclear. e.g., how do you estimate the Wasserstein distance? How is the noise level $s_t$ scheduled? (Figure 6 alone is not enough for replicating the result.) What is the tunable parameter in $C_t$? \n\n(3) Experiments: you should faithfully report your results on CIFAR10, and summarize them as a Table.  \n\n(4) To argue about the superiority of the sampling speed, the authors should early-stop the sampling process, e.g. at the 200 steps, and provide a visualization of examples. \n\n(5) About Figure 3: Why only compare your method with NCSN++? what about DDPM++, which is the previous SOTA as reported in Table 1?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "c91nIW9QRx", "reviewer": "Reviewer_dDmh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a diffusion-based generative model trained on reversing a noisy and corruption process. Using the idea is that denoising score matching also works with general corruptions, score matching for corruption is proposed. To use it in a generative model, a momentum sampler is proposed on top of the naive sampler (which seems to have inferior performance). Experiments results on CIFAR-10 and CelebA show promising results in generative modeling, specifically on CelebA 64x64.", "summary_of_the_review": "While learning diffusion models for blurring corruptions is interesting, the theorem is essentially rewriting the one in Vincent (2011), and the experiments are not comprehensive enough to demonstrate why one would favor this method than regular Gaussian diffusion. Even for a systematic study of soft diffusion alone, the work does not have a lot of CIFAR-10 results reported for CelebA, does not report NFE systematically (for many FID numbers except for the best ones, it is unclear how many NFE is used), does not have VP-SDE results, and does not explore ablation studies over the corruption process. Therefore, at the current stage, I am leaning towards rejection.", "strength_and_weaknesses": "*Strengths*\n\n- Proposed a diffusion process combined with corruptions, such as blurring. \n- A leap-frog like procedure is proposed for sampling, which addresses the issue of changing corruptions in the naive sampler.\n- Competitive FID results on generative modeling.\n- The authors are rather honest about the limitations of the method, as detailed in Appendix E.\n\n*Weaknesses*\n\n**Reverse SDE might not recover desired distribution** \n\nThe reverse SDE is based on an ad-hoc modification to the Gaussian diffusion SDE (specialized to the case where data distribution is delta), which makes it hard to justify why this would recover the desired data distribution (e.g., with Fokker-Planck). Wouldn't annealed Langevin dynamics / predictor-corrector be more principled?\n\n**Experiments are limited**\n\nAll the experiments are done in CIFAR-10 and CelebA-64. While that is not anything to blame alone, it is difficult to generalize the conclusions to higher resolutions or even another dataset. One of the reasons is in Figure 3, it shows that the claimed SOTA FID is obtained by heavily tuning the number of NFEs, which is quite different from general findings that sample quality improves with more sample steps in standard diffusion models. Is this finding true for CIFAR-10 as well? \n\nAlso, some results seem to be missing. \n- The Probability Flow ODE sampling with momentum is only listed for CIFAR-10 and not CelebA. \n- CIFAR10 results should also have a table, with results on naive sampler as well.\n- It does not seem unreasonable to train a model on VP-SDE / Elucidated DM given their superiority than VE.\n\nThe standard Gaussian diffusion models is a high bar to overcome, and it is unclear whether the experimental results would convince practitioners to switch to methods based on blurring. There does not seem to be advantages regarding architecture design, or sampling speed, or training speed. The most promising direction is to use these corruption models in inverse problems, as the authors suggested in Section 6, but there is no empirical evidence on this either. \n\n**Novelty of Theorem 3.1**\n\nAnything new about the proof of Theorem 3.1 that is not from Vincent (2011)? It appears all the tricks are the same, and $C_t$ does not even show up. The only change is $q_\\sigma(\\tilde{x} | x)$ (in Vincent's work that is a noise model) to $q(x_t | x)$ (that is the corrupted noise model), but the set of equations to prove the statements is basically the same.\n\n**Inflated claims**\n\nIt is said that \"Our method is superior in sampling time, for both CIFAR-10 and CelebA.\". First, it is not mentioned the superior is against which method (likely VE-SDE, but that is probably the worst baseline for comparing sampling time). Second, the claim is not backed up for CIFAR-10 in the main paper (even in Figure 8, it is only shown for NFE within 100 and 170); only CelebA results are shown. \n\n\n------\n\nThese are not weaknesses, but simple out of curiosity:\n\n- I would recommend an ablation study on the effect on blurring degrees to the generative model, since you can interpolate between the blur kernel that is being used and doing no corruption at all. This could substantially improve the arguments on section 3.3, and indeed show its promise in generative modeling. \n\n- The DDIM paper describes a series of solvers for the probability flow ODE and various SDEs (with different stochasticity in the sampling process). Is it possible to adopt that into this framework?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GSoQblXtVk", "reviewer": "Reviewer_pNiX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work explores general linear corruption processes for diffusion models and practical training and sampling to improve performance for the new diffusion models. However, there are several technical questions unclear.", "summary_of_the_review": "In summary I think the proposed method is interesting and technically sound. But the benefits over previous methods are not clear both in theory and practice. However, the benefits over previous methods are not clear both in theory and practice. I would like to increase my score if the authors can address my questions.", "strength_and_weaknesses": "Strength:\n\n1. Authors show denoising score matching~(DSM) can be used to learn score for noised diffusion with general linear corruption processes theoretically. \n\n2. With the proposed several training and sampling improvements, diffusion models with general linear corruption can achieve competitive sampling quality on CIFAR10 and CelebA. \n\nWeaknesses:\n\n**general corruption**\n\n1. Theoretically, extending diffusion models from the isotropic corruption process to general non-isotropic corruption is quite straightforward based on original DSM work or recent.\n\n2. Regarding performance, the claimed SOTA FID on CelebA 64 is not accurate,  where [3,4]  have lower FID. \n\n3. The performance of the proposed diffusion model in popular CIFAR-10 is worse than VPSDE or its variants. Authors compare with a weak baseline VESDE with reverse diffusion and ODE solver, whose performance is much better than the suggested Predictor-Corrector sampler. Authors claim \"it is known[5] that diffusion models with VP SDEs usually do better in CIFAR-10\". However, [5] suggests their VESDE can beat previous VPSDE and VESDE, and different diffusion models with isotropic corruption can be transformed into VESDE. The performance difference is due to various pre-condition and noise schedules during training. \n\n4. To the best of the reviewer's knowledge, the SDE of the general corruption, which has a general linear shift term, can also be transformed into VESDE with a similar technique in [5] or exponential transformation introduced in [6].\n\n**Momentum Sampler**\n\n1. The idea is quite interesting. It will be nice to compare with SOTA sampler in diffusion models, such as ODE solvers introduced in [5,6]. \n\n**Scheduling**\n\n1. The idea that choosing scheduling based on minimal distance from D0 to D1 is new. However, there is no theoretical evidence to justify why minimizing distance with the selected w2 distance is better. It is hard to predict the benefits of the choice or show the choice can minimize discretization error when solving sampling SDE or ODE. \n\n2. Besides, estimating Wasserstein distances accurately between two datasets is quite expensive, even in CIFAR10 32x32 datasets. It would be nice to include more details, such as whether authors use partial datasets or some approximations in solving optimal transportation problems in the image dataset scale.\n\n\n\n[3] Kim et al. Maximum Likelihood Training of Implicit Nonlinear Diffusion Model\n[4] Wang et al. Diffusion-GAN: Training GANs with Diffusion\n[5] Karras et al. Elucidating the Design Space of Diffusion-Based Generative Models\n[6] Zhang et al. Fast Sampling of Diffusion Models with Exponential Integrator", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject