Soft Diffusion: Score Matching with General Corruptions
Authors: Giannis Daras
Mauricio Delbracio
Hossein Talebi
Alex Dimakis
Peyman Milanfar
Venue: Accepted by TMLR ()

Abstract: We define a broader family of corruption processes that generalizes previously known diffusion models. To reverse these general diffusions, we propose a new objective called Soft Score Matching. Soft Score Matching incorporates the degradation process in the network and provably learns the score function for any linear corruption process. Our new loss trains the model to predict a clean image, that after corruption, matches the diffused observation. This objective learns the gradient of the likelihood under suitable regularity conditions for the family of linear corruption processes. We further develop an algorithm to select the corruption levels for general diffusion processes and a novel sampling method that we call Momentum Sampler. We show experimentally that our framework works for general linear corruption processes, such as Gaussian blur and masking. Our method outperforms all linear diffusion models on CelebA-64 achieving FID score 1.85. We also show computational benefits compared to vanilla denoising diffusion.

Published in Transactions on Machine Learning Research (05/2023)
Soft Diffusion:
Score Matching with General Corruptions
Giannis Daras UT Austin"
gionnisdornss@utees..el
Mauricio Delbracio
mdelbra&@googlee..
Google Research
Hossein Talebi
htalebi com
Google Research
Alexandros G. Dimakis UT Austin
dimakis@oustin wtwzass edu
Peyman Milanfar
milanfar@google.com com
Google Research
Reviewed on OpenReview: https: // net/ forum? id= W98rebBalQ
Abstract
We define a broader family of corruption processes that generalizes previously known dif- fusson models. To reverse these general diffusions, we propose a new objective called Soft Score Matching. Soft Score Matching incorporates the degradation process in the network and provably learns the score function for any linear corruption process. Our new loss trains the model to predict a clean image, that after corruption, matches the diffused observation. This objective learns the gradient of the likelihood under suitable regularity conditions for the family of linear corruption processes. We further develop an algorithm to select the corruption levels for general diffusion processes and novel sampling method that we call Momentum Sampler. We show experimentally that our framework works for general lincar corruption processes, such as Gaussian blur and masking. Our method outperforms all lin- ear diffusion models on CelebA-64 achieving FID score 1.85. We also show computational benefits compared vanilla denoising diffusion.
1 Introduction
Score-based models (Song & Ermon 2019 2020 Song et al. 2021b) and Denoising Diffusion Probabilistic Models (DDPMs) (Sohl- Bich-Diikstein et al.] 2015 Ho et al.) 2020 Song al.] (021a) are two powerful classes
of generative models that produce samples by inverting dilfusion process. These classes have been unified under single framework (Song et al. 2021b and are widely known diffusion models. Diffusion
modeling has found great success in wide range of applications (Croitoru et al. 2022 Yang et al. 2022
including image (Saharia et al.) 2022a Ramesh et al. 2022 Rombach et al. 2022
Dhariwal & Nichol
2021
audio (Kng et al. 2021 Richter et al. 2022 Serrà et al. 2022), video generation
Ho et al.
2022b) as well
as solving inverse problems (Daras et al. 2022: Kadkhodaie & Simoncelli
2021
Kawar et al.
2022
2021
Jalal al. 2021 Arvinte al. 2022 Saharia
2022b Laumont et al.
2022
Whang et al.] 2022 Chung
al. 2022b Ruiz et al. 2022
Daras & Dimakis 2022 Gal et al. 2022 Ren al. 2022
2022 analyze the design space of diffusion models. The authors identify three stages: i) the noise scheduling, ti) the network parametrization (each one leads to a different loss function), iii) the
Karras et al.
*The work done during an internship Google.

---

Published in Transactions on Machine Learning Research (05/2023)
Blur Noise Forward Process
Masking Noise Forward Process
Reverse Process
Reverse Process
Uncurated Samples (Blur+Noise)
Figure 1: Top two rows: Demonstration of our generalized diffusion method. Instead of corrupting by only adding noise, we propose framework to provably learn the score function reverse any linear diffusion (left: blur and noise, right: masking and noise). Our (blur and noise) models achieve slate-of-tho-ar FID score 1.85 for linear diffusson models on CelebA-64. Uncurated samples shown the last three rows.
sampling algorithm. We argue that there is one more important step: choosing how to corrupt. Typically, the diffusion additive noise of different magnitudes (and sometimes input rescalings). There have been a
few recent attempts to use different corruptions (Deasy et al. 2021 Hoogeboom al.] 2022a b Avrahami
et al. 2022 Nachmani et al. 2021 Johnson et al. 2021 Lee et al.
2022 Ye et al., 2022 but the results
are usually inferior to diffusion with additive noisc. Also, a common framework on how to properly design general corruption processes is missing.
We present such a principled framework for learning to invert a general class corruption processes. We propose a new objective called Soft Score Matching that provably learns the score for any regular lincar corruption process. Soft Score Matching incorporates the filtering process in the network and trains the model to predict clean image that after corruption matches the diffused observation.
Soft Score Matching learns the score (i.c. likelihood gradicnts) for corruption processes that satisfy a regu- larity condition that we identify: the diffusion must transform any image into any other image with nonzero likelihood. Using our method and Gaussian Blur paired with little noise as the diffusson mechanism, we achieve state-of-the-art FID on CelebA (FID 1.85) for linear diffusion models. We also show that our cor- ruption process leads to generative models that are faster compared vanilla Gaussian denoising diffusion.
Our contributions:
We propose learning objective that: i) provably learns the score for wide family of regular diffusion processes and ii) enables learning under limited randomness in the diffusion.
We present principled way to select the intermediate distributions. Our method minimizes the Wasserstein distance along the path from the initial to the final distribution.
2

---

Published in Transactions on Machine Learning Research (05/2023)
We propose novel sampling method that call Momentum Sampler: sampler uses convex combination of corruptions at. different. diffusion levels and is inspired by momentum methods in optimization.
We train models on CelebA and CIFAR-10. Our trained models on CelebA achieve a new state-of- the-art FID score of 1.85 for lincar diffusion models while being faster compared to models trained with vanilla Gaussian denoising diffusion.
2
Background
Diffusion models are generative models that produce samples by inverting corruption process. The cor- ruption level is typically indexed by time t, with t = 0 corresponding to clean and t = 1 to fully corrupted images. The diffusion process can be discrete or continuous. The two general classes of diffusion models are Score-Based Models (Song & Ermon 2019 2020 Song et al. 2021b) and Denoising Diffusion Probabilistic
Models (DDPMs) (Sohl-Dickstein et a.. 2015 Ho et al. 2020
The typical diffusion in score-based modeling is additive noise of increasing magnitude. The perturbation kernel at time is: (e()))(zo) Maaa æo; E = of I), where ano qo is clean image. Score models are trained with the Denoising Score Matching (DSM) objective:
min Jo..j, [e ) (ar)))
(m. (
[150(
Va, log q (1x
(1)
where wa scales differently the weights of the inner objectives. If we train for each noise level t indepen- dently, given enough data and model capacity, the network is guaranteed to recover the gradicnt of the log likelihood (Vincent 2011 Hyvärinen & Dayan 2005 known as the score function. In other words, the model so(2;; [t) is trained such that: so(x)))) () 2 Va, log (xx)) In practice, we use parameter sharing and conditioning on time to learn all the scores. Once the model is trained, we start from a sample of the final
distribution, Q1, and then use the learned score to gradually denoise (Song & Ermon 2019 2020 The
final variance of is selected to be very large such that the distribution gi is approximately Gaussian, i.e. the signal noise ratio tends to O.
DDPMs corrupt by rescaling the input images and by adding noise. The corruption can be modelled with Markov chain with perturbation kernel (=( (ax v1 - Bi E 3,T). Typically, 31 = and hence 91 = I). DDPMs are also trained with the DSM objective which is derived by minimizing an evidence lower bound (ELBO) (Ho et al. 2020).
In their seminal work, Song et al. (2021b) observe that the diffusions of both Basee models and DDPMs can be expressed as solutions of Stochastic Differential Equations (SDEs) of the form:
dæ = fax, t)dt + g()ddw,
(2)
where w is the standard Wiener process. Particularly, Score- models use: f(z,t)=
V do?
dt and DDPMs use: f(x, 1) = B, s()) vB. As explained earlier, for Score-Based models we need large noise at the end for the final distribution to approximate a normal distribution. Hence, the corresponding SDE is named Variance Exploding (VE) SDE (Song et al., 2021b). DDPMs usually have a final distribution of unit variance and hence their SDE is known as the Variance Preserving (VP) SDE (Song el. al. 2021b). Song et al. (2021b) also propose another SDE with bounded variance, the sub -uDDE,, that experimentally yields better likelihoods.
For both Score-Based models and DDPMs, Equation 2 known as the Forward SDE. This SDE is re- versible Anderson (9822) and the Reverse SDE is given below:
dæ [f( 9" (1) log q (x)] dt g(()dw
(3)
where is the reverse time standard Wiener process. Typically, Va log q (x) is approximated by sg(2) and samples are generated by solving the Reverse SDE (Sogg et al. 2021b)
3

---

Published in Transactions on Machine Learning Research (05/2023)
3 Method
Our framework for training diffusion models with more general corruptions includes three components: i) the training objective, ii) the sampling, iii) the scheduling of the corruption mechanism.
3.1
Training Objective
We study corruption processes of the form:
3: C, 8t77:
(4)
where C₁ : R" + I2" is deterministic linear operator, T): is Wiener process, and s, is non-negative scalar controlling the noise level at time i, and æo go(x). We further denote with of the variance of the noise at level and we assume that it is non non-decreasing function of t. Unless stated otherwise, we assume that time is continuous and runs from f = 0 to f = 1. Additionally, we assume that at = 0, we have Co = Laxn and og = O, Le. = 0 corresponds to natural images. We also assume that recovering 26 from 2; harder as gets larger (i.e., entropy of ge (e(2 ae;) increases with 1). Eq. defines a general class of diffusion processes, that includes (as special cases) the VE, VP and subVP SDEs used in (Snng et al. (2021b). Our diffusion is the sum of deterministic linear corruption of 2n and stochastic part that progressively adds noise. For any corruption process of this family, we are interested in learning the scores, i.e. Va, log (:(x)) for all t.
Crucially, the corruption processes of interest cannot be described by linear Ito SDEs as in the seminal work of [Song et al. (2021b). For any f(x, = A,æ, A + Iaxn, the marginal distributions, pr( (ma); are Gaussians with correlated noise. In our corruption model, we are interested in linear corruptions with uncorrelated noise. By decoupling the degradation and the noise at any time step, we can have better control of the forward process. To study such corruption processes, new framework is needed.
For the vanilla Gaussian denoising diffusion, the celebrated result of Vincent (2011 shows that only need access to the gradient of the conditional log-likelihood, Va, log da (xx mo), in order to learn the score, Va, log ge (.)). In fact, this result holds for more general corruption families, as long as some mild technical conditions are satisfied.
Theorem 3.1 (Vincent, (2011))- Let yo be two distributions in RR Assume that all conditional distribu- tions, qe (x (o)) are fully supported and differentiable in TR. Let:
Ex Exzra [115o(20.)- - Va, log ge (mx)))
(0)
(5)
2
J2( - 1E (20 go (mm [1100(2))) Vx log (20
(6)
Then, there a universal constant. (that does not depend on A) such that: ( (0) + +C
Theorem 3.1 implies that minimizing the second function is equivalent to minimizing the first one. The second function is nothing else than the DSM objective. necessary condition to maintain the score- matching guarantees of the DSM objective is that the corruption process gives non-zero probability to all e, for any image Eg. This is easily achieved by adding Gaussian noise, but this is not the only option. We include the proof of this Theorem in the Appendix for completeness.
Network parametrization. For the class of diffusion processes given by Eq. have that: qe (2x/200) N (z = C, == = I) and hence the objective becomes:
1
Ciay x; of
L(t)= (0))=:) (1 ma) (o(())t)
As shown, the objective of the model is to predict the (normalized) difference, C,æy Dt, which actually the noise, n.- We argue that even though this objective theoretically grounded, in many cases, would
4

---

Published in Transactions on Machine Learning Research (05/2023)
not work in practice because we would need infinite samples to actually learn the vector-field Va, log)( in way that would allow sampling.
Assume that the corruption process is blurring (at different levels) paired with additive noise of small magnitude. The objective written in Eq. 7 learns the distributions of blurry (and slightly noisy images) by just removing noise. Hence, in practice we might only learn these distributions locally (around the blurry images) and hence we might not be able reduce the blurriness. This point might be better understood after present our Sampling Method in Section 3.2
To account for this problem, we propose a network reparametrization which leverages that we know the linear corruption mechanism, C. Specifically, we propose the following parametrization:
(x(2)(t) Cetr((ai It
(8)
o
Crucially, the network incorporates the corruption process.
The loss becomes:
L(t)= E (000 o
[IC((ho(z)lt) -xw)l²]
(9)
When C. is a blurring matrix, this loss function is the MSE between the blurred prediction of he and the blurred clean image. Finally, observed in previous works (Song & Ermon 2019 Ho et al. 2020 Karras et al. 2022 the optimization landscape becomes smoother when we predicting the residual, instead of the clean image directly. This corresponds to the additional roparamctrization:
ho( (t) cbu (a:e()) + st.
(10)
which leads to the final form of our loss function:
I(t)= E (cco of [[ (00(22)1)) -
(11)
where r, is the residual with respect to the clean image, i.c. ri = Fo - xt, Following prior work, we use a single network conditioned on time that optimized for all L(t). Hence, the total loss is:
L= (xme((o.111(()) Beeeunee)) 20) nig (o (x. (20))0)) (zo)
[IC: (sba(xe)() (m)))))))))))]) -
(12)
where the weights are usually chosen to be 1 or 1/0, Karras
2022 Kingma et al. 2021
We call our training objective Soft Score Matching. The name is inspired from "soft filtering" term used in photography to denote image filter that removes fine details (e.g., blur, fading, etc). As in the Denoising Score Matching, the network is essentially trained to predict the residual to the clean image, but in our case, the loss is in the filtered space. When there is no filtering matrix, i.e. C, I, we recover the DSM objective used in (Song & Ermon 2019
2020 Song et al. 2021b).
Comparison with objectives used in other works. Previous (Anonymous) 2022) or concurrent (Bansal
et al. 2022 Rissanen et al., 2022 Hoogeboom & Salimans 2022) works that consider different degradations the heuristic objective of predicting the clean image, i.e. they minimize: | - This is actually
than Gaussian Diffusion,
our loss, i.e. ||C, (oo( rx) || < [|Calll(@o(za(t)) rell. Since the spectral norm, ||Cr]l is fixed, one can optimize for the upper-bound by minimizing || (zalt) rell- Instead, Soft Score Matching optimizes directly for learning the score. Experimentally, Soft Score Matching outperforms (ours FID: 1.85, theirs: 5.91), under the exact same setting, this simple baseline (see Experiments).
upper-bound
3.2 Sampling
5

---

Published in Transactions on Machine Learning Research (05/2023)
Algorithm 1 Naive Sampler
Naive Sampler. Once the model is trained, need a way to generate samples. The simplest idea (that does not come with any guarantees of sampling from the right distribution) is to use recursively our model, (o( to get estimates the clean image, Po- To move from corruption level to corruption level At, we feed the image at the model to get an estimate of the clean image, do, and then corrupt back to level At. This idea is shown in Algorithm 1
Require: Pl, 4o, Ct, At
21 (1)) for t= to 0 with step At do du
7) ~
J:- +- Ct-Ac 01-207
end for return
Momentum Sampler, As we will show in the Experiments
section, the naive sampler presented above leads to generated images that lack diversity. We propose an simple, yet novel, alternative method for sampling from the general linear diffusion model presented in Eq. Our method is inspired by the continuous formulation of diffusion models that is introduced in Song et al. 2021bb.
The first step is to find a Markovian stochastic process that has marginal distributions that are "close" the ones defined in Eq. Consider the following SDE:
dæ. = C.E(zol + d(a?) dw, dt
(13)
where w is the standard Wiener process. This is a special case of the Itô SDE: dæ = f(x, t)dt + g(t) that appears in Song et al. (2021b), ie. f(x, t) C,E[a,]zz] [z,] and g(t) d(a?) Crucially, E| does not depend previous values of æ and our SDE is indeed an Ito SDE.
To build some intuition, it is useful to think of the toy setting where the dataset contains one image, a E IR". Under this setting, the Markovian corruption described by the SDE of Eq. 13 has the same marginal distributions as the process of Eq. As we explain in the Appendix (Section E in the general case, the SDE 1 introduces an approximation error with respect to the corruption process of Eq. the former uses the conditional expectation, |[ while the latter corrupts directly 20- Nevertheless,, the approximation error for the considered corruptions seemss to be small, given the experimental success of the derived sampler. Intuitively, this is because for low corruption levels, the conditional expectation and æo are close, while for high corruption levels, the distance is contracted by the multiplication with the corruption matrix C, (see also Appendix, Section Eb. Similar approximations have been made in the literature of solving linear inverse problems with diffusion models, e.g. we refer the interested reader to the work Chung et al.) (0022a). We use the notation ie to explicitly show that there a drift between this corruption process and the one of Equation
Eq. 13 describes reversible diffusion process
Anderson, 1982
The reverse is also Ito SDE:
dē: = C.Eoool d(o a., log 2 (w:) di + V
d(o?) dt
duc.
(14)
dt
where is a standard Wiener process when time flows backwards from ( = 1 to = 0. In practice, to solve Eq. [4] we discretize the SDE (i.e., apply Euler Mally-Maruyama, and approximate the function derivatives with finite differences). The Euler-Maruyama discretization is given below:
8. -. -C -C,-AA-C.,)Eloooo C.)oo(o (oi - V (0))) + v of - o
(15)
where 77 ~ N(O, In this update Equation, there are two unknowns: the conditional expectation, EF [[mo@a.)) and the score function, Va, (Z,)) We show that these are actually connected, through the Stein 1981) formula. Specifically, it holds that:
Tweedie's [Efron 2011 Robbins 1956
C,E|oo/ + of x. log qe (xx..
(16)

---

Published in Transactions on Machine Learning Research (05/2023)
The proof is given for completeness in the Appendix, Lemma model that provably learns the score according to Theorem 3.1 Putting everything together:
A.2
To estimate Væ, log ( use
A It
-1-A1
Zt
(17)
oi - (c,
(C,-at Ci) (xx + Zz
@1( +1+ -it) + Vooo o-
30
Our sampler is summarized in Algorithm 2 Essentially, there is one update for deblurring and one for denoising. At the core of this update equation, is the prediction of the clean image, Lg. Once the clean image is predicted, we blur it back to two different corruption levels, t and At. The deblurring gradient is the residual between the blurred images at levels t At and t. Interestingly, the denoising update is the
A Algorithm Momentum Sampler
Require: pl, Os. Ct, ds, At
#1 ~ (xr)
for = to 0 with step - At do
t (0(xx))) +
Coarse prediction of the clean image.
ÿt + Ciăo
> Coarse prediction of filtered image at t.
n 1)0
é, + ÿ: Ft
D. Estimate of noise al. L. b. Filtered image at with noise at At. Coarse prediction of filtered image At. Filtered image at At with noise at At.
(0 10-00-00 )
Et Vot - oi
2t < Et
Ú1-11 + Ci-si0
T1-A1 + 21 + (i1-21- -9:) 9:)
end for return 22
same the one used in typical score-based models (that only use additive noise). In fact, if there is no blur (C, I), our sampler becomes exactly the sampler used for the Variance Exploding (VE) SDE in Song et al. (001bb)
We call our sampler Momentum Sampler because we can think of as a generalization of the update of the Naive Sampler, where there momentum term. To understand this better, we look at the setting where there no noise. Then, the update rule of the Momentum Sampler is:
Ar, = C,-A: Cião.
(18)
As seen, the first term is what the Naive Sampler would use to update the image at level t and the second term what the Naive Sampler would use to update the image level -- At. If these two directions are aligned, then the gradient Az, is small. Hence, there is a notion of momentum, analogous how the term is used in classical optimization. In the Appendix, Section B.2 also present DDIM-type (Song et al. 2021a sampler for which the momentum term also appears.
Probability Flow Momentum Sampler. The update rule of sampler was derived using the dis- crelization of the backward SDE of our corruption, given in Eq, 14. Similarly to Song et al. (2021b), can also consider the Ordinary Differential Equation (ODE) associated with this SDE:
1 d( Ve, log ((.)) dt.
dē; C,E/ [.,]
(19)
2 dt
We can approximate C,Eoo]@.], Va, lo (ă:) with our trained network and get a deterministic version of the Momentum Sampler, which we call Probability Flow Momentum Sampler, as [Song et al. (2021b). We
detail our derivations in the Appendix, Section B.I
3.3 Scheduling
The last piece of our framework is how to choose the corruption levels, i.c. the scheduling of the diffusion. For example, for Gaussian Blur, the scheduling decides how much time spent in the diffusion in the very

---

Published in Transactions on Machine Learning Research (05/2023)
blurry, somewhat blurry and almost no blurry regimes. We provide an algorithm for choosing the corruption levels for arbitrary corruption processes.
Let Do the distribution of real images and Dj be a known distribution that we know how to sample from, e.g. a Normal Distribution. In the design phase of score-based modeling, the goal is to choose a set of intermediate distributions, (Dr); that smoothly transform images from Do to samples from the distribution (2,,., ... "..... be the space of diffusion parameters, i.e. each wi corresponds to a distribution Dur. In the case of blur for example, wi controls how much we blur the image. Let also M : X' X + R be metric that measures distances between distributions, e.g. M might be the Wasserstein Distance of distributions with support X.
Di. Let Q = (
We construct a weighted graph G. with the nodes being the distributions and the weights given by:
w (Du, Dws) = { M (Du,, Dw,), if M (Du., Dug) se 00, otherwise.
(20)
For a lixed 6, we choose the distributions that minimize the cost of the poth belween Do and Dj. The parameter e expresses the power of the best neural network we can train reverse one step of the diffusion. If = oo, then for any metric M, the shortest path is to go directly from Do to Dr. However, it is impossible to denoise in one step super noisy image. Hence, we need to go through many intermediate steps, which is forced by setting e to a smaller value. As we increase the number of the candidate distributions we are getting closer to finding the geodesic between Do and DJ. However, the computational cost of the method increases since we need to estimate all the pairwise distances M(Dw,, Dw,). In practice, we use a relatively small number of candidate distributions, e.g. T = 256 and once the path is found, we do linear interpolation to extend to the continuous case.
We note that the connection of diffusion models and optimal transport problems has been previously ex- plored in Kwon et al. (2022) where the authors show that minimizing the DSM objective also minimizes the Wasserstein distance between the data and the model distribution. Inspired by this connection, we propose to change the scheduling to make the optimization problem easier.
4 Experiments
We evaluate our method in CelebA-64 and CIFAR-10. We show that by just changing the corruption mechanism (and using our framework for scheduling, learning and sampling) we improve the FID score and reduce the sampling time. We use the architocture and the training hyperparameters from Song et al. (2021b) (full details can be found in the Appendix).
For most of our experiments, we use Gaussian Blur our primary corruption mechanism. To illustrate that our method works more generally, we also show results with masking which is discussed separately later. Consistent with the description of our method, our deterministic corruptions are also paired with additive low magnitude noise. This is required by our thcoretical results, otherwise the conditional log-likelihood, log q( (r)) would be undefined. We also find the addition of noise beneficial in practice (see Appendix 1. for ablation studies on the role of noise).
Scheduling. We use the standard geometric scheduling for the noise (Song & Ermon 2019 2020 Song et al. 2021b) and use the methodology described Scction 3.3 to select the blur levels. We use the (regularized) 12- Wasserstein distance as the metric M to measure how close are the different distributions. To clearly illustrate the switch to different corruption, our diffusion has an initial stage that increases the noise (with no blur) and then we fix the noise (to small value, e.g. = 0.1) and change (using our scheduling framework) the amount of blur. Our diffusion spends less than 20% of the total time in the initial stage that increases the noise. We ablate those choices in Section of the Appendix.
One important property of our framework is that the scheduling is dataset specific. Intuitively, the way we corrupt depends on the nature of the data we are modelling. The interested reader can find the found schedulings for each dataset in Figure 3 of the Appendix.
8

---

Published in Transactions on Machine Learning Research (05/2023)
Results. We train our networks CelebA-64 and CIFAR-10 using the found schedulings and the training
objective of Eq. 12 We start by showing uncurated samples of our models in Figure 2 The generated images
Figure 2: Uncurated samples from our trained models on CIFAR-10 (left) and CelebA (right).
have high diversity and fidelity in both datasets. We compare the FID (Heusel et al. 2017 obtained by our method on CelebA with many natural baselines that use any of the VE, VP or subVP SDEs. Specifically, we compare against DDPM (Ho et al., 2020) that uses the VP SDE, DDIM (Song et al.
2021a that uses the same model but with a different sampler, + (Kim et al. 2022b) that is the statc-of-the-art model for VP-SDE and the NCSN++ models (Song et al., 2021b) trained with the VE and subVP-SDEs. For a fair comparison, we only use reported numbers in published papers for the baselines and we do not rerun them ourselves. Our model outperforms all other linear diffuusion models in CelebA, achicving FID score 1.85. The results summarized in Table | For CIFAR-10, we obtain FID score 3.86 with our Probability Flow Momentum Sampler and 3.91 with our Momentum Sampler. We summarize the results in Table 2 Our best FID is competitive with similar samplers to similar methods, e.g. with NCSN++ (VE SDE) with Reverse SDE sampling. However, it is behind other state-of-the-art models such as LSGM (Vahdat et al. 2021 and DDPM (Ho et al., 2020). We believe that this performance gap can be decreased in the future by further research the area of diffusion models with general corruptions.
Our method is also superior in sampling time, for both CIFAR-10 and CelebA. Figure shows how FID changes based on the Number of Function Evaluations (NFEs). Our method requires significant less steps to achieve the same or better quality than NCSN++ (VE SDE) (Song et al. 2021b), using the same architecture and training hyperparameters (FID values taken from (Ma et al. 2022 ). We note that our method is not explicitly designed to reduce sampling time, this only a byproduct of switching to a different corruption.
We refer the reader to the Related Work section for a non-exhaustive list of methods to further reduce sampling time that could be used in conjunction with Soft Diffusion.
Model
FID
FCSh
DDPM (VP SDE) (o et 2020
3.26
128
DDIM (VP SDE) (Song et al.
2021 la
3.51
DDPM++ (VP SDE) Kim et al 2022b
1.90
0- 32
NCSN++ (subVP-SDE Song
b021b
3.95
NCSN++ (VE SDE) (Song et al.
20215
3.25
Ours (VE SDE + Blur)
1.85
1000
Number of Functionn Evaluations (NFEs)
Table 1: FID results on CelebA-61.
Figure 3: FID versus NFEs (CelebA-64).
A Ablation Studies for Sampling. For all the results we presented so far, we used the Momentam Sampler that we introduced in Section 3.2 and Algorithm 2 In this section, we ablate the choice of the sampler. We show that the choice
Specifically, we compare with the intuitive Naive Sampler described in Algorithm [ of the sampler has a dramatic effect in the quality and the diversity of the generated images. Results are shown in Figure 0 The images from the Naive Sampler seem repetitive and lack details. This is reflected in
9

---

Published in Transactions on Machine Learning Research (05/2023)
the poor FID score. The Momentum Sampler leads to images with greater variety and detail, dramatically improving FID from 27.82 to 1.85.
Figure 5: Momentum Sampler (uncurated). 1.85.
FID:
Figure 4:
Naive Sampler (uncuratod). FID: 27.82.
Figure 6: Effect of sampling method on the quality of the samples. Naive Sampler gives repetitive images that lack details. Momentum Sampler dramatically improves the sampling quality and the FID score.
In the Appendix, we further compare our Momentum Sampler with DDIM-type sampler (Song et al., 2021a) and Predictor-Correcttor (Song et al. 2021b) samplers (see Sections B.2 are that alternative samplers outperform the Momentum Sampler for low Number of Function Evaluations (NFEs), but the Momentum Sampler achieves the best performance for higher NFEs. We expect that further research on generalized diffusion models will result to improved sampling methods, as it happened for diffusion models with additive Gaussian noise, e.g. see the work of Zhang & Chen (002)).
F.3 In short, the lindings
Training Objective. Concurrent works (Hoogeboom & Salimans 2022 Bansal et al. 2022) that also consider different corruption processes have used as their loss a simple objective: given an input image, they predict the residual to the clean. As explained in Section 3.1 this is equivalent to optimizing for an upper- bound of the Soft Score Matching objective (which is minimizing the error to the score-function.
Theorem 3.1. We show that if we use our exact pipeline (model, scheduling, sampler, training lyperparametess and so on) and we replace our Soft Score matching Loss with the loss introduced in Bansal et al. (2022) FID score increases from 1.85 to 5.91. This experiment shows the effectiveness of the Soft Score Matching objective.
Ablation Studies for Scheduling. We perform extensive ablations on the scheduling to understand the role of noise in the framework and whether our scheme outperforms other natural basclines. The results are detailed in the Appendix, Section F.1 Our main findings are: i) the Momentum Sampler works even if noise and blur are changing simultaneously (i.e. for schedulings with non-fixed noise), ii) lowering the maximum value of noise leads to important performance degradation - for very small noise the method completely fails and iii) our found scheduling outperforms significantly (bascline FID: 8.35, ours: 1.85) natural bascline that sets blur parameters such that MSE between the corrupted and the clean image decays in the same rate for Gaussian Denoising and Soft Diffusion.
Masking Diffusion Models. To show the generality of our framework, we also train models with (dis- crete) masking diffusion paired with noise. Figure 1 (top 2 rows) shows the forward and the (learned) reverse process for blur on the left and masking on the right. We train the model with our Soft Score Matching objective on CelebA. Unconditional samples from the model trained with masking can be found in Figure 15 of the Appendix. In Figure 7 we show the predictions of our two trained models (blur and masking) for the conditional mean, E| [.)) at different times of the diffusion. Soft Score Matching trains the model to make prediction that matches the real images in the filtered space. Hence, given masked images the masking model is incentived to predict right only the observed noisy region. As diflusion time becomes smaller (cleaner images), the observed region grows and the model predicts bigger windows. Although it is interesting that we can train Masking Diffusion models, there are several limitations compared to Blur Diffusion (and even Gaussian Diffusion). We observe that these models are very slow sample from: with 1024 sampling steps, FID is 30.92 while with 4096, FID improves to 12.37.
10

---

Published in Transactions on Machine Learning Research (05/2023)
5
Related Work
We showed that by just changing the corruption mechanism, observe important computational benefits. Reducing the number of function cvaluations for sample generation with diffusion models is an active area of research. Jolicoeur-Martineau et al. (2021); ;; Liu et al. (2022a]; Song et al. (2021a) propose more efficiint samplers for diffusion models. Rombach et al. (0022); Daras et al. (022) train diffusion models on low-
dimensional latent spaces. [ et al. (2022) combine diffusion models with Generative Adversarial Networks (GANs) to allow for bigger denoising steps. Ryu & Ye (2022 Ho ct al. (2022a) generate high-resolution images progressively, from coarse to fine quality. [Salimans & Ho (2022) train progressively student network that mimics the teacher diffusion model with fewer sampling steps. The authors of gDDIM (Zhang et al, 2022 develop sampling algorithm that extends the idca of DDIM to general diffusion models. We note that all these works are orthogonal to ours and therefore can be used in combination with our framework for even faster sampling.
On scheduling, there closely related work by Bao et al. (2022) The authors find a closed form solution (w.r.t. the score function) for the variance of the reverse SDE for Gaussian Diffusion. Then, they select a noise scheduling that minimizes the KL along the path from the initial to the final distribution. In our work, use Wasserstein distances and consider general corruption processes for which it is unclear whether such closed form solution exists. Instead, we estimate the distances in a data-driven way. This allows us to schedule arbitrary diffusion processes in a principled way.
There is significant recent Anonymous 2022 Liu et al.
2022b Peluchetti) and concurrent work (Rissanen
et al. 2022 Bansal et al., 2022 Hoogeboom & Salimans 2022) that proposes diffusion with other degrada- tions. These works have significant differences since they use different loss functions and sampling methods. Soft Score Matching experimentally outperforms (under the exact same sctting) the loss functions used in the concurrent works (ours FID: 1.85, theirs: 5.91). ). Our (blur) models oblain state-of-the-arl FID for linear diffusion models on CelebA (FID 1.85). For CIFAR10, we outperform (FID: 3.86) Gaussian diffusion with Variance Exploding SDE (Song et al. 2021b). HHoogeboom & Salimans (2022) also use blurring (but with Variance Preserving SDE) to further push the CIFAR-10 performance to FID 3.17. These advancements show that there are multiple diffusions with promising potential. Diffusion processes with general Gaussian Paths have also been explored in the context of Normalizing Flows (Lipman al. 2022).
6 Conclusions and Future Work
We presented a framework to train and sample from diffusion models that reverse general linear corruptions. We showed that by changing the corruption process, can get sample quality improvements and compu- tational benefits. This work opens several future research directions. For example, it is possible to optimize or learn the corruption process for solving a specific type of inverse problem. It is also worth exploring if mixing different corruptions (blur, noise, masking, etc.) improves performance. Our work could also be extended to the non-linear case, leveraging the techniques introduced in Rombach et al. (2022); Kim et al. (002a);; Wang et al. (2022). Finally, it important to understand the role of noise, from both a theoretical and practical standpoint.
Acknowledgements
This work was done during an interuship at Google Research. Giannis Daras has been supported by the Onassis Fellowship (Scholarship ID: F ZS 012-1/2022-2023) 2023),, the Bodossaki Fellowship and the Leventis Fellowship. This rescarch has been partially supported by NSF Grants CCF 1763702, 1934932, AF 1901281, 2008710, 2019844 the NSF IFML 2019844 award as well as research gifts by Western Digital, WNCG and MLL, computing resources from TACC and the Archie Straiton Fellowship.
The authors would like to thank Constantinos Daskalakis, Yuval Dagan, Sergey Ioffe and José Lezama for useful discussions. We would also like to thank Vikram Voleti and Simon Welker for useful comments on an earlier version of our work.
11

---

Published in Transactions on Machine Learning Research (05/2023)
References
Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982.
Anonymous. Diffusion models in space and time via the discrctized heat equation. In Submitted ICLR Workshop on Deep Generalive Models for Highly Struclured Dala, 2022. URL htttp:://ppnreeveee.nr net/
forum'id=Br1yyy4uuDD
Marius Arvinte, Ajil Jalal, Giannis Daras, Eric Price, Alex Dimakis, and Jonathan I Tamir. Single-shot adap- tation using score-based models for mri reconstruction. In International Society for Magnetic Resonance in Medicine, Annual Meeting, 2022.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2022.
Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold Diffusion: Inverting arbitrary image transforms without noise. arXiv preprint Xrx:22208.. 09399, 2022.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-DPM: Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models. arXiv preprinl ar arXi:2201.065003 2022.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXio preprint arXiv:2209. 01022099.16677 14687, 2022a.
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint XrXiv22206009411 2022b.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion Models in Vision: A Survey. arXiv preprint arXiv:2209.0473 7477, 2022.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In C.J. Burges, L. Botton,
M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Advances in Neural Information Processing
Systems, volume 26. Curran Associates, Inc., 2013. URL https://prooceedings neurips.cc/paper/ cc/paper/ pdf
Marco Cuturi, Lactitia Meng-Papaxanthos, Yingtao Tian, Charlotte Bunne, Geoff Davis, and Olivier Teboul. Optimal Transport Tools (OTT): A JAX Toolbox for all things Wasserstein. arXiv preprint
arXiv:2201.1233 2022.
Giannis Daras and Alexandros G Dimakis.
Multiresolution textual inversion. arXiv preprint
arXiv:2211. 171115 2022.
Giannis Daras, Yuval Dagan, Alexandros Dimakis, and Constantinos Daskalakis. Score-guided interme- diate layer optimization: Fast langevin mixing for inverse problem. arXiv preprint arXiv:2206.0911 2022.
Jacob Deasy, Nikola Simidjievski, and Pictro Liò. Hcavy-tailed denoising score matching. arXiv preprint arXiv:2112.091 02.09888, 2021.
Prafulla Dhariwal and Alexander Nichol. Diffusson models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8777 8794, 2021.
John R Dormand and Peter Prince. A family of embedded rungo-kutta formulae. Journal of computational and applied mathematics, 6(1):19 26, 1980.
12

---

Published in Transactions on Machine Learning Research (05/2023)
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106 (496):1602-1614, 2011.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. An image worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint Xrx0:22008.0 01618, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in neural infor- mation processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information. Processing Systems, 33:6840-660 6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23:47-1, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Flect. Video diffusion models. arXiv:2204.03458 2022b.
Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. arXiv preprint orXiv:2209.05533 2022.
Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In International Conference Learning Representations, 2022a.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pp. 1867-8887, PMLR, 2022b.
Aapo Hyvärinen and Peter Dayan. Estimation non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advonces in Newral Information Processing Systems, 34:14938 14954, 2021.
Daniel D Johnson, Jacob Austin, Rianne van den Berg, and Daniel Tarlow. Beyond in-place corruption: Insertion and deletion in denoising probabilistic models. arXio preprint arXiv:2107.07671 2021.
Alexia Jolicoeur-Martineau, Ke Li, Rémi Fich-TTailefees Tal Kachman, and Toannis Mitliagkas. Gotta go fast when generating data with score-based models. arXie preprint orXiv:2105.1400 14080, 2021.
Zahra Kadkhodaic and Ecro Simoncelli. Stochastic solutions for lincar inverse problems using the prior implicit in denoiser. Advances in Neural Information Processing Systems, 34:13242 -12244, 2021.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Flucidating the design space of diffusion-based generative models. orXiv preprint arXiv:2206. X00:2206.00364, 2022.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochastically. Advances in Neural Information Processing Systems, 34:21757-21769, 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration ion models. Advances in Newral Information Processing Systems, 2022.
Dongjun Kim, Byconghu Na, Se Jung Kwon, Dongsoo Lec, Wanmo Kang, and Il-Chul Moon. Maximum likelihood training of implicit nonlinear diffusion model. In Advances in Neural Information Processing Systems, 2022a.
13

---

Published in Transactions on Machine Learning Research (05/2023)
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and I1-Chul Moon. Soft truncation: A univer- sal training technique of score-based diffusion model for high precision score estimation. In International Conference on Machine Learning, pp. 11201-1101 11228. PMLR, 2022b.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696 21707, 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Represenlations, 2021.
Dohyun Kwon, Ying Fan, and Kangwook Lee. Score-based generative modeling secretly minimizes the wasserstein distance. arXiv preprint arXiv:2212.0633 06359, 2022.
Rémi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. Bayesian imaging using plug & play priors: when langevin meets tweedie. SIAM Journal on Imaging Sciences, (5(2)7701-737 2022.
Sangyun Lee, Hyungjin Chung, Jaehyeon Kim, and Jong Chul Ye. Progressive deblurring of diffusion models for coarse-to-fine image synthesis. arXiv preprint 1111:2207.11199, 2022.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Mall Le. Flow matching for generative modeling, orXiv preprint arXiv:2210.022: 027777, 2022.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint Xro:222002.097 2022a.
Xingchao Liu, Lemeng Wu, Mao Ye, and Qiang Liu. Let us build bridges: Understanding and extending diffusion generative models. arXiv preprint arXiv:2208. 14062,00..69,, 2022b.
Hengyuan Ma, Li Zhang, Xiatian Zhu, Jingfeng Zhang, and Jianfeng Feng. Accelerating score-based gener- ative models for high-resolution image synthesis. arXiv preprint arXiv:2206. 04029, 2022.
Dimitra Maoutsa, Sebastian Reich, and Manfred Opper. Interacting particle solutions of fokker fokker-planck equations through gradient-log density estimation. Entropy, 22(8):802, 2020.
Eliya Nachmani, Robin San Roman, and Lior Wolf. Denoising diffusion gamma models. orXiv preprint arXiv:2110.00 05948, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162 8171. PMLR, 2021.
William Peebles, Ilija Radosavovic, Tim Brooks, Alexei Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. arXiv preprint arXiv:2209 12892, 2022.
Stefano Peluchetti. Non-denoising forward-time diffusions.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.061 2022.
Mengwei Ren, Mauricio Delbracio, Hossein Talebi, Guido Gerig, and Peyman Milanfar. Image deblurring with domain generalizable diffusion models. arXiv preprint arXi::2212.017 01 2022.
Julius Richter, Simon Welker, Jean-Marie Lemercier, Bunlong Lay, and Timo Gerkmann. Speech enhance- ment and dereverberation with diffusion-based generative models. arXiv preprint arXiv:2208. 058300 2022.
Severi Rissanen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissipation. arXiv preprinl XrXio:2206.1 13397, 2022.
H Robbins. An empirical bayes approach statistics. In Proc. 3rd Berkeley Symp. Math. Statist. Probab., 1956, volume 1, pp. 157-163, 1956.
14

---

Published in Transactions on Machine Learning Research (05/2023)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the CEEE/CVR Conference on Compuler Vision and Pattern Recognition, pp. 10684 10695, 2022.
Natanicl Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprinl 2022.
Dohoon Ryu and Jong Chul Ye. Pyramidal denoising diffusion probabilistic models. aiv preprint arXiv:2208. 0tt.::2228000884,,, 01864, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to- image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 1 1487, 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via itcrative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022b.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022. /ttp::///////eeeeee.eee. wtt.:e/etoooomootooIIIII....
Joan Serrå, Santiago Pascual, Jordi Pons, R Oguz Araz, and Davide Scaini. Universal speech enhancement with score-based diffusion. arXiv preprint arXiv:2206.0300 03065, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, In International Conference on Machine Learning, pp. 2256 2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Representations, 2021a.
Yang Song and Stelano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Newral Information Processing Systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 111223--12445 2020.
Yang Song, Jascha Sohl-Dickstein, Diederík P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021b.
Charles M Stein. Estimation of the mean of a multivariate normal distribution. The annals of Statistics, pp. 1135-1151, 1981.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling latent space. In Neural Information Processing Systems (NeurIPS), 2021.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23 (7):1661 1674, 2011.
Zhendong Wang, Huangjie Zheng, Pengeheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-gaa gan: Train- ing gans with diffusion. arXiv preprint urXim:2206.0220 2022. URL https://arxiv .ow./ www./oos///////.....oo abs6
02262
Jay Whang, Mauricio Delbracio, Hossein Talcbi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16293 16293-16303, 2022.
15

---

Published in Transactions on Machine Learning Research (05/2023)
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference on Learning Represenlations, 2022.
Ling Yang, Zhilong Zhang, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Ming- Hsuan Yang, and Bin Cui. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint ar Xr1::220090000 00796, 2022.
Mao Ye, Lemeng Wu, and Qiang Liu. First hitting diffusion models. arXiv preprint arXiv:2209.01 011000900111,.....0 2022.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint Xrv::2204 13902, 2022.
Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. XrXiv preprint arXi::2266.08 055664 2022.
16

---

Published in Transactions on Machine Learning Research (05/2023)
A Proofs
Theorem A.1 33.1 Let qo. be two distributions in IR". Assume that all the conditional distributions, gr (o0), are supported and differentiable R2. Let:
Ji(0) Bei-a (100 - Va, log ge (xx)))
(21)
(2 18 (=) (xo)) (mrl))))))
[1180(0)))) Vx log ( (20) )|²
(22)
Then, there is universal constant C (that does not depend on 0) such that: J1 (1 J2 +C.
The proof of this Theorem is following the calculations of Vincent (2011). We observe that as long as the tochnical conditions listed are satisfied, the proof holds independent of the corruption type. We provide the proof below for the shake of completeness.
Proof of Theorem
(0) = Barr rag
[|s0(<=)]1 200(2x)) " Vaz, log (xxe) + VV,, log ae (x:)))
(23)
2
E Eerryg [||s0(<))111) Exerge (06()))) (" log (x+) C1.
(24)
Similarly,
(2) Panra [[1=0(2=)]13]] 1 (a0 [so(ar)) log (or + Cz.
(25)
It suffices to show that:
Ea [so(ax)) V., log (xx)) (o....)))ooo))aaaa [so(xx)) " Vaz [
(26)
We start with the second term.
(o [s0(x;)² V x, log (
20 (m)))))) V Va, log ge (ar, (mg dar
(27)
$o (z.) Va. log qe (20)) dæ
(28)
90 90(20)9 5 (xx)
V., (x) m2)) dx
(29)
Gr ( (0o)
oy (xx) (20 Ve dæ,
(30)
m
t (zx) (go V dændæ;
(31)
a. 80
5b (x:)
go(2o) V (2 dx.
(32)
20
(fr
$b (x1)
Va, Ve,
(2 dæy dx
(33)
Va. (/ go (o(0o) xo )dzo dæ:
b (It)
(34)
[ $ (zt) V m. ge (2,
(35)
)
( V log qe ddz
(36)
m;
E (ax)
[5 (xx V Va, log (x()))
(37)
17

---

Published in Transactions on Machine Learning Research (05/2023)
]
Lemma A.2 (Twcedic's formula). Consider the corruption process: a; Cizo om, where n NO, I). Denole with (xx) the densily of and assume that log (x(s)) is differentiable everynchere. Then, for the Minimum Mean- Mean-Square Estimation (MMSE) of xo given 2t, it holds that:
C.Elaollass] + oiV as log qt (are).
(38)
Proof.
(((x))) = (a((a//aoo))mmmmm)))
(39)
(x,)) = Vv. )do
(40)
(00 lo a no darp
(41)
We know that: (<))//0))) = (xri Cra0: E = oof1). Hence,
V (x,) = f (00 Croo - 2t dæo
(42)
oî
2t / (00(20)04 )e (we koo )daco
C
(0((20))
(43)
7
1
( (1) (20d - a)
C
(44)
o?
Ve,
(xx))
1 (C,E(zolaz)) xx.) +
(45)
(1()))
logg((#;)
(C,B/arg - xx).
(46)
2.
of
B Deterministic Samplers
B.1
Probability Flow ODE
In the main text, we derived our Momentum Sampler by analyzing the Backward SDE associated with our corruption process. Inspired by the works of [Son et al. (2021b); Maoutsa et al. (2020). we also consider deterministic sampling that is derived by looking at the ODE that describes our diffusion. Particularly, the ODE:
dæ, s( -00000599000 (1) log v (mx) di,
(47)
has the same marginal distributions 2018) with the Backward SDE:
Anderson 1982
Maoutsa et al. 2020 Song
2021b
Chen
dæ; = [f(x,, g2 (t) (x:) + g(t)dw.
(48)
For our case, Eq.
47 becomes:
1 dd(a?)
dx;
C,E [c.]
V., log (zx)) dt.
(49)
2 dt
18

---

Published in Transactions on Machine Learning Research (05/2023)
The first-order discretization of this ODE is given below:
(0î- at - -ot) Vz, log qt (xt).
-t- At - It (Ci-at C,)Blaa [20]
(50)
We estimate C.E(zplaz and V log qe (2++) with our noural network and we get the Neural ODE (Chn et al.
2018):
Ar. = Et 3t
(51)
o - oi
= (Ci-at - Ci) Φ0((x)
(c+ (2)())))) xt -xt)
Dt
of
which is the update rule our Probability Flow ODE Momentum Sampler.
We note that our simple discrctization is the only way to solve the ODE of Eq. sophisticated, e.g. see Dormand & Prince (1980).
We can use morc
B.2 DDIM
A popular sampling scheme is the DDIM method, introduced in Song et al. (022aa). The idea is to use the network to predict dgg from Fy and then use the forward model to move from to to at-At- This same with the Naive Sampler we introduced in the main paper but the difference in DDIM is that part of the stochasticity replaced by "simulated" noise. The main trick to simulate the noise is to observe that once know do and Ht, idea for Soft Diffusion sampling. Specifically, we propose the following DDIM-type sampler:
once again the forward model to estimate the noise. We can use the same
Ft = Ct Ai ho( V of - 12 Dt Ciho(zee +kn.
(52)
rolm simulated noise
This Equation very similar to Equation 12, page in the DDIM (Sog et al. 2021a paper. The parameter k controls the amount stochasticity, iee. how much of the noise simulated. For 0, we get a deterministic sampler. Experiments with the deterministic DDIM-type sampler can be found in Section F.3
Figure Conditional means
azo [ac.] predictions of our blur alur/masking models, different diffusion times.
C Schedulings
We use our framework select the blur levels an unsupervised way. For solving the optimal transport problem, we the Sinkhorn Distances by Cuturi (2013) that has been extensively used for image experi- ments. We use the implementation of this paper from the software package ott-jax (Cuturi al. 2022). We experimented with both using the whole dataset and using slices. As expected, dataset slices lead to higher ostimation errors, but we observe that the found scheduling doesn't change much, i.c. the estimation error increases approximately uniformly for all the pairs for reasonably sized dataset slices. For the schedules found in the paper, we used slices of 10% of the dataset. We start with 256 different blur levels and we tune such that the shortest path contains 32 distributions. We then use lincar interpolation to extend to the continuous case. Full experimental details can be found in the Appendix. These choices seem to work well in practice, but further optimization could be made in future work.
19

---

Published in Transactions on Machine Learning Research (05/2023)
Figure shows the found schedulings for the CelebA and the CIFAR-10 datasets. Notice that the scheduling is slightly different between the two datasets - the diffusion depends on the nature of the data are trying to model. We underline that the parameters for the blur are selected without any supervision, by solving the optimization problem we defined in Scction 3.3
60
CelebA (64x64)
CIFAR10
0.08
GaussianB (ur(a)
50
Noise(d)
0.06
30
0.04
20
CelebA (64x64)
CIFARI0
0.0
0.2
0.4
0.6
0.8
8.0
0.2
0.4
0.6
0.8
1.0
diffusion time
diffusion time
Noise Scheduling
Blur Scheduling CelebA
Figure 8: Diffusion scheduling for CelebA-64 and CIFAR-10. The blur corruption levels are selected without supervision to minimize the sum of the Wasserstein distances between consecutive distributions. Notice that the scheduling is slightly different between the two datasets the diffusion depends on the nature of the data we are trying to model. The support of the Gaussian blur kernel was set to 65 x 65 and 161 x 161 for CIFAR-10 and CelebA-64 datasets respectively.
D Training Details
For our trainings, we use Adam optimizer with learning rate 2e 4, 31 0.9, B2 0.999, c = le 8. We additionally gradient clipping for gradient norms bigger than 1. For the learning rale scheduling, use 5000 steps of linear warmup. We use batch size 128 and we train for 1 1 22 ilerations (based on observed FID performance).
Hyperparameters.
Blur parameters. For the blurring operator, we use Gaussian blur with fixed kernel size and we vary the variance of the kernel. For CelebA-64, we keep the kernel half size fixed to 80 and vary the standard deviation from 0.01 to 23. For CIFAR-10, keep the kernel half size fixed to 32 and vary the standard deviation from 0.01 to 18. For both datasets, we implement blur with zero-padding. We chose the final blur level such that the final distribution is casy to sample from. In both cases, the final distribution becomes noise on top of (almost)) single color.
Final distribution For the blurring models, the final distribution is the distribution of very blurry images with additive Gaussian noise (we first blur, then add noise). At the limit, each blurry image becomes constant image having single color (i.e., the average color of the image). Hence, to sample from this final distribution, we first have to sample single color (from the distribution of average colors in the dataset), generate a constant image having that color, and then add little noise to the constant image. The distribution of average colors for all the considered datasets is very simple and we model it with a Gaussian distribution. Specifically, we a three-dimensionall gaussian distribution (one dimension for each color channel, diagonal covariance) the average color of the dataset. To begin the inference process, we sample one color from this distribution, create an image where every pixel has this color and then add Gaussian noise.
Architecture.
We use the architecture of Song et al. 2021bb without any changes.
Training Objective For all our experiments, we scale the loss level with w(t) 1/a? as in Song & Ermon (2019: 2020 Song et al. (021b)
20

---

Published in Transactions on Machine Learning Research (05/2023)
Compute and Training Time We train models 16 v2-TPUs. Our bhur models CelebA had an average speed of 6 iterations per second. For CIEAR-10, the average speed was 11 iterations per second We note that there is no overhead over the N paper other than projecting to the measurements space, which can be done very efficiently for both blur and masking.
Evaluation We keep one checkpoint every 10000 steps and we keep the best model among the kept checkpoints based on the obtained FID score. We use 50000 samples to evaluate the FID, as it is typically done in prior work. Regarding the number of steps, we selected the best FID among models evaluated with steps ranging from 200 to 1000 with step size of 80 (10 experiments total).
The Momentum Sampler seems to have a U-shaped performance plot, i.e. there is sweet spot in the number of function evaluations that gives the lowest FID score. We observed this in both CelebA and CIFAR-10.
However, this is not unique to our sampler the general belicf that sample quality improves with the number of steps has been called into question by recent works. Specifically, Karras et. al. do an extensive evaluation of the role of NFEs for different samplers and they find that many stochastic samplers behave similarly to our Momentum Sampler. For example, refer the interested reader to Figure 4, page 8 of the paper Elucidating the Design Space of Diffusion-Based Generative Models that clearly shows that for many stochastic samplers performance slightly deteriorates after a certain threshold of function evaluat ions and the optimum is at some intermediate point. We emphasize, however, that the deterioration we observed in CelebA is bigger compared to other stochastic samplers and therefore selecting the correct number of NFEs is important for our method. Consistent with the observations by Karras et al. (2022) (e.g. Figure 2, page 4); for deterministic samplers the performance flattens (DDMM) deteriorates slightly (Probability Flow Momentum Sampler) in the regime of very high Number of Function Evaluations (NFEs).
We also want to note that selecting the optimal number of function evaluations for reporting FID is not uncommon in the literature. It is done in both the landmark papers Elucidating the Design Space of Diffusion-Based Generative Models (Karras et al. 2022) and Score- Generative Modeling through Stochastic Differential Equations (Song et al. 2021b).
E Limitations and things that did not work
Our method has several limitations. First,, it requires the diffusion operator to be known. This is not always the case, e.g. see Peebles et al. (2022) where diffusion is applied to checkpoints of different models. Another limitation is that our framework does not offer any guidance on which diffusion operators are actually more or less useful for learning the data distribution. Particularly, we already showed that blurring is a much more powerful diffusion mechanism than masking, in the sense that it leads to better FID scores and faster generations. It is yet to be seen whether blurring is going to be outperformed by some other corruption method. Our method also only concerns linear diffusions (however, the extension to non-linear is relatively straightforward).
On the theoretical side, our method has also some shortcomings. First, it only intuitively explains why the reparametrization to the Denoising Score Matching is needed. Second, since our method is based on the Denoising Score Matching, it only works when the conditional log-likelihood is defined everywhere. There are distributions for which such condition is not satisfied, but still, can be Icarned (to some extent) with heuristic methods (Bansal et al. 2022).
Another limitation of our work is the derivation of the Momentum Sampler and its sampling distribution. Consider the forward process:
dæ; Cr [oo] + g())d
(53)
This process described by an Tto SDE, i.c. an SDE of the form:
dæ, = f(a t)dt + g(t)dw,
(54)
where f(x,, mog [2c)
21

---

Published in Transactions on Machine Learning Research (05/2023)
The sampling process we write in the main body of the paper (that leads to the Momentum Sampler) is exactly the reverse of this forward process, where P [2a is approximated by the neural network. Hence, the Bolker-Planck equations hold and we are guaranteed that (as long as the approximation of the conditional expectation and the approximation of the score function are accurate), we are sampling from the correct distribution (Song et al., 2021b).
During training, we cannot use this forward process because we do not have the conditional expectation. Instead, replace the conditional expectation of to with 2o itself. The mismatch between agand the conditional expectation of zo leads to an additional approximation error in the learning of the score. For small t, the conditional expectation will be very close to Zo and hence this approximation error is small. For large values of i, the distance between the conditional expectation and To is bigger. However, we are multiplying with the corruption matrix Ct, which is removing information as t grows. Therefore the distance of the corrupted conditional expectation and the corrupted Ho also expected to be small. Our training process learns the correct score for the corruption process applied directly Zp. Our sampler, however, assumes that we have learned the score using the conditional expectation of æ6- We intuitively expect that these two are not far away (for all t) as we explained. However, precisely characterizing this mismatch remains open.
On the practical side, we believe that our objective, Soft Score Matching, sometimes leads to slower sampling compared to the simpler objective of predicting the clean image. For example, for masking, since the model is only penalized in the observed region, there is no incentive in expanding this region. Hence, to achieve smooth transition between different masking levels we need to run many steps.
Experimentally, we tried using our framework with even less stochasticity, but it did not work, e.g. see It would be interesting to undersland better what is causing the failure and also what is the proper amount of randomness required at each diffusion step.
F
Additional Results
F.1
Scheduling Ablations
F.1.1
Ablation Studies for noise
In the experiments of the main paper, our diffusion involves an initial stage where only noise is added. Then noise is fixed and the images getting corrupted by the deterministic operator (e.g. blur masking). In this section, we show two ablations regarding the noise.
Magnitude of noisc. In this ablation study, we still keep the noise fixed for a significant part of the diffusion, but we ablate the magnitude of the noise. Specifically, we altempt to study to what extent noise is needed in order to learn to reverse corruption processes with Soft Score Matching. We train two additional models CelebA-64 where attempt to decrease the maximum noise to a lower value. The corruptions for both models involves initial stage where noise grows geometrically rate from the initial value (0.01)
to the maximum value. Then, all the models keep the noise fixed at their maximum value for the rest of the diffusion. We use the following maximum values: i) 0max = 0.1 (model used in the paper), ii) Tmax = 0.05,
and iii) Fmax = 0.025. Unconditional samples from the two ablation models are shown in Figure 9 As shown, both models fail to produce realistic samples. The quality of samples deteriorates significantly as the noise decreases the samples from the ablation models are significantly worse than the ones produced by our state-of-the-art model (see Figure 2 (right)). We want to underline that this is not a conclusive study. It might be the case that with different hyperparameters with lower values of noise. For example, we might need to tune the weights w(t) since for the ablations (and the state-of-the-art model), we use w(t) = 1/0? (as in Song & Ermon (2019 2020): Song et al. (2021b))
can make Soft Score Matching work
which might be causing instabilities for low values of noise (Nichol & Dhariwal 2021
Noise changing throughout the diffusion. We also train a model where noise and blur are changing simultaneously throughout the diffusion. This is a sanity check to verify that our framework (learning and sampling) still works when the model needs to deblur and denoise at the same time. For the noise scheduling,
22

---

Published in Transactions on Machine Learning Research (05/2023)
Images generated with model trained with max 0.025 Figure 9: Ablation study for the magnitude of noise.
Images generated with model trained with maax 0.05
we simply use geometric scheduling from 0.1 to 0.01. We keep the blur parameters the same with the state- of-the-art model, i.e. we use the blur parameters shown Figure [ This model achieves competitive FID score, 4.31. This score could probably be further improved (by jointly selecting the blur and the noise scheduling with our framework), but this is beyond the scope of this ablation.
F.1.2 Ablation Study for Blur
For all our experiments so far, we chose the blur corruption levels based on the scheduling method we described in Section 3.3] We show the benefits of our approach by comparing to a natural bascline for selecting the diffusion levels. For this natural baseline, we use the scheduling of Variance Exploding (VE) guidance. Specifically, we choose the blur parameters such that the MSE between the corrupted image and the clean image decays with the same rate for the Gaussian Denoising Diffusion and our Soft Diffusion (blur and low magnitude noise). Formally, let {g'li= be the (noisy) distributions used in Song et al. (2021b) for the Variance Exploding (VF) SDE and let {4:)1_ the blurry (and noisy) distributions we want. to select. At level t, we choose the blur parameters such that:
E, Em (2.
a xe xxe1
(a ) (1t ( [|:00- x=
(55)
E E( ( (a) (2o))
[ x1 |2] (ooo:,))~-ff (a) (m)))( [|x00 x 11
We retrain on CelebA using this natural bascline method for selecting the diffusion paramcters. For a fair comparison, we keep the architecture and all the hyperparameters the same and we only ablate the scheduling of the blur. We mensure FID for both the trained model with the baseline scheduling and we observe it increases from 1.85 to 8.35. Apart from this large deterioration in performance, the baseline model obtains its best FID score after 2000 steps, while with our scheduling we only need 280 steps obtain the best FID. This experiment shows that the choice of scheduling is really important for the model performance but for also the computational requirements of the sampling.
F.2 CIFAR-10: Additional Results
We report how FID is changing based on the Number of Function Evaluations for CIFAR-10 with Momentum
Sampling in Figure 10 The baseline model we are comparing against is NCSN++ trained with VE SDE, since our model also uses VE SDF. For the baseline, we use the Reverse SDE sampler with Euler Maruyama discretization (similar to our Momentum Sampler). The numbers are taken directly from Ma et al. 2022).
We also show visual samples from our blurring model, NCSN++ (VE SDE) and DDPM++ (VP SDE) in Figure 11 For all the models we fix the NFFs to 200 and we show samples generated with the Fuler- Maruyama discrctization of the associated reverse ODE. As shown, the bascline models generate samples with artifacts for low NFEs while our model leads to images of superior visual quality. We note that different samplers can be used to accelerate sampling for all models, e.g. samplers from Karras et al. (022) or DDIM-type samplers (Song et al. 2021a), as we show in Scction B.2
We also report best FID performance for different samplers for ours and competing methods in Table 2 Our best FID is competitive with similar samplers to similar methods, e.g. with NCSN++ trained with VE SDE and using the Reverse SDE sampler. However, is significantly behind the stato-of-thh-art LSGM (aahdat et al. (021)) model. We believe that this performance gap can be decreased in the future by further rescarch in the area of diffusion models with general corruptions.
23

---

Published in Transactions on Machine Learning Research (05/2023)
CIFAR10
32
ncsn++
ours
FID
2
100 110
120
130
140
150
160
Number of Function Evaluations (NFEs)
Figure 10: Demonstration of how FID changes based Number of Function Evaluations (NFEs) for CIFAR- 10 for our (blur) model on CIFAR-10 with momentun sampling. Our model offers significant performance benefits for low number of function evaluations.
Ours (VE SDE)
NCSN+ (VE SDE)
DDPM- (VP SDE)
Figure 11: Visual comparison samples from our model and baselines for 200 NFEs. For all the models,
we are using the Muler-Maruyama discretization of the associated reverse ODE to generate the samples. As shown, our model leads superior visual quality.
Method
FID
Ours (VE SDE) Naive Sampler
40.07
Ours (VE SDE) Momentum Sampler
3.91
Ours (VE SDE) Probability Flow Momentum Sampler
3.86
NCSN ++ (VE SDE) Reverse SDE
4.79
NCSN++ (VE SDE) Probability Flow
10.54
DDPM (VP SDE)
3.17
LSGM (Vahdat et al. 2021
2.10
Table 2: FID Results on CIFAR-10 for different samplers. For each of our samplers, we report the best result obtained among 10 runs with different NFEs, ranging from 200 to 1000 with step size of 80. The results for competing methods are taken directly from the papers.
F.3 Sampling Ablations
We perform several ablations regarding the sampling method, additional the results mentioned in the paper.
24

---

Published in Transactions on Machine Learning Research (05/2023)
We experiment with the deterministic DDIM-type sampler and report results in the Table for CelebA-64. Our DDIM-type sampler is very effective when the number of function evaluations is very low it achieves FID 5.08 with only 50 steps while in comparison, the Momentum Sampler, achieves FID with 50 steps. The efficiency of DDIM is consistent with what has been observed in (Song et al. 2021a; Karras et al., 2022). DDIM's performance also not gelting worse as we increase the number of steps. Instead, the performance of the Momentum Sampler, as observed
DDIM
30.31
Figure g has a U-shape: there is an intermediate optimal number of steps to achieve the best FID score. We believe this could be related to the fact that Momentum Sampler is not deterministic. Specifically, in Karras et al. (2022) it is observed that for deterministic samplers performance usually flattens after point (e.g. see Figure 2, page 4) while for some stochastic samplers performance goes again up after a certain number of function evaluations (e.g. see Figure 4, page 8).
NFEs
FID
25
10.54
50
5.08
100
3.40
200
2.92
300
2.83
600
2.80
1000
2.79
Table 3: Results of DDIM (Song et al. 2021a) sampler for CelebA-64 dataset using our blurring model. The DDIM sampler is particularly ellective for low number of function evaluations and maintains its performance as we increase the number of steps.
Probability Flow Momentum Sampler For CelebA-64, we reported results only for the (stochastic) Momentum Sampler in the main body of the paper (see Figure 3p. We report here results for the deterministic counterpart of this sampler, Probability Flow Momentum Sampler, summarized in Table The deterministic version of the Momentum Sampler seems to be performing slightly worse than the stochastic version FID jumps from 1.85 to 2.14. However, this sampler, similar to the DDIM-type sampler, maintains more performance as we increase the number of steps. This an important advantage for the deterministic sampling methods since practitioners need to put less effort into tuning the NFEs to get the best result.
NFE
FID
50
13.77
100
3.27
200
2.14
300
2.21
600
2.59
1000
2.49
Table 4: Results of the Probability Flow Momentum Sampler for Celeba-64 for our blurring models.
Predictor-Correeti Samplers Finally, we perform experiments with Predietor-Correctt samplers, as proposed in [Sng al. (2021b). The idea is that we are alternating each diffusion step between two different samplers. We experiment with DDIM Predictor and Probability Flow Momentum Sampler corrector. The results are summarized in Table 5 The Predictor-Corrector sampler maintains some of the benefits of both samplers. Namoly, for low number of function evaluations it has better performance than the Probability Flow Momentum Sampler (benefit coming from the DDIM sampler) and for higher number of function evaluations performance is better than the DDIM sampler (benefit coming from the Probability Flow Momentum Sampler). There spot, at 100 NFEs, where the Predictor-Corrcctor Corrector sampler is better
than both the Predictor and the Corrector. We encourage future research identifying even better pairs of that might outperform both the Predictor and the Corrector in some regime.
25

---

Published in Transactions on Machine Learning Research (05/2023)
NFE
FID
25
10.56
50
5.21
100
3.21
200
2.86
300
2.71
600
2.73
1000
2.73
sampler for Celeba-64 for our blurring models, The Predictor is the DDIM-type sampler and the corrector Probability Flow Momentum Sampler.
Table 5: Results of a Predictor-Correete (Song al. 2021b)
F.4 Nearest Samples in Training Data
To verify that our model does not simply memorize the training dataset, we present generated images from our model and their nearest neighbor (L2 pixel distance) from the dataset. The results are shown in Figure
I2
Generated images.
Nearest neighbors from dataset.
Generated images and nearest neighbors (L2 pixel distance) from the training dataset. As shown, the model produces new samples and does not simply memorize the training dataset.
Figure 12:
F.5
Uncurated samples
Figures
13 and
show more uncurated samples from our trained models on CelebA and CIFAR-10.
26

---

Published in Transactions on Machine Learning Research (05/2023)
Figure 13: More uncurated samples from our blur model trained on CelebA. These samples are obtained using our Momentum Sampler with 280 NFEs.
27

---

Published in Transactions on Machine Learning Research (05/2023)
Figure 14: More uncurated samples from our blur model trained on CIEAR-10. These samples are obtained using the Probabilistic Flow Momentum Sampler with 200 NFEs.
28

---

Published in Transactions on Machine Learning Research (05/2023)
Figure 15: Uncurated samples from our masking model trained on CelebA. These samples are obtained using
the Probabilistic Flow Momentun Sampler with 1024 NFEs.
29

---

Published in Transactions on Machine Learning Research (05/2023)
G Implementation of Degradation Operators
Blurring. The blurring operator is implemented as convolution with Gaussian kernel. The Gaussian kernel is truncated to have fixed support size (161 x 161 pixels), and normalized to have area one. The standard deviation of the Gaussian kernel sigma_ __lurrbuurrrrrrrr (parameter controlling the strength of the blur) is computed using the optimal transport optimization for the scheduling. The obtained schedule for the parameter sigma blur in CIFAR10 can be accessed from the following anonymous url: lattps://drive. oogglee..coo///il////////tbggggp 2600277222 -x0g5Elg
def generate__guusiia.Keenma kernellleeeeeeeleeeeeeeee (aigma_ssimmaalll blur, half salf_size=80):
v np. np.arange--half_.tt _ize, half_size + 1)
x, y np.meshgriddv, v)
k np. exp((=xx**22 y*+2) / (2+sigma_blur
k k / np. (k)
return k
Masking. The masking operator is implemented using a centered binary mask that sets to zero some percentage of pixels in the image. The scheduling with the masking operators can be accessed from the following anonymous url: https: oato::/a.........././///////aaaa
Noise. The noise degradation is implemented by adding to the blurred or masked image white Gaussian noise having standard deviation sigma noise. The scheduling of the noise can be accessed from the following anonymous url: http://wwwgggg.oooooooccc/////ta
30


Reviews:
[{"review_id": "qCAZPLl35h", "reviewer": "Reviewer_5XYr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n* First, I want to emphasize that the paper is very well-written and easy to read. The introduction of diffusion models is very clear as is the presentation of the loss following [1]. I also think that the idea presented in the paper is of interest for the diffusion model community (and more generally for the ML community). Indeed, setting an optimal forward degradation operator has been a long standing problem in diffusion models since their introduction. Most of the existing papers focus on VE-SDE or VP-SDE but this paper offers a nice alternative which is also competitive (and for some datasets state-of-the-art).\n* I think the idea of interpolating between the clean observation and the noisy data is a very good one. The simplicity of the interpolation comes at a price when defining a forward diffusion model but it also gives a better interpretation of the quantity $x_t$. It seems to me that the procedure considered by the authors could be linked with recent work extending diffusion models (flow matching [2] in the deterministic setting or bridge matching in the stochastic setting [3,4], for instance). Could the authors comment on the possible links here? \n* The experimental results are very good. Not only the authors obtained state-of-the-art results (at the time) for diffusion models in the case of Celeba64 but they also provide a lot of ablation studies and the appendix section \"Limitations and things that did not work\" is very useful\n\nWEAKNESSES:\n* I found the justification of the naive sampler to be pretty weak. First, the SDE introduced in (13) is given so that in the \"toy setting where the dataset contains one image [...] the Markovian corruption has the same marginal distributions as the process\". Why is that the case? Also there is surely a mismatch between the solutions of this SDE and the interpolation of Equation (4) as stated by the authors. However, the authors blur the lines between the stochastic process (13) and the interpolation (4) by denoting them both $x_t$. If the training is done by sampling $x_t$ according to the interpolation, does it make sense to consider this time-reversal (since the forward does not match the interpolation). I think it is necessary to comment on the mismatch between the interpolation process and the Markovian corruption.\n* The naive sampler does not make a lot of sense to me. Indeed, the authors call (13) the \"Markovian corruption\" and therefore, the SDE (13) is run in forward mode. However, at sampling time (Algorithm 1), the sampling is done backward. This is wrong because it is well known that in that case we do not recover the time-reversal of the forward. The time-reversal of the forward is in fact given by the momentum sampler (Algorithm 2).\n* This is a minor point but I am not sure that the name \"Momentum sampler\" is appropriate here. Setting $C_t=\\mathrm{Id}$, then Equation (18) is equal to zero, and it is a bit weird to say that there is no momentum in a VESDE. \n* I wish the authors could give more justifications to the setting considered in 3.3 (Scheduling). It is not clear what $\\mathcal{D}\\theta$ is here. In particular, I don't understand what is meant by \"each $\\theta_i$ corresponds to a distribution $\\mathcal{D}_{\\theta}$. Could the authors elaborate on that? Also the choice of the Wasserstein distance (which one? I assume the quadratic one. With regularization or not?) is weird for images as it compares images pixel-wise which is known to be a poor measure of comparison.\n* It is a bit underwhelming that no methodological/theoretical reason is given for why adding blur helps the model. It seems to work in practice but could someone choose another linear (or non-linear) operator and improve on the results or is the blur kernel special in some sense?\n\n[1] Karras, Aittala, Aila, Laine (2022) -- Elucidating the design space of diffusion-based generative models\n\n[2] Lipman, CHen, Ben-Hamu, Nickel, Le (2022) -- Flow Matching for Generative Modeling\n\n[3] Liu, Wu, Le, Liu (2022) -- Let us Build Bridges: Understanding and Extending Diffusion Generative Models\n\n[4] Peluchetti (2021) -- Non-Denoising Forward-Time Diffusions", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors introduce Soft Diffusion, a diffusion model framework which allows for general corruptions. First, the authors recall some of the basics of diffusion models as well as some good practices introduced by [1]. Then, they present their diffusion model setting. To do so, they define an interpolation between the clean data and some noise. Contrary to typical diffusion model (like VP-SDE or VE-SDE) the linear operator acting on the clean data is not simply a scaled version of the identity. Indeed, in this setting, any linear operator can be chosen. In particular, one can choose to blur the data. In the experimental part, it is found that blurring the data is beneficial in order to reach state-of-the-art FID results on CelebA. This interpolation procedure is associated with some forward SDE (note that contrary to the VP-SDE framework the SDE does not have a linear expression w.r.t. $x_t$ but instead depends on $\\mathbb{E}[x_t|x_0]$. From this SDE they derive a sampling procedure \"Naive Sampler\". Considering the time-reversal of this SDE, they introduce another sampler (which corresponds to the Euler-Maruyama discretization of this time-reversal) called the \"Momentum Sampler\". The name momentum sampler comes from some analogy with the deterministic setting. In order to determine the blur used in the process, the authors consider a graph based approach. Finally, they illustrate their approach for unconditional image modeling on CIFAR and CelebA64.\n\n[1] Karras, Aittala, Aila, Laine (2022) -- Elucidating the design space of diffusion-based generative models", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please answer the questions listed in the \"strengths\" / \"weaknesses\" section above. ", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1sduJKwIfG", "reviewer": "Reviewer_iwsn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[+] Though there are some similar works, Soft Diffusion is one the of first works to explore different corruption processes and conduct empirical experiments to validate the effectiveness of the proposed method.\n\n[+] The work has valid theoretical support. Authors how the proposed scheme can be formulated with SDE such that it can inherit lots of theoretical results and advantages from existing works.\n\n[+] The proposed momentum sampler is novel to me and works well in empirical experiments.\n\n[+ / -] The method outperforms existing methods in CELEBA, but does not show advantages in other datasets.\n\n[+ / -] Authors also discuss concurrent works and limitations, I do not find obvious weakness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work explores the design of the noising scheme in diffusion models. The authors propose a general noising scheme, which adds noise to the time-dependent blur images instead of clean images.  Theoretically, authors construct a corresponding SDE for the proposed method and show ingredients in existing diffusion models can be used in the new framework with minimal or no modifications. The authors further propose a new sampler tailored for soft diffusion and validate the advantages in empirical experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Personally, I prefer moving the results of ODE sampler such as DDIM into the main content.  ", "broader_impact_concerns": "I do have any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vHcE07ZJk5", "reviewer": "Reviewer_AJba", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of this paper is that it proposes new diffusion models, with promising experimental evaluations, which is of great contemporary interest. Also, the paper is well-written.\n\nHowever, I have several questions regarding the justification of the methodology. First of all, in eq. (13), the SDE is only an approximation to the corruption process. Presumably this is done in order to be able to reverse the SDE in time (please correct me if I am wrong; otherwise I do not see the difficulty in finding an exact SDE that describes the corruption process in (4)). However, in that case, the framework of Song and Ermon allows for a general SDE for the forward process; why not consider an SDE with linear coefficients, such as $\\mathrm d \\mathbf x_t = A_t \\mathbf x_t\\\\,\\mathrm d t + \\partial_t (\\sigma_t^2) \\\\, \\mathrm d \\mathbf w_t$? It seems to me that this would afford the same flexibility as (4), but would fit within existing frameworks for diffusion models, not have to rely on an approximate SDE, and not require special samplers. Can you compare your work with that of Q. Zhang, M. Tao, and Y. Chen, gDDIM: generalized denoising diffusion implicit models?\n\nFor eq. (17), why not use the exponential integrator of Zhang and Chen (2022)?\n\nIn Section 3.3, how is $\\mathcal M(\\mathcal D_{\\theta_i}, \\mathcal D_{\\theta_j})$ calculated? Also I do not understand the relevance of the Kwon et al. (2022) work – their paper only shows an upper bound on the Wasserstein distance, and moreover this upper bound scales exponentially with the Lipschitz constants of the vector fields, which could be large in practice.\n\nOverall I found this paper to be very heuristic, with poorly justified choices. I would not recommend acceptance unless the authors give a clear explanation of what their work achieves that the framework of Song and Ermon does not.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a more general family of diffusion models in which the corruption process is a linear mapping applied to the initial sample, plus some Gaussian noise (when the linear mapping is the identity or a rescaling, this corresponds to the usual diffusion models). An algorithm is proposed to sample from these more general models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In Section 3.3, there is an inconsistency between whether or not mathcal is used for D.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, the reviewer consensus is that: (i) the soft diffusion idea is a novel and interesting one, (ii) the paper is well-written and clear, and (iii)  the various initial concerns were sufficiently addressed by the author response.  A reviewer noted that perhaps handling more challenging data sets would be beneficial, and I agree this would be helpful but I would stop short of call it essential.

I have entered "minor revisions" to give the authors opportunity to make any final changes based on the reviews, but there are no changes that I am specifically marking as "mandatory".
Decision: Accept (poster)