Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion Models
Authors: Yangming Li
Boris van Breugel
Mihaela van der Schaar
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion Models
Expressive Bottleneck
Soft Mixture Denoising (SMD)
Primary area: generative models
TL;DR: We proved that current diffusion models have limited approximation capabilities and proposed soft mixture denoising (SMD), an expressive and efficient backward denoising model.

Abstract: Because diffusion models have shown impressive performances in a number of tasks, such as image synthesis, there is a trend in recent works to prove (with certain assumptions) that these models have strong approximation capabilities. In this paper, we show that current diffusion models actually have an expressive bottleneck in backward denoising and some assumption made by existing theoretical guarantees is too strong. Based on this finding, we prove that diffusion models have unbounded errors in both local and global denoising. In light of our theoretical studies, we introduce soft mixture denoising (SMD), an expressive and efficient model for backward denoising. SMD not only permits diffusion models to well approximate any Gaussian mixture distributions in theory, but also is simple and efficient for implementation. Our experiments on multiple image datasets show that SMD significantly improves different types of diffusion models (e.g., DDPM), espeically in the situation of few backward iterations.

Published as a conference paper at ICLR 2024
SOFT MIXTURE DENOISING: BEYOND THE EXPRES- SIVE BOTTLENECK OF DIFFUSION MODELS
Yangming Li, Boris van Breugel, Mihaela van der Schaar Department of Applied Mathematics and Theoretical Physics University of Cambridge y18 cam. uk
ABSTRACT
Because diffusion models have shown impressive performances in number of tasks, such as image synthesis, there is a trend in recent works to prove (with certain assumptions) that these models have strong approximation capabilities. In this paper, we show that current diffusion models actually have an expressive bot- ıleneck in backward denoising and some assumption made by existing theoretical guarantees is too strong. Based this finding, we prove that diffusion models have unbounded errors both local and global denoising. In light of our the- oretical studies, we introduce soft mixture denoising (SMD), an expressive and efficient model for backward denoising. SMD not only permits diffusion models well approximate any Gaussian mixture distributions in theory, but also is sim- ple and efficient for implementation. Our experiments on multiple image datasets show that SMD significantly improves different types of diffusion models (e.g.. DDPM), espeically in the situation of few backward iterations.
INTRODUCTION
Diffusion models (DMs) (Sohl-Dickstein et al. (2015) have become highly popular generative mod- els for their impressive performance in many research iiciiinn-incuuingg high-resolution image
synthesis (Dhariwal & Nichol 2021), natural language generation (Li et al. 2022), speech process- ing (Kong et al.] 2021), and medical image analysis (Pinaya et al. 2022).
Current strong approximator theorems. To explain the effectiveness of diffusion models, re- cent work (Lee et al., 2022a bj Chen et al. 2023) provided theoretical guarantees (with certain assumptions) show that diffusion models can approximate rich family of data distributions with arbitrarily small errors. For example, Chen et al. (202)) proved that the generated samples from dif- fusion models converge (in distribution) to the real data under ideal conditions. Since it is generally intractable to analyze the non-convex optimization of neural networks, a potential weakness of these works is that they all supposed bounded score estimation errors, which means the prediction error of denoising functions (i.c.. reparameterized score functions) are bounded.
Our limited approximation theorems. In this work, we take first step towards the opposite direction: Instead of explaining why diffusion models are highly effective, we show that their ap- proximation capabilities are in fact limited and the assumption of bounded score estimation errors (made by existing theoretical guarantees) is too strong.
In particular, we show that current diffusion models suffer from an expressive bottleneck the Gaussian parameterization of backward probability Po (x(x11 xx) is not expressive enough to fit the (possibly multimodal) posterior probability ((x-11 xt). Following this, we prove that diffusion models have arbitrarily large denoising errors for approximating some common data distributions q(xo) (e.g., Gaussian mixture), which indicates that some assumption of prior works-bounded score estimation errors-is too strong, which undermines their theoretical guarantees. Lastly and importantly, we prove that diffussion models will have an arbitrarily large error in matching the learnable backward process Pa (XyrT) with the predefined forward process q(Xe:r), even though matching these is the very optimization objective of current diffusion models (Ho et al., 2020) Soog al. 2021b). This finding indicates that diffusion models might fail to fit complex data distributions.

---

Published as a conference paper at ICLR 2024
(a) Baseline: vanilla LDM; FID: 11.29.
(b) Our model: LDM w/ SMD; FID: 6.85.
Figure 1: SMD improves quality and reduces the number of backward iterations. Results for CelebA-HQ 256 x 256 with only 100 backward iterations, for LDM with and without SDM. SDM achieves better realism and FID. Achieving the same FID with vanilla LDM would require 8x more steps (see Fig. 33 Note that SMD differs from fast samplers (e.g., DDIM (Song et al. 2021a) and DPM (Lu et 2022)): while those methods focus on deterministic sampling and numerical stability, SMD improves the expressiveness of diffusion models.
Our method: Soft Mixture Denoising (SMD). In light of theoretical findings, propose Soft Mixture Denoising (SMD), which aims to represent the hidden mixture components of the posterior probability with continuous relaxation. We prove that SMD permits diffiusion models to accurately approximate any Gaussian mixture distributions. For efficiency, we reparameterize SMD and derive an upper bound of the negative log-likel for optimization. All in all, this provides a new backward denoising paradigm to the diffusion models that improves expressiveness and permits few backward iterations, yet retains tractability.
Contributions. In summary, our contributions are threefold:
In terms of theory, we find that current diffusion models suffer from an expressive bottle- neck. We prove that the models have unbounded errors in both local and global denoising, demonstrating that the assumption of bounded score estimation errors made by current theoretical guarantees is too strong;
2. In terms of methodology, Not only does SMD permit the diffusion models to accurately fit Gaussian mixture distri- butions, but also simple and efficient to implement;
introduce SMD, expressive backward denoising model.
3. In terms experiments, ity of different diffusion models (DDPM (Ho et al. 2020). DDIM (Song et al. 2021a), ADM (Dhariwal & Nichol, (02T), and LDM (Rombach al. 2022)). especially for few backward iterations-set Fig. [or a preview. Since SMD lets diffusion models achieve competitive performances at a smaller number denoising steps, can speed up sampling and reduce the cost of existing models.
show that SMD significantly improves the generation qual-
2 BACKGROUND: DISCRETE-TIME DIFFUSION MODELS
In this section, briefly review the mainstream architecture of diffusion models in discrete time (e.g.. DDPM (Ho et al.) 22020). The notations and terminologies introduced below are necessary preparations for diving into subsequent sections.
2

---

Published as a conference paper at ICLR 2024
A diffusion model typically consists of two Markov chains of T steps. One of them is the forward process- also known as the diffusion process- which incrementally adds Gaussian noises to the real sample Xo E RR,, D E N, giving chain of variables x1.7 = [x] X2,"""
qx1T xo) I g(x, xx-1), g(x: xx-1) = =N(x;; V B,x;-1.
(I)
where N denotes Gaussian distribution, represents identity matrix, and 92, 7 are predefined variance schedule. By properly defining the variance schedule, the last variable Xy will approximately follow normal Gaussian distribution.
The second part of diffusion models is the backward (or reverse) process. Specifically speaking, the process first draws initial sample xT from standard Gaussian p(xt) (N and then gradually denoises it into sequence of variables XT-100 = Xx ;x]]
(x(xx:)) p(x)) I po (x--1 xt), po (x xx) (xx o.J),
where o,I is predefined covariance matrix and μg is a learnable module with the parameter 0 to predict the mean vector. Ideally, the learnable backward probability po(x,-1 x,) is equal to the inverse forward probability q(x+ xx) at every iteration i [1, such that the backward process just a reverse version of the forward process.
Since the exact negative log-likelihood B[- [x)) is computationally intractable, common practices adopt its upper bound as the loss function
_xx=q(xo) |- log pe (xo)] < Eg[Pku[g(xr' xo). (().((())]] + Egg log (xo x1]]
(3)
Bg [DKL[g(xx-11 X1, Xo), po (x
xx]]] f.
LeeT
C-1
where DKL. denotes the KL divergence. Every term of this loss has an analytic form so that it is computationally optimizable. Нo et al. (2020) further applied some reparameterization tricks the loss L for reducing its variance. As result, the module p4g is reparameterized as
Mo (x( t)
1
B. a
to(x),t))
(4)
at
where a₁ = 1- Br. a₁ = = -I' ar, and Eg parameterized by neural networks. Under this popular scheme, the loss L is finally simplified as
T
oxo
e- colva + (T,
(5)
where the denoising function eg is tasked to fit Gaussian nosie c.
3 THEORY: DMs SUFFER FROM AN EXPRESSIVE BOTTLENECK
In this section, we first show that the Gaussian denoising paradigm leads to an expressive bottleneck for diffusion models to multimodal data distribution ((xa). Then, we properly define error Me. & that measure the approximation capability of general diffusion models and prove that they can both be unbounded for current models.
3.1 LIMITED GAUSSIAN DENOISING
The core of diffusion models is let the learnable backward probability pa(x--1 xx) at every iteration fit the posterior forward probability q(x;-1 xx). From Eq. (2). we see that the learnable probability is configured
simple Gaussian A pa (x,, t), ozI). While this setup is analyt- ically tractable and computationally efficient, our proposition below shows that its approximation goal q(x,-1 x,) might be much more complex.
3

---

Published as a conference paper at ICLR 2024
Proposition 3.1 Gau-Saus Inverse Probability). For the diffusion process defined in Eq. (7) suppose that the real data follow Gaussian mixture: q(xo) _ WxN(Xx; H3. Ex), which consists of K Gaussian components with mixture weight Whs mean vector dkx and covariance matrix En. then the posterior forward probability g(xt-1 xx) at every iteration t E [1,7] is another mixture of Gaussian distributions:
K
x x1) = Σ W Ex)
(6)
where w/. 14, depend on both variable and Jac.
Remark 3.1. The Gaussian mixture in theory is universal approximator of smooth probability densities (Dalal & Hall 1983, Goodfellow et al., 2016). Therefore, this proposition implies that the
posterior forward probability g(Xx-1 Xx) can be arbitrarily complex.
Proof. The proof to this proposition is fully provided in Appendix
While diffusion models perform well in practice, we can infer from above that the Gaussian de- noising paradigm po(x;-1 xx) = (N(x-11: (x,, t), oxI) causes a bottleneck for the backward probability to the potentially multimodal distribution g(x,-1 xt). Importantly, this problem is not since real-world data distributions
commonly non-Gaussian and multimodal. For exam-
ple, classes in a typical image dataset are likely to form separate modes, and possibly even multiple modes per class (e.g. different dog breeds).
The posterior forward probability g(xx-1 xx) can be arbitrarily complex for the Gaussian backward probability pe ( x,) = (xx=111 11:::a4ss (x(,,, o.I) to approximate, We call this problem the expressive bottleneck of diffusion models.
Takeaway:
3.2 DENOISING AND APPROXIMATION ERRORS
To quantify the impact of this expressive bottleneck, we define two error measures terms of local and global denoising errors, i.e., the discrepancy between forward process ((xu:)) and backward process pa( (Xo:r).
Derivation of the local denoising error. Considering the form of loss term L.-1 in Eq. 3). apply the KL. KI. divergence to estimate the approximation error of every learnable backward probability
xe). po (x,-1 x,)]- In Appendix A we prove that the single-step backward model [=(x:-1 x.) is optimized towards
po(xi-1
x, x E [1, 7) to its reference q(x,-1 xx) as DKL[q(xx-)
q(x,-1 X for training with loss L. Since the error depends on variable Xr, we normalize it with density ((x,) into Fl1 = Sx, <(x,)DKL l1xx. Importantly, we take the infimum of this error the parameter space θ inface ((xx)DDRL. [a(-), po (-)]dx;), which means neural networks globally optimized. In light of the above derivation, have the below definition.
Definition 3.1 (Local Denoising Error). For every learnable backward probability po (x x,), 1 t & T in a diffusion model, its error of best approximation (i.e., parameter is glob- ally optimized) to the reference qxx+-1 x,) is defined as
inf M₁
x1), po()-po xx)1l)
dee
(f
inf le8
g(x+)
DKL[g(x4-1 xx), Se).po(xe (x x dx:))
Density Weight
Denoising Error w.t.t the Input x
where space θ represents the set of all possible parameters. Note that the inequality Me > 0 always
holds because KL divergence is non-negative
Significance of the global denoising error. Current practices (Ho et al. 2020) expect the backward process po (Xy:r) to exactly match the forward process q( x such that their marginals at iteration 0 are equal: q(xo) process as the reverse-time diffusion equation. Hence, we have the following error definition to measure the global denoising capability of diffusion models.
(x().). For example, (ong et al. (2021b) directly configured the backward
4

---

Published as a conference paper at ICLR 2024
Definition 3.2 (Global Denoising Error). The discrepancy between learnable backward process Po(Xo:T) and predefined forward process (xXu:r) is estimated as
E inf (DKLl((XO.T)) (Xo:r) )1). BEB
(8)
where again & 3/ 0 always holds since KI. divergence is non-negative.
3.3 LIMITED APPROXIMATION THEOREMS
In this part, we prove that the above defined errors are unbounded for current diffusion models
Theorem 3.1 (Uniformly Unbounded Denoising Error). For the diffusion process defined in Eq. and the Gaussian denoising process defined in Eq. 2 there exists a continuous data distribution xxx)) (more specifically, Gaussian mixture) such that M, uniformly unbounded- given any real number N E IR, the inequality M, >> N holds for every denoising iteration t E [1,T].
Proof. We provide complete proof to this theorem in Appendix C
The above theorem not only implics that current diffusion models fail to fit some multimodal data distribution (xx)) because of their limited expressiveness in local denoising. but also indicates that the assumption of bounded score estimation errors (i.e., bounded denoising errors) is too strong.
Consequently, this undermines existing theoretical guarantees (Lee et al., 2022a Chen et al. 2023 that aim to prove that diffusion models are universal approximates.
The denoising error M₁ of current diffusion models can be arbitrarily large at every denoising step t E [1, 7j. Thus, the assumption of bounded score estimation errors made by existing theoretical guarantees is too strong.
Takeaway:
Based on Theorem 3.1 and Proposition 3.1 we finally show that the global denoising error & of current diffusion models is also unbounded.
Theorem 3.2 (Unbounded Approximation Error). For the forward and backward processes respec- rively defined in Eq. ( and Eq. (2). given any real number N R. there exists continuous data distribution ((xo) (specifically, Gaussian mixture) such that N.
Proof. A complete proof to this theorem is offered in Appendix
Since the negative likelihood E[ log [xa)] is computationally feasible, current practices (e.g., DDPM (Ho al. 2020) and SGM (Song al. 202 (b) optimize the diffusion models by matching the backward process pa (o.T) with the forward process y g(xo.T). This theorem indicates that this optimization scheme will fail for some complex data distribution g
Why diffuuion models already perform well in practice. The above theorem may bring uncase- how can this be true when diffusion models are considered highly-realiste data generators? The key lies in the number denoising steps. As indicated Eq. (29), the more steps are used, the more the backward probability, Eq. (2) is centered around single mode, hence the more the simple Gaussian assumption holds (Sohl-Dickstein et al., (015). As a result, we will see in Sec. Sec.5.3 th3 our own method, which makes no Gaussian posterior assumption, improves quality especially for few backward iterations.
Standard diffusion models (e.g. DDPM) with simple Gaussian denoising poorly approximate some multimodal distributions (e.g. Gaussian mixture). This is problematic, these distributions are very common in practice.
Takeaway:
It is also worth noting that these emrors already overestimate the performances of diffusion models, since their definitions involve infimum operation infoee.
5

---

Published as a conference paper at ICLR 2024
4 METHOD: SOFT MIXTURE DENOISING
Our theoretical studies showed how current diffusion models have limited expressiveness to approx- imate multimodal data distributions. To solve this problem, we propose soft mixture denoising (SMD), a tractable relaxation of a Gaussian mixture model for modelling the denoising posterior.
4.1 1
MAIN THEORY
Our theoretical analysis highlight an expressive bottleneck of current diffusion models due to its Gaussian denoising assumption. Based on Proposition 3.1, an obvious way to address this problem directly model the backward probability p= Xr) Gaussian mixture. For example, we could model:
mixture Pe (x--1
xx)
zo, (x,, (x4 (Xt, 1), Eo, (xx;1))
where 0 = UK_ Oz. the number Gaussian components K is hyperparameter, and where weight =f +.), mean 12 (-), and covariance Σg (.) are learnable and determine each of the mixture compo- nents. While the mixture model might be complex enough for backward denoising, it is not practical for two reasons: 1) it is often intractable to determine the number of components K from observed data; 2) mixture models notoriously hard to optimize. Actually, Jin al. (2016) proved that Gaussian mixture model might be optimized into an arbitrarily bad local optimum.
Soft mixture denoising. To efficiently improve the expressiveness of diffusion models, in- troduce soft mixture denoising (SMD) pr (x.-1 xt). soft version of the mixture model pe (.))xture.),, which avoids specifying the number of mixture components K and permits effective optimization. Specifically, we define a continuous latent variable z,, as an alternative to mixture weight =/. that represents the potential mixture structure of posterior distribution q(x,-1 | x.). Under this scheme, we model the learnable backward probability as
SMD (-) - 10 SMD (x xx)dzz = SMD xx) SMD (x Xx, de P8
(10)
where I denotes the set of all learnable parameters. We model pa multivariate Gaussian and expect that different values of the latent variable z, will correspond to differently parameterized Gaussians:
Xx, 2x) learnable
SMD P9
(x:11 x,, z.) =
t).
())
(11)
where 0 0 is set of vanilla learnable parameters and fo(z, t) is another collection of parameters
computed from neural network fa with learnable parameters & = 0. Both 0 and fo (z,, i) constitute the parameter set of mean and covariance functions μ4, Σ. for computations, but only and ø will be optimized. This type of design similar to the hypernetwork (Ha et al., 2017 Krueger et al., 2018). For implementation, we follow Eq. (2) to constrain the covariance matrix E. to the form 0, and parameterize mean ut, (x,,)) () similar to Eq. (4):
1 _(xx- Va,
B. (oU 1x::
(o (x,,,
(12)
VI ar
where e, is neural network. For image data, we build it as a U-Net (Ronneberger et al. 2015) (i.e., 0) with several extra layers that computed from (z(ze:
For the mixture component Pa(z, xt), we parameterize it with a neural network such that it can be an arbitrarily complex distribution and adds great flexibility into the backward probability SMD
pe (x:-1 xx). For implementation, we adopt a mapping 9g tt, - z,, = 0 with
I.d.d. "
(O 1). which converts standard Gaussian into non-Gaussian distribution.
Theoretical guarantee. We prove that SMD MMD xx) improves the expressiveness of diffusion models resolving
models--rsooling the limitations highlighted in Theorems and 3.2
Theorem 4.1 (Expressive Soft Mixture Denoising). For the diffusion process defined in Eq. [T), sup- pose soft mixture model p (SMD(x,- xx) is applied for backward denoising and data distribution
((x)) is Gaussian mixture, then both Mr 0, 0.40e [1,T] and E 0 hold.

---

Published as a conference paper at ICLR 2024
Algorithm Training
Algorithm 2 Sampling
1: repeat
xy p(xr) = (O, 2: for t = T...., do
2:
Xo ~ g(xo)
3:
[ <~ [1.TT) (O.II
3:
(0 ift else = 1 (0
4:
X, vū;X0 v1 dGE
5:
7 (O,
Latent variable sampling: z, 96 (1 x1.
Latent variable sampling: a. 96 (7777: xx,
Param. for computation: / 0 fe(ez))
7: Param. for computation: 0 (a(a,1))
7: X -
(xt
L-aL <1-
cg (Xx, + OSE
8: Param. to optimize: - U
Val
8: end for
9: Update w.r.t. Vole eg (x,,
9: return Xo
until converged
Proof.
The proof to this theorem is fully provided in Appendix
E
Remark 4.1. The Gaussian mixture is universal approximator for continuous probability distri- butions (Dalal & Hall 1983). Therefore, this theorem implies that our proposed SMD permits the diffusion models well approximate arbitrarily complex data distributions.
Takeaway: Soft mixture denoising (SMD) parameterizes the backward probability tinuously relaxed Gaussian mixture, which potentially permits the diffusion models to well approximate any continuous data distribution.
4.2 EFFICIENT OPTIMIZATION AND SAMPLING
While Theorem 4.1 shows that SMDs are highly expressive, it assumes the neural networks are globally optimized. Plus, the latent variable in SMD introduces more complexity the computation and analysis of diffusion models. To fully exploit the potential of SMD, we thus need efficient optimization and sampling algorithms.
Loss function. The negative log-likelihood for a diffusion model with the backward probability SMD x x,) of latent variable model is formally defined as Pa
E, [- In p
SMD (xo)] Bxo= In ( p(xr) (=T
11
SMD pe
(X xx) x))xx1 (13)
Like vanilla diffusion models, this log-likelihood term is also computationally infeasible. In the following, we derive its upper bound for optimization.
Proposition 4.1 (Upper Bound of Negative Log-likelinhe likeliikiih.od.. Suppose the diffusion process defined
as Eq. ( and the soft mixture model pe (x, xx) is applied for backward denoising, then upper bound of the expected negative log- ligeelild E, In Inp,
SMD (xq)] is
LSMD_ = =C Σ
Bne..x, r.|e- EoL (Varx + VI- ((,,
(14)
1-1
where g-(-) 0: V0xX0 + +I +I- C a constant that does not involve any learnable parameter = 0 US, Xo ~ g(xo), n, are two independent variables drawn from standard Gaussians, and P, 12/(20101e1 - ā.)).
Proof. The detailed derivation to get the upper bound LSMD is in Appendix F
Compared with the loss function of vanilla diffusion models, Eq. (5) our upper bound mainly differs in the hypernetwork f parameterize the denoising function e, and expectation operation E,

---

Published as a conference paper at ICLR 2024
DD DDW
DDdPN w. SMID
Training Data
10 training steps 5x10* training steps 10t training steps
Loss Dynamics
Visualising the expressive bottleneck of standard diffusson models. Experimental results on synthetic dataset with 7 x 7 Gaussians (right), for DDPM with 7' = 1000. Even though DDPM has converged, we observe that the modes are not easily distinguishable. On the other hand, SMD converges much faster and results in distinguishable modes.
Figure 2:
Table I: SMD consistently improves generation quality. FID score different models across common image datasets and resolutions. We use T 1000 for all models.
Dataset Model
DDPM
DDPM w/ SMD
ADM
ADM w/ SMD
CIFAR-I0 (32 32)
3.78
3.13
2.98
2.55
SSUN-Conference (64 64)
4.15
3.52
3.85
3.29
LSUN-Church (64 x 64)
3.65
3.17
3.41
2.98
CelebA-HQ (128 128)
6.78
6.35
6.45
6.02
The former is computed by neural networks and the latter is approximated by Monte Carlo sampling, which both add minor computational costs.
Training and Inference. The SMD training and sampling procedures are respectively shown in Algorithms D and 2 with blue highlighting differences with vanilla diffusion. For the training pro- cedure, we follow common practices of (Ho et al. 2020; Dhariwal & Nichol 2021), and (1) apply Monte Carlo sampling to handle iterated expectations E Eyy.e.x in Eq. (4). and (2) reweigh loss term
L-- 6, (x,, I) by ignoring coefficient F,, One can also sample more noises (e.g., m)) in one training step to trade run-time efficiency for approximation accuracy. The source code of this work is pub- licly available at a personal repository: https//gittub.com///uist33..
and our lab repository:
https://github.c http://iituub.coommaneeeeee comm/vandeershaarlbbssddd
EXPERIMENTS
Let us verify how SMD improves the quality and speed of existing diffusion models. First, a toy example to visualise that existing diffusion models struggle to learn multivariate Gaussians, whereas SMD does not. Subsequently, we show how SMD significantly improves the FID score across different types of diffusion models (e.g.. DDPM, ADM (Dhariwal & Nichol, (02T). LDM) and datasets. Then, we demonstrate how SMD significantly improves performance low num- ber of inference steps. This enables reducing the number of inference steps, thereby speeding up generation and reducing computational costs. Lastly, we show how quality can be improved even further by sampling more than one for loss estimation at training time, which further improves the performance causes an extra time cost.
5.1 VISUALISING THE EXPRESSIVE BOTTLENECK
From Proposition 3.1 and Theorems 3.2 3.1 it follows that vanilla diffusion models would strug- gle with learning à Gaussian Mixture model, whereas Theorem 4.1 proves SMD does not. Let us visualise this difference using simple toy experiment. In Figure 2 plot the learnt distribution of DDPM over the training process, with and without SMD. We observe that DDPM with SMD converges much faster, and provides accurate distribution at time of convergence.
8

---

Published as a conference paper at ICLR 2024
5.2 SMD IMPROVES IMAGE QUALITY
We select three of the most common diffusion models and four image datasets to show how our proposed SMD quantitatively improves diffusion models. Baselines include DDPM Нo al. (020)) ADM (Dhariwal & Nichol (02T); and Latent Diffusion Model (LDM) (Pinaya et al. 2022). Datasets include CIFAR-10 (Krizhevsky
LSUN-Church (Yu al. 2013), and CelebA-HQ (Liu et al. 2013). For all models, we set the backward iterations T as 1000 and generate 10000 images for computing FID scores.
al., 2009),
In Table [ we show how the pro- posed SMD significantly improves both DDPM and ADM on all datasets, for a range of resolu- tions. For example, SDM outper- forms DDPM by 15.14% on LSUN- Church and ADM by 16.86%. Sec- ond, in Table 2 we include results for high-resolution image datasets, Fig. [ for example images (T' = 100). Here we employed LDM as baseline to reduce memory footprint, where we use a pretrained and frozen VAE. We observe that SMD improves FID scores significantly. These results strongly indicate how SMD is effective in improving the performance for different baseline diffusion models.
Table 2: SMD improves LDM generation quality. FID score of latent diffusion with and without SMD on high- resolution image datasets (T 1000).
Dataset Model
LDM
LDM w/ SMD
LSUN- -SUN-Church (256 x 256)
5.86
5.21
CelebA-HO (256 256)
6.13
5.48
5.3 SMD IMPROVES INFERENCE SPEED
Intuitively, for few denoising iterations the distribution q(1-1 x,) is more of a mixture, which leads to the backward probability Pa(x--1 x simple Gaussian-beiin being worse approximation. Based on Theorems [.T] we anticipate that our models will be more robust to this effect than vanilla diffusion models.
Met
LEN SMD
The solid blue and red curves in Fig. B respectively show how the Fl scores of vanilla LDM and LDM w/ SMD change with respect to increasing backward iterations. We can see that our proposed SMD improves the LDM much more at fewer backward iterations (e.g.. T = 200). We also include LDM with DDIM (Song et al. 2021a), a popular fast sampler. We see that the advantage of SDM consistent across samplers.
Figure 3: SMD reduces the number of sampling steps. Latent DDIM and DDPM for different iterations on CelebA-HO (256 x 256).
5.4 SAMPLING MULTIPLE 7y: QUALITY-COST TRADE-OFF
In Algorithm we only sample one at a time for maintaining high computational efficiency. We sample multiple 1) to estimate the loss better. Figure 4 shows how the training time of one training step and FID score DDPM with SMD changes as a function of the number of 17 samples. While the time cost linearly goes up with the increasing sampling times, FID monotonically decreases (6.5% for 5 samples).
6 FUTURE WORK
We have proven that there exists an expressive bottleneck in pop- ular diffusion models. Since multimodal distributions are so com- mon, this limitation does matter across domains (e.g., tabular. im- ages, text). Our proposed SMD, as general method for expressive backward denoising, solves this problem. Regardless of network architectures, SMD can be extended to other tasks, including text- to-image translation, video generation, and speech synthesis. Be- cause SMD provides better quality for fewer steps, we also hope it will become standard part of diffusion libraries, speeding up both training and inference.
Number
Figure 4: SMD quality is
further improved by sam- pling multiple 77, see Alg. [ on
(64
64) for DDPM w/ SMD.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank the anonymous ICLR reviewers for their kind and constructive reviews, Yangming Li also thanks Accenture for their sponsorship and support.
REFERENCES
Peter Ahrendt. The multivariate gaussian probability distribution. Technical University of Denmark, Tech. Rep, pp. 203, 2005.
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Representations, 2023. wttptttts///peeeeeeeeee
net/forum? wtt.rrmmmmddyyyyyy20000000
SR Dalal and WJ Hall, Approximating priors by mixtures of natural conjugate priors. Journal of the Royal Statistical Society: Series (Methodological), 45(2):278-255 1983.
Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Annals of operations research, 134,19-67, 2005.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer. Y. Dauphin, P.S. Liang. and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 8780-8794. Curran Asso- cc cc/papprr/02///11e.
ciates, Inc., 2021. URL http:///pppooeeeeenge.eeeee pdf
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
David Ha, Andrew M. Dai, and Quoe V. Le. Hypernttworks. In International Conference Learn ing Representations, 2017. URL
net/ htte//oormmiideee wtd.rummmdddrrreeeee11
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
Neural Information Processing Systems, 33:6840-6857 2020.
Marco F Huber, Tim Bailey, Hugh Durrant-Whye Whyte, and Uwe D Hanebeck. On entropy approxima- tion for gaussian mixture random vectors. In 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pp. 181 188-188 IEEE, 2008
Chi Jin, Yuchen Zhang, Sivaraman Balakrishnan, Martin J Wainwright, and Michael I Jordan. Local maxima in the likelihood of gaussian mixture models: Structural results and algorithmic conse- quences. Advances in neural information processing systems, 29, 2016.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-2663 2022.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learming Representations, 2021.
httprrewwwwwwweeeeeeeeeeeeeeeeee www.:oorrmmi/eeeeeeeeeeeee.r x xF Ymz
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks, 2018. htttts:://penreriew.ntttt S1fcY-20-
Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of score-based generative modeling for general data distributions. In NewrIPS 2022 Workshop on Score-Based Methods, 2022a. URL httpps://spenreviee... net wwwp:/eee/fooommidssttomo 9 9m8
10

---

Published as a conference paper at ICLR 2024
Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for boore-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems, 05:22870-22882, 2022b.
Xiang Lisa Li, John Thickstun, Ishaan Guirajani, Percy Liang, and Tatsunori Hashimoto. Diffusion- LM improves controllable text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: ftttp:://openrevieee net forum?id=dsm
Ziwei Liu, Ping Luo. Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
Walter HL Pinaya, Petru-Daniel Tudosiu, Jessica Dafflon, Pedro F Da Costa, Virginia Fernandez, Parashkev Nachev, Sebastien Ourselin, and Jorge Cardoso. Brain imaging generation with la- tent diffusion models. In MICCA/ Workshop on Deep Generative Models, pp. 117-126. Springer, 2022.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learn- ing, volume 37 of Proceedings of Machine Learning Research, pp. 1530-1538, Lille, France, 07-09 Jul 2015. PMLR. URL httt:s::///rcceeddnng. m press www.pppssss/33311m vv7 3277reeeedde1... html
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assisted Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part III 18, pp. 234-241. Springer, 2015.
Peter J Rousseeuw and Annick M Leroy. Robust regression and outlier delection. sons, 2005.
John wiley &
Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILE mobile computing and communications review, 5(1):3-55, 2001.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibriun thermodynamics. In International Conference on Machine Learn- ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021a. URL
net/
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pp. 574 584. PMLR, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021b. URL atttps://openreviev.. net/
Corum?1.=PXTTG112RR
11

---

Published as a conference paper at ICLR 2024
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1106.00330 2015.
Yufeng Zhang, Wanwei Liu, Zhenbang Chen, Kenli Li, and Ji Wang. On the properties of kullback- leibler divergence between gaussians. arXiv preprint arXiv: 20n::2202.05855 2021.
12

---

Published as a conference paper at ICLR 2024
A SIGNIFICANCE OF THE LOCAL ERROR M₁
In this part, we show that the backward model pe(t 1 t), [1, is optimized towards the inverse probability g(t 1 t) in terms of the loss function L as defined Eq. (3)
Proposition A.1. For a perfectly optimized reverse process {po(t ea1.1 (as defined in Eq. (2)) that minimizes the loss function L (as defined in Eq. (3), equality po ( (1) q(1
holds for every denoising iteration E [1
Proof Following DDPM (Ho et al.
2020) and SGM (Song & Ermon 2019 Song et al. 2020
Karras et al. 2022). the loss function L can be formulated as
Pe (X).T)
C E [xxrr
Ex (x(x)) (x)) I [(xo]] (15)
g( (x1.T xo)
Note that the second expectation term EE[] is a constant and the first KL-divergence term D reaches its minimum when po (xp:r) equals ((x:r)) Therefore, for perfectly optimized diffusion model, we have po(Xo.7) (o:r) = g(Xo.T). Then, for every iteration I E [1,7]. we get
Pa (x
Pu (Xo:T -dx0==2dxx++0 2dx,+1.7 = (xx0
x(x),1,x,). (16)
Similarly, we also have the following equality:
px().) po -1er))xxxxxxxx///.
((xx g(xx).
(17)
Based on the above two equations, we finally derive
pe(X(-1,Xe)
g(x(-1,X)) g(xx-1 xx),
(x xx)
(18)
(x()
((x))
which proves the proposition.
The above conclusion indicates that the definition of local denoising error M, is proper for its use: quantifying the performance of backward module po 1 t).
B PROOF OF PROPOSITION 3.1
By repeatedly applying basic operations (e.g., chain rule) of probability theory conditional distri- bution of backward variable g(x(-1 xt), we have
g(xt-1 xx) = qx,,, x1-1) q(x; x x(-1)g(x(11)
g(xx x(-1)
[
g()x-1,x0ddx0
g(xi)
y(x:)
g(xe)
(19)
((x,) v(xx x(-1)
g(x--1 xo)g(xo)dxo
Jxg
Based on Eq. (1 and g(x; xo) N(xi - from (Ho al. 2020), posterior
probability g(x;11 Xt) can be expressed as
q(x--1 xx, Nxx:; vI Bix. 311)
M Ti (ggx))))()))dx))) (20)
((x)
Note that for multivariate Gaussian, the following holds:
N(x; A4, E) (2T)
- exp
(x Tx) 1x 211)
2
(2
Σ
(m
x ,) E
(1(2)) 1--(0 (4 x (--
(21)
exp
22)
13

---

Published as a conference paper at ICLR 2024
where À RR, je denotes vector with dimension D, and Σ is positive semi-definite matrix. Fromt that, and B = 1 - Cgg. the following identities follow:
N(xi V1 Bix, 3x1) at & N (x--1 Vas at
X 1- 1-a11)
(22)
ar-1)l) (ixx--ivaa--ee (1-- (ac-1) ( &N((xo:
XL-1 1- -
va-11 az~1
Therefore, refomulate Eq.
200g as
<(-) (a,a;-1) + N 11-15 N(xx X2
N N(x0:
X1-1
1 1-at-1) (xx))xx0 (23)
(xx))
va--1
ar-1
Now, we let (Xx) be a mixture of Gaussians ((xo) = >xx WxN(xxo pe: Ex), where K is the number of Gaussian components, wi E [0, 1J. En Wi = 1, and vector μ4 and matrix En respec- tively denote the mean and covariance of component k.
For the the mixture of Gaussians distribution g(Xn) and by exchanging the operation order of sum- mation > and integral Sxg' we have
K Ww w
X
1 - 1-mr]
((x,-1
x1)
N X1-1;
(xx)
Vai"
(24)
N
(xx:
Xo:
Xe-
1 ār- 1).
Xoi pax En _x) dxo
var-1 a:-1
Xa
nice property of Gaussian distributions is that the product of two multivariate Gaussians also follows Gaussian distribution (Ahrendt 2005). Formally, have
N(x; pI, E, (xx E -- #1 + Eg (2), (ET') カーリ
(xx u2: Σg NN4z; pi, E + Σ2)
(25)
where 11, 142 vectors of the same dimension and E1, Eg positive-definit matrices. There- fore, the integral part Sxo in Eq. (24) can be computed as
N
Xoi
X
1-ac-1
Xni pex Ex dxo 1- 06-11 N
va--1
ō4-1
N
pkk
X1-1
1- or
0
I+ Exe
N(xo: :)xo;;;)dxx
(26)
vār=1"
ō4-1
X0
(a=-1)
N
/ 1t-1MR (1 ō.-1) + ā Ex)
where the last equation is derived by Eq.
21
With this result, we have
K
D W
((x--1 xx)
Σ
II +
(27)
(xx)
By applying Eq. (25) and Eq. (T). and a;-10; azz the product of two Gaussian distributions in the above equality can be reformulated as
N Xt-li
X1
1-41) * *N Xt- (x / /aiiiaa-iii aax (1-ar))) 1i-1Ex
Var
ar
= N N(xx; Vāetue (1 Tz ōz Dt
=
(28)
*N(xx-1: (I A A=
+ (I+ As) -
a1-
1- (I +I+ A=1)
Va
az
14

---

Published as a conference paper at ICLR 2024
where matrix A₁- (an
ai)/(11 a1)I I 0z/(1 - -1).x. With this result, we have
g(x:-1
xx)) Σ
wxt Eiy)
k-1
w//
wk Nx v Volkk (1 -x xx
g(x;)
(29)
14 (I A )
X (I+AA)
Vai-1uu
Vas
Σx -
at (I 1 A A
where _ w/1 1. To conclude, from this equality it follows that posterior probability p(x:-1 x;) also mixture of Gaussians. Therefore, our proposition holds.
C PROOF OF THEOREM 3.1
Let us rewrite metric Mr as
Me = inf (f{ v(xi)( [ g(xc-1 xx) In g(x--1 Xx) Jx.-1
dxx- Jdx,)
6E8
po xx-1
xx)
(30)
(fc g(xi( - H[]g(x01 xx)] + Dex[[(xx-1) po xx)1)
inf 6e
where H[[.] is information entropy (Shannon 2001):
H[q(xx-11 | xx) =
a(x xx) In q
X1 dx
(31)
X
and | denotes the cross-entropy De Boer et al.
2005
Deelg(x:-11 xx), (x xt)] =
((x--1 xz) In po
x xx)dxx-1-
(32)
Xx-1
Note that the entropy term Hl.- does not involve parameter 0 and can be regarded as a normalization term for adjusting the minimum of -
Our goal is to analyze error metric MM defined in Eq. 77. Regarding its decomposition derived
in Eq. (30). we first focus on cross-entropy De xx). x.)]. Suppose q( Xg
follows Gaussian mixture, then x(x,-1 x,) is also such distribution as formulated in Eq. 29
Therefore, we can expand the above cross entropy DcE as
DeEl.
q(x--1
x²) In X dxx-11
Xx--1
(x=N((x-1::1 En,) In po(xe-1 xx)dxx-1
X1-1
(33)
K
Σ
w/ DeEDDL E%e po xi)]
K
K
Σ
w/. DKL
(xx (N
Ex po))po xx)] + Σ
w WHH.N [[[[[((xx) (x:-15 plg. E'1
Suppose Σg
I 5k
0, then have
1 + (ōx 1)xx-11
(1 -
v 0x +
1+ (d, 1)a+ #1
1 + (ōz- 1)at
(34)
+ (6x 1)a;-1
Σ
(1 at)
(dx 1)āx
15

---

Published as a conference paper at ICLR 2024
With this equation, simplify entropy sum Σ w/ H[[-1 as
K
K In 2 = In(2Te) +
we
K
w 2
w/ H[W (x:11 M
In -x (35)
2
Term DRI. is in fact the KL divergence between two multivariate Gaussians, (x and N (x, (,)(xx,)) 1); I, which has an analytic form (Zhang et al. 2021:
11 2x
DKL.--
= 2 In
D +
1 Ot
114e - (x f)1 + Tr((o,)) Σg
2
Σ
(36)
1 Diioo - in E D)
1 1- at 1 + (5x- 1) D
14 - t)l
2
20,
20, 1 (6% 1)a;
With the above two equalities and the fact that ā Den[1(xi-11 Po),po(x;11 xt)] as
> Tr because ot
1, we reduce term
1
K
D wkllek (x(x,, t)11 In(2ro)) + 1 at D. 20
DCEl1 []
(37)
20
Since entropy H[g(x;-1 xx)] does not involve model parameter 0, the variation of error metric Me
from cross-entropy DeEL... more specifically, sum Σk Let's focus on how this term contributes to error metric M, as formulated in Eq.
K Σ (f
ICE = [ (xx
wkll''s (x(x,, dx
wk9(x)) 1 (xx(x,,))) t)x/d(xx).. (38)
Considering that Eq. 2 and En has been set as I1,, we have
K ICE = Σ
(Je
wwN XXi Varus, (1 + (ōx - 1)a+)1) | uo(xs,,) dxxx)
k-1
(39)
K = NC()
anlva=-1 (1--a)aar-)) (1+ + (dx- 1)ār, pel (Ho(x),t)= (.)vss)) )oss
Σ
Sum 1= wk|l 2| is essentially problem called weighted least squares (Rousseeuw & Leroy, 2005 for model (xx()))) - (o)a/x,,,, which achieves a minimum error when the model is 1 (=((-) (-)(-))e. For convenience, we suppose ..K 10//t//111 + (ōx - 1)),) = 0 and we have
ICE > (s N(-)dxx
(Σ wi (-)mm
K
(
K
az) 2 a=-1
wk
(40)
+ (6x - 1)
Term M[q(x:-1 I x.) is in fact the differential entropy of a Gaussian mixture. Considering our previous setup and its upper bound provided by (Huber al. 2008), we have
HT-1
Σ
K
w/
1 In w/ + In (2mr)
D
1 + (6x 1)at-1 (1 - -a)))
2
1 (6x
1)a
k=1
(41)
K
2Te (1 oz) Σ
1 - -1) + In K
2 In
w/ In wh & 2
In
2nme
a4
where the second ineqaulity holds since (1 + s3/(1 + xy) 1/v, RR,y (0,1) and the last inequality is obtained by regarding term > as the entropy of discrete variables [w], w/y, ... ]. Therefore, its contribution to error metric Mr is
D 2ne
(2me (1 --ai)) - In K.
TEnt g(x))( xx))]xxx;
In
(42)
az
Combining this inequality with Eq. (37 and Eq. 40
we have
Me* (1 ax)
Σ
K
wk
In K
+
D)
In
0100
+
1- as
1). (43)
20
1+ (0x
1)ū,
2
Cr
16

---

Published as a conference paper at ICLR 2024
with constraint Σ 1///k/ (1 (ox- 1)ā,) 0. Since wi 0, < & K, there exists group of non-zero vectors [11, [42 px satisfying this linear equation, corresponds to a Gaussian mixture p(xo). With this result, can always find another group of solution [i Ap2,,.. A for À E IR, which corresponds to a new mixture of Gaussians. By increasing the value of x, the first term of this inequality can be arbitrarily and uniformly large in terms of iteration i.
D PROOF OF THEOREM 3.2
Due to the first-order markov property of the forward and backward processes and the fact ((xT) =
pe(xT) = NO I), oc, we first have
((x:T)
] = Eg
((xT) IT (xxx-1 1 xx)
DRL.-1 = Exo:7~((x.F))
Po Px(x)T))
Po Po(XT) (XT) [T= Po 20(x xx)
(44)
T Σ
In g(xe-1 xx)
T
= 1Eg
Σ
Exs PxLl9(xx-11 Xx). pel (x xx)]
po (x Xx)
where the last equality holds because of the following derivation:
Eg In g(x1-1 x1)
q(Xo:T) In 9 xi)
dxo:T
pa(xx- xx)
xxxxx111 Xx)
g(xi)) 9(x;--1
xx) In q(xx-1 Xx)
dxx- dxc;
(45)
Po(x,-1 x²)
x
xx
Exs o(xe)
DKI.(((x--1
xx), po (x x:)]
3.1 then we can infer that there is a continuous data distribution ((x)) such that the inequality Mr (N 1)/T holds for t [1. For this distribution, we have
Based on Theorem
DKLL-1
inf (Ex, PKL[g(> xx), po
x
Σ M, > N+1
(46)
Finally, we get E inf(DKL|l) & N 1> N for the data distribution g(xx).
PROOF OF THEOREM E
We split the proof into two parts: one for Me, [] and the other for E.
Zero local denoising errors. For convenience, we denote integral Sx. q(x;) DKL[-jdx, in the def- inition of eиrоr measure Mi Mz(0). Immediately, have M₁ inf M.(O). With this equality, it suffices to prove two assertions: (, > O, vO θ and 30 E θ ()) = 0.
The first assertion trivially true since KL divergence DKL always nen-tegative For the second assertion, we introduce two lemmas: 1) The assertion is true for the mixture model pa mixxturr (x, x.): 2) Any mixture model can be represented by its soft version yg SMD (x,-1 x.). If we can prove the two lemma, it is sufficient to say that the second assertion also holds for SMD.
We prove the first lemma by construction. According to Proposition 3.1 the inverse forward prob- ability q(x,-1 xx) is also a Gaussian mixture as formulated in Eg. (29 By selecting a proper number K, the mixture model pa pixittur (xt-1 xx) defined in Eq. 9 will be of the same distribu-
tion family as its reference q(x-1 xx), which only differ in the configuration of different mixture components. Based on Eq. 29
specifically set parameter 0 = UlEkEk 8g
28. (x., t) K WgNN(xx; (Xi: ait (1 C4 Z₁Ex)
M4.. (xr, t) (1 + A
X1
(I+ A+) 1 Va-1K4
Ckt
1 0t (I
(47)
Es, (xe,t)
A 1
0t
T
I+
at
1
aq
17

---

Published as a conference paper at ICLR 2024
such that the backward probability pa phixturee (X--1 (x-- xx) is the same its reference q(x;-1 xi) and thus DRI.(e (xxx11 Xt): Pi (x--1 xt)] by definition is O. In this sense, we also have M,(0)= = 0, which exactly proves the first lemma.
We also prove the second lemma by construction. Given any mixture model pe p xx) as defined in Eq. (9), we divide the space R-- (where L is the vector dimension of variable z,) into K disjoint subsets [2,,1 2,,,, 2r,x) such that:
SMD Pg
(Ze xx)dzz (xe,,)), O2 = folze,)) 1), Vz. E Zz,
(48)
2,,62
where & {1, K3. The first equality can be true for any continuous density pa AND and the second one can be implemented by a simple step function. By setting
Ø. we have
SMD Pg
(x1-1
xx)
Pe
SMD (ze 1i (Xe, (x,,t), t), Eo, fa (mn o) f))
SMD (ze (x t), ().() 1) dzz)
Pe
(49)
(x Mo, (xx, Eo, t)) [
pe
pg (ze xxx xz)
Ze,
(ox t..,)) Eee (xx, ()) (xx:
Pa
mixture
Xx--1 x1)
which actually proves the second lemma.
Zero global denoising error. We can see from above that there is always a properly parameterized backward probability py MDD for any Gaussian mixture (Xxu) such that q(x;-1 xx) -P
SMD
(x,-1
x1), VL E [1, 7.. Considering q(XT)
p
SMD
(xT). we have
SMD (xT-1,XT) Pg P8
SMD
(XT)P? SMD
(xT-1 xT) q(x))((() = -(xx--1,x7))
(50)
Immediately, we can get ((xT-1) = ps
SMD
(XT-1) since
ph (xT-1)
SMD Pe
(x XT
qXxT-1, )x1 q(xx-1).
(51)
Xr
XT
With the above results, we can further prove that Fi
gMD
(XT (xT 2T X,7) = q(xT XT)
and pe SMD SMMD(XT (2) q(xT-2). By iterating this process for the subscript f from T to 1, we will finally have Po (Xu:T) ) ((xur)) such that & O.
F PROOF OF PROPOSITION
While we have introduced new family of backward probability pe (x.((x.-
xx) in Eq. 10
upper bound C = >i'" Lu defined in Eq. (3) is still valid for deriving the loss function. To avoid confusion, we add a superscript SMD to new loss terms. An immediate conclusion is that LSMD because p(x;) by definition is standard Gaussian and q(xT xo) also well approximates this distribution for large T. Therefore, the focus of this proof is on terms of KL divergence CSMD < t & T and negative log- likelihood LSMD.
Based on the fact that q(x;-1 X2, Xo) has a closed-form solution:
xx Xx, Xn) = N (Xt, xo), 3.1),
(52)
where mean (x,, xx) and variance 3, are respectively defined as
V0t-181 13,
Vaill ar-1)
1 1 Ti 1-7r-11 Be:
(x,, xo) 1 α xo
X1, Br
(53)
18

---

Published as a conference paper at ICLR 2024
expand term L-
Ex]DDLL((Xx/- x(,x,) 11 P2 poDD (0, (x,-1 xx))]
g(xx-1 1 xx) In 9( 91x,-1 Xx, xo)
2-1 0(x0
dx,-1
(x(-1 xx)
(54)
F - X xo) + DCE ((xx-1 X,: xx:xo): SMD (x
xx]]
Considering our new definition of backward probability pa Jensen's inequality, we can infer
p)MD((x;-1 xx) in Eq.
100
and applying
DeEl-] -Ex;- -al 1 (x)))0))
In
SMD P0
(z
SMD xx)Pp
Xx, 2, ddz (x
J2c
-Ex, (x( (x))))
In Bg 2, p (z.
[p]
SMD
(x--1
X 2 dzz)
<
- -
- |xc. 20)
-1 aMD (2.1x,) (x)))))))))))))))))))))))))))))) p3
SMD
[x--
Xx, 2z dZz
(55)
Ex mp (xx1)
g(xx-1 X1,
SMD
(xt-1 X Ze dxx
Xx-1
Exg -yg (a (xc) Combining the above two equations, we have
DeRlu(xx-> (x-[ X1, X1) SMD
( (x1-1 xx. 2[)l
2-1 &
Ea H[
Xx, xo)] + Ex, [PeFl((x-11 Xt. xo). SMD (Xe-1 Xxx 2+)]]
= -Ey,e. H[ Xe. x0)] + DCE[9(X:-1 X x1, xo). Pa
SMD
(xx-1 X1, [i)]
(56)
SMD [x X 24) Ezz Exo x(xx))a (xo) [DKL[g(x--)) x2, xn), Pr
Considering 2, = ( m. X4, t) and applying the law the unconscious statistician (LO- simplify the above inequality as
TUS) (Rezende & Mohamed 2015),
LSMD & (.. [K,, [q(x;-1 xx. Xo), SMD (x--1 xt, x1; []]]
(57)
The inner term of expectation F, essentially the same as the old definition of LSMD in
Eq. 3), except that term Pa(-) is additionally conditional on 2μ. Hence, we follow the procedure of DDPM Ho et al. (2020) to reduce it. The result is given without proving:
B8
CMMP & C. E,,,,00
c ( + vI a1,
20,a a.)
(58)
fo(:) følge (n, v T6x0 + v1 ã.e, t),
where C, a constant, n, c NN(0, I). , and parameters 0, db E are learnable.
For the negative log-likelihood LSMD = Eyl- In PSMD (xu x1)], we expand it as
[- In (S
L Ex Xi x((x))((x1)x)))
Po (MDD(2)
x x1)r
SMD SMD (xo X1, 21 21)1))
(59)
By applying Jensen's 's inequality, we have
Exx.x1
Pe (z) x1) In p
SMD
SMD (xo X1, 21 )2z
Exoxx
Ex Eni
Inpo x1,,
(60)
xx)
C + Ep BMD xx))
Exx.x
X
(o.folmn.t) (x) 1x1, 1)
20
where C is constant that does not involve with the model parameter 0 U US. Considering Eq. ( and Eq. (12). we can convert this inequality into
87
SMD
& C + Ey -p (z)xx)) x1)
Ex,,
60,fo(e..0) 1)1 6
20 (1
ãn)
(61)
87
= C1 + Eyye.x
le- E0, VI-
20 -a))
19

---

Published as a conference paper at ICLR 2024
where 77, (N, I), and the second equality is also derived by LOTUS.
Finally, by combining Eq.
558 and Eq. 61p. we have
1g - log
SMD [xo]]
&
LMD
SMD
(62)
T
+
R 1,,,000
- (0.0
- vl-a,e, 1)
where C Ei= Ct and r, (21(20;a(1-aa))) -
G
GENERATED SAMPLES
Some images generated by our models (e.g., LDM w/ SMD) are in Fig. 5 and Fig.
20

---

Published as a conference paper at ICLR 2024
(a) Synthesized images of LSUN Church
(b) Synthesized images of LSUN Conference
Figure 5: 64 x 64 images generated by DDPM w/ SMD.
Figure 6: Generated images CelebA-HQ 128 128 (left) and 256 x 256 (right). The left samples are from DDPM w/ SMD and the right ones from LDM w/ SMD.
21


Reviews:
[{"review_id": "0fBpTE0Jxt", "reviewer": "Reviewer_izts", "summary": "This paper identifies an expressive bottleneck in the backward denoising process of current diffusion models, challenging the strong assumptions underlying their theoretical guarantees. The authors demonstrate that these models can incur unbounded errors in both local and global denoising tasks. To address this, they introduce Soft Mixture Denoising (SMD), a more expressive model that theoretically can approximate any Gaussian mixture distribution. The effectiveness of SMD is validated through experiments on various image datasets, particularly noting significant improvements in diffusion models, like DDPM, with few backward iterations.", "questions": "1. There seems to be a typographical error involving a comma in the superscript at the end of Equation (3).\n2. Could you detail the noise schedule utilized in your algorithm? The experiments section suggests that the original DDPM scheduling is retained while only the model component is modified. Considering that your paper emphasizes the importance of shorter chains and the expressiveness issue within them, it would be beneficial to see experimentation with significantly fewer steps to underscore the advantages of your proposed Soft Mixture Denoising (SMD).\n3. The SMD approach bears resemblance to a Variational Autoencoder (VAE) in its structure. Could you confirm if this observation is accurate or elaborate on the distinctions between SMD and VAE?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is articulate and presents a clear logical progression.\n\n2. Both theoretical exposition and experimental verification are provided to substantiate the authors’ arguments.", "weaknesses": "1. The critique leveled against existing diffusion models seems to be somewhat overstated. These models have achieved considerable success across various applications and can represent complex distributions effectively. The alleged expressive bottleneck is contingent upon the noise scheduling strategy deployed. For instance, in typical diffusion models, a small value of $\\beta_t$, such as 0.0001, is assumed to be used as the initial. As indicated in Equation (25), the transition probability $q(x_{t-1} | x_t)$ approaches a Gaussian distribution as $\\beta_t$ tends toward zero, which contradicts the claim of an inherent expressive limitation.\n\n2. The selection of datasets for experimentation—LSUN and CelebA—seems narrow given the criticism of diffusion models' multimodality capacity. For a robust evaluation, a more complex and varied dataset like ImageNet, encompassing 1k categories, would be more appropriate.\n\n3. There appears to be a flaw in the derivation of local denoising error $M_t$. The associated loss term in $L_{t-1}$ is predicated on the KL divergence $KL[q(x_{t-1} | x_t, x_0) || p_\\theta(x_{t-1} | x_t)]$. Here, $q(x_{t-1} | x_t, x_0)$, which is a known Gaussian distribution, should not be conflated with $q(x_{t-1} | x_t)$, which represents an unknown distribution. The validity of Theorems 3.1 and 3.2 is reliant on the accurate definition of $M_t$.\n\n4. The paper does not reference the FID (Fréchet Inception Distance) results from the Latent Diffusion Model (LDM) study. In the LDM research, the reported FID scores were 4.02 for LSUN-Church and 5.11 for CelebA-HQ, which are superior to the performance metrics achieved by SMD as presented in this paper. This omission is significant as it pertains to the comparative effectiveness of the proposed model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kCXc7ADffr", "reviewer": "Reviewer_eLFe", "summary": "this submission introduces soft mixture denoising for improving the expressive bottleneck of diffusion models. It first shows that diffusion models have an expressive bottleneck in the backward denoising steps, when approximating p(xt-1|xt) using a Gaussian distribution, that leads to unbounded local and global denoising. It then proposes soft mixture denoising (SMD) that approximate the backward step p(xt-1|xt) using a  gaussian mixture distribution, where the number of modes is infinity. This soft gaussian mixture is a universal approximator for continuous probability distributions and the result shows that the local and global errors would be bounded. Experiments with image datasets indicate that SMD improves different diffusion models such as DDPM and DDIM.", "questions": "The mean parameterization in eq. 11 needs to be clarified? What is the hyper network representation? what is \\theta \\cup f_{\\phi}?\n\nWhile the theory supports that soft gaussian mixture to be a universal approximator. However, the performance gains compared with single Gaussian are not significant. What are the limitations and approximations that led to that? Could it be the identity assumption for the covariance matrix?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Improving the design of diffusion models and make them more efficient is a timely problem\n\nIdentifying the expressiveness bottleneck of single gaussian approximation, and the unbounded denoising errors is novel for denoising diffusion models\n\nThe experiments are extensive", "weaknesses": "The performance gains for the new soft mixture models are not significant. One would expect a significant reduction in number of steps if soft mixture is a better approximation for p(xt-1|xt), but that is not the case in the experiments. \n\nThe architectural changes for the new denoising networks are not discussed well. It’s a bit confusing how the", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ihRqp6ghZO", "reviewer": "Reviewer_mAPH", "summary": "This paper studied the reverse process in the diffusion models. Specifically, the authors theoretically showed that the Gaussian assumption in the reverse process of the original diffusion models is not expressive enough for complicated target distribution, and proposed a soft-mixture model for the reverse denoising process. The authors theoretically demonstrated the expressiveness of the new model, and derived training and sampling algorithms for it. Experiments have been conducted to demonstrate the effectiveness of the proposed method.", "questions": "To my knowledge, there are studies considers better parameterizing the distribution in the reverse process, such as:\n\n1. Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. ICLR, 2022.\n2. Yanwu Xu, Mingming Gong, Shaoan Xie, Wei Wei, Matthias Grundmann, Kayhan Batmanghelich, and Tingbo Hou. Semi-Implicit Denoising Diffusion Models (SIDDMs). arXiv:2306.12511, 2023.\n\nThe authors should discuss these studies, and better to empirically compare with them.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is new and reasonable.\n\n2. The authors provided theoretical foundations for their proposed method.\n\n3. The effectiveness of the proposed method has been empirically verified.", "weaknesses": "To me, there is no significant weakness of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jixmHVgPj3", "reviewer": "Reviewer_nQBi", "summary": "This paper...\n- theoretically shows DMs suffer from an expressive bottleneck due to the assumption that the denoising distribution is Gaussian,\n- proposes soft mixture denoising to address this problem,\n- shows SMD improves the performance of DMs on various datasets.", "questions": "- Is SMD compatible with fast samplers such as EDM [1]? If it is, can the authors provide results? If not, can the authors suggest how SMD could be modified to be compatible with such fast samplers?\n- How does the performance of SMD vary as we change the size of $g_\\xi$ and $f_\\phi$ networks? Does SMD work better if we use larger networks or is it sufficient to use small networks?\n\n[1] Elucidating the Design Space of Diffusion-Based Generative Models, Karras et al., NeurIPS, 2022.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes a novel approach to improving the sampling efficiency of diffusion models.\n- SMD is well-motivated through rigorous theoretical analysis.\n- This paper well-written and I had no trouble following the logic.\n- There are non-trivial performance improvements after applying SMD.", "weaknesses": "- SMD requires training of additional $g_\\xi$ and $f_\\phi$ networks, so I would expect training SMD requires more VRAM and time compared to training standard diffusion models. A comparison of VRAM / training time / inference time of SMD vs. standard diffusion would be insightful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)