On Analyzing Generative and Denoising Capabilities of Diffusion-based Deep Generative Models
Authors: Kamil Deja
Anna Kuzina
Tomasz Trzcinski
Jakub Mikolaj Tomczak
Venue: NeurIPS 2022 Accept (2022)
Keywords: Diffusion generative models
DDGM
denoising autoencoders

Abstract: Diffusion-based Deep Generative Models (DDGMs) offer state-of-the-art performance in generative modeling. Their main strength comes from their unique setup in which a model (the backward diffusion process) is trained to reverse the forward diffusion process, which gradually adds noise to the input signal. Although DDGMs are well studied, it is still unclear how the small amount of noise is transformed during the backward diffusion process. Here, we focus on analyzing this problem to gain more insight into the behavior of DDGMs and their denoising and generative capabilities. We observe a fluid transition point that changes the functionality of the backward diffusion process from generating a (corrupted) image from noise to denoising the corrupted image to the final sample. Based on this observation, we postulate to divide a DDGM into two parts: a denoiser and a generator. The denoiser could be parameterized by a denoising auto-encoder, while the generator is a diffusion-based model with its own set of parameters. We experimentally validate our proposition, showing its pros and cons.

On Analyzing Generative and Denoising Capabilities of Diffusion-based Deep Generative Models
Kamil Deja*1
Anna Kuzina" Vrije Universiteit Amsterdam Amsterdam, the Netherlands a. kuzina@vu, nl
Warsaw University of Technology
Warsaw, Poland
kamil kamil.dejaaa deja@pw...... edu.pl pl
Tomasz Trzciáski
Jakub M. Tomezak Vrije Universiteit Amsterdam Amsterdam, the Netherlands j m. tomczak@vu. nl
Warsaw University of Technology
Jagiellonian University of Cracow
Tooploox, IDEAS NCBR
tomasz. wwwcinskiiiiii...o..... edu. pl
Abstract
Diffusion-basee Deep Generative Models (DDGMs) offer state-of-the-art perfor- mance in generative modeling. Their main strength comes from their unique setup in which model (the backward diffusion process) is trained to reverse the for- ward diffusion process, which gradually adds noise to the input signal. Although DDGMs well studied, it is still unclear how the small amount of noise is transformed during the backward diffusion process. Here, we focus on analyzing this problem gain ing and generative capabilities. We observe a fluid transition point that changes the functionality of the backward diffusion process from generating a (corrupted) image from noise to denoising the corrupted image to the final sample. Based on this observation, we postulate divide a DDGM into two parts: denoiser and generator. The denoiser could be parameterized by a denoising auto-encoder, while the generator a Liffusion-based model with its own set of parameters. We experimentally validate our proposition, showing its pros and cons.
insight into the behavior of DDGMs and their denois-
Introduction
Diffusion- based Deep Generative Models 22 (DDGM) have recently attracted increasing attention,
due to the unprecedented quality of generated samples [5
9 The general idea behind this set of methods is generate samples using diffusion processes g 10
23 24] In the forward diffussin process, an image is passed through a number of steps that consecutively add a small portion of noise direct reverse the forward process, where generative
it. The backward diffusion process model is trained to gradually denoise the image. With a sufficient number of the forward diffusion steps, noisy images approach isotropic Gaussian noise. Then, generating new examples possible by applying the backward diffusion to the noise sampled from the standard Gaussian distribution.
While the performance of DDGMs is impressive, not all of their aspects are fully understood. Intuitively, a DDGM is trained to remove small amounts noise from many intermediary corrupted images. Although this perspective is reasonable and complies with the interpretation of DDGMs using stochastic differential equations [10 24]. it is still unclear how the small amount of noise
"Equal Contribution
Work done while visiting Vrije Universiteit Amsterdam
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Standard diffusion model (coninuation)
Pe
Pa
Ps
Xg00
X100
Po
Pa
X4000
X500
Poz
X
Diffusion model
DAED step denosing autoencoder
Figure 1:
Overview of the proposed Denoising Auto-Encodee with Diffusion (DAED). To validate our hypothesis that DDGMs can be understood composition of generator and denoiser, propose to explicitly model the denoising part with separate denoising autoencoder.
is removed during the backward diffusion process where images are composed of almost entirely random values. The more adequate intuition might be that in its initial steps, a diffusion model does not only remove noise but also introduces a new signal according to the distribution learned from the data. In this work, further investigate this observation understand the balance between the generative and denoising capabilities of DDGMs.
In particular, aim to answer the following three questions in this paper: (i) Is there transition in the functionality of the backward diffusion process that switches from generating to denoising? (ii) How does this split of functionality affect the performance? (fii) Does the denoising part in DDGMs generalize to other data distributions? As a result, the contribution of the paper is threefold:
First, we analyze the noise distribution in the forward diffusion process and how steps of the diffusion process are correlated with the reconstruction error.
Second, based and a generator. As a result, we propose a new class of models that consist of a Denoising Auto-Encoder and Diffusion-based generator shortened DAED. DAED could be considered as a variation of DDGMs with an explicit split into the denoising part and the generating part.
analysis, we postulate that DDGMMs composed of two parts: denoiser
Third, we empirically assess the performance of DDGMs and DAED on three datasets (FashionM- NIST, CIFAR10, CelebA) in terms of data generation and transferability (i.e., how DDGMs behave on different data distribution).
2 Background
2.1 Diffusion-Based Deep Generative Models (DDGMs)
Model formulation We follow the formulation of the Diffusion- based Deep Generative Models (DDGMs) as presented in 8 22 DDGMs could be seen as infinitely deep hierarchical VAEs with specific family of variational posteriors [T0 Given data point and latent variables X....., Xy, we want to optimize the marginal likelihood po(xo) f (Po........ Xy )xxx. .x. We define the backward (or reverse) process as a Markov
25 26), namely, Gaussian diffusion processes [22
chain with Gaussian transitions starting with p(x7) = N NNx7; 0, I), that is:
T
po( = p(x)) II
pe (xx))
where po px(x(-1/x;) sion process a Markov chain that gradually adds Gaussian noise the data according variance schedule 31, ..., 3r, namely, q(xi,..., (xo) = I 4(x, [x--), where q(x; (x-1) = Nxx; /I - 5,x--1, 3,I). Let us further define = 1 - 3r and a = - Qg Since the condi- tionals in the forward diffusion can be seen as Gaussian linear models, we can analytically calculate the following distributions:
t), Eg (xe, t)). Additionally, we define the forward diffur-
gxx:(xo) = =N(xxi Vārxo, (I)
(2)

---

and
gxx X1, x0) = N (x P (x)x xo), 3rI),
(3)
where [x,,, Xo)
2 1-a. Xp +
far(2-a, " x,, and Br 1-30-1
Be.
We can use 2 and (3) to
1-ac
1-3,
define the variational lower bound follows:
DEL (((xx/xx))llee) In px(o) (0) = Rxx(x (x0) (x
ler
T
gxx (x)) DKL (9(x 1 x Ilpo (x)-1
(4)
L--1
that we further optimize with respect to the parameters of the backward diffusion.
The conditional likelihood In this paper, we focus on images, thus, data is represented by integers from 0 to 255. Following [8). we scale them linearly to [-,, As result, obtain discrete log-likelihoods, we consider the discretized (binned) Gaussian conditional likelihood (
(()(
p (#6)
pe (xo (1) = III
N (z; pà (x1, oî) dr,
(5)
(=1 (26)
where D is the data dimensionality of xo, and i denotes one coordinate of Xo, and:
(x)
00
if
6_(z)
-00
=
(6)
+ 255
if <
x 255
if
Noise scheduling Originally, 8 propose to linearly scale the noise parameters Br (linear schedul- ing), e.g., scaling linearly from 3 10 to BT = 0.02. In 17. authors suggest to increase the
number of less noisy steps through cosine scheduling: a. 7 f()
f(0)
1/T+c Itc
,
with clipping the values of Be to 0.999 to prevent potential instabilities at the end of the diffusion.
Training details In authors notice that a single part of the variational lower bound equal
3
(, Exo,e 207a, (1 σ.) ||e Es (vaixo + vI - t|l?
where é ~ (0, I) and eg is neural network predicting the noise e from x,, Since we use (3) in the variational lower bound objective (4). and x, could be sampled from the forward diffusion for given data, see (2). we can optimize one layer at a time. In other words, we can randomly pick a specific component of the objective, L,, and update the parameters by optimizing Le without running the whole forward process from Xo to Xy. As a result, the training becomes very efficient and learning very deep models (with hundreds or even thousands of steps) possible.
In g it also proposed to train a simplified objective that is a version
7 without scaling, namely:
L simple (0) Exo.e |||E Es (Vā (I- []]]
where is uniformly sampled between and T. To further reduce computational and memory costs, typically, a single. shared neural network is used for modeling eø [8] [1][]] that is parameterized by an architecture based on U-Net type neural net [8]. The U-Net could be seen as specific auto-encoder that passes all codes from the encoder to the decoder.
2.2 Denoising Auto-Encoders
Another class of models, Denoising Auto-Encoders (DAEs), is similar to DDGMs in the sense that they also revent a known corruption process. However, DAEs are trained to remove the noise single pass, and unlike DDGMs, they cannot generate new objects. Specifically, DAEs are auto-encoders that reconstruct data point X9 from its corrupted (noisy) version
4 28 Let us

---

denote the auto-encoder by f.).. Using the same notation as for DDGMs, the Gaussian corruption distribution is q(x) (xa). Then, DAE maximizes the following objective function:
{(xo <p) = Egtts (xx) [In (xolfe (x1)).
and, in particular, for the Gaussian distribution with the identity covariance matrix, we get the original objective for DAEs 28 Iap (xolSp (x1)) = - x fo (x1))l const.
3 An analysis of DDGMs
The core idea behind DDGMs is the gradual noise injection to images as we go forward in time such that the final object a sample from the standard Gaussian distribution. Then, in the backward diffusion process model reverts this procedure and, as result, generates new objects. Therefore, un- derstanding the success of DDGMs relies heavily on understanding how the injected noise influences the behavior of both training and the model itself.
The noise distribution in the forward difffusion process The first question we ask is how much
corrupted an image gets after applying specific noise schedule. Following g 12 17,, we can utilize the signal-to-noise ratio (SNR), expressed as the squared mean of a signal (here: image) divided by the variance of signal, to quantify the amount of noise in xx. For this purpose, the quantity of interest is the forward diffusion for given Xo. namely, g(x, xo). that results in the following SNR:
SNR(x6,0) 1) [x
(10)
02.
Similarly to [T]. we formulate the forward diffusion in such way that the SNR is strictly monotoni- cally decreasing in time, namely, SN R(x0, t) < SN R((oo, s) for t s. This means that an image becomes more noisy as we go forward in time.
log SNR
FabKKoMNIST CIFATLO
J Jon SNN
log SNR
FallimMNiSS CIFARIO
FaciimMNIST CIFAR1O
FadiieMMNIT CIPARUO
00
00
05
t/T
t/T
f/T
t/T
(a) Linear noise schedule
(b) Cosine noise schedule
Figure 2: Logarithm the ignal-to-noise ratio averaged over the dataset (solid line) and its standard deviation, and the difference of the log SNR within two consecutive time steps.
In Figure 2 (left) we plot the logarithm of the SNR for both linear (Figure a) and cosine (Figure b)) noise schedules for two datasets (PashionMNIST and CIFAR10). We average SNR over the Xy 's (from the corresponding dataset). The right column depicts the change of the log SNR, i.e., its discrete derivative A log SNR(t) log SNR(Io, t) log SNR(# t 1). First of all, we can notice a point at which the log-SNR drops below 0. This corresponds to the situation of the noise overshadowing the signal. In the case of the linear noise schedule, this happens after about 20% of steps, while for the cosine noise schedule, it appears after about 25 - 50% of steps. However, the transition occurs in both cases. The biggest changes in the log-SNE are noticeable within the first 10% of steps. This may suggest that the signal is the strongest within the first 10 - 20% of the forward diffusion process steps, and then it starts being overshadowed by the noise.
The reconstruction error of DDGMs Since we know that the signal is not lost within the first. 10- 20% of steps, the next question is about the reconstruction capabilities of DDGMs, namely, what is the reconstruction error of x, (xxx (x)). To be clear, we are not interested in how much each step of a DDGM contributes to the final objective (c.g., see Figure 2 in (17) but rather how well a DDGM reconstructs a noisy image xt, In Figure 3 we plot the Mean Absolute Error (MAE) and the Multi-Scale Structural Similarity (MS-SSIM) [29] that both measure the difference between an
original image Xy and corrupted image at the tth step X2 reversed by the backward diffusion.

---

We present the values on two datasets (FashionMNIST and CIFARIO) for the first 20% of steps. Apparently, after around 10% of the steps, the reconstruc- tion error starts growing, and the MAE in- creases linearly above 0.1 (i.e., about 6% of error per pixel). At the same time, the MS-SSIM drops below 0.9 0.95 (i.e.. the discrepancy between original images and reconstructions becomes perceptu- ally evident). This observation might suggest that DDGMs could be roughly divided into two parts: fraction of steps of DDGM (e.g., first 10% of the steps) constitute denoiser that turns a corrupted image into a clear image, and the remaining steps of the DDGM are responsible for turning noise into noisy structure corrupted image), i.e., generator that generates meaningful patterns. In other words, we claim that DDGM can be interpreted as a composition of denoiser and generator, but the boundary between those two parts is fluid. Moreover, the denoiser gradually removes the noise in generative manner (i.e., by sampling x ~ p(x-1 (x))).
CIFARLO
MS SSIM
PallsnNNNST
CIPARIO
1/7
Figure 3: The averaged reconstruction error calculated using (left) the MAE, and (right) the MS-SSIM at different steps of DDGM.
DDGMs hierarchical VAEs In this paper, we postulate that DDGMs could be composition of parts that serve different purposes. We can get additional insight into our claim by noticing a close connection between DDGMs and hierarchical VAEs. As presented in [00 II 22].. if we treat all X₂ 's with & > 0 as latents, and see the forward diffusion process as a composition of (non-trainable) variational posteriors, DGGMs become a specific formulation of hierarchical VAEs. On the other hand, we can start with VAE with a single latent variable, X1, for which the variational lower bound equal to:
In p(xo)) 2 Exi xx) [In p(xo x1)] [xx [x0))l((x]]]-
(11)
Then, similarly to [2 30), the marginal p(x1) could be further modeled by a DDGM. By keeping the dimensionality of the same Xg, and taking the variational posterior {(x] (xx) to be fixed and part the forward diffusion, we get the DDGM model. This perspective of combining a VAE with a DDGM opens new possibilities for developing hybrid models.
4 DAED: Denoising Auto-Encoder with Diffusion
In this work, we propose specific combination that distinctly splits the DDGM into generative and denoising parts. As noted in the previous section, the signal in the forward diffusion process is the strongest within the first 10 - 20% of steps, and, thus, we postulate to perceive this first part of a DDGM as a denoiser. Together with the observation about the combination of VAE with a DDGM-based prior, we consider turning a denoising auto-encoder into a generative model as presented Figure We bring a DDGM-basce part into DAE for generating corrupted images. The resulting objective is the following:
E(xo: sp,0) 0) = Exirg(xi|xo) (xo) [lnp (xolfp (x1)) + In po
(12)
In po (x1 px X,) g(x),..., X7 xo)
2 Exx (xo) [In (xol.f (x1)) + + xx)xx)).x
(13)
(XA=()):P))
(n(x):0)
where in (13) we introduce additional latent variables and the variational posterior over them, that yields the variational lower bound. We call the resulting model DAE with a Diffusiion, or DEAD for short. In sense, DAED is a DDGM with distinct parameterizations of the part between Xo and X1, and the part for the remaining x's. Thus, DEAD is almost identical to a DDGM, but there are (i) We can control the amount of noise in q(x, (x). It can correspond the first step of the forward diffusion model, or we can introduce more noise once that would correspond to several steps in the DDGM. (ii) We use two different parameterizations, namely, an auto-encoder (e.g., U-Net architecture) for f-(-) and separate, shared U-Net for modeling the DDGM from X1 to XT- Since there are two neural networks, the lower bound to the objective 7 in fact composition of two objectives with disjunctive parameters, namely, the objective for the
the following differences:
5

---

denoiser. (DAE, and the objective for the generator (i.e., the diffusion- based generative model), ÉD- (iii) In the DAED, we introduce the denoiser explicitly and make a clear distinction between the denoising and the generating parts while, as discussed earlier, this boundary is rather fluid in DDGMs. By introducing DAED, we can analyze what happens if we distinctly divide those two aspects with two separate parametrizations.
Moreover, we hypothesize that the resulting model may better generalize across various data distribu- tions due to decoupling the parameterization of the denoiser and the generator. The training dataset may bias a single, shared parameterization a DDGM, and while denoising an image from a different domain, it may add some artifacts from the source. While with two distinct parameterizations, there might be lower chance for that. We evaluate this hypothesis in the experiments.
5 Related work
DDGM for image generation Various modifications of DDGMs were recently proposed to improve their sampling quality. This includes simplifying the learning objective and proposing new noise schedulers, which allow DDGMs to achieve state-of-the-art results. In this work, we show that splitting the decoder into two parts, namely, a denoiser and a generator, can benefit the performance, especially when training with the variational lower bound.
Properties of DDGMs In 8 authors notice that DDGMs can be beneficial for lossy compression. observing (Figure 5 in [8] that most of the bits are allocated to the region of the smallest distortion that corresponds to the first steps of a DDGM. We draw similar conclusion when discussing the denoising ability of the diffusion model in Section 3 However, we base our analysis on the signal-to- noise ratio rather than compression. On the other hand 21 focus on the computational complexity of DDGM and propose a progressive distillation that iteratively reduces the number of diffusion steps. The work shows that it is possible to considerably reduce the number sampling steps without losing performance. We believe that their results support our intuition that it is reasonable to combine several initial steps into a single denoiser model. In 33, authors evaluate how the diffusion process changes in time when model is trained with different objectives (Eq. Orr Eq. 8 They observe that the image generation process differs significantly and that is more beneficial to switch between those two approaches at different stages of the diffusion. In this work, we also investigate changes in the diffusion process, but we focus on the generative and denoising capabilities the model instead.
Connection to hierarchical Variational Autoencoders Several works have noted the connection of DDGM to VAEs. In [00 authors focus on the continuous diffusion models and draw the connection to the infinitely deep hierarchical VAEs. In [I] authors further explore this connection, formulate a VLB objective in terms of the signal-to-noise ratio and propose to learn noise schedule, which brings the forward diffusion process even closer to the encoder of a VAE. Recently a latent score-based generative model (LSGM) was proposed [27), which can seen as a VAE with the score-based prior. We follow similar direction and propose to see a DDGM as combination of a denoising auto-encoder with an additional diffusion-basee generator of corrupted images.
6 Experiments
Experimental setup In all the experiments, we use a U-Net-based architecture with timestep embeddings as proposed in [7]. We train all the models with a linear 3 scheduler and uniform steps sampler to simplify the comparison. All implementation details and hyperparameters included in the Appendix (A.4) and code repository 3] For DAED, we use the same architecture for both the diffusion part and the denoising autoencoder. We experiments on three standard benchmarks with different complexity: FashionMNIST (11 of gray-scale 28 x 28 images, CIFAR- 10 [13 of 32 x 32 natural images, and CelebA (14) of 64 64 photographs of faces. We do not use any augmentations during training for any dataset. We report results for both variational lower bound loss (VLB) 22 and simplified objective [8]. Following [7 we evaluate the quality of generations with Fréchet Inception Distance (FID) (6) and distributions Precision (Prec) and Recall (Rec) metrics [19 that disentangle FID score into two aspects: the quality of generated results (Precision) and their diversity (Recall).
httt:://ggtaa..cca///nllll/////p

---

6.1
Is there transition in functionality of the backward diffusion process that switches from generating denoising?
In section 3 we investigate how the signal-to-nove noise ratio and the reconstruction error of a DDGM change with the increasing number of diffusion steps (see Figure 4. Based on this analysis, we postulate that DDGMs can be divided into two parts: denoiser and generator. To determine the switching point, we propose an experiment that answers the follow-
0.2
ColebA CIFAR10
MAD
0.1
ing question:
0.0
0.11
0.2
there a denoising part of a DDGM that is agnostic to the signal from the data?
t/T
Figure The MAE for a DDGM trained on CI- FARI0 10 and evaluated on CIFARIO & CelebA, with a 0.95 confidence interval.
To that end, we refer once the reconstruction error (e.g.. MAE) from different diffusion steps. This time, however, we compare the quality of reconstructions with single DDGM model trained on the CIFARIO dataset and then evaluated on CIFARIO and CelebA. The result of this experiment presented in Figure 4 Interestingly. we notice that for approximately 10% of the initial steps of the DDGM, there is a negligible difference in the reconstruction error between these datasets. This fact may suggest that, indeed, the model does not require any information about the background data signal in the first steps, and it capable of denoising corrupted images. However, after this point (about 10% of steps), the reconstruction eтrоr starts growing faster for the dataset the model was not trained on. This indicates that information about the domain becomes important and affects performance.
to the analysis of
6.2 How does splitting DDGMs into generative and denoising parts affect the performance?
The results so far confirm our claims that DDGMs could be divided into denoising and generative parts. Independently of dataset, there appears to be a transition point at which a DDGM stops generating a corrupted image from noise and starts denoising it in a generative manner. Here, we aim to verify whether it is possible to do clear split into denoising part and generating part. For this purpose, we use the introduced DAED approach that consists of a DAE part (the denoiser) and a DDGM (the generator) parameterized by two distinct U-Nets.
First, we consider a situation which we train a DDGM using the simplified objective (8) and then replace the first steps with a DAE. In other words, we train a DAED in two steps: first the DDGM and then the DAE. This experiment aims to check how the decoupling of the DDGM into two parts influences the model performance. In Figure 5 we present the dependency between the log-SNR at the splitting point and the FID score. In all cases, the performance of DAED comparable to the DDGM if we replace the DAE with up to the 10% of the steps that correspond to (og(SNF R) is equal around 4. For more complicated datasets like CIFARIO and CelebA, fewer steps could be replaced. This effect could be explained by the fact that images in these datasets have three channels (RGB), and removing noise is more problematic. That outcome reconfirms our presumptions that is reasonable to split the DDGM since the final performance is not significantly affected by the division for an adequately chosen splitting point.
30
Nish
CIEARIO
30
Cileiy
FID
20
10-
10
10-
4
2
2
0
lng(SNN)
log(SNR)
(a) FaShionMNIST
(b) CIFARIO
(c) CelebA
Figure 5: The performance (FID) of DAED with different switching points with respect to the logarithm of the signal to noise ratio (T0) on three different datasets.
7

---

Table I: FID Precision (Prec) and Recall (Rec) scores. For each row, we indicate the length of the diffusion process (T) and the training objective (Loss). Best results in bold.
Model
Fashion Moist
CIFARIO
CelebA
Loss
FID
Prec
T
FID
Prec
FID
Prec
DDGM
VLB
500
8.9
68
53
1000
26
53
54
1000
23
51
21
DAED 0.1
VLB
468
71
900
59
900
63
DAED 0.001
VLB
499
7,5
71
999
15
60
999
16
70
27
DDGM
Simple
500
7.8
72
65
1000
7.2
65
61
1000
4,9
66
57
DAED
Simple
73
900
62
67
27
DAED 61 0.001
Simple
499
5.7
69
64
999
14.8
65
53
999
7.4
67
54
To get further insight into the qual- itative performance, in Figure 6 we demonstrate how the selection of
splitting point with respect to the Sig-
nal to Noise Ratio (SNR) affects the quality of final generations We present non-cherry-pic -eieerey samples from DAED trained in the same ner as described the previous para- graph. As expected, the more noise the DAE part (the denoiser) must deal with (see the values of (3 in Figure the fewer details in the generations there are. These samples again in- dicate that by replacing some steps with denoiser, we get trade-off be- tween "cleaning" the corrupted image or, in fact, further generating details. It seems that there is sweet spot for perceptually appealing images that contain details and are "smooth" at the same time, see 3, 0.025 in Figure However, as it is typically difficult to provide convincing arguments by staring at samples, we further propose to analyze quantitative measures.
B 02 (eg(SMMM 028
Bs =01 leg(SWRy = 1.1
B = 0.05 Ing(SKA) 1.81
As = 0,025 (ng(SWR) = 2,67
P. = 0.001 leg/ste
Figure 6: Examples of generations from DAED with the same noise value and different switching points.
In Table we compare the performance of DAED against the DDGM on FashionMNIST, CIFARIO. and CelebA terms of FID, Precision and Recall scores. We want to highlight that our goal is not achieve SOTA results on the before- mentroned datasets but to verify whether we can gain some further understanding and, potentially, some improvement by splitting the denoising and generative parts. We consider two scenarios, namely, learning DDGM and DAEDs using either the variational lower bound (VBL) or the simplified objective (Simple) with various lengths of the diffusion. Interestingly. DAED outperforms the DDGM when these models are trained using the VBL loss. For the simplified objective, DAED trained with the same number of diffusion steps yields slightly lower performance than standard DDGMs. As indicated by the Precision/Recall, generations from DAED are as precise as those from DDGM. However, they lack certain diversity, probably due the smoothing effect of the DAE part. Detailed results for other setups are presented in Appendix
6.3 Does the noise removal in DDGMs generalize to other data distributions?
The last question
interested in is the generalizability of DDGMs to other data distributions. We refer to this concept as transferability for short. other words, the goal of this experiment is to determine whether possible. In this experiment, we rely on the results presented in Section where roughly the first 10% of steps could be utilize DAED with an explicit division into the denoising and generating parts.
model its part new data with good performance as
the denoising part. To further strengthen this perspective, also
First, we consider the case in which we compare the reconstruction errors measured by the MAE and the MS-SSIM. In this scenario, we train DDGM on source dataset and then assess it on
'Generations for all datasets are presented in Appendix A.2 "In Appendix A.6 (..enndixA...wwrr show that increasing the number of parameters of DDGMs to be comparable to DAED does not lead significant performance improvements.

---

Table 2: Reconstruction errors measured by MAE (D), MS-SSIM (1) for images noised with 3 = 0.1.
*To evaluate models trained on CIFARIO, we downscale CelebA to 32 x 32. Best results in bold.
Target dataset
CIFARIO
CIFARI00
CelebA
Source Dataset
Model
MAE
MS-SSIM
MAE
MS-SSIM
MAE
MS-SSIM
DDGM VLB
0.091
0.94
0.097
0.94
0.093
0.95
CIFARIO
DDGM Simple
0.085
0.95
0.097
0.94
0.096
0.95
DAED
0.065
0.97
0.074
0.97
0.068
0.97
DDGM VLB
0.113
0.93
0.110
0.93
0.077
0.96
ImageNet
DDGM Simple
0.113
0.94
0.111
0.93
0.068
0.96
DAED
0.071
0.97
0.071
0.97
0.050
0.98
a target dataset. We use CIFARIO or ImageNet (32x32 or 64x64) as source data and CIFAR1O CIFARIO0 or CelebA target data. For each image from the target dataset, we apply the DAE part of DAED to obtain the reconstruction or 793 steps of the forward and backward diffusion in the case of the DDGM, which corresponds to the same level of added noise. For this experiment, use the pre-trained DDGM from 17 that consists of 4000 steps and uses the cosine noise scheduler. The results outlined in Table 2 First of all, there no significant difference in the performance of DDGMs trained with either the VBL objective or the simplified objective. They achieve quite satisfactory MAE and MS-SSIM scores. However, DAED outperforms the DDGMs, obtaining much better transferability. We explain it by the fact that probably, with each step in the denoising part DDGM adds details that are typical for source data while DAED focuses on removing noise and produces smoother output. This outcome may further suggest that splitting DDGMs into two parts with two separate parameterizations reasonable and even beneficial.
To get further insight into the transferability behavior, we present few (non-cherry-pice -picked examples from CelebA in Figure and four toy examples Figure Ib. We use the same setup as explained in the previous paragraph (i.e., the pre- pre-trained DDGM provided in [17], and the images are noised with 81 0.1. In columns 3-6 Figure a, we present reconstructions for the DDGM trained on CelebA, the DDGM trained on ImageNet. DAED trained on CelebA, and DAED trained on ImageNet, respectively. becomes apparent that the DDGM trained on CelebA denoises the image by generating new details while DAED denoises by smoothing. Interestingly, DAED performs better than the DDGM when we use ImageNet-traie several toy examples that were denoised with the DDGM and DAED trained on CIFARIO. We see that the DDGM adds many details that are artifacts from the source data. seems that DAED does not suffer from that behavior.
models to denoise CelebA. In Figure b,,, we depict
DDGM Celeb4
DDGM ImageNet
DAED CelebA
DAED ImageNet
DAED
DDGM
& 0.1
(a) Reconstructions CelebA Figure 7: (a) Denoising of image with 0.1 noise using either DAED or the corresponding number of the DDGM steps. (b) Four noisy toy examples denoised by DAED and the DDGM.
(b) Toy examples

---

7 Conclusion
In this work, we investigate the generative and denoising capabilities of the Diffusion- Deep Generative Models. We observe and experimentally validate that it is reasonable to understand DDGMs as a combination of two parts. The first one generates noisy samples from the pure noise by inputting more signal from learned data distribution, while the second one removes the remaining noise from the signal. Although for standard DDGMs, the exact switching point between those two parts is fluid, propose new approach dubbed DAED that is explicitly built combination of generative component (a DDGM) and denoising one (a DAE). In the experiments, we observe that DAED simplifies training with standard VLB loss function that leads to improved performance. On the other hand, with increasing noise processed by DAE, DAED smoothens the generations resulting in lower performance when training with the simplified objective. We further show that DDGMs, and DAED especially, generalize well to unseen data, what opens new possibilities for further research in terms of transfer or continual learning of DDGMs.
Acknowledgements
This research was funded by National Science Centre, Poland (grant 2018/31/N/ST6/0233 and 2020/39/B/STT/01551 Foundation for Polish Science (grant no POIR.04.00 1.....4....0000000 00-0000011--0000 carried out within the Team-Net program co-financed by the European Union under the European Regional Development Fund), and the Hybrid Intelligence Center, a 10-year programme funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research. Computation was carried out on the Dutch national e-infrastructure with the support of SURF Cooperative.
References
[1] G. Alain and Y. Bengio. What regularized auto-encoders learn from the data- satta-enerating distribution. The Journal of Machine Learning Research, 15(1):3563-359) 2014.
[2] Y. Bengio. L. Yao, G. Alain, and P. Vincent. Generalized denoising auto-encoders as generative models. Advances in neural information processing systems, 26, 2013.
[3] Y. Benny and L. Wolf. Dynamic dual-output diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11482-11491, 2022.
[4] M. Chen, K. Weinberger, F. Sha, and Y. Bengio. Marginalized denoising auto-encoders for nonlinear representations. In International conference machine learning, pages 1476-1484. PMLR, 2014.
[5] Dhariwal and A. Nichol. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
[6] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances neural information processing systems, 30, 2017.
[7] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Im- proving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv: 1007.02070000 2012.
[8] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[9] Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
[10] C.WW. Huang, H. Lim, and A. Courville. A variational perspective on diffusion-based generative models and score matching. Advances in Neural Information Processing Systems, 34, 2021.
[11] D. Kingma, T. Salimans, B. Poole, and Ho. Variational diffusion models. Advances in Neural Information Processing Systems, 2021.
10

---

[12] D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. In ICLR, 2014.
[13] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. In Cileseer, 2009.
[14] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[15] Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
[16] C. Nash, J. Menick, S. Dieleman, and P. W. Battaglia. Generating images with sparse represen- tations. arXiv preprini arXiv:2103.03844 2021.
[17] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In Interna- tional Conference on Machine Leariin, pages 8162-8171. PMLR, 2021.
[18] O. Ronneberger, Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.
[19] M. S. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly, Assessing generative models via precision and recall. arXiv preprint 180iv:8800.000 2018.
[20] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[21] T. Salimans and J. Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Reprexentations, 2022.
[22] Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[23] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
[24] Y. Song, Sohl-Dickstein, D. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[25] M. Tomezak. Deep Generative Modeling. Springer Cham, 2022.
[26] B. Tzen and M. Raginsky. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv preprint arXiv: ArXiv:1905.09883, 2019.
[27] A. Vahdat, K. Kreis, and Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34, 2021.
[28] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages (096-1103, 2008
[29] Z. Wang. E. P. Simoncelli, and A. C. Bovik. Multiscale structural similarity for image quality assessment. In The Shrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pages 1 1398-1402. IEFF, 2003.
[30] A. Wehenkel and G. Louppe. Diffusion priors variational autoencoders. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021.
[31] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: Novel Image Dataset for Benchmarking Machine Learning Algorithms, 2017.
11

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See supplemen-
tary material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See supplementary material
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See supplementary material
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use pretrained
model for the transferability experiment and cite the authors correspondingly
(b) Did you mention the license of the assets? [Yes] See supplementary material
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "elRL0Ck8moC", "reviewer": "Reviewer_4jY9", "summary": "The presented work makes the observation that the reverse diffusion step of DDGMs can be decomposed into two distinct parts: one corresponding to denoising and the other corresponding to generation. Furthermore, the authors extensively study different aspects of such a decomposition and motivated by it, propose a new setup called DAED combining DDGMs with DAE. The proposed DAED seems to have performance on par with DDGMs for most setups while it has a better performance compared to DDGMs regarding generalizabilitry to different data distributions. ", "questions": "Please refer to the weaknesses described above. It would be nice to hear what the authors think regarding the mentioned weaknesses. \n\nIn addition, I would be interested to see some analysis on the diversity of generated samples from the proposed DAED compared to traditional DDGMs.\n\nFinally, I would also be interested to see what impact different noise schedules have on the performance of DAED and how the transition points change with respect to different noising schedules (e.g.- linear, cosine etc.)", "limitations": "I enjoyed reading the paper and find it a well written paper with novel observations. Some limitations have been mentioned in the weaknesses section of the review.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "***Strengths***\n1) The work makes an interesting and novel observation that the generative reverse diffusion step of DDGMs can be decomposed into two separate parts for denoising and generation. This is a significant observation and will help the community better understand DDGMs many aspects of which till date are not fully understood. \n2)  The proposal of DAED leverages on the observations made in this paper and is a sensible approach while achieving at par or better performance to traditional DDGMs.\n\n***Weaknesses***\n1) The authors show there is a transition point between the denoising and generative capabilities of traditional DDGMs and choosing when to switch from one mode to another affects performance significantly. Can the authors recommend how this transition point can be automatically selected for a given dataset?\n\n2) The proposal of composing a DDGM with DAE is sensible. I am wondering if stacking DAEs (stacked denoising autoencoders) can further help with performance of DAED. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ck3fxnJDe-O", "reviewer": "Reviewer_QKFo", "summary": "The paper investigate the behaviour of diffusion-based deep generative models (DDGM) and argues that DDGM can be separated into a generative and a denoising part. The authors combine DDGMs with denoising auto encoders to introduce DAED and show that this model can improve the overall sample and reconstruction quality.", "questions": "- The metrics show precision and recall values for the images. What does this mean?", "limitations": "see above.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- The paper is well written and easy to follow.\n- The interpretation of DDGM as generator and denoiser seems novel and offers interesting insights into the field of diffusion-based models.\n- The experimental results seem to indicate improved generative performance.\n\nCons:\n- The authors rely on t/T vs reconstruction to define the denoising and generation stages. However, this is a bit arbitrary and and should be further discussed. Additionally, it would be interesting to show standard deviations for this behaviour to further support this distinguishment.\n- The improved results might be due to the increased model capacity instead of an actually better model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FF3WrOb73vK", "reviewer": "Reviewer_aLmz", "summary": "- In this study, the authors observe that score-based denoising diffusion models can actually be decomposed into two phases. They argue that by substituting the first half of the two phases with VAE, higher quality images can be generated. They then propose DAED (Denoising Auto-Encoder + Diffusion), a tandem connection of VAE and Diffusion model.\n- The authors examine the fundamentals of the division into these two phases. For example, what level of SNR should be set at the transition point is considered.\n- Many other basic issues related to this problem are also carefully considered.\n", "questions": "- It would be nice if the points raised in the Cons section above is clarified.\n", "limitations": "- This study is more interested in understanding the fundamental issues of denoising diffusion models rather than updating SOTA. So the performance is not competitive with the latest SOTA. This is one of the limitations of this study, and is acknowledged by the authors.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### pros\n\n- The authors found that the diffusion model can actually be divided into a first half and a second half, with the first half being used to roughly generate the noisy images, and the second half being used for denoising. The authors also found that solving the first half with VAE is more effective than solving it with diffusion, and they verified this experimentally. The argument is clear and intuitive.\n- As can be seen from the loss of the denoising diffusion model, one needs to solve a problem of estimating $\\varepsilon'$ from $\\varepsilon$ (where $\\varepsilon$ and $\\varepsilon'$ are very close) when SNR is very low, which is seemingly redundant. It is a natural idea to combine all the first half of the \"noise to noise prediction\" process into a single module to reduce redundancy, and increase the gain per step. The authors have verified that this is indeed the case.\n- The authors' findings also mean that 80-90% of the total diffusion steps can be replaced by a single pass VAE, which would also make the synthesis more efficient.\n\n\n### cons\n\n- In the usual denoising diffusion models, the parameters are shared by a single model regardless of the log SNR value. However, in the proposed method, separate models are used for the VAE and diffusion. Thus, the number of parameters in the total model would be roughly doubled. This raises the question whether it is fair to compare the proposed tandem method directly with a vanilla denoising diffusion model. The dependence on the total model size should be examined.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FDONmpdCWvg", "reviewer": "Reviewer_s9uB", "summary": "In this work, the authors investigate the generative and denoising capabilities of the Diffusion-based Deep Generative Models (DDGMs), and claimed that a DDGM is a combination of two parts. The first one generates noisy samples from the pure noise by inputting more signal from a learned data distribution, while the second one removes the remaining noise from the signal. Based on the observation, the authors proposed a new approach dubbed DAED that is explicitly built as a combination of a generative component (a DDGM) and a denoising one (a DAE).  Empirical experiments are performed to validate the claim.", "questions": "Please explain the relation to the aforementioned reference [1] by Benny et al.  Furthermore, the introduction of the new variational bounds and the denoising autoencoder should be justified as they appear redundnat.", "limitations": "Limitations are not stated.", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: Various empirical validation of the authors' claim.\n\nWeakness: The approach and observations are not novel. Similar observations are already made in the following work. The experimental verifications appear similar. \n\n[1] Yaniv Benny, Lior Wolf, \"Dynamic Dual-Output Diffusion Models\", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 11482-11491\n\nMoreover, the variational lower bound (11) has been already used in the derivation of the DDPM, so it is not clear why the new bound and related discussion are necessary. Furthermore, the way of training score functions is closely related to the denoising author encder. Therefore, the decomposition of the generative parts is denoising parts in DDGMs are trivial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept