UDPM: Upsampling Diffusion Probabilistic Models
Authors: Shady Abu-Hussein
Raja Giryes
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion models
generative models
Primary area: generative_models
TL;DR: We propose an efficient diffusion model that is based on adding noise + upsampling and a novel training strategy, which leads to state-of-the-art generation results on CIFAR-10 and other datasets

Abstract: Denoising Diffusion Probabilistic Models (DDPM) have recently gained significant attention. DDPMs compose a Markovian process that begins in the data domain and gradually adds noise until reaching pure white noise. DDPMs generate high-quality samples from complex data distributions by defining an inverse process and training a deep neural network to learn this mapping. However, these models are inefficient because they require many diffusion steps to produce aesthetically pleasing samples. Additionally, unlike generative adversarial networks (GANs), the latent space of diffusion models is less interpretable. In this work, we propose to generalize the denoising diffusion process into an Upsampling Diffusion Probabilistic Model (UDPM). In the forward process, we reduce the latent variable dimension through downsampling, followed by the traditional noise perturbation. As a result, the reverse process gradually denoises and upsamples the latent variable to produce a sample from the data distribution. We formalize the Markovian diffusion processes of UDPM and demonstrate its generation capabilities on the popular FFHQ, AFHQv2, and CIFAR10 datasets. UDPM generates images with as few as three network evaluations, whose overall computational cost is less than a single DDPM or EDM step while achieving an FID score of 6.86. This surpasses current state-of-the-art efficient diffusion models that use a single denoising step for sampling. Additionally, UDPM offers an interpretable and interpolable latent space, which gives it an advantage over traditional DDPMs. Our code is available online: \url{https://github.com/shadyabh/UDPM/}

UDPM: Upsampling Diffusion Probabilistic Models
Shady Abu-Hussein Department Electrical Engineering Tel Aviv University shady. abbagya com
Raja Giryes Department Electrical Engineering Tel Aviv University rajaêtauex taw.ac.
Abstract
Denoising Diffusion Probabilistic Models (DDPM) have recently gained significant attention. DDPMs compose Markovian process that begins in the data domain and gradually adds noise until reaching pure white noise. DDPMs generate high- quality samples from complex data distributions by defining an inverse process and training a deep neural network to learn this mapping. However, these models are inefficient because they require many diffusion steps to produce aesthetically pleasing samples. Additionally, unlike generative adversarial networks (GANs) the latent space of diffusion models less interpretable. In this work. we propose to generalize the denoising diffusion process into an Upsampling Diffusion Prob- abilistic Model (UDPM). In the forward process, we reduce the latent variable dimension through downsampling, followed by the traditional noise perturbation. As a result, the reverse process gradually denoises and upsamples the latent vari- able to produce sample from the data distribution. We formalize the Markovian diffusion processes UDPM and demonstrate its generation capabilities on the popular FFHQ, AFHQv2, and CIFARIO datasets. UDPM generates images with few as three network evaluations, whose overall computational cost less than a single DDPM or EDM step while achieving FID score of 6.86. This surpasses current state-of-te the-art efficient diffusion models that use a single denoising step for sampling. Additionally. UDPM offers interpretable and interpolable latent space, which gives it an advantage over traditional DDPMs. Our code available online: /tttts:://iiee
com http://h/haabbbDDD...
X9
((x, (xo)
X1
((x2(x1)
x2)
X2
(x(x))Xa)
Po (X1 X2)
po (xo(x)) x1)
Figure The Upsampling Diffusion Probabilistic Model (UDPM) scheme for 3 diffusion steps (L = 3). In addition to the gradual noise perturbation in traditional DDPMs, UDPM also downsamples the latent variables. Accordingly, in the reverse process, UDPM denoises and upsamples the latent variables to generate images from the data distribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
In recent years, Denoising Diffusion Probabilistic Models (DDPMs) have become popular for image generation due to their ability to learn complex data distributions and generate high-fidelity fidelity images. These models work by starting with data samples and gradually adding noise through Markovian process until pure white noise is reached. This process is known as the forward diffusion process, defined by the joint distribution g(Xo.r.). The reverse diffusion process, used for generating new samples, is defined by the learned reverse process (o(xo.1. using deep neural network. This methodology allows them to achieve impressive performance in learning data distributions and sampling from them.
Although DDPMs have shown impressive results in image generation, they possess some limitations. One major limitation that they require a large number of denoising diffusion steps produce aesthetically pleasing samples. This can be quite intensive computationally. which makes the sampling process slow and resource-intensive
Additionally, the latent space of these models is not interpretable, which limits their utility for certain types of image generation tasks, such video generation animation, especially when used in an unconditional setting. The vast majority of works using diffusion models for editing rely on manipulating the CLIP 28 embeddings used with these models and not the latent space itself.
In this work, we propose a generalized scheme of DDPMs called the Upsampling Diffusion Proba- bilistic Model (UDPM). In addition to the gradual noise addition in the forward diffusion process, we downsample the latent diffusion variables to "dissolve" the data information spatially, as demonstrated in Figure I We thoroughly formulate the generalized model and derive the assumptions required for obtaining a viable scheme.
Using our approach, we can sample images from CIFAR10 [22], AFHQQ2 6 and FFHQ 19 datasets, with as few as UDPM steps, where the cost of all three steps together is ~30% of single regular diffusion step. This is less than two orders of magnitude compared to standard DDPMs: guided diffusion
[9 typically requires 1000 iterations, denoising diffusion implicit models (35] require 250 iterations, stable diffusion [29] requires at least 50 iterations with an additional decoder, EDM [8] can reduce the number of steps to 39, and other recent works (24 25 10 can sample with 10-20 network evaluations, which is still considerably more than UDPM. Indeed, in 41. an integration of discriminative loss with diffusion models has shown great generation performance while requiring only diffusion steps. Recent works 4S 26 B1] have demonstrated that sampling can be performed with as few as a single denoising diffusion step while maintaining competitive generation performance. Yet, UDPM achieves better generation quality than these works the CIFARIO dataset 22 while requiring a smaller computational cost (1/3 /3 of a typical diffusion step).
Because UDPM gradually reduces the dimensions of the latent variables in each step, the size of the random noise added is considerably smaller. Specifically, all the dimensions of the latent variables together are smaller than the dimensions of the original images. As result, UDPM is much more interpretable compared to conventional DDPMs. In our experiments, we show how one may manipulate the generated images by changing the latent variables, which is similar to what has been done in Generate Adversarial Networks (GANs) [9].
Our contributions may be summarized as follows: (i) A novel efficient diffusion model for image generation that achieves a significant improvement over current state-of-the-ar methods by reducing the number and cost of diffusion steps required to generate high-quality images; (ii) achieving good interpretability and interpolability of the latent space.
2 Related Work
Diffusion models are latent variable models defined through a diffusion probabilistic model. On one side we have the data Xo v(Xn) and on the other side, we have pure noise x,, g(xx.). Both are related to each other using a Markovian diffusion process, where the forward process is defined by the joint distribution g Xo) and the reverse process by p(xo-L-1(xx) Using the Markov chain

---

Figure Generated 64 x 64 images of AFHQv2 with FID=7.10142, produced using unconditional
UDPM with only steps, which are equivalent to 0.3 of a single typical 64 x 64 diffusion step.
property, ((x1:1/xx) and p xL) be expressed by
L
p(XU_L-I(LL) = II ((x))))(x)),
(1)
and
L. g(xi.1 Xo) = II 1=1
(2)
As stated in previous literature 14 34 the common approach is to assume that the Markov chain constructed using normal distributions defined by
((x XI = = N(V1 1.x1-1,
(3)
and
- == (m(())) Ex(x)))
(4)
where 32,32.... By are hyperparameters that control the noise levels of the diffusion process, while (4, Σn denote the mean and variance of the reverse process, respectively.
By learning the reverse process p(x xx) using deep neural network, one can generate samples from the data distribution by running the reverse process: starting from pure noise Xr. ~ (O, I),
then progressively predicting the next step of the reverse process using network trained to predict Xt-1 from X2, until reaching Xo-
Over the past couple of years, this principle has shown marvelous performance in generating realistic- looking images 99 29 34 27 20 32 []. However, as noted by [14, the number of diffusion steps L is required to be large for the model to produce pleasing-looking images.
Recently, many studies have utilized diffusion models for image manipulation and reconstruction tasks [40 30). where a denoising network trained to learn the prior distribution of the data. At test time, some conditioning mechanism is combined with the learned prior for solving highly challenging imaging tasks 3 2 7 Note that our novel adaptive diffusion ingredient can be incorporated into any conditional sampling scheme that based diffusion models.
The works 440 30) addressed the problems of deblurring and super-resolution using diffusion models. These works try to deblur [40] or increase the resolution (30) of a blurry or low-res input image. Unlike our work, their goal is not image generation, but rather image reconstruction from a given degraded image. Therefore, their trained model is significantly different from ours.
Cold diffusion performs diffusion steps by replacing the steps of noise addition with steps of blending with another image or other general steps. This approach differs significantly from ours, as their goal to demonstrate that denoising can be replaced with other operations without developing the corresponding diffusion equations. In our case, we formally show what operations can be used without losing the Markovian property of the forward and reverse diffusion processes, and without omitting the noise component.

---

Soft diffusion 8 proposes to blur the signal before adding noise to it in the forward process. Then, for solving the reverse process they train a network to deblur and denoise the signal. However, the addition of the blur operation to the forward process prohibits explicit access to the reverse process and therefore relies solely on the network to predict the clean sharp sample. contrast, in our method, the reverse process is explicitly accessible and can be sampled easily.
Another effort [7 suggests replacing the denoising steps by learning the wavelet coefficients of the high frequencies. They show that this can reduce the number of diffusion steps. Our work differs from theirs in the fact that we rely on upsampling with an additive noise step. We also employ advanced loss functions and present state-of-the- generation results. This is in addition to our interpretable latent space, a component missing from many of the recent diffusion models.
Many works tried to accelerate the sampling procedure of the denoising diffusion model 35 1
23 [0] However, they only focus reducing the number of sampling steps, while ignoring the diffusion structure itself. By contrast, in this work we propose to degrade the signal not only over the noise domain but also in the spatial domain, thereby "dissolving" the signal much faster.
3 Method
Traditional denoising diffusion models assume that the probabilistic Markov process is defined by 33) and (4). These equations construct forward and backward processes that progress by adding and removing noise, respectively. In this work, generalize this scheme by adding degradation element to the forward process. Specifically, we downsample the spatial dimension of the latent variable and upsample it when reversing the process.
3.1 Upsampling Diffusion Probabilistic Model (UDPM)
We begin by redefining the marginal distributions ((x)) and p(x1-1(x)) of the forward and reverse processes:
g(x(xx1)) :oai)::=aaa(xaxaaaa Na
(5)
and
p(x: x1) == (nx(x;;;))) Et),
where in contrast to previous diffusion models that used H as downsampling operator, defined by applying a blur filter W followed by subsampling with stride y. As a result, the forward diffusion process decreases the variables' dimensions addition to the increased noise levels.
in this work we define the operator
In diffusion models, the goal is to match the joint distributions pe (x xo) (learned) and q(x Xx) under some statistical distance. One particular choice is the Kullback-Leibler (KL) divergence, which we adopt here. Formally.
E.g log ((x1L (xo) po ( [xo)]
po (xo:L) ] (x111 (x0)
log Po (xxo) E.g
(7)
ELBO
Thus, can minimize the KL-divergence between po ( Xg) and g(X (xo) by minimizing the Evidence Lower Bound (ELBO). As we show in Appendix B.I this is equivalent to
log (=(X (x0)
Eg
E, En
DKL (px xo) log Pog.eel (x
(8)
1=2
The right-hand side of (8) can be then minimized stochastically w.r.t. 0 using gradient descent, where at each step a random is chosen and a single term of (8) is optimized.
In order to be able to use ( for training po(.), one needs explicit access to q(x) |xt, Xo), for which we need to obtain (xx (x) first. Then, using Bayes' theorem, we can derive x(x1-11 Xt, Xo). To do we first present Lemmal (the proof is in Appendix B.2

---

OS
Figure 3: Generated 64 64 images of FFHQ with FID=7741065, produced using unconditional
UDPM with only steps, which are equivalent to 0.3 of a single typical 64 x 64 diffusion step.
Lemma 1. Let e sed IOO,1) E RN and H. = S.W, where S., is subsampling operator with stride Y and W is blur operator with blur kernel w. Then, if the support of w is most ", have sdd He ~ ((0, [w]
Lemma holds, then by assuming that ||w 1|2 get the following result (see Appendix B.3
xx) (aa Xo, I where ã
II age and 0i
Using 99. can obtain x, from Xa simply by applying H I-times Xo, followed by the addition of white Gaussian noise with standard deviation ø
Given (xx()x0), can use Bayes' theorem and utilize the Markov chain property get
xb) = ((x/(x/110 ((x((x(-1(((xxx=xx))
(xx)
which, as shown in Appendix B.4 of the following form
q(x x.) (N(xx,, Xo. 1), E.),
(10)
where
Ej =
H
1
(11)
o
Ø1-1
and
n(x; x). E₁ 2 Ht x
(12)
Although expression seems be implacable, in practice it can be implemented efficiently using the Discrete Fourier Transform and the poly-phase filtering identity used in [1] where H H is equivalent convolution between w and its flipped version, followed by subsampling with stride y.
More details are provided in Appendix B.5
Following (TO), the true posterior / Xo) in this new setup is Gaussian distribution with parameters (μ(X), Xn, 1), E,). Therefore, one may assume that (.).) is also Gaussian with parameters (no, Σx), where μo is parameterized by deep neural network with learned parameters 0.
For normal distributions, a single term of (8) is equivalent to
(())
xo)
Ci 2 (No n0 ', " 14),
(13)
where C7 is constant value independent of 0.
5

---

Algorithm UDPM training algorithm
Algorithm UDPM sampling algorithm
Require: fo()), Lx)(). (+
Require: fo(.). L
I: while Not converged do
1: X
Xy r q(x)
2: for all L....;
..,
do
3:
E 21, }}
a HHH H.
-1
4:
~ 1)
3: Σ
8-11
X; = a.H xo ōre
6: f = Add Exiple A per + Lny fadr
(1)
4: 14 E
o H
o (xi)
61 1
ADAM step on 0
5: x--1 (Ho, Σ)
Adversarial ADAM step on @
6: end for
9: end while
7: return X0
10:
return fo(+)
From our experiments and following 19 14]. training Pe to predict μ directly leads to worse results. Therefore, we train the network to predict the second term in (12), i.e. to estimate H Xp from x;. As result, minimizing (13) can be simplified to minimizing the following term
7 simple
(fo())) 76 xx) "E, (fo(x)) - xo))
(14)
where fo(-) is deep neural network that upsamples its input by scale factor of y. By the definition of Eg, it is easy to show that it is a diagonal positive matrix, and therefore it can be dropped. As a result, one may simplify the objective to the following term
[ - H'''xuli
(15)
simple
Unlike denoising diffusion models, UDPM tackles a super-resolution task at each reverse step. Consequently, relying solely on simple for training fo(-) produces softer images, as shown in Figure
and discussed in the ablation studies section 5 To address this, we propose to incorporate two
additional regularization terms, following 38 The first term a perceptual loss [42 denoted by
l per. which aligns the VGG features of H Xn and fx().. The second term is an adversarial loss
denoted as aavv. which is similar to the one used in [38 23). Both are used to ensure sharp detailed
results. Overall, fo(-) is trained using the following objective function
=
nd Simple + Xpa S + X ad fadv-
(16)
Algorithm and Figure 4 provide an overview of the UDPM training scheme.
3.2 Image Generation using UDPM
The UDPM scheme presented in the previous section introduces a new approach for capturing the true implicit data distribution q(x,) of given dataset. Given that trained UDPM, the remaining question is how it can be utilized for sampling from ((Xo).
Given a deep neural network fo trained to predict H' from Xt, we start with a pure Gaussian noise sample x (O, 7). Subsequently, by substituting fo(x)) into H in (12), get an estimate of μ4r Then, the next reverse diffusion step XL-1 can be obtained by sampling from M(L,, Σz). By iteratively repeating these steps L times, we can acquire sample Xn po (Xn), as outlined in Algorithm 2
Note that sampling the posterior requires parameterizing N N(m, Ex) in the form of
X l4 + 2)
where e ~ NO, T) has the same dimensions as X This requires having access to Σ Due to the structure of E,, it is possible to apply it efficiently, as shown in Appendix B.6
4 Experiments
In this section, we present the evaluation of UDPM under multiple scenarios. We tested our method on CIFARIO 22 FFHQ 19), and AFHQv2 (6) datasets. Here we focus on the qualitative performance

---

Training:
Sampling:
Xo
9x
01
X01
A/ 13
XL--1
JPO
2
XL
76'-1
74-1 Xa
10
12
3
A700
Figure 4: The training and sampling procedures of UDPM. During the training phase, an image Xo randomly selected from the dataset. It is then degraded using (9) to obtain a downsampled noisy version Xt, which is then plugged into I (-), that is trained to predict H'-1xo- In the sampling phase, we start from pure noise X₁, ~ (O, I). This noise is passed through the network f s(D) to estimate H used to compute faz. through (12), with Σ,. obtained from (TT). Afterwards, Xt-1 drawn from W(u, Et) using the technique described in Appendix B.6 By repeating this procedure for I. iterations, the final sample Xg is obtained.
Figure 5: Latent space interpolation for 64 x 64 generated images. The four corner images are interpolated by weighted mixture of their latent noises, such that the other images are in-bbewween" images from the latent perspective, similar to what has been done in GANs [99.
of UDPM and demonstrate its interpolatable latent space. In the appendix, provide additional quantitative and qualitative results.
We set L 3 and fix 7 for all datasets. We also a uniform box filter of size 2 x 2 as the downsampling kernel w as satisfies the condition Lemma T We then normalize it w.r.t. its norm ||w|| and use to construct the downsampling operator H. Unlike previous diffusion approaches that are limited to analytically defined noise schedulers due to the large diffusion steps number, UDPM allows the noise scheduler to be fine-tuned manually by setting only 6 values (far)/, and {ar H²)) In our tests set {ai 13 {0.5, 0.2, 10-33 and { {0.1, 0.2, 0.3) for all datasets.
7

---

noise deviation
Figure 6: Latent space perturbation for 64 x 64 generated images. The original image is on the left. To its right we present images that were generated by adding a small noise the latent noise from diffusion step 1. As can be seen, the initial diffusion step (1 1) controls the fine details of the image, while the final diffusion step (1 = 3) changes the semantics of the image.
We use the same UNet architecture proposed by [7 and utilized in [8] for all diffusion steps (the specific implementation details presented in Table We increase the number of output channels of the network by a factor of 7² and use the depth-to-space layer [33] to rearrange the output pixels to the desired dimension, which is equivalent to a scale-up by factor of y. For all datasets, we train the network for 600K training steps using ADAM [21] with learning rate and batch size set to 10-4 and 64, respectively. We save the model weights every 10K training steps and pick the model with the best Fréchet inception distance (FID) 13 For stabilizing the training we use exponential moving averaging (EMA) with the dampening parameter set to 0,9999, We also set Anu
(1, 1, 0), Aper = (4, 4, 0), and Aadv = (0.2, 0.5, 1). Particularly, we found that setting And Xper 0 for l achieves the best results with minimal mode-collapse. For the discriminator network, we use the discriminator architecture used in [4],, which is variant of the original discriminator network proposed by [99] We train all models using single NVIDIA RTX A6000 GPU.
4.1 Unconditional Generation
In the unconditional scheme, evaluate UDPM on FFHQ and AFHQv2 datasets, where each dataset contains 50K and 14K images, respectively. We resize the images to 64x64 and train the UNet
using Algorithm [
The number of sampling steps required by UDPM is two orders of magnitude smaller than the original denoising diffusion models 14 34] one order of magnitude less than [35 18], and up-to 4 times
smaller than many recent sophisticated samplers [24. 25 10 designed for accelerating diffusion generation. Particularly, since the size of latent variables increases progressively when sampling, UDPM requires less total computations than a single denoising step used in traditional denoising diffusion, detailed in 4.3 While reducing the computational costs, UDPM retains great image generation quality, as can be seen in Figures
andd]
In addition to the decreased number of sampling steps, UDPM has much smaller latent dimensions. This property allows us to smoothly interpolate the latent space, similar to what has been done in GANs [19] Figures 5 TO and [nd show the results, where the corner images are randomly generated and the be-between images are generated by averaging the noises used generate the corner images:
e, =. + v1 (1-53eb) + 11 (6 12-n2(0jef
+ /1 11 (1-8)))).
where e/ e/, e/,, e are the noises at diffusion step used to generate the corner images, (i.j) are the indices of the interpolated image in the figure, and (n: 6y) are the interpolation coefficients that lie in the range [0, 1]. In the supplementary material, we investigate other interpolation regimes, such as latent variable swapping between two generated images (see Figure [3}.
Another benefit of the smaller latent dimensions is that it allows better interpretability of the generative model. While in previous diffusion approaches one needed to add a conditioning mechanism to the network to be able to control the generation, UDPM, one may control the generation by modifying only the noise maps of the diffusion, can be seen in Figures 6 and 12
8

---

steps
FID
DDIM R5
10Y5
13.36/93.51
DPM-Solver 25
10/5
6.96/288.99
EDM [18
35/5
1.79/35.54
GENIE 110
5
11.20
DEIS [16
5
15.37
GGDM [39
5
13.77
DDGAN [41
2
4.08
TDPM (3
8.91
CT 36
8.70
UDPM (ours)
<1
6,86
Table I: FID scores the CIFARIO dataset (22.. UDPM uses steps, which equivalent in terms
of complexity to 0.3 of a single denoising step used in typical diffusion models like DDPM or EDM.
4.2 Class Conditional Generation
Similarly previous works, also propose conditional generation scheme for UDPM, where use the label encoding block used in [8] to control the generation class. We evaluate UDPM on the popular CIFARIO [2 dataset, which has 50K 32 32 images of 10 different classes.
Figure shows the generation results UDPM on CIFARIO. We also compare the performance of UDPM empirically to many other state-of-the-an non-distillation diffusion methods in Table where we examine the FID results on 50K images generated from the classes equally. For the baseline methods, we show the original results presented in their papers and the results when the number of steps reduced to 5 diffusion steps. As can be seen, when the number of steps is limited to 5, UDPM outperforms all the methods, while also requiring considerably less computation; specifically, 1/3 of a single typical diffusion step.
4.3 Runtime
Unlike typical denoising diffusion models, UDPM decreases the dimensions of the latent variables when proceeding with the diffusion process. Thus, the computations required to run the reverse diffusion process for sampling are significantly less extensive. Specifically, sampling a 64 x 64 image using a single denoising diffusion step requires 40.62 GFLOPS, while UDPM samples images with the same network structure using 13.35 GFLOPs, which is equivalent to 30% the computations. The reason is that in its 3 steps, the input dimensions of the UDPM network are 8 x 8, 16 x 16, and 32 x 32
When comparing the runtimes, UDPM can sample 64 x 64 images from FFHQ and AFHQv2 datasets the rate of 765.21 FPS (frames per second) when benchmarked on NVIDIA RTX A6000 GPU, while a single denoising diffusion step can be run at the rate of 255 FPS using very similar network.
5 Ablation studies
To further understand the diffusion process captured by UDPM, we present two ablation studies in which we examine the effect of each diffusion step on the generation results and demonstrate the effectiveness of the perceptual loss terms used to train the network.
PCA of the generations. Given the noise maps of a generated image, we marginally change the single diffusion step and then perform Principal Component Analysis (PCA) on the generated images, we then examine the top principal components to understand what features are captured in each diffusion step. In Figure 8 we present the top 8 principal components when the 1-th noise map is using small noise, as can be seen, the initial diffusion step controls the fine details while the final step controls the semantics and the overall structure of the image. Similarly, in Figure 9 we show the top 8 principal components of the generated images when a single noise map is replaced entirely.
Perceptual loss. From our experiments we saw that using || ||pp for training fa(-) leads to very soft results. Hence we use the more sophisticated loss term in (16). which promotes perception over distortion. For evaluating the contribution of each loss term, we examine three different networks:

---

(1 + Eper + luiv
61 &per
f FID=72.91
FID=7.41
FID=32.76
Figure 7: Visual comparison of the loss terms effect the FFHQ64 dataset generation results.
The first one is trained using limple while the second has fer in addition to simple in its objective, and finally the network trained using (16). We show qualitative and quantitative comparisons in Figure the advantage of the perceptual terms can be seen clearly, where the FID score and the visual comparison show that the results of using the full loss significantly better.
?
6
Conclusion
We have proposed a new diffusion-based generative model called the Upsampling Diffusion Prob- abilistic Model (UPDM). Our approach reduces the number of diffusion steps required to produce high-quality images, which makes it significantly more efficient than previous solutions.
We have demonstrated the effectivvnees of our approach three different datasets: CIFARIO, FFHQ, and AFHQv2. capable of producing high-quality images with only 3 diffusion steps, whose computational cost is less than for step of the original diffusion models, while significantly outperforming current saate-of-the-se methods dedicated to diffusion sampling acceleration.
Furthermore, we have shown that our interpolatable latent space has potential for further exploration, partieularly in the realm image editing. Based on the initial results shown in the paper, it contains semantic directions, such making people smile changing their age as shown in Figure 6 Future research may explore using UDPM to perform editing operations similar to those performed in styleGAN while maintaining better generation capabilities and favorable diffusion properties.
7
Limitation
While our approach produces impressive generative performance, has some limitations. One limitation of our work is the evaluation of relatively small datasets, such as CIFAR10, FFHQ, and AFHQv2, which is a consequence of our limited computational resources. This restricted us from evaluating our approach on larger and more diverse datasets. Future work could overcome this limitation by leveraging more powerful computational infrastructure to conduct experiments on larger datasets, providing more comprehensive validation of the model's capabilities. Additionally, while UDPM offers improved interpretability over "standard" denoising diffusion models, it still falls short compared to the interpretability shown for GANs. To address this, further research could explore enhancing the latent space structure of UDPMs to make it more interpretable, perhaps by incorporating techniques from GANs further analyzing the currently generated latent space.
Acknowledgement
This research partially supported by the Isracli Innovation Authority through the meta consortium.
The authors are grateful to the Neubauer family for their kind support.
10

---

References
[1] Shady Abu-Hussein, Tom Tirer, and Raja Giryes. Adir: Adaptive diffusion for image recon- struction. 2022.
[2] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. arXiv preprint arXiv:2206.02779, 2022.
[3] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18208-18218, 2022.
[4] Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S. Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv:2208.0985 2022.
[5] Stanley H Chan, Xiran Wang, and Omar A Elgendy. Plug-and-pll admm for image restoration: Fixed-point convergence and applications. IEEE Transactions Computational Imaging. 3(1):84-99) 18,88899-6600 2016.
[6] Yunjey Choi, Youngjung Uh, Jacjun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8188 8197,197, 2020.
[7] Hyungjin Chung, Eun Sun Lee, and Jong Chul Ye. Mr image denoising and super-resolution using regularized reverse diffusion. arXiv preprint (7Xii:2203.1262). 2022.
[8] Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G Dimakis, and Peyman Milan- far. Soft diffusion: Score matching for general corruptions. arXiv preprint arXiv:2209.05442.
2022.
[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794 2021.
[10] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Genie: Higher-order denoising diffusion solvers. arXiv preprint arX1v:2210.05475, 2022.
[11] Giorgio Giannone, Didrik Nielsen, and Ole Winther. Few-shot diffusion models.
10. 10445550ARXX12220555566. 2022.
[12] Florentin Guth, Simon Coste, Valentin De Bortoli, and Stephane Mallat. Wavelet score-based generative modeling, 2022.
[13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[15) Shady Abu Hussein, Tom Tirer, and Raja Giryes. Correction filter for single image super- resolution: Robustifying off-the-shelf deep super-resolvers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1428-1437, 2020.
[16] Alexia Martincaa, Ke Li, Rémi Tiché-Taileffe Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105. 14080, 2021.
[17] Zahra Kadkhodaie, Florentin Guth, Stéphane Mallat, and Eero Simoncelli. Learning multi- scale local conditional probability models of images, 2023.
[18] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In NewrIPS, 2022.

---

[19] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 4401 -400- 2019.
[20] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv: arXiv:2210.09276, 10.0927, 2022.
[21] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[22] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[23] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken. Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681 14690,, 2017.
[24] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li. and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint AXXiv:2066000227 2022.
[25] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint urXiv:221..01099 2022.
[26] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388, 2021.
[27] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162 8171. PMLR, 2021.
(28) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 8748-8763. PMLR, 2021.
[29] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[30] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Patlern Analysis and Machine Intelligence, 2022.
[31] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00511 2022.
[32] Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. Knn-diffusion: Image generation via large-scale retrieval. arXiv:2204.02844 2022.
[33] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1874-1883, 2016.
[34] Jascha vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-22666 2265. PMLR, 2015.
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper-
[35] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2022.
12

---

[36] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023.
[37] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13355, 2020.
[38] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1905-1914, 2021.
[39] Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality. In International Conference on Learning Representationss 2021.
[40] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16293-16303, 2022.
[41] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXiv:2112.07804, 2021.
[42] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features conference on computer vision and pattern recognition, pages 586-595, 2018.
perceptual metric. In Proceedings of the IEEE
[43] Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion probabilistic models. arXiv preprint arXiv:2202.09671, 1(3.1):2, 2022.
13

---

UDPM: Upsampling Diffusion Probabilistic Models SSppllementarry Material
Here we provide our social impact statement, extended derivations of our UDPM model, ablation studies on the latent space, and additional generation results.
A Social Impact Statement
The development of Upsampling Diffusion Probabilistic Models (UDPM) represents a significant advancement in the field of image generation, offering the ability to produce high-quality images with fewer computational resources. However, as with any powerful technology. there are potential risks and ethical considerations to address. One major concern is the potential misuse of this technology for creating realistic but deceptive images, such as deepfakes, which can be used to spread misinformation for malicious purposes. Additionally, the ability to generate high-quality synthetic images raises issues of copyright and ownership. potentially impacting artists and content creators whose work might be replicated modified without their consent.
To mitigate these risks, it essential to implement safeguards and establish ethical guidelines for the of UDPM and similar technologies. This can include developing robust detection methods for synthetic content, ensuring transparency in the creation and distribution of Al-generated media, and promoting responsible use among developers and end-users. Collaborating with policymakers to create regulations that address the misuse of generative models can also help in preventing harm. Furthermore, fostering open dialogue within the Al research community about the ethical implications and potential societal impacts of such technologies will be crucial in ensuring that advancements in this field are aligned with broader societal values and public interest.
B
Extended Derivations
B.1 Evidence Lower Bound
From the Evidence Lower Bound (ELBO) we have E[- log log (X S By [Log (x(o) hence
log (x(X).))
1.
po (x1-1 (x1)
E - log Po (xx)) s Eg
Eg
log logp(xx)
Σ
log g(xi/ x1-
q(x1.L (x0)
1=1
pa (x| (xi)
po (250 (x1)
Bg
log x(x))
log
log g(x1 (xn)
8(xi| X1
log po po (x)) (xo)
loo po (oxo xx1)
Bg
log p(xx)
9(x:-1 X1, N ) (xo)
q(x11 (xo)
Po (x))
By
log ((g)
log
log (0)((111110)) + Σ
log g( x (x))
(x, (1, Xo)
1=2
1=2
Po >n((x/ (x)|xx))
-log
g(x, xx)
po px (x))
B.g
log p(x))
log
log 4(xertx0))) + llog logg(xx(Xx))
(xx111 X1, Xo)
po (xo (x1]
llog
g(xxxx0)
log p(XL)
pe (x lo
E.g
log
a(x1.1 (xo)
al (xx:11 X1, Xo)
DEL -ox.(ooxx))l(((xxxx))
Dan xo)) E. [log pxolxx)))
1=2
14

---

where in (+) Bayes was used, particularly
((x((x[-)) = q(x((x/-1xx0) 9(x x2 xx|xx) 9(x4-11xx Xo ( (x))
gxx]-1|x0) g(x;-1 x0)
B.2
Lemmal
1
Proof. The characteristic function of Normal vector e NNu,, Σ) the form
(t) Eleex((it = explit" ETt,),
(17)
when μ 0 and Σ = we get
(c(t) = exp(-xtt t²t..
To prove the first claim of the lemma, it is sufficient prove that He has a characteristic function of the form (17). Denote the transpose operator of H. by H which is defined zero padding operator followed by applying flipped version the kernel of H. denoted by w. We have
(te(t) Elexp(it" (He))) exp(i(H() exp(i(?("t)) t
expp(it Hu 1 ('"EH"") (-=0,==1) exp(-211" HH("),,
2
He NO, HHHT.
All that remains is to express HH. under the assumption the support of the downsampling kernel. . The operator H. is defined as applying blur kernel w = [w-L1/2 11/2]+1..... .1..... We w1 |//2|
followed by subsampling with stride 7, which can be represented in matrix form by H S.W. where S, M and W R Specifically
1
00
0
0
10
0
0
S,
0
wo
W1
W [T/2]
0
w
w/22+1
W-L-/2j42
W111
W, =
W
Wp
W /1/
0
0
w
17/2|+1
W-2
Therefore we have
Wo
[1/2]
0
0
12/2]+1
S,W
0
to
10-L7/21+1
WL/22
(18)
can be observed from
18) the of W do not intersect with each other, result get
[|w]] 10
HH. =,,W(S,W)²=
S"
|| 12... (19)
15

---

B.3 Efficient representation of the forward process
Similar to earlier works, for efficient training, one would like to sample x; directly using xo. Which is viable when assuming that the support of w is at most and ||w|| = 1. Formally
X a; rai1 =; 0 1l10
+ 01-102) 0r11
-2:0 x7 =
Lemma
x a + d/e1,
where C1, 02 idd. I). By definition, e₁ and é₂ are independent identically distributed vectors, therefore one may write
x = - v ofoo-1 + offe
repeating the opening - 2 times leads to
X aj xo + afa aão
+ afja 10001100000 a30 2.. af off
af
0f
+ o9
e a x0 + t11 of
= - xo +1
+
ā1
02
ai
-ai x0 + o
where I On, of of Σ
k==1
and I..
B.4 Derivation of the posterior
Using Bayes theorem, we have for /
9xx;-1, xo)
g(x) (10-1)10)(((x-11 (20) Markav g(x//x/-))9( (x0)
qxx/1|xi,
(xx)
((x)(x))
x(xi
(+*)
1
x exp
||x 017
||x - 17'-77'' xxo12
2
207-1
expl
(x] Xi 20 H x1 + aix (xxxxx1 Hx
201
2
(x--X
T
X|-1
1
20 x|
71 xo + 0z (H- (711-197711-12 xx))
(+*) x
exp
2 x
H
T H
+
1
X1 + x
0n H
xi
+
o
ā 76 x0
which is Normal distribution form w.r.t. the random vector -. Note that in (+*) the proportional equivalence with relation to x;-1. Let q( (x 1 xo) Nu, Σ), then one may write
a H H
1 I
Σ Σ
(20)
of
0
and
Σ # n
H.
+
22
7
-xxo
X0.
(21)
of
0-
B.5
Mean and variance of the posterior
The terms in 20 and 21 seem hard to evaluate, however, in the following we present an efficient way to compute them. By definition H is structured from a circular convolution with blur filter followed by a subsampling with stride a. Therefore, one may use the poly-phase identity used in
15 (5 which states that HTH in this case is circular convolution between the blur kernel and its flipped version w followed by subsampling with factor a, formally
h (w ① flip(w)) La:
16

---

therefore, one may write H in the following form
=
(22)
where F is the Discrete Fourier Transform (DFT), F is the inverse DFT, and An diagonal operator representing the DFT transform of h. By plugging (22 into
20
we get
2
AnF +
I
of F AnF + FF=
An +
1)5,
1
1-1
of
0/-1
01-1
cquivalently
== F
Ah
1 F.
(23)
Therefore, as can be observed from (23). applying Σ is equivalent to applying the inverse of the filter
An + 7t o. o..1). which can performed efficiently using Fast Fourier Transform (FFT). Finally have
P
H x of
07-1
H1' xx
(24)
B.6 Sampling the posterior
As discussed in section 3.2 the proposed generation scheme requires sample from MM,, Et) each diffusion step, which due to the dimensions of Σ, is not naive, since E0.5 is needed in order to use the parameterization Xt-1 = (x;) + 20.5e where e N0, l). However, as we saw in section B.5 the operator 2, is equivalent to applying linear filter h, therefore one may seek to find a filter such that
Σ =
equivalently, we can break the DFT of the filter h to magnitude and phase such that
DFT(h} [DFT(h))e
|FFT (DFT
therefore
[d) / /DFF(h)] [DDFT(d) 1
Im mm [h]
arctan [DF
(25)
As result, applying 00.5 can be performed by employing the filter d defined in (25).
C Additional results
C.1 Latent space interpolation
As shown previously [19. generative models have very interesting property, where may modify the latent variable of the generative model order to control the "style" the generated result, for instance, given the latent variables of two images, one may interpolate the latent space and get an "in-between" images that in contrast to the pixel-domain interpolation, transition smoothly and naturally, as be seen in Figures and
C.2 Latent space perturbation
Similar to what is presented the main paper, we show additional latent variable perturbation in order to obtain better understanding of the latent space, where we add small noise single diffusion step and see how affects the generated result, as can be seen Figures and
C.3 Latent variable swapping
In addition to what we presented above, we provide an additional ablation study that shows the benefit of UDPM, that to swap the latent variables (noise maps) between two generated images, and study the effect on the generated result, can be seen in Figure B3
17

---

C.4
Generation Results
We provide below additional generation results UDPM on the CIFARIO dataset, as can be seen in Figure 14 We also provide an additional quantitative comparison to EDM [8] on the FFHQ [99
and AFHQv2 [6] datasets, as can be seen in Table 2
FFHQ
AFHQv2
steps
FID
steps
FID
EDM L18
79/5
2.39/344.763
79/5
1.96/266.024
UDPM (ours)
<1
7.41
<1
7.10
Table 2: FID scores comparison between UDPM and EDM 18 on the FFHQ [19 and AFHQv2 6
datasets. UDPM requires diffusion steps, which is equivalent 0.3 denoising steps of EDM.
18

---

CIFARIO
AFHQV2
FFHQ
Learning rate
10
10
10
Warmup steps
5000
5000
5000
Batch Size
64
64
64
Dropout
0.1
0.1
O
Optimizer
Adam
Adam
Adam
Number of GPUS
Table 3: Training hyperparameters.
CIFARIO
AFHQv2
FFHQ
Architecture
NCSN [37
NCSN [37
NCSN [37
Parameters
57.73M
65.41M
65.41M
Base channels
128
128
128
Channels Multiplier
2,
1,2,
1,2, 2,
Attention resolution
8, 4.2
16, 8,4
16, 8,
In channels
3
Out channels
12
12
12
Blocks per scale
4
4
Table 4: Model hyperparameters.
CIFAR
AFHQV2 DDGAN AI
FFHQ
Anchitecture
DDGAN 41
DDGAN AII 23.86M 192, 384, 312, 512, 512, 512
Parameters
18.95M
23. 86M
Channels
192, 384, 512, S12, 512
192, 384, 512, 512, 312, 312
In channels
Blocks per scale
Blocks type
NCSN down
NCSN down 0.1, 0.4, 0.8
NCSN down 0.1, 0.4, 0.8 0.3
Grad Penalty weight [41
0.025. 0.4
Dropout
0.4
0.4
Table 5: Discriminator network hyperparameters.
19

---

1=1
1=2
original image
1=3
Figure 8: The first 8 principal components of the covariance matrix computed over 128 images generated by fixing two diffusion steps and adding small perturbation noise to the third (indexed above).
1=1
[ 1= 2
original image
1=3
Figure 9; The first 8 principal components of the covariance matrix computed over 128 images generated by fixing two diffusion steps and re-randomizing the third (indexed above).
20

---

Figure 10: AFHQv2 6
latent space interpolation example. The four corner images are interpolated by weighted mixture of their latent noises, such that the other images are "in-between" images from the latent perspective, similar to what has been done in GANs 19]. All the images are of size 64 x 64.
21

---

Figure 11: FFHQ 19 latent space interpolation example. The four corner images are interpolated by weighted mixture of their latent noises, such that the other images are "in-between" images from the latent perspective, similar to what has been done in GANs 19]. All the images are of size 64 x 64.
22

---

noise deviation
Latent space perturbation for 64 x 64 generated images. The original image is the left, then the images generated by adding a small noise to the latent noise from diffusion step l. As can be seen, the initial diffusion steps (1 1) controls the fine details of the image, while the final diffusion step = 3) changes the semantics of the image.
Figure 12:
23

---

l= 3
l=2 2
1=1
1=3
1=
Figure 13: Latent variable swapping: Given the left and right images with the noise maps used for generating them, we replace the 1-th noise map of the image on the right with the I-th noise map of the image on the left to see how each diffusion step affect the result (middle columns).
24

---

FID=6.86149
Figure 14: Generated 32 32 images of CIFARIO 2 using conditional UDPM, requiring only diffusion steps; equivalent to 0.3 traditional denoising step.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See the limitation section in the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

Every assumption was explicitly mentioned in the paper, and the proof of every claim was added the supplemental material.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In the appendix we present a table of the configurations used the experiments shown in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The code and the models will be released publicly upon acceptance. The implementation details mentioned the paper allow the results be faithfully reproduced.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https://nipp cc/
Subblic/an httpg/uddde//eeeeeeeeeeeee
ss wessoonooliii ollcy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips. cct pcwppplpppccccccccccccccccccccll i c/ des CodeSubri ss i onPol icy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Tables and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: we report results using the same metrics used in all previous works.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All the details are reported in the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:s:///urriipssss. / http:///wwrrprr.oopaa..caaaaaaaa
Answer: [Yes]
Justification:
All the points are satisfied.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See the social impact statement.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: See the social impact statement.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets used the paper, including code, data, and models, are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paaperswis wwwoosocittcceeeeoo com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
30

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will publish code and models upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "kbKVv63ZDo", "reviewer": "Reviewer_9qFt", "summary": "This paper proposes a new training and sampling scheme for a diffusion model. The motivation is to enhance the effectiveness and interpretability of the diffusion model. Building upon the methods of DDPM, this paper introduces an upsampling operation into the Markov process, enabling the model to denoise and upsample simultaneously. Furthermore, through mathematical derivations, the reliability of this process is demonstrated. Experimental results ultimately show that in certain specific scenarios, the model outperforms existing alternatives.", "questions": "- What is the computational logic behind steps less than 1 in Table 1? Could you please provide a clear explanation?\n- It appears that this method involves special design considerations for the network structure. What is the actual inference latency of this model compared to baselines?\n- Traditional diffusion models, such as EDM, achieve significantly better results with more sampling steps due to their scalability. How scalable is the proposed method, and how does its performance compare?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The paper is well-written with a clear organizational structure, making it easy to follow.\n+ The paper propose a new diffusion model framework, complete with mathematical derivations, resulting in a loss function analogous to that used in DDPM.\n+ The discussion and comparison with related work, such as cold diffusion and soft diffusion, are clearly articulated, effectively highlighting the technical contributions of this paper.", "weaknesses": "- The motivation behind the study is not sufficiently clear, and the interpretability of the model has not been well demonstrated.\n- There is a lack of ablation studies on the loss function. The complexity of the loss, especially with adversarial training, may lead to instability during training.\n- The experiments were conducted only at a 64x64 resolution, leaving the scalability of the method unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's insightful comments. Our response is provided below.\n\n**Weaknesses**:\n1) It is well known that diffusion models severely suffer from heavy computations to produce pleasing-looking images due to two aspects: (i) The large number of diffusion steps and (ii) the large dimensions of the latent variables at each step. Compared to GANs, where a single inference step is used, with latent space much smaller than the generated image size; DDPMs have significant drawbacks.\nAs a result, in this paper, we narrow this gap significantly, reducing the computations using the same network by $\\sim 65\\\\%$, while outperforming the fastest SOTA diffusion-based models.  \nAdditionally, it has been shown in many GAN papers that the latent space of the generative model is interpolatable and interpretable, which was not the case with DDPMs, where even a single step DDPM has latent space with the same dimensions as the input image; making the latent space very redundant. In UDPM however, the whole latent space is much smaller than the generated image, making the latent space smooth for interpolation (as shown in Figures 5, 7, and 8) and interpretable (as shown in Figures 6, 9, 10). For further understanding of the interpretability of the model, we present an ablation study where we generate an image, then fix two of its three noise maps, and perturb/rerandomize the third noise map 128 times to produce 128 different images. We then take these images and analyze them by examining the principal components of their covariance matrix to understand how each diffusion step affects the generated image (check the rebuttal PDF file).\nAs can be seen in Figures 1 and 2 in the attached PDF file, the noisiest latent variable controls the semantics of the image (i.e. class, pose, background, etc.), while the initial and the middle noise level controls the fine details of the generation. As a result, one may get similar behavior to what has been done in StyleGAN simply by modifying the last diffusion variable; as demonstrated in Figure 3.\n\n2) Complexity: The reverse process in UDPM requires super-resolving the latent variable from the previous step, which necessitates the use of a sophisticated loss term, as shown by previous super-resolution literature [1, 2, 3], which we find crucial in our case for obtaining sharp and detailed results.  \nStability: In contrast to pure adversarial loss used in GANs, using it as a regularization term for guiding the network to sharp solutions is in fact very stable. The only part that needs tuning is the weights of each term, which from our experiments remained fixed to the reported values in the paper for all datasets. \nTo demonstrate the effect of each loss term, in Figure 4 in the attached rebuttal PDF, we show how each term contributes to the generation results. This ablation will be added and discussed in the revised version.\n\n3) As mentioned in the limitation section, training diffusion models require heavy computational resources, particularly when increasing the images resolution; therefore, due to our limited resources we leave such exploration to future research. Yet, for completeness, we ran our approach on LSUN-horses dataset with resolution 128x128 and reported the results in the attached rebuttal PDF file.\n\n\n**Questions**:\n\n1) Because the size of the diffusion variables changes throughout the diffusion process in UDPM, it is not possible to directly compare it to the denoising steps of DDPM. Therefore, we compare the total computations required by each algorithm; from which we obtain that UDPM uses $\\sim30\\\\%$ of the computations used in a single denoising step with the same network, or equivalently $\\sim0.3$ the time of a **single** diffusion denoising step.\n\n2) UDPM needs a network architecture that upsamples its input while being aligned to other DDPM works to allow direct comparison. Therefore, we use the popular SongUNet [4] used in many diffusion works while increasing the number of output channels from $3$ to $3\\times \\gamma^2$, followed by a depth-to-space layer that upsamples the output by rearranging the pixels (https://pytorch.org/docs/stable/generated/torch.nn.PixelShuffle.html). This architecture makes minimal changes to the network, assuring minimal additional latency over the baseline. This is reflected by the run time that we measure which indeed shows that.\n\n3) UDPM generalizes traditional denoising diffusion schemes, therefore one can omit $\\mathcal{H}$ in part of the diffusion steps (e.g. the last ones), such that some diffusion steps become denoising without upsampling. This may  enable increasing the diffusion steps to arbitrary choice, similar to traditional diffusion. However, we did not examine such an approach as we focused on efficiency, and we leave it for further research.\n\n[1] Photo-realistic single image super-resolution using a generative adversarial network   \n[2] Esrgan: Enhanced super-resolution generative adversarial networks  \n[3] Real-esrgan: Training real-world blind super-resolution with pure synthetic data  \n[4] Score-Based Generative Modeling through Stochastic Differential Equations\n\n\n\nGiven the substantial improvements made in response to your feedback, we kindly request you to reconsider the score you initially assigned to our submission. We believe that the revised version of our paper now better aligns with the high standards of the NeurIPS conference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "19swsn2euY", "reviewer": "Reviewer_AGqF", "summary": "The paper discusses the Gaussian diffusion modeling at different dimensionality by incorporating downsampling in the forward process. As a solution, the authors propose a new model called Upsampling Diffusion Probabilistic Model (UDPM), which reduces the latent variable dimension before adding noise. The reverse process then gradually denoises and upsamples the latent variable to produce a final image and tackles the computationally inefficient problem in previous diffusion models like DDPM. In the experiment, UDPM can generate images within 3 stages, which is computationally cheaper than a single step in DDPM and achieves better results.", "questions": "Please see the Weakness.", "limitations": "The authors have discussed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The method is novel and explores a diffusion process across variance scale and dimensionality. The proposed solution is technically sound.\n\n- The proposed method is computationaly efficient compared to previous diffusion models, which were designed on a fixed dimensionality and relied a following cascade of upsampling models to reach higher dimension. \n\n- The paper is overall easy to read and well organized.", "weaknesses": "- The authors do not elaborate on how do we determine the number of upsampling stages are needed. Moreover, how do we choose the resolutions in the training in order to balance the performance and computation cost? The authors may need to provide heuristics, theoretical analysis or empirical studies to guide the authors on the choices. \n\n- The expression \"steps $<1$\" is not rigorous. Using NFEs (numbers of fuction evaluation) and GPU time (or FLOPs) at different resolution stages may be more informative to the readers. \n\n- Some important related works are missing and lack of discussion. For example, Simple Diffusion studies the diffusion schedule in terms of the image dimensionality; LEGO diffusion and Matryoshka diffusion also discuss the diffusion modeling with variable dimensionality and the solutions are closely related. \n\nEmiel Hoogeboom, Jonathan Heek, and Tim Salimans. \"simple diffusion: End-to-end diffusion for high resolution images.\" In International Conference on Machine Learning, pp. 13213-13232. PMLR, 2023.\n\nHuangjie Zheng, Zhendong Wang, Jianbo Yuan, Guanghan Ning, Pengcheng He, Quanzeng You, Hongxia Yang, and Mingyuan Zhou. \"Learning stackable and skippable LEGO bricks for efficient, reconfigurable, and variable-resolution diffusion modeling.\" In The Twelfth International Conference on Learning Representations. 2023.\n\nJiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M. Susskind, and Navdeep Jaitly. \"Matryoshka diffusion models.\" In The Twelfth International Conference on Learning Representations. 2023.\n\n---------------\n\nSome minors:\n\n- Some references are not precisely cited. For example, Adir [1] and Wavelet SGM [12] missed the conference/journal title; ; score-sde [36] was published in ICLR 2021, DDGAN [40] was published in ICLR 2022, TDPM [42] was published in ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s valuable points. Our response to the reviewer’s comments are given below.\n\n**Weaknesses**:\n\n1) The number of steps is determined by the smallest noise resolution you want to start with. Then everything is fixed. From our experiments, we found out that beyond the 3 diffusion steps, the diffusion steps did not benefit the generation quality, hence we use 3 diffusion steps. We will make sure to clarify this is the revised version.\n2) Because the size of the diffusion variables changes throughout the diffusion process in UDPM, it is not possible to directly compare it to the denoising steps of DDPM. Therefore, we compare the total computations required by each algorithm; from which we obtain that UDPM uses $\\sim30\\\\%$ of the computations used in a single denoising step with the same network, or equivalently $\\sim0.3$ the time of a **single** diffusion denoising step.  \nYet, for the completeness of the comparison, below we report the average runtimes on CIFAR10, of UDPM with 3 steps and DDPM with a single denoising step when a similar network is used. Benchmarked on a single NVIDIA RTX A6000 GPU and averaged on 100K image generations for eliminating any unwanted overhead:  \nUDPM: 2008.21 FPS  \nDDPM: 735.68 FPS  \n$\\Rightarrow$ Speedup = 2.73x  \nThe runtimes will be added to Table 1 in the revised version.\n\n3) **Simple Diffusion**: This work investigates the noise scheduling and the network architecture, and their relation to the image's resolution. However, this approach complements our method, since it does not modify the diffusion structure itself and only optimizes the empirical setup.  \n**LEGO diffusion**: This approach proposes a LEGO bricks architecture for improving the training, efficiency, and resolution generalization of diffusion models. This approach is also complementary to UDPM since it can be used alongside our approach for further improvements.  \n**Matryoshka diffusion**: This method proposes to train a diffusion model on different resolutions simultaneously. The effectiveness of this approach shines when the desired generation resolution is relatively high, where it can generate images with resolutions similar to what Stable-Diffusion can produce but with no need for a separate image encoder. This approach however is very different from UDPM, since it does not modify the formulation of the diffusion model itself.   \nAll the missing references will be added to the revised version and discussed accordingly. We thank the reviewer for the effort.\n\nGiven the substantial improvements made in response to your feedback, we kindly request you to reconsider the score you initially assigned to our submission. We believe that the revised version of our paper now better aligns with the high standards of the NeurIPS conference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J94Ue5vY5J", "reviewer": "Reviewer_ZKuR", "summary": "This paper introduces a novel generative model called the Upsampling Diffusion Probabilistic Model (UPDM). UPDM aims to decrease the number of diffusion steps needed to generate high-quality images, resulting in a significantly improved efficiency compared to previous methods.", "questions": "1.\tIn ColdDIffuison, blur and noise can be utilized to train a diffusion model. In your method, downsample and noise is utilized at the same time to train a diffusion model. Please re-clarify your main contribution expect for this.\n2.\tPlease re-clarify the details for your network to handle images with different resolution.\n3.\tCan you explain how you balance the weights of L{simple}, L{per}, and L{adv}. Please provide more ablation studies.\n4.\tPlease compare the generation speed with other methods that speed up DDPM.\n5.\tPlease provide more details about your network.\n6.\tHow did the authors assure that 3 steps will obtain best performance?", "limitations": "The authors have addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThis paper is well-written, and the organization is great.\n2.\tThe motivation is clear enough.", "weaknesses": "1.\tSome symbols are not fully explained when they are used at the first time.\n2.\tThe datasets might be fully able to valid the effectiveness of your method.\n3.\tThe compared methods are relatively out-of-date.\n4.\tThe comparison metric is only FID.\n5.\tSome commas and labels in several equations are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the valuable comments. Our response is detailed below.\n\n**Weaknesses**:\n\n1) Will be fixed in the revised version. We thank the reviewer for the effort.\n2) Although the three datasets we examined UDPM on are diverse, with CIFAR10 and AFHQv2 being multi-class datasets; we additionally trained UDPM on the LSUN horses dataset at 128x128 resolution as can be seen in the attached rebuttal PDF file. \n3) The methods shown in the paper are state-of-the-art accelerated diffusion-based generative models. We will be glad to compare our methods to other works we might have missed.  There is a recent class of generative models derived from diffusion named consistency models [1], which can generate images with a single denoising step with an FID score of 8.70 on CIFAR10, which is still inferior to UDPM in both generation quality and efficiency.  \nNote that we do not compare UDPM to distillation-based approaches and specifically to the distillation version of the consistency models, as distillation is complementary to our technique. Thus, we compare only to direct training. Note also that even if distillation-based approaches are taken into account, they require at least a single denoising step, leading to much longer runtimes ($\\sim300\\\\%$) than UDPM.\n4) Following the reviewer's comment, we have the following inception score comparison:  \nDenoising student: 8.36  \nTDPM: 8.65  \nUDPM: **9.01**  \nWhich shows that UDPM outperforms current SOTA efficient diffusion models also in the inception score measure. In the revised version of the paper we will make sure to add this comparison to Table 1.  \n5) We will make sure to polish the paper in the revised version.\n\n**Questions**:\n1) Indeed in ColdDiffusion they propose multiple approaches for defining the forward diffusion process. However, ColdDiffusion does not address the existence of the reverse diffusion process in their formulation. While in UDPM, Lemma 1 allows explicit access to the reverse process defined in equations (10-12).\n2) UDPM needs a network architecture that upsamples its input while being aligned to other DDPM works to allow direct comparison. Therefore we use the popular SongUNet [2] used in many diffusion works while increasing the number of output channels from $3$ to $3\\times \\gamma^2$, followed by a depth-to-space layer that upsamples the output by rearranging the pixels (https://pytorch.org/docs/stable/generated/torch.nn.PixelShuffle.html). This architecture makes minimal changes to the network, assuring minimal additional latency over the baseline. This is reflected by the run time that we measure which indeed shows that.\n3) The practical objective function we use is built from three different terms:  \n$\\ell_1$: Promotes high fidelity and agreement between the diffusion steps.  \n$\\ell_{per}$: Guides the network to more perceptual pleasing estimations.  \n$\\ell_{adv}$: Complementary term to the $\\ell_{per}$, that makes sure the reconstructed variable statistics matches the true one.   \nWe found the combination of the three terms above very crucial for getting sharp and detailed generations. For instance, on using only $\\ell_1$ when training UDPM on CIFAR10 we got an FID score of $\\sim 60$, and when we added the $\\ell_{per}$ it reached $\\sim 30$. In the attached PDF file we show the effect of each loss term on the generation results. \nWe will make sure this ablation study is presented in the revised version of the paper.\n4) For the completeness of the comparison, below we report the average runtimes on CIFAR10 of UDPM with 3 steps, and DDPM with a single denoising step when a similar network is used. Benchmarked on a single NVIDIA RTX A6000 GPU and averaged on 100K image generations for eliminating any unwanted overhead:  \nUDPM: 2008.21 FPS  \nDDPM: 735.68 FPS  \nSpeedup = 2.73x  \nThe runtimes will be added to Table 1 in the revised version.\n5) We use the popular SongUNet used in many diffusion works, particularly the implementation of EDM [3] (https://github.com/NVlabs/edm/). The specific hyperparameters used are detailed Table 4 in the supplementary.\n6) The number of steps is determined by the smallest noise resolution you want to start with. Then everything is fixed. From our experiments, we found out that beyond the 3 diffusion steps, the diffusion steps did not benefit the generation quality, hence we use 3 diffusion steps. We will make sure to clarify this is the revised version.\n\n[1]  Consistency models by Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  \n[2] Score-Based Generative Modeling through Stochastic Differential Equations  \n[3] Elucidating the Design Space of Diffusion-Based Generative Models (EDM)  \n\nGiven the substantial improvements made in response to your feedback, we kindly request you to reconsider the score you initially assigned to our submission. We believe that the revised version of our paper now better aligns with the high standards of the NeurIPS conference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
In the following PDF file, we present the following ablation studies and results:

1) Additional demonstration of the interpretability of the model.
2) Ablation study on the contribution of each loss term.
3) Additional results on more a diverse dataset with higher resolution.

We hope after this significant improvement we will make the reviewers reconsider their initial scores.

Best regards

Meta-review: The paper introduces a generalized approach to the denoising diffusion process called the Upsampling Diffusion Probabilistic Model. The forward process reduces the dimensionality of the latent variable through downsampling, followed by the addition of noise. The reverse process progressively denoises and upsamples the latent variable to generate a sample from the data distribution. The reviewers were rather positive about this work, many indicating the novelty of the proposed approach, its theoretical grounding and computational efficiency. Several concerns raised by the reviewers were adequately answered during the discussion, including the scalability issue of testing with only small resolution dataset.
Decision: Accept (poster)