Pyramidal Denoising Diffusion Probabilistic Models
Authors: Dohoon Ryu
Jong Chul Ye
Venue: Submitted to ICLR 2023 (2023)
Keywords: Diffusion Model
Image Generation
Super Resolution

Abstract: Recently, diffusion model have demonstrated impressive image generation performances, and have been extensively studied in various computer vision tasks. Unfortunately, training and evaluating diffusion models consume a lot of time and computational resources. To address this problem, here we present a novel pyramidal diffusion model that can generate high resolution images starting from much coarser resolution images using a {\em single} score function trained with a positional embedding. This enables a neural network to be much lighter and also enables time-efficient image generation without compromising its performances. Furthermore, we show that the proposed approach can be also efficiently used for multi-scale super-resolution problem using a single score function.

Under review as a conference paper at ICLR 2023
PYRAMIDAL DENOISING DIFFUSION PROBABILISTIC MODELS
Anonymous authors Paper under double-bling review
ABSTRACT
Recently, diffusion model have demonstrated impressive image generation per- formances, and have been extensively studied in various computer vision tasks. Unfortunately, training and evaluating diffusion models consume a lot of time and computational resources. To address this problem to allow training with even a single GPU, here we present a novel pyramidal diffusion model that can gener- ate high resolution images starting from much coarser resolution images using a single score function trained with a positional embedding. This enables a neu- ral network be much lighter and also enables time-efficient image generation without compromising its performances. Furthermore, we show that the proposed approach can be also efficiently used for multi-scale super-resolution problem us- ing single score function.
INTRODUCTION
Diffusion models produce high quality images via reverse diffusion processes and have achieved impressive performances in many computer vision tasks. Score-based generative models (Song cet al. 2021b) produce images by solving a stochastic differential equation using a score function estimated by neural network. Denoising diffusion probabilistic models (DDPMs) (Ho al. 2020 SShh-Dickstein et al.) 2015) can be considered as discrete form of score- score-based generative models. Thanks to the state-of-art image generation performance, these diffusion models have been widely investigated for various applications.
For example, Rombach al. (202T) trained a diffusion model on the latent space of convolu- tional neural network (CNN)-basee generative model, which enabled various of tasks. Diffusion- CLIP (Kim & Ye| (22TT) leveraged contrastive language-image pretraining (CLIP) loss (Radford al. 2021) and the denoising diffusion implicit model (DDIM) (Song et al. 202 for text-driven style transfer. ILVR (Choi et al. 202T) proposed conditional diffusion models using unconditionally trained score functions, and CCDF (Chung al., 2021) developed its generalized frameworks and their acceleration techniques. Also, recently proposed models (Nichol et al.] 2021 Ramesh et al. 2022 have achieved incredible performances text-conditioned image generation and editing.
In spite of the amazing performance and flexible extensions, slow training and generation speed remains critical drawback. To resolve the problem, various approaches have been investigated. Rombach et al.) (2021) I): Vahdat et al. (2021) trained a diffusion model in low-dimensional rep- resentational space provided by pre-trained autoencoders. DDIM (Song et al. (221a)) proposed deterministic forward and reverse sampling schemes to accelerate the generation speed. [Son & Ermon (2020) proposed parameterization covariance term to achieve better performance and Taster sampling speed. Jolicoeur-Martinauu et al. (202T) used adaptive step size without any tuning. PNDM (Liu et al. (022) devised pseudo numerical method by slightly changing classical numer- ical methods (Sauer) (011T) for speed enhancement, [Salimans & Ho (2022) reduced the sampling time by progressively halving the diffusion step without losing the sample quality. Denoising dif- fusion GANs (Xiao et al. 2021) enabled large denoising steps through parameterizing the diffusion process by multimodal conditional GANs. For conditional diffusion, a short forward diffusion steps of corrupted input reduce the number of reverse diffusion step in SDEdit (Meng et al. 2021 and RePaint (Lugmayr et al.] 2022), whose theoretical justification was discovered in in CCDF (Chung al. 2021 using the stochastic contraction theory.

---

Under review as a conference paper at ICLR 2023
Progressive image generation from noises using the proposed method trained on FFHQ (Choi et al.) 2020) dataset. Three different resolution images are generated from noise through reverse diffusion processes using single model. In red boxes, the preservation of the semantic information at different resolution images is observed.
Figure 1:
Alternatively, this paper addresses the slow sampling time issue similar manner to the method in Saharia et al. (2021) Ho et al. (2022a) that refine low resolution images to high resolution using cascaded applications of multiple diffusion models. However, in contrast to (Saharia et al. 2021 Ho et al. 2022a), our model does not need to train multiple models, and can be implemented much lighter single architecture which results in speed enhancement in both training and inference without compromising the generation quality.
Specifically, in contrast to the existing diffusion models that adopt encoder-decoder architecture for the dimensional input and output, here propose score function using positional information, which gives flexibility in the sampling process of reverse diffusion. Specifically, our pyramidal DDPM can generate multiple resolution images using single score function by utilizing positional information as a condition for training and inference. Fig. shows the result of generated images in three different resolutions using only one model in the reverse diffusion process, which clearly demonstrates the flexibility of our method. Furthermore, as byproduct, we also demonstrate multi-scale super-resolution using single diffusion model.
conditional training method for the
The contribution of this work can be summarized as following:
We propose novel method of conditionally training diffusion model for multi-scale image generation by exploiting the positional embedding. In contrast to the existing diffusion model, in which the latent dimension and the output dimension are the same, in our method the output dimension can be arbitrarily large compared to the latent input dimension.
Using single score network, mitigate high computation problem and slow speed issue reverse diffusion process using a coarse-to-fi to-fie refinement while preserving the genera- tion quality. The key element for this is again the positional encoding condition for the diffusion model.
We present multi-scale super-resolution which recursively refines the image resolution ing single score model.
BACKGROUND
2.1 DENOISING DIFFUSION PROBABILISTIC MODELS
In DDPMs Ho et al. 2020 Sohl Dickstein et al. 2015), for a given data distribution xp ~ q(x)),
define forward diffusion process (( ( [ Markov chain by gradually adding Gaussian
2

---

Under review as a conference paper at ICLR 2023
2 channeels
(GB(3) channels
Avairoad Fondling
Random Resize
Model
Figure 2: Our training scheme. Two dimensional coordinate information is concatenated with the input image and randomly resized one of the target resolution. Then, two channels of coordinate values are encoded with the sine and cosine functions, and expanded to 2 x channels where L is the degree of positional encoding.
noise at every time steps where {5} variance schedule:
q( II (x, where g(x,) N( vI ,, t=1
(1)
With well scheduled {B)T=0- the forward process converts data distribution to an isotropic Gaus- sian distribution - T. Using the notation a; 1 3r and de II O's. sample from qa,, (D) in a closed form:
2t 2 /I 1+z where NO
As the reverse of the forward step q( q( intractable, DDPM learns parameterized Gaussian transitions po (0 The reverse process is defined Markov chain with learned mean and fixed variance, starting from p(x1) = NN(z:: 0, I):
(o( == (o(T)) II
Po (z, xx), where po 1x,) = (o (3)
where
1
1 1) = Va,
a (z))s((ee()) s0(
(0(
(4)
VI a
Here, so(0, t) is a score function and z t) is trained by optimizing the objective
(
min I(0), where I(O) R
z (v + v o,
After the optimization, by plugging the learned score function into the generative (or reverse) diffu- sion process, one can simply sample from po (0 2 by
Ft-1
æt + (1 ar (00()))))))))))))))))))))))))( + o.z.
(6)
2.2 POSITIONAL EMBEDDING
Positional embedding encoding is widely used in many recent studies in order give locational information to deep neural networks. For example, Devlin et al. (0018;; Dosovitskiy et al. (2020) add position embedding to every entry of inputs in the form of trainable parameters. Unlike trainable
embedding, Tancik et al. (2020) proposed method applying sinusoidal positional encoding of the coordinate values for various tasks such as image regression, 3D shape regression, or MRI recon- struction. As the wave is continuous and periodic, low dimensional information can be expanded to a high dimensional space of different frequencies. In particular, the distance between periodi- cally encoded vectors can be easily calculated by simple dot product, so that the relative positional information of the data is provided without any additional effort.
The positional information is also useful for training neural network for image generation. Authors in (Anokhin et al. 202I) modified StyleGAN2 (Karras et al., (220)) into pixel-wise image gener- ation model, which outputs RGB values from the input of 2D coordinate of pixels image. The

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Pyramidal reverse diffusion for image generation. It starts from initialization with Gaussian noise at the coarsest resolution, and ends HR.
Require: Ty, T,, Ata {a? 22 (ot) {ot 1: T R ~ NN(,00r
D Gaussian sampling at the low resolution
2: for TT to do
> Full reverse diffusion
3:
at Val
lat + (1 of satt ,l, pos pos(()) + of
4: end for
5: 2o <- 21
6: while æn is not HR do
7: H0, j, (xo), (i), (x
Upsample image and coordinate value Sample at A, Scaled reverse diffusion
8: Dr,At, 5 / Eu + v 01,4,
9: for t T, T, to do
10:
Øt-1 ag
Ft + (1 (aze, p,s(sp)) pos(s))) + 0f2
end for
12: end while
13:
return Zog
models of (Yu et al. 2022 Skorokhodov al. 2021 generate continuous video frame by giving position encoded temporal information to the image generative models. The decoder in (Lin et al., 2019) generates full images despite the model being only trained with image patches and their center coordinate. Also, [Lin et al. (2021); Ntavelis et al. (2022) train their models through patches and their coordinates, which allows the generative model to bring out arbitrary size of images.
PYRAMIDAL DENOISING DIFFUSION PROBABILISTIC MODELS
In this section, we provide detailed explanation of our method, called the Pyramidal Denoising Diffusion Probabilistic Models (PDDPM).
3.1 MULTI-SCALE SCORE FUNCTION TRAINING
Training with multi-scale images is feasible CNN-based models they rely on the convolution calculation with spatially invariant filter kernels (Albawi et al. (017). Leveraging this simple but strong characteristic of the architecture, our goal to train diffusion model such that it can under- stand different scale of the input by giving coordinate information as condition. Specifically, we concatenate an input image and coordinate values of each pixels (f,j),, while i, [0, 1] are nor- malized value of its ry coordinate. Then, random resizing the target resolution, 64/ 128/ 256 in our case, is applied on the merged input. The resized coordinate values are encoded with sinusoidal wave, expanded to high dimensional space, and act conditions when training shown in Fig. Specifically, the positional encoding function given by
pos(r) = sin (2 cos (2"y), sin (2*r)), cos (2² .......in sin (24-17), cos (2 (
172 72
7n.]"
where E [0, 1] and L denotes the degree of positional encoding and is dimension of vector. By denoting [8] is iN]" .j = [i j2 jNl as the collection of the normalized
and y coodinates from the N pixels of 24, the training cost function (5) can be converted as
L(0) = Er. [| 20(01, pos(s) pos(s))
Benefited from the UNet-like model structure (Ronneberger al. 2015), the cost function Eq. (8) is invariant to all different resolutions that the optimization can be performed with only a single network. This simple idea scale-free training of the score network significantly improves the network's flexibility of sampling process which will be discussed later. Importantly, this can also alleviate the problem slow training and low batch size problems especially when training with limited resources, the latter of which significant for higher performance generative tasks.
4

---

Under review as a conference paper at ICLR 2023
(+) Image generation
1
ene-stop forward diffuronn
Conebritts
(b) Super-resolution
1
one stup forword diflssion
mulli sllii-llpprn roverse diflusion
upssmpling
crawetep dithaenn
nulli stop reverse difusionn
Figure 3: Proposed inference procedure for (a) image generation and (b) super- super-resolution. At the lowest resolution, full reverse diffusion performed, which then upscaled and forward diffused with additional noise. The CCDF (Chung et al. (021) acceleration scheme is used as an acceleration scheme. For super-resolution, we imposes constraints (TO) at every step of the reverse process.
3.2 FAST REVERSE DIFFUSION THROUGH PYRAMIDAL REVERSE SAMPLING
Thanks to the multi-scale score function, the sampling speed, which is the most critical disadvantage of the diffusion models, can be also made much faster compared to single full DDPM (Song al.
2021a) reverse process. Although one may think that our method is similar to (Ho et al. 2022a Jing al. 2022) which trains additional score networks for sampling in lower dimensions, there eral important improvement our method. First, our method only uses single score model, which trained with the positional encoding as explained in (5). Second, inspired by the CCDF accelera- tion scheme (Chung et al. 2021), the reverse sampling process can be further accelerated by using the lowortresolution reconstruction as an initialization for the next higher resolution reconstruction.
Specifically, as shown in Algorithm and Fig. 3(a), we first set two different number of time steps, T, and Tg. Here, Ty is a total time steps for full reverse diffusion process without any short- cut path which is applied on generating images of the lowest resolution; T, is scaled time step for fast higher resolution reverse processes using Mar-aan diffusion process suggested in DDIM (Sng et al. (2021a). More specifically, starting from low resolution (LR) random Gaussian noise LL ~ (0, I), the full reverse diffusion process is first performed, which is much faster compared to the reverse diffusion process at the maximum resolution. Then the generated LR image upscaled twice to produce the initialization for the next resolution. Then, the noises are added to the scaled image through forward diffusion with T., after which the reverse diffusion process is performed from t T,At, to t generated with At, E 1). This procedure is recursively applied to the next higher resolution images. Using this pyramidal image generation, the total sampling time can be significantly reduced compared to the single-resolution image generation at the highest resolution. Furthermore, accord- ing the CCDF theory (Chung et al.) (202T), the reverse diffusion process a stochastic contraction mapping so that it reduces the estimation error from the forward diffused initialization at exponen- tially fast speed. Therefore, the required number of the reverse diffusion can significantly reduced when a better initialization is used. This is why we use the previous resolution reconstruction as an initialization for the next finer resolution for further accelcration. For theoretical details, see (Chung ct al. 2021).
and its position values
until the next higher resolution image is
3.3
PYRAMIDAL SUPER RESOLUTION WITH STABLE GRADIENT GUIDANCE
Recall that SR3 Saharia et al. 2021) iteratively refines images from low to high resolution using two different score network modules. Likewise, Ho et al. (022a); Ramesh et al. (2022); Song & Ermon (2020) generate low resolution images, and uses one or two different pretrained checkpoints for super-resolution. These diffusion models showed impressively high performance, but still
5

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Pyramidal reverse diffusion for super-resolution. It starts from initialization with input LR image Φi and ends at HR.
Require: Fo. Ta, 23 {az 1 (oi)
1: 2o < 20
Initialization using LR input image
HR do 3: Tu. i t (xo), ( ())
2: while æn
Upsample image and coordinate value Sample at At
4: 2T, - /0 At. Tu / TT,,,
5:
for = TgAls to do
Scaled reverse di diffusion
6:
1t f- Va
[t + (1 ; ( t, pos(t). pos(s))) + ofz
7: vo (0t) = Va?
(t + (1 aie ) s0 ( (Ct, L, pos(t), pos(()))
8: 2t-1 + 2t-1 Aa æz (21) 2o112
9: end for
10: end while
11:
return 26g
quires separately trained networks. However, in our case, only a single model with small number of forward and reverse processes is sufficient as explained in the following and Fig. b).
Here, the reverse step can be guided towards the target by subtracting the gradient to the intended direction as suggested in (Ho et al.] [2022b] Dhariwal Nichol, [202]] |Avrahami et al. 2022))
Specifically, starting from (2), the denoised prediction of æ given 2; is first computed by
Co (1)))) ==
Vāg (x. + (1 - ā,) (2, t. pos(t), pos(j)))
(9)
which mixed with the reverse diffusion samples x;-1 in (6) terms of additional gradient to update the sample:
Lt-1 + 2t-1 2 Xvx. (zt) 2ol12
(10)
where D denotes the samplign operator and Ty is the low-resolution measurement. Addi-
tionally, we use the CCDF acceleration scheme (Chung et al. (021) similar to the aforementioned
pyramidal image generation. One difference from the image generation is that even at the coarsest
resolution, the full reverse sampling at the coarsest level is not necessary as the lower-resolution measurement can be utilized as an initialization after upsampling. See Algorithm and Fig. (b).
EXPERIMENTS
4.1 DATASETS AND IMPLEMENTATION
We trained each score models for 1.2M iterations using the proposed method for FFHQ 256x256 (Choi et al., (020) and LSUN-Church 256x256 (Yu al. 2015) dataset, and proceeded 500k iteration for AFHQ-Dog 256x dataset. The model was trained with the batch size of 48/12/3 for 64/128/256 image resolution. FFHQ and L.SU--Chur used for super-resolution. Our model is To alleviate the memory and training speed problem in
dataset were used to evaluate
generation performances while AFHQ-Dop dataset based on an improved version of DDPM I the limited resource environment using one GeForce 1080 Ti for all model training, chose small size model and the details are described in supplementary materials. Also we used 1000 diffusion steps for all training. We used Adam optimizer of 0.0001 learning rate. The degree of positional encoding L in ( was set to 6. For the base setting of the inference, we set T'- 1000 and T. 100.
4.2 IMAGE GENERATION
We compared the quality of generated images among different reverse diffusion methods devised for fast sampling. Specifically, we trained a new model for numerical method (FON), DDIM, S S-PNDM
https:://gtthbb.com http:///pppeeanm/p///eeeeeeeee htmmmmooreeddddoo aat.rrredddddduffoooooo

---

Under review as a conference paper at ICLR 2023
(e)
(b)
[c]
[o]
f1
E
SR
SRt
SR 88
SRt
Figure 4: Result of super-resolution FFHQ and AFHQ-dog dataset. Upper row shows the results of x8 SR and bottom row is x4 result. (a) Ground Truth, (b) low resolution images, the results by (c) cubic interpolation, (d) SRGAN, (c) ILVR, (f) MCG on AFHQ and SR3 FFHQ, and (g) the proposed method.
Frechet Inception distance (FID)) comparison using 10k generated images. Sampling speed was also calculated by comparing with a full reverse diffusion process. Speed for the baselines were calculated using 100 sampling steps (gray-colored cells). The models are trained for 1.2M iterations for fair comparison.
Table 1:
Dataset
Method
50
100
200
Params
Speed
FON
26.71
18.91
16.70
16.13
16.22
x9.13
DDIM
37.87
27.71
20.45
18.30
16.85
x10.1
FFHQ
S-PNDM
28.59
22.62
18.13
16.71
16.48
114M
x9.92
F-PNDM
25.10
19.16
16.30
16.09
16.32
x9.20
Ours
15.78
16M
x18.1
FON
28.93
27.44
27.11
25.29
25.72
DDIM
26.72
26.65
25.89
24.36
25.30
LSUN
S-PNDM
29.79
27.15
26.82
26.93
26.33
114M
Church
F-PNDM
33.80
27.07
26.95
25.94
25.78
Ours
14.07
16M
and F-PNDM (Liu et al. 2022) for 1.2M iteration. The settings and training details are described in supplementary materials. As the model consists of 7 times larger parameters than ours, the training was performed on Quadro RTX 6000 to handle the low batch size problem. This means that the baselines are trained on better condition than ours. The model uses linear noise schedule and we chose 10, 20, 50, 100 and 200 steps for the total sampling steps. When sampling from ours, we set Al, = 0.3 so that all samplings procedure of higher resolution images can be done in only 30 steps. The sampling speed was calculated by comparing the full diffusion step of the larger model.
httpp::///g thu.. cow/ popping--- www.gggini/ii 1

---

Under review as a conference paper at ICLR 2023
Table FID(L), LPIPS()), PSNR(()))) STIM(1) score evaluation on super-resolution task. Bold face: best, underline: second best. (*:Unofficial re-implementation.)
FFHQ (256 256)
AFHQ (256 256)
SR facotor
x8
x4
Method
FID
LPIPS
PSNR SSIM
FID
LPIPS PSNR
SSIM
FID
LPIPS
PSNR
SSIM
FID
LPIPS PSNR
SSIM
Bicubic
125.7
0.278
28.84
0.851
151.3
0.446
24.73
0.703
35.34
0.280
29.10
0.816
69.47
0.429
25.34
0.671
SRGAN
46.85
0.204
29.45
0.857
71.69
0.296
26.20
0.753
22.61
0.247
29.48
0.826
45.35
0.329
25,91
0.688
SR3*
59.31
0.291
20.56
0.725
91.57
0.404
18.90
0.612
27.89
0.345
20.15
0.730
35.09
0.401
18.85
0.622
HLR
54.73
0.224
29.15
0.851
71.47
0.295
24.92
0.712
26.61
0.234
28.76
0.800
37.39
0.326
25.10
0.650
MCG
22.10
0.246
27.34
0.782
32.96
0.323
24.78
0.652
Ours
49.06 0.192
28.70
0,860
66.80
0.289
24.83
0.725
21.62
0.237
27.74
0.796
33.14
0.311
25.38
0.649
We sampled 10k images from each method and evaluated visual quality using Frechet Inception distance based pytorch-fiq The result on [ that method produces superior results compared to the baselines despite the faster sampling speed with much smaller architecture. This implies that the model focuses on generating realistic images lower resolution and adding fine details at higher resolution. Also the results on LSUN-Church dataset show that the training of the baselines was incomplete, whereas our method can produce qualitative results even with much limited resources.
We can further speed up the reverse process by re- spacing total diffusion steps of lower dimension im- ages. We changed 7 Т, 100 and studied the effect of At, image quality. By fixing to- tal diffusion steps same for all resolution, we com- pared FID score on generated image when At, value changes from 0.1 to 0.5. The sampling speed com- putation was done similar as done in Ta Setting T, = 100 and M, = 0.1 increased the speed to be 82.4 times faster than original full diffusion process, while generating convincing images. Table shoows that large sampling steps on high resolution produce higher FID scores, but the sampling speed decreases proportionally to At.. Also, compared to the baselines in Table [] our method produced comparable results to the baseline despite the faster sampling speed.
Table 2: Self-Comparison results. Ty is fixed to 100 which is shorter than the setting in Table T FID score and sampling speed are measured in the same way as in Table T
A
0.2
FFIQ
23.91
22.42
22.11
21.91
21,87
I.UUN Church
27.39
18.10
15.38
14.57
14.63
Speed
x8244
x50.3
x36.1
x22.9
4.3 SUPER RESOLUTION
Experiment was performed on two SR factors: x4. and x8. We have tested on FFHQ, AFHQ-Dog x 256 dataset with our models and other baselines: SRGAN, SR3, ILVR, MCG Chung et al. (2022) and traditional bicubic upsampling method for comparison. For SRGAN and SR3, there was no checkpoint for appropriate evaluation, so the models were trained on 32->256, 64 64-+256 for each dataset. Additionally, the original work of SR3 (Saharia et al. (021) cascaded two x4 upsamplers for super-resolution, but in this case we reported on the result super-resolution with one model for each SR factors. We made comparison through FIDJ, LPIPS PSNR and SSIM Here, LPIPS was calculated using the open source of the perceptual similarity of VGG (Simonyan & Zisserman 2014). As shown in Table 3 our method produced the best result FID and LPIPS for most of the cases, but not on PSNR and SSIM. Although SRGAN provides the best PSNR and SSIM values, it contains many image artifacts and the results of our method are more realistic show in Fig. 4
4.4 ABLATION STUDY
We performed ablation study pyramidal super-resolution and positional encoding. The ab- lation study on super-resolution was done by removing some steps of the resolution while up- sampling, in order see the effect of pyramidal super-resolution. For positional encoding. we
8

---

Under review as a conference paper at ICLR 2023
Figure 5: Generated images of at full resolution (1024x 1024) by our method trained with only 256 256, 512x512 patches. The model had never seen full resolution image.
first trained new model with multi-scale images without positional encoding. Second, we trained score model with patches of 256 256 and 512 512 images for generation of 1024 1024 images.
Pyramidal super-resolution was evaluated by com- paring the original method - 64 + 128 + 256 with the up-sampling procedures of 32 - 64 + 256, 32 + 128 + 256 and 32 - 256. The eval- uation was performed on both FFHQ and AFHQ- Dog dataset. The result in Table 4 shows that using all the resolution step of pyramidal super-resolution has achieved the best result. The CCDF (Chung et al. 2 acceleration scheme of every resolution has shown quality improvement of reconstructed im- ages.
Table 4: FID(.), LPIPS(() scores of ablation study on super-resolution tasks.
FFHQ
AFHQ
Method
FID
LPIPS
LPIPS
32 256
72.78
0.308
45.99
0.340
32
0.340
32 128 + 256
68.33
0.291
36,80
0.328
Original
0.311
For the first ablation experiment on the positional en- coding, we trained the model with the same experi- ment setting except for the positional encoding. It was trained for 800k iterations. Fig. [ shows the re- sult of the first experiment. Odd looking faces with multiple eyes, crushed face or dislocated facial fea- tures were generated. The result confirmed the im- portance of positional encoding.
Experiment patch-wise learning for very high- resolution image generation was performed using one NVIDIA RTX 3090. The model was trained for 400k iterations with batch size of 6/2 on 256/512 scaled image patches of CelebA-HQ (Karras al. 2017) dataset. When training, random resizing and cropping were used. Sampling was performed by setting Ty = T. 100. As seen in Fig. 55 although the score model had never a maximum resolu- tion input, 1024 x 1024 images were generated through our fast pyramidal image generation.
Figure 6: Images generated from a pyramidal DDPM without positional encoding. Facial features are not in proper location and some crushed.
5 CONCLUSION
In this work, we proposed a novel Pyramidal DDPM which is trained with conditions on the po- sitional information. We showed that this simple change improves the speed of reverse diffusion process and the performance of super-resolution and image generation. We also tested the effect of positional encoding by additional ablation experiments. Especially, without positional encoding, the model lost the ability to predict proper images at different resolution. Also patch-wise training further improved the flexibility of the score model, generating very high resolution images without using full resolution images. Given the significant advantages from simple modification, be- lieve that our method may further mitigate high computation problems in the diffusion models and be used for many other applications.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Saad Albawi, Tareq Abed Mohammed, and Saad Al-Zawi. Understanding of convolutional neural network. In 2017 international conference on engineering and technology (ICET), pp. 1-6. Icee, 2017.
Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Ko- rzhenkov. Image generators with conditionall--nndeeeane pixel synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16208-18218, 2022.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: Con- ditioning method for denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wor woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 1188-8197, 2020.
Hyungjin Chung. Byeongsu Sim, and Jong Chul Ye. Come-cloer-ddiffus--fsseee Accelerating con- ditional diffuusion models for inverse problems through stochastic contraction. arXiv preprint arXiv:2112.05146, 2021.
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint arXiv:2206.0094 2022.
Jacob Devlin, Ming-Wos Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 2018.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs image synthesis. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. arXiv preprint MtXiv:2010.11929, 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research. 23(47)11333 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458, 2022b.
Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion gener- ative models. arXiv preprint arXiv:2205.01990. 2022.
Alexia Obliceeu-Maatineau Ke Li, Rémi Piché-Taillefés Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models, arXiv preprint arXiv:2105. 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 10/96, 2017.
Tero Karras, Samuli Laine, Miia Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 81 10-8119, 2020.
10

---

Under review as a conference paper at ICLR 2023
Gwanghyun Kim and Jong Chul Ye. Diffusionclip: Text-guided image manipulation using diffusion models. arXiv preprint arXiv:2110.02711, 2021.
Chieh Hubert Lin, Chia-Che Chang, Yu-Sheng Chen, Da-Cheng Juan, Wei Wei, and Hwann-Tzomt Chen. Coco-gan: Generation by parts via conditional coordinating. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2019.
Chieh Hubert Lin, Hsin-Ying Ying Lee, Yen-Chi Cheng, Sergey Tulyakov, and ding-Hsuan Yang. Infin- itygan: Towards infinite-resolution image synthesis. arXiv e-prints, pp. arXiv-2104, 2021.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.0977 099000 2022.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. arXiv preprint arXiv:2201.09863 2022.
Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun Van Zhu, and Stefano Ermon.
SDEdit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01003, 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Evangelos Ntavelis, Mohamad Shahbazi, lason Kastanis, Radu Timofte, Martin Danelljan, and Luc Van Gool. Arbitrary-scale image synthesis. arXiv preprint arXiv:2204.02277 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. arXiv preprint arXiv:2112.1003 2021.
Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer. 2015.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint (00iv:2104.076:33 2021.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00511. 2022.
Timothy Sauer. Numerical analysis. Addison-Wesley Publishing Company, 2011.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. arXiv preprint arXiv:21 14683, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Leam- ing, pp. 2256-2265. PMLR, 2015.
11

---

Under review as a conference paper at ICLR 2023
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th International Conference on Learning Representations. ICLR, 2021a.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Advances in Neural Information Processing Systems, volume 33, pp. 12438-12448, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and
Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations, ICLR, 2021b.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Process- ing Systems, 33:7537-7547, 2020.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling latent space. Advances in Neural Information Processing Systems, 34: 1287-113800 2021.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans, arXiv preprint arXiv:21 2021.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1506.03365 2015.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Wor Woo Ha, and Jinwoo Shin. Generating videos with dynamics-awaar aware implicit generative adversarial networks. arXiv preprint arXiv:2202.0057,, 2022.
12

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL RESULTS
A.1 IMAGE GENERATION
Visualization of generated samples from the comparison methods are provided in Fig. 7 We also provided additional samples of different resolution generated from a single score model on FFHQ and LSUN-Chured dataset in Fig. D and Fig. [0] To verify the benefit of using positional encoding as condition, we added continuous magnification image generation (x3.5, x5) samples at Fig. B Especially, 320 320( x5) resolution results, which are larger than the maximum size of images used for training, shows the benefit of the proposed method.
A.2 SUPER- REPPERRESOLLTTON: INTERMEDIATE RESULTS
In this section, we provide intermediate results on pyramidal super-resolution task. Fig. 11 shows up-scaled images of each resolution starting from 32x32 on FFHQ and AFHQ-dog dataset.
A.3
SELF-COMPARISON RESULTS
We visualized the result in Table 4 in Fig. 12 and Fig. 13 Both of the result show that direct path from the low resolution image to the maximum resolution produce more details in the images. This implies the effect of iterative refinement method is not trivial.
B EXPERIMENTAL DETAILS
B.1 COMPARISON METHODS
B.1.1 IMAGE GENERATION
Pyramdial DDPM We implemented Pvramdial DDPM based on the original code of improved diffusion model (Nichol & Dhariwal 2021 We used the basic setting of the UNet model for generation of 64 x 64 images. Its stages use (64,128,256.512] channels from highest to lowest resolution with residual block for each layer. We employ attention 4 x 4 and 8 x 8 resolution to achieve better image generation quality on the lowest resolution. Exponential moving average over model parameters with rate of 0.999 is used for evaluation. The model was trained using on GeForce 1080 Ti.
FON, DDIM, PNDM We used the score function from PNDM (Liu al. 2022 official github repository 6
a baseline. Its stages use 2 residual blocks, from highest to lowest resolution with residual blocks for each layer and employed attention at 16 x 16. Exponential moving average over model parameters with rate of 0.999 is used for eval- uation. Adam optimizer with learning rate 0.0002 was used. This setting is exactly the same as the original setting when training LSUN-Church dataset suggested by the author. Each model for L.SUN-hhurd and FFHQ dataset was trained for 1.2M iterations for fair comparison with our model. FON, DDIM, S-PNDM and F-PNDM methods are used. The training was done with the batch size of 8 as the model was trained using Quadro RTX 6000.
channels
B.1.2 SUPER RESOLUTION
We trained a model from scratch on FFHQ and AFHQ-dog dataset using a code from The model was trained for 500k iteration for each dataset. The model was trained with batch size of 8, 0.001 learning rate, 0.9 31 and 0.999 B3 using Adam optimizer. We used one GeForce 1080 Ti for training.
SRGAN
SRGAN repository |
htteaes//attbbb.cccmm/mm//////ee ffusion https:///qittbbb.cmm///gggggg liu/PNN
13

---

Under review as a conference paper at ICLR 2023
SR3 We used unofficial implementation for SR3 model from github repository The official model trained two separate model for x4 super-resolution, but we trained single upsampler for 32 256 and 64 + 256. The training was done for 500k iteration with batch size of 2 for each dataset. The model consists of inner channels starting from 64 and 2 residual blocks. It was trained with 0.0001 learning rate using Adam optimizer. We used one GeForce 1080 Ti for training.
We used pretrained network for both dataset from ILVR repository It was trained for 1M iterations for FFHQ, and 500k for AFHQ-dog. The checkpoint of AFHQ-dog used for super-resolution task using MCG (Chung et al. 2022).
ILVR, MCG
httt:///tth..aco//eeee/r///aaaar -tt-n---ia----taatttataa ive Ref nnment
14

---

Under review as a conference paper at ICLR 2023
FON
DDIM
S-PNDD
F-PNDM
Figure 7: Generated images from comparison baselines(FON, DDIM, S-PNDM, F-PNDM) FFHQ and LSUN-Church dataset using 100 reverse diffusion steps.
224 224
320 x30
25
Figure 8: Continuous magnification image generation on 224 x 224(x3.5) and 320 x 320(3 x5) via
Pyramidal DDPM.
15

---

Under review as a conference paper at ICLR 2023
64 x 59
128 128
256 x 256
Figure 9: Additional generated images by our method trained from FFHQ dataset. Resolution increases starting from top bottom row.
16

---

Under review as a conference paper at ICLR 2023
64 x 64
shu Chadt ck
128 128
shupplartck
256x x 256
Figure 10: Additional generated images by method trained from LSUN-Church dataset. Reso- lution increases starting from top to bottom row.
17

---

Under review as a conference paper at ICLR 2023
Ground Truth
Low resolution
64
128 128
256 x 256
Figure 11: Intermediate results of pyramidal super-resolution on FFHQ and AFHQ-dog dataset.
18

---

Under review as a conference paper at ICLR 2023
Ground Truth
Low resolution
32->256
32->64
32->128>>256
Full
Figure 12: Visualization of the results of x8 super-resolution Table on FFHQ dataset.
19

---

Under review as a conference paper at ICLR 2023
Ground Truth
Low resolution
32->256
32->44>256
32>>12->2266
Full
Figure 13: Visualization of the results x8 super-resolution Table AFHQ-dog dataset.
20


Reviews:
[{"review_id": "iRYpbQYLsNk", "reviewer": "Reviewer_VnnC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper develops a variant (positional encoding) of conditional denoising diffusion probabilistic model (DDPM) with the goal of efficient sampling and multi-scale image generation while still maintaining the performance. By enforcing the gradient of the data-consistency during multi-stage sampling, the proposed diffusion model can also be used for image super-solution (SR). Both quantitative and qualitative results of this work are provided, comparing against several existing methods. The authors compared their method with several existing methods in terms of  sampling speed, image generation quality and SR performance.", "summary_of_the_review": "Consequently, given the pros and cons on balance, I feel this is a very borderline paper, and I vote for borderline accept tentatively", "strength_and_weaknesses": "Strengths:\n\nI think the overall idea is interesting: designing a diffusion model that assembles the positional encoding as condition for fast sampling and multi-scale image generation. I believe this is an overall well-written paper with some new results that could spark further research in the interesting topic of image continuous scale generation. But I fear that the justification and benefit for the usage of positional encoding as condition input is not well justified through the experiments. Please find my technical comments below.\n\nWeaknesses:\n\n1, As cooperated by the authors, using positional information for training image generator networks is not new.\n\n2, More importantly, for both image generation quality and SR restoration, the comparison with existing methods seems to be inaccurate. For image generation comparison, it is unclear why only the unofficial implementation results are reported. For example, in the original DDIM [Song.etal2021], the authors reported FID score of 10.84 on the LSUN-Church dataset when sampling 50 iterations.  Besides, the SR3 results show serious mean shift even for SRX4. The baseline methods seem to be re-implemented inappropriately. The pretrained DDPMs are public-available and reported similar results to the official implementation, such as https://github.com/openai/improved-diffusion. \n\n3, Table 1. reports improved sampling speed of this work, comparing to baseline methods.  However, the reasons lead to such speedup is not clearly presented through ablation studies. As mentioned by the authors the CCDF (Chung et al., 2021) acceleration scheme is used as an acceleration scheme. Since CCDF is a general acceleration scheme, for fair comparison, all methods should use the same acceleration settings.\n\n4, No continuous magnification results are showed in this paper. Since this work takes coordinate information as a condition and uses fully CNN architecture, it would be very informative to show the image generation or SR  results for continuous scale results (e.g., X3.2, X6.5).\n\n5, Some implementation details are confusing. “We trained our model using FFHQ 256×256 (Choi et al., 2020), LSUN-Church 256×256 (Yu et al.,2015) datasets for 1.2M iterations”. Is this means that a single model is trained on multiple dataset ?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UJGzL051ueN", "reviewer": "Reviewer_4NKS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new method to progressively generate or super-resolve to high-resolution images. It uses a diffusion approach for every up-sampling step, an idea it shares with Saharia et al. (SR3) and Ho et al. Its main innovation is in using a single model for all up-sampling steps (while SR3 used a separate model for each step).", "summary_of_the_review": "Given the weaknesses in the evaluation as mentioned above, I do not think the paper is ready for publication. A new revision containing evaluations comparing to baselines in settings reported in those papers needs to undergo a fresh round of reviews. Moreover, the paper should include a discussion of bias and fairness concerns.", "strength_and_weaknesses": "# Strengths\n\n- The idea of using positional embeddings to learn a single model to be applied across scales is interesting.\n\n# Weaknesses\n\nThe main problem with the paper is in its evaluation.\n\n- For the image generation task, the paper trains its own models for the competing methods, and the results are extremely surprising. While the competing methods are clustered together in their score, the proposed method does significantly better while also being faster and having an order of magnitude fewer parameters. For example, on the LSUN church dataset, the four other methods have scores ranging from 25.30 to 25.78, while the proposed method has a score of 14.07. The paper also doesn't include any visual examples of the images generated by the other methods, only its own.\n\nTraining requires setting and choosing many hyper-parameters, and given the significant gap in scores, it is possible that this difference is because of some hyper-parameter setting. Note that the other papers report their own numbers on different benchmarks. Why not train the proposed model under those settings, and report numbers comparing to those methods that way?\n\n- For the super-resolution experiments, the closest method to the proposed one is SR3 --- which also does iterative scale diffusion, except with separate networks for each scale. The authors report results with their own trained version of SR3, and it appears to perform significantly worse. Note that the reported results show SR3 with a 7-9db worse PSRN than SRGAN while in SR3's own evaluation, it performed equivalently / slightly better than SRGAN (also on face images). Indeed, SR3 has 6-8db worse PSNR than bicubic interpolation! Moreover, visually inspecting the results, there seems to be a very visible change in the color temperature of the SR3 outputs from those of all other methods. This again suggests that there might be some issue with the implementation of SR3 used for testing.\n\nIn this case again, the SR3 paper reports results on a benchmark with a clear description of the training and evaluation sets. Why not replicate those settings and compare to SR3 on those reported numbers?\n\n- In addition to the problematic comparisons to the baseline, the ablation studies could be improved. In particular, the ablation study should evaluate its specific novel contributions over previous methods. For example, it could show results from training a separate network for each scale, and demonstrate that with the proposed positional encoding, a single network comes close to matching results from such separate networks. It could show the benefit of CCDF and the projection steps, showing results when those were not applied.\n\n- Given that the paper focuses on image generation, and in particular evaluates on face images, it should discuss the potential for bias in generated images.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "1Wk5TC7qKKM", "reviewer": "Reviewer_HWig", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to solve the low-efficiency problem of diffusion models and presents a pyramidal diffusion model that can generate high-resolution images starting from much coarser resolution images using a single score function.\nThe key idea is to use a positional embedding, which enables a neural network to be much lighter and also enables time-efficient image generation without compromising its performance. The paper also shows that the proposed approach can be used for super-resolution using a single score function.", "summary_of_the_review": "The usage of positional encoding in the diffusion model is interesting. However, the results are not quite convincing. Also, the paper has writing issues. See more details in Strength And Weaknesses.", "strength_and_weaknesses": "Strength:\n\n- The core idea of incorporating positional encoding into the diffusion model is a valid contribution. It allows multi-scale sampling with a single score function model. The effectiveness of this strategy is verified with ablation study.\n\n- The paper shows good sampling speed for diffusion-based image generation.\n\n- The paper makes a good literature review about denoising diffusion models in the introduction, especially the works on applications of diffusion models.\n\n\nWeakness:\n\n- The results of the proposed algorithm do not look very good. While I understand this might be due to the small model size used in this paper, it will be important to show that the algorithm has the potential to achieve better performance by slightly enlarging model sizes.\n\n- For the speed comparison, it is important to compare with existing methods that are also based on multi-scale sampling, for example, SR3 and Cascaded Diffusion Model.\n\n- The paper is not well written, and some contents are hard to understand. For example:\n\n(1) What is 2x2 in positional encoding in Figure 2? \n\n(2) On Line 6 of Algorithm 1, I think it should be \"while $x_0$ is 'not' HR do\". Please clarify if I misunderstand this.\n\n(3) On P6, the paper says \"1000 diffusion steps for all training\". It is not clear how these steps are split between different scales. Also, I don't know why the number of steps is different in inference: $T_f$=1000, $T_s$=100, which is more than 1000 in total.\n\n(4) On P8, the paper mentioned Table 2(b) and Table 1(a) which however are never presented in the paper.\n\n(5) On P9, the paper presents an experiment where the score model is trained with patches of 256×256 and 512×512 images for the generation of 1024×1024 images. What is this experiment for? How do we interpret this result?\n\n(6) Typo: on P6, $T^f$ and $T^s$ should be $T_f$ and $T_s$; in Figure 3, diffsion --> diffusion", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "11NWjpUOF-s", "reviewer": "Reviewer_REZk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about training a multi-scale diffusion model in a unified framework. Specifically, it uses a casacaded mutli-scale generation process (from lower-resolution to higher-resolution). Unlike prior work which uses several score networks at each scale level. This paper uses a single network and proposes to use spatial positional encoding to make the network scale-aware and location aware. This spatial positional encoding also enables patch-wise training, which makes it feasible for large-scale image generation. Results demonstrated its ability to generate high-fidelity images at various scales.", "summary_of_the_review": "I'm not fully aware of the field of diffusion models, so maybe I missed some prior works. From my point of view, this paper is novel and interesting. The contribution of this paper is clear. It uses spatial positional encoding to solve multi-scale cascaded generation with a single score function. Besides, this also enables patch-wise training.\nThe results also seem good.", "strength_and_weaknesses": "Strength.\n1. The idea of using a single score function to progressively generate higher-resolution images is interesting. And the use of spatial positional encoding seems an elegant solution for this task.\n\n2. Introducing spatial positional encoding also enables patch-wise training of the diffusion model, which makes the training process much more flexible.\n\nWeakness.\n1. I do not fully agree with the claim of the super-resolution capability of the proposed method, although I do agree with the cascaded generation method. As seen from the red box of Fig.1, the 2x super-resolution breaks the consistency between the input lower-resolution image and the output image. (the mouth part). I think this is because every time a generated image is super-resolved, Gaussian noise is added during the forward diffusion process.  Although this is arguable because super-resolution can be ill-posed, however 2x super-resolution should not produce such a mismatch. I believe that's also why only on 4x~ super-resolution task, can this method outperforms existing works.\n\n2. The comparison with SR3 seems weird. As seen in Figure.4, the generated images of SR3 have severe chromatic distortion. I think the unofficial implementation of SR3 may be buggy.\n\n\n\nOther comments\nTypos: 1. missing closing brackets in the reverse diffusion process in Algorithm.1 and Algorithm.2 ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject