Relay Diffusion: Unifying diffusion process across resolutions for image synthesis
Authors: Jiayan Teng
Wendi Zheng
Ming Ding
Wenyi Hong
Jianqiao Wangni
Zhuoyi Yang
Jie Tang
Venue: ICLR 2024 spotlight (2024)
Keywords: generative models
diffusion model
image synthesis
Primary area: generative models
TL;DR: An efficient cascaded diffusion framework with sota sFID on ImageNet

Abstract: Diffusion models achieved great success in image synthesis, but still face challenges in high-resolution generation. Through the lens of discrete cosine transformation, we find the main reason is that *the same noise level on a higher resolution results in a higher Signal-to-Noise Ratio in the frequency domain*. In this work, we present Relay Diffusion Model (RDM), which transfers a low-resolution image or noise into an equivalent high-resolution one for diffusion model via blurring diffusion and block noise. Therefore, the diffusion process can continue seamlessly in any new resolution or model without restarting from pure noise or low-resolution conditioning. RDM achieves state-of-the-art FID on CelebA-HQ and sFID on ImageNet 256$\times$256, surpassing previous works such as ADM, LDM and DiT by a large margin. All the codes and checkpoints are open-sourced at \url{https://github.com/THUDM/RelayDiffusion}.

Published as a conference paper at ICLR 2024
RELAY DIFFUSION: UNIFYING DIFFUSION PROCESS
ACROSS RESOLUTIONS FOR IMAGE SYNTHESIS
Jiayan Teng"', Wendi , Ding*lat,
Wenyi Hong', Jianqiao Wangni", Zhuoyi Yang', Jie Tangl!
"equal contribution 'ssnnhuua University *Zhipu AI corresponding authors fteng 2y @ma 1 1s, zhengwd2 3mails, jietangemail) ts tsinghua edu mingding. thuâ gma 1. com
ABSTRACT
Diffusion models achieved great success in image synthesis, but still face chal- lenges in high-resolution generation. Through the lens of discrete cosine transfor- mation, find the main reason is that the same noise level on a higher resolution results in a higher Signal-to-Nore Ratio in the frequency domain. this work, we present Relay Diffusion Model (RDM), which transfers low-rowollllo resolution image or noise into an equivalent high-resolution one for diffusion model via blurring diffusion and block noise. Therefore, the diffusion process continue seam- lessly in any new resolution or model without restarting from pure noise or low- resolution conditioning. RDM achieves state-of-the-art FID on CelebA-HQ and sFID on ImageNet 256x256, surpassing previous works such as ADM, LDM and DiT by a large margin. All the codes and checkpoints are open open-sourced https: //tph/ug.ttb/../////// com/ THUDM/ HHUDM/RelayDiffUSS
LDM-4
10
DIT-XL/2
FDD.50k
AQM-U
MDT-XL/2
RDMoour
200M
600M
10
1.40
1.00
Number of Training Images
Figure 1: (left): Generated Samples by RDM on ImageNet 256x256 and CelebA-HQ 256x256. (right): Benchmarking recent diffusion models class-conditional ImageNet 256x256 generation without any guidance. RDM can achieve FID of 1.99 (and a class-balanced FID of 1.87) if with classifier-free guidance.
INTRODUCTION
Diffusion models et al. 2020 Rombach et al.
2022) succeeded GANs (Goodfellow et al. 2020)
and autoregressive models (Ramesh al. 2021 Ding et al. 2021) to become the most prevalent

---

Published as a conference paper at ICLR 2024
generative models in recent years. However, challenges still exist in the training of diffusion models for high-resolution images. More specifically, there are two main obstacles:
Training Efficiency. Although equipped with UNet to balance the memory and computation cost across different resolutions, diffusion models still require a large amount of resources to train on high-resolution images. One popular solution is train the diffusion model a latent (usually 4x compression rate in resolution) space and map the result back as pixels (Rombach et al. 20222) which is fast but inevitably suffers from some low-level artifacts. The cascaded method (Ho al.
2022 Saharia et al. 2022) trains a series of varying-size super-resolution diffusion models, which effective but needs a complete sampling for each stage separately.
Noise Schedule. Diffusion models need a noise schedule to control the amount of the isotropic Gaussian noise at each step. The setting of the noise schedule shows great influence over the perfor- mance, and most current models follow the linear (Ho et al. 2020) or cosine (Nichol & Dhariwal, 2021] schedule. However, an ideal noise schedule should
resolution- deppution-deeendent (See Figure 2 or Chen (2023)), resulting in suboptimal performance to train high-resolution models directly with common schedules designed for resolutions of x22 or 64 x4 pixels.
These obstacles hindered previous researchers from establishing an effective end-to-end diffusion model for high-resolution image generation. DDhrrwwl&& Nichol (202T) attempted to directly train 256 ADM but found that it performs much worse than the cascaded pipeline. Chen (2023) and Hoogeboom et al. (2033) carefully adjusted the hyperparameters of the noise schedule and architecture for high-resolution cases, but the quality is still not comparable to the state-of-the-ar cascaded method (Saharia et al. 2022).
In our opinion, the cascaded method contributes in both training efficiency and noise schedule: (1) It provides flexibility to adjust the model size and architecture for each stage to find the most efficient combination. (2) The existence of low-resolution condition makes the early sampling steps easy, so that the common noise schedules (optimized for low-resolution models) can be applied as a feasible baseline to the super-resolution models. Moreover, (3) high-resolution images are more difficult to obtain on the Internet than low-resolution images. The cascaded method leverages the knowledge from low-resolution samples, meanwhile keeps the capability to generate high-resolution images. Therefore, it might not be a promising direction to completely replace the cascaded method with an end-to-end one at the current stage.
The disadvantages of the cascaded method are also obvious: (1) Although the low-resolution part determined, complete diffusion model starting from pure noise is still trained and sampled for super-resolution, which is time-consuming. (2) The distribution mismatch between ground-truth and the generated low-resolution condition will hurt the performance, so that tricks like conditioning augmentation (Ho et al. 2022) become vitally important to mitigate the gap. Besides, the noise schedule of high-resolution stages are still not well studied.
Present Work. Here present the Relay Diffusion Model (RDM), new cascaded framework to improve the shortcomings of the previous cascaded methods. In each stage, the model starts diffusion from the result of the last stage, instead of conditioning it and starting from pure noise. Our method is named as the cascaded models work together like a "relay race". The contributions of this paper can be summarized as follows:
We analyze the reasons of the difficulty of noise scheduling in high-resolution diffusion models in frequency domain. Previous works like LDM (Rombach et al.) (022) assume all image signals from the same distribution when analyzing the SNR, neglecting the differ- ence in frequency domain between low-resolution and high-resolution images. Our analysis successfully accounts for phenomenon that the same noise level shows different perceptual effects on different resolutions, and introduce the block noise to bridge the gap.
We propose RDM disentangle the diffusion process and the underlying neural networks in the cascaded pipeline. RDM gets rid of the low-resolution conditioning and its distribu- tion mismatch problem. Since RDM starts diffusion from the low-resolution result instead of pure noise, the training and sampling steps can also be reduced.
We evaluate the effectiveness of RDM on unconditional CelebA-HQ 256x 256 and condi- tional ImageNet 256 256 datasets. RDM achieves state-of ofte---artttttttt FID on CelebA-HO and sFID on ImageNet.
2

---

Published as a conference paper at ICLR 2024
2 PRELIMINARY
2.1 DIFFUSION MODELS
To model the data distribution Pduto (xo), denoising diffusion probabilistic models (DDPMs, Ho iet al. (2020) define the generation process as Markov chain of learned Gaussian transitions. DDPMs first assume forward diffusion process, corrupting real data X6 by progressively adding Gaussian noise from time steps 0 to T, whose variance { } is called the noise schedule:
(xxx(xx1)) N(x;; -
(I)
The reverse diffusion process learned by a time-dependent neural network to predict denoised results at each time step, by optimizing the variational lower bound (ELBO).
Many other formulations for diffusion models include stochastic differential equations (SDE, Song et al. (020bb), denoising diffusion implicit models (DDIM, [Song et al.) (2020a)), etc. Karras et al. (2022) summarizes these different formulations into the EDM framework. In this paper, we gener- ally follow the EDM formulation and implementation. The training objective of EDM is defined as L error terms:
Exx-p /oo.://o +e, -
where p(o) represents the distribution of continuous noise schedule. D(x + oc, represents the denoiser function depending on the noise scale. We also follow the EDM precondition for D(x a) with o-dependent skip connection (Karras al. 2022).
Cascaded diffusion model (CDM, eo al. 2022) is proposed for high-resolution generation. CDM divides the generation into multiple stages, where the first stage generates low-resolution images and the following stages perform super-resolution conditioning on the outputs of the previous stage. f- DM (Gu al. 2022) unifies multiple resolutions of image generation with a linear interpolation in single model. Cascaded models are extensively adopted in recent works of text-to-iman to-image generation, e.g. Imagen (Saharia et al. 2022), DALL-E-2 (Ramesh et al. 2022) and eDiff-I (Balaji al. 2002).
2.2 BLURRING DIFFUSION
The Inverse Heat Dissipation Model (IHDM) RRissanen et al. 2022) generates images by reversing the heat dissipation process. The heat dissipation is a thermodynamic process describing how the temperature u(z, y, at location (x, y) changes in (2D) space with respect to the time t. The dynamics can be denoted by PDE
du St
g2
Dr
By
Blurring diffusion (Hoogeboom & Salimans, 2022) is further derived by augmenting the Gaus- sian noise with heat dissipation for image corruption. Since simulating the heat equation up to time t is equivalent to a convolution with a Gaussian kernel with variance o² = 2t in an infinite plane (Bredies al., 2018), the intermediate states æ; become blurry, instead of noisy in the stan- dard diffusion. If Neumann boundary conditions are assumed, blurring diffusion in discrete 2D) pixel space can be transformed the frequency space by Discrete Cosine Transformation (DCT) conveniently as:
quu (ua) N(u Dau, o!I)
where u, = DDT((;)) and D. All is a diagonal matrix with - coordinate (i. Here Gaussian noise with variance of is mixed into the blurring diffusion process to transform the deterministic dissipation process to a stochastic one for diverse generation.
W for
METHOD
3.1 MOTIVATION
The noise schedule is vitally important to the diffusion models and is resolution-dependdnn. A certain noise level appropriately corrupting the 64 x 64 images, could fail to corrupt the 256 x 256 (or higher resolution) images, which is shown in the first row of Figure [(a)(b). Chen (2023) and but we found that an analysis
Hoogeboom et al. (2023) attributed this to the lack of schedule-tuning from the perspective of frequency spectrum help better understand this phenomenon.
3

---

Published as a conference paper at ICLR 2024
(a) 64x64 Gaussian noise.
(b) 256x256 Gaussian noise, the same variance with (a).
(c) 256x256 Block/4] noise, the vuriance with (a).
Images
Noise
DOS Conton 200
Happ
slet Cessiion vebetigh leel
Signal Noise Ratio(SNR) across frequencies
The ligio Enguncy anad S.xx
light auiiido
Tlack with 6000
Figure 2: Illustration of spatial and frequency results after adding independent Gaussian and block noise. (a)() At the resolution of 64 x 64 and 256 x 256, the same noise level results in different perceptual effects, and in the frequency plot, the SNR curve shifts upward. (c) The independent Gaussian noise at the resolution 64 x 64 and block noise (kerell size = 4) at the resolution 256 x 256 produce similar results in both spatial domain and frequency domain. The noise is NO, 0.3?) for (a). These SNR curves are universally applicable to most natural images.
Frequency spectrum analysis of the diffusion process. The natural images with different reso- lutions can be viewed the result of visual signals sampled varying frequencies. To compare the frequency features of a 64 x 64 image and a 256 x 256 image, we can upsample the x 64 one to 256 256, perform DCT and compare them in the 256-point DCT spectrum. The second row of Figure 2(a) a) shows the signal noise ratio (SNR) at different frequencies and diffusion steps. In Figure bb), clearly find that the same noise level on higher resolution results in higher SNR in (the low-frequency part of) the frequency domain. Detailed frequency spectrum analysis are included in Appendix D
At a certain diffusion step, a higher SNR means that during training the neural network presumes the input image more accurate, but the early steps may be able generate such accurate images mismatch will accumulate over step by step during
after the increase in SNR. This sampling. leading the degradation performance.
Block noise as the equivalence at high resolution. After the upsampling from 64 x 64 to 256 x 256, the independent Gaussian noise on 64 64 becomes noise on grids, thus greatly changes its frequency representation. To find variant of the x s-grid noise without deterministic boundaries, we propose Block noise, where the Gaussian noises are correlated for nearby positions. More specifically, the covariance between noise Ezo.90 and Crr.gn is defined as
Cov(Exo.so:E
max (0, dis(ro, max (0, s
dis (yoo vi)),
(4)
where o2 is the noise variance, and & is a hyperparameter kernel size. The dis(:.)) function here is the Manhattan distance. For simplicity, we "connect" the top and bottom edges and the left and right edges of the image, resulting in
dis(ro, x1) = min (Iru- xzl. Zmar
x1
(5)
The block noise with kernel size can be generated by averaging s x s independent Gaussian noise. Suppose we have an independent Gaussian noise matrix c, the block noise construction function [-ock[s][)) is defined as
Block[s](<) =
Ez-
(6)
j=0
where Block[s((e)x,,, is the block noise at the position (z, y), and c-z Czmax -p- Figure (
shows that the block noise with kernel size s = 4 on 256 x 256 has a similar frequency spectrum as the independent Gaussian noise 64 x 64 images.
4

---

Published as conference paper at ICLR 2024
The analysis above seems to indicate that design end-to-end model for high-resolution images by introducing block noise in early diffusion steps, while cascaded models already achieves great success. Therefore, revisit of the cascaded models is necessary.
Why does the cascaded models alleviate this issue? Experiments in previous works (Nichol & Dhariwal 2021 Dhariwal & Nichol 2021) have already shown that cascaded models perform better than end-to-end models under a fair setting. These models usually use the same noise schedule in all stages, why are the cascaded models not affected by the increase of SNR? The reason that in the super-resolution stages, the low-resolution condition greatly eases the difficulty of the early steps, so that although the higher SNR requires more accurate input, the accuracy is within the capability of the model.
natural idea is that since the ooe-freuenccy information in the high-resolution stage has already been determined by the low-resolution condition, we can continue generating directly from the up- sampled result to reduce both the training and sampling steps. However, the generation of low- resolution images is not perfect, and thus the solution of the distribution mismatch between ground- truth and generated low oow-resolution images a priority to "continue" the diffusion process.
3.2 RELAY DIFFUSION
Ordinary diffusion
Blurring diffusion: Continue from block noisy upsampled images 256x256 images
from noise 64x64 images
Upsample
Block noise
XT
X
Xt-1
Xo
[
"baton passing the relay race"!
64 diffusion model
256 256 diffusion model
Figure 3: Pipeline of Relay Diffusion Models (RDM).
We propose relay diffusion model (RDM), cascaded pipeline connecting the stages with block noise and (patch-level) blurring diffusion. Different from CDM, RDM considers the equivalence of the low-resolution generated images when upsampled to high resolution. Suppose that the generated 64 x 64 low-resolution image x x" + e,, can be decomposed into a sample in real distribution x- and remaining noise ε9, (O, I As mentioned in section 3.1 the 256 256 equivalence of ez is Block[4] noise with variance Pg denoted by еş. After (nearest) upsampling, x" becomes x", where each x 4 grid share the same pixel values. We can define the starting state of
patch-wise blurring diffusion.
Unlike blurring diffusion models (Rissanen al. 2022 [Hoogeboom Salimans, 2022) that per- form the heat dissipation on the entire space of images, we propose to implement the heat dissipation on each 4 patch independently, which is of the same size as the upsampling scale. We first define series of patch-wise blurring matrices {DP), which is introduced in detail in Appendix A. The forward process would have a similar representation with equation 3
g( (z)) NNx VD Vx,,,, 313),
t (),
where V is the projection matrix DCT and o, is the variance of noise. Here the D chosen to guarantee V DJ, V xp in the same distribution as " meaning that the blurring process ultimately makes the pixel value in each 4 patch the same.
5

---

Published as a conference paper at ICLR 2024
The training objective of the high-resolution stage of RDM generally follows EDM Karras al. 2022) framework in our implementation. We replace the Gaussian noise in equation with a mixture of Gaussian noise and block noise in section 3.1 The loss function is defined the prediction of denoiser function D to fit with true data a, which is written as:
Po--aate:t-1100 N(0. (D D(xx 01) x11²;
(c+ a Block[s)(()) where Et = -VDVV VI+
(8)
burring
block noise
where and e' are two independent Gaussian noise. The main difference in training between RDM and EDM is that the corrupted sample æ; is not simply 2; = e. but mixture of the blurred image, block noise and independent Gaussian noise. Ideally, the noise should gradually transfer from block noise to high-resolution independent Gaussian noise, but we find that a weighting average strategy perform well enough, because the low-frequency component of the block noise is much larger than the independent Gaussian noise, and vice versa for eeghfrreuency component. is hyperparameter and the normalizer Vi+a? is used to keep the variance of the noise, o² unchanged.
The advantages of RDM compared CDM includes:
RDM is more efficient, because RDM skips the re-generation of low- low-frequency information in the high-resolution stages, and reduce the number training and sampling steps.
RDM simpler, because gets rid of the low-resolution conditioning and conditioning augmentation tricks. The consumption from cross attsstattntiion with the low-resolution condi- tion is also spared.
RDM more potential in performance, because RDM is a Markovian denoising process (if with DDPM sampler). Artifacts in the low-resolution images could be corrected in the high-resolution stage, while CDM is trained to correspond to the low-resolution condition.
Compared to end-to-end models
(Chen 2023
Hoogeboom et al. 2023).
RDM more flexible to adjust the model size and leverage more low-resolution data.
3.3 STOCHASTIC SAMPLER
Since RDM differs from traditional diffusion models in the forward process, we also need adapt the sampling algorithms. In this section, we focus on the EDM sampler (Karras et al., 2022) due to its flexibility to switch between the first and second order (Heun's) samplers.
Heun's method introduces an additional step for the correction of the first-order sampling. The updating direction of a first-order sampling step is controlled by the gradient term d, dd,
(.. The correction step updates current states with an averaged gradient term Cre
di Heun's 's method takes account of the change of gradient term dæ between f, and f,-1. Therefore,
achieves higher quality while allowing for fewer steps of sampling.
We adapt the EDM sampler to the blurring diffusion of RDM's super-resolution stage following the derivation of DDIM (Song al. 2020Aa. We define the indices of sampling steps J. in corresponding to the noisy states of images {x; Jeo- To apply blurring diffusion, images transformed into frequency space by DCT u, V' Song al. (2020a) uses family of inference distributions to describe the diffusion process. We can write it for blurring diffusion as:
qs(u (u1 /wo) = q5( (uu (u))
II n=2
go 16(u-11/u uo).
(9)
where 5 R. denotes the index vector for the distribution. For all > 1. the backward process is:
9o( (n, 1u,
Un uo) = (u
oi, Vo 82 un + (0, (.. - Vo 57, 63.D;, (.)) 5
(10)

---

Published as a conference paper at ICLR 2024
The mean of the normal distribution ensures the forward process to be consistent with the formula- tion of blurring diffusion in Section 3.2, which is (() = (u,|Di, Ug of I). We provide sampler and the formulation of blurring diffusion
detailed proof of the consistency between in Appendix A.3 When the index vector ó is O, the sampler degenerates into an ODE sampler. We set 6, ,n- for our sampler, where 17 [0, 1) is fixed scalar controlling the scale of random- ness injected during sampling. We substitute the definition into equation I0 to obtain our sampler function as:
Un doo
un- = (Di.., + 7n D,, Je, (TnD Dr
(11)
Ttn
-1-1, As in the section 3.1 we also need to consider block noise besides where 7n VI- 112 0th blurring diffusion. The adaptation is just replace isotropic Gaussian noise e with é, which is weighted sum of the block noise and isotropic Gaussian noise. Ug = upg Un, or,) is predicted by the neural network.
Finally, a stochastic sampler for the super-resolution stage of RDM is summaried in Appendix A.4
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Dataset. We CelebA-HQ and ImageNet in our experiments. CelebA-HQ (Karras et al. 2018) is a high-quality subset of CelebA (Liu et al. 2015) which consists of 30,000 images of faces from human celebrities. ImageNet (Deng et al.] 2009) contains 1,281,167 images spanning 1000 classes and is widely-used dataset for generation and other vision tasks. We train RDM on these datasets to generate 256 256 images. See Appendix C.T for further experiments on higher resolutions.
Architecture and Training. RDM adopts UNet (Ronneberger et al. 2013) as the backbone of dif- fusion models for all stages. The detailed architectures largely follow ADM (Dhariwal Nichol, 2021 for fair comparison. We train unconditional models on CelebA-HQ and class-conditional models on ImageNet respectively. Since we follow the EDM implementation, we directly use the released checkpoint from EDM in ImageNet in the 64 x 64 stage. We calculate the training consump- tion by the number of training samples at 256 x 256 resolution, while also including the training cost of the 64 x 64 stage in the total calculation. According Appendix B.I the FLOPs of the 64 x 64 model are less than 1/10 that of the 256 x 256 model. So we add 1/10 of the first stage's ' number of training samples to the 256 256 stage's to be the total training consumption. See Appendix B.] for more information about the architecture and hyperparameters.
Evaluation. We metrics including FID (Heusel et al. (017). sFID (Nash al. 2021). IS (Sali- mans et al. 2016), Precision and Recall [Kynkäännicmi et al. (019)) for comprehensive evaluation of the results. FID measures the difference between the features of model generations and real images, which is extracted by a pretrained Inception network, sFID differs from FID by using inter- mediate features, which better measures the similarity of spatial distribution. IS and Precision both measure the fidelity of the samples, while Recall indicates the diversity. We compute metrics with 50,000 and 30,000 generated samples for ImageNet and CelebA-HQ respectively.
Table 2: Effect of stochasticity in the sampler on ImageNet 256 x 256 (top) and CelebA-HI 256 x 256 (bottom). We explored different values of 1) in Eq. II
Table 1: Benchmarking unconditional image genera- tion on SclebA-HQ 256 x 256.
UnconditSonal CelebA-HO 256
206
Model
FED1
Precisiont
Recalff
Cost
0.10
0.15
0.20
0.25
0.30
0.40
0.50
LSGM
Valder et el 20211
7.22
470Kx 125
FIDJ
5.65
5.44
5,31
5.27
5.48
5.91
6.91
9.17
WiweD
5.94
0.37
234kxx64 x64
LDM4
5.11
0.72
0.49
410kxx48
StylsSu
3,25
256001 x32
0.10
0.15
0.20
0.25
0.30
0.40
0.50
RDM
3,15
0.77
0.53
46kx 1004
FIDL
3.74
3.15
3,23
4.79
4.2 RESULTS
CelebA-HQ We compare RDM with the existing methods on CelebA-HQ 256 x 256 in Table I 512 512 in Table 6 and 1024 x 1024 in Table 7 RDM outperforms the state-of-the-art model
class-balance means making the number of images generated for each class same among 50,000 images.

---

Published as a conference paper at ICLR 2024
Table 3: Benchmarking class-conditional image generation on ImageNet 256 256. The cost of RDM in the table has taken the first-stage model into consideration and made equivalent conver- sions according to Section 4.1] The cost of latent diffusion model's vae not taken into considera- tion. The calculation process of NFE is clarified in sampling steps part of Section 4.3
Class-Conditioaal ImageNet 256
256
Model
FID
sFID]
ISt
Precisiont
Recall
Cost(IeerX BS)
Sampling NFE
BigGAN-deep (Brock et al.
2018
6.95
7.36
171.4
0.87
0.28
x65kx 2048
StyleGAN- XL. (Sauer et al. 2022
2.30
4.02
265.12
0.78
0.53
ADM Dhariwal & Nichol
2021
10,94
6,02
100.98
0.69
0.63
x980kx256
250
LDM [Kombach et al
2022
10.56
103 49
0.71
0.62
178kx
250
CDM Ho al. 2022
4.88
158.71
100
DiT-X Pocbles & Xie
2022
9.62
6.85
121.50
0.67
0.67
7000k 256
250
MDT-XL/2 (Gao et al. 2025
6.23
5.23
143.02
0.71
0.65
6500kx256
250
RDM
5.27
4.39
153.43
0.75
0.62
2996
125
ADM-U,
3.94
6.14
215.84
0.83
0.53
1980k 256
500
LDM-4G (CFG=1.50)
3.60
247.67
0.87
0.48
178k 1200
500
DIT-XL/2-G (CFG=1.50)
2.27
4.60
278.24
0.83
0.57
7000k
500
MDT-XL/2-G (dynamic CFG)
1.79
4.57
283.01
0.81
0.61
6500kx
500
MDT-XL/2-G (CFG=1.325)
2.26
4.28
246.06
0.81
0.59
6500kx255
500
RDM (CFG=3.50)
1.99
3.99
260.45
0.81
0.58
4096
250
+ class-balanc
1.87
3.97
278.75
0.81
0.59
4096
250
StyleSwin (Zhang et al.]| 2022) with a remarkably fewer training iterations (50M versus 820M trained images). We also achieve the best precision and recall among the existing works.
ImageNet Tshoes the performance of class-conditional generative models on ImageNet 256x 256. We report the best results possible of the existing methods with classifier-free guidance (CFG) (Ho & Salimans) 2022). RDM achieves the best sFID and outperforms all the other methods by FID except MDT-XL/2 (Gao al. 2023) with dynamic CFG scale. If with a fixed but best- picked CFG scale MDT-XL/2 can only achieve an FID of 2.26. While achieving competitive results, RDM is trained with only 70% of the iterations of MDT-XL/2 (1.2B versus 1.7B trained images), indicating that the longer training and a more granular CFG strategy are potential directions to further optimize the FID of RDM.
Training Efficiency We also compare the performance of RDM with existing methods along with the training cost in Figure I When CFG is disabled, RDM achieves a better FID than previous state-of-the-ar diffusion models including DiT (Peebles & Xie 2022) and MDT (Gao et al. 2023 RDM outperforms them even with only about 1/3 training iterations.
4.3 ABLATION STUDY
In this section, conduct ablation experiments the designs of RDM to verify their effectiveness. Unless otherwise stated, we report results of RDM on 256 x 256 generation without CFG.
The Effectiveness of block noise. We compare the performance of RDM with and without adding block noise in Figure 4 With a sufficient phase of training, RDM with block noise outperforms the model without block noise by a remarkable margin on both ImageNet and CelcbA-HQ. This demonstrates the effectiveness of the block noise. The addition of block noise introduces higher modeling complexity of the noise pattern, which contributes to slower convergence of training in the initial stage, as illustrated by Figure 4a.. We assume that training on a significantly smaller scale of samples leads to fast convergence of the model, which obliterates such a feature, therefore similar phenomenon cannot be observed in the training of CelebA-HQ.
The scale of stochasticity. As previous works (Song et al.) 2020b) have shown, SDE samplers usually perform better than ODE samplers. We want to quantitatively measure how the scale of the stochaticity affects the performance in the RDM sampler (Algorithm []. Table 2 shows results with 77 varying from 0 to 0.50. For both CelebA-HQ and ImageNet, the optimal FID is achieved by 77 = 0.2. We hypothesize a small is insufficient for the noise addition to cover the bias formed
"The best CFG scale is 1.325 with hyperparameter sweep from 1.0 to 1.8. We observed the FID increases greatly if CFG scale 1.5 for MDT-XI./2.
8

---

Published as a conference paper at ICLR 2024
18.0
W/ Block Noise
w/ Block Noise
w/o Block Noise
w/o Block Noise
FID 3K
16.5
15.0
60M
Number of Training Images
Number of Training Images
(a)
Figure 4: The effectiveness block noise. We compare the performance of RDM along the training on (a) ImageNet 256 256 and (b) CelebA-HQ 256 x 256. To apply block noise in RDM, we set a = 0.15 and kernel size 8 =
earlier sampling steps, while a large 1 introduces excessive noise into the process sampling, which makes moderate 7) to be the best choice. Within a reasonable scale of stochasticity, an SDE sampler always outperforms the ODE sampler by a significant margin.
Sampling steps. To show the efficiency of our model, we compare the performance RDM and other methods with fewer sampling steps. Number of Function Evaluations (NFE), iee, the number that neural network is called dur-
14
MDTXL/2
RDM,50 +
12
RDM,100+(--10
ing sampling, is used as the index of the com-
RDM,150+
parison for fairess. For RDM, the NFE con-
sists of the NFE in the second stage and 1/10
FID-S0K
10
the NFE in the first stage, according to the pro-
portion of the FL.OPs. As shown in Figure 5 the performance of DiT-XI-/2 and MDT-XLL/2 both drop significantly with lower NFE, while RDM barely declines. Considering that the steps in different stages may contribute differ- ently in FID, we demonstrates three FLOPs al- location strategies in Figure With more NEE allocated in the first stage, RDM achieves a bet- ter FID. In all settings, RDM performs better than MDT-XL/2 and DiT- if NFE < 200.
6-
NFE
Figure 5; Comparison of FID ImageNet with different sampling steps. For allocation of NFE N in RDM, 10n + {2 n) means 10n for the first stage and n for the second.
CONCLUSION AND DISCUSSION
In this paper, we propose relay diffusion optimize the cascaded pipeline. The diffusion process can now continue when changing the image resolution model architectures. We anticipate that our method can reduce the cost of training and inference, and help create more advanced text-to-image model in the future.
The frequency analysis the paper reveals the relation between noise and image resolution, which might be helpful to design better noise schedule. However, our numerous attempts to theoretically derive the optimal noise schedule on the dataset from a frequency perspective did not yield good results. The reason might be that the optimal noise schedule is also related the size of the model, inductive bias, and the nuanced distribution characteristics of the data. Further investigation is left for future work.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work is supported by Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grant 2022ZD01 18600, the NSFC for Distinguished Young Scholar 61825602, Tsinghua University Initiative Scientific Research Program 20233080067 and the New Comerstone Science Foundation through the XPLORER PRIZE. The authors also thank Ting Chen from Google DeepMind and Junbo Zhao from Zhejiang University for their valuable talks and com- ments.
AUTHOR CONTRIBUTIONS
Ming Ding proposes the methods and leads the project. Jiayan Teng and Wendi Zheng conduct most of the experiments. Wenyi Hong works together on early experiments. Jianqiao Wangni, Wenyi Hong and Zhuoyi Yang contribute to the writing of the paper. Jie Tang provides guidance and supervision.
The work is partly done during the internship of Jiayan Teng and Wendi Zheng at Zhipu Al.
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. ediffi: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Kristian Bredies, Dirk Lorenz, et al. Mathematical image processing. Springer, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 11096, 2018
Ting Chen. On the importance of noise scheduling for diffusion models. arXiv preprint 1097:2 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 14:8780-87944 2021.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34: 19822-19835. 2021.
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is strong image synthesizer. arXiv preprint arXiv:2303. 1tte:2333..4899 14389, 2023.
Ian Goodfellow, Jean Pbuget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):133 144, 2020.
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Miguel Angel Bautista, and Josh Susskind. f-dm: A multi- stage diffusion model via progressive signal transformation. arXiv preprint arXiv:2210.04955, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans.
Classifier-fre diffusion guidance.
arXiv preprint
arXiv:2207.12599 2022.
10

---

Published as a conference paper at ICLR 2024
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6833 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, (511:22449-2883 2022.
Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. arXiv preprint arXiv2220900557.. 2022.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. arXiv preprini arXiv:2301.11099, 2023.
Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. Polarity sampling: Quality and diversity control of pre-trained generative networks via singular values. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10641-10650, 2022.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Mika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint arXiv:2103.0388 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171, PMLR, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint 12240922 2022.
Hao Phung, Quan Dao, and Anh Tran. Wavelet diffusion models are fast and scalable image gener- ators. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10199-10208, 2023.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pp. 8821-8831 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Severi Rissanen, Markus Heinonen, and Amo Solin. Generative modelling with inverse heat dissi- pation. arXiv preprint arXiv:2206.1339) 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.

---

Published as a conference paper at ICLR 2024
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assiser Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part II 18, pp. 234-241. Springer, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479-36494, 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPHI 2022 conference proceedings, pp. 1-10, 2022.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Yang Song. Jascha Sohl-Dicksteni Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020b.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:1 14:712 11877111022 302, 2021.
Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Bain- ing Guo. Styleswin: Transformer-haat based gan for high-resolution image generation. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 11304-11314, 2022.
Long Zhao, Zizhao Zhang, Ting Chen, Dimitris Metaxas, and Han Zhang. Improved transformer for high-resolution gans. Advances in Neural Information Processing Systems, 34:18367-18380 2021.
12

---

Published as a conference paper at ICLR 2024
A DERIVATION
A. PATCH-WISE BLURRING
The forward process of blurring diffusion defined as Eq. 3 where un =V 2o denotes the representation of the image Zo in the frequency space. The diagonal matrix D, e defines non-isotropic blurring projection, where A(i W + j. +
2 --2 H + W corresponds to the coordinate (i.j) in the 2D frequency space. In the equation q( ut (u)) = N (u De un of I),
we can utilize the dot product of matrices to transform D, and uo into 2D matrices, D, and úg in
the shape of H x W for calculation: D + D. -uo
(12)
In the super-resolution stage of RDM, we apply blurring each & x k patch independently. We name it as patch-wise blurring and define the diagonal blurring matrix in the shape of k: x for each patch as:
De,e = exp(Axx (ix)(i,)) = =
(13)
where i E [0, E [0, For any patch, D remains the same. The blurring matrix D of the patch-wise blurring is combination of all the independent blurring matrices Dr, The relationship between the elements of DP, and Di, can be expressed as:
(i. = Dr, mod k,j mod k),
(14)
where (i, corresponds the coordinate in the 2D frequency space. Finally, Dř in Eq. 7 can be formulated as:
D. diag(unfold D))).
(15)
where unfold (D,) means unfolding the H x W matrix into vector of HW dimensions and diag(v) denotes the diagonal matrix with vector v as its diagonal line.
A.2 COMBINATION OF SCHEDULE
We follow Karras et al. 2022 to set the noise schedule for standard diffusion as In((r) ((Pmean: Pata). We use FD and Fi to denote the cumulative distribution function (CDF) and the inverse distribution function (IDF) for distribution D in the following description. With t sampled
from uniform distribution ((00, 1), the noise scale is formulated as:
(t) (Fsp(FN(Pmma Para (t)).
(t)).
(16)
For the super-resolution stage of RDM, we apply truncated version of diffusion noise schedule d'(t), ((t,,6~((0 (0, 1). we set lg as the starting point of the truncation, the new noise schedule can be formally expressed as:
= ( (uan)tt)ttt
(17)
which means we only sample the noise scale a' from positions of the normal distribution ([ p2.)) where its CDF is less than t..
For the process of blurring, we set its schedule following the setting of Hoogeboom & Salimans (002). They found that the heat dissipation is equivalent to a Gaussian blur with the variance of its kernel o.. 2Tc. They set the blurring scale U,L as:
B = B sin 2 to
2 (TT))
(18)
where is also sampled from the uniform distribution (0, 1) and OB mama denotes a fixed hyperpa- rameter. Empirically, set B,maa 3 for ImageNet 256 x 256 and d, 2 for CelebA-HQ
256 256.
The blurring matrix formulated as D, A where T =e. As illustrated in
Section 2.2 A is diagonal matrix and Axx 2 1
2 (r for coordinate (i.j)-
13

---

Published as a conference paper at ICLR 2024
A.3 SAMPLER DERIVATION
In this section, we prove the consistency between the design of our sampler and the formulation of blurring diffusion. We need to prove that the jointly distribution qo( (o(un-1 Ut) we define in Eq. matchess with the marginal distribution
Q6 (un, (40) -N(un Di uo o., I)
(19)
under the condition of Eq.
proof. Given that q (uN) uo) = N(ux Di uo, Jiy I). proceed with mathematical induction
approach. Assuming that for any S N, (t (2g) = (u,, Dr, Mo, o., I) holds. We only need to prove qo (un,1 (2)) (N(nnnilD
uo of I). and then the conclusion above will be proved
based on the induction hypothesis.
Firstly, based on
qs(un-1leo)) = qu(un-ilemn uo) (u)
(20)
we introduce
(1mn111u,, uo) N =N(un-1
1 _ (V o7.-1 53 + (o,, Di
Vo of,
8% Di, )uo):
8
Ttn
(21)
and
qo(u, (uo) =N(un, Dr, uo, oi I)-
(22)
Then according Bishop & Nasrabadi (2006), 98 16(un-11/20 is also Gaussian distribution:
qs (u un) N( (/n En-1)
(23)
Therefore, from Eq.
20 we can derive that
Mn-1
of 82 D., us + (or, Dr.
Vo 8, 52.D )uo)
D.P up (24)
otn
and
o hn-1
82
Σ
o2 I. 0
I
52. I
(25)
0²
Summing up, qs( (unn-1/u6) =N(un Dr uo 2 P
I). The inductive proof is complete.
14

---

Published as a conference paper at ICLR 2024
A.4 STOCHASTIC SAMPLER
Algorithm the RDM second-order stochastic sampler
sample z'y from results of the first stage
i.c. images at the resolution of 64 x 64 upsample 64px images to blurry
Nx interpolate (N'N, 256, mode nearest") + ot
256px images and apply the truncated schedule in Appendix A.2
UN VTEN
transformed into the frequency domain
for n E {N
1 do -n-1
nn. = V1 17
5n = 105t-1
b coefficient of the random term
Jen
tig uol uo(unn Tt..)
D model prediction tx b first-order gradient term t,
d, un-th
Ten
Up-1 1 (Dir + Val1 -DD Mun + +or, (In DD )d, + Ō,,
> from t, to t- using Euler's 's method
if n + then
the second-order part model prediction at 6
Ug = Ug 11 0tn-1)
d, p1
D gradient term at -
d' du +d
second-order gradient term
u (Dr + tI un, + ot, (rr D Dr dd, E correction
end if
Un = 4/a-1
end for
Zp Vuo
As for the sampler of the first stage, we follow the EDM sampler (Karras et al., 2022). Of course, samplers such as DDPM are also capable. After all, the first stage is just a standard diffusion model.
B MODEL DETAILS
B. HYPERPARAMETERS
Hyperparameters we use for the training of RDM are presented in Table 4 We set the architecture hyperparameters for diffusion models following Dhariwal & Nichol (202I), in corresponding to the input resolutions. For the experiments CelebA-HQ, set the model dropout be larger (0.15 and 0.2 for two stages respectively), and enable sample augmentation to prevent RDM from overfitting.
B.2 TRAINING COST
On ImageNet, the first stage model was trained on 32 V100 for 13 days according to EDM (Karaas al. 2022) and the second stage model (64 -> 256) was trained on 64 10G-A100 for 12.5 days. On CelebA-HQ we trained the first stage model on 32 40G-A100 100 for 16 hours and the second stage model (64 + 256) on 32 40G-A100 for 25.5 hours.
C ADDITIONAL EXPERIMENTS
C.1 FURTHER EXPERIMENTS ON HIGHER RESOLUTIONS
To further show that RDM can easily scale high-resolution image generation without carefully adjusting the hyperparameters of noise schedule and architecture. We take dataset CelebA-HO as an example and conduct experiments higher resolutions: 512 and 1024. Except for the different hyperparameters shown in Table other settings remain the same as the 64 4256 model in the second stage.
15

---

Published as a conference paper at ICLR 2024
Table 4: Hyperparameters for RDM.
ImageNet 64
ImageNet 64 4456
CelebA-HQ 64
CelehA-HQ 64 >256
Diffusion steps
256
100
120
53
Model size
295M
553M
295M
553M
GFLOPs
104
1117
104
1117
Mixed-precision (FP16)
V
V
V
Channels
192
256
192
256
Channels multiple
1,2,3,4
1,1,2,2,4,4
1,2,3,4
1,1,2,2,4,4
Heads Channels
64
64
64
64
Attention resolution
32,16,8
32,16,8
32,16,8
32,16,8
Dropout
0.1
0.1
0.15
0.2
Augment probability
0
0
0.2
0.2
Blurring mmae
3.0
2.0
Batch size
4096
4096
1024
1024
Training Images
2500M
1000M
70M
40M
Learning Rate
le-4
le-4
le-4
le-4
Table 5: Hyperparameters for RDM on higher resolutions of 512 and 1024.
CelebA-HQ 256-+512
CelebA-HQ 102-
Diffusion steps
35
35
Model size
558M
562M
GFL.Pss
1987
4509
Mixed-precisioe (FP16)
V
V
Channels
256
256
Channels multiple
0,2,1,1,
0.5.0.5,111,.2,444
Heads Channels
64
64
Attention resolution
32,16,8
32,16,8
Dropout
0.2
0.1
Augment probability
0.2
0.2
Blurring Maz
1.25
2.0
Batch size
256
256
Training Images
15M
11M
Learning Rate
le-4
le-4
The cost in Table 6 and Table 7 contains the training of three stages, with the same equivalent conversion Table and Table 3 according to GFLOPs. As shown the table, RDM achieves state-of-the-art FID at the resolution of both 512 and 1024, and only requires a small amount of training the third stage according to Table This demonstrates that RDM can be easily extended from two stages to three stages and higher resolutions. Examples are shown in Figure
Table 6: Benchmarking unconditional image generation CelebA-HQ 512 512.
Unconditional CelebA-HQ
512
Model
FID1
Cost(Number of Training Images)
WafeDiff (Phung et al. RDM
2023)
6.40
12M
3.47
41M
C.2 ADD BLOCK NOISE TO END-TO-END DIFFUSION MODEL
To further demonstrate the effectiveness of block noise, we conduct ablation experiments on an end- to-end model followed by the setting of ADM (Dhariwal & Nichol 2021). We mixture of
16

---

Published as a conference paper at ICLR 2024
Table 7: Benchmarking unconditional image generation on CelebA-HQ 1024 1024.
Unconditional CelebA-HQ 1024 x 1024
Model
FIDJ
Cost(Number of Training Images)
HiT-B (Zhao et al.
2021 Polarity-ProUne ProGAN Humayun et al. 2022
8.83
16M
7.28
StyleGAN (Karras et al 2019
5.06
StyleSwin (Zhagg at
4.43
819M
RDM
3.85
23M
fixed ratio block noise and Gaussian noise as illustrated in section 3.2 with block noise outperforms the model without block noise by remarkable margin CelebA-HQ 256 x 256.
As shown in Figure ADM
23
FID-5K
19
w/ Block Noise
w/o Block Noise
15
15K
25K
35K
45K
Iterations
Figure 6: The effectiveness of block noise. We compare the performance of ADM along the training on CelebA-HQ 256 256. For block noise, we set a 0.15 and kernel size
C.3 CORRECT ARTIFACTS IN THE LOW-RESOL.UTIO IMAGES
As illustrated in section 3.2 the super-resolution generation of RDM is a Markovian process, in comparison of CDM and ADM-U using low-resolution conditioning all along the generation pro- cess. This could improve the robustness of RDM on handling artifacts from low-resolution stages. the comparison of super-resolution generation between RDM and ADM-U, We use 64 x 64 samples from ImageNet low-resolution inputs, adding 0.05 scale of Gaussian Noise to introduce artifacts. While RDM successfully handles the noise to generate clean 256 x 256 samples, ADM-U preserves the noise to the 256 x 256 samples.
D DETAILS ABOUT THE POWER SPECTRAL DENSITY
D.1 CALCULATION PROCEDURE OF THE PSD
We follow the setting of Rissanen et al. (202)) to calculate the PSD the frequency space. The PSD certain frequency is defined the square of the DCT coefficient that frequency. Firstly,
17

---

Published as a conference paper at ICLR 2024
Gaussian Noise
to generation
RDM
84px
256px
Figure 7: Input 64px images with artifacts into the 64-+256 model of RDM (right top) and ADM-U (right bottom) generate 256px images.
we transform the image into the 2D frequency space by DCT and set the frequency range to [0, x]. To obtain the ID curve of the PSD, we calculate the distance from each point (x, y) the origin
in the frequency space, i.e. V2² + v², considering as a ID frequency value. Subsequently, we uniformly divide the frequency domain into N intervals, and take the midpoint of each interval as its representative frequency value. Finally, we take the mean the PSD) values for all points within the interval the PSD value for that interval, in order to get N coordinate pairs for plotting. The SNR curve in Figure 2 can be obtained in a similar approach, while the only difference that the vertical axis values replaced with the absolute value of the ratio between the DCT coefficients for the image and noise in the frequency space.
D.2 ANALYSIS OF THE PSD
As shown in Figure g the PSD of real images gradually decreases from low frequency to high fre- quency. And the intensity of Gaussian noise components across all frequency bands is generally equal. Therefore, when corrupting real images, Gaussian noise initially drowns out high-frequency components until the noise intensity becomes high enough drown the low-frequency compo- nents of real images. And it is demonstrated in Figure 2 that, as the resolution of images increases, less information corrupted under the same noise intensity. Correspondingly. as shown Fig- ure ()) and FigureBib) b), the low-frequency portion of the PSD gets drowned out more slowly as the resolution increases. It is indicated that we will introduce excessive high- geghfrryuency components of noise when corrupting the low- low-frequency information of real images, especially for high-resolution images.
Differently, the low-frequency portion of the PSD from block noise is notably higher than that of Gaussian noise with the same intensity. Furthermore, the PSD of block noise exhibits a decreasing trend as frequency increases, and its curve is quite similar the PSD curve of Gaussian noise the resolution 64 upsampled to the resolution of 256. This leads to the PSD curves of high- images with added block noise and that of low-resolution images with added Gaussian noise also
18

---

Published as a conference paper at ICLR 2024
being quite similar. As result, the low- low-frequency portion of the PSD from images with added block noise gets drowned out more quickly than that from images with added Gaussian noise. We conclude that block noise corrupt the wwwfrequency components of images more easily.
Piane Spectra Denviily
Phaee Spentrul
Iow Iaggarrrry pofec
30
1811
EPT
127
Trequency
Precuencyy
frequency
Figure 8: The power spectral density (PSD) of real images after adding (a) 64px Gaussian noise, (b) 256px Gaussian noise and (c) 256px block noise with block size of 4. The black curve represents the PSD of real images. The red curves, from dark to light, represent adding noise with increasing intensity. order to make comparisons within the same frequency space, for the images at the resolution of 64, firstly upsample them to the pixel space at the resolution of 256.
E ADDITIONAL SAMPLES
Section 4.3 quantitatively compares performance RDM with other models under the same NFE and demonstrates the superiority of RDM with fewer sampling steps. Figure shows quali- tative comparison results. While other models achieve competitive quality of generation with suf- ficient NFE, their performances degenerate noticeably with the decrease of NFE. In contrast, RDM still maintains comparable generation quality with low NFE.
Figure 10 compares visualized samples generated by the best settings of StyleGAN-XL (Sauer al., 2022]. DiT (Peebles & Xie 2022) and RDM. StyleGAN-XL is in the framework of GAN, while DiT and RDM are diffusion models. RDM achieves the best quality of images synthesis. Figure exhibits more examples generated by our model RDM on ImageNet 256 256.
19

---

Published as a conference paper at ICLR 2024
NFE = 20
NFE 40
NFE = 80
NFE = 160
Figure 9: Comparison of ImageNet samples with varied NFE, DiT-XL/2 (left) vs MDT-XL/2 (mid- dle) vs RDM (right). The allocation of NFE between the two stages of RDM is: [2, 18), [8, 32), [20, 60), [40, 120].
20

---

Published as a conference paper at ICLR 2024
class 207
class 388
class 511
class 812
Figure 10: Comparison of best ImageNet samples. StyleGAN-XL (FID 2.30, left) vs DIT-XL/2
(FID 2.27, middle) vs RDM (FID 1.87, right).
21

---

Published as a conference paper at ICLR 2024
Figure 11: Additional ImageNet samples generated by RDM. Classes 279: Aretic fox, 90:
lorikeet, 301: ladybug, 973: coral reef, 980: volcano, 497: church, 717: pickup truck, 927: trifle.
22

---

Published as a conference paper at ICLR 2024
Figure 12: Samples generated by RDM on CelebA-HQ5122 512 (left) and CelebA-HQ 1024 1024 (right).
23


Reviews:
[{"review_id": "VufGvbyQeo", "reviewer": "Reviewer_pR8K", "summary": "This paper presents the Relay Diffusion Model (RDM), a new cascaded framework to improve the shortcomings of the previous cascaded methods. Contributions are (1) The difference of noise addition process between low-resolution and high-resolution image diffusion is analyzed from the perspective of frequency domain. Based on the analyzation, this paper further introduces the block noise to bridge the gap; (2)  By combing a low-resolution ordinary diffusion model and a high-resolution blurring diffusion model, RDM starts diffusion from the low-resolution result instead of pure noise, reducing the training and sampling steps; and (3) This paper also evaluates the effectiveness of RDM on unconditional CelebA-HQ 256×256 and conditional ImageNet 256×256 datasets.", "questions": "1. On page 2, “Training Efficiency”, the author takes the cascaded method as a solution to mitigate memory and computation costs, but the cascaded method still needs to train and inference in the highest resolution.  How does such a setting mitigate memory and computation cost? For RDM, the problem still exists.\n2. I can’t understand why, for Fig.2, the high-frequency period is meaningless, and the difference can be neglected. More explanation may be required.\n3. How to choose $D_{T}^{p}$ to guarantee $VD_{T}^{P}V^{T}x_{0}$ in the same distribution as $x^{H}$; this is the key to connect low-resolution diffusion and high-resolution diffusion.\n4. Adding the training iterations of each method in Tables 1, and 3 may make it more readable. The comparison may be a little unfair because RDM’s low-resolution diffusion model is pre-trained.\n5. For the text description below equation (11), why blurring corruption and block noise corruption can be considered independently, i.e., why can we just replace $\\epsilon$ with $\\tilde{\\epsilon}$? Is there any formal proof?\n6. For Algorithm 1 in A.4, it seems that the sampling algorithm does not contain the process of the low-frequency diffusion model. Please clarify.\n7. Some minor errors:\n    * On page 4, two “the” in “the same noise level on a higher resolution results in a higher SNR in the (low-frequency part of) the frequency domain.”\n    *  For equation (6), the denominator should be $s$ or $s^2$?\n    *  Equation 8 may be not appropriate, the $\\epsilon^{'}$ should be below the expectation symbol too.\n\n\n-----------\nThank the authors for their responses. Most of my concerns have been addressed. I'd like to raise my score.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. For originality, this paper first combines a low-resolution ordinary diffusion model with a high-resolution blurring diffusion model. To some extent, this solves the problem of noise schedule in high-resolution diffusion. Besides, the analysis of the noise gap between low-resolution and high-resolution is interesting.\n2. For clarity, this paper’s writing is well-structured, and the problem was presented and solved straightforwardly.", "weaknesses": "1. The transition from section 3.1 to section 3.2 is a little incomprehensible; why combining block noise and blurring diffusion model in the second stage needs further explanation. Besides, from the equation and algorithm, I can’t find an explicit connection between low-resolution diffusion and high-resolution diffusion; such an explicit statement may be necessary for reader to understand.\n2. Missing experiments: How does an end-to-end model for high-resolution images by introducing block noise in early diffusion steps perform? The author mentions it on page 5 but does not explain it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HS589BwenW", "reviewer": "Reviewer_htaY", "summary": "This work designs a diffusion framework for medium-resolution image generation and explores it on unconditional CelebA-HQ $256^2$ and class-conditional ImageNet $256^2$. The framework generally follows EDM and cascaded diffusion model and develops some novel contributions on top of them. First, it develops a novel diffusion/denoising process formulation, where the noise, added to an image, is locally correlated for the higher resolution images. Then, instead of concatenating the upsampled low-resolution images channel-wise, the diffusion process is unified for low and higher resolution images. As such, it does not need to train the low-resolution generator from scratch and uses the pre-trained EDM checkpoints. The framework achieves SOTA FID results on ImageNet $256^2$ and CelebA-HQ $256^2$. The paper also contains interesting analysis of the frequencies of noised/clean images in low/medium resolutions and also some curious insights about the cascaded diffusion models.", "questions": "- Would the method work on $512^2$, $1024^2$ and/or higher resolutions?\n- How does the method perform with from-scratch training? If the model cannot train well from scratch, then the claim about its simplicity compared to CDMs does not hold and should be removed. Also, it's not fair to claim the reduced amount of training steps for a non-from-scratch trained model.\n- Can it be the case that the reduced amount of sampling steps is due to the 2-nd order sampler? For example, MDT uses DDPM sampler, which was shown by EDM to require more sampling steps.\n- For a fair comparison, please report the training/inference costs of the developed model vs the baselines (where possible).\n- Please, update the claims about the convergence speed (especially the teaser paper) to specify that the model is not trained from scratch.\n- Please, update the claims about SotA FID based on the unbalanced FID instead of the class-balanced FID.\n- Why not try dynamic CFG as well to further improve FIDs? Or it was tried and didn't work?\n- It would be good to describe the differences with f-DM.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The work sets a new state-of-the-art results on ImageNet $256^2$ for constant CFG.\n- Analysing SNRs across frequencies is interesting and might inspire some subsequent works to explore this direction as well.\n- The work has not only proposed the idea of block-wise locally correlated noise, but also derived the sampler for it (while the derivations are not particularly involved, they still require some carefulness).\n- The exposition is very clear and the paper is easy to follow.", "weaknesses": "- $256^2$ resolution is not \"high-resolution\" (as claimed in the paper), but rather \"medium-resolution\". Even the $512^2$ resolution can be fit with end-to-end architectures — e.g. simple diffusion [1] or VDM++ [2] (they can even fit 512x512 in the end-to-end fashion). In this way, it's no clear whether the method would easily scale for high-resolution generation without re-tuning the hyperparameters. \n- Scaling Laws clearly demonstrated that the amount of compute spent on training the model has great influence on the final performance — just like the developed novel techniques or model size. However, there is no information about the training cost in the paper (only in terms of the amount of iterations). The paper mentioned that the $256^2$-resolution generator is ~10x more expensive that the $64^2$-resolution one, so I would guess that it might be way more expensive in terms of compute compared to the baselines.\n- Figure 1 (and the accompanying claims about faster convergence in the text) seems misleading: the model is not trained from scratch, but compares the convergence with from-scratch trained models. The vanilla EDM was trained on \n- class-balanced FID is the known trick to improve the performance, and, since it's unclear whether prior works used it as well, shouldn't be used in the claims about SotA FID (having it in the table as a separate row is not an issue).\n- f-DM (Gu et ICLR'22) can also be seen as a \"relay\" diffusion, but it is not even mentioned in the paper.\n\n[1] simple diffusion: End-to-end diffusion for high resolution images\n[2] Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation\n[3] f-DM: A Multi-stage Diffusion Model via Progressive Signal Transformation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R5HQhulmF9", "reviewer": "Reviewer_Jazo", "summary": "This paper focuses on the task of using a diffusion model for high-resolution image generation. Specifically, the authors locate the issue in the SNR in the frequency domain and introduce the block noise to bridge the gap and build an RDM upon a cascaded pipeline to get rid of the reliance on the low-resolution condition. The authors show state-of-the-art FID on CelebA-HQ and ImageNet on 256x256.", "questions": "This paper is well-written and the proposal shows some impressive results in experiments. My major concerns are about the experiment on really high-resolution images and some experiments to verify the superiority of the RDM further. Please see more details in the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. this paper has excellent presentation quality and it is easy for readers to follow.\n    \n2. this paper provides a comprehensive analysis of the limitation of existing cascaded model for high-resolution image generation and locate the issue in the SNR of higher-resolution image. To address these issues, the authors introduce RDM to solve this issue. I think this highlight is insightful and could inspire others in the community.\n    \n3. the authors have shown extensive comparison in the paper to verify the effectiveness of the RDM.", "weaknesses": "1. This paper focuses on high-resolution image generation. However, the biggest image resolution used in this paper is only 256x256, which is much smaller than the existing definition of ‘high-resolution‘. I would expect an experiment result that has a resolution at least 512 or 1024 to see whether RDM still works.  \n    \n2. The authors claim that any artifacts in the low-res images can be corrected in the high-res stage, we expect some qualitative cases in experiments to verify such a claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h4iTQ8tci7", "reviewer": "Reviewer_edjv", "summary": "This paper proposes a Relay Diffusion Model (RDM) with the aid of block noise to overcome the drawback of the existing cascaded models. The proposed method is claimed to reduce the training and sampling steps. The experiments show that RDM outperforms other methods on FID and sFID on two datasets CelebA-HQ and ImageNet with resolutions of 256x256.", "questions": "1) What is the specific of each group in Table 3? It shows that the first column (Model) presents 5 groups (separated by \\midrule) but unclear why is it divided like that.\n\n2) I wondering if the number of data used in this paper is wrong, for example, Table 4 stated that 2500M training images were used for ImageNet64, is that 2.5 billion images? To my knowledge, ImageNet contains more than one million images, am I wrong? The same for all other datasets., does CelebA-HQ have 70 million images for training? Also, in Figure 1, is the horizontal axis the real number of images for training or it is just the iterations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The finding that the same noise level can give a mismatch between low- and high-resolution images is interesting.\n+ The design of the method can facilitate the training speed where it avoids the conditioning on low-resolution images.\n+ Some improvements are sound", "weaknesses": "+ The paper mentioned that \"most current models follow the linear (Ho et al., 2020) or cosine (Nichol & Dhariwal, 2021) schedule\" and \"an ideal noise schedule should be resolution-dependent (See Figure 2)\" but looking into Figure 2, I cannot see which noise schedule is used there, either cosine or linear, making it very confusing to capture what the authors want to say. Again, the next sentence said \"train high-resolution models directly with common schedules designed for resolutions of 32×32 or 64×64 pixels\" but Figure 2 only shows the 64px and 256px (what about 32x32) --> What is the relationship between 32x32 or 64x64 pixels and the one shown in Figure 2 (64px/256px)?\n\n+ The advantage of the proposed method is emphasized with training efficiency, however, it only considers the setting without CFG where some competitors do not yield the optimal output. I believe that it is better also more meaningful to compare all methods in the setting with CFG where the existing methods achieve the best performance (optimal setting). Furthermore, when claiming the efficiency, I would also expect the comparison of the inference time/steps of the proposed method with the existing ones in their best optimal regime (including sampling steps in Table 3.).\n\n+ While the proposed method achieves slightly better sFID compared to MDT-XL/2-G on class-conditional generation ImageNet 256x256 (Table 3) (3.97 vs. 4.57, not really \"a large margin\" as stated in the abstract), however, most other metrics lag behind MDT-XL/2-G (such as FID, IS, and Recall). This indicates that to show its advantages over the existing approaches, it may need to present more evidence.\n\n+ Section 4.2 talking about CeleA-HQ, mentioned fewer training iterations while it is given with 50M and 820M trained images, making it a bit confusing that the dataset contained a hundred million images or that is just the total training steps. I recommend revising it more clearly and putting a column on the side of that table about the total training iterations or total training images.\n\n``[Post rebuttal update]: After the author's response, somehow it resolved some of my concerns, I increased my rating score from 5 to 6.``", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)