f-DM: A Multi-stage Diffusion Model via Progressive Signal Transformation
Authors: Jiatao Gu
Shuangfei Zhai
Yizhe Zhang
Miguel Ángel Bautista
Joshua M. Susskind
Venue: ICLR 2023 poster (2023)
Keywords: diffusion models
progressive signal transformation

Abstract: Diffusion models (DMs) have recently emerged as SoTA tools for generative modeling in various domains. Standard DMs can be viewed as an instantiation of hierarchical variational autoencoders (VAEs) where the latent variables are inferred from input-centered Gaussian distributions with fixed scales and variances. Unlike VAEs, this formulation constrains DMs from changing the latent spaces and
learning abstract representations. In this work, we propose f-DM, a generalized family of DMs which allows progressive signal transformation. More precisely, we extend DMs to incorporate a set of (hand-designed or learned) transformations, where the transformed input is the mean of each diffusion step. We propose a generalized formulation and derive the corresponding de-noising objective with a modified sampling algorithm. As a demonstration, we apply f-DM in image generation tasks with a range of functions, including down-sampling, blurring, and learned transformations based on the encoder of pretrained VAEs. In addition, we identify the importance of adjusting the noise levels whenever the signal is sub-sampled and propose a simple rescaling recipe. f-DM can produce high-quality samples on standard image generation benchmarks like FFHQ, AFHQ, LSUN, and ImageNet with better efficiency and semantic interpretation.

Published as a conference paper at ICLR 2023
fDM: A MULTI-STAGE DIFFUSION MODEL VIA PRO- GRESSIVE SIGNAL TRANSFORMATION
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Miguel Angel Bautista, Josh Susskind Apple
{igu32, szhai, yizhe.zhang, mbaut itbautistaaariee abbaat/sssaaarrti...aassssaeappp jsusskindeesppee.ccoom
ABSTRACT
Diffusion models (DMs) have recently emerged as SoTA tools for generative mod- eling various domains. Standard DMs can be viewed as an instantiation of hi- erarchical variational autoencoders (VAEs) where the latent variables are inferred from input-centered Gaussian distributions with fixed scales and variances. Un- like VAEs, this formulation constrains DMs from changing the latent spaces and learning abstract representations.
this work, we propose f-DM, a generalized family of DMs, which allows progressive signal transformation. More precisely, we extend DMs to incorporate a set of (hand-designed or learned) transformations, where the transformed input is the mean of each diffusion step. We propose gen- eralized formulation of DMs and derive the corresponding de-noising objective to- gether with a modified sampling algorithm. As demonstration, we apply f-DM in image generation tasks with range of functions, including down-sampling blurring, and learned transformations based on the encoder of pretrained VAEs. In addition, we identify the importance of adjusting the noise levels whenever the signal sub-sampled and propose a simple rescaling recipe. f-DM can pro- duce high-quality samples standard image generation benchmarks like FFHQ, AFHQ, LSUN and ImageNet with better efficiency and semantic interpretation. Please check videos at http: //tttt/aaaauuuuuaooaaa
d.mm/
Dosmaeppe (Lsun-church 28x223)
Gaunsian AARO 256x358)
VS- (CLass- GLLass-condLiomes Znagomet 2501236)
t=1
Stage Lecent P0ace ESPPusion
t=0
Figure 1: Visualization reverse diffusion from fDDMs with various signal transformations. æ; is the denoised output, and z, is the input to the next diffusion step. We plot the first three channels of VQVAE latent variables. Low-resolution images are resized to 256 for ease of visualization.
INTRODUCTION
Diffusion probabilistic models (DMs, Dicks-Dicsttin et al. 2015 Ho et al. 2020 Nichol & Dhari- wal 2021) and score-based (Song et al. 2021b) generative models have become increasingly popular

---

Published as a conference paper at ICLR 2023
the tools for nigh-quality image (Dhariwal & Nichol 2021). video (Ho al., 2022b). text-to- speech (Popov et al., 2021) and text-to-image (Rombach et al.) 2021; Ramesh et al. 2022 Saharia et al. (2022a) synthesis. Despite the empirical success, conventional DMs are restricted to operate in the ambient space throughout the Gaussian noising process. On the other hand, common generative models like VAEs (Kingma & Welling 2013) and GANs (Goodfellow al., 2014 Karras et al. 2021) employ a coarse-to-fine process that hierarchically generates high-resolution outputs.
We are interested in combining the best of the worlds: the expressivity of DMs and the benefit of hierarchical features. To this end, we propose f-DM, a generalized multi-stage framework of DMs to incorporate progressive transformations to the inputs. As an important property of our formulation, f-DM does not make any assumptions about the type of transformations. This makes it compatible with many possible designs, ranging from domain-specific ones to generic neural networks. In this work, we consider representative types of transformations, including down-sampling blurring, and neural-based transformations. What these functions share in common is that they allow one to derive increasingly more global, coarse, and/or compact representations, which we believe can lead to better sampling quality as well as reduced computation.
Incorporating arbitrary transformations into DMs also brings immediate modeling challenges. For instance, certain transformations destroy the information drastically. and some might also change the dimensionality. For the former, derive anterpolation-busee formulation smoothly bridge consecutive transformations. For the latter, we verify the importance of rescaling the noise level, and propose a resolution-agnonne signal-to-noise ratio (SNR) practical guideline for noise rescaling.
Extensive experiments are performed on image generation benchmarks, including FFHQ, AFHQ, LSUN Bed/Church and ImageNet. f-DMs consistently match or outperform the baseline perfor- mance, while requiring relatively less computing thanks to the progressive transformations. Further- more, given pre-trained f-DM, we can readily manipulate the learned latent space, and perform conditional generation tasks (c.g., super-resolution) without additional training.
2
BACKGROUND
Diffusion Models (DMs, Sohl-Dicksteii et al.) 2015 [Sng & Ermon 2019 Ho al. (020)) are deep generative models which can be viewed as special case hierarchical VAEs (Kingma et al. 2021). In this paper, we consider diffu- sion in continuous time similar Song et 2b2 Kingma et al. (2021).
.V10.
Nio.r.
N10.7
Given datapoint æ RN, a DM models time-dependent latent variables z {2, E [0, 1]. 20 = x} based on a fixed signal-noise schedule {at, 0:):
a(2)|za)= -N(z;;0) o2e0)
where atjs ai/a,, of of ag also defines the marginal distribution q( (2)(z)) as:
(a) DMs
(b) VAES
(c) f- (Ours)
((2,22) (2z;; 0, of
Figure 2: (a) the standard DMs; (b) bottom-up
hierarchical VAEs; (c) our proposed f-DM.
By default, we assume the variance preserving form (Ho et al. 2020 That is, of of 1, aF 1, and the signal-to-noise-ratio (SNR, ///o decreases monotonically with f. For generation, a parametric function is optimized to reverse the diffusion process by denoising 2, a,æ + d,E to the clean input I, with weighted reconstruction loss Lg. For example, the "simple loss" proposed in Ho et al. 2020) is equivalent to weighting residuals by wA /o?
Lo E (x)m)=((()))_ w || t) 21
In practice, 0 is parameterized as a U-Net (Ronneberger et al. 2015). As suggested in Ho et al. 2020), predicting the noise eg empirically achieves better performance than predicting Des where
2

---

Published as a conference paper at ICLR 2023
zo (z) 02 /a) Sampling from such learned model can be performed from ancestral sampling (DDPM, Ho et al. 2020). or a deterministic DDIM sampler (Song et al. (0211))
Starting from 2 ~ (O I], sequence of timesteps 1
... iN 0 are sampled for iterative generation, and we can readily summarize both methods for each step as follows:
0s No (z() + Vo 11802 (Z())) + njo 6~ c~, <1
(2)
where ē = 01 o, and 7 controls the proportion of additional noise. (i.c., DDIM rj 0).
As the score function Eş is defined in the ambient space, it is clear that all the latent variables are forced to be the same shape as the input data æ (IR-). This not only leads to inefficient training, especially for steps with high noise level (Jing al. 2022), but also makes DMs hard to learn abstract and semantically meaningful latent space as pointed out by Preechakul et al. (2022).
3 METHOD
In this section, we introduce f-DM, an extended family of DMs to enable diffusion on transformed signals, in way similar to a standard hierarchical VAE. We start by introducing the definition of the proposed multi-stage formulation with general signal transformations, followed by modified training and generation algorithms (Section 3.1p. Then, specifically apply f-DM with three categories of transformations (Section 3.2)
3.1 MULTI-STAGE DIFFUSION
Signal Transformations We consider a sequence of deterministic funetions f (fo..... SK...fK) fk), where fo progressively transforms the input signal æ E R.N into x = (a) Me. We assume x" = fo(x) = æ. In principle, f can be any function. In this work, we focus on transformations that gradually destroy the information contained in (e.g.. down-sampling). leading towards more compact representations. Without loss of generality, we assume Mo 2 M1 2
MK- A sequence of inverse mappings g (9u..... 9000....-99--- -K-1) is used to connect corresponding sequence of pairs of consecutive spaces. Specifically, we define Φg as:
San (fa+1( s s x
k k=
th
The approximation of Equation 3 (k < K) is not necessarily (and sometimes impossibly) accurate. For instance, f& downsamples an input image x from 128 into 642 with average pooling, and gk can be bilinear interpolation that upsamples back to 1282, which is a lossy reconstruction.
The definition of f and g can be seen as a direct analogy of the encoder (e) and decoder (0) in hierarchical VAEs (see Figure 2 (b)). However, there are still major differences: (1) the VAE en- coder/decoder is stochastic, and the encoder's 's outputs are regularized by the prior. In contrast, f and g are deterministic, and the encoder output x does not necessarily follow simple prior; (2) VAES directly use the decoder for generation, while f. g are fused in the diffusion steps of f-DM.
Forward Diffusion We extend the continuous- time DMs for signal transformations. We split the diffusion time 0 -> into K+1 stages, where for each stage, a partial diffusion process is performed. More specifically, we define a set of time boundaries 0 = To < Ti <... < <TK < TK+1 = 1, and for torte E [0, 1), the latent 2, has the following marginal probability:
(<<e<=)= = (1 where x; (t Tk )&t + (Th-1 D)at
Th St Th- (4)
Tk+1 Tk
As listed above, æ, the interpolation of t and approximation ** when t falls in stage k. A simple illustration for the relationship of æt, at xx and is shown in Figure 10 We argue that interpolation is crucial as it creates a continuous transformation that slowly corrupts information inside each stage. In this way, such change can be easily reversed by our model. Also, it is non- trivial to find the optimal stage schedule тy for each model as it highly depends on how much the information is destroyed in each stage fx- In this work, tested two heuristics: (1) linear schedule Tk = k/(K + 1): (2) cosine schedule тą = cos(1 k/ (K + 1)). Note that the standard DMs can be seen special case of our f-DM when there is only one stage (K = 0).
3

---

Published as a conference paper at ICLR 2023
Bescaling
SS0
Beu Entopail
(b) Varisnce Preserved Suocaling
Sigral Preserved Resoaling
Scandard Forvard Diffusion
Figure 3: Left: an illustration of the proposed SNR computation for different sampling rates; Right: the comparison of rescaling the noise level for progressive down-sampling. Without noise rescaling, the diffused images in low-resolution quickly become too noisy distinguish the underline signal.
Equation does not guarantee a Markovian transition. Nevertheless, our formulation only need q( 2, }, which has the following simple form focusing diffusion steps within a stage:
[( x) (z = + at (x, . Tk S s < Tk+1
(5)
From Equation we further re-write T; x, = -5, - (t-s)/(tt Tk), where 6, = x² I, is the signal degradation. Equation 5 also indicates that the reverse diffusion distribution q( / (Zz x: ( (x)) can be written as the function of a and 6, which will be our learning objectives.
z, ar) xx
Boundary Condition To enable diffusion across stages, we need the transition at stage boundaries Tk- More specifically, when the step approaches the boundary -- (the left limit of 7), the transi- tion g(z, ž,,-, x should be as deterministic (ideally invertible) & smooth as possible to minimize information loss First, we can easily expand 2, and 2,- as the signal and noise combination:
Before: z,- = 0ty- Zr-- + to,-.c, (.p((c)= -o, I), After: 2+ = 077 Br + (((), p(((e)) = ().
Based on definition, x,- = -k-1 = g(x²) = g(x,). which means the signal part is invertible. Therefore we only need to find Ç. Under the initial assumption of Mr < MM this can be achieved easily by dropping elements from e. Take down-sampling (M-- = 4.M2) as an example. We can directly drop 3 out of every 2 x 2 values from More details are included in Appendix A.4
The second requirement of a smooth transition is not as straightforward as looks, which asks remain unchanged across the boundary. We argue that the conven-
the "noisiness" of latents tional measure - the signal-to-noise-ratio (SNR)-in in DM literature is not compatible with resolution change it averages the signal/noise power element-wise. In this work, propose a generalized resolution-agnoste SNR by viewing data as points sampled from a continuous field:
SNR(2) = En En.-I|(Ei-ASSNNAA(()))
En- [OII
where / the data range, SIGNAL represents the real data value (such image pixels), and NOISE is the unstructed Gaussian noise added to the data. Q is patch relative to I, which can be any size as long as invariant to different sampling rates (resolutions). As shown in Figure (left), we can obtain reliable measure of noisiness by averaging the signal/noise inside patches. We derive o,, o, from Q--,0,- for any transformations by forcing SNR(2,) definition. Specifically, if dimensionality change is solely caused by the change of sampling rate (e.g., down-sampling, average RGB channels, deconvolution), we can get the following relation:
SNR(2,-) under this new
2²/0² dx 7 a2./0?-
where d2 M is the total dimension change, and 7 E E| | is the change of signal power. For example, we have d2 = 4, 7k Rs for down- nown-sampling. Following Equation the straightforward rule is rescale the magnitude of the noise, and keep the signal part unchanged:
For simplicity, we omit the subscript for Tk in the following paragraphs.
4

---

Published as a conference paper at ICLR 2023
Algorithm 1: Reverse diffusion for image generation using f-DM
Input: model 0, f.g. stage schedule (To...., ). rescaled noise schedule functions ax)).
step-size åt, emall ~ N( I), DDPM ratio
Initialize 2 from éfull
for (k K; 0; k= k- 1) do
for (t = Th+1:t Txit=t At Ar) do
Eo, 6g (z,t); we = (z (1) E0)/a(();;
if > then
z as)) (we + 8g (1 s) - Tx)) + (o²(s)) 1/82022.000 eu too e,
if k then
Re-sample noise 6:s from eo and ftulli == a(T)) (x0))) + o(T))) Exs
return
/ o dd,, which we refer as signal preserved (SP) rescaling. Note that, to ensure the noise schedule is continuous over time and close to the original schedule, such rescaling is applied to the noises of the entire stage, and will be accumulated when multiple transformations are used. As the comparison shown in Figure 3 the resulting images are visually closer the standard DM. However, the variance of 2; becomes very small, especially when - 1, which might be hard for the neural networks distinguish. Therefore, we propose the variance preserved (VP) alternative to further normalize the rescaled a, a so that a² + 0² = 1. We show the visualization in Figure (b).
a f a. aaaataa.... t
Training We train a neural network to denoise. We also show the training pipeline Fig- ure TO In f-DM, noise is caused by two factors: (1) the perturbation from noise injection; (2) the degradation due to signal transformation. Thus, we propose to predict 2o and da jointly, which simultaneously remove both noises from 2, with "double reconstruction" loss:
Lo = E -=.1) [W+ (1 (zt,1) t) xx 112 + ]6 f) 5,11
(9)
where the denoised output is No (z,,1) 60(2,,)) 1). Unlike standard DMs, the denoising goals are the transformed signals of each stage rather than the final real images, which are generally simpler targets to recover. The same as standard DMs, we also choose to predict Eg. and compute Xp = (z, oree)/ar. We adopt the same U-Net architecture for all stages, where input will be directed to the corresponding inner layer based on spatial resolutions (see Appendix Figure I1 for details).
Unconditional Generation We present the generation steps in Algorithm ! where I, and 8, are replaced by model's 's predictions òg. Thanks to the interpolation formulation (Fquation 4. gen- eration independent of the transformations f. Only the inverse mappings g - which might be simple and easy to compute is needed to map the signals at boundaries. This brings flexibility addition, Al-
and efficiency to learning complex or even test-time inaccessible transformations. mnoise-resampling step" for each stage boundary, which is the reverse process for ((e) in Equation ( While Ç is deterministic, the reverse process needs additional randomness. For instance, if drops elements in the forward process, then the reverse step should inject standard Gaussian noise back to the dropped locations. Because we assume Mo 2 ... 2 Mr. . we propose to sample full-size noise 6full before generation, and gradually adding subsets of efull to each stage. Thus, Etull encodes multi-scale information similar to RealNVP (Dinh et al. 2016)).
gorithm [ includes
Conditional Generation Given an unconditional f-DM, we can do conditional generation by replacing the denoised output Zy with any condition x, suitable time (T), and starting diffusion from T. For example, suppose ff is amwnsample, and x, is a low-resolution image, f-DM enables super-resolution (SR) without additional training. To achieve that, is critical initialize zT, which implicitly asks Z7 & OrT + dTeg(27) (2T) In practice, we choose T to be the corresponding stage boundary, and initialize by adding random noise Tre to @T A gradient- -basedd-baaddddde method used to iteratively update 2T + zT AV xr (2(2T) xell for few steps before the diffusion starts.
3.2 APPLICATIONS ON VARIOUS TRANSFORMATIONS
With the definition in Section 3.1 next show (-DM applied with different transformations. In this paper, we consider the following three categories of transformations.
Downsampling. As the motivating example in Section 3.1 we let f a sequence of nownsampe operations that transforms a given image (e.g., 256²) progressively down 162, where each ss(())
5

---

Published as a conference paper at ICLR 2023
reduces the length by 2, and correspondingly (x(.) upsamples by 2. Thus, the generation starts from a low-resolution noise and progressively performs super-resolution We denote the model as f-DM-DS, where d2 4, 7 = 1 in Equation 8 and K 4 for 256² images.
Blurring. JDDM also supports general blur transformations. Unlike recent works (Rissanen
et al. 2022 Hoogeboom & Salimans| 2022) that focuses on continuous-tin
blur (heat dissipation), Equation can be scen an instantiation of progressive blur function if we treat ã as a blurred ver- sion of æ", This design brings more flexibility in choosing any kind of blurring functions, and using the blurred versions as stages. In this paper, we experiment with two types of blurring functions. (1) f-DM-Blur-U: utilizing the same downsample operators as f f-DM-DS, while always up-sampling the images back to the original sizes; (2) f-DM-Blur-G: applying standard Gaussian blurring ker- nels following Rissanen et al. (2022). In both cases, we use gu (2) æ. As the dimension is not changed, no rescaling and noise resampling required.
Image + Latent Trans. We further consider diffusion with learned non-linear transformations such as VAEs (see F f: VAE encoder, g: VAE decoder). By inverting such an encoding process, we are able to generate data from low-dimensional latent space similar RRmbach et al. (LDM,, 2021). As a major difference, LDM operates only on the latent variables, while J-DM learns diffusion in the latent and image spaces jointly Because of this, our performance will not be bounded by the quality of the VAE decoder. In this paper. we consider VQVAE (Van Den Oord et al.) 2017 together with its GAN variant (VQGAN, Esser et al. 202T). For both cases, we transform 256 x 3 images into 32² X 4 (i.e., d₂ 48) latent space. The VQVAE encoder/decoder is trained on ImageNet (Deng et al. 2009), and is frozen for the rest of the experiments. For J-DM-VQGAN, we directly take the checkpoint provided by RRombach et al. (2021). Besides, we need to tune 7 separately for each encoder due to the change in signal magnitude.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Datasets. We evaluate f-DMs on five commonly used benchmarks testing generation on range of domains: FFHQ (Karras et al. 2019), AFHQ (Choi et al. 2020). LSUN Church & Bed (Yu al.) 2015). and ImageNet (Deng al. 2009). All images are center-cropped and resized to 256 x 256.
Training Details. We implement the three types of transformations with the same architecture and hyper-parameters except for the stage-specific adapters. We adopt lighter version of ADM (Dhari- wal & Nichol 202I) as the main U-Net architecture. For all experiments, we adopt the same training scheme using AdamW (Kingma& & Ba 2014) optimizer with learning rate of 5 and EMA de- cay factor of 0.9999. We set the weight W/ = sigmoid(- (ognoidd(lo(((23 (og((?//?)) following P2-weighting (Choi et al. 2022). The cosine noise schedule 0₃ cos(0.5.t) is adopted for diffusion working in the 256 x 3 image space. As proposed in Equation g noise rescaling (VP by default) is applied for f-DMs when the resolutions change. All our models trained with batch-size 32 images for 500K (FFHQ, AFHQ, LSUN Church), 1.2M (LSUN Bed) and 2.5M (ImageNet) iterations, respectively.
Baselines & Evaluation. We compare f-DMs against a standard DM (DDPM, Ho al. 2020) on all five datasets. To ensure fair comparison, we train DDPM following the same settings and continuous-timg formulation as our approaches. We also include transformation-speeiff baselines: (1) we re-implement the cascaded DM (Cascaded, HHo et al. 2022a) to adapt f f-DM-DS setup from 162 progressively to 2562, where for each stage a separate DM is trained conditioned on the con- secutive downsampled image; (2) we re-train latent-diffusion model (LDM, RRmbach et al.) (021)) on the extracted latents from our pretrained VQVAE; (3) to compare with J-DM-Blur-Gi we in- clude the scores and synthesised examples of IHDM (Rissanen al. (022) We set 250 timesteps (At = 0.004) for f-DMs and the baselines with 7 = (Algorithm Tp. We use Frechet Inception Distance (FID, Heusel et al., (0177) and Precision/Recall (PR, Kynkäänniemi al. 2019) as the measures of visual quality, based on 50K samples and the entire training set.
4.2 RESULTS
Qualitative Comparison To demonstrate the capability of handling various complex datasets, Figure 4 (1) presents an uncurated set of images generated by f-DM-DS. We show more samples
from all types of f-DMs in the Appendix E.4 We also show a comparison between f-DMs and the
6

---

Published as a conference paper at ICLR 2023
FFHQ 256x256
AFHQ 256x256
LSUN-Church 256x256
LSUN- BoN-Bedroon 256x256
InageNet 256x256
VQ-VAE/VO GAN
(Our implamentation)
IIDH (Rissanen at 2122)
LOW (VQVAE Decoder]
LOW (VQBAN Decoder)
Ourss (f-DN-05)
Ours [T-DN- 8lur
ours (T-0M- VQWAE)
Ours (T- 9P- VQUAA))
Figure 4: t Random samples from f-DM-DS trained various datasets; J. Comparison of f-DMs and the corresponding baselines under various transformations. Best viewed when zoomed in. All faces presented synthesized by the models, and real identities.
Table 1: Quantitative comparisons various datasets. The speed compared to DDPM is calculated with bsz= 1 on CPU. Best performing DMs are shown bold.
Models
FID
P
Rt
FID.
Pt
R
Speed
Models
FID.
FFHQ
256 x 256
AFHQ
256 x 256
ASUN-Chureh 256 x 256
DDPM
10.8
0.76
0.53
9.3
0.74
0.51
x1.0
DDPM
9.7
DDPM (1/2)
16.8
0.74
0.45
15.2
0.64
0.44
x2.0
f-DM-DS
8.2
--MA-VQVAE
8.0
Cascaded
49.0
0.10
0.09
24.2
0.37
0.13
J-DM-DS
10.8
0.74
0.50
6.4
0.81
0.48
x2.1
LSUN-Bed 256 x 256
IHDM
64.9
43.4
DDPM
8.0
f-DM-Blur-G
11.7
0.73
0.51
6.9
0.76
0.49
x1.0
F-DM-DS
6.9
J-DM-Blur-U
10.4
0.74
0.52
0.77
0.53
x1.0
f-DM-VQVAE
7.1
LDM
48.0
0.31
0.07
29.7
0.07
0.11
x9.8
ImageNet 256 256
LDM (GAN)*
8.6
0.72
0.60
6.5
0.63
0.61
x9.2
DDPM
10.9
J-DM-VQVAE
12.7
0.77
0.47
8.9
0.76
0.40
x1.7
f-DM-DS
8.2
JDM-VQGAN
11.7
0.74
0.51
5,6
0.76
0.53
x1.7
/-DM-VQVAE
6.8
baselines with various transformations FFHQ (Figure L). Our methods consistently produce better visual results with more coherence and without noticeable artifacts.
Quantitative Comparison. We measure the generation quality (FID and precision/recall) and rel- ative inference speed of f-DMs and the baselines in Table [ Across all five datasets, f-DMs sistently achieves similar or even better results for the DDPM basclines, while gaining near x2 inference speed for f-DM-(DS, VQVAE, VQGAN) due the nature of transformations. As comparison, having fewer timesteps (DDPM 1/2) greatly hurts the generation quality of DDPM. We also show comparisons with transformation-seae specific baselines FFHQ & AFHQ
v.s. Cascaded DMs. Although cascaded DMs have been shown effective in literature Nichol & Ho al. 2022a), it is underexplored apply cascades in sequence of consecu-
Dhariwal 2021

---

Published as a conference paper at ICLR 2023
Input Image S2x22
spssopled Teage 2504256
T-BR-IE -R
Taruet Image 2005290
Treat Latents 32x32e4
vquse
T-2N VENIE
Terget lrag 2501266
(b)
Interslation of the Initial noise
(c)
nean image
Fixing noise
Figure 5: Random DDIM samples (1 0) from (a) f-DMs AFHQ and LSUN-Churde by given (downsampled, blurred, latent) images as conditions;
by interpolating the initial noise of the latent stage; fc)f-DM-DS starting from the same initial noise of the 16 16 stage. For (c), we also show the "mean image" of 300 random samples using the same initial noise.
tive resolutions (16 + 32 + 64 - ...) like ours. such cases, the prediction errors get easily accumulated during the generation, yielding serious artifacts in the final resolution. To ease this, Cascaded DM (Ho et al. 2022a) proposed to apply "noise conditioning augmentation" which re- duced the domain gap between stages by adding random noise the input condition. However, it is not straightforward to tune the noise level for both training and inference time, By contrast, f-DM by-design non-cascaded, and there are domain gaps between stages. That is, can train our model end-to-end without worrying the additional tuming parameters and achieve stable results.
v.s. LDMs. We show comparisons with LDMs (Rombach et al. 2021 in Table LDMs generate more efficiently as the diffusion only happens in the latent space. However, the generation is heavily biased by the behavior of the fixed decoder. For instance, it is challenging for VQVAE decoders to synthesize sharp images, which causes low scores in Table [ However, LDM with VQGAN decoders is able to generate sharp details, which are typically favored by Inception V3 (Szegedy al. 2016) used in FID and PR. Therefore, despite having artifacts (see Figure 4 below, right- most) in the output, LDMs (GAN) still obtain good scores. In contrast, f-DM, as pure DM, naturally bridges the latent and image spaces, where the generation is not restricted by the decoder.
v.S. Blurring DMs. Table I compares with a recently proposed blurring- based method (IHDM, Rissanen et al.) (2022). Different from approach, IHDM formulates fully deterministic forward process. We conjecture the lack randomness is the cause of their poor generation quality. In- stead, f-DM proposes natural way of incorporating blurring with stochastic noise, yielding better quantitative and qualitative results.
Conditional Generation. In Figure a), we demonstrate the example of using pre-trained f-DMs to perform conditional generation based on learned transformations. We downsample and blur the sampled real images, and start the reverse diffusion following Section 3.1 with f-DM-DS and Blur-U, respectively. Despite the difference in fine details, both our models faithfully generate high-fidelity outputs close the real images. The same algorithm is applied the extracted latent representations. Compared with the original VQVAE output, f-DM-VQVA is able to obtain bet- ter reconstruction. We provide additional conditional generation samples with the ablation of the "gradient-based" initialization method in Appendix E.3
Latent Space Manipulation
To demonstrate f-DMs have learned certain abstract representations by modeling with signal transformation, show results of latent manipulation in Figure 5 Here we assume DDIM sampling (n 0), and the only stochasticity comes from the initially sampled noise ffull In (b), obtain semantically smooth transition between cat faces when linearly
8

---

Published as a conference paper at ICLR 2023
interpolating the low-resolution noises; the other hand, we show samples of the same identity with different fine details (c.g., expression, poses) in (c), which is achieved easily by sampling f f-DM-DS with the low-resolution (16²) noise fixed. This implies that f-DM is able to allocate high-level and fine-grained information in different stages via learning with downsampling.
4.3
ABI ABLATION STUDIES
Table 2: Ablation of design choices for f-DMs trained on FFHQ All faces are not real identities.
Model
Eq. 4
Rescale
Stages
FID
No
VF
cosine
26.5
0.70
0.25
f-DM- DS
Yes
No
cosine
14.5
0.73
0.43
Yes
SP
cosine
12.1
0.75
0.47
Yes
VP
linear
13.5
0.73
0.46
(a) withost Interpniation (Ft.1)
Yes
VP
cosine
10.8
0.74
0.50
f-DM-
Yes
No
1inear
24.0
0.79
0.29
Yes
VP
cosine
13.8
0.78
0.45
VQVAE
Yes
linear
12.7
0.77
0.47
Intergolation
Table 2 presents the ablation of the key design choices. As expected, the interpolation formulation (Equation 4 effectively bridges the information gap between stages, without which the prediction emrors get accumulated, resulting in blurry outputs and bad scores. Table also demonstrates the importance of applying correct scaling. For both models, rescaling improves the FID and recall by large margins, where SP works slightly worse than VP. In addition, we also empirically explore the difference of stage schedules. Compared VAE-based models, we usually have more stages in DS/Blur- based models to generate high-resolution images. The cosine schedule helps diffusion move faster in regions with low information density (e.g., low-resolution, heavily blurred).
RELATED WORK
Progressive Generation with DMs. Conventional DMs generate images in the same resolutions. Therefore, existing work generally adopt cascaded approaches (Nichol & Dhariwal [222] Ho et al.) 2022a Saharia al. 2022a) that chains series of conditional DMs to generate coarse-to- to-fine, and have been used in super-resolution (SR3, /Saharia et al., (022b). However, cascaded models tend to suffer error propagation problems. More recently, Ryu Ye (2022) dropped the need of conditioning, and proposed to generate images in pyramidal fashion with additional reconstruction guidance; Jing et al. (2022) explored learning subspace DMs and connecting the full space with Langevin dynamics. By contrast, the proposed f-DM is distinct from all the above types, which only requires one diffusion process, and the images get naturally up- sampled through reverse diffusion.
Blurring DMs. Several concurrent research (Rissanen al. 2022
Daras et al. 2022 Lee et al.
2022 have recently looked into DM alternatives to combine blurring into diffusion process, some of which also showed the possibility of deterministic generation (Bansal al. 2022). Although sharing similarities, our work starts from a different view based on signal transformation. Furthermore, our empirical results also show that stochasticity plays a critical role in high-quality generation.
Latent Space DMs. Existing work also investigated combining DMs with standard latent variable models. To our best knowledge, most of these works adopt DMs for learning the prior of latent space, where sampling is followed by a pre-trained (Rombach et al.) (2021) or jointly optimized (Vahdat al. 2021) decoder. Conversely, f f-DM does not rely on the quality decoder.
CONCLUSION
We proposed f-DM, a generalized family of diffusion models that enables generation with signal transformations. As demonstration, apply f-DM image generation tasks with range of transformations, including downsampling, blurring and VAEs, where fDMMs outperform the base- lines in terms of synthesis quality and semantic interpretation.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Our work focuses on technical development, i.e., synthesizing high-quality images with a range of signal transformations (e.g., downsampling, blurring). Our approach has various applications, such as movie post-production, gaming, helping artists reduce workload, and generating synthetic data as training data for other computer vision tasks. Our approach can be used to synthesize buman-related images (e.g., faces), and it is not biased towards any specific gender, race, region, or social class.
However, the ability of generative models, including our approach, to generate high-quality images that are indistinguishable from real images, raises concerns about the misuse of these methods, e-g., generating fake images. To resolve these concerns, we need to mark all the generated results as "synthetic". In addition, we believe it is crucial to have authenticity assessment, such as fake image detection and identity verification, which will alleviate the potential for misuse. We hope our approach can be used to foster the development of technologies for authenticity assessment. Finally, we believe that creating a set of appropriate regulations and laws would significantly reduce the risks of misuse while bolstering positive effects on technology development.
REPRODUCIBILITY STATEMENT
We assure that all the results shown in the paper and supplemental materials can be reproduced. We believe we have provided enough implementation details in the paper and supplemental materials for the readers to reproduce the results.
REFERENCES
Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Gold- blum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.09392 2022.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11472-1148)) 2022.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pp. 11188889,7 2020.
Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G. Dimakis, and Peyman Milanfar. Soft diffusion: Score matching for general corruptions, 2022. URL httts::/arxiv..r
abs/2209. 05442
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 0009/CVPR.2200.520664..
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.0888 2016.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEFE/CVE conference on computer vision and pattern recogni- tion, pp. 12873-12883 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahra- mani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances
10

---

Published as a conference paper at ICLR 2023
in Neural Information Processing Systems, volume 27, pp. 2672 2680. Curran Asso- ciates, Inc., 2014. URL /tttttsss///rroccdanss..eee /ttpppppe//11//1ie///// httpp//www.....ppppp/////////el.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. wrXiv:2207.12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. Mach. Learn. Res., 23: 2022a.
Jonathan Ho, Tim Salimans, Alexey A Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In ICLR Workshop on Deep Generative Models for Highly Struc- fured Data, 2022b.
Emiel Hoogeboom and Tim Salimans. Blurring diffusion models, 2022. URL bttps:::////xx org/ ab///b///22 2aa.///229.05555 05557
Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion gener- ative models. arXiv preprint urXiv:2205.01490 2022.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. arXiv preprint arXiv:2106. 12423, 2021.
Diederík Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances in neural information processing systems, 34:21696-21707 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.61 2013.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Sangyun Lee, Hyungjin Chung, Jaehyeon Kim, and Jong Chul Ye. Progressive deblurring of diffu- sion models for coarse-to- fine image synthesis. arXiv preprint arXiv:2207.11192. 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffiusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Vadim Popoy, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad- tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pp. 8599-8608. PMLR, 2021.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Dif- fusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10619-10629, 2022.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 06125, 2022.
11

---

Published as a conference paper at ICLR 2023
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Severi Rissanen, Markus Heinonen, and Amo Solin. Generative modelling with inverse heat dissi- pation. arXiv preprint arXiv:2206.133: 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Dohoon Ryu and Jong Chul Ye. Pyramidal denoising diffusion probabilistic models. arXiv preprint MrXiv:2208.01866 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. / 1487, 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022b.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv::202. 005.000522222 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- fional Conference on Learning Representations, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. Interna- tional Conference on Learning Representations, 2021b.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 2818-2826, 2016.
Arash Vahdat and Jan Kautz. Nvae: deep hierarchical variational autoencoder. Advances Neural Information Processing Systems, 33: 19679 2020.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Neural Information Processing Systems (NeurIPS), 2021.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Fisher Yu, Ari Seff. Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of lcrge-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 506 03365, 2015.
12

---

Published as a conference paper at ICLR 2023
APPENDIX
A DETAILED DERIVATION OF f-DMs
A. ((ze(zs: x)
We derive the definition in Equation 5 with the change-of-variable trick given the fact that 3t. and x² are all deterministic functions of x.
More precisely, suppose 2ạ N Thus, it is equivalent to have Ut ~ N(a,," , u, ~ NNaax o).1,, u = 2e - a2 x*) ) u. 2, a2, )²). From the above definition, reasonable assume Uz U, follow the standard DM transitionm which means that:
Naaa,,, of where To S
Tk+1-
u = Orls 2 + Ozjsc, 6 <~ ((,,
2, ar(xx - x²) = adls (ze as - x²)) + opse E~ NO0,1)
2t = arls + a, - x.) + oz c~N(0,,))
As typically 22 j x, and both æs, v are the functions of æ. Then ž is dependent on both 2, and xx (x), resulting in a non mon-Morkovian transition:
q(z R. x) N (z a4 22 ai (x) xx) o
Note that, this equation stands only when 22, a. and my in the same space, and did make specific assumptions to the form of 24.-
A.2 qZz,/ze, x)
The reverse diffusion distribution follows the Bayes' Theorem: qzz,|z,, x) x ((zz()))((z)l x),, where both q(z.)e) (e) and g(=z) æ) are Gaussian distributions with general forms of (z.lu, o2)) and [Zz (zz]Aza A2. +b, o respectively. Based on Bishop Nasrabadi (2006) (2.116), we can derive:
( z,, 2) (0-2 (z. b) a 2 ,0
where σ² = (2- 2 0' A 1 Therefore, we can get the exact form by plugging our variables 0, 0s A 0/ T. b = (x: x,,, of otls into above equation, we get:
(,,,|,,, x) N(z + v Voz o²er, 0²1);
where = (2-- - //2 and ē = 0..r///o
Alternatively, if we assume o; take the interpolation formulation in Equation 4 we can also re- write 2, with x, + i-s o,, where we define a new variable 6, = x² xt. As stated in the main context (Section 3.1 such change makes q(z [=,,)) avoid computing x, which may be potentially costly. In this way, we re-write the above equation as follows:
T-
(22./21, x) = (2x),, (x. + 6. (1-) + Vo? Fe, 0'1,
(10)
A.3 DIFFUSION INSIDE STAGES
In the inference time, we generate data by iteratively sampling from the conditional distribution p(z,(z,) = E. (g( 2,, Zz, .)) based on Equation 10 In practice, the expectation over x is approx- imated by our model's prediction. As shown in Equation 9 in this work, we propose a "double- prediction" network 0 that reads z,, and simultancously predicts æ; and ô, with Øg and 8g. respec- tively. The predicted Gaussian noise is denoted as Eg = (z, - agzp)/or Note that the prediction æs and EΦ are interchangable, which means that Therefore, by replacing Z;, d,, Er, with Dg. dg, Eg in Equation [0, we obtain the sampling algorithm shown in Algorithm 1 Line
readily derive one from the other's 's prediction.
13

---

Published as a conference paper at ICLR 2023
SP Ilinar)) VP (inear)
SP (cosinel (cosina)
rescale
0.8
Sigma
Alpha
0.6
0.4
02
0.2
(linear
(cosine) VP (cosine)
VP (linear)
00
0.0
NO rescale
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.6
0.8
Illustration of noise schedule (a, and σ₂) for f-DM-DS models with 5 stages (16? 256²). We use the standard cosine noise schedule a₂ = cos(0..5t)) We also show the difference between the linear/cosine stage schedule, well the proposed SP/VP re-scaling methods.
Figure 7:
A.4 NOISE AT BOUNDARIES
In this paper, the overall principle is to handle the transition across stage boundary is to ensure the forward diffusion to be deterministic and smooth, therefore almost no information is lost during the stage change. Such requirement is important as it directly correlated to the denoising performance. Failing recover the lost information will directly affect the diversity of the model generates.
Forward diffusion As described in Sec- tion 3.1. since we have the control of the signal and the noise separately, we can directly apply the deterministic transformation on the signal, and dropping noise elements.
Eo
E1
E2
E3
X
E2
E3
Drop
Mean, x2
Alternatively, we also implemented different ((E) based on averaging. As shown in F if the transformation is down-sampling, we can use the fact that the mean of Gaussian noises is still Gaussian with lower variance: (Eo + E1 E2 + es)/4 ~ NO !)). Therefore, x2 rescaling is needed on the resulted noise.
Figure 6: Two naive ways for down- www--samling.
Reverse diffusion Similarly, we also define the reverse process if ( is chosen to be averaging. Different from "dropping" where the reverse process is simply adding independent Gaussian noises, the reverse of "averaging" requests to sample Σi e; = 2e given the input noise e, while having p(G)) (0, I), 0, 2, Such problem has closed solution and can be implemented in autoregressive fashion:
a 2c;
6o a/4 /4 + v 3/4 é1, en, él N(O,1);
a/3 + 2/3 é2, a E1, é2 NO0 ();
E a/2 /2 + 1 é3,
E21 és (O,
Similar to the case of "dropping", we also need 3 additional samples é1:3 to contribute to four noises, therefore it can be implemented the same way as described Section 3.1 Empirically, reversing the "averaging" steps tends to produce samples with better FID scores. However, since it introduces correlations into the added noise, which may cause undesired biases especially in DDIM sampling.
Intuition behind Re-scaling Here we present a simple justification of applying noise rescaling. Suppose the signal dimensionality changes from Ma-1 to Mj when crossing the stage boundary, and such change is caused by different sampling rates. Based the proposed resolution -agnustti SNR (Equation 7 the number of sampled points inside Ω is proportional to its dimensionality. Generally, is safe to assume signals are mostly low- www-freue.cy. Therefore, averaging signals will not change its variance. By contrast, shown above, averaging Gaussian noises results in lower variance, where
14

---

Published as a conference paper at ICLR 2023
Roise
Molae
(4) Standard 80IM
(b) D0IN generation with F-DR-05
Figure 8: We show the comparison of the DDIM sampling-
in our case, the variance proportional M- Therefore, suppose the signal magnitude does change, we can get the re-scaling low by forcing SNR(=+) SNR(2,-) at the stage boundary:
M = o M
which derives the signal preserving (SP) rescaling in Equation In Figure 7 of the change of and with and without applying the re-scaling techqnique for f-DM-DS models.
we show an example
A.5 DDIM SAMPLING
The above derivations only describe standard ancestral sampling (7) = 1) where [2, x) is determined by Bayes' Theorem. Optionally, one arbitrarily define any proper reverse diffusion distribution as long as the marginal distributions match the definition. For example, f-DM can also perform deterministic DDIM (Song al. 2021a) setting 7 in Algorithm Similar Song et al. 2021a), we can also obtain the proof based on the induction argument.
Figure 8 shows the comparison of DDIM sampling between the standard DMs and the proposed f-DM. In DDIM sampling (1) 0), the only randomness comes from the initial noise at f = 1. Due the proposed noise resampling technique, f-DM enables multi-scale noising process where the sampled noises are splitted and sent to different steps of the diffusion process. In this case, compared standard DMs, we gain the ability of controlling image generation at different levels, resulting in smooth semantic interpretation.
B DETAILED INFORMATION OF TRANSFORMATIONS
We show the difference of all the transformations used this paper in Figure
B. DOWNSAMPLING
In early development of this work, we explored various combinations of performing down-sampling: f= (bilinear, nearest, Gaussian blur + subsample). g = (bilinear, bicubic, nearest, neural-based) While all these combinations produced similar results, empirically on FFHQ found that both choosing bilinear interpolation for both f. achieves most stable results. Therefore, all the main experiments of f-DM-DS conducted bilinear interpolation. As discussed in Section 3.2 choose = 4, which progressively downsample a 256 into 162.
B.2 BLURRING
We experimented two types of blurring functions. For upsampling-based blurring, we use the same number of stages as the downsampling case; for Gaussian-based blurring, we adopt K with corresponding kernel sizes 0g 15 sin s (3 Tk), where тı follows the cosine stage schedule. In practice, we implement blurring function in frequency domain following Rissanen et al. (2022) based discrete cosine transform (DCT).
15

---

Published as a conference paper at ICLR 2023
Downsample
VQ VAE
Updown Blur
v@ VQ-GAN
Gaussian Blur
Figure 9: We show examples of the five transformations (downsample, blur, VAEs) used in this paper. For downsampling, we resize the image with nearest upsampler; for VQ-VAE/VQ-GAN, we visualize the first 3 channels of the latent feature maps.
B.3 VAES
In this paper, we only consider vector quantized (VQ) models with single layer latent space, while our methods can be readily applied to hierarchical (Razavi et al. 2019) and KL-regularized VAE models (Vahdat & Kautz, 2020). Following RRobaach et al. (2021), take the feature vectors before the quantization layers as the latent space, and keep the quantization step in the decoder (g) when training diffusion models.
We follow an open-sourced implementation to train our VQVAE model on ImageNet. The model consists of two strided convolution blocks which default downsamples the input image by factor of 8. We use the default pperm-parameters and train the model for 50 epochs with a batch-size of 128. For a fair comparison match the latent size of VQVAE,
the pre-trained autoencoding model (Rombach et al. (2021) with the setting of (f=8, VQ (Z=256, d=4)]. We directly use the checkpoint provided by the authors. Note that the above setting is not the best performing model (LDM-4) in the original paper. Therefore, it generates more artifacts when reconstructing images from the latents.
Before training, we compute the signal magnitude ratio "Y (Equation 8) over the entire training set of FFHQ, where we empirically set ?/k 2.77 for VQ-GAN and 7 2.0 for VQ-VAE, respectively.
C DATASET DETAILS
FFHQ (https://github Ccap///aabs/fffg-daataaaeeee contains 70k images of real human faces in resolution of 1024". For most of experiments, resize the images to 256².
AFHQ httt::///ggtuuub..oo///////a///s
contains 15k images of animal faces including cat, dog and wild three categories in resolution of 5122. We train conditional diffusion models by merging training images with the label information. All images are resized to 256
animal- -arad---aae-------------------e-
LSUN (https:/ /tt::://www.... www..yf... wtw.:yff...o/....o ho/p//sunl is collection of large-scale image dataset contain- ing 10 scenes and 20 object categories. Following previous works Rombach et al. (2021), we choose the two categories Church (126k images) and Bed (3M images), and train separate unconditional models on them. As LSUN-Boc is relatively larger, we set the iterations longer than other datasets. All images are resized 256² with center-crop.
n256.
16

---

Published as a conference paper at ICLR 2023
x
Ansat!
fual3
AROEOD EHED-AL
transfore the signal
Unterpobaticn
double prediction
derolved
Figure 10: An illustration of the training pipeline.
the standard ImageNet- IK dataset which contains 1. 1.28M images across 1000 classes. We directly merge all the training images with class-labels. All images are resized 256 with center-crop. For both f-DM and the baseline models, we adopt the classifier-frs frreefee--reeeeeee guidance (Ho & Salimans) (202) with the unconditional probability 0.2. In the inference time, we use the guidance scale (s 2) for computating FIDs, and = 3 to synthesize examples for comparison.
ImageNet (https: http://mmage-eet.... org down Load...pp php)
D IMPLEMENTATION DETAILS
D.1 ARCHITECTURE CONFIGURATIONS
We implement f-DM strictly following standard U-Net architecture in Nichol & Dhariwal (2021). As shown in Figure II input 2, will be directed to the corresponding inner layer based on spa- tial resolutions, and a stage-specific adapter is adopted to transform the channel dimension. Such architecture also allows memory-effiie various resolutions, and split the computation based on the resolutions.
batching across stages where we can create batch with
D.2 HYPER-PARAMETERS
In our experiments, we adopt the following two sets of parameters based on the complexity of the dataset: base (FFHQ, AFHQ, LSUN-CCurch/Bed) and big (ImageNet). For base, we use residual block per resolution, with the basic dimension 128. For big, we use residual blocks with the basic dimension 192. Given one dataset, all the models with various transformations including the baseline DMs share the same paperpparameters except for the adapters. We list the hyperparameter details in T
Hyper-param.
FFHQ
AFHQ
LSUN-Church
LSUN-Bed
ImageNet
image res.
256
2562
256 None
256 None
256
# of classes guidance
None
1000
No
Yes
#channels broskblocks channel multi.
128
128
128
128
192
[1, 1, 2, 4,4]
attention res.
16,8 8
batch size
32
32
32
32
64
Ir iterations
2e-5
500K
500K
500K
1200
2500
Table 3: Hyperparameters and settings for fDM on different datasets.
17

---

Published as a conference paper at ICLR 2023
128
.28
69
64
Bottleneck
64x64
Adapter
Adapter
Figure 1I: illustration of the modified U-Net architecture. Time conditioning is omitted. The parameters partially shared across stages based the resolutions. Stage-specifie adapters adopted to transform the input dimensions.
Cascaded OM (16->32->64 11----6---1------ -228-2266 condition on previous nodel's 's output.) )
T-DM-DS >128 -256, conditioning)
X1
25
Cascaded ON
Ours (F-DNDSS
Figure 12: Additional comparisons with Cascaded DM AFHQ. Comparison of the reverse diffusion process from 162 to 256 We visualize the denoised outputs (æ;) and the corresponding next noised input (z,) near the start end of each resolution diffusion. Comparison of random samples generated by Cascaded DM and -DDM-DS.
ADDITIONAL. RESULTS
E. QUANTITATIVE COMPARISON WITH DDIM
We also include comparison of f f-DMwith the standard DM using DDIM sampling (7 = 0) in Table Similar to the conclusion drawn from Table I the proposed f-DM can achieve comparable or even better performance than baseline DM even with 0 (generation only controlled by the initial noise, see Figure 8), while having better scores for DDIM with half generation steps.
18

---

Published as a conference paper at ICLR 2023
LON (VQVAE Decoder)
(F-DN-VQVAE)
(VQBAN Decoder)
Ours (T-DR-VQ6AM)
Figure 13: Additional comparisons with LDMs AFHQ.
Models
FID.
Pt
R1
FID.
Pt
R
Speed
FFHQ
256 x
256
AFHQ
256
256
DDIM
11.4
0.71
0.53
12.1
0.58
0.65
x1.0
DDIM (1/2)
13.0
0.70
0.51
16.8
0.48
0.64
x2.0
f-DM-DS (n 0)
12.6
0.76
0.55
5,8
0.76
0.55
x2.1
Table 4: Comparison on FFHQ and AFHQ for DDIM sampling (17 = 0)
E.2 V.S. TRANSFORMATIONNSSEEIII BASELINES
We include more comparisons Figure 12 and 13 From Figure 12 we compare the generation process of f-DM and the cascaded DM. It is clear that f-DM conducts coarse-to-fine generation in a more natural way, and the results will not suffer from eтror propagation. As shown in Figure [3] LDM outputs easily affected by the chosen decoder. VQVAE decoder tends output blurry im- ages; the output from VQGAN decoder has much finer details while remaining noticable artifacts eyes,
eyes, furs). By contrast, f-DM perform stably for both latent spaces.
19

---

Published as a conference paper at ICLR 2023
E.3 CONDITIONAL GENERATION
We include additional results of conditional generation, i.e., super-resolution (Figure 14) and de- blurring (Figure 13). We also show the comparison with without the proposed gradient-based initialization, which greatly improves the faithfulness of conditional generation when the input noise high (e.g.. 16 16 input).
E.4 ADDITIONAL QUALITATIVE RESULTS
Finally, we provide additional qualitative results for our unconditional models for FFHQ (Figure 16 AFHQ (Figure 17). LSUN (Figure 18 and our class-condi ImageNet model (Figure 19 20
F LIMITATIONS AND FUTURE WORK
Although f-DM enables diffusion with signal transformations, which greatly extends the scope of DMs to work in transformed space, there still exist limitations and opportunities for future work. First, an empirical question to find the optimal stage schedule for all transformations. Our ablation studies also show that different heuristics have differences for DS-based and VAE-based models. A metric that can automatically determine the best stage schedule based on the property of each transformation is needed and will be explored in the future. addition, although the current method achieves faster inference when generating with transformations like down-sampling, the speed-up is not very significant we still take the standard DDPM steps. How to further accelerate the inference process of DMs is a challenging and orthogonal direction. For example, it has great potential to combine f-DM with speed-up techniques such as knowledge distillation (Salimans & Ho 2022. Moreover, no matter hand-des or learned, all the transformations used in f-DM are still fixed when training DM. is, however, different from typical VAEs, where both the encoder and decoder are jointly optimized during training. Therefore, starting from random/imperfeet transformation and training f-DM jointly with the transformations towards certain target objectives will be studied future work.
20

---

Published as a conference paper at ICLR 2023
Input Image
without
Grad-Init
Target Irage
F-DN- (256x266)
(36x16)
(250x206)
Figure 14: Additional examples of super-resolution (SR) with the unconditional f-DM-DS trained on AFHQ. 1 The same input image with various resolution 162 .22 642 1282. We sample 3 random seeds for each resolution input. We also show the difference with and without applying gradient- based initialization IGitd-Init) J. SR results of various 162 inputs.
21

---

Published as a conference paper at ICLR 2023
Ingut Image
Without Grad-Init
With Grad-Init
Target Image
Input Image (blur a=15)
T-DM-Blur- O Output (Doblur)
Target Image (No blur?
Additional examples of de-blurring with the unconditional f-DM-Blur Blur-G trained on AFHQ. 1 The same input image with various Gaussian kernel sizes o 15, 9, 4, 1.4. We sample 3 random seeds for each resolution input. We also show the difference with and without applying gradient-based initialization (Grad-Init) on z. +. Deblurred results of various blur images.
Figure 15:
22

---

Published as a conference paper at ICLR 2023
F-DM-Blur-U
DN VOVAE
Figure 16: Random samples generated by five f-DMs trained on FFHQ 256 x 256. All faces presented synthesized by the models, and are not real identities.
23

---

Published as a conference paper at ICLR 2023
F-DN-DS
f-DM- Blur
F-VA-VQVAE
FNN VOSAN
Figure 17: Random samples generated by five f-DMs trained on AFHQ 256 x 256.
24

---

Published as a conference paper at ICLR 2023
DS-
11
Noot
WIEK
--Du-BUur-U
VQVVAE
F-
F- -QVAE
Figure 18: Random samples generated by f f-DMs trained on LSUN-Church & -Bed 256 x 256.
25

---

Published as a conference paper at ICLR 2023
T-DN-DS
T- T-DN-VOVAE
CI/IVE WILL
MA
EU L
MARR
Figure 19: Random samples generated by f-DM-DS/VQVAE trained on ImageNet 256 x 256 with classifier-free guidance (s = 3). Classes from top to bottom: red panda, robin, daisy, valley, trifle, comic book.
26

---

Published as a conference paper at ICLR 2023
F-DM-DS
f- DM-VVVVEE
WOMLC
Figure 20: Random samples generated by J-DM-DS/VQVAE trained on ImageNet 256 x 256 with classifier-free guidance (s = 3). Classes from top to bottom: school bus, pizza, seashore, photo- copier, golden retriever, axolotl.
27


Reviews:
[{"review_id": "wGrayXbxvEU", "reviewer": "Reviewer_C5sD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to train diffusion models via multi-stage progressive signal transformations. Compared to traditional diffusion models, the signal transformation here is more general and can be any transformation from coarse to fine. The authors focus on three main transformations (down-sample, Gaussian blur and transformations learned from a neural network) in the experiments, and demonstrate the effectiveness of the approach.", "summary_of_the_review": "This work considers learning diffusion models on data processed with a sequence of signal transformations. Similar ideas have been explored in other generative models, including normalizing flows, GANs and VAEs. It is almost straightforward to apply similar ideas to diffusion models. Although the proposed approach is claimed to work for any transformation, it remains unclear how effective it would be for random transformations besides the ones the authors performed experiments on. At the same time, there are already works that learn diffusion models on down-sampled images or latent spaces. Given these related works, the novelty of this work is questionable. ", "strength_and_weaknesses": "Strength:\n1. The paper is well-written with thorough experimental results.\n2. This paper considers a very interesting direction of diffusion models by incorporating progressive signal transformation.\n\nWeakness:\n1. Limited novelty: Similar ideas (e.g., incorporating progressive signal transformation into generative modeling) have be extensively explored for other generative models including GANs [1], normalizing flows [2,3], and VAEs. For the three major transformations considered in the paper (down-sample, gaussian blur and VQ-VAE/VQ-GAN), there are already existing work on diffusion models using each of the transformation as mentioned in the paper. Specifically, for transformations learned by neural networks, diffusion models trained on a latent space fit into this category: there have been various works in this direction and it is not a new idea. For instance, besides LDM, [4,5,6] also perform diffusion on data transformed with an encoder. In some sense, the contribution of this paper seems to unify these approaches into one framework by calling them \"diffusion models learned with progressive signal transformation\".\n\n2.  Limited theoretical insights: the contribution of this paper is mainly on the empirical side. There are not enough theoretical insights or guarantee.\n\n3. It seems that the performance (e.g., sample quality/speed) depends heavily on the transformation $f$ and the stage scheduling. More discussion should be provided on the selection of the optimal tranformations. Although the transformation are claimed to be any transformation, it remains unclear how effective/useful the other transformations are besides the ones used in the paper  (down-sample, Gaussian blur and transformations learned from a neural network). However, there are already existing work using these transformations. \n \n4. In the experiment, the reimplemented Cascaded DM (figure 4) seems to have much worse performance than the ones reported in the original paper. It is unclear how trustworthy the results are. \n\n5. For comparison with DDPM with fewer steps (DDPM 1/2), the authors should also consider comparing with DDIM 1/2 [7] in Table 1.\n\n6. How expensive it is to find the stage scheduling used in the experiments? Is there any theoretical connection to ODE/SDE? \n\n[1] Progressive Growing of GANs for Improved Quality, Stability, and Variation: https://arxiv.org/abs/1710.10196\n\n[2] Wavelet Flow: Fast Training of High Resolution Normalizing Flows: https://arxiv.org/abs/2010.13821\n\n[3] Improving Continuous Normalizing Flows\nusing a Multi-Resolution Framework: https://arxiv.org/abs/2106.08462\n\n[4] D2C: Diffusion-Denoising Models for Few-shot Conditional Generation: https://arxiv.org/abs/2106.06819\n\n[5] Diffusion-LM Improves Controllable Text Generation: https://arxiv.org/abs/2205.14217\n\n[6] Symbolic Music Generation with Diffusion Models: https://arxiv.org/abs/2103.16091\n\n[7] Denoising Diffusion Implicit Models: https://arxiv.org/abs/2010.02502\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OC0moFCj_b", "reviewer": "Reviewer_Z8a2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission deals with designing diffusion models by using progressive signal transformation. It proposes a generalized formulation of diffusion models, termed f-DM, with a modified sampling that is applied to image generation tasks with a range of signal transformations such as down-sampling, blurring, and learn compression based on pretrained VAEs. The reported experiments show better efficiency, quality and interpretation for image generation based on a few standard image datasets. \n", "summary_of_the_review": "This paper deals with an important and timely problem. The proposed solution is also a solid generalization of denoising diffusions to general signal transformations and distortions. The reported experimental results also demonstrate significant FID improvement over some benchmarks. However, comparison with LDMs is not quite convincing and clear. ", "strength_and_weaknesses": "Strength\n- It is a solid idea that generalized denoising diffusions to other signal transformation tasks\n- Experiments are performed a diverse set of generation tasks and datasets\n\nWeakness\n- The comparison with LDM, based on the experiments, is not conclusive. LDM (GAN) seems to be achieving the best FID, while Fig. 4 shows images with artifacts. Also, for more realistic and diverse datasets such as LSUN and ImageNet no comparison is performed with LDMs, while LDMs tend to achieve much better speed. \n\nQuestions and comments\n- About f-DMs versus LDMs, it is not theoretically convincing if progressive transformation (such as down-sampling) can be more efficient. - At least from the experiments one can see that LDMs are much more efficient, and one needs to see the LDM trained with strong encoder and decoders on diverse datasets such as LSUN and ImageNet to make a conclusion.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "6nRIb6v6K2k", "reviewer": "Reviewer_FRYF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a diffusion model which incorporates functions other than the addition of Gaussian noise in the forward process. Concretely, they show results with downsampling functions, blurring, and a learned neural encoding. ", "summary_of_the_review": "The proposed method seems to be well-thought out and of interest to the ICLR community. I recommend acceptance, although the clarity/presentation could still be improved.", "strength_and_weaknesses": "Strengths:\n- At a high level, I think this is an elegant method to integrate transformations into a diffusion process. This could be of considerable interest to the ICLR community.\n- Well thougt-out design choices including interpolating between $\\hat{x}_k$ and $\\hat{x}_k$ at each stage between transformations (as described in Equation 4), parameterisation of the network to predict both $\\epsilon$ and $\\delta$, and definition of $q(z_t|x)$.\n- Good experimental results.\n\n\nWeaknesses:\n - Slightly confusing presentation - reading Section 3.1 was reasonably difficult. I think I understand it after spending considerable time on it, but suspect that it could be made clearer with more explanation (or perhaps a diagram/expanded version of Fig 2b) of the relationships between each $x$, $\\hat{x}$, $z_t$, etc. \n- In Equation 7, $\\Omega$ is said to be the \"minimal interested patch\", with no further explanation. And the \"SIGNAL\" and \"NOISE\" functions in Eq. 7 are not explained at all. The conclusions reached about setting $\\alpha_{\\tau}$, $\\alpha_{\\tau^-}$, $\\sigma_{\\tau}$, and $\\sigma_{\\tau^-}$ seem reasonable, but their relationship to Eq. 7 needs more explanation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster