Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidance
Authors: Chen Henry Wu
Fernando De la Torre
Venue: Submitted to ICLR 2023 (2023)
Keywords: diffusion models
generative models
image-to-image translation
controllable generation
zero-shot learning
text-to-image synthesis

Abstract: Diffusion models have achieved unprecedented performance in generative modeling. The commonly-adopted formulation of the latent code of diffusion models is a sequence of gradually denoised samples, as opposed to the simpler (e.g., Gaussian) latent space of GANs, VAEs, and normalizing flows. This paper provides an alternative, Gaussian formulation of the latent space of various diffusion models, as well as an invertible DPM-Encoder that maps images into the latent space. While our formulation is purely based on the definition of diffusion models, we demonstrate several intriguing consequences. (1) Empirically, we observe that a common latent space emerges from two diffusion models trained independently on related domains. In light of this finding, we propose CycleDiffusion, which uses DPM-Encoder for unpaired image-to-image translation. Furthermore, applying CycleDiffusion to text-to-image diffusion models, we show that large-scale text-to-image diffusion models can be used as zero-shot image-to-image editors. (2) One can guide pre-trained diffusion models and GANs by controlling the latent codes in a unified, plug-and-play formulation based on energy-based models. Using the CLIP model and a face recognition model as guidance, we demonstrate that diffusion models have better coverage of low-density sub-populations and individuals than GANs.

Under review as a conference paper at ICLR 2023
UNIFYING DIFFUSION MODELS' LATENT SPACE, WITH APPLICATIONS TO CYCLEDIFFUSION AND GUIDANCE
Anonymous authors
Paper under double-blind review
ABSTRACT
Diffusion models have achieved unprecedented performance in generative model- ing. The commonly-adopted formulation of the latent code diffusion models a sequence of gradually denoised samples, opposed to the simpler (c.g., Gaussian) latent space of GANs, VAEs, and normalizing flows. This paper provides an alter- native, Gaussian formulation of the latent space of diffusion models, well reconstructable DPM-Encoder that maps images into the latent space. While our formulation is purely based on the definition of diffusion models, demonstrate several intriguing consequences. (1) Empirically, we observe that a common latent space emerges from two diffusion models trained independently related domains. for
In light of this finding, we propose CycleDiffusion, which uses unpaired image-to-iman to-image translation. Furthermore, applying CycleDiffusion eext-to-image diffusion models, we show that large-scale text-to-image diffusion models can be used as zero-shot image-to-image editors. (2) One can guide pre- trained diffusion models and GANs by controlling the latent codes in unified, plug-and-pll formulation based on energy-based models. Using the CLIP model and face recognition model as guidance, we demonstrate that diffusion models have better coverage of low-density sub-populations and individuals than GANs.
INTRODUCTION
Diffusion models (Song & Ermon, 2019; Ho et al., 2020) have achieved unprecedented results in generative modeling and are instrumental text-to-image models such as DALL-E 2 (Ramesh et al., 2022). Unlike GANs (Goodfellow et al., 2014), VAEs (Kingma & Welling, 2014), and normalizing flows (Dinh et al., 2015), which have a simple (e.g.. Gaussian) latent space, the commonly-adopee formulation of the "latent code" of diffusion models is sequence of gradually denoised images. This formulation makes the prior distribution the "latent code" data-dependent, deviating from the idea that generative models mappings from simple noises to data (Goodfellow al., 2014).
This paper provides a unified view of generative models of images by reformulating various diffusion models as deterministic maps from Gaussian latent code to an image æ (Figure 1, Section 3.1). A question that follows is encoding: how to map an image 2 to a latent code x. Encoding has been studied for many generative models. For instance, VAEs and normalizing flows have encoders by design, GAN inversion (Xia et al., 2021) builds post hoc encoders for GANs, and deterministic diffusion probabilistic models (DPMs) (Song et al., 2021a;b) build encoders with forward ODEs. However, it is still unclear how to build an encoder for stochastic DPMs such as DDPM (Ho et al., 2020), -on-deterministic DDIM (Song et al., 20) and latent diffusion models (Rombach et al., 2022). We propose DPM-Encoder (Section 3.2), reconstructable encoder for stochastic DPMs.
We show that some intriguing consequences emerge from our definition of the latent space of diffusion models and our DPM-Encode First, observations have been made that, given two diffusion models, a fixed "random seed" produces similar images (Nichol al., 2022). Under our formulation, we formalize "similar images" via an upper bound of image distances. Since the defined latent code contains all randomness during sampling, DPM-Encoder is similar-in-spirit to inferring the "random seed" from real images. Based on this intuition and the upper bound of image distances, we propose CycleDiffusion (Section 3.3), a method for unpaired image-to mange-to-image translation using
OOr codes will be publicly availahle.

---

Under review as a conference paper at ICLR 2023
GAN/ / VAE
INN
Daterministic DPMs
DDGAN
latent component
2T
2-T-1
2
-
intermediate variable
(w/ grad.)
Zyy
6''
Stochastic DPMs
image
2
T--1
Zo
Diffusion autoencoder
Latent diffusion models (LDMs)
AT
DDIM
high dim.
DDPM/ DDIM/ / (an abstraction)
Zy
250
20
DDIMx
$
low dim.
Z0
25
Once trained, various types of diffusion models can be reformulated deterministic maps from latent code z to image æ, like GANs, VAEs, and normalizing flows.
Figure I:
our -ncoeer Like the GAN- UNIT method (Liu et al., 2017), CycleDiffusion encodes and decodes images using the common latent space. Our experiments show that CycleDiffusion outperforms previous methods based on GANs or diffusion models (Section 4.1). Furthermore, by applying large-scale text-to-image diffusion models (e.g., Stable Diffusion; Rombach et al., 2022) to CycleDiffusion, we obtain zero-shot image-to-image editors (Section 4.2).
With a simple latent prior, generative models can be guided in plug-and-play manner by means of energy-based models (Nguyen et al., 2017; Nie et al., 2021; Wu et al., 2022). Thus, our unification allows unified, plug-and-play guidance for various diffusion models and GANs (Section 3.4), which avoids finetuning the guidance model on noisy images for diffusion models (Dhariwal & Nichol, 2021; Liu et al., 2021). With the CLIP model and face recognition model as guidance, show that diffusion models have broader coverage of low-density sub-populations and individuals (Section 4.3).
RELATED WORK
Recent years have witnessed a great progress in generative models, such as GANs (Goodfellow et al., 2014), diffusion models (Song& & Ermon, 2019; Ho et al., 2020; Dhariwal & Nichol, 2021), VAEs (Kingma& & Welling, 2014), normalizing flows (Dinh et al., 2015), and their hybrid extensions (Sinha et al., 2021; Vahdat et al., 2021; Zhang& & Chen, 2021; Kim et al., 2022a). Previous works have shown that their training objectives are related, e.g., diffusion models as VAEs (Ho et al., 2020; Kingma et al., 2021: Huang et al., 2021); GANs and VAEs as KL divergences (Hu et al., 2018) or mutual information with consistency constraints (Zhao et al., 2018); a recent attempt (Zhang et al., 2022b) has been made to unify several generative models GFlowNets (Bengio et al., 2021). In contrast, this paper unifies generative models as deterministic mappings from Gaussian noises to data (aka implicit models) once they are trained. Generative models with non-Gaussian randomness (Davidson et al., 2018; Nachmani et al., 2021) can be unified as deterministic mappings in similar ways.
One of the most fundamental challenges in generative modeling is to design an encoder that is both computationally efficient and invertible. GAN inversion trains an encoder after GANs are pre-trained (Xia et al., 2021). VAEs and normalizing flows have their encoders by design. Song et al. (2021a;b) studied encoding for ODE-based deterministic diffusion probabilistic models (DPMs). However, it remains unclear how to encode for general stochastic DPMs, and DPM-Encoder fills this gap. Also, CycleDiffusion can be seen as an extension of Su et al. (2022)'s DDIB approach to stochastic DPMs.
Previous works have formulated plug-and and-play guidance of generative models as latent-space energy- based models (EBMs) (Nguyen et al., 2017; Nie et al., 2021; Wu et al., 2022), and our unification makes it applicable various diffusion models, which are effective for modeling images, audio (Kong et al., 2021), videos (Ho et al., 2022: Hoppe al., 2022), molecules (Xu et al., 2022), 3D objects (Luo & Hu, 2021), and text (Li et al., 2022). This plug-and-ple -lly guidance can provide principled, fine-grained model comparisons of coverage of nub-populations and individuals on the same dataset.
concurrent work observed that fixing both (1) the random seed and (2) the cross-attention map in Transformer-base based text-to-in to-image diffusion models results in images with minimal changes (Hertz et al., 2022). The idea of fixing the cross- attentionnnionn map is named Cross Attention Control (CAC) in that work, which can be used to edit model- sendel-generated images when the random seed is known.
2

---

Under review as a conference paper at ICLR 2023
For real images with stochastic DPMs, they generate masks based on the attention map because the random seed unknown for real images. In Section 4.2, we show that CycleDiffusion and CAC can be combined to improve the structural preservation of image editing.
Table I: Details of redefining various diffusion models' latent space (Section 3.1).
Latent code
Deterministic map G(z)
2T-1 ur (TT, T) + oT O CT
Stochastic DPMs
( eq
@ 61)
31 pr (Tt 1) + oi ② E (t TTT, a == Tig
#T-1 ur(ar.I)
Deterministic DPMs == Tr (T' Tg with gradient)
F4-1 p (2 1) (t < T): x = an.
LDM
of G Ganees
Zy Glassu (2); x = D(z)).
DifFAE
2
(Zy mr)
Zg
( a 3y (
-r pr 21, T) or ET;
2 t (eT (1) ør ① eT Cp-..
DDGAN
R1-1 = po ( 21, f) + a. 3 (1 < <t< 7),
1 22 Eg 1 21)
x wo ur( 21.
METHOD
3.1
GAUSSIAN LATENT SPACE FOR DIFFUSION MODELS
Generative models such as GANs, VAEs, and normalizing flows can be seen as a family of implicit models, meaning that they are deterministic maps G Fd - X' from latent codes to images At inference, sampling from the image prior x ~ Paz (a:) implicitly defined as 2 ~ Pa (2), x = G(z). The latent prior (=(z) is commonly chosen to be the isometric Gaussian distribution. In this section, we show how to unify diffusion models into this family. Overview is shown in Figure and Table 1.
Stochastic DPMs:
Stochastic DPMs (Ho al., 2020; Song & Ermon, 2019; Song et al., b;a; Wat- son et al., 2022) generate images with a Markov chain. Given the mean estimator jer (see Appendix A) and ar (0, T), the image
Fo is generated through -; NN (ar l), diag (o()). Us- ing the reparamcterization trick, we define the latent code z and the mapping G recursively as
MT (x t) of Et, t t= T, ,,,,,,,, 1, (T @ ET + E1) ~ I), It-1
(I)
where 3 is concatenation. Here, has dimension d x (T + where dy is the image dimension.
Deterministic DPMs: Deterministic DPMs (Song et al., 2021 Salimans & Ho, 2022; Liu et al., 2022; Lu et al., 2022; Karras al., 2022: Zhang & Chen, 2022) generate images with the ODE formulation. Given the mean estimator pr, deterministic DPMs generate == De via
== ET ~ N (O, I), xt-1 = (T t), t t=
(2)
Since backpropagation through Eq. (2) is costly, we use fewer discretization steps Tg when computing gradients. Given the mean estimator μT, with number of steps Tg. the image x 2o is generated as
(with gradients)
ZT. (O,,I), I), I;-1 prz (z,, t), t t= ==
(3)
Latent diffusion models (LDMs): An LDM (Rombach et al., 2022) first uses diffusion model Giment to compute a "latent code" 2n (z), which then decoded x D(z)). Note that Gratent is an abstraction of the diffusion models that are already unified above.
Diffusion autoencoder (DiffAE):: DiffAE (Preechakul et al., 2022) first uses a deterministic DDIM to generate a "latent code" 2o.² 2 which is used as condition for an image-space deterministic DDIM:
(ZT @T) (O, I), Zg DDIM (ZT),
2g DDIM XDIMx(xT,20) (#T,
(4)
DDGAN: DDGAN (Xiao et al., 2022) models each reverse time step as a GAN conditioned on the output of the previous step. We define the latent code and generation process G of DDGAN as
(T zT 1
22 62 @ 2 NOO
(5)
Øt-1 pr t)
t = T 2,
x = 2g = p1(21, æ1, 1).
Quotation marks stand for "latent code" in the eited papers, different from latent code in Section 3.1.
3

---

Under review as a conference paper at ICLR 2023
Algorithm 1: CycleDiffusion for zero-shot image-to-image translation
Input: source image zo; source text E; target text é; encoding step Tos S T Sample noisy image æT. = VT. ~ (TT (z))
for { = .a...., do
2. 0 ~ 9( Ft. ve)
3. 6, = [i 10t11-r(app
, (1) 0
4. 2;-1 ( t|t) 0 Ci
Output: 2 = L0
3.2 DPM-ENCODER: A RECONSTRUCTABLE ENCODER FOR DIFFUSION MODELS
In this section, we investigate the encoding problem, i.e., x Enc( z x. G). The encoding problem has been studied for many generative models, and contribution is DPM-Encoder, stochastic DPMs. DPM-Fnccdder is defined as follows. For each image æ == De, stochastic DPMs define posterior distribution ((0)))()))) (Ho et al., 2020; Song al., 2021a). Based g(>1:) (ano) and Eq. (I), we can directly derive ~ DPMEnc(* (2|x, [x, G) as (see details in Appendices A and B)
encoder for
Bh.... Cir.,,TTii -T- ØT ( 6: (a:-1 uT f)) oe, t TT.....,
(6)
z [r Cy 1 ... ① C2 e2 G.).
A property of DPM-Encoder is perfect reconstruction, meaning that we have a: G(z) for every Eno(z|a; G). proof by induction is provided Appendix
3.3 CYCLEDIFFUSION: IMAGE-TO-IMACE TRANSLATION WITH DPM-ENCODER
Given two stochastic DPMs G1 and G2 that model two distributions D3 and D2, several researchers and practitioners have found that sampling with the same "random seed" leads to similar images (Nichol et al., 2022). To formalize "similar images", based on assumptions about the trained DPMs, shown at the end of this subsection. Based on this finding, we propose simple unpaired image-to-imet to-image translation method, CycleDiffusiin.. Given source image æ E Dj, we use DPM-Encoder to encode it as and then decode it as a = G2(z):
provide an upper bound of image distances
DPMEnc(= G1), Φ = G2(x).
We can also apply CycleDiffusion text-to-image diffusion models by defining D1 and D: image distributions conditioned on two texts. Let Ge be a text-to-image diffusion model conditioned on text t. Given source image ær, the user writes two texts: source text describing the source image and target text & describing the target image & to be generated. We can then perform zero-shot image-to-imane editing via (zero-shot means that the model has never been trained on image editing)
DPMEnc 2 x, Gt), Φ = Gg(z).
Inspired by the realism-faithful tradeoff in SDEdit (Meng et al., 2022), we can truncate = towards a specified encoding step Tes < T. The algorithm CycleDiffusion is shown Algorithm 1.
Why does fixed leads to similar images? We illustrate with text-to-image diffusion models. Suppose the text-to-image model has the following two properties:
1. Conditioned on the same text, similar noisy images lead to similar enough mean predictions. Formally, (2,, ttt is K2- Ky-Lipschitz, i.e., | t/t) pr (2., t|E]]| < K2 Kella, a. ||-
2. Given the same image, the two texts lead to similar predictions. Formally, I| (æ, t|t) Pr (zz, +| < S. Intuitively, a smaller difference between t and gives us a smaller S..
Let B, be the upper bound of ||ا, æ at time step t when the same latent code sampling (i.e., 2g (z) and 2. = Gz(z)). We have B; = 0 because || - a. 22 = 0, and Bi is the upper bound for the generated images ||ær - #l12. The upper bound B, can be propagated through time, from T to O. Specifically, by combining the above two properties, we have
used for
B (Ke + 11) + S.
4

---

Under review as a conference paper at ICLR 2023
3.4 UNIFIED PLUG-AND-PLAE PLAYY GUIDANCE FOR GENERATIVE MODELS
Prior works showed that guidance for generative models can be achieved in the latent space (Nguyen al., 2017: Nie al., 2021; Wu et al., 2022). Specifically, given condition one can define the guided image distribution as an energy-based model (EBM); p(=[C) Pa (xx)e (x Sampling for ~ p(=+CC) equivalent z pa (=IC), x G(z), where p(=[C) ps (=
Examples of the energy function E(x (C) are provided in Section 4.3. To sample 2 ~ p(=|C), one can use any -addel-agnostic samplers. For example, Langevin dynamies (Welling & Teh, 2011) starts from z (0) (0, r) and samples
(m) iteratively through
(l++) (A) + 2 v= (ogg=(=(1)) - E(G(= Vow!!!, w ~ ~10, (10)
Table 2: Quantitative comparison for unpaired image-to-image translation methods. Methods in the second block use the same pre-trained diffusion model in the target domain. Results of CUT, ILVR, SDEdit, and EGSDE are from Zhao et al. (2022). Best results using diffusion models are in bold. CycleDiffusion has the best FID and KID among all methods and the best SSIM among methods with diffusion models. Note that it has been shown that SSIM is much better correlated with human visual perception than squared distance- based metrics such as Lg and PSNR (Wang et al., 2004).
Cal
Dog
Wild - Dog
FID
KIDx10
PSNRT
SSIM
FIDI
KIDx10
PSNRT
SSIM
CUT (GAN SOTA; Park al., 2020)
76.21
17.48
0.601
92.94
17.20
0.592
ILVR (Choi al., 2021)
74.37
17.77
0.363
75.33
16.85
0.287
SDEdit (Meng 2022)
74.17
19.19
0.423
68.51
17.98
0.343
EGSDE (Zhao al., 2022)
65.82
19.31
0.415
59.75
18.14
0.343
CycleDiffusion w/ DDIM (1/ 0.1)
58.87
20.3
18.50
0.557
56.45
19.5
17.82
0.479
EXPERIMENTS
This section provides experimental validation of the proposed work. Section 4. shows how CycleDif- fusion achieves competitive results on unpaired mmage-to-image translation benchmarks. Section 4.2 provides a protocol for what we call zero-shot image-to-image translation; CycleDiffusion outper- forms several image-to-image translation baselines that we re-purposed for this new task. Section 4.3 shows how diffusion models and GANs can be guided a unified, plug-and-plas formulation.
4.1 CYCLEDIFFUSION FOR UNPAIRED IMAGE-TO-IMAGE TRANSLATION
Given two unaligned image domains, unpaired image-to-image translation aims at mapping images in one domain the other. We follow setups from previous works whenever possible, detailed below. Following previous work (Park et al., 2020; Zhao et al., 2022), we conducted experiments on the test set of AFHQ (Choi et al., 2020) with resolution 256 256 for Cat Dog and Wild Dog. For each source image, each method should generate target image with minimal changes. Since CycleDiffusion sometimes generates noisy outputs, we used Teteii steps of SDEdit for denoising. When T' 1000, we set Tassin = 100 for Cat + Dog and Tadsii- = 125 for Wild + Dog.
To evaluate realism, we reported Frechet Inception Distance (FID; Heusel et al., 2017) and Kernel Inception Distance (KID; Bićikowki et al., 2018) between the generated and target images. To evaluate faithfulness, we reported Peak Signal-to-Nonn -Nisee Ratio (PSNR) and Structural Similarity Index Mcasure (SSIM; Wang et al., 2004) between each generated image and its source image.
Metries:
We compared CyclcDiffusion with previous ofate-of-hhe-e the-art unpaired image-to-image translation methods: CUT (Park et al., 2020), ILVR (Choi et al., 2021), SDEdit (Meng et al., 2022), and EGSDE (Zhao et al., 2022). CUT is based on GAN, and the others use diffusion models.
Baselines:
Pre-trained diffusion models: ILVR, SDEdit, and EGSDE only need the diffusion model trained on the target domain, and we followed them to use the pre- pre-trrined model from Choi et al. (2021) 1) for Dog. CycleDiffusion needs diffusion models both domains, trained them Cat and Wild.
5

---

Under review as a conference paper at ICLR 2023
Seen in Table 2 are the results. CycleDiffusion has the best realism (i.e., FID and KID). There is mismatch between the faithfulness metrics (i.e., PSNR and SSIM), and note that SSIM is much better correlated with human perception than PSNR (Wang et al., 2004). Among all diffusion model-based methods, CycleDiffusion achieves the highest SSIM. Figure 2 displays some image samples from CycleDiffusion, showing that our method can change the domain while preserving local details such as the background, lighting, pose, and overall color the animal.
Wild
Dog
Dog
Figure 2: Unpaired image-to-image translation (Cat Dog, Wild Dog) with CycleDiffusion.
4.2 TEXT-TO-IMAGE DIFFUSION MODELS CAN BE ZERO-SHOT IMAGE-TO-IMACE EDITORS
This section provides experiments for zero-shot image editing. We curated set of 150 tuples (æ, t. t) for this task, where æ the source image, is the source text (e.g., "an aerial view of autumn scene." in Figure 3 second row on the right), and & is the target text (c.g., "an aerial view of winter scene."), The generated image is denoted as â. We also demonstrate that CycleDiffusion can be combined with the Cross Attention Control (Hertz et al., 2022) to further preserve the image structure.
Metrics: To evaluate faithfulness to the source image, we reported PSNR and SSIM. To evaluate au- thenticity to the target text, we reported the CLIP score (c f) = cos (CLIP ume (æ), ) CLIP (t)). where the CLIP embeddings are normalized. We note trade-off between PSNR/SSIM and SCIPP: by copying the source image we get high PSNR/SSIM but low SCLIP, and by ignoring the source image (e-g., by directly generating images conditioned on the target text) we get high ScuP but low PSNR/SSIM. To address this trade-off, we also reported the directional CLIP score (Patashnik et al., 2021) (both CLIP embeddings and the embedding differences are normalized):
SD.CLIP( ar, t, f) cos ( (CLIPing mg ( CL.IP ing (æ)), CLIP (f) CLIPPEu(t).
(11)
The baselines include SDEdit (Meng et al., 2022) and DDIB (Su et al., 2022). We used the same hyperparameters for the baselines and CycleDiffusion whenever possible (e.g., the number of diffusion steps, the strength of classifier-free guidance; see Appendix C).
Baselines:
Pre-trained text-to-image diffusion models: To investigate how the zero-shot performance changes with data size, data quality, and training details, we used the following models: (1) LDM-400M, a 1.45B-paramete model trained on LAION-400M (Schuhmann al., 2021), (2) SD-v1-1, 0.98B- parameter Stable Diffusion model trained on LAION-5B (Schuhmann et al., 2022), (3) SD-v1-4, finetuned from SD-v1-1 for improved aesthetics and classifier frreee guidance.
Results: Table 3 shows the results for zero-shot image-to to-image translation. CycleDiffusion excels being faithful the source image (i.e., PSNR and SSIM); by contrast, SDEdit and DDIB have comparable authenticity to the target text (i.e., SC.IP) but their outputs are much less faithful. For all methods, we find that the pre-trained weights SD-v1-1 and SD-v1-4 have better faithfulness than LDM-400M. Figure 3 provides samples from CycleDiffusion, demonstrating that CycleDiffusion achieves meaningful edits that span (I) replacing objects, (2) adding objects, (3) changing styles, and (4) modifying attributes. See Figure 7 (Appendix E) for qualitative comparisons with the baselines.
CycleDiffusion Cross Attention Control: Besides fixing the random seed, Hertz et al. (2022) shows that fixing the cross attention map (i.e., Cross Attention Control, CAC) further improves the similarity between synthesized images. CAC is applicable to CycleDiffusion: in Algorithm 1, we can apply the attention map of (2 . t)t) to pr (r t(f). However, we cannot apply to all samples because CAC puts requirements on the difference between t and f. Figure 4 shows that CAC helps CycleDiffusion when the intended structural change is small. For instance, when the intended change color but not shape (left), CAC helps CycleDiffusion preserve the background; when the intended change horse elephant, CAC makes the generated elephant to look more like horse in shape.

---

Under review as a conference paper at ICLR 2023
Table 3: sero-shot image editing. We did not fixed hyperparameterrs and neither did plot the trade-off curve. The reason is that every input can have its best hyperparameters and even random seed (such per-sample tuning for Stable Diffusion is quite common nowadays). Instead, for each input, we ran 15 random trials for each hyperparameter and report the one with the highest Sp.CLIP-
Method
ScLF
SpecIPt
PSNRT
SSIM
SDEdit (Meng et al., 2022)
0.332
0.264
13.68
0.390
LDM-480M
DDIB (Su et al., 2022)
0.324
0.195
15.82
0.544
CycleDiffusion w/ DDIM (q) 0.1; ours)
0.333
0.275
18.72
0.625
SDEdit (Meng ct al., 2022)
0.339
0,248
15.50
0.498
SD-v1-1
DDIB (Su et al., 2022)
0.324
0.189
18.58
0.658
CycleDiffusiin w/ DDIM (g = 0.1; ours)
0.331
0.262
21.98
0.731
SDEdit (Meng al., 2022)
0.344
0.258
15.93
0.512
SD-v1-4
DDIB (Su et al., 2022)
0.331
0.209
18.10
0.653
CycleDiffusion w/ DDIM (g) 0.1; ours)
0.334
0.272
21.92
0.731
Nock bear walking
srass.
peat red flowess.
Tbe spound
with snow
Oil paiming of
Makoto Shinkai.
A of loughing wormas.
man.
applec Randi Ineckpacc flooti
Lu
valley.
apples
car in side of the strect)
A Acmertible car
An car
astresu niing home.
I sn clephurs
boslevand wilhi strme statuss at siles, at auturnn
Ien elephants
HB
phono
dog.
sitting
I..
boy lse-creau it night. Anime
I.. and
on
Figure 3: CycleDiffusion for zero-shot image editing. Source images æ are displayed with purple margin; the other images are the generated æ. Within each pair of source and target texts, overlapping text spans are marked in purple in the source text and abbreviated as [ in the target text.
A green apple and black backpack on Boor A.red
An astronaut riding a horse,
Iee un elephant.
Source
CyclDiiflusinn
CycleDiffusion CAC
Image
CycleDiflusion
Cycleiiftssinn CAC
Figure 4: Cross Attention Control (CAC; Hertz et al., 2022) helps CycleDiffussin when the intended structural change small. For instance, when the intended change is color but not shape (left), CAC helps CycleDiffusion preserve the background; when the intended change is horse - elephant, CAC makes the generated elephant look more like horse in shape.
7

---

Under review as a conference paper at ICLR 2023
Fcuplatf?
Feurlatee
0.30
0.28
SAA-AAN ERYAGANAL (124
DEEIGAANAL CHON)
0.27
TeyidED OXD4
CIRAFES
0.28
GIRABTE-EED
0.26
0.22
100
900
700
ACLIP
ACLIP
(a) a baby
(b) an old person
SOMIGNNAL
0.33
CIRARRE HED
0.30
DERAR (55))
Eaupler
0,30
0.29
SRYAGANAA 1124
0:28
0.28
0.27
DEBAE
100
900
900
ACLIP
ACLIP
(c) person with eyeglasses
(d) a person with eyeglasses and a yellow hat
Figure 5: Unified plug-and-plar guidance for diffusion models and GANs with text and CLIP. The text description used in each plot is photo of [ Image samples and more analyses are in Figure and Appendix G. When the guidance becomes complex, diffusion models surpass GANs.
4.3 UNIFIED PLUG-AND-PLL PLAY GUIDANCE FOR DIFFUSION MODELS AND GANS
Previous methods for conditional sampling from (aka guiding) diffusion models require training the guidance model on noisy images (Dhariwal & Nichol, 2021; Liu et al., 2021), which deviates from the idea of plug-and-play guidance by leveraging the simple latent prior of generative models (Nguyen et al., 2017). In contrast, our definition of the Gaussian latent space of different diffusion models allows for unified plug-and and- guidance of diffusion models and GANs. It facilitates principled comparisons over sub-pot populations and individuals when models are trained on the same dataset.
We used the text to specify sub-population. For instance, a photo baby represents the baby sub- population in the domain of human faces. We instantiate the energy in Section 3.4 as FCLPP (a:u((r))))) ())
E 1- cos (CLIP img (DiffAug, (x)), CLIP (o)) where DiffAug: stands for differentiable augmentation (Zhao et al., 2020) that mitigates the adversarial effect, and we sample from the energy- based distribution using Langevin dynamics in Eq. (10) with 200, 0.05. We enumerated the guidance strength (i.e., the coefficient de in Section 3.4) ACLPP E {100, 300, 500, 700, 1000). For evaluation, we reported (1 ÉCLI (x] averaged over 256 samples. This metric quantifies whether the sampled images are consistent with the specified text t. Figure plots models with pre-trained weights FFHQ (Karras et al., 2019) (citations in Table 5, Appendix G). In Figure 6, visualize samples for SN-DDPM and DDGAN trained on CelebA. We find that diffusion models outperform 2D/3D GANs for complex text, and different models represent the same sub-popul population differently.
Broad coverage of individuals is an important aspect of the personalized use of generative models. To analyze this coverage, guide different models generate images that close to reference
8

---

Under review as a conference paper at ICLR 2023
T, in the identity (ID) space modeled by the IR-SE50 face embedding model (Deng al., 2019), denoted as R. Given an ID reference image 2r we instantiated the energy defined in Section 3.4 Eap(22),))
1 cos [R(x), R(x,)) with strength Ap 2500 (i.c.. A in Section 3.4). For sampling, we used Langevin dynamics detailed in Eq. (10) with n = 200 and a = 0.05. To measure ID similarity to the reference image 3y, reported cos [( R(x,)) averaged over 256 samples. In Table 4, we report the performance of StyleGAN2, StyleGAN-XL. GIRAFFE-HD, EG3D, LDM- DDIM, DDGAN, and DiffAE. DDGAN trained on CelebAHQ. while others are trained FFHQ. We find that diffusion models have much better coverage of individuals than 2D/3D GANs. Among diffusion models, deterministic LDM-DDIM (1) 0) achieves the best identity guidance performance. We provide image samples of identity guidance in Figure (Appendix G).
(a) StyleGAN2 m
(b) StyleGAN-XL
(c) GIRAFFE HD
(d) StyleNeRF DU
(e) LDM-DDIM (v)
DifIAE
(g) DDGAN
(h) SN-DDPM
(i) SN-DDPM TO
Figure 6: Sampling sub- cab-populations from pre-trained generative models. Notations follow Figure 5.
Table 4: Guiding diffusion models and GANs with ID. (A, B, C, D) are images from FFHQ The metric the ArcFace cosine similarity (Deng et al., 2019). See samples in Figure 9 (Appendix G).
2D GAN
3D GAN
Diffussion model
StyleGAN2
StyleGAN-XL
GIRAFFE-HD
EG3D
LDM-DDIM
DDGAN
DiffAE
ID-A
0.561
0.681
0.616
0.468
0.904
0.837
0.873
ID-B
0.604
0.688
0.590
0,454
0.896
0.805
0,838
DD-C
0.495
0.636
0.457
0.403
0.892
0.795
0.852
ID-D
0.554
0.687
0.574
0.436
0.911
0.831
0.873
CONCLUSIONS AND DISCUSSION
This paper provides a unified view of pre-trained generative models by reformulating the latent space of diffusion models. While this reformulation is purely definitional, we show that it allows us to use diffusion models in similar ways as CycleGANs (Zhu et al., 2017) and GANs. Our CycleDiffusion translation (with two diffusion models
achieves impressive performance on unpaired trained on two domains independently) and zero-shot image-to-image translation (with rext-to-image diffusion models). Our definition of latent code also allows diffusion models to be guided in the same way as GANs (i.e., plug-and-plas without finetuning on noisy images), and results show that diffusion models have broader coverage of sub-populations and individuals than GANs.
Besides the interesting results, it is worth noting that this paper raised more questions than provided answers. We have provided formal analysis of the common latent space of stochastic DPMs via the bounded distance between images (Section 3.3), but it still needs further study. Notably, Khrulkov & Oseledets (2022) and Su et al. (2022) studied deterministic DPMs based on optimal transport. Furthermore, efficient plug-and -nd--lay guidance for stochastic DPMs on high-resolution images with many diffusion steps still remains open. These topics can be further explored in future studies.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. ICML, 2022.
Yoshua Bengio, Tristan Deleu, Edward J. Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio. GFlowNet foundations. ArXiv, 2021.
Mikolaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. ICLR, 2018.
Andrew Brock. Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. ICI.R, 2019.
Han K. Cao. Cheng Tan, Zhangyang Gao, Guangyong Chen, Pheng-Ann Heng, and Stan Z. Li. A survey on generative diffusion model. ArXiv, 2022.
Eric Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio
Gallo, Leonidas Guibas, Jonathan Tremblay, S. Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. CVPR, 2022.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR:
Conditioning method for denoising diffusion probabilistic models. ICCV, 2021.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. StarGAN v2: Diverse image synthesis for multiple domains. CVPR, 2020.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: A survey. ArXiv, 2022.
Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomezak. Hyperspherical variational auto-encoders. UAI, 2018.
Jiankang Deng. J. Guo, and Stefanos Zafeiriou. ArcFace: Additive angular margin loss for deep face recognition. CVPR, 2019.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. NeurlPS, 2021.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation. ICLR, Workshop Track Proceedings, 2015.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu. David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, 2014.
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. StyleNeRF: A style-based 3D aware generator for high-resolution image synthesis. ICLR, 2022.
Amir Hertz, Ron Mokady, Jay M. Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. ArXiv, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. NIPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPS, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. NeurIPS, 2022.
Tobias Hoppe. Arash Mchrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infilling. NeurIPS, 2022.
Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, and Eric Xing. On unifying deep generative models. ICL.R 2018.
10

---

Under review as a conference paper at ICLR 2023
Chin-Wor Huang, Jae Hyun Lim, and Aaron C. Courville. A variational perspective llffuusionbbsed generative models and score matching. NeurlPS, 2021.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. CVPR, 2020.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucciating the design space of diffusion- based generative models. NeurIPS, 2022.
Valentin Khrulkov and I. Oseledets. Understanding DDPM latent codes through optimal transport. ArXiv, 2022.
Dongjun Kim, Byeonghu Na, Se Jung Kwon, Dongsoo Lee, Wanmo Kang, and Il-Chul Moon. Maximum likelihood training of implicit nonlinear diffusion models. NeurlPS, 2022a.
Gwanghyun Kim, Taesung Kwon, and long-Chul Ye. DiffusionCLLIP: Text-guided diffusion models for robust image manipulation. CVPR, 2022b.
Diederik Kingma and Max Welling. Auto-cncoding variational Bayes. ICLR, 2014.
Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. NeurlPS, 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A versatile diffusion model for audio synthesis. ICLR, 2021.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffussin- LM improves controllable text generation. NeurlPS, 2022.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. /CLR, 2022.
Ming-Yu Yu Liu, Thomas M. Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. NIPS, 2017.
Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! Image synthesis with semantic diffusion guidance. ArXiv, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. ICCV, 2015.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. NeurIPS, 2022.
Shitong Luo and Wei Hu. Diffusion probabilistic models for point cloud generation. CVPR, 2021.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. ICLR, 2022.
Eliya Nachmani, Robin ArXiv, 2021.
and Lior Wolf. Non Gaussian denoising diffusion models.
Anh M Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. CVPR, 2017.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. ICML, 2022.
Weili Nie, Arash Vahdat, and Anima Anandkumar. Controllable and compositional generation with latent-space energy-based models. NeurlPS, 2021.

---

Under review as a conference paper at ICLR 2023
Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher- Shlizerman. StyleSDF: rigo-uesotttion 3D-consistent image and geometry generation. CVPR, 2022.
Taesung Park, Alexei A. Efros, Richard Zhang, and Jun- Yan Zhu. Contrastive learning for unpaired image-to-image translation. ECCV, 2020.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and D. Lischinski. StyleCLII. Text- driven manipulation of StyleGAN imagery. ICCV, 2021.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasom Suwajanakom. Diffu- sion autoencoders: Toward a meaningful and decodable representation. CVPR, 2022.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. ArXiv, 2022.
Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. CVPR, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani. Yacl Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. ArXiv, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. ICLR, 2022.
Axel Sauer, Katja Schwarz, and Andreas Geiger. StyleGAN-XL: Scaling StyleGAN to large diverse datasets. SIGGRAPH, 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs. ArXiv, 2021.
Christoph Schuhmann, Romain Beaumont, Cade W Gordon, Ross Wightman, mehdi cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Richard Vencu, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. NeurIPS Datasets and Benchmarks, 2022.
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. InterFaceGAN: Interpreting the disentan- gled face representation learned by GANs. TPAMI, 2022.
Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2C: Diffusion -ennosiinn models for sew-shot conditional generation. NeurIPS, 2021.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurlPS, 2019.
Yang Song, Jascha Sohk-Dickstein, Diederík Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2021b.
Xu Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. ArXiv, 2022.
Cristian Vaccari and Andrew Chadwick. Deepfakes and disinformation: Exploring the impact of synthetic political video deception, uncertainty, and trust in news. Social Media Society, 6, 2020.
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. NeurlPS, 2020.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. NeurIPS, 2021.
12

---

Under review as a conference paper at ICLR 2023
Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion- GAN: Training GANs with diffusion. ArXiv, 2022.
Zhou Wang, Alan Conrad Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13:600-612, 2004.
Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality. ICLR, 2022.
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient Langevin dynamics. ICML, 2011.
Mika Westerlund. The emergence of Deepfake technology: A review. Technology Innovation Management Review, 9(11), 2019.
Chen Henry Wu, Saman Motamed, Shaunak Srivastava, and Fermando De la Torre. Generative visual prompt: Unifying distributional control pre-trained generative models. NeurIPS, 2022.
Weihao Xia, Yulun Zhang, Yujuu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. GAN inversion: A survey, ArXiv, 2021.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. ICLR, 2022.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. GeoDiff: A geometric diffusion model for molecular conformation generation. ICI.R, 2022.
Yang Xue, Yuheng Li, Krishna Kumar Singh, and Yong Jae Lee. GIRAFFE HD: high-resolution 3D-aware generative model. CVPR, 2022.
Ling Yang, Zhilong Zhang, Shenda Hong. Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Ming-Hsuan Yang, and Bin Cui. Diffusion models: A comprehensive survey of methods and applications. ArXiv, 2022.
Bo Zhang, Shuyang Gu, Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo. StyleSwin: Transformer-bat GAN for high-resolution image generation. CVPR, 2022a.
Dinghuai Zhang, Ricky T. Q. Chen, Nikolay Malkin, and Yoshua Bengio. Unifying generative models with GFlowNets. ArXiv, 2022b.
Qinsheng Zhang and Yongxin Chen. Diffusion normalizing flow. NeurlPS, 2021.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. ArXiv, 2022.
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. EGSDE:
Unpaired image-to-image translation via energy-guided stochastic differential equations. NeurIPS, 2022.
Shengjia Zhao, Jiaming Song. and Stefano Ermon. The information- autonncoiinn family: A La- grangian perspective on latent variable generative modeling. ArXiv, 2018.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient GAN training. NeurIPS, 2020.
Jun-Yan Zhu, Taesung Park. Phillip Isola, and Alexei A. Efros. Unpaired sange-to-image translation using cycle-consistent adversarial networks. ICCV, 2017.
13


Reviews:
[{"review_id": "iwiG6eV_7Qo", "reviewer": "Reviewer_2RYz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the latent space of diffusion models compared to a variety of other generative models such as VAEs, Normalizing Flows, and especially GANs. First of all, this paper provides a very comprehensive and nice overview of latent space across deep generative models and motivates the study of latent space in diffusion models. Then, it traces down to the question about the latent space of stochastic DPMs rather than other methods which leverage \"fixed\" processes. The authors leverage the finding such that the randomness in the model captures the image-specific information and further utilize the defined latent space for an unpaired image translation task. It explores more tasks such as zero-shot image-to-image editor and latent space guidance with energy-based models.", "summary_of_the_review": "Overall, my main concern about this paper is the flow of the paper and solid experiments to support the method empirically as theoretical results are missing or challenging. I value the novelty of this paper to formulate this important yet not fully explored problem (latent space of diffusion models). I am leaning towards accepting this paper. ", "strength_and_weaknesses": "Strength:\n* This paper is mostly well-written with a clear literature review, leading to its motivation. \n* The studied problem is important, it is significant to understand the latent space of diffusion models as it has been done in GANs and other generative models. It can lead to more understanding and empirical applications of diffusion models.\n* The experimental results empirically support the claims and findings of the paper. \n\nWeaknesses\n* Even though the motivation to study the latent space of diffusion models and how this paper studies it with a simple yet effective approach is well-written, the applications to work as a CycleDiffusion, zero-shot image-to-image editing and latent space guidance sounds a little indirect to me, though this may be due to my biased perspective. I directly thought about GAN inversion and related literature, e.g. [1-3], after I read the motivation of this paper. I don't mean the applications in this paper are not okay, but I would suggest making the motivation for the applications tighter.\n* As a long-standing problem in understanding the latent space of deep generative models, even though this paper achieves great experimental results, I would still suggest the authors study theoretical guarantees of it.\n* The proposed D-CLIP score is a bit confusing, is the relative change meaningful without absolute value?\n* Unified Plug-and-Play Guidance is a bit misleading, why would the proposed or defined latent space unify diffusion models and GANs?\n\n[1] Härkönen, E., Hertzmann, A., Lehtinen, J. and Paris, S., 2020. Ganspace: Discovering interpretable gan controls. Advances in Neural Information Processing Systems, 33, pp.9841-9850.\n[2] Shen, Y., Yang, C., Tang, X. and Zhou, B., 2020. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence.\n[3] Shen, Y. and Zhou, B., 2021. Closed-form factorization of latent semantics in gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1532-1540).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cN8SNreVz9", "reviewer": "Reviewer_fAbM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to store the random noises along the generative trajectory of diffusion probabilistic model as a latent code, and use this latent code for image-to-image translation.", "summary_of_the_review": "The paper leverages an existing strategy to encode images generated by DPM, then directly apply this latent code to another DPM to achieve image-to-image translation. The novelty is limited and the results are not satisfying.", "strength_and_weaknesses": "Strength:\n(1) The paper provides a clear summary for previous works.\n(2) The paper is easy to digest.\n\n\nWeaknesses:\n(1) Limited Novelty: DPM-Encoder, which stores all the noises along the generative process, is an obvious way to encode an image to the latent space of a stochastic DPM. Since it is obvious, it appears in the famous work [1], Appendix B.2, to interpolate between images. Therefore, the encoding strategy is not new to me.\n(2) Limited Novelty: CycleDiffusion is a simple generalization of SDEdit [2] and the observation in [3]. The authors made no additional theoretical justification on the method, making its improvement over previous works limited.\n(3) Unsatisfying results: In Figure 3, many of the parts in the images, which we do not expect to change, changed. For example,  the astronaut figure on the bottom-left changes not only in the style, but also the content. This, in my opinion, is not fixable with the current method. \n\n[1] Improved Techniques for Training Score-Based Generative Models, https://arxiv.org/pdf/2006.09011.pdf\n[2] SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations https://arxiv.org/abs/2108.01073\n[3] Understanding DDPM Latent Codes Through Optimal Transport https://arxiv.org/abs/2202.07477\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "as__pqCeBU", "reviewer": "Reviewer_JmnA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides a unified setting among generative models. In particular it offers a perspective  on diffusion models (DM) that cast them in a framework more similar to GANs, which admit a Gaussian formulation of the latent space. This point of view is then leveraged in three settings. \nContributions: 1. provide a formulation of DM with Gaussian latent code  (introducing the concept of DPM-encoder that maps images into a latent space). 2. Propose CycleDiffusion for a) image-to-image translation and b) to show that text-to-image diffusion models can be used as zero-shot image-to-image editors. 3. Rely on the proposed perspective which unifies Diffusion Models and GANs to perform plug-and-play guidance.   ", "summary_of_the_review": "I think the paper could improve in its clarity. The novelty seems pretty limited, although CycleDiffusion used as zero-shot image-to-image editor is interesting in my opinion.", "strength_and_weaknesses": "Section 3.2: do I understand correctly that your DPM-Encoder is a proposal for how to build a z such that x = G(z) for z ~ DPMEnc(z | x, G), which amount to just concatenating x_T with \\eps_t as described?  \n\nCan you clarify the following: in section 3.1 I understood that in your unified framework, z is drawn from a Gaussian. In equation (6), it doesn't seem to be the case. In section 3.2, is the Gaussian latent  abandoned? \n \nI am a bit confused about the proof in Proposition 1. Do you use induction ((show it’s true for 0, assume it’s true for t, show it’s true for t+1)\n) or backward induction (show it’s true for T, assume it’s true for t, show it’s true for t-1)  ?\nAlso,  if the goal here to show that G(z) = x for any z sampled from DPMEnc ( | x), in the statement I would write x = G(z) =: \\bar{x}, just for clarity. \n\nSection 3.3: \nCan you clarify what you mean with “our defined latent codes contain all randomness?”\n“ we can formalize the above finding as a common latent space emerges...\" :can you clarify this sentence (what does it mean pragmatically?) ?  \nHow is the concept of  ‘common’ defined and measured? \n\nWhat do you mean “D_1 and D_2 do not need to be two datasets?” \n\nSection  4.1 \nPeak Signal-to-Noise Ratio: what is this an indicator of? Meaning CycleDiffusion has excellent  SSIM (which  is more correlated to human perception) than baselines, but lower PSNR (actually the lowest). What is this an indicator of? \n\n Section 4.3\nAm I correct to understand that your contribution in section 4.3 is the fact that you can “work on a latent space” for diffusion models (e.g. apply Langevin) similarly to what one can do with GANs? \nThe findings reported are interesting but reporting them without a critical explanation is maybe limiting. E.g “3D GANs are worse at guidance than 2D GANs and diffusion models”: why so? Or at least, what’s your opinion on this?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4UIunqmxcN", "reviewer": "Reviewer_NUmH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers a concatenation of all intermediate variables through a forward diffusion as a latent code.\n\nThe authors consider the forward process of diffusion models as an encoder.\n\nUsing two independently trained diffusion models, the proposed method encodes an image with a fixed random seed for one model and run reverse process on the other model to achieve image-to-image translation.", "summary_of_the_review": "This paper shows interesting phenomena but does not contribute to building general knowledge with logical development.", "strength_and_weaknesses": "\nStrengths\n\n(+) This paper shows interesting application of diffusion models.\n\n(+) The proposed latent code (concatenation of all variables) is versatile across different models.\n\n\nWeaknesses\n\nPlease refer to the below section.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2ViRC57QuF", "reviewer": "Reviewer_dmqB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper under consideration proposes a method for inverting stochastic diffusion models. Based on this method the authors come up with an (conditional) unpaired image-to-image translation approach called “CycleDiffusion”. The validity of the proposed methodology is verified by several applications.", "summary_of_the_review": "The authors provide a new method for solving (conditional) unpaired im-to-im problems based on stochastic diffusion models. In spite of the interesting methodological proposition and good practical results, the method requires further theoretical investigation.", "strength_and_weaknesses": "Strength:\n1) The proposed idea is clear and practically applicable. One can easily apply the proposed framework for a general stochastic diffusion model.\n\n2) The experiments look quite convincing.\n\nWeaknesses:\n1) The idea of DPM-Encoder itself seems to be quite easy and straightforward. All of the papers dealing with stochastic diffusion models (like DDPM) actually mentioned the forward form of the stochastic dynamic (i.e. how to obtain latent code $x_T$ from the original image $x_0$) but didn’t stress the importance of the dynamic. Therefore, the only theoretical contribution of the paper under consideration is that the authors highlight the importance of forward stochastic dynamics (for the problem of unpaired image-to-image generation related applications)\n\n2) As it even mentioned in the article itself, the methodology of stochastic diffusion based unpaired image-to-image generation is under question and requires theoretical verification.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1KnUw2Wm53S", "reviewer": "Reviewer_Nh8B", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a way of deriving an encoder from a pretrained diffusion model. In particular, this encoder is stochastic and invertible, which makes it different from existing attempts. The authors show that this encoder can be deployed in an image to image translation setting, and demonstrates interesting results. Moreover, a unified guidance method is proposed based on the derived latent representations. ", "summary_of_the_review": "Overall I think this is a promising paper with a simple idea and interesting results. I'd like to hear clarifications on my questions during the rebuttal phase. ", "strength_and_weaknesses": "1. The idea of defining the stochastic encoder is simple but interesting. \n2. Being able to encode an image and using the code for image to image translation or editing is clever.\n3. The empirical results on both image to image translation and text guided edits are impressive. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject