Diffusion Models Already Have A Semantic Latent Space
Authors: Mingi Kwon
Jaeseok Jeong
Youngjung Uh
Venue: ICLR 2023 notable top 25% (2023)
Keywords: diffusion models
semantic image editing

Abstract: Diffusion models achieve outstanding generative performance in various domains. Despite their great success, they lack semantic latent space which is essential for controlling the generative process. To address the problem, we propose asymmetric reverse process (Asyrp) which discovers the semantic latent space in frozen pretrained diffusion models. Our semantic latent space, named h-space, has nice properties for accommodating semantic image manipulation: homogeneity, linearity, robustness, and consistency across timesteps. In addition, we measure editing strength and quality deficiency of a generative process at timesteps to provide a principled design of the process for versatility and quality improvements. Our method is applicable to various architectures (DDPM++, iDDPM, and ADM) and datasets (CelebA-HQ, AFHQ-dog, LSUN-church, LSUN-bedroom, and METFACES).

Published as a conference paper at ICLR 2023
DIFFUSION MODELS ALREADY HAVE A SEMANTIC LATENT SPACE
Mingi Kwon, Jaeseok Jeong, Youngjung Uh"
Department of Artificial Intelligence Yonsei University
Scoul, Republic of Korea
{kwonmingi, jete- wi..hlyjynueeeeiee. ac. .r
ABSTRACT
Diffusion models achieve outstanding generative performance in various domains. Despite their great success, they lack semantic latent space which is essen- tial for controlling the generative process. To address the problem, we propose asymmetric reverse process (Asyrp) which discovers the semantic latent space in frozen pretrained diffusion models. Our semantic latent space, named It-pacc, has nice properties accommodate semantic image manipulation: homogeneity, linearity, robustness, and consistency across timesteps. In addition, we introduce a principled design of the generative process for versatile editing and quality boost- ing by quantifiable measures: editing strength an interval and quality deficiency at timestep. Our method is applicable various architectures (DDPM++, iD- DPM, and ADM) and datasets (CelebA-HQ. AFHQ-dog, LSUN-church, LSUN- bedroom, and METFACES). Project page: http:://wwomiiikiihttbb////////
INTRODUCTION
In image synthesis, diffusion models have advanced achieve ofate-of-the-rt performance regard- ing quality and mode coverage since the introduction of denoising diffusion probabilistic models (Ho al.] 2020). They disrupt images by adding noise through multiple steps of forward process and generate samples by progressive denoising through multiple steps of reverse (i.e., generative) process. Since their deterministic version provides nearly perfect reconstruction of original images (Song et al. 2020a), they suitable for image editing, which renders target attributes the real images. However, simply editing the latent variables (i.c., intermediate noisy images) causes de- graded results (Kim & Ye 2021). Instead, they require complicated procedures: providing guidance in the reverse process or finetuning models for an attribute.
Figure a-c) briefly illustrates the existing approaches. Image guidance mixes the latent variables of the guiding image with unconditional latent variables (Choi et al.) 2021 Lugmayr et al.) 2022 control, it ambiguous to specify which attribute to
Meng et al. 2021). Though it provides reflect among the ones in the guide and the unconditional result, and it lacks intuitive control for the magnitude of change. Classifier guidance manipulates images by imposing gradients on the latent variables in the reverse process to match the target class (Dhariwal & Nichol 202T Avrahami et al. [022 [iuu et al. (021). requires training an extra classifier for the latent variables, i.e., noisy images. Furthermore, computing gradients through the classifier during sampling is costly. Finetuning the whole model can steer the resulting images to the target attribute without the above
classifier
problems (Kim & Ye, 2021). Still, it requires multiple models to reflect multiple descriptions.
On the other hand, generative adversarial networks (Goodfellow et al. 2020) inherently provide straightforward image editing in their latent space. Given latent vector for original image, can find the direction the latent space that maximizes the similarity of the resulting image with target description in CLIP embedding (Patashnik al. 202T). The latent direction found on one
corresponndng author

---

Published as a conference paper at ICLR 2023
Image manifold
15h
Happy + +hh
Happy
Happy
A4N
+
2000
X
Orginal
-3A
(a) Image guidance
Classifier guidance
(c) DiffusionCLIP
(d) Ours
Manipulation approaches for diffusion models. (a) Image guidance suffers ambiguity while controlling the generative process. (b) Classifier guidance requires an classifier, is hardly editable, degrades quality, or alters the content. (c) DiffusionCI.IP requires fine-tuning the whole model. (d) Our method discovers semantic latent space of a frozen diffusion model.
Figure 1:
image leads to the same manipulation of other images. However, given real image, finding its exact latent vector often challenging and produces unexpected appearance changes.
It would allow admirable image editing if the diffusion models with the nearly perfect inversion property have such a semantic latent space. Preechakul et al. (2022) introduces an additional input to the reverse diffusion process: latent vector from original image embedded by an extra encoder. This latent vector contains the semantics to condition the process. However, it requires training from scratch and does not match with pretrained diffusion models.
In this paper, we propose an asymmetric reverse process (Asyrp) which discovers the semantic latent space of a frozen diffusion model such that modifications in the space edits attributes of the original images. Our semantic latent space, named h-space, has the properties necessary for editing applica- tions follows. The same shift in this space results in the same attribute change in all images. Linear changes in this space lead to linear changes in attributes. The changes do not degrade the quality of the resulting images. The changes throughout the timesteps are almost identical to each other for desired attribute change. Figure d(d) illustrates some of these properties and $ 5.3 provides detailed analyses. To the best of our knowledge, the frozen pretrained diffusion models. Spoiler alert: our semantic latent space is different from the intermediate latent variables the diffusion process. Moreover, we introduce a principled design of the generative process for versatile editing and quality boosting by quantifiable measures: edit- ing strength of an interval and quality deficiency at a timestep. Extensive experiments demonstrate that our method is generally applicable to various architectures (DDPM++, iDDPM, and ADM) and datasets (CelebA-HQ, AFHQ-dog, LSUN-church, bedooo,, and METFACES).
the first attempt discover the semantic latent space in
BACKGROUND
We briefly describe essential backgrounds. The rest of the related work is deferred Appendix A
2.1 DENOISING DIFFUSION PROBABILITY MODEL (DDPM)
DDPM a latent variable model that learns a data distribution by denoising noisy images (Ho et al. 2020). The forward process diffuses the data samples through Gaussian transitions parameterized with a Markov process:
g (xt Ft-1) =N a0 / V B,x, 3.1) = N (Var at a- 81-1,
at or- )
(I)
where (B.TT1 is the variance schedule and IT - (1 - -3). Then the reverse process becomes
Pa (Zo.T) == p (xT) [ Pe (x,-1 xx)), starting from IT ~N(O.)) with noise predictor e.:
Be
at-1 V1 3, [
ol (xx)
(2)
V
at
where 2, ~N( and of is variance of the reverse process which is set of B, by DDPM.
2

---

Published as a conference paper at ICLR 2023
2.2 DENOISING DIFFUSION IMPLICIT MODEL (DDIM)
DDIM redefines Eq. 1 as dr ( Fe, vo) = N 04 10 + V 04-1 0,
o
I)
Y
which is a non-Mann Man-Mankon process (Song et al. 2020a Accordingly, the reverse process becomes
Tt vI arej (xx)
x--1 vor-1
0h1 of ei (ate)
vat
"direction pointing to zzy"
"predicted uEgg
where o, = -1 a²) /
04/01-1 When 1j = for all t, it becomes DDPM. As the process becomes deterministic and guarantees nearly perfect inversion.
7
2.3 IMAGE MANIPULATION WITH CLIP
CLIP learns multimodal embeddings with an image encoder F. and a text encoder Ery whose sim- ilarity indicates semantic similarity between images and texts (Radford al. [02T]. Compared to directly minimizing the cosine distance between the edited image and the target description (Patash-
nik al. 2021b, directional loss with cosine distance achieves homogeneous editing without mode
collapse (Gal et al. 2021):
Al AT A
Edirection
(xdi larget, source gource) =
(4)
AT
Ey (ouou)))) for edited image x ndis target description yiarget, original image "ource, and source description "ouure. We use the prompts 'smiling face' and 'face" as the target and source descriptions for facial attribute smi ling.
where AT = Er (y Ex (y)ource) and AT = Ey (x
DISCOVERING SEMANTIC LATENT SPACE IN DIFFUSION MODELS
This section explains why naive approaches do not work and proposes new controllable reverse process. Then we describe the techniques for controlling the generative process. Throughout this paper, we abbreviated version of Eq. (3):
(5) where (c (xx)) denotes the predicted zy and De (x:)) denotes the direction pointing xt. We omit o,z, for brevity, except when 1) * O. We further abbreviate P,(e, (ef(() (a:)) as P, and (e((()))) as D, when the context clearly specifies the arguments.
2t-1 va--1 Pa (e() + (x)) + t
3.1 PROBLEM
We aim allow semantic latent manipulation of images Ze generated from Vy given pretrained and frozen diffusion model. The casiest idea to manipulate 2g is simply updating #T to optimize the directional CL.IP loss given text prompts with Eq. However, it leads to distorted images incorrect manipulation (Kim & Ye 2021).
An alternative approach is to shift the noise ef predicted by the network at each sampling step- However, it does not achieve manipulating In because the intermediate changes in P, and D, cancel each other resulting in the same po (No.T), similarly to destructive interference.
Theorem 1. Let ef be a predicted noise during the original reverse process at t and El be its shifted counterpart. Then, Ar, _ It-1 negligible where Ιt-1 vai-1 P.(Ef + (i(E((z))) Lee. the shifted terms of ج. in P, and D, destruct each other in the reverse process.
Appendix Clroves above theorem. Figure 13 b- shows that wo is almost identical Zn-
3.2 ASYMMETRIC REVERSE PROCESS
In order to break the interference, we propose a new controllable reverse process with asymmetry:
ft-1 von-1 (E() Di(e((at)))
3

---

Published as a conference paper at ICLR 2023
DDIM process
Po(s)) D. Er P((F))
image
10M
S/+
Smiling
21-1
Asymmetric reverse prooess
20
Generative process of Asyrp. The green box on the left illustrates Asyrp which only alters P, while preserving D, shared by DDIM. The right describes that Asyrp modifies the original reverse process toward the target attribute reflecting the change in h-space.
Figure 2:
i.e. modify only P, by shifting c, to EÂ while preserving D,. Intuitively, it modifies the original reverse process according to A = 2. cf while does not alter the direction toward æ; so that - follows the original flow D, at each sampling step. Figure Siguureiillssratess the above intuition.
As in Avrahami et al. (2022). we use the modified pedit and the original Prource as visual inputs for the directional CLIP loss Eq. (4), and regularize the difference between the modified Pedit and the original P Prouree We find Ae arg min se Et ((t) where
C Edirection (Peilit ,y" .y"' Paouree youurce) + Areoon Pedit
Paource
Although Ae indeed renders the attribute in the æi e-space lacks the necessary properties of the semantic latent space in diffusion models that will be described in the following.
3.3 h-space
Note that ef is implemented as U-Net in all state-of-the-art diffusion models. We choose its bottle- neck, the deepest feature maps h4 control c. By design, h, has smaller spatial resolutions and high-level semantics than ef. Accordingly, the sampling equation becomes
Vai-1 (x, - De(e, + diZt,
(8)
where of (ax, Ah;)) adds Ad, to the original feature maps h. The Ah, minimizing the same loss in Eq. 7 with Pr(e; (w//As))) instead P, (E (zx)) successfully manipulates the attributes.
We observe that h-space in Asyrp has the following properties that others do not have.
The same Ah leads to the same effect on different samples.
Linearly scaling Ah controls the magnitude of attribute change, even with negative scales.
Adding multiple Ah manipulates the corresponding multiple attributes simultaneously.
Ah preserves the quality of the resulting images without degradation.
Ah; is roughly consistent across different timesteps t.
The above properties are demonstrated thoroughly in 5.3 Appendix D.3 provides details of h-space
and suboptimal results from alternative choices.
3.4 IMPLICIT NEURAL DIRECTIONS
Although Ah succeeds in manipulating images, directly optimizing Ah, multiple timesteps quires many iterations training with a carefully chosen learning rate and its scheduling. Instead, define implicit function f. (h+) which produces Ah; for given h4 and t. fo is implemented as a small neural network with two x 1 convolutions concatenating timestep See Appendix for the details. Accordingly, optimize the same loss in Eq. with pedit (c, (x, f.)).
Learning f. more robust to learning rate settings and converges faster than learning every Ah, In addition, as learns implicit function for given timesteps and bottleneck features, it generalizes

---

Published as a conference paper at ICLR 2023
aditinn
quality deficicncy
Uboost
14
strengit
LPIPS(a.)
1.2
LPIPS(z.))
1.0
0.8
066
APH2
P.
LPIPS A
ARHQ
Ji
CáihA
P
(sy) 1000
800
600 400
200
0 (Vo)
Figure 3: Intuition for choosing the intervals for editing and quality boosting. We choose the intervals by quantifying two measures (top left inset). The editing strength of an interval [T, f mea- sures perceptual difference from until We set [T. to the interval with the smallest editing strength that synthesizes P, close to i..., LPIPS(( P.) 0.33. Editing flexibility of an interval [f. 0] measures the potential amount of changes after t. Quality deficiency at t measures the amount of noise in æ;. We set (t, 0] to handle large quality deficiency (i.e., LPIPS(x, x;) 1.2) with small editing flexibility.
unseen timesteps and bottleneck features. The generalization allows to borrow the accelerated
training scheme of DDIM defined on subsequence (ær,, |eE[, where {T:} is subsequence of ?"] and S < T. Then, we can use the generative process with a custom subsequence } with length S < T through normalization: Ah; fz (h:) SS/S. It preserves the amount of Σ Ah., Ah; S S. Therefore,
f. trained on any subsequence for any length of the genera- tive process. See Appendix F for details. We use fc to get Ah; for all experiments except Figure 6
GENERATIVE PROCESS DESIGN
This section describes the entire editing process, which consists of three phases: editing with Asyrp, traditional denoising, and quality boosting. We design formulas to determine the length of each phase with quantifiable measures.
4.1 EDITING PROCESS WITH ASYRP
Diffusion models generate the high-level context in the early stage and imperceptible fine details in the later stage (Choi et al., 20022.. Likewise, we modify the generative process in the early stage to achieve semantic changes. We refer the early stage as the editing interval [T,, lodit].
LPIPS( (c. Pr) and (z, P,) calculate the perceptual distance between the original image and the predicted image at time steps T and respectively. Intuitively, the high-level content is already determined by the predicted terms at the respective timesteps and LPIPS measures the remaining component be edited through the remaining reverse process. Consequently, we define editing strength of an interval [T,
= =PIPS P LPIPS(x,
indicating the perceptual change from timestep T to in the original generative process. Figure illustrates I.P.PS(x..) for P, and æ, with examples and the inset depicts editing strength. The shorter editing interval has the lower & and the longer editing interval brings more changes to the resulting images. We seek the shortest editing interval which will bring enough distinguishable changes in the images in general. We empirically find that feda with LPIPS (z, P ttmar:) = 0.33 builds the shortest editing interval with enough editing strength as Planr has nearly all visual attributes in x.
However, some attributes require more visual changes than others, e.g., pixar > smile. For such attributes, increase the editing strength by 5 0.33d Frr (Usource), (ytarge))) where
5

---

Published as a conference paper at ICLR 2023
Er(-) produces CLIP text embedding, V() denotes the descriptions, and d(.,) computes the cosine distance between the arguments. Choosing ledit with LPIPS(xr, Preas) = 0.33 - 6 expands the editing interval suitable length. consistently produces good results in various settings. The supporting experiments shown in Appendix
4.2 QUALITY BOOSTING WITH STOCHASTIC NOISE INJECTION
Although DDIM achieves nearly perfect inversion by removing stochasticity (7 = 0), Karras et al. 2022) demonstrate that stochasticity improves image quality. Likewise, we inject stochastic noise in the boosting interval [bboost 0j.
Though the longer boosting interval would achieve higher quality, boosting over excessively long intervals would modify the content. Hence, we want to determine the shortest interval that shows enough quality boosting to guarantee minimal change in the content. We consider the noise in the image as the capacity for the quality boosting and define quality deficiency at t: 7/- = (P xx) indicating the amount of noise in æt compared to the original image. We æ; instead of P, because we consider the actual image rather than the semantics. Figure 3 inset depicts editing flexibility and quality deficiency. We empirically find that boost with Yhowe 1.2 achieves quality boosting with minimal content change. We confirmed that the editing strength of the intervals [Eboost, 0) is guaranteed be less than 0.25. In Figure after boostt LPIPSS(2,, æx,) sharply drops in the original generative process while LPIPS(x:. (x, Pt) changes little. Note that most of the quality degradation of the resulting images is caused by DDIM reverse process, not by Asyrp. We use this quality boosting for all experiments except ablation in Appendix H
4.3 OVERALL PROCESS OF IMAGE EDITING
Using fedit and boost determined by the above formulas, we modify the generative process of DDIM with
N (N (yar- a P (r(e (mn (()) + D,, of 1= Va P.(e, + o -
if 2 1 ledit
Pg (It-1 220)
lodit > 1 2 boost
a P:(e, + DD, n
boost >
The visual overview and comprehensive algorithms of the entire process are in Appendix
EXPERIMENTS
In this section, we show the effectiveness of semantic latent editing in ht-sace with Asyrp various attributes, datasets and architectures in $ $5.1] Moreover, we provide quantitative results including user study in & [.2) Lastly, we provide detailed analyses for the properties of the semantic latent space on h-space and alternatives
Implementation details. We implement our method on various settings: CelebA-HQ Karras al. 2018) and
d Yu et al., 2015) DDPM+ (Song et al. 2020b) (Meng et al. 202T): AFHQ dogg-dog (Choi et al. (0200 on iDDPM (Nichol & Dhariwal 2021) ): and MET-
FACES (Krras et al., (2020) on ADM with pe-ghttgting (Dhariwal & Nichol 2021) (Choi et al., 2022). Please note that all models are official pretrained checkpoints and are kept frozen. De- tailed settings including the coefficients for ACLIP and Ayecons and source/target descriptions can be found in Appendix [.]] We train ft with S 40 for 1 epoch using 1000 samples. The real samples are randomly chosen from each dataset for in-domain-ll like attributes,
attributes. For out-of-domain- randomly draw 1,000 latent variables æ (O. I). Details
described in Appendix J.2] Training takes about 20 minutes with three RTX 3090 GPUs. All the images in the figures are not used for training. We set = 1, 000 for inference. The code is available at http:://ghhbb.oom///oommmt/ty//p
5.1 VERSATILITY OF f-spuce WITH ASYRP
Figure shows the effectiveness our method on various datasets and any existing U-Net based architectures. Our method can synthesize the attributes that are not even included in the training

---

Published as a conference paper at ICLR 2023
Real image
Recon.
Smiling
Sad
Angry
Man Woman
Y Young
Curly bair
Real image
Recon.
Department
Factory
Gothic
bricks
Temple
Wooden
HILD
TIEE
Real image
Smiling
Disgusted
Real Image
Sleepy
Happy
Real image
Hotel
Princess
Figure 4: Editing results of Asyrp on various datasets. conduct experiments on CelebA-HQ. LSUN-church, METFACES, AFHQ-dog, and LSUN-bedroom.
Real image
Nicolas Cage
Pixar
Modigliani
Neanderthal
Frida
Figure 5: Editing results of Asyrp for unseen domains in CelebA-HQ dataset.
dataset, such as church + [department, factory, and temple]. Even for dogs, our method synthesizes smi miiling Poodle and Yorkshire, the species that barely smile in the dataset. Figure 5 provides results for changing human faces to different identities, painting styles, and ancient pr- mates. More result be found in Appendix N Versatility of our method is surprising because we do not alter the models but only shift the bottleneck feature maps in h-space with Asyrp during inference.

---

Published as a conference paper at ICLR 2023
CelehA-HQ in-domain
CelehA-HQ unseen-domain
LSUN-church
quality
ultribute
overall
quality
attribute
overall
quality
atiribute
divensity
overall
Asyrp (ours) DiffusionCLIP
98.36%
88.13%
94.9%%
71.6%
59.84%
63.13%
73.19%
7181% 28.19%
87.50%
76.81%
1.64%
1188%
5.08%
28.44%
40.16%
36.88%
26.81%
12.50%
23.19%
Table I: User study with 80 participants. The details are described in
K.
Real image
A/
Ac
Real image
Alu
A
Real image
A/t
Art
(a) Result of optimization
(b) Result of adaptation to other images
Figure 6: Optimization for smil amiling on h-space and e-space. (a) Optimizing Ah, for sample with smi 1 ling results in natural editing while the change due to optimizing Ae, is relatively small. (b) Applying Ahe from (a) other samples yields the same attribute change while Ae, distorts the images. The result of de; is the best sample we could find.
Real image
Smiling (untrained)
Smiling (trained)
Figure 7: Linearity of htspace. Linear interpolation and extrapolation h-space lead to grad- ual changes even to the unseen intensity and directions. Right side shows the interpolation results by positive scaling of A Left side shows the extrapolation results by negative scaling of Ahe A smiling Note that we did not train f. for the negative direction.
5.2 QUANTITATIVE COMPARISON
Considering that our method can be combined with various diffusion models without finetuning, do not find such versatile competitor. Nonetheless, compare Asyrp against DiffusionCLIP using the official code that edits the real images by finetuning the whole model. We asked 80 partic- ipants choose the images with better quality, natural attribute change, and overall preference for given total of 40 sets of original images, ours, and DiffusionCLIP. Table that Asyrp outper- forms DiffusionCLIP in the all perspectives including the attributes unseen in the training dataset. We list the settings for fair comparison including the questions and example images in Appendix K.I See K.2 for more evaluation metries: segmentation consistency (SC) and directional CLIP similarity(S))
5.3
ANALYSIS ON h-space
We provide detailed analyses to validate the properties of semantic latent space for diffusion models: homogeneity, linearity, robustness, and consistency across timesteps.
Homogeneity. Figure illustrates homogeneity of h-space compared to e-space. One Ah, opti- mized for an image results the same attribute change other input images. On the other hand, one Ae; optimized for an image distorts other input images. In Figure I0 applying Ahmean
N Σ Ahi produces almost identical results where i indicates indices of N 20 random samples.
Linearity. In Figure 7 we observe that linearly scaling Ah reflects the amount of change in the visual attributes. Surprisingly, it generalizes to negative scales that are not seen during train- ing. Moreover, Figure g shows that combinations of different Ah's yield their combined semantic changes in the resulting images. Appendix N.2 provides mixed interpolation between multiple at- tributes.
8

---

Published as a conference paper at ICLR 2023
Real image
Young
Sed
Neanderhal
N+S
N+Y
Y46
Y+64N
Figure 8:
Linear combination. Combining multiple Ahs leads to combined attribute changes.
Real image
(a) Ah
(b) A
(c) Ah (x3)
Figure 9: Ramdom manipulation on h-space and e-space. (a) Adding random noise with mag- nitude of Ah; shmiinng and random direction in h-space leads to realistic images with small changes. (b) Adding random noise with magnitude of ee; smillig and random direction in 6-space leads to severely distorted images. (c) Adding random noises with tripled magnitude produce diverse visual changes in realistic images.
Real image
Ah
A
Ahglobal
Real image
Ahr
A Almeun
A Ah Ahglobal
Young
Pixar
Figure 10: Consistency on h-space. Results of Ahe Ahtuean and Ahglobal are almost identical for in-domain samples. However, we choose Ah, over others to prevent unexpected small difference in the unseen domain. We provide more results in L.I
Robustness. Figure 9 compares the effect of adding random noise in h-space and e-space. The random noises are chosen to be the vectors with random directions and magnitude of the example Δh, and Ae; in Figure 6 on each space. Perturbation in h-space leads realistic images with minimal difference or some semantic changes. On the contrary, perturbation in e-space severely distorts the resulting images. See Appendix D.2 for more analyses.
Consistency across timesteps. Recall that Ah, for all samples are homogeneous and replacing them by their mean Ahtuaa yields similar results. Interestingly, in Figure| we observe that adding time-invariant Ahglobal T. Σ, Ahqusu instead of Ah, also yields similar results where T. denotes the length of the editing interval [T, ledit. Though we use Ah, to deliver the best quality and manipulation, using Ahmean or even Ahglobal with some compromise would be worth trying for simplicity. We report more detail about mean and global direction in Appendix L.]
6 CONCLUSION
We proposed generative process, Asyrp, which facilitates image editing in semantic latent space h-space for pretrained diffusion models. h-space has nice properties as in the latent space of GANs: homogeneity, linearity, robustness, and consistency across timesteps. The full editing process is designed to achieve versatile editing and high quality by measuring editing strength and quality deficiency at timesteps. We hope that our approach and detailed analyses help cultivate new paradigm of image editing in the semantic latent space of diffusion models. Combining previous finetuning or guidance techniques would be an interesting research direction.
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation Korea (NRF) grant funded by the Korea govermment (Ministry of Science and ICT) (No. 2021-0-00155)

---

Published as a conference paper at ICLR 2023
REFERENCES
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18208-18218, 2022.
Hila Chefer, Sagie Benaim, Roni Paiss, and Lior Wolf. Image-based clip-guided essence transfer. arXiv preprint arXiv:2110.12422 2021.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Con- ditioning method for denoising diffusion probabilistic models. arXiv preprini AXXi:2108..02938, 2021.
Jooyoung Choi, Jungbeom Lee, Chachun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11481, 2022.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1188-197, 2020.
Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. Style transformer: Unpaired text style transfer without disentangled latent representation. arXiv preprint arXiv:1905.055: 1905.05621, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan- nada: Clip- guided domain adaptation of image generators. arXiv preprint (xXix:2108.00940 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. Advances Neural Information Processing Systems, 33:9841-98501 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neurul Information Processing Systems, 33:6840-6851, 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 13:12104-12114, 2020.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint arXiv:2206.005: 2022.
Gwanghyun Kim and Jong Chul Ye. Diffusionclip: Text-guided image manipulation using diffusion models. 2021.
Diederik Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Cheng- Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and 2020.
Pattern Recognition, pp.
Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralaa, and Sanja Fidler. Edit- gan: High-precision semantic image editing. Advances in Neural Information Processing Systems, 34:16331-16345, 2021.
10

---

Published as a conference paper at ICLR 2023
Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. arXiv preprint arXiv:2112.05744, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.
Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1pp.1445 11461-11471, 2022.
Chenlin Meng, Yang Song, Jiaming Song. Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Im- age synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112. 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 3162-8171. PMLR, 2021.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 2085-2094, 2021.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakom. Dif- fusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10619-10629, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent difffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by eиror propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Vikash Sehwag, Caner Hazirbas, Albert Gordo, Firat Ozgenel, and Cristian Canton. Generating high fidelity data from low-density regions using diffusion models. In Proceedings the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 11492-1 11011, 2022.
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentan- gled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence, 2020.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 133456, 2020b.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-basee based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:1 1287-1 11302, 2021.

---

Published as a conference paper at ICLR 2023
Daniel Watson, William Chan, Jonathan Ho. and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality. In International Conference on Learn- ing Representations, 2022. httttp:://oeeeeeiieeeeeee net/ www.:eeee///tmiiiee PI juF BHEp
Zipeng Xu, Tianwei Lin, Hao Tang, Fu Li, Dongliang He. Nicu Sebe, Radu Timofte, Luc Van Gool, and Errui Ding. Predict, prevent, and evaluate: Disentangled text-driven image manipulation empowered by pre-trained vision-language model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18229-18222 18238, 2022.
Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilat- eral segmentation network for real-time semantic segmentation. Proceedings of the European conference on computer vision (ECCV), pp.
2018.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:/506.0355 2015.
Oğuz Kaan Yüksel, Enis Simsar, Ezgi Gülperi Er, and Pinar Yanardag. Latentelr: A contrastive learning approach for unsupervised discovery of interpretable directions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14263-14272, 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattemn recognition, pp. 633-641, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):302-33) 22122,1112222022 2019.
12

---

Published as a conference paper at ICLR 2023
Appendix
Table of Contents
A Related work
14
B More discussion
15
Proof Theorem
D Additional supports h-space with Asyrp
D.1 Random perturbation on e-space without Asym
17
D.2 Robustness and semanties in spsce and e-sace with Asyrp
17
D.3 Choice of h-xpace in U-Net
17
E
Implicit neural directions
17
F
Quality improvements by non-accelerated sampling with scaled Ah,
G Editing strength and editing flexibility
18
G.1
Editing strength and fedit
18
2
Editing flexibility and thoost
18
Quality boosting
19
Algorithm
23
Training details
23
J.1
Loss coefficients
23
2 Training with random sampling instead of the training datasets
24
Evaluation
24
User study
24
Segmentation consistency and directional CLIP similarity
25
L Directions
26
Global direction
26
Compare three methods
29
Random sampling
30
More samples
32
N.I ImageNet
32
N22 Multi-ina Multi-interpolation
32
3 More results on all datasets
32
13

---

Published as a conference paper at ICLR 2023
Real images
Smiling
Figure 11: Editing dog to smile. We provide high-resolution results of editing real images in the teaser and more.
A RELATED WORK
After Eiter/Soll-Diceeee et al. (2015), denoising diffusson probabilistic models (DDPMs) provide uni- versal approach for generative modeling (Ho et al.) 22200) On the other hand, [Sng et al. (2020b) suggests score-based model and unifies SDEs incorporating diffusion models with score- based mod- els. Subsequent works renovate diffusion models by focusing on architectures, scheduling, weight- ing, and fast sampling (Nichol & Dhariwal (202I). Karras al. (2022), Choi et al. (2022), Song et al. (2020a). Watson et al. (2022)). They mainly consider random generation rather than controlled generation.
In the meantime, Dhariwal & Nichol (2021) introduces classifier guidance not only improving the quality of images but also retrieving specific class images. Since it can apply any guidance, its variants have emerged (Sehwag et al. (2022), AArahami et al.) (2022), Liu al. (2021). NNichol al. (2011). However requires a noise depended classifier (or any the-the-shelf models) and additional cost compute gradients for the guidance during its sampling process. The other works try to control the generative process using image-space guidance (Choi et al. (2021), Meng et al. (2021). Lugmayr et al. (2022). Avrahami
(2022)). They manipulate resulting images by matching noisy images with target images during the reverse process. Still, it is hard to expect delicate control of the reverse process from the image guidance. Furthermore, |Preechakul (rreemoare
al. (2022) introduces an extra encoder which encodes the semantic features of real image in order to condition the generative process. Although the semantics allow one to control diffusion models, requires additional training from scratch with the encoder and inherently can not use the other pretrained diffusion models.
For controllability, Rombach et al. (2022) and Vahdat et al. (202T) apply another approach which adapts VAE (Kingma & Welling 2013) and autoencoder (Rumelhart et al. 1985) to diffusion mod- els. In spite their great success in editing, their diffusion models learn the distribution of the learned embeddings in VAE or autoencoder, not the images. Kim & Yel (2021) proposes another strategy: fine-tuning a whole diffusion model for image editing. It shows valid performance but requires each fine-tuned model corresponding each attribute.
In comparison, Asyrp enables outstanding manipulation without high computation, specifically de- signed architectures, or fine-tuning whole models.
Meanwhile, generative adversarial network Goodfellow et al. (2020) address their latent space for image editing (Ling al. (2021), Härkönen et al. (2020). Chefer al. (2021). Shen al. 2020
14

---

Published as a conference paper at ICLR 2023
Yüksel et al. (2021). Patashnik et al. (2021). Gal et al. (2021). Dai al. (2019). Xu et al.) (2022) However they have to conduct 'inversion" to their latent space for real image editing and 'GAN
inversion' is often challenging and produces unexpected appearance changes.
On the contrary, Asyrp enables to use latent space real images by nearly perfect easy inversion of DDIM.
B MORE DISCUSSION
In this section, we discuss the pros and cons of diffussion-moddllbased and GAN-based methods. And we provide guidelines for further improvements.
GAN-based latent manipulation methods Patashnik et al. (021)) Gal et al. (202 rrquiree careful inversion from real images latent codes for real image editing. the contrary, our proposed
method based on diffusion models has a powerful advantage; the sophisticated inversion method is not necessary. This means that we can obtain the latent code of arbitrary real image even the image is not in the trained domain. On the other hand, several inversion methods have been proposed for GANs to obtain the latent of the real image, and the corresponding latent manipulation method should be considered for each inversion method. For example, it is difficult to apply the method of editing w space to the method inversion using w+ space.
However, GANs have the advantage of fast sampling. In addition, diffusion models have relatively slow sampling time. Additionally, we have be aware of the time steps of diffusion models, which is still less well known.
The advantage being free from Inversion provides the following milestones. The manipulation in the latent of the diffusion models is the same the editing in real images. It can be expanded to segmentation, clustering, classification, etc. h-space for real-world images.
It would be an interesting research direction to employ previous techniques. Our method can be used in conjunction with gradient guidance methods. Although we do not focus on random sampling, ours works effectively for sampling with stochastic. (See M It may bring more diverse methods to steer diffusion models.
h-space in the latent diffusion models such stable diffusion, is another interesting research di- rection. The main contribution of our paper is only modifying P. while preserving D,, and can be adapted with latent diffusion models. However, since the latent meaning may be different due to structural differences, research on this is needed.
Furthermore, all of the properties of h-space according to the time step has been fully discussed so far. Research on them can be expected to expand further.
Limitations. Editing with Asyrp seldom yields changes in overall style or peripheral objects but edits attributes of the main object. Style transfer using frozen diffusion models is our future work.
Societal impact / Ethics statement. Techniques for high-quality image manipulation such as Asyrp should be accompanied by social and/or technical solutions to prevent abuse. We acknowledge the potential ethical implications that may arise from the use of image manipulation technique, Asyrp. We advocate for the development and implementation of social and technical solutions to prevent potential abuses such spreading disinformation or propaganda. We are committed to en- suring fairness and non non-discrimination, legal compliance, and research integrity in our work.
C PROOF THEOREM
Proof of Theorem Let of be a predicted noise during the original reverse process at and ès be its shifted counterpart. Then, Ax; pr-1 #;-1 is negligible where _.-1 var- P,(e) (ef(( (x:)) (
D, (E{(()))
15

---

Published as a conference paper at ICLR 2023
Pt((t)
+Dr(a)
Pt(et)
C
+De(F))
Et
1r-1
Figure 12: Illustration of Theorem 1. Upper blue line describes applying noise é e, + Der to produce Pa(e)) = the shifted predicted øg- However, the shift due to Ae, is canceled out by the shift in (&,))) due to d As results, applying Ae, both on P, and D, brings identical outputs to the original.
Define (x:)) = (xxe) + Ae,, (B,)=11 = (31 = Smin:...,,87 Br = 3mux ), and oz = I (.). Note that Aoux is defined as a small value (e.g., Aaux = 0.001) and { are defined by a decreasing schedule from 87 Amnx to 31 Bmin 0 (e.g., Amin 0.00001). Then,
H 0 (((E(()))) + (.(e
(10)
- I ai (cx + AGi))
+ VI a-- (e (c))(()) (x,) + Der)
/az-1
(11)
Vat
v0t-IvI- at
00 (x))) + De(e:(at)))
De + V - 11-04-1 AAE
(12)
V
VI- aq
= - +
-
V1 3r
+ V I - a1-1
DE:
(13)
vI at
(1 Ba) 1 Br
= xz-14 +
.
(14)
VI 3r
VI- 3r
VI- Br
Ft-1
II (1 11-88 B.)
(15)
VI 3r
VI- 04 Dj VI ai
Az = 2t-1
Ae; is negligible
:: Br < Bmaz
V1 11- 3r
(16)
16

---

Published as a conference paper at ICLR 2023
D ADDITIONAL SUPPORTS FOR h-space WITH ASYRP
D.1 RANDOM PERTURBATION ON c-space WITHOUT ASYRP
In 3.1 we argue that if both P. and D, are shifted, we can not manipulate æo. Figure 13 do not observe the noticeable difference between (a) and (b) which are the result of the original reverse process of DDIM and the one with shifting both terms, respectively.
D.2 ROBUSTNESS AND SEMANTICS IN h-space AND e-space WITH ASYRP
In 5 3.2 also argue that h-space is more robust than 6-space with Asyrp. In Figure 13 observe that small random noise 2 ~ (NO,, I) c-space degrades the resulting image without semantic changes (c) and much larger random noise in h-space yields random semantic changes without severe artifacts (d).
Note that diffusion models designed as latent variable models with learned Gaussian transitions and the reverse process should also be close to Gaussian. Based on the assumption, we manage of follow the original Gaussian distribution as follows. Adding
(0, o1) ef expands the distribution of the predicted noise and may produce distorted images. To preserve the distribution, we scale εy (ef + 21/v12 + o², Still, the resulting images are almost identical compared to the original images where e e + as shown Figure 13) It no wonder that the scaling does not improve the distorted results since the additive random noise disturbs the denoising operation of the predicted random noise.
Zo where
L1 distance between and Z0
ZO
0.03
0.4
distancc
10?
E-space hespace
10P
Real image
(a)
(b)
(c)
(d)
10-2
(e)
10-1 Variance of 2 (a?)
10
Figure 13: (a) The reconstructed image by the original DDIM inversion process which is almost indistinguishable to the real image. (b) The result from adding random noise & both on P, and D,- (a) looks identical to (b). The insets of (b, c) depict the full SSIM image from (a). It shows that simply shifting ee without Asyrp does not affect the result. (c) Adding ~ N to e-space with Asyrp easily degrades image with little semantic change. (d) Adding A h-space with Asyrp yields random semantic change without image degradation. (e) Correlation between image degradation and noise strength in the two spaces.
D.3 CHOICE OF h-space IN U-NET
As shown in Figure 14 there are many other candidates for h-space in the architecture. Among the layers, choose the 8th layer, the bridge of the U-Net based architecture. The layer is not influ- enced by any skip connection, has the smallest spatial dimension with compressed information, and is located just before the upsampling blocks. Thus, assume that could possibly be considered as the most suitable latent embedding. To confirm the assumption, we train ft on the other layers. The results shown in Figure We carefully tuned the training hyperparameterss (ACLIP and Arecon) for fair comparison. The 1st to the 6th layers hardly bring visible changes. The 7th and 9th layers bring not only the desired changes but also difficulty in finding optimal hyperparameters. After the 9th layer, the results bear severe artifacts.
E IMPLICIT NEURAL DIRECTIONS
Figure 16 illlsstttse the neural implicit function ft. It has only two 1xl convolution layers with 512 channels, Note that we haven't explored the network architecture much.
17

---

Published as a conference paper at ICLR 2023
1
2
14 15
E (x t)
2567
16 XF1222
#Px53x
#Fx512
16'x512
32Px25H
6ePx225
64Px256
126 x128
120% 120
x128
258
t
Sinusoidal encoding
Res block
Concatenation
Res block Att black
Positional embedding
Conv3x3
Figure 14: Location of h-space. The U-Net architecture of diffusion models outputs 256 x 256 images. Each layer is indexed with number along the operating sequence of the model. The 8th layer is our h-space which is not directly influenced by a skip connection.
F QUALITY IMPROVEMENTS BY NON-ACCHLERATED SAMPLING WITH SCALED Ahe
Figure 17 shows the quality improvements by con-acceleetted sampling with scaled Ah; described in [..4 Even with different number of inference steps, we observe similar changes an attribute if we preserve the sum of Ah, This scaling technique allows non-accelerated sampling with 1000 steps for the models trained by accelerated training with 40 steps. Using non-accelerated sampling with scaled Ah, leads the same magnitude of manipulation and higher-quality images. In our experiments, it takes about 1.5 seconds to sample for 40 steps and 40 seconds for 1000 steps.
G EDITING STRENGTH AND EDITING FLEXIBILITY
G.1 EDITING STRENGTH AND tedit
shows the results according to fedit- If tedit is too high, the length of the editing process becomes too short resulting in insufficient changes. On the contrary, too low tedit causes excessively unnecessary manipulation from the long editing process.
Figure 18
We observe that fedit is one of the important byperparameters. We argue that the formula for choos- ing fedit using editing strength is reasonable because it applies to all five different datasets despite its sensitivity, even though the choice LPIPS = 0.33 is empirical. We provide ablation of thresholds in Figure 22
Additionally, these results imply why we need to use sufficiently low fedit in the unseen domains. Editing interval with insufficient editing strength struggles escape from the training domain.
G.2 EDITING FLEXIBILITY AND Eboot
Table shows the prompts, fedit, and tboost Intuitively, the attributes with larger visual changes have smaller cosine similarity and require longer editing interval. Figure [9 shows the average T.PIPS(æ,. P.) and LPIPS( 2x) of 100 samples all datasets. Note that lodit and boost dif-
18

---

Published as a conference paper at ICLR 2023
Original
I Layer 1-6
Layer
Layer (i-space)
Layer
Layer 10
Layer 11
Layer 12
Layer 13
Layer 14
Layer 15
Layer 16
Figure 15: Exhaustive enumeration over the choices for semantic latent space. show result of the training data. We observe that the eighth layer (h-space) of the U-net suits the best for the semantic latent space.
Ah
1x1 DIAY
group
siiish
4
512
lneer
(ha)
Figure 16: Illustration ft, We use group norm and swish following DDPM.
fer across datasets and attributes, and they are chosen by the formulas We provide ablation of boost in Figure 23
H QUALITY BOOSTING
We validate the effectiveness of our quality boosting ($ 4.2 the original DDIM process and in
Asyrp.
19

---

Published as a conference paper at ICLR 2023
Smiling with
Real image
40 steps
1000 steps
CABGIT
s
Figure 17: Quality improvements by non-accelerated sampling with scaled Ah,. We observe quality improvements by non-accelerated (1000-step) sampling with scaled Ah, from accelerated (40-step) training described in 3.4 Please zoom in for detailed comparison.
fedit 900 fadit 800 tesiit 700 fede 600 taxi 500 tedkt
400 feait 300 tedit 200 fedit 100
Figure 18: Importance of choosing proper fedit- We explore various fodit with smil smiling. Too short editing interval struggles manipulate attributes. Excessive editing strength results in de- graded images.
20

---

Published as a conference paper at ICLR 2023
Dataset
trg_Ixt
CLIP similarity
Addt
buuee
Person
Young person
0.905
515
167
Face
Smiling face
0.899
513
Face
Sad face
0.894
513
167
Face
Angry face
0.892
512
167
Face
Tanned face
0.886
512
167
Face
Disgusted face
0.880
511
167
Person
Person with makeup
0.875
509
Human
Zombie
0.868
506
Person
Person with bald head
0.861
505
167
Person
Person with curly hair
0.835
499
167
CelebA-HQ
Human
Neanderthal
0.802
490
167
Person
Mark Zuckerberg
0.797
489
167
Person
Nicolas Cage
0.710
461
167
Human
Painting the style of Pixar
0.667
446
Photo
Painting in Modigliani style
0.565
403
Photo
Self-portrait by Frida Kahlo
0.443
321
167
Church
Gothic Church
0.912
371
293
Church
Temple
0.898
367
293
Church
Department
0.841
349
293
Church
Wooden House
0.793
333
293
LSUN-church
Church
Ancient traditional Asian tower
0.784
330
293
Church
Red brick wall Church
0.774
326
293
Church
Factory
0.702
301
293
Bedroom
Princess Bedroom
0.912
370
221
LSUN-bedroom
Bedroom
Hotel Bedroom
0.917
371
221
Dog
Happy Dog
0.883
430
167
Dog
Sleepy Dog
0.866
422
167
Dog
Angry Dog
0.860
419
167
AFHQ
Dog
Wolf
0.850
412
167
Dog
Yorkshire Terrier
0.690
302
167
Painting person
Painting Sad person
0.921
330
Painting of a person
Painting of Smiling person
0.908
320
METFACES
Painting of a person
Painting of a Disgusted person
0.879
291
180
Table 2: Prompts, CLIP similarity, fodie, and boost for all attributes in the experiments.
14
LPIPS(r,) - 1.2
LPIPS
0.
AFHQ
AFHQ
LSUN-BaC P.
Belk-Bedooom
CelebA P.
LPIPS(a.)) 0.33
CelebA Fr
LSUN-Church P.
0.
LSUN-Church d;
METFACES P.
METFACES
2000
800
Figure 19: Average LPIPS(x. P.) and LPIPS(x,,, xz) of 100 samples on all datasets.
Figure 20 shows the effect of quality boosting with the original DDIM reverse process. Although the reverse process of DDIM has nearly-perfect inversion property, we observe some noise by zooming in. Our quality boosting improves the quality of sample and concurrently keeps nearly
21

---

Published as a conference paper at ICLR 2023
Real image
w/o Quality boosting
w/ Quality boosting
Figure 20: Ablation study of quality boosting the original DDIM process without Asyrp. Our quality boosting enhances fine details and prevents images from being noisy in the original DDIM process. Please zoom in for detailed comparison.
Smiling
Real image
w/o Quality boosting
w/ Quality boosting
Figure 21: Ablation study of quality boosting with Asyrp. Our quality boosting enhances fine details and prevents images from being noisy. Note that the source of degradation is DDIM process, not Asyrp. confirmed in Figure 20 Please zoom in for detailed comparison.
22

---

Published as a conference paper at ICLR 2023
LPIPS(R, Proce
Real image
0.1
0.2
0.3
0.33
0.36
0.4
0.5
Pixar
Young
Smiling
Figure 22: Analyzing the effect of the hyperparameters The figure shows that the calculated parameter works effectively as the maximum boundary editing while maintaining the quality of the image.
Thoost
Real image
0.5
0.8
1.0
1.2
1.3
1.35
1.38
Pixar
Young
Smiling
Figure 23: Analyzing the effect of the hyperparameters We observe that the quality of results has robustness to "boou except for too large "oooott
perfect inversion property. We observe that tooest is not sensitive, but the larger interval brings the less preservation.
Figure 21 shows quality improvements by our quality boosting in Asyrp.
I
ALGORITHM
Figure 24 illustrates generative process. Algorithm I dedcrierri algorithm of Asyrp, respectively.
describe training algorithm and inference
TRAINING DETAILS
J.1 LOSS CORFFICIENTS
Table reports loss coefficients for each attribute. ACLIPS near 0.8 are suitable for most in-domain attributes. For unseen domains, higher coefficient leads to more noticeable changes. Note that we Areron CLIP similarity 3 which reduces II loss when attribute needs lot of change.
23

---

Published as a conference paper at ICLR 2023
TT
Mhoost
Ilboost
T0
Real image
X
X
X
c
atdy Smiling
24
Prozen U. Not
Semantic encoder
Adding AA
(a) Editing
(b) Denoising
(c) Quality boosting
Figure 24: Overview of our generative process.
J.2 TRAINING WITH RANDOM SAMPLING INSTEAD OF THE TRAINING DATASETS
Apparently, for training, inverting real-imager can be replaced by random sampling. refers to using ær ~ (0, I) instead of æy g II q (x; xx-1) where Wo Pdata (x). It allows to train Asyrp only with the pretrained network and without extra dataset. Using random samples has tradeoff between preservation of contents and possible amount in editing. It take advantages when a target attribute requires large amount changes. We assume that the inversion of real-image is in the long tail of Gaussian distribution because of realistic background or detailed clothes. On the other hand, random noise is considered to be closer to the mean of the normal, so it is easier to find directions. can easily bring larger changes but also easily alter the contents. On the contrary, training with inversion shows the opposite property.
We train with random sampling for attributes whose identity preservation is not important to take advantage of these properties. Thee rightmost column in Table shows the choices.
K
EVALUATION
K.
USER STUDY
We conduct user study to compare the performance of Asyrp and DiffusionCLIP (Kim & Ye 202T on Celeba-HQ (Liu et al., 2015) and LSUN-church (Yu et al. 2013). We use official checkpoints provided by DiffusionCLIP except for some facial attributes whose checkpoint do not exist. We tried our best to tune their hyperparameters following the manual for fair comparison. Example
images are shown in Figure 25 27
We use smi i ng and sad for in-domain CelebA-HQ attributes, Pixar and Neanderthal for unseen-domain CelebA-HQ attributes, and department store, ancient, and wooden for LSUN-church.
In unseen domain and Lsun-church, we use official checkpoints provided by DiffusionCLIP. We also randomly select images for each CelebA-HQ attribute and 12 images for each LSUN-church attribute.
We observe that DiffusionCLIP works better in changing the holistic style of images. At the same time, it is short of the ability to bring semantic changes and suffers noisy results and lack of diversity. The problems would be caused by fine-tuning the whole diffusion model.
We use the following questions for the survey. 1) Quality: Which image quality do you think is better? (clear and less noisy) 2) Attribute: Which image do you think is "Attributece.g. Smiling) naturally"? 3) Overall: Which image do you think is better considering the above evaluation criteria?
24

---

Published as a conference paper at ICLR 2023
Algorithm 1: Editing(Inference)
Input: cnnputt image), { (M Neural implicit functions for M attributes), fcz i= (user defined M scaling coefficients for M attributes), co (frozen pretrained model), foor (# of inversion steps), S gen (ff of inference steps), bedit (computed from 4.1 boost
(computed from 4.2)
Function Editor (Zo, EØ, {Gi) LM for S +) *)
// step 1: Semantic encoding
Define {=a] Sfor s.t 0, TSyor =
for s 1,2,... Stor c (T(
do
2r.+1 = Va, + vI,
// step 2: Manipulation
Define {Fe)xer Spen s = 0, T = bedits T Snoise
boost and Suen = T
Tiggen Itsjor
for Sgen: Sgen 1,..., do // phase 1: editing
if Sedit then
10
Extract feature map he, from c0(,))
Ah+ Stor
(Ei=1 cxft, (h+,))
Spen
12
غ = co (ī;,
Ah;.)
14
oF, =0 0
// phase 2: dencising
else if Snoise then
16
== Eo (i+,) 0F, =
// phase 3: quality boosting
18
else
= co(ix,)
OF, ((1- (11- (5..v - aat,
20
21
N()
St, - + 0F, - 0 OF,
i Vor
23
return ëo (manipulated image)
As for LSUN-church, we provide a set of four images at once and add a question: 3) Diversity: Which group do you think has more diverse style? 4) Overall: Which image do you think is better considering the above evaluation criteria?
K.2 SEGMENTATION CONSISTENCY AND DIRECTIONAL CLIP SIMILARITY
We compare Asyrp and DiffusionCLIP using directional CLIP similarity (Sar) and segmentation
consistency (SC) fol following the protocols in DiffusionCLIIP in Table 4 and Table 5 A pretrained
CLIP (Radford al. 2021) and segmentation models (Yu al. (2018): Zhou et al. 2019 2017 Lee al. 2020)) are used compute Sair and SC, respectively. We choose three attributes (smi 1 ing. sad, tanned) for CelebA BalebA-HQ-re Leee-----in-ooone -nn-ann,,,i,, two attributes (P (Pixar,Neaddere Maa for CelebA-HQ- unseen-domain and three attributes (department. store, ancient, red brick) for LSUN- church.
For fair comparison, tributes (tanned, zed brick) following Kim & Ye (202T). Regarding attributes without the official checkpoints (smi 1 lingsadd), train DiffusionCLIIP by ourselves with the official code.
official checkpoints of DiffusionCLIP and provide scores of the at-
25

---

Published as a conference paper at ICLR 2023
Algorithm 2: Training Neural implicit function ft
Input:
epppretrained model), {To (i) \N (images to precompute), fi J:eeural implicit function of attribute), (soure text). Vtor(targe text). for of inversion steps), K of training epochs), tedit (computed from (4.I). bbost (computed from 3 4.2)
// step 1: Precompute latents
Sfor S. T1 0.7Syor =T Define 24ar
for i 1,2,.., N do
for s = 1, Sfor do (i)
c Ta)
3t.+1 V + VI 0r,6
Save the latent 2755
TS.a = bedit
// step 2: Update ft
for epoch= 1,2, ..., K do
for i 1,2, N do
Clone the latent ((1) for s = Sfor, Sfor for
10
I IT
11
1, Sedit do
Extract feature map hr, from es (ū!!)
Ah. Fr. (h,,)
P 1-aa, ca ( (Aht)) Pare
21 / a- (1
14
var,
var,
a/4 = VOT.-1 P + v OT 1 (0(
15
1/ = VOT.-1 Pare + V1 at OC. 60 (0(())))))
16
Lootal Pitrilaiirction (. ytar: Pares Vsvc) A -Paa Panel
Take gradient step Lootal and update fr
We use 100 samples per attribute. Asyrp outperforms DiffusionCLIP on Sair for all attributes. On SC, DiffusionCL.IP achieves better competitive scores. Because DiffusionCLIPP manipulates im- ages mostly by focusing on texture or color while preserving structure and shape, it takes advantage of getting higher SC scores. However, in Figure 28 results of smi 1 iing show that is not proper to edit attributes which require structural manipulation. Note that better SC scores do not guarantee bet- ter qualitative performance. Results of Pixar also show the similar tendency of each method. We allow more structural changes than DiffusionCI.IP while editing. Lower SC of our method comes from desirable structural changes as shown in Figure 28
(AI, AT)
Sair (I Ytar' &eef's Yret)
(17)
|
L DIRECTIONS
L.1 GLOBAL DIRECTION
Figure 29 and Figure 30 show that the effects of mean direction and global direction quite similar with Ah, by ft in various attributes. We compute mean direction and global direction from 20 different images.
We argue that h-space is roughly homogeneous across samples and timesteps. However, we ob- serve that h-space is not completely independent to the conditions especially unseen-domain
26

---

Published as a conference paper at ICLR 2023
Dataset
Irg.IKC
ACLIP
Avecon
from random noise
Person
Young person
0.8
0,905
Face
Smiling face
0.8
0.899*3
Face
Sad face
0.8
0.894*3
Face
Angry face
0.8
0.892*3
Face
Tanned face
0.8
0.886*3
X
Face
Disgusted face
0.8
0.880*3
X
Person
Person with makeup
0.8
0.875*3
CelebA-HQ
Person
Person with curly hair
0.8
0.835*3
Photo
Self-portrait by Frida Kahlo
0.5
0.4433
Human
Zombie
0.6
0.868*3
Person
Nicolas Cage
0.8
0.710*3
Human
Painting the style of Pixar
0.8
0.667*3
Photo
Painting in Modigliani style
0.565*3
Human
Neanderthal
0.802*3 1.2
Church
Gothic church
0,912*3
Church
Temple
0.898*3
Church
Department store
0,841*3
Church
Wooden house
0.793*3
LSUN-church
Church
Ancient traditional Asian tower
0.8
0.784*3
Church
Red brick wall Church
0.8
0.774*3
Church
Factory
0.702*3 0.8
Bedroom
Princess bedroom
0.912*3 1.5
LSUN-bedroom
Bedroom
Hotel hedroom
0.917*3 1.5
Dog
Happy dog
0.883$3 1,5
Dog
Sleepy dog
0.866*3 1.5
AFHQ-dog
Dog
Wolf
0.850*3 1.7
Dog
Yorkshire terrier
0.690*3 2
Painting person
Painting of Sad person
1.5
0.921
Painting person
Painting of Smiling person
1.5
0.908
METFACES
Painting person
Painting of Disgusted person
1.5
0.879*3
Table 3: The coefficients range from 0.5 to 0.8 for in-domain attributes. Unseen domains need slightly stronger ACLISS We also report which attributes train with random noise sampling. The criterion which attributes require relatively less maintenance of identity. We use Arecon CLIP similarity which reduces Ll1 loss when attribute needs lot of change.
Mileb--H--indoomee Sad
CelebA Wew----ssen-domaee
Smiling
Tanned
Pixar
Neanderthal
Sair
SC
S
SC
S
SC
S
SC
S
SC
Asyrp (ours)
0.921
89.02%
0.964
88. 90%
0.991
85.71%
0.956
76.51% %
0.805
79.03
DiffusionCLIIP
0.813
91.41 %
0.760
89.93% %
0.888
92.85%
0.811
89.91 %
0.661
81.23%
Table 4: Quantitative evaluation on CelebA-HQ.
27

---

Published as a conference paper at ICLR 2023
Real image
Asyip (ours)
DiffusionCLIP
Real image
Asyip (ours)
DiffusionCLIP
Stilina
Figure 23: Asyrp vs. DiffusionCLIP on CelebA-HQ in-domain attributes. We observe that Dif- fusionCLIP struggles change semantic facial attributes. Their official checkpoints do not exist and we ran careful hyperparameter tuning for training.
Real image
Asyip (ours)
DiffusionCLIP
Real image
Asyrp (ours)
DiffusionCLIP
Neandaathal
Pixar
ON
Figure 26: Asyrp vs. DiffusionCLIP on CelebA-HQ unseen-domain attributes. We use the offi- cial checkpoint provided by DiffusionCLIP. Asyrp works well even for the unseen-domains.
LSUN church ancient
Department store
red brick
Sair
SC
Sair
SC
Sair
SC
Asyrp (ours)
0.778
57.62%
0.943
62.6 22.5%%
0.989
65.83 %
DIfIusionCLIP
0.661
54.50%
0.907
64.82 %
0.964
65.02%
Table 5: Quantitative evaluation on LSUU--hurch.
(See Figure 29 Note that unseen-domains require longer editing interval with small fodit- There- fore, we conjecture that the consistency h-space decreases at the end of the generative process. is supported by additional experiments that L2 distance between Ah, and global direction gradually increases along with timesteps. We leave a more detailed analysis on h-space at different timesteps future work.
28

---

Published as a conference paper at ICLR 2023
Real image
syrp (ours)
DiffusionCLIP
Real image
Asyip (ows)
DiffusionCLIi LIP
Deportmmett store
Woodes
Real image
ssypp (ours)
DiffusionCLIP
Real image
syrp (ours)
DiffusionCLIP
Anxioni
Red brick
Figure 27: Asyrp vs. DiffusionCLIP on LSUN-church. Note that the official checkpoint provided by DiffusionCLIP. We observe DiffusionCLIP produces narrow range of styles while Asyrp produces diverse styles.
L.2 COMPARE THREE METHODS
In this section, compare three methods: implicit neural direction fi, optimized Ah, and opti- mized gloobul
Training time ft = Ahglobal < Ahh
We have optimize each Ah, for each time step t. Additionally, it needs specific hyperparameters for each Aht e.g., higher learning rates for larger L. On the contrary, time-consuming for fa is similar optimizing Ahglobot
Quality J2 Ah, Alglobal
ft and Ah, where directions can be obtained for each timestep. have the best quality. As can be shown in Figure 10, Ahglobal is sometimes accompanied by slight differences in hair, etc.
Extensibility fa Ah,
globoa!
29

---

Published as a conference paper at ICLR 2023
CelebA-HQ Smiling
CelebA-HQ
Pixar
LSUN-church Department store
Original
Asyrp (Ours)
DiffusionCLIP
Figure 28: Example segmentations for computing segmentatiion-conistennyy (SC).
Ar, can be obtained from fc. and Ahglobal can be obtained by aggregating Ahe
We opt to use ft for above three advantages.
M RANDOM SAMPLING
We conduct extra experiments: generating images with target attributes using Asyrp not from inver- sion but from random Gaussian noises. As consequence, the generative process be used for conditional random sampling. We provide the results in Figure 31 However, it is beyond the scope of this paper.
30

---

Published as a conference paper at ICLR 2023
Real image
A
A Amennn
Ahglobal
Real image
A/u
A
menn
Ahglobai
Smiling
2
Pixar
Angry
Modigliani
E
6
Neanderthal
Figure 29; We compute mean direction and global direction from 20 other images on CelebA- HQ. The effect of mean direction and global direction are quite similar with Ah, by ft at diverse attributes.
Real image
A
Ak meon
Alglobal
Real image
A
Ahmeon
Department store
Gothic
Temple
Figure 30: We obtain mean direction and global direction from 20 other images LSUN-church. The effect of mean direction and global direction are quite similar with Ah, by fo at diverse at- tributes.
Young
Pixar
Smiling
Frida
(a) DDIM process
(b) DDPM process
Figure 31: Uncurated random sampling. We generate images from random noise with Asyrp. Although do not focus these results, Asyrp can be used for conditional sampling.
31

---

Published as a conference paper at ICLR 2023
N
MORE SAMPLES
N.1 IMAGENET
We conduct extra experiments: editing images with target class using Asyrp with ImageNet pre- trained model. We verified that models trained on large datasets, such as ImageNet, can be edited using Asyrp. However, we also observed that in this case the latent space is not partitioned by classes. For an orange, we have different latents for single orange, for many oranges, for cross-section of cut orange, and a single piece of orange. Therefore, we learned the implicit function by collecting similar images to find the direction.
Real image
Apple
Rurger
Real image
School bus
House
Figure 32: Result of Asyrp in ImageNet. The result shows that Asyrp works even in ImageNet dataset.
N.2 MUL.T-INTTEPOLLATIOO
Figure 33 provides mixed interpolation between multiple attributes. We observe that any interpola- tion with any attribute is possible.
N.3 MORE RESULTS ON ALL DATASETS
We provide more results on CelebA-HQ (Figure 34p. LSUN-church (Figure 35 AFHQ, LSUN- bedroom, METFACES (Figure
32

---

Published as a conference paper at ICLR 2023
-smiligg (untrained)
+smiling (trained)
-young (untrained)
tyoung (trained)
-smiling (untrained)
+smiling (trained)
-young (untrained)
+young (trained)
Figure 33: Combination of multiple attributes. The result shows that Asyrp works for combined
Ah of smi 1 ing and young.
33

---

Published as a conference paper at ICLR 2023
Real image Smiling
Sad
Angry
Man++ Woman
Young
Curly hair
Nicolas
Pixar
Modigliani Neanderthal Frida
Man+ Woman
Real image Smiling
ngry
Young
Curly hair
Nicolas
Pixar
Modigliani Neanderthal Frida
Figure 34: We provide more results on CelebA-Ho AQ dataset.
34

---

Published as a conference paper at ICLR 2023
image Departoent
Factory
Temple
image Deportment
Factory
Temple
Figure 35: We provide more results on LSUN-church dataset.
Real image
Sleepy
Happy
Wolf
Yorkshire
Real image
Hotel
Princess
Real Image
Smiling
Disgusted
Figure 36: We provide more results on AFHQ, LSUN-bedroom and METFACES datasets respec- tively.
35


Reviews:
[{"review_id": "uwjNbb7dGnm", "reviewer": "Reviewer_4Ber", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper discovers a semantic latent space (termed h-space) for pretrained diffusion models. The proposed h-space is a shift/residual in the middle-layer feature of the UNet, and is discovered via finetuning. The authors propose asymmetric reverse process (Asyrp) for both finetune and sampling. The proposed h-space has several nice properties: homogeneity, linearity, robustness, and consistency across timesteps.", "summary_of_the_review": "I am willing to amend my score is my major concerns are addressed.", "strength_and_weaknesses": "## strengths\n- Finding a semantic latent space for pre-trained diffusion models is an important problem and is relatively less studied. The proposed Asyrp is a promising endeavor in this direction.\n- The proposed h-space has several nice properties: homogeneity, linearity, robustness, and consistency across timesteps.\n- The empirical study in generative process design suggests an interesting analysis for editing flexibility.\n\n## weakness and detailed questions\n- Theorem 1 seems a little bit weak to me, it says that $\\Delta x_{t-1} \\approx 0$ given a shift $\\Delta \\epsilon_t$. However, this conclusion is trivial given the well-known result from DDIM Eq. 13. The resulted loss function Eq. 7 seems same as the GPU-efficient finetuning in DiffusionCLIP [1].\n- In Eq. 7, is $x_t^{edit}$ $P_t^{edit}$?\n- Can we think of the proposed training of Asyrp as a constrained version of DiffusionCLIP (only the middle layer of UNet can be finetuned)? If so, how does the generative process of Asyrp compare to DiffusionCLIP, i.e. unmodified (symmetric) DDIM sampling?\n- Linear combination (Figure 8) is quite interesting. How does this compare to a linear combination of scores (as in DiffusionCLIP)? This comparison would be interesting since it saves multiple UNet forward passes.\n- This won't impact my rating. I am curious is it possible to extend the proposed h-space in text-to-image diffusion models such as StableDiffusion?\n\n[1] Kim, Gwanghyun, Taesung Kwon, and Jong Chul Ye. 2021. “DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/2110.02711.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "d5yWfNVhDp", "reviewer": "Reviewer_KRCt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce an algorithm that manipulates images using diffusion models.\nThey do so by manipulating the representation in the bottle-neck U-net layer of the diffusion model over several (but not all) timesteps. The loss function that they optimize is the same as for DiffusionCLIP.", "summary_of_the_review": "Overall, I think this is a nice contribution that carefully explains the authors' procedure and provides visually striking experimental results. The contribution is novel and presents a significant step towards practical image editing with diffusion models. With additional theoretical justifications, this would be even better. I would recommend it for publication at ICLR.\n", "strength_and_weaknesses": "### Strengths\n\nThe visual results are striking. There are plenty of examples of different attributes for editing. A user study (n=80) was conducted and showed very clearly superior performance to DiffusionCLIP. The algorithm can be applied to pretrained diffusion models.\n\n\n### Weaknesses\nTheorem 1 seems to be formulated a bit sloppy... $ \\tilde{x}_t \\approx  x_t $\nisn't really saying anything. Defining a bound $||\\tilde{x}_t -  x_t|| \\leq \\delta$ and then showing that $\\delta$ is small would be better suited.\nFor the proof in appendix B, I would prefer if all symbols were defined in that section. In equation (13) you introduce $\\beta$ and $\\eta$ without stating their relation to previous terms.\nIn general, theoretical motivation for the procedure and parameter choices is lacking a bit.\n\nThe only quantitative evaluation is the user study. Results for DiffusionCLIP seem a bit lower quality than in the original paper.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "91ewEqM099K", "reviewer": "Reviewer_pRhF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes asymmetric reverse process (Asyrp) to exploit the semantic latent space of pre-trained diffusion models. With the guidance of CLIP, the proposed method can find semantically meaningful directions in the latent space for image editing. ", "summary_of_the_review": "Overall, the goal of the paper is intriguing, but the paper contains a number of heuristics without enough strict ablation study and quantitative results, making the claims doubtful.", "strength_and_weaknesses": "Strength:\n(1) As far as I know, this paper is the first work to directly find the semantic directions in pre-trained diffusion models.\n(2) The empirical results look impressive.\n\nWeaknesses:\n(1) The choice of h-space needs more clarification. The Figure 15 and Appendix C.3. could not explain everything. My questions are: [1] Are images in Figure 15 training images or test images? [2] If they are training images, why does the layers other than layer 8 have bad performance? I imagine they are all able to minimize the loss in Eq. 7. [3] If they are test images, why does layer 1-6 does not change the generated images at all? At least there should be some changes.\n(2) The implicit neural direction looks unnecessary. Could the authors provide more ablation study to compare using $f_t$ and $h_t$? Moreover, I cannot understand the results in Figure 6 (a). Since $\\epsilon$ has a higher dimension than $h$, I can hardly imagine it yields higher loss when optimizing Eq. (7). \n(3) There are too many magic numbers in Section 4, making me doubt the generalizability of the proposed generative process. Though the authors claim that these magic numbers work across the five datasets, I still think quantitative results should be provided on analyzing the sensitivity of these magic numbers.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0NnqaAdGFQf", "reviewer": "Reviewer_6SAm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposed a new method called Asyrp (asymmetric reverse process) to discover the semantic directions in the latent space of pre-trained diffusion models. Specifically, this work first theoretically and empirically showed that a shift added to the UNet noise prediction results in almost the same reverse process trajectory. To this end, an asymmetric reverse process is proposed to manipulate the real image by controlling the latent variable. Besides, this work found that the bottleneck (or the deepest) feature map in the UNet, termed h-space, has the most salient properties than other latent spaces, and thus considered to learn implicit neural directions in the h-space. Finally, in order to achieve the best tradeoff between editing strength and generation quality, this work designed a full editing process, consisting of three phases: editing with Asyrp, traditional denoising, and quality boosting.\n", "summary_of_the_review": "Overall, I liked the idea of discovering the latent semantics in diffusion models and I think the experiments well support the claimed contributions in the work. Since I still have some concerns about the theory and experiments (see the weaknesses above), I put my initial rating as “weak accept”. I’m open to increasing my score if the rebuttal well addresses my concerns.\n", "strength_and_weaknesses": "Strengths:\n- This work is clearly written and it is overall very easy to follow. \n- I liked the idea of discovering the semantic directions in the h-space of existing diffusion models, since the state-of-the-art diffusion models use the UNet backbone. Compared with the noise space which has been considered in previous works, the h-space is more compact and contains higher semantic information. Moreover, the work semantically demonstrated in various experiments several nice properties of h-space: homogeneity, linearity, robustness and rough consistency across timesteps. \n- Other important components in the proposed method, including asymmetric reverse process, implicit neural directions, three-stage generative process design, have also shown via experiments the respective effectiveness in contributing to the success of the proposed method.\n- The experiments were conducted across different diffusion model architectures and different datasets to show the general applicability of the proposed method, and also demonstrated that it outperformed the previous fine-tuning method DiffusionCLIP.\n\nWeaknesses:\n- In the proof of Theorem 1, it assumed that $\\beta_t \\approx 0$. This might be true only when $t \\to 0$. As $\\beta_t$ monotonically increases from $t=0$ to $T$, I wonder if this assumption still holds when $t$ is large. I think more justifications are needed here to clarify the possible confusions.\n- In experiments, many “less-diverse” datasets have been considered, such as CelebA-HQ, LSUN bedroom/church, AFHQ-dog. I wonder why not consider more diverse datasets, such as ImageNet? In particular, we know diffusion models have achieved good generation results on ImageNet. Compared with previous latent space traversal works based on StyleGAN that cannot work on ImageNet, this result will show a big advantage of the proposed method.\n-  Since we already have many StyleGAN-based methods (StyleCLIP, StyleGAN-NADA, etc.) for latent semantic discovery, how does the proposed method based on diffusion models compare with them? I can tell that for the diffusion-model-based method, we still need many more tricks (e.g., asymmetric process, three-stage scheduling, implicit direction, etc.) to make it work reasonably well. I’m not saying that we have to beat those methods in this paper, but even not, it will still be interesting to see the pros and cons of both two methods and to provide guidelines for further improvements.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%