Boundary Guided Learning-Free Semantic Control with Diffusion Models
Authors: Ye Zhu
Yu Wu
Zhiwei Deng
Olga Russakovsky
Yan Yan
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion probabilistic models
learning-free applications
high-dimensional semantic boundary
markov mixing
TL;DR: We introduce one of the first learning-free methods for effective and efficient image semantic control and editing via pre-trained unconditional denoising diffusion base models.

Abstract: Applying pre-trained generative denoising diffusion models (DDMs) for downstream tasks such as image semantic editing usually requires either fine-tuning DDMs or learning auxiliary editing networks in the existing literature. In this work, we present our BoundaryDiffusion method for efficient, effective and light-weight semantic control with frozen pre-trained DDMs, without learning any extra networks. As one of the first learning-free diffusion editing works, we start by seeking a more comprehensive understanding of the intermediate high-dimensional latent spaces by theoretically and empirically analyzing their probabilistic and geometric behaviors in the Markov chain. We then propose to further explore the critical step in the denoising trajectory that characterizes the convergence of a pre-trained DDM and introduce an automatic search method. Last but not least, in contrast to the conventional understanding that DDMs have relatively poor semantic behaviors (in generic latent spaces), we prove that the critical latent space we found already forms semantic subspace boundaries at the generic level in unconditional DDMs, which allows us to do controllable manipulation by guiding the denoising trajectory towards the targeted boundary via a single-step operation. We conduct extensive experiments on multiple DPMs architectures (DDPM, iDDPM) and datasets (CelebA, CelebA-HQ, LSUN-church, LSUN-bedroom, AFHQ-dog) with different resolutions (64, 256), achieving superior or state-of-the-art performance in various task scenarios (image semantic editing, text-based editing, unconditional semantic control) to demonstrate the effectiveness.

Boundary Guided Learning-Free Semantic Control with Diffusion Models
Ye Zhu Yu Wu Zhiwei Deng", Olga Russakovsky", Yan Yan'
Department of Computer Science, Illinois Institute of Technology
2 Department of Computer Science, Princeton University School of Computer Science, Wuhan University 'Google Research
yezhupprinceton edu, wuyucs@whu oggarus@princeton. edu, yyan340iit.
wwwp://wwwuuuuuu...e edu. en, zhiveiieeggggoggle.ccom
Abstract
Applying pre-trained generative denoising diffusion models (DDMs) for down- stream tasks such image semantic editing usually requires either fine-tuning DDMs or learning auxiliary editing networks in the cxisting literature. In this work, we present our BoundaryDifiasion method for efficient, effective and light- weight semantic control with frozen pre-traine pre-trained DDMs, without learning any extra networks. As of the first learning-fis -reee diffusion editing works, start by seeking a comprehensive understanding of the intermediate high-dimensional latent spaces by theoretically and empirically analyzing their probabilistic and geometric behaviors in the Markov chain. We then propose to further explore the critical step for editing in the denoising trajectory that characterizes the convergence of pre-trained DDM and introduce an automatic search method. Last but not least, in contrast to the conventional understanding that DDMs have relatively poor semantic behaviors, we prove that the critical latent space we found already exhibits semantic subspace boundaries the generic level in unconditional DDMs, which allows us to do controllable manipulation by guiding the denoising trajectory towards the tar- geted boundary via a single-step operation. We conduct extensive experiments on multiple DPMs architectures (DDPM, iDDPM) and datasets (CelebA, CelebA-HO, AFHQ-dog) with different resolutions (64, 256), achieving superior state-of-the-art performance in various task scenarios (image semantic editing, text-based editing, unconditional semantic control) demonstrate the effectiveness. Project page at
LSUN-church,
Introduction
The denoising diffusion models (DDMs) (53 have been successfully applied various tasks such as image and video synthesis 21
41 22
48
14
23
52
20 audio generation 33
40
37
71].
image customization [50] reinforcement learning 26
and recently in scientific applica- tions (63, 34, 58). Among various applications of DDMs, one popular downstream task is to use pre-trained unconditional DDMs for image manipulation and editing, by either re- traiiinning the diffu- sion models [31], learning extra auxiliary neural networks for the editing signals [44] 33). However, as these methods require additional learning processes, we argue that they have not yet fully leveraged the great potential of pre- pre-trained DDMs. In this work, show that the latent spaces of pre-trained DDMs when leveraged properly, can be directly used to perform various manipulation tasks without
'The latent spaces in our context are different from concept LDMs (a.k.a.. StableDiffudion 4S While the main idea of LDMs is to operate the diffusion denoising process on the latent space of raw data, we investigate the latent spaces represented by the intermediate diffusion states along the
Markov chain.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Original
Ours
Asym
DiffusionCLIP
Original
glaxr"
"+ smile
(a) Compartson sementic editing given real imagees
add "smmle"
(b) Semantic editing on other attributes
Original
Recons
Red brick wall church
Temple
Gothic Gothic.chrrch charch
Church nigh:
Wonden church
Original
Recons.
Prinooss bodroom
Original
Recons
Smiling dog
Original
Recons
Smiling dog
beste semartic aditing given images, including unsesn domains (in red)
(d) Unconditinal semantic control from sampled latent encdingss on the attributa smile
We present the BoundaryDiffusion for efficient (single-step), effective and light-weigh semantic control on multiple application scenarios with frozen pre-trained DDMs. Different from existing works, our BoundaryDifussion is a learning-free method that achieves SOTA performance without learning any extra networks, in contrast e.g. Asyrp (35) which requires training auxiliary editing neural networks, and DiffusionCI.II [31] which fine-tunes the pre-trained DDMs. We include more randomly selected non-cherry-picked results in Appendix G as qualitative demonstrations.
Figure 1:
learning any extra networks and achieve superior or SOTA performance as shown in Fig. | This makes our work the first to achieve learning-free free diffusion editing with unconditional base models that are trained without additional semantic supervision or conditioning.
The understanding of latent spaces has been a challenging yet critical factor for better explaining. interpreting, and utilizing diffusion models. On the one hand, the DDMs are formulated as a long Markov chain (usually with 1,000 diffusion steps) 21,, which introduces the same number of generic latent spaces within a single model. In contrast to single unified latent space in GANs [16], this unique formulation imposes extra difficulties for studying the latent spaces of DDMs. leading to additional research questions such as "which latent spaces should we focus on to analyze their behaviors? ?" On the other hand, abundant methodology designs for multiple downstream applications have been motivated by their corresponding understanding in terms of latent spaces 44
31 35). For
example, DiffAutoeneoder [44) proposes to learn auxiliary encoders based on existing latent variables to obtain semantically meaningful representations for image attribute manipulations, mainly because the generic latent spaces in DDMs are initially believed to lack semantics.
the same reason, DiffusionCIIIP 31 proposes to achieve semantic manipulation by fine-tuning the entire pre-trained DDMs each time given new editing target attribute. More recently, Asyrp (35) observes that DDMs already form semantic spaces, but only in the bottleneck level of U-Net implementations 49

---

referred to as h-space. Accordingly, they propose to learn an auxiliary editing network that operates on the h-space for each manipulation attribute. To this end, in this work, we feature our first branch of contributions in terms of latent spaces understanding from the following two aspects, by supplementing and correcting the current literature: i) Through the geometric and probabilistic analysis of the latent encodings from the departure space at diffusion step T, we demonstrate that the stochastic denoising and deterministic inversion (the process to convert image to noisy latent variables, which is essential for semantic image editing (31 trajectories along the Markov chain are asymmetric, as opposed to previous understanding [35] (see Sec. 3). ii We further propose a theoretically supported automatic search method to identify the critical diffusion step (f.e., the mixing step) along the chain, which starts to exhibit semantics at the generic level (referred as e-space), as
supplementary and/or correction to existing literature 44 31 35
(see Sec. 4 and Sec. 5).
Based on the better understanding of latent spaces, we then propose our BoundaryDiffusion method to achieve semantic control with pre-trained frozen and unconditional diffusion models in a learning- free manner in Sec. 5 Specifically, our approach first locates the semantic boundaries in the form of hyperplanes via SVMs within the latent space at the critical mixing step. We then introduce mixing trajectory with controllable editing strength, which guides the original latent encoding to cross the semantic boundary at the same diffusion step to achieve manipulation given a target editing attribute as in Fig. 3 We also optimize the image quality via a combination of deterministic and stochastic denoising processes. We conduct extensive experiments using different DDMs architectures (DDPM 21. iDDPM (41) and datasets (CelabA (39], CelebA-HQ [27]. AFHQ-dog [10], LSUN- church (66). LSUN-bedroom (66]) for multiple applications (image semantic editing, text-basee editing, unconditional semantic control), achieving either superior or SOTA performance in both quantitative scores (Sair 45). segmentation consistency 65
68 67 face identity similarity [13 and
FID [9] and qualitative user study evaluations, as presented in Sec. Compared to previous learning- based methods [3 (3), our CoundaryDiffusion method features light-weight and unified single-step operation without any additional training cost. In addition, different from DiffusionCLIP [3 and Astrp 35 which take about 30 min of learning time per attribute, our Boundary takes approximately negligible time about is and as few as 100 images to locate and identify the semantic boundaries.
Overall, our work contributes to both latent space understanding and technical methodology designs. Firstly, we provide multiple novel findings and analysis that correct or supplement the current under- standing of latent spaces of DDMs. Notably, reveal that the previous symmetric assumption of stochastic denoising and deterministic inversion directions does not hold after explicitly analyzing the geometric and probabilistic properties in the departure latent space diffusion step T. We also demonstrate that the unconditionally trained DDMs exhibit meaningful semantics at our located mixing step along the Markov chain in generic e-space level. Secondly, our proposed BoundaryDifsi- sion method achieves semantic control in a learning- free manner, providing a promising direction for resources friendly and efficient downstream applications. The above contributions are further supported and validated by strong performance through extensive experiments and comprehensive evaluations. Code available at https://githu..com/L Ycwhh//oondaayDiitt...ooo
2 Background
We briefly describe essential background here, and include more detailed related work in Appendix A
Denoising Diffusion Models. Denoising Diffusion Probabilistic Models (DDPMs) 21 are one of the principal formulations for diffusion generative models (53). The core design of DDPMs consists of stochastic Markov chain in two directions. The forward process adds stochastic noises (usually parameterized with Gaussian kernel) to data sample zo following:
1x) = .V(1V - 3 ByLa),
where {8 = 1 are usually scheduled variance.
The reverse direction, which corresponds the generative process. denoises a latent sample Xy (often sampled from a standard Gaussian distribution) to a data sample xn as:
3.
X
VI - aq (((xx)) ot2t,
(2)
VI
--
3
where ef the learnable noise predictor, 2, ~ NO I), and the variance of the reverse process of set to be o² B.
3

---

Alternatvely, denoising diffusion implicit models (DDIMs) consider a non Mon-Marko process with the same forward marginals as DDPMs, with slightly modified deterministic sampling process as follows:
Xt-1 = va--1 Xt vI ard ) V1 04-1 - -o -x(()) + t
Vor
where 02 = (1-0t-1) at) 04/ as the v control the degree of stochasticity. Specifically, the first term indicates the directly predicted X, at arbitrary diffusion step 4, and the second term denotes the direction pointing to Xt. and the last term is the random noise.
In this work, we focus on the study of denoising diffusion models. While the pre-trained models are learned based on DDPMs, we adopt DDIMs for inverting real images Xy E X' to obtain their corresponding latent encodings X E as in [1 B5 However, different from conventional understanding, we reveal this inversion process is asymmetric to the stochastic denoising process from DDPMs in Sec. E which is an important fact to understand the reason why the previous works require extra learning to achieve the semantic editing effects.
Semantic Understanding and Trajectory in High Dimensional Space. Previous works on the semantic understanding of latent space for generative models have largely focused on variants of GANs [16, such as the e space for generic GANs [16]. P space from ProgressiveGAN [27], and W space from StyleGAN [29]. More recently, B35 study the latent spaces of diffusion models and show that the pre-trained DDMs already have semantic latent space at the bottleneck level of U-Net, and name it h-space. We observe that there exist two unique components for DDMs compared to previous studies for GANs [2]: the relatively high- -igednmmeesionlltyy of intermediate latent spaces and the trajectory along the Markov chain.
Firstly, unlike GANs that usually sample from a lower dimensional Gaussian distribution and upsam- ple the latent encoding to the higher data space [16]. DDMs (33 maintain the same dimensionality for all the intermediate latent encodings ((1r)) throughout the entire Markov chain. The above formulation regularizes the latent encodings to stay in higher dimensional spaces, which motivates us to tackle the problem using existing mathematical tools from high dimensional space theory (8), such as the concentration mass distribution and Gaussian radius estimation. In addition, DDMs model random walk on the data space as a Markov stochastic process [56] 57, 32 55 24 60), and form a trajectory as the diffusion step proceeds. This trajectory characterizes the property of the considered Markov and imposes nature discussion on the convergence question, which inspires us to draw the connection between the mixing time study in Markov chain 38 and diffusion models.
3 High-Dimensional Latent Spaces in Diffusion Models
Notations. Given a pre-trained DDM p (we omit 0 from common notation pe given the fact that the parameters are frozen in this work) with T diffusion steps. We define the raw data space as X' and a data sample Xo X' in d-dimension. The intermediate latent variables X(1:T) are samples in the same dimensionality as Xn from their corresponding spaces E(1:T)- Specifically, as previous works suggest there are two methods to obtain the latent encodings, either from direct sampling as most generative works do [21 14 48 33, or from a given raw data after inversion via DDIMs B5) We note x² and x as the latent encodings at the t-th step from 6, via sampling and inversion, respectively. Similarly, we distinguish two sampling processes with the stochastic DDPMs and the deterministic DDIMs as ps and pi. Note the difference between DDPMs and DDIMs only exists in sampling process, while the training remains identical (i.e., single p may be used in two sampling ways). We use x6 to represent the denoised image after editing.
Sampling vs. Inversion. We start the analysis of high-dimensional latent spaces of DDMs by considering different sources of latent encodings from the departure space er at the diffusion timestep T. Theoretically, the directly sampled latent encodings x² follow standard Gaussian distribution AN Ia) by definition. In contrast, the inverted latent encodings, as they adopt the DDIMs formulation, do not follow an identical trajectory, but the marginal distribution of the forward DDPMs to go from the image space X' to the latent space éT, leading to a non-standard Gaussian distribution different from the direct sampling. The above describes a critical statement that differs from existing work 35), where DDIMs- inversion has been previously considered as symmetric process to the generative trajectory.

---

Table 2: 100-stepwise variation of Gaussian radius for different sample sources and sampling com- binations using pre-trained DDM. With different latent encodings and sampling processes, the mix-
Table 1: Estimation of the Gaussian radius r for sampled and inverted latent encodings in éT from different pre-trained DDMs, datasets and image resolutions (64 and 256).
ing step appears at approximately the same time
around 500.
Steps
1000
900
800
700
600
500
Models
Sampling
Inversion
Ar
Xi + P-
0.02
0.01
0.25
0.75
4.63
DDPM-CelebA-64
110.84
95.06
15.78
Xx + pc
0.02
0.02
0.21
0.74
2.03
4.76
DDPM-LSUN-256
443.42
430.88
12.54
IDDPM-AFHQ-250
443.41
438.07
5.34
Xy + Pa
1.76
1.74
1.42
1.45
2.08
3.81
xT + pc
1.72
1.73
1.45
1.41
2.18
3.70
We can also empirically demonstrate the above conclusion by estimating the radius r of the high-
dimensional Gaussian space. In mathematics, the radius of a high dimensional Gaussian space defined as the square root of the expected squared distance: r = ovd. Specifically, given a d-dimensional Gaussian centered at the origin with variance o². For a point x = (z), 22. Ta) chosen at random from Gaussian, the expected squared length is:
E(a + 22 + + x3) = d=( do².
For large d, the value of the squared length of x is tightly concentrated about its mean, and the square root of the expected squared distance (namely ov d) is called Gaussian radius, denoted by r.
We calculate the mean of squared length from N samples from xi and x as a EN ((TT)) For the sampled latent encoding from eT ~ (O Ia), the estimated radius r Vd, as shown in Tab. [ In contrast, for the inverted latent encodings from real images, the previously expected radius does not hold. Interestingly, we note that the inverted latent encodings from iDDPMs [41] endure less radius shift compared to DDPMs (2I), indicating that this radius estimation may imply the goodness of a pre-trained DDM. The above discussion on the distributions of er helps to better interpret the revealed "distance effect" below. More details about the marginal discussion via DDIMs inversion
are included in Appendix [A]
Distance Effect by Deterministic Inversion and Mitigation. As the next step, we study the influence of asymmetric inversion trajectory by following the marginal distribution via DDIMs. Previous studies [3 35 suggest that despite the latent encodings inverted via DDIMs allowing for near-perfect reconstruction,
rather empirically difficult directly edit x for controlling the final denoised output. Similarly, we observe the phenomena that for the inverted latent encodings xi., a small modification often leads to severely degraded and distorted denoised images following the deterministic sampling Pt, as shown in the second row of Fig. 2 a). As a comparison, the directly sampled encoding x shows a smooth transition after p; (see the top row in Fig. aa). This distance effect harms the performance in downstream applications that requires the inversion operation, such as image editing and manipulation tasks where the input is given raw image xo instead of directly sampled latent encoding xij
Having observed that the inverted latent encoding x tends to suffer from distorted denoised images during the interpolation, we aim to further understand and interpret the distance effect from properties of the latent high-dimensional spaces. Intuitively, the sampled latent encodings x locate in the area with higher probability concentration mass eT space, while the spatial locations of inverted ones x following the marginal diffusion trajectories, have relatively lower concentration mass. This spatial difference remains after the deterministic denoising via pt due to the homogeneity of intermediate latent spaces [35, resulting in denoised samples from x to be rather uut-of-distriuutio and low- fideeiility after distance shift in latent spaces. The homogeneity of latent spaces from (35] states that "the same shift in this space results in the same attribute change in all images". while in our context, we consider the spatial difference (i.e., inverted latent encodings have a smaller radius than sampled ones) in terms of Gaussian radius along the denoising trajectory.
From the geometric point of view, the probabilistic concentration mass of high-dimensional sphere is essentially located within thin slice at the equator and contained in narrow annulus at the surface [8]. In addition, one needs to increase the radius of the sphere to nearly vd before there is
5

---

0
4 P5
Concentaation
Amuuus width 3
(a) Interpolation neults
(b) Geametiic Unssttation concentraion high dimensional Causican.
Figure 2: Illustration of distance effect from the qualitative results and geometric properties. (a) The interpolation are conducted on CelabA-64, with different combinations for latent encoding sources and sampling methods. Following the same denoising process p;, inverted latent encodings lead to distorted images. (b) The inversion process reverses images to the latent encodings at the area of less concentration mass with a smaller radius. O Va
is the width of the slice for the concentration mass.
We explain the cause of the distance effect via the geometric illustration in Fig b). More theoretical details can be found in Appendix B
non-zero concentration measure 8
In contrast to the deterministic generation that suffers from the distance effect, the stochastic denoising is more robust and helps to alleviate the issue, as shown in Fig. a). Similarly, previous works also reveal that stochasticity is beneficial for the quality of denoised data [28 31 35]. Therefore, adding stochasticity is mitigation solution for the distance effect. From the perspective of spatial properties, the stochasticity brings the inverted latent encodings x from the area with lower concentration mass to higher ones, thus making the denoising process more robust.
Mixing Step in Diffusion Models
Problem Formulation. In the mathematical and statistical study of the Markov process, the mixing time is defined as key step f, at which the intermediate distribution converges to the stationary distribution 38 step problem for diffusion models as the search for the critical diffusion step E {1, T), where the Gaussian distribution converges to the final data distribution the reverse denoising process under some pre-defined distance measures (usually in total variation distance, noted as || ||rv). It is worth noting that, unlike the distance effect which induced by the deterministic inversion method, the mixing step is characteristic carried by generic DDMs, as we show that the mixing step appears at the same time both for the directly sampled latent encodings and the inverted ones. In practice, we show in the experiments in Sec. 6 that imposing editing and manipulations on the mixing step is more effective for downstream tasks. Meanwhile, similar concepts have been empirically studied as apper-parameter "return step" [1] "editing step" 335 without further investigations into its theoretical meanings. Proof and details of the following property can be found the supp.
Inspired by the established mathematical formulations, we introduce the mixing
Property 4.1. Under the total variation distance measure || ||rv: the mixing step fm for a DDM with data dimensionality d is formed during training (i.e., irrelevant the sampling methods). fmm is mainly related to the transition kernels, the stationary distribution (i.e., datasets), and the dimensionality d. In practice, a radius shift of approximately can be considered as a searching criterion for the mixing step in the denoising chain.
Automatic Search via Radius Estimation. While the rigorous theoretical deviation for the mixing step fmm. depends on many factors and is a complex and non-trivial problem (see our supp. for details), we present here effective yet simple method to search for the mixing step by estimating the radius r of latent high-dimensional Gaussian spaces. We can estimate the radius of the latent spaces along the denoising trajectory and show that the change of (corresponding to the transition kernel in Property
4.1 reveals the appearance of the mixing step for a pre-trained DDM. Our proposed method automates the search for this critical diffusion step, while it has been previously determined by manually checking the quality of final denoised images [31 35 More detailed discussion and proof about the empirical search method can be found in Appendix
See more details about the Markov mixing time in Appendix

---

XT
mm
X
Pi
Pi
Pi
Pi
Sample led or Inverted
Semantic
boundary
x' 0
Mixing trajectory
Ps
Ps
Initial trajectory after mixing step
Boundary guidance
x' mm
Figure 3: BoundaryDiijusion for semantic control in one-step editing. We propose to guide the initial deterministic generative trajectory via the semantic boundary at the mixing step m. for image manipulation.
Specifically, we follow the procedure described in Sec. 3 where we calculate the mean of the squared distance of a latent sample from the origin as the estimation of the Gaussian square. We compute the radius of the latent encodings following the generative trajectory, the earliest step with the largest radius shift (Ar = 4) is an approximation of the mixing step. We show the change of radius estimation for different combinations of latent encoding sources and sampling methods, demonstrating the mixing step is a generic characteristic irrelevant to the above factors in Tab. 2
BoundaryDiffusion for Semantic Control
In this section, we propose BoundaryDiffusion method to achieve one-step semantic control and editing by guiding the denoising trajectory using the semantic boundaries in high dimensional latent spaces with pre-trained and frozen DDMs. Specially, leverage the stochasticity and the mixing step bm for better qualitative results and more effective manipulation.
Semantic Boundary Search. In the case of attribute manipulation with given annotations (e.g.- face attributes modifications). we propose to search for the semantic separation hyperplane as a classification problem as in (51). Specifically, we use the linear SVM [18] to classify the latent encodings using the label annotations of their corresponding raw data, and the parameters in SVM are used the hyperplanes. The validation of using data ground truth annotation at the latent space level is guaranteed by the deterministic inversion and reconstruction via DDIMs [54]. We consider this application scenario to be conditional semantic editing, which allows us to edit semantic
attribute given a real raw image. It is worth noting that the objective of using SVM is to fit the existing semantic hyperplanes rather than learning classifiers or networks.
In addition to the above application scenario, we can further extend our semantic boundary search to more flexible settings such text-based image editing by leveraging the popular large-scale cross-modality generative models such as DALLE-2 [46] and Imagen [20]. Particularly, we can easily generate high-fidelity images based on given textual prompt and utilize the synthetic images as samples to search for the semantic boundary using the same classification method.
An extra application for our semantic boundary searching is to achieve semantic control for uncondi- tional image synthesis. due to the reason that the semantic boundary feature learned and formed during the training. In contrast, other learning-based methods like DiffusionCLIIP [31 that change the original parameters of pre-trained DMMs are not suitable for this downstream task.
Mixing Trajectory for Semantic Control. After having located the semantic boundaries, we propose achieve semantic manipulation by guiding the trajectory via the identified boundaries using the projected distance control. We note the unit normal vector of the hyperplane as E IRd, and define the sign-sensitive "distance" of the latent encoding Xt.. to the boundary similar [5]
d(n, Xt... n Xt...
(5)
7

---

Meanwhile, we add the stochasticity after the editing the latent encodings at the mixing step tm Overall, the mixing trajectory with boundary guidance can be summarized as:
[p((x(-1\x))
1T2t>> Im
Pmic (xt (xx) (p=(x(-1(x)
(6)
if tm 2t
where x =X,,, + (d((n, xt,.), and is the editing strength parameter that controls the projected distance to the target semantic boundary. We illustrate our proposed boundary-guided mixing trajectory method in Fig. [ Note this operation can also be extended multi-attribute semantic control, either by iteratively applying guidance to multiple boundaries (which is a linear operation), or
by modifying Eq.5 to d(N, Xt.. X where N [n1, D2 Dm for different attributes. The concrete algorithm of our proposed BoundaryDifttsion is presented in Appendix F
6 Experiments
6.1 Experimental Setup
Task Settings. We conduct our experiments on multiple variants of semantic control tasks, including real image conditioned semantic editing, real image conditioned text-based editing, and unconditional image synthesis with semantic control. Among those tasks, the first two scenarios are conditioned on given images and therefore involve the inversion process, while the last one is extension application for image synthesis without given image input.
Model Zoo and Datasets. We test different pre-trained DDMs on various datasets with different image resolutions for experiments. Specifically, we test the DDPMs on the CelabA-64 (3), CelebA- HQ-256 (27). LSUN-Church-25 and SUUN-edroom-2255 256
[66]. We also experiment with pre- trained improved DDPM 41 on the AFHQ-Dog-233 [00. In particular, we emphasize here that the experiments with different image resolutions are critical in our work, especially for the high- dimensional space analysis since the resolutions represent the dimensionality of the latent spaces as shown in Tab. and Fig. 2 However, in all the editing experiments, we use a resolution of 256 x 256 as the default setting for better visualization quality.
Operational Latent Spaces. Previous works have either operated on the e-space [31 or on the h- space from the bottleneck layer of the U-Net [35], showing that h-space has better semantic meanings with homogeneity, linearity, and robustness (35] In our main experiments,
impose the guidance on both latent encodings in both c and h levels. All the boundary guidance is imposed in a single-step the mixing step fm We show in experimental results later that the single-step operation is more effective at the e-level, while the h-level may require iterative guidance at multiple diffusion steps.
Implementations. We use the linear SVM classifier for searching the semantic boundary. We implement the SVM via the sklearn python package with the number of parameters equal to the total dimensionality of the latent spaces. For e-space, the dimensionality d, 3 x 256 x 256 196, 608. For the h-space, the dimensionality depends on the pre-trained DDMs architecture implementation for the U-Net [49]. In our experiments, we use the same level of latent spaces in [35, which have a dimensionality of d, = 8 x x 512 = 32, 768. In practice, we observe approximately 100 images are sufficient for finding effective semantic boundary. For the text-based semantic editing scenario, we use synthetic images generated from Stable Diffusion using text- eext-proppt [48]. These image samples can be obtained via any other praitee text-to-image -magg generative models.
In practice, the hyperplanes are found via linear SVMs [8] with almost negligible learning time of about 1 second on single RTX3090 GPU. For the inference, the time cost remains at the same level as other SOTA methods. Specifically, by using the skipping step techniques, we can already generate high-quality denoised images using approximately 40-100 steps, which take from 1.682 13.272 seconds, respectively on a single RTX-3090 GPU.
Comparison and Evaluations. We mainly compare our proposed method with the most recent of image manipulation works with diffusion models: the DiffusionCLIP [I] and the Asyrp 331. Both works consist of extra learning processes. Our evaluations include quantitative and qualitative assessments for different experimental settings, similar to most existing generation works. For the inversion and reconstruction part, since we adopt the same technique in 31 35
via DDIMs (54] and achieve near-perfection reconstruction results, we show qualitative samples in Fig.
I As for the conditional editing, we show quantitative results the CelebA-HQ using
8

---

Table 3: Evaluation results on CelebA-HQ-256 for real image conditioned semantic editing. FID scores are reported on the test set with 500 raw images, averaged on "add or remove smile" editing. The user study follows similar evaluation questions in B5
Methods
Sair f
SC
ID f
FID + User Quality 1 User Attribute f
StyleCLIP 44
0.13
86.8%
0.35
StycGGA-NAAA 15
0.16
89.4%
0.42
DiffusionCLIP 31
0.17
93.7%
0.70
86.23
3.2%
8.2%
Asyrp 35
0.19
87.9%
68.38
41.3%
44.9%
Ours Boundary Diffusiin
0.17
90.4%
0.73
63.14
55.5%
46.9%
three metrics: Directional CLIP similarity (Sarr) computed via CLIP [45). segmentation consistency (SC) computed via [63] 68 67]. and face identity similarity (ID) using [3]. We also report the FID
scores [9] of edited images the testing set of CelebA-HQ [27]. We further conduct human studies
as subjective evaluations. Additionally, we also report the classification accuracy for the semantic
boundary separation validation in Appendix G
6.2 Experimental Results
Conditional Semantic Manipulation. We show the qualitative results for conditional semantic
manipulation on the CelebA-HQ-256 [27] using the pre-trained DDPM [21] in the top row of Fig. [ The quantitative evaluations are shown in Tab. where we achieve SOTA performance. In particular,
we emphasize the fact both SOTA methods 31 351 directly use the directional CLIP loss as part
of the loss function to align the direction between the embeddings of the reference and generated image in the CLIP space [45] in their learning processes, which helps to increase the score of Sair Additionally, they also optimize the identity loss Lia("h; z0) to preserve the facial identity consistency between the original image Zi and the edited one zo- thus boosting the ID score. While our proposed BoundaryDifsasion method does use any learning loss function for semantic control, yet achieves very competitive scores in all the metrics.
Text-Guided Semantic Manipulation. With the success of large-scale cross-modality generative models [46] 20), we are able to extend the proposed BoundaryDiffusion method to other downstream tasks such as rext-guided semantic manipulations. Qualitative examples presented in the middle part of Fig. D where we show the original image input, reconstruction results, and the edited samples. the unseen domain transfer and show the example of editing a dog to afox, as indicated using the red text prompt in Fig. [ We note one limitation our BoundiryDifiusion is relatively difficult for unseen domain editing tasks compared other learning- -based methods, which aligns with the expectation since the frozen DDMs do not have the knowledge for unknown domain distributions.
We also perform experiments
Unconditional Semantic Control. One unique application scenario we present in the bottom row of Fig. iis to apply the identified semantic boundary to the unconditional synthesis setting, where the departure latent encoding Xy is now directly sampled from a standard Gaussian without a given real image input. Since the semantic boundary is an intrinsic characteristic formed during the training process of DDMs, we can easily re-use it for semantic control under a general unconditional denoising trajectory. In comparison, the existing SOTA methods that modify the parameters of pre-trained DDMs during additional learning are no longer applicable to this task.
Ablation on Operational Latent Spaces and Editing Strength. We can easily control the editing strength via the editing distance as defined in Sec. S Fig. shows an example for editing the smile attribute CelcbA-HQ. We use the typer-parameter Ç to control the degree of modification. From the same figure, we observe that we achieve a stronger editing effect at the same distance when applying the boundary guidance to both latent levels at ét,, and ht...
User Study. We also conducted a subjective user study to compare the performance with Diffusion- CLIP [3I] and Asyrp [35), where we interviewed 20 human evaluators to rank the edited images in terms of general quality and attribute editing effects. Specifically, we asked the evaluators to pick the best-edited result in terms of two aspects: 1). General quality: which image quality do you think is the best (clear, fidelity, photorealistic)? 2). Attribute: which image do you think achieves the best attribute editing effect (natural, identity preservation with respect to the given raw image)? Tab

---

Editing results both levels strength distanoe range of 150)
(bl Rai Bintting resltss on single generio e lntent level from strungth distoron range of [-150, 150)
[c] Editing nesults on single h latent level from strungth distanos mane of [-150, 150)
Figure 4: Ablation editing strength control via distance with different operational spaces. We can modify the editing strength by changing the distance between the latent encoding X to the target boundary. We show that the one-step guidance more effective when applied in the e-level space. Images in green boxes denote the given raw image.
shows the percentage of each method picked as the best result. Details about the user study and more randomly selected, non-cherry-picker qualitative samples are included in Appendix G
Discussion and Conclusion
Our work explores and leverage the great potential of pre-trained unconditional diffusion models without learning any additional network model modules. We provide a rather comprehensive analysis from the nigh-dimensional space perspective better understand the latent spaces and the trajectory for pre- pre-trained DDMs. We introduce the mixing step for DDMs, which characterizes the convergence of generic DDMs, and present simple yet effective method search for the mixing step via Gaussian radius estimation in high- high-dimensional latent spaces. Last but not least, we propose BoundaryDiffusion. an effective learning- free method with coundar--guidddd mixing trajectory realize effective semantic control and manipulation for downstream tasks using multiple pre-trained DDMs and datasets.
Broader Impact and Limitation. We diseuss the broader impact of this work. Firstly, the primary goal of this work is not to create new generative models or generate synthetic data, but to explore the potential of the current generative models for better usage. To do so, we also propose new perspective to better understand and interpret the DDMs, which is the analysis of high--dienniionll latent space behaviors using the thcoretical tools from mathematics and statistics. In the meanwhile, during the process of exploring and separating the semantic boundary, we leverage the current popular cross-modality generative models synthesize images with text prompt. However, all the generated images are only used for boundary detection. We believe our work brings valuable insights to the research community in terms of a better understanding and further exploration via training-free free methods to apply diffusion generative models.
As for limitations, since BoundaryDifiusion method is learning-free and does not introduce any extra parameters to the base diffusion models, the ability for unseen domain image editing is relatively limited compared to other learning-base based methods.
10

---

Acknowledgements
This work is supported by NSF 233-2309073. This article solely reflects the opinions and conclusions of its authors and not the funding ageney. The first author of this paper would also like to thank Dr. Alain CHILLÈS for the insightful discussions on the theoretical and mathematical analysis.
References
[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the
stylegan latent space? In ICCV, 2019.
[2] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegantt: How to edit the embedded images? In CVPR, 2020.
[3] Yuval Alaluf, Or Patashnik. and Daniel Cohen-Or. Restyle: A residual-based stylegan encoder
via iterative refinement. In ICCV, 2021.
[4] Jacob Austin, Daniel Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In NeurlPS, 2021. [5] Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. Towards open-set identity preserving face synthesis. In CVPR, 2018
[6] David Bau, Jun Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Torralba. Inverting layers of a large generator. In ICLR Workshop, 2019.
[7] David Bau, Jun- Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Torralba. Seeing what a gan cannot generate. In ICCV, 2019.
[8] Avrim Blum, John Hopcroft, and Ravindran Kannan. Foundations of data science. Cambridge University Press. 2020.
[9] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1006, 2018.
[10] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020.
[11) Matthew Coleman, Olga Russakovsky, Christine Allen- Mlen-BBlacchetee, and Ye Zhu. Discrete diffusion reward guidance methods for offline reinforcement learning. ICML Workshop. 2023.
[12] Antonia Creswell and Anil Anthony Bharath. Inverting the generator of a generative adversarial network. IEEE TNNLS, 2018
[13] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular
margin loss for deep face recognition. In CVPR, 2019.
[14] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In NeurlPS, 2021.
[15] Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.
Stylegan-nade -ada:: dlip-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG), 41(::11-11 113, 2022.
[16] Ian Goodfellow, Jean Abbdee- Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
[17] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR, 2022.
[18] Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. Support vector machines. IEEE Intelligent Systems and their applications. 1998.
[19] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. NeulPS, 2017.
[20] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole. Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.022 2022. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[21]
[22] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 2022.
[23] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J
Fleet. Video diffusion models. NeurIPS Workshop, 2022.
[24] Chin-Wer Huang, Jae Hyun Lim, and Aaron C Courville, variational perspective on diffusion- based generative models and score matching. In NeurIPS, 2021.

---

[25] Minyoung Huh, Richard Zhang, Jun- Yan Zhu, Sylvain Paris, and Aaron Hertzmann, Trans- forming and projecting images into conditionl generative networks. In ECCV. Springer. 2020.
[26]
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. ICMI, 2022.
[27] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv::710. 10196, 2017.
[28] Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basee generative models. arXiv preprint arXiv:2206 00364, 2022.
[29] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[30] Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020.
[31] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion
models for robust image manipulation. In CVPR, 2022.
[32] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models.
In NeurlPS, 2021.
[33] Zhifeng Kong. Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In ICLR, 2020.
[34] Karsten Kreis, Tim Dockhorn, Zihao Li, and Ellen Zhong. Latent space diffusion models of
cryo-em structures. NeurlPS Workshop. 2022.
[35] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. In ICLR, 2023.
[36] Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, and
Marc Aurelio Ranzato. Fader networks: Manipulating images by sliding attributes. NIPS, 2017. [37] Junhyeok Lee and Seungu Han. Nu-wave: A diffusion probabilistic model for neural audio
upsampling. Proc. Interspeech 2021, 2021.
[38] David A Levin and Yuval Peres. Markov chains and mixing times, volume 107. American
Mathematical Soc., 2017.
[39] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In /CCV, December 2015.
[40] Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprint arXiv:2103. 16091, 2021.
[41] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic
models. In JCML. PMLR, 2021.
[42] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In ICML, 2017.
[43] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip:
Text-driven manipulation of stylegan imagery. In CVPR, 2021.
[44] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakom. Diffusion autoencoders: Toward meaningful and decodable representation. In CVPR, 2022. [45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin. Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML. PMLR, 2021.
[46] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprini arXiv:2204.0612 2022.
[47]
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-image translation. In CVPR, 2021.
[48] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer.
High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[49] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 2015.
[50] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500 22510, 2023.
[51] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In CVPR, 2020.
12

---

[52]
Uriel Singer, Adam Polyak. Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video: Texe-tovvideo generation without text-video data. arXiv preprint arXiv:2209, 14792, 2022.
[53]
Jascha
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In ICML.. PMLR, 2015.
[54] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR,
2021.
[55] Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In NeurIPS, 2021.
[56] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models.
In NeurlPS. 2020.
[57] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2020.
[58] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. bioRxiv, pages 2022-11, 11, 2022.
[59] Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning gan for pose- invariant face recognition.
CVPR, 2017.
[60] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space.
In Advances in Neural Information Processing Systems, 2021.
[61] Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Weiming Zhang. Lu Yuan, Gang Hua, and Nenghai Yu. E2style: Improve the efficiency and effectiveness of stylegan inversion. IEEE TIP. 2022.
[62] Weihao Xia, Yulun Zhang, Yujiu Yang. Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: A survey, IEEE TPAMI, 2022.
Duo Xu, Jonathan C Tan, Chia-Jung Hsu, and Ye Zhu. Denoising diffusion probabilistic models to predict the density of molecular clouds. The Astrophysical Journal, 950(2): 146, 2023, [64] Yongqi Yang, Ruoyu Wang, Zhihao Qian, Ye Zhu, and Yu Wu. Diffusion in diffusion: Cyclic generation. arXiv preprint arXiv:2306.0822 08247,,
[63]
one-way diffusion for cext-vision-conditoon 2023.
[65] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 325-341, 2018.
[66] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arviv: 0336, 2015.
[67] Bolei Zhou, Hang Zhao, Xavier Puig. Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. arXiv preprint arXiv: 1608. 05444, 2016.
[68] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[69] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In ECCV Springer, 2020.
[70] Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipulation on the natural image manifold. In ECCV. Springer, 2016.
[71] Ye Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, and Yan Yan. Discrete contrastive diffusion for gross-moda and conditional generation. ICLR, 2023.
13

---

We present the detailed related work about the denoising diffusion models including the marginal discussion in Appendix A The high-dimensional space properties and lemmas are introduced in Appendix B and we explicitly describe how those established theoretical theorems are used and connected to our analysis in the main paper Sec. In Appendix [ we provide the theoretical foundations of the Markov mixing study, which inspires us to formulate the mixing step problem for DDMs. More details about the mixing step problem, formulation, proof and discussion are included in Appendix D Appendix E includes details about semantic boundary search method and further discussions in terms of two latent space levels (e.g., generic e-space and ht-sacc from the U-Neu bottleneck (35 )- We show the algorithm of proposed boundary-guidee mixing trajectory method in Appendix More randomly selected and non cherry-picked experimental results, details about user study, and some failure cases analysis are shown in Appendix G
A Detailed Related Work
A.1 Denoising Diffusion Models
While we have briefly introduced the preliminaries on DDPMs [1] and DDIMs (4 in the main paper, we re-organize and present more details here. We note that the relevant background is mainly from the original papers, we only include the relevant background information to better illustrate our ideas in this work.
The key idea for generative tasks is to approximate a data distribution g( with a model learned distribution pe (2o) that can be easily sampled from. The original Denoising Diffusion Probabilistic Models (DDPMs) [53] propose to use latent variable models to fulfill the goal with the following specific form:
Pa : (d
where Ti,..., TT are variables modeled by the latent states of Markov chain, which have the same dimensionality as the actual data Ty g( Specifically, we have:
T
pa (To.r) == pe (xT))
II
Pb (Ie-
(8)
1=1
The training objective is the variational lower bound on negative log likelihood:
L = El-log Po(o))]
Po Po(20:1 (ZO:T)
$ E log q(I (Ta)
Po(m, a1wr)) (x,, (rt-1)
E,[-log pl (T))
log
The above formulation indicates that the DDPMs be learned with a pre-defined inference procedure q( In the case of (21), the authors propose to model the Markov chain with Gaussian transitions parameterized by decreasing sequence a1.7 E (0, 1)T follows:
q( (o) = II (() 1=1
(10)
where g(xx/=1-1) == NW
)I)
We often refer to the above-mentioned processes from Up to Ir and from Ty to Zp as forward process and reverse process (or generative process), respectively. Intuitively, the forward process adds noise to data To, while the reverse process denoises a noisy latent variable T1.7. The reverse denoising is stochastic based this formulation.
A.2 Marginal Discussion for Deterministic Inversion
Motivated to reduce the iteration numbers from the original DDPMs 53 21 Denoising Diffusion Implicit Models (DDIMs) (54) propose generalize the inference process (L.e., forward process) from
14

---

a Markov chain to Non-Markov one. The theoretical support for the proposed generalization lies within the fact the learning objective of DDPMs only depends on the conditional (on zp) marginals (Fre)ro), instead of the conditional (on rn) joint ((11:1]
Based on the previous fact, DDIMs consider family of inference distribution Q, indexed by a real vector a E 2220
T
qo (Z1: (o) == qo [TT 20) II do (11,1
(11)
where yo (ZT]20) = (VTTT (1- orr) Specifically, the qo (Ix-1 lxt, 2o) is carefully designed in way that the mean function satisfies the above Gaussian kernel as:
La (at--114,,0 mo) = MMVaaaiing + VI a4-1 o² It /2 .o
(12)
vI 04
Using the Bayes' rule, Eq. 12 can be further rewritten as:
ao (1)) = 9a (o To 40 (2)
(13)
le (T (T))
The above Eq. 12 and Eq. 13 show that the Non-Markov process da considered in DDIMs is marginal and also Gaussian (but not a standard one).
After having specified the forward process, DDIMs propose different variant of the sampling process where the model expected first predict the corresponding noiseless Zig given noisy observation It. and use the prediction to obtain - through Eq. 13] Specifically, the iteration can be written as follows:
VI
(4) orep (at)) Vou
31 = V
+ V a: 111---1-00 07000 (xt) + o
(14)
where ~ NNO. I). By choosing the = O for all steps I, the random noise induced by the last term from Eq. [4ii removed, and therefore changing the stochastic process from the original DDPMs formulation to deterministic one.
|
By connecting the Eq. 14 to the Euler integration for solving ordinary differential equations (ODEs), it can be further rewritten as:
(v1-000 1-041
Tt-1
Ft
+
04-1
a4)
e' c((x),
(15)
/a-1
Vas
at-1
ac
which is the Fuler solution for the following:
x(t)
d =/0
do
(16)
Therefore, when we adopt the deterministic inversion method to convert Zy to Zy,, we preserve the marginal property of the considered family of inference distribution Q. Intuitively, extending this distribution family to the d-dimensional space, it ensembles a group of Gaussian distributions parameterized by the defined sequence. Given the denoising process can be considered trajectory, the deterministic inversion follows and stays at the border of the space ensemble.
A.3 Other Related Works
Other studies related to this work include the areas of GANs inversion, image editing and manipula- tions.
GAN Inversion. GAN inversion problem 622 is proposed to tackle the lack of inference capability in GANs [6). As another powerful model other than DMs for data generation, many GAN models [27 9. 29 30 have been proposed for high-quality image synthesis. With high-level objectives to invert a given image and to apply it in downstream tasks like image editing, the two problems are often studied separately. Specifically, due the intractability of GAN generation, many works have been
15

---

focused solely on the first objective to invert an image to the latent space and to reconstruct from the latent encoding, which corresponds to the initial and primary goal of GAN inversion. There are three main technical directions for the inversion and reconstruction problem, which consists
of learning an additional deterministic encoder 770 61 3 directly solving the optimization
problem 1 2 25 12,
hybrid way that combines the above two techniques [69 7 6].
Different from existing GAN inversion works, we leverage the better tractability of DMs and use the deterministic property from the denoising diffusion implicit models (DDIMs) 54 to achieve the
inversion and reconstruction when studying the diffusion direction.
Image Manipulation and CLIP Guidance. Image manipulation based on generative models mainly covers two categories. While one branch of existing works often requires retraining of generative model (e.g., GANs [16)
42
59
36
5), others are studied as a downstream task application for For image manipulation using the GAN inversion technique,
GAN inversion works 69
29
prerequisite for effective editing is a disentangled understanding of latent spaces from pre-trained
GAN models. The analysis on the latent space addresses several different separate latent spaces
such as the 2 space for generic GANs [16] and the W space from StyleGAN [29]. The current
SOTA methods for diffusion-based editing like DiffusionCLIP 3I and Aysrp 35 all adopt the CLIP guidance as part of their loss function during the learning process.
In this work, we adopt similar semantic disentanglement idea as the tool to interpret and understand the latent space along the chain. At the same time, we are able to leverage our analysis and a better understanding of the latent space to achieve real-face image editing.
B High Dimensional Space
In this section, we provide the necessary theoretical foundations for understanding the geometric and probabilistic properties of nigh-dimensional spaces. The majority of the properties and lemmas we describe here are established theorems from nigh-dimensional space studies in mathematics and statistics from [8. We omit the detailed proof's for the following properties and lemmas, and kindly ask readers to refer to the original book if interested.
Property B.1. For a unit-radius sphere in high dimensions, as the dimension d increases, the volume of the sphere goes to O, and the maximum possible distance between two points stays at 2.
Lemma B.2. The surface area A(d) and the volume V V(d) of unit-radius sphere in d-dimensions are given by:
2T1/2 A(d) r(d/2) v(d)
4/2
(17)
(d//2)
2
where [(x) is a generalization of the factorial function for noninteger values of z.
The above Property B.T and Lemma B are generic geometric properties for high-dimensional spheres, but also applicable to high-dimensional Gaussian in which we are interested in the context of DDMs. To draw the connections with our context for studying the latent spaces of DMMs, with higher dimensionality, the latent Gaussian spaces of pre-trained DDMs become more difficult to operate due to decreased volume and mass concentration, as empirically suggested in [31 35)
Property B.3. The volume of a high-dimensional sphere is essentially all contained in a thin slice at the equator and simultaneously contained in narrow annulus at the surface, with essentially no interior volume. Similarly, the surface area is essentially all at the equator.
The Property B.3 implies the connection with the standard Gaussian in cr from direct sampling. In 2 (b) of the main paper where we illustrate the geometric and probabilistic properties of samples in the 6T space, the inverted ones locate in the inner border area of the narrow annulus, which is also empirically verified in our Tab. in the main paper. As those inverted latent encodings have smaller radius than the expected standard Gaussian case.
Lemma B.4. For any >> O, the fraction of the volume of the hemisphere above the plane x Vd-I
is less than _e- 2.
The above Lemma B.4 explains the volume range we show in Fig.2 (b) of the main paper in the left side of the Gaussian sphere to show the concentration mass, which is in the order of O r
(aa
16

---

Lemma B.5. The maximum likelihood spherical Gaussian for a set of samples is the one over center equal the sample mean and standard deviation equal to the standard deviation of the sample.
The above Lemma B.5 provides the theoretical justifications for using the mean of squared distance estimate the radius of Gaussian high-dimensional space.
C Markov Mixing
The mixing time defines a parameter that measures the time required by a Markov chain for the distance to stationary to be small [38]. The study of Markov mixing time aims to quantify the speed of convergence for Markov chains, and requires some other necessary preliminary knowledge on the total variance distance and the Convergence Theorem, which will briefly describe below.
Firstly, to quantify the convergence characteristic of Markov chains, an appropriate distance measure metric is prerequisite. In the literature, the total variation distance is the metric used to define the distance.
Definition C.1. The total variation distance between two probability distributions y and y on X' is defined by:
Fl u/|rV maxAgx//xAA) z(A4)]-
(18)
In the above definition, A is a probabilistic event, indicating that the distance between μ and is the maximum difference between probabilities assigned to a single event by the two distributions. This initial definition not very practical to estimate the actual distance, which further induces the following propositions.
Proposition C.2. Let μ and be two probability distributions on Then
|| v||TV
Σ TEX
lu(x) v(x)).
(19)
Proof.
Let B = (x p(x) 2 v(x)} and let A C X be any event. Then we have
μ(A) v(A) S y(An B) (B)))
(20)
The first inequality holds since any An B satisfies p(x) - vxx) < O, and thus the difference in probability cannot decrease when such elements of B are eliminated. For the second inequality, we note that including more elements of B can not decrease the difference in probability.
By the same reasoning, we have:
((A) u(A) < (BB) - -vBB).
(21)
The upper bounds on the right sides of Equation 20) and 21 are the same. Furthermore, by taking A B or A B then (μ(AA) v(A) is equal the upper bound. Therefore, arrive at:
1 |1 vllrv 2 (u(B) v(B) + (B") (BB))
1 Σ (l(x))u(x)) NEt
- v(=))-
(22)
The above proposition reduces total variance distance to a simple sum over the state space, which is an important theoretical support formulate our mixing step problem and empirical search method. The proof process also reveals the following remark.
Remark C.3. || TV = _mex,((n)<(()) [m(x) v(x)).
We then proceed to introduce the convergence theorem, which claims that aperiodic Markov chains converge to their stationary distributions at a key step, which is the direct thcoretical foundation for us introduce the mixing step problem for DDMs.
Theorem C.4. Convergence Theorem Suppose that P is irreducible and aperiodic, with stationary distribution π. Then there exit constants a E (0, 1) and C' > 0 such that:
max (
T||rv Ca'.
(23)
17

---

There exist multiple mathematical versions for the proof of the convergence theorem, which we omit in this appendix. Note that the assumptions for P to be irreducible and aperiodic are essential. We recall here the definition of an irreducible chain P.
Definition C.5. A chain P is called irreducible if for any two states z, X, there exists an integer t (possibly depending on x and y) such that P (z. y) O.
Intuitively, this means that it is possible to get from any state to any other state using only transitions of positive probability. This is verified in the current formulation of DDMs, indicating that DDMs satisfy the pre-requite be an irreducible Markov chain.
Next, we recall the definition of the period for a Markov chain.
Definition C.6. Let (=)) == (t 1t2:0 1 (r,,, > 0} be the set of times when is possible for the chain to return to starting position z. The period of state x is defined to be the greatest common divisor of (z)).
For an irreducible chain, the period of the chain is defined to be the period that is common to all states, and the chain is aperiodic if all states have period 1. Intuitively, the above definition and property match the actual formulation and implementations of DDMs, given the fact that plenty of existing DDMs 54
propose an auxiliary loss to predict directly the denoised 2 at arbitrary step.
Having introduced the above definitions, we are now ready to present the formal definition of mixing time in Markov chain studies.
Definition C.7. Definition of Mixing Time The mixing time is parameter that measures the time required by a Markov chain for the distance to the stationary distribution to be small, following the definition below:
(e) min(l: d(() S and Imiz :- Amxx (1/4).
(24)
In particular, taking above yields
d(ltmio) S 2 and (miz(() < [log2 e e tmiz.
(25)
In addition to the initial definition of mixing time, we also need the background knowledge on the time reversal to search for the actual mixing step in more practical way.
Definition C.8. For distribution y group G, the reversed distribution ju is defined by ji()) =
((g )) for all g E
The time reversal is directly related to the two-direction design of DDMs, and ensures that the mixing step remains at fixed position in two directions for both diffusion and generative processes. This property is also critical to better understand the DDMs, and provides theoretical justifications for us to search for the mixing step along the generative direction using the Gaussian radius estimation. In fact, the Gaussian radius estimation search method can only be valid and applied in the generative direction but not the inverse diffusion process. The reasons are discussed in the following section when we show more empirical results.
Lemma C.9. Let P be the transition matrix of random walk on group G with increment distribution μ and let P be that of the walk G with increment distribution ft. Let at be the uniform distribution on G. Then for any 2 0,
||Pt (id. * +lllv = [| - <||rv.
(26)
The lemma above implies the remark below, which will be used in our proof for the Property Remark C. Emir is the mixing time of a random walk on a group and fyiit is the mixing time of the reversed walk, then Emiz = tmiz-
We hereby finish introducing the necessary background on Markov mixing studies, and continue to a more detailed discussion of the mixing step problem of DDMs.
18

---

D More Discussion on Mixing Step
Inspired by the Markov mixing studies, we remark that the current formulation of DDMs satisfies several key assumptions as described in Appendix C including most importantly DDMs model an irreducible and aperiodic chain. Note that our current exploration and formulation for the mixing step of DDMs are not absolutely thorough and complete, which can be considered as an approximate and adapted version of the mathematical Markov mixing time.
D.1 Proof for Property of Mixing Step
We rewrite the Property of mixing step for DDMs here before going to the detailed proof. Property D.I. Under the total variation distance measure || llzv, the mixing step fm for a DDM with data dimensionality d is formed during training (i.e., irrelevant to the sampling methods). tm is mainly related to the transition kernels and the stationary distribution (i.e., datasets), and less dependant on the dimensionality d.
Proof. The proof for the above property consists of several steps.
Existence justification. Firstly, we have shown that DDMs model a group of chains that are irreducible and aperiodic, and thus the convergence theorem holds for DDMs. This fact establishes the theoretical foundation to find such critical convergent step that theoretically characterizes the convergence of pre-trained DDMs.
Directions to approach. Secondly, we show that the mixing step is large and mostly dependent on the transition kernel. Here, we have clarify the direction of the DDMs we are tackling. Fortunately, based on the time reversal from Lemma C.9 and Remark IC.0 whichever direction gives the same mixing step, provides us with the flexibility to study either direction. However, in practice, the easiest way to approach the mixing step is to theoretically infer the transition kernel in the diffusion direction, and then empirically search for it in the denoising direction. We will first provide the method and explain the reasons for such design.
Theoretical based transition kernel study. We hereby restrict ourselves in considering the diffusion process. Given pre-trained DDMs, according to Lemma C.9 we have an irredueible transition matrix P on space X. In the current scenario of diffusion direction from xy xy, the stationary distribution is the standard Gaussian (0, Ia) in ey. The transition matrix is a pre-defined Gaussian with known mean value and variance, thus we have
P = (x,, x) = := 11- 3,I).
(27)
For an irreducible transition matrix P with stationary distribution म, define:
or(,, P'(z, v)
(28)
with (x y) = (1, when P is reversible with respect to æ. We also have:
< <,, > Σ ()(,,))<()) =
(29)
Next, we have the definition of [P-distance (p) as:
d(e) (t) == max - 111p-
(30)
To replace the above notations with the notations from DDMs, we have:
( (t) == max max,ex|lo, y) 1/11;
(31)
and
Pt v)
x ~ N(x;; v1 Bi 3eT)
(x y)
(32)
π Tlv)
1~((0,,))
Based on the definition of mixing time in 24 we then have:
(1)
(E) inf > 0; d (t) <)
(33)
19

---

Table 4: Gaussian radius estimation for empirical search of the mixing step for pre-trained DDPM on CelebA-64.
Steps
1000
900
800
700
600
500
400
300
X7
Ps
110.84
110.82
110.83
110.58
109.83
107.85
103.22
94.64
A rl
0.02
0,01
0,25
0.75
1.98
4.63
8,58
X7 + Pi
110.88
110.86
110.84
110.63
109.89
107.86
103.10
94.45
A rl
0.02
0.02
0.21
0.74
2.03
4.76
8.65
XT + Pe
95.06
93.30
91.56
90.14
88.69
86.61
82.80
76.46
A
1.76
1.74
1.42
1.45
2.08
3.81
6.36
x Pi
95.06
93.34
91.61
90.16
88.75
86.57
82.87
76.53
A rl
1.72
1.73
1.45
1.41
2.18
3.70
6,34
We take the value be 1, and thus arrive at:
2'
mi inf(t 2 0, 01) 1
(34)
Now, we return back to Equation and replace the Equation 34 with:
Na1; -1 3,1) - -
max
(35)
(O)
By using the Proposition C.2 as follows:
we can now substitute the above Equation using the approximation
||x ((x /11 1, = 4.
(36)
This above gives an approximation of the transition kernel the mixing step the diffusion direction we would expect, with a radius change at approximately 4.
We observe that there is explicit dependency the dimensionality of the latent spaces, but directly related to the formulation of the transition kernel, which is mostly the Gaussian as used in existing DDMs implementations. In the meanwhile, note the intermediate latent encodings x; actually dataset dependent. Therefore, we verify our claim that the mixing step is more dependent on the transition kernel and dataset. However, despite no explicit dependency between the mixing step and dimensionality, we empirically observe that the appearance of mixing step still differs in pre-trained diffusion models on different resolutions as in Tab.
Interestingly, the above proof gives us a numerical approximation for the transition kernel when the mixing step appears, which is the radius variation approximately 4. We hereby finish demonstrating the fact that the mixing step appears at around diffusion step = 500 in our main paper. In the
meanwhile, other related works 31 35 report similar conclusions that editing on step 500 shows better empirical performance in different experimental settings.
D.2 More Empirical Results on Mixing Step
For the empirical verification of the mixing step, we use a pre-trained DDPM model 21 the CelebA dataset 339 with 3 x 64 x 64 resolution. Therefore, for standard Gaussian space in the dimensionality of d = 3 x 64 x 64 = 12, 288, the expected Gaussian radius is ovd = 110.85. The full radius estimation results are listed in Tab. 4 we also show the difference in Gaussian radius between consecutive 100 steps. Note we are slightly "abusing" the estimation results for steps after the mixing step, since the distributions of the latent spaces after 6,,, are no longer considered as Gaussian, but rather converge to the actual data distributions in X,, therefore, estimating the Gaussian radius of those latent spaces are not theoretically sound. This also explains the reason why we do not report the numbers for step numbers less than 300. The above also explains our design to derive the theoretical proof the diffusion direction, but proposes empirically search for the mixing step the denoising direction.
20

---

Table 5: More results on Gaussian radius estimation for the empirical search of the mixing step for pre-trained DDMs
Model
Setting
1000
900
800
700
600
500
400
DDPM-CClchA--QQ220
Xi +
443.42
443.36
443.30
442.49
440.16
432.55
416.77
0.06
0.06
2.33
7.61
15.78
DDPM--ellch--HQ2256
Xi
443.40
443.29
443.06
442.22
439.44
431.66
413.96
0.11
0.23
0.84
2.78
7.78
17.70
IDDPM+AFHQ-256
X7. +
443.34
443.21
442.84
441.72
439.31
429.16
408.69
A 0.13
0.37
1.12
2.41
10.15
20.47
D.3 Connection with Existing Works
We notice that existing SOTA methods [1 35 have proposed similar ideas in their works, by empirically exploring the diffusion steps that obtain better qualitative results. However, the mixing step has been studied influences the downstream qualities without formal definition. In this work, we formally define and introduce the concept of the mixing step. which originated from sound mathematical studies on the Markov mixing time, and provide a comprehensive perspective to re- re-think this More excitingly, we discover that the theoretically driven deviation and our Gaussian radius estimation method come consistent conclusion and echo with previous literature in actual experimental tests.
hyper-parameter (i.e., "return step" in (31 and "edit step" (35) that
E
Boundary Search Discussion
In this section, we present more details about our proposed boundary search method, and discuss the connections between different latent space levels from the perspective of the Projection theorem.
E.1 Projection Theorem
In theory, we expect the projected lower-dimensional subspace to preserve the same properties of its original higher-dimension space such the projected distances between pairs of samples should have the same ordering in two spaces. In mathematics, we can ensure the validity of this projection design using the existing projection theorems.
Theorem E. Theorem of the Random Projection. Let v be a fixed unit length vector a d- dimensional space and let W be random k-dimensional subspace. Let w be the projection of ho²
onto W. For any 0 S c S 1, Prob[]|w|" 2 e S
4e
One way to interpret the random projection theorem is that one chooses a random k-dimensional subspace from a higher- sigher-dimensional space in d-dimension, then indeed all the projected distances in the d-diiensional space are approximately within a known scale factor of the distances in the d-dimensional space.
present the projection theorem here to draw connections between the above boundary search and different operational latent space levels (i.e., e-space and h-space). As we describe in the main paper, the classification results from Tab. 6 show that even though the accuracy score is generally lower in e-space, it does carry meaningful semantic boundaries. This above observation and claim differ from the previous literature 44,, where the latent spaces of DDMs are considered to lack semantic meaning. In fact, given the recent study from [35], which first reveals the semantic behaviors of pre-trained DDMs in h-space, it provides evidence to imply that the same semantic meanings might also exist in the digees-dimensionall e-space. As h-space is subspace of corresponding c-space with higher dimensionality.
F BoundaryDiffusion with Mixing Trajectory
We show the algorithm implementation for our proposed BoundaryDiffusion with mixing trajectory under the conditional application scenario in Algo. [ For the unconditional scenario, the only difference that we can directly sample the latent encodings from the Gaussian distribution as the initial xy, and get the corresponding h-level latent encoding hy from the given DDPM at T step.
21

---

Algorithm BoundaryDiffusion (Conditional)
Input: input image Xp, target boundaries b. and b, for the editing attribute m, pre-trained DDM
p, inversion steps Sine. denoising steps Syen, mixing step tm. user defined editing distance Ç. and Sh. and editing space steps K.
// Step 1: Inversion via DDIMs to get the latent encoding at bm
Define {Tx) Sin s.t. 71 = 0, TS..v bm
a=1
for 1,2, 2,.... Sine do
6 x(x Ts)
Ta+1 Var
a
end for
hen, extract feature map from
I/ Step 2: Boundary guidance
I/ Step 2.1: Define initial editing space and h latent levels
{ s.t. de = -ce d.t = Ge
{d s.t. dr. = dit Sh
// Step 2.2: Compute projection distance to the boundaries
(dp.e} = {(de) by Xtm
(dp,h) } by....
Step3: Denoising with mixing trajectory
for k = 1, ... K do
x = =X,m + df b.
h h + di bb
X f- x
h, f- hi
for Sgens gen 1,....2 do
E f- ps Xxx 11,88
(00
x x,-1 Va
Xx 2--vI-are +
as-1 o² 0g
Var
end for end for
G More Experimental Results
We present more experimental results and discussion in this section.
G.1 Pre-trained Models and Datasets
The pre-trained DDMs we use for experiments mainly include the DDPM 21 and the improved DDPM (iDDPM) [II] The main difference between the original DDPM and the improved version lies within the fact that iDDPMs use hybrid learning objective that obtains better log-likelihoods than directly optimizing it.
We conduct experiments on multiple datasets, which includes CelabA-64 39] CelebA-HQ-256 2
AFHQ-dog-256 [10), LSUN-church-255 [66], LSUN- bedrooom-256 66 Different from existing works that usually pay little attention to the image resolutions in the experiments, the resolutions play an important role in our experiments since they define the actual dimensionality of the latent spaces for pre-trained DDMs. However, the 642 resolution model mainly used the high-dimensional analysis and interpolation observations, for the image editing and semantic control experiments, we use 2562 as the default resolution for visualization quality.
G.2 Semantic Boundary Validation
We search the semantic boundaries via linear classifiers on both c-space and h-space using 100 images, and we show the semantic behaviors via the testing classification accuracy on different attributes in Tab. 6 We observe from the classification results that the boundaries in /tspace are in general better defined compared to the e-space, which is consistent with previous findings from 35). Notably, for certain attributes such as glass and mustache, both space levels perform well in defining the
22

---

boundaries, which implies and aligns with our empirical finding that guidance on both levels of latent spaces helps for more effective semantic control.
Table 6: Classification accuracy separation boundaries in different latent spaces at the mixing step.
Lalent space
Smile
Glass
Age
Mustache
0.86
0.95
0.87
0.96
0.98
0.95
0.93
0.96
G.3 User Study
As subjective evaluations, we conduct a user study compare our proposed method with Asyrp 33 and DiffusionCLIP [3] on CelebA-HO-256 [39]. We use the official codebases from previous works and follow the exact default commands, using the smile attribute as the editing target, either add or remove smiles from 100 raw images that are randomly selected from the dataset. Our human evaluators include 20 graduate students volunteers, with 15 males and 5 females. Specifically, we show the original images together with edited images to the human evaluator and ask them to pick the best result with respect to general quality and attribute editing effectiveness.
More non-cherry picky editing results from three different methods are included in Fig. and Fig.
Fig. 7 shows more interpolation results from different combinations of latent encoding sources and sampling methods on CelebA-64 during our investigation on the distance effect. More qualitative samples for different editing settings included in Fig. 8
Fig. 10 and Fig. 11
23

---

image
Bounday Diffusion
Asyp [CC1R23]
DiflosinCLIP [CVPR22]
Raw image
Bounday Diffreden
Asep [ICLR23]
DillusiunCIPP [CVP
AII
US
S
Add smiling
Remove smiling
Figure 5: Non-Cherry-Pickee randomly selected results for add smiling and remove smiling editing operations from our proposed BoundaryDiffusion, Asyrp [35), and DiffusionCLIP B1.
24

---

Raw image
DiflusionCLIP
Boundarytifftusion
Raw image
DiflusionCLIP
Boundaaytiiltsinn
BKIn
Figure 6: More Non-Cherry-Picked randomly selected results for LSUN-Church dataset given "red brick church" editing text prompt. Comparing against learning-based SOTA method DiffusionCLIP, our learning-free method preserves better image details and achieve better fidelity and more natural editing effects.
25

---

x9 4
4 +.
+7
+
x)
Figure 7: More interpolation results from different combinations of latent encoding sources and sampling methods on CelebA-64.
streagh
Figure 8: More qualitative results for editing strength modification. We use the Cel2bA-HQ-256 dataset and the attributes glass and age as examples. In particular, also show samples with distortions and lower quality when the editing distance becomes too large. The optimal editing distance range is also related the properties of the high-dimensional spaces.
26

---

Figure 9: More qualitative results for text-based conditional editing on the LSUN-Bedroom-256 dataset.
Figure 10: More qualitative results for text-based conditional editing on the LSUN-Church-256 dataset.
27

---

Figure 11: More qualitative results for unconditional semantic control on AFHQ-Dog-256 and C6125A-HQ-226 datasets with smile semantic.
28


Reviews:
[{"review_id": "PM2bpqzQei", "reviewer": "Reviewer_5XEa", "summary": "This paper proposes Boundary Diffusion, a method for controlling the output of frozen pre-trained diffusion models. Boundary Diffusion contains two parts: (1) an analysis from the characteristics of Gaussian distribution to determine the key mixing step (2) a boundary guidance method to control the output of the diffusion models. Empirical results are presented to showcase the effectiveness of the proposed method.", "questions": "N/A", "limitations": "Limitations and broader impacts are properly discussed.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and the contents are easy to digest.\n\n2. The analysis based on the characteristics of Gaussian distribution is an interesting angle.\n\n3. The empirical results in Figure 1 is intriguing.", "weaknesses": "**1. Missing important baselines**: Since in the proposed method needs to train an SVM with noisy data x_{tm}, classifier guidance should be compared as an important baseline. Classifier guidance requires the user to train extra classifiers upon all the noisy data over all the time steps, which makes it more expensive than the proposed method, but the computational cost of SVMs are small in the deep learning era and I think it should be compared due to the great similarity with the proposed method. The computational cost of classifier guidance can be further reduced by applying it on a fraction of time steps (instead of all the 1000 steps). Anyway, it should not be ignored.\n\n**2. Need additional experiments**: the quantitative results are only available for CelebA+smile in Table 3. This makes readers doubt the generalization ability of the proposed method. Because the emphasis of this paper is empirical contribution rather than theoretical analysis, the authors should provide more quantitative results and comparison on different models, datasets and semantic attributes. For example, the LSUN-Church models.\n\n\nIf the authors could provide the required experimental results, I am happy to increase my ratings.\n\nTypos: L222, Tab 3 ==> Tab 2.\n\nAdditionally, the following recent work is related to this paper and the authors may be willing to discuss it as a concurrent work:\n\nFlowGrad: Controlling the Output of Generative ODEs With Gradients (https://openaccess.thecvf.com/content/CVPR2023/papers/Liu_FlowGrad_Controlling_the_Output_of_Generative_ODEs_With_Gradients_CVPR_2023_paper.pdf)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for the valuable comments, especially with the suggestion to include more experimental results to further enhance our paper.\n\n---\n- **Comment 1:** Baseline comparison with the classifier guidance.\n\n- **Response to Comment 1:** \n\n   We appreciate the comment. As suggested, we implement the CG following the original code [2], but with the **same base models trained on CelebA-HQ-256** to ensure a fair comparison, and report the quantitative results in Tab.1 below.\n\n   In terms of the computational cost, Classifier guidance [2] learns the gradients along the trajectory at each intermediate step to achieve the guided control. Conceptually it is similar to Asyrp [3] which modifies the intermediate latent variables via auxiliary neural networks. In our rebuttal, we train the classifier for approximately 30 mins with CG, similar to DiffusionCLIP [3] and Asyrp[4]. It is worth noting that our *BoundaryDiffusion* **does not train classifiers**, but rather **search** and **fit** the existing boundaries with SVM as hyperplanes in **negligible time about 1s**, which is also one of the key differences that distinguish our work from other learning-based methods since we are exploring the **intrinsic semantic properties** of pre-trained DMs formed in training. We believe this high-level take-away is critical since we proved that for a probabilistic generative model trained without semantic supervision, it forms meaning semantics by itself. On the contrary, researchers used to think that unconditionally trained generative models are not good at learning data semantics.\n\n| Method             | $S_{dir}$ $\\uparrow$     | SC $\\uparrow$        | ID $\\uparrow$       | FID  $\\downarrow$      |\n|--------------------|----------|-----------|----------|-----------|\n| ClassifierGuidance [2] | 0.15     | 86.5%     | 0.65     | 70.71     |\n| DiffusionCLIP [3]      | 0.17     | **93.7%** | 0.70     | 86.23     |\n| Asyrp [4]              | **0.19** | 87.9%     | -        | 68.38     |\n| FlowGrad+RF [1]       | -        | -         | **0.74** | -         |\n| *Our BoundaryDiffusion*  | 0.17     | 90.4%     | 0.73     | **68.14** |\nTab. 1 More baseline comparisons with additional baselines. Note among all the comparing methods, *BoundaryDiffusion* is the only learning-free method with **one-step** guidance operation.\n\n---\n- **Comment 2:** Need additional quantitative results. For example, LSUN-Church dataset.\n\n- **Response to Comment 2:**\n\n   We are happy to provide more quantitative results to demonstrate the effectiveness and SOTA performance of our method. As suggested, we report additional results on more datasets with different attributes. Note that the face identity similarity metric is no longer applicable to datasets other than human faces, as shown in Tab. 2, 3, and 4. In particular, we note our *BoundaryDiffusion* achieves higher segmentation consistency (SC) for LSUN-Church, partially due to the reason that we are modifying the latent encoding with **minimum shift** in latent space to cross the boundary, thus better preserves the original structure compared to learning based SOTA. We also included **more non-cherry-picky** qualitative examples for LSUN-Church in our one-page rebuttal for your reference, to ensure the effectiveness and SOTA performance as claimed.\n\n| Method            | $S_{dir}$  $\\uparrow$   | SC $\\uparrow$    | ID $\\uparrow$      |\n|-------------------|----------|-----------|----------|\n| DiffusionCLIP [3]    | 0.17     | **93.1%** | 0.63     |\n| Asyrp   [4]          | **0.18** | 88.3%     | 0.67     |\n| *Our BoundaryDiffusion* | **0.18** | 92.5%     | **0.69** |\nTab. 2 Results on CelebA-HQ-256, with attribute “add or remove glass”.\n\n\n| Method            | $S_{dir}$ $\\uparrow$    | SC  $\\uparrow$     |\n|-------------------|----------|------------|\n| DiffusionCLIP  [3]   | 0.96     | 65.02%     |\n| Asyrp          [4]   | **0.98** | 65.83%     |\n| *Our BoundaryDiffusion* | 0.97     | **66.01%** |\nTab. 3 Results on LSUN-Church-256, with example attribute “red brick church”.\n\n| Method            | $S_{dir}$  $\\uparrow$   | SC  $\\uparrow$       |\n|-------------------|----------|------------|\n| DiffusionCLIP [3]    | 0.91     | 64.82% |\n| Asyrp  [4]           | **0.94** | 62.65%     |\n| *Our BoundaryDiffusion* | **0.94** | **65.17%**     |\nTab. 4 Results on LSUN-Church-256, with example attribute “ancient church”.\n\n---\n- **Others:** [1]  is related to this paper and the authors may be willing to discuss it as a concurrent work.\n\n- **Response:** Thank you for the suggestion! We have read the FlowGrad paper and agree that it is a very interesting work related to ours and would like to discuss it in our final version! FlowGrad tackles the ODE-based generative models and proposes to control the generated output by controlling the back-propagated gradient to intermediate time steps on the ODE trajectory. \n\n   Essentially, FlowGrad features a method that optimizes the editing gradient along the ODE trajectory of pre-trained models. In that sense, we are similar in a way to seek control of the generative models in a more disentangled way. While FlowGrad proposed to do so via decomposing the back propagation process, we seek to track the gradual evolution of high-dimensional latent space along the trajectory.\n\n---\n[1] FlowGrad: Controlling the Output of Generative ODEs with Gradients. In CVPR 2023.\n\n[2] Diffusion Models Beat GANs on Image Synthesis. In NeurIPS 2021.\n\n[3] Diffusionclip: Text-guided diffusion models for robust image manipulation.\" In CVPR 2022.\n\n[4] Diffusion models already have a semantic latent space. In ICLR 2023.\n\n"}, {"review_id": "0id9nr7uTW", "reviewer": "Reviewer_EhMb", "summary": "This paper proposes a method for semantic control in image editing using pre-trained denoising diffusion models (DDMs) without the need for additional network modules. The method involves optimizing the denoising trajectory solely via frozen DDMs and exploring the critical step in the denoising trajectory that characterizes the convergence of a pre-trained DDM. The paper also presents a method to search for the semantic subspaces boundaries for controllable manipulation by guiding the denoising trajectory toward the targeted boundary at the critical convergent step. The paper leverages ideas in high-dimensional statistics, estimating the Gaussian radius and studying the difference between DDPM sampling and DDIM inversion. The method can perform semantic manipulation in one step and is extensively validated through experiments and datasets.", "questions": "* Regarding Equations 4 and 5, could you provide a more detailed explanation? Although the introduction is well-articulated, the presentation of these methods seems somewhat brief and rapid. Could you break them down further for better understanding?\n\n* As for Figure 3, could you specify what exactly you are 'mixing'? Furthermore, could you clarify the term 'mixing' within this context? What is its precise meaning in this scenario?", "limitations": "* Extensive experiments and grounded method. I do not see any relevant limitation to this method that can be applied to any pretrained DDPM.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper provides a comprehensive analysis of the latent spaces and the trajectory for pre-trained DDMs from a high-dimensional space perspective. This analysis can help better understand the behavior and properties of these models. Particularly interesting is the analysis of DDPM sampling vs DDIM inversion.\n\n* The authors propose a method to achieve semantic control and editing by guiding the denoising trajectory using the semantic boundaries in high-dimensional latent spaces with pre-trained and frozen DDMs. This approach leverages the stochasticity and the mixing step for better qualitative results and more effective manipulation. The method is grounded in theory and effective in practice.\n\n* The paper explores the potential of pre-trained diffusion models without learning any additional network model modules. This approach can be more efficient and less complex than methods that require additional learning.", "weaknesses": "* In this paper, certain assumptions are made about the structure of the latent space which are difficult to verify or justify. Hierarchical latent spaces are generally difficult to comprehensively analyze and validate due to their non-obvious, often high-dimensional, characteristics. \n\n* Furthermore, the terminology used for the methods, namely 'semantic boundary search' and 'trajectory mixing', creates an additional layer of complexity and is confusing. In particular trajectory mixing might be misunderstood or misinterpreted. You claim to perform semantic control in a single step but then your main method is called trajectory mixing. I would simplify or clarify the notation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the acknowledgement and the insightful questions, which we would like to include the clarifications in our final paper.\n\n---\n- **Comment 1:** In this paper, certain assumptions are made about the structure of the latent space which are difficult to verify or justify. Hierarchical latent spaces are generally difficult to comprehensively analyze and validate due to their non-obvious, often high-dimensional, characteristics.\n\n- **Response to Comment 1:** \n\n   While we believe the analysis from the novel perspective of high-dimensional latent space helps to build a more comprehensive and better understanding of diffusion models, we also agree with the reviewer that arbitrary high-dimensional distributions usually exhibit very complex spatial behaviors and characteristics in mathematics and statistics. \n\n   Luckily, Gaussian has many well-established properties that help us to better interpret DMs, which also motivates us to consider the Gaussian prior as a **reference** to understand DMs, and restrict our analytical studies mostly to the latent space of step $T$, which exhibits a **standard Gaussian by definition**. We are **being very prudent to make assumptions and in the choice of mathematical tools**. For instance, when we propose the automatic research method for the mixing step using the Gaussian radius as the metric, we indeed notice and explicitly state that the intermediate latent spaces do not exhibit standard Gaussian as the trajectory goes more toward real data space (see details in Appendix D.2), but the Gaussian radius remains to be a reasonable approximation to trace the latent spatial behaviors (turns out to be also pretty good as demonstrated in our empirical experiments in various downstream tasks by achieving strong SOTA performance) before the mixing step.\n\n   We do not boastfully claim to fully understand the DMs in this work, but we are also confident that it should help to build a better and more comprehensive understanding of diffusion generative models for interested researchers in the community, with abundant theoretical and empirical evidence.\n\n---\n- **Q1:** Regarding Equations 4 and 5, could you provide a more detailed explanation? Although the introduction is well-articulated, the presentation of these methods seems somewhat brief and rapid. Could you break them down further for better understanding?\n\n- **A1:** For sure. The overall idea of our *BoundaryDiffusion* consists of: \n\n   **i)** Locate and fit the semantic boundaries (characterized by their unit normal vector $\\bf n$ at the mixing step $t_m$ via SVM using a few labeled samples (the parameters from SVM characterize the boundary itself, the unit normal vector can be directly computed using zero dot-product). Note this is for locating the semantic boundaries that *formed and stayed invariant during the training* of DMs.\n\n   **ii)** Impose boundary-guided editing signals $d$ to a latent encoding $x_{t_m}$ at mixing step. Note the latent encoding can come from different sources at step $T$: for conditional semantic manipulation, it comes usually from an inverted image; for unconditional semantic control, it comes from a random sample of Gaussian. The denoising trajectory from $T$ to $t_m$ is deterministic following DDIM.\n\n   **iii)** Follow the remaining stochastic denoising trajectory to step $0$. \n\n   **For Eqn. 4**, it defines the editing signal $d$ with respect to an identified semantic boundary characterized by its unit normal vector $\\bf n$.\n\n   **For Eqn. 5**, it defines the entire mixing trajectory process with the mixing step $t_m$ as a separation point. Before $t_m$, the sampling/generation process follows a deterministic denoising trajectory via DDIM [1]. After $t_m$, the denoising trajectory is stochastic via the original DDPM [2].\n\n---\n- **Q2:** As for Figure 3, could you specify what exactly you are *mixing*? Furthermore, could you clarify the term *mixing* within this context? What is its precise meaning in this scenario?\n\n- **A2:**  Thank you for the question. There are two main reasons for us to use the term *mixing* within the context of *BoundaryDiffusion* in Fig. 3. **Firstly**, the editing operation we proposed is a one-step modification on the latent encoding at the **mixing step $t_m$**, so we find it appropriate to introduce the idea of the mixing step in the methodology design and naming. **Secondly**, we use two denoising techniques, namely the deterministic and stochastic, along the trajectory, which is also a **mixture** that relates to the idea of *mixing*.\n\n---\n[1] Denoising Diffusion Implicit Models. In ICLR, 2021.\n\n[2] Denoising diffusion probabilistic models. In NeurIPS 2020.\n\n\n"}, {"review_id": "QiA0zGRc9w", "reviewer": "Reviewer_sSmU", "summary": "The authors argue they can disentangle image attributes in the *e* and *h*-level latent spaces at the discovered mixing step $t_m$. The mixing step can be determined by \"the largest radius shift (e.g., 4)\" based on \"the mean of the squared distance of a latent sample from the origin (and following the generative trajectory) as the estimation of the Gaussian square.\" The semantic boundaries, editing direction vectors $\\mathrm{n}$, are found by applying SVM to classify the corresponding latent encodings using the label annotations of their corresponding inputs. The authors emphasize that the boundary guidance performs in the single step at $t_m$ without any extra fine-tuned network. They show competitive results compared with the other methods using finetuning or guidance supervision, while the proposed method does not use them both.", "questions": "Q1. In L70, omitted a space after \"segmentation consistency.\"\n\nQ2. In L105, normal text for real images $x_0$ instead of a vector representation $\\mathrm{x}_0$ for consistency. In L120, $x$ is the latent encodings. The inconsistency makes me confused.\n\nQ3. In L136, \"the stochastic DDPMs and the deterministic DDIMs\" may link to Appendix A.2. But the authors rarely refer to their corresponding Appendix sections. For DDIMs, $\\eta$ controls the degree of stochasticity. Why do you call \"deterministic DDIMs\"? Did you set $\\eta=0$?\n\nQ4. In L297, \"The quantitative evaluations are shown in (Table) 3\", missing \"Table\"?", "limitations": "None to mention.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1. Without finetuning with an extra network nor the supervised learning techniques, they show remarkable qualitative results compared with recent other works. \n\nS2. The method is simple yet effective. Without extra training procedures, downstream tasks may exhibit faster generations.", "weaknesses": "W1. The major concern of this work is clarity and reproducibility. Section 5 is the most critical part, but the authors did not allocate enough space to describe the key details of their method. Unfortunately, the background and motivation (Sec. 2-4) are also unclear due to the lack of rigorousness. And the authors should indicate the corresponding Appendix section number to refer to (I hope the authors did not circumvent the deadline in this way). Overall, the manuscript needs a major revision for the following issues (along with W2 and W3):\n\n* W1a. In L175-176, what do you mean by \"The spatial difference remains after the deterministic denoising via $p_i$ due to the homogeneity of intermediate latent spaces\" in the context? The contextual understanding is laborious here. Could you elaborate on this? The paper should be self-contained and self-explainable. You should restate the definition of \"homogeneity\" from Asyrp [22] (\"The same shift in this space results in the same attribute change in all images.\") and explain or elaborate on why the homogeneity and remaining spatial differences are connected. \n\n* W1b. In L181-182, which section in Appendix? And what is in there?\n\n* W1c. In Fig. 2, what is \"$0(\\frac{r}{\\sqrt{d}})$\"?\n\n* W1d. In L217-222, how do you define \"Gaussian square,\" \"squared distance,\" and \"the radius shift\"? There are ambiguities to resolve.\n\nW2. In L149-153, it is hard to understand how you define the radius $r$. In L151, the authors said, \"the radius of a high dimensional Gaussian space is defined as the square root of the expected squared distance: $r = \\sigma \\sqrt{d}$\"; however, the text and the equation are not matched. The radius is only determined by the \"expected squared distance\" $d$? But, $d$ is the dimension of a sample (L131). In L152, what is \"squared length\"? Do you mean the squared vector length of a sample vector $\\mathrm{x}$? \n\nW3. In L228-243, \"Semantic Boundary Search\" omitted major details. How do you select the \"semantic separation hyperplane\" to the corresponding annotations, e.g., \"smile\"? The authors failed to relate and indicate to refer to Appendix E. In Appendix E, Theorem E.1. has a typo $\\|$ or $|$ and a missing proof for the Theorem. Even though consulting with Appendix, the reproducibility seems to be infeasible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We have submitted our **anonymized code to AC to eliminate the reproducibility concerns** of the reviewer, and ensure that the reported results and performance are easily reproducible. We are also thankful for the careful reading and helpful suggestions on the writing, such as indicating the specific appendix sections in the main paper, which we would like to revise accordingly in the final version. We note that the reviewer is concerned with the ambiguous definitions of multiple metrics and concepts in high-dimensional Gaussian, we would like to politely clarify that all the concepts and calculations of adopted metrics such as Gaussian radius are **well established in mathematics [2]**, which we utilize as tools to better analyze and understand DMs, but **didn’t define or invent those by ourselves**.\n\n---\n- **Q1:** In L175-176, what do you mean by \"The spatial difference remains after the deterministic denoising via due to the homogeneity of intermediate latent spaces\" in the context? Explain or elaborate on why the homogeneity and remaining spatial differences are connected.\n\n- **A1:** This echoes the statement “The same shift in this space results in the same attribute change in all images” from [1]. In our context, the homogeneity means that spatial difference in terms of the Gaussian radius (i.e., inverted latent encodings have a smaller radius than sampled ones) along the trajectory, as shown in Tab. 2 in the main paper and Tab. 1 and 2 in the appendix.\n \n   The connection between the semantic homogeneity and the homogeneous spatial difference is due to the intrinsic nature of semantics formed during the training. In other words, the semantic boundaries in $\\epsilon$-space **remain invariant** after its formation in training at our introduced mixing step, which in return makes our BoundaryDiffusion method possible with **one single step** operation. Otherwise, if the homogeneity of semantics and the spatial boundaries vary, we would not be able to preserve the editing effect on target attributes with one-step editing.\n\n---\n- **Q2:** In L181-182, which section in Appendix? And what is in there?\n\n- **A2:** In Appendix B High Dimensional Space, we provide necessary theoretical foundations for understanding the geometric and probabilistic properties of high-dimensional Gaussian. In particular, we show the **theoretical support for the thin annulus pattern of concentration mass illustrated in Fig. 2**, and demonstrate the statistical significance of the spatial difference computed in Tab.1 and 2. \n\n---\n- **Q3:** In Fig. 2, what is $O(\\frac{r}{\\sqrt{d}})$?\n\n- **A3:** This is **the width of the slice** for the concentration mass, as stated in L178-179, “ the probabilistic concentration mass of a high-dimensional sphere is located within a thin slice at the equator”. This is an established property of high-dimensional Gaussian, with theoretical support given in Lemma B.4 in appendix.\n\n---\n- **Q4 & Q5:** In L217-222, how do you define \"Gaussian square\", \"squared distance,\" and \"the radius shift\"? There are ambiguities to resolve. In L149-153, it is hard to understand how you define the radius $r$.\n\n- **A4 & A5:** We didn’t define the above concepts by ourselves, they are **established metrics and concepts** in mathematics, as in the book [2]. Hope the explanation below will give a better definition of these metrics, and we are glad to include the clarification in our final version.\n\n   *Consider a $d$-dimensional Gaussian centered at the origin with variance $\\sigma^2$. For a point $\\bf{x} = (x_1, x_2, ..., x_d)$ chosen at random from the Gaussian, the expected squared length of $\\bf x$ is: $E(x^2_1+x^2_2+...+x^2_d)=dE(x^2_1)=d\\sigma^2$. For large $d$, the value of the squared length of $\\bf x$ is tightly concentrated about its mean. The square root of the expected squared distance (namely $\\sigma\\sqrt{d}$) is called Gaussian radius.* \n\n---\n- **Q6:** In L228-243, how do you select the \"semantic separation hyperplane\" to the corresponding annotations, e.g., \"smile\".\n\n- **A6:** As stated in L230, we use an SVM linear classifier to fit the classification boundary using a few annotated images. For example, we prepare 100 human face images with 50 annotated as smile and 50 as non-smile. We fit an SVM to classify those samples, then the parameters in SVM are the hyperplanes. We can further use those parameters to find the unit normal vector $\\bf n$ and use it as the directional guidance in Eqn. 4.\n\n---\n- **Q7:** In L136, \"the stochastic DDPMs and the deterministic DDIMs\". For DDIMs, $\\eta$ controls the degree of stochasticity. Why do you call \"deterministic DDIMs\"? Did you set $\\eta=0$?\n\n- **A7:** Yes, $\\eta$ controls the stochasticity, we use $\\eta=0$ for deterministic DDIMs.\n\n---\n- **Other suggestions on format including space and notations.**\n\n- We appreciate the suggestions and will revise accordingly.\n\n---\n[1] Diffusion models already have a semantic latent space. In ICLR, 2023.\n\n[2] Foundations of data science. Cambridge University Press, 2020.\n\n"}, {"review_id": "5CdLNgFu75", "reviewer": "Reviewer_TWGA", "summary": "This paper introduces the BoundaryDiffusion method for semantic image editing with frozen pre-trained Denoising Diffusion Models. The authors provide an analysis showing that the inverted latent variables have a different distribution from the prior Gaussian. From this observation, this paper suggests a method for identifying the critical step for semantic editing. This paper proposes a method for discovering semantic control perturbations through the linear SVM classifier of the corresponding semantics on this critical step.", "questions": "- **Q1-2. (L175-177)** The homogeneity of latent space in [1] was mainly about the semantic property of $h$-space. However, L175-177 discusses the spatial property (radius) of the inverted latent variable $x_{T}^{i}$ and the sampled latent variable $x_{T}^{s}$. Can we discuss the spatial property of $\\mathcal{X}$-space from the semantic homogeneity of $h$-space? If so, how can we understand the mixing step under the maintained spatial difference after the deterministic denoising $p_{i}$? Aren’t the inverted latent variables and sampled latent variables supposed to be close enough in the mixing step?\n\n- **Q3-4. (L180)** Lemma B.2 states that the volume of a *unit-radius* sphere decreases to near-zero as the dimensionality grows. However, the mass of high-dimensional Gaussian concentrates around the radius of $r=\\sigma\\sqrt{d}$. In this case, it appears that the volume of sphere, i.e., $V(d) r^{d}$ does not converge to zero when approximated with Stirling's formula. It would be beneficial if the authors could provide further clarification on this matter to enhance our understanding. Moreover, if the volume decreases to zero, isn’t it harder for the stochastic denoising to stay in the higher concentration mass (**L186-188**)?\n\n[1] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. In ICLR, 2023.", "limitations": "The authors addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- This work is overall well-written.\n- This work showed that $\\mathcal{X}$ also provides a semantic property.\n- The analysis in Sec 3 is interesting as it shows the difference between the inverted latent variables and the Gaussian samples.", "weaknesses": "- The BoundaryDiffusion does not require training a neural network. But, it requires the linear SVM classifier optimized from the annotated images.\n- The discussions of Distance effect in Sec 3 requires some clarifications (Please see my Questions below.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the feedback regarding the theoretical part of the distance effect.\n\n---\n**Comment 1:** *BoundaryDiffusion* does not require training a neural network but requires the linear SVM classifier optimized from the annotated images.\n\n**Response to Comment 1:** \n\n&nbsp;&nbsp;&nbsp; We are more of **searching** and **fitting** the existing boundaries with SVM as hyperplanes, rather than **learning classifiers** since we are exploring the **intrinsic semantic properties** of pre-trained DMs formed in training. Note this high-level take-away is critical since we proved that **an unconditionally probabilistic generative model learns meaningful semantics by itself**. This is **not evident** in previous literature where probabilistic generative models, including DMs, are conventionally believed to be not good at learning data semantics **without explicit semantic supervisions** such as text.  Also, our *BoundaryDiffusion* is much more computationally economical than learning-based SOTA [1, 2] with negligible fitting time. As for annotated images, [1,2] also require annotated images for fine-tuning or learning the editing neural networks. \n\n---\n**Questions on the distance effect:** \n\n- **Q1:** The homogeneity of latent space in [1] was mainly about the semantic property of $h$-space. L175-177 discusses the spatial property (radius) of the inverted and the sampled latent variable. Can we discuss the spatial property from the semantic homogeneity of $h$-space?  \n- **Q2:**  If so, how can we understand the mixing step under the maintained spatial difference after deterministic denoising $p_i$? Aren’t the inverted and sampled latent variables supposed to be close enough in the mixing step?\n- **Q3:** Lemma B.2 states that the volume of a unit-radius sphere decreases to near-zero as dim grows. However, the mass of high-dimensional Gaussian concentrates around the radius $r$. In this case, it appears that the volume of sphere does not converge to 0 when approximated with Stirling's formula. \n- **Q4:** If the volume decreases to 0, isn’t it harder for the stochastic $p_s$ to stay in the higher concentration mass?\n\n**Our responses:** \n\n&nbsp;&nbsp;&nbsp; Firstly, we would like to emphasize **an important precondition** for the studies on the distance effect: all the spatial/probabilistic properties and empirical analytical tools (*e.g.*, the mass concentration and radius) are built upon the **Gaussian distribution prior**. This is because an arbitrary high-dimensional distribution is extremely complex to parse and understand, while Gaussian has many well-established properties that help us to better interpret DMs. Therefore we consider the Gaussian prior as a **reference** to understand DMs, and restrict our studies mostly to the latent space of step $T$, which exhibits a **standard Gaussian by definition**.\n\n- **A1: We can discuss the semantic homogeneity but not the spatial property in $h$-space.** For the spatial property, it is unrealistic to conduct similar studies in $h$-space at step $T$, since $h$-space is extracted from the bottleneck layer of U-Net, and the Gaussian prior no longer holds after convolutional layers from U-Net, and thus we do not have a *reference or expectation* on what the spatial properties should be. However, we can indeed conduct the semantic homogeneity analysis in $h$-space using the SVM classifier as the tool and found that the semantic properties are preserved in $h$-space, as demonstrated in **Appendix G.2, Tab. 3**.\n\n- **A2: Distance effect is observed for inverted latent variables after distance shift following deterministic denoising; the mixing step is an intrinsic property formed during training to characterize the formation time of meaningful semantics.**\nTo observe the distance effect, there are two preconditions based on our experience: the departure latent encoding locates in a non high concentration area, and the denoising process does not bring it back along the trajectory. However, the mixing step is a slightly different concept with distance effect. Our high-level take-away for the mixing step is that this characterizes the time when the semantics start to form in a separable manner along the denoising trajectory, which is an intrinsic property formed in training given a pre-trained model.\n\n- **A3:  Please see the clarifications below.**\n\n   *i)* It is true that the mass of high-dimensional Gaussian concentrates around the $r$, but **within an annulus** of width $1/d$, as shown in right of Fig.2(b). Therefore, when approximating with Lemma B.2, we need to take the width into consideration.\n\n   *ii)* A simple way to check the formula from Lemma B.2: we have $A(2)=2\\pi$, $V(2)=\\pi$ and $A(3)=\\frac{2\\pi^{3/2}}{1/2\\sqrt{\\pi}}=4\\pi$, $V(3)=\\frac{2}{3}\\frac{\\pi^{3/2}}{\\Gamma(3/2)}=\\frac{4}{3}\\pi$. Note that $\\pi^{3/2}$ is an exponential in $\\frac{d}{2}$, and $\\Gamma(\\frac{d}{2})$ grows as the factorial of $\\frac{d}{2}$, which implies that $lim_{d\\to\\infty }V(d)=0$.\n\n   *iii)* In addition, essentially, we know the volume of concentration mass goes to zero for **an arbitrary high-dimensional distribution** according to the **curse of dimensionality**. \n\n- **A4: Yes and no.**\n\n   *i)* Yes, it is always harder to train deep learning models with data in higher dimensionality **in general**, which is also part of the curse of dimensionality. \n\n   *ii)* No because the DMs are trained with variational loss with their max probabilistic nature. If we are comparing $p_s$ and $p_i$, which are both **sampling process** that does not touch the training, $p_s$ is more consistent with training.  For DDIM $p_i$, the training is identical to DDPM $p_s$, and bears a slight quality decrease as explicitly discussed in its original paper [3].\n\n---\n[1] Diffusion models already have a semantic latent space. In ICLR, 2023.\n\n[2] Diffusionclip: Text-guided diffusion models for robust image manipulation.\" In CVPR, 2022.\n\n[3] Denoising Diffusion Implicit Models. In ICLR, 2021."}]

Author rebuttal:
### Overall responses to all reviewers and ACs

We thank all reviewers and ACs for the valuable feedback and for initiating insightful discussions. We also appreciate the acknowledgment of our work from both theoretical and experimental aspects, with the *theoretical analysis to better understand diffusion models from a novel perspective* (R-5XEa, R-EhMb, R-TWGA), and *strong experimental performance even without additional network learning* (R-5XEa, R-sSmU). (Note our **non-cherry-picky** results show better fidelity and more natural editing effects than existing learning-based SOTA methods due to minimum and effective shift in latent spaces.)

From our perspective as authors, our **key take-away/biggest contribution** in this work is: **we proved that an unconditionally trained generative diffusion model can well capture the data semantics in a unified latent space along the denoising trajectory and form invariant and separable boundaries at mixing steps.** 

This is **not evident** in previous literature where probabilistic generative models, including DMs, are conventionally believed to be not good at learning data semantics in unconditional settings **without explicit semantic supervisions** such as text (see examples with multiple popular text-to-image generation DMs). We demonstrate the above **strong but important claim** with both theoretical and empirical support via proposed analysis from the high-dimensional angle and *BoundaryDiffusion* method and believe this should greatly benefit our community with a better understanding of diffusion models. In addition, this should also help to **bridge the gap between generative and discriminative models**, with respect to their different emphasis to **learn distributions and discriminative decision boundaries**, and shall benefit even **discriminative downstream tasks** such as segmentation [1], which uses pre-trained DMs as representation learning models to extract semantic features.

We are also confident in the SOTA performance claim and have submitted our anonymized code to AC in a separate comment during rebuttal, and would like to invite any interested reviewers/AC to check and test the code to ensure reproducibility. 

We note different reviewers are intrigued with either theoretical or empirical aspects of our work, but we found both to be indispensable in proving the above claim and inspiring our methodological design of a **simple, elegant, effective yet non-intuitive** framework. 
To restate the contributions and novelties of our work, we make the table below as further clarifications.

|   Work  |        Study on the latent space     |         Study on mixing step   |    Methodology logic                           |      Applications    |      Auxiliary models     |     Trainable param.   |      Training time     |
|---------|----------------------------------------|-----------------------------|-----------------------|--------------------|--------------------|------------------------|-----------------|
| DiffAutoencoder [CVPR22 oral] |                No semantic meaning                      |                                                         Non                                                         |              Learn an attached autoencoders on $x$-level             |           Unconditional synthesis          |          MLP based autoencoder          |   Several million based on architecture   |                    ~28 GPU hours            |
|        DiffusionCLIP [CVPR22]       |                    Non                   |                Pure empirical, manual selection based on final output                             | Fine-tune the entire model on $x$-level, operates on iterative steps |            Guided image editing            | No, but fine-tune the pre-trained model | Same as base model, thus several millions |             ~20-30min for each attribute            |
|      Asyrp [ICLR2023 spotlight]     |                     Already have a semantic space, but only on $h$-level                     |                              Pure empirical, manual selection based on final output                             |                Learn editing signal on iterative steps               |            Guided image editing            |        Auxiliary editing networks       |                   ~800k                   |             ~20-30min for each attribute            |
|                Ours      | **Both $h$ and $x$ levels have semantic space**, can be well leveraged to control via our method | **Theoretical interpolation and deviation, automatic search via Gaussian radius  (thus not dependant on final output)** |             Mixing step boundary guidance,  **single step operation**             | **Unconditional synthesis + Guided image editing** |                    **No**                   |                    **0** (we are fitting existing hyperplanes instead of learning classifiers)                    | **Negligible, all attributes at the same latent space** |

---
[1] Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models, In CVPR 2023.

Meta-review: The authors have put significant effort into their rebuttal and there were intensive discussions between the reviewers. In the end, all critical reviewer concerns have been addressed and the reviewers now unanimously vote for acceptance.
-> accept. Congratulations!
When preparing your camera-ready, please take all reviewer comments into account as promised in the rebuttal.
Decision: Accept (poster)