SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Property 4.1. Under the total variation distance measure  $||\cdot ||_{TV}$ , the mixing step  $t_m$  for a DDM with data dimensionality  $d$  is formed during training (i.e., irrelevant to the sampling methods).  $t_m$  is mainly related to the transition kernels, the stationary distribution (i.e., datasets), and the dimensionality  $d$ . In practice, a radius shift of approximately 4 can be considered as a searching criterion for the mixing step in the denoising chain.
Automatic Search via Radius Estimation. While the rigorous theoretical deviation for the mixing step  $t_m$  depends on many factors and is a complex and non-trivial problem (see our supp. for details), we present here an effective yet simple method to search for the mixing step by estimating the radius  $r$  of latent high-dimensional Gaussian spaces. We can estimate the radius of the latent spaces along the denoising trajectory and show that the change of  $r$  (corresponding to the transition kernel in Property 4.1) reveals the appearance of the mixing step for a pre-trained DDM. Our proposed method automates the search for this critical diffusion step, while it has been previously determined by manually checking the quality of final denoised images [31, 35]. More detailed discussion and proof about the empirical search method can be found in Appendix D.
See more details about the Markov mixing time in Appendix C.
Figure 3: BoundaryDiffusion for semantic control in one-step editing. We propose to guide the initial deterministic generative trajectory via the semantic boundary at the mixing step  $t_m$  for image manipulation.
Specifically, we follow the procedure described in Sec. 3, where we calculate the mean of the squared distance of a latent sample from the origin as the estimation of the Gaussian square. We compute the radius of the latent encodings following the generative trajectory, the earliest step with the largest radius shift  $(\triangle r \approx 4)$  is an approximation of the mixing step. We show the change of radius estimation for different combinations of latent encoding sources and sampling methods, demonstrating the mixing step is a generic characteristic irrelevant to the above factors in Tab. 2.
5 BoundaryDiffusion for Semantic Control
In this section, we propose our BoundaryDiffusion method to achieve one-step semantic control and editing by guiding the denoising trajectory using the semantic boundaries in high dimensional latent spaces with pre-trained and frozen DDMs. Specially, we leverage the stochasticity and the mixing step  $t_m$  for better qualitative results and more effective manipulation.
Semantic Boundary Search. In the case of attribute manipulation with given annotations (e.g., face attributes modifications), we propose to search for the semantic separation hyperplane as a classification problem as in [51]. Specifically, we use the linear SVM [18] to classify the latent encodings using the label annotations of their corresponding raw data, and the parameters in SVM are used as the hyperplanes. The validation of using raw data ground truth annotation at the latent space level is guaranteed by the deterministic inversion and reconstruction via DDIMs [54]. We consider this application scenario to be conditional semantic editing, which allows us to edit a semantic attribute given a real raw image. It is worth noting that the objective of using SVM is to fit the existing semantic hyperplanes rather than learning classifiers or networks.
In addition to the above application scenario, we can further extend our semantic boundary search to more flexible settings such as text-based image editing by leveraging the popular large-scale cross-modality generative models such as DALLE-2 [46] and Imagen [20]. Particularly, we can easily generate high-fidelity images based on a given textual prompt and utilize the synthetic images as samples to search for the semantic boundary using the same classification method.
An extra application for our semantic boundary searching is to achieve semantic control for unconditional image synthesis, due to the reason that the semantic boundary is a feature learned and formed during the training. In contrast, other learning-based methods like DiffusionCLIP [31] that change the original parameters of pre-trained DMMs are not suitable for this downstream task.
Mixing Trajectory for Semantic Control. After having located the semantic boundaries, we propose to achieve semantic manipulation by guiding the trajectory via the identified boundaries using the projected distance control. We note the unit normal vector of the hyperplane as  $\mathbf{n} \in \mathbb{R}^d$ , and define the sign-sensitive "distance" of the latent encoding  $\mathbf{x}_{t_m}$  to the boundary similar to [51]:
$$
d \left(\mathbf {n}, \mathbf {x} _ {t _ {m}}\right) = \mathbf {n} ^ {T} \mathbf {x} _ {t _ {m}}. \tag {5}
$$
Meanwhile, we add the stochasticity after the editing the latent encodings at the mixing step  $t_m$ . Overall, the mixing trajectory with boundary guidance can be summarized as:
$$
p _ {m i x} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}\right) = \left\{ \begin{array}{l l} p _ {i} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}\right) & \text {i f} T \geq t > t _ {m} \\ p _ {s} \left(\mathbf {x} _ {t - 1} ^ {\prime} \mid \mathbf {x} _ {t} ^ {\prime}\right) & \text {i f} t _ {m} \geq t \end{array} \right. \tag {6}
$$
where  $\mathbf{x}_{t_m}' = \mathbf{x}_{t_m} + \zeta d(\mathbf{n},\mathbf{x}_{t_m})$ , and  $\zeta$  is the editing strength parameter that controls the projected distance to the target semantic boundary. We illustrate our proposed boundary-guided mixing trajectory method in Fig. 3. Note this operation can also be extended to multi-attribute semantic control, either by iteratively applying guidance to multiple boundaries (which is a linear operation), or by modifying Eq. 5 to  $d(\mathbf{N},\mathbf{x}_{t_m}) = \mathbf{N}^T\mathbf{x}_{t_m}$ , where  $\mathbf{N} = [\mathbf{n_1},\mathbf{n_2},\dots,\mathbf{n_m}]$  for  $m$  different attributes. The concrete algorithm of our proposed BoundaryDiffusion is presented in Appendix F.
Task Settings. We conduct our experiments on multiple variants of semantic control tasks, including real image conditioned semantic editing, real image conditioned text-based editing, and unconditional image synthesis with semantic control. Among those tasks, the first two scenarios are conditioned on given images and therefore involve the inversion process, while the last one is an extension application for image synthesis without given image input.
Model Zoo and Datasets. We test different pre-trained DDMs on various datasets with different image resolutions for experiments. Specifically, we test the DDPMs on the CelabA-64 [39], CelebAHQ-256 [27], LSUN-Church-256 and LSUN-Bedroom-256 [66]. We also experiment with pretrained improved DDPM [41] on the AFHQ-Dog-256 [10]. In particular, we emphasize here that the experiments with different image resolutions are critical in our work, especially for the high-dimensional space analysis since the resolutions represent the dimensionality of the latent spaces as shown in Tab. 1 and Fig. 2. However, in all the editing experiments, we use a resolution of  $256 \times 256$  as the default setting for better visualization quality.
Operational Latent Spaces. Previous works have either operated on the  $\epsilon$ -space [31] or on the  $h$ -space from the bottleneck layer of the U-Net [35], showing that  $h$ -space has better semantic meanings with homogeneity, linearity, and robustness [35]. In our main experiments, we impose the guidance on both latent encodings in both  $\epsilon$  and  $h$  levels. All the boundary guidance is imposed in a single-step at the mixing step  $t_m$ . We show in our experimental results later that the single-step operation is more effective at the  $\epsilon$ -level, while the  $h$ -level may require iterative guidance at multiple diffusion steps.
**Implementations.** We use the linear SVM classifier for searching the semantic boundary. We implement the SVM via the sklearn python package with the number of parameters equal to the total dimensionality of the latent spaces. For  $\epsilon$ -space, the dimensionality  $d_{\epsilon} = 3 \times 256 \times 256 = 196,608$ . For the  $h$ -space, the dimensionality depends on the pre-trained DDMs architecture implementation for the U-Net [49]. In our experiments, we use the same level of latent spaces as in [35], which have a dimensionality of  $d_h = 8 \times 8 \times 512 = 32,768$ . In practice, we observe approximately 100 images are sufficient for finding an effective semantic boundary. For the text-based semantic editing scenario, we use synthetic images generated from Stable Diffusion using text-prompt [48]. These image samples can be obtained via any other pre-trained text-to-image generative models.
In practice, the hyperplanes are found via linear SVMs [18], with almost negligible learning time of about 1 second on a single RTX3090 GPU. For the inference, the time cost remains at the same level as other SOTA methods. Specifically, by using the skipping step techniques, we can already generate high-quality denoised images using approximately 40-100 steps, which take from 1.682 - 13.272 seconds, respectively on a single RTX-3090 GPU.
Comparison and Evaluations. We mainly compare our proposed method with the most recent state-of-the-art image manipulation works with diffusion models: the DiffusionCLIP [31] and the Asyrp [35]. Both works consist of extra learning processes. Our evaluations include quantitative and qualitative assessments for different experimental settings, similar to most existing generation works. For the inversion and reconstruction part, since we adopt the same technique as in [31, 35] via DDIMs [54] and achieve near-perfection reconstruction results, we show qualitative samples in Fig. 1. As for the conditional editing, we show quantitative results on the CelebA-HQ using
Table 3: Evaluation results on CelebA-HQ-256 for real image conditioned semantic editing. FID scores are reported on the test set with 500 raw images, averaged on "add or remove smile" editing. The user study follows similar evaluation questions in [35].
Methods	Sdir↑	SC↑	ID↑	FID↓	User Quality↑	User Attribute↑
StyleCLIP [43]	0.13	86.8%	0.35	-	-	-
StyleGAN-NADA [15]	0.16	89.4%	0.42	-	-	-
DiffusionCLIP [31]	0.17	93.7%	0.70	86.23	3.2%	8.2%
Asyrp [35]	0.19	87.9%	-	68.38	41.3%	44.9%
Ours BoundaryDiffusion	0.17	90.4%	0.73	63.14	55.5%	46.9%
three metrics: Directional CLIP similarity  $(S_{dir})$  computed via CLIP [45], segmentation consistency (SC) computed via [65, 68, 67], and face identity similarity (ID) using [13]. We also report the FID scores [19] of edited images on the testing set of CelebA-HQ [27]. We further conduct human studies as subjective evaluations. Additionally, we also report the classification accuracy for the semantic boundary separation validation as in Appendix G.
6.2 Experimental Results
Conditional Semantic Manipulation. We show the qualitative results for conditional semantic manipulation on the CelebA-HQ-256 [27] using the pre-trained DDPM [21] in the top row of Fig. 1. The quantitative evaluations are shown in Tab. 3, where we achieve SOTA performance. In particular, we emphasize the fact both SOTA methods [31, 35] directly use the directional CLIP loss as part of the loss function to align the direction between the embeddings of the reference and generated image in the CLIP space [45] in their learning processes, which helps to increase the score of  $S_{dir}$ . Additionally, they also optimize the identity loss  $\mathcal{L}_{id}(x_0', x_0)$  to preserve the facial identity consistency between the original image  $x_0$  and the edited one  $x_0'$ , thus boosting the ID score. While our proposed BoundaryDiffusion method does not use any learning loss function for semantic control, yet achieves very competitive scores in all the metrics.
Text-Guided Semantic Manipulation. With the success of large-scale cross-modality generative models [46, 20], we are able to extend the proposed BoundaryDiffusion method to other downstream tasks such as text-guided semantic manipulations. Qualitative examples are presented in the middle part of Fig. 1, where we show the original image input, reconstruction results, and the edited samples. We also perform experiments on the unseen domain transfer and show the example of editing a dog to a fox, as indicated using the red text prompt in Fig. 1. We note one limitation of our BoundaryDiffusion is relatively difficult for unseen domain editing tasks compared to other learning-based methods, which aligns with the expectation since the frozen DDMs do not have the knowledge for unknown domain distributions.
Unconditional Semantic Control. One unique application scenario we present in the bottom row of Fig. 1 is to apply the identified semantic boundary to the unconditional synthesis setting, where the departure latent encoding  $\mathbf{x}_T$  is now directly sampled from a standard Gaussian without a given real image as input. Since the semantic boundary is an intrinsic characteristic formed during the training process of DDMs, we can easily re-use it for semantic control under a general unconditional denoising trajectory. In comparison, the existing SOTA methods that modify the parameters of pre-trained DDMs during additional learning are no longer applicable to this task.
Ablation on Operational Latent Spaces and Editing Strength. We can easily control the editing strength via the editing distance as defined in Sec. 5. Fig. 4 shows an example for editing the smile attribute on CelebA-HQ. We use the hyper-parameter  $\zeta$  to control the degree of modification. From the same figure, we observe that we achieve a stronger editing effect at the same distance when applying the boundary guidance to both latent levels at  $\epsilon_{t_m}$  and  $h_{t_m}$ .
User Study. We also conducted a subjective user study to compare the performance with Diffusion-CLIP [31] and Asyrp [35], where we interviewed 20 human evaluators to rank the edited images in terms of general quality and attribute editing effects. Specifically, we asked the evaluators to pick the best-edited result in terms of two aspects: 1). General quality: which image quality do you think is the best (clear, fidelity, photorealistic)? 2). Attribute: which image do you think achieves the best attribute editing effect (natural, identity preservation with respect to the given raw image)? Tab 3
(a) Editing results on both latent levels from strength distance range of [-150, 150]
(b) Editing results on single generic  $\epsilon$  latent level from strength distance range of [-150, 150]
(c) Editing results on single  $h$  latent level from strength distance range of [-150, 150]
Figure 4: Ablation on editing strength control via distance with different operational spaces. We can modify the editing strength by changing the distance between the latent encoding  $\mathbf{x}_{t_m}$  to the target boundary. We show that the one-step guidance is more effective when applied in the  $\epsilon$ -level space. Images in green boxes denote the given raw image.
shows the percentage of each method picked as the best result. Details about the user study and more randomly selected, non-cherry-picked qualitative samples are included in Appendix G.
Our work explores and leverage the great potential of pre-trained unconditional diffusion models without learning any additional network model modules. We provide a rather comprehensive analysis from the high-dimensional space perspective to better understand the latent spaces and the trajectory for pre-trained DDMs. We introduce the mixing step for DDMs, which characterizes the convergence of generic DDMs, and present a simple yet effective method to search for the mixing step via Gaussian radius estimation in high-dimensional latent spaces. Last but not least, we propose BoundaryDiffusion, an effective learning-free method with boundary-guided mixing trajectory to realize effective semantic control and manipulation for downstream tasks using multiple pre-trained DDMs and datasets.
Broader Impact and Limitation. We discuss the broader impact of this work. Firstly, the primary goal of this work is not to create new generative models or generate synthetic data, but to explore the potential of the current generative models for better usage. To do so, we also propose a new perspective to better understand and interpret the DDMs, which is the analysis of high-dimensional latent space behaviors using the theoretical tools from mathematics and statistics. In the meanwhile, during the process of exploring and separating the semantic boundary, we leverage the current popular cross-modality generative models to synthesize images with a text prompt. However, all the generated images are only used for boundary detection. We believe our work brings valuable insights to the research community in terms of a better understanding and further exploration via training-free methods to apply diffusion generative models.
As for limitations, since our BoundaryDiffusion method is learning-free and does not introduce any extra parameters to the base diffusion models, the ability for unseen domain image editing is relatively limited compared to other learning-based methods.
This work is supported by NSF IIS-2309073. This article solely reflects the opinions and conclusions of its authors and not the funding agency. The first author of this paper would also like to thank Dr. Alain CHILLÉS for the insightful discussions on the theoretical and mathematical analysis.
We present the detailed related work about the denoising diffusion models including the marginal discussion in Appendix A. The high-dimensional space properties and lemmas are introduced in Appendix B, and we explicitly describe how those established theoretical theorems are used and connected to our analysis in the main paper Sec. 3. In Appendix C, we provide the theoretical foundations of the Markov mixing study, which inspires us to formulate the mixing step problem for DDMs. More details about the mixing step problem, formulation, proof and discussion are included in Appendix D. Appendix E includes details about our semantic boundary search method and further discussions in terms of two latent space levels (e.g., generic  $\epsilon$ -space and  $h$ -space from the U-Neu bottleneck [35]). We show the algorithm of our proposed boundary-guided mixing trajectory method in Appendix F. More randomly selected and non cherry-picked experimental results, details about user study, and some failure cases analysis are shown in Appendix G.
A Detailed Related Work
A.1 Denoising Diffusion Models
While we have briefly introduced the preliminaries on DDPMs [21] and DDIMs [54] in the main paper, we re-organize and present more details here. We note that the relevant background is mainly from the original papers, we only include the relevant background information to better illustrate our ideas in this work.
The key idea for generative tasks is to approximate a data distribution  $q(x_0)$  with a model learned distribution  $p_{\theta}(x_0)$  that can be easily sampled from. The original Denoising Diffusion Probabilistic Models (DDPMs) [53] propose to use latent variable models to fulfill the goal with the following specific form:
$$
p _ {\theta} := \int p _ {\theta} \left(x _ {0: T}\right) d x _ {1: T}, \tag {7}
$$
where  $x_{1}, \ldots, x_{T}$  are variables modeled by the latent states of a Markov chain, which have the same dimensionality as the actual data  $x_{0} \sim q(x_{0})$ . Specifically, we have:
$$
p _ {\theta} \left(x _ {0: T}\right) := p _ {\theta} \left(x _ {T}\right) \prod_ {t = 1} ^ {T} p _ {\theta} ^ {(t)} \left(x _ {t - 1} \mid x _ {t}\right). \tag {8}
$$
The training objective is the variational lower bound on negative log likelihood:
$$
\begin{array}{l} L := \mathbb {E} \left[ - \log p _ {\theta} \left(x _ {0}\right) \right] \\ \leq \mathbb {E} \left[ - \log \frac {p _ {\theta} \left(x _ {0 : T}\right)}{q \left(x _ {1 : T} \mid x _ {0}\right)} \right] \tag {9} \\ = \mathbb {E} _ {q} \left[ - \log p \left(x _ {T}\right) - \sum_ {t \geq 1} \log \frac {p _ {\theta \left(x _ {t - 1} \mid x _ {t}\right)}}{q \left(x _ {t} \mid x _ {t - 1}\right)} \right]. \\ \end{array}
$$
The above formulation indicates that the DDPMs can be learned with a pre-defined inference procedure  $q(x_{1:T}|x_0)$ . In the case of [21], the authors propose to model the Markov chain with Gaussian transitions parameterized by a decreasing sequence  $\alpha_{1:T} \in (0,1]^T$  as follows:
$$
q \left(x _ {1: T} \mid x _ {0}\right) := \prod_ {t = 1} ^ {T} q \left(x _ {t} \mid x _ {t - 1}\right), \tag {10}
$$
where  $q(x_{t}|x_{t - 1})\coloneqq \mathcal{N}(\sqrt{\frac{\alpha_{t}}{\alpha_{t - 1}}} x_{t - 1},(1 - \frac{\alpha_{t}}{\alpha_{t - 1}})\mathbf{I}).$
We often refer to the above-mentioned processes from  $x_0$  to  $x_T$  and from  $x_T$  to  $x_0$  as forward process and reverse process (or generative process), respectively. Intuitively, the forward process adds noise to data  $x_0$ , while the reverse process denoises a noisy latent variable  $x_{1:T}$ . The reverse denoising is stochastic based on this formulation.
A.2 Marginal Discussion for Deterministic Inversion
Motivated to reduce the iteration numbers from the original DDPMs [53, 21], Denoising Diffusion Implicit Models (DDIMs) [54] propose to generalize the inference process (i.e., forward process) from
a Markov chain to a Non-Markov one. The theoretical support for the proposed generalization lies within the fact the learning objective of DDPMs only depends on the conditional (on  $x_0$ ) marginals  $q(x_t|x_0)$ , instead of the conditional (on  $x_0$ ) joint  $q(x_{1:T}|x_0)$ .
Based on the previous fact, DDIMs consider a family of inference distribution  $\mathcal{Q}$ , indexed by a real vector  $\sigma \in \mathbb{R}_{\geq 0}^{T}$ :
$$
q _ {\sigma} \left(x _ {1: T} \mid x _ {0}\right) := q _ {\sigma} \left(x _ {T} \mid x _ {0}\right) \prod_ {t = 2} ^ {T} q _ {\sigma} \left(x _ {t - 1} \mid x _ {t}, x _ {0}\right), \tag {11}
$$
where  $q_{\sigma}(x_T|x_0) = \mathcal{N}(\sqrt{\alpha_T} x_0, (1 - \alpha_T)\mathbf{I})$ . Specifically, the  $q_{\sigma}(x_{t-1}|x_t,x_0)$  is carefully designed in a way that the mean function satisfies the above Gaussian kernel as:
$$
q _ {\sigma} \left(x _ {t - 1} \mid x _ {t}, x _ {0}\right) = \mathcal {N} \left(\sqrt {\alpha_ {t - 1}} x _ {0} + \sqrt {1 - \alpha_ {t - 1} - \sigma_ {t} ^ {2}} \frac {x _ {t} - \sqrt {\alpha_ {t}} x _ {0}}{\sqrt {1 - \alpha_ {t}}}, \sigma_ {t} ^ {2} \mathbf {I}\right). \tag {12}
$$
Using the Bayes' rule, Eq. 12 can be further rewritten as:
$$
q _ {\sigma} \left(x _ {t} \mid x _ {t - 1}, x _ {0}\right) = \frac {q _ {\sigma} \left(x _ {t - 1} \mid x _ {t} , x _ {0}\right) q _ {\sigma} \left(x _ {t} \mid x _ {0}\right)}{q _ {\sigma} \left(x _ {t - 1} \mid x _ {0}\right)}. \tag {13}
$$
The above Eq. 12 and Eq. 13 show that the Non-Markov process  $q_{\sigma}$  considered in DDIMs is marginal and also Gaussian (but not a standard one).
After having specified the forward process, DDIMs propose a different variant of the sampling process where the model is expected to first predict the corresponding noiseless  $x_0$  given a noisy observation  $x_{t}$ , and use the prediction to obtain  $x_{t - 1}$  through Eq. 13. Specifically, the iteration can be written as follows:
$$
x _ {t - 1} = \sqrt {\alpha - 1} \left(\frac {x _ {t} - \sqrt {1 - \alpha_ {t}} \varepsilon_ {\theta} ^ {(t)} \left(x _ {t}\right)}{\sqrt {\alpha_ {t}}}\right) + \sqrt {1 - \alpha_ {t - 1} - \sigma_ {t} ^ {2}} \varepsilon_ {\theta} ^ {(t)} \left(x _ {t}\right) + \sigma_ {t} \varepsilon_ {t}, \tag {14}
$$
where  $\varepsilon_t \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ . By choosing the  $\sigma_t = 0$  for all steps  $t$ , the random noise induced by the last term from Eq. 14 is removed, and therefore changing the stochastic process from the original DDPMs formulation to a deterministic one.
By connecting the Eq. 14 to the Euler integration for solving ordinary differential equations (ODEs), it can be further rewritten as: