SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ye Zhu $^{1,2}$ , Yu Wu $^{3}$ , Zhiwei Deng $^{4}$ , Olga Russakovsky $^{2}$ , Yan Yan $^{1}$ $^{1}$ Department of Computer Science, Illinois Institute of Technology  
 $^{2}$ Department of Computer Science, Princeton University  
 $^{3}$ School of Computer Science, Wuhan University  
 $^{4}$ Google Research  
yezhu@princeton.edu, wuyucs@whu.edu.cn, zhiweideng@google.com, olgarus@princeton.edu, yyan34@iit.edu
Applying pre-trained generative denoising diffusion models (DDMs) for downstream tasks such as image semantic editing usually requires either fine-tuning DDMs or learning auxiliary editing networks in the existing literature. In this work, we present our BoundaryDiffusion method for efficient, effective and lightweight semantic control with frozen pre-trained DDMs, without learning any extra networks. As one of the first learning-free diffusion editing works, we start by seeking a comprehensive understanding of the intermediate high-dimensional latent spaces by theoretically and empirically analyzing their probabilistic and geometric behaviors in the Markov chain. We then propose to further explore the critical step for editing in the denoising trajectory that characterizes the convergence of a pre-trained DDM and introduce an automatic search method. Last but not least, in contrast to the conventional understanding that DDMs have relatively poor semantic behaviors, we prove that the critical latent space we found already exhibits semantic subspace boundaries at the generic level in unconditional DDMs, which allows us to do controllable manipulation by guiding the denoising trajectory towards the targeted boundary via a single-step operation. We conduct extensive experiments on multiple DPMs architectures (DDPM, iDDPM) and datasets (CelebA, CelebA-HQ, LSUN-church, LSUN-bedroom, AFHQ-dog) with different resolutions (64, 256), achieving superior or state-of-the-art performance in various task scenarios (image semantic editing, text-based editing, unconditional semantic control) to demonstrate the effectiveness. Project page at https://l-yezhu.github.io/BoundaryDiffusion/.
The denoising diffusion models (DDMs) [53] have been successfully applied in various tasks such as image and video synthesis [21, 41, 22, 48, 14, 23, 52, 20], audio generation [33, 40, 37, 71], image customization [50, 64], reinforcement learning [26, 11], and recently in scientific applications [63, 34, 58]. Among various applications of DDMs, one popular downstream task is to use pre-trained unconditional DDMs for image manipulation and editing, by either re-training the diffusion models [31], or learning extra auxiliary neural networks for the editing signals [44, 35]. However, as these methods require additional learning processes, we argue that they have not yet fully leveraged the great potential of pre-trained DDMs. In this work, we show that the latent spaces of pre-trained DDMs<sup>1</sup>, when leveraged properly, can be directly used to perform various manipulation tasks without
<sup>1</sup>The latent spaces in our context are different from the concept of LDMs (a.k.a., StableDiffusion [48]). While the main idea of LDMs is to operate the diffusion denoising process on the latent space of raw data, we investigate the latent spaces represented by the intermediate diffusion states along the Markov chain.
Figure 1: We present the BoundaryDiffusion for efficient (single-step), effective and light-weight semantic control on multiple application scenarios with frozen pre-trained DDMs. Different from existing works, our BoundaryDiffusion is a learning-free method that achieves SOTA performance without learning any extra networks, in contrast to e.g. Asyrp [35], which requires training auxiliary editing neural networks, and DiffusionCLIP [31], which fine-tunes the pre-trained DDMs. We include more randomly selected non-cherry-picked results in Appendix G as qualitative demonstrations.
learning any extra networks and achieve superior or SOTA performance as shown in Fig. 1. This makes our work the first to achieve learning-free diffusion editing with unconditional base models that are trained without additional semantic supervision or conditioning.
The understanding of latent spaces has been a challenging yet critical factor for better explaining, interpreting, and utilizing diffusion models. On the one hand, the DDMs are formulated as a long Markov chain (usually with 1,000 diffusion steps) [21], which introduces the same number of generic latent spaces within a single model. In contrast to a single unified latent space in GANs [16], this unique formulation imposes extra difficulties for studying the latent spaces of DDMs, leading to additional research questions such as "which latent spaces should we focus on to analyze their behaviors?" On the other hand, abundant methodology designs for multiple downstream applications have been motivated by their corresponding understanding in terms of latent spaces [44, 31, 35]. For example, DiffAutoencoder [44] proposes to learn auxiliary encoders based on existing latent variables to obtain semantically meaningful representations for image attribute manipulations, mainly because the generic latent spaces in DDMs are initially believed to lack semantics. For the same reason, DiffusionCLIP [31] proposes to achieve semantic manipulation by fine-tuning the entire pre-trained DDMs each time given a new editing target attribute. More recently, Asyrp [35] observes that DDMs already form semantic spaces, but only in the bottleneck level of U-Net implementations [49],
referred to as  $h$ -space. Accordingly, they propose to learn an auxiliary editing network that operates on the  $h$ -space for each manipulation attribute. To this end, in this work, we feature our first branch of contributions in terms of latent spaces understanding from the following two aspects, by supplementing and correcting the current literature: i) Through the geometric and probabilistic analysis of the latent encodings from the departure space at diffusion step  $T$ , we demonstrate that the stochastic denoising and deterministic inversion (the process to convert image to noisy latent variables, which is essential for semantic image editing [31, 35]) trajectories along the Markov chain are asymmetric, as opposed to previous understanding [35] (see Sec. 3). ii) We further propose a theoretically supported automatic search method to identify the critical diffusion step (i.e., the mixing step) along the chain, which starts to exhibit semantics at the generic level (referred as  $\epsilon$ -space), as supplementary and/or correction to existing literature [44, 31, 35] (see Sec. 4 and Sec. 5).
Based on the better understanding of latent spaces, we then propose our BoundaryDiffusion method to achieve semantic control with pre-trained frozen and unconditional diffusion models in a learning-free manner in Sec. 5. Specifically, our approach first locates the semantic boundaries in the form of hyperplanes via SVMs within the latent space at the critical mixing step. We then introduce a mixing trajectory with controllable editing strength, which guides the original latent encoding to cross the semantic boundary at the same diffusion step to achieve manipulation given a target editing attribute as in Fig. 3. We also optimize the image quality via a combination of deterministic and stochastic denoising processes. We conduct extensive experiments using different DDMs architectures (DDPM [21], iDDPM [41]) and datasets (CelabA [39], CelebA-HQ [27], AFHQ-dog [10], LSUN-church [66], LSUN-bedroom [66]) for multiple applications (image semantic editing, text-based editing, unconditional semantic control), achieving either superior or SOTA performance in both quantitative scores ( $S_{dir}$  [45], segmentation consistency[65, 68, 67], face identity similarity [13] and FID [19]) and qualitative user study evaluations, as presented in Sec. 6. Compared to previous learning-based methods [31, 35], our BoundaryDiffusion method features a light-weight and unified single-step operation without any additional training cost. In addition, different from DiffusionCLIP [31] and Astrip [35], which take about  $30\mathrm{min}$  of learning time per attribute, our Boundary takes approximately negligible time about 1s and as few as 100 images to locate and identify the semantic boundaries.
Overall, our work contributes to both latent space understanding and technical methodology designs. Firstly, we provide multiple novel findings and analysis that correct or supplement the current understanding of latent spaces of DDMs. Notably, we reveal that the previous symmetric assumption of stochastic denoising and deterministic inversion directions does not hold after explicitly analyzing the geometric and probabilistic properties in the departure latent space at diffusion step  $T$ . We also demonstrate that the unconditionally trained DDMs exhibit meaningful semantics at our located mixing step along the Markov chain in generic  $\epsilon$ -space level. Secondly, our proposed BoundaryDiffusion method achieves semantic control in a learning-free manner, providing a promising direction for resources friendly and efficient downstream applications. The above contributions are further supported and validated by strong performance through extensive experiments and comprehensive evaluations. Code is available at https://github.com/L-YeZhu/BoundaryDiffusion.
We briefly describe essential background here, and include more detailed related work in Appendix A.
Denoising Diffusion Models. Denoising Diffusion Probabilistic Models (DDPMs) [21] are one of the principal formulations for diffusion generative models [53]. The core design of DDPMs consists of a stochastic Markov chain in two directions. The forward process adds stochastic noises (usually parameterized with Gaussian kernel) to a data sample  $x_0$  following:
$$
q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t - 1}\right) = \mathcal {N} \left(\sqrt {1 - \beta_ {t}} \mathbf {x} _ {t - 1}, \beta_ {t} \mathbf {I} _ {d}\right), \tag {1}
$$
where  $\{\beta_t\}_t^T = 1$  are usually scheduled variance.
The reverse direction, which corresponds to the generative process, denoises a latent sample  $\mathbf{x}_T$  (often sampled from a standard Gaussian distribution) to a data sample  $\mathbf{x}_0$  as:
$$
\mathbf {x} _ {t - 1} = \frac {1}{\sqrt {1 - \beta_ {t}}} \left(\mathbf {x} _ {t} - \frac {\beta_ {t}}{\sqrt {1 - \alpha_ {t}}} \epsilon_ {t} ^ {\theta} (\mathbf {x} _ {t})\right) + \sigma_ {t} z _ {t}, \tag {2}
$$
where  $\epsilon_t^\theta$  is the learnable noise predictor,  $z_{t}\sim \mathcal{N}(0,\mathbf{I})$ , and the variance of the reverse process  $\sigma_t^2$  is set to be  $\sigma^2 = \beta$ .
Alternatively, denoising diffusion implicit models (DDIMs) consider a non-Markovian process with the same forward marginals as DDPMs, with a slightly modified deterministic sampling process as follows:
$$
\mathbf {x} _ {t - 1} = \sqrt {\alpha_ {t - 1}} \left(\frac {\mathbf {x} _ {t} - \sqrt {1 - \alpha_ {t}} \epsilon_ {t} ^ {\theta} (\mathbf {x} _ {t})}{\sqrt {\alpha_ {t}}}\right) + \sqrt {1 - \alpha_ {t - 1} - \sigma_ {t} ^ {2}} \cdot \epsilon_ {t} ^ {\theta} (\mathbf {x} _ {t}) + \sigma_ {t} z _ {t}, \tag {3}
$$
where  $\sigma_{t} = \eta \sqrt{(1 - \alpha_{t - 1}) / (1 - \alpha_{t})}\sqrt{1 - \alpha_{t} / \alpha_{t - 1}}$ , as the  $\eta$  control the degree of stochasticity. Specifically, the first term indicates the directly predicted  $\mathbf{x}_0$  at arbitrary diffusion step  $t$ , and the second term denotes the direction pointing to  $\mathbf{x}_{t}$ , and the last term is the random noise.
In this work, we focus on the study of denoising diffusion models. While the pre-trained models are learned based on DDPMs, we adopt DDIMs for inverting real images  $\mathbf{x}_0\in \mathcal{X}$  to obtain their corresponding latent encodings  $\mathbf{x}_t\in \epsilon_t$  as in [31, 35]. However, different from conventional understanding, we reveal this inversion process is asymmetric to the stochastic denoising process from DDPMs in Sec. 3, which is an important fact to understand the reason why the previous works require extra learning to achieve the semantic editing effects.
Semantic Understanding and Trajectory in High Dimensional Space. Previous works on the semantic understanding of latent space for generative models have largely focused on variants of GANs [16], such as the  $\epsilon$  space for generic GANs [16],  $\mathcal{P}$  space from ProgressiveGAN [27], and  $\mathcal{W}$  space from StyleGAN [29]. More recently, [35] study the latent spaces of diffusion models and show that the pre-trained DDMs already have a semantic latent space at the bottleneck level of U-Net, and name it  $h$ -space. We observe that there exist two unique components for DDMs compared to previous studies for GANs [51, 12]: the relatively high-dimensionality of intermediate latent spaces and the trajectory along the Markov chain.
Firstly, unlike GANs that usually sample from a lower dimensional Gaussian distribution and upsample the latent encoding to the higher data space [16], DDMs [53] maintain the same dimensionality for all the intermediate latent encodings  $\mathbf{x}_{\{1:T\}}$  throughout the entire Markov chain. The above formulation regularizes the latent encodings to stay in higher dimensional spaces, which motivates us to tackle the problem using existing mathematical tools from high dimensional space theory [8], such as the concentration mass distribution and Gaussian radius estimation. In addition, DDMs model a random walk on the data space as a Markov stochastic process [56, 57, 32, 55, 24, 60], and form a trajectory as the diffusion step proceeds. This trajectory characterizes the property of the considered Markov and imposes nature discussion on the convergence question, which inspires us to draw the connection between the mixing time study in Markov chain [38] and diffusion models.
3 High-Dimensional Latent Spaces in Diffusion Models
Notations. Given a pre-trained DDM  $p$  (we omit  $\theta$  from common notation  $p_{\theta}$  given the fact that the parameters are frozen in this work) with  $T$  diffusion steps. We define the raw data space as  $\mathcal{X}$  and a data sample  $\mathbf{x}_0 \in \mathcal{X}$  in  $d$ -dimension. The intermediate latent variables  $\mathbf{x}_{\{1:T\}}$  are samples in the same dimensionality as  $\mathbf{x}_0$  from their corresponding spaces  $\epsilon_{\{1:T\}}$ . Specifically, as previous works suggest there are two methods to obtain the latent encodings, either from direct sampling as most generative works do [21, 14, 48, 33], or from a given raw data  $\mathbf{x}_0$  after inversion via DDIMs [31, 35]. We note  $\mathbf{x}_t^s$  and  $\mathbf{x}_t^i$  as the latent encodings at the  $t$ -th step from  $\epsilon_t$  via sampling and inversion, respectively. Similarly, we distinguish two sampling processes with the stochastic DDPMs and the deterministic DDIMs as  $p_s$  and  $p_i$ . Note the difference between DDPMs and DDIMs only exists in sampling process, while the training remains identical (i.e., a single  $p$  may be used in two sampling ways). We use  $\mathbf{x}_0'$  to represent the denoised image after editing.
Sampling vs. Inversion. We start the analysis of high-dimensional latent spaces of DDMs by considering different sources of latent encodings from the departure space  $\epsilon_T$  at the diffusion timestep  $T$ . Theoretically, the directly sampled latent encodings  $\mathbf{x}_T^s$  follow a standard Gaussian distribution  $\mathcal{N}(\mathbf{0},\mathbf{I}_d)$  by definition. In contrast, the inverted latent encodings, as they adopt the DDIMs formulation, do not follow an identical trajectory, but the marginal distribution of the forward DDPMs to go from the image space  $\mathcal{X}$  to the latent space  $\epsilon_T$ , leading to a non-standard Gaussian distribution different from the direct sampling. The above describes a critical statement that differs from existing work [35], where DDIMs-based inversion has been previously considered as a symmetric process to the generative trajectory.
Table 1: Estimation of the Gaussian radius  $r$  for sampled and inverted latent encodings in  ${\epsilon }_{T}$  from different pre-trained DDMs, datasets and image resolutions (64 and 256).
Models	Sampling	Inversion	△r
DDPM-CelebA-64	110.84	95.06	15.78
DDPM-LSUN-256	443.42	430.88	12.54
iDDPM-AFHQ-256	443.41	438.07	5.34
Table 2: 100-stepwise variation of Gaussian radius for different sample sources and sampling combinations using pre-trained DDM. With different latent encodings and sampling processes, the mixing step appears at approximately the same time around  $t = 500$ .
Steps	1000	900	800	700	600	500
xTs + ps	0.02	0.01	0.25	0.75	1.98	4.63
xTt + pi	0.02	0.02	0.21	0.74	2.03	4.76
xti + ps	1.76	1.74	1.42	1.45	2.08	3.81
xti + pi	1.72	1.73	1.45	1.41	2.18	3.70
We can also empirically demonstrate the above conclusion by estimating the radius  $r$  of the high-dimensional Gaussian space. In mathematics, the radius of a high dimensional Gaussian space is defined as the square root of the expected squared distance:  $r = \sigma \sqrt{d}$ . Specifically, given a  $d$ -dimensional Gaussian centered at the origin with variance  $\sigma^2$ . For a point  $\mathbf{x} = (x_1, x_2, \ldots, x_d)$  chosen at random from Gaussian, the expected squared length of  $\mathbf{x}$  is:
$$
E \left(x _ {1} ^ {2} + x _ {2} ^ {2} + \dots + x _ {d} ^ {2}\right) = d E \left(x _ {1} ^ {2}\right) = d \sigma^ {2}. \tag {4}
$$
For large  $d$ , the value of the squared length of  $\mathbf{x}$  is tightly concentrated about its mean, and the square root of the expected squared distance (namely  $\sigma \sqrt{d}$ ) is called Gaussian radius, denoted by  $r$ .
We calculate the mean of squared length from  $N$  samples from  $\mathbf{x}_T^s$  and  $\mathbf{x}_T^i$  as  $\frac{1}{N}\sum_{N}\sum_{j}^{d}((x_{T,j})^2)$ . For the sampled latent encoding from  $\epsilon_T\sim \mathcal{N}(\mathbf{0},\mathbf{I}_d)$ , the estimated radius  $r = \sqrt{d}$ , as shown in Tab. 1. In contrast, for the inverted latent encodings from real images, the previously expected radius does not hold. Interestingly, we note that the inverted latent encodings from iDDPMs [41] endure less radius shift compared to DDPMs [21], indicating that this radius estimation may imply the goodness of a pre-trained DDM. The above discussion on the distributions of  $\epsilon_T$  helps to better interpret the revealed "distance effect" below. More details about the marginal discussion via DDIMs inversion are included in Appendix A.
Distance Effect by Deterministic Inversion and Mitigation. As the next step, we study the influence of an asymmetric inversion trajectory by following the marginal distribution via DDIMs. Previous studies [31, 35] suggest that despite the latent encodings inverted via DDIMs allowing for near-perfect reconstruction, it is rather empirically difficult to directly edit  $\mathbf{x}_T^i$  for controlling the final denoised output. Similarly, we observe the phenomena that for the inverted latent encodings  $\mathbf{x}_T^i$ , a small modification often leads to severely degraded and distorted denoised images following the deterministic sampling  $p_i$ , as shown in the second row of Fig. 2(a). As a comparison, the directly sampled encoding  $\mathbf{x}_T^s$  shows a smooth transition after  $p_i$  (see the top row in Fig. 2(a)). This distance effect harms the performance in downstream applications that requires the inversion operation, such as image editing and manipulation tasks where the input is a given raw image  $\mathbf{x}_0$  instead of directly sampled latent encoding  $\mathbf{x}_T^s$ .
Having observed that the inverted latent encoding  $\mathbf{x}_T^i$  tends to suffer from distorted denoised images during the interpolation, we aim to further understand and interpret the distance effect from properties of the latent high-dimensional spaces. Intuitively, the sampled latent encodings  $\mathbf{x}_T^s$  locate in the area with higher probability concentration mass in  $\epsilon_T$  space, while the spatial locations of inverted ones  $\mathbf{x}_T^i$ , following the marginal diffusion trajectories, have relatively lower concentration mass. This spatial difference remains after the deterministic denoising via  $p_i$  due to the homogeneity of intermediate latent spaces [35], resulting in denoised samples from  $\mathbf{x}_T^i$  to be rather out-of-distribution and low-fidelity after distance shift in latent spaces. The homogeneity of latent spaces from [35] states that "the same shift in this space results in the same attribute change in all images", while in our context, we consider the spatial difference (i.e., inverted latent encodings have a smaller radius than sampled ones) in terms of Gaussian radius along the denoising trajectory.
From the geometric point of view, the probabilistic concentration mass of a high-dimensional sphere is essentially located within a thin slice at the equator and contained in a narrow annulus at the surface [8]. In addition, one needs to increase the radius of the sphere to nearly  $\sqrt{d}$  before there is a
Figure 2: Illustration of distance effect from the qualitative results and geometric properties. (a) The interpolation are conducted on CelabA-64, with different combinations for latent encoding sources and sampling methods. Following the same denoising process  $p_i$ , inverted latent encodings lead to distorted images. (b) The inversion process reverses images to the latent encodings at the area of less concentration mass with a smaller radius.  $\mathcal{O}\left(\frac{r}{\sqrt{d}}\right)$  is the width of the slice for the concentration mass.
non-zero concentration measure [8]. We explain the cause of the distance effect via the geometric illustration in Fig.2(b). More theoretical details can be found in Appendix B.
In contrast to the deterministic generation that suffers from the distance effect, the stochastic denoising is more robust and helps to alleviate the issue, as shown in Fig. 2(a). Similarly, previous works also reveal that stochasticity is beneficial for the quality of denoised data [28, 31, 35]. Therefore, adding stochasticity is a mitigation solution for the distance effect. From the perspective of spatial properties, the stochasticity brings the inverted latent encodings  $\mathbf{x}_T^i$  from the area with lower concentration mass to higher ones, thus making the denoising process more robust.
4 Mixing Step in Diffusion Models
Problem Formulation. In the mathematical and statistical study of the Markov process, the mixing time is defined as a key step  $t$ , at which the intermediate distribution converges to the stationary distribution [38]². Inspired by the established mathematical formulations, we introduce the mixing step problem for diffusion models as the search for the critical diffusion step  $t \in \{1, \dots, T\}$ , where the Gaussian distribution converges to the final data distribution in the reverse denoising process under some pre-defined distance measures (usually in total variation distance, noted as  $|| \cdot ||_{TV}$ ). It is worth noting that, unlike the distance effect which is induced by the deterministic inversion method, the mixing step is a characteristic carried by generic DDMs, as we show that the mixing step appears at the same time both for the directly sampled latent encodings and the inverted ones. In practice, we show in the experiments in Sec. 6 that imposing editing and manipulations on the mixing step is more effective for downstream tasks. Meanwhile, similar concepts have been empirically studied as a hyper-parameter as "return step" [31] or "editing step" [35] without further investigations into its theoretical meanings. Proof and details of the following property can be found in the supp.