SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {x _ {t - 1}}{\sqrt {\alpha_ {t - 1}}} = \frac {x _ {t}}{\sqrt {\alpha_ {t}}} + \left(\sqrt {\frac {1 - \alpha_ {t - 1}}{\alpha_ {t - 1}}} - \sqrt {\frac {1 - \alpha_ {t}}{\alpha_ {t}}}\right) \epsilon^ {t} (x _ {t}), \tag {15}
$$
which is the Euler solution for the following:
$$
d \bar {x} (t) = \varepsilon_ {\theta} ^ {(t)} \left(\frac {\bar {x} (t)}{\sqrt {\sigma^ {2} + 1}}\right) d \sigma (t). \tag {16}
$$
Therefore, when we adopt the deterministic inversion method to convert  $x_0$  to  $x_T$ , we preserve the marginal property of the considered family of inference distribution  $\mathcal{Q}$ . Intuitively, extending this distribution family to the  $d$ -dimensional space, it ensembles a group of Gaussian distributions parameterized by the defined  $\alpha$  sequence. Given the denoising process can be considered as a trajectory, the deterministic inversion follows and stays at the border of the space ensemble.
A.3 Other Related Works
Other studies related to this work include the areas of GANs inversion, image editing and manipulations.
GAN Inversion. GAN inversion problem [62] is proposed to tackle the lack of inference capability in GANs [16]. As another powerful model other than DMs for data generation, many GAN models [27, 9, 29, 30] have been proposed for high-quality image synthesis. With high-level objectives to invert a given image and to apply it in downstream tasks like image editing, the two problems are often studied separately. Specifically, due to the intractability of GAN generation, many works have been
focused solely on the first objective to invert an image to the latent space and to reconstruct from the latent encoding, which corresponds to the initial and primary goal of GAN inversion. There are three main technical directions for the inversion and reconstruction problem, which consists of learning an additional deterministic encoder [70, 47, 61, 3], directly solving the optimization problem [1, 2, 25, 12], or a hybrid way that combines the above two techniques [69, 7, 6].
Different from existing GAN inversion works, we leverage the better tractability of DMs and use the deterministic property from the denoising diffusion implicit models (DDIMs) [54] to achieve the inversion and reconstruction when studying the diffusion direction.
Image Manipulation and CLIP Guidance. Image manipulation based on generative models mainly covers two categories. While one branch of existing works often requires retraining of a generative model (e.g., GANs [16]) [42, 59, 36, 5], others are studied as a downstream task application for GAN inversion works [69, 51, 29, 1]. For image manipulation using the GAN inversion technique, a prerequisite for effective editing is a disentangled understanding of latent spaces from pre-trained GAN models. The analysis on the latent space addresses several different separate latent spaces such as the  $\mathcal{Z}$  space for generic GANs [16] and the  $\mathcal{W}$  space from StyleGAN [29]. The current SOTA methods for diffusion-based editing like DiffusionCLIP [31] and Aysrp [35] all adopt the CLIP guidance as part of their loss function during the learning process.
In this work, we adopt a similar semantic disentanglement idea as the tool to interpret and understand the latent space along the chain. At the same time, we are able to leverage our analysis and a better understanding of the latent space to achieve real-face image editing.
B High Dimensional Space
In this section, we provide the necessary theoretical foundations for understanding the geometric and probabilistic properties of high-dimensional spaces. The majority of the properties and lemmas we describe here are established theorems from high-dimensional space studies in mathematics and statistics from [8]. We omit the detailed proofs for the following properties and lemmas, and kindly ask readers to refer to the original book if interested.
Property B.1. For a unit-radius sphere in high dimensions, as the dimension  $d$  increases, the volume of the sphere goes to 0, and the maximum possible distance between two points stays at 2.
Lemma B.2. The surface area  $A(d)$  and the volume  $V(d)$  of a unit-radius sphere in  $d$ -dimensions are given by:
$$
A (d) = \frac {2 \pi^ {d / 2}}{\Gamma (d / 2)}, V (d) = \frac {\pi^ {d / 2}}{\frac {d}{2} \Gamma (d / 2)}, \tag {17}
$$
where  $\Gamma(x)$  is a generalization of the factorial function for noninteger values of  $x$ .
The above Property B.1 and Lemma B.2 are generic geometric properties for high-dimensional spheres, but also applicable to high-dimensional Gaussian in which we are interested in the context of DDMs. To draw the connections with our context for studying the latent spaces of DMMs, with higher dimensionality, the latent Gaussian spaces of pre-trained DDMs become more difficult to operate due to decreased volume and mass concentration, as empirically suggested in [31, 35].
Property B.3. The volume of a high-dimensional sphere is essentially all contained in a thin slice at the equator and is simultaneously contained in a narrow annulus at the surface, with essentially no interior volume. Similarly, the surface area is essentially all at the equator.
The Property B.3 implies the connection with the standard Gaussian in  $\epsilon_T$  from direct sampling. In Fig.2 (b) of the main paper where we illustrate the geometric and probabilistic properties of samples in the  $\epsilon_T$  space, the inverted ones locate in the inner border area of the narrow annulus, which is also empirically verified in our Tab. 1 in the main paper. As those inverted latent encodings have a smaller radius than the expected standard Gaussian case.
Lemma B.4. For any  $c > 0$ , the fraction of the volume of the hemisphere above the plane  $x_{1} = \frac{c}{\sqrt{d - 1}}$  is less than  $\frac{2}{c} e^{-\frac{c^2}{2}}$ .
The above Lemma B.4 explains the volume range we show in Fig.2 (b) of the main paper in the left side of the Gaussian sphere to show the concentration mass, which is in the order of  $O\left(\frac{r}{\sqrt{d}}\right)$ .
Lemma B.5. The maximum likelihood spherical Gaussian for a set of samples is the one over center equal to the sample mean and standard deviation equal to the standard deviation of the sample.
The above Lemma B.5 provides the theoretical justifications for using the mean of squared distance to estimate the radius of Gaussian high-dimensional space.
C Markov Mixing
The mixing time defines a parameter that measures the time required by a Markov chain for the distance to stationary to be small [38]. The study of Markov mixing time aims to quantify the speed of convergence for Markov chains, and requires some other necessary preliminary knowledge on the total variance distance and the Convergence Theorem, which we will briefly describe below.
Firstly, to quantify the convergence characteristic of Markov chains, an appropriate distance measure metric is a prerequisite. In the literature, the total variation distance is the metric used to define the distance.
Definition C.1. The total variation distance between two probability distributions  $\mu$  and  $\nu$  on  $\mathcal{X}$  is defined by:
$$
\left\| \mu - \nu \right\| _ {T V} = \max  _ {A \subseteq \mathcal {X}} | \mu (A) - \nu (A) |. \tag {18}
$$
In the above definition,  $A$  is a probabilistic event, indicating that the distance between  $\mu$  and  $\nu$  is the maximum difference between probabilities assigned to a single event by the two distributions. This initial definition is not very practical to estimate the actual distance, which further induces the following propositions.
Proposition C.2. Let  $\mu$  and  $\nu$  to be two probability distributions on  $\mathcal{X}$ . Then
$$
\left| \left| \mu - \nu \right| \right| _ {T V} = \frac {1}{2} \sum_ {x \in \mathcal {X}} | \mu (x) - \nu (x) |. \tag {19}
$$
Proof. Let  $B = \{x : \mu(x) \geq \nu(x)\}$  and let  $A \subset \mathcal{X}$  be any event. Then we have
$$
\mu (A) - \nu (A) \leq \mu (A \cap B) \leq \mu (B) - \nu (B). \tag {20}
$$
The first inequality holds since any  $x \in A \cap B^c$  satisfies  $\mu(x) - \nu(x) < 0$ , and thus the difference in probability cannot decrease when such elements of  $B$  are eliminated. For the second inequality, we note that including more elements of  $B$  can not decrease the difference in probability.
By the same reasoning, we have:
$$
\nu (A) - \mu (A) \leq \nu \left(B ^ {c}\right) - \mu \left(B ^ {c}\right). \tag {21}
$$
The upper bounds on the right sides of Equation (20) and (21) are the same. Furthermore, by taking  $A = B$  or  $A = B^c$ , then  $|\mu(A) - \nu(A)|$  is equal to the upper bound. Therefore, we arrive at:
$$
\left| \left| \mu - \nu \right| \right| _ {T V} = \frac {1}{2} (\mu (B) - \nu (B) + \nu \left(B ^ {c}\right) - \mu \left(B ^ {c}\right)) = \frac {1}{2} \sum_ {x \in \mathcal {X}} | \mu (x) - \nu (x) |. \tag {22}
$$
The above proposition reduces total variance distance to a simple sum over the state space, which is an important theoretical support to formulate our mixing step problem and empirical search method. The proof process also reveals the following remark.
Remark C.3.  $||\mu - \nu||_{TV} = \sum_{x \in \mathcal{X}, \mu(x) \geq \nu(x)} [\mu(x) - \nu(x)]$ .
We then proceed to introduce the convergence theorem, which claims that aperiodic Markov chains converge to their stationary distributions at a key step, which is the direct theoretical foundation for us to introduce the mixing step problem for DDMs.
Theorem C.4. Convergence Theorem Suppose that  $P$  is irreducible and aperiodic, with stationary distribution  $\pi$ . Then there exist constants  $\alpha \in (0,1)$  and  $C > 0$  such that:
$$
\max  _ {x \in \mathcal {X}} \| P ^ {t} (x,.) - \pi \| _ {T V} \leq C \alpha^ {t}. \tag {23}
$$
There exist multiple mathematical versions for the proof of the convergence theorem, which we omit in this appendix. Note that the assumptions for  $P$  to be irreducible and aperiodic are essential. We recall here the definition of an irreducible chain  $P$ .
Definition C.5. A chain  $P$  is called irreducible if for any two states  $x, y \in \mathcal{X}$ , there exists an integer  $t$  (possibly depending on  $x$  and  $y$ ) such that  $P^t(x, y) > 0$ .
Intuitively, this means that it is possible to get from any state to any other state using only transitions of positive probability. This is verified in the current formulation of DDMs, indicating that DDMs satisfy the pre-requisite to be an irreducible Markov chain.
Next, we recall the definition of the period for a Markov chain.
Definition C.6. Let  $\tau(x) \coloneqq \{t \geq 1 : P^t(x, x) > 0\}$  be the set of times when it is possible for the chain to return to starting position  $x$ . The period of state  $x$  is defined to be the greatest common divisor of  $\tau(x)$ .
For an irreducible chain, the period of the chain is defined to be the period that is common to all states, and the chain is aperiodic if all states have period 1. Intuitively, the above definition and property match the actual formulation and implementations of DDMs, given the fact that plenty of existing DDMs [54, 4, 17, 71] propose an auxiliary loss to predict directly the denoised  $x_0$  at arbitrary step.
Having introduced the above definitions, we are now ready to present the formal definition of mixing time in Markov chain studies.
Definition C.7. Definition of Mixing Time The mixing time is a parameter that measures the time required by a Markov chain for the distance to the stationary distribution to be small, following the definition below:
$$
t _ {m i x} (\epsilon) := \min  \{t: d (t) \leq \epsilon \} \text {a n d} t _ {m i x} := t _ {m i x} (1 / 4). \tag {24}
$$
In particular, taking  $\epsilon = \frac{1}{4}$  above yields
$$
d \left(l t _ {m i x}\right) \leq 2 ^ {- l} \text {a n d} t _ {m i x} (\epsilon) \leq \left\lceil \log_ {2} \epsilon^ {- 1} \right\rceil t _ {m i x}. \tag {25}
$$
In addition to the initial definition of mixing time, we also need the background knowledge on the time reversal to search for the actual mixing step in a more practical way.
Definition C.8. For a distribution  $\mu$  on a group  $G$ , the reversed distribution  $\hat{\mu}$  is defined by  $\hat{\mu}(g) \coloneqq \mu(g^{-1})$  for all  $g \in G$ .
The time reversal is directly related to the two-direction design of DDMs, and ensures that the mixing step remains at a fixed position in two directions for both diffusion and generative processes. This property is also critical to better understand the DDMs, and provides theoretical justifications for us to search for the mixing step along the generative direction using the Gaussian radius estimation. In fact, the Gaussian radius estimation search method can only be valid and applied in the generative direction but not the inverse diffusion process. The reasons are discussed in the following section when we show more empirical results.
Lemma C.9. Let  $P$  be the transition matrix of a random walk on a group  $G$  with increment distribution  $\mu$  and let  $\hat{P}$  be that of the walk on  $G$  with increment distribution  $\hat{\mu}$ . Let  $\pi$  be the uniform distribution on  $G$ . Then for any  $t \geq 0$ ,
$$
\left\| P ^ {t} (i d, \cdot) - \pi \right\| _ {T V} = \left\| \hat {P} ^ {t} (i d, \cdot) - \pi \right\| _ {T V}. \tag {26}
$$
The lemma above implies the remark below, which will be used in our proof for the Property 4.1.
Remark C.10. If  $t_{\text{mix}}$  is the mixing time of a random walk on a group and  $t_{\hat{\text{mix}}}$  is the mixing time of the reversed walk, then  $t_{\text{mix}} = t_{\hat{\text{mix}}}$ .
We hereby finish introducing the necessary background on Markov mixing studies, and continue to a more detailed discussion of the mixing step problem of DDMs.
D More Discussion on Mixing Step
Inspired by the Markov mixing studies, we remark that the current formulation of DDMs satisfies several key assumptions as described in Appendix C, including most importantly, DDMs model an irreducible and aperiodic chain. Note that our current exploration and formulation for the mixing step of DDMs are not absolutely thorough and complete, which can be considered as an approximate and adapted version of the mathematical Markov mixing time.
D.1 Proof for Property of Mixing Step
We rewrite the Property of mixing step for DDMs here before going to the detailed proof.
Property D.1. Under the total variation distance measure  $||\cdot ||_{TV}$ , the mixing step  $t_m$  for a DDM with data dimensionality  $d$  is formed during training (i.e., irrelevant to the sampling methods).  $t_m$  is mainly related to the transition kernels and the stationary distribution (i.e., datasets), and less dependant on the dimensionality  $d$ .
Proof. The proof for the above property consists of several steps.
Existence justification. Firstly, we have shown that DDMs model a group of chains that are irreducible and aperiodic, and thus the convergence theorem holds for DDMs. This fact establishes the theoretical foundation to find such a critical convergent step that theoretically characterizes the convergence of pre-trained DDMs.
Directions to approach. Secondly, we show that the mixing step is large and mostly dependent on the transition kernel. Here, we have to clarify the direction of the DDMs we are tackling. Fortunately, based on the time reversal from Lemma C.9 and Remark C.10, whichever direction gives the same mixing step, provides us with the flexibility to study either direction. However, in practice, the easiest way to approach the mixing step is to theoretically infer the transition kernel in the diffusion direction, and then empirically search for it in the denoising direction. We will first provide the method and explain the reasons for such a design.
Theoretical based transition kernel study. We hereby restrict ourselves in considering the diffusion process. Given pre-trained DDMs, according to Lemma C.9, we have an irreducible transition matrix  $P$  on space  $\mathcal{X}$ . In the current scenario of diffusion direction from  $\mathbf{x}_0$  to  $\mathbf{x}_T$ , the stationary distribution is the standard Gaussian  $\mathcal{N}(0,\mathbf{I}_d)$  in  $\epsilon_T$ . The transition matrix is a pre-defined Gaussian with known mean value and variance, thus we have
$$
P = q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t - 1}\right) := \mathcal {N} \left(\mathbf {x} _ {t}; \sqrt {1 - \beta_ {t}} \mathbf {x} _ {t - 1}, \beta_ {t} I\right). \tag {27}
$$
For an irreducible transition matrix  $P$  with stationary distribution  $\pi$ , define:
$$
\sigma_ {t} (x, y) := \frac {P ^ {t} (x , y)}{\pi (y)}, \tag {28}
$$
with  $\sigma_t(x,y) = \sigma_t(y,x)$  when  $P$  is reversible with respect to  $\pi$ . We also have:
$$
<   \sigma_ {t} (x, \cdot), 1 > _ {\pi} = \sum_ {y} q _ {t} (x, y) \pi (y) = 1. \tag {29}
$$
Next, we have the definition of  $l^p$ -distance  $d^{(p)}$  as:
$$
d ^ {(p)} (t) := \max  _ {x \in \mathcal {X}} | | \sigma_ {t} (x, \cdot) - 1 | | _ {p}. \tag {30}
$$
To replace the above notations with the notations from DDMs, we have:
$$
d ^ {(1)} (t) := \max  _ {x \in \mathcal {X}} \| \sigma_ {t} (x, y) - 1 \| _ {1}, \tag {31}
$$
$$
\sigma_ {t} (x, y) = \frac {P ^ {t} (x , y)}{\pi (y)} = \frac {x \sim \mathcal {N} \left(x _ {t} ; \sqrt {1 - \beta_ {t}} x _ {t - 1} , \beta_ {t} I\right)}{y \sim \mathcal {N} \left(0 , I _ {d}\right)}. \tag {32}
$$
Based on the definition of mixing time in 24, we then have:
$$
t _ {m i x} ^ {(1)} (\varepsilon) := \inf  \{t \geq 0; d ^ {(1)} (t) \leq \varepsilon \}. \tag {33}
$$
Table 4: Gaussian radius estimation for empirical search of the mixing step for pre-trained DDPM on CelebA-64.
Steps	1000	900	800	700	600	500	400	300
xS_T + ps| △r|	110.84	110.82	110.83	110.58	109.83	107.85	103.22	94.64
0.02	0.01	0.25	0.75	1.98	4.63	8.58	-
xS_T + pi| △r|	110.88	110.86	110.84	110.63	109.89	107.86	103.10	94.45
0.02	0.02	0.21	0.74	2.03	4.76	8.65	-
xI_T + ps| △r|	95.06	93.30	91.56	90.14	88.69	86.61	82.80	76.46
1.76	1.74	1.42	1.45	2.08	3.81	6.36	-
xI_T + pi| △r|	95.06	93.34	91.61	90.16	88.75	86.57	82.87	76.53
1.72	1.73	1.45	1.41	2.18	3.70	6.34	-
We take the value  $\varepsilon$  to be  $\frac{1}{2}$ , and thus arrive at:
$$
t _ {m i x} ^ {(1)} := \inf  \{t \geq 0; d ^ {(1)} (t) \leq \frac {1}{2} \}. \tag {34}
$$
Now, we return back to Equation 32 and replace the Equation 34 with:
$$
\max  _ {x \in \mathcal {X}} | | \frac {x \sim \mathcal {N} \left(x _ {t} ; \sqrt {1 - \beta_ {t}} x _ {t - 1} , \beta_ {t} I\right)}{y \sim \mathcal {N} \left(0 , I _ {d}\right)} - 1 | | \leq \frac {1}{2}. \tag {35}
$$
By using the Proposition C.2, we can now substitute the above Equation 35 using the approximation as follows:
$$
\left| \left| x \sim \mathcal {N} \left(x _ {t}; \sqrt {1 - \beta_ {t}} x _ {t - 1}, \beta_ {t} I\right) \right| \right| \leq 4. \tag {36}
$$
This above gives an approximation of the transition kernel at the mixing step in the diffusion direction we would expect, with a radius change at approximately 4.
We observe that there is no explicit dependency on the dimensionality of the latent spaces, but directly related to the formulation of the transition kernel, which is mostly the Gaussian as used in existing DDMs implementations. In the meanwhile, we note the intermediate latent encodings  $x_{t}$  are actually dataset dependent. Therefore, we verify our claim that the mixing step is more dependent on the transition kernel and dataset. However, despite no explicit dependency between the mixing step and dimensionality, we empirically observe that the appearance of mixing step still differs in pre-trained diffusion models on different resolutions as in Tab. 5.
Interestingly, the above proof gives us a numerical approximation for the transition kernel when the mixing step appears, which is the radius variation at approximately 4. We hereby finish demonstrating the fact that the mixing step appears at around diffusion step  $t = 500$  in our main paper. In the meanwhile, other related works [31, 35] report similar conclusions that editing on step 500 shows better empirical performance in different experimental settings.
D.2 More Empirical Results on Mixing Step
For the empirical verification of the mixing step, we use a pre-trained DDPM model [21] on the CelebA dataset [39] with  $3 \times 64 \times 64$  resolution. Therefore, for a standard Gaussian space in the dimensionality of  $d = 3 \times 64 \times 64 = 12$ , 288, the expected Gaussian radius is  $r = \sigma \sqrt{d} = 110.85$ . The full radius estimation results are listed in Tab. 4, we also show the difference in Gaussian radius between consecutive 100 steps. Note we are slightly "abusing" the estimation results for steps after the mixing step, since the distributions of the latent spaces after  $\epsilon_{t_m}$  are no longer considered as Gaussian, but rather converge to the actual data distributions in  $\mathcal{X}$ , therefore, estimating the Gaussian radius of those latent spaces are not theoretically sound. This also explains the reason why we do not report the numbers for step numbers less than 300. The above also explains our design to derive the theoretical proof in the diffusion direction, but proposes to empirically search for the mixing step via the denoising direction.
Table 5: More results on Gaussian radius estimation for the empirical search of the mixing step for pre-trained DDMs
Model	Setting	1000	900	800	700	600	500	400
DDPM-CelebA-HQ-256	xs^s + ps| △r|	443.42	443.36	443.30	442.49	440.16	432.55	416.77
0.06	0.06	0.81	2.33	7.61	15.78	-
DDPM-CelebA-HQ-256	xs^s + pi| △r|	443.40	443.29	443.06	442.22	439.44	431.66	413.96
0.11	0.23	0.84	2.78	7.78	17.70	-
iDDPM+AFHQ-256	xs^s + pi| △r|	443.34	443.21	442.84	441.72	439.31	429.16	408.69
0.13	0.37	1.12	2.41	10.15	20.47	-
D.3 Connection with Existing Works
We notice that existing SOTA methods [31, 35] have proposed similar ideas in their works, by empirically exploring the diffusion steps that obtain better qualitative results. However, the mixing step has been studied as a hyper-parameter (i.e., "return step" in [31] and "edit step" [35]) that influences the downstream qualities without formal definition. In this work, we formally define and introduce the concept of the mixing step, which originated from sound mathematical studies on the Markov mixing time, and provide a comprehensive perspective to re-think this "hyper-parameter". More excitingly, we discover that the theoretically driven deviation and our Gaussian radius estimation method come to a consistent conclusion and echo with previous literature in actual experimental tests.
E Boundary Search Discussion
In this section, we present more details about our proposed boundary search method, and discuss the connections between different latent space levels from the perspective of the Projection theorem.
E.1 Projection Theorem
In theory, we expect the projected lower-dimensional subspace to preserve the same properties of its original higher-dimension space such as the projected distances between pairs of samples should have the same ordering in two spaces. In mathematics, we can ensure the validity of this projection design using the existing projection theorems.
Theorem E.1. Theorem of the Random Projection. Let  $\mathbf{v}$  be a fixed unit length vector in a  $d$ -dimensional space and let  $W$  be a random  $k$ -dimensional subspace. Let  $\mathbf{w}$  be the projection of  $\mathbf{v}$  onto  $W$ . For any  $0 \leq \epsilon \leq 1$ ,  $\text{Prob}(|\mathbf{w}|^w - \frac{k}{d}) \geq \epsilon \frac{k}{d}$  and  $\epsilon \leq e^{-\frac{k\epsilon^2}{64}}$ .
One way to interpret the random projection theorem is that if one chooses a random  $k$ -dimensional subspace from a higher-dimensional space in  $d$ -dimension, then indeed all the projected distances in the  $k$ -dimensional space are approximately within a known scale factor of the distances in the  $d$ -dimensional space.