Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometry
Authors: Yong-Hyun Park
Mingi Kwon
Jaewoong Choi
Junghyo Jo
Youngjung Uh
Venue: NeurIPS 2023 poster (2023)
Keywords: diffusion models
semantic image editing
differential geometry
TL;DR: We unveil new insights into the latent space of diffusion models through a geometrical analysis. and the influence of text conditioning in image editing.

Abstract: Despite the success of diffusion models (DMs), we still lack a thorough understanding of their latent space. To understand the latent space $\mathbf{x}_t \in \mathcal{X}$, we analyze them from a geometrical perspective. Our approach involves deriving the local latent basis within $\mathcal{X}$ by leveraging the pullback metric associated with their encoding feature maps. Remarkably, our discovered local latent basis enables image editing capabilities by moving $\mathbf{x}_t$, the latent space of DMs, along the basis vector at specific timesteps. We further analyze how the geometric structure of DMs evolves over diffusion timesteps and differs across different text conditions. This confirms the known phenomenon of coarse-to-fine generation, as well as reveals novel insights such as the discrepancy between $\mathbf{x}_t$ across timesteps, the effect of dataset complexity, and the time-varying influence of text prompts. To the best of our knowledge, this paper is the first to present image editing through $\mathbf{x}$-space traversal, editing only once at specific timestep $t$ without any additional training, and providing thorough analyses of the latent structure of DMs.
The code to reproduce our experiments can be found at the [link](https://github.com/enkeejunior1/Diffusion-Pullback).

Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometry
Yong-Hyun Park"), Mingi Kwon" Jaewoong Choi Junghyo Jo Youngjung Uh
Scoul National University "Yonsei University KKorea Institute for Advanced Study
Abstract
Despite the success of diffusion models (DMs), still lack thorough understand- ing of their latent space. To understand the latent space x, E A' we analyze them from a geometrical perspective. Our approach involves deriving the local latent basis within X' by leveraging the pullback metric associated with their encoding feature maps. Remarkably, our discovered local latent basis enables image editing capabilities by moving x;, the latent space of DMs, along the basis vector at specific timesteps. We further analyze how the geometric structure of DMs evolves over diffusion timesteps and differs across different text conditions. This confirms the known phenomenon such as the discrepancy between across timesteps, the effect of dataset complex- ity, and the time-varyee varying influence of text prompts. To the best of our knowledge, this paper is the first to present image editing through x-space traversal, editing only once at specific timestep without any additional training, and providing thorough analyses of the latent structure of DMs. The code to reproduce experiments found at https: / github. com/ /ee/eeeeeeiiiooo www.umoiiooo//iiioooooooo - back.
coarse-to-fine generation, as well as reveals novel insights
Introduction
The diffusion models (DMs) are powerful generative models that have demonstrated impressive performance [22, 51, 52, 37]. DMs have shown remarkable applications, including text-to-image synthesis [45, 46, 5, 36), inverse problems [14, 31], and image editing [21, 54, 39, 35).
Despite their achievements, the research community lacks comprehensive understanding of the latent space DMs and influence on the generated results. So far, the completely diffused images considered as latent variables but it does not have useful properties for controlling the results. For example, traversing along direction from latent produces weird changes in the results. Fortunately, Kwon al. [26] consider the intermediate feature space of the diffusion kernel, referred to H. as a semantic latent space and show its usefulness on controlling generated images. In the similar sense, some works investigate the feature maps of the self-attention for controlling the results [21, 54, 39], improving sample quality [8], or downstream tasks such as semantic segmentation [32, 55].
attentonn operations
Still, the structure of the space X, where latent variables {xr} live remains unexplored despite its crucial role in understanding DMs. It is especially challenging because 1) the model is trained to estimate the forward noise which does not depend on the input, as opposed to other typical supervisions such as classification or similarity, and 2) there are lots of latent variables over multiple recursive timesteps. this paper, we aim to analyze X' in conjunction with its corresponding
"Equal Contribution
Corresponding authors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Txx
Diflusion model (U-Net;
The
V2
Ja. Janshion of at
V1
54h right singule vector de
Latent-ppoo structure
Tu spenl Vaure val
54h left singilar wetor Ja
Feature space structure
Th span(u.... u..)
Latent space (2)
Intermediate Feature Space (H)
(a) Conceptual illustaaionn
(b) Notation
Figure I: Conceptual illustration of local geometric structure. (a) The local basis {VI, V2, - of the local latent subspace Tx, within the latent space X' is interlinked with the local basis { u..) of the local tangent space The in the feature space H. (b) The derivation of these local bases is facilitated through the singular value decomposition (SVD) of the Jacobian, which emanates from the U-Net responsible for encoding the feature map f. linking X' and H.
representation H. by incorporating a local geometry to X' using the concept of pullback metric in Riemannian geometry.
First, we discover the local latent basis for X' and the corresponding local tangent basis for H. The local basis is obtained by performing singular value decomposition (SVD) of the Jacobian of the mapping from X' to H. To validate the discovered local latent basis, we demonstrate that walking along the basis can edit real images in a semantically meaningful way. Furthermore, we can use the discovered local latent basis vector to edit other samples by using parallel transport, when they exhibit comparable local geometric structures. Note that existing editing methods manipulate the self-attention map cross-attention map over multiple timesteps [21, 54, 39J. On the other hand, we manipulate only x, once at a specific timestep.
Second, we investigate how the latent structures differ across different timesteps and samples as follows. The frequency domain of the local latent basis shifts from low-frequency to high frequency along the generative process. We explicitly confirm it using power spectral density analysis. The difference between local tangent spaces of different samples becomes larger along the generative process. The local tangent spaces at various diffusion timesteps are similar to each other if the model trained on aligned datasets such as CelebA-HQ or Flowers. However, this homogeneity does not occur on complex datasets such as ImageNet.
Finally, we examine how the prompts affect the latent structure of text-to-image DMs as follows. Similar prompts yield similar latent structures. Specifically, we find a positive correlation between the similarity prompts and the similarity of local tangent spaces. The influence of on the local tangent space becomes weaker along the generative process.
Our work examines the geometry of X' and H using Riemannian geometry. We discover the latent structure of and how evolves during the generative process and is influenced by prompts. This geometric exploration deepens our understanding of DMs.
Related works
Diffusion Models. Recent advances DMs make great progress the field image synthesis
and show state-of-the-art performance [50, 22, 51J. An important subject in the diffusion model is introduction of gradient guidance, including classifier-free guidance, to control the generative process [17, 47, 4, 30, 36, 46]. The work by Song et al. [52] has facilitated the unification of DMs with score-based models using SDEs, enhancing our understanding of DMs as a reverse diffusion process. However, the latent space is still largely unexplored, and our understanding is limited.
The study of latent space in GANs. The study of latent spaces has gained significant attention in recent years. In the field Generative Adversarial Networks (GANs), researchers have proposed various methods manipulate the latent space achieve the desired effeet in the generated images

---

[44, 41, 1, 20, 49, 59, 38]. More recently, several studies [60, 10] have examined the geometrical properties of latent space in GANs and utilized these findings for image manipulations. These studies bring the advantage of better understanding the characteristics of the latent space and facilitating the analysis and utilization of GANs. contrast, the latent space of DMs remains poorly understood, making it difficult to fully utilize their capabilities.
Image manipulation in DMs. Early works include Choi et al. [11] and Meng al. [33] have attempted to manipulate the resulting images of DMs by replacing latent variables, allowing the generation of desired random images. However, due the lack of semantics in the latent variables of DMs, current approaches have critical problems with semantic image editing. Alternative approaches have explored the potential of using the feature space within the U-Net for semantic image manipula- tion. For example, Kwon et al. [26] have shown that the bottleneck of the U-Net, H. can be used as a semantic latent space. Specifically, they used CLIP [43] to identify directions within H that facilitate genuine image editing. Baranchuk et al. [6] and Tumanyan et al. [54] use the feature map of the U-Net for semantic segmentation and maintaining the structure of generated images. Unlike previous works, our editing method finds the editing direction without supervision, and directly traverses the latent variable along the latent basis.
Riemannain Geometry. Some studies have applied Riemannian geometry to analyze the latent spaces of deep generative models, such Variational Autoencoders (VAEs) and GANs [2, 48, 9, 3, 27, 28, 57]. Shao et al. [48] proposed a pullback metric on the latent space from image space Euclidean metric to analyze the latent space's s geometry. This method has been widely used in VAEs and GANs because it only requires a differentiable map from latent space to image space. However, no studies have investigated the geometry of latent space of DMs utilizing the pullback metric.
3 Discovering the latent basis of DMs
In this section, we explain how to extract a latent structure of X' using differential geometry. First, we introduce key concept in our method: the pullback metric. Next, by adopting the local Euclidean metric of H. and utilizing the pullback metric, we discover the local latent basis of the X. Moreover, although the direction we found is local, we show how it can be applied to other samples via parallel transport. Finally, we introduce x-space guidance for editing data in the X' to enhance the quality of image generation.
3.1 Pullback metric
We consider a curved manifold, X, where our latent variables x, exist. The differential geometry represents X' through patches of tangent spaces, Tx, which are vector spaces defined at each point x. Then, all the geometrical properties of X' can be obtained from the inner product of ||dx|| (dx, dx)x in Tx. However, we do not have any knowledge of (dx, dx)x. is definitely not Euclidean metric. Furthermore, samples of x, at intermediate timesteps of DMs include inevitable noise, which prevents finding semantic directions in Tx-
Fortunately, Kwon et al. [26] observed that H. defined by the bottleneck layer of the U-Net, exhibits local linear structure. This allows us to adopt the Euclidean metric on H. In differential geometry, when a metric is not available on a space, pullback metric is used. If smooth map exists between the original metric-unavailable domain and a metric-available codomain, the pullback metric is used to measure the distances in the domain space. Our idea is to use the pullback Euclidean metric on H to define the distances between the samples in X.
DMs are trained infer the noise e, from a latent variable x, at each diffusion timestep L. Each x has different internal representation h, the bottleneck representation of the U-Net, at different t's. The differentiable map between X' and H. is denoted as f X + H. Hereafter, we refer to x, as x for brevity unless it causes confusion. It is important to note that our method can be applied at any timestep in the denoising process. We consider linear map. Tx -> The between the domain and codomain tangent spaces. This linear map can be described by the Jacobian Jx = Vxh which determines how a vector v E Tx is mapped into a vector u E Th by u J&V.
Using the local linearity of H. we assume the metric, ||ch||2 (dh, dhi, = dh't dhh as a usual dot product defined in the Euclidean space. To assign geometric structure to X, use the pullback
3

---

C DRIM Leverson
E DDIN genetation
Identify local besis vector (v) ( w-sjaco guidance
IDIM generiioon
web prospe
Impul image
MW
Tiger
Boar
Loon
Ostricc
p(Ky))
.Pribailly dow injectuctes
P(G))
(a) Method overvlew
(b) Editing results
Figure 2: Image editing with the discovered latent basis. (a) Schematic depiction of our image editing procedure. ① An input image is subjected to DDIM inversion, resulting in an initial noisy sample XT.
( The sample Xy is progressively denoised until reaching the point through DDIM generation. @ Subsequently, the local latent basis {V1, ........ vn is identified by using the pullback metric. 4 This enables the manipulation of the sample x, along one of the basis vectors using x-space guidance. ( The DDIM generation concludes with the progression from the modified latent variable x, (b) Examples of edited images using selected basis vector. The latent basis vector could be conditioned on prompts and it facilitates text-aligned manipulations.
metric of the corresponding H. In other words, the norm of v E Tx is measured by the norm of corresponding codomain tangent vector:
11v1 - (u, u)n =v v'''''x
3.2 Finding local latent basis
Using the pullback metric, we define the local latent vector v E Tx that shows a large variability in Th- We find unit vector V1 that maximizes | By maximizing ||v| while remaining orthogonal to V1, one can obtain the second unit vector v2. This process can be repeated to have Tx- In practice, V, corresponds to the i-th right singular n latent directions of (V1, V2, ......... vn
vector from the singular value decomposition (SVD) of Jx = UAV", i.e., J&v; = Au, Since the Jacobian of too many parameters is not tractable, we use power method [18, 34, 19] approximate the SVD of Jx (See Appendix D for the time complexity and Appendix F for the detailed algorithm).
Henceforth, we refer to Tx as a local latent subspace, and {V1, V2, ; vn) the corresponding local latent basis.
Tx A spantv1, V2 ".,n., where v, is i-th right singular vector of Jx-
Using the linear transformation between Tx and To via the Jacobian Jx, one can also obtain corre- sponding directions in The In practice, u, corresponds to the i-th left singular vector from the SVD of Jx. After selecting the top n (e.g., n 50) directions of large eigenvalues, we can approximate any vector in Tn with a finite basis, {u,, u2 ... J. When we refer to a local tangent space henceforth, it means the n--iiennsonaal low-rank approximation of the original tangent space.
The A span un U2 un), where u; is the i-th left singular vector of Jx.
(3)
The collection of local latent basis vectors, {Vi, V2, ... VaJ. obtained through our proposed method, can be interpreted as a signal that the model is highly response to for a given x. On the other hand, the basis of the local tangent space, denoted as {u, ... } can be viewed as the corresponding representation associated with the signal.
In Stable Diffusion, the prompt also influences the Jacobian, which means that the local basis also depends on it. We can utilize any prompt obtain local latent basis, and different prompts create distinct geometrical structures. For the sake of brevity, we will omit the word local unless it leads to confusion.
3.3 Generating edited images with x-space guidance
naive approach for manipulating a latent variable x using a latent vector v is through simple addition, specifically x + yv. However, using the naïve approach sometime leads to noisy image

---

generation. To address this issue, instead of directly using the basis for manipulation, we use basis vector that has passed through the decoder once for manipulation. The x-space guidance is defined as follows
Xxo = x r + + - cx(x)]
(4)
where y is a hyper--praneee controlling the strength of editing and eg is a diffusion model. Equation inspired by classifier-free guidance, but the key difference that it is directly applied in the latent space X. Our x-space guidance provides qualitatively similar results to direct addition, while it shows better fidelity. (See Appendix C for ablation study.)
3.4 The overall process of image editing
In this section, we summarize the entire editing process with five steps: 1) The input image is inverted into initial noise Xy using DDIM inversion. 2) Xr is gradually denoised until / through DDIM generation. 3) Identify local latent basis [V1. .nn using the pullback metric at t. 4) Manipulate X2 along the one of basis vectors using the x-space guidance. 5) The DDIM generation is then completed with the modified latent variable &, Figure 2 illustrates the entire editing process.
In the context of a text-to-image model, such as Stable Diffusion, it becomes possible to include textual conditions while deriving local basis vectors. Although we do not use any text guidance during DDIM inversion and generation, a local basis with text conditions enables semantic editing that matches the given text. Comprehensive experiments can be found in Section 4.1.
is noteworthy that our approach needs no extra training and simplifies image editing by only adjusting the latent variable within single timestep.
3.5 Editing various samples with parallel transport
Let us consider a scenario where our aim is to edit ten images, changing straight hair to curly hair. Due to the nature of the unsupervised image editing method, it is becomes imperative to manually check the semantic relevance of the latent basis vector in the edited results. Thus, to edit every samples, we have manually find straight-to-ce -urrlyly basis vector for individual samples.
One way to alleviate this tedious task is apply the curly hair vector obtained from one image to other images. However, the basis vector v E Tx obtained at x cannot be used for the other sample x' because é Txx Thus, in order to apply the direction we obtained another sample, is necessary to relocate the extracted direction to new tangent space.
To achieve this, we use parallel transport that moves v; onto the new tangent space Tx In the realm of differential geometry, parallel transport is a technique to relocate a tangent vector to different position with minimal direction change. while keeping the vector tangent on the manifold [48]. is notable that in curved space, parallel transport can significantly modify the original vector. Therefore, it is beneficial apply the parallel transport in H. where relatively flatter than X.
The process of moving a tangent vector V E Tx to v' E Tx through parallel transport in H. is summarized as follows. First, we convert the latent direction V; Tx to the corresponding direction of u, E The Second, we apply the parallel transport u, E Th to u'; E Th', where h = fxx). In the general case, parallel transport involves iterative projection and normalization along the path connecting two points [48]. However, in our case, we assume that H has Euclidean geometry. Therefore, we move directly onto Th through projection, without the need for an iterative process. Finally, transform uf into v E X. Using this parallel transport of v; + v via H. we can apply the local latent basis obtained from edit modify the input x'.
the tangent space
4 Findings and results
In this section, we analyze the geometric structure of DMs with our method. 9 4.1 demonstrates that the latent basis found by our method can be used for image editing. In $ 4.2, we investigate how the geometric structure of DMs evolves as the generative process progresses. Lastly, in 3 4.3, we examine how the geometric properties of the text-condition model change with a given text.
The implementation details of our work are provided on Appendix B. The source code for our experiments can be found at https: //te:///b.tttt///// com/
Pullbacck.
5

---

1T
0.87
0.67
FFHQ
image
(Old)
Real image
(Woman)
Real image-
(Red nose)
Flows
Real image
(Sunflower)
Real image
(Yellow)
Real image
(Pinky)
AFHQ
Real image
(Bullmastiff)
Real image
(Open mouth)
Real image
(Wink)
LSUN-Churcll
Real image
(Two-story)
Real image
(Long roof)
Real image
(White paint)
0.77
0.67
Stable Diffusion
e
Real image
(Woman)
Real image
(Overweighted)
Figure 3: Examples of image edition using the latent basis. The attributes manually interpreted since the editing directions are not intentionally supervised. For Stable Diffusion, we used an empty string as a prompt. Each column represents edits made at different diffusion timesteps (0.67, 0.ST. and 7 for the unconditional diffusion model; 0.67' and 0.7T for Stable Diffusion).
0.7T V2
0.6T V2
Real image
V1
V3
V1
V3
w/ "Lion" w/o prtroe
Figure 4: Examples of image edition using top-3 latent basis vectors. Each column is edited using different latent vector [V1, V2, va). Each group of columns represents edits made different diffusion timesteps (0.67' and 0.71'). Notably, when given the "Lion" prompt, is evident that all the top latent basis vectors align with the direction of the prompt.
4.1 Image editing with the latent basis
In this subsection, we demonstrate the capability of our discovered latent basis for image editing. To extract latent variables from real images for editing purposes, we use DDIM inversion. In experiments with Stable Diffusion (SD), we do not use guidance, i.e., unconditional sampling, for both DDIM inversion and DDIM sampling processes. This ensures that our editing results solely depend on the latent variable, not other factors such as prompt conditions.

---

Real image
"Tiger"
"Bear"
"Lion"
"Running dog"
"Sitting dog"
"Rumning cat"
"Standing cat"
Figure 5: Examples of image edition using latent basis vectors discovered with various prompts. Each column is edited using the latent basis vector obtained from a different prompt. Importantly, our method employs each prompt only once to derive the local latent basis.
Figures 2 and 3 illustrate the example results edited by the latent basis found by our method. The latent basis clearly contains semantics such as age, gender, species, structure, and texture. Note that editing at timestep T yields coarse changes such as age and species. On the other hand, editing at timestep 0.67 leads to fine changes, such nose color and facial expression.
Figure demonstrates the example results edited by the various latent basis vectors. Interestingly, using the text "lion" condition, the entire latent basis captures lion-related attributes. Furthermore, Figure shows that the latent basis aligns with the text not only in terms of object types but also in relation to pose action. For qualitative comparison with other state-of-the-art art image editing methods, refer to Appendix D. For more examples editing results, refer to Appendix G.
4.2 Evolution of latent structures during generative processes
In this subsection, we demonstrate how the latent struc- ture evolves during the generative process and identify three trends. 1) The frequency domain of the latent ba- sis changes from low to high frequency. agrees with the previous observation that DMs generate samples in coarse-to- fine manner. 2) The difference between the tangent spaces of different samples increases the generative process. implies finding generally appli- cable editing direction in latent space becomes harder in later timesteps. 3) The differences tangent spaces between timesteps depend the complexity of the dataset.
Density
10-1
0.757
10-3
Power Spectral
0.37
10-5
-0.25
10
T
10"
10
V1
Frequency
Figure 6: Power Spectral Density (PSD) of latent basis. The PSD at ( T (pur- ple) exhibits a greater proportion of low- frequency signals, while the PSD at smaller t (beige) reveals larger proportion of high- frequency signals. The latent vectors v, are min-max normalized for visual clarity.
Latent bases gradually evolve from low- to high- frequency structures. Figure is the power spectral density (PSD) of the discovered latent basis over var- ious timesteps. The early timesteps contain a larger portion of low frequency than the later timesteps and the later timesteps contain a larger portion of high frequency.
This suggests that the model focuses low-frequency signals at the beginning of the generative process and then shifts its focus to high-frequency signals over time. This result strengthens the common understanding about the coarse-to-fine behavior of DMs over the generative process [12, 15].
The discrepancy tangent spaces from different samples increases along the generative process. To investigate the geometry of the tangent basis, employ metric the Grassmannian manifold.
7

---

The Grassmannian manifold is a manifold where each point is a vector space, and the metric defined above represents the distortion across various vector spaces. We use geodesic metric [10, 56] define the discrepancy between two subspaces (T(1), ]:
D200 (T(),, T(2))
Σ
(5)
where Or denotes the k-th principle angle between ( and ((2)_ Intuitively, the concept of geodesic metric can be understood as an angle between two vector spaces. Here, the comparison between two different spaces was conducted for (Th, The J. Unlike the X, the assumes Euclidean space which makes the computation of geodesic metric that requires an inner product between tangent spaces easier. The relationship between tangent space and latent subspace is covered in more detail in Appendix E.
11-
10-
Themrtsal spper
Den Tb., The
Ransom basis
9-
Dilfoon (ver 2.1)
ImageNet
de
LSUN church
LeeNN
LSUN car
7-
LSUN- Sonne
Celebd IQQ
Figure 7 demonstrates that the tangent spaces of the different samples are the most similar at & T and diverge as timestep becomes zero.
Flowess
FFHQ"
0.757'
0.57
0.257
Diffusion Timestep
Moreover, the similarity across tangent spaces allows us effectively transfer the latent basis from ple to another through parallel transport as shown in Figure 8. In T, where the tangent spaces are homo- geneous, we consistently obtain semantically aligned editing results. On the other hand, parallel transport at t = 0.67' does not lead to satisfactory editing because the tangent spaces are hardly homogeneous. Thus, should examine the similarity of local subspaces to ensure consistent editing across samples.
Figure 7: Geodesic distance across tan- gent space of different samples various diffusion timesteps. Each point represents the average geodesic distance between pairs of 15 samples. It is notable that the similar- ity of tangent spaces among different sam- ples diminishes the generative process progresses.
DMs trained on simpler datasets exhibit more consistent tangent spaces over time. In Figure 9 (a), we provide a distance matrix the tangent spaces across different timesteps, measured by the geodesic metric. We observe that the tangent spaces are more similar to each other when model trained on CelebA-HQ, compared to ImageNet. To verify this trend, we measure the geodesic distances between tangent spaces of different timesteps and plot the average distances of the same
IT
0.6T +v,
Real image
+v;
(Man)
Real image
+v.
(Woman)
Real image
(Red lip)
t
Figure 8: Examples of image edition using parallel transport. The first demonstrates the results of editing with their respective latent vectors, while the subsequent rows exhibit the results of editing through the parallel transport (P.T) of the latent vectors used in the first row. The latent vector performs effectively when & = I' (left and middle), but comparatively less satisfactorily for 0.6T (right).

---

0.17
0.17
Ma
10-
8-
Irageke
Dawi Thu Tay)
6-
1SUh- bedron
Aog Peo
LSUN-chards
4-
Tloxes
2-
0:17
0-
CelebA-HQ
ImageNet
0.757
|A7]
(b)
Figure 9: Simpler datasets lead to more similar tangent spaces across diffusion timesteps. (a)
Distance matrix visualization of tangent space measured by geodesic metric across various timesteps. (b) Average geodesic distance based on timestep differences, indicating that the complexity of the dataset correlates with greater distances between tangent spaces.
10-
0.45
Thential lignelleant flound Danken Bhacs
Daol (Thoury Thooe!
10-
g-
0.40
D DavaTTo Thous Thoms
8
score
0.35
6
0.97
0.25
or
D.ST
0.20
0.77
0.15
02
04
0.8
0.17
0.8T
0.67
0.4I
0.27
Similarily (text)
Diffusion Timestop
Dillusion Timestep
(6) Tangent space distance and bext similarity (b) Coeflicienl of determination
(c) Geodesic distance accoss Lizzelleps
Figure 10: Similar prompts create similar tangent spaces, and the impact of the prompt decreases as the generative process progresses. (a) The horizontal axis represents the CLIP similarity between two different prompts, while the vertical axis represents the geodesic distance in the tangent space from each prompt. Different colors represent various diffusion timesteps. negative relationship is observed between prompt similarity and tangent space distance. (b) The R² score the linear regression between clip similarity and geodesic distance of tangent spaces decreases throughout the generative process. (c) Each point represents the distance between tangent spaces created from different prompts. Until around = 0.77', the distance between tangent spaces is very large, but it gradually decreases thereafter. This indicates that the influence of the prompt the tangent space diminishes.
difference in timestep in Figure (b). As expected, find that DMs trained on datasets, that generally considered simpler, have similar local tangent spaces over time.
4.3 Effect of conditioning prompts the latent structure
In this subsection, we aim to investigate how prompts influence the generative process from a geometrical perspective. We randomly sampled 50 captions from the MS-COCO dataset [29] and used them as text conditions.
Similar text conditions induce similar tangent spaces. In Figure 10 (a), we observe a negative correlation between the CLIP similarity of texts and the distance between tangent spaces. In other words, when provided with similar texts, the tangent spaces are more similar. The linear relationship between the text and the discrepancy of the tangent spaces particularly strong in the early phase of the generative process as shown by R² score in Figure 10 (b).
The generative process depends on text conditions in later timesteps. Figure 10 (c) illustrates the distances between local tangent spaces for given different prompts with respect to the timesteps.

---

Image
(Beard)
Real image
+V;
(a) Entanglement of semantic directions
(b) Abrupt changes
Figure 11: Limitations. (a) Entanglement between attributes due to dataset biases (b) Abrupt changes in Stable Diffusion.
Notably, as the diffusion timestep approaches values below 0.7T, the distances between the local tangent spaces start to decrease. It implies that the variation due walking along the local tangent basis depends less on the text conditions, i.e., the text less influences the generative process, in later timesteps. It is a possible reason why the correlation between the similarity of prompts and the similarity of tangent spaces reduces over timesteps.
5 Discussion
In this section, we provide additional intuitions and implications. It interesting that our latent basis usually conveys disentangled attributes though we do not adopt attribute annotation to enforce disentanglement. We suppose that decomposing the Jacobian of the encoder in the U-Nets naturally yields disentanglement
some extent. However, it does and some directions are entangled. For example, the editing for beard converts a female subject to a male shown in Figure 11 (a). This kind of entanglement often occurs in other editing methods due to the dataset bias: female faces seldom have beard.
guarantee the perfect disentanglement
While our method has shown effectiveness in Stable Diffusion, more research is needed to fully validate its potential. We have observed that some of the discovered latent vector occasionally leads to abrupt changes during the editing process in Stable Diffusion, depicted in Figure 11 (b). This observation highlights the complex geometry of X in achieving seamless editing. Exploring this topic in future research is interesting area to delve into.
Our approach is broadly applicable when the feature space the DM adheres to a Euclidean metric, as demonstrated by H. This characteristic has been observed in the context of U-Net within Kwon al. [26]. would be intriguing to investigate if other architectural designs, especially those similar to transformer structures as introduced in [42, 53], also exhibit Euclidean metric.
Despite these limitations, our method provides significant advance in the field of image editing for DMs, and provides deep understanding of DM through several experiments.
6 Conclusion
We have analyzed the latent space of DMs from a geometrical perspective. We used the pullback metric to identify the latent and tangent bases in X and H. The latent basis found by the pullback metric allows editing images by traversal along the basis. We have observed properties of the bases in two aspects. First, we discovered that the latent bases evolve from low- gighffreuencc components; 2) the discrepancy of tangent spaces from different samples increases along the generative process; and 3) DMs trained on simpler datasets exhibit more consistent tangent spaces over timesteps. Second, we investigated how the latent structure changes based on the text conditions in Stable Diffusion, and discovered that similar prompts make tangent space analogous but its effect becomes weaker over timesteps. We believe that a better understanding of the geometry of DMs will open up new possibilities for adopting DMs useful applications.
7 Acknowledgement
This work was supported in part by the Creative-Pioneering Researchers Program through Seoul National University, the National Research Foundation of Korea (NRF) grant (Grant No. 2022R1 1 AARRIAACC106811 (J. J.). KIAS Individual Grant [AP087501] via the Center for Al and Natural Sciences at Korea Institute for Advanced Study, and the National Research Foundation of Korea (NRF) grant (RS- (RS-2023-00223062)
10

---

References
[1] Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. Styleflow: Attribute-onddttoone exploration of Transactions on Graphics (ToG), 1-21, 2021.
images using conditional continuous normalizing flows. ACM
[2] Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. arXiv preprint arXiv:1710.11379, 2017.
[3] Georgios Arvanitidis, Søren Hauberg, and Bernhard Schölkopf. Geometrically enriched latent spaces. arXiv preprint ar.iv:2008.0055550 2020.
[4] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208-10 18208-18218, 2022.
[5] Yogesh Balaji. Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-iman to--iage- diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.0132. 2022.
[6] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. arXiv preprint arXiv:21 2021.
[7] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogmition, pages 18392-18402, 2023.
[8] Hila Chefer, Yuval Alaluf, Yacl Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and- excite: Attention-based semantic guidance for text-to-image diffusion models. arXiv preprint 2rXiv:2301.13826 2023.
[9] Nutan Chen, Alexej Klushyn. Richard Kurle, Xueyan Jiang, Justin Bayer, and Patrick Smagt. Metrics for deep generative models. In International Conference on Artificial Intelligence and Statistics, pages 1540-1550. PMLR, 2018.
[10] Jaewoong Choi, Junho Lee, Changyeon Yoon, Jung Ho Park, Geonho Hwang, and Myungjoo Kang. Do not escape from the manifold: Discovering the local coordinates on the latent space of gans, arXiv preprint arXiv:2106.0699
2021.
[11] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. IIvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.0m 02938, 2021.
[12] Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11472-11481, 2022.
[13] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3188-8197, 2020.
[14] Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint arXiv:2206. 00941, 2022.
[15] Giannis Daras and Alexandros Dimakis. Multiresolution textual inversion. arXiv preprint arXiv:2211.17115. 2022.
[16] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li. Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[17] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.

---

[18] Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
[19] René Haas, Inbar Huberman-Spiegelgee Rotem Mulayoff, and Tomer Michaeli. Discover- ing interpretable directions in the semantic latent space of diffusion models. arXiv preprint 1rX1v32303.1103, 2023.
[20] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. Advances in Neural Information Processing Systems, 33:9841-9850 2020.
[21] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-promy image editing with cross attention control. arXiv preprint vrXiv22208.0122) 2022.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-683 6851, 2020.
[23] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations. 2018.
[24] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pultern recognition, pages 4401-4410, 2019.
[25] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364 2022.
[26] Mingi Kwon, Jaescok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. arXiv preprint arXiv:2210. 1n..22000.09000. 2022.
[27] Yonghyeon Lee and Frank C Park. On explicit curvature regularization in deep generative models. 2023.
[28] Yonghyeon Lee, Seungyeon Kim, Jinwon Choi, and Frank Park. A statistical manifold frame- work for point cloud data. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pages 12402. PMLR, 17-23 Jul 2022. URL https: //pro/ sttpss//pprcceedingg mlr. press prrss/vv62// htep/vvv2/1eee/ 2ed/dd..e22d2222eeee v162//ee222eeee html.
[29] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt 13, pages 740-755. Springer, 2014.
[30] Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. arXiv preprint arXiv:2112.05744, 2021.
[31] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461-11471, 2022.
[32] Chaofan Ma, Yuhuan Yang. Chen Ju, Fei Zhang, Jinxiang Liu, Yu Wang, Ya Zhang, and Yanfeng Wang. Diffusionseg: Adapting diffusion towards unsupervised object discovery. arXiv preprint arX1v:2303.09813, 2023.
[33] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.0m 010033 2021.
[34] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint 152.57780.. 2018
12

---

[35] Ron Mokady, Amir Hertz, Kfrr Aberman, Yael Pritch, and Daniel Cohen- Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794, 2022.
[36] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 10741, 2021.
[37] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[38] Xingang Pan, Ayush Tewari, Thomas Leimküüler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your gan: Interactive point-based manipulation on the generative image manifold. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11, 2023.
[39] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun- Zhu. Zero-shot mmage-to-imagee translation. arXiv preprint arXivv23322.33027, 2023.
[40] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic diffcrentiation in pytorch. In NIPS- W, 2017.
[41] Or Patashmik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2085-2094, 2021.
[42] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748- 8763. PMLR, 2021.
[44] Aditya Ramesh, Youngduck Choi, and Yann LeCun. A spectral regularizer for unsupervised disentanglement. arXiv preprint arXiv:1812.0116/ 2018.
[45] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. 2022.
[46] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[47] Vikash Sehwag, Caner Hazirbas, Albert Gordo, Firat Ozgenel, and Cristian Canton. Generating high fidelity data from low-density regions using diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 1449-11110,, 2022.
[48] Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The riemannian geometry of deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 315 1135-22,,20 2018.
[49] Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 1540-1540, 2021.
[50] Jascha Sohl-Dickstein Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[51] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 2010:22010.02502, 2020.
13

---

[52] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprini arXiv:201 13456, 2020.
[53] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209.1499 2022.
[54] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven imaeeto-immage to-image translation. arXiv preprint arXiv:2211.12572 2022.
[55] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. arXiv preprint arXiv:2303.04803 2023.
[56] Ke Ye and Lek-Heng Lim. Schubert varieties and distances between subspaces of different dimensions. SIAM Journal on Matrix Analysis and Applications, 37(3):1 2016.
[57] LEE Yonghyeon, Sangwoong Yoon, MinJun Son, and Frank C Park. Regularized autoencoders for isometric representation learning. In International Conference on Learning Representations, 2021.
[58] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 150.. 03366 2015.
[59] Oğuz Kaan Yüksel, Enis Simsar, Ezgi Gülperi Er, and Pinar Yanardag. Latentelr: contrastive learning approach for unsupervised discovery of interpretable directions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14263-14272, 2021.
[60] Jiapeng Zhu, Ruili Feng, Yujun Shen, Deli Zhao, Zheng-Jun Zha, Jingren Zhou, and Qifeng Chen. Low-rank subspaces in gans. Advances in Neural Information Processing Systems, 34: 16648-16658, 2021.
14


Reviews:
[{"review_id": "zDYH9EnidJ", "reviewer": "Reviewer_cKEc", "summary": "The paper proposes a study on the latent space of diffusion models and on how to manipulate it. It takes advantage of an observation made by previous work [22] on the flatness and semantic structure of the U-Net model used in  DDIM and uses pullback metric from the latent space of the U-Net to the space of diffusion to measure some properties of the latter under different conditions. \nThe paper also proposes a method to manipulate the diffusion space through the different time steps, so as to carry out semantically meaningful edits.", "questions": "I was wondering what would happen if, instead of moving along one of the principal axis of T_H, you use directly the principal axis of T_x. \n\nI would also discuss a bit more method 22 in the related work since it seems related to the proposed method.\n\nWriting issues:\n - Check the sentence at rows 74-75\n- Row 152: we aims\n- general grammar check", "limitations": "I don’t foresee any particular negative societal impact. A discussion on how the proposed study may generalize to other domains and architectures would be of value.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper is one of the first to study the behavior of the space diffusion models. It presents some interesting studies on the behavior of the process during time, showing that the early stages convey higher frequency while the last steps are more concerned with higher frequencies. Another interesting observation is that tangent spaces of different samples tend to be more aligned at T=1, while they diverge toward T=0 (end of the generative process). \n\nThe paper also shows (even if it had already been observed in 22) that the pullback metric is effective in transferring the shift along the semantically meaningful principal components of the U-Net latent space into the diffusion process, thus resulting in meaningful edits of the generated image, which frequency depends on the time the edit was performed.", "weaknesses": "I may have misunderstood or missed some important information, but the method described is not really clear. Specifically, it is not really clear to me how the editing process works (sections 3.3 and 3.4):\n- In 3.3, the letter v is used to indicate elements of both T_x and T_h, so it is not always clear to which space they are referring.\n- In general, it is not clear why the idea expressed in 3.3 is useful and where it was adopted.\n- In eq 4 what is the epsilon function? In general, isn’t the vector toward which to shift selected from T_H (so it should be u) and then transferred to T_x?\n\n\nAnother concern is about the generalization of the proposed method to other diffusion techniques, or with other score models (i.e. not UNet). I think that this point needs more discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the strengths of our paper: first to study the behavior of the space of diffusion models (DMs), e.g., the coarse-to-fine behavior, the divergence of tangent space across different samples, and the meaningful image editing with pullback metric.\n\n---\n\n> [W1] It is not really clear to me how the editing process works (sections 3.3 and 3.4)\n\n[W1] Please refer to the global rebuttal 2.\n\n> + [W1-a] In 3.3, the letter v is used to indicate elements of both T_x and T_h, so it is not always clear to which space they are referring.\n\n[W1-a] We thank you for identifying the typo, $\\mathbf{v} \\in \\mathcal{T_\\mathbf{h}}$, in Section 3.3 Line 148. It should be corrected to $\\mathcal{T_\\mathbf{h}} \\rightarrow \\mathcal{T_\\mathbf{x}}$. The vector notations, $\\mathbf{v}\\in\\mathcal{T_\\mathbf{x}}$ and $\\mathbf{u}\\in\\mathcal{T_\\mathbf{h}}$ are specifically employed for each space.\n\n> + [W1-b] In general, it is not clear why the idea expressed in 3.3 is useful and where it was adopted.\n\n[W1-b] Parallel transport (P.T.) is a concept in differential geometry that involves transporting a vector between spaces while maintaining its direction relative to the space. We use the P.T. for two purposes:\n\n* First, we use it for image editing. Considering the inherent characteristics of the unsupervised image editing method, it becomes imperative to manually inspect the semantic relevance of the latent basis vector within the edited results. Let us consider a scenario where our aim is to edit ten images of straight hair into curly hair. If we do not use P.T., we have to manually find a straight-to-curly basis vector for individual samples. P.T. allows transporting a straight-to-curly basis vector in one sample to all other samples to edit all images with only one manual inspection. \n\n* Second, we employ it to verify the similarities among local geometrical structures across various samples as shown in Fig. 7. This empirical demonstration substantiates that the geodesic distance among local geometrical structures holds tangible significance rather than being merely a conceptual measure as depicted in Fig. 6. \n\n> + [W1-c-i] In eq 4 what is the epsilon function? [W1-c-ii] In general, isn’t the vector toward which to shift selected from T_H (so it should be u) and then transferred to T_x?\n\n[W1-c-i] $\\epsilon$ is the denoising function of the pretrained DM. For clarity, we will explicitly state this as $\\epsilon^{\\theta}$ in the revised manuscript:\n\n>> (L166) ... where $\\epsilon^{\\theta}$ is the denoising function of the pretrained DM, and $\\gamma$ is ...\n\n[W1-c-ii] Through the decomposition of the Jacobian, we can simultaneously obtain both $\\mathbf{v}$ and its corresponding $\\mathbf{u}$. However, we use only $\\mathbf{v}$ during the editing procedure. \n\n---\n\n> [W2] Another concern is about the generalization of the proposed method to other diffusion techniques, or with other score models (i.e. not UNet). I think that this point needs more discussion.\n\n> [L1] A discussion on how the proposed study may generalize to other domains and architectures would be of value.\n\n[W2, L1] Thank you for raising this important issue. We add the discussion in the revised manuscript as follows: \n\n>> Our approach exhibits broad applicability in cases where a feature space within the DM possesses a Euclidean metric, as exemplified by $\\mathcal{H}$. This characteristic has been observed in the context of U-Net within [Asyrp]. The question of whether alternative architectures, such as those resembling the structures of [DiT] or [MotionDiffusion], could also manifest a Euclidean metric, presents an intriguing avenue for future investigation.\n\n---\n\n> [Q1] I was wondering what would happen if, instead of moving along one of the principal axis of T_H, you use directly the principal axis of T_x.\n\n[Q1] As previously mentioned in [W1-c-ii], we directly manipulate the latent variable $\\mathbf{x}_t$ with the discovered latent basis vector $\\mathbf{v}$. This approach offers several advantages over the editing of $\\mathcal{H}$ as described in [Asyrp]. First, direct control over $\\mathbf{h}_t$ disrupts the coherence between inner features and skip connections within the U-Net, which can lead to artifacts, particularly when making substantial adjustments [DiffStyle]. As a result, [Asyrp] implements gradual changes across multiple timesteps. In contrast, the manipulation of $\\mathbf{x}_t$ enables more substantial alternations within a single timestep. Second, unlike $\\mathbf{u}$ which exclusively affects the deepest feature map, $\\mathbf{v}$ exerts influence not only on the latent variable $\\mathbf{x}_t$ but also on all the feature map inside the DMs.\n\n---\n\n> [Q2] I would also discuss a bit more method [25] in the related work since it seems related to the proposed method.\n\n[Q2] To enhance the distinction between our paper and [Asyrp], we modified the related work as follows:\n\n>> ... Kwon et al. [25] demonstrated that the bottleneck of the U-Net, $\\mathcal{H}$, can be used as a semantic latent space. Specifically, they used CLIP to identify directions within $\\mathcal{H}$ that facilitate genuine image editing. ... In contrast to the method, our editing approach involves the direct manipulation of latent variables within the latent space. Furthermore, we autonomously identify editing directions in an unsupervised manner.\n\n---\n\n> [Q3] Writing issues:\n> + Check the sentence at rows 74-75\n> + Row 152: we aims\n> + general grammar check\n\n[Q3] Thank you for the careful comments. We will address and incorporate these revisions in the upcoming manuscript.\n\n**References**\n\n[Asyrp] : Diffusion Models already have a Semantic Latent Space, Kwon et al., 2022\n\n[DiT] : Scalable Diffusion Models with Transformers, Peebles et al., 2022\n\n[MotionDiffusion] : Human Motion Diffusion Model, Tevet et al., 2022\n\n[DiffStyle] : Training-free Style Transfer Emerges from h-space in Diffusion models, Jeong et al., 2023"}, {"review_id": "J9lE3G6Alu", "reviewer": "Reviewer_vuQy", "summary": "This paper studies the geometry of latent spaces of diffusion models (Dms) using the pullback metric. In the analyses, they mainly examine change of the frequency representations in latent spaces over time and the change of the structure based on text conditioning.\n\nAfter the rebuttal:\n\nI checked all reviewer comments and responses.\n\nI agree with the other reviewers regarding limited algorithmic novelty of the work. Therefore, I keep my original score.", "questions": "It is stated that “To investigate the geometry of the tangent basis, we employ a metric on the Grassmannian manifold.” However, the space could be identified by another manifold as well.  Why and how did you define the space by the Grassmannian manifold?\n\nIt is claimed that “the similarity across tangent spaces allows us to effectively transfer the latent basis from one sample to another through parallel transport”. How this improves effectiveness was not analyzed. In general, how do the proposed methods improve training and inference time? Indeed, the additional transformations can increase training and inference time. Could you please provide an analysis of the footprints?", "limitations": "Some of the limitations were addressed but potential impacts were not addressed.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is well written in general (there are several typos but they can be fixed in proof-reading).\n\n- The proposed methods and analyses are interesting. Some of the theoretical results highlight several important properties of diffusion models.", "weaknesses": "1. Some of the statements and claims are not clear as pointed in the Questions.\n\n2. The results are given considering the Riemannian geometry of the latent spaces and utilizing the related transformations (e.g. PT) among tangent spaces on the manifolds. However, vanilla DMs do not employ these transformations. Therefore, it is not clear whether these results are for vanilla DMs or the  DMs utilizing the proposed transformations.\n\n3. A major claim is that the proposed methods improve effectiveness of the DMs. However, the employed transformations can increase the footprints of DMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> [W1] Some of the statements and claims are not clear as pointed in the Questions.\n\n> [Q1] It is stated that \"we employ a metric on the Grassmannian manifold.” Why and how did you define the space by the Grassmannian manifold?\n\n[Q1] These subspaces (as a vector space) of $\\mathcal{T_{\\mathbf{x}}}$ and $\\mathcal{T_{\\mathbf{h}}}$ assigned at the points $\\mathbf{x} \\in \\mathcal{X}$ and $\\mathbf{h} \\in \\mathcal{H}$ of diffusion models. Given that the Grassmannian manifold is characterized as a manifold comprising subspaces, it appears well-suited for representing the manifold of $\\mathcal{T_{\\mathbf{h}}}$. Additionally, the geodesic metric defined on this manifold quantifies the separation between two subspaces based on their principal angles. Hence, we posit that this geodesic metric offers a robust means of evaluating the similarity between different $\\mathcal{T_{\\mathbf{h}}}$s.\n\n> [Q2-a] It is claimed that “the similarity across tangent spaces allows us to effectively transfer the latent basis from one sample to another through parallel transport”. How this improves effectiveness was not analyzed. [Q2-b] In general, how do the proposed methods improve training and inference time? (...) Could you please provide an analysis of the footprints?\n\n[Q2-a] Figure 7 shows the impact of employing parallel transport (P.T.). The transported vectors can edit distinct samples while manipulating the same attributes. We will modify L227-299 as follows to better reflect our intent:\n\n>> (L227-229) $\\rightarrow$ ... us to successfully transfer the latent basis vector from one sample to another through parallel transport. Figure 7 shows that the transported vectors can edit distinct samples while manipulating the same attributes.\n\n[Q2-b] We would like to emphasize that our approach does not require any form of training. Furthermore, the utilization of P.T. is not a default procedure in the editing process; its application is limited to specific instances where local basis vectors from other samples are used. The details regarding the typical editing footprint in the absence of P.T. are outlined in the global rebuttal 3. Here, for your reference, we focus solely on presenting the result table:\n\n|Image Edit Method|running time|Preprocessing|\n|:-:|:-:|:-:|\n|Ours| 11 sec |N/A|\n|[SDEdit]|4 sec|N/A|\n|[Pix2Pix-zero]|25 sec|4 min|\n|[PnP]|10 sec|40 sec|\n|[Instruct Pix2Pix]|11 sec|N/A|\n\nThe time taken for editing using P.T. can be broken down into three components: 1) DDIM inversion and generation, 2) identification of the local basis, and 3) the parallel transport process. For optimal results when employing P.T., it is advisable to use a sufficiently large value of $n$ to mitigate distortion. However, this choice necessitates more computation time for generating the local basis. In our case, we opt for $n=50$, with the number of DDIM steps set at $100$, and utilizing an unconditional diffusion model trained on CelebA-HQ.\n\n|DDIM inversion + generation|Identification of local basis|Parallel transport|\n|:-:|:-:|:-:|\n| 10sec | 100sec | 0.002sec|\n---\n> [W2] The results are given ... utilizing the related transformations (e.g. PT) among tangent spaces on the manifolds. However, vanilla DMs do not employ these transformations. Therefore, it is not clear whether these results are for vanilla DMs or the DMs utilizing the proposed transformations.\n\n[W2] We wish to emphasize that our method works on frozen vanilla DMs without necessitating any fine-tuning or architectural modifications. It offers unsupervised image editing capabilities that are applicable to both unconditional DMs and conditional DMs. Furthermore, as explained in [Q2], it is important to note that the inclusion of P.T. is not a default step in our editing process. This technique is selectively employed in particular cases where local basis vectors are transferred for editing other samples.\n\n> [W3] A major claim is that the proposed methods improve the effectiveness of the DMs. However, the employed transformations can increase the footprints of DMs.\n\n[W3] To provide clarification, our primary focus centers on image editing utilizing DMs, without an emphasis on augmenting the performance of DMs themselves. \n\nWhen performing image editing with a single latent basis vector (i.e., $n=1$), our editing process consumes approximately 11 seconds. This represents roughly 15% of the time required for vanilla inversion and reconstruction. Furthermore, even when employing Jacobian approximation with $n=1$, the computational overhead associated with our approach remains comparable with other state-of-the-art editing methods, as illustrated in the aforementioned table.\n\n> [L1] Some of the limitations were addressed but potential impacts were not addressed.\n\n[L1] We appreciate your observation and would like to address this concern by incorporating a societal impact and ethics statement into the revised manuscript, which is presented below:\n\n>> **Societal Impact / Ethics Statement.** Our research endeavors to unravel the geometric structures of the diffusion model and facilitate high-quality image editing within its framework. While our primary application resides within the creative realm, it is important to acknowledge that image manipulation techniques, such as the one proposed in our method, hold the potential for misuse, including the dissemination of misinformation or potential privacy implications. Therefore, the continuous advancement of technologies aimed at thwarting or identifying manipulations rooted in generative models remains of utmost significance.\n\n**References**\n\n[SDEdit] : SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations, Meng et al., 2021\n\n[Pix2Pix-zero] : Zero-shot Image-to-Image Translation, Parmar et al., 2023\n\n[PnP] : Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation, Tumanyan et al., 2022\n\n[Instruct Pix2Pix] : InstructPix2Pix: Learning to Follow Image Editing Instructions, Brooks et al., 2022"}, {"review_id": "heUXC7Xx2C", "reviewer": "Reviewer_6jtZ", "summary": "In this submission, the authors probe the latent space, xt ∈ X, of diffusion models (DMs) from a geometric perspective, utilizing the pullback metric to identify local latent basis in X and corresponding local tangent basis in H. To confirm their findings, they edit images via latent space traversal. The authors provide a two-pronged analysis, investigating the evolution of geometric structure over time and its variation based on text conditioning in Stable Diffusion. Notably, they discovered that in the generative process, the model prioritizes low-frequency components initially, moving to high-frequency details later, and that the model's dependence on text conditions reduces over time. The paper introduces image editing through x-space traversal and to offer comprehensive analyses of the latent structure of DMs, with a specific emphasis on the use of the pullback metric and the SVD of the Jacobian in computing a basis.", "questions": "What is the complexity of computing the Jacobian? What is the the actual runtime (in seconds) of the approach compared to other editing methods? I think answering these questions can provide a good context for readers. ", "limitations": "Societal impact has been properly address but I would like to see a deeper analysis of the computational complexity and runtime of the approach. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "\nThe paper presents a distinctive idea that provides an alternative method for editing in diffusion models, as well as enhancing comprehension of the latent space dynamics. By utilizing a geometric perspective, the authors make use of the pullback metric to investigate the latent space, offering insights into its structure and operation.  The exploration of the evolution of geometric structure over time and its response to various text conditions offers additional insights into the dynamics of the latent space is interesting. ", "weaknesses": "\nThe first area where the paper could see improvement is in terms of the clarity of its analysis. Given its nature as an analysis paper, it's crucial that the analysis presented is as comprehensible as possible. However, the method and notation used in this work can lead to some confusion. For instance, Section 3, in its current form, may not be as accessible to all readers as it could be, and it could benefit from being revised for clearer communication of the ideas contained therein. Additionally, Fig. 1, which is presumably intended to illustrate key concepts, is perhaps too dense with information. Dividing Fig. 1 into two separate figures could make it easier to digest, enabling a clearer explanation of the approach.\n\nA second aspect that could be improved upon is the overall presentation and proofreading of the paper. While the approach is relatively simple, its translation into the written form has not been as clear as one would hope. The text could benefit from a thorough proofreading to ensure that it is not just grammatically correct, but also that it conveys the authors' ideas in a way that is accessible to the wider machine learning community. As it stands, the paper's usefulness to this community may be hindered by its presentation.\n\nLastly, the paper could do more to address the computational implications of its approach. The authors use the power method to approximate the Jacobian, which, while effective, can be computationally costly. It would be beneficial if the authors were more transparent about this fact, allowing readers to fully understand the computational demands of the approach and evaluate whether or not it would be feasible in their own applications. Being upfront about such limitations can help to build a more honest and comprehensive understanding of the paper's methodologies and implications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the strengths of our paper: the distinctive idea for editing in diffusion models (DMs), and enhancing comprehension of the latent space dynamics, e.g., the evolution of geometric structure over time and the influence of various text conditions.\n\n---\n\n> [W1] ... the method and notation used in this work can lead to some confusion. For instance, [W1-a] Section 3, in its current form, may not be as accessible to all readers as it could be, and it could benefit from being revised for clearer communication of the ideas contained therein. [W1-b] Additionally, Fig. 1, which is presumably intended to illustrate key concepts, is perhaps too dense with information. Dividing Fig. 1 into two separate figures could make it easier to digest, enabling a clearer explanation of the approach.\n\n[W1] Thank you for the constructive suggestion.\n\n[W1-a] To clarify the image editing process in Section 3, we added a new subsection summarizing the overall procedure. This includes a visual overview in Fig. R2 and clear textual explanations. Please refer to global rebuttal 2 for a more detailed description.\n\n[W1-b] To make Fig. 1 more comprehensible, we divide it into two figures, Fig. R1 and R2. Please refer to the global rebuttal 1 for more detailed improvements.\n\n---\n\n> [W2] A second aspect that could be improved upon is the overall presentation and proofreading of the paper. While the approach is relatively simple, its translation into the written form has not been as clear as one would hope. The text could benefit from a thorough proofreading to ensure that it is not just grammatically correct, but also that it conveys the authors' ideas in a way that is accessible to the wider machine learning community. As it stands, the paper's usefulness to this community may be hindered by its presentation.\n\n[W2] Thank you for the advice on readability. We made the following improvements to help readers understand more easily:\n\n+ We relocate the discussion on parallel transport (P.T.) from section 3.3 to the final subsection of section 3. This alteration was implemented to prevent any potential misunderstanding that using P.T. is the default approach in our editing method. By introducing P.T. as a special case after concluding the explanation of the image editing method, we aim to mitigate such misconceptions.\n\n+ We consolidate all elements related to the image editing method within section 3. For instance, the process of generating initial noise $\\mathbf{x}_T$ through DDIM inversion, previously outlined in section 4 (L178-182), has been now moved to the newly introduced subsection referenced in global rebuttal 2. This organization ensures that readers can fully comprehend the entire editing process by solely referring to section 3.\n\n+ We make adjustments to Figure 8. Specifically, we have relocated Fig. 8(b) and the content spanning from L243 to L248 to the appendix. This decision was based on the rationale that Figure 8(b) primarily serves to validate the findings in (a) and (c). Including this information in the appendix was deemed a more suitable arrangement.\n\nWe will also carefully identify further shortcomings and revise them in the camera-ready version.\n\n---\n\n> [W3] Lastly, the paper could do more to address the computational implications of its approach. The authors use the power method to approximate the Jacobian, which, while effective, can be computationally costly. It would be beneficial if the authors were more transparent about this fact, allowing readers to fully understand the computational demands of the approach and evaluate whether or not it would be feasible in their own applications. ...\n\n> [L1] I would like to see a deeper analysis of the computational complexity and runtime of the approach.\n\n[W3, L1] In Appendix A, we present the computation time required for approximating the Singular Value Decomposition (SVD) of the Jacobian using the power method. The runtime of this power method varies based on the parameter $n$, which denotes the low-rank approximation of the original tangent space. Smaller values of $n$ lead to shorter computation times. For instance, when $n=3$, the process takes approximately 10 seconds. In contrast, for $n=50$, the computation time extends to around 3 to 4 minutes, particularly in the context of Stable Diffusion.\n\nNotably, when only a single basis vector is needed, as is the case in scenarios like text conditional editing, the time taken to approximate the SVD of the Jacobian is remarkably brief—approximately 2.5 seconds.\n\n---\n\n> [Q1] What is the complexity of computing the Jacobian? What is the actual runtime (in seconds) of the approach compared to other editing methods? I think answering these questions can provide a good context for readers.\n\n[Q1] Thank you for your valuable question. We conduct all comparisons on the Nvidia 3090. To ensure a fair comparison, we set $n=1$ and performed 50 steps of the DDIM algorithm. The time taken by each method is as follows:\n\n| Image Edit Method | running time  | Preprocessing |\n|:-----------------:|:-------------:|:-------------:|\n|  Ours             |     11 sec    |N/A            |\n|  SDEdit           |      4 sec    |N/A            |\n|  Pix2Pix-zero     |     25 sec    |4 min*             |\n|  PnP              |     10 sec    |40 sec**  |\n|  Instruct Pix2Pix |     11 sec    |N/A     |\n\n(*: generating 100 prompts with GPT, obtaining embedding vectors, **: storing feature vectors, queries, and key values)\n\nFor a more detailed discussion, please refer to global rebuttal paragraph 3.\n\n---\n\n**References**\n\n[SDEdit] : SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations, Meng et al., 2021\n\n[Pix2Pix-zero] : Zero-shot Image-to-Image Translation, Parmar et al., 2023\n\n[PnP] : Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation, Tumanyan et al., 2022\n\n[Instruct Pix2Pix] : InstructPix2Pix: Learning to Follow Image Editing Instructions, Brooks et al., 2022\n"}, {"review_id": "rHMZaoHnws", "reviewer": "Reviewer_iEgM", "summary": "The paper presents an analysis of the latent structure of diffusion models using differential geometry. The authors propose a method to define a geometry in the latent space by pulling back the Euclidean metric from the U-Net bottleneck space *H* via the network encoder. This approach enables the identification of directions of maximum variation in the latent space. The paper also explores the application of the proposed latent structure guidance for image editing. Finally, the evolution of the geometric structure over time steps and its dependence on text conditioning are investigated.", "questions": "To make the paper applicable to real-world scenarios right away, the authors should include a comparative analysis with other image editing techniques.\n\n**Some potentially interesting references**\n- Pan, Xingang, et al. \"Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold.\" arXiv preprint arXiv:2305.10973 (2023).\n- Brooks, Tim, Aleksander Holynski, and Alexei A. Efros. \"Instructpix2pix: Learning to follow image editing instructions.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n- Meng, Chenlin, et al. \"Sdedit: Guided image synthesis and editing with stochastic differential equations.\" arXiv preprint arXiv:2108.01073 (2021).\n\n**Typos**\n- line 20: double citation\n- line 160 editted\n- Figure 2 caption: editing", "limitations": "The authors have addressed the limitations of their approach.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- I found the analysis presented in the paper interesting. It both uncovers unknown details about diffusion models (effect of text prompt and complexity of the dataset on the latent space) and confirm some previous observations (e.g., coarse-to-fine behaviour). This exploration can potentially reveal new capabilities of diffusion models, contributing to the advancement of the field.\n- The paper is technically sound, and the claims made by the authors in sections 4 are supported by experiments. This experimental validation enhances the credibility of the proposed approach.", "weaknesses": "- The paper lacks comparisons with other diffusion-based image editing techniques, like [7,18]. Including such comparisons would have provided a more comprehensive evaluation and demonstrated the advantages of the proposed method.\n- The presentation and clarity of the paper could be improved. For example, the abstract contains too much detail, making it challenging to understand upon initial reading. Also the explanation of the image editing technique could be improved: what is DDIM (section 4)? what is epsilon in Equation 4? Finally, Figure 1 is not sufficiently clear to me, it may hampers the reader's comprehension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging our strengths:\n\n- uncovering the effect of text prompt and dataset complexity on the latent space.\n- confirming coarse-to-fine behavior of diffusion models (DMs).\n- enhancing credibility through experimental validation.\n\n---\n\n> [W1] The paper lacks comparisons with other diffusion-based image editing techniques. \n\n> [Q1] To make the paper applicable to real-world scenarios right away, the authors should include a comparative analysis with other image editing techniques.\n\n[W1, Q1] Thanks for the great suggestion. We conduct qualitative comparisons with text-guided image editing methods. Our state-of-the-art baseline methods include: (i) [SDEdit], (ii) [Pix2Pix-zero], (iii) [PnP], and (iv) [Instruct Pix2Pix]. All comparisons were performed using the official code. Please refer to the global rebuttal paragraph 3 and Fig. R3 in the global rebuttal PDF for the results.\n\n---\n\n> [W2] The presentation and clarity of the paper could be improved. [W2-a] For example, the abstract contains too much detail, making it challenging to understand upon initial reading. [W2-b] Also the explanation of the image editing technique could be improved: [W2-b-i] What is DDIM (section 4)? [W2-b-ii] What is epsilon in Equation 4? [W2-b-iii] Finally, Figure 1 is not sufficiently clear to me, it may hamper the reader's comprehension.\n\n[W2-a] Thank you for your constructive suggestion. We revise the abstract to be concise and digestible as follows:\n\n>> Despite the success of diffusion models (DMs), we still lack a thorough understanding of their latent space. To understand the latent space $\\mathbf{x}_t \\in \\mathcal{X}$, we analyze them from a geometrical perspective. Our approach involves deriving the local latent basis within $\\mathcal{X}$ by leveraging the pullback metric associated with their encoding feature maps. Remarkably, our discovered local latent basis enables image editing capabilities by moving $\\mathbf{x}_t$, the latent space of DMs, along the basis vector at specific timesteps.\n>> \n>> We further analyze how the geometric structure of DMs evolves over diffusion timesteps and differs across different text conditions. This confirms the known phenomenon of coarse-to-fine generation, as well as reveals novel insights such as the discrepancy between $\\mathbf{x}_t$ across timesteps, the effect of dataset complexity, and the time-varying influence of text prompts. To the best of our knowledge, this paper is the first to present image editing through $\\mathbf{x}$-space traversal, editing only once at specific timestep $t$ without any additional training, and providing thorough analyses of the latent structure of DMs.\n\n\n[W2-b] Thank you for pointing out the clarity issue. \n\n[W2-b-i] To make the image editing process easier to understand, we created a summary subsection and an overview figure for the whole process. Please refer to global rebuttal paragraph 2 and Fig. R2 in the global rebuttal PDF for more details. In our method, [DDIM] is used to invert the image into the initial noise $\\mathbf{x}_T$, and again for denoising to generate the image.\n\n\n[W2-b-ii] $\\epsilon$ is a function representing the pretrained diffusion model. For clarity, in the revised manuscript we will write it as $\\epsilon^{\\theta}$, and state it explicitly as below :\n\n>> (L166) ... where $\\epsilon^{\\theta}$ is the denoising function of the pretrained diffusion model, and $\\gamma$ is ...\n\n[W2-b-iii] Thank you for the valuable suggestion. Please refer to global rebuttal paragraph 1 and Fig. R1, R2 in the PDF for improvements to Fig. 1. Below is an excerpt from the content of the global rebuttal paragraph 1, extracted to enhance readability for the reviewers:\n\nIn order to make Fig. 1 more comprehensible, we divide Fig. 1 into two separate figures. Please refer to Fig. R1 and R2 in the global rebuttal PDF. First, **Figure R1** conceptually visualizes the local basis found through the pullback metric and provides a summary of the process for obtaining a local basis. Second, **Figure R2** provides an overview of the image editing method using the discovered local basis and briefly presents its results. \n\n---\n\n> Some potentially interesting references\n\nThank you for your valuable references. We will add those suggested references to the revised manuscript.\n\n> Typos\n\nThank you for the careful comments. We will address and incorporate these revisions in the upcoming manuscript.\n\n\n---\n\n**References**\n\n[SDEdit] : SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations, Meng et al., 2021\n\n[Pix2Pix-zero] : Zero-shot Image-to-Image Translation, Parmar et al., 2023\n\n[PnP] : Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation, Tumanyan et al., 2022\n\n[Instruct Pix2Pix] : InstructPix2Pix: Learning to Follow Image Editing Instructions, Brooks et al., 2022\n\n[DDIM] : Denoising Diffusion Implicit Models, Song et al., 2020"}]

Author rebuttal:
We thank the reviewers for their valuable advice. Here, we compile reviews we want to share with all the reviewers. Please see our responses addressing the specific concerns below:

---

### 1. Improving Fig. 1 for clarity
**Reviewers *6jtZ* and *iEgM* suggested modifying Fig. 1 since it has too dense information.**

To enhance the comprehensibility of Fig. 1, we divide Fig. 1 into two figures. Please refer to Fig. R1 and R2 in the attached PDF file. **Figure R1** gives a conceptual visualization of the local basis derived from the pullback metric. This figure succinctly outlines the methods involved in obtaining the local basis. **Figure R2** delivers an overview of the image editing method using the discovered local basis. It also provides a concise presentation of the outcomes.

---

### 2. Additional subsection to overview the whole image editing process
**Reviewers *cKEc*, *6jtZ*, and *iEgM* asked the clear explanation of the editing process.**

We create a new subsection in section 3 that summarizes the entire procedure of image editing. This subsection provides clear explanations and also visually illustrates the method in Fig. R2. The detailed contents are as follows:

>> ### 3.4. The overall process of image editing
>> In this section, we summarize the entire editing process with five steps: 1) The input image is inverted into initial noise $\mathbf{x}_T$ using DDIM inversion. 2) $\mathbf{x}_T$ is gradually denoised until $t$ through DDIM generation. 3) Identify local latent basis $\{ \mathbf{v}_1, \cdots, \mathbf{v}_n \}$ using the pullback metric at $t$. 4) Manipulate $\mathbf{x}_t$ along the one of basis vectors using the $\mathbf{x}$-space guidance. 5) The DDIM generation is then completed with the modified latent variable $\tilde{\mathbf{x}}_t$. Figure R2 illustrates the entire editing process. 
>>
>>In the context of a text-to-image model, such as Stable Diffusion, it becomes possible to include textual conditions while deriving local basis vectors. It aligns all the local basis vectors with the condition text. Comprehensive experiments can be found in Section 4.1.
>>
>>It is noteworthy that while our approach involves moving the latent variable within a single timestep, it achieves semantically meaningful image editing. In addition to the image manipulation within a single timestep, the direct manipulation of the latent variable, $\mathbf{x}_t$, of diffusion models is a pioneering approach to the best of our knowledge. 
 

---


### 3. Comparative experiment to other state-of-the-art (SoTA) editing methods

**Reviewers *iEgM* and *6jtZ* provided constructive discussion about including comparative analysis for qualitative comparisons and runtime aspects.**

We conduct qualitative comparisons with text-guided image editing methods. Our state-of-the-art baseline methods include: (i) [SDEdit], (ii) [Pix2Pix-zero], (iii) [PnP], and (iv) [Instruct Pix2Pix]. All comparisons were performed using the official code. **Please refer to Fig. R3 in the PDF for the qualitative results.** 


We also compare the time complexity of each method. For a fair comparison, we only identify the first singular vector $\mathbf{v}_1$, i.e., $n=1$, and set the number of DDIM steps to 50. All experiments were conducted on an Nvidia RTX 3090. The runtime for each method is as follows:

| Image Edit Method | running time  | Preprocessing |
|:-------------------------:|:------------------:|:-------------------:|
|  Ours             |     11 sec    |N/A           |
|  SDEdit           |      4 sec    |N/A            |
|  Pix2Pix-zero     |     25 sec    |4 min*   |
|  PnP              |     10 sec    |40 sec**        |
|  Instruct Pix2Pix |     11 sec    |N/A            |

(*: generating 100 prompts with GPT, obtaining embedding vectors, **: storing feature vectors, queries, and key values)

The computation cost of our method remains comparable to other approaches, although the Jacobian approximation takes around 2.5 seconds for $n=1$. This is because we only need to identify the latent basis vector once at a specific timestep. Furthermore, our approach does not require additional preprocessing steps like generating 100 prompts with GPT and obtaining embedding vectors (as in Pix2Pix-zero), or storing feature vectors, queries, and key values (as in PnP). Our method also does not require finetuning (as in Instruct Pix2Pix). This leads to a significantly reduced total editing process time in comparison to other methods.

**References**

[SDEdit] : SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations, Meng et al., 2021

[Pix2Pix-zero] : Zero-shot Image-to-Image Translation, Parmar et al., 2023

[PnP] : Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation, Tumanyan et al., 2022

[Instruct Pix2Pix] : InstructPix2Pix: Learning to Follow Image Editing Instructions, Brooks et al., 2022

Meta-review: This paper utilizes a Riemannian pullback metric to analyze the latent space of diffusion models, utilizing this structure to demonstrate how the latent structure changes over time in the generation process, as well as how the latent structure changes under text conditioning in Stable Diffusion. This gives a form of interpretability of the diffusion model as well as a new understanding of the effect of its standard operations. The employed mathematical method also gives new insights.

The reviewers do have several concerns, most notably clarity and computational cost. The latter is perhaps less crucial for a post hoc interpretation method such as this one, but the authors would indeed need to use the feedback from the reviews to improve clarity. The authors should also carefully discuss the limitations of their method, including those brought forward in the review process.

In the post-rebuttal period, one reviewer steps forward as a champion of the paper, in particular arguing 

"I like to see this kind of exploratory papers, since they could lead to a better understanding of the diffusion models and possibly other interesting applications. In particular, in this paper, the observations made about the geometric properties of the diffusion latent space are exploited in a novel (unsupervised) semantic editing, which is probably not as good as SOTA-supervised techniques, but it is still useful to validate the observed properties and appreciate the possible applications."

The AC agrees with this reviewer's opinion, and therefore recommends acceptance. The authors should note that this was a borderline paper, and they should carefully utilize the review feedback to improve clarity and downstream impact.
Decision: Accept (poster)