Title: Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometry
Venue: NeurIPS 2023
Accepted: yes
Keywords: diffusion models

semantic image editing

differential geometry
Abstract: Despite the success of diffusion models (DMs), we still lack a thorough understanding of their latent space. To understand the latent space $\mathbf{x}_t \in \mathcal{X}$, we analyze them from a geometrical perspective. Our approach involves deriving the local latent basis within $\mathcal{X}$ by leveraging the pullback metric associated with their encoding feature maps. Remarkably, our discovered local latent basis enables image editing capabilities by moving $\mathbf{x}_t$, the latent space of DMs, along the basis vector at specific timesteps. We further analyze how the geometric structure of DMs evolves over diffusion timesteps and differs across different text conditions. This confirms the known phenomenon of coarse-to-fine generation, as well as reveals novel insights such as the discrepancy between $\mathbf{x}_t$ across timesteps, the effect of dataset complexity, and the time-varying influence of text prompts. To the best of our knowledge, this paper is the first to present image editing through $\mathbf{x}$-space traversal, editing only once at specific timestep $t$ without any additional training, and providing thorough analyses of the latent structure of DMs.
The code to reproduce our experiments can be found at the [link](https://github.com/enkeejunior1/Diffusion-Pullback).

Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometry

Abstract

Despite the success of diffusion models (DMs), we still lack a thorough understanding of their latent space. To understand the latent space x t ∈ X , we analyze them from a geometrical perspective. Our approach involves deriving the local latent basis within X by leveraging the pullback metric associated with their encoding feature maps. Remarkably, our discovered local latent basis enables image editing capabilities by moving x t , the latent space of DMs, along the basis vector at specific timesteps. We further analyze how the geometric structure of DMs evolves over diffusion timesteps and differs across different text conditions. This confirms the known phenomenon of coarse-to-fine generation, as well as reveals novel insights such as the discrepancy between x t across timesteps, the effect of dataset complexity, and the time-varying influence of text prompts. To the best of our knowledge, this paper is the first to present image editing through x -space traversal, editing only once at specific timestep t without any additional training, and providing thorough analyses of the latent structure of DMs. The code to reproduce our experiments can be found at https://github.com/enkeejunior1/Diffusion-Pullback .

1 Introduction

The diffusion models (DMs) are powerful generative models that have demonstrated impressive performance [22, 51, 52, 17, 37]. DMs have shown remarkable applications, including text-to-image synthesis [45, 46, 5, 36], inverse problems [14, 31], and image editing [21, 54, 39, 35].

Despite their achievements, the research community lacks a comprehensive understanding of the latent space of DMs and its influence on the generated results. So far, the completely diffused images are considered as latent variables but it does not have useful properties for controlling the results. For example, traversing along a direction from a latent produces weird changes in the results. Fortunately, Kwon et al. [26] consider the intermediate feature space of the diffusion kernel, referred to as H , as a semantic latent space and show its usefulness on controlling generated images. In the similar sense, some works investigate the feature maps of the self-attention or cross-attention operations for controlling the results [21, 54, 39], improving sample quality [8], or downstream tasks such as semantic segmentation [32, 55].

Still, the structure of the space X t where latent variables { x t } live remains unexplored despite its crucial role in understanding DMs. It is especially challenging because 1) the model is trained to estimate the forward noise which does not depend on the input, as opposed to other typical supervisions such as classification or similarity, and 2) there are lots of latent variables over multiple recursive timesteps. In this paper, we aim to analyze X in conjunction with its corresponding

Latent space

structure

Feature space

structure

Txt

Diffu

X

f

H

Tht

V

112

Jxt:Jacobian of f at xt

Vi： i-th right singular vector of Jxt

V

Tx:span{V1,...,Vn}

u: i-th left singular vector of Jxe

Th: span{u,...,un}

Latent space ()

Intermediate Feature Space (H)

(a) Conceptual illustration

(b) Notation

Diffusion

model

(U-Net)

representation H , by incorporating a local geometry to X using the concept of a pullback metric in Riemannian geometry.

First, we discover the local latent basis for X and the corresponding local tangent basis for H . The local basis is obtained by performing singular value decomposition (SVD) of the Jacobian of the mapping from X to H . To validate the discovered local latent basis, we demonstrate that walking along the basis can edit real images in a semantically meaningful way. Furthermore, we can use the discovered local latent basis vector to edit other samples by using parallel transport, when they exhibit comparable local geometric structures. Note that existing editing methods manipulate the self-attention map or cross-attention map over multiple timesteps [21, 54, 39]. On the other hand, we manipulate only x t once at a specific timestep.

Second, we investigate how the latent structures differ across different timesteps and samples as follows. The frequency domain of the local latent basis shifts from low-frequency to high-frequency along the generative process. We explicitly confirm it using power spectral density analysis. The difference between local tangent spaces of different samples becomes larger along the generative process. The local tangent spaces at various diffusion timesteps are similar to each other if the model is trained on aligned datasets such as CelebA-HQ or Flowers. However, this homogeneity does not occur on complex datasets such as ImageNet.

Finally, we examine how the prompts affect the latent structure of text-to-image DMs as follows. Similar prompts yield similar latent structures. Specifically, we find a positive correlation between the similarity of prompts and the similarity of local tangent spaces. The influence of text on the local tangent space becomes weaker along the generative process.

Our work examines the geometry of X and H using Riemannian geometry. We discover the latent structure of X and how it evolves during the generative process and is influenced by prompts. This geometric exploration deepens our understanding of DMs.

2 Related works

Diffusion Models. Recent advances in DMs make great progress in the field of image synthesis and show state-of-the-art performance [50, 22, 51]. An important subject in the diffusion model is the introduction of gradient guidance, including classifier-free guidance, to control the generative process [17, 47, 4, 30, 36, 46]. The work by Song et al. [52] has facilitated the unification of DMs with score-based models using SDEs, enhancing our understanding of DMs as a reverse diffusion process. However, the latent space is still largely unexplored, and our understanding is limited.

The study of latent space in GANs. The study of latent spaces has gained significant attention in recent years. In the field of Generative Adversarial Networks (GANs), researchers have proposed various methods to manipulate the latent space to achieve the desired effect in the generated images

[44, 41, 1, 20, 49, 59, 38]. More recently, several studies [60, 10] have examined the geometrical properties of latent space in GANs and utilized these findings for image manipulations. These studies bring the advantage of better understanding the characteristics of the latent space and facilitating the analysis and utilization of GANs. In contrast, the latent space of DMs remains poorly understood, making it difficult to fully utilize their capabilities.

Image manipulation in DMs. Early works include Choi et al. [11] and Meng et al. [33] have attempted to manipulate the resulting images of DMs by replacing latent variables, allowing the generation of desired random images. However, due to the lack of semantics in the latent variables of DMs, current approaches have critical problems with semantic image editing. Alternative approaches have explored the potential of using the feature space within the U-Net for semantic image manipulation. For example, Kwon et al. [26] have shown that the bottleneck of the U-Net, H , can be used as a semantic latent space. Specifically, they used CLIP [43] to identify directions within H that facilitate genuine image editing. Baranchuk et al. [6] and Tumanyan et al. [54] use the feature map of the U-Net for semantic segmentation and maintaining the structure of generated images. Unlike previous works, our editing method finds the editing direction without supervision, and directly traverses the latent variable along the latent basis.

Riemannain Geometry. Some studies have applied Riemannian geometry to analyze the latent spaces of deep generative models, such as Variational Autoencoders (VAEs) and GANs [2, 48, 9, 3, 27, 28, 57]. Shao et al. [48] proposed a pullback metric on the latent space from image space Euclidean metric to analyze the latent space's geometry. This method has been widely used in V AEs and GANs because it only requires a differentiable map from latent space to image space. However, no studies have investigated the geometry of latent space of DMs utilizing the pullback metric.

3 Discovering the latent basis of DMs

In this section, we explain how to extract a latent structure of X using differential geometry. First, we introduce a key concept in our method: the pullback metric . Next, by adopting the local Euclidean metric of H and utilizing the pullback metric, we discover the local latent basis of the X . Moreover, although the direction we found is local , we show how it can be applied to other samples via parallel transport. Finally, we introduce x -space guidance for editing data in the X to enhance the quality of image generation.

3.1 Pullback metric

We consider a curved manifold, X , where our latent variables x t exist. The differential geometry represents X through patches of tangent spaces, T x , which are vector spaces defined at each point x . Then, all the geometrical properties of X can be obtained from the inner product of || d x || 2 = ⟨ d x , d x ⟩ x in T x . However, we do not have any knowledge of ⟨ d x , d x ⟩ x . It is definitely not a Euclidean metric. Furthermore, samples of x t at intermediate timesteps of DMs include inevitable noise, which prevents finding semantic directions in T x .

Fortunately, Kwon et al. [26] observed that H , defined by the bottleneck layer of the U-Net, exhibits local linear structure. This allows us to adopt the Euclidean metric on H . In differential geometry, when a metric is not available on a space, pullback metric is used. If a smooth map exists between the original metric-unavailable domain and a metric-available codomain, the pullback metric is used to measure the distances in the domain space. Our idea is to use the pullback Euclidean metric on H to define the distances between the samples in X .

DMs are trained to infer the noise ϵ t from a latent variable x t at each diffusion timestep t . Each x t has a different internal representation h t , the bottleneck representation of the U-Net, at different t 's. The differentiable map between X and H is denoted as f : X → H . Hereafter, we refer to x t as x for brevity unless it causes confusion. It is important to note that our method can be applied at any timestep in the denoising process. We consider a linear map, T x →T h , between the domain and codomain tangent spaces. This linear map can be described by the Jacobian J x = ∇ x h which determines how a vector v ∈ T x is mapped into a vector u ∈ T h by u = J x v .

Using the local linearity of H , we assume the metric, || d h || 2 = ⟨ d h , d h ⟩ h = d h T d h as a usual dot product defined in the Euclidean space. To assign a geometric structure to X , we use the pullback

w/o prompt

w/ prompt

Bear

Ostrich

Lion

Tiger

Input image

①

DDIM

inversion

②

DDIM

generation

⑤

DDIM

generation

: Probability flow ODE trajectories

④

x

-space guidance

③

Identify the local basis vector (

𝐯

)

(a) Method overview

(b) Editing results

𝐱𝑡

෤

𝐱𝑡

𝐱𝑡

෤

𝐱𝑡

𝐱0

𝑝(𝐱0)

𝑝(𝐱𝑇)

metric of the corresponding H . In other words, the norm of v ∈ T x is measured by the norm of corresponding codomain tangent vector:

3.2 Finding local latent basis

Using the pullback metric, we define the local latent vector v ∈ T x that shows a large variability in T h . We find a unit vector v 1 that maximizes || v || 2 pb . By maximizing || v || 2 pb while remaining orthogonal to v 1 , one can obtain the second unit vector v 2 . This process can be repeated to have n latent directions of { v 1 , v 2 , · · · , v n } in T x . In practice, v i corresponds to the i -th right singular vector from the singular value decomposition (SVD) of J x = U Λ V T , i.e., J x v i = Λ i u i . Since the Jacobian of too many parameters is not tractable, we use a power method [18, 34, 19] to approximate the SVD of J x (See Appendix D for the time complexity and Appendix F for the detailed algorithm).

Henceforth, we refer to T x as a local latent subspace, and { v 1 , v 2 , · · · , v n } as the corresponding local latent basis.

Using the linear transformation between T x and T h via the Jacobian J x , one can also obtain corresponding directions in T h . In practice, u i corresponds to the i -th left singular vector from the SVD of J x . After selecting the top n (e.g., n = 50 ) directions of large eigenvalues, we can approximate any vector in T h with a finite basis, { u 1 , u 2 , · · · , u n } . When we refer to a local tangent space henceforth, it means the n -dimensional low-rank approximation of the original tangent space.

The collection of local latent basis vectors, { v 1 , v 2 , · · · , v n } , obtained through our proposed method, can be interpreted as a signal that the model is highly response to for a given x . On the other hand, the basis of the local tangent space, denoted as { u 1 , u 2 · · · , u n } , can be viewed as the corresponding representation associated with the signal.

In Stable Diffusion, the prompt also influences the Jacobian, which means that the local basis also depends on it. We can utilize any prompt to obtain a local latent basis, and different prompts create distinct geometrical structures. For the sake of brevity, we will omit the word local unless it leads to confusion.

3.3 Generating edited images with x -space guidance

A naïve approach for manipulating a latent variable x using a latent vector v is through simple addition, specifically x + γ v . However, using the naïve approach sometime leads to noisy image

generation. To address this issue, instead of directly using the basis for manipulation, we use a basis vector that has passed through the decoder once for manipulation. The x -space guidance is defined as follows

where γ is a hyper-parameter controlling the strength of editing and ϵ θ is a diffusion model. Equation 4 is inspired by classifier-free guidance, but the key difference is that it is directly applied in the latent space X . Our x -space guidance provides qualitatively similar results to direct addition, while it shows better fidelity. (See Appendix C for ablation study.)

3.4 The overall process of image editing

In this section, we summarize the entire editing process with five steps: 1) The input image is inverted into initial noise x T using DDIM inversion. 2) x T is gradually denoised until t through DDIM generation. 3) Identify local latent basis { v 1 , · · · , v n } using the pullback metric at t . 4) Manipulate x t along the one of basis vectors using the x -space guidance. 5) The DDIM generation is then completed with the modified latent variable ˜ x t . Figure 2 illustrates the entire editing process.

In the context of a text-to-image model, such as Stable Diffusion, it becomes possible to include textual conditions while deriving local basis vectors. Although we do not use any text guidance during DDIM inversion and generation, a local basis with text conditions enables semantic editing that matches the given text. Comprehensive experiments can be found in Section 4.1.

It is noteworthy that our approach needs no extra training and simplifies image editing by only adjusting the latent variable within a single timestep.

3.5 Editing various samples with parallel transport

Let us consider a scenario where our aim is to edit ten images, changing straight hair to curly hair. Due to the nature of the unsupervised image editing method, it is becomes imperative to manually check the semantic relevance of the latent basis vector in the edited results. Thus, to edit every samples, we have to manually find a straight-to-curly basis vector for individual samples.

One way to alleviate this tedious task is to apply the curly hair vector obtained from one image to other images. However, the basis vector v ∈ T x obtained at x cannot be used for the other sample x ′ because v / ∈ T x ′ . Thus, in order to apply the direction we obtained to another sample, it is necessary to relocate the extracted direction to a new tangent space.

To achieve this, we use parallel transport that moves v i onto the new tangent space T x ′ . In the realm of differential geometry, parallel transport is a technique to relocate a tangent vector to different position with minimal direction change, while keeping the vector tangent on the manifold [48]. It is notable that in curved space, parallel transport can significantly modify the original vector. Therefore, it is beneficial to apply the parallel transport in H , where relatively flatter than X .

The process of moving a tangent vector v ∈ T x to v ′ ∈ T x ′ through parallel transport in H is summarized as follows. First, we convert the latent direction v i ∈ T x to the corresponding direction of u i ∈ T h . Second, we apply the parallel transport u i ∈ T h to u ′ i ∈ T h ′ , where h ′ = f ( x ′ ) . In the general case, parallel transport involves iterative projection and normalization on the tangent space along the path connecting two points [48]. However, in our case, we assume that H has Euclidean geometry. Therefore, we move u directly onto T h ′ through projection, without the need for an iterative process. Finally, transform u ′ i into v ′ i ∈ X . Using this parallel transport of v i → v ′ i via H , we can apply the local latent basis obtained from x to edit or modify the input x ′ .

4 Findings and results

In this section, we analyze the geometric structure of DMs with our method. § 4.1 demonstrates that the latent basis found by our method can be used for image editing. In § 4.2, we investigate how the geometric structure of DMs evolves as the generative process progresses. Lastly, in § 4.3, we examine how the geometric properties of the text-condition model change with a given text.

The implementation details of our work are provided on Appendix B. The source code for our experiments can be found at https://github.com/enkeejunior1/Diffusion-Pullback .

Real image

(Old)

Real image

(Woman)

Real image

(Red nose)

Real image

(Sunflower)

Real image

(Yellow)

Real image

(Pinky)

Real image

(Bullmastiff)

Real image

(Open mouth)

Real image

(Wink)

Real image

(Two-story)

Real image

(Long roof)

Real image

(White paint)

FFHQ

Flowers

AFHQ

LSUN-Church

1T

0.8T

0.6T

→

0.7T

Real image

(Woman)

Real image

(Overweighted)

Stable Diffusion

0.6T

Real image

w/ 'Lion'

w/o prompt

0.7T

0.6T

V2

V2

V3

4.1 Image editing with the latent basis

In this subsection, we demonstrate the capability of our discovered latent basis for image editing. To extract the latent variables from real images for editing purposes, we use DDIM inversion. In experiments with Stable Diffusion (SD), we do not use guidance, i.e., unconditional sampling, for both DDIM inversion and DDIM sampling processes. This ensures that our editing results solely depend on the latent variable, not on other factors such as prompt conditions.

'Tiger'

'Bear'

'Running dog'

'Sitting dog'

'Lion'

'Running cat'

'Standing cat'

Figures 2 and 3 illustrate the example results edited by the latent basis found by our method. The latent basis clearly contains semantics such as age, gender, species, structure, and texture. Note that editing at timestep T yields coarse changes such as age and species. On the other hand, editing at timestep 0 . 6 T leads to fine changes, such as nose color and facial expression.

Figure 4 demonstrates the example results edited by the various latent basis vectors. Interestingly, using the text 'lion' as a condition, the entire latent basis captures lion-related attributes. Furthermore, Figure 5 shows that the latent basis aligns with the text not only in terms of object types but also in relation to pose or action. For a qualitative comparison with other state-of-the-art image editing methods, refer to Appendix D. For more examples of editing results, refer to Appendix G.

4.2 Evolution of latent structures during generative processes

In this subsection, we demonstrate how the latent structure evolves during the generative process and identify three trends. 1) The frequency domain of the latent basis changes from low to high frequency. It agrees with the previous observation that DMs generate samples in coarse-to-fine manner. 2) The difference between the tangent spaces of different samples increases over the generative process. It implies finding generally applicable editing direction in latent space becomes harder in later timesteps. 3) The differences of tangent spaces between timesteps depend on the complexity of the dataset.

10

Densit

0.75T

D

10-3

Spectral

0.5T

10-5

Power

0.25T

10-7

T

100

10-1

10-2

Vi

Frequency

Latent bases gradually evolve from low- to highfrequency structures. Figure 6 is the power spectral density (PSD) of the discovered latent basis over various timesteps. The early timesteps contain a larger portion of low frequency than the later timesteps and the later timesteps contain a larger portion of high frequency.

This suggests that the model focuses on low-frequency signals at the beginning of the generative process and then shifts its focus to high-frequency signals over time. This result strengthens the common understanding about the coarse-to-fine behavior of DMs over the generative process [12, 15].

The discrepancy of tangent spaces from different samples increases along the generative process. To investigate the geometry of the tangent basis, we employ a metric on the Grassmannian manifold.

The Grassmannian manifold is a manifold where each point is a vector space, and the metric defined above represents the distortion across various vector spaces. We use geodesic metric [10, 56] to define the discrepancy between two subspaces {T (1) , T (2) } :

where θ k denotes the k -th principle angle between T (1) and T (2) . Intuitively, the concept of geodesic metric can be understood as an angle between two vector spaces. Here, the comparison between two different spaces was conducted for {T h 1 , T h 2 } . Unlike the X , the H assumes a Euclidean space which makes the computation of geodesic metric that requires an inner product between tangent spaces easier. The relationship between tangent space and latent subspace is covered in more detail in Appendix E.

Figure 7 demonstrates that the tangent spaces of the different samples are the most similar at t = T and diverge as timestep becomes zero.

Moreover, the similarity across tangent spaces allows us to effectively transfer the latent basis from one sample to another through parallel transport as shown in Figure 8. In T , where the tangent spaces are homogeneous, we consistently obtain semantically aligned editing results. On the other hand, parallel transport at t = 0 . 6 T does not lead to satisfactory editing because the tangent spaces are hardly homogeneous. Thus, we should examine the similarity of local subspaces to ensure consistent editing across samples.

11

10-

.....

Theoretical upper bound

Random basis

9

Stable-Difusion (ver 2.1)

ImageNet

LSUN-church

geo

8

LSUN-bedroom

D

LSUN-cat

7

LSUN-horse

CelebA-HQ

Flowers*

6

FFHQ*

T

0.75T

0.5T

0.25T

Diffusion Timestep

DMs trained on simpler datasets exhibit more consistent tangent spaces over time. In Figure 9 (a), we provide a distance matrix of the tangent spaces across different timesteps, measured by the geodesic metric. We observe that the tangent spaces are more similar to each other when a model is trained on CelebA-HQ, compared to ImageNet. To verify this trend, we measure the geodesic distances between tangent spaces of different timesteps and plot the average distances of the same

Real image

(Man)

Real image

(Woman)

Real image

(Red lip)

P.T

P.T

1T

0.6T

+Vi

+Vi

+Vi

(b)

(a)

CelebA-HQ

2/3

1/3

0

ImageNet

𝑇

0.1𝑇

𝑇

0.1𝑇

0.1𝑇

𝑇

0

0.25𝑇

0.5𝑇

0.75𝑇

ImageNet

LSUN-horse

LSUN-cat

LSUN-bedroom

LSUN-church

FFHQ*

Flowers*

CelebA-HQ

|Δ𝑇|

0

2

4

6

8

10

M

D

（Tht

geo

D

0

0.2

0.4

0.6

0.8

Theoretical Upper Bound

Random Basis

10-

0.45

10

9

0.40

8

score

0.35

8

7

0.30

6

D

geo

6

5

T

0.9T

0.8T

2

0.25

0.20

D

geo

4

0.7T

0.15

2

T

→0.1T

T

0.8T

0.6T

0.4T

0.2T

Similarity (text)

Diffusion Timestep

Diffusion Timestep

(a)Tangent space distance and text similarity (b) Coefficient of determination

(c) Geodesic distance across timesteps

difference in timestep in Figure 9 (b). As expected, we find that DMs trained on datasets, that are generally considered simpler, have similar local tangent spaces over time.

4.3 Effect of conditioning prompts on the latent structure

In this subsection, we aim to investigate how prompts influence the generative process from a geometrical perspective. We randomly sampled 50 captions from the MS-COCO dataset [29] and used them as text conditions.

Similar text conditions induce similar tangent spaces. In Figure 10 (a), we observe a negative correlation between the CLIP similarity of texts and the distance between tangent spaces. In other words, when provided with similar texts, the tangent spaces are more similar. The linear relationship between the text and the discrepancy of the tangent spaces is particularly strong in the early phase of the generative process as shown by R 2 score in Figure 10 (b).

The generative process depends less on text conditions in later timesteps. Figure 10 (c) illustrates the distances between local tangent spaces for given different prompts with respect to the timesteps.

Real image

Real image

(a) Entanglement of semantic directions

(b) Abrupt changes

P.T

+Vi

Notably, as the diffusion timestep approaches values below 0 . 7 T , the distances between the local tangent spaces start to decrease. It implies that the variation due to walking along the local tangent basis depends less on the text conditions, i.e., the text less influences the generative process, in later timesteps. It is a possible reason why the correlation between the similarity of prompts and the similarity of tangent spaces reduces over timesteps.

5 Discussion

In this section, we provide additional intuitions and implications. It is interesting that our latent basis usually conveys disentangled attributes even though we do not adopt attribute annotation to enforce disentanglement. We suppose that decomposing the Jacobian of the encoder in the U-Nets naturally yields disentanglement to some extent. However, it does not guarantee the perfect disentanglement and some directions are entangled. For example, the editing for beard converts a female subject to a male as shown in Figure 11 (a). This kind of entanglement often occurs in other editing methods due to the dataset bias: female faces seldom have beard.

While our method has shown effectiveness in Stable Diffusion, more research is needed to fully validate its potential. We have observed that some of the discovered latent vector occasionally leads to abrupt changes during the editing process in Stable Diffusion, as depicted in Figure 11 (b). This observation highlights the complex geometry of X in achieving seamless editing. Exploring this topic in future research is an interesting area to delve into.

Our approach is broadly applicable when the feature space in the DM adheres to a Euclidean metric, as demonstrated by H . This characteristic has been observed in the context of U-Net within Kwon et al. [26]. It would be intriguing to investigate if other architectural designs, especially those similar to transformer structures as introduced in [42, 53], also exhibit a Euclidean metric.

Despite these limitations, our method provides a significant advance in the field of image editing for DMs, and provides a deep understanding of DM through several experiments.

6 Conclusion

Wehave analyzed the latent space of DMs from a geometrical perspective. We used the pullback metric to identify the latent and tangent bases in X and H . The latent basis found by the pullback metric allows editing images by traversal along the basis. We have observed properties of the bases in two aspects. First, we discovered that 1) the latent bases evolve from low- to high-frequency components; 2) the discrepancy of tangent spaces from different samples increases along the generative process; and 3) DMs trained on simpler datasets exhibit more consistent tangent spaces over timesteps. Second, we investigated how the latent structure changes based on the text conditions in Stable Diffusion, and discovered that similar prompts make tangent space analogous but its effect becomes weaker over timesteps. We believe that a better understanding of the geometry of DMs will open up new possibilities for adopting DMs in useful applications.

References

Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows. ACM Transactions on Graphics (ToG) , 40(3):1-21, 2021.

Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. arXiv preprint arXiv:1710.11379 , 2017.

Georgios Arvanitidis, Søren Hauberg, and Bernhard Schölkopf. Geometrically enriched latent spaces. arXiv preprint arXiv:2008.00565 , 2020.

Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18208-18218, 2022.

Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 , 2022.

Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. arXiv preprint arXiv:2112.03126 , 2021.

Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18392-18402, 2023.

Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-andexcite: Attention-based semantic guidance for text-to-image diffusion models. arXiv preprint arXiv:2301.13826 , 2023.

Nutan Chen, Alexej Klushyn, Richard Kurle, Xueyan Jiang, Justin Bayer, and Patrick Smagt. Metrics for deep generative models. In International Conference on Artificial Intelligence and Statistics , pages 1540-1550. PMLR, 2018.

Jaewoong Choi, Junho Lee, Changyeon Yoon, Jung Ho Park, Geonho Hwang, and Myungjoo Kang. Do not escape from the manifold: Discovering the local coordinates on the latent space of gans. arXiv preprint arXiv:2106.06959 , 2021.

Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02938 , 2021.

Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11472-11481, 2022.

Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8188-8197, 2020.

Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint arXiv:2206.00941 , 2022.

Giannis Daras and Alexandros G Dimakis. Multiresolution textual inversion. arXiv preprint arXiv:2211.17115 , 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Gene H Golub and Charles F Van Loan. Matrix computations . JHU press, 2013.

René Haas, Inbar Huberman-Spiegelglas, Rotem Mulayoff, and Tomer Michaeli. Discovering interpretable directions in the semantic latent space of diffusion models. arXiv preprint arXiv:2303.11073 , 2023.

Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. Advances in Neural Information Processing Systems , 33:9841-9850, 2020.

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations , 2018.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364 , 2022.

Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. arXiv preprint arXiv:2210.10960 , 2022.

Yonghyeon Lee and Frank C Park. On explicit curvature regularization in deep generative models. 2023.

Yonghyeon Lee, Seungyeon Kim, Jinwon Choi, and Frank Park. A statistical manifold framework for point cloud data. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 12378-12402. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/lee22d.html .

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. arXiv preprint arXiv:2112.05744 , 2021.

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11461-11471, 2022.

Chaofan Ma, Yuhuan Yang, Chen Ju, Fei Zhang, Jinxiang Liu, Yu Wang, Ya Zhang, and Yanfeng Wang. Diffusionseg: Adapting diffusion towards unsupervised object discovery. arXiv preprint arXiv:2303.09813 , 2023.

Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073 , 2021.

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 , 2018.

Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794 , 2022.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your gan: Interactive point-based manipulation on the generative image manifold. In ACM SIGGRAPH 2023 Conference Proceedings , pages 1-11, 2023.

Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. arXiv preprint arXiv:2302.03027 , 2023.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W , 2017.

Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2085-2094, 2021.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Aditya Ramesh, Youngduck Choi, and Yann LeCun. A spectral regularizer for unsupervised disentanglement. arXiv preprint arXiv:1812.01161 , 2018.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Vikash Sehwag, Caner Hazirbas, Albert Gordo, Firat Ozgenel, and Cristian Canton. Generating high fidelity data from low-density regions using diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11492-11501, 2022.

Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The riemannian geometry of deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 315-323, 2018.

Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1532-1540, 2021.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209.14916 , 2022.

Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. arXiv preprint arXiv:2211.12572 , 2022.

Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. arXiv preprint arXiv:2303.04803 , 2023.

Ke Ye and Lek-Heng Lim. Schubert varieties and distances between subspaces of different dimensions. SIAM Journal on Matrix Analysis and Applications , 37(3):1176-1197, 2016.

LEE Yonghyeon, Sangwoong Yoon, MinJun Son, and Frank C Park. Regularized autoencoders for isometric representation learning. In International Conference on Learning Representations , 2021.

Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015.

O˘ guz Kaan Yüksel, Enis Simsar, Ezgi Gülperi Er, and Pinar Yanardag. Latentclr: A contrastive learning approach for unsupervised discovery of interpretable directions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14263-14272, 2021.

Jiapeng Zhu, Ruili Feng, Yujun Shen, Deli Zhao, Zheng-Jun Zha, Jingren Zhou, and Qifeng Chen. Low-rank subspaces in gans. Advances in Neural Information Processing Systems , 34: 16648-16658, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an analysis of the latent structure of diffusion models using differential geometry. The authors propose a method to define a geometry in the latent space by pulling back the Euclidean metric from the U-Net bottleneck space *H* via the network encoder. This approach enables the identification of directions of maximum variation in the latent space. The paper also explores the application of the proposed latent structure guidance for image editing. Finally, the evolution of the geometric structure over time steps and its dependence on text conditioning are investigated.\n\nSTRENGTHS:\n- I found the analysis presented in the paper interesting. It both uncovers unknown details about diffusion models (effect of text prompt and complexity of the dataset on the latent space) and confirm some previous observations (e.g., coarse-to-fine behaviour). This exploration can potentially reveal new capabilities of diffusion models, contributing to the advancement of the field.\n- The paper is technically sound, and the claims made by the authors in sections 4 are supported by experiments. This experimental validation enhances the credibility of the proposed approach.\n\nWEAKNESSES:\n- The paper lacks comparisons with other diffusion-based image editing techniques, like [7,18]. Including such comparisons would have provided a more comprehensive evaluation and demonstrated the advantages of the proposed method.\n- The presentation and clarity of the paper could be improved. For example, the abstract contains too much detail, making it challenging to understand upon initial reading. Also the explanation of the image editing technique could be improved: what is DDIM (section 4)? what is epsilon in Equation 4? Finally, Figure 1 is not sufficiently clear to me, it may hampers the reader's comprehension.\n\nQUESTIONS:\nTo make the paper applicable to real-world scenarios right away, the authors should include a comparative analysis with other image editing techniques.\n\n**Some potentially interesting references**\n- Pan, Xingang, et al. \"Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold.\" arXiv preprint arXiv:2305.10973 (2023).\n- Brooks, Tim, Aleksander Holynski, and Alexei A. Efros. \"Instructpix2pix: Learning to follow image editing instructions.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n- Meng, Chenlin, et al. \"Sdedit: Guided image synthesis and editing with stochastic differential equations.\" arXiv preprint arXiv:2108.01073 (2021).\n\n**Typos**\n- line 20: double citation\n- line 160 editted\n- Figure 2 caption: editing\n\nLIMITATIONS:\nThe authors have addressed the limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this submission, the authors probe the latent space, xt ∈ X, of diffusion models (DMs) from a geometric perspective, utilizing the pullback metric to identify local latent basis in X and corresponding local tangent basis in H. To confirm their findings, they edit images via latent space traversal. The authors provide a two-pronged analysis, investigating the evolution of geometric structure over time and its variation based on text conditioning in Stable Diffusion. Notably, they discovered that in the generative process, the model prioritizes low-frequency components initially, moving to high-frequency details later, and that the model's dependence on text conditions reduces over time. The paper introduces image editing through x-space traversal and to offer comprehensive analyses of the latent structure of DMs, with a specific emphasis on the use of the pullback metric and the SVD of the Jacobian in computing a basis.\n\nSTRENGTHS:\nThe paper presents a distinctive idea that provides an alternative method for editing in diffusion models, as well as enhancing comprehension of the latent space dynamics. By utilizing a geometric perspective, the authors make use of the pullback metric to investigate the latent space, offering insights into its structure and operation.  The exploration of the evolution of geometric structure over time and its response to various text conditions offers additional insights into the dynamics of the latent space is interesting.\n\nWEAKNESSES:\nThe first area where the paper could see improvement is in terms of the clarity of its analysis. Given its nature as an analysis paper, it's crucial that the analysis presented is as comprehensible as possible. However, the method and notation used in this work can lead to some confusion. For instance, Section 3, in its current form, may not be as accessible to all readers as it could be, and it could benefit from being revised for clearer communication of the ideas contained therein. Additionally, Fig. 1, which is presumably intended to illustrate key concepts, is perhaps too dense with information. Dividing Fig. 1 into two separate figures could make it easier to digest, enabling a clearer explanation of the approach.\n\nA second aspect that could be improved upon is the overall presentation and proofreading of the paper. While the approach is relatively simple, its translation into the written form has not been as clear as one would hope. The text could benefit from a thorough proofreading to ensure that it is not just grammatically correct, but also that it conveys the authors' ideas in a way that is accessible to the wider machine learning community. As it stands, the paper's usefulness to this community may be hindered by its presentation.\n\nLastly, the paper could do more to address the computational implications of its approach. The authors use the power method to approximate the Jacobian, which, while effective, can be computationally costly. It would be beneficial if the authors were more transparent about this fact, allowing readers to fully understand the computational demands of the approach and evaluate whether or not it would be feasible in their own applications. Being upfront about such limitations can help to build a more honest and comprehensive understanding of the paper's methodologies and implications.\n\nQUESTIONS:\nWhat is the complexity of computing the Jacobian? What is the the actual runtime (in seconds) of the approach compared to other editing methods? I think answering these questions can provide a good context for readers.\n\nLIMITATIONS:\nSocietal impact has been properly address but I would like to see a deeper analysis of the computational complexity and runtime of the approach.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the geometry of latent spaces of diffusion models (Dms) using the pullback metric. In the analyses, they mainly examine change of the frequency representations in latent spaces over time and the change of the structure based on text conditioning.\n\nAfter the rebuttal:\n\nI checked all reviewer comments and responses.\n\nI agree with the other reviewers regarding limited algorithmic novelty of the work. Therefore, I keep my original score.\n\nSTRENGTHS:\n- The paper is well written in general (there are several typos but they can be fixed in proof-reading).\n\n- The proposed methods and analyses are interesting. Some of the theoretical results highlight several important properties of diffusion models.\n\nWEAKNESSES:\n1. Some of the statements and claims are not clear as pointed in the Questions.\n\n2. The results are given considering the Riemannian geometry of the latent spaces and utilizing the related transformations (e.g. PT) among tangent spaces on the manifolds. However, vanilla DMs do not employ these transformations. Therefore, it is not clear whether these results are for vanilla DMs or the  DMs utilizing the proposed transformations.\n\n3. A major claim is that the proposed methods improve effectiveness of the DMs. However, the employed transformations can increase the footprints of DMs.\n\nQUESTIONS:\nIt is stated that “To investigate the geometry of the tangent basis, we employ a metric on the Grassmannian manifold.” However, the space could be identified by another manifold as well.  Why and how did you define the space by the Grassmannian manifold?\n\nIt is claimed that “the similarity across tangent spaces allows us to effectively transfer the latent basis from one sample to another through parallel transport”. How this improves effectiveness was not analyzed. In general, how do the proposed methods improve training and inference time? Indeed, the additional transformations can increase training and inference time. Could you please provide an analysis of the footprints?\n\nLIMITATIONS:\nSome of the limitations were addressed but potential impacts were not addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a study on the latent space of diffusion models and on how to manipulate it. It takes advantage of an observation made by previous work [22] on the flatness and semantic structure of the U-Net model used in  DDIM and uses pullback metric from the latent space of the U-Net to the space of diffusion to measure some properties of the latter under different conditions. \nThe paper also proposes a method to manipulate the diffusion space through the different time steps, so as to carry out semantically meaningful edits.\n\nSTRENGTHS:\nThis paper is one of the first to study the behavior of the space diffusion models. It presents some interesting studies on the behavior of the process during time, showing that the early stages convey higher frequency while the last steps are more concerned with higher frequencies. Another interesting observation is that tangent spaces of different samples tend to be more aligned at T=1, while they diverge toward T=0 (end of the generative process). \n\nThe paper also shows (even if it had already been observed in 22) that the pullback metric is effective in transferring the shift along the semantically meaningful principal components of the U-Net latent space into the diffusion process, thus resulting in meaningful edits of the generated image, which frequency depends on the time the edit was performed.\n\nWEAKNESSES:\nI may have misunderstood or missed some important information, but the method described is not really clear. Specifically, it is not really clear to me how the editing process works (sections 3.3 and 3.4):\n- In 3.3, the letter v is used to indicate elements of both T_x and T_h, so it is not always clear to which space they are referring.\n- In general, it is not clear why the idea expressed in 3.3 is useful and where it was adopted.\n- In eq 4 what is the epsilon function? In general, isn’t the vector toward which to shift selected from T_H (so it should be u) and then transferred to T_x?\n\n\nAnother concern is about the generalization of the proposed method to other diffusion techniques, or with other score models (i.e. not UNet). I think that this point needs more discussion.\n\nQUESTIONS:\nI was wondering what would happen if, instead of moving along one of the principal axis of T_H, you use directly the principal axis of T_x. \n\nI would also discuss a bit more method 22 in the related work since it seems related to the proposed method.\n\nWriting issues:\n - Check the sentence at rows 74-75\n- Row 152: we aims\n- general grammar check\n\nLIMITATIONS:\nI don’t foresee any particular negative societal impact. A discussion on how the proposed study may generalize to other domains and architectures would be of value.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}