SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figures 2 and 3 illustrate the example results edited by the latent basis found by our method. The latent basis clearly contains semantics such as age, gender, species, structure, and texture. Note that editing at timestep  $T$  yields coarse changes such as age and species. On the other hand, editing at timestep  $0.6T$  leads to fine changes, such as nose color and facial expression.
Figure 4 demonstrates the example results edited by the various latent basis vectors. Interestingly, using the text "lion" as a condition, the entire latent basis captures lion-related attributes. Furthermore, Figure 5 shows that the latent basis aligns with the text not only in terms of object types but also in relation to pose or action. For a qualitative comparison with other state-of-the-art image editing methods, refer to Appendix D. For more examples of editing results, refer to Appendix G.
4.2 Evolution of latent structures during generative processes
In this subsection, we demonstrate how the latent structure evolves during the generative process and identify three trends. 1) The frequency domain of the latent basis changes from low to high frequency. It agrees with the previous observation that DMs generate samples in coarse-to-fine manner. 2) The difference between the tangent spaces of different samples increases over the generative process. It implies finding generally applicable editing direction in latent space becomes harder in later timesteps. 3) The differences of tangent spaces between timesteps depend on the complexity of the dataset.
Latent bases gradually evolve from low- to high-frequency structures. Figure 6 is the power spectral density (PSD) of the discovered latent basis over various timesteps. The early timesteps contain a larger portion of low frequency than the later timesteps and the later timesteps contain a larger portion of high frequency.
Figure 6: Power Spectral Density (PSD) of latent basis. The PSD at  $t = T$  (purple) exhibits a greater proportion of low-frequency signals, while the PSD at smaller  $t$  (beige) reveals a larger proportion of high-frequency signals. The latent vectors  $\mathbf{v}_i$  are min-max normalized for visual clarity.
This suggests that the model focuses on low-frequency signals at the beginning of the generative process and then shifts its focus to high-frequency signals over time. This result strengthens the common understanding about the coarse-to-fine behavior of DMs over the generative process [12, 15].
The discrepancy of tangent spaces from different samples increases along the generative process. To investigate the geometry of the tangent basis, we employ a metric on the Grassmannian manifold.
The Grassmannian manifold is a manifold where each point is a vector space, and the metric defined above represents the distortion across various vector spaces. We use geodesic metric [10, 56] to define the discrepancy between two subspaces  $\{\mathcal{T}^{(1)},\mathcal{T}^{(2)}\}$ :
$$
D _ {\mathrm {g e o}} \left(\mathcal {T} ^ {(1)}, \mathcal {T} ^ {(2)}\right) = \sqrt {\sum_ {k} \theta_ {k} ^ {2}}, \tag {5}
$$
where  $\theta_{k}$  denotes the  $k$ -th principle angle between  $\mathcal{T}^{(1)}$  and  $\mathcal{T}^{(2)}$ . Intuitively, the concept of geodesic metric can be understood as an angle between two vector spaces. Here, the comparison between two different spaces was conducted for  $\{\mathcal{T}_{\mathbf{h}_1},\mathcal{T}_{\mathbf{h}_2}\}$ . Unlike the  $\mathcal{X}$ , the  $\mathcal{H}$  assumes a Euclidean space which makes the computation of geodesic metric that requires an inner product between tangent spaces easier. The relationship between tangent space and latent subspace is covered in more detail in Appendix E.
Figure 7 demonstrates that the tangent spaces of the different samples are the most similar at  $t = T$  and diverge as timestep becomes zero.
Moreover, the similarity across tangent spaces allows us to effectively transfer the latent basis from one sample to another through parallel transport as shown in Figure 8. In  $T$ , where the tangent spaces are homogeneous, we consistently obtain semantically aligned editing results. On the other hand, parallel transport at  $t = 0.6T$  does not lead to satisfactory editing because the tangent spaces are hardly homogeneous. Thus, we should examine the similarity of local subspaces to ensure consistent editing across samples.
Figure 7: Geodesic distance across tangent space of different samples at various diffusion timesteps. Each point represents the average geodesic distance between pairs of 15 samples. It is notable that the similarity of tangent spaces among different samples diminishes as the generative process progresses.
DMs trained on simpler datasets exhibit more consistent tangent spaces over time. In Figure 9 (a), we provide a distance matrix of the tangent spaces across different timesteps, measured by the geodesic metric. We observe that the tangent spaces are more similar to each other when a model is trained on CelebA-HQ, compared to ImageNet. To verify this trend, we measure the geodesic distances between tangent spaces of different timesteps and plot the average distances of the same
Figure 8: Examples of image edition using parallel transport. The first row demonstrates the results of editing with their respective latent vectors, while the subsequent rows exhibit the results of editing through the parallel transport (P.T) of the latent vectors used in the first row. The latent vector performs effectively when  $t = T$  (left and middle), but comparatively less satisfactorily for  $0.6T$  (right).
Figure 9: Simpler datasets lead to more similar tangent spaces across diffusion timesteps. (a) Distance matrix visualization of tangent space measured by geodesic metric across various timesteps. (b) Average geodesic distance based on timestep differences, indicating that the complexity of the dataset correlates with greater distances between tangent spaces.
(a) Tangent space distance and text similarity
b) Coefficient of determination
(c) Geodesic distance across timesteps
Figure 10: Similar prompts create similar tangent spaces, and the impact of the prompt decreases as the generative process progresses. (a) The horizontal axis represents the CLIP similarity between two different prompts, while the vertical axis represents the geodesic distance in the tangent space from each prompt. Different colors represent various diffusion timesteps. A negative relationship is observed between prompt similarity and tangent space distance. (b) The  $R^2$  score of the linear regression between clip similarity and geodesic distance of tangent spaces decreases throughout the generative process. (c) Each point represents the distance between tangent spaces created from different prompts. Until around  $t = 0.7T$ , the distance between tangent spaces is very large, but it gradually decreases thereafter. This indicates that the influence of the prompt on the tangent space diminishes.
difference in timestep in Figure 9 (b). As expected, we find that DMs trained on datasets, that are generally considered simpler, have similar local tangent spaces over time.
4.3 Effect of conditioning prompts on the latent structure
In this subsection, we aim to investigate how prompts influence the generative process from a geometrical perspective. We randomly sampled 50 captions from the MS-COCO dataset [29] and used them as text conditions.
Similar text conditions induce similar tangent spaces. In Figure 10 (a), we observe a negative correlation between the CLIP similarity of texts and the distance between tangent spaces. In other words, when provided with similar texts, the tangent spaces are more similar. The linear relationship between the text and the discrepancy of the tangent spaces is particularly strong in the early phase of the generative process as shown by  $R^2$  score in Figure 10 (b).
The generative process depends less on text conditions in later timesteps. Figure 10 (c) illustrates the distances between local tangent spaces for given different prompts with respect to the timesteps.
(a) Entanglement of semantic directions
(b) Abrupt changes
Figure 11: Limitations. (a) Entanglement between attributes due to dataset biases (b) Abrupt changes in Stable Diffusion.
Notably, as the diffusion timestep approaches values below  $0.7T$ , the distances between the local tangent spaces start to decrease. It implies that the variation due to walking along the local tangent basis depends less on the text conditions, i.e., the text less influences the generative process, in later timesteps. It is a possible reason why the correlation between the similarity of prompts and the similarity of tangent spaces reduces over timesteps.
In this section, we provide additional intuitions and implications. It is interesting that our latent basis usually conveys disentangled attributes even though we do not adopt attribute annotation to enforce disentanglement. We suppose that decomposing the Jacobian of the encoder in the U-Nets naturally yields disentanglement to some extent. However, it does not guarantee the perfect disentanglement and some directions are entangled. For example, the editing for beard converts a female subject to a male as shown in Figure 11 (a). This kind of entanglement often occurs in other editing methods due to the dataset bias: female faces seldom have beard.
While our method has shown effectiveness in Stable Diffusion, more research is needed to fully validate its potential. We have observed that some of the discovered latent vector occasionally leads to abrupt changes during the editing process in Stable Diffusion, as depicted in Figure 11 (b). This observation highlights the complex geometry of  $\mathcal{X}$  in achieving seamless editing. Exploring this topic in future research is an interesting area to delve into.
Our approach is broadly applicable when the feature space in the DM adheres to a Euclidean metric, as demonstrated by  $\mathcal{H}$ . This characteristic has been observed in the context of U-Net within Kwon et al. [26]. It would be intriguing to investigate if other architectural designs, especially those similar to transformer structures as introduced in [42, 53], also exhibit a Euclidean metric.
Despite these limitations, our method provides a significant advance in the field of image editing for DMs, and provides a deep understanding of DM through several experiments.
We have analyzed the latent space of DMs from a geometrical perspective. We used the pullback metric to identify the latent and tangent bases in  $\mathcal{X}$  and  $\mathcal{H}$ . The latent basis found by the pullback metric allows editing images by traversal along the basis. We have observed properties of the bases in two aspects. First, we discovered that 1) the latent bases evolve from low- to high-frequency components; 2) the discrepancy of tangent spaces from different samples increases along the generative process; and 3) DMs trained on simpler datasets exhibit more consistent tangent spaces over timesteps. Second, we investigated how the latent structure changes based on the text conditions in Stable Diffusion, and discovered that similar prompts make tangent space analogous but its effect becomes weaker over timesteps. We believe that a better understanding of the geometry of DMs will open up new possibilities for adopting DMs in useful applications.
This work was supported in part by the Creative-Pioneering Researchers Program through Seoul National University, the National Research Foundation of Korea (NRF) grant (Grant No. 2022R1A2C1006871) (J. J.), KIAS Individual Grant [AP087501] via the Center for AI and Natural Sciences at Korea Institute for Advanced Study, and the National Research Foundation of Korea (NRF) grant (RS-2023-00223062).
[52] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
[53] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209.14916, 2022.
[54] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. arXiv preprint arXiv:2211.12572, 2022.
[55] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. arXiv preprint arXiv:2303.04803, 2023.
[56] Ke Ye and Lek-Heng Lim. Schubert varieties and distances between subspaces of different dimensions. SIAM Journal on Matrix Analysis and Applications, 37(3):1176-1197, 2016.
[57] LEE Yonghyeon, Sangwoong Yoon, MinJun Son, and Frank C Park. Regularized autoencoders for isometric representation learning. In International Conference on Learning Representations, 2021.
[58] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
[59] Oğuz Kaan Yüksel, Enis Simsar, Ezgi Gülperi Er, and Pinar Yanardag. LatentCLR: A contrastive learning approach for unsupervised discovery of interpretable directions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14263-14272, 2021.
[60] Jiapeng Zhu, Ruili Feng, Yujun Shen, Deli Zhao, Zheng-Jun Zha, Jingren Zhou, and Qifeng Chen. Low-rank subspaces in gans. Advances in Neural Information Processing Systems, 34: 16648-16658, 2021.