Exploring Low-Dimensional Subspace in Diffusion Models for Controllable Image Editing

Authors: siyi  chen
huijie  zhang
minzhe  guo
yifu  lu
peng  wang
qing  qu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 22:03:54

Abstract: Recently, diffusion models have emerged as a powerful class of generative models. 
Despite their success, there is still limited understanding of their semantic spaces. This makes it challenging to achieve precise and disentangled image generation without additional training, especially in an unsupervised way. 
In this work, we improve the understanding of their semantic spaces from intriguing observations: among a certain range of noise levels, (1) the learned posterior mean predictor (PMP) in the diffusion model is locally linear, and (2) the singular vectors of its Jacobian lie in low-dimensional semantic subspaces. We provide a solid theoretical basis to justify the linearity and low-rankness in the PMP. These insights allow us to propose an unsupervised, single-step, training-free **LO**w-rank **CO**ntrollable image editing (LOCO Edit) method for precise local editing in diffusion models. LOCO Edit identified editing directions with nice properties: homogeneity, transferability, composability, and linearity. These properties of LOCO Edit benefit greatly from the low-dimensional semantic subspace.
Our method can further be extended to unsupervised or text-supervised editing in various text-to-image diffusion models (T-LOCO Edit). Finally, extensive empirical experiments demonstrate the effectiveness and efficiency of LOCO Edit. The code and the arXiv version can be found on the [project website](https://chicychen.github.io/LOCO).

Main paper: 
 introduction: 
 Recently, diffusion models have emerged as a powerful new family of deep generative models with remarkable performance in many applications such as image generation across various domains [1, 2, 3, 4, 5, 6] , audio synthesis [7, 8] , solving inverse problem [9, 10, 11, 12, 13, 14] , and video generation [15, 16, 17] . For example, recent advances in AI-based image generation, revolutionized by diffusion models such as Dalle-2 [18] , Imagen [19] , and stable diffusion [4] , have taken the world of "AI Art generation", enabling the generation of images directly from descriptive text inputs. These models corrupt images by adding noise through multiple steps of forward process and then generate samples by progressive denoising through multiple steps of the reverse generative process.Although modern diffusion models are capable of generating photorealistic images from text prompts, manipulating the generated content by diffusion models in practice has remaining challenges. Unlike generative adversarial networks [20] , the understanding of semantic spaces in diffusion models is still limited. Thus, achieving disentangled and localized control over content generation by direct manipulation of the semantic spaces remains a difficult task for diffusion models. Although effective, some existing editing methods in diffusion models often demand additional training procedures and are limited to global control of content generation [21, 22, 23] . Some methods are trainingfree or localized but are still based upon heuristics, lacking clear mathematical interpretations, or for text-supervised editing only [24, 25, 26, 27, 28] . Others provide analysis in diffusion models [29, 30, 31, 32, 33] , but also have difficulty in local edits such as hair color. In this study, we address the above problem by studying the low-rank semantic subspaces in diffusion models and proposing the LOw-rank COntrollable edit (LOCO Edit) approach. LOCO is the first local editing method that is single-step, training-free, requiring no text supervision, and having other intriguing properties (see Figure 1 for an illustration). Our method is highly intuitive and theoretically grounded, originating from a simple while intriguing observation in the learned posterior mean predictor (PMP) in diffusion models: for a large portion of denoising time steps,The PMP is a locally linear mapping between the noise image and the estimated clean image, and the singular vectors of its Jacobian reside within low-dimensional subspaces.The empirical evidence in Figure 2 consistently shows that this phenomenon occurs when training diffusion models using different network architectures on a range of real-world image datasets.Theoretically, we validated this observation by assuming a mixture of low-rank Gaussian distributions for the data. We then prove the local linearity of the PMP, the low-rank nature of its Jacobian, and that the singular vectors of the Jacobian span the low-dimensional subspaces.By utilizing the linearity of the PMP, we can edit within the singular vector subspace of its Jacobian to achieve linear control of the image content with no label or text supervision. The editing direction can be efficiently computed using the generalized power method (GPM) [30, 34] . Furthermore, we can manipulate specific regions of interest in the image along a disentangled direction through efficient nullspace projection, taking advantage of the low-rank properties of the Jacobian.Benefits of LOCO Edit. Compared to existing editing methods (e.g., [29, 35, 23, 24] ) based on diffusion models, the proposed LOCO Edit offers several benefits that we highlight below:• Precise, single-step, training-free, and unsupervised editing. LOCO enables precise localized editing (Figure 1a ) in a single timestep without any training. Further, it requires no text supervision based on CLIP [36] , thus integrating no intrinsic biases or flaws from CLIP [37] . LOCO is applicable to various diffusion models and datasets (Figure 5 ). • Linear, transferable, and composable editing directions. The identified editing direction is linear, meaning that changes along this direction produce proportional changes in a semantic feature in the image space (Figure 1d ). These editing directions are homogeneous and can be transferred across various images and noise levels (Figure 1b ). Moreover, combining disentangled editing directions leads to simultaneous semantic changes in the respective region, while maintaining consistency in other areas (Figure 1c ). • An intuitive and theoretically grounded approach. Unlike previous works, by leveraging the local linearity of the PMP and the low-rankness of its Jacobian, our method is highly interpretable. The identified properties are well supported by both our empirical observation (Figure 2 ) and theoretical justifications in Section 4.Moreover, LOCO Edit is generalizable to T-LOCO Edit for T2I diffusion models including DeepFloyd IF [19] , Stable Diffusion [4] , and Latent Consistency Models [38] , with or without text supervision (Figure 4 ). A more detailed discussion on the relationship with prior arts can be found in Appendix B.Notations. Throughout the paper, we use X t ⊆ R d to denote the noise-corrupted image space at the time-step t ∈ [0, 1]. In particular, X 0 denotes the clean image space with distribution p data (x), and x 0 ∈ X 0 denote an image. X 0,t denote the posterior mean space at time-step t ∈ (0  
preliminaries on diffusion models: 
 In this section, we start by reviewing the basics of diffusion models [1, 2, 39] , followed by several key techniques that will be used in our approach, such as Denoising Diffusion Implicit Models (DDIM) [3] and its inversion [40] , T2I diffusion model, and classifier-free guidance [41] .Basics of Diffusion Models. In general, diffusion models consist of two processes:• The forward diffusion process. The forward process progressively perturbs the original data x 0 to a noisy sample x t for t ∈ [0, 1] with the Gaussian noise. As in [1] , this can be characterized by a conditional Gaussian distribution p t (xt |x 0 ) = N (x t ; √ α t x 0 , (1-α t )I d ).Particularly, parameters {α t } 1 t=0 sastify: (i) α 0 = 1, and thus p 0 = p data , and (ii) α 1 = 0, and thus p 1 = N (0, I d ).• The reverse sampling process. To generate a new sample, previous works [1, 3, 42, 43] have proposed various methods to approximate the reverse process of diffusion models. Typically, these methods involve estimating the noise ϵ t and removing the estimated noise from x t recursively to obtain an estimate of x 0 . Specifically, the sampling step from x t to x t-∆t with a small ∆t > 0 can be described as:EQUATIONwhere ϵ θ (x t , t) is parameterized by a neural network and trained to predict the noise at time t.According to [1] , the denoiser ϵ θ (x t , t) is optimized by solving the following problem:min θ ℓ(θ) := E t∼[0,1],xt∼pt(xt|x0),ϵ∼N (0,I) ∥ϵ θ (x t , t) -ϵ∥ 2 2 ,where θ denotes the network parameters of the denoiser. Once ϵ θ is well trained, recent studies [44, 45] show that the posterior mean E[x 0 |x t ], i.e., predicted clean image at time t, can be estimated as follows:EQUATIONHere, f θ,t (x t ; t) denotes the posterior mean predictor (PMP) [45, 44] , and x0,t ∈ X 0,t denotes the estimated posterior mean output from PMP given x t and t as the input. For simplicity, we denote f θ,t (x t ; t) as f θ,t (x t ).DDIM and DDIM Inversion. Given a noisy sample x t at time t, DDIM [3] can generate clean images by multiple denoising steps. Given a clean sample x 0 , DDIM inversion [3] can generate a noisy x t at time t by adding multiple steps of noise following the reversed trajectory of DDIM. DDIM inversion has been widely in image editing methods [40, 46, 29, 35, 47, 26] to obtain x t given the original x 0 and then performing editing starting from x t . In our work, after getting x t given x 0 via DDIM inversion, we edit x t to x ′ t only at the single time step t with the help of PMP, and then utilize DDIM to generate the edited image x ′ 0 . For ease of exposition, for any t 1 and t 2 with t 2 > t 1 , we denote DDIM operator and its inversion as x t1 = DDIM(x t2 , t 1 ) and x t2 = DDIM-Inv(x t1 , t 2 ).Text-to-image (T2I) Diffusion Models & Classifier-Free Guidance. So far, our discussion has only focused on unconditional diffusion models. Moreover, our approach can be generalized from unconditional diffusion models to T2I diffusion models [38, 4, 48, 19] , where the latter enables controllable image generation x 0 guided by a text prompt c. In more detail, when training T2I diffusion models, we optimize a conditional denoising function ϵ θ (x t , t, c). For sampling, we Low-rankness of the Jacobian J θ,t (x t ) and Local linearity of the PMP f θ,t (x t ). We evaluated DDPM (U-Net [49] ) on CIFAR-10 dataset [50] , U-ViT [51] (Transformer) on CelebA [52] , ImageNet [53] datasets and DeepFloy IF [19] trained on LAION-5B [54] dataset. (a) The rank ratio of J θ,t (x t ) against timestep t. (b) The norm ratio (Top) and cosine similarity (Bottom) between f θ,t (x t + λ∆x) and l θ (x t ; λ∆x) against step size λ at timestep t = 0.7. employ a technique called classifier-free guidance [41] , which substitutes the unconditional denoiser ϵ θ (x t , t) in Equation ( 1) with its conditional counterpart εθ (x t , t, c) that can be described as follows:EQUATIONHere, ∅ denotes the empty prompt and η > 0 denotes the strength for the classifier-free guidance. 
exploring linearity & low-dimensionality for image editting: 
 In this section, we formally introduce the identified low-rank subspace in diffusion models and the proposed LOCO Edit method with the underlying intuitions. In Section 3.1, we present the benign properties in PMP that our method utilizes. Followed by this, in Section 3.3 we provide a detailed description of our method.First, let us delve into the key intuitions behind the proposed LOCO Edit method, which lie in the benign properties of the PMP f θ,t (x t ). At one given timestep t ∈ [0, 1], let us consider the first-order Taylor expansion of f θ,t (x t + λ∆x) at the point x t :EQUATIONwhere ∆x ∈ S d-1 is a perturbation direction with unit length, λ ∈ R is the perturbation strength, and J θ,t (x t ) = ∇ xt f θ,t (x t ) is the Jacobian of f θ,t (x t ). Interestingly, we discovered that within a certain range of noise levels, the learned PMP f θ,t exhibits local linearity, and the singular subspace of its Jacobian J θ,t is low rank. Notably, these properties are universal across various network architectures (e.g., UNet and Transformers) and datasets.We measure the low-rankness with rank ratio and the local linearity with norm ratio and cosine similarity. Specifically, (i) rank ratio is the ratio of rank(J θ,t (x t )) and the ambient dimension d; (ii) norm ratio is the ratio of ∥f θ,t (x t + λ∆x)∥ 2 and ∥l θ (x t ; λ∆x)∥ 2 ; (iii) cosine similarity is between f θ,t (x t + λ∆x) and l θ (x t ; λ∆x). The detailed experiment settings are provided in Appendix D.1, and results are illustrated in Figure 2 , from which we observe:• Low-rankness of the Jacobian J θ,t (x t ). As shown in Figure 2 (a), the rank ratio for t ∈ [0, 1] consistently displays a U-shaped pattern across various network architectures and datasets: (i) it is close to 1 near either the pure noise t = 1 or the clean image t = 0, (ii) J θ,t (x t ) is low-rank (i.e., rank ratio less than 10 -1 ) for all diffusion models within the range t ∈ [0.2, 0.7], (iii) it achieves the lowest value around mid-to-late timestep, slightly differs depending on architecture and dataset. • Local linearity of the PMP f θ,t (x t ). Moreover, the mapping f θ,t (x t ) exhibits strong linearity across a large portion of the timesteps; see Figure 2 (b) and Figure 10 . Specifically, in Figure 2 (b), we evaluate the linearity of f θ,t (x t ) at t = 0.7 where the rank ratio is close to the lowest value. We can see that f θ,t (x t + λ∆x) ≈ l θ (x t ; λ∆x) even when λ = 40, which is consistently true among different architectures trained on different datasets.In addition to comprehensive experimental studies, we will also demonstrate in Section 4 that both properties can be theoretically justified.The two benign properties offer valuable insights for image editing with precise control. Here, we first present the high-level intuitions behind our method, with further details postponed to Section 3.3. Specifically, for any given time-step t ∈ [0, 1], let us denote the compact singular value decomposition (SVD) of the Jacobian J θ,t (x t ) asEQUATIONwhere r is the rank of J θ,t(x t ), U = [u 1 • • • u r ] ∈ St(d, r) and V = [v 1 • • • v r ] ∈ St(d, r)denote the left and right singular vectors, and Σ = diag(σ 1 , • • • , σ r ) denote the singular values. We write J θ,t (x t ) = J θ,t in short for a specific x t , and denote range(J ⊤ θ,t ) = span(V ) and null(J θ,t ) = {w | J θ,t w = 0}.• Local linearity of PMP for one-step, training-free, and supervision-free editing. Given the PMP f θ,t (x t ) is locally linear at the t-th timestep, if we perturb x t by ∆x = λv i , using one right singular vector v i of J θ,t (x t ) as an example editing direction, then by orthogonalityEQUATION) This implies we can achieve one-step editing along the semantic direction u i . Notably, the method is training-free and supervision-free since v i can be simply found via the SVD of J θ,t (x t ).• Local linearity of PMP for linear, homogeneous, and composable image editing. (i) First, the editing direction v = v i is linear, where any linear λ ∈ R change along v i results in a linear change ρ i = λσ i along u i for the edited image. (ii) Second, the editing direction v = v i is homogeneous due to its independence of x0,t , where it could be applied on any images from the same data distribution and results in the same semantic editing. (iii) Third, editing directions are composable. Any linearly combined editing direction v = i∈I λ i v i ∈ range J ⊤ θ,t is a valid editing direction which would result in a composable change i∈I ρ i u i in the edited image. On the contrary, w ∈ null (J θ,t ) results in no editing since f θ,t (x t + λw) ≈ f θ,t (x t ).• Low-rankness of Jacobian for localized and efficient editing. J θ,t (x t ) is for the entire predicted clean image, thus J θ,t (x t ) finds editing directions in the entire image. Denote Jθ,t the Jacobian only for a certain region of interest (ROI), and Jθ,t the Jacobian for regions outside ROI. Similarly, v ∈ range J ⊤ θ,t can edit mainly regions within the ROI, and null J ⊤ θ,t contain directions that do not edit regions outside of ROI. Further projection of v onto null J ⊤ θ,t can result in a more localized editing direction for ROI. To perform such nullspace projection, computing the full SVD can be very expensive. But we can highly reduce the computation by the low-rank estimation of Jacobians with rank r ′ ≪ d. The estimation is efficient yet effective with t ∈ [0.5, 0.7] when the rank of the Jacobian achieves the lowest value.In this subsection, we provide a detailed introduction to LOCO Edit, expanding on the discussion in Section 3.1. We first introduce the supervision-free LOCO Edit, where we further enable localized image editing through nullspace projection with masks. Second, we generalize to T-LOCO Edit for T2I diffusion models w/wo text-supervision to define the semantic editing directions.LOCO Edit. We first introduce the general pipeline of LOCO Edit. As illustrated in Figure 3 , given an original image x 0 , we first use x t = DDIM-Inv(x 0 , t) to generate a noisy image x t . In particular, we choose t ∈ [0.5, 0.7] so that the PMP f θ,t (x t ) is locally linear and its Jacobian J θ,t (x t ) is close to its lowest rank. From Section 3.1, we know that we can edit the image by changing x ′ t = x t + λv p , where v p is the identified editing direction. After editing x t to x ′ t , we use x ′ 0 = DDIM (x ′ t , 0) to generate the edited image. In many practical applications, we often need to edit only specific local regions of an image while leaving the rest unchanged. As discussed in Section 3.2, we can achieve this task by finding a precise local editing direction with localized Jacobians and nullspace projection. Overall, the complete method is in Algorithm 1. We describe the key details as follows. Given an image x 0 , we perform DDIM-Inv until time t to get x t , and estimate x0,t from x t . After masking to get the region of interest (ROI) x0,t and its counterparts x0,t , we find the edit direction v p via SVD and nullspace projection based on their Jacobians (Algorithm 1). By denoising x t + λv p , an image x ′ 0 with localized editing is generated. In this paper, the variables and notions related to ROI, nullspace, and final direction are respectively highlighted by green, blue, and red colors. ( Ũ , Σ, Ṽ) of Jθ,t = ∇x t PΩ(f θ,t (xt)) 5: Compute the top-r ′ SVD ( Ū , Σ, V) of Jθ,t = ∇x t P Ω C (f θ,t (xt)) 6: Pick direction v ← Ṽ[:, i]▷ 1 Pick the k th singular vector for the editing direction • Finding localized Jacobians via masking. To enable local editing, we use a mask Ω (i.e., an index set of pixels) to select the region of interest,2 with P Ω (•) denoting the projection onto the index set Ω. For picking a local editing direction, we calculate the Jacobian of f θ,t (x t ) restricted to the region of interest, Jθ,t = ∇ xt P Ω (f θ,t (x t )) = Ũ Σ Ṽ ⊤ , and select the localized editing direction v from the top-r singular vectors of Ṽ (e.g., v = Ṽ [:, k] ∈ range J ⊤ θ,t for some index k ∈ [r]). In practice, a top-r rank estimation for Ṽ is calculated through the generalized power method (GPM) Algorithm 2 with r = 5 to improve efficiency.EQUATION• Better semantic disentanglement via nullspace projection. However, the projection P Ω (•) introduces extra nonlinearity into the mapping P Ω (f θ,t (x t )), causing the identified direction to have semantic entanglements with the area Ω C outside of the mask. Here, Ω C denotes the complimentary set of Ω. To address this issue, we can use the nullspace projection method [56, 57] . Specifically, given Jθ,t =∇ xt P Ω C (f θ,t (x t )) = Ū Σ V ⊤ , nullspace projection projects v onto null J ⊤ θ,t .The projection can be computed as v p = proj null( Jθ,t) (v) = (I -V V ⊤ )v so that the modified v p does not change the image in Ω C . In practice, we calculate a top-r ′ rank estimation for V through the generalized power method (GPM) Algorithm 2 with r ′ = 5.T-LOCO Edit. The unsupervised edit method can be seamlessly applied to T2I diffusion models with classifier-free guidance (3) (Algorithm 3). Besides, we can further enable text-supervised image editing with an editing prompt (Algorithm 4). See results in Figure 4(a) . This is useful because the additional text prompt allows us to enforce a specified editing direction that cannot be found easily in the semantic subspace of the vanilla Jacobian J θ,t . As illustrated in Figure 4 (b), this includes adding glasses or changing the curly hair of a human face. For simplicity, we introduce the key ideas of text-supervised T-LOCO Edit based upon DeepFloyd IF [19] . Similar procedures are also generalized to Stable Diffusion and Latent Consistency Models with an additional decoding step [4, 38] . We discuss the key intuition below, see Appendix E.2 and Appendix E.3 for method details. We first introduce some notations. Let c o denote the original prompt, and c e denote the editing prompt. For example, in Figure 4 (b), c o can be "portrait of a man", while c e can be "portrait of a man with glasses". Correspondingly, given the noisy image x t for the clean image x 0 generated with c o , let f o θ,t (x t ) and J o θ,t (x t ) be the estimated posterior mean and its Jacobian conditioned on the original prompt c o , and let f e θ,t (x t ) and J e θ,t (x t ) be the estimated posterior mean and its Jacobian conditioned on both the editing prompt c e and c o .According to the classifier-free guidance (3), we can estimate the difference of estimated posterior means caused by the editing prompt as d = f e θ,t (x t ) -f o θ,t (x t ), and then set v = J e θ,t (x t ) ⊤ d as an initial estimator of the editing direction. 3 Based upon this, to enable localized editing, similar to the unsupervised case, we can apply masks Ω to select ROI in d and calculate localized Jacobian to get v. After that, similarly, we can perform nullspace projection of v for better disentanglement to get the final editing direction v p . 
justification of local linearity, low-rankness, & semantic direction: 
 In this section, we provide theoretical justification for the benign properties in Section 3.1. First, we assume that the image distribution p data follows mixture of low-rank Gaussians defined as follows. Assumption 1. The data x 0 ∈ R d generated distribution p data lies on a union of K subspaces. The basis of each subspace {M k ∈ St(d, r k )} K k=1 are orthogonal to each other with M ⊤ i M j = 0 for all 1 ≤ i ̸ = j ≤ K, and the subspace dimension r k is much smaller than the ambient dimension d. Moreover, for each k ∈ [K], x 0 follows degenerated Gaussian withP (x 0 = M k a k ) = 1/K, a k ∼ N (0, I r k ). Without loss of generality, suppose x t is from the h-th class, that is x t = √ α t x 0 + √ 1 -α t ϵ where x 0 ∈ range(M h ), i.e. x 0 = M h a h . Both ||x 0 || 2 , ||ϵ|| 2 is bounded.Our data assumption is motivated by the intrinsic low-dimensionality of real-world image dataset [58] .Additionally, Wang et al. [59] demonstrated that images generated by an analytical score function derived from a mixture of Gaussians distribution exhibit conceptual similarities to those produced by practically trained diffusion models. Given that f θ,t (x t ) is an estimator of the posterior mean E[x 0 |x t ], we show that the posterior mean E[x 0 |x t ] can analytically derived as follows. Lemma 1. Under Assumption 1, for t ∈ (0, 1], the posterior mean isEQUATION)Lemma 1 shows that the posterior mean E [x 0 |x t ] could be viewed as a convex combination of M k M ⊤ k x t , i.e. x t projected onto each subspace M k . This lemma leads to the following theorem:Theorem 1. Based upon Assumption 1, we can show the following three properties for the posterior mean E[x 0 |x t ]:• The Jacobian of posterior mean satisfies rank (∇ xt E[x 0 |x t ]) ≤ r := K k=1r k for all t ∈ (0, 1].• The posterior mean E[x 0 |x t ] has local linearity such thatEQUATIONwhere ∆x ∈ S d-1 and λ ∈ R is the step size.• ∇ xt E[x 0 |x t ] is symmetric and the full SVD of ∇ xt E[x 0 |x t ] could be written as ∇ xt E[x 0 |x t ] = U t Σ t V ⊤ t , where U t = [u t,1 , u t,2 , . . . , u t,d ] ∈ St(d, d), Σ t = diag(σ t,1 , . . . , σ t,r , . . . , 0) with σ t,1 ≥ • • • ≥ σ t,r ≥ 0 and V t = [v t,1 , v t,2 , . . . , v t,d ] ∈ St(d, d). Let U t,1 := [u t,1 , u t,2 , . . . , u t,r ] and M := [M 1 , M 2 , . . . , M K ].It holds thatlim t→1 I d -U t,1 U ⊤ t,1 M F = 0.The proof is deferred to Appendix F. Admittedly, there are gap between our theory and practice, such as the approximation error between f θ,t (x t ) and E[x 0 |x t ], assumptions about the data distribution, and the high rankness of J θ,t for t < 0.2 and t > 0.9 in Figure 2 . Nonetheless, Theorem 1 largely supports our empirical observation in Section 3 that we discuss below:• Low-rankness of the Jacobian. The first property in Theorem 1 demonstrates that the rank of∇ xt E[x 0 |x t ]is always no greater than the intrinsic dimension of the data distribution. Given that the intrinsic dimension of the real data distribution is usually much lower than the ambient dimension [58] , the rank of J θ,t on the real dataset should also be low. The results align with our empirical observations in Figure 2 when t ∈ [0.2, 0.7]. • Linearity of the posterior mean. The second property in Theorem 1 shows that the linear approximation error is within the order of λα t /(1 -α t ) • O(λ). This implies that when t approaches 1, α t /(1 -α t ) becomes small, resulting in a small approximation error even for large λ. Empirically, Figure 2 shows that the linear approximation error of f θ,t (x t ) is small when t = 0.7 and λ = 40, whereas Figure 10 shows a much larger error for t = 0.0 under the same λ. These observations align well with our theory. • Low-dimensional semantic subspace. The third property in Theorem 1 shows that, when t is close to 1, left singular vectors associated with the top-r singular values form the basis of the image distribution. Since the editing direction consists of basis, the edited image remains within the image distribution. This explains why u i found in Equation ( 6) is a semantic direction for image editing. 
experiments: 
 In this section, we perform extensive experiments to demonstrate the effectiveness and efficiency of LOCO Edit. We first showcase LOCO Edit has strong localized editing ability across a variety of datasets in Section 5.1. Moreover, we conduct comprehensive comparisons with other methods to show the superiority of the LOCO Edit method in Section 5.2. Besides, we provide ablation studies on multiple components in our method in Appendix C.1, and analyze the editing directions in Appendix C.2, with extra experimental details postponed to Appendix G.First, we demonstrate benign properties of LOCO Edit in Algorithm 1 on a variety of datasets, including LSUN-Church [60] , Flower [61] , AFHQ [62] , CelebA-HQ [52] , and FFHQ [63] .As shown in Figure 5 and Figure 1a , our method enables editing specific localized regions such as eye size/focus, hair curvature, length/amount, and architecture, while preserving the consistency of other regions. Besides the ability of precise local editing, Figure 1 demonstrates the benign properties of the identified editing directions and verify our analysis in Section 4:• Linearity. As shown Figure 1 (d), the semantic editing can be strengthened through larger editing scales and can be flipped by negating the scale. • Homogeneity and transferability. As shown Figure 1 • Composability. As shown Figure 1 (c), the identified disentangled editing directions in the low-rank subspace allow direct composition without influencing each other.We compare LOCO Edit with several notable and recent image editing techniques, including Asyrp [29] , Pullback [30] , NoiseCLR [23] , and BlendedDifusion [24] . We also compare with an unexplored method using the Jacobians ∂ϵt ∂xt to find the editing direction, named as ∂ϵt ∂xt . Metrics. We evaluate our method using the below metrics and summarize the results in Table 1 . Besides the image generation quality, we also compared other attributes such as the local edit ability, efficiency, the requirement for supervision, and theoretical justifications.• Local Edit Success Rate evaluates whether the editing successfully changes the target semantics and preserves unrelated regions by human evaluators. • LPIPS [64] and SSIM [65] measure the consistency between edited and original images.• Transfer Success Rate measures whether the editing transferred to other images successfully changes the target semantics and preserves unrelated regions by human evaluators. • Learning time to measure the time required to identify the edit directions.• Transfer Edit Time to measure the time required to transfer the editing to other images directly.• #Images for Learning measures the number of images used to find the editing directions.• One-step Edit, No Additional Supervision, Theoretically Grounded, and Localized Edit are attributes of the editing methods, where each of them measures a specific property for the method.Moreover, we visualize the editing results on non-cherry-picked images in Figure 6 . The detailed evaluation settings are provided in Appendix G.2.Benefits of Our Method. Based upon the qualitative and quantitative comparisons, our method shows several clear advantages that we summarize as follows.• Superior local edit ability with one-step edit. it is supervision-free, thus integrating no biases from other modules such as CLIP [36] . [37] shows CLIP sometimes can't capture detailed semantics such as color. We can observe failures in capturing detailed semantics for methods that utilize CLIP guidance such as BlendedDiffusion and Asyrp in Figure 6 , where there are no edits or wrong edits. 
conclusion: 
 We proposed a new low-rank controllable image editing method, LOCO Edit, which enables precise, one-step, localized editing using diffusion models. Our approach stems from the discovery of the locally linear posterior mean estimator in diffusion models and the identification of a lowdimensional semantic subspace in its Jacobian, theoretically verified under certain data assumptions.The identified editing directions possess several beneficial properties, such as linearity, homogeneity, and composability. Additionally, our method is versatile across different datasets and models and is applicable to text-supervised editing in T2I diffusion models. Through various experiments, we demonstrate the superiority of our method compared to existing approaches. 
 
 Appendix: We identify several future directions and limitations of the current work. The current theoretical framework explains mainly the unsupervised image editing part. A more solid and thorough analysis of text-supervised image editing is of significant importance in understanding T2I diffusion models, which is yet a difficult open problem in the field. For example, there is still a lack of geometric analysis of the relationship between subspaces under different text-prompt conditions [4, 19, 38, 66] . Based on such understandings, it may be possible to further discover benign properties of editing directions in T2I diffusion models, or design more efficient fine-tuning [67, 68] accordingly. Besides, the current method has the potential to be extended for combining coarse to fine editing across different time steps. Furthermore, it is worth exploring the direct manipulation of semantic spaces in flow-matching diffusion models and transformer-architecture diffusion models. Lastly, it is possible to connect the current finding to image or video representation learning in diffusion models [69, 70, 71, 72] , extend to 3D editing of pose or shape [73, 74] , or utilize the low-rank structures to build dictionaries [75] .Study of Latent Semantic Space in Generative Models. Although diffusion models have demonstrated their strengths in state-of-the-art image synthesis, the understanding of diffusion models is still far behind the other generative models such as Generative Adversarial Networks (GAN) [76, 57] , the understanding of which can provide tools as well as inspiration for the understanding of diffusion models. Some recent works have identified such gaps, discovered latent semantic spaces in diffusion models [29] , and further studied the properties of the latent space from a geometrical perspective [30] . These prior arts deepen our understanding of the latent semantic space in diffusion models, and inspire later works to study the structures of information represented in diffusion models from various angles. However, their semantic space is constrained to diffusion models using UNet architecture, and can not represent localized semantics. Our work explores an alternative space to study the semantic expression in diffusion models, inspired by our observation of the low-rank and locally linear Jacobian of the denoiser over the noisy images. We provide a theoretical framework for demonstrating and understanding such properties, which can deepen the interpretation of the learned data distribution in diffusion models.Image Editing in Unconditional Diffusion Models. Recent research has significantly improved the understanding of latent semantic spaces in diffusion models, enabling global image editing through either training-free methods [29, 30, 31] or by incorporating an additional lightweight model [30, 77] . However, these methods result in poor performance for localized edit. In contrast, our approach achieves localized editing without requiring supervised training. For localized edits, [25] builds on [30] , enabling local edits by altering the intermediate layers of UNet. However, these approaches are restricted to UNet-based architectures in diffusion models and have largely ignored intrinsic properties like linearity and low-rankness. In comparison, our work provides a rigorous theoretical analysis of low-rankness and local linearity in diffusion models, and we are the first to offer a principled justification of the semantic significance of the basis used for editing. Moreover, our method is independent of specific network architectures.Other recent works, such as [32] , introduce training-free global audio and image editing based on a theoretical understanding of the posterior covariance matrix [33] , also independent of UNet architectures. However, our approach offers a distinct perspective, providing complementary insights and new findings. We explore the low-rank nature and local linearity in PMP, offering rigorous theoretical analyses. Based on this, our proposed LOCO Edit method allows unsupervised and localized editing, which enables several advantageous properties including transferability, composability, and linearity -benign features that have not been explored in prior work. Further, we extend the method to unsupervised and text-supervised editing in various text-to-image models. Additionally, while [24] supports localized editing, it requires supervision from CLIP, lacks a theoretical basis, and is time-consuming for editing each image. In contrast, our method is more efficient, theoretically grounded, and free from failures or biases in CLIP. The CLIP-supervised may also exhibit a bias We evaluate the numerical rank of the denoiser function x θ (x t , t) for DDPM (U-Net [49] architecture) on CIFAR-10 dataset [50] (d = 32 × 32 × 3), U-ViT [51] (Transformer based networks) on CelebA [52] (d = 64 × 64 × 3), ImageNet [53] datasets (d = 64 × 64 × 3) and DeepFloy IF [19] trained on LAION-5B [54] dataset (d = 64 × 64 × 3). Notably, U-ViT architecture uses the autoencoder to compress the image x 0 to embedding vector z 0 = Encoder(x 0 ), and adding noise to z t for the diffusion forward process; and the reverse process replaces x t , x t-∆t with z t , z t-∆t in Equation ( 1). And the generated image x 0 = Decoder(z 0 ). The PMP defined for U-ViT is:EQUATION)The J θ,t (z t ; t) = ∇ zt f θ,t (z t ; t) for f θ,t (z t ; t) defined above. For DeepFloy IF, there are three diffusion models, one for generation and the other two for super-resolution. Here we only evaluate J θ,t (z t ; t) for diffusion generating the images.Given a random initial noise x T , diffusion model x θ generate image sequence {x t } follows reverse sampler Equation (1) . Along the sampling trajectory {x t }, for each x t , we calculate J θ,t (z t ; t) and compute its numerical rank via rank(J θ,t (x t )) = arg min r r :EQUATION)where σ i (A) denotes the ith largest singular value of A. In our experiments, we set η = 0.99. We random generate 15 initialize noise x t (z t for U-ViT). We only use one prompt for DeepFloyd IF. We use DDIM with 100 steps for DDPM and DeepFloyd IF, DPM-Solver with 20 steps for U-ViT, and select some of the steps to calculate rank(J θ,t (x t ; t)), reported the averaged rank in Figure 2 . To report the norm ratio and cosine similarity, we select the closest t to 0.7 along the sampling trajectory and reported in Figure 2 , i.e. t = 0.71 for DDPM, t = 0.66 for U-ViT and t = 0.69 for DeepFloyd IF. The norm ratio and cosine similarity are also averaged over 15 samples.We illustrated the norm ratio and cosine similarity for more timesteps in Figure 10 , more text prompts, and flow-matching-based diffusion model in Figure 11 . More specifically, for the plot of t = 0.0, we exactly use t = 0.04 for DDPM, t = 0.005 for U-ViT and t = 0.09 for DeepFloyd IF; for the plot of t = 0.5, we exactly use t = 0.49 for DDPM, t = 0.50 for U-ViT and t = 0.49 for DeepFloyd IF. The results aligned with our results in Theorem 1 that when t is closer the 1, the linearity of f θ,t (x t , t) is better. This section is an extension of Section 3.1. We study the low rankness and local linearity of more mappings between spaces of diffusion models. The sampling process of diffusion model involved the following space:x t ∈ X t , x0,t ∈ X 0,t , h t ∈ H t , ϵ t ∈ E t, where H t is the h-space of U-Net's bottleneck feature space [29] and E t is the predict noise space. First, we explore the rank ratio of Then, we also study the linearity of h t and x0,t given x t , using DDPM with U-Net architecture trained on CIFAR-10 dataset. We change the step size λ defined in Equation ( 4). Results are shown in Figure 13 , both h t and x0,t have good linearity with respect to x t ..In Theorem 1, the jacobian ∇ xt E[x 0 |x t ] is a symmetric matrix. Therefore, we also verify the symmetry of the jacobian over the PMP J θ,t . We use DDPM with U-Net architecture trained on CIFAR-10 dataset. At different timestep t, we measure ||J θ,t -J ⊤ θ,t || F . Results are shown on the right of Figure 13 . J θ,t has good symmetric property when t < 0.1 and t ∈ [0.6, 0.7]. Additionally, J θ,t is low rank when t ∈ [0.6, 0.7]. So J θ,t aligned with Theorem 1 t ∈ [0.6, 0.7].To the end, we want to based on the experiments in Figure 12 and Figure 13 The Generalized Power Method [34, 30] for calculating the op-t singular vectors of the Jacobian is summarized in Algorithm 2. It efficiently computes the top-k singular values and singular vectors of the Jacobian with a randomly initialized orthonormal V ∈ R d×k .The overall method for DeepFloyd is summarized in Algorithm 3. For T2I diffusion models in the latent space such as Stable Diffusion and Latent Consistency Model, at time t, we additionally decode ẑ0 into the image space x0 to enable masking and nullspace projection. The editing is still in the space of z t .Algorithm 2 Generalized Power Method1: Input: f : R d → R d , x ∈ R d and V ∈ R d×k 2: Output: U , Σ, V ⊤ -k top singular values and vectors of the Jacobian ∂f ∂x 3: y ← f (x) 4: if V is empty then 5: V ← i.i.d. standard Gaussian samples 6: end if 7: Q, R ← QR(V ) ▷ Reduced QR decomposition 8: V ← Q ▷ Ensures V ⊤ V = I 9: while stopping criteria do 10: U ← ∂f (x + aV ) ∂a at a = 0 ▷ Batch forward 11: V ← ∂ U ⊤ y ∂x 12: V , Σ 2 , R ← SVD( V )▷ Reduced SVD 13: end while 14: Orthonormalize U Algorithm 3 Unsupervised T-LOCO Edit for T2I diffusion models 1: Input: Random noise xT , the mask Ω, edit timestep t, pretrained diffusion model ϵ θ , editing scale λ, noise scheduler αt, σt, selected semantic index k, nullspace approximate rank r, original prompt co, null prompt cn, classifier free guidance scale s. EQUATIONBefore introducing the algorithm, we define:EQUATIONandEQUATIONto be the posterior mean predictors when using classifier-free guidance on the original prompt c o , and both the original prompt c o and the edit prompt c e accordingly.Algorithm. The overall method for DeepFloyd is summarized in Algorithm 4. For T2I diffusion models in the latent space such as Stable Diffusion and Latent Consistency Model, at time t, we additionally decode ẑ0 into the image space x0 to enable masking and nullspace projection. The editing is in the space of z t for Stable Diffusion and Latent Consistency Model. The proposed method is not proposed as an approach beating other T2I editing methods, but as a way to both understand semantic correspondences in the low-rank subspaces of T2I diffusion models and utilize subspaces for semantic control in a more interpretable way. We hope to inspire and open up directions in understanding T2I diffusion models and utilize the understanding in versatile applications. Here, we want to find a specific change direction v p in the x t space that can provide target edited images in the space of x 0 by directly moving x t along v p : the whole generation is not conditioned on c e at all, except that we utilize c e in finding the editing direction v p . This is in contrast to the method proposed in [30] , where additional semantic information is injected via indirect x-space guidance conditioned on the edit prompt at time t. We hope to discover an editing direction that is expressive enough by itself to perform semantic editing.Intuition. Let xo 0,t be the estimated posterior mean conditioned on the original prompt c o , and xe 0,t be the estimated posterior mean conditioned on both the original prompt c o and the edit prompt c e . Let J o θ,t and J e θ,t be their Jacobian over the noisy image x t accordingly. The key intuition inspired by the unconditional cases are: i) the target editing direction v in the x t space is homogeneous between the subspaces in J o θ,t and J e θ,t ; ii) the founded editing direction v can effectively reside in the direction of a right singular vector for both J o θ,t and J e θ,t ; iii) xe 0,t and xo 0,t are locally linear.Define xe 0,t -xo 0,t = d as the change of estimated posterior mean. Let J e θ,t = U e t S e t V e T t , then v = ±v e i for some i. Besides, we have xe 0,t = xo 0,t + λ o J o θ,t v and xo 0,t = xe 0,t + λ e J e θ,t v due to homogeneity and linearity. Hence, d = -λ e J e θ,t v = ±λ e s e i u e i and then J e T θ,t d = ±λ e s e i s e i v e i = ±λ e s e i s e i v, which is along the desired direction v. And this v identified through the subspace in J e θ,t can be effectively transferred in J o θ,t for controlling the editing of target semantics. We further apply nullspace projection based on J o θ,t to obtain the final editing direction v p .F.1 Proofs of Lemma 1Proof of Lemma 1. Under the Assumption 1, we could calculate the noised distribution p t (x t ) at any timestep t,p t (x t ) = 1 K K k=1 p t (x t |"x 0 belongs to class k") = 1 K K k=1 p t (x t |x 0 = M k a k , "x 0 belongs to class k")N (a k ; 0, I r k )da k . Because a k ∼ N (0, I r k ), p t (x t |x 0 = M k a k , "x 0 belongs to class k") ∼ N ( √ α t M k a k , (1 - α t )I d ).From the relationship between conditional Gaussian distribution and marginal Gaussian distribution, it is easy to show that p t (x t |"x 0 belongs to class k") ∼ N (0,α t M k M ⊤ k + (1 -α t )I d ) Then, we have p t (x t ) = 1 K K k=1 N (0, α t M k M ⊤ k + (1 -α t )I d ).Next, we compute the score function as follows:∇ xt logp t (x t ) = ∇ xt p t (x t ) p t (x t ) = K k=1 N (0, α t M k M ⊤ k + (1 -α t )I d ) - 1 1 -α t x t + α t 1 -α t M k M ⊤ k x t K k=1 N (0, α t M k M ⊤ k + (1 -α t )I d ) = - 1 1 -α t x t + α t 1 -α t K k=1 N (0, α t M k M ⊤ k + (1 -α t )I d )M k M ⊤ k x t K k=1 N (0, α t M k M ⊤ k + (1 -α t )I d ).Based on Tweedie's formula [45, 82] , the relationship between the score function and posterior isEQUATIONTherefore, the posterior mean isE[x 0 |x t ] = √ α t K k=1 N (0, α t M k M ⊤ k + (1 -α t )I d )M k M ⊤ k x t K k=1 N (0, α t M k M ⊤ k + (1 -α t )I d ) = √ α t K k=1 exp - 1 2 x ⊤ t α t M k M ⊤ k + (1 -α t )I d -1 x t M k M ⊤ k x t K k=1 exp - 1 2 x ⊤ t α t M k M ⊤ k + (1 -α t )I d -1 x t = √ α t K k=1 exp - 1 2(1 -α t ) ∥x t ∥ 2 -α t ∥M ⊤ k x t ∥ 2 M k M ⊤ k x t K k=1 exp - 1 2(1 -α t ) ∥x t ∥ 2 -α t ∥M ⊤ k x t ∥ 2 = √ α t K k=1 exp α t 2(1 -α t ) ∥M ⊤ k x∥ 2 M k M ⊤ k x t K k=1 exp α t 2(1 -α t ) ∥M ⊤ k x∥ 2 ,where the third equation is obtained by Woodbury formula [83] (α t M k M ⊤ k + (1 -α t )I d ) -1 = 1 1 -α t I d -α t M k M ⊤ k . ⊔ ⊓ F.2 Proofs of Theorem 1Lemma 2. The jacobian of the poster mean isEQUATIONwhereω k (x t ) := exp α t 2 (1 -α t ) ∥M ⊤ k x t ∥ 2 K l=1 exp α t 2(1 -α t ) ∥M ⊤ l x∥ 2 Proof of Lemma 2. Let ω k (x t ) := exp α t 2 (1 -α t ) ∥M ⊤ k x t ∥ 2 K l=1 exp α t 2(1 -α t ) ∥M ⊤ l x∥ 2, so we have:E [x 0 |x t ] = √ α t K k=1 ω k (x t )M k M ⊤ k x t ∇ xt ω k (x t ) = α t (1 -α t ) ω k (x t ) M k M ⊤ k x t - K l=1 ω l (x t )M l M ⊤ l x tSo:∇ xt E [x 0 |x t ] = √ α t K k=1 ω k (x t )M k M ⊤ k + √ α t K k=1 ∇ xt ω k (x t )x ⊤ t M k M ⊤ k = √ α t K k=1 ω k (x t )M k M ⊤ k + α t √ α t (1 -α t ) K k=1 ω k (x t )M k M ⊤ k x t x ⊤ t M k M ⊤ k - α t √ α t (1 -α t ) K k=1 ω k (x t )M k M ⊤ k x t x ⊤ t K k=1 ω k (x t )M k M ⊤ k ⊤ .⊔ ⊓ Lemma 3. Assume second-order partial derivatives of p t (x t ) exist for any x t , then the posterior mean∇ xt E [x 0 |x t ] satisfied ∇ xt E [x 0 |x t ] = ∇ xt E ⊤ [x 0 |x t ].Proof of Lemma 3. By taking the gradient of Equation ( 13) with respect to x t for both side, because the second-order partial derivatives of p t (x t ) exist for any x t , we have:∇ xt E[x 0 |x t ] = I + (1 -α t )∇ 2 xt logp t (x t ) √ α t .The hessian of logp t (x t ) is symmetric, so we have:∇ xt E ⊤ [x 0 |x t ] = I + (1 -α t ) ∇ 2 xt logp t (x t ) ⊤ √ α t = I + (1 -α t )∇ 2 xt logp t (x t ) √ α t = ∇ xt E[x 0 |x t ].Notably, the symmetric of ∇ xt E[x 0 |x t ] holds without the Assumption 1.Proof of Theorem 1. First, let's prove the low-rankness of the posterior mean. From Lemma 2,∇ xt E [x 0 |x t ] = √ α t A + α t √ α t (1 -α t ) B - α t √ α t (1 -α t ) C = K k=1 M k M ⊤ k √ α t A + α t √ α t (1 -α t ) B - α t √ α t (1 -α t ) C ,where the second equation is obtained due to the fact thatEQUATIONThen, we prove the linearity:1 ⃝ : ||E [x 0 |x t + λ∆x] -E [x 0 |x t ] -λ∇ xt E[x 0 |x t ]∆x|| 2 =|| √ α t K k=1 (ω k (x t + λ∆x) -ω k (x t )) M k M ⊤ k (x t + λ∆x) -λ K k=1 ∇ xt ω k (x t )x ⊤ t M k M ⊤ k ∆x|| 2 =|| √ α t K k=1 λ∇ ⊤ xt ω k (x t + λ 1 ∆x)∆x M k M ⊤ k (x t + λ∆x) -λ K k=1 ∇ xt ω k (x t )x ⊤ t M k M ⊤ k ∆x|| 2 ≤λ K k=1 √ α t ∇ ⊤ xt ω k (x t + λ 1 ∆x)∆x||M ⊤ k (x t + λ∆x) || 2 + x ⊤ t M k M ⊤ k ∆x||∇ ⊤ xt ω k (x t )|| 2 ≤λ K k=1 √ α t ||∇ xt ω k (x t + λ 1 ∆x)|| 2 ||M ⊤ k (x t + λ∆x) || 2 + ||∇ xt ω k (x t )|| 2 ||M ⊤ k x t || 2where the first equation plug in the formula of∇ xt E [x 0 |x t ] = √ α t K k=1 ω k (x t )M k M ⊤ k + √ α t K k=1 ∇ xt ω k (x t )x ⊤ t M k M ⊤ k and the second equation use the mean value theorem ω k (x t + λ∆x) -ω k (x t ) = λ∇ ⊤ xt ω k (x t + λ 1 ∆x)∆x, λ 1 ∈ (0, λ). 2 ⃝ : ||∇ xt ω k (x t + λ 1 ∆x)|| 2 = α t (1 -α t ) ω k ||M k M ⊤ k (x t + λ 1 ∆x) - K l=1 ω l M l M ⊤ l (x t + λ 1 ∆x)|| 2 ≤ α t (1 -α t ) ω k ||M ⊤ k x t || 2 + K l=1 ω l ||M ⊤ l x t || 2 + λ 1 ||M ⊤ k ∆x|| 2 + λ 1 K l=1 ω l ||M ⊤ l ∆x|| 2 ≤ α t (1 -α t ) ω k ||M ⊤ k || F ||x t || 2 + K l=1 ω l ||M ⊤ l || F ||x t || 2 + λ 1 ||M ⊤ k || F + λ 1 K l=1 ω l ||M ⊤ l || F ≤ α t (1 -α t ) ω k r k + K l=1 ω l r l √ 2 max{||x 0 || 2 , ||ϵ|| 2 } + λ 1 ≤ α t (1 -α t ) ω k (x t + λ 1 ∆x) •2 • max k r k • √ 2 max{||x 0 || 2 , ||ϵ|| 2 } + λ 1 C1:= ,where the third inequality use the fact thatEQUATION}, we simplified ω k (x t + λ 1 ∆x) as ω k in this prove, and C 1 defined in the last inequality is independent of t. Similarly, we could prove that: ⃝ : ||M k M ⊤ k (x t + λ∆x) || 2 ≤ max k r k • √ 2 max{||x 0 || 2 , ||ϵ|| 2 } + λ C2:= , 4 ⃝ : ||∇ xt ω k (x t )|| 2 ≤ α t (1 -α t ) ω k (x t ) 2 √ 2 • max k r k • max{||x 0 || 2 , ||ϵ|| 2 } C3:= , 5 ⃝ : ||M k M ⊤ k x t || 2 ≤ √ 2 max k r k • max{||x 0 || 2 , ||ϵ|| 2 } C4:= .Here,C 1 = O(λ), C 2 = O(λ), C 3 = O(λ), C 4 = O(λ). After plugin 2 ⃝, 3 ⃝, 4 ⃝, 5 ⃝ to 1 ⃝, we could obtain: ||E [x 0 |x t + λ∆x] -E [x 0 |x t ] -λ∇ xt E[x 0 |x t ]∆x|| 2 ≤λ √ α t K k=1 α t (1 -α t ) ω k (x t + λ 1 ∆x)C 1 C 2 + λ K k=1 α t (1 -α t ) ω k (x t )C 3 C 4 =λ α t (1 -α t ) O(λ)Finally, let's prove the property of the left singular vector of∇ xt E [x 0 |x t ]: From Lemma 3, the eigenvalue decomposition of ∇ xt E [x 0 |x t ] could be written as ∇ xt E [x 0 |x t ] = U t Λ t U ⊤ t, where Λ t = diag(λ t,1 , . . . , λ t,r , . . . , 0), and the relation between eigenvalue decomposition and singular value decomposition of ∇ xt E [x 0 |x t ] could be summarized as for all i ∈ [r]: σ t,i = |λ t,i |, v i = sign (λ t,i ) u i , where sign (•) is the sign function. Therefore, we have:EQUATIONgiven V t,1 := [v t,1 , v t,2 , . . . , v t,r ]. From Lemma 2, we define:∇ xt E [x 0 |x t ] = √ α t K k=1 ω k (x t )M k M ⊤ k + √ α t K k=1 ∇ xt ω k (x t )x ⊤ t M k M ⊤ k ∆t:= .From the full singular value decomposition of∇ xt E [x 0 |x t ] and √ α t K k=1 ω k (x t )M k M ⊤ k : ∇ xt E [x 0 |x t ] = [U t,1 U t,2 ] Σ t,1 0 0 Σ t,2 V t,1 V t,2 ⊤ , √ α t K k=1 ω k (x t )M k M ⊤ k = Ût,1 Ût,2 Σt,1 0 0 Σt,2 Vt,1 Vt,2 ⊤ .where:Σ t,1 =    σ t,1 . . . σ t,r    , Σ t,2 =    σ t,r+1 . . . σ t,n    , Σt,1 =    σt,1 . . . σt,r    , Σt,2 =    σt,r+1 . . . σt,n    σ t,1 ≥ σ t,2 ≥ . . . ≥ σ t,r ≥ . . . ≥ σ t,d , σt,1 ≥ σt,2 ≥ . . . ≥ σt,r ≥ . . . ≥ σt,d , r = K k=1 r k .From Equation ( 15), we know that σ t,r+1 = . . . = σ t,d = 0. It is easy to show that:M := Vt,1 = [M s1 M s2 . . . M s K ] ,where {s 1 , s 2 , . . . , sK } = {1, 2, . . . , K} satisfied ω s1 (x t ) ≥ ω s2 (x t ) ≥ . . . ≥ ω s K (x t ). And σt,r = √ α t ω s K (x t ) = √ α t min k ω k (x t ).Based on the Davis-Kahan theorem [84] , we have:|| I d -V t,1 V ⊤ t,1 M || F ≤ ||∆ t || F min 1≤i≤r,r+1≤j≤d |σ t,i -σ t,j | = || √ α t K k=1 ∇ xt ω k (x t )x ⊤ t M k M ⊤ k || F √ α t min k ω k (x t ) ≤ K k=1 ||∇ xt ω k (x t )|| F ||x ⊤ t M k M ⊤ k || F min k ω k (x t ) = α t 1 -α t C 3 C 4 min k ω k (x t ) . Because lim t→1 min k ω k (x t ) = 1 K , lim t→1 α t 1 -α t = 0, so: lim t→1 || I d -V t,1 V ⊤ t,1 M || F = 0.And from Equation ( 16), we have:lim t→1 || I d -U t,1 U ⊤ t,1 M || F = 0.⊔ ⊓All the experiments can be conducted with a single A40 GPU having 48G memory.Datasets. We demonstrate the unconditional editing method in various dataset: FFHQ [63] , CelebaA-HQ [52] , AFHQ [62] , Flowers [61] , MetFace [85] , and LSUN-church [60] .Models. Following [30] , we use DDPM [1] for CelebaA-HQ and LSUN-church, and DDPM trained with P2 weighting [86] for FFHQ, AFHQ, Flowers, and MetFaces. We download the official pre-trained checkpoints of resolution 256 × 256, and keep all model parameters frozen. We use the same linear schedule including 100 DDIM inversion steps [3] as [30] . Further, we apply quanlity boosting after t = 0.2 as proposed in [87] .Edit Time Steps. We empirically choose the edit time step t for different datasets in the range [0.5, 0.8]. In practice, we found time steps within the above range give similar editing results. In most of the experiments, the edit time steps chosen are: 0.5 for FFHQ, 0.6 for CelebaA-HQ and LSUN-church, 0.7 for AFHQ, Flowers, and MetFace.Editing Strength. In the empirical study of local linearity, we observed that the local linearity is well-preserved even with a strength of 300. In practice, we choose the edit strength λ in the range of [-15.0, 15.0], where a larger α leads to stronger semantic editing and a negative α leads to the change of semantics in the opposite direction.Existing Methods We compare with four existing methods: NoiseCLR [23] , BlendedDiffusion [24] , Pullback [30] , and Asyrp [29] .Alternative Manifolds. Evaluation Dataset Setup. In human evaluation, for each method, we randomly select 15 editing direction on 15 images. Each direction is transferred to 3 other images along both the negative and positive directions, in total 90 transferability testing cases. Learning time and transfer edit time are averaged over 100 examples. LPIPS [64] and SSIM [65] are calculated over 400 images for each method.Human Evaluation Metrics. We measure both Local Edit Success Rate and Transfer Success Rate via human evaluation on CelebA-HQ. i) Local Edit Success Rate: The subject will be given the source image with the edited one, if the subject judges only one major feature among {"eyes", "nose", "hair", "skin", "mouth", "views", "Eyebrows"} are edited, the subject will respond a success, otherwise a failure. ii) Transfer Success Rate: The subject will be given the source image with the edited one, and another image with the edited one via transferring the editing direction from the source image. The subject will respond a success if the two edited images have the same features changed, otherwise a failure. We calculate the average success rate among all subjects for both Local Edit Success Rate and Transfer Success Rate. Lastly, we have ensured no harmful contents are generated and presented to the human subjects.Learning Time. Learning time is a measure of the time it takes to compute local basis(training free approaches), to train an implicit function, or to optimize certain variables that help achieve editing for a specific edit method.Models. We generalize our method to three types of T2I diffusion models: DeepFloyd [19] , Stable Diffusion [4] , and Latent Consistency Model [38] . We download the official checkpoints and keep all model parameters frozen. The same scheduling as that in the unconditional models is applied to DeepFloyd and Stable Diffusion, except that no quality boosting is applied. We follow the original schedule for Latent Consistency Model [38] with the number of inference steps set as 4.Edit Time Steps. We empirically choose the the edit time step t as 0.75 for DeepFloyd and 0.7 for Stable DIffusin. As for Latent Consistency Model, image editing is performed at the second inference step.Editing Strength. The paper originally presents a new image manipulation method, with a theoretical framework to deepen the understanding of diffusion models. However, there exist potential social impacts that the proposed methods can be misused in generating and manipulating harmful content. Therefore, we will release our code and models with license and ethics commitments in the future. Besides, methods for identifying and preventing such harmful behaviors are of great significance in generative models.For datasets that have predefined masks, we can use them directly. For other datasets that lack predefined masks as well as generate images, we can utilize Segment Anything (SAM) to generate masks[55].The idea is to identify the editing direction in the Xt space based on changes in the estimated posterior mean caused by the editing prompt. More details are provided in Appendix E.3. 


Decision: This paper proposes a diffusion image editing framework called LOw-rank COntrollable edit (LOCO Edit) based on two observations: 1) the learned posterior mean predictor (PMP) is locally linear during the middle timesteps during denoising, and 2) the singular vectors of the PMP's Jacobian lie in low-dimensional semantic subspaces. These observations are backed both empirically and theoretically. To conduct LOCO edit, an image is inverted with DDIM inversion to an intermediate timestep. SVD is conducted on the Jacobian of the PMP w.r.t. to this noisy image in order to discover semantic directions. Furthermore, a mask can be applied along with a null space projection in order to achieve more disentangled and localized edits. A variant of LOCO edit is also proposed to enable text direction.

The reviewers found the observations in terms of local linearity of the denoiser interesting and well-validated across architectures and dataset. They found the method novel, well-motivated, and appreciated its support on theoretical arguments, finding the discovery of homogeneity and transferability of the editing direction insightful. They also pointed out that LOCO tackles an important challenge in diffusion models: achieving precise and disentangled image editing without additional training, and found the paper to be well written.

There were some concerns about missing citations and comparisons to previous works and, most importantly, limited experimental validation. While the paper considers various architectures and performs various ablations, there were limited examples of the success of the edits. Various questions of detail were also raised. The authors addressed these issues in their rebuttal, providing several new examples of edits. While these are still only for object centric images, they show a wealth of image modifications that can be of interest for applications. In the end, all reviewers were positive towards the paper, and the ACs consur that the paper merits publication.

Reviews: review: paper_summary: paper_summary: This paper made an interesting observation on local linearity of the diffusion model's denoiser. Based on the observation, the author introduces a novel method of performing one-step closed-forrm operation to achieve semantic image editing. Empirical and numerical results are given to demonstrate the effectiveness of the method
strength_weakness: strength_weakness: 1. The computation time of the method is still a bit long (taking around 70s if I understand correctly).
questions: questions: 1. I wonder whether the homogeneity of the editing can be observe on samples that are very different: e.g., 1) computing editing direction on an image with face at the left, and transfer to an image with face on the right; 2) computing editing direction on an image with realistic style, and transfer to an image with cartoon style.
2. In addition, since current models such as SD3 are using flow-matching objective to train the model, I wonder if such observed phenomenon would be more prominent on these models?
limitations: limitations: Yes, the limitations have been adequately addressed.
paper_summary: paper_summary: The paper presents a method for steering the generation in a diffusion model, without any further training. The proposed method is based on two insights about the Posterior Mean Predictor (PMP) and its Jacobian -- that is, the former being locally linear and the latter having singular vectors lying on low-dimensional subspaces. Some theoretical results are provided towards the justification of the above. Local linearity of the PMP allows for a single-step, training-free method for local editing of regions of interest, while the low-rank nature allows for the effective identification of semantic directions using subspace power methods. Some qualitative and quantitative experimental results are provided.
strength_weakness: strength_weakness: The weakest aspect of the paper concern the reported experimental results. Both qualitative and quantitative results are extremely limited, whilst comparisons with existing work are also not convincing. Besides, the significance of the empirical results is not convincing. For instance, in Fig. 6, when the proposed method (LOCO) is compared to Asyrp, it's not clear at all why localized editing with LOCO is better than Asyrp. Similarly, in Table 1, the Transfer CLIP Score (an important metric in the context of the task studied in the paper) of Asyrp is significantly better that the proposed method's. The authors do discuss this, but the fact that the proposed framework is learning-free, in contrast to Asyrp that learns how to perform editing, does not justify the significant difference wrt to CLIP Score. 

Finally, as the authors acknowledge in the Appendix H ("Future Direction", where limitations are also discussed), the provided theoretical framework concerns mainly the undirected image editing part; how text-directed image editing behaves is not addressed by the proposed method. This is a certain limitation, that weakens the generality of the proposed method, yet it doesn't reduce the importance of the theoretical results provided.
questions: questions: Please see weaknesses.
limitations: limitations: The authors discuss adequately the limitations of the proposed method in the Appendix H (Future Direction). A separate "Limitations" section is missing, but the authors discuss the limitations of the paper honestly and comprehensively.
paper_summary: paper_summary: The paper examines the use of low-dimensional subspaces in diffusion models for precise and disentangled image editing. The paper observes that the Posterior Mean Predictor (PMP) in diffusion models shows local linearity across various noise levels, and the singular vectors of its Jacobian exist in low-dimensional semantic subspaces. Building on these observations, they introduce LOw-rank COntrollable image editing (LOCO Edit), a technique that enables efficient, training-free, and precise localized image manipulation by utilizing the Jacobian's low-rank nature. This approach has a strong theoretical basis and has been shown to be effective across different architectures and datasets.
strength_weakness: strength_weakness: 1. The experimental evidence appears highly limited. The quantitative and qualitative comparison results (confined to Table 1 and Fig. 6 with just one example in the main paper) seem inadequate. The study would be strengthened by a more comprehensive comparison with additional state-of-the-art image editing methods, beyond just [24] and [25]. Highlighting specific advantages and potential weaknesses in relation to these methods would provide a more balanced perspective.

2. The assumptions regarding low-rank Gaussian distributions used for theoretical validation warrant more rigorous examination. Expanding the discussion to include potential limitations or circumstances where these assumptions may not hold would add depth and nuance to the analysis.
questions: questions: 1. The paper should clarify that the Posterior Mean Predictor (PMP) is a network function within diffusion models, and detail how it is computed. Specifically, it should describe how PMP is derived and utilized in the context of local linearity and low-rankness.
   
2. Elaborate on what P_{\omega} and P_{\omega^C} represent in Fig.3 and lines 193-195, and clarify the decomposition of x_{0,t}^{\hat} into ROI and null space using masking techniques.

3. Explain how performing the Jacobian over different masked images helps in disentangling features for precise local editing.
limitations: limitations: In addition to the listed weakness above, the sensitivity of the method to various parameters (e.g., noise levels, perturbation strengths) could be explored in greater detail. Understanding the robustness of the approach under different conditions would be valuable for practical applications.
paper_summary: paper_summary: This paper proposes an diffusion image editing framework called LOw-rank COntrollable edit (LOCO Edit) based on two observations: 1)  the learned posterior mean predictor (PMP) is locally linear during the middle timesteps during denoising, and 2) the singular vectors of the PMP's Jacobian lie in low-dimensional semantic subspaces. These observations are backed both empirically and theoretically. To conduct LOCO edit, an image is inverted with DDIM inversion to an intermediate timestep. SVD is conducted on the Jacobian of the PMP w.r.t. to this noisy image in order to discover semantic directions. Furthermore, a mask can be applied along with a null space projection in order to achieve more disentangled and localized edits. A variant of LOCO edit is also proposed to enable text direction. A variety of experiments are conducted for the unconditional LOCO edit on various domains using human evaluation and CLIP score and edit discovery time, achieving superior results against other methods. Qualitative results are shown for various diffusion model variants (e.g., Stable Diffusion, Latent Consistency Model) and domains, demonstrating linearity and the ability to modulate strength, the ability to compose with other directions, as well as the ability to transfer the edits to other images.
strength_weakness: strength_weakness: * The qualitative results for editing are not quite the best I have seen, and they are compared against relatively weak baselines (e.g. Asyrp). However, I find that this is okay since the approach is quite principled.  Many impressive diffusion based image editing papers have been released in the past year [1,2]. I don't think it is necessary to compare against them, but including them in the paper in the related works section would keep this paper up-to-date with the current state of image editing. 

* Since the linearity and low-rankness properties are more apparent in the middle time-steps, applying more global edits (e.g. changing the shape of something while keeping appearance) would be difficult since more coarse level features are constructed in the earlier timesteps of denoising. I see this effect in Fig. 7 in the appendix, as other parts of the image are edited as well. 

[1] @inproceedings{dalva2024noiseclr,
  title={Noiseclr: A contrastive learning approach for unsupervised discovery of interpretable directions in diffusion models},
  author={Dalva, Yusuf and Yanardag, Pinar},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={24209--24218},
  year={2024}
}
[2] @article{gandikota2023concept,
  title={Concept sliders: Lora adaptors for precise control in diffusion models},
  author={Gandikota, Rohit and Materzynska, Joanna and Zhou, Tingrui and Torralba, Antonio and Bau, David},
  journal={arXiv preprint arXiv:2311.12092},
  year={2023}
}
questions: questions: * I am quite surprised that an edit at a single timestep has enough effect to edit the image. For instance, [1,2] require over multiple timesteps. Is there an intuition for why this edit at a single timestep has enough causal effect?

* For more open-domain diffusion models like Stable Diffusion, can the edit direction be applied to another subject? For instance, in Figure 1, you transfer human opening eyes to different images. Would this transfer to other domains, such as an animal?

* A visualization of all the edit directions discovered via SVD would be quite interesting. Are they all semantically meaningful?
limitations: limitations: A more substantive discussions of limitations is missing. Although Appendix Section H discusses future directions, there is little mention of limitations of the current method.
paper_summary: paper_summary: Even though there have been many research papers on conditional Diffusion Models, sampling-based disentangling method remains a challenge. The authors empirically/theoretically show that PMP is locally linear and the singular vectors of the gradient of PMP are in low-rank space.  Based this, the authors propose LOCO Edit that can benefit homogeneity, composability and linearity in semantic controls. Empirical studies are provided to prove the effectiveness of the method.
strength_weakness: strength_weakness: 1. Literature review of the recent related works
2. Lack of baselines and comparisons with other methods (Please add comparisons with other local/global editing papers.)
3. Uninterpretable semantic changes. For example, in Fig. 5, color, light angle, tower architecture are not controllable. Is it red? Blue? What is the meaning of color? It would be more interesting if the authors can add some interpretable ways to control the semantics (e.g., using label information).
4. Almost everything this paper describes is based on PMP while the actual results are not from PMP but from DDIM.
5. Lack of ablation studies; null space projection
6. The metrics used in Fig. 2 are not intuitive and descriptions are not clear.
questions: questions: 1. (Fig. 2) 
- Can the results be generalized? It seems like only a few x_T’s are used, and only one prompt is used (as mentioned in Appendix D.1) 
- fig2(a) The authors mentioned in L79 that “rank (A) denotes the numerical rank of A”.  However, in Eq. 10 and Fig. 2,  it is shown as rank ratio. First of all, what is the rank ratio? and how Eq. 10 can be interpreted? Please elaborate them.
- fig2(b) To my understanding, $f$ and $l$ (in L146) seem to be a sort of the denoised images. Why not directly computing their distance? It would be more natural measure for the statement $f \sim l$ in L142.

2. (L80) the x_0 sampled from p_data might need to be the input to get x_t, not the output of the posterior distribution.

3. (L37) What are the various other unexplored aspects?

4. (Fig. 4) What are the undirected t2i edit? If text is not given, why is it called as t2i edit?

5. (L159-165) Are the three properties shown to be true in the experiments? For example, what about composing multiple singular vectors?
limitations: limitations: 1. (L157) and (Alg. 1) The authors mentioned that the proposed method benefits the one-step editing while the actual algorithm needs to sample by DDIM twice (DDIM inv and DDIM).

2. (L189-L190) I think there would have been many papers regarding local editing based on Diffusion Models such as Blended diffusion [1].  


[1] Blended Diffusion: Text-driven Editing of Natural Images, CVPR’22

Comments: title: looking forward to your reply
Looking forward to your reply
comment: Thanks for the questions, we would like to further make clarifications on these points.

1. We have extended the comparisons with **two additional baselines** BlendedDiffusion [2] and NoiseCLR [1] for comparison. See global rebuttal Q&A 1 and pdf Tab. A and Fig. E.

2. (1) The human evaluation dataset is randomly selected have high diversity to cover various semantic directions; (2) Asyrp [3] uses only 40 data samples for human evaluation on CelebA (Appendix K.1), Pullback [4] only shows qualitative results; (3) BlendedDiffusion and NoiseCLR do not mention the size of evaluation dataset; (4) A further detail is, we used 400 samples for other added metrics LPIPS and SSIM. Hence, we think the evaluation dataset size is robust enough to provide fair comparisons.

3. The mentioned FID and IS are good metrics measuring whether generated image distributions are similar to the training images, but **not standard metrics for image editing methods**. (1) FID and IS will be increased if additional features are edited in comparison to the original one; (2) Representative baslines including NoiseCLR, BlendedDiffusion, Asyrp, and Pullback use neither of the FID and IS; (3) As in global rebuttal Q&A 1 and pdf Tab. A, we have conducted a comprehensive comparison using 7 metrics and 4 attributes to show the superiority of our method. This indeed supports that we have conducted comprehensive empirical evidence in addition to the solid theoretical basis which is lacking in the previous papers. 

4. For more ablation studies, as summarized in Q2&A2 for the global response, we do more ablation studies on Noise levels (i.e., time steps), Perturbation strength, Nullspace projection, and ranks. We have shown representative examples, and more results will be included in the revision.

We appreciate the reviewers' prompt responses and make further clarification and references on how we conduct comprehensive experiments to show the superiority of our method. We kindly request the reviewer evaluate our work fairly based on the rebuttal and our further clarifications.

[1] Yusuf et al., Noiseclr: A contrastive learning approach for unsupervised discovery of interpretable directions in diffusion models. CVPR 2024.

[2] Omri et al., Blended diffusion for text-driven editing of natural images. CVPR 2022.

[3] Mingi et al. Diffusion models already have a semantic latent space. ICLR 2023.

[4] Yong et al. Understanding the latent space of diffusion models through the lens of riemannian geometry. NeurIPS 2023.
I appreciate the authors' efforts in addressing concerns through their rebuttal. While some issues have been clarified, I still have reservations about the depth of empirical evidence presented in the original paper:

1. Limited comparative analysis: The paper only includes two comparison methods ([24, 25]), which may not provide a comprehensive benchmark.

2. Small sample sizes: The evaluation relies on relatively few samples (90 cases for transferability testing, 100 examples for other metrics), potentially limiting the robustness of the findings.

3. Absence of standard metrics: Apart from using the CLIP score, the study doesn't utilize other widely accepted measures such as Fréchet Inception Distance and Inception Score, which could have strengthened the evaluation.

4. Insufficient ablation studies: The paper lacks thorough ablation studies to demonstrate the contribution of individual components.

While the rebuttal has partially addressed these concerns, the original paper's limitations in empirical validation remain significant. Consequently, I am inclined to either maintain my initial score or consider a slight downward adjustment.
Dear reviewer,

Thanks for the positive response and we appreciate your acknowledgment of our responses in addressing the existing problems.

For the proposed precise point: (1) We have tested transferring edit direction from “white to red” to flowers in other colors including blue, pink, and orange. We saw the colors change to darker blue, darker pink, and darker orange. Considering the effects of transferring, the edit direction can be described as “darker color”, which is semantically meaningful for flowers in general. (2) To further discuss the point, when transferring “enlarging eyes” to images of a person wearing sunglasses, only the details of the sunglasses are changed which is hard to notice. This is because “enlarging eyes” is not semantically meaningful for people wearing sunglasses. (3) Such edge cases are interesting, but since we consider practical editing scenarios where the edit directions are semantically meaningful, we do not present them in the paper. Besides, we can not show additional figures, but we will add those in our revision to make the discussion more complete.

Thanks for the question and hope we have further addressed your concerns. We are happy to answer any other questions.

Best Regards,

Authors
Thank you for clarifying my concerns during the rebuttal.

Overall, most of my concerns have been resolved.
I will raise my score to 5.

One of the remaining issues (which could be important) is:
if the semantic direction means white to red, what happens if input image does not have anything related to white and red, e.g., black and blue?
Dear Reviewer hJrH,

We have worked diligently to resolve your concerns.

As the rebuttal period comes to an end, please do not hesitate to contact us if you have any last-minute questions or require further clarification.

Best regards,

The Authors
Dear Reviewers and ACs,

Thank you again for your comprehensive review and valuable suggestions for improvement. We have carefully considered your constructive feedback, and provided detailed responses. With the discussion period concluding in less than two days, we kindly request you to read our rebuttal and we’d appreciate your prompt responses. We are hopeful that our rebuttal has effectively addressed your concerns and you can reconsider your score. If you need any further clarification, we are willing to provide any additional information to ensure your confidence in making responses.

Sincerely,

Authors
Thank you for all the valuable suggestions and for recognizing the value of our work! We will expand the discussion on global editing and transferability in conditional diffusion models and will discuss BoundaryDiffusion in the paper. We have carefully read through BoundaryDiffusion, which is an interesting supervised one-step global edit method. BoundaryDiffusion uses linear SVMs to classify image latent by hyperplanes using label annotations, and about 100 images are required for each class. The edit intuitively moves the image latent in both $\epsilon_t$ and $h_t$ spaces to the negative or positive side of the hyperplane, so that the corresponding semantic is reduced or enhanced. Although the linear and one-step edit properties are similar, our method is (a) localized, (b) requires no label annotation, (c) can find transferable directions using a single image, (d) requires edits only in $x_t$ space, and (e) has a theoretical basis to support the approach. Thanks for the constructive feedback, we will make sure all of them are reflected in our final manuscript.
Thank you for the detailed response. I have read through the global rebuttal and PDF, as well as other responses. Overall, I am satisfied with the responses. I suggest that the authors include the limitations of global editing and the difficulty of transferring edits in conditional diffusion models. After going through this paper again and revisiting other papers, I think there are some similarities to [1], such as linear properties of intermediate noise space, and the one-step editing. I suggest that the authors include this paper in their work and discuss it. I keep my overall score as is. 

[1] "Boundary Guided Learning Free Semantic Control with Diffusion Models." Zhu et al. 2023.