Exploring Low-Dimensional Subspace in Diffusion Models for Controllable Image Editing
Authors: Siyi Chen
Huijie Zhang
Minzhe Guo
Yifu Lu
Peng Wang
Qing Qu
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion model
precise image manipulation
low-rank
Primary area: diffusion_based_models

Abstract: Recently, diffusion models have emerged as a powerful class of generative models. 
Despite their success, there is still limited understanding of their semantic spaces. This makes it challenging to achieve precise and disentangled image generation without additional training, especially in an unsupervised way. 
In this work, we improve the understanding of their semantic spaces from intriguing observations: among a certain range of noise levels, (1) the learned posterior mean predictor (PMP) in the diffusion model is locally linear, and (2) the singular vectors of its Jacobian lie in low-dimensional semantic subspaces. We provide a solid theoretical basis to justify the linearity and low-rankness in the PMP. These insights allow us to propose an unsupervised, single-step, training-free **LO**w-rank **CO**ntrollable image editing (LOCO Edit) method for precise local editing in diffusion models. LOCO Edit identified editing directions with nice properties: homogeneity, transferability, composability, and linearity. These properties of LOCO Edit benefit greatly from the low-dimensional semantic subspace.
Our method can further be extended to unsupervised or text-supervised editing in various text-to-image diffusion models (T-LOCO Edit). Finally, extensive empirical experiments demonstrate the effectiveness and efficiency of LOCO Edit. The code and the arXiv version can be found on the [project website](https://chicychen.github.io/LOCO).

Exploring Low-Dimensional Subspaces in Diffusion Models for Controllable Image Editing
Siyi Chen
Huijie Zhang'*
Minzhe Guo
Yifu Lu
Peng Wang'
Qing Qu
University of Michigan
(siyich,huile huijiezh, vincegmz, pengwa, qingqu) edu
Abstract
Recently, diffusion models have emerged powerful class of generative models. Despite their success, there is still limited understanding of their semantic spaces. This makes it challenging to achieve precise and disentangled image generation without additional training, especially in an unsupervised way. In this work, we improve the understanding of their semantic spaces from intriguing observations: among a certain range of noise levels, (1) the learned posterior mean predictor (PMP) in the diffusion model locally linear, and (2) the singular vectors of its Jacobian lie in low- oow-dimensional semantic subspaces. We provide a solid theoretical basis to justify the linearity and low-ram wowranknesss the PMP. These insights allow us propose an unsupervised, single-step, training-free LOw-rank COntrollable image editing (LOOO Edit) method for precise local editing diffusion models. LOCO Edit identified editing directions with nice properties: homogeneity, transferability, composability, and linearity. These properties of LOCOO Edit benefit greatly from the now-dimensional semantic subspace. Our method can further be extended unsupervised or text-supervised editing in various text-to-image diffusion models (T-LOCO Edit). Finally, extensive empirical experiments demonstrate the effective- ness and efficiency of LOCO Edit. The code and the arXiv version can be found the project website."
Introduction
Recently, diffusion models have emerged as powerful new family of deep generative models with remarkable performance in many applications such as image generation across various domains [1, 4, 5, 6), audio synthesis [7, 8), solving inverse problem [9, 10, 11, 12, 13, 14], and video generation [15, 16, 17J. For example, recent advances in Al-based image generation, revolutionized by diffusion models such Dalle-2 [18], Imagen [19]. and stable diffusion [4], have taken the world of "AI Art generation", enabling the generation of images directly from descriptive text inputs. These models corrupt images by adding noise through multiple steps of forward process and then generate samples by progressive denoising through multiple steps of the reverse generative process.
Although modern diffusion models are capable of generating photorealistic images from text prompts, manipulating the generated content by diffusion models in practice has remaining challenges. Unlike generative adversarial networks [20], the understanding of semantic spaces in diffusion models is still limited. Thus, achieving disentangled and localized control over content generation by direct manipulation of the semantic spaces remains difficult task for diffusion models. Although effective, some existing editing methods in diffusion models often demand additional training procedures and are limited to global control of content generation [21, 22, 23]. Some methods are training- free or localized but are still based upon heuristics, lacking clear mathematical interpretations, or for text- supervisdd editing only [24, 25, 26, 27, 28]. Others provide analysis in diffusion models [29, 30, 31, 32, 33], but also have difficulty in local edits such as hair color.
hntops://chiccchen -iigtgtthhh 1o/LOCO
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Mouth shape
Hair curvature
Hair amount
shape
(a) Precise and localized image editing.
Original
Transfer (other)
Original 0.5 ................................
Transfer 0.8
(b) Homogeneity and transferability of the editing direction.
eye smile
smile hair color
Open
Close mouth
(c) Composability of disentangled directions.
(d) Linearity in the editing direction.
Figure 1: LOCO Edit. (a) The proposed method can perform precise localized editing in the region of interest. The editing direction is (b) homogeneous, (c) composable, and (d) linear.
In this study, we address the above problem by studying the low-rank semantic subspaces in diffusion models and proposing the LOw-rank COntrollable edit (LOCO Edit) approach. LOCO is the first local editing method that is single-step, training-free, requiring no text supervision, and having other intriguing properties (see Figure! for illustration). Our method is highly intuitive and theoretically grounded, originating from simple while intriguing observation in the learned posterior mean predictor (PMP) in diffusion models: for large portion of denoising time steps,
The PMP is locally linear mapping between the noise image and the estimated clean image, and the singular vectors of its Jacobian reside within low-dimensional subspaces.
The empirical evidence in Figure consistently shows that this phenomenon occurs when training diffusion models using different network architectures on range of real-world image datasets. Theoretically, we validated this observation by assuming a mixture low-rank Gaussian distributions for the data. We then prove the local linearity of the PMP, the low-rank nature of its Jacobian, and that the singular vectors the Jacobian span the low-dimensional subspaces.
By utilizing the linearity of the PMP, we can edit within the singular vector subspace of its Jacobian to achieve linear control of the image content with label or text supervision. The editing direction can be efficiently computed using the generalized power method (GPM) [30, 34). Furthermore, we can manipulate specific regions of interest in the image along a disentangled direction through efficient nullspace projection, taking advantage of the low-rank properties of the Jacobian.
Benefits of LOCO Edit. Compared to existing editing methods (e.g., [29, 35, 23, 24)) based on diffusion models, the proposed LOCO Edit offers several benefits that we highlight below:
Precise, single-step, training- and unsupervised editing, LOCO enables precise localized editing (Figure la) single timestep without any training. Further, it requires no text supervision based on CLIP [36]. thus integrating no intrinsic biases or flaws from CLIP [37]. LOCO is applicable various diffusion models and datasets (Figure 5).
Linear, transferable, and composable editing directions. The identified editing direction is linear, meaning that changes along this direction produce proportional changes semantic feature in the image space (Figure 1d). These editing directions homogeneous and can be transferred across various images and noise levels (Figure 1b). Moreover, combining disentangled editing directions leads to simultaneous semantic changes in the respective region, while maintaining consistency in other areas (Figure Ic).
An intuitive and theoretically grounded approach. Unlike previous works, by leveraging the local linearity of the PMP and the low-rankness of its Jacobian, our method is highly interpretable. The identified properties are well supported by both our empirical observation (Figure 2) and theoretical justifications in Section 4.
Morcover, LOCO Edit is generalizable to T-LOCO Edit for T2I diffusion models including DeepFloyd IF [19], Stable Diffusion [4], and Latent Consistency Models [38], with or without text supervision (Figure 4). A more detailed discussion on the relationship with prior arts can be found in Appendix B.

---

Notations. Throughout the paper, we use X S R to denote the noise- coise-corrupted image space at the time-step t 10, 1]. In particular, X, denotes the clean image space with distribution Panta (x). and 20 E Xo denote an image. No,, denote the posterior mean space time-step & (0, 1]. Here, S denotes a unit hypersphere in R4, and rt) :- {Z Rdxr
z } denotes the Stiefel manifold. rank A) denotes the numerical rank of A. I P (e)) [To [xr] denotes the posterior mean and is written as [zz)) rango(A) denotes the span the columns of A. mull(A) denotes the set of solutions to Ax = 0. projmull a (æ) denotes the projection of onto mull(A).
2 Preliminaries on Diffusion Models
In this section, we start by reviewing the basics of diffusion models [1, 2, 39), followed by several key techniques that will be used in our approach, such as Denoising Diffusion Implicit Models (DDIM) [3] and its inversion [40], T2I diffusion model, and classifier-fres guidance [41].
Basics of Diffusion Models. In general, diffusion models consist of two processes:
The forward diffusion process. The forward process progressively perturbs the original data 2o to noisy sample o, for & E [0, 1] with the Gaussian noise. As in [1], this can be characterized by conditional Gaussian distribution Pe Pe(zylazz)
(a / Varagg (1 ax) La). Particularly, parameters {a+}i=o sastify: (i) ag 1, and thus pa = Pdata and (ii) 01
and thus p1 =N Iz).
The reverse sampling process. To generate new sample, previous works [1, 42, 43] have proposed various methods to approximate the reverse process of diffusion models. Typically, these methods involve estimating the noise 6, and removing the estimated noise from a; recursively to obtain an estimate of To. Specifically, the sampling step from to -t-At with a small At > 0 can be described as:
CrE0 (Tt, + V1 04 01-11
Pt-An At V 01-Ai
(1)
Va
where c0(2), () is parameterized by a neural network and trained to predict the noise at time I.
Denoiser and Posterior Mean Predictor (PMP). According to [1]. the denoiser c0(2,, t) is
optimized by solving the following problem:
min & (0) == 1E (eop
[ () <
where O denotes the network parameters of the denoiser. Once Eg well trained, recent studies [44, 45] show that the posterior mean [[(zo|< i.e., predicted clean image at time t, can be estimated as follows:
io,: fe,((x;)) = VI (
Vaz
Here, (a, t) denotes the posterior mean predictor (PMP) [45, 44), and w No. denotes the estimated posterior mean output from PMP given æn and as the input. For simplicity, we denote fo.((we; t) as (x:)
DDIM and DDIM Inversion. Given a noisy sample 2, at time I, DDIM [3] can generate clean images by multiple denoising steps. Given a clean sample vo. DDIM inversion [3] can generate a noisy æ; at time by adding multiple steps of noise following the reversed trajectory of DDIM. DDIM inversion has been widely in image editing methods [40, 46, 29, 35, 47, 26] to obtain 2; given the original To and then performing editing starting from Fe. In our work, after getting given wo via DDIM inversion, we edit æ; to x only at the single time step with the help of PMP, and then utilize DDIM to generate the edited image 2'
For ease of exposition, for any t and to with to t1, denote DDIM operator and its inversion as 2t, = DDIM( f1) and It,- = MOIM-Inv(2,,, tz).
Text-to-image (T2I) Diffusion Models & Classifier-Free Guidance. So far, our discussion has only focused on unconditional diffusion models. Moreover, our approach can be generalized from unconditional diffusion models to T2I diffusion models [38, 4, 48, 19], where the latter enables controllable image generation 2 guided by a text prompt e. more detail, when training T21 diffusion models, we optimize conditional denoising function e e, t).. For sampling,
3

---

CIFAR-10-DOPM
CeHbA-UAVIT
DeepPlayd-FR Saiioddd---PPrmppp P-rompp--
ImageNeetUAAi
10a
Cosine similarity Norm ratio
1,000
0,077
Rank rabio
10"
0,050
0
16
20
301
40
1.000
0.975
10-2
0,000
0.0
0.2
0.1
0.5
0.5
10
(a) Low-rankness of the Jacobian
(b) I Local linearity of PMP
Figure 2: Low-rankness of the Jacobian (a,((x,) and Local linearity of the PMP fo.( (Zx). We evaluated DDPM (U-Net [49]) CIFAR-I0 dataset [50], U-VIT [51] (Transformer) on CelebA [52], ImageNet [53] datasets and DeepFloy IF [19] trained on LAION-5B [54] dataset. (a) The rank ratio of Jo,, (xx) against timestep f. (b) The norm ratio (Top) and cosine similarity (Bottom) between fo. + >Ax) and lo (z;; ^Az) against step size at timestep = 0.7.
employ a technique called classifier-fret guidance [41], which substitutes the unconditional denoiser c0(zi,t) t) in Equation (1) with its conditional counterpart ēØ (z,, t, c) that can be described follows:
Eg Ea t, c) = E (v,, t, Ø) + nT(e (z;, E, c) Eg (z, t, Ø))
Here, Ø denotes the empty prompt and > 0 denotes the strength for the classifier-fre guidance.
3 Exploring Linearity & Low-Dimensionality for Image Editting
In this section, we formally introduce the identified low-rank subspace in diffusion models and the proposed LOCO Edit method with the underlying intuitions. Section 3, we present the benign properties in PMP that our method utilizes. Followed by this, in Section 3.3 we provide detailed description of our method.
3.1 Local Linearity and Intrinsic Now-Dimensionalit in PMP
First, let delve into the key intuitions behind the proposed LOCO Edit method, which lie in the benign properties of the PMP (xt)) At one given timestep & E [0, 1]. let us consider the first-order Taylor expansion of fa. AAz) at the point 2;;
Le (2q; AAx) = f0.) + (x)) Ax,
where Av E Sd-1 1 is perturbation direction with unit length, X E the perturbation strength, and (x:) Va, fo, (o, is the Jacobian of (xz)) Interestingly, discovered that within certain range of noise levels, the learned PMP fe.: exhibits local linearity, and the singular subspace of its Jacobian Jo.: is low rank. Notably, these properties are universal across various network architectures (e.g., UNet and Transformers) and datasets.
We measure the low-rankness with rank ratio and the local linearity with norm ratio and cosine similarity. Specifically, (i) rank ratio is the ratio of rank(Je, (x::)) and the ambient dimension d; (ii) norm ratio is the ratio of f. (xx, + AAx) ) la and || Adx) I|2; (iii) cosine similarity is between
fo, + >Ax) and to (Iz; AAx)) ). The detailed experiment settings provided in Appendix D.I,
and results are illustrated in Figure 2, from which we observe:
As shown in Figure 2(a), the rank ratio for t E [0, 1] consistently displays a U-shaped pattern across various network architectures and datasets: (i) is close to 1 near either the pure noise = 1 or the clean image t 0, (ii) Je. (xx;) is low-rank (i.e., rank ratio less than 10-1) for all diffusion models within the range E [0.2, 0.7], (iii) achieves the lowest value around mid-to-late timestep, slightly differs depending on architecture and dataset.
Low-renkness of the Jacobian Je,,
Local linearity of the PMP f0.( Moreover, the mapping fo. exhibits strong linearity across a large portion of the timesteps; see Figure 2(b) and Figure 10. Specifically, in Figure 2(b), we evaluate the linearity of (xr, at = 0.7 where the rank ratio is close to the lowest value. We can see that f + AAz) lo (mr; >Ax) even when 40, which is consistently true among different architectures trained on different datasets.

---

In addition to comprehensive experimental studies, we will also demonstrate in Section 4 that both properties can be theoretically justified.
3.2 Key Intuitions for Our Image Editing Method
The two benign properties offer valuable insights for image editing with precise control. Here, we first present the high-level intuitions behind our method, with further details postponed to Section 3.3. Specifically, for any given time-step & E [0, 11, let denote the compact singular value decomposition (SVD) of the Jacobian (o,) as
Jo, = UEV = Σ o.u.v
(5)
where r is the rank of ( U [u] U E St r) and V = [o Jr St(d, r) denote the left and right singular vectors, and Σ diag(d).... σ,) denote the singular values. We write (x,)
Jø.r short for specific Iy, and denote range(J span(V) and noll(Jo.:) = (w Jo = 0}.
Local linearity of PMP for one-step, training-free and supervision-free editing. Given the PMP (x+) is locally linear the t-th timestep, if we perturb æ, by Ax Xu; using one right singular vector v. of (Zx) as an example editing direction, then by orthogonality
fe.((1, + Xu,) ~ fa.( + (x,)(
fa,) + tayu = Lo.t + P:u.
(6)
This implies we can achieve one-step editing along the semantic direction u, Notably, the method is training- faining-free and supervision-free since v; be simply found via the SVD of (
Local linearity of PMP for linear, homogeneous, and composable image editing, (i) First, the editing direction v v; is linear, where any linear À E I. change along v, results in linear for the edited image. (ii) Second, the editing direction v
change Pi = λo, along homogeneous due its independence of &o.c. where could be applied on any images from the same data distribution and results in the same semantic editing. (iii) Third, editing directions are
is
composable. Any linearly combined editing direction v = =Eiez X E range (Jg.) is a valid editing direction which would result in a composable change Σiez Pilk; in the edited image. On the contrary, E null (Je.)) results in no editing since fa. + Xw) fa,)(w.).
Low-rankness of Jacobian for localized and efficient editing. + ) is for the entire predicted clean image, thus Jo, (zx) finds editing directions in the entire image. Denote Jo, the Jacobian only for certain region of interest (ROI), and Jo.: the Jacobian for regions outside ROL Similarly, E range
gott can edit mainly regions within the ROI, and null J contain directions that
do not edit regions outside of ROL. Further projection of onto null (Jo.) can result in more localized editing direction for ROI. To perform such nullspace projection, computing the full SVD can be very expensive. But we can highly reduce the computation by the low-rank estimation of Jacobians with rank r' < d. The estimation is efficient yet effective with / E [0.5,0.a when the rank of the Jacobian achieves the lowest value.
3.3 Low-rank Controllable Image Editing Method with Nullspace Projection
In this subsection, we provide detailed introduction to LOCO Edit, expanding on the discussion in Section 3. We first introduce the supervision-Fe LOCO Edit, where we further enable localized image editing through nullspace projection with masks. Second, we generalize to T-LOCO Edit for T21 diffusion models w/wo text-supervision to define the semantic editing directions.
LOCO Edit. We first introduce the general pipeline LOCO Edit. As illustrated in Figure 3, given DDIM-Inv(aD)) f) to generate noisy image wt, In particular, we choose E [0.5, 0.7] so that the PMP fa. (xx) is locally linear and its Jacobian Ja,( is close to its lowest rank. From Section 3.1, we know that we can edit the image by changing x² = α, + Xup. where Un is the identified editing direction. After editing Ty to x' we use x DDIM (æ), 0) to generate the edited image.
an original image Lo. we first use
In many practical applications, often need to edit only specific local regions of an image while leaving the rest unchanged. As discussed in Section 3.2, we can achieve this task by finding a precise local editing direction with localized Jacobians and nullspace projection. Overall, the complete method is in Algorithm 1. We describe the key details as follows.
5

---

DDIM-Inv
mull( Jo.))
Jas
fox Tos
Up
DDIM
22 + Xup
F0,4
Joz
① rangel (Tag)
Bng
Por
Up Je
Find Up
Figure 3: Illustration of the unsupervised LOCO Edit for unconditional diffusion models. Given an image To. we perform DDIM- Inv until time to get æt, and estimate uo from 2t. After masking to get the region of interest (ROI) T and its counterparts To,t. we find the edit direction Up via SVD and nullspace projection based on their Jacobians (Algorithm 1). By denoising æ; + A an image a' with localized editing is generated. In this paper, the variables and notions related to ROI, nullspace, and final direction are respectively highlighted by green, blue, and red colors.
Algorithm 1 Unsupervised LOCO Edit
1: Input: original image Vo. the mask Zt pretrained diffusion model cæ, editing strength À, semantic index k. number of semantic directions r,, editing timestep E [0.5, 0.7]. rank 5.
2: Output: edited image wh.
3: Generate me < DDIM- Inv (zo. t)
b noisy image at 1-th timestep
4: Compute the top-r SVD ÜŪ Σ V of Jo. = V Pa(fe.
5: Compute the top-r' SVD (U, Σ, VJ of Jo.: = Va, Poo (fo, (x..)
6: Pick direction + v/.
Pick the hthh singular vector for the editing direction
7: Compute Up < (I VV 1
D Nullspace projection for editing within the mask ? Normalize the editing direction & Editing with forward DDIM along the direction up
8: Up f- upp up/l|uppl
9: Return: wg < DDIM(#; + AMp, 0)
Finding localized Jacobians via masking. To enable local editing, we mask Q (i.e., an index set of pixels) to select the region of interest," with Pa(.) denoting the projection onto the index set Ω. For picking a local editing direction, caleulate the Jacobian of fa.: restricted the region of interest, Jo.. = aa (fo, (x+) = Ü.VT and select the localized editing direction v from the top-r singular vectors of V (e.g., u = Vi. k] E range Jj. for some index k E [r]). In practice, top-r rank estimation for V is calculated through the generalized power method (GPM) Algorithm 2 with r= 5 to improve efficiency.
Better semantic disentanglement via nullspace projection. However, the projection P-(() introduces extra nonlinearity into the mapping (fo causing the identified direction have semantic entanglements with the area Ω2 outside of the mask. Here, Ω2 denotes the complimentary set of f. To address this issue, we can use the nullspace projection method [56, 57]. Specifically, given Jo.: Vz, Pae (x))( U.VT, nullspace projection projects onto
mill (Jo.)) The projection can be computed Up (Jo.)
(I VV yo that the modified op does not change the image in Ωº In practice, we calculate a top-r' rank estimation for V through the generalized power method (GPM) Algorithm 2 with r' 5.
T-LOCO Edit. The unsupervised edit method can be seamlessly applied to T2I diffusion models with classifier-fre frree guidance (3) (Algorithm 3). Besides, can further enable text-supervised image editing with an editing prompt (Algorithm 4). See results in Figure 4(a). This is useful because the additional text prompt allows to enforce specified editing direction that cannot be found casily in the semantic subspace of the vanilla Jacobian Jo.t- As illustrated in Figure 4(b), this includes adding glasses or changing the curly hair of human face. For simplicity, introduce the key ideas of text-super T-LOCO Edit based upon DeepFloyd IF [19]. Similar procedures are also generalized Stable Diffusion and Latent Consistency Models with additional decoding step [4, 38]. We discuss the key intuition below, see Appendix E.2 and Appendix E.3 for method details.
For datasets that have predefined masks, we can use them directly. For other datasets that lack predefined masks well generate images, utilize Segment Anything (SAM) to generate masks [55].

---

Stable Diffusion
DeepFloyd
Latent Consistency
up
Remove beard
mask
Side
(a) Unsupervised T21 Edit
mask
+ "glasses"
mask
"glasses"
mask
"curty heir"
(b) Text-supervised T21 Edit
Figure 4: T-LOCO Edit T2I diffusion models. (a) Unsupervised editing direction is found only
via the given mask without editing prompt. (b) Text-supervised editing direction is found with both mask and an editing prompt such as "with glasses". Experiment details can be found in Appendix G.3.
We first introduce some notations. Let co denote the original prompt, and Ce denote the editing prompt. For example, in Figure 4(b), ca be "portrait of man", while ce can be "portrait of man with glasses". Correspondingly, given the noisy image D for the clean image 2o generated with (2t)
Cos let (xt) and (axe) be the estimated posterior mean and its Jacobian conditioned the original prompt ca. and let (ax.) and J (ax:) be the estimated posterior mean and its Jacobian conditioned on both the editing prompt Ce and Cg.
According to the classifier-free guidance (3), we can estimate the difference of estimated posterior means caused by the editing prompt as d = fg (xx) f (3te. and then set (z,(())))) as an initial estimator of the editing direction." Based upon this, enable localized editing, similar the unsupervised case, we can apply masks Q2 to select ROI in d and calculate localized Jacobian to get v. After that, similarly, we can perform nullspace projection of for better disentanglement get the final editing direction Up..
Justification of Local Linearity, Low-rankness, & Semantic Direction
In this section, we provide theoretical justification for the benign properties in Section 3.1. First, we assume that the image distribution Pdata follows mixture of low-rank Gaussians defined as follows. Assumption 1. The data ao E I2d generated distribution Paato lies on a union of K subspaces. The basis of each subspace {M, E St(d, (t) are orthogonal to each other with M, M, 0 for all $ S K, and the subspace dimension Tt is much smaller than the ambient dimension Moreover, for each k E [K] Do follows degenerated Gaussian with P (mo Mga;) = 1/K, as ~ N(O., Without loss of generality, suppose D, is from the h-th class, that I Vai,0 VI- are where æo (n.) i.e. Wo = Mgaas- Both || | 22 bounded.
Our data assumption motivated by the intrinsic oow-dimensionality of real-world image dataset Additiolll,.l,lly Wang et al. [59] demonstrated that images generated by analytical score function derived from a mixture of Gaussians distribution exhibit conceptual similarities to those produced by practically trained diffusion models. Given that (x:.)) is estimator of the posterior mean El 22). we show that the posterior mean E| [+ can analytically derived as follows.
Lemma 1. Under Assumption 1, for t (0, 1). the posterior mean is
Σ exp
at
M xell3)
M,M x
2 a1))
E [xpllaee] Vou
ΣK K
exp
2 a1)
0t:
|M 2:.1?
Lemma shows that the posterior mean E [ could be viewed as convex combination of M,,, Zt, i.e. I; projected onto each subspace Mj. This lemma leads to the following theorem:
'The idea is to identif'y the editing direction in the X, space based on changes in the estimated posterior caused by the editing prompt. More details provided in Appendix E.3.
7

---

Theorem 1. Based upon Assumption 1, we can show the following three properties for the posterior
mean B(zo|
K rh all t E (0,
The Jacobian of posterior mean satisfies rank (Vx, (200) S rr:- = Σ
The posterior mean EE [as] has local linearity such that
|IB [2o xx. + AAæ] E [xo] - AV B Blaoo/ee Axll = (1 (4.)
at
O
(8)
where Ax 1 and À IR is the step size.
V a. E/oo xi) is symmetric and the full SVD of V ((zuloo could be written as Vw, 1/20 [1
U. V, where U. [u,, 2,,,,-... Hr,al E St(d, d), E, = ding(or.1.... Otr...... 0)
with 0r,1 2
and Vi [U U2,2,.... vr,al St(d, d). ttr and M = [M, MR]. It holds that
Let U, = [u,, 2
Ut.r
lim (U,
M
0.
The proof is deferred to Appendix F. Admittedly, there are gap between our theory and practice, such as the approximation error between fo.)(2;) and E. [2o aaa]. assumptions about the data distribution, and the high rankness of Jo.: for t < 0.2 and t > 0.9 in Figure 2. Nonetheless, Theorem 1 largely supports our empirical observation in Section that we discuss below:
Low-rankness of the Jacobian. The first property in Theorem demonstrates that the rank of V a, E (zo xx] is always greater than the intrinsic dimension of the data distribution. Given that the intrinsic dimension of the real data distribution is usually much lower than the ambient dimension [58], the rank of Jo., on the real dataset should also be low. The results align with empirical observations in Figure when E [0.2, 0.7].
Linearity of the posterior mean. The second property in Theorem 1 shows that the linear approxi- mation error is within the order of (1 ax)) (()). This implies that when approaches 1, a1/(1 ( (<) becomes small, resulting in a small approximation error even for large A. Empirically, Figure 2 shows that the linear approximation error of fe, (xx)) is small when = 0.7 and À 40, whereas Figure 10 shows a much larger error for 0.0 under the same A. These observations
align well with theory.
Low-dimensional semantic subspace. The third property in Theorem shows that, when / is close I, left singular vectors associated with the top-r singular values form the basis of the image distribution. Since the editing direction consists of basis, the edited image remains within the image distribution. This explains why u, found in Equation (6) a semantic direction for image editing.
5 Experiments
In this section, we perform extensive experiments to demonstrate the effectiveness and efficiency of LOCO Edit. We first showcase LOCO Edit has strong localized editing ability across variety of datasets Section 1. Moreover, we conduct comprehensive comparisons with other methods to show the superiority of the LOOCO Edit method in Section 5.2. Besides, we provide ablation studies on multiple components in our method in Appendix C. and analyze the editing directions in Appendix C.2, with extra experimental details postponed to Appendix G.
5.1 Demonstration on Localized Editing and Other Benign Properties
First, we demonstrate benign properties of LOCO Edit in Algorithm on a variety of datasets, including LSUN-Church [60], Flower [61], AFHQ [62], CelebA-HQ [52], and FFHQ [63].
As shown in Figure and Figure 1a, our method enables editing specific localized regions such as eye size/focus, hair curvature, length/amount, and architecture, while preserving the consistency of other regions. Besides the ability of precise local editing, Figure demonstrates the benign properties of the identified editing directions and verify our analysis in Section 4:
Linearity, As shown Figure (dd), the semantic editing can be strengthened through larger editing scales and can be flipped by negating the scale.
Homogeneity and transferability. As shown Figure 1(b), the discovered editing direction can be transferred across samples and timesteps in Xt
8

---

FFIQ
Iorrh shrew
Fvse Iocus
Tyehmwammout amsunt
Par amount
doda
Caebi
Fyn shaps
Hair length
Mouth shaaps
Harr curveture
Lip thickrwss
AFTIQ
Backglooadd
Samiie
Iongua 207
Fye hagiht
Hight eer coler
Flower
Closed vs. ocen
Color
ammount
MedSoee
Eys locus
Hall amount
Mouth
Het stya
Eye shrpe
Church
Tower Tnigght
Archteciure
Tower archiosture
Window R20
Gor shapo
Figure 5: Benchmarking LOCO Edit across various datasets. For each group of three images, in the center is the original image, and on the left and right are edited images along the negative and the positive directions accordingly.
Composability. As shown Figure (c)), the identified disentangled editing directions in the low-rank subspace allow direct composition without influencing each other.
5.2
Comprehensive Comparison with Other Image Editing Methods
We compare LOCO Edit with several notable and recent image editing techniques, including Asyrp [29], Pullback [30], NoiseCLR [23], and BlendedDifusion [24]. We also compare with an unexplored method using the Jacobians de
dæ. to find the editing direction, named as 0æ:
de:
Metrics. We evaluate our method using the below metrics and summarize the results in Table 1. Besides the image generation quality, we also compared other attributes such the local edit ability, efficiency, the requirement for supervision, and theoretical justifications.
Local Edit Success Rate evaluates whether the editing successfully changes the target semantics and preserves unrelated regions by human evaluators.
LPIPS [64] and SSIM [65] measure the consistency between edited and original images.
Transfer Success Rate measures whether the editing transferred other images successfully
changes the target semantics and preserves unrelated regions by human evaluators.
Learning time to measure the time required to identify the edit directions.
Transfer Edit Time measure the time required to transfer the editing other images directly.
#lmages for Learning measures the number of images used find the editing directions.
One-step Edit, No Additional Supervision, Theoretically Grounded, and Localized Edit are attributes of the editing methods, where each them measures specific property for the method.
Moreover, we visualize the editing results on non-cherry-picke images in Figure The detailed evaluation settings are provided in Appendix G.2.
Benefits of Our Method. Based upon the qualitative and quantitative comparisons, our method shows several clear advantages that summarize follows.
Superior local edit ability with one-step edit. Table shows LOCO Edit achieves the best Local Edit Success Rate. Such local edit ability only requires one-step edit at specific time t. For LPIPS and SSIM, our method performs better than most methods worse than BlendedDiffusion. However, BlendedDiffusion sometimes fails the edit within the masks (as visualized in Figure 6, rows 1, 3, 4, and 5). Other methods find semantic direction more globally, leading to worse performance in Local Edit Success Rate, LPIPS, and SSIM for localized edits.

---

Method Name
Pullback
der/
NoiseCLR
Asyrp
BlendedDiffusion
LOCO (Ours)
Local Edit Success Rate
0.32
0.37
0.32
0.47
0.55
0.80
LPIPSL
0.16
0.13
0.14
0.22
0.03
0.08
SSIMT
0.60
0.66
0.68
0.68
0.94
0.71
Transfer Success Ratet
0.14
0.24
0,66
0.58
Can't Transfer
0.91
Transfer Edit Timel
4s
28
5s
3s
Can't Transfer
2s
#lmages for Learning
1
1
100
100
Learning Time
8s
44s
day
475s
120s
One-step Edit?
X
X
No Additional Supervision?
V
X
X
Theoretically Grounded?
X
X
X
X
Localized Edit?
X
Table Comparisons with existing methods. Our LOCO Edit excels in localized editing, transfer- ability and efficiency, with other intriguing properties such as one-step edit, supervision-free, and theoretically grounded.
Oridin
NaiseCLR
Asyrp
BlendedDiifesion
LOCO (Ours)
Origin
NotseCL
Asyrp
Blendeliifiiin
1CC (CC0((
Add durker syebron
Add red lppstik
Figure 6: Compare local edit ability with other works on non-cherry-picker images. LOCO has consistent and accurate local edit ability, while other methods have wrong, global, or no edits.
Transferability and efficiency. First, LOCO Edit requires less learning time than most of the other methods and requires learning only for a single time step with single image. Moreover, LOCO Edit is highly transferable, having the highest Transfer Success Rate in Table A. In contrast, BlendedDiffusion cannot transfer and requires optimization for each individual image. NoiseCLR has the second-best yet lower transfer success rate, while other methods exhibit worse transferability.
Theoretically-gronndee and supervision-free. LOCO Edit theoretically grounded. Besides. it is supervision-free. thus integrating no biases from other modules such as CLIP [36]. [37] shows CLIP sometimes can't 't capture detailed semantics such as color. We can observe failures in capturing detailed semantics for methods that utilize CLIP guidance such as BlendedDiffusion and Asyrp in Figure 6, where there are no edits or wrong edits.
Conclusion
We proposed new low-rank controllable image editing method, LOCO Edit, which enables precise, one-step, localized editing using diffusion models. Our approach stems from the discovery of the locally linear posterior mean estimator in diffusion models and the identification of a low- dimensional semantic subspace its Jacobian, theoretically verified under certain data assumptions. The identified editing directions possess several beneficial properties, such as linearity, homogeneity, and composability. Additionally, our method versatile across different datasets and models and is applicable to text-supervised editing in T2I diffusion models. Through various experiments, we demonstrate the superiority of our method compared existing approaches.
10

---

Acknowledgement
We acknowledge support from NSF CAREER CCF-2143904, NSF CCF-2212066. NSF CCF- 2212326, NSF IIS 2312842, NSF IIS 2402950, ONR N00014-22-1-2529, a gift grant from KLA, an Amazon AWS AI Award, MICDE Catalyst Grant. The authors acknowledge valuable discussions with Mr. Zekai Zhang (U. Michigan), Dr. Ismail R. Alkhouri (U. Michigan and MSU), Mr. Jinfan Zhou (U. Michigan), and Mr. Xiao Li (U. Michigan).
References
[1] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:8840-8888
2020. 2020.
[2] Yang Song, Jascha Sohl- Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poolc. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[3] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[4] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pottern Recognition, pages 10684-10695, 2022.
[5] Huijic Zhang, Yifu Lu, Ismail Alkhouri, Saiprasad Ravishankar, Dogyoon Song, and Qing Qu. Improving training efficiency of diffusion models via multi-stage framework and tailored multi-decader architectures. In Conference on Computer Vision and Pattern Recognition 2024, 2024.
[6] Ismail Alkhouri, Shijun Liang, Rongrong Wang, Qing Qu, and Saiprasad Ravishankar. Diffusion-based adversarial purification for robust deep mri reconstruction. ArXiv preprint urXiv:2309.05794 2023.
[7] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. International Conference on Learning Representations, 2021.
[8] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2021.
[9] Hyungjin Chung, Bycongsu Sim, Dohoon Ryu, and Jong Chul Yc. Improving diffusion models for inverse problems using manifold constraints. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors. Advances in Neurul Information Processing Systems, 2022.
[10] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guudde diffusion models for inverse problems. In International Conference on Learning Representations, 2023.
[11] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, 2023.
[12] Xiang Li, Soo Min Kwon, Ismail R Alkhouri, Saiprasad Ravishanka, and Qing Qu Decoupled data consistency with diffusion purification for image restoration. ArXiv preprint arXiv:2403.0603 2024.
[13] Ismail Alkhouri, Shijun Liang, Rongrong Wang. Qing Qu, and Saiprasad Ravishankar. Robust physics- based deep mri reconstruction via diffusion purification. In Conference on Parsimony and Learning (Recent Spotlight Track), 2023.
[14] Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models via hard data consistency. In The Twelfth International Conference on Learning Representations, 2024.
[15] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 18456-18466, 2023.
[16] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. ArXiv preprint arXiv:2311.15127, 2023.
11

---

[17] Levon Khachatryan. Andranik Movsisyan. Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Texi2video-zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 55954-15964 2023.
[18] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. ArXiv preprint arXiv:2204.06122. 2022.
[19] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorcalistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36694, 2022.
[20] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4396-4405, 2018.
[21] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[22] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning texl-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 22500-22510, 2023.
[23] Yusuf Dalva and Pinar Yanardag. Noiseeir: A contrastive learning approach for unsupervised discovery of interpretable directions in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 24209-24218. 2024.
[24] Omri Avrahami. Dani Lischinski. Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18208-18218, June 2022.
[25] Theodoros Kouzelis, Manos Plitsis, Mihalis A. Nicolaou, and Yannis Panagakis. Enabling local editing in diffusion models by joint and individual component analysis, 2024.
[26] Guillaume Couairon, Jakoh Verbeek, Holger Schwenk, and Matthicu Cord. Diffedit: Diffusion-basee semantic image editing with mask guidance. In The Eleventh International Conference on Learning Representations, 2023.
(27] Manuel Brack, Felix Friedrich, Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, and Kristian Kersting. SEGA: Instructing text-to to ext-to-image models using semantic guidance. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[28] Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, and Shiyu Chang. Uncovering the disentanglement capability in text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1900-1910, 2023.
[29] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. In The Eleventh International Conference on Learning Representations, 2023.
[30] Yong- Hyun Park, Mingi Kwon, Jacwoong Choi, Junghyo Jo, and Youngjung Uh. Understanding the latent space of diffusion models through the lens of riemannian geometry. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
[31] Ye Zhu, Yu Wu, Zhiwei Deng, Olga Russakovsky, and Yan Yan. Boundary guided learning- free semantic control with diffusion models. In Conference on Neural Information Processing Syxtems (NeurIPS), 2023.
[32] Hila Manor and Tomer Michaeli. Zero-shot unsupervised and text-based audio editing using DDPM inversion. Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors. Proceedings of the 41st International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pages 34603-3465 34629, PMLR, 21-27 Jul 2024.
[33] Hila Manor and Tomer Michaeli. On the posterior distribution in denoising: Application to uncertainty guantification. In The Twelfth International Conference Learning Representations, 2024.
[34] Yousef Saad. Numerical methods for large eigenvalue problems: revised edition. SIAM, 2011.
12

---

[35] Yong-Hyun Park, Mingi Kwon, Jaewoong Choi, Junghyo Jo. and Youngjung Uh. Understanding the latent space of diffusion models through the lens of riemannian geometry. Advances in Neural Information Processing Systems, 36:24129-244 241422442222200 2023.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Muchine Learning, pages 3748-8763. PMLR. 2021.
[37] Shengbang Tong. Zhuang Liu, Yuexiang Zhai, Yi Ma. Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal Ilms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568-9578, 2024.
[38] Simian Luo, Yigin Tan, Longbo Huang. Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. ArXiv preprint arXiv:2310.04378 2023.
[39] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion based generative models. Advances in Neural Information Processing Systems, 35:26565 26577, 2022.
[40] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038-60888 003-6047,22 2023.
[41] Jonathan Ho and Tim Salimans. Classifier- free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[42] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-575 2022.
[43] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion based generative models. Advances in Neural Information Processing Systems, 35:26565 26577, 2022.
[44] Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Peng Wang. Liyue Shen, and Qing Qu The emergence of reproducibility and consistency diffusion models. Forty-first International Conference on Machine Learning, 2024.
[45] Calvin Luo. Understanding diffusion models: A unified perspective. ArXiv preprint arXiv:2208.11970. 2022.
[46] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426-2435, 2022.
[47] René Haas, Inbar Huberman-Spiegelglaa Rotem Mulayoff, and Tomer Michaeli. Discovering interpretable directions in the semantic latent space of diffusion models. International Conference on Automatic Face and Gesture Recognition, abs/2303. 11033, 2024.
[48] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel. Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first Interna- tional Conference on Machine Learning, 2024.
[49] Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Conputer-assisted Intervention MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[50] Alex Krizhevsky, Geoffrey Hinton, et Learning multiple layers features from tiny images. 2009.
[51] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22669-226002 22679, 2023.
[52] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference Computer Vision, pages 3730-3738, 2015.
[53] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. leee, 2009.
13

---

[54] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-Sb: An open large-scale dataset for training next generation image-text models. Advances in Nearal Information Processing Systems, 35:25278 33:25278-25294, 2022.
[55] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yer Lo, Piotr Dollar, and Ross Girshick, Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015-4026, October 2023.
[56] S. Banerjee and A. Roy. Linear Algebru and Matrix Analyxis for Statistics. Chapman Hall/CRC Texts in Statistical Science. CRC Press, 2014.
[57] Jiapeng Zhu, Ruili Feng. Yujun Shen, Deli Zhao, Zhengjun Zha, Jingren Zhou, and Qifeng Chen, Low-rank subspaces in gans. In Neural Information Processing Systems, 2021.
[58] Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. In International Conference on Learning Representations, 2021.
[59] Binxu Wang and John Vastla.. The hidden linear structure score-based models and its application. ArXiv preprint arXiv:2311. 2023.
[60] Fisher Yu, Yinda Zhang. Shuran Song, Ari Seff, and Jianxiong Xiao, Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. ArXiv, abs/1506.033655 2015,
[61] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722 729, 2008.
[62] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8 8188-8197, 2020.
[63] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[64] Richard Zhang, Phillip Isola, Alexci Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a pereeptual metrie. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pages 586-595, 2018.
[65] Zhou Wang. Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEF Transactions on Image Processing, 13(4):600-612, 2004.
[66] Peng Wang, Huikang Liu, Druv Pai, Yaodong Yu, Zhihui Zhu, Qing Qu. and Yi Ma. A global geometric analysis of maximal coding rate reduction. In Forty-first International Conference on Machine Learning, 2024.
[67] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[68] Can Yaras, Peng Wang, Laura Balzano, and Qing Qu. Compressible dynamics in deep overparameterized low-rank learning & adaptation. In Forty-first International Conference on Machine Learning, 2024.
[69] Michael Fuest, Pingchuan Ma, Ming Gui, Johannes S Fischer, Vincent Tao Hu, and Bjom Ommer. Difffusion models and representation learning: A survey. ArXiv preprint arXiv:2407.00788 2024.
[70] Xiang Li, Yixiang Dai, and Qing Qu. Understanding generaiiziiy of diffusion models requires rethinking the hidden gaussian structure. 2024.
[71] Peng Wang, Xiao Li, Yaras Can, Zhihui Zhu, Laura Balzano, Wei Hu, and Qing Qu Understand- ing deep representation learning via layerwise feature compression and discrimination. ArXiv preprint 1xXv:23111.0996 2023.
[72] Siyi Chen, Minkyu Choi, Zesen Zhao, Kuan Han, Qing Qu, and Zhongming Liu. Unfolding videos dynamics via taylor expansion, 2024.
14

---

[73] Ayaan Haque, Matthew Tancik, Alexei Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct- nerf2nerf: Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[74] Shengyi Qian, Linyi Jin, Chris Rockwell, Siyi Chen, and David F. Fouhey. Understanding 3d object articulation internet videos. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1589-1599, 2022.
[75] Jinqi Luo, Tianjiao Ding. Kwan Ryan Chan, Darshan Thaker, Aditya Chattopadhyay, Chris Callison- Burch, and Rene Vidal. Pace: Parsimonious concept engineering for large language models. ArXiv preprint arXiv:2406.0m 04331, 2024.
[76] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Neural Information Processing Systems, 2014.
[77] René Haas, Inbar Huberman-Spiegelguae Spiegelglas, Rotem Mulayoff, and Tomer Michaeli, Discovering interpretable directions in the semantic latent space of diffusion models. International Conference on Automatic Face and Gesture Recognition, abs/2303. 11073, 2024.
[78] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-or. Prompt-to- prompt image editing with cross-attention control. In The Eleventh International Conference on Learning Representations, 2023.
[79] Qian Wang, Biao Zhang, Michael Birsak, and Peter Wonka. Instructedit: Improving automatic masks for diffusion-baas based image editing with user instructions. ArXiv, abs/2305. 18047, 2023.
[80] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. 202. IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18392-18402, 2022.
[81] Shanglin Li, Bo- Bo-Wen Zeng Yutang Feng, Sicheng Gao, Xuhui Liu, Jiaming Liu, Li Lin, Xu Tang. Yao Hu, Jianzhuang Liu, and Baochang Zhang. Zone: Zero-shot instruction-guider local editing. 2024 JEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[82] Bradley Efron. Tweedic's 's formula and selection bias. Journal of the American Statistical Association, 106(496):1602-1666 1614, 2011.
[83] A Wondhury Max. Inverting modified matrices. In Memorandum Rept. 42, Statistical Research Group, page 4. Princeton Univ., 1950.
[84] Chandler Davis and William Morton Kahan. The rotation of cigenvectors by perturbation. iii. SIAM Journal on Numerical Analysis, 7(1):144 1970.
[85] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc.
[86] Jooyoung Choi, Jungbeom Lee, Chachun Shin, Sungwon Kim, Hyunwoo J. Kim, and Sung-Hoon Yoon. Perception prioritized training of diffusion models. 2022 JEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1462-11471, 2022.
[87] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
15

---

Appendix
A Future Direction
We identify several future directions and limitations of the current work. The current theoretical framework explains mainly the unsupervised image editing part. A more solid and thorough analysis of text-supervised image editing is of significant importance in understanding T2I diffusion models, which is yet difficult open problem in the field. For example, there is still lack of geometric analysis of the relationship between subspaces under different text- eext-proppt conditions [4, 19, 38, 66]. Based on such understandings, it may be possible to further discover benign properties of editing directions in T21 diffusion models, or design more efficient fine-tuning [67, 68] accordingly. Besides, the current method has the potential to be extended for combining coarse fine editing across different time steps. Furthermore, it is worth exploring the direct manipulation of semantic spaces in flow matching diffusion models and current finding to image or video representation learning in diffusion models [69, 70, 71, 72], extend to 3D editing of pose or shape [73, 74], or utilize the low-rank structures to build dictionaries [75].
diffusion models. Lastly, it is possible connect the
B Discussion on Related Works
Study of Latent Semantic Space in Generative Models. Although diffusion models have demon- strated their strengths in state-of-the-ar image synthesis, the understanding of diffusion models is still far behind the other generative models such as Generative Adversarial Networks (GAN) [76, 57], the understanding of which can provide tools well as inspiration for the understanding of diffusion models. Some recent works have identified such gaps, discovered latent semantic spaces in diffusion models [29], and further studied the properties of the latent space from a geometrical perspective [30]. These prior arts deepen our understanding of the latent semantic space in diffusion models, and inspire later works to study the structures of information represented in diffusion models from various angles. However, their semantic space is constrained to diffusion models using UNet architecture, and can not represent localized semanties. Our work explores an alternative space to study the semantic expression in diffusion models, inspired by our observation of the low-rank and locally linear Jacobian of the denoiser over the noisy images. We provide a theoretical framework for demonstrating and understanding such properties, which can deepen the interpretation of the learned data distribution in diffusion models.
Image Editing in Unconditional Diffusion Models. Recent research has significantly improved the understanding of latent semantic spaces in diffusion models, enabling global image editing through either training-free methods [29, 30, 31] or by incorporating an additional lightweight model [30. 77]. However, these methods result in poor performance for localized edit. In contrast, our approach achieves localized editing without requiring supervised training. For localized edits, [25] builds on [30]. enabling local edits by altering the intermediate layers of UNet. However, these approaches are restricted to UNet-based architectures in diffusion models and have largely ignored intrinsic properties like linearity and low-rankness. In comparison, our work provides a rigorous theoretical analysis of low-rankness and local linearity in diffusion models, and we are the first to offer principled justification of the semantic significance of the basis used for editing. Moreover, our method is independent of specific network architectures.
Other recent works, such as [32], introduce training-free global audio and image editing based on theoretical understanding of the posterior covariance matrix [33], also independent of UNet architectures. However, our approach offers distinct perspective, providing complementary insights and new findings. We explore the low-rank nature and local lincarity in PMP, offering rigorous theoretical analyses. Based on this, our proposed LOCO Edit method allows unsupervised and local- ized editing, which enables several advantageous properties including transferability, composability, and linearity benign features that have not been explored in prior work. Further, we extend the method to unsupervised and texe-supervised editing in various text-to-image models. Additionally, while [24] supports localized editing, it requires supervision from CLIP, lacks theoretical basis, and time-consuming for editing each image. In contrast, our method is more efficient, theoretically grounded, and free from failures or biases in CLIP. The CLIP-superviee may also exhibit bias
16

---

0.67
(a) Ablation on time step.
(b) Ablation nullspace and rank.
0..07
0.27
(c) Ablation on edit strengths.
Figure 7: Ablation Study. (a) Effects of one-step edit time. (b)Effects of using nullspace projection and rank. Ec)effects of editing strengths.
toward the CLIP score, leading to suboptimal editing results, as shown in Figure 6, comparison, our method consistently enables high-quality edits without such bias.
Image Editing in T21 Diffusion Models. T21 image editing usually requires much more compli- cated sampling and training procedures, such as providing certainly learned guidance in the reverse [11], training an extra neural network [21], or fine-tuning the models for certain attributes [22]. Although effective, these methods often require extra training even human inter- vention. Some other T21 image editing methods are training- free [46, 27, 28], and further enable editing with identifying masks [46], or optimizing the soft combination of text prompts [28]. These methods involve a continuous injection the edit prompt during the generation process to gradually refine the generated image to have the target semantics. Though effective, all of the above methods (either training-free -ree or not) as well as instruction cnaructtoonuuieedd guided ones [78, 79, 80. 81] lack clear mathematical interpretations and requires text supervision. [23] discovers editing directions in T2I diffusion models through contrastive learning without text supervision, but is not generalizable to editing with text supervision. [30] has some theoretical basis and extends to editing approach in T21 diffusion models with text supervision, but such supervision is only for unconditional sampling. In contrast, our extended T-LOCO Edit, which originated from the understanding of diffusion models, is the first method exploring single-step editing with or without text supervision for conditional sampling.
sampling process
C More Experiment Results on LOCO-Edit
C.1 Ablation Studies
We conduct several important ablation studies on noise levels, the rank of nullspace projection, and editing strength, which demonstrates the robustness of our method.
Noise levels (i.e., editing time step (). We conducted an ablation study on different noise levels, with representative examples shown in Figure 7a. The key observations are summarized as follows: (a) Larger noise levels (i.e., edit on 2, with larger t) perform more coarse edit while small noise levels perform finer edit; (b) LOCO Edit is applicable to generally large range of noise levels ([O.2T, 0.7TJ) for precise edit.
Rank of nullspace projection r'. Ablation study on nullspace projection is in Figure 7b (definition of r' is in Algorithm 1). We present the key observations: (a) the local edit ability with no nullspace projection is weaker than that with nullspace projection; (b) when conducting nullspace projection, an effective low-rank estimation with r' = 5 can already achieve good local edit results.
Editing strength The linearity with respect to editing strengths is visualized in Figure 7c, with the key observations in addition to linearity: LOCO Edit is applicable to generally wide range of editing strengths ([-15, 15]) to achieve localized edit.
17

---

Eye
Up
Eyebrow
Nose
Dog ear
Dog mouth
Figure 8: Visualizing edit directions identified via LOCO Edit. The edit directions semantically meaningful.
Original
Transfer (different position)
Trunsfer (different position)
Original
Transfer (same dataset)
Transfer (different dataset)
Mouth
Original
Transfer (different flower)
Transfer (different flower)
Original
Transfer (with eyes)
Transfer (without eyes)
Eye
Figure 9: Analyzing transferability of edit directions to objects with different positions and shapes, images from different datasets, or images with no corresponding semanties.
C.2 Visualization and Analysis of Editing Directions
We visualize the identified editing direction Up (see Algorithm 1) in Figure 8. The editing directions are semantically meaningful to the region of interest for editing. For example, the editing directions for eyes, lips, nose, etc., have similar shapes to eyes, lips, nose, etc.
Further, since the objects in datasets Flower, AFHQ. CelebA-HQ, and FFHQ usually positioned at the center, the identified editing directions also tend to be at the center. Besides, objects could have different shapes, and semanties some images do not exist in other images. To further study the robustness of transfcrability for the editing directions, we transfer editing directions to images with objects different positions, from different datasets, with different shapes, and with no corresponding semantics. We present the results in Figure 9, with key observations that: (a) the edit directions are generally robust to gender differences, shape differences, moderate position differences, and dataset differences. illustrated in the first five rows of Figure 9 (b) transferring editing direction to images without corresponding semantics results almost no editing (shown the last row Figure 9). Therefore, in practical applications, meaningful transfer editing scenarios for LOCO Edit occur when the transferred editing directions correspond to existing semanties in the target image (e.g.. transferring the editing direction of "eyes" is effective only if the target image also contains eyes).
18

---

10 DDPM U-Vi
Saa.eoooooo-PPoonnnn T-Prrmpt-1 U--UVVI
DespFloyd-Il
CIFAR-10-DDRP
Ceepployd-IF-Promppe
-+-
CelebA-U-Vi
ImageNet. U-VT
Cosine similarity Norm ratio
Cosine similarity Norm ratio
1.000
0.9
0:075
0,950
0
10
20
20
40
30
40
1.0
1,000
0.975
0.8
0,950
A
(a) t= 0.0
(b) t= 0.5
Figure 10: More results on the linearity of fe, ( t).
D More Empirical Study on Low-rankness & Local Linearity
D.1 Experiment Setup for Section 3.1
We evaluate the numerical rank of the denoiser function zn (ax, t) for DDPM (U-Net [49] architecture) on CIFAR-I0 10 dataset [50] (d = 32 x 32 x 3), U-ViT [51] (Transformer based networks) on CelebA [52] (d 64 x 64 3), ImageNet [53] datasets (d 64 x 64 x 3) and DeepFloy IF [19] trained on LAION-SB [54] dataset (d = 64 x 64 3). Notably, U-VIT architecture uses the autoencoder to compress the image Be to embedding vector Zy = Excoder(Zxu) and adding noise to for the diffusion forward process; and the reverse process replaces æt. ai-At with 21, #t-At in Equation (1). And the generated image To Decoder(zu). (20). The PMP defined for U-ViT is:
2: - V 04ED (z,, Zt, 1)
L. = fo,(Zz;t) = Decoder
Va
The Jo.c(zrit) V =, (z;; for (z;;(t diffusion models, one for generation and the other two for super-resolution. Here we only evaluate (z;; for diffusion generating the images.
t) defined above. For DeepFloy IF, there are three
Given random initial noise Ir, diffusion model ag generate image sequence {x, follows reverse sampler Equation (1). Along the sampling trajectory {at)), for each æ,, we calculate Jo. (z;; () and compute its numerical rank via
arg min {r. Σ. of (Jo, ER of (x(:t))
17 2
(10)
where (A) denotes the ith largest singular value of A. our experiments, we set 1) = 0.99. We random generate 15 initialize noise 2. (z, for U-ViT). We only use one prompt for DeepFloyd IF, We use DDIM with 100 steps for DDPM and DeepFloyd IF, DPM-Solver with 20 steps for U-ViT, and (x;;
select some of the steps calculate rank(Jo, (=x;; f)), reported the averaged rank in Figure 2. To report the norm ratio and cosine similarity, we select the closest t to 0.7 along the sampling trajectory and reported in Figure 2, Le. t 0.71 for DDPM, 0.66 for U-ViT and t 0.69 for DeepFloyd IF. The norm ratio and cosine similarity are also averaged over 15 samples.
D.2 More Experiments for Section 3.1
We illustrated the norm ratio and cosine similarity for more timesteps in Figure 10, more text prompts, and Now-matching-bae based diffusion model in Figure 11. More specifically. for the plot of t = 0.0, we exactly use / 0.04 for DDPM, / = 0.005 for U-VIT and 0.09 for DeepFloyd IF; for the plot of I 1= 0.5, we exactly use = 0.49 for DDPM, /. = 0.50 for U-ViT and / = 0.49 for DeepFloyd IF The results aligned with our results Theorem that when / closer the the linearity of fo.((m,, () better.
19

---

DeepFloyd-IF-addee lodaiill----ddiionna--aaa -51-0mmppss55s-s-ps
Sim-ImageNet (Flow Matching)
10"
1.05
Norm ratio
1.00
Rank ratio
10-1
0.95
Cosine similarity
10
20
30
40
1.0
0.9
10-2
0.8
0.2
0.4
0.6
0.8
1.0
10
20
30
40
X
Figure 11: More empirical study on low-rankness and local linearity on more prompts and models trained with flw-matching objectives.
Bhs d7a
dr 7AE,
Bc: 24,
2/1,
10"
10
Rank ratio
10-1
Je
10
10-2
10
10-4
10
0.4
0.8
0.2
0.6
0.8
Figure 12: (Left) Numerical rank of different jacobian J different timestep (Right) Frobe- nius norm of different jacobian J at different timestep
D.3 Comparison for Low-rankness & Local Linearity for Different Manifold
This section is an extension of Section 3.1. We study the low rankness and local linearity more mappings between spaces of diffusion models. The sampling process of diffusion model involved the following space: x; E Xr Ø E Xo,s, h2 E Has E, E &x. where H. the h-space of U-Net's bottleneck feature space [29] and E, is the predict noise space. First, explore the rank ratio of
Jacobian J and Frobenius norm Jo.(|l for: Th, de, ddo.t de, dā.. We use DDPM with
dx. Oh;" Oh. Ox. dz. U-Net architecture, trained on CIFAR-IO dataset, and other experiment settings are the same as Appendix D.I, results shown in Figure 12. The conclusion could be summarized as:
dh, de, dan. dāo. Oz," Ōh," Ōh, dx, are low rank jacobian when [0.2,0.7]. As shown in the left of Th, de, dāo, 0 is less than 0.1. It should be noted that:
Figure 12, rank ratio for dx," Oh,' Oh, Ox;
der, 2 d rank taos This is because dz:
rrank( Oot
at dE,,
rank
> rank
Ia) rank
Vai
dx,
Vas
Or,
De, is high rank when
Fāo, is low rank.
Therefore, dx.
dx
20

---

hy
1002
1.34
Cosire sim arby
0,5
Norm atto
1201
Symmetry
0.00
0.00
1000
0.4
01500
00.1
11.2
0.5
Figure 13: (Left, Middle) Cosine similarity and norm ration of different mappings with respect to A. (Right) Symmetric property of Ezo.
with respect timestep (.
da,
Gdo. rank
ddo.t dx.
This is because to,t = at vI (nE and
Oa:
rank
8h.
Vai
Oh:
When It fixed, E0,1, will change little when changing h4. As shown in the right of Figure 12, Oeo.
|| and dc.
< dc, This means when It fixed, Eo,t, Ee will change little Th. dx.
ll
||e <<
Ōh, when changing hes
Then, we also study the linearity of h, and æo.: given It, using DDPM with U-Net architecture trained on CIFAR-10 dataset. We change the step size X defined in Equation (4). Results are shown in Figure 13, both h, and æ have good linearity with respect to Z...
In Theorem the jacobian V E[ [x.] is symmetric matrix. Therefore, we also verify the symmetry of the jacobian over the PMP Jo.r. We use DDPM with U-Net architecture trained on CIFAR-10 dataset. At different timestep i, we measure || Jo,, Results are shown on the right of Figure 13. Jo.: has good symmetric property when t < 0.1 and t E [0.6, 0.7]. Additionally, Jo. is low rank when t [0.6, 0.7]. So Jo. aligned with Theorem t E [0.6, 0.7].
To the end, we want to based on the experiments in Figure 12 and Figure 13 to select the best space for out image editing method. de;
is the high-rank matrix, not suitable for efficiently estimate the Öh
Ox,
de: and Üæg.t has too small Frobenius norm to edit the image. Therefore, only Dæ. Th, 3h,
nullspace:
and Fw.. are low-rank and linear for image editing. What's more, h space is restricted to UNet
dx.
architecture, but the property of the Oātot dx; does depend on the UNet architecture and verified in diffusion models using transformer architectures. Additionally, we could only apply masks on t but cannot h, Therefore, the PMP fa.: is the best mapping for image editing.
E
Extra Details of LOCO Edit and T-LOCO Edit
E.1
Generalized Power Method
The Generalized Power Method [34, 30) for calculating the op-/ singular vectors of the Jacobian is summarized in Algorithm 2. It efficiently computes the top-k singular values and singular vectors of the Jacobian with randomly initialized orthonormal V E Rdd
E.2 Unsupervised T-LOCO Edit
The overall method for DeepFloyd summarized Algorithm For T2I diffusion models in the latent space such as Stable Diffusion and Latent Consistency Model, at time we additionally decode Eo into the image space &n to enable masking and nullspace projection. The editing is still in the space of
21

---

Algorithm 2 Generalized Power Method
1: Input: /: R E R and V R
top singular values and vectors of the Jacobian af
2: Output: EU,E,
3: y + -f())
4: if V is empty then
5:
V + i.i.d. standard Gaussian samples
6: end if
7: Q,
QR(V)
Reduced QR decomposition Ensures V V
8: V
9: while stopping criteria do
U 0f(m aV) at
10:
Batch forward
a aa V
v)
11:
dx 12: R SVD(V)
Reduced SVD
end while
14: Orthonormalize U
Algorithm 3 Unsupervised T-LOCO Edit for T2I diffusion models
1: Input: Random noise ær,, the mask S2, edit timestep pretrained difflusion model Eø, editing scale noise scheduler oe,, ot, selected semantic index k nullspace approximate rank r, original prompt co, null prompt Cu. classifier free guidance scale s.
2: Output: Edited image æh.
3: <- ODIM(#r: 1, 6o (wrr, ca) + s(Fo
co cn)))
4: Φo. fö. (mx)
5: Masking by Tho 6 (a( and mo, + .n,1 - F0,4
Use the mask for local image editing Efficiently computed generalized power method
6: The top-k SVD (Ü.,,, Ei,,, Vi,e) of Ja, Zāo.t
dx.
7: The top-r SVD (Ut,,, Ee,r, Vi.r) of Ja.-
Efficiently computed via generalized power method
dei
8: Pick direction < V.
& Pick the 8th singular vector for editing within the mask Ω
9: Compute Up + (1 Vor Verr)
> Nullspace projection for editing within the mask Ω Normalize the editing direction
Up f- |1
11: a/ +- 2i + XUp
12: # <- DDIM(w,, O, co(as, L, en) ((,a(ax,4,,e)) ca(wi, cx)))
E.3 Text-suprvised T-LOCO Edit
Before introducing the algorithm, we define:
fa.( St (,o,(eo( 4, c,) + s(eo (,) (,(
(11)
at
and
(x.( (x.) m(Ea( (,))) - c Go Cn)))
(12)
at
to be the posterior mean predictors when using classifier-free guidance on the original prompt co, and both the original prompt c, and the edit prompt ea accordingly.
Algorithm. The overall method for DeepFloyd is summarized in Algorithm For T2I diffusion models in the latent space such as Stable Diffusion and Latent Consistency Model, at time t, we additionally decode &n into the image space â, to enable masking and nullspace projection. The editing is in the space of z, for Stable Diffusion and Latent Consistency Model. The proposed method not proposed as an approach beating other T2I editing methods, but as way to both understand semantic correspondences in the low-rank subspaces of T21 diffusion models and utilize subspaces for semantic control in more interpretable way. We hope to inspire and open up directions in understanding T2I diffusion models and utilize the understanding in versatile applications.
22

---

Algorithm 4 Text-supervised T-LOCO Edit for T2I diffusion models
1: Input: Random noise ær, the mask Q, edit timestep t, pretrained diffusion model cø, editing scale noise scheduler oc,, selected semantic index k nullspace approximate rank original prompt Co. edit prompt null prompt Cn. classifier free guidanee scale
2: Output: Edited image z/g-
3: a; + DDIM(#T, 1, so (ær,, i, cn) + ((E,(aT, Eo(
4: Fo. - (ā.(
5: Lo. t (xt)
6: d + Pn ( 60.e)
7: Zo.t t -Pa(
8: v + Odd' To.1)
>. Get text-supervised editing direction within the mask
9: Z.. 0. Pa(
0āa.
The lop-r SVD (U..: Ei,.. Ve.r) of Jo.
Efficiently computed via generalized power method
Do.
11: Up + (I- -. V.T.)
nullspace projection for editing within the mask Normalize the editing direction
12: Up Toplla
a + It + X0p
14: To < DDIM(a f, O, co 4, cn) ((, ca( cn)))
Here, we want to find specific change direction Up in the space that can provide target edited images in the space of æa by directly moving x; along vp: the whole generation is not conditioned on at all, except that we utilize ce in finding the editing direction up. This is in contrast to the method proposed in [30], where additional semantic information is injected via indirect x-space guidance conditioned on the edit prompt at time t. We hope to discover an editing direction that is expressive enough by itself to perform semantic editing.
Intuition. Let G.. be the estimated posterior mean conditioned on the original prompt Co. and # be the estimated posterior mean conditioned both the original prompt c, and the edit prompt c.. Let J and J be their Jacobian over the noisy image æ; accordingly. The key intuition inspired by the unconditional cases are: i) the target editing direction in the æ, space is homogeneous between the subspaces in Jg,, and J : ii) the founded editing direction v can effectively reside in the direction of right singular vector for both Jg and J iii) L5 and a are locally linear.
Define a5 0 d as the change of estimated posterior mean. Let Jq. U; SV then =o for some i. Besides, we have th #6, + Xo Ja. and a0. 26. X X due to X J5 sf and then JØ d =t ofo + $ffe, which is along the desired direction v. And this identified through the subspace in Jj.t can be effectively transferred in JH,c for controlling the editing of target semantics. We further apply nullspace projection based on J r to obtain the final editing direction Up.
homogeneity and linearity. Hence,
Proofs in Section
F.1
Proofs of Lemma 1
Proof of Lemma I. Under the Assumption could calculate the noised distribution pe (xx.) at any timestep f,
Dr (xx)))
1
Σ
pr (axx/"aa l'wo belongs class k²)
K
K
/ pe( Iag = Maac, "xn belongs to class l²") (ax: 0, I, da
K
23

---

Because a (O, pr(zi/zo = Mæag. "z belongs to class k²) M (1 aa). From the relationship between conditional Gaussian distribution and marginal Gaussian distribution, is easy show that (z zo belongs to class k*) ~ NV(O, a,MM +(1-a)
Then, we have
K Σ
(x()))
a + (1 La)
K
k=1
Next, we compute the score function as follows:
V a. (o. = (x+))
(x()
Ek N(0,,,MM, + + +1--a)) (- (-1- It + at
M,M, 21)
at
1-
N(0, (1)-)))
VO,, + +(1 L.)
Et +
at
1 1 at
NO.. +1 aa)
Based on Tweedie's 's formula [45, 82], the relationship between the score function and posterior is
24 + (1 (xe)
V æ. (x;)
[00 [200
(13)
Voa
Therefore, the posterior mean is
NIO, M +11 MM,
E| [x]]
Var
k=
N(0 + +1-aa))La)
Σk exp
- (a. + a1) xz M& xt
Vai
ΣK
2 (a.M.M + (1 -at) x4
ΣR=1 exp
EK K exp
-a.l xx 1f2)) M&M x
2(1 - -aa)
Voi
1
(1122112 - arell M xx 2x.13)
Lk=1 exp
2(1 ozs)
K
Am
exp
2(1
a₁
a+)
M. z011
MrM xc.
Vai
ΣK exp
0t 2(1 -11-a)) 0z)
MM x012)
where the third equation obtained by Woodbury formula [83] (a. (1 -a)
i-a aq (Ia - M.M.M
24

---

F.2 Proofs of Theorem 1
Lemma 2. The jacobian of the poster mean is
k
22, E [00 = vot
Wk (1)) M
A=
K
+ atyy C2
(k( MMMM xx M
(1 a1)
(14)
B:=
01v0
Σ k=1
(WM Da Σ (
(1 (1-aa) a1)
exp
|M x
2(1 11--0 a1)
where wa (xt) =
Σ
21=1 exp
K
M 2:12
22(1 ar))
2(1 a2
M æell?
exp
Proof of Lemma 2. Let wet))
=
so we have:
ΣK exp
0tt
1|,, x012
2(11 at)
E [20o Vor
wk (a.:) xc
Va, (x(4)
an (1 - oz))
wx(
M&M az Σ (M
So:
K
V E. [aoolar) [rc] = Vot
wk (x(rr))MMM
VoL Σ n, (x:)),) M,M
k=1
K
Voz
(x
k=1
0
W/ (M x. M,M
a1
K [ul
Σ wx(z))
(1-ar)
Lemma 3. Assume second-order partial derivatives of pe (xx)) exist for any D,, then the posterior mean V n, E [wo] satisfied Vn. n, E [250 [x:]] Vn, E [x).
Proof of Lemma 3. By taking the gradient of Equation (13) with respect to æ, for both side, because the second-order partial derivatives of (x exist for any Tt, we have:
I + (1 04 2 logp. (æ:)
I Dg [rt]
1i
25

---

The hessian of (a;) is symmetric, so we have:
I+ (1 ar) (V.,
logp (xx)
I+ (1 at) V2 log
Vx, E [
Vx, 1 [+r]
Vor
Var
Notably, the symmetric of V 2 holds without the Assumption 1.
Proof of Theorem I. First, let's prove the low-rankness of the posterior mean. From Lemma 2,
ar
01ty an C
V m E
[
V A +
B
at)
(1)
K Σ
M (VorA+
ai
a1 ai (1-a)) a4) c)
+ (1-04)
B
where the second equation is obtained due to the fact that Σ M&M A A, M.M, == B E M =C Therefore, we have:
rank (V,, E (200 (x)) rank
M (VoiA+
ar
B an C
(1 (x.) (1 04
(15)
K
K
S rank
M&M
= - Th
Then, we prove the linearity:
① |E (yl D + Adx)
- E [2pp
XV E(zul Axl122
Ot
(wk (xt + AAx) - (x(x))) M (x: + AAx) A
V n. (x))) M Aæl12
k=1
R=
=IIVo 04
Wk (a + )1 M (xt + AAx) A
|2 M Azl
20
K
a1 V T
m. Wh + A1 (x; + +Ax) 1l2 +x M, A V (x)))l
/ V Va, Ar)aalaa|MM (x, + [
M x l12)
where the first equation plug in the formula of Va, [oo Vai Σ ui (a: yo, Σx= (a.wa(a.))e T M. and the second equation use the mean value theorem wi( A).
AAx) (x( = A wr(x; AZ) \1
26

---

2 V 2: wx + >11x))12
K
a1
wx Mr (xt + A1A))
Σ
wi
M (ar + A1)llll2
(1 oz)
K [ moll2 + Σ
K
$ (1 a1))
wk
will M æ2l12 + A Az + A Σ
wil MM, A dzll 1z
1=1
a1
[|Me 1|11|00l12 +
Σ will ||||2z211 + 11e + A1 Σ wi
Mi 11F
wk
(1 ai)
K
< 0x
Tk + Σ wi)))
(V2max[[ + +11)
Wk
[1 a1)
1=1
01
(eraxrt-(VEmaxiiiii (V2max([ [|lll/ ||e|12] + 21).
W/ (x' + \1 2 maxx=
k
Ci==
where the third inequality use the fact that ||æz||2 Vl + vT-aiell S lI
[ S 2 max /200/// ||l]]l//2, we simplified wa (æt + Å Ax) as wa in this prove, and Ci defined in the last inequality is independent of I. Similarly, we could prove that:
(V2 (ax || + +2):
M (a,, AAx) lla
max Ti b
V a. wxx))ll S (1 a1)
wR (xxx)) 2 max T max xol l2, ||E|12].
ar)
: MrM xz 1/2 S V2 max Fh max {|| 20lle, || [2
C
Here, C1 O(A, = 0(A), O(A), C. O()). After plugin 2.2.C. 1 to !. could obtain:
EE
[ [ + \Aæc - E [2o|=x]] - AV a. 1/20 [ Ax |2
K
K
<\Vor
Wk +C
at
wk (x(xx))331
(1
aa)
k= (1 al))
k=
1 O2
O(A)
Finally, let's prove the property of the left singular vector of V 22, 100o2x13
From Lemma 3, the eigenvalue decomposition of V Va, E [xo]<x]]] could be written as V E [z]] U,N.U,T, where A, diagg(Ar.......... At,r.- ... 0), and the relation between eigenvalue decomposi- tion and singular value decomposition of Va,, [an a could be summarized as for all E [r]:
ot,i = = v; sign (Ae,s) u.
where sign (-) is the sign function. Therefore, we have:
U,, V., VIJ
(16)
given Vr, = [Vr,1, U1,2, V. From Lemma 2, we define:
K
K
V E [x) = Vos
wk (xx M + Voe V Va, wk (I) (x))az M
1=1
1=
A
27

---

From the full singular value decomposition of V E [
and Vou (2))
E,,1
0
V1,
E
= 1
U122
E
E1,
Vi.2
Σ
0
V4,1
a1
(
[111
U222]
V22
where:
0t11
Øtr
Σt,1
E1,
tt.
1tn
ōt,1
Øirt1
Σt,1
E
ōt.r
Ōt.
0t11 2 0t, ... oer 2 ... 2 dt.d ōz,1 2 0z,2 2 ... 2 oz, ... 2r,ds
Tke
From Equation (15), we know that 0r,4+ = Ot,d = O. It is easy to show that:
M D2,1 M,.
M
M
where 821,82 .8] = (1,2..... ..... K} satisfied was (æx)) 2 Way (a:e) 2 2 Wax (ar)). And
āt. Varwss (1+) = /0x, ming wa (2te). Based on the Kahis-Kahan theorem [84], we have:
|
-Ia- - Vv.V. M v
min 0 Øz.s
IIvor Ek=1
K
V Vn, wA (20 M
Var mina Wh (20e)
>k=1 k=1
K
a, wi (x:)) M&M ||F
min& wk (2:e)
C3CA ae ming (1xe
1
04 = 0, SO:
Because lim,-1 ming (xx))) =
lim
K'
1 1 ag
lim t-ll
(Ia V4, V.T1) M |F = 0.
And from Equation (16), we have:
lim
(Ia U,,U,T)) M|le 0.
G Image Editing and Evaluation Experiment Details
All the experiments can be conducted with single A40 GPU having 48G memory.
G.1 Editing in Unconditional Diffusion Models of Different Datasets
Datasets. We demonstrate the unconditional editing method in various dataset: FFHQ [63],
CelebaA-HQ [52], AFHQ [62], Flowers [61], MetFace [85], and LSUN-church [60].
28

---

Models. Following [30], we use DDPM [1] for CelebaA-HQ and LSUN-churchs and DDPM trained with P2 weighting [86] for FFHQ, AFHQ, Flowers, and MetFaces. We download the official pre-trained checkpoints of resolution 256 x 256, and keep all model parameters frozen. We use the same linear schedule including 100 DDIM inversion steps [3] as [30]. Further, we apply quanlity boosting after t 0.2 as proposed in [87].
Edit Time Steps. We empirically choose the edit time step t for different datasets in the range [0.5, 0.8). In practice, we found time steps within the above range give similar editing results. In most of the experiments, the edit time steps chosen are: 0.5 for FFHQ, 0.6 for CelebaA-HQ and LSUN-church, 0.7 for AFHQ, Flowers, and MetFace.
Editing Strength. In the empirical study of local linearity, observed that the local linearity is well-preserved even with a strength of 300. In practice, we choose the edit strength À in the range of [5.0,, 15.0), where larger leads to stronger semantic editing and a negative leads to the change of semantics the opposite direction.
G.2 Comparing with Alternative Manifolds and Methods
Existing Methods We compare with four existing methods: NoiseCLR [23], BlendedDiffussoon [24], Pullback [30], and Asyrp [29].
Alternative Manifolds. There are two alternative manifolds where similar training-free free approaches
dc: (equivalently
can be applied, and each of the alternative involves evaluation of the Jacobians Oh. aæu and de: Oh," Ox, accordingly.
de: (or equivalently
0e.. up scale) calculates the Jacobian of the noise residual 6, with
Th,
Th,
respect to the bottleneck feature of mt.
De:
calculates the Jacobian of the noise residual with respect to the input zt. oæ;
dc: has hardly notable editing results on images. and hence we present the editing results de: also leads hardly notable changes
Notably, Ōh.
of de: Besides, with masking and nullspace projection, de; Oæ.
on images, thus the final comparison without masking and nullspace projection.
Evaluation Dataset Setup. In human evaluation, for each method, we randomly select 15 editing direction 15 images. Each direction is transferred to other images along both the negative and positive directions, in total 90 transferability testing cases. Learning time and transfer edit time are averaged over 100 examples. LPIPS [64] and SSIM [65] are calculated over 400 images for each method.
Human Evaluation Metrics. We measure both Local Edit Success Rate and Transfer Success Rate via human evaluation on CelebA-HQ. i) Local Edit Success Rate: The subject will be given the source image with the edited one, if the subject judges only major feature among { |"eyes", "nose", "hair", "skin", "mouth", "views". "Eyebrows") } are edited, the subject will respond a success, otherwise a failure. ii) Transfer Success Rate: The subject will be given the source image with the edited one, and another image with the edited one via transferring the editing direction from the source image. The subject will respond success if the two edited images have the same features changed, otherwise failure. We calculate the average success rate among all subjects for both Local Edit Success Rate and Transfer Success Rate. Lastly, we have ensured harmful contents are generated and presented to the human subjects.
Learning Time. Learning time is measure of the time it takes to compute local basis(training free approaches), to train an implicit function, or to optimize certain variables that help achieve editing for specific edit method.
29

---

G.3 Editing in T21 Diffusion Models
Models. We generalize our method to three types of T2I diffusion models: DeepFloyd [19], Stable Diffusion [4], and Latent Consistency Model [38]. We download the official checkpoints and keep all model parameters frozen. The same scheduling as that in the unconditional models applied to DeepFloyd and Stable Diffusion, except that no quality boosting is applied. We follow the original schedule for Latent Consistency Model [38] with the number of inference steps set as
We empirically choose the the edit time step t as 0.75 for DeepFloyd and 0.7 for Stable DIffusin. As for Latent Consistency Model, image editing is performed the second inference step.
Edit Time Steps.
Editing Strength. For unsupervised image editing, we choose E (-5.0, 5.0) in Stable Diffusion, A E [-15.0, 15.0) in DeepFloyd, and À E [-5.0, 5.0) in Latent Consistency Model. For text- supervised image editing, we choose À E [-100, 10.0] in Stable Diffusion, [-50.0, 50.0) in DeepFloyd, and À E [-10.0, 10.0) in Latent Consistency Model.
H Social Impacts and Safeguards
The paper originally presents new image manipulation method, with a theoretical framework to deepen the understanding of diffusion models. However, there exist potential social impacts that the proposed methods can be misused in generating and manipulating harmful content. Therefore, we will release code and models with license and ethies commitments in the future. Besides, methods for identifying and preventing such harmful behaviors are great significance in generative models.
30

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We present the empirical observation Section 3.1, image edit method in Section theoretical proof in Section 4 and Appendix F, and experiment results in Section in details in the paper and appendix.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations and future direction of our works in Appendix A.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: For the theory presented in Section 4, we provide detailed proofs in Appendix F.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed experiment setup in Appendix G for reproducing our result.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide codes and documentations https://gittubb.com www./oo///hhhhhiiiieecccccccceee LOCO-EdiE Edit.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed experiment setup, evaluation setup, and metrics setup in Appendix G for better interpretation of our results.
7. Experiment Statistical Significance
Question: Does the paper report eтror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Our generation experiments conducted randomly for hundreds of times across different dataset and models.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification: We provide computation resources information in Appendix G.
31

---

9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethies https://neurrps. http:://p/ppplccciiiiiiiiieeeeee
Answer: [Yes]
Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the paper's 's impacts in introduction and related works, as well as potential misuse and our commitment in preventing harmful behaviors in Appendix H.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators. or scraped datasets)?
Question:
Answer: [Yes]
Justification; We discuss potential misuse and our commitment in preventing harmful behaviors in Appendix H.
12. Licenses for existing assets
Question: Are the creators or original owners assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have properly credited all existing models and datasets that related to the
paper.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide detailed descriptions and implementation details for the proposed new method. We have also released codes for reproducibility.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We provide details on human evaluation for the generated images in details in Appendix G.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [Yes]
32


Reviews:
[{"review_id": "3MwOP7yfCc", "reviewer": "Reviewer_eLFT", "summary": "This paper made an interesting observation on local linearity of the diffusion model's denoiser. Based on the observation, the author introduces a novel method of performing one-step closed-forrm operation to achieve semantic image editing. Empirical and numerical results are given to demonstrate the effectiveness of the method", "questions": "1. I wonder whether the homogeneity of the editing can be observe on samples that are very different: e.g., 1) computing editing direction on an image with face at the left, and transfer to an image with face on the right; 2) computing editing direction on an image with realistic style, and transfer to an image with cartoon style.\n2. In addition, since current models such as SD3 are using flow-matching objective to train the model, I wonder if such observed phenomenon would be more prominent on these models?", "limitations": "Yes, the limitations have been adequately addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The observations in terms of local linearity of the denoiser is interesting, and is well-validated across different architecture and datasets.\n2. The presented method is novel and well-motivated, and simple.\n3. The discovery of homogeneity and transferability of the editing direction is insightful.\n4. The empirical and numerical result demonstrate consistent image editing.", "weaknesses": "1. The computation time of the method is still a bit long (taking around 70s if I understand correctly).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer’s constructive feedback, and in the following we address the reviewer’s concerns one-by-one.\n>**Q1: The computation time of the method is still a bit long (taking around 70s if I understand correctly)**  \n\n**A1:** \n1. Compared with the global edit method Pullback [25], the additional cost is introduced from finding local edit direction, yet the addition cost is not significant. \n2. Besides, we add additional comparisons with a local edit method BlendedDiffusion [1] in Table A of the global response PDF, where our method is more efficient in learning time.\n3. Moreover, our method can transfer the local edit direction to other images with a high successful rate and no additional time cost in learning. On the contrary, other local/global edit methods cannot transfer the edit direction (BlendedDiffusion) or have weaker transferability as summarized in Table A.\n\n>**Q2: Whether the homogeneity of the editing can be observed on samples that are very different:** e.g., \n>- computing editing direction on an image with a face at the left, and transfer to an image with a face on the right; \n>- computing editing direction on an image with realistic style, and transfer to an image with cartoon style.\n\n**A2:** \nThis is a good question. We present more results and extend the discussions.\n1. - For unconditional diffusion models on FFHQ and CelebA, we can transfer edit directions to faces on the different side, as presented in **Figure B of the global response PDF**. But we do notice that these datasets tend to have the object in the center, which benefits the transferability. \n\t- We have further attached the visualization of editing directions in Figure C, and the edit direction has semantic correspondences to the target edit region. From our observation, the transferability is robust to gender differences, facial feature shape differences, and moderate position differences in FFHQ and CelebA.\n2. - To achieve goal (2), conditional diffusion models are required. We have previously explored such transferability in stable diffusion models, and the transfer success rate is low. By our analysis, the homogeneity needs the images lie in the same subspace, but the manifold of images with different styles in conditional diffusion models may violate this. \n\t- Understanding the feature space of conditional diffusion models is a challenging and unsolved problem in the area, since the feature space is related to text prompts and noisy images. We are interested in exploring the question, and hope our theoretical analysis in unconditional diffusion models can provide some inspiration for future works. \n   \n\n>**Q3: In addition, since current models such as SD3 are using the flow-matching objective to train the model, I wonder if such observed phenomenon would be more prominent on these models?** \n\n**A3:** \n1. This is an interesting question and we have explored it further. Due to time constraints, we study the low-rankness and local linearity of SiT [2], a simpler diffusion model using the flow-matching objective at training. As presented in **Figure F of the global response PDF**, we do see generalized local linearity, and a similar low-rank trend in SiT, though the rank is not exactly as low as other diffusion models. It would be interesting to study the differences in low-rank subspace between diffusion models trained under different objectives, and test other diffusion models trained using flow-matching objectives such as Stable Diffusion 3.\n\n[1] Omri et al., Blended diffusion for text-driven editing of natural images. CVPR 2022.\n\n[2] Nanye et al., \"Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers.\" 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tUIcYwyYW6", "reviewer": "Reviewer_h6gb", "summary": "The paper presents a method for steering the generation in a diffusion model, without any further training. The proposed method is based on two insights about the Posterior Mean Predictor (PMP) and its Jacobian -- that is, the former being locally linear and the latter having singular vectors lying on low-dimensional subspaces. Some theoretical results are provided towards the justification of the above. Local linearity of the PMP allows for a single-step, training-free method for local editing of regions of interest, while the low-rank nature allows for the effective identification of semantic directions using subspace power methods. Some qualitative and quantitative experimental results are provided.", "questions": "Please see weaknesses.", "limitations": "The authors discuss adequately the limitations of the proposed method in the Appendix H (Future Direction). A separate \"Limitations\" section is missing, but the authors discuss the limitations of the paper honestly and comprehensively.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and sound. The provided theoretical results wrt local linearity of the PMP, along with the low-rankness of its Jacobian are interesting (however not extremely surprising) and might be useful to the research community.", "weaknesses": "The weakest aspect of the paper concern the reported experimental results. Both qualitative and quantitative results are extremely limited, whilst comparisons with existing work are also not convincing. Besides, the significance of the empirical results is not convincing. For instance, in Fig. 6, when the proposed method (LOCO) is compared to Asyrp, it's not clear at all why localized editing with LOCO is better than Asyrp. Similarly, in Table 1, the Transfer CLIP Score (an important metric in the context of the task studied in the paper) of Asyrp is significantly better that the proposed method's. The authors do discuss this, but the fact that the proposed framework is learning-free, in contrast to Asyrp that learns how to perform editing, does not justify the significant difference wrt to CLIP Score. \n\nFinally, as the authors acknowledge in the Appendix H (\"Future Direction\", where limitations are also discussed), the provided theoretical framework concerns mainly the undirected image editing part; how text-directed image editing behaves is not addressed by the proposed method. This is a certain limitation, that weakens the generality of the proposed method, yet it doesn't reduce the importance of the theoretical results provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer’s constructive feedback, and in the following we address the reviewer’s concerns one-by-one.  \n   \n>**Q1: Not enough qualitative and quantitative results and comparisons with existing work.**  \n\nThanks to the reviewer's suggestions, we conduct more qualitative and quantitative comparisons with more baselines, and add more results to better evaluate our method. We refer detailed results and discussion to **Q&A 1 of the global response.**\n\n>**Q2. Not convincing empirical results.**\n>1. Fig. 6 is not convincing to show better localized editing with LOCO than Asyrp   \n>2. Why Transfer CLIP Score of Asyrp is better than LOCO, and whether it’s an issue\n\n   \n**A2:** We thank the reviewer for raising these constructive points which improve our work.\n1. **More convincing visualizations.**\n\t- To demonstrate the effectiveness of our method, we selected examples in Fig. 6 where Asyrp achieves its best performance, yet our method still performs even better. For instance, in Fig. 6, to edit lips, Asyrp noticeably alters undesired regions such as face color, hair shape, and face shape more than our method. \n\t- There are many other instances where Asyrp's editing changes the image much more significantly than ours. We add more random examples in **Figure E of the global PDF** to visualize the local edit ability of our method in comparison of others.  \n2. **Discussion on the Transfer CLIP Score in Table 1\\.** **Very good question for a deeper discussion.**  \n   - The higher transfer CLIP score of Asyrp is because it is directly supervised by the CLIP score to learn each concept and predict editing directions. In contrast, our method is unsupervised and learning-free, and has the best CLIP score among other unsupervised methods.   \n   - Moreover, the transfer CLIP score is biased by large changes in Asyrp: For successful edit in both Asyrp and LOCO, Asyrp tends to have larger global changes resulting in a much higher transfer CLIP score, sometimes leading to corruption in other parts though the desired region is edited. Examples are shown in **Figure E of the global PDF**.  \n   - Additionally, the transfer CLIP score relies on CLIP's intrinsic failures to capture detailed semantics as discovered in [1], which may lead to edit failures of Asyrp. Failure examples that are potentially related are shown in **Figure E (row 5-8) of the global PDF**, where Asyrp fails to edit darker eyebrows for all random examples.  \n   - Therefore, to compensate for the biases of the transfer CLIP score, we also measure the local edit success rate, where our method outperformed all others. Thanks to the reviewer for raising the point, we also add LPIPS [2] and SSIM [3] as guardians of dramatic changes and use the less biased local edit success rate as the major evaluation metric for local edit ability.\n\nIn the revision, we will include those discussions in the paper to clarify our findings.\n\n>**Q3: The provided theoretical framework concerns mainly the undirected image editing part; how text-directed image editing behaves is not addressed by the proposed method. This is a certain limitation, that weakens the generality of the proposed method, yet it doesn't reduce the importance of the theoretical results provided.**\n\n**A3:** We thank the reviewer for raising the points, but we want to clarify this further.\n1. In Figure 4, we showed that our method can be extended to text-directed image editing. This demonstrates the generality of our edit approach.  \n2. Second, the empirical observation of low-rankness and local linearity is generalizable to text-to-image diffusion models as shown in Figure 2 and Figure 9.\n3. Although our theoretical study part is for unconditional diffusion models, to the best of our knowledge, our work is the first theoretically ground method compared to all previous diffusion model based editing methods. It's an advantage compared to other image edit methods.\n4. Besides, for studying text-directed diffusion models, the challenges lie in the more complicated feature space caused by a mixture of image and text distributions. Understanding their feature space is yet an unsolved problem in the area. Hope our work can provide some inspiration for future exploration.\n\n[1] Shengbang et al., Eyes wide shut? exploring the visual shortcomings of multimodal llms. CVPR 2024\n\n[2] Richard et al. The unreasonable effectiveness of deep features as a perceptual metric. CVPR 2018.\n\n[3] Zhou et al., Image Quality Assessment: From Error Visibility to Structural Similarity. 2004", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ncuMweTzBD", "reviewer": "Reviewer_pgJR", "summary": "The paper examines the use of low-dimensional subspaces in diffusion models for precise and disentangled image editing. The paper observes that the Posterior Mean Predictor (PMP) in diffusion models shows local linearity across various noise levels, and the singular vectors of its Jacobian exist in low-dimensional semantic subspaces. Building on these observations, they introduce LOw-rank COntrollable image editing (LOCO Edit), a technique that enables efficient, training-free, and precise localized image manipulation by utilizing the Jacobian's low-rank nature. This approach has a strong theoretical basis and has been shown to be effective across different architectures and datasets.", "questions": "1. The paper should clarify that the Posterior Mean Predictor (PMP) is a network function within diffusion models, and detail how it is computed. Specifically, it should describe how PMP is derived and utilized in the context of local linearity and low-rankness.\n   \n2. Elaborate on what P_{\\omega} and P_{\\omega^C} represent in Fig.3 and lines 193-195, and clarify the decomposition of x_{0,t}^{\\hat} into ROI and null space using masking techniques.\n\n3. Explain how performing the Jacobian over different masked images helps in disentangling features for precise local editing.", "limitations": "In addition to the listed weakness above, the sensitivity of the method to various parameters (e.g., noise levels, perturbation strengths) could be explored in greater detail. Understanding the robustness of the approach under different conditions would be valuable for practical applications.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces an approach to image editing within diffusion models by identifying and exploiting low-dimensional subspaces. The idea of using local linearity and low-rank properties of the Jacobian in diffusion models for controllable editing sounds novel.\n\n2. The methodology looks sound, supported by robust empirical evidence demonstrating local linearity and low-rankness of the PMP's Jacobian. \n\n3. The proposed method tackles a key challenge in diffusion models: achieving precise and disentangled image editing without additional training. This has wide-ranging implications for various image generation and manipulation applications.\n\n4. Extensive empirical evaluations are presented, showcasing the method's effectiveness across diverse network architectures (UNet and Transformers) and datasets (e.g., CIFAR-10, CelebA, ImageNet). The results show performance improvements and validate the approach's generalizability. \n\n5. The paper is well-organized and clearly written. Explanations are thorough, and the visuals (e.g., figures) effectively convey the main concepts and outcomes.", "weaknesses": "1. The experimental evidence appears highly limited. The quantitative and qualitative comparison results (confined to Table 1 and Fig. 6 with just one example in the main paper) seem inadequate. The study would be strengthened by a more comprehensive comparison with additional state-of-the-art image editing methods, beyond just [24] and [25]. Highlighting specific advantages and potential weaknesses in relation to these methods would provide a more balanced perspective.\n\n2. The assumptions regarding low-rank Gaussian distributions used for theoretical validation warrant more rigorous examination. Expanding the discussion to include potential limitations or circumstances where these assumptions may not hold would add depth and nuance to the analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for constructive feedback. During rebuttal, we address the reviewer’s concerns on the experiments and other questions as follows.\n>**Q1. Limited experimental evidence and quantitative and qualitative comparisons.**\n\n**A1:** Thanks to the reviewer's suggestions, we added more results, and conducted more qualitative and quantitative comparisons with more baselines to better evaluate our method. We refer detailed results and discussion to **Q&A 1 of the global response**.\n\n>**Q2. Detailed ablation study on sensitivity to various parameters...**\n\nA2: Thanks to the reviewer's suggestions, we conducted more detailed ablation studies and gave a more comprehensive summary. We refer detailed results and discussion to **Q&A 2 of the global response**.\n\n> **Q3: Validation of using the low-rank Gaussian distributions.**\n\n**A3:** In [1], the authors conducted extensive empirical experiments and found that for diffusion models trained on the FFHQ and AFHQ datasets, the learned score can be approximated by the linear score of a Gaussian, particularly at high noise levels. Furthermore, [2] confirms the low-dimensional nature of various image datasets, such as CIFAR-10 and ImageNet, and calculates their intrinsic dimensions. Building on these findings, we incorporate the concept of Gaussian distribution and low-dimensional properties to study the low-rank Gaussian case. This approach enables our model to effectively capture the core structure of image data while remaining tractable for theoretical analysis, thus serving as a practical foundation for theoretical studies on real-world image datasets.\n\nWe will integrate these discussions into the final version to enhance motivation.\n\n\n> **Q4. Clarification on Posterior Mean Predictor (PMP).**\n\n**A4:**\n1.  **Definition & computation of PMP and reference for it’s derivation:** Indeed, PMP is a network function in diffusion models, which takes an input pair ($x_t$, $t$), and outputs the predicted posterior mean $\\hat{x}\\_{0,t}$. Specifically, the computation of PMP in diffusion models is defined in Equation (2), where $\\epsilon_{\\theta}$ is the learned unet denoiser in the diffusion model. We refer how PMP is derived to the derivation of Equation (12) in the DDIM paper [3].\n2.  **How local linearity and low-rankness in PMP is utilized:** (1) PMP’s local linearity is the key assumption for finding edit direction via SVD of PMP’s Jacobian, since directions found via SVD are meaningful only if PMP is linear; (2) PMP’s local linearity also leads to the linear and composable properties of the LOCO edit method; (3) PMP’s low-rankness ensures us to use a low-rank estimation to find the nullspace, achieving efficient and effective nullspace projection.\n\nWe will make the above points clear in the final version.\n\n> **Q5: Clarification on finding the precise and disentangled edit direction.**\n> - Elaborate on what $P_{\\Omega}$... using masking techniques.\n> - Explain how performing the Jacobian... helps... for precise local editing\n\n**A5:** We would like to clarify concepts, process, and intuition in finding such local edit direction. We will also revise the writing to make the points clearer.\n\n1.  **Definition of ROI, $P_{\\Omega}$ and $P_{\\Omega^C}$:** Here, $\\Omega$ is an index set that covers the region of interest (ROI) and  $\\Omega^C$ covers any other region in the image outside ROI. Based upon this, $P_{\\Omega}$ and $P_{\\Omega^C}$ denote the projections onto $\\Omega$ and $\\Omega^C$, respectively. Intuitively speaking, $P_{\\Omega}(I)$ crops the content of an image $I$ within the mask, and $P_{\\Omega^C}(I)$ crops the contents of $I$ outside of the mask. \n2.  **Decomposition of $\\hat{x}\\_{0,t}$ and calculation of Jacobians:** Based on the above definition, we decompose PMP’s output $\\hat{x}\\_{0,t}$ into $\\tilde{x}\\_{0,t}$ and $\\bar{x}\\_{0,t}$ as visualized in Figure 3, where $\\tilde{x}\\_{0,t} = P_{\\Omega}(\\hat{x}\\_{0,t})$ and $\\bar{x}\\_{0,t} = P_{\\Omega^C}(\\hat{x}\\_{0,t})$. We further define $\\tilde{J}\\_{\\theta,t} = \\partial \\tilde{x}\\_{0,t} /\\partial x_t$ and $\\bar{J}\\_{\\theta,t} = \\partial \\bar{x}\\_{0,t} /\\partial x_t$.\n3.  **Finding image editing directions and nullspace via SVD of Jacobians:** Let $\\tilde{J}\\_{\\theta,t} = \\tilde{U}\\tilde{S}\\tilde{V}^T$, and $\\bar{J}\\_{\\theta,t} = \\bar{U}\\bar{S}\\bar{V}^T$ be the compact SVD. Intuitively, $span(\\tilde{V}) = range(\\tilde{J}\\_{\\theta,t}^T)$ is the subspace containing change directions of $x_t$ that leads to change within the $\\tilde{x}\\_{0,t}$. Besides, $span(\\bar{V}) = range(\\bar{J}\\_{\\theta,t}^T)$ is the subspace containing change directions of $x_t$ that leads to edit in $\\bar{x}\\_{0,t}$. Moreover, $nullspace(\\bar{J}\\_{\\theta,t})$ is the subspace leads to no edit in $\\bar{x}\\_{0,t}$.\n4.  **Nullspace projection for more precise and disentangled image editing:** For a direction $v \\in range(\\tilde{J}\\_{\\theta,t}^T)$, nullspace projection means projecting $v$ onto $nullspace(\\bar{J}\\_{\\theta,t})$. In practice, this nullspace projection can be calculated as $v_p = (I - \\bar{V}\\bar{V}^T) v$. Such nullspace projection can further eliminate the effect of $v_p$ to change within $\\bar{x}\\_{0,t}$, and disentangle the change to be only within $\\tilde{x}\\_{0,t}$. Then by denoising $x_t + \\lambda v_p$, the new image is more precisely edited within the ROI $\\Omega$.\n\n[1] Binxu et al., \"The Hidden Linear Structure in Score-Based Models and its Application.\" 2023.\n\n[2] Phillip et al., \"The intrinsic dimension of images and its impact on learning.\" ICLR 2021.\n\n[3] Jiaming et al., Denoising diffusion implicit models. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bX3suslyko", "reviewer": "Reviewer_9wnQ", "summary": "This paper proposes an diffusion image editing framework called LOw-rank COntrollable edit (LOCO Edit) based on two observations: 1)  the learned posterior mean predictor (PMP) is locally linear during the middle timesteps during denoising, and 2) the singular vectors of the PMP's Jacobian lie in low-dimensional semantic subspaces. These observations are backed both empirically and theoretically. To conduct LOCO edit, an image is inverted with DDIM inversion to an intermediate timestep. SVD is conducted on the Jacobian of the PMP w.r.t. to this noisy image in order to discover semantic directions. Furthermore, a mask can be applied along with a null space projection in order to achieve more disentangled and localized edits. A variant of LOCO edit is also proposed to enable text direction. A variety of experiments are conducted for the unconditional LOCO edit on various domains using human evaluation and CLIP score and edit discovery time, achieving superior results against other methods. Qualitative results are shown for various diffusion model variants (e.g., Stable Diffusion, Latent Consistency Model) and domains, demonstrating linearity and the ability to modulate strength, the ability to compose with other directions, as well as the ability to transfer the edits to other images.", "questions": "* I am quite surprised that an edit at a single timestep has enough effect to edit the image. For instance, [1,2] require over multiple timesteps. Is there an intuition for why this edit at a single timestep has enough causal effect?\n\n* For more open-domain diffusion models like Stable Diffusion, can the edit direction be applied to another subject? For instance, in Figure 1, you transfer human opening eyes to different images. Would this transfer to other domains, such as an animal?\n\n* A visualization of all the edit directions discovered via SVD would be quite interesting. Are they all semantically meaningful?", "limitations": "A more substantive discussions of limitations is missing. Although Appendix Section H discusses future directions, there is little mention of limitations of the current method.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper tackles an important problem in diffusion models: disentangled and continuous editing. \n\n* This is done in a principled way, based on two important observations of the PMP and its Jacobian across different timesteps. Furthermore, there are both empirical experiments as well as theoretical analysis validating these. \n\n* These edits can be composed, transferred, and modulated in a continuous manner. Additionally, the time required for discovering these directions is less than other methods, such as Asyrp. \n\n* The qualitative and quantitative results demonstrate strong editing capabilities. \n\n* A variety of ablations are provided in the paper and the appendix, such as the effect of nullspace projection, timestep used for editing, etc.", "weaknesses": "* The qualitative results for editing are not quite the best I have seen, and they are compared against relatively weak baselines (e.g. Asyrp). However, I find that this is okay since the approach is quite principled.  Many impressive diffusion based image editing papers have been released in the past year [1,2]. I don't think it is necessary to compare against them, but including them in the paper in the related works section would keep this paper up-to-date with the current state of image editing. \n\n* Since the linearity and low-rankness properties are more apparent in the middle time-steps, applying more global edits (e.g. changing the shape of something while keeping appearance) would be difficult since more coarse level features are constructed in the earlier timesteps of denoising. I see this effect in Fig. 7 in the appendix, as other parts of the image are edited as well. \n\n[1] @inproceedings{dalva2024noiseclr,\n  title={Noiseclr: A contrastive learning approach for unsupervised discovery of interpretable directions in diffusion models},\n  author={Dalva, Yusuf and Yanardag, Pinar},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  pages={24209--24218},\n  year={2024}\n}\n[2] @article{gandikota2023concept,\n  title={Concept sliders: Lora adaptors for precise control in diffusion models},\n  author={Gandikota, Rohit and Materzynska, Joanna and Zhou, Tingrui and Torralba, Antonio and Bau, David},\n  journal={arXiv preprint arXiv:2311.12092},\n  year={2023}\n}", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for constructive feedback and interesting questions. During rebuttal, we address the reviewer’s concerns on the experiments and other questions as follows.\n\n> **Q1: The qualitative resultes are not quite the best... up-to-date with the current state of image editing.**\n\n**A1:** Thank the reviewer for pointing out the interesting works NoiseCLR and Concept sliders. We have looked into them and will cite and add discussions on NoiseCLR and Concept sliders in the revised manuscript. Besides, based on the reviewer's suggestion, we conducted more qualitative and quantitative baseline comparisons with NoiseCLR. Moreover, we extend both qualitative and quantitative results per the reviewer's suggestion. We refer detailed results and discussion to **Q&A 1 of the global response**.\n\n>**Q2: Since the linearity and low-rankness properties are more apparent in the middle time-steps, applying more global edits... would be difficult since more coarse level features are constructed in the earlier timesteps of denoising...are edited as well.**  \n\n**A2:** Thanks for raising this point here, and we would like to discuss further.\n1. For coarse features that are controlled in early time steps close to random noise, LOCO is not guaranteed to disentangle these features in the high-rank space. This is closely related to our method is principled in the low-rank subspace of diffusion models in the middle time steps, and hence we mainly focus on unsupervised precise local edit. \n2. It would be interesting to non-trivially model the space at those high-rank timepoints, maybe taken inspiration from works such as NoiseCLR. Our current focus is to study the low-rank subspaces in diffusion models and leave the understanding of semantics in high-rank spaces to be explored in future studies.\n\n> **Q3: I am quite surprised that an edit at a single timestep has enough effect to edit the image. For instance, [1,2] require over multiple timesteps. Is there an intuition for why this edit at a single timestep has enough causal effect?**\n\n**A3:** Thanks for the interesting question and we try to provide possible intuitions. \n1. From the perspective of method principles, LOCO initiates from the local linearity of PMP, and tries to find directions that lead to the largest changes in the posterior mean. Under the assumption of local linearity within the low-rank semantic subspace, such identified direction $v$ can achieve one step edit with appropriate edit strength. Following the intuition above, the one-step edit ability is experimentally verified.\n2. In contrast, NoiseCLR optimizes a conditional variable representing specific features to be used in classifier-free guidance, and Concept sliders use LoRA to finetune a Slider that steers the generation to be more aligned with using conditional variable $c+$ instead of $c-$. Such indirect optimization and finetuning based on conditional variables may benefit from multiple timesteps to achieve faster convergence in optimization and better performance in editing.\n\n> **Q4: For more open-domain diffusion models like Stable Diffusion, can the edit direction be applied to another subject? For instance, in Figure 1, you transfer human opening eyes to different images. Would this transfer to other domains, such as an animal?**\n\n**A4:** Thanks for the interesting question, and we would like to discuss it further.\n1. The transferability has a high success rate for unconditional diffusion, which matches our theoretical analysis for unconditional diffusion models. However, it’s currently hard to transfer edit directions for more open-domain T2I diffusion models as we have experimented. \n2. It is potentially because the feature space of these diffusion models is more complicated correlating with various text prompts. Such feature space does not align with the assumptions in the theoretical analysis for unconditional diffusion models.\n3. The modeling of feature space in T2I diffusion models is still an open and interesting question in the area. The exploration of transferability in these models requires further studies that may involve conditional text variables, noisy images, and various other non-trivial aspects.\n\n> **Q5: A visualization of all the edit directions discovered via SVD would be quite interesting. Are they all semantically meaningful?**\n\n**A5:** Thanks for proposing the interesting question. We have attached the identified editing directions for different regions of interest in **Figure C of the global response PDF**. These editing directions are selected from CelebA, FFHQ, and AFHQ. \n1. We observe that the editing directions demonstrate semantics correspondences to the region of interest. We also notice for these datasets, the position of objects is biased in the center of images, which benefits the transferability of editing directions. \n2. However, from our observation, the transferability is robust to gender difference, facial feature shape difference, and moderate position difference, as presented in Figure 1(b) of the main paper and **Figure B of the global response PDF**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nVu45mRxmQ", "reviewer": "Reviewer_hJrH", "summary": "Even though there have been many research papers on conditional Diffusion Models, sampling-based disentangling method remains a challenge. The authors empirically/theoretically show that PMP is locally linear and the singular vectors of the gradient of PMP are in low-rank space.  Based this, the authors propose LOCO Edit that can benefit homogeneity, composability and linearity in semantic controls. Empirical studies are provided to prove the effectiveness of the method.", "questions": "1. (Fig. 2) \n- Can the results be generalized? It seems like only a few x_T’s are used, and only one prompt is used (as mentioned in Appendix D.1) \n- fig2(a) The authors mentioned in L79 that “rank (A) denotes the numerical rank of A”.  However, in Eq. 10 and Fig. 2,  it is shown as rank ratio. First of all, what is the rank ratio? and how Eq. 10 can be interpreted? Please elaborate them.\n- fig2(b) To my understanding, $f$ and $l$ (in L146) seem to be a sort of the denoised images. Why not directly computing their distance? It would be more natural measure for the statement $f \\sim l$ in L142.\n\n2. (L80) the x_0 sampled from p_data might need to be the input to get x_t, not the output of the posterior distribution.\n\n3. (L37) What are the various other unexplored aspects?\n\n4. (Fig. 4) What are the undirected t2i edit? If text is not given, why is it called as t2i edit?\n\n5. (L159-165) Are the three properties shown to be true in the experiments? For example, what about composing multiple singular vectors?", "limitations": "1. (L157) and (Alg. 1) The authors mentioned that the proposed method benefits the one-step editing while the actual algorithm needs to sample by DDIM twice (DDIM inv and DDIM).\n\n2. (L189-L190) I think there would have been many papers regarding local editing based on Diffusion Models such as Blended diffusion [1].  \n\n\n[1] Blended Diffusion: Text-driven Editing of Natural Images, CVPR’22", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed idea is theoretically interesting.\n- Introduction is well organized.", "weaknesses": "1. Literature review of the recent related works\n2. Lack of baselines and comparisons with other methods (Please add comparisons with other local/global editing papers.)\n3. Uninterpretable semantic changes. For example, in Fig. 5, color, light angle, tower architecture are not controllable. Is it red? Blue? What is the meaning of color? It would be more interesting if the authors can add some interpretable ways to control the semantics (e.g., using label information).\n4. Almost everything this paper describes is based on PMP while the actual results are not from PMP but from DDIM.\n5. Lack of ablation studies; null space projection\n6. The metrics used in Fig. 2 are not intuitive and descriptions are not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's constructive feedback that helps improve the quality of our work. Below, we address the reviewer's major concerns and clarify potential misunderstandings in some parts of our work. And we will incooperate those valuable points into our final version.\n>**Q1:\n>Literature review of related works.   \n       (L189-190) I think ... such as Blended diffusion.** \n       \n**A1:** \n1. First, a detailed discussion of related works is provided in Appendix B for space limitations. There, we thoroughly review the existing literature on (1) semantic latent spaces of diffusion models, (2) global image editing in unconditional diffusion models, and (3) local/global image editing in conditional diffusion models.  \n2. Several existing methods address local editing in conditional diffusion models, such as CLIP-guided diffusion in BlendedDiffusion and classifier-free guidance diffusion in SEGA [22]. However, local editing remains challenging for purely unconditional diffusion models. Thanks the reviewer. We will clarify this point in lines 189-190 and include BlendedDiffusion in related works.\n\n>**Q2: Lack of baselines and comparisons...**\n\n**A2:** Thanks to the suggestions, we conduct comparisons with BlendedDiffusion and NoiseCLR [1], with details in **Q&A 1 of the global response.** \n\n>**Q3: Lack of ablation studies; null space projection**\n\n**A3:** We showed the role of nullspace projection in Fig. 8 in Appendix, and refer the summary of ablation studies to **Q&A 2 in the global response**. \n\n>**Q4: Uninterpretable semantic change…**\n\n**A4: These are great points, but we want to give further clarification:**\n1. **Unsupervised vs supervised edit.** Unsupervised edit finds edit directions without any label or text prompt such as Pullback [25] and NoiseCLR, while supervised edit utilizes labels or text supervision such as BlendedDiffusion and SEGA [22]. Both are important research directions. Our method mostly focuses on unsupervised edit (Fig. 1 and 5), but can be extended to supervised edit with a target text prompt as shown in Fig. 4(b).   \n2. **Interpretation of semantic changes.**  Even in unsupervised edit, the semantic change directions can be interpreted only after being identified. For example, in Fig. 5, these directions indeed have semantic meanings: colors are changed from white to red, tower architecture are changed from simple to complex. We will make the descriptions more concrete. \n\n>**Q5: Misunderstandings on our paper.**\n>1. The actual results are not from PMP but from DDIM  \n>2. (L80) The $x_0$ sampled from $p_{data}$ might need to be the input to get $x_t$, not the output of the posterior distribution  \n>3. (L157) The proposed method is not one-step editing...  \n>4. (L159-165) Are the three properties... \n>5. What are the undirected t2i edit...\n\n**A5:** We want to clarify these misunderstandings:  \n1. **PMP and DDIM.**    We use PMP at time step $t$ to find the direction $v_p$ and edit $x_t$ as $x_t + \\lambda v_p$. In contrast, DDIM-Inv is used to get $x_t$ from $x_0$, and DDIM is to generate the edited image by denoising $x_t + \\lambda v_p$. We use both PMP and DDIM, whereas PMP takes the most important role in finding the edit direction $v_p$.\n2. **Clarification on L80.** $\\mathbb{E}_{x_0 \\sim p\\_{data}(x)}[x_0|x_t]$ is the expectation of $x_0$ given the observed $x_t$ and the prior distribution $p\\_{data}(x)$. Its output is indeed **not** $x_0$. \n3. **One-step editing.** In [25], “one step” means only changing $x_t$ at one timestep $t$ for image editing. In comparison, as mentioned by reviewer 9wnQ, NoiseCLR requires edits across multiple timesteps.  \n4. **Experimental verification of three properties.** These properties have been demonstrated in Fig. 1 (b,c,d) and discussed in L 64 - 69. Particularly, in Fig. 1\\(c\\), the disentangled editing directions are composable: linear combination of two editing directions result in changes of two attributes simultaneously.\n5. **Undirected and text-directed T2I edit.**\n\t Suppose T2I diffusion models generate an image based on text prompt $c_o$. \n\t- **In Undirected T2I edit,** image is edited without additional editing prompt for the target edit direction. \n\t-  **In Directed T2I edit,** an extra editing prompt $c_e$ is provided. So it is called “text-directed T2I edit”. \n\n>**Q6: For Fig. 2:**\n>1. Whether the results be generalized to more data samples and prompts  \n>2. Define rank ratio and interpret Eq. 10  \n>3. Why use the norm ratio\n\n**A6:** Thanks for raising these points.\n1. We test 15 more text prompts. As shown in **Fig. F of the global response PDF**.  Results demonstrate the generalizability of more prompts and initial noises.\n2. The rank ratio is defined in Line 144, which is the ratio between the numerical rank and the ambient dimension $d$. Eq.10 can be interpreted as finding the smallest $r$ such that: compared to the top-$r$ largest singular values of the given Jacobian, the remaining singular values are smaller than a threshold and can be neglected.   \n3. The norm ratio $||l - f||_2/||l|_2$ measures relative distance to better reflect linearlity. If $l$ and $f$ are apart from each other but $||l||_2$ and $||f||_2$ are too small, the absolute norm differences $||l - f||_2$ may be too small to reflect violations of linearity, but the norm ratio can.\n\n\n>**Q7: (L37) What are the various other unexplored aspects?**\n\n**A7:** Specifically, these aspects include but not limited to (1) whether diffusion models have semantic spaces; (2) what features lie in the semantic spaces; (3) theoretically justify those semantic features; (4) utilize those features unsupervised local edit in unconditional diffusion models.  In this work, we looked into all these aspects and proposed LOCO edit that tackles many of the questions raised. We will make the discussion clearer.\n\n[1] Yusuf et al., \"Noiseclr: A contrastive learning approach for unsupervised discovery of interpretable directions in diffusion models.\" CVPR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for carefully reviewing our work with constructive and positive feedback. Most reviewers find our empirical observation “interesting”, “well-validated”, “important” (eLFT, h6gb, 9wnQ), our theoretical analysis “useful”, “strong”, (h6gb, pgJR), our edit method “novel”, “quite principled”, “well-motivated”, “insightful” (eLFT, pgJR, 9wnQ, hJrH), and our presentation “well-written”, “sound”, “well-organized” and “clear” (h6gb, pgJR, hJrH).

**Summary of our results.** Our work introduced a simple yet effective method for local edit in diffusion models by exploring (i) the linearity of the posterior mean predictor (PMP), and (ii) low-rankness of its Jacobian. The advantages of our method can be highlighted as follows:
-   **One-step, local editing.** To our knowledge, this is the first work on image editing using unconditional diffusion models that allow for one-step, localized editing. In contrast, most works require editing on multiple (all) timesteps and only perform global edit.
-   **An intuitive and theoretically grounded approach.** Our method is highly interpretable, leveraging the benign properties of PMP. The identified properties are well supported by empirical observation (Fig, 2 in paper) and theoretical justifications in Section 4. In contrast, most previous methods are heuristic and lack theoretical justifications.

**Addressing reviewers’ major concerns.** We appreciate the reviewer's comments on our limited experiments. During the revision, we addressed reviewers’ concerns with more comprehensive results, comparisons, and ablation studies as follows. **They are presented in Fig. A-F and Tab. A of the global response PDF**, and will be included in the revised paper, with code released.
>**Q1: More evaluation of our method.**

**A1:**
 1.  **More qualitative results:** We add more qualitative results on different datasets, shown in Fig. A of the attached PDF.
 2.  **More quantitative metrics:** The quantitative comparison is extended in Tab. A of the attached PDF using the additional metrics:
	 - *LPIPS* [4] and *SSIM* [5] to measure the consistency between edited and original images.
	 - *Learning time*, *Transfer Edit Time* to measure the time each method requires to find the editing direction and apply it to edit an image.
     - *#Images for Learning* to measure the number of images used to find directions.
     - *One-step edit*, *No Additional Supervision*, *Theoretically Grounded*, and *Localized Edit* are different properties for each editing method.
3.  **More qualitative comparisons:** We also extend the qualitative comparison in Fig. E to showcase our method's strong local editing capability.
4.  **Detailed comparison with more baselines:** We compare our work with two other studies: NoiseCLR [1] and BlendedDiffusion [2], together with the previous baselines. We discuss key observations as follows.
	-  **Local edit ability:** Tab. A shows LOCO achieves the best Local Edit Success Rate. For LPIPS and SSIM, our method performs better than global edit methods but worse than BlendedDiffusion. However, BlendedDiffusion sometimes fails the edit within the masks (as visualized in Fig. F, rows 1, 3, 4, and 5). We discuss the potential causes from CLIP bias in the last point below. Other methods like NoiseCLR find semantic direction more globally, such as style and race, leading to worse performance in Local Edit Success Rate, LPIPS, and SSIM for localized edits.
    
	-  **Efficiency and transferability.** First, LOCO requires less learning time than most other methods, and the learning needs only a single time step and a single image. Moreover, LOCO is highly transferable, having the highest Transfer Success Rate in Tab. A. In contrast, BlendedDiffusion can't transfer and requires optimization for each image. While NoiseCLR excels at open-domain transfer, it performs worse than LOCO in closed-domain transfer (e.g., on the CelebA dataset). Other methods exhibit even weaker transferability.
	-  **Theoretically grounded and supervision-free.** LOCO is theoretically grounded. Besides, it is supervision-free, thus integrating no biases from other modules such as CLIP. [3] shows CLIP sometimes can't capture detailed semantics such as color. We observe failures to capture detailed semantics in methods that utilize CLIP guidance such as BlendedDiffusion and Asyrp in Fig. E.
>**Q2: More ablation studies.**

**A2:**
1.  **Noise levels (i.e., time steps).** We test different noise levels and show results in Fig. 7. Results for more noise levels are in Fig. D with key observations: (a) Edit at large noise level (i.e., large time step) perform coarse changes while small noise level performs finer edit; (b) LOCO applies to a large range of noise levels ([0.2T, 0.7T]) for precise edit.
2.  **Perturbation strength.** The linearity concerning edit strengths is visualized in Fig. 1d, and detailed ablation results of perturbation strength at 0.6T are in Fig. D as an example, with key observations: LOCO applies to a generally wide range of perturbation strengths ([-15, 15]) to achieve localized edit.
3.  **Nullspace projection and ranks.** Ablabtion study on nullspace projection is in Fig. 8, with key observations: (a) the local edit ability with no nullspace projection is weaker than that with nullspace projection; (b) when conducting nullspace projection, an effective low-rank estimation with $r=5$ can achieve good local edit results.

[1] Yusuf et al., Noiseclr: A contrastive learning approach for unsupervised discovery of interpretable directions in diffusion models. CVPR 2024.

[2] Omri et al., Blended diffusion for text-driven editing of natural images. CVPR 2022.

[3] Shengbang et al., Eyes wide shut? exploring the visual shortcomings of multimodal llms. CVPR 2024

[4] Richard et al. The unreasonable effectiveness of deep features as a perceptual metric. CVPR 2018.

[5] Zhou et al., Image Quality Assessment: From Error Visibility to Structural Similarity. 2004

Meta-review: This paper proposes a diffusion image editing framework called LOw-rank COntrollable edit (LOCO Edit) based on two observations: 1) the learned posterior mean predictor (PMP) is locally linear during the middle timesteps during denoising, and 2) the singular vectors of the PMP's Jacobian lie in low-dimensional semantic subspaces. These observations are backed both empirically and theoretically. To conduct LOCO edit, an image is inverted with DDIM inversion to an intermediate timestep. SVD is conducted on the Jacobian of the PMP w.r.t. to this noisy image in order to discover semantic directions. Furthermore, a mask can be applied along with a null space projection in order to achieve more disentangled and localized edits. A variant of LOCO edit is also proposed to enable text direction.

The reviewers found the observations in terms of local linearity of the denoiser interesting and well-validated across architectures and dataset. They found the method novel, well-motivated, and appreciated its support on theoretical arguments, finding the discovery of homogeneity and transferability of the editing direction insightful. They also pointed out that LOCO tackles an important challenge in diffusion models: achieving precise and disentangled image editing without additional training, and found the paper to be well written.

There were some concerns about missing citations and comparisons to previous works and, most importantly, limited experimental validation. While the paper considers various architectures and performs various ablations, there were limited examples of the success of the edits. Various questions of detail were also raised. The authors addressed these issues in their rebuttal, providing several new examples of edits. While these are still only for object centric images, they show a wealth of image modifications that can be of interest for applications. In the end, all reviewers were positive towards the paper, and the ACs consur that the paper merits publication.
Decision: Accept (poster)