DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Models
Authors: Zhengyang Yu
Zhaoyuan Yang
Jing Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion
Personalization
Editing
Primary area: generative_models
TL;DR: A plug-in method for improved editability of any existing T2I personalization baseline

Abstract: Recent text-to-image (T2I) personalization methods have shown great premise in teaching a diffusion model user-specified concepts given a few images for reusing the acquired concepts in a novel context. With massive efforts being dedicated to personalized generation, a promising extension is personalized editing, namely to edit an image using personalized concepts, which can provide more precise guidance signal than traditional textual guidance. To address this, one straightforward solution is to incorporate a personalized diffusion model with a text-driven editing framework. However, such solution often shows unsatisfactory editability on the source image. To address this, we propose DreamSteerer, a plug-in method for augmenting existing T2I personalization methods. Specifically, we enhance the source image conditioned editability of a personalized diffusion model via a novel Editability Driven Score Distillation (EDSD) objective. Moreover, we identify a mode trapping issue with EDSD, and propose a mode shifting regularization with spatial feature guided sampling to avoid such issue. We further employ two key modifications on the Delta Denoising Score framework that enable high-fidelity local editing with personalized concepts. Extensive experiments validate that DreamSteerer can significantly improve the editability of several T2I personalization baselines while being computationally efficient.

DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Models
Zhengyang Yu!
Zhaoyuan Yang²
Jing Zhang
Australian National University'
GE Research
(zhengyang. yu, jing. zhang) @ @anu.edu.au
zhaoyuan yangage.. com
Abstract
Recent text-to-image personalization methods have shown great promise in teach- ing a diffusion model user-specified concepts given few images for reusing the acquired concepts in novel context. With massive efforts being dedicated to personalized generation, a promising extension personalized editing, namely to edit an image using personalized concepts, which can provide more precise guidance signal than traditional textual guidance. address this, a straightforward solution is to incorporate personalized diffusion model with text-driven editing framework. However, such a solution often shows unsatisfactory editability on the source image. To address this, we propose DreamSteerer, a plug-in method for augmenting existing T2I personalization methods. Specifically, we enhance the source image conditioned editability of personalized diffusion model via a novel Editability Driven Score Distillation (EDSD) objective. Moreover, we identify mode trapping issue with EDSD, and propose mode shifting regular- ization with spatial feature guided sampling to avoid such an issue. We further employ two key modifications the Delta Denoising Score framework that en- able high-fidelity local editing with personalized concepts. Extensive experiments validate that DreamSteerer can significantly improve the editability of several T2I personalization baselines while being computationally efficient. Project page:
Introduction
Text-to-Image Diffusion Probabilistic Models (T2I DPMs) [58 61 have revolutionized novel content creation due to their superior capacity in both sample fidelity and mode coverage, as well as their flexibility in achieving effortless concept composition 44 and user-friendly control [35 Despite the success of these models, the limited expressiveness natural language may lead to ambiguity in real-world scenarios where users demand greater specificity and engagement in the creation process.
This challenge has sparked extensive research T2I Personulization I0 16 62
Specifically, personalization is the process of teaching T21 DPMs novel visual concept with few (usually 3-8) reference images by fine-tuning the pre-trained model parameters. After personalization, the personal concept is linked to rare token in the text encoder dictionary, e.g., "[my_dog]" dog]", enabling flexible reuse of the visual concept in new contexts, c.g., "a photo of suit". Despite the success of these models, the majority of previous works have been focusing personalized generation, emphasizing the preservation of concept identity across varying textual conditions. In this work, explore natural extension these methods perform image editing with acquired concepts, namely personalized editing, which can be promising in enabling higher level
wearing an astronaut
Work was partially done while Zhengyang Yu was an intern at GE Research.
Correspondigg author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Reference
Source
Textual Inversion
Ours
DreamBooth
Ours
Custom Diffusien
Ours
*A photo of tall eat (cat_statue
photo cat [brown_cat] standing next mirror"
"A painting of [black_cat] and rebbit [black_cat] the grass"
"A painting of dog [fat__dapt wearing armor"
"A painting of a turtle ceertoise_plusyye swimming the sea"
"A photo of temple. [wooden_barme showstorm
Figure 1: DreamStecrer enables efficient editability enhancement for a source image with any existing T2I personalization models. leading significantly improved editing fidelity in various challenging scenarios. When the structural difference between source and reference images are significant, it can naturally adapt to the source while maintaining the appearance learned from the personal concept.
control over the editing direction than traditional text-driven editing frameworks 21 primary goal synthesize high-fidelity image that aligns the appearance and content of the target concept, as well as the structural layout and background of the source image without blindly resorting copying learned reference images. straightforward solution this problem is incorporate personalized diffusion model into an existing text-driven editing model, by which personalized editing appears be trivial via swapping source subject token in the source prompt by the special token, e.g., "a photo (dog - [my_ [my_dog]) wearing an astronaut suit". However, such a naïve incorporation often leads to severe distortion or failure in natural adaptation the source image layout. We attribute the essential causes for such failure to the limited scope of reference images in personalization. Under lack of data diversity, existing personalization methods are prone to collapsing into the patterns of reference images [71 and entangling subject-relevant and subject-irrelevan information [9] This causes a significant loss the model's prior knowledge related to the source category, resulting in poor editability [48] 60 in new context. Additionally, the stricter demands for maintaining
34 55
The
structural layout (54 72 and preserving subject-irrelevant information [12 21 necessitate a higher level of editability than personalized generation. In more challenging editing scenarios, certain level of extrapolation on the attributes of personalized concepts may be required, e.g. the editing process may require significant change in subject structure, pose style to achieve high-fidelity

---

editing result on the source image. Without information on the source image during personalization, existing personalization methods can hardly be guaranteed be editable on arbitrary source images.
To overcome these challenges, in this paper, we propose the first approach that enhances the source image conditioned editability using existing personalized T21 DPMs (DreamSteerer). Inspired by the one-step Bayes optimal denoising using Tweedie's ' formula [15 59 with DPMs [3] 7 47 69], and
the probabilistic score distillation sampling [22 57 74,, we formulate editability enhancement on the source image as a novel score distillation objective, dubbed Editability Driven Score Distillation (EDSD). Then we identify the existence of mode trapping issue when directly optimizing EDSD. Inspired by recent success in zero-shot semantic correspondence [ 6 80] based on the strong spatial awareness of UNet attention features, we propose a spatial guided sampling strategy that produces a regularization set which alleviates the mode trapping issue via shifting the mode of personalized DPMs distribution. We further make two major modifications on the Delta Denoising Score (DDS) [2 framework for valid adaptation from text-driven editing to personalized editing.
We summarize our main contributions as: 1) we identify and analyze the lack of editability in existing T21 personalization methods for editing real-world images, 2) we propose the DreamSteerer framework, plug-in method that is compatible with arbitrary personalization baselines and requires only a small number of fine-tuning steps (~~ 10) to achieve significant improvement in editability on a source image, 3) we validate the effectiveness of DreamSteerer on different baselines []6 39 62
demonstrating its efficacy, especially in challenging personalized editing scenarios such as significant structural gaps and data-hungry cases (Fig. T.. See Fig. 3 for an overall framework.
2 Related Work
T2I Personalization. Building upon the success of T2I DPMs 52
recent works have
shown the promise of personalized T2I synthesis [16] 19 39 62). Textual Inversion [16] encapsulates personalized concepts concepts by word embedding optimization, which is further improved by more expressive
representation spaces 22] 773. DreamBooth (2) fine-tunes the full model parameters of a pre-trained
DPM that conditions on a rare word token. Efficient fine-tuning methods 9 18
45 71 77 have
been introduced for improving efficiency and generalizability. Custom Diffusion [1) optimizes
both a word embedding as well as the UÑet cross -ateentnn key and value projections for improved these three different types of models.
compositionality. We explore enhancement in editability
More recent studies in encoder-based personalization [26 63 65 79 propose new training paradigms to condition Diffusion Model single multiple input images. Although these works can enable faster inference, they necessitate extensive pre- tre-training and typically limit application to particular domains. New forms of conditioning, objectives adaptors are proposed for different purposes such as stylization (23, 29 51, improved identity preservation [13 43], composability [37 [56 82 85) or
generation fidelity (50) Unlike these works, we focus on addressing an inherently different task of improving the editability of personalized Diffusion Models.
Image Editing, The conditioning mechanisms [4] 24 and inversion techniques 49
68 have
enabled DPMs to achieve image editing. Earlier works [32 34 40) relied on heavy optimization and often failed in local editing, while other works [4, achieved this using user-provided mask guidance. Prompt-to-Promp [2 achieves both local and global editing using cross-attention map injection without requiring extra guidance. Similarly, PnP (72) proposes employing UNet attention features to achieve image-to-imad mutual self attenttnn to manipulate subject pose or view. Despite the success of these works, most of them are designed only for text-driven editing, leaving image editing with personalized concepts still under-explored. Unlike preliminary attempts [10 [7] that rely on specific baselines, we consider editing with visual concepts acquired by an arbitrary existing personalization method.
translation. Masactrl 6 extends
Commp-to-Prompp [2]] with
Score Distillation. Pooled et al. (57 proposes a novel Score Distillation Sampling (SDS) method that can synthesize 3D assets without requiring 3D training data. For improving sample fidelity and mode coverage of SDS, VSD 74) utilizes LoRA adapters to model a Wasserstein gradient flow, NFSD (31 employs negative prompts, CSD (78) introduces Classifier Guidance. SDS has been improved and extended for different downstream purposes, such as conditional generation of different
modality assets [28] 36), text-driven visual editing [22, 67] and text-aligned generation (3)
3

---

3 Preliminaries
Diffusion Probabilistic Models (DPMs). With an input image latent state X E X, a corresponding text prompt y E ) and a diffusion process defined as (x, xo) == N (x:; /a,Xu, (1 oz)
where a; represents the forward process variance at time t and x, is the noised latent state of the input Xo, a diffusion probabilistic model parameterized by d can be trained using the denoising objective:
(X y; Φ)
[lles y, -1
(I)
T21 Personalization with DPMs. Given a diffusion model 60o pre-trained for Text-to-Image (T2I) generation using Eq. and a small set of image latent states and prompt pairs D yrroff )n Jn which encapsulate the personalized concepts the user wishes the diffusion model to capture.
{( ,
general approach of T21 personalization methods 16 62 is to fine-tune a subset of the source model parameters Diet by optimizing the denoising objective as:
arg min (gget y CDPM j ()
where à is initialized with the pre-trained weights Φn- The text prompt yæee takes the form of "a photo of a [S]", where the placeholder [S] corresponds to a new word embedding representing the specific subject. At inference time, the fine-tuned model co can be used to generate creative images of the specific subject novel contexts, such as "a photo of a [S] sitting next to a mirror".
Score Distillation. Score Distillation is mechanism for sampling from source diffusion model under predefined constraints, achieved by performing probability density distillation from the source diffusion model Eg to parameterized differentiable function. In the context of image domains, represent this differentiable function as z0)), which renders the image with parameter 0. SDS (57) is the pioneering approach in score distillation, which optimizes z=() through mode-seeking process. Specifically, given a target prompt y that describes the desired edit, the gradient of the distillation objective w.r.t. parameter 0 is computed as follows:
0x(00) ae
V V g) = EE..U(..).. w(t) (co (co (2,()),
(3)
where (0) is noised latent state of (0) at time step t, and (t) is constant determined by the forward process variance, SDS is known to be a mode-seeking process and suffers from issues such as low diversity, over-saturation, and over-smoothness (74), which lead to sub-optimal performance on downstream tasks. Thus, various SDS variants have been proposed to enhance performance.
DDS 22 extends SDS to handle text-driven image editing. Specifically, given source image latent state xBrC a source prompt yare that is aligned with x and a target prompt y that describes the desired edit, the parameter 0 is initialized using the source image such that (x() xaro, For certain differentiable function z(-), 0 can then be updated using the following delta score direction:
dx(())
VeLops ( = -E.2u(1))) (t) (Eo (I, t) - c (xa's, y t))
80
To simplify the notations, we incorporate all constants associated with score computations into (t) for the remainder of the paper.
From text-driven editing to personalized editing Existing text-driven editing works like DDS
generate high-quality output for image editing with open-world vocabulary; however, their capability of personalized concepts editing remains underexplored. In this work, we focus on bridging such gap. Given a diffusion model e personalized via Eq. 2 and the latent state a source image x with aligned source prompt y (e.g., "a photo of cat sitting next to mirror"), our objective is to edit x² using the personalized concept [s] captured by eg. For example, an edit might "a photo of (cat [s)) sitting next to mirror," where [s] represents your childhood pet cat. We define the target prompt as ref The desiderata for personalized editing are concluded as follows

---

the overall structure of edited image should align with the source image.
the edited part should capture the appearance and content of the personal subject,
the instruction-irrelevaar part should be preserved as much as possible.
3.1 Preliminary Editing with Existing Personalization Baseline
We explore preliminary solution to personalized editing by employing SDS in Eq. and DDS in
Eq Eq.Abased based on the personalized DPM Eg, where the differentiable function initialized by the source image latent state, i.e., x00 xi and can be updated with VoL (0, (0)), yref). However, this
straightforward solution yields unsatisfactory performance as shown in Fig. 3 (b)- Specifically, the editing result with SDS suffers from the same low fidelity issues as mentioned by prior work [22 (74).
With DDS, although the overall layout preservation is improved. there still reveals a lack of editability. Additionally, we observe that DDS leads severe bias of certain attributes on the source class, leading to unsatisfactory editing results.
Source Score Bias Correction. We suspect that such bias caused by the distribution shift of co during personalization. As shown in Fig. 2 DreamBooth trained for the concept "plushie_tor tortoise" shows significant bias towards the yellow color in the corresponding source class "tortoise". which leads to the incorrect color of the edited image as shown in Fig. (c). To address this, we use the pre-trained diffusion model 6gg to conduct the source score prediction in Eq. 4 yielding the modified delta score named as DDS-S,i...
VeLoDsS = ELe wi)) (co (z.( ,t) coo (xc. t)) Ox(()
(5)
00
(a) Generations of 'tortoise' souree model. (b) Generations of 'tortoise" DrcamBooth.
Figure Source class bias of DreamBooth trained for "plushie_toreee tortoise".
As shown in Fig. (d), the modified delta score effectively alleviate the source score bias issue.
4
The DreamSteerer Method
4.1 Editability Driven Score Distillation (EDSD)
Direct incorporation between existing personalized model and DDS-S in Eq. Bjeelvelseees the bias caused by inaccurate source score. However, as shown in (5g and (d), a deficiency editability still persists, evident from the structural misalignment between source and edited images. We hypothesize that this issue stems from the misalignment between score estimations of the pre-trained and personalized models, underscoring the necessity for adjustments the personalized model. An intriguing direction is to adjust the personalized model such that the objective of the DDS-S is further optimized. For this purpose, we first recover the loss from the gradient form of the DDS-S in 5 as CDDS-S = Er,, Ill (Ir(0), re ,t) Edu (xirc, " t) [12]. For the purpose of editability enhancement, instead of performing score distillation w.r.t. the rendering parameter 6, we perform score distillation w.r.t. the personalized model parameters ø. Specifically, we first introduce perturbation on the souree latent state using the personalized DPM co as follows:
(x) := SIC /I at (Es src (t)
(6)
Single-step denoising direction
which is equivalent to the noised latent state of Tweedie's estimation model es w.r.t. x5 (Sec. B
1.5 of the personalized
5

---

photo turtle
swimming the
paimiing
DOM Inversion MCT
X
[forTone_
Eda
Add nolse
photo of
[tortoisk_ swmming the sea.
photo [torteise__ the
LED
X
Spatial guidance
E
noise
Denoising X(T 1)
min,, Agru.
① Spatial Feature Guided Sampling
b Mode Shitling Repularizatin
a Ediabilit Driven Score Distilation
Figure 3: Overall framework of DreamSteerer (the gradient flows are illustrated with dashed lines).
Then, we define the editability loss as LED = Ee,ellle ( rf cdo (x, which is similar to the CDsS-S fons-s but with different parameters to optimize. reduce the computational cost, with slight abuse of notation, we use VoLED 0Cpn ae da
des (0a(d), ytef, yref.) 0i(())
VOLED = Er,e (() (co (oe(0), y .t) Cáo (xxre, , 1))
(7)
(z)
dΦ
where all constants are absorbed in (t). Intuitively, the perturbation introduced by the single denoising step in caan be interpreted small editing step x by cs. Therefore, optimizing CED can be understood as distilling information from the source model Edu into the personalized diffusion model Eg through (Ø). Specifically, without bias to the reference dataset, the source model Ego has better editability on the source image x, than the personalized model Es, thus its score estimation, i.e., the noising prediction, tends to be more accurate. As co. trained on D via personalization, in order for the personalized model c to achieve similar score estimation, the "edited" (τ) would be prone to have characteristics similar to the personal subject without losing the overall fidelity, through which enhancement in editability can be achieved. In Eq. / with parameter sharing, s naturally serves as the score estimator for (o))), without requiring additional training. The detailed gradient flow is shown in Fig. 3
4.2
Mode Shifting regularization with spatial feature guided sampling.
The mode trapping issue of EDSD. Directly optimizing the personalized model parameter via Eq. | results in a peculiar characteristic observed in the edited and generated images, specifically generated and edited results show patterns that fall between the reference images Dief and the source image xare As shown in Fig. 4 (f), when using a silver cat image as source and brown cat as reference subject, the generated images reveal a hybrid appearance compared to the source model generations in Fig. 4 (c), showcasing features of both silver and brown cats. This observation suggests that EDSD has induced Eg to collapse to a trivial trapping point between the modes of the source image and the reference images.
Spatial feature guided sampling. To avoid such mode trapping issue and maintain the concept of the personal subject, we regularize EDSD by jointly training the model on a set of personal subject images. Instead of accessing the reference dataset, we use images generated by the personalized model co(. y We find that guiding the generated samples to have a structural layout akin to the source image xir not only serves as an effective regularization but also shifts the model distribution p-(- yff towards more editable region than the original mode centered around reference images.
show that the Self Self-Attention (SA) features of the T2I DPMs are embedded with detailed spatial information, thus having strong sense of spatial layout that allows building inter-image semantic correspondence using these features [ 80] Motivated by these findings, we propose spatial feature guided sampling strategy using these features from the source image.
Recent works 66 72
As shown in Fig. we begin by performing the DDIM inversion 68 on the source image using the pre-trained DPM Egg At each time step t, for the f-th SA layer from Edg. intermediate feature

---

(a)
(b) Reference
(c) wo mode shifting
(d) mode shifting
Source
(e)
(n) mode shifting
(g) wt mode shifting
(h) Guided samples
Baseline
Figure 4: The effect of different regularization strategies the editing and generation results of DreamBooth baseline. The source prompt is photo of a cat sitting next to mirror".
vector dby (x,) is extracted, which is further projected linearly into queries Q = f& (sle (xx)), keys KJ fk ( (xx)) and values V ft (whe (xx)). The final output SA feature computed via h Softmax (of (K?)* /VC)) ve, where C is the channel size of the keys and queries. The output spatial features fi E RRSxxx xo* are encoded with localized semantic information, where S is the number of spatial locations in the f-th layer. These features are cached and serve as guidance for sampling with the personalized model ea(- y=ef). Specifically, after running DDIM inversion with the source model 6 (- yarc), we obtain the inverted initial latent state of the source image xare starting from which the reverse diffusion process is conducted using the personalized model <(: zef). At each time step i, the same set of spatial features (h() are extracted, and we aim to condition the model prediction on the source image x5 by matching the pairs hf and h, at corresponding spatial locations. For this purpose, following previous work [3 84] we compute the patchwise contrastive loss between the spatial features follows
LPatchNCF (h,, h.)
LNCF (he lifi", (4f",,,.5) R...0N0).
(8)
==1
where we treat the spatial feature vectors from the same spatial location as positive pairs, those from different spatial locations are treated as negative pairs and LNCE (h, ht h
exp(h-h+/r)
0. With Eq. 8 for the genera- tion with the personalized model at time step we model the likelihood of a noisy latent state X; matching the structural layout of the source image at time step t as Xx, log bog | x;) s
- -logl ex EA /xp(h.h//r) with some coefficient
Vx, LPateaNCC (h,, h.). Following the Classifier Guidance 14 we modify the noise prediction of Classifier- Guidance follows
X1. y ref fng) = c clg (xc, drro CraechNCE (nr. h:
(9)
o. V.V. Vx. log Po (x, y Aogg (h, xx)]
where À is weight that balances two scores. Using such spatial feature guided sampling strategy. we can sample set of images gd As shown Fig. ((). these images are prone have a similar structure to the source image without losing the appearance of the personal concept [s].
7

---

(a) Source Reference
(b) SDS
(c) DDS
(d) DDS-S
(o) DDS-S+ EDSD
DDS-S+EDSD
(g) DDS-SM-EDSD Mode Shifting
Mode Shitting
Figure 5: Illustration on the effect of the proposed components on editing with DreamBooth baseline (Ist row shows the editing results; 2nd row shows the editing directions, where brown means zero).
Table Comparison with different baselines (DreamSteerer uses the same model as baseline).
Michad
CLP B52
CI CP)
L/14
1IPSS
SSIM
MS-SSIM
Togig
Musiq
LIQE
Teauel Inresioo
10
0.785
0.746
0.133
0.181
0.523
0.580
577
68.4
4.23
DreamStrerte
0.788 (1.48)
0.753 (1)9%)
0.115
(-13.4%)
B.167 (-8.2%)
0,833 (+1.3%)
0,898 (+2.0%)
593 (+2.8%)
70.1 (+2.5%)
4,39 (138%)
DecmnBouth
(+1.8%)
Custom Diftusoon
59
0.781
0.745
0,808
.991
4.24
DreamStrerre
0.746
(8899))
(+1.5%)
0,83
71.5
4,41 (+4.5)
Mode shifting regularization. With the set of guided samples Ega, a mode shifting regularization term is jointly optimized with EDSD as
- 1 CED VaLMs), LMs Exse CDMM (xx5 y y ())
(10)
where is the learning rate. As shown in Fig. 4 (g). the mode trapping issue in Fig. (f) can be avoided with the mode shifting regularization in Eq. [0] where the generated images maintain appearance fidelity comparable to source model generations in Fig. (e); meanwhile, the generations exhibit patterns akin to the source image. e.g., features like the blue collar, the presence "two cats" and subject pose closely resemble those in the source image in Fig. 4 (a), indicating that the mode of (-) has been effectively steered to enhance editability for the source image. Furthermore, this enhancement is evidenced by the noticeable improvement in appearance fidelity of edited images from Fig. (c) to Fig. (d). See Fig. for an overall framework.
4.3 Automatic Subject Masking
Given the final steered personalized model eg after optimization through Eq. 10 although Eq, leads to pleasant target concept alignment, we observe that the subject irrelevant part may not be properly maintained due to the structural layout difference between source and target subjects as shown in Fig. 5) Inspired by recent work [2i 70) showing that the Cross-Attentio (CA) maps concentrate on the relevant regions of the corresponding prompt token, automatically extract subject masks M (xxrc (refer to Sec. | for details) and we define such masked delta score as DDS-SM:
V VDLDSSSMM E,, Mx O (ez v .t) Eso (X,, y
(x)
(11)
00
which better focuses on editing the subject-relevant regions as shown in Fig. 5 (g).
Experiments
Evaluation metrics. In accordance with the desired editing properties discussed in Sec. S} we
evaluate the effectiveness of DreamSteerer from three perspectives: 1) semantic similarity with the reference images using CLIP image similarity with CLIP ViT-B/32 and CLIP ViT-L/14 [58]. 2) perceptual similarity with the source image using LPIPS with AlexNet [8 and VGG (66), 3) structural similarity with the source image using SSIM [76] and MS-SSIM (75]. Additionally, to
validate the editing fidelity of our proposed method, we report the No-Reference Image Quality Assessment (IQA) metries Topiq (8] Musiq (33 and LIQE [8I)

---

Reference
Source
wo EDSD
Ours
Reference
Source
wo mode shilting
Durs
Figure Ablation study on EDSD and Mode Shifting Regularization.
Table 2: Ablation study on EDSD and Mode Shifting, the best and second best results are highlighted.
Mode Shifting
CLIP-I (f)
I.PIPS (4)
Baseline
EDSD
SSIM (1)
MS-SSIM (t)
CLIP1/14
VGG
0.788
0.753
0.115
0.167
0.833
0.898
Textual Inversion 16
X
/ 0.788
0.749
0.132
0.182
0.821
0,880
0.779
0.746
0.090
0.144
0.850
0.922
V
0.796
0.761
0.182
0.229
0.768
0.815
DreamBooth 62
X
V
0.794
0.755
0.200
0.246
0.753
0.792
0.782
0.745
0.134
0.186
0.805
0.869
XX
0.784
0.746
0.182
0.227
0.779
0.833
Custom Diffusion
39
V 0.779
0.737
0.217
0.262
0.748
0.795
0.760
0,730
0.100
0.152
0.841
0.917
Implementation details. We evaluate our plug-in method on three personalization baselines: Textual Inversion [16). DreamBooth (22 and Custom Diffusion (39]. which include the 3 main- stream types of models outlined in Sec. 2 We use the pre-trained checkpoints provided by Dream- Matcher (50), with 16 concepts for each baseline encompassing living, non-living, in-door, and outdoor subjects. For each baseline, 70 random real-world images are used, focusing on the challeng- ing editing scenarios as shown in Fig T For fair comparison, all experiments use DDS-SM (Eq. as the editing method.
Comparison with baseline methods. Table. compares DreamSteerer against baselines [16 39
62). Our work shows clear improvement in all types of metrics, with substantial gains in the perceptual and structural alignment with the source images. Even for challenging editing scenarios (see Fig. [T) such mirror reflections and significant structural changes, where baseline methods often result distortions and unfaithful structure maintenance, our method effectively calibrates these issues to achieve high-fidelity results. Refer to Supp. [F] for more editing results. We find that the automatic metrics not fully reflect the superior performance our method compared to the baselines, particularly in terms of the quality of edited images. Therefore, we conduct user preference study using the same criteria Supp. [ our work preferred by the users by a significant margin against the baselines.
Ablation study. We ablate EDSD and Mode Shifting Regularization to demonstrate their effec- tiveness in our framework. As shown in Table. 2 and Fig. 6 without EDSD, the CLIP-I scores remain comparable to those of the full model. However, the performance in source image alignment deteriorates significantly, indicated by the lower LPIPS and SSIM scores. Without Mode Shifting

---

Reference
Source
DreamBooth
Ours
Custom Diffusion
Ours
Figure 7: Comparison of one-shot performance.
Table 3: Comparison with baselines under one-shot scenario.
CLIP-I (t)
LPIPS (L)
Method
SSIM (t)
MS-SSIM
CLIPB/S
CLIP
Alex
VGG
DreumBooth
0.790
0.717
0,222
0,264
0.742
DreamStecrer
0,801 (+1.4%)
0,748 (+4.3%)
0,160
(-27.9%)
0.212 (-16.7.%) %)
0.781 (+5.2%)
0.847 (+9.0%
Custom Diffusion
39
0.796
0,740
0.155
0.210
0,782
0.856
DreamStecrer
0.799 (+.4 %
0.753 (+188)
0.145
(-6.4%)
0,200
(-4.7%)
0.796 (+1.7
0,873 (+19%)
regularization, exceptionally high structural and perceptual alignment scores are achieved. However, as depicted Fig. 6 this often results in a severe loss of target subject appearance information, reflected in consistently poor CLIP-I scores. Overall, combining EDSD and Mode Shifting Regu- larization achieves the best trade-off between source image alignment and target concept alignment. Fig. provides component analysis, showing that EDSD effectively improves the editability of the baseline Dream Booth model, resulting in higher structural alignment with the source image in terms of the edited results and the editing directions (refer to Supp. [].
One-shot performance. We further evaluate the performance DreamSteerer against baselines under extreme lata-hungry scenario of one-shot personalization. We observe that Textual Inversion, which does not update the Diffusion Model parameters, cannot provide valid editing results under this setting. Therefore, DreamBooth and Custom Diffusion are used. As shown in Table DreamSteerer maintains superior performance under these conditions. Notably. DreamBooth, relying on full fine- tuning, exhibits severe bias towards the pose and structure of the reference image. Despite such strong bias, DreamSteerer improves its performance by a significant margin, as shown in Fig.
More comparisons with existing works. See Sec. for discussion the setting-level differences between our work and existing subject swapping works [17 42). We use a modified Delta Denoising Score the base editing model, this method provides stable editing results with all the personalized models we use. However, DreamSteerer is not restricted to a specific type of editing pipeline. We also evaluate the effectiveness of method as plug-in for personalized editing with Prompt- to-Prompt [21]. See Sec. D a comparison and a discussion on how Trompt-toPProme may be incompatible with certain base personalization models due the limitations of the existing latent state inversion method.
6 Conclusion
In this work, we identify that existing T2I personalization models fail to deliver satisfactory image editing results. Therefore, we present DreamSteerer, an efficient plug-in method designed to enhance the editability of images conditioned on the source image. DreamSteerer fine-tunes the personalization parameters by training novel Editability Driven Score Distillation objective under the constraint of a Mode Shifting regularization term based on spatial feature-guide samples. Through experiments, we show that DreamSteerer can significantly improve the editing fidelity of various existing baselines, particularly in challenging editing cases and data-hungry personalization scenarios. We consider DreamSteerer pivotal bridge from text-driven image editing to personalized image editing.
10

---

7 Acknowledgement
Special thanks Peter Tu for providing with his valuable inputs and support that greatly helped improving our work and for motivating and encourage us on further implementations of our work. This research was, in part, funded by the U.S. Government DARPA ECOLE HR001 12390061 and DARPA TIAMAT of the authors and should not be interpreted as representing the official policies. either expressed or implied, of the U.S. Government.
The views and conclusions contained in this document are those
References
[1] Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch-ENe and Daniel Cohen-Or. Cross-image attention for zero-shot appearance transfer. arXiv preprint 0rXiv:2311.0333 2023.
[2] Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or. A neural space-time representation for text-to-image personalization. ACM Transactions on Graphics (TOG), 42(6);1-10, 2023.
[3] Moab Arar, Andrey Voynov, Amir Hertz, Omri Avrahami, Shlomi Fruchter, Yacl Pritch, Daniel Cohen- Or, and Ariel Shamir. Palp: Prompt aligned personalization of text-to-image models. arXiv preprint arXiv:2401. 2024.
[4] Omri Avrahami, Ohad Fried, and Dani Lischinski. axXiv:220.002779, 2022.
Blended latent diffusion. arXiv preprint
[5] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instractpixppix: Learning to follow image editing instructions. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402, 2023.
[6] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qic, and Yinqiang Zheng. Masactrl: Tuning-free free mutual self-attention control for consistent image synthesis and editing. arXiv preprint arXiv:2304.0 084655, 2023.
[7] Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 23206-23217, 2023.
[8] Chaofeng Chen, Jiadi Mo, Jingwen Hou, Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi Lin. Topiq: A top-down approach from semantics distortions for image quality assessment. IEEE Transactions on Image Prucessing, 2024.
[9] Hong Chen, Yipeng Zhang, Xin Wang. Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Disenbooth: Identity-preserinng disentangled tuming for subject-driven text-to-image generation, 2023.
[10] Jooyoung Choi, Yunjey Choi, Yunji Kim, Junho Kim, and Sungroh Yoon. Custom-edit: Text-guided image editing with customized diffusion models. arXiv preprint arXiv:2305. 157799 2023.
[11] Hyungjin Chung. Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint arXiv:2206.0094/ 2022.
[12] Guillaume Couairon, Jakob Verbeek. Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-bassd semantic image editing with mask guidance. In ICLR 2023 (Eleventh International Conference on Learning Representations), 2023.
[13] Siying Cui, Jia Guo, Xiang An, Jiankang Deng, Yongle Zhao, Xinyu Wei, and Ziyong Feng. Idadapter: Learning mixed features for tuning- free personalization of text-to to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages
2024.
[14] Prafulla Dhariwal and Alexander Nichol. Diffusson models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
[15] Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106 (196):1002 1614, 2011.
[16] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Bermano, Gal Chechik, and Daniel Cohen-Or An image is worth one word: Personalizing text-to- text-to-image generation using textual inversion. arXiv preprint arXiv:2208. 01618, 2022.

---

[17] Jing Gu, Yilin Wang, Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu. Zhifei Zhang, He Zhang, Jianming Zhang. HyunJoon Jung, et al. Photoswap: Personalized subject swapping images. arXiv preprint arXiv:2305. 18286, 2023.
[18] Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[19] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Mctaxas, and Feng Yang. Svdiff: Compact paramcter space for diffusion fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7323-7334, 2023.
[20] Shaozhe Hao, Kai Han, Shihao Zhao, and Kwan- Yee K Wong. Vico: Detail preserving visual condition for personalized lext-to-image generation. arXiv preprint arXiv:2366.00977, 2023.
[21] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman. Yael Pritch, and Daniel Cohen-Or Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208.00 01626, 2022.
[22] Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta demoising score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2328-2337, 2023.
[23] Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or. Style aligned image generation via shared attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4775-4785, 2024.
[24] Jonathan Ho and Tim Salimans. Classificr-free diffusion guidance, 2022.
[25] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[26] Mengqi Huang, Zhendong Mao, Mingcong Liu, Qian Hc, and Yongdong Zhang. Realcustom: Narrowing real text word for real-time open-domain text-to-image customization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7476-7485, 2024.
[27] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pages 1501-1510, 2017.
[28] Ajay Jain, Amber Xie, and Pieter Abbeel. Vectorfusion: Texi-to-svg by abstracting pixel-based diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1911-1920, 2023.
[29] Maxwell Jones, Sheng-Y. Yu Wang, Nupur Kumari, David Bau, and Jun-Yan Zhu. Customizing texl-to-image models with single image pair. arXiv preprint arXiv:2405.01536 2024.
[30] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basca generative models. Advances in Neural Information Processing Systems, 0-226655-26777 2022.
[31] Oren Katzir, Or Patashnik, Daniel Cohen-Or, and Dani Lischinski. Noise-free score distillation. arXiv preprint 2rX1::2310.17590 2023.
[32] Bahjat Kawar. Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang. Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text- based real image editing with diffusson models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007-6017, 2023.
[33] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVE international conference on computer vision, pages 5148-5157, 2021.
[34] Gwanghyun Kim and Jong Chul Ye. Diffusionclip: Text-guided image manipulation using diffusion models. arXiv preprint arXiv:2110.02711. 2021.
[35] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 2426-2435, 2022.
[36] Subin Kim, Kyungmin Lee, June Suk Choi, Jongheon Jeong, Kihyuk Sohn, and Jinwoo Shin. Collaborative score distillation for consistent visual synthesis. arXiv preprint arXiv:2307.04787 2023.
12

---

[37] Zhe Kong, Yong Zhang, Tianyu Yang. Tao Wang, Kaihao Zhang. Bizhu Wu, Guanying Chen, Wei Liu, and Wenhan Luo. Omg: ceclusion-friendle personalized multi-concept generation in diffusion models. arXiv preprint arXiv:2403. 10983, 2024.
[38] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[39] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings the the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931-1941, 2023.
[40] Gihyun Kwon and Jong Chul Ye. Diffusion based image translation using disentangled style and content representation. arXiv preprint arXiv:2209. 1524,4 2022.
[41] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre- pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[42] Tianle Li, Max Ku, Cong Wei, and Wenhu Chen. Dreamedit: Subject-driven image editing. arXiv preprint arXiv:2306. 12624, 2023.
[43] Xiaoming Li, Xinyu Hou, and Chen Change Loy. When stylegan meets stable diffusion: a w+ adapter for personalized image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2187-2196, 2024.
[44] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pages 423 439. Springer, 2022.
[45] Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang. Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept neurons in diffusion models for customized generation. arXiv preprint arXiv:2303.05125. 2023.
[46] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1xX1v:1711.05100,,000 2017.
[47] Chenlin Meng, Yang Song, Wenzhe Li, and Stefano Ermon. Estimating high order gradients of the data distribution by denoising. Advances Neural Information Processing Systems, 14-25399-25366 2021.
[48] Chenlin Meng, Yang Song. Jiaming Song, Jiajun Wu, Jun Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01072 2021.
[49] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Ou Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 6038-6047, 2023.
[50] Jisu Nam, Heesu Kim, DongJae Lee, Siyoon Jin, Seungryong Kim, and Seunggyu Chang. Dreammatcher: Appearance matching self-attention for semantically-consiste text-to-image personalization. arXiv preprint 2001 2024.
[51] Thao Nguyen, Yuheng Li, Utkarsh Ojha, and Yong Jac Lee. Visual instruction inversion: Image editing via image prompting, Advances Neural Information Processing Systems, 36, 2024.
[52] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 1rXiv:2112.10744 2021.
[53] Taesung Park, Alexei A Efros, Richard Zhang, and Jun- Yan Zhu. Contrastive learning for unpaired image-to-image translation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK August 23-28, 2020, Proceedings, Part IX 16, pages 319-345. Springer, 2020.
[54] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zcro-shot image-to-image image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11, 2023.
[55] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer 2021.
Vision, pages
13

---

[56] Ryan Po, Guandao Yang, Kfir Aberman, and Gordon Wetzstein. Orthogonal adaptation for modular customization of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1963,7973, 2024.
[57] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall, Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
[58] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[59] Herbert E Robbins. An empirical bayes approach to statisties. In Breakthroughs in Statistics: Foundations and basic theory, pages 388-394. Springer, 1992.
[60] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Transactions graphics (TOG), 42(1):1-13, 2022.
[61] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695 2022.
[62] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2023.
[63] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingho Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization text-to-imaa ti-ae models. arviv preprint arXiv:2307.0663 069499 2023.
[64] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes. Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in Neural Information P Processing Systems, 01136479-369990 2022.
[65] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304.03411 2023.
[66] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv:1409. 1556, 2014.
[67] Uriel Singer, Amit Zohar, Yuval Kirstain. Shelly Sheynin, Adam Polyak, Devi Parikh, and Yaniv Taigman. Video editing via factorized diffusion distillation. arXiv preprint arXiv:2403.09334 2024.
[68] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint axXiv:2010.02502, 2020.
[69] Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming-Yu Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss gusied diffusion models for plug-and-pll controllable generation. In International Conference on Machine Learning, pages 32483 32498. PMLR, 2023.
[70] Raphael Tang, Linging Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp. Jimmy Lin, and Ferhan Ture. What the daam: Interpreting stable diffusion using cross attention. arXiv preprint Xrv: 20.22220.04400 04885..............11..11 2022.
[71] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personalization. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11, 2023.
[72] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text- driven image-to-image translation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1921-1930, 2023.
[73] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, Or, and Kfir Aberman. p+: Extended textual conditioning in text-to-image generation. arXiv preprint arXiv:2303.0953 09522, 2023.
[74] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation, 2023.
14

---

[75] Zhou Wang, Bero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Sevity-Seeenth Asilomar Conference Signals, Systems & Computers, 2003, volume pages 1398-1402, leee, 2003.
[76] Zhou Wang. Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from visibility to structural similarity. IEEE transactions on image processing, 13(4):600-6) 612. 2004.
[77] Chendong Xiang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. closer look parameter-efficient tuning in diffusion models. arXiv preprint arXi::2303.1811 2023.
[78] Xin Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Song-Hai Zhang, and Xiaojuan Qi. Text-to-3d with classifier score distillation. arXiv preprint arXiv:2310.1942 2023.
[79] Yu Zeng. Vishal M Patel, Haochen Wang. Xun Huang. Ting-Chun Wang, Ming-Yu Liu, and Yogesh Balaji. Jedi: Joint-image diffusion models for finetuning free personalized text fext-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6786-6795 2024.
[80] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances in Neural Information Processing Systems, 36, 2024.
[81] Weixia Zhang, Guangtao Zhai, Ying Wei. Xiaokang Yang. and Kede Ma. Blind image quality assessment via rision-language correspondence: multitask learning perspective. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14071-14081, 2023.
[82] Yanbing Zhang, Mengping Yang. Qin Zhou, and Zhe Wang. Attention calibration for disentangled text- to-image personalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4764-4774, 2024.
[83] Yuechen Zhang, Jinbo Xing, Eric Lo, and Jiaya Jia. Real-world image variation by aligning diffusion inversion chain. arXiv preprint arXiv:2305. 18729, 2023.
[84] Jia-Xing Zhao, Yang Cao, Deng-Ping Fan, Ming-Ming Cheng, Xuan- Li, Le Zhang. Contrast prior and fluid pyramid integration for rgbd salient object detection. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3927-3936, 2019.
[85] Jing Zhao, Heliang Zheng, Chaoyue Wang. Long Lan, and Wenjing Yang. Magicfusion: Boosting text-to to image generation performance by fusing diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 22592-22602, 2023.
15

---

A More Background on Diffusion Probablistic Models
Denoising Difffusion Implicit Model (DDIM). Given a diffusion probabilisticc model parameter- ized by Ø and diffusion process defined as q (Xt Xo) ;= . (Xr; Va,Xo, (1 - a2)) I) where ar represents the forward process variance at time and x, the noised latent state of the input Xo. DDIM [68] defines the following update rule in the reverse diffusion process
x 0
X1
O (xx)
07-1 oft (1) (xx)
(12)
v 04
"direction pointing
"random noise"
"predicted %5"
where oe is free variable that controls the stochasticity the reverse process.
DDIM Inversion. By setting Ø to O, obtain deterministic update rule which can be reversed to give a deterministic mapping between Xy and its latent state Xy. We refer this inverse mapping as DDIM inversion:
0++1 (v1
ate V1-10 Ott
X
Xr
co (1) (xx)
(13)
0++1
04t
0+++
Classifier-free Guidance (CFG). Given a diffusion model jointly trained on conditional and unconditional embeddings, at inference time, samples can be generated using CFG 22). The prediction with the conditional and unconditional estimates are defined as
cfc Y: B (, + (1 9)&s (x,, 1,())
(14)
where 3 is the guidance scale that controls the trade-off between mode coverage as well as sample fidelity and Ø is a null token used for unconditional prediction.
B Details about single-step denoising direction
We parameterize the single-step perturbed latent state (o) based on Tweedie's estimation 13
which computes the posterior estimation for the denoised data, i.e., Xy == E(x, X,
(x, + (1 - Vx, logp(x;)) Voi, with which we define single-step perturbed latent state (o) as follows
ax(d)= = Vapoo + V1 arC Xt V1 at (co (xe, y)
(15)
Single-step denoising direction
where the same noise added x, and (Ø).
C More Implementation details
The pre- pre-trained checkpoints provided by ViCo 20 have 16 concepts, which include 6 toys, live animals, 2 types of accessories, types of containers, and building. The corresponding text prompts of source images are generated by BLIP2 [1]]. For editing with one-shot personalization. performance evaluations are conducted on subset of 20 cat and dog images.
Dataset
Editability Driven Score Distillation The EDSD gradient in Eq. 7 includes a Jacobian term
(e which is computationally expensive and unstable for optimization. Previous work 577
(2,(0)
shows that this term cause instability in optimization and omits the computation of it by setting it
to I. As shown in Fig. g we find that by setting it to I following previous work, the editing process tends destroy the structural layout and background of the source image. Meanwhile, the edited
image tends to be over-saturated. This indicates that by setting 8eo (0)
I, the discrepancy between personalized and source model score estimations has been maximized. We find that setting
(8x(d)
de, (e.(e)))) -I leads to significantly better results with natural adaptation to the layout of the
(0,(0)
16

---

Reference
Source
Positive Jacobian Negative Jacobian
XXI
Figure 8: Results with different Jacobian omitting strategy.
source image. We suspect that this because, in the former work [57] the input noisy latent state is independent from the diffusson model. However, in Eq. there exists entanglement between negative score estimation in the single-step denoising function (Eq. 6) and the score estimation. We leave further investigation this interesting scenario future study.
Spatial feature guided sampling During spatial feature guided sampling, to avoid potential numerical instability in the sampling process due the drift in statistics caused by the addi- tional guidance gradient, we rescale the noise prediction via an adaptive normalization [27] i.e.,
agd y h, = AdaaN gd y hi
rfg yte)) Following earlier works
S3 we
introduce spatial feature guidance only for the early denoising steps >> tearly. After fearly. we change the ODE sampler [68] to an SDE sampler (25) to inject stochasticity for improved sample quality (30). we employ DDIM with a total step 50, the CFG is set as for the inversion process, 3.5 for the sampling process, and negative prompts "oversaturated color, ugly, tiling, low quality, noisy" are employed replace the null text token. We set the early stopping step as fearly
The
size of Aga is set to be the same as the size of the reference images.
Optimization details For fine-tuning with each baseline, we employ the same set of trainable parameters as the original personalization process. We use an Adam W [46) optimizer for all baselines; the learning rates are set as le-3, le-6, and 5e-5 for Textual Inversion, DreamBooth, and Custom Diffusion respectivell. The total optimization step is set 10 with 10 cumulative gradient steps. Experiments run on a single NVIDIA RTX3090 GPU take fine-tuning time of around minute for batch size of For all experiments, we use a latent space diffusion model [6]], with pre-trained checkpoints from Stable- sialeeDiffusion------- which is augmented with safety modules to mitigate NSFW content.
D Further comparison with existing works
Difference with subject swapping Text-driven image editing methods generally fall into two
categories: rigid editing (5 21 that emphasize the preservation of the source image, and non-rigid editing [6] that focus on changing the view or pose of a subject in the source image while preserving the background. Our work is motivated by bridging the gap in editability between specific concept and natural language conditioning for rigid editing. Subject swapping methods like DreamEdit [42] and Photoswap [7] diverge from our approach in their main criteria. While these methods prioritize alignment with the source subject's 's location and pose, they do not necessitate maintaining the original structural details our method does. Furthermore. these works demand a stricter preservation of
Source from https://ttgiiingssss.ooooomiiiia
oriiiiaa
17

---

Reference
Source
DreamEdit
Photoswap
Ours
Figure Comparison with subject swapping methods.
Source
Reference
Custom Edit
Custom Edit-ours
Reference
Custom-Edit
Custom-Edit-os
Figure 10: Comparison with Custom- Eddi..
subject identity and typically do not require significant level of concept extrapolation like our work, e.g.. from short cat tall cat. Fig. g we provide a comparison with DreamEdit and Photoswap in scenarios involving significant structural gaps. Compared with our work, these works often exhibit severe distortion or fail to maintain structural consistency with the source image.
Comparison using different image editing method We employ variant of Delta Denoising Score as the base editing model as this method is compatible with all the personalized models we use. However, DreamSteerer is not restricted to a specific type of editing pipeline. We also evaluate the effectiveness of our method as a plug-in for Custom-Edi Edit [10], which directly combines Custom Diffusion with Soompt-to-Promppe (2I]. The results are shown in Fig. I0 and Table. Meanwhile, as shown in Fig. II combining Prompt-to- Prrompp with DreamBooth can introduce significant appearance artifacts the edits, which is the main reason did not as the base editing method. Prompt-to- trompt-to-Prompt relies on a source latent state inversion process, typically through Null-Text Inversion (NTI). However, parameter updates during personalization can shift the model distribution for the source class, compromising the editability of the inverted latent state chain with NTI. In comparison, the Delta Denoising Score-based edited method employed in our work does not require an inversion process, providing more robust performance across different types of personalization baselines. We believe this phenomenon worth further investigation and encourage future works to develop new inversion techniques specifically tailored for the personalized models.
18

---

Table 4: Comparison with Custom-Edit.
CLIP- (1)
LPIPS (1)
IQA (1)
Method
SSIM (t)
MS-SSIM (t)
L14
VGG
Topiq
Musiq
LIQE
Custom Edit 100
.748
.727
.141
.210
.793
.899
564
67.94
3.97
DreamStecrer
.750
.729
.141
.209
.793
.899
.565
67.95
3,98
Reference
Source
Prompt-to-Prompt
Ours
Figure 11: Failure of Prompt-to- Hoompttto-Prompt when integrated with DreamBooth.
c.5 3.5
cg-77
Source
Source
Reference
Reference
d
Source
Source
dfg=15
Reference
Figure 12: Sensitivity analysis on and cfg for spatial feature guided sampling
19

---

E
More ablation study
More results of ablation study on EDSD and Most Shifting regularization are provided in Fig.
Sensitivity analysis on spatial feature guided sampling We ablate on two aaper-parameters and cfg introduced in Eq. for spatial feature guided sampling. As shown in Fig. 13 contrast to the usual selection of large cfg, e.g., 7 or 15. A relatively small cfg of 3.5 leads to better naturalness of the generated samples that avoids obvious subject appearance drift from the reference images. Meanwhile, a large spatial feature guidance scale À produces samples that better preserve the structure of the source image. Thus we set cfg as 3.5 and 15.
F
More results compared with the base personalization methods
More results compared with the baseline models are provided in Fig. 14 15 16
G More results with one-shot personalization
More comparison of one-shot performance with the DreamBooth baseline and Custom Diffusion baseline is provided in Fig. 17
H More spatial feature guided sampling results
More spatial feature guided sampling results used in Eq. are provided in Fig. Fig. 19 and Fig.
I Explanation on editing direction visualization
In Fig. []]
we visualize the score-based editing directions obtained by taking the editing scores as input the Stable Diffusion decoder. This similar visualization to Katzir et al. [31]. We note that the brown color in the visualization indicates an editing score value of O. An example of zero-valued latent state decoding is illustrated in Fig.
J Automatic subject mask extraction using cross-attention
Figure 21: Zero-valued latent decod- ing.
As shown in Fig. 22 we observe that the attention maps from decoder layers with resolutions 16 and 32 have accurate subject layout. Thus, we use the averaged CA maps from these layers as the subject mask. Similar to the spatial feature extraction process in Sec. 4.2 DDIM inversion is conducted on the source image x by which the averaged CA map that corresponds to the source subject token extracted, i.e.,
M (xfr) = E Softmax (of (Kt)* /CC)..
K
Failure case and limitation
Although DreamSteerer achieves high-fidelity editing results, its performance still limited by the baseline model. For example, as shown in Fig. 23] the DreamBooth model trained on the concept [duck_to] toy] lacks subject appearance invariance preservation. With DreamSteerer, although improved source image alignment is achieved, the editing result loses part of the subject appearance information. We emphasize the importance of devising reliable T21 personalization pipelines with improved editability and sample fidelity for future works.
L
User study
10 participants conduct user preference study of DreamSteerer against different person alization baselines. For each bascline with without DreamSteerer, asked evaluate
20

---

Reference
Source
EDSD
mode shifting
(a) Baseline model: Textual Inversion.
Reference
Source
EDSO
mode shifting
Durs
(b) Baseline model: DreamBooth
Reference
Source
wo EDSD
wo mode shifting
Ours
(c) Baseline model: Custom Diffusion.
Figure 13: More ablation study.
21

---

Reference
Source
Textual Inversion
Ours
Figure 14: More comparison with Textual Inversion
22

---

Reference
Source
Textual Inversion
Ours
Figure 15: More comparison with DreamBooth
23

---

Reference
Source
Custom Diffusion
Ours
Figure 16: More comparison with Custom Diffusion
24

---

Reference
Source
DreamBooth
Reference
Source
Custom Diffusion
Figure 17: More comparison of one-shot performance with the DreamBooth baseline and Custom Diffusion baseline.
Reference
Source
Guided samples
Figure 18: More spatial feature guided sampling results with Textual Inversion baseline model.
25

---

Reference
Source
Guided samples
Figure 19: More spatial feature guided sampling results with DreamBooth baseline model.
35 random examples. The examples were given in random order. For each example, the user was asked to rate the overall quality of the edited image on a 1-5 scale (higher is better) with respect to the following criteria:
1. How well does the right image preserve the structural information and the background information of the left image? Consider the accuracy in subject part alignment between two images.
2. How well does the edited subject in the right image preserve the appearance subject in the reference images above it?
3. How well is the overall realism and quality of the right image?
screenshot of the user study page is shown in Fig. 24 As shown in Table. 5 DreamSteerer is preferred by users with significant margin. By employing the Stable-Diffusion-v-1 safety modules, we have prevented the inclusion of NSFW content that could potentially harm the participants.
M Broader Impacts
Our research on personalized image editing, particularly in fine-tuning the model to understand personal concepts and applying it image editing, has significant potential for broad societal impacts. By enabling individuals to seamlessly integrate their personal concepts, styles, and cultural backgrounds into their visual content, our approach democratizes the creative process digital media. This accessibility enables users across diverse demographics, including artists, content creators, educators, and individuals with limited artistic expertise, to express themselves authentically through content creation. Moreover, by facilitating the creation of personalized content, our method
26

---

Reference
Source
Guided samples
Figure 20: More spatial feature guided sampling results with Custom Diffusion baseline model.
(a) Enc16 16
(b) Enc32 x 32
(c) Enc64 x 64 (d) Dec16 x 16 (c) Dec32 x 32 (f) Dec64 x 64
Figure 22: Visualization averaged cross-attention maps of the DPM UNet encoder and decoder
different resolutions corresponding to 'astronaut' token in the prompt 'an astonaut riding a horse'.
Reference
Source
DreamBooth
Ours
Figure 23: Failure case of DreamSteerer.
27

---

This the set of referance Images
You shown a comparison the source and edited Image.
Please read Instructions below carefully you asked
photo on
How well does the edited (right) image the structural information and the background information the source (leff) image?
Consider the accuracy in subject part alignment between two Images.
2. How well does the edited subject the edited (right)) image preserve the appearance the subject the reference images above?
How well the overall reallem and quality of the edited (right) image?
Hore (left) and edited photo (right)
Based on those criteria, please rate the right photo on scale of to 5 (higher means better).
5 texoltett
Figure 24: A screenshot of the user study page.
28

---

Table User preference study on 1-5 scale (DreamSteerer uses the same model as baseline).
User preference (t)
Textual Inversion [66
2.48
DreamSteerer
3.78 (+44.5%)
DreamBooth DreamSteerer
622
2.90
3.76 (+29.7%)
Custom Diffusion 39 DreamSteerer 3.78 (+49.7%)
2.62
has the potential to enhance user engagement and satisfaction in various domains, ranging from social media advertising to educational materials and storytelling. While our research offers novel approach to personalized content creation, there is a potential for misuse in generating false and harmful content, emphasizing the need for greater caution.
N
Image attribution
ViCo dataset: http:///gthuub.cm//uoo/o///
PlE-Bench: http:p////ttthhioooomm//m/aaaaaa
Textual Inversion http://www..//.eeetttt..../////a
in
DreamBooth dataset: httpss//rreaabocch..ttttbbob/
Custom httpo//otgg//a////////////////o/
DreamBench: http:///hh.attee//...oo//ooo/ooa
Cat riding scooter: http::///eeerrraa.aaaooo////////
Teddybear: http::///aeeeee....dadaaaa///aaa
A highly detailed zoomed-in digital painting of cat dressed as a witch wearing a wizard hat
in a haunted house, artstation:
photo of a corgi dog riding a bike in times square http:://inggen.reerrd..oo.g
a photo of a cute corgi lives house made out of sushi: btaps://imaaen.rrt researcc
google
29

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Experimental results in Sec. justifies the claims in the abstract and introduc-
tion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
An analysis on limitation of this work is provided in Sec. K
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
30

---

Justification:
Every equation is referenced or derived, accompanied by necessary approxi- mations or assumptions. all of which are clearly motivated and justified.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All implementation details provided in the main paper and sec.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer:
[No]
Justification: the code this work will be open-sourced upon acceptance of paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
wwt.Subbmisiioooiiioooooooio for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training and testing details are provided in the main paper and Sec. C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: For all experiments, we present the mean values of the evaluation metrics. We adhere to the evaluation criteria established in the literature, where error bars are not commonly utilized.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
32

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computational source requirement information is provided in Sec. C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have thoroughly gone through the NeurlPS Code of Ethics and confirm that the research conducted in this work conforms with every aspect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: We have provided discussion on the potential social impact of the work in Sec. M
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification: The safety modules used in Stable Diffusion 1-4 is mentioned in Sec.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we have explicitly respected the owner of the pre-trained models and images.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
34

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
The paper does not introduce new assets; however, it utilizes pre-trained checkpoints and safety modules from umentation and details accessible from the provided source. Furthermore, the code associated with this work will be made available upon paper acceptance, accompanied by comprehensive documentation.
Justification: Justification:
for which doc-
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: For user preference study of our work, we include the full text of instruction in Sec. L
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: As discussed Sec. L we have addressed the potential risk associated with the inclusion of NSFW content, which could potentially harm the participants. We mitigate this risk by utilizing safety modules provided by the owner of the pre-trained model.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
35

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "SoF4BPycut", "reviewer": "Reviewer_AnAq", "summary": "DreamSteerer is a proposed fine-tuning pipeline for personalized diffusion models, designed to enhance the custom editability of these models. The authors point out that naively incorporating existing image editing and personalization methods—such as employing score distillation and DDS where the differentiable function is initialized by the source image latent space—results in low fidelity and lack of editability relative to the source image. To address these issues, they propose the following additions that comprise DreamSteerer:\n1. Editability driven score distillation (EDSD), to increase the editability of the model by distillation information from the source model $\\epsilon_{\\phi_0}$ to the personalized diffusion model  $\\epsilon_{\\phi_0}$ through a single-step perturbed source latent state\n2. Spatial Feature Guided Sampling and mode shifting regularization to overcome the mode trapping issue introduced by EDSD (commonly observed in score distillation)\n3. Source score bias correction and automatic subject masking to address the distribution shift from personalization, and to focus on the relevant aspects of the image during training respectively.\n\nThe authors present thorough qualitative and quantitative results across three metrics: semantic similarity, perceptual similarity, and structural similarity, along with a human preference evaluation. They demonstrate that DreamSteerer consistently improves upon existing baseline methods and include ablation studies to justify various components of the pipeline.", "questions": "I have minor questions regarding the implementation details to clarify some confusion:\n1. Source of Real-World Images: Can you provide more details on the 70 random real-world images used for editing? Were these images scraped from the web or sourced from an existing dataset? Additionally, were images containing a shared superclass used to enhance the feasibility of the editing tasks?\n2. Clarification on Figure 5: In Figure 5, it’s unclear where the results in subfigures (e) to (g) originate from. Are these images based on specific source images, or are they entirely generated without any reference images, from the finetuned personalization model?", "limitations": "The authors have adequately addressed the limitations and potential societal impact of their work. Specifically, they describe and visualize the limitations of DreamSteerer, which can be constrained by the baseline personalization model, and suggest ways to avoid fidelity inconsistencies in future similar works.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Originality: The work is well-motivated and addresses a limitation in the intersection of personalization and image editing, which is a relatively underexplored area. Although DDS has been employed for general text-driven editing, the application of DDS to enhancing editability in the personalization setting is a novel approach. In addition to combining the existing ideas, the authors introduce further modifications to improve final results. \n\nClarity: The paper clearly explains each learning objective with detailed formulations, supported by an overarching diagram. The results are well-organized and articulated, presenting both qualitative and quantitative analyses. This structure provides a cohesive narrative that justifies the design decisions made throughout the work.\n\nQuality: The proposed algorithm introduces a novel approach which improves over existing methods in the personalized image editing domain, and the work presents sufficient ablation studies over core components to reinforce the reliability of the proposed approach. \n\nSignificance: The proposed method for personalized image editing with few fine-tuning steps required has potential for widespread adoption. DreamSteerer's ability to maintain structural integrity while enabling personalization edits makes the work applicable across various creative domains, which may demand precise edits.", "weaknesses": "The paper’s results would benefit from benchmarking on more extensive personalization datasets, such as CustomConcept101. Currently, the evaluation is based on only 16 concepts from the ViCo repository, a design decision that lacks strong justification. While the selected concepts show a reasonable level of diversity, evaluating the method across a broader set of personalized concepts would enhance the robustness and generalizability of the results.\n\nAdditionally, there are minor implementation details that could be clarified in the Experiments section. Specifically, the paper mentions using pre-trained checkpoints provided by ViCo. However, the ViCo GitHub repository does not link to checkpoints for Textual Inversion, DreamBooth, and Custom Diffusion, leading to some confusion about the source of these pre-trained checkpoints. Providing clear information on where these checkpoints were obtained would help improve the reproducibility of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and questions. Following are responses for your questions.\n\n> **W1 Benchmarking on more extensive personalization datasets** \n\nThank you for your feedback. Our decision to evaluate based on 16 concepts from the open repository[1] was driven by the availability of comparable, publicly accessible checkpoints. These checkpoints, specifically for Textual Inversion, DreamBooth, and Custom-Diffusion, are trained on the same concepts within the ViCo dataset and demonstrate generation fidelity consistent with their original publications. This approach allowed us to ensure a fair and direct comparison across different personalization methods. We acknowledge the importance of evaluating our method across a broader set of personalized concepts to enhance the robustness and generalizability of our results. We plan to explore this in future work, aiming to provide a more comprehensive evaluation across additional concepts and personalization baselines.\n\n\n> **W2 Pre-trained checkpoints** \n\nThank you for pointing out the need for clarification regarding the implementation details. We apologize for any confusion caused by the oversight in our manuscript. The pre-trained checkpoints used in our work were sourced from the DreamMatcher GitHub repository[1], that has been trained on the ViCo dataset. We will update our manuscript to include this accurate source information to enhance the reproducibility of our work.\n\n> **Q1  Source of Real-World Images**\n\nThe images we use are collected from 3 main sources: \n* DreamBench[2]\n* PIE-Bench[3]\n* Web data\n\nOur selection criteria were particularly focused on images where the source subject exhibited a significant structural difference from the reference subjects, yet maintained a shared superclass with these references. Such selection criteria enable us to better evaluate the robustness of the personalized editing with significant structural misalignments between source and reference subjects. We will include these details in our updated paper.\n\n> **Q2  Clarification on Figure 5**\n\nThank you for seeking clarification regarding Figure 5 in our manuscript. We understand the confusion and are happy to provide a more detailed explanation.\n\nThese figures are random generations **using different diffusion models** based on the prompt 'A photo of a sks cat standing next to a mirror', where sks corresponds to the personalized concept in Fig.5b. Specifically:\n\n* Fig.5e is generated by a base personalized model trained on the images shown in Figure 5b.\n* Fig.5f results from fine-tuning the base model with only the Editability Driven Score Distillation (EDSD) term.\n* Fig.5g results from fine-tuning the base model with both the EDSD term and the Mode Shifting term.\n\nOur analysis reveals that without the Mode Shifting term, the images in Fig.5f tend to lose a portion of appearance fidelity compared to Fig.5e, resulting in a hybrid appearance closer to the source image. Conversely, with the inclusion of the Mode Shifting term, the images in Fig.5g maintain appearance fidelity comparable to the base model (Fig.5e) while displaying patterns more akin to the source image, such as the blue collar and the presence of \"two cats.\" This demonstrates the effectiveness of the Mode Shifting term in steering the model to enhance editability for the source image without compromising the personalized subject's appearance information. We will include this in our revised paper.\n\n[1] https://github.com/KU-CVLAB/DreamMatcher\n\n[2] https://github.com/nousr/dream-bench\n\n[3] https://github.com/cure-lab/PnPInversion", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qnF8RgMz2A", "reviewer": "Reviewer_zDhi", "summary": "Aiming at addressing unsatisfactory editability on the source image, this paper proposes a novel plug-in method for augmenting existing T2I personalization methods.  Specifically, this framework finetunes the personalization parameters by training a novel Editability Driven Score Distillation objective under the constraint of a Mode Shifting regularization term based on spatial feature-guided samples. Extensive experiments validate the effectiveness of the proposed method.", "questions": "See weakness.", "limitations": "Although DreamSteerer achieves high-fidelity editing results, its performance is still limited by the baseline model.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method can be applied current T2I personalization models to perform custom editing as a plug-in.\n\n2. Efficiently, the framework  DreamSteerer requires only a small number of fine-tuning steps (∼ 10) to achieve significant improvement in editability on a source image.", "weaknesses": "1. In related work, the authors lack a discussion of recent works for T2I personalization. Most of the works are in the first half of last year.\n\n2. This framework introduces many computing loads in there modules in Fig.3, which worries me about the inference efficiency of this method. It is suggested that the authors can provide some explanation about computational complexity in experiment section.\n\n3. There are some excellent methods for custom editing, such as [1] [2]. It is recommended to compared with these strong baseline to show the effectiveness of this method.\n\n4. As for qualitative results in this paper, obviously, the proposed method forces consistency of the object shape between the source object and target object, resulting in the destruction of the edited object identity compared the reference. This may not be the editing result that the user wants. It is suggested the authors can provide analysis for this.\n\n[1] Custom-edit: Text-guided image editing with customized diffusion models.\n\n[2] DreamEdit: Subject-driven Image Editing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and questions. Following are responses to your concerns.\n\n> **W1 Insufficient related work of T2I personalization**\n\nThank you for the suggestion. We provide an additional literature review related to T2I personalization mostly from this year below:\n\nRecent studies in encoder-based personalization[1-4] propose new training paradigms to condition a Diffsuion Model on single or multiple input images. Although these works can enable faster inference, they necessitate extensive pre-training and typically limit application to particular domains. New forms of conditionings, objectives or adaptors are proposed for different purposes such as stylization[5-7], improved identity preservation[8,9], composability[10-13], or generation fidelity[14]. Unlike these works, we focus on addressing an inherently different task of improving the editability of personalized Diffusion Models.\n\nWe will include these in our revised paper.\n\n[1] JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation, CVPR 2024.\n\n[2] Instantbooth: Personalized text-to-image generation without test-time finetuning, CVPR 2024.\n\n[3] RealCustom: Narrowing Real Text Word for Real-Time Open-Domain Text-to-Image Customization, CVPR 2024.\n\n[4] Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models, CVPR 2024.\n\n[5] Customizing Text-to-Image Models with a Single Image Pair, arXiv 2024.\n\n[6] Visual instruction inversion: Image editing via image prompting, NeurIPS 2023.\n\n[7] Style aligned image generation via shared attention, CVPR 2024.\n\n[8] When stylegan meets stable diffusion: a w+ adapter for personalized image generation, CVPR 2024.\n\n[9] IDAdapter: Learning Mixed Features for Tuning-Free Personalization of Text-to-Image Models, CVPR 2024.\n\n[10] Attention Calibration for Disentangled Text-to-Image Personalization, CVPR 2024.\n\n[11] Orthogonal adaptation for modular customization of diffusion models, CVPR 2024.\n\n[12] Omg: Occlusion-friendly personalized multi-concept generation in diffusion models, ECCV 2024.\n\n[13] Magicfusion: Boosting text-to-image generation performance by fusing diffusion models, ICCV 2023.\n\n[14] Dreammatcher: Appearance matching self-attention for semantically-consistent text-to-image personalization, CVPR 2024.\n\n> **W2 Computational complexity**\n\nPlease refer to general comment part3. \n \n> **W3 Comparison with existing methods** \n\n**Comparison with DreamEdit[2]**:\nPlease refer to general comment part 1 for a discussion on how our work differ from subject-driven editing works like DreamEdit[2]. As illustrated in Fig.C of the attached rebuttal pdf, compared with our work, DreamEdit often exhibits severe distortion or failure in maintaining structural consistency with the source image.  \n\n**Comparison with Custom-Edit[1]**: Please refer to general comment part 2.\n\n> **W4 Forcing shape consistency with source**\n\n* Please refer to general comment part 1 for a clarification on how our work connects with rigid text-driven image editing and differs from subject swapping. \n* Here we also illustrate several advantages that the feature of aligning with the source structure might offer for real-world applications:\n    * Enhanced creative experimentation: content creators have the freedom to explore different appearances based on different reference concepts without compromising the essence of the source image. This facilitates the exploration of diverse aesthetic outcomes with ease.\n    * Consistency and recognizability: in branding and media content creation, maintaining the original structure ensures that if viewers are familiar with the original content, it enhances brand recognition and ensures continuity of visual content. \n    * Contextual integrity preservation: in fields such as cultural heritage, aligning edits with the source structure can be crucial for maintaining the context and authenticity of visual information.\n\n[1] Custom-edit: Text-guided image editing with customized diffusion models\n\n[2] DreamEdit: Subject-driven Image Editing", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UdmEgWLKNY", "reviewer": "Reviewer_2C3N", "summary": "The paper introduces DreamSteerer, a method to enhance the editability of source images using personalized diffusion models in text-to-image (T2I) personalization. Existing methods often fail to maintain edit fidelity when applied to new contexts due to limited reference images and adaptability issues. DreamSteerer addresses this by introducing the Editability Driven Score Distillation (EDSD) objective, which enhances image editability. To mitigate a mode trapping issue in EDSD, the paper proposes mode shifting regularization with spatial feature guided sampling. This technique aligns the model more closely with the source image structure while preserving personalized concept appearances. Extensive experiments show that DreamSteerer significantly improves editability and efficiency across various T2I personalization baselines.", "questions": "Figure 14 and 13 is same.\nInsufficient resolution of figures provided in the paper", "limitations": "The paper discussed the limitations of this work.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1、The method is designed as a plug-in compatible with arbitrary personalization baselines.  This versatility enhances its applicability across various models and use cases\n2、DreamSteerer requires only a small number of fine-tuning steps (~10) to achieve significant improvements, making it computationally efficient.", "weaknesses": "1、While the paper presents a novel approach, it would benefit from a more thorough discussion regarding its innovative aspects in comparison to existing work in Object Driven Image Editing, such as DreamEdit[1] and SWAPANYTHING[2]. It appears that similar mechanisms are already in place within these frameworks, and a detailed exploration of how the current method distinguishes itself or builds upon these concepts would be valuable.\n2、A broader experimental evaluation is suggested.  This should include comparative studies with existing image editing works and other Object Driven Image Editing techniques mentioned in [1,2].\n3、The proposed method relies on certain heuristics, such as spatial feature guided sampling and the choice of specific parameters for mode shifting regularization.  These heuristics may not generalize well across different datasets or tasks, leading to suboptimal performance in some scenarios.\n\n[1] DreamEdit: Subject-driven Image Editing\n[2] SWAPANYTHING: Enabling Arbitrary Object Swapping in Personalized Visual Editing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable feedbacks. Following are responses to your questions.\n\n> **W1 Comparison to Object Driven Image Editing works**\n\nWe thank the reviewer for the constructive suggestion and acknowledgement on the novelty of our method. For a detailed discussion on how our approach differs from previous works in subject swapping, including a comparison with DreamEdit[1], please refer to general comment part 1. SwapAnything[3] is also an interesting concurrent work follows a similar setting as DreamEdit, which is different from the setting of our work. We will defer detailed comparison of SwapAnything to future work once they release their codes.\n\n> **W2 Broader experimental evaluation**\n\n**Comparison with different subject swapping work**: Please refer to general comment part 1 and response to W1.\n\n**Comparison using different image editing method**: \nPlease refer to general comment part 2.\n\n>**W3 Dependence on heuristics**\n\nWe thank the reviewer for mentioning the concern. Our method does not require a careful tuning on the hyper-parameters. We use the same learning rates as the original personalization baselines. Beyond this, the same set of hyper-parameters are shared among the personalization baselines, irrespective of the concept or source image involved. \n\n> **Q1 Figure 14 and 13 is same. Insufficient resolution of figures provided in the paper**\n\n* Thanks for noticing the duplicated image, we will avoid this for the camera-ready version. \n* We apologize for the insufficient resolution of images in our main paper. To avoid potential network issues and ensure a smooth submission process, we reduced the size of the images, which resulted in a lower resolution. To ensure that our work is presented with clarity and detail necessary for thorough evaluation and understanding, we will replace all figures with their original, high-resolution versions for camera-ready version. \n\n[1] DreamEdit: Subject-driven Image Editing\n\n[2] Custom-edit: Text-guided image editing with customized diffusion models\n\n[3] Swapanything: Enabling arbitrary object swapping in personalized visual editing", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yTAsfHEONF", "reviewer": "Reviewer_kb7M", "summary": "This paper proposes DreamSteerer, a plug-in method that enhances existing text-to-image personalization techniques by improving the editability of source images.\n DreamSteerer proposes a novel Editability Driven Score Distillation (EDSD) objective to improve the structural alignment between the source and edited image by performing score distillation with respect to personalized model parameters. \nIt identifies and addresses mode-trapping issues in EDSD using regularization with spatial feature-guided sampling. The proposed method enhances the editability of several T2I personalization methods.\nMoreover, they introduce two key modifications to the Delta Denoising Score framework to enable high-fidelity local editing with personalized concepts.", "questions": "* In the Method section, how is the given personalized model trained, and on which data?\n* In Figure 2, what are the reference images? Are they the same as those in Figure 3?\n* In Table 1, what do the user preference numbers indicate? \n* What is the computational overhead compared to a base personalization method like DreamBooth? How many UNet calls are required to edit one single image?\n* How would the performance of the proposed method compare with the recent personalization methods like DreamMatcher?\n\n* The paper is missing some important recent works in personalization:\n  * DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization (CVPR 2024)\n  * Style Aligned Image Generation via Shared Attention (CVPR 2024)\n  * Visual Instruction Inversion: Image Editing via Visual Prompting (NeurIPS 2023)\n  * MagicFusion: Boosting Text-to-Image Generation Performance by Fusing Diffusion Models\n\n* References for Weakness section:\n  * [ref1] C. Chen, J. Mo, J. Hou, H. Wu, L. Liao, W. Sun, Q. Yan, and W. Lin. Topiq: A top-down approach from semantics to distortions for image quality assessment. IEEE Transactions on Image Processing, 2024.\n  * [ref2] J. Ke, Q. Wang, Y. Wang, P. Milanfar, and F. Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF international conference on computer vision, 2021.\n  * [ref3] W. Zhang, G. Zhai, Y. Wei, X. Yang, and K. Ma. Blind image quality assessment via vision-language correspondence: A multitask learning perspective. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.", "limitations": "Discussed in Appendix J", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The problem of enhancing the editability of personalized image editing conditioned on source images is significant and underexplored. This idea is original since it modifies the score distillation methods to better align with the personalized image editing task. They provide an easy-to-use model-agnostic method for any type of personalization.\n* The quantitative experiment results are solid and demonstrate the method's effectiveness over the three baselines.\n* The presentation and writing are reasonably clear and easy to follow.", "weaknesses": "* The method lacks comparisons with editing baselines outside of personalization. It also lacks qualitative or quantitative comparison to the cited preliminary works [8, 14]. For the images that also appear in the Custom-Edit [8] such as rows 1,2,5 in Figure 1 of this paper, the result images in the Custom-Edit [8] paper appear to have much better quality.\n* The  \"mode shifting regularization\" approach is not convincing on the effectiveness of the proposed(Figure 5). In Fig.5, the claimed improvement in editing fidelity from Fig.5 (c) to Fig. 5 (d) does not seem obvious. (also  between \"(f)\" and \"(g)\" )\n* The paper is missing some important metrics for evaluation. In Section 5, the paper finds that automatic metrics do not fully reflect the superior performance of the proposed method. There is no image quality assessment (IQA) metrics used such as [ref1, ref2, ref3]. Also, the structural similarity metrics globally compare the edited image to the source image, which is unsuitable in this scenario since you want to maintain the similarity in the unedited background region, and a local similarity comparison is better suited. [Please check Questions]\n* The method is claimed to be efficient in the abstract section. However, the proposed method's speed and cost are not analyzed or numerically measured. The method appears to require significant computation for a single edit. It is recommended that the computational overhead be reported compared to base personalization methods like DreamBooth.\n* The example images in the paper have very low resolution, especially when zoomed in.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedbacks. Following are responses to your questions.\n>**W1 Further comparison study**\n\nWe are targeting image editing with personalized concepts (not exist in vocabulary); thus, editing baselines outside of personalization are not applicable for our task.\n\n**Comparison with Photoswap[1]**: Please refer to general comment part 1 for a setting-level difference and comparison of our work with subject swapping works like Photoswap[1].\n\n**Comparison with Custom-Edit[2]**: In Fig.1, only row 5 uses the same images and the performance is comparable with the same base personalized model, Custom Diffusion. The others (row 1 and 2) either have different source or reference images. Meanwhile, for the cat-statue example in the ablation study shown in Fig.6, we use DreamBooth, which differs from Custom-Edit. For a fairer comparison, in Fig. B (lower) of the attached PDF, we use the same base model, Custom Diffusion, to compare with Custom-Edit. The result of Custom-Edit is more similar to a subject swapping (please refer to general comment part 1), lacking in structural preservation compared to our method (e.g., missing helmet, scarf and incorrect facial structure, etc.). Please refer to general comment part 2 for more comparisons with Custom-Edit and further discussions.\n\n>**W2 Effectiveness in Fig.5**\n\nTo emphasize the difference, we provide a higher-resolution version of Fig.5 in Fig.A of the attached pdf. Without the Mode Shifting term, the images in Fig.5f tend to lose a portion of appearance fidelity compared to Fig.5e, resulting in a hybrid appearance closer to the source image (tends to be more silver rather than brown). Conversely, with the inclusion of the Mode Shifting term, the images in Fig.5g maintain appearance fidelity comparable to Fig.5e while displaying patterns akin to the source image, such as the blue collar and the presence of \"two cats.\" This shows the effectiveness of the Mode Shifting term in steering the model to enhance editability for the source image without compromising the personalized subject's appearance information.\n\nA similar observation can be obtained by comparing Fig.5c and Fig.5d, which depict editing outcomes derived from the same prompt. Without the mode shifting term, Fig.5c adheres to the structure of the source image yet displays inaccurate appearance, such as the incorrect coloration of the cat's face.\n\n>**W3 IQA metrics & globally computed structural metrics**\n\n**IQA metrics:** We have conducted user study, which is the mainstream way to assess edited image quality. The evaluation is enriched by using the mentioned Topiq, Musiq and LIQE with [3]. We employed the No-Reference (NR) versions of these metrics, as they are better aligned with the requirements of image editing.\n\n|                        | Topiq | Musiq  | LIQE  |\n|------------------------|-------|--------|-------|\n| Textual Inversion      |  .577 | 68.391 | 4.234 |\n| Textual Inversion+ours |  **.593** | **70.064** | **4.390** |\n| DreamBooth             |  .604 | 70.432 | 4.350 |\n| DreamBooth+ours        |  **.608** | **71.857** | **4.441** |\n| Custom Diffusion       |  .591 | 69.760 | 4.240 |\n| Custom Diffusion+ours  |  **.612** | **71.450** | **4.414** |\n\nOur work consistently improves the overall image quality of the baselines.\n\n**Structural similarity metrics computed globally:** Maintaining a structural similarity for the unedited regions are also important as the editing may modify subject-irrelevant regions, which is undesired. \n\n>**W4 Computational overhead**\n\nPlease refer the general comment part 3. \n\n>**W5 Low resolution of images**\n\nTo ensure a smooth submission process, we reduced the size of the images, which resulted in a lower resolution. To ensure that our work is presented with clarity and detail necessary for thorough evaluation and understanding, we will replace all figures with their original, high-resolution versions for camera ready version (similar to the image quality in the attached pdf). \n\n>**Q1 Training and data of personalized model**\n\nWe conduct evaluation based on publicly accessible checkpoints provided by the open repository[4]. These checkpoints, specifically for Textual Inversion, DreamBooth, and Custom-Diffusion, have been trained on 16 identical concepts from the ViCo dataset. They exhibit generation fidelity that aligns with the results reported in the original papers, which ensure a fair and direct comparison across different personalization methods.\n\n>**Q2 Reference images in Fig.2**\n\nApologies for the oversight regarding the reference images in Fig.2. Indeed, they are identical to those presented in Fig.3. We will ensure that Fig.2 is updated accordingly in the camera-ready version.\n\n>**Q3 User preference numbers**\n\nThese ratings assess the overall quality of the edited images where a higher score indicates better quality. The rating criteria include (1) structure and background preservation from the source image, (2) appearance and concept preservation from the reference images, (3) overall realism and quality of the edited image. Please refer to appendix K of the main paper for details.\n\n>**Q4 Computational overhead compared to a base method** \n\nPlease refer to the general comment part 3.\n\n>**Q5 Compare with DreamMatcher**\n\nDreamMatcher is an interesting concurrent work. However, its focus is on boosting generation fidelity rather than editing, which is the core of our work. Due to this fundamental difference in purpose, it is not directly comparable to our work. We have acknowledged this work in our extended literature review.\n\n>**Q6 Missing recent works**\n\nWe have added an extended literature review on T2I personalization, where the mentioned works are discussed (response to reviewer zDhi W1).\n\n[1] Photoswap: Personalized subject swapping in images\n\n[2] Custom-edit: Text-guided image editing with customized diffusion models\n\n[3] https://github.com/chaofengc/IQA-PyTorch\n\n[4] https://github.com/KU-CVLAB/DreamMatcher", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Global comment
We thank all reviewers for their great effort and suggestions. Some general clarifications are provided below for better understanding of both our task and our method. We will include these details in our revised paper. 
> **Part 1**

***Connection with text-driven image editing***

Traditional text-driven image editing methods generally fall into two categories: rigid editing (e.g., [1,2]) and non-rigid editing like[3]

We conclude the desiderata of rigid editing as follows: 
* the overall structure of edited image should align with the source image, 
* the edited part should follow the input instruction,
* the instruction-irrelevant part should be preserved as much as possible.

Non-rigid editing focuses on changing the view or pose of a subject in the source image while preserving the background. 

In this work, **we consider extending text-driven rigid editing** to editing scenario like 'a photo of a (silver → $v*$) cat', where $v*$ represents a specific concept derived from reference images. This concept is more detailed than generic descriptions (e.g., 'brown'), capturing intricate appearance and semantic information. We provide a flexible plug-in to bridge the gap in editability between specific concept and textual conditionings using Diffusion Models, offering a unique contribution not addressed by existing methods.

***Difference with subject swapping***

Subject swapping methods like DreamEdit[4] and Photoswap[5] diverge from our approach in their main criteria. While these methods prioritize alignment with the source subject's location and pose, they do not necessitate maintaining the original structural details as our method does. Furthermore, these works demand a stricter preservation of subject identity and typically do not require a significant level of concept extrapolation like our work, e.g., from a short cat to a tall cat. In Fig.C of the attached rebuttal pdf, we provide a comparison with DreamEdit and Photoswap in scenarios involving significant structural gaps. Compared with our work, these works often exhibits severe distortion or fails to maintain structural consistency with the source image. 

[1] Prompt-to-prompt image editing with cross attention control

[2] Instructpix2pix: Learning to follow image editing instructions

[3] Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing

[4] Dreamedit: Subject-driven image editing

[5] Photoswap: Personalized subject swapping in images
>**Part 2**

***Comparison using different image editing method***

* We employ a variant of Delta Denoising Score as the base editing model as this method provides stable editing results with all the personalized models we use. 
* However, DreamSteerer is not restricted to a specific type of editing pipeline. We also evaluate the effectiveness of our method as a plug in for Custom-Edit[3], which directly combines Custom Diffusion with Prompt-to-Prompt. The numerical results are as follows:
 
|                 | CLIP B/32  | CLIP L/14 | LPIPS (Alex) | LPIPS (VGG) | SSIM | MS-SSIM | Topiq | Musiq  | LIQE  |
|-----------------|------------|-----------|--------------|-------------|------|---------|----------|--------|-------|
| Custom-Edit      |       .748 |      .727 |         .141 |        .210 | .793|    .899|     .564 | 67.942 | 3.974 |
| Custom-Edit+ours |       **.750** |      **.729** |         .141 |        **.209** | .793|    .899 |     **.565** | **67.948** | **3.981** |
* As shown in Fig.B (upper) of the attached pdf, Custom-Edit might not adequately preserve the appearance information of the reference subject. Our proposed method can effectively improve the reference subject appearance preservation when implemented as a plug-in, without compromising the structural alignment with the source image.
* Meanwhile, as shown in Fig.D of the attached pdf, combining Prompt-to-Prompt with DreamBooth can introduce significant appearance artifacts in the edits, which is the main reason we did not use it as the base editing method. Prompt-to-Prompt relies on a source latent state inversion process, typically through Null-Text Inversion (NTI). However, parameter update during personalization can shift the model distribution for the source class, compromising the editability of the inverted latent state chain with NTI. In comparison, the Delta Denoising Score based edited method employed in our work does not require a inversion process, providing more robust performance across different types of personalization baselines. We believe this phenomenon worth further investigation and encourage future works to develop new inversion techniques specifically tailored for the personalized models. 

>**Part 3**

***The computational cost of DreamSteerer***

The total sampling & fine-tuning time of DreamSteerer takes ~2 minutes on a single V100 GPU, that comes from two parts: (1) spatial feature guided sampling: this part is tuning-free and contains a DDIM inversion process on the single source image and a guided generation process. With a DDIM scheduler step size of 50, this process takes ~1 minute on a single v100 GPU, (2) the EDSD fine-tuning on personalized model parameters: this process takes a small number of 10 optimization steps with a batch size of 1 and a gradient accumulation step of 10, the fine-tuning time depends on the exact base personalized model. For the largest model DreamBooth, which uses full fine-tuning, it takes ~1 minute. We provide a comparison of the total number of UNet prediction steps (NFE) between our fine-tuning strategy and the baseline personalized models. 
|                   | NFE  |
|-------------------|------|
| Textual Inversion | 5000 |
| DreamBooth        | 1600 |
| Custom Diffusion  | 2000 |
| Ours              | 200  |

The fine-tuning and inference of the whole pipeline is also able to be conducted on a single 3090 GPU for the largest base model DreamBooth.

Meta-review: Though the paper has mixed reviews, the majority of reviewers are leaning toward acceptance. The reviewers appreciate that DreamSteerer enhances the users ability to edit personalized diffusion models, and that the authors have conducted a thorough quantitative and qualitative analysis of the results, including a human evaluation of results. The AC team agrees that the value of the contribution is sufficient for publication at NeurIPS and thus recommend acceptance.
Decision: Accept (poster)