Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing
Authors: Kai Wang
Fei Yang
Shiqi Yang
Muhammad Atif Butt
Joost van de Weijer
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion Models; Text-guided Image Edit; Textual Inversion; Localization
TL;DR: Existing text-guided image editing methods suffer from cross-attention leakage, we solve this problem by updating new added dynamic tokens for each noun word with the proposed losses. By this way, we achieve enhanced text-guided editing performance.

Abstract: Large-scale text-to-image generative models have been a ground-breaking development in generative AI, with diffusion models showing their astounding ability to synthesize convincing images following an input text prompt. The goal of image editing research is to give users control over the generated images by modifying the text prompt. Current image editing techniques are susceptible to unintended modifications of regions outside the targeted area, such as on the background or on distractor objects which have some semantic or visual relationship with the targeted object. According to our experimental findings, inaccurate cross-attention maps are at the root of this problem. Based on this observation, we propose $\textit{Dynamic Prompt Learning}$ ($DPL$) to force cross-attention maps to focus on correct $\textit{noun}$ words in the text prompt. By updating the dynamic tokens for nouns in the textual input with the proposed leakage repairment losses, we achieve fine-grained image editing over particular objects while preventing undesired changes to other image regions. Our method $DPL$, based on the publicly available $\textit{Stable Diffusion}$, is extensively evaluated on a wide range of images, and consistently obtains superior results both quantitatively (CLIP score, Structure-Dist) and qualitatively (on user-evaluation). We show improved prompt editing results for Word-Swap, Prompt Refinement, and Attention Re-weighting, especially for complex multi-object scenes.

Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing
Kai Wang Fei Y Shiqi Yang Muhammad Atif Butt Joost van de Weijer
Computer Vision Center, Barcelona, Spain
2 Universitat Autonoma de Barcelona, Barcelona, Spain
College of Computer Science, Nankai University, Tianjin, China Okwang, syang, mabutt, maautt,joostteeeeeeeeee joost) uab.es, feiyangdnankai. wewaaaiieddcc....
Abstract
Large-scale text-to-image generative models have been a ground-breaking develop- ment in generative AL, with diffusion models showing their astounding ability synthesize convincing images following input text prompt. The goal of image editing research to give users control over the generated images by modifying the text prompt. Current image editing techniques are susceptible to unintended modifications regions outside the targeted area, such as on the background or on distractor objects which have some semantic or visual relationship with the targeted object. According to our experimental findings, inaccurate cross-attention maps are at the root of this problem. Based this observation, we propose Dynamic Prompr Learning (DPL) to force cross- attention maps to focus on correct nown words the text prompt. By updating the dynamic tokens for nouns in the textual input with the proposed leakage repairment losses, we achieve fine-grained image editing particular objects while preventing undesired changes to other image regions. Our method DPI. based on the publicly available Stable Diffusion, extensively evaluated wide range of images, and consistently obtains superior results both quantitatively (CLIP score, Structure-Dirs Dist) and qualitatively (on user-evaluation). We show improved prompt editing results for Word-Swap. Prompt Refinement, and Attention Re-weighting, especially for complex multi-object scenes.
Introduction
Text-to-Image (T2I) is advancing at a revolutionary pace and has demonstrated an unprecedented ability to generate diverse and realistic images [37 34 35). The state-of-the-art T2I models trained on extremely large Lunguage-image datasets, which require huge computational resources. However, these models do allow for straightforward text-guided image editing and generally lack the capability to control specific regions of a given image.
Recent research on text text-guided image editing allows users to easily manipulate an image using only 24 42 43). this paper, we focus on prompt based image editing, where we aim to change the visual appearance of target object (or objects) in the image. Several of the existing methods use DDIM inversion 40) to attain the initial latent code of the image and
text prompts (5] 9 14 [5]
then apply their proposed editing techniques along the denoising phase 228 30 42 Nevertheless, current text- text-guided editing methods are susceptible to unintentional modifications of image areas. We distinguish between two major failure cases: unintended changes of background regions and unintended changes of distractor objects (objects that are semantically or visually related to the target object), shown in Fig. The functioning of existing editing methods greatly depends
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Promph cat and dog
pixipix-zem
GSAM inpaint InstructPicPPix Plug and ald-Pand
NTI P2P
DPL P2P
dog -
lion
dog + lion
Average Cross-Attention maps
DiffEdic
11N
DPL
<jelartoftextl>
<jendofled|p
cat mask
Figure 1: Compared with other image editing methods, our DPI. achieves more consistent layout when modifying one more objects in the image and keeping other content frozen. And also the desired editing is correctly delivered to the corresponding area. DiffEdit [9] method thoroughly fails to detect the editing region in multi-object cases. And Null-Text inversion (NTI (28) is unable to perfectly distinguish the objects in the given image since the cross-attention maps are suffering background leakage (yellow circle) and distractor object leakage (red circle). DPL can more successfully localize the objects in the textual prompts, thus benefits for future editing. Here the 16 x 16 cross-attention maps are interpolated the image size for better view. (image credits: gettyimages)
the accuracy of the cross-attention maps. When visualizing the cross-attention maps (see Fig. D.. we can clearly observe that for DDIM [40] and Null-Text inversion (28) these maps do not only correlate with the corresponding objects. We attribute this phenomenon cross-attention leakage background and distractor objects. Cross-attention leakage is the main factor to impede these image editing methods work for complex backgrounds and multi-object images. Similar observations for failure cases are also shown in 99 [14 19 [2 28). For example, Custom Diffusion [22] fails for multi-concey composition and DiffEdit does not correctly detect the mask region for a given concept.
We conduct a comprehensive analysis of the Stable Diffusson (SD) model and find that the inaccurate attribution of cross-attention (resulting in undesirable changes to background and distractor objects) can be mitigated by exploiting the semantic clusters that automatically emerge from the self-attention maps in the diffusion model. Instead of fine-tuning the diffusion model, which potentially leads to catastrophic (eglect[],, we propose replacing prompts related to scene objects with dynamic prompts that optimized. We propose two main losses for dynamic prompt learning (DPL), one tailored to reduce attention leakage to distractor objects and one designed to prevent background attention leakage. Moreover, better cross-attention maps are obtained when we update the prompts for each timestamp (as implied by dynamic). Furthermore, to facilitate meaningful image editing, we use Null- Text Inversion 28 with classifier-free guidance [17. In the experiments, we quantitatively evaluate the performance of DPL images from LAION-5B [3 dataset. And DPL shows superior evaluation metrics (including CLIP-Score and Structure- Dist) and improved user evaluation. Furthermore, we show grompt-editing results for Word-Swap, Prompt Refinement, and Attention we-weighting on complex multi-object scenes.
2 Related work
leverage CLIP [32] for image editing based on pretrained unconditional diffusion model. However, they are limited to the generative prior which only learned from visual data of specific domain, whereas the CLIP text-image alignment information is from much broader data. This prior knowledge gap is
For text-guided image editing with diffusion models, various methods 22
20 23 24

---

mitigated by recent progress of text-to-image (T2I) models 66 10 18 33 34 38
Nevertheless, these
T2I models offer little control over the generated contents. This creates great interest in developing methods to adopt such T2I models for controllable image editing. SDEdit [26], a recent work utilizing diffusion models for image editing, follows a two-step process. introduces noise to the input image and then applies denoising using the SDE prior to enhance realism and align with user guidance. Imagic [19] and P2P
[5] attempt airucture-preservee However, Imagic [99 requires fine-tuning the entire model for each image and focuses generating variations for the objects. P2P 15] has no need to fine-tune the model, it retrains the image structure by assigning pross-attention maps from the original image to the edited one in the corresponding text token. InstructPix2Pix [4] is an extension of P2P by allowing human- like instructions for image editing. To make the P2P capable of handling real images, Null-Tex inversion (NTI) [28 proposed to update the null-tex/ embeddings for accurate reconstruction to accommodate with the classifier free guidance [17]. Recently, pix2pix-zero to retrain the structure of a given image. However, it only supports image translation from one concept to another one and is not able to deal with multi-concept cases. DiffEdit ( automatically generates a mask highlighting regions of the input image by contrasting predictions conditioned on different text prompts. Plug-and-Play (PnP) [42] demonstrated that the image structure can be preserved by manipulating spatial features and self-attention maps in the T2I models. MnM [31] clusters self-attention maps and compute their similarity to the cross attention maps to determine the object localization, thus benefits to generate object-level shape variations. StyleDiffusion (4 adds up a mapping network to modify the value component in cross cros-attentionn computation for regularizing the attentions and utilizes P2P for image editing.
editing via Stable Diffusion (SD) models.
30 propose noise regularization and cross-attention guidance
These existing methods highly rely on precise cross-attention maps and overlook an important case where the cross-attention maps may not be perfectly connected to the concepts given in the text prompt, especially if there are multiple objects in the given image. Therefore, these imperfect cross-attention maps may lead to undesirable editing leakage to unintended regions. Essentially, our method DPL complements existing methods to have more semantic richer cross-attention maps. There are various diffusion bbaed image editing methods relying on giving a mask manually [29] or detecting a mask automatically (GSAMMinpaint 44) using pretrained segmentation model. On the contrary. our method only requires textual input and automatically detects the spatial information. This offers a more intuitive editing demand by only manipulating the text prompts.
There are some papers 21 12] concentrated on object removal or inpainting using user- user-defined masks 1 20 44 In contrast, the act of placing an object in suitable position without relying on a mask is an active research domain. This direction usually entails fine-tuning models for optimal outcomes and runs parallel to the research direction explored in this paper.
In this paper, we are also related to the transfer learning for T2I models (22 [ 25), which aims at adapting given model to new concept by given images from the users and bind the new concept with a unique token. However, instead of finetuning the T2I model [25] [3] 37], we learn new concept tokens by personalization in the text embedding space, which is proposed in Textual Inversion [1I. By this means, we avoid updating a T2I model for each concept separately. Furthermore, step forward to the scenario with complex background or multiple objects, where the previous methods have not considered. In contrast to transfer learning approaches, which aim to ensure that the learned concepts can accurately generate corresponding objects in given images, our proposed approach DPL focuses on enhancing image editing tasks by regularizing cross-attention to disentangle their positions.
3 Method
In this section, we first provide short preliminary and then describe our method to achieve this demand. An illustration of our method is shown in Fig. and Algorithm TT
3.1 Preliminary
Latent Diffusion Models. We use Stable Diffusion v1.4 which Latent Diffusion Model (LDMM) 35). The model is composed of two main components: an autoencoder and a diffusion model. The encoder& from the autoencoder component of the LDMSs maps an image I into latent code 20 {T) and the decoder reverses the latent code back to the original image as D(E(I)) x I. The
3

---

Editing
22'
Caption
Diffusion Model
A and robbit"
Tokenizer
320 Sech
40 370 Codo
Embedding Lookup
Sec. 3.3
Lig soo
* Las "i
1000
1300
Soft ottondion
AA
P-pOy
Diffusion Model
Text Transformer
Cross omerton
Zy-1
Allow
null-ttext
Figure 2: Dynamic Prompt Learning (DPL) first transforms the words in the text prompt into dynamic tokens. We register them in the token dictionary and initialize the representations by their original words at time T. Using DDIM inversion, we condition the DM-UNet with the original text condition C to get the diffusion trajectory {2:j0 17 and background mask B. In each denoising step --1 + 2g, first update the dynamic token set V, with background leakage loss, a disjoint object attention loss and an attention balancing loss, in order to ensure high-quality cross-attention maps. Then we apply Null-Text inversion approximate the diffusion trajectory.
diffusion model can be conditioned on class labels, segmentation masks or textual input. Let To (v) be the conditioning mechanism which maps condition y into conditional vector for LDMs, the LDM model is updated by the loss:
[|| co(21, (o()))
LLDM E
The neural backbone eg is typically a conditional UNet [36] which predicts the added noise. More specifically, text-guided diffusion models aim to generate an image from random noise 27 and a conditional input prompt P. To distinguish from the general conditional notation in LDMs, we itemize the textual condition as C (P))
DDIM inversion. Inversion entails finding an initial noise zт that reconstructs the input latent code zo upon sampling. Since we aim at precisely reconstructing given image for future editing, employ the deterministic DDIM sampling 40
2++1 Va C) / @041co(21,
(2)
where ăz+1 is noise scaling factor defined in DDIM 40 and fo(2,, t, predicts the final denoised latent code 2o as 10(,,, = -T 0 2,, L, /Vo..
Null-Text inversion. To amplify the effect of conditional textual input, classifier-fret guidance [17 proposed extrapolate the conditioned noise prediction with an unconditional noise prediction. Let Ø = To( (")) denote the null text embedding, then the classifier-fres guidance is defined as:
Eo(2,, t, C, Ø) = w co(2,, t, C) + (1 w) cc((2,, f, 2)
where set the guidance scale 7.5 as is standard for Stable Diffusion 17 the introduction of amplifier-free free guidance complicates the inversion process, and the generated image based on the found initial noise 2y deviates from the original image. Null-text inversion (28) proposes a novel optimization which updates the null text embedding 2, for each DDIM step / E [1, T to approximate the DDIM trajectory [=:]ō according
28 35
However,
min | (0, t,, 21)1
(4)
where { is the backward trace from Null-Text inversion. Finally, this allows to edit real images starting from initial noise žy 2T using the learned null-text 2; in combination with P2P (15).

---

Note, that the cross-attentiin maps of Null-Tex, Text inversion (w = 7.5) and DDIM inversion (w = 1.0) are similar, since NTI is aiming to approximate the same denoising path.
Prompt2Prompt. P2P [1] proposed achieve editing by manipulating the cross-attention Ai from target input P according to the cross-attention Ar from the source input P each timestamp t. The quality of the estimated attention maps A, is crucial for the good functioning of P2P. This paper addresses the problem of improved Ar estimation for complex images.
3.2 DPL: Dynamic Prompt Learning for addressing cross-attention leakage
Prompt-based image editing takes image I described by an initial prompt P. and aims to modify it according to an altered prompt P in which the user indicates desired changes. The initial prompt is used compute the cross-attention maps given the conditional text input P. As discussed in Sec. [ the quality of cross-attention maps plays a crucial role for prompt-based image editing methods
(see Fig. D since these attention maps will be used during the generation process with the altered user prompt [15]. We identify two main failures in the attention map estimation: (1) distractor object leakage, which refers to wrong assignment of attention to a semantically or visually similar foreground object and (2) background leakage, referring to undesired attention on the background. Both of these problems result in artifacts or undesired modifications in generated images.
The cross-attention maps in the Diffusion Model UNet are obtained from co [2,, f, C),, which the first component in Eq. 3 They computed from the deep features of noisy image (z+) which
projected to a query matrix Q; = (g(x(()) (=)), and the textual embedding which is projected to key matrix K- 1x (C). Then the attention map is computed according
A = so softmaz(Qq KTT (KT/Va) /V)
(5)
where d is the latent dimension, and the cell [Arli defines the weight of the j-th token on the pixel i. In this paper, we propose to optimize the word embeddings corresponding to the initial prompt P. in such a way that the resulting cross-attention maps A, do not suffer from the above-mentioned problems, i.e. distractor-object and background leakage. We found that optimizing the embedding i separately for each timestamp / leads to better results than merely optimizing a single e for all timestamps. The initial prompt P contains K noun words and their corresponding learnable tokens at each timestamp Vi = [u/ of..., ...., v,55 }. We update each specified word in V2 for each step t.
The final sentence embedding C, now varies for each timestamp t and is computed by applying the text transformer on the text embeddings: some of which are learned (those referring to nouns) while the rest remains constant for each timestamp (see Fig. ( For example, given a prompt "a puppy and a rabbit" and the updatable noun set (puppy, rabbit), in each timestamp & we have transformed the textual input into trainable ones as "a <puppy> and a <rabbit>" and we aim to learn word embeddings 2/ <puppy>> and i yrrabbi>> in each timestamp
We propose several losses to optimize the embedding vectors Vy: we develop one loss to address cross-attention leakage due to object distractors, and another one address background leakage. We also found that the attention balancing loss, proposed by Attend-am and-ancitte 77. was useful to ensure each specified noun word from P focusing on its relevant concept.
Disjoint Object Attention Loss. To address the problem of distractor object attention leakage, propose a loss that minimizes the attention overlap between the attention maps corresponding to the different objects. that is an updatable noun set Vr. This prevents specific objects related regions be linked to multiple nouns, which is one of the sources of distractor object attention. We propose to minimize a cosine similarity loss among the K cross-attention maps (referring the noun words in the input prompt P) to avoid their overlapping in cross-attention maps through:
Caj
(A AAt (At)
(6)
Background Leakage Loss. In order to alleviate the background leakage issue, we propose utilize background masks to keep the background area unchanged. Details on how to get the background mask are discussed in Sec. 3.3 After obtaining the BG mask B of the given image, for each of e V.,
5

---

Cat
CA.Dog
Self-Attention
Query
Key
Value
16:16
32'32
64"64
Figure 3: After DDIM inversion, we visualize the average of each component over ' timestamps for the input image Fig. 1 (with the prompt: "a cat and dog"). From left to right, they are cross-attention (CA) for each now word, then PCA&clustering of the self-attention (SA), query, key and value representations. Here, the maps are simply resized without any interpolation to demonstrate their original views.
we force its cross-attention map have minimal overlap with the background region:
Cbg cos(A,) B)
Attention Balancing Loss. For each oy E Ve, we pass its cross-attention map through a Gaussian filter F and formulate set of losses L.p for all words in Vr, The attention loss is defined as the maximum value in Lez:
Lat max C,f EV.
where =1- max ((A )]
(8)
The attention loss will strengthen the cross-attention activations for localizing token v/. thus force Aj to not leak attentions to other irrelevant concepts in the image. In other words, the resulting cross-attention maps will be more concentrated on the corresponding object region. Different from Attend-and- -xxiitte-- 7 where the refinement loss applied to update the latent code to augment the multiple object generation,
the first to explore its role in learning new conceptual tokens.
With all these three losses, we aim to update the learnable token of as:
arg min L where L= >at Lat + daj Laj + Abg Lbg
V2
Note that, when there is only one updatable noun token in the text input P, we only apply the Lon to relieve the background leakage since other losses rely on least two words. In this paper, we apply DPL over the 162 attention following the common practices [15] for a fair comparison.
Gradual Optimization for Token Updates. So far, we introduced the losses to learn new dynamic tokens at each timestamp. However, the cross-attention leakage is gradually accumulated in the denoising phase. Hence, we enforce all losses to reach a pre-defined threshold at each timestamp to avoid overfitting of the cross- cross-attention maps. We express the gradual threshold function. For the losses proposed above, the corresponding thresholds at time are defined as:
exponential
THE Bat exp(-t/Aaat:
TH = Bay exp(-t/o.j): TH = 3bg
(10)
We verify the effectiveness of this mechanism together with other hyperparameters in our ablation experiments (see Fig. (a) and Supplementary Material.).
Null-Text embeddings. The above described token updating ensures that the cross-attention maps highly related to the noun words in the text prompt and minimize rros-attention leakage. To

---

Self-Atiention
Cross-Attn-1
Cross-Attn-2
Background
Foreground 1
Foreground 2
Figure 4: Although cross-attention maps produce acceptable results in certain cases (up) with minor leakage, the object masks from self-cross attention matching are not always accurate due to the leakage problem (down). However, as the cross-attention maps tend to concentrate on the foreground objects, the background masks remain reliable.
be able to reconstruct the original image, we use Null-Text inversion 28 in addition to learn set of null embeddings 2, for each timestamp Then at the time we have a set of learnable word embeddings V, and null text Ø, which can accurately localize the objects and also reconstruct the original image.
3.3 Attention-based background estimation
To address the background attention leakage, investigate the localization information of self- attention and feature representations, which are
Algorithm 1:
Dynamic Prompt Learning
Input: A source prompt P, input image I Output: A noise vector žy, a set of updated tokens [V:J and null-text embeddings {Ø,}
present in the diffusion model. Besides cross- attention (CA), during the inversion of im- age with the DDIM sampler, we can also obtain
self-attention maps (SA), directly analyze the
Set T = 50 and scale w = 1;
query, key and value in the self-attention maps.
{ )T BB DDDIM-nnn(())
In Fig. [3 we visualize these maps. The main ob-
Set guidance scale w 7.5:
servation is that self-attention maps more closely
Initialize Vr with original noun tokens;
resemble the semantic layout. We therefore ex-
7 Initialize žт = 2T, PT = P, ØT = To (""); T T, T 1, 1.... do
ploit these to reduce background leakage.
for t
Initialize P. by V2, then C, = To (P));
To acquire the background via self-cross atten-
tion matching, we use the agreement score [31] between cross-attention A" of noun words n
while Las THe Laj TH,
Dog 2 TH do
and each of the self-attention clusters Ma, v E
= Aut Lat + Aaj Las Aog Log
[1. VJ. defined as Ree (AA- Σ M
12
V1 - Vj Vvy
However, should be noted that the agreement score does not accurately segment the objects,
13 end
14
Update P. by V,, then Ci = (o(PP); ã, = Eo(ā,, Cr, Ø.)
since there is still attention leakage between ob- jects and distractor objects (see Fig. for exam- ples). Therefore, other than localizing objects 17 in (31, we use the agreement score only for background mask estimation. We label the clus- 19 Return ēT, (V.17 {0.)
15
16 2,-1, + NTI(E,, Rx) Initialize 0,-1 = E2 Vi-1 Vv
18 end
ter M being background (BG) if for all if owe < TH, where H' is a threshold. Empirically, we choose ooff-attention with size 32, cross- attention with size 16 and TH 0.2, V 5 to obtain the background mask B.
4 Experiments
We demonstrate our method in various experiments based on the open-source T2I model Stable Diffusion 355 following previous methods [4]] 30 (28). All experiments are done on an A40 GPU.
7

---

26-
LIN
83
02
DPL
60
02
3.0
1.3
Thrasheld
<bookE>
cclecko
<pitae
<hidd
(a) JoU curves Valid-spiit (lef)) Test (right)
(b) attention maps comparison
Figure 5: (a) IoU curves drawn by varying the threshold from 0.0 to 1.0 for MO-Set. (b) We show some images for comparisons between the Null-Tex: inversion 28 and our method DPL demonstrated in cross-attention maps.
Dataset. We focus here on real-image editing, but also provide examples of generated image editing in the Supplementary Material. We clip retrieval [3] to obtain experimental multi-object real images from the LAION-5B [39] dataset by matching CLIP embeddings of the source text description. For the ablation study, select 100 multi-object images from various search prompts (including concepts as bird, dog, cat, book, clock, pen, pizza, table, horse, sheep, etc.). This Multi-Object set is abbreviated with MO-Ser. Of the images, 40 images used for validation and the other 60 test set. We manually verify their segmentation maps which serve as groundtruth in the ablation study loU computation. Finally, for user study, create URS-Set of 60 multi-object images of semantically related objects from various concepts.
Comparison methods. For qualitative comparison, we compare with pix2pix-7ero 30 GSAM- inpaint 44], InstructPix2Pix []) Plug-and-Play [42] and NTI+P2P [28] For quantitative evaluation, we benchmark our approach against NTI+P2P, considering it as a robust baseline for maintaining content stability and achieving superior conceptual transfer.
4.1 Ablation study
The ablation study is conducted on the MO-Set set. To quantitatively evaluate the performance of DPL in localization, we vary the threshold from 0.0 1.0 obtain the segmentation mask from the crsss-attnntinn maps and compute the loU metric using the segmentation groundtruth for comparison.
In Fig- (a),, compare the localizing performance with Null-Text inversion (NTT), DPI. with each proposed loss separately and our final model. We can observe that the disjoint object attention loss and the background leakage loss are able to work independently improve the cross-attention map. The attention balancing loss can enhance the cross-attention quality and can be further boosted by the other two proposed losses. We the validation set to optimize the various hyperparameters (see Eq. andd Eq. (O). Using these parameters, we show similar results on the independent test-set (Fig. 5
(a)). These parameters are for all further results (more details are provided Supplementary). In
Fig. (b), we qualitatively verify our method on some examples of comparison with cross- aross-attention
maps. DPI. improves the cross-attention quality than NTI as the baseline method.
4.2 Image editing evaluation
For image editing, combine our proposed DPL with the P2P (15] image editing method. Here we mainly focus on Word-Swap but also include more results for Prompt Refinement, and Attention Re-weighting in the supplementary material.
Word-Swap. In the word swapping scenario, we quantitatively compare NTI and DPL under two scenarios shown in Table For each case, we use the collected 20 images with searching prompts "a cat and a dog" and "a box and a clock" estimate the evaluation metrics, including CLIP-Score [16] and Structure Dist [1.. both cases, DPL is getting better performance than the NTI baseline (more shown in Supplementary).
Furthermore, we conduct a forced-choice user study involving 20 evaluators and the assessment of 60 image editing pairs. Users were asked to evaluate 'which result does better represent the prompt change?' (more protocol choices can found Supplementary). We can observe a significantly higher level of satisfaction among the participants with our method when compared to NTI.

---

Table 1: Comparison between method DPL +P2P and the baseline NTI+P2P by modifying multiple objects simultaneously. we examined the quality of editing by transforming an image depicting "a cat and dog" into "a leopard and tiger". We evaluated the editing quality of both approaches using objective metrics, such as CLIP-Score and Structure-Dist, across these multi-object collections. Additionally, capture the subjective opinions of users, we conducted user study involving 60 pairs of multi-object image translations.
leopard and dog - tiger
book + box clock sunflower
Method
User Study
CLIP-Score (1)
Structure Dist (1)
CLIP-Scone (f)
Structure Dist (J)
Original Images NTI 28 + P2P
32.50%
16.46%
31.50%
0,0800
12.91 %
0.0191 0.0163
12.8% 87.2 %
DPL (ours) P2P
32.23%
0.0312
14.29 %
piece ccokes on a wooden plate
cakon
Jenon (1)
Temon oj
podlo core
posto care
pasta oo
Cross Artentios
4E.d TAN
Ca
2044
Sysk
Figure 6: Attention refinement and reweighting. the given text prompt, only one noun word is learnable. The flowers below the cake are playing as distractors which distort the cross-attention maps while Null-Text Inversion [28] is applied. As a comparison, our method DPL successfully filters the cross-attention by our background leakage loss.
Qualitatively, we show word-swap examples on multi oujett images in Fig. 7 In contrast to other methods, our proposed approach (DPL) effectively addresses the issue of inadvertently editing the background or distractor objects in the provided images.
Attention refinement and re-weighting. We also compare NTI and DPL by only tuning a single concept token with additional adjective descriptions in the image in Fig. As can be observed, DPL keeps better details of the "cake" and modifies its appearance according to the new word. NTI suffers from the cross attention leakage to the background, thus it cannot successfully edit only the "cake" region and distorts the "flowers".
5 Conclusions
We presented DPL solve the cross-attention background and distractor object leakage problem in image editing using text-to-image diffusion models. We propose to update the dynamic tokens for each noun word in the prompt in such way that the resulting cross-attention maps suffer less from attention leakage. The greatly improved gross-attention maps result in considerably better results for text-guided image editing. The experimental results, confirm that DPL obtains superior results, especially on complex multi-object scenes. In the Supplementary Material, we further provide comprehensive discussion on the limitations of our work, its broader impact, and potential directions for future research.
Limitations and future works. One limitation of our work is that the smaller cross-attention maps, specifically those with size of 16x 16, contain a greater amount of semantic information compared to maps of larger sizes. While this rich information is beneficial, it limits our ability to achieve precise fine- -rainnd structure control. Furthermore, our current approach has not addressed complex scenarios where a single object in the image corresponds to multiple noun words in long caption. This remains an unexplored area and presents an opportunity for future research. Additionally, the Stable Diffusion model faces challenges when reconstructing the original image with intricate details due to the compression mechanism in the first stage autoencoder model. The editing of high-frequency information remains significant and ongoing challenge that requires

---

Input Image
piczpix-zero
inpaint
InstructPliPiix
Plug and-Play
<cat> and <dog>
Leoppedd
<apple>
PPoepple
Pinappple
<birds
Cell
Drd Hasicapee
<baskerballl> cfootballi. <tennis balls and cracket>
Figure 7: Word-Swap by only modifying the regions corresponding to the target concept. We list the source prompts on the left with dynamic tokens in the curly brackets. Each line corresponding to modifyyng one multiple concepts with various methods.
further investigation and development. Addressing these limitations and advancing our understanding in these areas will contribute to the improvement and refinement of image editing techniques.
Broader impacts. The application of text-to-image (T2I) models in image editing offers extensive potential for diverse downstream applications, facilitating the adaptation of images different contexts. The primary objective of our model is to automate and streamline this process, resulting in significant time and resource savings. It is important acknowledge that current methods have inherent limitations, as discussed in this paper. However, our model can serve an intermediary solution, expediting the creation process and offering valuable insights for further advancements. is crucial to remain mindful of potential risks associated with these models, including the dissemination of misinformation, potential for abuse, and introduction of biases. Broader impacts and ethical considerations should be thoroughly addressed and studied in order to responsibly harness the capabilities of such models.
Acknowledgments.
We acknowledge the support from the Spanish Government funding for projects PID2022-143257NV
TED2021-1325 130-100 funded by MCIN/AEI/IO. 13039/501 10001 1033 and by FSE+ and the European Union NextGenneraioonE//PRRR. and the CERCA Programme of Generalitat de Catalunya. We also thank for the insightful discussion with Juan Rodriguez, Danna Xue, and Mare Paz.
10

---

References
[1] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18208-18218, 2022.
[2] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In European Conference on Computer Vision, pages 707-723. Springer, 2022.
[3] Romain Beaumont. Clip retrieval: Easily compute clip embeddings and build clip retrieval system with them. httpp:///githul: http:///..ttt..cccooo/o/o/////// 2022.
[4] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
[5] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. arXiv preprint arXiv:2304.0846. 08440044046 2023.
[6] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint 1rXxi:2301.00000 2023.
[7] Hila Chefer, Yuval Alaluf, Yacl Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and- excite: Attention-based semantic guidance for text-to-image diffusion models. arXiv preprint arXiv:2301.13826, 2023.
[8] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. llvr: Conditioning method for denoising diffusion probabilistic models. In Proceedings of the International Conference on Computer Vision, pages 14347-14356. IEEE, 2021.
[9] Guillaume Couairon, Jakob Verbeek, Holger Sehwenk, and Matthicu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. In The Eleventh International Conference on Learning Representations, 2023.
[10] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based lext-to-image generation with human priors. In European Confer- ence on Computer Vision, pages 89-106. Springer, 2022.
[11] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth word: Personalizing text-to-image generation using textual inversion. arXiv preprint ar arXiv22288.0166 2022.
[12] Rohit Gandikota, Joanna Materzynska, Jaden Frotto-Kaufmaa and David Bau. Erasing concepts from diffusiin models. Proceedings of the International Conference on Computer Vision, 2023.
[13] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning, 2023.
[14] Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. arXiv preprint arXiv:2304.07096 2023.
[15] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-promp image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
[16] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514-7528, 2021.
[17] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. axX1v:220..12598, 2022.
arXiv preprint

---

[18] Susung Hong, Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. arXiv preprint XXiv:22100009334 2022.
[19] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[20] Gwanghyun Kim, Taesung Kwon, and Jong Chul Yc. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2426- 2435, June 2022.
[21] Nupur Kumari, Bingliang Zhang, Sheng- Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu, Ablating concepts in text-to-image diffusion models. Proceedings of the International Conference Computer Vision 2023.
[22] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi- concept customization of text-to-image diffusion. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[23] Gihyun Kwon and Jong Chul Ye. Diffusion-based image translation using disentangled style and content representation. In The Eleventh International Conference on Learning Representations, 2023.
[24] Senmao Li, Joost van de Weijer, Taihang Hu, Fahad Shahbaz Khan, Qibin Hou, Yaxing Wang, and Jian Yang. Stylediffusion: Prompt-embedding inversion for text-based editing, 2023.
[25] Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept neurons in diffusion models for customized generation. arXiv, 2023.
[26] Chenlin Meng. Yutong He, Yang Song, Jiaming Song. Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
[27]
Midjourney.com. Midjourney. https: /ttts::/www.... www. wwwdmmiddjor.y.. com 2022.
[28] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or Or. Null-text inversion for editing real images using guided diffusion models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[29] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image genera- tion and editing with text-guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th Inter- national Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages
PMLR, 17-23 Jul 2022.
[30] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. Proceedings of the ACM SIGGRAPH Conference on Computer Graphics, 2023.
[31] Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elon and Daniel Cohen-Or. Localizing object-level shape variations with text-to-image diffusion models, 2023.
[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[33] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[34] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
12

---

[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages (0684-10695, June 2022.
[36] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assiseed Cnanvention-MICCAN 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[37] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[38] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 1487, 2022.
[39] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[40] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[41] Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10748-10757, 2022.
[42] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-plas diffusion features for text-driven image-to-image translation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[43] Qian Wang, Biao Zhang, Michael Birsak, and Peter Wonka. Mdp: A generalized framework for text-guided image editing by manipulating the diffusion path, 2023.
[44] Tao Yu, Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, and Zhibo Chen. Inpaint anything: Segment anything meets image impainting, arXiv preprint arXiv:2304.06790 2023.
13


Reviews:
[{"review_id": "NRkpT2tY0Q", "reviewer": "Reviewer_4SLV", "summary": "This paper works on the fidelity problem (i.e., 'unintended changes of background and distractor objects') in text-based image editing of text-to-image diffusion models. The authors attribute the fidelity problem to cross-attention leakage and propose dynamic prompt learning to force cross-attention maps to focus on correct noun words in the text prompt. The paper shows a comparison with a couple of approaches, comprehensive image editing evaluations and ablations.", "questions": "1. Please clarify the similarities and differences between DPL and Imagic [2]. In my opinion, both are looking for optimized text embedding, while Imagic seems to remain open to control objectives, while DPL adds too many restrictions through attention maps.\n\n2. How long do training and inference take in the proposed method? Can the authors detail the advantages of DPL compared to the training-free methods (e.g., StructureDiffusion [3])?\n\n3. What are the dimensions of the attention maps, the query/key matrix, and the deep features of the noisy image.  It will be great if the authors could indicate those dimensions in the paper.", "limitations": "The authors have provided discussion on the limitations and broader impact in the Supplementary Material. However, as mentioned in the weaknesses, this work has the following potential limitations: \n- 1) The attention map limits the size of the generated target.\n- 2) Not applicable to the scenarios that the noun describes the background, or changes are reflected in verb word, such as adding an item or deleting objectives.\n- 3) Not training-free and may have high latency in practical applications.", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The fidelity problem that this work focuses on is important. The overall idea of the paper is really simple and seems effective to this problem.\n\n2. The authors present a good number of experiments validating the effectiveness of their approach.  The results from the paper are promising. The editing quality is decent with observable changes. \n\n3. The paper is overall well-written and easy to follow.", "weaknesses": "\n1. This work has the following potential limitations: \n- The attention map limits the size of the generated target, as shown in Figure 7.\n- Not applicable to the scenarios in that the noun describes the background, or changes are reflected in the verb word, such as adding an item or deleting objectives. Thus, more experiments in the above scenarios should be provided and discussed.\n- Not training-free and may have high latency in practical applications.\n2. Both qualitative and quantitative results miss comparison with important methods such as DiffEdit [1], Imagic [2], StructureDiffusion [3].\n\n3. The manuscript lacks explanations for some symbols, such as symbols in Equation (6). It would be great if the author could highlight these symbols in the diagram to refine the explanation of the method.\n\n[1] Diffedit: Diffusion based semantic image editing with mask guidance. ICLR 2023.\n\n[2] Imagic: Text-based real image editing with diffusion models. CVPR 2023.\n\n[3] Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis. ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and will integrate the discussions to improve. We use citations for references in the main paper and any omitted references are listed below.\n\n$\\textbf{W1.1}:$ Imposing strict DPL indeed limits the editing region, which aligns with practical scenarios. Our goal is to position the generated target object at the source object's original spot while maintaining overall stability. This constraint is akin to attention injection seen in cross-attention[4,9,14,26,28] and self-attention methods[5,40]. Adjusting the size of the generated target remains unexplored. Nonetheless, the constraint can be relaxed using partial attention injection as depicted in Fig.6 of P2P[14]. The effect of attention injection across varied steps for DPL is shown in Fig. 19 of the rebuttal PDF.\n\n$\\textbf{W1.2}:$ Your observations are correct: $\\textbf{(1)}$ In the context of background descriptions, the current research refers to it as “global editing”. While we've included global editing performance in Fig.14 of the supplementary, our main emphasis is on identifying object positions corresponding to noun words. Consequently, we don't exhibit significant advancements in this case. $\\textbf{(2)}$ Editing involving verb words often requires T2I model fine-tuning, as seen in [5,18], aligning with our approach as parallel contributions. Like previous works [4,9,14,26,28,40], we freeze T2I models to prevent overfitting to specific images. However, it's important to recognize the limitations associated with model freezing. $\\textbf{(3)}$ Some papers [A,B] have concentrated on object removal or inpainting using user-defined masks[1,27,42]. In contrast, the act of adding an object in a suitable position without relying on a mask is an active domain. This direction usually entails fine-tuning models for optimal outcomes and runs parallel to DPL. \n\n$\\textbf{W1.3}:$ Training-free methods[D,E] focus on augmenting T2I models to enable compositional generation but lack capabilities for image editing. In contrast, text-guided editing methods (NTI[26], DPL, etc.) drastically reduce editing time from hours of human editing to mere minutes. Consequently, these techniques find suitability in practical domains such as creative arts and publicity editing. However, we agree that these techniques are not applicable for real-time editing. \n\n$\\textbf{W2}:$ DiffEdit[9] proves unsatisfactory for the cases under our consideration, as evident in Fig. 1. Thus, a direct comparison is omitted from the main paper. Nonetheless, we append a comprehensive DiffEdit comparison in the rebuttal PDF, along with an extended user study below. Note that DPL is the most preferred method (by 47.1%). Regarding StructureDiffusion[D], it is essential to clarify that this is a generation method and not designed for image editing. Imagic[18] is a closed-source approach focusing on non-rigid object variation generation, necessitating per-image T2I model fine-tuning. Our method aligns with text-guided editing pipelines[9,14,26,28,40] by freezing T2I models. Notably, Imagic takes about 20 minutes per image on an A40 GPU, while our DPL achieves approximately 4.5 minutes. Qualitative Imagic comparisons are included in our rebuttal PDF. These clarifications aim to underscore the differences between our method and the aforementioned approaches.\n\n| method | DPL (ours) | NTI | DiffEdit | InstructP2P | Pix2Pix | PnP |\n| :--------: | :--------: |:--------: |:--------: |:--------: |:--------: |:--------: |\nUser study (%) | 47.1 | 3.8 | 0.9 | 11.1 | 7.3 | 29.8 |\n\n$\\textbf{Q1}:$ It is correct that DPL and Imagic share similarities in updating text embeddings, but it is crucial to emphasize the differences: $\\textbf{(1)}$ DPL updates specific tokens in the CLIP dictionary, which grants it better fine-grained control over individual objects. On the other hand, Imagic updates the entire text embedding, which may result in limited control over individual objects, especially in complex scenes with multiple objects. $\\textbf{(2)}$ DPL indeed updates only the token embeddings in the CLIP dictionary. In contrast, Imagic requires updating the entire T2I model per image, which is computationally expensive and potentially catastrophic forgetting when the T2I models are finetuned[20,35]. $\\textbf{(3)}$ DPL is designed to address a more general text-guided editing task. By comparison, Imagic focuses on non-rigid editing while keeping the same object, e.g., it can not be applied for Word-Swap. \n\n$\\textbf{Q2}:$ In training, our DPL exclusively focuses on updating token embeddings, leaving the UNet of the LDM untouched. Notably, DPL's extended training duration, relative to NTI[26], stems from the intent to regulate attention maps. In our experiments, NTI takes 2 minutes per image, while DPL requires about 4.5 minutes. However, both methods introduce no noticeable inference time overhead compared to the base LDM. Diverging from training-free methods[C,D] augmented on LDM, which primarily target compositional generation, DPL stands distinct. Aligning with previous pipelines [9,18,26,28,40], we empower tangible image editing driven by textual cues. Notably, DPL's ability for specific region editing further distinguishes it from existing text-guided editing methods.\n\n$\\textbf{Q3}:$ The attention map employed in DPL is 16x16, as indicated in Fig.1, Line 186, and Supp.A. The determination of this cross attention size adheres to practices from prior works[4,14,26]. Moreover, our investigation into various attention sizes is shown in Fig. 3 and the supplementary. Notably, this 16x16 cross attention encompasses QKV and feature shapes of 16x16x160 and 16x16x1280, respectively.\n\n[A] Ablating Concepts in Text-to-Image Diffusion Models. ICCV 2023\n\n[B] Erasing Concepts from Diffusion Models. ICCV 2023\n\n[C] Compositional visual generation with composable diffusion models. ECCV 2022\n\n[D] Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis. ICLR 2023."}, {"review_id": "fajgsXyA7g", "reviewer": "Reviewer_XeHW", "summary": "This paper proposes Dynamic Prompt Learning (DPL) to address the cross-attention leakage issue for text-based image editing. Based on the observation that inaccurate cross-attention maps cause unintended modifications of regions outside of the targeted area for text-based image editing, the authors propose Dynamic Prompt Learning to force cross-attention maps to focus on correct noun words in the text prompt. By updating the dynamic tokens for nouns in the textual input with the proposed leakage repairment losses, the proposed approach achieves fine-grained image editing over particular objects while preventing undesired changes to other image regions. Experiments of Stable Diffusion models are conducted for word-swap, prompt refinement, and attention re-weighting.", "questions": "Please refer to the weakness section.", "limitations": "The authors addressed the limitations adequately.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes an interesting approach to address the attention leakage problem for text-based image editing. By adding constraints on the cross-attention maps, the proposed approach regularizes the attention regions of the diffusion models and thus achieves detail-preserving and high-fidelity image editing.\n2. The proposed approach is well-motivated and the discovery of the attention leakage problem is interesting and inspiring.\n3. The proposed approach achieves better text-based image editing performance on various editing scenarios, as demonstrated in the extensive experiments.", "weaknesses": "1. The proposed DPL approach seems like a combination of several components (including the Disjoint Object Attention Loss, Background Leakage Loss, Attention Balancing Loss, Gradual Optimization for Token Updates, and the trick from null-text inversion), and it is not clear about the contribution and necessity of each component. More ablation study is needed to demonstrate the necessity of combining those methods and loss functions.\n2. It is not shown how accurate and how robust the attention-based background estimation is. When does the background estimation fail and how this will affect the model if this part fails? The choice of hyperparameters (self-attention with size 32, cross-attention with size 16 and TH = 0.2, V = 5) for this subsection is based on empirical study, and ablation study is not presented.\n3. The writing can be improved. The structure of section 3 can be organized more logically and more clearly. The paragraph \"Gradual Optimization for Token Updates\" is unclear to me. \n4. Experiments are conducted on a self-built dataset with a small size (only 100 images for ablation study and 60 images for user study).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and will incorporate the discussions mentioned below to enhance the quality of our paper. Note that we utilize the numerical references to cite sources within the main paper.\n\n$\\textbf{W1}:$ This study primarily addresses cross-attention leakage issues in the realm of real image editing, encompassing both background and distractor object leakage concerns. These issues have been insufficiently tackled by existing methods. To effectively mitigate leakage, we propose dynamic prompt learning (DPL), aiming to alleviate cross-attention leakage's influence on the editing process. To combat distractor object leakage, we introduce specialized loss functions: Disjoint Object Attention loss and Attention Balancing loss. These are meticulously designed to improve object focus and attention distribution balance. Background leakage is addressed through the Background Leakage Loss. Ablation of each loss component is detailed in Section 4.1 (Fig. 5-(a)) of the main paper, revealing that standalone use of the disjoint object attention loss and background leakage loss underperforms compared to NTI [26]. Combining them is key for enhanced performance. Moreover, the attention balancing loss enhances cross-attention quality, especially when coupled with the other two proposed losses. While detailed hyperparameter ablation is presented in Supp.D, Fig.8, and Fig.9 of the supplementary material, providing a comprehensive analysis of various experimental setups and corresponding outcomes.\n\n$\\textbf{W2}:$ Due to page limits we decided to move this part to the supplementary section (Supp.E); we regret any resulting misunderstandings. To address the critical aspect of background estimation in our method, we have extensively investigated this component in Supp.E of the supplementary material. The success of DPL in filtering background leakage relies heavily on accurate background estimation. Hence, we have thoroughly examined various factors, including attention size, feature components, threshold values, and other relevant parameters, to ensure the robustness and reliability of this essential component of our approach.\n\n$\\textbf{W3}:$ Given the multifaceted nature of diffusion-based text-guided image editing, we deeply regret any potential confusion arising from our content organization. Regarding \"Gradual Optimization for Token Updates,\" its central aim is to alleviate update pressures at each step of the process, particularly due to the accumulation of cross-attention leakage during denoising. We've introduced a mechanism ensuring all losses attain predefined thresholds at each step, aiming to prevent overfitting of cross-attention maps. Such overfitting could lead to erroneous cross attentions, detrimentally affecting the editing process, as can be observed in Fig.18 of our rebuttal PDF. Hence, the gradual optimization strategy enhances robustness and accuracy throughout editing.\n\n$\\textbf{W4}:$ When constructing our multi-object real-image dataset, we encountered the challenge of lacking standardized benchmarks tailored to real-image text-guided editing. To overcome this limitation, we adopted the protocol employed in the pix2pix-zero framework [28]. We gathered images from the LAION-5B dataset [37] per our dataset protocols (Supp.C in the supplementary material). Through predefined search templates, along with requisite preprocessing steps (like watermark removal and manual image selection without complete objects), we curated a collection of 327 images derived from 32 distinct prompts. However, these 32 prompts involve image numbers spanning from 1 to 37. Thus, for meaningful comparison with other methods, we retained only the prompts with higher image counts for our MO-Set and URS-Set. The remaining ones serve as qualitative candidates, showcased in Fig.5, Fig.7, Fig.12, Fig.13, etc. This contrasts with NTI[26] and pix2pix-zero[28] datasets, which encompass 100/250 single-object images respectively. Our dataset's scale is thus considerable within the context of text-guided image editing problems. Furthermore, our method's efficacy on generated images is evaluated, as demonstrated in Supp.H and Fig.15. Here, the number of images can potentially be infinite due to the generative nature of the T2I models. This framework permits us to test and showcase our method's versatility and applicability beyond fixed dataset constraints.\nThe comprehensive dataset statistics are presented as follows:\n\n| Prompts | Image Number | Dataset |\n| -------- | -------- | -------- |\n| a clock and a book | 24 | MO-Set; URS-Set |\n| a dog and a bird | 19 | MO-Set; URS-Set |\n| a ball and a cat | 19 | MO-Set |\n| a book and a pen | 17 | MO-Set |\n| a cat and a dog | 16 | MO-Set; URS-Set |\n| a knife and a fork | 13 | MO-Set |\n| a cat and a bird | 13 | MO-Set; URS-Set |\n| a person on a bike | 13 | MO-Set |\n| a horse and a sheep | 11 | MO-Set |\n| a cake in a plate | 4 | URS-Set |\n| a keyboard and a mouse | 4 | MO-Set |\n| a cat and a dog on the grass | 3 | MO-Set |\n| a piano and a chair in the room | 2 | URS-Set |\n| a pear and an apple | 2 | URS-Set |\n| a pizza on a table | 2 | URS-Set |\n"}, {"review_id": "3ClCiD0pGC", "reviewer": "Reviewer_qcZn", "summary": "The paper propose DPL to solve the cross-attention background and distractor object leakage problem in\nimage editing using text-to-image diffusion models. The presentation is well written and easy to follow. The discussion and analysis is extensive and interesting. But the experiment dataset is too small to clarify the robustness of this method. ", "questions": "null", "limitations": "null", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The presentation is well written and easy to follow. \n2. The discussion and analysis is extensive and interesting. \n3. The framwork to solve the cross-attention background and distractor object leakage problem is novel.", "weaknesses": "1. The experiment dataset is too small to clarify the robustness of this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and will incorporate the discussions mentioned below to enhance the quality of our paper. Note that we utilize the numerical references to cite sources within the main paper.\n\n$\\textbf{W1}:$ For the creation of our multi-object real-image dataset, we faced the challenge of the absence of standardized benchmarks specifically tailored to real-image text-guided editing tasks. To overcome this limitation, we adopted the protocol used in the pix2pix-zero [28] framework. In doing so, we retrieved images from the LAION-5B dataset [37] as detailed in our dataset protocols section (Supp.C in the supplementary material). With various predefined search templates followed by necessary preprocessing (including watermark removal and manual selection of images without complete objects, etc.), we collected a total of 327 images from 32 different searches. However, these 32 prompts have image numbers ranging from 1 to 37. Therefore, for a meaningful comparison with other methods, we selected only the prompts with more images for our MO-Set and URS-Set, and the others are serving as qualitative candidates, which we already included in Fig.5, Fig.7, Fig.12, and Fig.13. The detailed information is listed in the table below.\nIn comparison, the NTI [26], pix2pix-zero [28] datasets contain 100/250 single-object images respectively. Therefore, our data scale is not small in text-guided image editing problems, and we aim to publish our created dataset as a new benchmark for public usage in the future.\nFurthermore, in our evaluations, we also assess the effectiveness of our proposed method, DPL, on generated images. This is demonstrated in Supp.H and Fig.15. In this context, the number of images can potentially be infinite due to the generative nature of the task. This allows us to test and showcase the versatility and applicability of our method beyond the constraints of a fixed dataset.\nThe comprehensive dataset statistics are presented as follows:\n\n| Prompts | Image Number | Split |\n| -------- | -------- | -------- |\n| a clock and a book | 24 | MO-Set; URS-Set |\n| a dog and a bird | 19 | MO-Set; URS-Set |\n| a ball and a cat | 19 | MO-Set |\n| a book and a pen | 17 | MO-Set |\n| a cat and a dog | 16 | MO-Set; URS-Set |\n| a knife and a fork | 13 | MO-Set |\n| a cat and a bird | 13 | MO-Set; URS-Set |\n| a person on a bike | 13 | MO-Set |\n| a horse and a sheep | 11 | MO-Set |\n| a cake in a plate | 4 | URS-Set |\n| a keyboard and a mouse | 4 | MO-Set |\n| a cat and a dog on the grass | 3 | MO-Set |\n| a piano and a chair in the room | 2 | URS-Set |\n| a pear and an apple | 2 | URS-Set |\n| a pizza on a table | 2 | URS-Set |"}, {"review_id": "BifHxBhTwO", "reviewer": "Reviewer_sXMN", "summary": "The authors first point out that inferior cross-attention maps regarding noun text tokens are the main causes of failures cases in prompt-based editing methods, such as prompt-to-prompt (P2P).\nTo tackle this, they propose to optimize noun text features with three objectives at each denoising timestep;\ni) minimize the overlap between different cross-attention maps, ii) keep the background mask intact, which is automatically inferred from both cross- and self-attention maps, and iii) keep the max value of cross-attention maps related to nouns high.\nThis way, they achieve better editing results with cross-attention maps that are better aligned with the object regions.", "questions": "- regarding the algorithm 1, isn’t it better to use updated text features by implementing the line 9 after the line 13?\n- the thresholds defined by Equation 10 appear to prevent overfitting of the model. what happens if you don't use these thresholds?", "limitations": "please refer to the Weaknesses section", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the authors tackle a meaningful task\n- it is easy to understand the motivation and method with various visualizations\n- the proposed method presents improved performance on a variety of evaluation metrics", "weaknesses": "- increased inference time\n- only can solve cases \brelated to noun text tokens representing the type of object", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and will incorporate the discussions mentioned below to enhance the quality of our paper. Note that we utilize the numerical references to cite sources within the main paper and any references not included in the main paper are provided in a list at the end of this response in alphabetical order.\n\n$\\textbf{W1}:$ $\\textbf{(1)}$ In the training phase, the DPL method focuses solely on updating the token embeddings that are registered in the language model dictionary, without updating the UNet of the Stable Diffusion models [33]. More specifically, DPL exhibits an extended training duration relative to Null-Text Inversion [26] due to our objective of updating token embeddings to enforce regularization on attention maps for specific objects. In our single-GPU experiments using the A40, NTI requires 130 seconds to process one image, while DPL necessitates approximately 280 seconds. $\\textbf{(2)}$ However, during inference, both DPL and NTI do not incur any discernible additional time overhead compared to the fundamental Stable Diffusion model [33].\n\n$\\textbf{W2}:$ The reviewer is right to point this out (and we will improve our limitations section): \nIndeed, there have already been papers related to image editing that involve verbs. They typically necessitate fine-tuning the T2I models [5,18], which aligns with our approach and runs parallel to our contributions. As a common practice, we freeze the T2I models, similar to previous works [4,9,14,26,28,40], to prevent the models from overfitting to a specific image. However, it is essential to acknowledge that freezing the T2I models comes with some shortcomings, as pointed out by the reviewer. These limitations should be carefully considered when evaluating the overall performance and effectiveness. There are also certain papers [A,B] focusing on removing objects or inpainting using user-defined masks [1,27,42] in given images. However, the exploration of adding an object in an appropriate position without relying on a mask is currently a very active research area. This direction typically requires model fine-tuning to achieve desirable results and is a parallel research to text-guided image editing as our method DPL. We will include a discussion on this in our limitations section.\n\n$\\textbf{Q1}:$ Thanks for pointing it out. You are correct here. The text conditions' initialization appears on line 9. Then the updating of token representations causes changes in the text conditions. To avoid misunderstandings, it is preferable to add another line to describe this process after Line 13 in the future version.\n\n$\\textbf{Q2}:$ By using Equation 10, we aim to prevent overfitting of the cross-attention maps, which could otherwise lead to erroneous cross attentions and adversely impact the editing process, as can be observed in Fig.18 of our rebuttal PDF. The gradual optimization approach is thus strategically designed to enhance the robustness and accuracy of our method throughout the editing procedure.\n\n[A] Ablating Concepts in Text-to-Image Diffusion Models. ICCV 2023\n\n[B] Erasing Concepts from Diffusion Models. ICCV 2023\n"}, {"review_id": "ZFIzEU42iU", "reviewer": "Reviewer_m2Wr", "summary": "The paper proposes a new method to improve the attention masking for attention-based local editing.", "questions": "Please see the weakness part.", "limitations": "No", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed loss functions are novel, intuitive and effectively improves the inversion process of text-to-image diffusion model.", "weaknesses": "1. Please show the quantitative evaluation of the object masking performance between the mask generated from Diffedit.\n\n2. Quantitative comparison with baseline models are not enough. Please put more comparison between other baselines such as Plug-and-play and Pix2pix-zero, instructPix2Pix. At least include user study.\n\n3. Does it improve other attention-based editing mechanism? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and will incorporate the discussions mentioned below to enhance the quality of our paper. Note that we utilize the numerical references to cite sources within the main paper.\n\n$\\textbf{W1}:$ As requested, we performed an evaluation of the binary masks obtained from the DiffEdit method on our MO-Set Test-Split. The resulting Intersection over Union (IoU) score for DiffEdit is 0.41, which is similar to NTI but notably lower than our method, DPL (as depicted in Fig. 5-(a)). Moreover, we observed that DiffEdit generated masks with numerous noisy points, as illustrated in Fig. 1. As a consequence, the editing results were not satisfactory according to our user study and qualitative comparisons, as detailed in the author rebuttal PDF. This comparison demonstrates the superior performance of DPL in accurately matching desired editing areas, highlighting its effectiveness in comparison to other methods.\n\n$\\textbf{W2}:$ As requested, we incorporated four additional popular text-guided image editing methods into the user study, and the results are presented in the table below. Based on the evaluation, we can infer that our method, DPL, and Plug-and-Play [40] are the two approaches that primarily satisfy the evaluators' subjective preferences. These findings demonstrate the strong performance and user satisfaction of DPL and underscore its competitiveness in comparison to the other methods.\n\n| method | DPL (ours) | Null-Text Inversion [26] | DiffEdit [9] | InstructPix2Pix [4] | Pix2Pix-zero [28] | Plug-and-Play [40] |\n| :--------: | :--------: |:--------: |:--------: |:--------: |:--------: |:--------: |\nUser study (%) | 47.1 | 3.8 | 0.9 | 11.1 | 7.3 | 29.8 |\n\n$\\textbf{W3}:$ Our method DPL is regularizing the cross attention maps during the denoising phases of the DDIM inversion. As a result, it has the potential to enhance the performance of other existing cross-attention based editing mechanisms.  As of the NeurIPS 2023 deadline, we found only one available method, Null-Text inversion [26], that works on real image inversion and text-guided image editing. Moreover, their image editing approach is based on references from P2P [14].  On the other hand, Pix2pix-zero [28] is also working on cross-attention injection, but it adopts noise-regularization, which can disrupt the reconstruction property of DDIM inversion. Furthermore, Pix2pix-zero is primarily focused on changing the entire image style by discovering edit directions using GPT and CLIP models. In our case, we share more similarities with P2P streams, where we modify textual prompts to guide the editing process.\n"}, {"review_id": "0RMMxq2fZM", "reviewer": "Reviewer_zBnt", "summary": "This paper proposes a dynamic prompt learning approach for image editing that modifies self-attentions to more accurately attend to the correct nouns given a text prompt. The proposed approach is used along with null text inversion where the dynamic tokens corresponding with the noun words are updated with a background leakage loss, a disjoint object attention loss and an attention balancing loss. Then p2p is used with the modified cross-attention masks to edit the image. The main benefit of this approach is in editing an image which includes two or more semantically related objects like cat and dog. The authors have collected a set of 60 multi-object images from LAION as a test set and a set of 60 images with semantically related objects for their human evaluations, have shown improvements through qualitative and some quantitative evaluations.", "questions": "1. Please describe your evaluation datasets and their editing prompts more clearly. Is the URS-Set similar to the MO-Set? what is the \ndifference between these two sets? Since LAION-5B is pretty large, I'd suggest the authors to expand their test set to include more varied image editing prompts and examples.\n2. I wonder how much gain this method introduces on image editing problems with one object or two unrelated objects in the image (if any).", "limitations": "The authors have neither discussed the limitations of their work, nor its negative societal impact. A broader study on different editing types as mentioned above could clarify the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well written and explains the method and related works clearly.\n2. A novel method as dynamic prompt learning for image editing has been proposed. The paper is mainly focused on a specific axis of image editing where two or more semantically related objects are present in the image and cross attention maps for each object leak to other objects or the background. This approach modifies the attention masks and thus can be used along with p2p for image editing.\n3. A test set containing images of multiple objects from LAION-5B is collected for quantitative and qualitative evaluations and a couple of examples are presented in the paper and supplemental showing the superiority of the proposed method.", "weaknesses": "1. The scope of the image editing problems this paper aims to improve is narrow mainly useful for cases when there are two or multiple semantically-related objects in the image and for word-swapping editing prompts like cat to dog, cat to tiger, etc. \n2. Through only a couple of image editing examples shown in the paper, I am not convinced that this paper is beneficial in general image editing problems where there are not necessarily related objects in the scene or for other types of editing prompts rather than word swapping. \n  -  It is unclear what type of editing prompts are included in the collected test set, but from discussions, it looks like it is mainly focusing on word swapping. Also, size of the test set is fairly small, and the studies are all done in the above narrowed down scope in image editing, thus not conclusive. \n- The quantitative clip scores are only reported in two scenarios (1) cat → leopard and dog → tiger and (2)  book → box and clock → sunflower and improvements are still marginal.\n- On the human evaluation study, I'd expect it to be done on other image editing types with single or multiple objects either semantically related or unrelated, on a larger set, and to compare with other baselines such as instructpix2pix and plug-and-play. \n3. On auto evaluations, one potential metric to measure the correctness/accuracy of this method could be applying an object segmentation or object detection method on the edited image, predict label for the edited objects, check if the predicted label matches with the given prompt. Compute the average accuracy score and compare with the baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and will use the discussion to improve. Below we use the references in the main paper.\n\n$\\textbf{W1}:$ Actually DPL also works for Word-Swap with less related concepts. As we have shown in Fig.7, Fig.13, Fig.17, we successfully edit the bird into airplane/helicopter, the ball/clock into wheel/balloon, the chair into book/window, a person into a cat, etc. As to the scope of text-guided image editing, P2P[14] is designed to solve prompt-based editing in the denoising phases of a pretrained T2I diffusion models. After, NTI[26] generalized this technique to real-images. Experimenting with these codes, we quickly found that they fail for more complex scene with semantically related objects and background. We consider this one of the main short-comings of T2I and addressing this relevant problem can be important for many realistic image editing applications.\n\n$\\textbf{W2.1}:$  It is true that DPL is most beneficial when there are leakage issues in the image. In scenarios where no leakage issues are present, using DPL may not be necessary, as it would incur additional time complexity. DPL can be seen as an augmentation to existing cross-attention based image editing methods[4,14,26]. Its purpose is to improve the accuracy and reliability of cross-attention maps, which is particularly essential for tasks like Word-Swap. For other types of image editing tasks, Fig.6 in the main paper illustrates attention refinement, highlighting the effectiveness of DPL in improving attention maps by filtering the background leakage. Fig.14 in the supplementary material showcases global editing, where DPL may have shown comparatively less superiority. We also include more instances in the rebuttal PDF.\n\n$\\textbf{W2.2, Q1}:$  We encountered the challenge of lacking benchmarks tailored for multi-object image editing. We adopted the protocol employed in pix2pix-zero[28] and gathered images from the LAION-5B[37] per our dataset protocols (Supp.C). Through search templates and preprocessing steps, we curated a collection of 327 images from 32 distinct prompts. Finally, we retained only the prompts with more images. This contrasts with NTI and pix2pix-zero datasets, which encompass 100/250 single-object images respectively. Thus, our dataset's scale is considerable. Moreover, our method's efficacy on generated images is evaluated in Supp.H and Fig.15. Here, the number of images can potentially be infinite. This framework permits us to showcase DPL's applicability beyond fixed dataset constraints.\n \n| Prompts | Image Number | Split |\n| -------- | -------- | -------- |\n| a clock and a book | 24 | MO-Set; URS-Set |\n| a dog and a bird | 19 | MO-Set; URS-Set |\n| a ball and a cat | 19 | MO-Set |\n| a book and a pen | 17 | MO-Set |\n| a cat and a dog | 16 | MO-Set; URS-Set |\n| a knife and a fork | 13 | MO-Set |\n| a cat and a bird | 13 | MO-Set; URS-Set |\n| a person on a bike | 13 | MO-Set |\n| a horse and a sheep | 11 | MO-Set |\n| a cake in a plate | 4 | URS-Set |\n| a keyboard and a mouse | 4 | MO-Set |\n| a cat and a dog on the grass | 3 | MO-Set |\n| a piano and a chair in the room | 2 | URS-Set |\n| a pear and an apple | 2 | URS-Set |\n| a pizza on a table | 2 | URS-Set |\n\n$\\textbf{W2.3,W2.4}:$ To assess the performance of our editing method, we adopt a similar approach to pix2pix-zero[28], creating two scenarios that demand changes to both objects. However, the improvements achieved in these cases are limited due to the narrow improvement spaces. Additionally, we observe that even the original multi-object images do not closely match the textual prompts, which further impacts the editing outcomes. To gain a more comprehensive evaluation, we conducted a user study that covered a broad spectrum of image editing tasks, going beyond these two setups. As you requested, we extended the user study with more methods as below. Importantly, DPL received higher satisfaction ratings from the participants.\n\n| method | DPL (ours) | NTI | DiffEdit | InstructP2P | Pix2Pix | PnP |\n| :--------: | :--------: |:--------: |:--------: |:--------: |:--------: |:--------: |\nUser study (%) | 47.1 | 3.8 | 0.9 | 11.1 | 7.3 | 29.8 |\n\n$\\textbf{W3}:$ In our evaluation of editing accuracy, we established ground truths for our dataset images. These ground truths serve to compute IoU with corresponding cross-attention maps from our DPL inversion. This assessment is presented in Fig.5 of the main paper and Fig.8 in the supplementary. Further evaluation encompasses CLIP semantic similarity analysis and a user study, with summarized outcomes in Table 1. As per your request, we incorporated the Segment-Anything model for segmentation map generation based on prompts from edited images. The resultant IoU scores for our DPL method and the baseline NTI stand at 0.48 and 0.37, respectively, assessed over the MO-Set Test-Split. These scores affirm DPL's superior segmentation accuracy compared to the baseline.\n\n$\\textbf{Q2}:$ In cases where images feature a single main object, our DPL may exhibit limited improvement over other methods, as depicted in Fig.6 of the main paper and Fig.13 in the supplementary. This observation arises from the fact that in such scenarios, the cross-attention mechanism might lack additional objects to attend to, thus offering minimal influence on the editing process. However, DPL shines in scenarios with multiple disparate objects. Here, DPL excels in effectively filtering both background and distractor object leakage, showcased in Fig.12 and Fig.13 of the supplementary. Furthermore, DPL is specifically tailored to address situations where background and distractor object leakage significantly impact editing performance. It is an augmentation to existing editing methods. We do not assert DPL as a comprehensive solution to all existing editing challenges.\n\n$\\textbf{Limitations}:$ We have presented our limitations and broader impacts in Supp.A and Supp.B and aim to move these to the main paper in the future version.\n"}]

Author rebuttal:
In the author rebuttal PDF file, we include three additional figures for the reviewer's reference: 

$\textbf{(1)}$ Fig.17: extended comparison with DiffEdit and Imagic for image editing; 

$\textbf{(2)}$ Fig.18: ablation study of the Gradual Optimization for Token Updates; 

$\textbf{(3)}$ Fig.19: progressively infusing the attention maps across diverse diffusion steps.

Meta-review: This paper starts with experimental findings that the problem of unintended modifications of regions outside the targeted area proposes is caused by inaccurate cross-attention maps. dynamic prompt learning (DPL) to tackle in text-based image editing. This method employs leakage repairment losses to update the dynamic tokens for nouns, thereby forcing cross-attention maps to focus on correct nouns in the prompt.

Reviewers have identified a series of strengths and weaknesses of this paper as follows:

Pros:
1. The tackled problem is important. 
2. The idea is interesting, simple yet effective. The method is well motivated and technically sound.
3. The paper is generally well written and easy to follow.
4. The effectiveness of the proposed method is demonstrated in extensive experiments.

Cons:
1. The method is not applicable when nouns do not correspond to intended areas, and is not applicable for real-time editing.
2. Lack of comparisons with important baselines.
3. Experiments are conducted on small-size datasets.
4. Lack of some ablation evaluations.

The authors have provided a detailed rebuttal, and most reviewers acknowledged that their major concerns are addressed. The AC agrees with the strengths and recommends accept, though would be preferable if necessary revisions could be provided according to reviewer’s comments.
Decision: Accept (poster)