Title: Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing
Venue: NeurIPS 2023
Accepted: yes
Keywords: Diffusion Models; Text-guided Image Edit; Textual Inversion; Localization
Abstract: Large-scale text-to-image generative models have been a ground-breaking development in generative AI, with diffusion models showing their astounding ability to synthesize convincing images following an input text prompt. The goal of image editing research is to give users control over the generated images by modifying the text prompt. Current image editing techniques are susceptible to unintended modifications of regions outside the targeted area, such as on the background or on distractor objects which have some semantic or visual relationship with the targeted object. According to our experimental findings, inaccurate cross-attention maps are at the root of this problem. Based on this observation, we propose $\textit{Dynamic Prompt Learning}$ ($DPL$) to force cross-attention maps to focus on correct $\textit{noun}$ words in the text prompt. By updating the dynamic tokens for nouns in the textual input with the proposed leakage repairment losses, we achieve fine-grained image editing over particular objects while preventing undesired changes to other image regions. Our method $DPL$, based on the publicly available $\textit{Stable Diffusion}$, is extensively evaluated on a wide range of images, and consistently obtains superior results both quantitatively (CLIP score, Structure-Dist) and qualitatively (on user-evaluation). We show improved prompt editing results for Word-Swap, Prompt Refinement, and Attention Re-weighting, especially for complex multi-object scenes.

Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing

Abstract

Large-scale text-to-image generative models have been a ground-breaking development in generative AI, with diffusion models showing their astounding ability to synthesize convincing images following an input text prompt. The goal of image editing research is to give users control over the generated images by modifying the text prompt. Current image editing techniques are susceptible to unintended modifications of regions outside the targeted area, such as on the background or on distractor objects which have some semantic or visual relationship with the targeted object. According to our experimental findings, inaccurate cross-attention maps are at the root of this problem. Based on this observation, we propose Dynamic Prompt Learning ( DPL ) to force cross-attention maps to focus on correct noun words in the text prompt. By updating the dynamic tokens for nouns in the textual input with the proposed leakage repairment losses, we achieve fine-grained image editing over particular objects while preventing undesired changes to other image regions. Our method DPL , based on the publicly available Stable Diffusion , is extensively evaluated on a wide range of images, and consistently obtains superior results both quantitatively (CLIP score, Structure-Dist) and qualitatively (on user-evaluation). We show improved prompt editing results for Word-Swap, Prompt Refinement, and Attention Re-weighting, especially for complex multi-object scenes.

1 Introduction

Text-to-Image (T2I) is advancing at a revolutionary pace and has demonstrated an unprecedented ability to generate diverse and realistic images [27, 33, 34, 35]. The state-of-the-art T2I models are trained on extremely large language-image datasets, which require huge computational resources. However, these models do not allow for straightforward text-guided image editing and generally lack the capability to control specific regions of a given image.

Recent research on text-guided image editing allows users to easily manipulate an image using only text prompts [5, 9, 14, 15, 24, 42, 43]. In this paper, we focus on prompt based image editing, where we aim to change the visual appearance of a target object (or objects) in the image. Several of the existing methods use DDIM inversion [40] to attain the initial latent code of the image and then apply their proposed editing techniques along the denoising phase [28, 30, 42]. Nevertheless, current text-guided editing methods are susceptible to unintentional modifications of image areas. We distinguish between two major failure cases: unintended changes of background regions and unintended changes of distractor objects (objects that are semantically or visually related to the target object), as shown in Fig. 1. The functioning of existing editing methods greatly depends on

Plug-and-Play

pix2pix-zero

InstructPix2Pix

NTI + P2P

DPL + P2P

Prompt: a cat and a dog

dog

lion

cat

zebra

dog

lion

GSAM inpaint

cat

mask

Average Cross-Attention maps

DifEdit

NTI

DPL

=

<|startoftextI>

a

cat

and

a

bop

<lendoftextI>

=

the accuracy of the cross-attention maps. When visualizing the cross-attention maps (see Fig. 1), we can clearly observe that for DDIM [40] and Null-Text inversion [28] these maps do not only correlate with the corresponding objects. We attribute this phenomenon to cross-attention leakage to background and distractor objects. Cross-attention leakage is the main factor to impede these image editing methods to work for complex backgrounds and multi-object images. Similar observations for failure cases are also shown in [9, 14, 19, 22, 28]. For example, Custom Diffusion [22] fails for multi-concept composition and DiffEdit does not correctly detect the mask region for a given concept.

We conduct a comprehensive analysis of the Stable Diffusion (SD) model and find that the inaccurate attribution of cross-attention (resulting in undesirable changes to background and distractor objects) can be mitigated by exploiting the semantic clusters that automatically emerge from the self-attention maps in the diffusion model. Instead of fine-tuning the diffusion model, which potentially leads to catastrophic neglect [7], we propose replacing prompts related to scene objects with dynamic prompts that are optimized. We propose two main losses for dynamic prompt learning (DPL), one tailored to reduce attention leakage to distractor objects and one designed to prevent background attention leakage. Moreover, better cross-attention maps are obtained when we update the prompts for each timestamp (as implied by dynamic ). Furthermore, to facilitate meaningful image editing, we use NullText Inversion [28] with classifier-free guidance [17]. In the experiments, we quantitatively evaluate the performance of DPL on images from LAION-5B [39] dataset. And DPL shows superior evaluation metrics (including CLIP-Score and Structure-Dist) and improved user evaluation. Furthermore, we show prompt-editing results for Word-Swap, Prompt Refinement, and Attention Re-weighting on complex multi-object scenes.

2 Related work

For text-guided image editing with diffusion models, various methods [2, 8, 20, 23, 24] leverage CLIP [32] for image editing based on a pretrained unconditional diffusion model. However, they are limited to the generative prior which is only learned from visual data of a specific domain, whereas the CLIP text-image alignment information is from much broader data. This prior knowledge gap is

mitigated by recent progress of text-to-image (T2I) models [6, 10, 18, 33, 34, 38]. Nevertheless, these T2I models offer little control over the generated contents. This creates a great interest in developing methods to adopt such T2I models for controllable image editing. SDEdit [26], a recent work utilizing diffusion models for image editing, follows a two-step process. It introduces noise to the input image and then applies denoising using the SDE prior to enhance realism and align with user guidance. Imagic [19] and P2P [15] attempt structure-preserving editing via Stable Diffusion (SD) models. However, Imagic [19] requires fine-tuning the entire model for each image and focuses on generating variations for the objects. P2P [15] has no need to fine-tune the model, it retrains the image structure by assigning cross-attention maps from the original image to the edited one in the corresponding text token. InstructPix2Pix [4] is an extension of P2P by allowing human-like instructions for image editing. To make the P2P capable of handling real images, Null-Text inversion (NTI) [28] proposed to update the null-text embeddings for accurate reconstruction to accommodate with the classifier-free guidance [17]. Recently, pix2pix-zero [30] propose noise regularization and cross-attention guidance to retrain the structure of a given image. However, it only supports image translation from one concept to another one and is not able to deal with multi-concept cases. DiffEdit [9] automatically generates a mask highlighting regions of the input image by contrasting predictions conditioned on different text prompts. Plug-and-Play (PnP) [42] demonstrated that the image structure can be preserved by manipulating spatial features and self-attention maps in the T2I models. MnM [31] clusters self-attention maps and compute their similarity to the cross attention maps to determine the object localization, thus benefits to generate object-level shape variations. StyleDiffusion [24] adds up a mapping network to modify the value component in cross-attention computation for regularizing the attentions and utilizes P2P for image editing.

These existing methods highly rely on precise cross-attention maps and overlook an important case where the cross-attention maps may not be perfectly connected to the concepts given in the text prompt, especially if there are multiple objects in the given image. Therefore, these imperfect cross-attention maps may lead to undesirable editing leakage to unintended regions. Essentially, our method DPL complements existing methods to have more semantic richer cross-attention maps. There are various diffusion-based image editing methods relying on giving a mask manually [29, 1] or detecting a mask automatically (GSAM-inpaint [44]) using a pretrained segmentation model. On the contrary, our method only requires textual input and automatically detects the spatial information. This offers a more intuitive editing demand by only manipulating the text prompts.

There are some papers [21, 12] concentrated on object removal or inpainting using user-defined masks [1, 29, 44]. In contrast, the act of placing an object in a suitable position without relying on a mask is an active research domain. This direction usually entails fine-tuning models for optimal outcomes and runs parallel to the research direction explored in this paper.

In this paper, we are also related to the transfer learning for T2I models [22, 37, 11, 13, 25], which aims at adapting a given model to a new concept by given images from the users and bind the new concept with a unique token. However, instead of finetuning the T2I model [25, 13, 37], we learn new concept tokens by personalization in the text embedding space, which is proposed in Textual Inversion [11]. By this means, we avoid updating a T2I model for each concept separately. Furthermore, we step forward to the scenario with complex background or multiple objects, where the previous methods have not considered. In contrast to transfer learning approaches, which aim to ensure that the learned concepts can accurately generate corresponding objects in given images, our proposed approach DPL focuses on enhancing image editing tasks by regularizing cross-attention to disentangle their positions.

3 Method

In this section, we first provide a short preliminary and then describe our method to achieve this demand. An illustration of our method is shown in Fig. 2 and Algorithm 1.

3.1 Preliminary

Latent Diffusion Models. We use Stable Diffusion v1.4 which is a Latent Diffusion Model (LDM) [35]. The model is composed of two main components: an autoencoder and a diffusion model. The encoder E from the autoencoder component of the LDMs maps an image I into a latent code z 0 = E ( I ) and the decoder reverses the latent code back to the original image as D ( E ( I )) ≈ I . The

Editing Image

Caption

Tokenizer

''A puppy and a rabbit''

320

42 320

<    >

<    >

Text Transformer

?

?

?

?

?

?

?

?

Self-attention

maps clustering

Cross-attention maps

puppy

rabbit

puppy

rabbit

Diffusion Model

Sec. 3.3

NTI

NTI

Allow for gradient

Our DPL

Null-text Inversion

Updated

Diffusion Model

null-text

20

ZT-1

ZT

Lbg

B

U

*

ET-1

*0

8

diffusion model can be conditioned on class labels, segmentation masks or textual input. Let τ θ ( y ) be the conditioning mechanism which maps a condition y into a conditional vector for LDMs, the LDM model is updated by the loss:

The neural backbone ϵ θ is typically a conditional UNet [36] which predicts the added noise. More specifically, text-guided diffusion models aim to generate an image from a random noise z T and a conditional input prompt P . To distinguish from the general conditional notation in LDMs, we itemize the textual condition as C = τ θ ( P ) .

DDIM inversion. Inversion entails finding an initial noise z T that reconstructs the input latent code z 0 upon sampling. Since we aim at precisely reconstructing a given image for future editing, we employ the deterministic DDIM sampling [40]:

where ¯ α t +1 is noise scaling factor defined in DDIM [40] and f θ ( z t , t, C ) predicts the final denoised latent code z 0 as f θ ( z t , t, C ) = [ z t - √ 1 -¯ α t ϵ θ ( z t , t, C ) ] / √ ¯ α t .

Null-Text inversion. To amplify the effect of conditional textual input, classifier-free guidance [17] is proposed to extrapolate the conditioned noise prediction with an unconditional noise prediction. Let ∅ = τ θ ('') denote the null text embedding, then the classifier-free guidance is defined as:

where we set the guidance scale w = 7 . 5 as is standard for Stable Diffusion [17, 28, 35]. However, the introduction of amplifier-free guidance complicates the inversion process, and the generated image based on the found initial noise z T deviates from the original image. Null-text inversion [28] proposes a novel optimization which updates the null text embedding ∅ t for each DDIM step t ∈ [1 , T ] to approximate the DDIM trajectory { z t } T 0 according to:

where { ¯ z t } T 0 is the backward trace from Null-Text inversion. Finally, this allows to edit real images starting from initial noise ¯ z T = z T using the learned null-text ∅ t in combination with P2P [15].

Note, that the cross-attention maps of Null-Text inversion ( w = 7 . 5 ) and DDIM inversion ( w = 1 . 0 ) are similar, since NTI is aiming to approximate the same denoising path.

Prompt2Prompt. P2P [15] proposed to achieve editing by manipulating the cross-attention A ∗ t from target input P ∗ according to the cross-attention A t from the source input P in each timestamp t . The quality of the estimated attention maps A t is crucial for the good functioning of P2P. This paper addresses the problem of improved A t estimation for complex images.

3.2 DPL : Dynamic Prompt Learning for addressing cross-attention leakage

Prompt-based image editing takes an image I described by an initial prompt P , and aims to modify it according to an altered prompt P ∗ in which the user indicates desired changes. The initial prompt is used to compute the cross-attention maps given the conditional text input P . As discussed in Sec. 1, the quality of cross-attention maps plays a crucial role for prompt-based image editing methods (see Fig. 1), since these attention maps will be used during the generation process with the altered user prompt [15]. We identify two main failures in the attention map estimation: (1) distractor object leakage, which refers to wrong assignment of attention to a semantically or visually similar foreground object and (2) background leakage, referring to undesired attention on the background. Both of these problems result in artifacts or undesired modifications in generated images.

The cross-attention maps in the Diffusion Model UNet are obtained from ϵ θ ( z t , t, C ) , which is the first component in Eq. 3. They are computed from the deep features of noisy image ψ ( z t ) which are projected to a query matrix Q t = l Q ( ψ ( z t )) , and the textual embedding which is projected to a key matrix K = l K ( C ) . Then the attention map is computed according to:

where d is the latent dimension, and the cell [ A t ] ij defines the weight of the j -th token on the pixel i .

In this paper, we propose to optimize the word embeddings v corresponding to the initial prompt P , in such a way that the resulting cross-attention maps A t do not suffer from the above-mentioned problems, i.e. distractor-object and background leakage. We found that optimizing the embedding v t separately for each timestamp t leads to better results than merely optimizing a single v for all timestamps. The initial prompt P contains K noun words and their corresponding learnable tokens at each timestamp V t = { v 1 t , ..., v k t ..., v t K } . We update each specified word in V t for each step t .

The final sentence embedding C t now varies for each timestamp t and is computed by applying the text transformer on the text embeddings: some of which are learned (those referring to nouns) while the rest remains constant for each timestamp (see Fig. 2). For example, given a prompt "a puppy and a rabbit" and the updatable noun set {puppy, rabbit}, in each timestamp t we have transformed the textual input into trainable ones as "a <puppy> and a <rabbit>" and we aim to learn word embeddings v <puppy> t and v <rabbit> t in each timestamp t .

We propose several losses to optimize the embedding vectors V t : we develop one loss to address cross-attention leakage due to object distractors, and another one to address background leakage. We also found that the attention balancing loss, proposed by Attend-and-Excite [7], was useful to ensure each specified noun word from P is focusing on its relevant concept.

Disjoint Object Attention Loss. To address the problem of distractor object attention leakage, we propose a loss that minimizes the attention overlap between the attention maps corresponding to the different objects, that is an updatable noun set V t . This prevents specific objects related regions to be linked to multiple nouns, which is one of the sources of distractor object attention. We propose to minimize a cosine similarity loss among the K cross-attention maps (referring to the noun words in the input prompt P ) to avoid their overlapping in cross-attention maps through:

̸

Background Leakage Loss. In order to alleviate the background leakage issue, we propose to utilize background masks to keep the background area unchanged. Details on how to get the background mask are discussed in Sec. 3.3. After obtaining the BG mask B of the given image, for each v k t ∈ V t ,

Self-Attention

Query

Key

Value

16*16

32*32

64*64

CA-Cat

CA-Dog

we force its cross-attention map to have minimal overlap with the background region:

Attention Balancing Loss. For each v k t ∈ V t , we pass its cross-attention map through a Gaussian filter F and formulate a set of losses L v k t for all words in V t . The attention loss is defined as the maximum value in L v k t :

The attention loss will strengthen the cross-attention activations for localizing token v k t , thus force A v t to not leak attentions to other irrelevant concepts in the image. In other words, the resulting cross-attention maps will be more concentrated on the corresponding object region. Different from Attend-and-Excite [7] where the refinement loss is applied to update the latent code to augment the multiple object generation, we are the first to explore its role in learning new conceptual tokens.

With all these three losses, we aim to update the learnable token v k t as:

Note that, when there is only one updatable noun token in the text input P , we only apply the L bg to relieve the background leakage since other losses rely on at least two words. In this paper, we apply DPL over the 16 2 attention following the common practices [15, 7] for a fair comparison.

Gradual Optimization for Token Updates. So far, we introduced the losses to learn new dynamic tokens at each timestamp. However, the cross-attention leakage is gradually accumulated in the denoising phase. Hence, we enforce all losses to reach a pre-defined threshold at each timestamp t to avoid overfitting of the cross-attention maps. We express the gradual threshold by an exponential function. For the losses proposed above, the corresponding thresholds at time t are defined as:

We verify the effectiveness of this mechanism together with other hyperparameters in our ablation experiments (see Fig. 5-(a) and Supplementary Material.).

Null-Text embeddings. The above described token updating ensures that the cross-attention maps are highly related to the noun words in the text prompt and minimize cross-attention leakage. To

Self-Attention

Cross-Attn-1

Background

Foreground 1

Cross-Attn-2

be able to reconstruct the original image, we use Null-Text inversion [28] in addition to learn a set of null embeddings ∅ t for each timestamp t . Then at the time t , we have a set of learnable word embeddings V t and null text ∅ t which can accurately localize the objects and also reconstruct the original image.

3.3 Attention-based background estimation

To address the background attention leakage, we investigate the localization information of selfattention and feature representations, which are present in the diffusion model. Besides crossattention (CA), during the inversion of an image with the DDIM sampler, we can also obtain self-attention maps (SA), or directly analyze the query, key and value in the self-attention maps. In Fig. 3 we visualize these maps. The main observation is that self-attention maps more closely resemble the semantic layout. We therefore exploit these to reduce background leakage.

To acquire the background via self-cross attention matching, we use the agreement score [31] between cross-attention A n of noun words n and each of the self-attention clusters M v , v ∈ [1 , V ] , defined as σ nv = ∑ ( A n · M v ) / ∑ M v .

However, it should be noted that the agreement score does not accurately segment the objects, since there is still attention leakage between objects and distractor objects (see Fig. 4 for examples). Therefore, other than localizing objects as in [31], we use the agreement score only for background mask estimation. We label the cluster M v as being background (BG) if for all n

Algorithm 1: Dynamic Prompt Learning

1 Input: A source prompt P , an input image I

2 Output: A noise vector ¯ z T , a set of updated tokens {V t } T 1 and null-text embeddings { ∅ t } T 1

3 Set T = 50 and scale w = 1 ;

4 { z t } T 0 , B ← DDIM-inv( I );

5 Set guidance scale w = 7 . 5 ;

6 Initialize V T with original noun tokens;

7 Initialize ¯ z T = z T , P T = P , ∅ T = τ θ ('')

;

8 for t = T, T - 1 , . . . , 1 do

9

Initialize

P

t

by

V

t

, then

C

t

=

τ

θ

(

P

t

)

;

10

while

L

at

TH

at

t

or

L

dj

TH

dj

t

or

L

bg

TH

bg

t

do

11

L

=

λ

at

· L

at

+

λ

dj

· L

dj

+

λ

bg

· L

bg

12

V

t

←V

t

-∇

V

t

L

13

end

14

Update

P

t

by

V

t

, then

C

t

=

τ

θ

(

P

t

)

;

15

˜

z

t

= ˜

ϵ

θ

(¯

z

t

, t,

C

t

,

∅

t

)

16

¯

z

t

-

1

,

∅

t

NTI

(˜

z

t

,

∅

t

)

17

Initialize

∅

t

-

1

=

∅

t

,

V

t

-

1

=

V

t

18

end

19

Return

¯

z

T

,

{V

t

}

T

1

,

{

∅

t

}

T

1

if σ nv < TH , where TH is a threshold. Empirically, we choose self-attention with size 32, crossattention with size 16 and TH = 0 . 2 , V = 5 to obtain the background mask B .

4 Experiments

We demonstrate our method in various experiments based on the open-source T2I model Stable Diffusion [35] following previous methods [42, 30, 28]. All experiments are done on an A40 GPU.

(a)

IoU curves on the Valid-split (left) and the Test-Split (right)

(b)

Cross-attention maps comparison

NTI

DPL

0.7

ILN

0.5

NTI

0.6

Cat

Cdj

0.4

DPL

0.5

Cbg

loU

0.4

DPL

0.3

0.3

0.2

0.2

0.1

0.1

0.0

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0 1

Threshold

Threshold

<books>

<sheep>

<clock>

<plate>

<bird>

<dog>

Dataset. We focus here on real-image editing, but also provide examples of generated image editing in the Supplementary Material. We use clip retrieval [3] to obtain experimental multi-object real images from the LAION-5B [39] dataset by matching CLIP embeddings of the source text description. For the ablation study, we select 100 multi-object images from various search prompts (including concepts as bird, dog, cat, book, clock, pen, pizza, table, horse, sheep, etc.). This Multi-Object set is abbreviated with MO-Set . Of the images, 40 images are used for validation and the other 60 are test set. We manually verify their segmentation maps which serve as groundtruth in the ablation study IoU computation. Finally, for user study, we create a URS-Set of 60 multi-object images of semantically related objects from various concepts.

Comparison methods. For qualitative comparison, we compare with pix2pix-zero [30], GSAMinpaint [44], InstructPix2Pix [4], Plug-and-Play [42] and NTI+P2P [28]. For quantitative evaluation, we benchmark our approach against NTI+P2P, considering it as a robust baseline for maintaining content stability and achieving superior conceptual transfer.

4.1 Ablation study

The ablation study is conducted on the MO-Set set. To quantitatively evaluate the performance of DPL in localization, we vary the threshold from 0 . 0 to 1 . 0 to obtain the segmentation mask from the cross-attention maps and compute the IoU metric using the segmentation groundtruth for comparison.

In Fig. 5-(a), we compare the localizing performance with Null-Text inversion (NTI), DPL with each proposed loss separately and our final model. We can observe that the disjoint object attention loss and the background leakage loss are not able to work independently to improve the cross-attention map. The attention balancing loss can enhance the cross-attention quality and can be further boosted by the other two proposed losses. We use the validation set to optimize the various hyperparameters (see Eq. 9 and Eq. 10). Using these parameters, we show similar results on the independent test-set (Fig. 5(a)). These parameters are set for all further results (more details are provided in Supplementary). In Fig. 5-(b), we qualitatively verify our method on some examples of comparison with cross-attention maps. DPL improves the cross-attention quality than NTI as the baseline method.

4.2 Image editing evaluation

For image editing, we combine our proposed DPL with the P2P [15] image editing method. Here we mainly focus on Word-Swap but also include more results for Prompt Refinement, and Attention Re-weighting in the supplementary material.

Word-Swap. In the word swapping scenario, we quantitatively compare NTI and DPL under two scenarios shown in Table 1. For each case, we use the collected 20 images with searching prompts "a cat and a dog" and "a box and a clock" to estimate the evaluation metrics, including CLIP-Score [16] and Structure Dist [41]. In both cases, DPL is getting better performance than the NTI baseline (more shown in Supplementary).

Furthermore, we conduct a forced-choice user study involving 20 evaluators and the assessment of 60 image editing pairs. Users were asked to evaluate 'which result does better represent the prompt change?' (more protocol choices can be found in Supplementary). We can observe a significantly higher level of satisfaction among the participants with our method when compared to NTI.

a piece of <cake> on a wooden plate

... lemon cake ...

... lemon (↑) cake ...

... lemon (↓) cake ...

... pasta (↑) cake ...

... pasta (↓) cake ...

NTI + P2P

DPL + P2P

Cross Attention

OgCaboke

OgCahgke

OCalogke

Qualitatively, we show word-swap examples on multi-object images in Fig. 7. In contrast to other methods, our proposed approach ( DPL ) effectively addresses the issue of inadvertently editing the background or distractor objects in the provided images.

Attention refinement and re-weighting. We also compare NTI and DPL by only tuning a single concept token with additional adjective descriptions in the image in Fig. 6. As can be observed, DPL keeps better details of the "cake" and modifies its appearance according to the new word. NTI suffers from the cross attention leakage to the background, thus it cannot successfully edit only the "cake" region and distorts the "flowers".

5 Conclusions

We presented DPL to solve the cross-attention background and distractor object leakage problem in image editing using text-to-image diffusion models. We propose to update the dynamic tokens for each noun word in the prompt in such a way that the resulting cross-attention maps suffer less from attention leakage. The greatly improved cross-attention maps result in considerably better results for text-guided image editing. The experimental results, confirm that DPL obtains superior results, especially on complex multi-object scenes. In the Supplementary Material, we further provide a comprehensive discussion on the limitations of our work, its broader impact, and potential directions for future research.

Limitations and future works. One limitation of our work is that the smaller cross-attention maps, specifically those with a size of 16 × 16, contain a greater amount of semantic information compared to maps of larger sizes. While this rich information is beneficial, it limits our ability to achieve precise fine-grained structure control. Furthermore, our current approach has not addressed complex scenarios where a single object in the image corresponds to multiple noun words in a long caption. This remains an unexplored area and presents an opportunity for future research. Additionally, the Stable Diffusion model faces challenges when reconstructing the original image with intricate details due to the compression mechanism in the first stage autoencoder model. The editing of high-frequency information remains a significant and ongoing challenge that requires

a <cat> and a <dog>

Plug-and-Play

pix2pix-zero

InstructPix2Pix

NTI + P2P

DPL + P2P

GSAM inpaint

Cat

Bird

Cat

Leopard

Dog

Tiger

a <pear> and an <apple>

Pear

Pineapple

Pear

Pineapple

Apple

Grape

a <cat> and a <bird>

Cat

Leopard

Bird

Airplane

Cat

Fox

Bird

Helicopter

a <basketball>, a <football>, a

<tennis ball> and a <racket>

Football

Balloon

Basketball

Baseball

Basketball

Wheel

S

￥

?

?

further investigation and development. Addressing these limitations and advancing our understanding in these areas will contribute to the improvement and refinement of image editing techniques.

Broader impacts. The application of text-to-image (T2I) models in image editing offers extensive potential for diverse downstream applications, facilitating the adaptation of images to different contexts. The primary objective of our model is to automate and streamline this process, resulting in significant time and resource savings. It is important to acknowledge that current methods have inherent limitations, as discussed in this paper. However, our model can serve as an intermediary solution, expediting the creation process and offering valuable insights for further advancements. It is crucial to remain mindful of potential risks associated with these models, including the dissemination of misinformation, potential for abuse, and introduction of biases. Broader impacts and ethical considerations should be thoroughly addressed and studied in order to responsibly harness the capabilities of such models.

Acknowledgments. Weacknowledge the support from the Spanish Government funding for projects PID2022-143257NB-I00, TED2021-132513B-I00 funded by MCIN/AEI/10.13039/501100011033 and by FSE+ and the European Union NextGenerationEU/PRTR, and the CERCA Programme of Generalitat de Catalunya. We also thank for the insightful discussion with Juan A Rodriguez, Danna Xue, and Marc Paz.

References

Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18208-18218, 2022.

Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In European Conference on Computer Vision , pages 707-723. Springer, 2022.

Romain Beaumont. Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them. https://github.com/rom1504/clip-retrieval , 2022.

Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR , 2023.

Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. arXiv preprint arXiv:2304.08465 , 2023.

Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 , 2023.

Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-andexcite: Attention-based semantic guidance for text-to-image diffusion models. arXiv preprint arXiv:2301.13826 , 2023.

Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. In Proceedings of the International Conference on Computer Vision , pages 14347-14356. IEEE, 2021.

Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusionbased semantic image editing with mask guidance. In The Eleventh International Conference on Learning Representations , 2023.

Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In European Conference on Computer Vision , pages 89-106. Springer, 2022.

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 , 2022.

Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. Proceedings of the International Conference on Computer Vision , 2023.

Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning, 2023.

Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. arXiv preprint arXiv:2304.07090 , 2023.

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 , 2022.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 7514-7528, 2021.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Susung Hong, Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. arXiv preprint arXiv:2210.00939 , 2022.

Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2426-2435, June 2022.

Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu. Ablating concepts in text-to-image diffusion models. Proceedings of the International Conference on Computer Vision , 2023.

Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multiconcept customization of text-to-image diffusion. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Gihyun Kwon and Jong Chul Ye. Diffusion-based image translation using disentangled style and content representation. In The Eleventh International Conference on Learning Representations , 2023.

Senmao Li, Joost van de Weijer, Taihang Hu, Fahad Shahbaz Khan, Qibin Hou, Yaxing Wang, and Jian Yang. Stylediffusion: Prompt-embedding inversion for text-based editing, 2023.

Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept neurons in diffusion models for customized generation. arXiv , 2023.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations , 2022.

Midjourney.com. Midjourney. https://www.midjourney.com , 2022.

Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 16784-16804. PMLR, 17-23 Jul 2022.

Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. Proceedings of the ACM SIGGRAPH Conference on Computer Graphics , 2023.

Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, and Daniel Cohen-Or. Localizing object-level shape variations with text-to-image diffusion models, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821-8831. PMLR, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, June 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487 , 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations , 2021.

Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10748-10757, 2022.

Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Qian Wang, Biao Zhang, Michael Birsak, and Peter Wonka. Mdp: A generalized framework for text-guided image editing by manipulating the diffusion path, 2023.

Tao Yu, Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, and Zhibo Chen. Inpaint anything: Segment anything meets image inpainting. arXiv preprint arXiv:2304.06790 , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a dynamic prompt learning approach for image editing that modifies self-attentions to more accurately attend to the correct nouns given a text prompt. The proposed approach is used along with null text inversion where the dynamic tokens corresponding with the noun words are updated with a background leakage loss, a disjoint object attention loss and an attention balancing loss. Then p2p is used with the modified cross-attention masks to edit the image. The main benefit of this approach is in editing an image which includes two or more semantically related objects like cat and dog. The authors have collected a set of 60 multi-object images from LAION as a test set and a set of 60 images with semantically related objects for their human evaluations, have shown improvements through qualitative and some quantitative evaluations.\n\nSTRENGTHS:\n1. This paper is well written and explains the method and related works clearly.\n2. A novel method as dynamic prompt learning for image editing has been proposed. The paper is mainly focused on a specific axis of image editing where two or more semantically related objects are present in the image and cross attention maps for each object leak to other objects or the background. This approach modifies the attention masks and thus can be used along with p2p for image editing.\n3. A test set containing images of multiple objects from LAION-5B is collected for quantitative and qualitative evaluations and a couple of examples are presented in the paper and supplemental showing the superiority of the proposed method.\n\nWEAKNESSES:\n1. The scope of the image editing problems this paper aims to improve is narrow mainly useful for cases when there are two or multiple semantically-related objects in the image and for word-swapping editing prompts like cat to dog, cat to tiger, etc. \n2. Through only a couple of image editing examples shown in the paper, I am not convinced that this paper is beneficial in general image editing problems where there are not necessarily related objects in the scene or for other types of editing prompts rather than word swapping. \n  -  It is unclear what type of editing prompts are included in the collected test set, but from discussions, it looks like it is mainly focusing on word swapping. Also, size of the test set is fairly small, and the studies are all done in the above narrowed down scope in image editing, thus not conclusive. \n- The quantitative clip scores are only reported in two scenarios (1) cat → leopard and dog → tiger and (2)  book → box and clock → sunflower and improvements are still marginal.\n- On the human evaluation study, I'd expect it to be done on other image editing types with single or multiple objects either semantically related or unrelated, on a larger set, and to compare with other baselines such as instructpix2pix and plug-and-play. \n3. On auto evaluations, one potential metric to measure the correctness/accuracy of this method could be applying an object segmentation or object detection method on the edited image, predict label for the edited objects, check if the predicted label matches with the given prompt. Compute the average accuracy score and compare with the baselines.\n\nQUESTIONS:\n1. Please describe your evaluation datasets and their editing prompts more clearly. Is the URS-Set similar to the MO-Set? what is the \ndifference between these two sets? Since LAION-5B is pretty large, I'd suggest the authors to expand their test set to include more varied image editing prompts and examples.\n2. I wonder how much gain this method introduces on image editing problems with one object or two unrelated objects in the image (if any).\n\nLIMITATIONS:\nThe authors have neither discussed the limitations of their work, nor its negative societal impact. A broader study on different editing types as mentioned above could clarify the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper proposes a new method to improve the attention masking for attention-based local editing.

STRENGTHS:
The proposed loss functions are novel, intuitive and effectively improves the inversion process of text-to-image diffusion model.

WEAKNESSES:
1. Please show the quantitative evaluation of the object masking performance between the mask generated from Diffedit.

2. Quantitative comparison with baseline models are not enough. Please put more comparison between other baselines such as Plug-and-play and Pix2pix-zero, instructPix2Pix. At least include user study.

3. Does it improve other attention-based editing mechanism?

QUESTIONS:
Please see the weakness part.

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
The authors first point out that inferior cross-attention maps regarding noun text tokens are the main causes of failures cases in prompt-based editing methods, such as prompt-to-prompt (P2P).
To tackle this, they propose to optimize noun text features with three objectives at each denoising timestep;
i) minimize the overlap between different cross-attention maps, ii) keep the background mask intact, which is automatically inferred from both cross- and self-attention maps, and iii) keep the max value of cross-attention maps related to nouns high.
This way, they achieve better editing results with cross-attention maps that are better aligned with the object regions.

STRENGTHS:
- the authors tackle a meaningful task
- it is easy to understand the motivation and method with various visualizations
- the proposed method presents improved performance on a variety of evaluation metrics

WEAKNESSES:
- increased inference time
- only can solve cases related to noun text tokens representing the type of object

QUESTIONS:
- regarding the algorithm 1, isn’t it better to use updated text features by implementing the line 9 after the line 13?
- the thresholds defined by Equation 10 appear to prevent overfitting of the model. what happens if you don't use these thresholds?

LIMITATIONS:
please refer to the Weaknesses section
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
The paper propose DPL to solve the cross-attention background and distractor object leakage problem in
image editing using text-to-image diffusion models. The presentation is well written and easy to follow. The discussion and analysis is extensive and interesting. But the experiment dataset is too small to clarify the robustness of this method.

STRENGTHS:
1. The presentation is well written and easy to follow. 
2. The discussion and analysis is extensive and interesting. 
3. The framwork to solve the cross-attention background and distractor object leakage problem is novel.

WEAKNESSES:
1. The experiment dataset is too small to clarify the robustness of this method.

QUESTIONS:
null

LIMITATIONS:
null
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Dynamic Prompt Learning (DPL) to address the cross-attention leakage issue for text-based image editing. Based on the observation that inaccurate cross-attention maps cause unintended modifications of regions outside of the targeted area for text-based image editing, the authors propose Dynamic Prompt Learning to force cross-attention maps to focus on correct noun words in the text prompt. By updating the dynamic tokens for nouns in the textual input with the proposed leakage repairment losses, the proposed approach achieves fine-grained image editing over particular objects while preventing undesired changes to other image regions. Experiments of Stable Diffusion models are conducted for word-swap, prompt refinement, and attention re-weighting.\n\nSTRENGTHS:\n1. This paper proposes an interesting approach to address the attention leakage problem for text-based image editing. By adding constraints on the cross-attention maps, the proposed approach regularizes the attention regions of the diffusion models and thus achieves detail-preserving and high-fidelity image editing.\n2. The proposed approach is well-motivated and the discovery of the attention leakage problem is interesting and inspiring.\n3. The proposed approach achieves better text-based image editing performance on various editing scenarios, as demonstrated in the extensive experiments.\n\nWEAKNESSES:\n1. The proposed DPL approach seems like a combination of several components (including the Disjoint Object Attention Loss, Background Leakage Loss, Attention Balancing Loss, Gradual Optimization for Token Updates, and the trick from null-text inversion), and it is not clear about the contribution and necessity of each component. More ablation study is needed to demonstrate the necessity of combining those methods and loss functions.\n2. It is not shown how accurate and how robust the attention-based background estimation is. When does the background estimation fail and how this will affect the model if this part fails? The choice of hyperparameters (self-attention with size 32, cross-attention with size 16 and TH = 0.2, V = 5) for this subsection is based on empirical study, and ablation study is not presented.\n3. The writing can be improved. The structure of section 3 can be organized more logically and more clearly. The paragraph \"Gradual Optimization for Token Updates\" is unclear to me. \n4. Experiments are conducted on a self-built dataset with a small size (only 100 images for ablation study and 60 images for user study).\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nThe authors addressed the limitations adequately.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper works on the fidelity problem (i.e., 'unintended changes of background and distractor objects') in text-based image editing of text-to-image diffusion models. The authors attribute the fidelity problem to cross-attention leakage and propose dynamic prompt learning to force cross-attention maps to focus on correct noun words in the text prompt. The paper shows a comparison with a couple of approaches, comprehensive image editing evaluations and ablations.\n\nSTRENGTHS:\n1. The fidelity problem that this work focuses on is important. The overall idea of the paper is really simple and seems effective to this problem.\n\n2. The authors present a good number of experiments validating the effectiveness of their approach.  The results from the paper are promising. The editing quality is decent with observable changes. \n\n3. The paper is overall well-written and easy to follow.\n\nWEAKNESSES:\n1. This work has the following potential limitations: \n- The attention map limits the size of the generated target, as shown in Figure 7.\n- Not applicable to the scenarios in that the noun describes the background, or changes are reflected in the verb word, such as adding an item or deleting objectives. Thus, more experiments in the above scenarios should be provided and discussed.\n- Not training-free and may have high latency in practical applications.\n2. Both qualitative and quantitative results miss comparison with important methods such as DiffEdit [1], Imagic [2], StructureDiffusion [3].\n\n3. The manuscript lacks explanations for some symbols, such as symbols in Equation (6). It would be great if the author could highlight these symbols in the diagram to refine the explanation of the method.\n\n[1] Diffedit: Diffusion based semantic image editing with mask guidance. ICLR 2023.\n\n[2] Imagic: Text-based real image editing with diffusion models. CVPR 2023.\n\n[3] Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis. ICLR 2023.\n\nQUESTIONS:\n1. Please clarify the similarities and differences between DPL and Imagic [2]. In my opinion, both are looking for optimized text embedding, while Imagic seems to remain open to control objectives, while DPL adds too many restrictions through attention maps.\n\n2. How long do training and inference take in the proposed method? Can the authors detail the advantages of DPL compared to the training-free methods (e.g., StructureDiffusion [3])?\n\n3. What are the dimensions of the attention maps, the query/key matrix, and the deep features of the noisy image.  It will be great if the authors could indicate those dimensions in the paper.\n\nLIMITATIONS:\nThe authors have provided discussion on the limitations and broader impact in the Supplementary Material. However, as mentioned in the weaknesses, this work has the following potential limitations: \n- 1) The attention map limits the size of the generated target.\n- 2) Not applicable to the scenarios that the noun describes the background, or changes are reflected in verb word, such as adding an item or deleting objectives.\n- 3) Not training-free and may have high latency in practical applications.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-19"}