SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kai Wang $^{1,2}$ , Fei Yang $^{3*}$ , Shiqi Yang $^{1,2}$ , Muhammad Atif Butt $^{1,2}$ , Joost van de Weijer $^{1,2}$
<sup>1</sup>Computer Vision Center, Barcelona, Spain
2 Universitat Autonoma de Barcelona, Barcelona, Spain
<sup>3</sup> College of Computer Science, Nankai University, Tianjin, China
{kwang, syang, mabutt, joost}@cvc.uab.es, feiyang@nankai.edu.cn
Large-scale text-to-image generative models have been a ground-breaking development in generative AI, with diffusion models showing their astounding ability to synthesize convincing images following an input text prompt. The goal of image editing research is to give users control over the generated images by modifying the text prompt. Current image editing techniques are susceptible to unintended modifications of regions outside the targeted area, such as on the background or on distractor objects which have some semantic or visual relationship with the targeted object. According to our experimental findings, inaccurate cross-attention maps are at the root of this problem. Based on this observation, we propose Dynamic Prompt Learning (DPL) to force cross-attention maps to focus on correct noun words in the text prompt. By updating the dynamic tokens for nouns in the textual input with the proposed leakage repairment losses, we achieve fine-grained image editing over particular objects while preventing undesired changes to other image regions. Our method DPL, based on the publicly available Stable Diffusion, is extensively evaluated on a wide range of images, and consistently obtains superior results both quantitatively (CLIP score, Structure-Dist) and qualitatively (on user-evaluation). We show improved prompt editing results for Word-Swap, Prompt Refinement, and Attention Re-weighting, especially for complex multi-object scenes.
Text-to-Image (T2I) is advancing at a revolutionary pace and has demonstrated an unprecedented ability to generate diverse and realistic images [27, 33, 34, 35]. The state-of-the-art T2I models are trained on extremely large language-image datasets, which require huge computational resources. However, these models do not allow for straightforward text-guided image editing and generally lack the capability to control specific regions of a given image.
Recent research on text-guided image editing allows users to easily manipulate an image using only text prompts [5, 9, 14, 15, 24, 42, 43]. In this paper, we focus on prompt based image editing, where we aim to change the visual appearance of a target object (or objects) in the image. Several of the existing methods use DDIM inversion [40] to attain the initial latent code of the image and then apply their proposed editing techniques along the denoising phase [28, 30, 42]. Nevertheless, current text-guided editing methods are susceptible to unintentional modifications of image areas. We distinguish between two major failure cases: unintended changes of background regions and unintended changes of distractor objects (objects that are semantically or visually related to the target object), as shown in Fig. 1. The functioning of existing editing methods greatly depends on
Figure 1: Compared with other image editing methods, our  $DPL$  achieves more consistent layout when modifying one or more objects in the image and keeping other content frozen. And also the desired editing is correctly delivered to the corresponding area. DiffEdit [9] method thoroughly fails to detect the editing region in multi-object cases. And Null-Text inversion (NTI [28]) is unable to perfectly distinguish the objects in the given image since the cross-attention maps are suffering background leakage (yellow circle) and distractor object leakage (red circle).  $DPL$  can more successfully localize the objects in the textual prompts, thus benefits for future editing. Here the  $16 \times 16$  cross-attention maps are interpolated to the image size for better view. (image credits: Gettyimages)
the accuracy of the cross-attention maps. When visualizing the cross-attention maps (see Fig. 1), we can clearly observe that for DDIM [40] and Null-Text inversion [28] these maps do not only correlate with the corresponding objects. We attribute this phenomenon to cross-attention leakage to background and distractor objects. Cross-attention leakage is the main factor to impede these image editing methods to work for complex backgrounds and multi-object images. Similar observations for failure cases are also shown in [9, 14, 19, 22, 28]. For example, Custom Diffusion [22] fails for multi-concept composition and DiffEdit does not correctly detect the mask region for a given concept.
We conduct a comprehensive analysis of the Stable Diffusion (SD) model and find that the inaccurate attribution of cross-attention (resulting in undesirable changes to background and distractor objects) can be mitigated by exploiting the semantic clusters that automatically emerge from the self-attention maps in the diffusion model. Instead of fine-tuning the diffusion model, which potentially leads to catastrophic neglect[7], we propose replacing prompts related to scene objects with dynamic prompts that are optimized. We propose two main losses for dynamic prompt learning (DPL), one tailored to reduce attention leakage to distractor objects and one designed to prevent background attention leakage. Moreover, better cross-attention maps are obtained when we update the prompts for each timestamp (as implied by dynamic). Furthermore, to facilitate meaningful image editing, we use Null-Text Inversion [28] with classifier-free guidance [17]. In the experiments, we quantitatively evaluate the performance of DPL on images from LAION-5B [39] dataset. And DPL shows superior evaluation metrics (including CLIP-Score and Structure-Dist) and improved user evaluation. Furthermore, we show prompt-editing results for Word-Swap, Prompt Refinement, and Attention Re-weighting on complex multi-object scenes.
For text-guided image editing with diffusion models, various methods [2, 8, 20, 23, 24] leverage CLIP [32] for image editing based on a pretrained unconditional diffusion model. However, they are limited to the generative prior which is only learned from visual data of a specific domain, whereas the CLIP text-image alignment information is from much broader data. This prior knowledge gap is
mitigated by recent progress of text-to-image (T2I) models [6, 10, 18, 33, 34, 38]. Nevertheless, these T2I models offer little control over the generated contents. This creates a great interest in developing methods to adopt such T2I models for controllable image editing. SDEdit [26], a recent work utilizing diffusion models for image editing, follows a two-step process. It introduces noise to the input image and then applies denoising using the SDE prior to enhance realism and align with user guidance. Imagc [19] and P2P [15] attempt structure-preserving editing via Stable Diffusion (SD) models. However, Imagc [19] requires fine-tuning the entire model for each image and focuses on generating variations for the objects. P2P [15] has no need to fine-tune the model, it retrans the image structure by assigning cross-attention maps from the original image to the edited one in the corresponding text token. InstructPix2Pix [4] is an extension of P2P by allowing human-like instructions for image editing. To make the P2P capable of handling real images, Null-Text inversion (NTI) [28] proposed to update the null-text embeddings for accurate reconstruction to accommodate with the classifier-free guidance [17]. Recently, pix2pix-zero [30] propose noise regularization and cross-attention guidance to retrain the structure of a given image. However, it only supports image translation from one concept to another one and is not able to deal with multi-concept cases. DiffEdit [9] automatically generates a mask highlighting regions of the input image by contrasting predictions conditioned on different text prompts. Plug-and-Play (PnP) [42] demonstrated that the image structure can be preserved by manipulating spatial features and self-attention maps in the T2I models. MnM [31] clusters self-attention maps and compute their similarity to the cross attention maps to determine the object localization, thus benefits to generate object-level shape variations. StyleDiffusion [24] adds up a mapping network to modify the value component in cross-attention computation for regularizing the attentions and utilizes P2P for image editing.
These existing methods highly rely on precise cross-attention maps and overlook an important case where the cross-attention maps may not be perfectly connected to the concepts given in the text prompt, especially if there are multiple objects in the given image. Therefore, these imperfect cross-attention maps may lead to undesirable editing leakage to unintended regions. Essentially, our method  $DPL$  complements existing methods to have more semantic richer cross-attention maps. There are various diffusion-based image editing methods relying on giving a mask manually [29, 1] or detecting a mask automatically (GSAM-inpaint [44]) using a pretrained segmentation model. On the contrary, our method only requires textual input and automatically detects the spatial information. This offers a more intuitive editing demand by only manipulating the text prompts.
There are some papers [21, 12] concentrated on object removal or inpainting using user-defined masks [1, 29, 44]. In contrast, the act of placing an object in a suitable position without relying on a mask is an active research domain. This direction usually entails fine-tuning models for optimal outcomes and runs parallel to the research direction explored in this paper.
In this paper, we are also related to the transfer learning for T2I models [22, 37, 11, 13, 25], which aims at adapting a given model to a new concept by given images from the users and bind the new concept with a unique token. However, instead of finetuning the T2I model [25, 13, 37], we learn new concept tokens by personalization in the text embedding space, which is proposed in Textual Inversion [11]. By this means, we avoid updating a T2I model for each concept separately. Furthermore, we step forward to the scenario with complex background or multiple objects, where the previous methods have not considered. In contrast to transfer learning approaches, which aim to ensure that the learned concepts can accurately generate corresponding objects in given images, our proposed approach  $DPL$  focuses on enhancing image editing tasks by regularizing cross-attention to disentangle their positions.
In this section, we first provide a short preliminary and then describe our method to achieve this demand. An illustration of our method is shown in Fig. 2 and Algorithm 1.
Latent Diffusion Models. We use Stable Diffusion v1.4 which is a Latent Diffusion Model (LDM) [35]. The model is composed of two main components: an autoencoder and a diffusion model. The encoder  $\mathcal{E}$  from the autoencoder component of the LDMs maps an image  $\mathcal{I}$  into a latent code  $z_0 = \mathcal{E}(\mathcal{I})$  and the decoder reverses the latent code back to the original image as  $\mathcal{D}(\mathcal{E}(\mathcal{I}))\approx \mathcal{I}$ . The
Figure 2: Dynamic Prompt Learning (DPL) first transforms the noun words in the text prompt into dynamic tokens. We register them in the token dictionary and initialize the representations by their original words at time  $T$ . Using DDIM inversion, we condition the DM-UNet with the original text condition  $\mathcal{C}$  to get the diffusion trajectory  $\{z_t\}_0^T$  and background mask  $\mathcal{B}$ . In each denoising step  $\bar{z}_{t - 1}\rightarrow \bar{z}_t$ , we first update the dynamic token set  $\mathcal{V}_t$  with a background leakage loss, a disjoint object attention loss and an attention balancing loss, in order to ensure high-quality cross-attention maps. Then we apply Null-Text inversion to approximate the diffusion trajectory.
diffusion model can be conditioned on class labels, segmentation masks or textual input. Let  $\tau_{\theta}(y)$  be the conditioning mechanism which maps a condition  $y$  into a conditional vector for LDMs, the LDM model is updated by the loss:
$$
L _ {L D M} := \mathbb {E} _ {z _ {0} \sim \mathcal {E} (x), y, \epsilon \sim \mathcal {N} (0, 1), t \sim \operatorname {U n i f o r m} (1, T)} \left[ \| \epsilon - \epsilon_ {\theta} \left(z _ {t}, t, \tau_ {\theta} (y)\right) \| _ {2} ^ {2} \right] \tag {1}
$$
The neural backbone  $\epsilon_{\theta}$  is typically a conditional UNet [36] which predicts the added noise. More specifically, text-guided diffusion models aim to generate an image from a random noise  $z_{T}$  and a conditional input prompt  $\mathcal{P}$ . To distinguish from the general conditional notation in LDMs, we itemize the textual condition as  $\mathcal{C} = \tau_{\theta}(\mathcal{P})$ .
DDIM inversion. Inversion entails finding an initial noise  $z_{T}$  that reconstructs the input latent code  $z_{0}$  upon sampling. Since we aim at precisely reconstructing a given image for future editing, we employ the deterministic DDIM sampling [40]:
$$
z _ {t + 1} = \sqrt {\bar {\alpha} _ {t + 1}} f _ {\theta} \left(z _ {t}, t, \mathcal {C}\right) + \sqrt {1 - \bar {\alpha} _ {t + 1}} \epsilon_ {\theta} \left(z _ {t}, t, \mathcal {C}\right) \tag {2}
$$
where  $\bar{\alpha}_{t + 1}$  is noise scaling factor defined in DDIM [40] and  $f_{\theta}(z_t,t,\mathcal{C})$  predicts the final denoised latent code  $z_0$  as  $f_{\theta}(z_t,t,\mathcal{C}) = \left[z_t - \sqrt{1 - \bar{\alpha}_t}\epsilon_{\theta}(z_t,t,\mathcal{C})\right] / \sqrt{\bar{\alpha}_t}$ .
Null-Text inversion. To amplify the effect of conditional textual input, classifier-free guidance [17] is proposed to extrapolate the conditioned noise prediction with an unconditional noise prediction. Let  $\varnothing = \tau_{\theta}(\dots)$  denote the null text embedding, then the classifier-free guidance is defined as:
$$
\tilde {\epsilon} _ {\theta} \left(z _ {t}, t, \mathcal {C}, \varnothing\right) = w \cdot \epsilon_ {\theta} \left(z _ {t}, t, \mathcal {C}\right) + (1 - w) \cdot \epsilon_ {\theta} \left(z _ {t}, t, \varnothing\right) \tag {3}
$$
where we set the guidance scale  $w = 7.5$  as is standard for Stable Diffusion [17, 28, 35]. However, the introduction of amplifier-free guidance complicates the inversion process, and the generated image based on the found initial noise  $z_{T}$  deviates from the original image. Null-text inversion [28] proposes a novel optimization which updates the null text embedding  $\varnothing_{t}$  for each DDIM step  $t \in [1,T]$  to approximate the DDIM trajectory  $\{z_t\}_0^T$  according to:
$$
\min  _ {\varnothing_ {t}} \left\| \bar {z} _ {t - 1} - \tilde {\epsilon} _ {\theta} \left(\bar {z} _ {t}, t, \mathcal {C}, \varnothing_ {t}\right) \right\| _ {2} ^ {2} \tag {4}
$$
where  $\{\bar{z}_t\}_{0}^T$  is the backward trace from Null-Text inversion. Finally, this allows to edit real images starting from initial noise  $\bar{z}_T = z_T$  using the learned null-text  $\varnothing_{t}$  in combination with P2P [15].
Note, that the cross-attention maps of Null-Text inversion  $(w = 7.5)$  and DDIM inversion  $(w = 1.0)$  are similar, since NTI is aiming to approximate the same denoising path.
Prompt2Prompt. P2P [15] proposed to achieve editing by manipulating the cross-attention  $\mathcal{A}_t^*$  from target input  $\mathcal{P}^*$  according to the cross-attention  $\mathcal{A}_t$  from the source input  $\mathcal{P}$  in each timestamp  $t$ . The quality of the estimated attention maps  $\mathcal{A}_t$  is crucial for the good functioning of P2P. This paper addresses the problem of improved  $\mathcal{A}_t$  estimation for complex images.
3.2 DPL: Dynamic Prompt Learning for addressing cross-attention leakage
Prompt-based image editing takes an image  $\mathcal{I}$  described by an initial prompt  $\mathcal{P}$ , and aims to modify it according to an altered prompt  $\mathcal{P}^*$  in which the user indicates desired changes. The initial prompt is used to compute the cross-attention maps given the conditional text input  $\mathcal{P}$ . As discussed in Sec. 1, the quality of cross-attention maps plays a crucial role for prompt-based image editing methods (see Fig. 1), since these attention maps will be used during the generation process with the altered user prompt [15]. We identify two main failures in the attention map estimation: (1) distractor object leakage, which refers to wrong assignment of attention to a semantically or visually similar foreground object and (2) background leakage, referring to undesired attention on the background. Both of these problems result in artifacts or undesired modifications in generated images.
The cross-attention maps in the Diffusion Model UNet are obtained from  $\epsilon_{\theta}(z_t,t,\mathcal{C})$ , which is the first component in Eq. 3. They are computed from the deep features of noisy image  $\psi (z_{t})$  which are projected to a query matrix  $Q_{t} = l_{Q}(\psi (z_{t}))$ , and the textual embedding which is projected to a key matrix  $K = l_{K}(\mathcal{C})$ . Then the attention map is computed according to:
$$
\mathcal {A} _ {t} = \operatorname {s o f t m a x} \left(Q _ {t} \cdot K ^ {T} / \sqrt {d}\right) \tag {5}
$$
where  $d$  is the latent dimension, and the cell  $[\mathcal{A}_i]_{ij}$  defines the weight of the  $j$ -th token on the pixel  $i$ .
In this paper, we propose to optimize the word embeddings  $v$  corresponding to the initial prompt  $\mathcal{P}$ , in such a way that the resulting cross-attention maps  $\mathcal{A}_t$  do not suffer from the above-mentioned problems, i.e. distractor-object and background leakage. We found that optimizing the embedding  $v_t$  separately for each timestamp  $t$  leads to better results than merely optimizing a single  $v$  for all timestamps. The initial prompt  $\mathcal{P}$  contains  $K$  noun words and their corresponding learnable tokens at each timestamp  $\mathcal{V}_t = \{v_t^1, \dots, v_t^k, \dots, v_t^K\}$ . We update each specified word in  $\mathcal{V}_t$  for each step  $t$ .
The final sentence embedding  $\mathcal{C}_t$  now varies for each timestamp  $t$  and is computed by applying the text transformer on the text embeddings: some of which are learned (those referring to nouns) while the rest remains constant for each timestamp (see Fig. 2). For example, given a prompt "a puppy and a rabbit" and the updatable noun set {puppy, rabbit}, in each timestamp  $t$  we have transformed the textual input into trainable ones as "a <puppy> and a <rabbit>" and we aim to learn word embeddings  $v_t^{<\text{puppy}>}$  and  $v_t^{<\text{rabbit}>}$  in each timestamp  $t$ .
We propose several losses to optimize the embedding vectors  $\mathcal{V}_t$ : we develop one loss to address cross-attention leakage due to object distractors, and another one to address background leakage. We also found that the attention balancing loss, proposed by Attend-and-Excite [7], was useful to ensure each specified noun word from  $\mathcal{P}$  is focusing on its relevant concept.
Disjoint Object Attention Loss. To address the problem of distractor object attention leakage, we propose a loss that minimizes the attention overlap between the attention maps corresponding to the different objects, that is an updatable noun set  $\mathcal{V}_t$ . This prevents specific objects related regions to be linked to multiple nouns, which is one of the sources of distractor object attention. We propose to minimize a cosine similarity loss among the  $K$  cross-attention maps (referring to the noun words in the input prompt  $\mathcal{P}$ ) to avoid their overlapping in cross-attention maps through:
$$
\mathcal {L} _ {d j} = \sum_ {i = 1} ^ {K} \sum_ {\substack {j = 1 \\ i \neq j}} ^ {K} \cos \left(\mathcal {A} _ {t} ^ {v _ {t} ^ {i}}, \mathcal {A} _ {t} ^ {v _ {t} ^ {j}}\right) \tag{6}
$$
Background Leakage Loss. In order to alleviate the background leakage issue, we propose to utilize background masks to keep the background area unchanged. Details on how to get the background mask are discussed in Sec. 3.3. After obtaining the BG mask  $\mathcal{B}$  of the given image, for each  $v_t^k\in \mathcal{V}_t$
Figure 3: After DDIM inversion, we visualize the average of each component over  $T$  timestamps for the input image in Fig. 1 (with the prompt: "a cat and a dog"). From left to right, they are cross-attention (CA) for each noun word, then PCA&clustering of the self-attention (SA), query, key and value representations. Here, the maps are simply resized without any interpolation to demonstrate their original views.
we force its cross-attention map to have minimal overlap with the background region:
$$
\mathcal {L} _ {b g} = \sum_ {i = 1} ^ {K} \cos \left(\mathcal {A} _ {t} ^ {v _ {t} ^ {i}}, \mathcal {B}\right) \tag {7}
$$
Attention Balancing Loss. For each  $v_{t}^{k} \in \mathcal{V}_{t}$ , we pass its cross-attention map through a Gaussian filter  $\mathcal{F}$  and formulate a set of losses  $\mathcal{L}_{v_t^k}$  for all words in  $\mathcal{V}_t$ . The attention loss is defined as the maximum value in  $\mathcal{L}_{v_t^k}$ :
$$
\mathcal {L} _ {a t} = \max  _ {v _ {t} ^ {k} \in \mathcal {V} _ {t}} \mathcal {L} _ {v _ {t} ^ {k}} \quad \text {w h e r e} \quad \mathcal {L} _ {v _ {t} ^ {k}} = 1 - \max  [ \mathcal {F} \left(\mathcal {A} _ {t} ^ {v _ {t} ^ {k}}\right) ] \tag {8}
$$
The attention loss will strengthen the cross-attention activations for localizing token  $v_{t}^{k}$ , thus force  $\mathcal{A}_{t}^{v}$  to not leak attentions to other irrelevant concepts in the image. In other words, the resulting cross-attention maps will be more concentrated on the corresponding object region. Different from Attend-and-Excite [7] where the refinement loss is applied to update the latent code to augment the multiple object generation, we are the first to explore its role in learning new conceptual tokens.
With all these three losses, we aim to update the learnable token  $v_{t}^{k}$  as:
$$
\underset {\mathcal {V} _ {t}} {\arg \min } \mathcal {L} \quad \text {w h e r e} \quad \mathcal {L} = \lambda_ {a t} \cdot \mathcal {L} _ {a t} + \lambda_ {d j} \cdot \mathcal {L} _ {d j} + \lambda_ {b g} \cdot \mathcal {L} _ {b g} \tag {9}
$$
Note that, when there is only one updatable noun token in the text input  $\mathcal{P}$ , we only apply the  $\mathcal{L}_{bg}$  to relieve the background leakage since other losses rely on at least two words. In this paper, we apply DPL over the  $16^{2}$  attention following the common practices [15, 7] for a fair comparison.
Gradual Optimization for Token Updates. So far, we introduced the losses to learn new dynamic tokens at each timestamp. However, the cross-attention leakage is gradually accumulated in the denoising phase. Hence, we enforce all losses to reach a pre-defined threshold at each timestamp  $t$  to avoid overfitting of the cross-attention maps. We express the gradual threshold by an exponential function. For the losses proposed above, the corresponding thresholds at time  $t$  are defined as:
$$
T H _ {t} ^ {a t} = \beta_ {a t} \cdot \exp (- t / \alpha_ {a t}); \quad T H _ {t} ^ {d j} = \beta_ {d j} \cdot \exp (- t / \alpha_ {d j}); \quad T H _ {t} ^ {b g} = \beta_ {b g} \cdot \exp (- t / \alpha_ {b g}). \tag {10}
$$
We verify the effectiveness of this mechanism together with other hyperparameters in our ablation experiments (see Fig. 5-(a) and Supplementary Material.).
Null-Text embeddings. The above described token updating ensures that the cross-attention maps are highly related to the noun words in the text prompt and minimize cross-attention leakage. To
Figure 4: Although cross-attention maps produce acceptable results in certain cases (up) with minor leakage, the object masks from self-cross attention matching are not always accurate due to the leakage problem (down). However, as the cross-attention maps tend to concentrate on the foreground objects, the background masks remain reliable.