SlowGuess/NeurIPS_2023_OCR huggingface dataset:
be able to reconstruct the original image, we use Null-Text inversion [28] in addition to learn a set of null embeddings  $\varnothing_{t}$  for each timestamp  $t$ . Then at the time  $t$ , we have a set of learnable word embeddings  $\mathcal{V}_t$  and null text  $\varnothing_{t}$  which can accurately localize the objects and also reconstruct the original image.
3.3 Attention-based background estimation
To address the background attention leakage, we investigate the localization information of self-attention and feature representations, which are present in the diffusion model. Besides cross-attention (CA), during the inversion of an image with the DDIM sampler, we can also obtain self-attention maps (SA), or directly analyze the query, key and value in the self-attention maps. In Fig. 3 we visualize these maps. The main observation is that self-attention maps more closely resemble the semantic layout. We therefore exploit these to reduce background leakage.
To acquire the background via self-cross attention matching, we use the agreement score [31] between cross-attention  $\mathcal{A}^n$  of noun words  $n$  and each of the self-attention clusters  $\mathcal{M}_v$ ,  $v \in [1, V]$ , defined as  $\sigma_{nv} = \sum (\mathcal{A}^n \cdot \mathcal{M}_v) / \sum \mathcal{M}_v$ . However, it should be noted that the agreement score does not accurately segment the objects, since there is still attention leakage between objects and distractor objects (see Fig. 4 for examples). Therefore, other than localizing objects as in [31], we use the agreement score only for background mask estimation. We label the cluster  $\mathcal{M}_v$  as being background (BG) if for all  $n$
if  $\sigma_{nv} < TH$ , where  $TH$  is a threshold. Empirically, we choose self-attention with size 32, cross-attention with size 16 and  $TH = 0.2$ ,  $V = 5$  to obtain the background mask  $\mathcal{B}$ .
Algorithm 1: Dynamic Prompt Learning
1 Input: A source prompt  $\mathcal{P}$ , an input image  $\mathcal{I}$
2 Output: A noise vector  $\bar{z}_T$ , a set of updated tokens  $\{\mathcal{V}_t\}_1^T$  and null-text embeddings  $\{\varnothing_t\}_1^T$
3 Set  $T = 50$  and scale  $w = 1$ ;
4  $\{z_t\}_{0}^T,\mathcal{B}\gets \mathrm{DDIM - inv}(\mathcal{I})$
5 Set guidance scale  $w = 7.5$
6 Initialize  $\mathcal{V}_T$  with original noun tokens;
7 Initialize  $\bar{z}_T = z_T, \mathcal{P}_T = \mathcal{P}, \varnothing_T = \tau_\theta(^\prime$ "
8 for  $t = T, T - 1, \ldots, 1$  do
9 | Initialize  $\mathcal{P}_t$  by  $\nu_{t}$  , then  $\mathcal{C}_t = \tau_\theta (\mathcal{P}_t)$
10 while  $\mathcal{L}_{at}\geq TH_{t}^{at}$  or  $\mathcal{L}_{dj}\geq TH_t^{dj}$  or
$$
\begin{array}{l} \mathcal {L} _ {b q} \geq T H _ {t} ^ {b g} \mathbf {d o} \\ \left| \begin{array}{c} \mathcal {L} = \lambda_ {a t} \cdot \mathcal {L} _ {a t} + \lambda_ {d j} \cdot \mathcal {L} _ {d j} + \lambda_ {b g} \cdot \mathcal {L} _ {b g} \\ \mathcal {V} _ {t} \gets \mathcal {V} _ {t} - \nabla_ {\mathcal {V} _ {t}} \mathcal {L} \end{array} \right. \\ \end{array}
$$
14 Update  $\mathcal{P}_t$  by  $\nu_{t}$  , then  $\mathcal{C}_t = \tau_\theta (\mathcal{P}_t)$
$$
\begin{array}{l} \tilde {z} _ {t} = \tilde {\epsilon} _ {\theta} (\bar {z} _ {t}, t, \mathcal {C} _ {t}, \emptyset_ {t}) (15) \\ \bar {z} _ {t - 1}, \varnothing_ {t} \leftarrow N T I (\tilde {z} _ {t}, \varnothing_ {t}) (16) \\ \begin{array}{l l} \text {I n i t i a l i z e} \varnothing_ {t - 1} = \varnothing_ {t}, \mathcal {V} _ {t - 1} = \mathcal {V} _ {t} \end{array} \\ \end{array}
$$
18 end
19 Return  $\bar{z}_T, \{\mathcal{V}_t\}_{1}^T, \{\varnothing_t\}_{1}^T$
We demonstrate our method in various experiments based on the open-source T2I model Stable Diffusion [35] following previous methods [42, 30, 28]. All experiments are done on an A40 GPU.
(a) IoU curves on the Valid-split (left) and the Test-Split (right)
Figure 5: (a) IoU curves drawn by varying the threshold from 0.0 to 1.0 for MO-Set. (b) We show some images for comparisons between the Null-Text inversion [28] and our method DPL demonstrated in cross-attention maps.
(b) Cross-attention maps comparison
Dataset. We focus here on real-image editing, but also provide examples of generated image editing in the Supplementary Material. We use clip retrieval [3] to obtain experimental multi-object real images from the LAION-5B [39] dataset by matching CLIP embeddings of the source text description. For the ablation study, we select 100 multi-object images from various search prompts (including concepts as bird, dog, cat, book, clock, pen, pizza, table, horse, sheep, etc.). This Multi-Object set is abbreviated with MO-Set. Of the images, 40 images are used for validation and the other 60 are test set. We manually verify their segmentation maps which serve as groundtruth in the ablation study IoU computation. Finally, for user study, we create a URS-Set of 60 multi-object images of semantically related objects from various concepts.
Comparison methods. For qualitative comparison, we compare with pix2pix-zero [30], GSAM-inpaint [44], InstructPix2Pix [4], Plug-and-Play [42] and NTI+P2P [28]. For quantitative evaluation, we benchmark our approach against NTI+P2P, considering it as a robust baseline for maintaining content stability and achieving superior conceptual transfer.
4.1 Ablation study
The ablation study is conducted on the MO-Set set. To quantitatively evaluate the performance of DPL in localization, we vary the threshold from 0.0 to 1.0 to obtain the segmentation mask from the cross-attention maps and compute the IoU metric using the segmentation groundtruth for comparison.
In Fig. 5-(a), we compare the localizing performance with Null-Text inversion (NTI),  $DPL$  with each proposed loss separately and our final model. We can observe that the disjoint object attention loss and the background leakage loss are not able to work independently to improve the cross-attention map. The attention balancing loss can enhance the cross-attention quality and can be further boosted by the other two proposed losses. We use the validation set to optimize the various hyperparameters (see Eq. 9 and Eq. 10). Using these parameters, we show similar results on the independent test-set (Fig. 5-(a)). These parameters are set for all further results (more details are provided in Supplementary). In Fig. 5-(b), we qualitatively verify our method on some examples of comparison with cross-attention maps.  $DPL$  improves the cross-attention quality than NTI as the baseline method.
4.2 Image editing evaluation
For image editing, we combine our proposed DPL with the P2P [15] image editing method. Here we mainly focus on Word-Swap but also include more results for Prompt Refinement, and Attention Re-weighting in the supplementary material.
Word-Swap. In the word swapping scenario, we quantitatively compare NTI and  $DPL$  under two scenarios shown in Table 1. For each case, we use the collected 20 images with searching prompts "a cat and a dog" and "a box and a clock" to estimate the evaluation metrics, including CLIP-Score [16] and Structure Dist [41]. In both cases,  $DPL$  is getting better performance than the NTI baseline (more shown in Supplementary).
Furthermore, we conduct a forced-choice user study involving 20 evaluators and the assessment of 60 image editing pairs. Users were asked to evaluate 'which result does better represent the prompt change?' (more protocol choices can be found in Supplementary). We can observe a significantly higher level of satisfaction among the participants with our method when compared to NTI.
Table 1: Comparison between our method  $DPL + \mathrm{P2P}$  and the baseline NTI+P2P by modifying multiple objects simultaneously. we examined the quality of editing by transforming an image depicting "a cat and a dog" into "a leopard and a tiger". We evaluated the editing quality of both approaches using objective metrics, such as CLIP-Score and Structure-Dist, across these multi-object collections. Additionally, to capture the subjective opinions of users, we conducted a user study involving 60 pairs of multi-object image translations.
Method	cat → leopard and dog → tiger	book → box and clock → sunflower	User Study
CLIP-Score (↑)	Structure Dist (↓)	CLIP-Score (↑)	Structure Dist (↓)
Original Images	32.50%	-	16.46%	-	-
NTI [28] + P2P	31.50%	0.0800	12.91%	0.0191	12.8%
DPL (ours) + P2P	32.23%	0.0312	14.29%	0.0163	87.2%
Figure 6: Attention refinement and reweighting. In the given text prompt, only one noun word is learnable. The flowers below the cake are playing as distractors which distort the cross-attention maps while Null-Text Inversion [28] is applied. As a comparison, our method DPL successfully filters the cross-attention by our background leakage loss.
Qualitatively, we show word-swap examples on multi-object images in Fig. 7. In contrast to other methods, our proposed approach (DPL) effectively addresses the issue of inadvertently editing the background or distractor objects in the provided images.
Attention refinement and re-weighting. We also compare NTI and  $DPL$  by only tuning a single concept token with additional adjective descriptions in the image in Fig. 6. As can be observed,  $DPL$  keeps better details of the "cake" and modifies its appearance according to the new word. NTI suffers from the cross attention leakage to the background, thus it cannot successfully edit only the "cake" region and distorts the "flowers".
We presented  $DPL$  to solve the cross-attention background and distractor object leakage problem in image editing using text-to-image diffusion models. We propose to update the dynamic tokens for each noun word in the prompt in such a way that the resulting cross-attention maps suffer less from attention leakage. The greatly improved cross-attention maps result in considerably better results for text-guided image editing. The experimental results, confirm that  $DPL$  obtains superior results, especially on complex multi-object scenes. In the Supplementary Material, we further provide a comprehensive discussion on the limitations of our work, its broader impact, and potential directions for future research.
Limitations and future works. One limitation of our work is that the smaller cross-attention maps, specifically those with a size of  $16 \times 16$ , contain a greater amount of semantic information compared to maps of larger sizes. While this rich information is beneficial, it limits our ability to achieve precise fine-grained structure control. Furthermore, our current approach has not addressed complex scenarios where a single object in the image corresponds to multiple noun words in a long caption. This remains an unexplored area and presents an opportunity for future research. Additionally, the Stable Diffusion model faces challenges when reconstructing the original image with intricate details due to the compression mechanism in the first stage autoencoder model. The editing of high-frequency information remains a significant and ongoing challenge that requires
Figure 7: Word-Swap by only modifying the regions corresponding to the target concept. We list the source prompts on the left with dynamic tokens in the curly brackets. Each line is corresponding to modifying one or multiple concepts with various methods.
further investigation and development. Addressing these limitations and advancing our understanding in these areas will contribute to the improvement and refinement of image editing techniques.
Broader impacts. The application of text-to-image (T2I) models in image editing offers extensive potential for diverse downstream applications, facilitating the adaptation of images to different contexts. The primary objective of our model is to automate and streamline this process, resulting in significant time and resource savings. It is important to acknowledge that current methods have inherent limitations, as discussed in this paper. However, our model can serve as an intermediary solution, expediting the creation process and offering valuable insights for further advancements. It is crucial to remain mindful of potential risks associated with these models, including the dissemination of misinformation, potential for abuse, and introduction of biases. Broader impacts and ethical considerations should be thoroughly addressed and studied in order to responsibly harness the capabilities of such models.
Acknowledgments. We acknowledge the support from the Spanish Government funding for projects PID2022-143257NB-I00, TED2021-132513B-I00 funded by MCIN/AEI/10.13039/501100011033 and by FSE+ and the European Union NextGenerationEU/PRTR, and the CERCA Programme of Generalitat de Catalunya. We also thank for the insightful discussion with Juan A Rodriguez, Danna Xue, and Marc Paz.
[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[36] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[37] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[38] Chitwan Sahara, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022.
[39] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[40] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[41] Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10748-10757, 2022.
[42] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[43] Qian Wang, Biao Zhang, Michael Birsak, and Peter Wonka. Mdp: A generalized framework for text-guided image editing by manipulating the diffusion path, 2023.
[44] Tao Yu, Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, and Zhibo Chen. Inpaint anything: Segment anything meets image inpainting. arXiv preprint arXiv:2304.06790, 2023.