HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models
Authors: Hayk Manukyan
Andranik Sargsyan
Barsegh Atanyan
Zhangyang Wang
Shant Navasardyan
Humphrey Shi
Venue: ICLR 2025 Poster (2025)
Keywords: text-guided image inpainting
diffusion models
high-resolution image inpainting
Primary area: generative models
TL;DR: We present a text-guided image inpainting method which improves prompt alignment, prevents image quality loss and enables high-resolution inpainting

Abstract: Recent progress in text-guided image inpainting, based on the unprecedented success of text-to-image diffusion models, has led to exceptionally realistic and visually plausible results. However, there is still significant potential for improvement in current text-to-image inpainting models, particularly in better aligning the inpainted area with user prompts. Therefore, we introduce $\textit{HD-Painter}$, a $\textbf{training-free}$ approach that $\textbf{accurately follows prompts}$. To this end, we design the $\textit{Prompt-Aware Introverted Attention (PAIntA)}$ layer enhancing self-attention scores by prompt information resulting in better text aligned generations. To further improve the prompt coherence we introduce the $\textit{Reweighting Attention Score Guidance (RASG)}$ mechanism seamlessly integrating a post-hoc sampling strategy into the general form of DDIM to prevent out-of-distribution latent shifts. Our experiments demonstrate that HD-Painter surpasses existing state-of-the-art approaches quantitatively and qualitatively across multiple metrics and a user study. Code is publicly available at: [https://github.com/Picsart-AI-Research/HD-Painter](https://github.com/Picsart-AI-Research/HD-Painter)

Published as a conference paper at ICLR 2025
HD-PAINTER: HIGH-RESOLUTION AND PROMPT- FAITHFUL TEXT-GUIDED IMAGE INPAINTING WITH DIFFUSION MODELS
Hayk Manukyan Andranik Sargsyan" Shant Navasardyan'
Barsegh Atanyan'
Zhangyang Wang"
Humphrey Shi
Picsart AI Research (PAIR)
2TT Austin
Georgia Tech
ABSTRACT
Recent progress in text-guided image inpainting, based the unprecedented success of text-to-image diffusion models, has led to exceptionally realistic and visually plausible results. However, there is still significant potential for im- provement in current text-to-image inpainting models, particularly in better align- ing the inpainted area with user prompts. Therefore, we introduce Painter,, a training-free approach that accurately follows prompts. To this end, we design the Prompt-Aware Introverted Attention (PAInIA) layer enhancing self- attention scores by prompt information resulting in better text aligned genera- tions. To further improve the prompt coherence we introduce the Reweighting Attention Score Guidance (RASG) mechanism scamlessly integrating a post-hoc sampling strategy into the general form of DDIM to prevent out-of-distribution latent shifts. Our experiments demonstrate that HDD-Pintter surpasses existing state-of-the-an approaches quantitatively and qualitatively across multiple met- ries and a user study. Code publicly available at: https: //gittub//ttt./. com/ AiccarrtA--Rsseacche------
INTRODUCTION
The recent wave of diffusion models (Ho al. 2020 Song al., 2021 has taken the world by storm, becoming an increasingly integral part of our everyday lives. After the unprecedented suc- cess of text-to-image models (Rombach al. 2022; Ramesh et al. 2022 Saharia et al. [002;] Wul
et al. 2022) diffusion- based image manipulations such as prompt-condiiioned editing (Hertz et al.
2022 Brooks al. (023). controllable generation (Zhang & Agrawala 2023 Mou et all 2023)
personalized and specialized image synthesis (Ruiz et al.) [023)] Gal al. 2022 Lu et al. 2023
became hot topies in computer vision leading to a huge amount of applications. Particularly, text- guided image completion or inpainting (Wang et al. 2023 Wu et al. 2022 Avrahami et al. 2022)
allows users to generate new content in user-specified regions of given images based on textual prompts, leading to use cases like retouching specific areas of an image, replacing or adding objects, and modifying subject attributes such as clothes, colors, or emotion.
Pretrained text-to- eext-to-imaee image generation models such Stable Diffusion (Rombach al. 2022 Imagen (Saharia et al. 2022), and Dall-E (Ramesh et al.) (022) can be adapted for image completion by blending diffused known regions with generated (denoised) unknown regions during the backward diffusion process. Although such approaches (Avrahami et al. 2022 2023) produce visually plausi- ble completions, they are not well harmonized and lack global scene understanding, especially when denoising in high diffusion timesteps.
2021 Podell et 2023 Saharia et al. 2022), modify pretrained text-to-image models to take additional context information and tine-tune specifically for text- rext-guided image completion. GLIDE (Nichol et al. 202T) and Stable Inpainting (Rombach et al. (022) concatenate the mask and the masked image as additional chan- nels to the input of the diffusion UNet, initializing the new convolutional weights with zeros, then fine tune the modified model using random masks together with the initial prompt.
To address this, existing methods (Rombach et al. 2022 Nichol
Indicates equal contribution.

---

Published as a conference paper at ICLR 2025
flowers
shirt
castle
ion
Figure I:
Inpainting results with our approach. Results have been upscaled to 2048px large side super-resolution technique. The method is able to faith- fully fill the masked region according to the prompt even if the combination of the prompt and the known region is highly unlikely. Zoom in to view details.
using our developed aupaining-sppcializne
However, SmartBrush (Xie et al. 2023) and Imagen Editor (Wang al.] 2023) mention the weak image-text alignment of such models, attributing it to the random masking strategies, and the mis- alignment the global prompts used during training with the local context the masked region. In this paper, we will address this issue as prompt neglect. To alleviate this problem, both papers intro- duce novel, object-aware masking strategies. Additionally SmartBrush proposes BLIP captioning approach, to ensure better alignment of the inpainting prompt with the masked region. Nonethe- less, we find that while this approach reduces prompt neglect, it also decreases the generation quality.
We notice that prompt neglect is commonly expressed in two ways: either the model fills in the masked region with background (background dominance, fig. []] columns 1, 3, 5), or the model completes nearby object partially occluded by the mask (nearby object dominance, fig. 5 columns 2, 4, 6). In both cases the issue seems to be caused by the model preferring the local context of the known region the textual information provided by the prompt.
To address the mentioned problems we introduce Prompt-Aware Introverted Attention (PAInIA), without any training or fine-tuning requirements. PAIntA enhances the self-attention scores accord- ing to the given textual condition aiming to decrease the impact of non prompt-relevant information from the image known region while increasing the contribution of the prompt- prompt-aligned aligned known pixels.
To improve the text-alignment of the generation results even further we apply a post-hoc guidance mechanism by leveraging the cross- cros--attettion scores. However the vanilla post-hoe guidance mech- anism used by seminal works such as (Dhariwal & Nichol 2021; Epstein et al. 2023) may lead generation quality degradation due to -uitti--iitiibuttin shifts caused by the additional gradient term in the backward diffusion equation. To this end we propose Reweighting Attention Score Guid- ance (RASG), a post-hoe mechanism seamlessly integrating the gradient component in the general form of DDIM process. This allows simultancously guide the sampling towards more prompt- aligned latents and keep them in their trained domain leading to visually plausible inpainting results.
To summarize, main contributions
follows:
We introduce the Prompt-Aware Introverted Attention (PAInA) layer to alleviate the prompt neglect issues of background and nearby object dominance in text-guided image inpainting.
To further improve the text-alignment of generation present the Reweighting Attention Score Guidance (RASG) strategy which enables to prevent out-of-distribin shifts while performing post-hoc guided sampling.
2

---

Published as a conference paper at ICLR 2025
Our pipeline for text- text-guided image completion is training-free and demonstrates signifi- cant advantage over current state-of-thh-ai approaches quantitatively and qualitatively.
RELATED WORK
2.1
IMAGE INPAINTING
Early deep learning approaches for image inpainting Yuu et al. 2018 Yi et al. 2020 Navasardyan & Ohanyan 2020) introduce mechanisms propagate deep features from known regions. Later (Zhao ct al. 202 Zheng et al. 2022; Xu et al. 20233 Sargsyan et al. 2023) utilize StyleGAN-v2-iik
(Karras et 2020 decoder and discriminative training for better image detail generation.
Image inpainting also benefited from diffusion models, particularly with the emergence of text- guided inpainting. Given pre-trained extttto-mmage diffusion model Avrahami al. (2022) 2023) replace the unmasked region of the latent by the noised version of the known region during sam- pling. Nichol et al.] (2021 | Wang et al. (202); Podell et al. (2023) Xie al. (023) fine-tune pre- trained text-to-image models for text-guided image inpainting by conditioning the denoising model on the inpainting mask and the known region, concatenating them with the input latents. /Zhang & Agrawala (2023) obtain inpainting model by attaching trainable modules to the UNet, while keeping the base model unchanged. We propose a training-fire -freengg-eeeeeeeo approach leveraging plug-and-play components PAIntA and RASG, improving text- exxt-proppt alignment.
2.2 INPAINTING-SPECIE ARCHITECTURAL BLOCKS
Early deep learning approaches were designing special layers for better/more efficient inpainting. Particularly, Liu et al. (2018); Yu et al. (001);; Navasardyan & Ohanyan (2020) introduce special convolutional layers dealing with the known region of the image to effectively extract the informa- tion useful for visually plausible image completion. Yi et al. (2020) introduce the contextual atten- tion layer reducing the unnecessarily heavy computations of all-to-all self-attention for high- -ualliulity inpainting. this work we propose Prompt-Awar Introverted Attention (PAIntA) layer, specifi- cally designed for rext-guided image inpainting. It aims to decrease (increase) the prompt-irrelevan (-relevant) information from the known region for better text aligned inpainting generation.
2.3 POST-HoC GUIDANCE IN DIFFUSION PROCESS
Post-hoe guidance methods are backward diffusion sampling techniques which guide the next step latent prediction towards specific objective function minimization. Particularly|Dae Dhariwal & Nichol (202I) introduced classifier- classifier-guidance guidance aiming to generate images of a specific class. Later CLIP- guidance was introduced by Nichol et al. (2021) leveraging CLIP (Radford et al. 2021 as open- vocabulary classification method. Chefer et al. (2023) guide image generation by maximizing the maximal cross-attention score relying aligned results. Epstein et al. (2023) utilizes the cross-attention scores for object position, size, shape, and appearance guidances. All the mentioned post-hoe guidance methods shift the latent generation process by a gradient term (see eq. ( sometimes leading to image quality degradations.
multi-iteratio optimization process resulting in
text
To this end we propose the Reweighting Attention Score Guidance (RASG) mechanism allowing to perform post-hoc guidance with any objective function while preserving the diffusion latent domain. Specifically for inpainting, to alleviate the issue of prompt neglect, benefit from a guid- ance objective function based on the open- -pec-vocabulary segmentation properties of cross-attentions.
3 METHOD
We first formulate the text-guided image completion problem followed by introduction to diffu- sion models, particularly Stable Diffusion and Stable Inpainting (Rombach et al.] 2002). We then discuss the overview of our method and its components. Afterwards present our Prompt-Aware Introverted Attention (PAIntA) block and Reweighting Attention Score Guidance (RASG) mecha- nism in detail.
3

---

Published as a conference paper at ICLR 2025
Let I E =x be RGB image, M {0.1} #xW one wants to inpaint with a textual prompt т. The goal of text- -uided image inpainting is to output an image Is R xx3 such that Ie contains the objects described by the prompt in the region M while outside M it coincides with 7, i.e. 1e o - M) = T @ (1 M).
[0,1) be binary mask indicating the region in I
3.1 STABLE DIFFUSION AND STABLE INPAINTING
Stable Diffusion (SD) is diffusion model that functions within the latent space of autoencoder D(E(-)) (VQ-GAN (Esser et al. 2021) or VQ-VAE (Van Den Oord et al. the encoder and D the corresponding decoder. Specifically, let / "x* be an image and To (T), consider the following forward diffusion process with hyperparameters (3: ( C [0, 1]:
2017)) where & denotes
q(1)(2(-1) = -N(x;; v
t 1,.,,
(1)
where ( is the conditional density of 24 given 2t-1, and {ær T is Markov chain. Here T is large enough to allow an assumption øt ~ (0, 1). Then SD learns a backward process (below similarly, {ax _ is a Markov chain)
po( (x 1:::::o00o (ar, 1), o,), t..,.,
and hyperparameters {o=)T=1, allowing the generation of signal xo from the standard Gaussian noise Zy. Here (I( t) is defined by the predicted noise c) modeled neural network (Ho
et al. 2020 po (2 t) vp. 22 - (x))) Then i = D(æa) is returned.
The following claim can be derived from the main DDIM principle, (Song et al. 2021). Theorem 1.
CLAIM After training the diffusion backward process (eg. 2 the following {ar)T=--- parameterized family of DDIM sampling processes can be applied to generate high-quality images:
xt-1 = Va--1 It VI ((cg(zz))
ar-1 1 11-ay
V
(z)) + oie,
Vat
where et ~ (0, az [I' B.), and < or S VT 04-- can be arbitrary parameters.
Usually (e.g. in SD or Stable Inpainting described below) o2 = is taken to get a deterministic process:
- (xcg(1))) ) + 111--a--- (1-aa-1ch(x)) t = T.....,
Ft
Tt-1 = 0n-1
(4)
/
For text-to-image synthesis, SD guides the processes with textual prompt Hence the function <xx)) = <g (Zz, r), modeled by a UNet-like (Ronneberger et al., 2015) architecture, is also condi- tioned on by its cross-attention layers. For simplicity sometimes skip 7 in writing (Ia, r).
As mentioned earlier. Stable DIffusion can be modified and fine-tuned for lext-guided image inpaint- ing. To do so Rombach et al. (2022) concatenate the features of the masked image I 10 (1-M) obtained by the encoder ε, and the (downscaled) binary mask M to the latents Hg and feed the re- ((IM),downl down( (M)) 7). where down is
sulting tensor to the UNet to get the estimated noise < E (IMM) the downscaling operation to match the shape of the latent Tt. Newly added convolutional filters are initialized with zeros while the rest of the UNet from a pretrained checkpoint of Stable Diffu- sion. Training is done by randomly masking images and optimizing the model to reconstruct them based on image captions from the LAION-5B (Schuhmann al.) (022)) dataset. The resulting model shows visually plausible image completion and we refer to it as Stable Inpainting.
3.2 HD-PAINTER: OVERVIEW
The overview of HD-Painter is presented in fig. 2 To complete the missing region M according to the prompt take pre-trained inpainting diffusion model, replace the self-attention layers by PAIntA layers, and perform a diffusion backward process with our RASG mechanism. After getting the estimated latent Tg it is decoded resulting in an inpainted image Ie D(x,) R #xx33
Additionally, by leveraging high-resolution diffusion models and time-iterative blending, we design simple yet effective pipeline for up 2048 x 2048px resolution inpainting (see Appendix B.
4

---

Published as a conference paper at ICLR 2025
2t-1
8(7
Concalio
RASO
D
"white bear"
Convolution
PAlnIA
Crose-Atin
FFN
Reglection Laper Weight Shawlog for Qo Kr
Transformer Block x2
Figure 2: In each diffusion step denoise the latent F, by conditioning the inpainting mask M and the masked downscaled image A = down((]) @ (1 - M) E # x3 (encoded with the VAE encoder E). To make better alignment with the given prompt our PAInIA block applied instead of self-attention layers. After predicting the denoised refred in each step i, we provide it to our RASG guidance mechanism to estimate the next latent 1t-1-
3.3
PROMPT-AWARE INTROVERTED ATTENTION (PAINTA)
thorough analysis of existing text-guided image inpainting methods shows that they often overlook user-provided prompts, relying instead on the surrounding visual context of the inpainting area. For example, in fig. 3 most existing methods fail to create a vase matching the prompt to generate the boat. We hypothesize that the visual context dominance over the prompt is attributed to the prompt- free, only-spatial nature of self-attention layers. To support this we visualize the self-attention scores in fig. 3 (a). The heatmaps illustrate how, on average, pixels in the masked region attend to other pixels in the image (see Appendix B for implementation details). For example in the forth row (the image with the vase) see that the masked region pixels have high attention scores not only with themselves but also with other background pixels. This results in generated pixels that closely resemble the background, undermining the intended generation (given by the prompt). This behavior highlights a limitation of the self-attention layer: it not only disregards the prompt context (as it does not incorporate the prompt as input) but also reinforces similarity between the generated region and the background. Therefore, to alleviate the issue, we introduce plug-in replacement for self-attention, Prompt-Aware Introverted Attention (PAIntA, see fig. (a)) which adjusts the attention scores between mask pixels i and non non-mask pixels j according to the alignment of the pixels with the prompt (higher when more aligned, lower otherwise). fig. (b) shows the attention maps of PAIntA and here we clearly see that the mask region mostly attends to itself by so allowing the prompt influence the generation process later. This leads to outputs where the desired objects are accurately generated. Below we discuss PAIntA in detail.
Let X E (hxw)xd be the input tensor of PAIntA. Similar to self-attention, PAIntA first applies projection layers to get the queries, keys, and values we denote by Q.. K,, V, R(bxw)xd respec- tively, and the similarity matrix Aweyf vi
Q.KT Ig Then, as discussed above, we mitigate
the too strong influence the known region over the unknown by adjusting the corresponding atten- tion scores. To do so, for each unknown pixel i and known region pixel j we multiply the attention scores (Asntf) ji.j by a factor Cj E [0, 1). represents the amount of how much we want to suppress the impact of the known region pixel j on the completion of the missing region. As want the generation in the missing region to be more aligned with the provided textual prompt, we set oj based on the similarity between and the prompt in the embedding space. In other words, we set Cj low for such pixels j from the known region that are not semantically close to the given prompt, and we set ey high otherwise. Specifically, leveraging the prompt r, PAIntA defines new similarity matrix:
[cj (Aoetris M, = 1 and M, = 0, (Aeet))is
Asety E hhuwehw
(Asetf)iz
(5)
otherwise,
where M is the resized and flattened input mask and ej is defined as follows.
We define ter using the cross-attention spatio-textull similarity matrix Stross PaftMar(Q..K3 /Vd), where Qa E RR(xx)xxd K. TRixd are query and key tensors of cor-
5

---

Published as a conference paper at ICLR 2025
(a)
(b)
without PAIntA (vanilla DreumShuper)
with PAInIA
input
27
W
map
II
W
SAp
result
H
W
S. map
IT
W
map
result
antique greek vasc
white robot sallor hat
Figure 3: Comparison of self-attention similarity maps averaged across masked pixels for genera- tions without/with PAIntA's scaling the original self-attention scores. Images are generated from the same seed.
responding cross-attention layers, and is the number of tokens of the prompt т. Specifically, we consider CLIP text embeddings of the prompt 7 and separate the ones which correspond to the words of T and End of Text (EOT) token (in essence we just disregard the SOT token and the null-token embeddings), and denote the set chosen indices by ind(r) C (1,2,..... (in contrast with SOT) it contains information about the prompt according to the architecture of CLIP text encoder. For each jth pixel we define similarity with the prompt T by summing up it's similarity scores with the embeddings indexed from ind(T), i.c. Cj EkEind(r)) (Scross)jke Also,
We include EOT since
we found beneficial to normalize the scores Cj = clip (e)-medionn median (cri (ce: k= k=l...kko) 0, 1). where clip mea k= s=,,..,,ww)
the clipping operation between [0, 1].
Note that in vanilla SD cross-attention layers come after self-attention layers, hence in PAIntA to get query and key tensors Qc: Ke we borrow the projection layer weights from the next cross-attention module (see fig. 2}. Finally we get the output of the PAIntA layer with the residual connection with the input: Out X + SoftMaz(A
V.
3.4 REWEIGHTING ATTENTION SCORE GUIDANCE (RASG)
The experiments show that while PAIntA improves prompt-alignment generation by manipulat- ing the self-attention layers, the issue of prompt-alignment is not completely resolved. Therefore we additionally leverage the concept of post-hoc sampling guidance. The idea, introduced in (Dhari- wal & Nichol 202T) classifier-guidant guidance and generalized further (Rombach al. (2022). Chefer et al. (2023). Epstein et al. (2023)), is to guide the sampling (denoising) process - to the However, the vanilla post-hoc guidance, as noticed (Chefer) et al. 2023), may shift the domain of diffusion latents -x-1 resulting in image quality degradations. To this end we introduce the Reweighting Atlention Score Guidance (RASG) strategy which benefits from the general DDIM backward process, eq. B.. and introduces gradient reweighting mechanism resulting in latent domain preservation. Below we first discuss the vanilla post-hoc guidance with objective function S(x) which define later to be specifically in charge of prompt-alignment. Then we discuss RASG and how it approaches to the out-of-dommin latent shift problem of the vanilla guidance. And finally we present our choice for the post-hoc objective function S(x).
direction of more prompt-alignment
Let 5(x) be an objective function the post-hoc guidance mechanism should be applied with. Then, according (Dhariwal & Nichol 2021), the update rule for e (2r) will be
(x (x))) vI 04 VV, S(x1)
for brevity write (xx) instead of ((r,, ((T.) down(M)),

---

Published as a conference paper at ICLR 2025
unet Veul levin br
... ((te))
Away
2.47
vi
Ause
.. Slaa).
Autt
[idd:]
SooftAdea
(b)
Figure 4: (a) PAIntA block takes an input tensor X E IR hxxwxx and the CLIP embeddings of T. After computing the self- and cross-attention scores Asely and Across, we update the former (eq. 5 by scaling with the normalized values (c, )=r obtained from Scross = SoftMaaa (ar (crross) (.rrros)......s.. Finally the the updated attention scores Aself are used for the convex combination of the values V, to get the residual of PAIntA's output. (b) RASG mechanism takes the predicted scaled denoised latent var-1 pf pred /ar-1 (z: vI axca(1z)) and guides the I;-1 estimation process towards .or
minimization of (xx,) defined by eq. (T). Gradient reweighting makes the gradient term close to being sampled from N (0, 1) (green area) by ensuring the domain preservation (blue area).
where is a hyperparameter controlling the amount of the guidance. This, according to the (de- terministic) DDIM process eq. (4, will result in (by substituting eg (zt) with &g (zx)) the following update rule for xz-1:
VI (r))(
It-1 vai-1
VI an-1cb(())) V. S(xr),
anyan- V1- a1-1
& = v1 a4
Notice that in eq. 77 get the additional term - 2, S(x,) not present in the original sampling process given by eq. This term may shift the original distribution 14-1-
To this end introduce the Reweighting Attention Score Guidance (RASG) strategy which benefits from the general DDIM backward process (eq. (3) and introduces gradient reweighting mech- anism resulting in latent domain preservation. Specifically, according to Claim D _T-1 obtained either by eq. (4) or by eq. (3) remains in the required domain (see fig. 44). Hence in eq. (3) by re- placing the stochastic component e by the rescaled version of the gradient V.. S(xz) (to make closer to a sampling from (0, 1)), we can keep x4-1 in the required domain and at the same time guide its sampling towards minimization of S(x,). Rescaling of the gradient S(x,) is done by dividing it on its standard deviation (we do not change the mean to keep the direction of the S(x;) minimization, for more discussion see Appendix C). Thus, RASG sampling is done by the formula
St
VI -Tonee (x:))(z))
V V1- 04-1 - ofc)(x)) + +o, std( v. S(x)))
S(x.)
24-1
04
(8)
0
Now let us define the function S(x) (for more discussion on its choice see Appendix C). We want the objective S(x) guide the sampling to more prompt-alignment, at the same time keeping the generated object in the mask-indicated region. On the other hand we know that (Chefer et al. 2023) cross-attention maps contain similarities between the prompt and the pixels of latent xt. therefore the cross-attention maps can serve segmentation maps for the objects described via the given prompt. Thus we define S(x) as segmentation loss function between the cross-attention maps and the given inpainting mask. Below discuss the construction of S(x) in detail.
First we consider all cross-attention maps Across with the output resolution of H x W A1 Pross:.... Aprose W//2222 where is the number of such cross- layers, and
Am

---

Published as a conference paper at ICLR 2025
I is the number of token embeddings. attention maps and reshape to
!! W. Then for each & E ind(r) fl...., average the
32
32'
A Across (2t)
m Σ
Aornalli k] E
(9)
Using post-hoc guidance with S(z)) aim to maximize the attention scores in the unknown region determined by the binary mask M E {0, 1] $ $ hence we take the average negative binary cross entropy between A (xx) and M (M is downscaled with NN interpolation, here is sigmoid):
H w
S(z1) = Σ
[M, log o (A (x+),) + (1 - -M)) log(1 -
(10)
kgind(r)
4 EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
We apply HD-Painter 3 different Stable Diffusion models: Stable Diffusion 1.5, Stable Diffusion 2.0 and Dreamshaper-8 (Lykon 2023). PAIntA is used to replace the self attention layers on the H/32 x W/32 and H/ 16 x W 16 resolutions for the first half of generation steps. For RASG select only cross- -otssnattnnioon similarity matrices of the H //2 x W/32 resolution since utilizing higher resolutions did not offer significant improvements. For hyperparameters {o,)1=1 chose
n 01-1)/(1 ax)) V 01/0n-1, = 0.15.
(11)
In PAIntA's implementation, we reuse calculated artss-attention similarity maps, which results in very small performance impact. With PAIntA the model is about just 10 % slower, making ~ 3.3 seconds from ~ seconds of the baseline.
For RASG, naturally, the backward pass of the model increases the runtime about twice. However, optimizations, like using RASG only for subset of steps, etc., can potentially greatly decrease the runtime while keeping the generation distribution. We keep such investigations for future research.
4.2 EXPERIMENTAL SETUP
Here we compare with existing state-of-the-art methods such as GLIDE Nichol et al. 202 ),, Sta-
ble 2.0 Inpainting (Rombach et al. 2022). DreamShaper Inpainting (I (Lykon] 2023). Blended Latent
Diffusion (BLD) (Avrahami et al. 2023). ControlNet-Inpaintin
(Zhang & Agrawala 2023) (with
DreamShaper base), SDXL-Inpainting and SmartBrush (Xic et al. 2023). As authors of the Smart- Brush paper don't provide code and model, we reproduce it according to paper and refer to it as SmartBrush reprod.. We present the results of SmartBrush reprod. based on DreamShaper text-to- image model, since it had the best performance. We evaluate the methods on a random sample of 10000 (image, mask, prompt) triplets from the validation of MSCOCO 2017 (Lin et al.. 2014), where the prompt is chosen as the label of the selected instance mask. We noticed that when a precise mask of a recognizable shape is given Stable Inpainting, it tends ignore the prompt and inpaint based on the shape. To prevent this, we use the convex hulls of the object segmentation masks and compute the metries accordingly.
We evaluate the CLIP score on cropped region of the image using the bounding box of the input mask. As CLIP score can still assign high scores to adversarial examples, we additionally compute the generation class accuracy. So, we utilize pre-trained instance detection model for MSCOCO: MMDetection (Chen et al., 2019). We run on the cropped area of the generated image, and, as there might be more than one objects included in the crop, treat the example as positive if the prompt label is in the detected object list.
To measure the visual fidelity of the results employ the LAION aesthetic score The aesthetic score is computed by an MLP trained on 5000 image-rating pairs from the Simulacra Aesthetic Captions dataset (Pressman et al. 2022), and can be used assign a value from the (0, 10) range to images based on their aesthetic appeal.
com
8

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparison. 95% confidence interval of 5 runs with different sceds.
Model Name
CLIP score
Accuracy, % 1
Aesthetic score
GLIDE
25.09 0.01
43.08 0.30
4.476 = 0.002
BLD
25.64 0.05
55.64
4.822 = 0.006
SDXL. Inpainting
24.80
5.91
4.682 = 0.024
PreamShaper-ControNNee Inp.
25.73
58.74
4.946 = 0.005
SmartBrush reprod.
0.03
66.88
4.856 0.004
Stable 1.5 Inpainting
2.02
0.425
4.881 = 0.006
Stable 2.0 Inpainting
25.07 0.03
0.57
4.885 = 0.006
DreamShaper Inpainting
25.61 0.00
58.93
4.965 = 0.004
Stable 1.5 HID-Painter
2
0.557
4.864 0.006
Stable HD-Painter
26.48
5..74
4.846 = 0.011
Dreamshaper HD-Painter
2 0.03
68.05 0.48
4,980 = 0.003
4.3 QUANTITATIVE AND QUALITATIVE ANALYSIS
Table I shows that HD-Painter increases the prompt alignment of the corresponding baseline models. can be noticed that while SmartBrush trained over DreamShaper Inpainting improves the accuracy over the baseline, the CLIP score improvement is marginal and the overall quality is significantly dropped according to aesthetic score. On the other hand, our method significantly improves the prompt-alignment as measured by both CLIP score and accuracy while also maintaining the quality. Additionally, we performed user study (see Appendix Appendix D).
The examples in fig. 1...5des demonstrate qualitative comparison between our method and the other state- of-the-art approaches. In many cases the baseline DreamShaper Inp. generates background (fig. 5 columns 1, 3, 5) reconstructs the missing regions continuation of the known region objects disregarding the prompt (fig. [ columns 6, 7). while our method, thanks to the combination of PAIntA and RASG, successfully generates the target objects. Notice that even though DreamShaper- ControlNet-Inpaintigg and SmartBrush reprod. may also generate the required object, the quality of the generation is poor compared to ours.
Table 2: Ablation for PAIntA and RASG on the Dreamshaper 8 base. 95% confidence interval of 5 runs with different seeds.
Model Name
CLIP score
Accuracy
Aesthetic score
base (DreamShaper Inp.)
25.61 0.02
$8,93
4.965 0.004
only PAIntA
26.07 + 0.03
63.95 + 0.50
4.985 # 0.003
only RASG
25.94 + 0.02
63.75 + 0.48
4.965 # 0.003
RASG & PAIntA
26.32 2 0.03
68.05 + 0.48
4,980 0.003
4.4 ABLATION STUDY
In Mable[[we show that PAIntA and RASG separately provide substantial improvements to the model quantitatively. We also provide more discussion on each of them in our supplementary material, including thorough analyses on their impact, demonstrated by visuals. For qualitative ablation study see figs. and 7 in Appendices andd C
CONCLUSION
In this paper, we introduced a training-free method to frimpt-faithful text-guided image inpainting, addressing the prevalent challenges of prompt neglect: background and nearby object dominance. Our contributions, the Prompt-Aware Introverted Attention (PAIntA) layer and the Reweighting Attention Score Guidance (RASG) mechanism, effectively mitigate the mentioned issues leading our method to surpass the existing ofte-of-the-- art approaches qualitatively and quantitatively.
REPRODUCIBILITY STATEMENT
We have included the codebase, evaluation scripts, and comprehensive instructions for reproducing experiments in the supplementary material. This ensures the reproducibility of our results and facilitates independent validation. Furthermore, we report confidence intervals for all computed metrics, demonstrating the robustness of our findings to random factors.

---

Published as a conference paper at ICLR 2025
duck
antique greek
swimming
sunglasses
flowers
wooden boat
flames
shirt
in water
Masked
BLD
Stable 1.5 Inpainting
Stable Inpainting
SDXL Inpainting
Dreamshaper- ControlNet Inpainting
SmartBrush reprod.
DreamShaper Inpainting
Stable 1.5 +HD-Painter
Stable
*HD-Painter
DreamShaper *HD-Painter
Figure 5: Comparison with state-of- fftte-of-the-att text- text-guided inpainting methods. Zoom in for details. For more comparison see Appendix A
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18208-18218, 2022.
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM Transactions on Graphics (TOG), 42(4):1-11, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2piix Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-10, 2023.
K Chen, J Wang, J Pang, Y Cao, Y Xiong, Li, S Sun, W Feng, Z Liu, Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arxiv 2019. arXiv preprint 1XX6::900.077 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Dave Epstein, Allan Jabri, Ben Poole, Alexei A Efros, and Aleksander Holynski. Diffusion self- guidance for controllable image generation. arXiv preprint arXiv:2306.0095. 2023.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pp. 12873-12883, 2021.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. coompt-to-prompe image editing with cross attention control. arXiv preprint arXiv:2208.01622 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 01166800-85111 2020.
Itseez. Open source computer vision library. htttpp:://githuu.com//
2015.
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 81 10 81 19, 2020.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Guilin Liu, Fitsum Reda, Kevin Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European conference on computer vision (ECCV), pp. 85-100, 2018
Haoming Lu, Hazarapet Tunanyan, Kai Wang, Shant Navasardyan, Zhangyang Wang, and Humphrey Shi. Specialist diffusion: Plug-and-play sample-efficient fine-tuning of text-to-image diffusion models to learn any unseen style. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14267-14276, 2023.
Lykon.
Dreamshaper-8 inpainting.
https://civitale http://aaa..ccooooo////////////
mode MondelVerss on Id= 2023.

---

Published as a conference paper at ICLR 2025
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang. Zhongang Qi, Ying Shan, and Xiaohu Qie 12i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453, 2023.
Shant Navasardyan and Marianna Ohanyan. Image inpainting with onion convolutions. In proceed- ings of the asian conference on computer vision, 2020.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 2021.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01955, 2023.
John David Pressman, Katherine Crowson, and Simulacra Captions Contributors. Simulacra aes- thetic captions. Technical Report Version 1.0, Stability AI, 2022. url https//github..oo//D>
P/simulacreaessteticccaaptoi
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net:
Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Cosputer-Assisted Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part 18, pp. 234-241. Springer, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour; Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479-36494, 2022.
Andranik Sargsyan, Shant Navasardyan, Xingqian Xu, and Humphrey Shi. Mi-gan: A simple base- line for image inpainting on mobile devices. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7335-7345 October 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265. PMLR, 2015.
Jamming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2025
Su Wang. Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tusee Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 18359-18369, 2023.
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In European conference on computer vision, pp. 720-736. Springer, 2022.
Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22428 22437,, 2023.
Xingqian Xu, Shant Navasardyan, Vahram Tadevosyan, Andranik Sargsyan, Yadong Mu, and Humphrey Shi. Image completion with heterogeneously filtered spectral hints. In Proceedings of the IEEE/CVF Winter Conference Applications Computer Vision, pp. 4591-4601, 2023.
Zili Yi, Qiang Tang, Shekoofeh Azizi, Daesik Jang, and Zhan Xu. Contextual residual aggrega- tion for ultra high-resolution image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7508-7517, 2020.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5505-5514, 2018.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. Proceedings of the IEEE/CVF international conference on computer vision, pp. 4471 14800, 2019.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint wrxXi::3002.0543333 2023.
Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. arXiv preprint arXiv:2103. 10428, 2021.
Haitian Zheng, Zhe Lin, Jingwan Lu, Scott Cohen, Eli Shechtman, Comnelly Barnes, Jianming Zhang, Ning Xu, Sohrab Amirghodsi, and Jiebo Luo. Cm-gan: Image inpainting with cascaded modulation gan and object-aware training. arXiv preprint arXiv:2203. 1947, 2022.
13

---

Published as a conference paper at ICLR 2025
A EXTENDED QUALITATIVE COMPARISON
In fig. 18 show more visual comparison with the other ofte-of-thhearr methods. Figure [9 in- cludes more comparison on the validation set of MSCOCO 2017 (Lin et al. 2014). The results show the advantage of our method over the baselines.
B DISCUSSION ON PAINTA
In this section we discuss the effectiveness of the proposed PAIntA module a plug-in replacement for self-attention (SA) layers. To that end, first we visualize SA similarity maps averaged across masked locations from resolutions H/16 x W/16 and H/32 W/32 where PAIntA is applied (see fig. B) Then, we see that PAIntA successfully scales down the similarities of masked locations with prompt-unrelated locations from the known region, and, as result, prompt-specified object generated inside the mask.
For a given resolution (H/16 x W/16 or H/32 x W/32), in order to visualize the average SA similarity map across masked pixels, first we resize the input mask to match the dimensions of the corresponding resolution (we use nearest interpolation in resize operation). Then, for each SA layer in the given resolution, we form 2D similarity map by reshaping and averaging the simi- larity matrix rows corresponding to the masked region. Further, we average obtained 2D similarity maps across all SA layers (of the given resolution) and diffusion timesteps. More specifically, if A self...... A E hwu (h x w is either H/16 x W/16 or H H/32 x W W/32) are the self-attention matrices of Stable Inpainting layers of the given resolution, and, respectively, are being updated by PAIntA to the matrices À self (see eq. ( then we consider the following similarity maps:
A
Σ
Σ
(Abar)ss ERAW
M
i,M,=1
(12)
Á
Σ 1,M,=11=1
Σ
(Abar)ss R
M
and reshape them to 2D matrices of size h x w. So, Ai and Å. show the average amount in which masked pixels attend to to other locations in the cases the vanilla self-attention and PAIntA respectively. Finally, in order to visualize the similarity maps, to match it with the image dimensions and plot the similarity heatmap using JET colormap from OpenCV [Iteez 2015).
bicubic resize operation
Next, we compare the generation results and corresponding similarity maps obtained from above procedure when PAIntA's SA scaling is (the case of Á) or is not (the case of A) used. Because PAIntA's scaling only applied on H H/32 W/32 and H/16 W/16 resolutions, nearby object dominance
are interested
in those similarity maps. Rows 1-3 in fig. demonstrate visualizations issue (when known objects are continued to the inpainted region while ignoring the prompt) of the vanilla diffusion inpainting, while rows 4-6 demonstrate those of with background dominance issue (when nothing is generated, just the background is coherently filled in).
For example, on row (fig. 3 in case of Stable Inpainting without PAInIA generation, the average similarity of the masked region is dominated by the known regions of the car on both 16 and 32 resolutions. Whereas, region with the car is effectively reduced, and the masked region is generated accordance to the input prompt.
result of PAIntA scaling application, the average similarity of the masked
Row 4 (fig 3 demonstrates an example where the result without PAIntA continues the background based on visual context instead of following the user prompt. In this case, visualization shows that usage of PAIntA successfully reduces the similarity of the masked region with the unrelated background. As a result, by reducing the similarity of masked region with the unrelated known regions PAIntA enables prompt-faithif generation. You can find additional examples of PA effect on the final generation in fig.
14

---

Published as a conference paper at ICLR 2025
white duck
swimming water
moon
wine botttle
clock
boots
imput
DreamShaper Inpainting
with PAIntA
Figure 6: Visual ablation of PAIntA. Generated images use the same seed. row only PAIntA is used.
C DISCUSSION ON RASG
antique greek vase
clock
wooden boat
white fox
space helmet
input
DreamShaper Inpainting
with RASG
Figure 7: Visual ablation of RASG. Generated images the same seed. In row 3 only RASG is used.
In this section discuss the choice of RASG objective guidance function S(z), then demonstrate the effect of RASG and motivate the part gradient reweighting by its standard deviation. Finally, present additional examples of RASG's effect the final generation in fig.
15

---

Published as a conference paper at ICLR 2025
C.I THE OBJECTIVE FUNCTION ((x)
As we already mentioned in the main paper, Stable Inpainting may fail to generate certain objects in the prompt, completely neglecting them in the process. We categorized these cases into two types, namely background and nearby object dominance issues. Chefeer et al. (2023) also mentions these issues but for text-to-image generation task, and refers them as catastrophic neglect problem. To alleviate this problem Chefer al. (202)) propose mechanism called generative semantic nursing, allowing the users to "boost" certain tokens the prompt, ensuring their generation. In essence the mechanism is post-hoc guidance with chosen objective function maximizing the maximal cross- attention score of the image with the token which should be "boosted". This approach can be easily adapted the inpainting task by just restricting the maximum to be taken in an unknown region so that the object is generated there, and averaging the objectives across all tokens, since we don't 't have specific tokens to "boost", but rather care about all of them. In other words, by our notations from the main paper, the following guidance objective funciton can be used:
S(xe) ind((T]
Σ kEind(T)
max (A (1z)i).
(13)
d: M;=1
However we noticed that with this approach the shapes/sizes of generated objects might not be sufficiently aligned with the shape/size of the input mask, which is often desirable for text-guided inpainting (see fig. 99. Therefore, we utilize the segmentation property of cross-attention similarity maps, by so using Binary Cross Entropy the energy function for guidance (see eq. (IO) in the main paper). As can be noticed from fig. 9 the results with the binary cross-entropy better fit the shape of the inpaining mask.
C.2 EFFECT OF RASG STRATEGY
Although the objective function S(z) defined by eq. (10) (main paper) results in better mask shape/size aligned inpainting, the vanilla post-hoc guidance may lead the latents to become out of their trained domain also noted by [Chteer et al. (023): "many updates of 2z may lead to the latent becoming
resulting in incoherent images". Due to this the post-hoc guidance mechanism (semantic nursing) by Chefer et al. (2023) done using multiple iterations of very small, iterative perturbations of Tt, which makes the process considerably slow. In addition, the generation can still fail if the iterative process exceeds the maximum iteration limit without reaching the necessary thresholds.
Thanks to RASG's seamless integration of the v(,S(x)) gradient component into the general form of DDIM diffusion sampling, our RASG mechanism keeps the modified latents I, within the ex- pected distribution, while introducing large enough perturbations to Ha with only one iteration of guidance per time-step. This allows to generate the objects described in the prompts coherently with the known region without extra-cost of time.
Figure 8 demonstrates the advantage of RASG's strategy over the vanilla guidance mechanism. Indeed, the vanilla post-hoc guidance there is a hyperparameter s controlling the amount of guidance. When is too small (e.g. close to 0 or for some cases mechanism does not show much effect due to too small guidance from 8 Szz)). Then with in- creasing the hyperparameter (s 1000, 10000) one can notice more and more text/shape alignment with prompt/inpainting mask, however the generated results are unnatural and incoherent with the known region. This is made particularly challenging by the fact, that different images, even dif- ferent starting seeds with the same input image might require different values of the perturbation strength to achieve the best result. In contrast, RASG approach is hyperparameter-free allowing both: prompt/mask-alige alignnd and naturally looking results.
100) the vanilla guidance
C.3 RESCALING WITH STANDARD DEVIATION
The core idea of RASG is to automatically scale perturbation using certain heuristics, such that the guidance process has consistent effect the output, without harming the quality of the image. Our main heuristic relies on the fact that [Sog et al. (202I) have defined parametric family of stochastic denoising processes, which can all be trained using the same training objective as DDPM
16

---

Published as a conference paper at ICLR 2025
Dreamshaper Inpainting (w/o Guidance)
Guidance: s 100
Guidance: $ 1000
Guidanee:
Input
RASG
Figure 8: Comparison of RASG strategy with default Stable Inpainting and vanilla guidance mech- anism with different guidance scales. In contrast vanilla guidance, where the generation highly depends on the guidance scale, RASG consistently produces naturally looking and prompt-aligned results.
(Ho et al. 2020). Recall the general form of parametric family of DDIM sampling processes:
It - (cc
-11111--1 a1-1 offe((xr) + ores,
Tt-1 Va
(14)
Vor.
where e, AN 1). Particularly e, can be taken be collinear with the gradient S(, which will result in Øt-1 distribution preservation by at the same time guiding the generation process towards minimization of 5(xx).
Therefore we propose to scale the gradient V.,, S(zz) with value À and use instead of in the general form of DDIM. To determine we analyse the distribution of V.,S(r)) and found out that the values of the gradients have distribution very close gaussian distribution, with 0 mean and some arbitary a, which changes over time-step/image (fig. TÖp. Therefore, computing the standard deviation of the values of V a, S(x,). ). and normalizing it by
sdd( results in the standard So the final form of RASG guidance strategy is
normal distribution (see fig.
2t-1 Va--1
Ft
anch ( (21)
a: - ofc((1)) + 0, std V 2, Szz))
V S(x))
(15)
17

---

Published as a conference paper at ICLR 2025
dog
elephant
zebra
input
Attend&Exxins Objective
Binary Cross Entropy
Figure 9: Comparison of the Binary Cross Entropy engery function to modifed version of Attend & Excite. Images generated from the same seed.
400
Figure 10: Histogram V2,S((z)) values (i.e. before gradient standardization)
1100
200
S(m.) Figure 11: Histogram of std 2v ((m))) values (i.e. after gradient standardization)
D
USER STUDY
We perform user study for qualitative comparison with the competitor state-of-the art methods. The 12 participants were shown 20 (image, mask, prompr) triplets and the inpainting results of all methods random order. For each sample image we asked to select the best results based on (i) prompt alignment and (ii) overall quality, allowing the choice of no methods when all methods were bad, or multiple methods when the quality was similar. We calculate the total votes for all methods for each question. The results are presented in fig. demonsttrai clear advantage of our method in both aspects over all competitor methods.
E INPAINTING-SPECII..IE CONDITIONAL
Here we discuss our method for high-resolution inpainting utilizing pre-trained diffusion-based super-resolution model. We leverage the fine- fine-grained information from the known region to upscale the inpainted region (see fig. [3 In our experiments we utilized Stable Diffusion x4 Upscaler (Rombach et al. 2022).
Assume that E 1f XfXWx3 x3 is the hegh-resolution image we want to inpaint, M E Hxx is the inpainting mask, and & is the encoder of VQ-GAN (Esser et al. 202T). We start by downscaling
18

---

Published as a conference paper at ICLR 2025
prompt aligament
overall quality
153
150
146
146
Total Votes
110
120
115
100
.59
50
34
25
16 GLiDE
DreamShuper
DreamShuper Inpainting Inpainting IID-Painter
SmarlBrush reprod.
BLD
DreamShupee ControlNet Inpainting
Stable 2.0 Inpainting
SDXL- Inpainting
Figure 12: Total votes of each method based user study for prompt alignment and overall quality. Our method HD-Painter has a clear advantage over all competitors.
Original image: R Retrav
X
x
8(r*)
X2
Difusion Based Super Resalution
Bending
D
Out
HD Painter
M
R
Mis
1'
Widss hoar
Figure 13: For aapainting-speciiin super resolution condition the high nigh-resolution latent X, de- noising process by the lower resolution inpainted result fo followed by blending Xpred Moow + E(T) @ (1 - Moow). Finally we get Ie by Poisson blending the decoded output with the original image
Input
Bicubic
Stable Super-Resolution
Ours
sailor bench
Figure 14: Comparison of our inpainting-speciille super-resolution approach with vanilla upscal- ing methods for inpainting. Best viewed when zoomed in.
I and M to smaller resolution image How R x x3 and mask Maw # We then
use HD-Painter obtain low-resolution inpainted image - To perform conditional super-resolution, we consider X, E(T) and take standard Gaussian noise Xy
R # W. x1 Then apply backward diffusion process (eq. (4 X by using the upscale-
19

---

Published as a conference paper at ICLR 2025
specialized SD model and conditioning! it on Hoo After each diffusion step blend the estimated denoised latent X (X; vI (xi)) with Xo by using Moos
X +- Miow @ Xpred + (1 Miow) @ Xus
(16)
and use the new X Xgred to determine the latent 1 (by eq. 44) After the last diffusion step xpred is decoded and blended (Poisson blending) with the original image I.
It's worth noting that our blending approach inspired by seminal works (Sohl- et al. 2013 Avrahami et al.) 2022) blending X, with the noisy latents of the forward diffusion. However, in contrast to those works, blend high-frequue frggunrequeecess from Xo with the denoised prediction X.pred allowing noise-free image details propagate from the known region to the missing one during all diffusion steps.
In fig. 14 we compare our inpainting-specialized super-resolution method with vanilla approaches of Bicubic or Stable Super- -eseu--esouuiiobbbae upscaling of the inpainting results followed by Poisson blending in the unknown region. We can clearly see that our method, leveraging the known region fine-grained information,
seamlessly fill in with high quality.
In figs. and we show more visual comparisons between our method and the approach of Stable Super-Resolution.
F LARGE QUANTITY OF OBJECTS
In this section we examine the challenging case, when multiple instances of the target object are already present in the known region of the input image. Many inpainting methods often do not generate target object in this case when it is already present in the image. Usually this means that the input prompt is considered in the global scope of the image and not just the masked area. Increasing the number the existing objects makes this problem particularly challenging.
fig. 13 shows the effect of applying HD-Painter the Dreamshaper-8 base. As evident from the example, HD-Painter enforces that the requested object is generated inside the given mask, thus ensuring that the object is properly generated.
G MULTI-OBIECT MASKS
In this section we evaluate the ability of our method to handle masks consisting of multiple parts. This can be useful when trying to generate multiple objects at the same time. The results presented in fig. show that our method is able to generate objects in the case of multiple masks. Note that this is more relevant when all objects are of the same type, generating objects of multiple types also requires specifying which type of object should be generated in each region. However, the task we investigate in this paper is text-guided image inpainting, which assumes that the only inputs are the mask and the textual prompt.
H LIMITATIONS
Although our method improves the prompt-alignment of existing text-guided inpainting approaches, still has a dependency on the backbone model, hence inherits some quality limitations. Particularly it may generate extra limbs (the elephant in fig. I7 has 5 legs) or illogical appearances (the sheep appears have two bodies in fig. after the inpainting).
I POTENTIAL NEGATIVE IMPACTS
Our research strives to enhance the accuracy object generation within the scope of text- text-guided image inpainting. However, it is crucial to acknowledge the potential negative impacts. The technol- ogy could be exploited to create deceptive imagery or disseminate misinformation, raising ethical concerns. While our method is training-free and does not introduce new biases, is imperative to
20

---

Published as a conference paper at ICLR 2025
Masked
DreamShapar Inpainting
DreamShaper *HD-Painter
DreamShaper Inpeinting
DreamShapar IHD-Painler
sir belloon
Nervon
brush
dise
800%
bel
Figure 15: HDD-Panteer makes sure the target object generated inside the masked area, even when multiple copies of the object exist outside.
Christmas Toys
Apples
Oranges
Potted plants
Figure 16: HD-Painter has no trouble generating multiple objects in single mask
21

---

Published as a conference paper at ICLR 2025
epephant
sheep
Figure 17: Failure examples produced by our approach.
consider the potential propagation of biases from the base models we build upon. These biases could lead to the generation of content that inadvertently reflects societal historical prejudices.
To counter these issues, it is essential for the broader research community to establish ethical stan- dards and develop robust methods to detect Al-generated content. Furthermore, efforts should be made to diversify training datasets to reduce inherent biases. While these challenges are significant, the positive implications of our work in areas such creative arts, design and content creation, when used responsibly, have the potential to surpass the negative repercussions.
J MORE EXAMPLES OF OUR METHOD
We present more results of our method both for low-resolution (512 for the long side) images (fig. 22). well high-resoltuion (2048 for the long side) (figs. 23 to 25).
22

---

Published as a conference paper at ICLR 2025
moon
wine bottle
boots
clock
hamster
big horns
Input
BLD
Stable 1.5 Inpainting
Stable 2.0 Inpainting
SDXLL
Inpainting
DreamShaper- ControlNet Inpainting
SmartBrush reprod.
DreamShaper Inpainting
Stable 1.5 HDD-Panteer
Stuble 2.0 HD--ainter
DreamShaper HD-Painter
Figure 18: More qualitative comparison results. Zoom in view the details.
23

---

Published as a conference paper at ICLR 2025
truck
bear
remote
cup
chair
laptop
Input
BLD
Stable 2.0
Inpainting
Stable 1.5 Inpainting
SDXL
Inpainting
Dhaper ControlNet Inpainting
SmartBrush reprod.
DreamShaper Inpainting
Stable + HID-Painter
Stable 2.0
HDD-Painter
DreamShaper HD--Pinter
Figure 19: Qualitative comparison results MSCOCO 2017.
24

---

Published as a conference paper at ICLR 2025
Input
Stable Super-Resolution
Ours
yellow headphones
lake
teddy bear
Figure 20: Comparison between vanilla 2.0 upscale and our approach. In all examples the large side is 2048px. The cropped region is 256x256px. Best viewed when zoomed in.
25

---

Published as a conference paper at ICLR 2025
Input
Stable Super-Resolution
Ours
lion
ice cream
leather couch
Figure 21: Comparison between vanilla SD 2.0 upscale and our approach. In all examples the large side 2048px. The cropped region is 256x256px. Best viewed when zoomed in.
26

---

Published as a conference paper at ICLR 2025
Tricion
space helmet
antique greek vase
Japtop
hoodie
leather couch
fedora
bowtie
Figure 22: More results of our method.
27

---

Published as a conference paper at ICLR 2025
boots
parrot
Figure 23: More high-resolution results of our method. Zoom in view high-resolution details.
28

---

Published as a conference paper at ICLR 2025
potted roses
raccoon
Figure 24: More high-resolution results of our method. Zoom in to view high-resolution details.
29

---

Published as a conference paper at ICLR 2025
salad
castle
Figure 25: More high-resolution results of our method. Zoom in to view high-resolution details.
30


Reviews:
[{"review_id": "DJ3CHXzQT2", "reviewer": "Reviewer_sKdN", "summary": "The authors introduced Prompt-Aware Introverted Attention (PAIntA) block without any training or fine-tuning requirements, enhancing the self-attention scores according to the given textual condition aiming to decrease the impact of non-prompt-relevant information. They also proposed Reweighting Attention Score Guidance (RASG), a post-hoc mechanism seamlessly integrating the gradient component in the general form of DDIM process. This allows to simultaneously guide the sampling towards more prompt-aligned latents and keep them in their trained domain.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The writing is clear to understand with detailed formulations and figures.\n\n* The results are superior when compared to other methods.\n\n* The phenomena of Appendix B are very interesting, revealing that the original model maintains a similar visual pattern from other parts of images and the PAIntA would increase the probability to respond to the prompts.", "weaknesses": "Some questions below\n\n* Would this method be easily adapted to some modern models, for example, SDXL, SD3, or even FLUX?\n\n* The successful rate of one case with sufficient sampling of different seeds is not clear.\n\n* The experiments are conducted in cases with few instances, so if there are multiple instances (>5), what is the performance?\n\n* Could the method deal with the inpainting tasks with multiple masks in one inference?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FgxMVTFZAk", "reviewer": "Reviewer_Bisp", "summary": "This paper introduces a training-free approach to enhancing prompt-guided image inpainting with diffusion models. It proposes two key components: Prompt-Aware Introverted Attention (PAIntA) and Reweighting Attention Score Guidance (RASG), which improve alignment with text prompts. PAIntA adjusts self-attention layers to prioritize text-related regions, while RASG refines cross-attention scores for better prompt consistency. A specialized super-resolution technique ensures high-quality image scaling. Quantitative and qualitative results on MSCOCO confirm the method’s superiority.", "questions": "The inference time cost should be reported and compared.\n\nHow to derive Claim 1? How to define high-quality images?\n\nWill the proposed method work on transformer-based models like SD3 and FLUX?\n\nIn Table 2, the bolded aesthetic score is not the best one.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The discussion about prompt neglect is promising. \n\nThe proposed solution achieves strong results on evaluation metrics.", "weaknesses": "Some discussion and analysis should be included, see the question part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6LjK3eWmTy", "reviewer": "Reviewer_Lgey", "summary": "Paper addresses the problem of prompt neglect in text-guided image inpainting. Existing solutions (smartbrush, imagen editor) are argued to have reduced generation quality. The proposed method is training-free. The paper proposes to techniques, Prompt-aware Introverted Attention layer, and Reweighting Attention Score guidance for accurate prompt following and high image quality.", "questions": "Overall I found the visual results appealing. The quantitative improvement less so (maybe a new metric should be developed to better show the superiority of the method ?). I found the presentation of the crucial section 3.3-3.4 of bad quality and they need to be improved much. \n\n- see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- the hypothesis that the problems of existing methods is based because of the self-attention, and that this problem can be addressed there, is interesting and convincing (even though it is not well explained).\n- the qualitative results clearly show the superiority of the proposed method. Also the quantitative results are ok, but they do not seem to indicate the large improvement seen in the qualitative results.", "weaknesses": "---\n- the self-attention map analysis (in Appendix B) is important for the motivation of the paper and should be moved to main paper. This would help to provide a motivation in section 3.3. before just stating it in math, helping the reader understand the proposed method. \n\n- the explanation of the main idea behind section 3.3 is not well presented. The main idea is the introduction of c_j in the self attention, but what this represent is not well explained in words. \n\n- does the 'introverted' nature make it hard to use information from outside of the impainted region. For example if you ask for an object with a whole behind which the background should continue ? Or a bike in front of a fence, etc. \n\n- section 3.4 should also start out by stating the problem it addresses and how it is planning to address this. I found the presentation not good of this section, and very hard to understand. \n\n- in the user study the results of DreamShaper are better than SmartBrush, but in figure 5 the results of DreamShaper are very bad not following the prompt at all, and SmartBrush is much better. Any explanation, this makes me doubt the correctness/usefullness of the user study. \n\nMINOR points: \n\nThink would be better to directly put equation of c_j also in (5), and then explain. Try to first explain the main idea, then the details (SOT, EOT, clipping etc). Now the main idea is hard to distill. \n\nMore usage of \\citep might make reading easier (e.g. line 96).\n\nToo many forward references in introduction (to future tables and figures and appendices).\n\nReferences for relevant information to appendix are out of place in the introduction. The main information should be in the main paper; the introduction introduces the most relevant information of the paper.\n\nline 248. Maybe better to keep professional factual style (instead of diary-style 'we did this' 'than that'), so better 'a thorough analysis of Stable Inpainting led to the conclusion...'", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "498gQm9uHn", "reviewer": "Reviewer_Te6W", "summary": "The paper investigates the diffusion based image inpainting task. The proposed method is training-free, which modifies the self attention block and use post training alignment/guidance. The experimental results prove the effectiveness of the proposed method.", "questions": "Please check the weakness.\n\n(Minor comment) In line 183, it should be VQGAN or VAE within SD, where usually VAE is chosen.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is intuitively clear and reasonable. The self attention may contain irrelevant information regarding area to be inpainted. The modification to self attention block makes sense. The visualization of attention map proves the efficiency of the method.\n- Although a bit heuristic, the training-free nature makes the method easily extensible to other pretrained inpainting diffusion models.\n- The proposed RASG is cleverly simple yet effective, which transforms the post training alignment to the form of non-deterministic DDIM. It seems to efficiently avoid noisy latent deviating too far from the original trajectory.", "weaknesses": "- Lack of experiments on running time. What is the additional time cost associated with the proposed method?\n- Lack of ablation study on hyperparameters. How sensitive is the model to different hyperparameters? Espscially $\\eta$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)