HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models

Authors: Hayk Manukyan, Andranik Sargsyan, Barsegh Atanyan, Zhangyang Wang, Shant Navasardyan, Humphrey Shi
Primary area: generative models
Keywords: text-guided image inpainting, diffusion models, high-resolution image inpainting

Abstract:
Recent progress in text-guided image inpainting, based on the unprecedented success of text-to-image diffusion models, has led to exceptionally realistic and visually plausible results. However, there is still significant potential for improvement in current text-to-image inpainting models, particularly in better aligning the inpainted area with user prompts. Therefore, we introduce $\textit{HD-Painter}$, a $\textbf{training-free}$ approach that $\textbf{accurately follows prompts}$. To this end, we design the $\textit{Prompt-Aware Introverted Attention (PAIntA)}$ layer enhancing self-attention scores by prompt information resulting in better text aligned generations. To further improve the prompt coherence we introduce the $\textit{Reweighting Attention Score Guidance (RASG)}$ mechanism seamlessly integrating a post-hoc sampling strategy into the general form of DDIM to prevent out-of-distribution latent shifts. Our experiments demonstrate that HD-Painter surpasses existing state-of-the-art approaches quantitatively and qualitatively across multiple metrics and a user study. Code is publicly available at: [https://github.com/Picsart-AI-Research/HD-Painter](https://github.com/Picsart-AI-Research/HD-Painter)

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729671175115

thread_type:
official_review

<thread_object>
creator: reviewer_te6w

<content>
summary:
The paper investigates the diffusion based image inpainting task. The proposed method is training-free, which modifies the self attention block and use post training alignment/guidance. The experimental results prove the effectiveness of the proposed method.

strengths:
- The method is intuitively clear and reasonable. The self attention may contain irrelevant information regarding area to be inpainted. The modification to self attention block makes sense. The visualization of attention map proves the efficiency of the method.
- Although a bit heuristic, the training-free nature makes the method easily extensible to other pretrained inpainting diffusion models.
- The proposed RASG is cleverly simple yet effective, which transforms the post training alignment to the form of non-deterministic DDIM. It seems to efficiently avoid noisy latent deviating too far from the original trajectory.

weaknesses:
- Lack of experiments on running time. What is the additional time cost associated with the proposed method?
- Lack of ablation study on hyperparameters. How sensitive is the model to different hyperparameters? Espscially $\eta$.

questions:
Please check the weakness.

(Minor comment) In line 183, it should be VQGAN or VAE within SD, where usually VAE is chosen.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730374423218

thread_type:
official_review

<thread_object>
creator: reviewer_lgey

<content>
summary:
Paper addresses the problem of prompt neglect in text-guided image inpainting. Existing solutions (smartbrush, imagen editor) are argued to have reduced generation quality. The proposed method is training-free. The paper proposes to techniques, Prompt-aware Introverted Attention layer, and Reweighting Attention Score guidance for accurate prompt following and high image quality.

strengths:
- the hypothesis that the problems of existing methods is based because of the self-attention, and that this problem can be addressed there, is interesting and convincing (even though it is not well explained).
- the qualitative results clearly show the superiority of the proposed method. Also the quantitative results are ok, but they do not seem to indicate the large improvement seen in the qualitative results.

weaknesses:
---
- the self-attention map analysis (in Appendix B) is important for the motivation of the paper and should be moved to main paper. This would help to provide a motivation in section 3.3. before just stating it in math, helping the reader understand the proposed method. 

- the explanation of the main idea behind section 3.3 is not well presented. The main idea is the introduction of c_j in the self attention, but what this represent is not well explained in words. 

- does the 'introverted' nature make it hard to use information from outside of the impainted region. For example if you ask for an object with a whole behind which the background should continue ? Or a bike in front of a fence, etc. 

- section 3.4 should also start out by stating the problem it addresses and how it is planning to address this. I found the presentation not good of this section, and very hard to understand. 

- in the user study the results of DreamShaper are better than SmartBrush, but in figure 5 the results of DreamShaper are very bad not following the prompt at all, and SmartBrush is much better. Any explanation, this makes me doubt the correctness/usefullness of the user study. 

MINOR points: 

Think would be better to directly put equation of c_j also in (5), and then explain. Try to first explain the main idea, then the details (SOT, EOT, clipping etc). Now the main idea is hard to distill. 

More usage of \citep might make reading easier (e.g. line 96).

Too many forward references in introduction (to future tables and figures and appendices).

References for relevant information to appendix are out of place in the introduction. The main information should be in the main paper; the introduction introduces the most relevant information of the paper.

line 248. Maybe better to keep professional factual style (instead of diary-style 'we did this' 'than that'), so better 'a thorough analysis of Stable Inpainting led to the conclusion...'

questions:
Overall I found the visual results appealing. The quantitative improvement less so (maybe a new metric should be developed to better show the superiority of the method ?). I found the presentation of the crucial section 3.3-3.4 of bad quality and they need to be improved much. 

- see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730465304322

thread_type:
official_review

<thread_object>
creator: reviewer_bisp

<content>
summary:
This paper introduces a training-free approach to enhancing prompt-guided image inpainting with diffusion models. It proposes two key components: Prompt-Aware Introverted Attention (PAIntA) and Reweighting Attention Score Guidance (RASG), which improve alignment with text prompts. PAIntA adjusts self-attention layers to prioritize text-related regions, while RASG refines cross-attention scores for better prompt consistency. A specialized super-resolution technique ensures high-quality image scaling. Quantitative and qualitative results on MSCOCO confirm the method’s superiority.

strengths:
The discussion about prompt neglect is promising. 

The proposed solution achieves strong results on evaluation metrics.

weaknesses:
Some discussion and analysis should be included, see the question part.

questions:
The inference time cost should be reported and compared.

How to derive Claim 1? How to define high-quality images?

Will the proposed method work on transformer-based models like SD3 and FLUX?

In Table 2, the bolded aesthetic score is not the best one.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730469508769

thread_type:
official_review

<thread_object>
creator: reviewer_skdn

<content>
summary:
The authors introduced Prompt-Aware Introverted Attention (PAIntA) block without any training or fine-tuning requirements, enhancing the self-attention scores according to the given textual condition aiming to decrease the impact of non-prompt-relevant information. They also proposed Reweighting Attention Score Guidance (RASG), a post-hoc mechanism seamlessly integrating the gradient component in the general form of DDIM process. This allows to simultaneously guide the sampling towards more prompt-aligned latents and keep them in their trained domain.

strengths:
* The writing is clear to understand with detailed formulations and figures.

* The results are superior when compared to other methods.

* The phenomena of Appendix B are very interesting, revealing that the original model maintains a similar visual pattern from other parts of images and the PAIntA would increase the probability to respond to the prompts.

weaknesses:
Some questions below

* Would this method be easily adapted to some modern models, for example, SDXL, SD3, or even FLUX?

* The successful rate of one case with sufficient sampling of different seeds is not clear.

* The experiments are conducted in cases with few instances, so if there are multiple instances (>5), what is the performance?

* Could the method deal with the inpainting tasks with multiple masks in one inference?

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents ProFI-Painter, a training-free approach that accurately follows prompts for text-to-image inpainting. Moreover, they also proposed RASG, a post-hoc mechanism to guide the sampling towards latents aligned to prompts and prevent out-of-distribution latent shifts.

The major paper strengths: 1) The discussion on prompt neglect for inpainting and the proposed idea to improve prompt following is interesting.  2) Both ProFI-Painter and RASG are effective. 3) The experiments show promising results.  4) The extension to FLUX in authors' rebuttal is also helpful.

Considering these strengths, I will recommend "Accept (poster)".