Diffree: Text-Guided Shape Free Object Inpainting with Diffusion Model

Authors: Lirui Zhao, Tianshuo Yang, Wenqi Shao, Yuxin Zhang, Ping Luo, Kaipeng Zhang, Rongrong Ji
Primary area: generative models
Keywords: Image inpainting; Text-guided image editing;

Abstract:
This paper addresses an important problem of object addition for images with only text guidance. It is challenging because the new object must be integrated seamlessly into the image with consistent visual context, such as lighting, texture, and spatial location. While existing text-guided image inpainting methods can add objects, they either fail to preserve the background consistency or involve cumbersome human intervention in specifying bounding boxes or user-scribbled masks. To tackle this challenge, we introduce Diffree, a Text-to-Image (T2I) model that facilitates text-guided object addition with only text control. To this end, we curate OABench, an exquisite synthetic dataset by removing objects with advanced image inpainting techniques. OABench comprises 74K real-world tuples of an original image, an inpainted image with the object removed, an object mask, and object descriptions. Trained on OABench using the Stable Diffusion model with an additional mask prediction module, Diffree uniquely predicts the position of the new object and achieves object addition with guidance from only text. Extensive experiments demonstrate that Diffree excels in adding new objects with a high success rate while maintaining background consistency, spatial appropriateness, and object relevance and quality.

Decision: reject
Reviewer scores: 5, 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1729928561008

thread_type:
official_review

<thread_object>
creator: reviewer_js6x

<content>
summary:
This paper proposes an object addition method called Diffree. Diffree comprises an object mask prediction (OMP) module, which could be used to generate masks through textual inputs. Thus Diffree can be regarded as a mask-free inpainting model. The authors also provide a synthetic dataset called OABench with 74k real-world object removal tuples built by PowerPaint. Extensive experiments verify the effectiveness of Diffree.

strengths:
1. This paper proposed a sophisticated and reliable data collection process to achieve high-quality image removal tuples, which would potentially strengthen many other inpainting-based tasks. 

2. The authors show some promising and impressive object addition results in the appendix, including both real-world and AI-generated images.

weaknesses:
1. As mentioned in the main paper, Diffree is trained to predict precise object masks, which is not convincing for both object removal and object addition. Because most objects contain shadows or reflections, which are not included in object masks. Without proper shadow and reflection generation, the inserted objects suffer from "copy and paste" artifacts. For example, no shadows are generated in the "Coconut palm" of Figure 3, and no reflections are generated in the "boat" of Figure A2. The metric of background consistency is also problematic. I think this is the main drawback of this method. 

2. Missing discussion and ablation about the necessity of the joint training of OMP and diffusion model. Why not independently train OMP to predict the mask.

3. The generalization of the Diffree should be concerned. As shown in Figure 9 (a), the generation failed while the mask was very small.

4. For the object addition guided completely by texts, the authors should provide more results to verify whether the proposed method could handle different orientational descriptions. For example, these prompts should be considered: "on left/right/up/bottom of...".

questions:
1. Why not use Diffree to address mask-free removal? This should also be addressed by OABench.

2. What did "with/without mix with previous image" mean in Figure 8?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730276711958

thread_type:
official_review

<thread_object>
creator: reviewer_zvng

<content>
summary:
The paper introduces Diffree, a text-to-image (T2I) model aimed at text-guided object addition within images, a process where new objects are integrated into a scene while maintaining visual consistency in lighting, texture, and spatial orientation. Unlike previous approaches, Diffree avoids the need for manually defined masks, instead using a synthetic dataset (OABench) for training. OABench, composed of image pairs with and without specified objects, is created by advanced image inpainting techniques. Trained on OABench, Diffree incorporates a Stable Diffusion backbone with an Object Mask Predictor (OMP) module to predict object placement. Results indicate that Diffree achieves high success in preserving background coherence, achieving spatially plausible integration, and producing visually realistic objects.

strengths:
- **Originality**: Diffree’s approach of shape-free object addition guided solely by text is novel, significantly enhancing usability by eliminating the need for manual mask definitions. This innovation in user experience represents a unique contribution.
- **Clarity**: Overall, the dataset creation process, and evaluation metrics are well-described, with figures that aid understanding of Diffree’s operational and comparative performance.

weaknesses:
- **Minor Typographical Issue**: There is a missing period at the end of line 101, which should be corrected for clarity.
- **Dataset Limitation in Prompt Detail**: Since the dataset primarily relies on the COCO dataset, prompts are often generic object labels rather than detailed, fine-grained descriptions. This limitation can hinder the model's ability to respond to nuanced or interactive prompts, such as requests for specific object attributes or context-based interactions.
- **Methodology Clarity**: The description of the proposed methodology, especially the Object Mask Predictor (OMP) module, could benefit from further clarification. For example, more detail on why the image latent and latent noise are concatenated as inputs for the OMP module during both training and inference phases would help improve comprehension.
- **Figure 6 Presentation**: Figure 6 needs a more detailed caption to allow readers to follow the workflow easily by reading only the figure caption. The current figure layout lacks clarity, making it challenging for readers to understand the process.
- **Inconsistency in Visuals (Figure 1)**: In Figure 1, example 2 appears visually inconsistent, with an unrealistic rendering of the added object. This raises questions about the model's object realism under certain conditions.
- **Lack of User Study**: Including a user study would strengthen the evaluation of the model, particularly to capture subjective aspects of user satisfaction and perceived realism in object integration.
- **Limited Comparative Evaluation**: The model is only compared against PowerPaint and InstructPix2Pix. Including additional comparison baselines, such as recent text-guided inpainting methods or general object addition approaches, would provide a more comprehensive assessment.
- **Ambiguity in Object Placement in Complex Scenes**: The method struggles with complex images involving multiple entities (e.g., two people). When required to add an object to one specific entity, the model may face ambiguity, as there is limited functionality for users to adjust the mask and specify the target placement.

questions:
1. How does Diffree handle ambiguities in scenes with multiple entities (e.g., if there are two people in an image, how does it decide which one to add the object to)? Could user input be incorporated to clarify target placement in such cases?
2. Could more fine-grained prompts be incorporated or generated to enhance the model's ability to handle specific requests for object details and interactions within a scene?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730566215783

thread_type:
official_review

<thread_object>
creator: reviewer_ethc

<content>
summary:
The paper presents a novel model, Diffree, which facilitates the addition of objects to images. The authors have constructed a synthetic dataset named OABench, consisting of 74K real-world tuples, to train Diffree using a Stable Diffusion model augmented with an additional mask prediction module. Diffree predicts the target mask and generates inpainting results simultaneously. Extensive experiments demonstrate Diffree's good performance in adding new objects with high success rates while preserving background consistency, spatial appropriateness, and object relevance and quality.

strengths:
1. Diffree offers a user-friendly approach to inserting objects into images. The mask-free object insertion is particularly useful in practical applications.
2. The creation of OABench, a large-scale synthetic dataset, is a significant contribution, providing a rich resource for training and evaluating object addition models.
3. The OMP module's ability to predict the target mask and generate inpainting results simultaneously is a novel architectural advancement in this field.

weaknesses:
1. [1] proposed a method for mask prediction closely related to Diffree. An in-depth analysis and comparison with this work would be beneficial.
2. All the prompts used in this paper are in the form "add {object}". It is unclear how Diffree generalizes to more precise control, such as "add a dragon in the room".
3. While user-friendly for object insertion, it restricts users from adjusting the mask. In standard image processing, users or designers often need to make adjustments to achieve their desired results.

[1] SmartMask: Context Aware High-Fidelity Mask Generation for Fine-grained Object Insertion and Layout Control

questions:
1. In Fig. 1, the "add dragon" prompt results in "a dragon painting in the frame". Is this behavior as expected?
2. How does Diffree perform with out-of-distribution (OOD) objects not included in OABench?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620815850

thread_type:
official_review

<thread_object>
creator: reviewer_t9nw

<content>
summary:
This paper proposes solving the object painting problem with only text as guidance and does not rely on other constraints, e.g., shape mask. The authors first construct a synthetic dataset, called OABench, to facilitate the model’s training with the help of the existing SOTA inpainting model. Then, a diffusion-based model named Diffree is proposed to train on this OABench that can simultaneously output the image with the added object and the object mask. Finally, the authors also conduct comparisons to validate the superiority of their method.

strengths:
- Object inpainting using only text without relying on shape constraints.
- The authors built an OABench to facilitate text-guided object inpainting.
- The results are attractive and the supported applications are interesting.

weaknesses:
- Lack of ablation of the validation model design. For example, what happens to the output if the OMP is removed? BTW, integrating a mask head in the diffusion process is not new, e.g. in [1].
- The comparison is slightly unfair. In Figure-A10, comparing the model you trained on the curated dataset to other methods that were not retrained or fine-tuned is unfair.
- This paper is poorly written, especially the explanation of the charts. For example, in line 50 of the description section, the author provides links to three figures for the reader to view. However, these figures lack sufficient detail to explain what the figure depicts and what the symbols within the figures/captions represent.

[1]. Shadow Generation for Composite Image Using Diffusion Model–CVPR 2024

questions:
- How well does the model generalize? Since the model was trained on a synthetic dataset generated from an inpainting model, it is not surprising that the model may overfit the artifacts introduced by the inpainting model.
- In Line 468, the authors argue that their Local CLIP Score is slightly inferior to the InstructPixel2Pixel due to the latter considers only the successful results are taken into consideration. I wonder what the results will be if you combine both the successful and failure cases.
- Regarding the generalization problem, assuming that a real image contains two dogs, and I first use PowerPaint to delete the two dogs, and then let the proposed Diffree generate one dog, what will the output image show? Will the position of the new dog duplicate the position of the previously removed one?
- Were the compared methods (e.g., InstructPix2Pix) trained on the proposed dataset as well? or just directly copied their pre-trained weights for use?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732192461356

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Paper Updating

comment:
We have carefully addressed all reviewers’ comments and concerns. Major modifications are highlighted in blue in the updated manuscript. Below is a summary of the key changes in our updated version:

1. **We added more comparisons and discussion**

- Added in-depth discussion and explanations of the OMP module’s role and training process. (see Section A8 in Appendix).
- Added a user study on the overall satisfaction with all results between InstructPix2Pix and our Diffree. (see Section A9 in Appendix).
- Added discussions and additional experiments to demonstrate generalization and address specific concerns about artifacts. (see Section A10.1 in Appendix).
- Added discussions on the generalization of out-of-distribution (OOD) objects not included in OABench for Diffree. (see Section A10.2 in Appendix).
- Added discussions of the the generation from Diffree regarding the description with specific object attributes and context-based interactions. (see Section A10.3 in Appendix).

2. **We added more experimental results.**

- Explored Diffree's results on the generalization ability for artifacts. (see Figure A13 in Appendix).
- Explored the results regarding users' satisfaction through a user study. (see Figure A12 in Appendix).
- Conducted experiments using contextually detailed prompts specifying different locations within the same scene or involving multiple similar objects. (see Figure A14 in Appendix).
- Explored the results of combining Diffree and shadow generation work. (see Figure A11 in Appendix).

3. **Addressed other phrasing clarifications requested by reviewers.**
</content>
</thread_object>

</thread>
<thread>
creation date:
1732192528330

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response (About the Object Mask Predictor (OMP) module)

comment:
We sincerely thank all the reviewers for their valuable feedback and suggestions. To address concerns about the Object Mask Predictor (OMP) module in our method, we provide an in-depth discussion summarizing its role:

1. **Ensuring background consistency, which is crucial for iterative additions** 

   In mask-guided methods (e.g., PowerPaint), the instance mask is a required input. Typically, a post-processing step involves mix the synthesized object into the input image’s background using the instance mask, ensuring the background remains unchanged.

   In contrast, text-guided methods (e.g., InstructPix2pix and Diffree) do not require an instance mask as input, making this mix operation unavailable. This limitation can lead to quality degradation, especially in iterative additions. To the best of our knowledge, we are the first to introduce an output mask through the OMP module in text-guided inpainting methods. This enables the mix operation and allows for iterative additions. We provided an ablation study (Figure 8 in the paper) to evaluate the impact of omitting the OMP module on Diffree’s iterative results. Without the OMP module, the background deteriorates rapidly after multiple steps, rendering further additions infeasible.

2. **Integration with Existing Works for Exciting Applications**

   As explained in the paper, the instance mask can be used in many applications that require a mask as input. For example, when combined with AnyDoor [1], Diffree can achieve image-prompted object addition. We highlight a few additional points:

   (1) *Combining with shadow generation methods to produce realistic shadows*

   The task, shadow generation, aims to create plausible shadows for a composite foreground, given a composite image without foreground shadows and the foreground object mask [2]. However, existing mask-guided and text-guided inpainting methods pose challenges for shadow generation, especially for objects casting long shadows. These challenges arise for two reasons: first, there is a misalignment between the data masks and the objects and actual shadows (e.g., long shadows); second, for mask-guided method, it is difficult for users to draw the estimated masks of long shadow area with objects , which also is challenging for the model understands the respective parts of shadows and objects in the mask. Therefore, combining inpainting works with shadow generation works can lead to better results. 

   In mask-guided methods, the shadow generation method's input can be derived from the input mask and output image. Thanks the OMP module’s output mask, Diffree, as a text-guided method without mask input, effectively integrates with such methods to generate coherent objects and realistic shadows, as shown in Figure A11 of the Appendix.

   (2) *Providing a starting point for user or designer adjustments*

   In standard image processing, users or designers often need to make adjustments to achieve desired results. Diffree’s output mask serves as a good starting point, making it easier for designers to refine the outcome. We emphasize that the OMP’s mask output can be combined with evolving mask-based methods, serving as input for better results or continuous adjustments.

3. **Early Mask Output During Initial Decoding Steps**

   During training, diffusion model takes a concatenation of the input latent $z$ and the noisy output latent $\tilde{z_t}$ to estimate the noise at each timestep. The estimated noise is then used to denoise  $\tilde{z_t}$, producing a noise-free latent $\tilde{o_t}$. The concatenation of $z$ and $\tilde{o_t}$, including input and denoised output information, is passed to OMP to generate the object mask $m$:
   $$
   \tilde{o_t} = \frac{\tilde{z_{t}} - \sqrt{1-\bar{\alpha}t} \epsilon_\theta(\tilde{z{t}}, z, \mathrm{Enc}_{\rm txt}(d), t)}{\sqrt{\bar{\alpha}_t}}
   $$
   This $\tilde{o_t}$ can be derived from the denoising process and is available at each step, enabling mask prediction in the initial steps (see Figure A4 in the Appendix). This allows us to quickly obtain a reasonable mask of the added objects without waiting for complete generation, facilitating integration with various applications.  Training the OMP solely on image pairs and object masks would limit mask generation to after full denoising at inference time.  By aligning the OMP’s training inputs with those of the diffusion model and detaching gradients of $\tilde{o}_t$, we ensure independent optimization without interference, as noted on Line 297. The independence of their loss functions and input consistency make separate training theoretically equivalent to joint training.



[1] Chen, Xi, et al. "Anydoor: Zero-shot object-level image customization." CVPR 2024.

[2] Liu, Qingyang, et al. "Shadow Generation for Composite Image Using Diffusion Model." CVPR 2024.
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) This paper introduces Diffree, a diffusion-based model trained on the proposed synthetic OABench dataset for text-guided object painting without additional constraints, demonstrating superior performance through comparisons.

(b) Strengths: The paper presents a novel approach to object inpainting using only text guidance, eliminating the need for shape constraints and manual mask definitions, which significantly enhances usability. The creation of OABench, a large-scale synthetic dataset, is a major contribution, providing a valuable resource for training and evaluating object addition models. Diffree's user-friendly, mask-free object insertion method demonstrates strong practical applicability, supported by its innovative OMP module that predicts masks and generates inpainting results simultaneously. The results are visually appealing, and the well-documented dataset creation process and evaluation metrics ensure clarity and ease of understanding.

(c) Weaknesses: The paper has several weaknesses in the initial version, including a lack of ablation studies for validating model design, and insufficient comparisons with closely related works, particularly methods integrating mask prediction in diffusion models. The evaluation is criticized for unfair comparisons (e.g., using retrained models against non-fine-tuned baselines) and limited diversity in prompts, restricting the model's ability to handle nuanced or context-based interactions. The clarity of methodology descriptions, charts, and figures is lacking, with specific issues in explaining workflows (e.g., Figure 6) and inconsistent visual results (e.g., Figure 1). Additionally, the absence of a comprehensive user study and limited comparative evaluation against more recent inpainting or object addition methods reduce the strength of the paper's claims, while the inability to handle complex scenes or provide mask adjustments limits practical usability.

(d) The most important reasons for reject are: Although this work does show potential, the authors present too many flaws in the paper, which can easily lead to misunderstandings and needs significant refinement. E.g., Figure 1 is intended for stress-test, it is ok but could you please change a case that would not be challenged by the visual consistency issue? How effectively a paper minimizes controversy is also a key measure of its quality. Furthermore, the authors were overly passive during the rebuttal process and should have proactively provided relevant and necessary information in many cases (GPT-4O relabelling, user study, etc.) to convince readers and reviewers. E.g., as for the user study problem, providing the all the details for how to conduct the user study is very important at the first presentation. Although the authors provided additional details in response to the reviewers' further requests, the AC found that it remains unclear how many people participated in the user study and what their relationship is to the 1,000 cases. Were multiple people involved in evaluating a single case to avoid bias? 

The AC encourages the authors further minimize controversy and improve the overall quality. However, based on current states and rebuttal/discussion process, the overall evaluation still leans towards rejection. Sorry for this and wish the authors good luck in the next upcoming conference. Your valuable work will eventually be recognized by the community; perhaps it just needs a little more patience.