Diffree: Text-Guided Shape Free Object Inpainting with Diffusion Model

## Abstract

Abstract This paper addresses an important problem of object addition for images with only text guidance. It is challenging because the new object must be integrated seamlessly into the image with consistent visual context, such as lighting, texture, and spatial location. While existing text-guided image inpainting methods can add objects, they either fail to preserve the background consistency or involve cumbersome human intervention in specifying bounding boxes or user-scribbled masks. To tackle this challenge, we introduce Diffree, a Text-to-Image (T2I) model that facilitates text-guided object addition with only text control. To this end, we curate OABench, an exquisite synthetic dataset by removing objects with advanced image inpainting techniques. OABench comprises 74K real-world tuples of an original image, an inpainted image with the object removed, an object mask, and object descriptions. Trained on OABench using the Stable Diffusion model with an additional mask prediction module, Diffree uniquely predicts the position of the new object and achieves object addition with guidance from only text. Extensive experiments demonstrate that Diffree excels in adding new objects with a high success rate while maintaining background consistency, spatial appropriateness, and object relevance and quality.

###### Keywords:

## 1 Introduction

With the recent remarkable success of Text-to-Image (T2I) models (*e.g*., Stable Diffusion [], Midjourney [], and DALL-E [, ]), creators can quickly generate high-quality images with text guidance. The rapid development has driven various text-guided image editing techniques [, , , , ]. Among these techniques, text-guided object addition which inserts an object into the given image has attracted much attention due to its diverse applications, such as advertisement creation, visual try-on, and renovation visualization. While important, object addition is challenging because the object must be integrated seamlessly into the image with consistent visual context, such as lighting, texture, and spatial location.

Existing techniques for object addition in images can be broadly categorized into mask-guided and text-guided approaches (Fig. 2). Mask-guided algorithms typically require the specification of a region where the new object will be inserted. For example, traditional image inpainting methods [, , , , ] focus on seamlessly filling user-defined masks within an image to match the surrounding context. Recent advancements, such as PowerPaint [], have effectively incorporated objects into images given their shape and textual descriptions while maintaining background consistency. However, manually delineating an ideal region for all objects, considering shape, size, and position, can be labor-intensive and typically requires drawing skills or professional knowledge.
On the other hand, text-guided object addition methods, such as InstructPix2Pix [], attempt to add new objects using only text-based instructions. Despite this, these methods have a low success rate and often result in background inconsistencies, as demonstrated in Fig. 2 and Fig. 9. Additionally, when employing InstructPix2Pix for iterative object addition, the quality of the inpainted image tends to degrade progressively with each step, as illustrated in Fig. 10.

To tackle the above challenges, we introduce Diffree, a diffusion model with an additional object mask predictor module that can predict an ideal mask for a candidate inpainting object and achieve shape-free object addition with only text guidance.
Compared with previous works  [, , , ], our Diffree has three appealing properties.
First, Diffree can achieve impressive text-guided object addition results while keeping the background unchanged. In contrast, previous text-guided methods  [] struggle to guarantee this.
Second, Diffree does not require additional mask input, which is necessary for traditional mask-guided methods []. In real scenarios, high-quality masks are hard to obtain.
Third,
Diffree can generate the instance mask and thus can be further combined with various existing works [, ] to develop exciting applications.
For example, Diffree can achieve image-prompted object addition when combined with AnyDoor [] and plan to add objects suggested by GPT4V [], as shown in Fig. 11.

Figure: Figure 2: Qualitative comparisons of Diffree and different kinds of methods.
Refer to caption: x2.png

Towards high-quality text-guided object addition, we curate a synthetic dataset named Object Addition Benchmark (OABench) which consists of 74K real-world tuples including an original image, an inpainted image, a mask image of the object, and an object description.
The data curation process is illustrated in Fig. 5. Note that object addition can be deemed as the inverse process of object removal. We build OABench by removing objects in the image using advanced image inpainting algorithms such as PowerPaint  []. In this way, we can obtain an original image containing the object, an inpainted image with the object removed, the object mask, and the object descriptions. We use instance segmentation dataset COCO [, ] as the source data, which has two benefits. First, the source image captures comprehensive natural scenes where the location and shape of one individual object often exhibit intrinsic alignment with the overall scene. It helps guarantee the reasonability of new objects’ location. For instance, a monitor is commonly situated behind computer peripherals. Second, the ground-truth mask of the object already exists in the instance segmentation dataset, which can be directly utilized in removing objects with background consistency preserved. By contrast, InstructPix2Pix [] collects image pairs using proprietary T2I model [] under prompt pair with subtle modifications. While this approach maintains new objects’ reasonability, it poses difficulties in preserving background consistency.

With OABench, Diffree is trained to predict masks and images containing the new object given the original image and object text description. Thanks to the extensive coverage of objects in natural scenes in OABench, Diffree can add various objects to the same image while matching the visual context well as shown in Fig. 3. Moreover, Diffree can iteratively insert objects into a single image while preserving the background consistency using the generated mask as shown in Fig. 1 and Fig. 4.

For evaluation, we propose a set of evaluation rules through existing metrics [, , , , ], including consistency of background, reasonableness of object location, quality, diversity and correlation of generated object, and success rate. Extensive experiments show that Diffree performs better in object addition than previous mask-guided and text-guided techniques. For instance, Diffree obtains a significantly higher success rate than InstructPix2Pix. For successful cases, Diffree still outperforms InstructPix2Pix in various quantitative metrics.

Figure: Figure 3: Diffree adds objects to the same image, with different spatial relationships.
Refer to caption: x3.png

The contributions of this work are three-fold.
1) We proposed Diffree, a model that can achieve text-guided shape-free object addition to free users from drawing the appropriate mask of objects.
The inpainted image from Diffree includes the new objects with reasonable shapes and consistent visual context.
2) We introduced OABench, an exquisite synthetic dataset for object addition. OABench comprises 74K real-world training data for the task of object addition.
3) We evaluate this task with a set of rules for comprehensive assessment. Extensive experiments demonstrate the effectiveness of Diffree.
For example, Diffree achieves a high success rate (*e.g*., 98.5% in COCO) and superior unified score (*e.g*., 38.92 versus 4.48) compared with other methods.

## 2 Related Work

Text-to-Image Diffusion Models
Recently, text-to-image (T2I) diffusion models [, , ], have shown exceptional capability in image generation quality and extraordinary proficiency in accurately following text prompts, under the dual support of large-scale text-image dataset [] and model optimizations [, , ].
GLIDE [] incorporated text conditions into the diffusion model and empirically showed that leveraging classifier guidance leads to visually appealing outcomes.
DALLE-2 [] enhances text-image alignment via CLIP [] joint feature space, DALLE-3 [] further improves the prompt following abilities by training on highly descriptive generated image captions.
Stable Diffusion [], which is well-established and widely adopted, garners significant attention and application within and beyond the research community.
Given that T2I models generate comprehensive images from text prompts, even minor alterations in prompts can result in substantial changes to the resultant image [].
Consequently, there has been an increased focus not only on T2I generation but also on image editing based on additional conditions such as text inputs, masks, *et al*.

Text-Guided Image Editing
The effectiveness of the text-guided image editing methos [, , , ] largely depends on the composition of its dataset and how it is collected.
InstructPix2Pix [] combines two large pretrained models, a large language model [] and a T2I model [], to generate a dataset for training a diffusion model to follow written image editing text prompts.
Its innovative data collection method allows InstructPix2Pix to follow instructions and shows amazing effects, while makes its consistency is difficult to guarantee due to both input and output are generated by the T2I model.
InstructDiffusion [] treats all computer vision tasks as image generation with multiple output formats, and aligns these tasks with human instructions.
Emu Edit [] adapt its architecture for multi-task learning and train it an unprecedented range of tasks formulated as generative tasks, and displays strong and diverse results, especially in the combination of multiple tasks.
Emu Edit generate the output image and apply the mask-based attention to generate input iamge for object addtion task.
MagicBrush [] introduces a manually annotated dataset which both input and output are generated by the T2I model.
The image editing performance of fine-tuneing InstructPix2Pix on MagicBrush shows better.
Unlike the previous methods, we propose a novel and easily expandable collection method, thanks to the existing instance segmentation dataset, we use real images as output and synthetic images without a specific object as input.
Our work closely relates to the concurrent work PIPE [], which independently explores similar concepts and methodologies. Both studies involve removing objects to collect an object addition dataset and train a diffusion model for text-guided object addition. Our approach additionally trains an Object Mask Predictor (OMP) module to predict the mask of objects. We believe that the concurrent exploration of these ideas underscores the significance and timeliness of this research direction.

Mask-Guided Image Inpainting
Mask-guided image inpainting methods [, , , ] alter the image in specific areas under addition conditions (*e.g*., text), while maintaining the background in its original state.
SmartBrush [] achieves precise object inpainting guided by text and mask through a novel training and sampling strategy .
Imagen Editor [], finetuned on Imagen [], captures fine details in the input image by conditioning the cascaded pipeline to accomplish precise image inpainting through a user defined mask and text prompts.
AnyDoor [] employs a discriminative ID extractor and a frequency-aware detail extractor to characterize the target object, thereby facilitating effective object addition given an area and corresponding object image.
Powerpaint [] demonstrates superior performance on various inpainting benchmarks attributed to the introduction of learnable tokens to distinguish different tasks.
Although these methods have achieved amazing image inpainting effects, their commonality is the need for a mask.
For ordinary users, drawing an object mask with appropriate shape, size, and aspect ratio, corresponding accurately to the object and image, presents an unignorable challenge.

## 3 Methodology

Given an image and the object description, our goal is to add the object to the image while preserving the background consistency.
Following this, we initially introduce OABench, an synthetic dataset for this task, comprising image-text pairs (as input) with corresponding object masks and images containing the object (as output).
We provide an overview of our data collection pipeline in Sec. 3.1 and.
We next present Diffree, an architecture amalgamating a Stable Diffusion model with an Object Mask Predictor (OMP) module in Sec. 3.2.
The evaluation procedure is presented in Sec. 3.3.

Figure: Figure 4: Diffree iteratively generates results. Objects added later can relate to the earlier.
Refer to caption: x4.png

### 3.1 OABench

We combine existing instance segmentation dataset [, ] with powerful image inpainting method [] to generate the OABench.
Unlike other instructions follow methods [, ], generating both data pairs using existing text-to-image (T2I) models [, ] with prompt pairs and filtering, we use the real image with object to synthesize the image without the object, as depicted in Fig. 5.
This can greatly ensure the consistency of the background as in other image inpainting methods [] that require masks.
Furthermore, an object in the real image naturally aligns with its background, *i.e*., it is appropriate for generating the corresponding image without the same object.
In the following sections, we describe in detail the three steps of this process.

#### 3.1.1 Collection and Filtering

We gather and refine instances suitable for image inpainting by applying a set of rules from the LVIS dataset [], a large instance segmentation dataset annotated for COCO [] dataset.
As depicted in Fig. 5, in images containing multiple instances, we enforce size constraints to exclude instances that are too big or too small (typically related to object components or background elements like buttons on clothing or rivers).
Subsequently, incomplete instances are filtered out using edge detection and integrity assessments.
Instances that are partially obscured are identified through cavity inspection, iterative IOU algorithm application, and common part comparison among various instances.
Additionally, objects with exceptionally high aspect ratios, which tend to yield subpar inpainting outcomes, are also eliminated.

#### 3.1.2 Data Synthesis

We next employ a powerful image inpainting method, PowerPaint [], to eliminate specific instances obtained in the preceding stage.
Therefore, we can generate a synthetic image without specific objects with background consistency with the original image.
simultaneously, the original image, object mask, and corresponding name can be extracted from the LVIS and COCO.

Figure: Figure 5: The data collection process of OABench.
Refer to caption: x5.png

#### 3.1.3 Post-Processing

In the post-processing stage, we filter out the results with poor effects in image inpainting.
For some special cases (*e.g*., one of many dense and adjacent small cakes), image inpainting cannot effectively remove objects due to the complexity of the background.
Thus we calculate the clip score [] using the object name and the region of the inpainted image, setting a threshold to remove images with higher scores which are deemed suboptimal.
Finally, OABench includes 74,774 high-quality data pairs, each data pair includes a synthetic image and object caption as input, object masks and original images as output.

### 3.2 Diffree

For an image $x$ and a text prompt $d$, Diffree predicts a binary mask $m$ that specifies the region in $x$ and generates an image $\tilde{x}$. The masked region $\tilde{x}\odot m$ aligns with the text prompt $d$.
To this end, Diffree is instantiated with a pre-trained T2I diffusion model (*e.g*. Stable Diffusion []) with a object mask prediction (OMP) module as shown in Fig. 6.

#### 3.2.1 Diffusion Model

learns to generate data samples by iteratively applying denoising autoencoders that estimate the score function [] of a given data distribution [].
Stable Diffusion [] apply them in the latent space of powerful pre-trained variational autoencoder [], including encoder $\mathcal{E}$ and decoder $\mathcal{D}$, to reduce computing resources while maintaining quality and flexibility.
Stable Diffusion encompasses both forward and reverse processes.
Given an image $\tilde{x}$, the forward process adds noise to the encoded latent $\tilde{z}=\mathcal{E}(\tilde{x})$:

$$ $\displaystyle\tilde{z_{t}}=\sqrt{\bar{\alpha}_{t}}\tilde{z}+\sqrt{1-\bar{ \alpha}_{t}}\epsilon,\epsilon\sim\mathcal{N}(0,\text{I})$ (1) $$

where $\tilde{z_{t}}$ is the noisy latent at timestep $t$, $\bar{\alpha}_{t}$ denotes the associated noise level.

Figure: Figure 6: Overview of Diffree framework.
Refer to caption: x6.png

In the reverse process, we learn a network $\epsilon_{\theta}$ that predicts the noise added to the noisy latent $\tilde{z_{t}}$, conditioned on both the image $x$ and text $d$.
To fine-tune Stable Diffusion for inpainting, we extend the channel of the first convolution layer to concatenate latent $z=\mathcal{E}(x)$ of image $x$ with $\tilde{z_{t}}$.
This allows Diffree to generate images by denoising step by step from Gaussian noise concatenated with the latent of the input image.
At the same time, the denoising process is guided by the associated feature $\mathrm{Enc}_{\rm txt}(d)$ of text $d$ encoded through the CLIP text encoder [].
The network $\epsilon_{\theta}$ is optimized by minimizing the following objective function:

$$ $\displaystyle L_{\rm DM}=\mathbb{E}_{\mathcal{E}(\tilde{x}),\mathcal{E}(x),d, \epsilon\sim\mathcal{N}(0,\text{I}),t}\Big{[}\|\epsilon-\epsilon_{\theta}( \tilde{z_{t}},z,\mathrm{Enc}_{\rm txt}(d),t)\|_{2}^{2}\Big{]}.$ (2) $$

#### 3.2.2 OMP Module

and diffusion model are trained simultaneously and used to predict the binary mask $m$.
The OMP module comprises two convolutional layers, two ResBlocks, and an attention block, as illustrated in Fig. 6.
First, we calculate the predicted noise-free latent $\tilde{o_{t}}$ using the output of the diffusion model:

$$ $\displaystyle\tilde{o_{t}}=\frac{\tilde{z_{t}}-\sqrt{1-\bar{\alpha}_{t}} \epsilon_{\theta}(\tilde{z_{t}},z,\mathrm{Enc}_{\rm txt}(d),t)}{\sqrt{\bar{ \alpha}_{t}}}.$ (3) $$

Here, the concatenation of $z=\mathcal{E}(x)$ with $\tilde{o_{t}}$ serves as inputs to the OMP module.
The gradient of $\tilde{o_{t}}$ is detached to optimize the two models without affecting each other.
We conduct bilinear interpolation downsampling on the mask $m$ to obtain $m^{\prime}$, preserving its size identical to the input latent.
The OMP module’s network $\tau_{\theta}$ is optimized according to the following objective function:

$$ $\displaystyle L_{\rm OMP}=\mathbb{E}_{\mathcal{E}(\tilde{x}),\mathcal{E}(x),d, m}\Big{[}\|m^{\prime}-\tau_{\theta}(\tilde{o_{t}},z)\|_{2}^{2}\Big{]}.$ (4) $$

It is noteworthy that the OMP module can predict the mask through the reverse process of diffusion rather than after it, as $\tilde{o_{t}}$ is available at each step, enabling mask prediction in the initial steps, as illustrated in Fig. 7.

We train both the diffusion model and the OMP module simultaneously. Combining Eqs. 2 and 4, our final training objective can be expressed as follows:

$$ $\displaystyle L=L_{\rm DM}+\lambda L_{\rm OPS},$ (5) $$

where $\lambda$ is a hyper-parameter which balances the two losses.

Figure: Figure 7: Visualization of masks from OMP at different steps of diffusion inference process (total 100 steps). The mask of added objects can be acquired in the very beginning.
Refer to caption: x7.png

#### 3.2.3 Classifier-free Guidance

Classifier-free diffusion guidance [] is a method that involves the joint training of a conditional diffusion model and an unconditional diffusion model.
By combining the output score estimates from both models, this approach achieves a balance between sample quality and diversity.
Training for the unconditional diffusion model is achieved by fixing the conditioning value to a null variable intermittently throughout the training process.
We follow the approach of Brook *et al*. [] by stochastically and independently defining our input conditions $x$ and $d$ as null variables with a probability of 5%.

### 3.3 Evaluation Metric

Due to the absence of robust quantitative metrics for shape-free object inpainting except the success rate, we propose a set of evaluation rules leveraging exits metrics [, , , , ] to evaluate different methods in different aspects.

We first randomly select and manually inspect 1,000 evaluation data pairs from COCO [] and OpenImages [] independently to ensure the validity of the object in the image and generalizability of the evaluation dataset.
Each data pair comprises an original image $x_{\rm ori}$, a text prompt of an object $d$, and an inpainted image $x$.
The resulting output image $x_{\rm output}$ and the corresponding object mask $m_{\rm output}$ are outcomes derived from distinct methods.

#### 3.3.1 Background Consistency

We adapt LPIPS [], a widely adopted and robust metric for assessing the similarity between images, to evaluate this aspect:

$$ $\displaystyle s_{\rm con}\left(x,x_{\rm output},m_{\rm output}\right)={\rm LPIPS }\left(x,x\odot m_{\rm output}+x_{\rm output}\odot\left(1-m_{\rm output}\right )\right).$ (6) $$

Figure: Figure 8: GPT4V shows good distinguish ability in the reasonableness between objects.
Refer to caption: x8.png

#### 3.3.2 Location Reasonableness

Assessing the reasonableness of the object’s location is a challenging task due to its inherent subjectivity.
Surprisingly, we note GPT4V [] demonstrates strong discriminative abilities in assessing variations and evaluating different locations by providing $x$, $d$, $x_{\rm output}$ and an instruction $T$ as illustrated in Fig. 8. GPT4V rates the appropriateness of the object’s position on a scale from 1 to 5, while also providing justifications for these ratings:

$$ $\displaystyle s_{\rm rea}\left(x,x_{\rm output},d,T\right)={\rm GPT4V}\left(x, x_{\rm output},d,T\right)$ (7) $$

#### 3.3.3 Object Correlation

To quantify this relationship, we utilize CLIP Score [], a metric to assess the correlation between text and image, by calculating the cosine similarity of their embeddings from CLIP [].
we measure CLIP Score between the object area of $x_{\rm output}$ and $d$, which is referred to as “Local CLIP Score”:

$$ $\displaystyle s_{\rm cor}(d,x_{\rm output},m_{\rm output})={\rm CLIPScore} \left(d,\mathrm{Local}(x_{\rm output},m_{\rm output})\right).$ (8) $$

where $\mathrm{Local}(x,m)$ denotes obtaining a cropped region from $x$ using $m$.
To mitigate influences from background or mask shape, we compute an average of two Local CLIP Scores (one with background removal and another without).

#### 3.3.4 Object Quality and Diversity

Following [], we employ Local FID, measuring Fréchet Inception Distance (FID) [] on the local regions, to evaluate the quality and diversity of generated object:

$$ $\displaystyle s_{\rm qd}(LX_{\rm org},LX_{\rm output})=$ $\displaystyle{||\mu_{LX_{\rm org}}-\mu_{LX_{\rm output}}||}^{2}+$ (9) $\displaystyle\textrm{Tr}(\Sigma_{LX_{\rm org}}+\Sigma_{LX_{\rm output}}-2*( \Sigma_{LX_{\rm org}}*\Sigma_{LX_{\rm output}})^{\frac{1}{2}})$ $$

where $LX_{\rm org}$ and $LX_{\rm output}$ respectively denote the sets comprising all local regions of the original images and output images, $\mu$ and $\Sigma$ represent the mean and variance of the feature vectors obtained through a particular network [].

#### 3.3.5 Unified Metric

Drawing upon the evaluation metrics delineated above (Eqs. 6, 8, 7 and 9), we compute a unified score to holistically assess text-guided shape-free object inpainting.
We treat the derivative of inverse metric results (LPIPS and Local FID) as positive metrics and normalized the outcomes across different methods for each metric.
Ultimately, we average these normalized scores and multiply them by the success rate as a unified score.
The Unified metric not only considers success rate but also focuses on quantitative performances.

## 4 Experiment

We comprehensively evaluated our model, Diffree, by conducting experiments on two benchmark datasets: COCO [], and OpenImages []. Given the distinct input-output characteristics of our method compared to previous approaches, a quantitative comparison proves challenging. We align previous methods by adding auxiliary conditions, as depicted in Sec. 4.1.3, and provide quantitative comparison results (Sec. 4.2) to prove the effectiveness of Diffree more intuitively.
We then showcase visualizations of generated images and give corresponding analyses to offer an intuitive assessment of Diffree’s capabilities and comparisons in Sec. 4.3.
finally, we demonstrate some applications to prove that Diffree is highly compatible with existing methods (Sec. 4.4).

Figure: Figure 9: InstructionPix2Pix’s result statistics in object addition. (a): COCO, (b): OpenImages. Foreground Error denotes failing to add objects or transforming existing objects, Background Error denotes inconsistent background.
Refer to caption: x9.png

### 4.1 Experimental Settings

#### 4.1.1 Training Setups

we employ OABench to train Diffree, initializing the diffusion model with the Stable Diffusion 1.5 [] weights.
We set $\lambda=2$ in Eq. 5 and set a batch size of 256.
Our model was trained around 10K steps on 8 A100 GPUs.

#### 4.1.2 Evaluation Datasets and Metrics

As outlined in Sec. 3.3, we employ four metrics (LPIPS [], GPT4V [], Local CLIP Score and Local FID []) alongside the unified metric for evaluation on COCO [] and OpenImages [].

#### 4.1.3 Baselines

To facilitate comparison with prior methods [, ], we manually check and annotate the object masks for InstructPix2Pix [], and utilize our generated mask for PowerPaint [] for generation.
It is important to note that neither of these methods can complete evaluations independently.
thus, their quantitative metrics should be used as references only.

**Table 1: Main results on COCO and OpenImages. *: only calculate the successful cases’ results. †: use the masks from our Diffree as PowerPaint’s input.**
|  |  | InstructPix2pix [] | PowerPaint [] | Diffree |
| --- | --- | --- | --- | --- |
| COCO [] | Success rate | 17.4 | N/A | 98.5 |
| LPIPS $\downarrow$ | 0.11* | 0.06 | 0.07 |  |
| GPT4V Score $\uparrow$ | 3.13* | N/A | 3.47 |  |
| Local CLIP Score $\uparrow$ | 29.30* | 28.74 | 28.96 |  |
| Local FID $\downarrow$ | 156.25* | 58.08 | 57.43 |  |
| Unified Metric $\uparrow$ | 4.48 | 37.20† | 35.92 |  |
| OpenImages [] | Success rate | 18.9 | N/A | 98.0 |
| LPIPS $\downarrow$ | 0.11* | 0.06 | 0.07 |  |
| GPT4V Score$\uparrow$ | 3.36* | N/A | 3.50 |  |
| Local CLIP Score $\uparrow$ | 29.21* | 28.57 | 28.81 |  |
| Local FID $\downarrow$ | 143.82* | 62.40 | 60.07 |  |
| Unified Metric $\uparrow$ | 5.04 | 36.41† | 35.47 |  |

### 4.2 Main results

Tab. 1 shows the main results of Diffree with different baselines. We report the results of four powerful metrics and Unified Metric.
It is worth to highlight that only successful InstructPix2pix results are computed and PowerPaint is utilized for image inpainting under the masks provided by the results of our approach.
We can deduce the following conclusions from the results in several aspects.

#### 4.2.1 Success Rate

We achieved a success rate of over 98% on different dataset, while InstructPix2pix shows a lower success rate in object addition (17.2% and 18.9%).
As shown in Fig. 9, most of the results of InstructPix2pix involve replacing existing object, without adding or significant changes to the background.
This demonstrates our excellent ability to complete this task.
Meanwhile it is not applicable to PowerPaint as it necessitates a mask input.

#### 4.2.2 Consistency of Background

Diffree significantly outperforms InstructPix2pix in the LPIPS scores across all datasets (all decreased by 36% than InstructPix2pix).
In particular, only scores from carefully chosen successful cases of InstructPix2pix were computed, potentially leading to an overestimation.
Furthermore, Diffree, as a shape free inpainting method, yields LPIPS results comparable to PowerPaint, as a shape required inpainting method.
As discussed in Sec. 3.1, we expect achieving consistency of background like the image inpainting methods that necessitate masks.
These methods inherently excel on this aspect, given that their input and ground truth are the same image during the training process.
Therefore, we believe that we have a strong capability in this aspect.

#### 4.2.3 Reasonableness of object location

The results of GPT4V’s assessment demonstrate that Diffree has a considerable advantage in the reasonableness of object location (*e.g*., 0.34 higher than successful results from InstructPix2pix).
This is not avaliable for PowerPaint due to it requires a mask as input.

#### 4.2.4 Correlation, Quality and Diversity of Generated Object

We conduct an evaluation of the generated object across these three dimensions, utilizing both Local CLIP Score and Local FID.
Although Diffree exhibits a slightly lower Local CLIP Score in comparison to InstructPix2pix (*e.g*., 28.96 versus 29.30 on the COCO), this discrepancy can be rationalized by the fact that its successful results are inherently highly correlated while ours encompass all outcomes without any specific selection.
Intriguingly, we demonstrate superiority over PowerPaint in terms of correlation.
Furthermore, our performance according to the Local FID metric indicates a distinct advantage relative to all other methods.

#### 4.2.5 Unified Metric of Diffree

We combine the success rate with diverse metrics across various aspects to calculate a unified metric, thereby facilitating a more comprehensive comparison with extant text-guided methods.
It is discernible that Diffree exhibits a substantial superiority over InstructPix2pix, for instance, ours’ 35.92 as opposed to InstructPix2pix’s 4.48 on the COCO.
PowerPaint achieves superior results (*e.g*., 37.20 on the COCO dataset), the image inpainting of powerpaint buit upon the masks from our Diffree.
This further underscores the excellent scalability of Diffree when integrated with other methods.

Figure: Figure 10: Visualization of with/without mix with previous image using mask from OMP.
Refer to caption: x10.png

### 4.3 Visualization

We provide different types’ visualizations to more intuitively evaluate Diffree’s capabilities Figs. 1, 4, 3, 7, 10, 11 and 2, please refer to the respective image captions for detailed explanations. For more results, please refer to the appendix.

### 4.4 Application

Diffree can be well combined with other methods for more expansion.

#### 4.4.1 With GPT4V

GPT4V [] has a good ability to perceive and understand images, therefore we can use GPT4V for planning a object suitable for the image scene, seeing Fig. 11.
However, when task with adding corresponding object without altering the background, DALL-E-3 [] in GPT4, falls short.

#### 4.4.2 With Other Methods

AnyDoor [] can add a specific object to the designated area by providing a mask and object image. As depicted in Fig. 11, Diffree can combined with AnyDoor to further achieve adding a specific object to image.
DIffree also can effectively leverage the continuous progress in the image inpainting, to generate superior images, as demonstrated in Tab. 1.

#### 4.4.3 Iterative Operation

In Figs. 10 and 1, we present results of iterative inpainting.
Leveraging the predicted mask from OMP module, Diffree can preserve the image background from cumulative degradation during successive inpainting. This holds potential applications within architectural and interior design domains.

Figure: Figure 11: Applications combined with Diffree. (a): combined with anydoor to add a specific object. (b): using GPT4V to plan what should be added.
Refer to caption: x11.png

## 5 Conclusion

We propose a novel method, Diffree, that leverages a diffusion model with an object mask predictor for text-guided object addition.
Beyond the method, we build a high-quality synthetic dataset, OABench, through a novel data collection method for this task.
Diffree distinguishes itself by preserving background consistency without requiring additional masks, which solves shortcomings of previous text-guided and mask-guided object addition methods.
The quantitative and qualitative results demonstrate the superiority of our method.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: - How well does the model generalize? Since the model was trained on a synthetic dataset generated from an inpainting model, it is not surprising that the model may overfit the artifacts introduced by the inpainting model.
- In Line 468, the authors argue that their Local CLIP Score is slightly inferior to the InstructPixel2Pixel due to the latter considers only the successful results are taken into consideration. I wonder what the results will be if you combine both the successful and failure cases.
- Regarding the generalization problem, assuming that a real image contains two dogs, and I first use PowerPaint to delete the two dogs, and then let the proposed Diffree generate one dog, what will the output image show? Will the position of the new dog duplicate the position of the previously removed one?
- Were the compared methods (e.g., InstructPix2Pix) trained on the proposed dataset as well? or just directly copied their pre-trained weights for use?
soundness: 2
strengths: - Object inpainting using only text without relying on shape constraints.
- The authors built an OABench to facilitate text-guided object inpainting.
- The results are attractive and the supported applications are interesting.
summary: This paper proposes solving the object painting problem with only text as guidance and does not rely on other constraints, e.g., shape mask. The authors first construct a synthetic dataset, called OABench, to facilitate the model’s training with the help of the existing SOTA inpainting model. Then, a diffusion-based model named Diffree is proposed to train on this OABench that can simultaneously output the image with the added object and the object mask. Finally, the authors also conduct comparisons to validate the superiority of their method.
weaknesses: - Lack of ablation of the validation model design. For example, what happens to the output if the OMP is removed? BTW, integrating a mask head in the diffusion process is not new, e.g. in [1].
- The comparison is slightly unfair. In Figure-A10, comparing the model you trained on the curated dataset to other methods that were not retrained or fine-tuned is unfair.
- This paper is poorly written, especially the explanation of the charts. For example, in line 50 of the description section, the author provides links to three figures for the reader to view. However, these figures lack sufficient detail to explain what the figure depicts and what the symbols within the figures/captions represent.

[1]. Shadow Generation for Composite Image Using Diffusion Model–CVPR 2024

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. In Fig. 1, the "add dragon" prompt results in "a dragon painting in the frame". Is this behavior as expected?
2. How does Diffree perform with out-of-distribution (OOD) objects not included in OABench?
soundness: 3
strengths: 1. Diffree offers a user-friendly approach to inserting objects into images. The mask-free object insertion is particularly useful in practical applications.
2. The creation of OABench, a large-scale synthetic dataset, is a significant contribution, providing a rich resource for training and evaluating object addition models.
3. The OMP module's ability to predict the target mask and generate inpainting results simultaneously is a novel architectural advancement in this field.
summary: The paper presents a novel model, Diffree, which facilitates the addition of objects to images. The authors have constructed a synthetic dataset named OABench, consisting of 74K real-world tuples, to train Diffree using a Stable Diffusion model augmented with an additional mask prediction module. Diffree predicts the target mask and generates inpainting results simultaneously. Extensive experiments demonstrate Diffree's good performance in adding new objects with high success rates while preserving background consistency, spatial appropriateness, and object relevance and quality.
weaknesses: 1. [1] proposed a method for mask prediction closely related to Diffree. An in-depth analysis and comparison with this work would be beneficial.
2. All the prompts used in this paper are in the form "add {object}". It is unclear how Diffree generalizes to more precise control, such as "add a dragon in the room".
3. While user-friendly for object insertion, it restricts users from adjusting the mask. In standard image processing, users or designers often need to make adjustments to achieve their desired results.

[1] SmartMask: Context Aware High-Fidelity Mask Generation for Fine-grained Object Insertion and Layout Control

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. How does Diffree handle ambiguities in scenes with multiple entities (e.g., if there are two people in an image, how does it decide which one to add the object to)? Could user input be incorporated to clarify target placement in such cases?
2. Could more fine-grained prompts be incorporated or generated to enhance the model's ability to handle specific requests for object details and interactions within a scene?
soundness: 2
strengths: - **Originality**: Diffree’s approach of shape-free object addition guided solely by text is novel, significantly enhancing usability by eliminating the need for manual mask definitions. This innovation in user experience represents a unique contribution.
- **Clarity**: Overall, the dataset creation process, and evaluation metrics are well-described, with figures that aid understanding of Diffree’s operational and comparative performance.
summary: The paper introduces Diffree, a text-to-image (T2I) model aimed at text-guided object addition within images, a process where new objects are integrated into a scene while maintaining visual consistency in lighting, texture, and spatial orientation. Unlike previous approaches, Diffree avoids the need for manually defined masks, instead using a synthetic dataset (OABench) for training. OABench, composed of image pairs with and without specified objects, is created by advanced image inpainting techniques. Trained on OABench, Diffree incorporates a Stable Diffusion backbone with an Object Mask Predictor (OMP) module to predict object placement. Results indicate that Diffree achieves high success in preserving background coherence, achieving spatially plausible integration, and producing visually realistic objects.
weaknesses: - **Minor Typographical Issue**: There is a missing period at the end of line 101, which should be corrected for clarity.
- **Dataset Limitation in Prompt Detail**: Since the dataset primarily relies on the COCO dataset, prompts are often generic object labels rather than detailed, fine-grained descriptions. This limitation can hinder the model's ability to respond to nuanced or interactive prompts, such as requests for specific object attributes or context-based interactions.
- **Methodology Clarity**: The description of the proposed methodology, especially the Object Mask Predictor (OMP) module, could benefit from further clarification. For example, more detail on why the image latent and latent noise are concatenated as inputs for the OMP module during both training and inference phases would help improve comprehension.
- **Figure 6 Presentation**: Figure 6 needs a more detailed caption to allow readers to follow the workflow easily by reading only the figure caption. The current figure layout lacks clarity, making it challenging for readers to understand the process.
- **Inconsistency in Visuals (Figure 1)**: In Figure 1, example 2 appears visually inconsistent, with an unrealistic rendering of the added object. This raises questions about the model's object realism under certain conditions.
- **Lack of User Study**: Including a user study would strengthen the evaluation of the model, particularly to capture subjective aspects of user satisfaction and perceived realism in object integration.
- **Limited Comparative Evaluation**: The model is only compared against PowerPaint and InstructPix2Pix. Including additional comparison baselines, such as recent text-guided inpainting methods or general object addition approaches, would provide a more comprehensive assessment.
- **Ambiguity in Object Placement in Complex Scenes**: The method struggles with complex images involving multiple entities (e.g., two people). When required to add an object to one specific entity, the model may face ambiguity, as there is limited functionality for users to adjust the mask and specify the target placement.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Why not use Diffree to address mask-free removal? This should also be addressed by OABench.

2. What did "with/without mix with previous image" mean in Figure 8?
soundness: 2
strengths: 1. This paper proposed a sophisticated and reliable data collection process to achieve high-quality image removal tuples, which would potentially strengthen many other inpainting-based tasks. 

2. The authors show some promising and impressive object addition results in the appendix, including both real-world and AI-generated images.
summary: This paper proposes an object addition method called Diffree. Diffree comprises an object mask prediction (OMP) module, which could be used to generate masks through textual inputs. Thus Diffree can be regarded as a mask-free inpainting model. The authors also provide a synthetic dataset called OABench with 74k real-world object removal tuples built by PowerPaint. Extensive experiments verify the effectiveness of Diffree.
weaknesses: 1. As mentioned in the main paper, Diffree is trained to predict precise object masks, which is not convincing for both object removal and object addition. Because most objects contain shadows or reflections, which are not included in object masks. Without proper shadow and reflection generation, the inserted objects suffer from "copy and paste" artifacts. For example, no shadows are generated in the "Coconut palm" of Figure 3, and no reflections are generated in the "boat" of Figure A2. The metric of background consistency is also problematic. I think this is the main drawback of this method. 

2. Missing discussion and ablation about the necessity of the joint training of OMP and diffusion model. Why not independently train OMP to predict the mask.

3. The generalization of the Diffree should be concerned. As shown in Figure 9 (a), the generation failed while the mask was very small.

4. For the object addition guided completely by texts, the authors should provide more results to verify whether the proposed method could handle different orientational descriptions. For example, these prompts should be considered: "on left/right/up/bottom of...".

Meta-review: (a) This paper introduces Diffree, a diffusion-based model trained on the proposed synthetic OABench dataset for text-guided object painting without additional constraints, demonstrating superior performance through comparisons.

(b) Strengths: The paper presents a novel approach to object inpainting using only text guidance, eliminating the need for shape constraints and manual mask definitions, which significantly enhances usability. The creation of OABench, a large-scale synthetic dataset, is a major contribution, providing a valuable resource for training and evaluating object addition models. Diffree's user-friendly, mask-free object insertion method demonstrates strong practical applicability, supported by its innovative OMP module that predicts masks and generates inpainting results simultaneously. The results are visually appealing, and the well-documented dataset creation process and evaluation metrics ensure clarity and ease of understanding.

(c) Weaknesses: The paper has several weaknesses in the initial version, including a lack of ablation studies for validating model design, and insufficient comparisons with closely related works, particularly methods integrating mask prediction in diffusion models. The evaluation is criticized for unfair comparisons (e.g., using retrained models against non-fine-tuned baselines) and limited diversity in prompts, restricting the model's ability to handle nuanced or context-based interactions. The clarity of methodology descriptions, charts, and figures is lacking, with specific issues in explaining workflows (e.g., Figure 6) and inconsistent visual results (e.g., Figure 1). Additionally, the absence of a comprehensive user study and limited comparative evaluation against more recent inpainting or object addition methods reduce the strength of the paper's claims, while the inability to handle complex scenes or provide mask adjustments limits practical usability.

(d) The most important reasons for reject are: Although this work does show potential, the authors present too many flaws in the paper, which can easily lead to misunderstandings and needs significant refinement. E.g., Figure 1 is intended for stress-test, it is ok but could you please change a case that would not be challenged by the visual consistency issue? How effectively a paper minimizes controversy is also a key measure of its quality. Furthermore, the authors were overly passive during the rebuttal process and should have proactively provided relevant and necessary information in many cases (GPT-4O relabelling, user study, etc.) to convince readers and reviewers. E.g., as for the user study problem, providing the all the details for how to conduct the user study is very important at the first presentation. Although the authors provided additional details in response to the reviewers' further requests, the AC found that it remains unclear how many people participated in the user study and what their relationship is to the 1,000 cases. Were multiple people involved in evaluating a single case to avoid bias? 

The AC encourages the authors further minimize controversy and improve the overall quality. However, based on current states and rebuttal/discussion process, the overall evaluation still leans towards rejection. Sorry for this and wish the authors good luck in the next upcoming conference. Your valuable work will eventually be recognized by the community; perhaps it just needs a little more patience.

Decision: Reject