CLIPAway: Harmonizing focused embeddings for removing objects via diffusion models
Authors: Yiğit Ekin
Ahmet Burak Yildirim
Erdem Eren Caglar
Aykut Erdem
Erkut Erdem
Aysegul Dundar
Venue: NeurIPS 2024 poster (2024)
Keywords: Object Removal
Inpainting
Diffusion models
Primary area: diffusion_based_models
TL;DR: A training-free, plug-and-play method that uses CLIP embeddings to focus on background areas for seamless object removal in diffusion models without requiring specialized datasets.

Abstract: Advanced image editing techniques, particularly inpainting, are essential for seamlessly removing unwanted elements while preserving visual integrity. Traditional GAN-based methods have achieved notable success, but recent advancements in diffusion models have produced superior results due to their training on large-scale datasets, enabling the generation of remarkably realistic inpainted images.
Despite their strengths, diffusion models often struggle with object removal tasks without explicit guidance, leading to unintended hallucinations of the removed object. To address this issue, we introduce CLIPAway, a novel approach leveraging CLIP embeddings to focus on background regions while excluding foreground elements. CLIPAway enhances inpainting accuracy and quality by identifying embeddings that prioritize the background, thus achieving seamless object removal. Unlike other methods that rely on specialized training datasets or costly manual annotations, CLIPAway provides a flexible, plug-and-play solution compatible with various diffusion-based inpainting techniques.

CLIPAway: Harmonizing Focused Embeddings for Removing Objects via Diffusion Models
Yigit Ekin Ahmet Burak Yildirim', Erdem Eren Caglar',
Aykut Erdem Erkut Erdem* Aysegul Dundar
t Department of Computer Engineering, Bilkent University, Ankara, Turkey Department of Computer Engineering, Koç University, Istanbul, Turkey KUIS AI Center, Koç University, Istanbul, Turkey Department of Computer Engineering, Hacettepe University, Ankara, Turkey yigit.ekin bilkent.edu.tr
Correspondence:
Abstract
Advanced image editing techniques, particularly inpainting, are essential for seam- lessly removing unwanted elements while preserving visual integrity. Traditional GAN-based methods have achieved notable success, but recent advancements diffusion models have produced superior results due to their training large-scale datasets, enabling the generation remarkably realistic inpainted images. Despite their strengths, diffusion models often struggle with object removal tasks without explicit guidance, leading to unintended hallucinations of the removed object. To address this issue, we introduce CLIPAway, a novel approach leveraging CLIIP embeddings to focus on background regions while excluding foreground elements. CLIPAway enhances inpainting accuracy and quality by identifying embeddings that prioritize the background, thus achieving seamless object removal. Unlike other methods that rely on specialized training datasets or costly manual anno- tations, CLIPAway provides a flexible, plug-and-play solution compatible with various diffusion- based inpainting techniques. Code and models are available via our project website: https://yigiteknn. wwweeeiiiiiiiuuiiioo. github.io/CLIPAAa.
Introduction
In today's digital era, the demand for sophisticated image editing techniques has surged, with inpainting emerging as a fundamental method for seamlessly removing unwanted elements from images while maintaining visual coherence. Image inpainting has long been studied in both academia and industry. Traditionally, research has predominantly focused on Generative Adversarial Network (GAN)-based methods (2 17
26 32). which have shown notable success in inpainting tasks. However, recent advancements in diffusion models have attracted considerable interest due to their ability produce high-quality results (24 28 31). A key factor behind the effectiveness of diffusion models is their extensive training on large-scale datasets. By leveraging comprehensive collections of diverse image data, diffusion models can learn complex patterns and correlations, and intricate details, allowing them to inpaint missing regions with exceptional realism.
12 16. 14
Despite their strengths, diffusion-basee text-guided image inpainting models [28 challenges in object removal tasks without explicit guidance. When tasked with object removal without explicit text inadvertently hallucinate the removed object, substituting it instead of erasing it entirely. This issue contrasts with user expectations, as users typically anticipate the erased portion to be seamlessly filled with the background. For example, when removing a person on a surfboard (Figure [ second row), diffusion model might insert another person, given the context that surfboards often have people on
1 often encounter
insert a replacement with the text of "background"" these models may
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input
Mask
SDDInpaint CLIPAway
Input
Mask
SD-Inpaint CLIPAway
Figure I: Diffusion-basd inpainting methods often struggle with object removal tasks. Instead of seamlessly filling the erased area with background elements, diffusion models may unintentionally replace the removed object with another or add irrelevant objects. This outcome diverges from the user's s intention, which is typically to restore the area with the background alone, without introducing new elements. Our method, CLIPAway, aims at amending this deficiency by precisely focusing maintaining the integrity of the background, ensuring that the space is filled as intended by the user.
them. Similarly, when removing plane from image of airport field (Figure 1 fourth row), model might insert another plane due to the presence of other planes in the background. Alternatively, model might introduce shoes the floor (Figure [ first row) when the user's intention
fill the space with background elements. In this paper, to address this issue, we propose a novel approach distinguish between foreground (the object to be removed) and background regions. Our method aims to identify an embedding for the inpainted region that focuses on the background while excluding the foreground content, thereby enhancing the quality and accuracy of the inpainting process.
called CLIPAway that leverages AlphaCLIP (25] embeddings
Recent advancements in diffusion fraded inpainting for object removal have yielded notable works such as InstInpaint [31], MagicBrush [34], InstructPix2Pix [3] and concurrently ObjectDrop (27). These methods introduce training datasets with varying approaches: Some generate targets through existing inpainting methods (31. resulting imperfect targets, while others rely on synthetically generated pairs that may contain annotation errors. Certain methods resort costly manual annotations 34 or require extensive data collection setups where images of scenes before and after object removal are captured [27). Furthermore, these techniques typically involve either training model from scratch or fine-tuning existing models specifically for the removal task. In contrast, our model, CLIPAway, distinguishes itself by its lack of dependency a specialized training offers a plug-and-pla solution compatible with various diffusion-basee inpainting methods, ensuring seamless object removal without the need for costly or complex data preparation.
Our contributions can be summarized as follows:
We introduce CLIPAway, a method that utilizes AlphaCLIP embeddings to effectively differentiate between foreground and background regions for superior object removal.
Our approach offers simple plug-and-play solution that does not require specialized training datasets, making it adaptable to various diffusion-based inpainting methods.
By focusing background regions, CLIPAway significantly improves the quality and accuracy of inpainting results, avoiding the common issue of object hallucination.
We provide comprehensive evaluations on a standard dataset, demonstrating consistent improvements over state-of-the-art methods.

---

2
or extensive data collection setups Related Work
Image inpainting involves replacing missing pixels in an image with new ones that blend scamlessly with the surrounding content. Historically, Generative Adversarial Networks (GANs) have dominated this field where they demonstrated significant success across various image domains [21 17 33 12
[2]. However, GAN-based models are typically trained separately for specific image domains, such face inpainting using training datasets like FFHQ [10] scenery inpainting using datasets like Places [37]. This domain-specific training restricts their ability to generalize to diverse scenes, thereby limiting their versatility.
Recently, diffusion-based models have made strides, showing promising results 18] 24 For in- stance, the Repaint model [18] employs pretrained unconditional diffusion model to perform image inpainting by conditioning the gencration on the uncrased parts of the image. Despite its effective- ness, Repaint operates on the image space, thus computationally demanding and slow. Alternative approaches that work in the latent space, e.g. SD-Inpaint [24] and Blended Latent Diffusion [Л) adapt the Stable Diffusion (SD) model by adding a mask channel to the latent inputs. These methods, however, often introduce new objects into the scene based on context rather than removing existing ones, confficting with the user's intention of background restoration. Other diffusion- bbasd methods, such as GLIDE 20 and SmartBrush [28] are designed to add objects rather than remove them.
There has been growing interest in instruction-based inpainting methods for object removal, such as Instruct-Pix2Pig [3] and Inst-Inpaint [31], which use prompts instead of masks for object removal. These methods require datasets specifically tailored for this task. For example, Instruct-Pix2Pix generates paired datasets using the GPT-3 language model 4 and the text-to-image Stable Diffusson model [24] incorporating
techniques 66). While capable of object removal, Distuct-Pix2PPt performs this task with limited precision, possibly due to the synthetic data's 's lack of diversity inaccurate annotations. Inst-Inpaint the other hand, is trained with paired data where targets are inpainted images generated with GAN-based models. Hence, it inherits the artifacts of these GAN models.
Other works have relied on manual annotation efforts involving scenes captured with and without the object 27 However, the high cost of manual annotations limits the scale of these datasets. In contrast, our method, CLIPAway, sets itself apart by eliminating the need for specialized training sets. It offers a flexible, plug-and-play solution compatible with various diffusion- inpainting methods, ensuring seamless object removal without the necessity for costly complex data preparation.
3 Method
3.1 Preliminaries
Our framework leverages pretrained diffusion models, particularly the latent diffusion model [24]. chosen for its computational efficiency. This model includes encoder (F) and decoder (D). The encoder compresses images into a lower-dimensional latent space while the decoder reconstructs images from these latent codes. These components function similarly a variational autoencoder and are trained separately from the diffusion process.
The diffusion process, as described by [8), operates latent codes, denoted as zu E(x), where is the input image. Noise is gradually added to 20 over a series of time steps until, after T steps, 2r approximates normal distribution with zero mean and an identity covariance matrix.
Diffusion models act as denoising autoencoders, trained to reverse the noise addition process. They aim to predict a denoised version of their input, 2t, where 2, is a noisy version of zg. The objective function for this denoising task on the latent codes is defined as follows:
LIDD == Ee ca(2,, ()]]
Here, t is sampled from the range to T, and co(2). represents a neural network, specifically UNet that predicts the noise added to 2,, conditioned on the time step t. We specifically employ models fine-tuned for inpainting tasks. These methods involve adding a single-channel mask, which is downsampled to fit the latent space, to the denoising UNet. The diffusion models are commonly trained using text-image pairs, where the text information extracted from a frozen CLIP text encoder [23]. This encoded text data is then integrated into the UNet via attention layers.
3

---

image prompt
image prompt
text prompt "ackground"
IP-Adapter SD-Inpaint
IP-AAaater SD-Inpaint
IP IP-Adopter
SD SD-Irpaint
input
output
input
output
input
output
Figure 2: Limitations of IP-Adapter [30] for Inpainting. Direct of the IP-Adapter with the input image as the image prompt is ineffective for inpainting, as it predictably fills the erased area with the original object. In addition, directly giving the prompt "background" is also problematic as the background can also contain instances of the images that we want to remove, resulting in direct replacement of the foreground object. On the other hand, using an erased image the prompt results in the generation of black artifacts.
3.2 CLIPAway
Our objective is to seamlessly remove objects while maintaining the integrity of the background. Unlike conventional impainting methods that ignore the pixels from the erased area, our approach utilizes these pixels to guide the model on what not to fill in. This distinguishes our method significantly from others. To achieve this, we exploit the detailed pixel-level information available from both the regions to be erased and the unerased regions of the image. While popular Stable- Diffusion models typically rely solely on text conditioning, we explore conditioning the inpainting process embeddings derived from image pixels.
Recent advancements have introduced additional control signals via adapters, addressing the limi- tations of text in fully expressing desired outcomes. In some cases, edge maps, poses, or reference images are necessary effectively control the generation process 35 36 19]. The CLIP model utilized for text embedding is originally trained with contrastive objective jointly with CLIP image encoder. Adapters have demonstrated that rather than training an image encoder from scratch for reference image-based control, the existing CLIP image encoder can be utilized. UniControl [36] and T2IAdapter [19] extract features from the CLIP image encoder, map them to new features via a trainable network, and concatenate them with text features. These merged features are then fed into the UNet of the diffusion model to guide the image generation process. IP-Adapter (30] further shows that instead of merging image and text features in the cross atteentttnn layer, the features can pass through a small trainable projection network, which are then fed into the UNet via a decoupled attention layer. Our implementation is based on IP-Adapter but can be used with others as well 36
Our goal is to achieve inpainting by focusing on the background. However, directly using the IP- Adapter with the input image proves ineffective. Using the entire image the reference (prompt image), the method predictably fills the erased area with the original object (Figure left pane). Conversely, erasing the input image results black pixels in the masked area, leading to black artifacts in the filled regions (Figure 2 middle pane). Lastly, providing text prompt "background", also does not help in removing the object (Figure right pane). Therefore, we need an embedding that solely focuses on the background. To address this, we explore AlphaCLIP (25), which achieves region focus without altering the original image by incorporating regions of interest through an additional alpha channel input. Although it is initialized with the CL.IP (2 model, its training requires a substantial set of region-text paired data. By utilizing the Segment Anything Model (SAM)
[1I and multimodal large model BLIP-2 [13] for image captioning. millions of region-text pairs generated. AlphaCLIP model is pretrained on a mixture of region-text pairs and image-texi pairs. Their dataset has not been released; fortunately, set. Instead, we leverage existing models and techniques to achieve our inpainting objectives.
method does not require specialized training
We train a Multi-Layer Perceptron (MLP) model to adapt the publicly released AlphaCLIP image encoder (CLIP-L/14) to the CLIP image encoder used in the IP-Adapter (OpenCLIP ViT-H/14). The MIP consists of six blocks, each containing linear layer, layer normalization, and GELU activation. begins with 768 features and outputs 1024 features, matching the output dimensions of the CLIP-L/14 encoder and the OpenCLIP VIT-H/14 encoder, respectively. For this training, use

---

Fa Fo-focused embedding
AlphaCLIP
+ MLP
LMSE
CLIP encoder
Fg- focused embodding
Bi focused embbddding
projectiza
Finol ambbddinng
mask
Bg focused embedding
embedcing Projection Block
Training
Inference
Encodee mashed image
IP
Adopter
Alphoc_Es encoder
MLP
image
Projection Block
Zt
SD
Irpaint
AlphaCLIP
MLP
Zo
messk
image (1-mask)
CLIPAway
concateration
Figure 3: The overall framework of CLIPAway. Input images, comprising both foreground and background elements, are embedded via AlphaCLIP. These embedded images are then processed through an MLP trained to adapt features the IPAdapter input space. Through vector arithmetic on the features, background embedding without foreground influence is achieved. SDInpaint is depicted as if is working on image space for clarity; works on latent space.
the COCO image dataset with the alpha channel set to all 1s, corresponding to the full image rather than focusing on a specific region for AlphaCLIP. This setup aligns with AlphaCLIP training. where the authors occasionally set alpha channels all to indicate full images and sometimes to local regions. In our training, the target is the OpenCLIP embeddings for a given image. This allows us to train a projection layer so that AlphaCLII outputs features that the rest of the IP-Adapter expects. This part is shown in Figure 3 (Training). We show that AlphaCLIP can be aligned with other CLIP Image encoders without a special dataset.
One of the promises of AlphaCLIP is its ability to focus on a specific region while maintaining contextual awareness. For example, given an image and mask pointing to the background. it may primarily focus on the background while still encoding the foreground, albeit with reduced emphasis. This behavior illustrated in Figure 4 where we use image prompts as input images with alpha channels corresponding to the mask for foreground focus and the inverse of the mask for background focus. The results shown incorporate our projection layer, which bridges the AlphaCLIP and IP- Adapter. The first row displays the conditional image generations, while the second row shows the inpainting results. When the foreground focused, the foreground object appears more prominent. Conversely, when the background is focused, the foreground object is present but receives less attention. Therefore, even when background focused, the inpaintings still include the object one aims remove. To remove the foreground overall, propose to subtract the foreground embedding from the background via projection.
Given two vectors eb (background focused embedding) and er (foreground focused embedding), the final embedding Ciiua can calculated as the equation below:
eb ef
er
eb
(Ter
Terl
(2)
where Op Cr the dot product Cb and cr, and ||or|] is the norm cy. With this vector arithmetic. we find the final embedding that is orthogonal to the foreground embedding. After performing this subtraction, the embedding process predominantly focuses on the background, as illustrated in Figure i in our results. This tendency is evident in conditional image generation, where the resulting image predominantly exhibits the background style. Consequently, this translates into consistent background filling in the inpainting task for erased areas.
5

---

Conditional Image Generation Results
Image
FGi-Focused
BG-Focused
Ours
Image
FG-Focuued
BG-Foocused
Ours
Buvus
Image
FG-Focused
BG-Focused
Ours
Image
FG-Focused
BG-Focused
Ours
Conditional Image Inpainting
Mask
FG-Focused
BG-Focused
Ours
Mask
FG-Focused
BG-Focused
Ours
Mask
FG-Focused
BG-Focused
Ours
Mask
FG-Focused
BG-Focused
Ours
Figure 4: Starting with an input image and mask, we present our findings utilizing both foreground and background-focusee embeddings. The images the first row depict conditional image generation outcomes of the stable-diffusion model without the inpainting task. These visuals offer insights into the focus of the embeddings. While both embeddings capture features from various parts of the image, the foreground embedding tends to emphasize the foreground, whereas the background embedding predominantly focuses on the background but still contains the foreground. Our approach successfully removes the foreground in the generated results, yielding pure background. This outcome consistent with the image inpainting outputs, as demonstrated in the second row.
The overall framework is depicted in Figure 3 (Inference). Input images containing both foreground and background elements are embedded via AlphaCLIP. These embedded images are then fed into the MLP, which we trained to adapt the features to the IP-Adapter input space. By performing vector arithmetic on the features, we achieve a background embedding without the influence of the foreground. Notably, this method does not necessitate object removal dataset and can be readily utilized as a plug-and -nd-play feature.
Experiments
4.1 Baselines
We compare our method with state-of-the-art GAN-based and diffusion-based inpainting methods. The GAN based methods include ZITS++ [5) MAT [4), and LaMa (26) models, whereas diffusion
based models include Blended Latent Diffusion (.) Unipaint [29 SD-Inpaint
[24] We use the
models released by the authors that achieve the best scores.
For GAN- models, we use the models that are trained on the Places2 dataset. For the diffusion models, we provide an empty prompt. We also experiment with providing prompts as "background" but that does not change the results. In our experiments with Unipaint combined with CLIPAway, the masking mechanism proposed Unipaint is applied to the projected embeddings our network and then fed into the UNet with the help of IP-Adepter. To demonstrate the flexibility of our method across different CLIP and AlphaCLIIP embedding spaces, we also present comparative analyses of our approach when the projection step is applied both before and after the MLP layer. This comparison

---

employs three distinct AlphaCLIP backbones VVT-LL14, ViT-L/14@336px, and ViT-B/16 by training an MLP layer tailored each evaluated backbone.
4.2 Datasets
We evaluate the models on the COCO 2017 validation dataset [5]. which provides us with the collection of images indoor and outdoor and instance level annotations. For each image in the validation set, we set the masks that correspond to object instances, excluding stuff categories. We dilate the masks with kernel size of 5 sometimes the pixels from object remain in the image and result in artifacts for all models.
4.3 Metrics
We report the Frechet Inception Distance (FID) 77. Kernel Inception Distance (KID) [2] and the CLIP Maximum Mean Discrepancy (CMMD) metries [9] to assess the photorealism the generated images by comparing the source image distribution with the inpainted image distributions. However, these metrics do not evaluate the object is correctly removed.
To measure the accuracy of correct object removal, we use the CLIP metrics proposed by Inst- Instinn 31, namely CLIP Distance and CLIP Accuracy. The goal CLIP Distance evaluate how well the target object is removed. We extract the image regions indicated by bounding boxes from both the source and the inpainted images, then estimate the CLIP similarities 22 between these regions. We expect larger distance if the object is correctly removed. For CLIP Accuracy, we utilize CLIP as a zero-shot classifier. We identify the most likely semantic label of the image region extracted from the source image using the object's bounding box, considering the object classes in our dataset. Next, we perform another prediction for the image region extracted from inpainted image. We expect the class prediction to change after performing the object removal operation. If the predicted class based on the source image is not in the Top-1, Top-3, or Top-5 predictions the inpainted image, considered success. We report the percentage of successes.
We also conduct user study with participants on the first samples of the validation set. We include ZITS++, Unipaint, SD-Inpaint, and CLIPAway to provide a range of the best- best-performing models. Participants are asked to evaluate whether the object is correctly removed and to assess quality of the inpainting, and then select the best result among all choices. Details of the study are given the Appendix.
4.4 Results
We provide quantitative and qualitative comparisons of our method with state-of-the-arr inpainting models in Table [ and Figure 6 respectively. Our approach, when combined with various diffusion-based inpaint- ing methods, demonstrates consistent improvements in both FID and CLIP metries. Figure iillustrates that CLIPAway (with SD- SD-Inpaint) achieves significantly bet- ter CLIP@3 and FID scores, positioning it in the top left corner, indicating more accurate and higher quality results compared to other methods. Additionally, our user study shows a strong preference for our approach over competing methods. Human subjects preferred method 7 1..75% of the time, compared to 13 13.25% for ZITS++, 11.25% for SD-Inpaint, and 3.75% for Unipaint.
Scatter Plot of FID vs CLIP@3 Scores
80
ZTTS+ MAT LaMa Blended Unipuint
70
FD
SD Inpaint
CLIPAway
69
50
80
90
100
CLIP Score
Figure 5: Comparison of CLIPAway with state-of-the-an methods based on image quality and inpainting accuracy.
few predictions from competing models are presented in Figure GAN-based models, namely LaMa, MAT, and +ITS++, do not exhibit issues with object insertion. However, this problem evident in the methods. This may be because diffusion models are more powerful and
outputs of diffusion-basee better at modeling data distribution, leading them to generate objects that more closely match the real distribution. Although GAN-based models avoid object insertion object insertion, they fail to
7

---

Models
FID
KID
CMMD
CLIP Dist.
CLIP@1 f
CLIP@31
CLIP@51
Z 5
67.72
61.31
MAT 14
63.39
0.0278
0.93
0.76
80.62
65.85
60.10
LaMa 26
65.76
0.0195
0.81
0.66
78.34
64.42
56.85
SD-Inpaint LaMa
51.33
0.0117
0.45
0.75
72.29
57.61
50.01
Blended Diff. 1
72.24
0.0362
0.89
0.85
85.69
75.01
69.
+ CLIPAway
61. 61.66 (-10.58)
0.0194
(-0.0168)
0.78 (-0.11)
0.83 (-0.02)
87.28 (+1.59)
78.8 78.87 (+3.86)
73.20 (+3.86)
Unipaint
77.58
0.0360
0.98
74.48
67.44
+ CLIPAway
62. 18.18 (-15.40)
0.0199 (-0.0161)
0.79 (-0.19)
0.84 (+0.06)
88.25 (+2.88
78.6 28.65 (+4.17)
73.05 (+5.61)
SD-Iopaint [4
59.21
0.0145
0.54
0.75
70.45
57.14
49.88
CLIPAway
57.32 (-1.89)
0.0108 (40.0037)
0.53 (-0.01)
0.81
84.82 (+14.37)
74.42 (+17.28)
67.76 (+17.88)
Table I: Evaluation results.
Improvements of CLIPAway over the base models are given in
parenthesis for each metrie.
generate realistic backgrounds. Our method, CLIPAway, is the only one that effectively removes objects and fills the regions realistically. For example, in the first row, GAN-based models fill the inpainted area in a blurry way, and diffusion models insert person. In contrast, our method removes the third example, our approach the only one that
the person and fills the area seamlessly. realistically fills the kitchen background. Similarly, in the eighth example, while GAN-based models extend the object in a blurry way and diffusion models add unrelated content, our method provides sharp and realistic result.
Table iillssrates the adaptability of approach across various CLIP and AlphaCLII embedding spaces. Our method is not constrained to the CLIP embedding spaces employed in our initial experiments; with different backbones, the results consistently enhance the performance of the SD-Inpaint model across diverse CLIP embeddings. Additionally, our projection method is applicable beyond the OpenCLIP embedding space. Since AlphaCLIP's vision transformer is trained with objectives similar to those of the CLIP vision transformer, the resulting feature spaces are conceptually aligned. Consequently, projections can be effectively performed in the AlphaCLIP embedding space or in other CLIP embedding spaces with comparable properties. To validate this, we evaluated the projection method on the AlphaCLIIP feature space (projection on AlphaCIIIP space followed by MLP) using the same experimental setup.
Models
FID 1 CMMD 1
CLIP Dist.
CLIP@1
CLIP@31
CLIP@5
SD-Inpaint
59.21
0.54
0.75
70.45
57.14
49.88
projection before MLP
SD- 1D-Inpaint- CLIPAway (VIT L/14)
24
$7.32
0.53
0.81
84.82
74.42
67.76
SD 3D-Inpaint CLIPAway (VIT l/14
24
54.93
0.48
0.80
82.36
71.68
63.28
3D-Inpaint- CLIPAway (VIT-B/16
55.31
0.48
0.78
83.57
72.44
projection after MLP
SD Inpaint CLIPAway (VIT L/14)
24
56.15
0.42
0.86
85.31
74.26
68.58
SD Inpaint CLIIPAway (VIT //140333pp
24
54.46
0.36
0.82
82.13
70.02
63.28
Inpaint CLIPAway (VTT B/16)
$4.99
74.79
Table 2: Evaluation results. SD-Inpaint and SD-Inpaint CLIPAway across different CLIP embed- ding configurations, with projections evaluated before and after MLP.
Figure 7 shows our method integrated with various diffusion-based inpainting techniques, highlighting the significant performance enhancement our module offers. In Figure we present qualitative comparisons between our method and instruction based diffusion models. Since our method requires mask and these models require prompts, this is not a direct one-to-one comparison. The visual results reveal that Instruct-Pix2Pix B struggles to accurately remove object, and Inst- Insa-Inpaint [I] produces blurry inpainted images due to its training with GAN-generated targets. important to note that these competing methods rely on specialized datasets and specific model training for this task. In contrast, our method does not necessitate a special dataset and can seamlessly integrate with existing diffusion models.

---

Image
Mask
LaMa
MAT
ZIT
Blended
Unipaint
SDInpaint
CLiPAway
Diffusion models often replace the removed object or insert new content instead of simply removing it. baseb models avoid adding new objects but struggle to generate realistic backgrounds. Our method, CLIPAway, effectively removes objects and fills the regions with realistic background content.
Figure
Image
Mask
Blended
CLIPAway
Unipaint
+ CLIPAway
SDInpaint
+ CLIPAway
Figure 7: Qualitative results of biffusion-based models and our method combined with them.
5 Conclusion and Broader Impacts
In this paper, we introduced CLIPAway, novel approach for object removal in images using diffusion- based inpainting methods. Specifically, CLIPAway addresses the common issue of diffusion models hallucinating removed objects by focusing embeddings on the background. Our method leverages AIphaCL.IP embeddings to effectively distinguish between foreground and background regions, focusing background restoration to achieve seamless and realistic inpainting. By climinating

---

Image
Instruct Pix2Pix
Inst Inpaint
CLIPAway
Image
Instruct Pix2Pix
Inst Inpaint
CIiiAwway
Remove the woman on the right
Remove the man middle
Remove the bear the left
Remove the suitcase at the bottom
middle
man middle
Figure 8: Qualitative results comparing instruction-basee diffusion models with our method, which utilizes a mask for object removal.
the dependency specialized training datasets, CLIPAway provides flexible, plug-and woug-and-play solution compatible with various diffusion-based inpainting techniques. Our extensive experiments demonstrated that CLIPAway significantly improves the quality and accuracy of inpainting compared to state-of-the-art methods.
Broader Impacts. Our framework has the potential image restoration. editing, and completion. However, this technology also brings important ethical considerations. One potential misuse is the alteration falsification of visual content, leading to the creation of misleading deceptive images. We do not endorse such activities and emphasize the necessity establishing safeguards to ensure the ethical use of this technology.
Limitations. Our method demonstrated using latent-based diffusion models. A limitation is that, despite the diffusion occurring in the latent space, these models are still slower than GAN-based methods and do not run real-time. Another limitation of our model is the degradation performance when the preferred resolution is not used. For instance, when utilizing the SD-Inpaint pipeline the expected resolution for inference without quality loss is 512x512. provided, performance degrades. While this issue is inherent to based diffusion models rather than specific to our approach, it remains a present limitation. Additionally, our model, while removing objects, does not remove their shadows if they are not included in the mask. This can be seen in Figure in the first two examples on the right, where shadows are handled as patterns by the model instead of being removed.
different resolution is
Acknowledgments
This work was supported in part by KUIS Al Research Award. We thank all the reviewers for their valuable comments.
10

---

References
[1] Avrahami, O., Fried, O., Lischinski, D.; Blended latent diffusion. ACM Trans. Graph. 42(() (jul 2023).
https//445...ttttss/////... /tt.::/:://ol..oog//0111//////// 1ttp://0..114//39245555
[2] Binkowski, M., Sutherland, D.J., Arbel, M., Gretton, A.: Demystifying mmd gans (2021),
fttpp:://axxiw.org/aaaaap
[3] Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402 (2023)
[4] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877-1901 (2020)
[5] Cao, C., Dong, Q., Fu, Y:: Zits++: Image inpainting by improving the incremental transformer on structural priors. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)
[6] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208.011 221622008206666 (2022)
[7] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems 30 (2017)
[8] Ho, J.. Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33, 6840-6851 (2020)
[9] Jayasumana, S., Ramalingam, S., Veit, A., Glasner, D., Chakrabarti, A., Kumar, S.: Rethinking fid: Towards a better evaluation metric for image generation. arXiv preprint arXiv:2401.09603 (2023)
[10] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4401-4410 (2019)
[11] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint XXXv:23004002433 (2023)
[12] Li, J., Wang, N., Zhang, L., Du, B., Tao, D.: Recurrent feature reasoning for image inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp.
7760-7768 (2020)
[13] Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-iman pre-training with frozen image encoders and large language models. In: International conference machine learning. pp. 19730-19742. PMLR (2023)
[14] Li. W., Lin, Z., Zhou, K., Qi, L.. Wang, Y., Jia, J.: Mat: Mask-aware transformer for large hole image inpainting. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10758-10768 (2022)
[15] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D... Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740-755. Springer (2014)
[16] Liu. G., Dundar, A.. Shih, K.J., Wang, T.C., Reda, F.A., Sapra, K., Yu, Z., Yang, X., Tao, A., Catanzaro, B.: Partial convolution for padding, inpainting, and image synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)
[17] Liu, G., Reda, F.A., Shih, K.J. Wang, T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions, In: Proceedings of the European conference on computer vision (ECCV), pp. 85-100 (2018)

---

[18] Lugmayr, A.. Danelljan, M.. Romero, A., Yu, F., Timofte, R., Van Gool, L.: RePaint: Inpainting using denoising diffusion probabilistic models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1 11461-1177 (2022)
[19] Mou, C., Wang, X., Xie, L., Wu, Y., Zhang, J., Qi, Z., Shan, Y:: 12i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 4296-4304 (2024)
[20] Nichol, A.Q., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., Megrew, B., Sutskever, I., Chen,
M.: Glide: Towards photorealistic image generation and editing with text-guided diffusion models, In: International Conference on Machine Learning. pp. 16784-16804, PMLR (2022)
[21] Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: Proceedings the IEEE conference on computer vision and pattern recognition. pp.
(2016)
[22] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry. G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference Machine Learning. Proceedings of Machine Learning Research. vol. 139, pp. 8748-8763, PMLR (18-24 Jul 2021)
[23] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748-8763. PMLR (2021)
[24] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10684-10695 (2022)
[25] Sun, Z., Fang, Y., Wu, T., Zhang, P., Zang, Y., Kong, S., Xiong, Y., Lin, D., Wang, J.: Alpha-clip: clip model focusing on wherever you want. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)
[26] Suvorov, R., Logacheva, E.,, Mashikhin, A., Remizova, A., Ashukha, A., Silvestrov, A., Kong, N., Goka, H., Park, K., Lempitsky, V.: Resolution-robust large mask inpainting with fourier convolutions. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 2149-2159 (2022)
[27] Winter, D., Cohen, M., Fruchter, S., Pritch, Y., Rav-Acha, A., Bootstrapping counterfactuals for photorealistic object removal and insertion. arXiv preprint arXiv:2403.1 18818 (2024)
Hoshen, Y.: Objectdrop:
[28] Xic, S., Zhang, Z., Lin, Z., Hinz, T., Zhang, K.: Smartbrush: Text and shape guided object inpainting with diffusion model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp.
(2023)
[29] Yang, S., Chen, X., Liao, J.: Uni-paint: A unified framework for multimodal image inpainting with pretrained diffusion model. In: Proceedings of the 31st ACM International Conference on Multimedia. p. 1190-31199 MM '23, Association for Computing Machinery, New York, NY, USA (2023). https://doi.org// wttpg//...111135511 11144451183333611111
[30] Ye, H., Zhang, J., Liu, S., Han, X., Yang. W.: Ip-adapter: Text compatible image prompt adapter for rext-to-image diffusion models. arXiv preprint arXiv:2308.0677 066221 (2023)
[31] Yildirim, A.B., Baday, V., Erdem, E., Erdem, A., Dundar, A.: Inst-inpaint: Instructing to remove objects with diffusion models. arXiv preprint arXiv:2304.03246 (2023)
[32] Yildirim, A.B., Pehlivan, H., Bilecen, B.B., Dundar, A.: Diverse inpainting and editing with gan inversion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 23120-2311111 23130 (2023)
12

---

[33] Yu, Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4471-4480 (2019)
[34] Zhang, K., Mo, L., Chen, W., Sun, H., Su, Y.: Magicbrush: A manually annotated dataset for instruction guided image editing. Advances Neural Information Processing Systems 36 (2024)
[35] Zhang, L., Rao, A., Agrawala, M.: Adding conditional control text-to-image diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3836-3847 (2023)
[36] Zhao, S., Chen, D., Chen, Y.C., Bao, J., Hao, S., Yuan, L., Wong, K.Y.K.: Uni-controlnet: All- in-one control to text-to to--ggg--mg diffusion models. Advances in Neural Information Processing Systems 36 (2023)
[37] Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence 40(6), 1452-1464 (2017)
13

---

A Supplementary Material
In this supplementary material provide:
1. Training and inference algorithm
2. Architectural details of the MI.P network and training details
3. Conditional image generation examples for projected embeddings
4. Visual comparison with LaMa, MAT, ZITS++, Blended Latent Diffusion, Unipaint and SD-Inpaint
5. Details of the user study that we have conducted
6. Potential extension to SDXL-Inpainting pipeline
7. Our models flexibility across CLIP embedding styles
8. Our models contextual understanding
9. Reference-based background inpainting examples of our model
10. LaMa + SDInpainting comparison with our model
14

---

A.1 Training and Inference Algorithm
Algorithm Algorithm for our training and inference
1:
Training Algorithm Of MLP Projection Layer:
2: Input: Training data D = {(+)))=> Alpha-CLIP Image Encoder (a-CLIP), CLIP Image
Encoder (CLIP), MLP with parameters 8, learning rate 77. number of epochs E
3: Output: Model parameters 0
4: for epoch = to E do
5: for each (x:) E do
6: mask ones(
7: Ca-clip f- a- a-CLII(mask,..)))
8:
(prjectedd -- MMPP(ca-cli)
9: Eclip + CLIP(a;;)
LMSE elecppp ea-clip
11: O - VoLMSE
end for
13: end for
Inference Algorithm of CLIPAway:
15: Input: Image /nn Mask M, downscaled mask m. Alpha-CLIP Image Encoder (a-CLIP), Trained
MLP from 1.1, trained Stable Diffusion model SD, trained IP-Adapter, timestep
16: Output: Inpainted Image Tout
Z N/0, I)
18: ffg + a-CLIP(mask,, Iin)
19: Ebg - u-CLIP(I - mask.Iiv Iin.)
20: eʃg + MLP(esg)
21: ebg + MLP((Ebg)
Gnnal + ebg
23: Eip IP-AAdaperr(eaall (mnnal)
24: Lout < SD(Zr,t espp)
A.2 Architectural details of the MLP network and training details
As stated, we have trained a multi-layer perceptron (MLP) to project Alpha-CLIP embeddings into CLIP embeddings subspace. The network consists of blocks each containing linear layer followed by layer normalization followed by GELU activation function. The dimension of linear layers changes midway to adapt to different embedding dimension, transitioning from Alpha-CUl CLIP embedding dimension to IP-Adapter embedding dimension. It is also important to note that the final layer does not include GELU activation function. Complete implementation of the model Pytorch-like syntax is given in Table 3
The MLP model is trained single NVIDIA A40 GPU with batch size 8. Adam, optimizer is used with learning rate and weight decay are set to 1 and le-4, respectively. The model is trained for 300k steps COCO2017 which took approximately GPU hours.
15

---

Layer
Description
Linear(768,, 768) LayerNorm(768) GELUO
Linear(768,, 768) LayerNorm(768) GELU0
Linear(768,, 1024) LayerNorm(1024) GELUO
4
Linear(1024, 1024) LayerNorm(1024) GELU0)
Linear(1024, 1024) LayerNorm(1024) GELUO)
Linear( 1024, 1024) LayerNorm(1024) GELU()
Linear(1024, 1024)
Table 3: Layer-by-Layer Description of the MLP
16

---

A.3 Conditional Image Generation Examples
Image
Mask
Fg Focused Bg Focused Projected
BEENE
Figure Conditional image generation results (not inpainting), when condition is foreground, background, and projected embeddings.
17

---

A.4
Additional Qualitative Results
Image
LaMa
MAT
+
Blended
Unipaint
SDInpaint
CLiPaway
Figure 10: Qualitative results our and competing methods on the COC02017 Validation dataset. Diffusion models often replace the object insert new content instead of removing it, while GAN- based models, although not struggling with object insertion, fail to generate a realistic background. Our method is the only one that effectively removes objects and fills the regions in realistic manner.
18

---

A.5
User Study Details
have conducted a user study to obtain better comparison between our model and our competitors. 20 people voted volunteers. Random 20 samples included in user study without cherry- picking. We include ZITS++, Unipaint, SD-Inpaint, and CLIPAway to have a variety of the best- performing models. The voters were asked to select the option that achieved the best object removal regarding realism and consistency with the scene. The instructions and an example from the study are given in Figure 11 and 12 respectively. The order of the methods randomized for each question. There was no time limit set the study.
User Study- CCIPAWay
Image inpainting is technique used fill damaged or missing parts of an Image make it look complete and natural. This process can be used to remove unwanted objects from photos, repair old damaged photographs, or fill in gaps when parts of an image are missing. The goal is to make the final image appear as natural and seamless as possible, as if the missing or unwanted parts were never there.
Task: You will be provided with the original image and a mask indicating the region to be removed each question. Review the images in options through assessing them based on the following criteria:
Accuracy: If the object is correctly removed.
Smoothness: The texture and transitions the inpainted area, ensuring they are seamless.
Figure 11: Guideline given to the voters correctly completing the user study
A.6 Extension to SDXI.-Inpainting Pipeline
SDXL-Inpaintin is another widely-used pipeline for inpainting tasks, yet has notable limitations when applied to object removal. Upon examination, we observed that issues related to unintended object additions particularly pronounced. SDXLL-Ipainting recommends setting the strength parameter to 0.99, using the unerased image as input, and applying noise at this strength level The strength parameter, which ranges from 0 to 1, controls the degree of transformation applied to the masked area, with higher values introducing more noise and necessitating additional denoising steps. When set 1, the strength parameter applies maximum noise, often resulting in degradation of image quality, as noted in the documentation' 's limitations. Conversely, using a strength 0.99 frequently causes the model to regenerate erased foreground objects, residual information about the object remains. Under this configuration, SDXL-Inpainting achieves a CLIP-1 score of 67.22, the lowest among those reported in Table of the main paper. However, when combined with our CLIPAway method, the SDXL-Inpaint model achieves an improved CLIP-1 score of 85,91. These results are presented in Figure T7, which illustrates that a strength of the model tends to hallucinate or replace objects, while at strength of 0.99, it may partially regenerate the object.
A.7 Flexibility Across CLIP Embedding Styles
CLIPAway demonstrates robust adaptability across various CLIP embedding spaces. To explore this flexibility, we trained the MLP for AlphaCLIP with additional model Keys-VITT-L/14e 336px and VIT-B/16-alonsside VIT-L/14. Each model was subsequently evaluated, and the results underscored
1.0-inpainting-O0.
19

---

2. In which image the object is best removed? *
Fidelity
IND
T Fidulity
Fidelity
O
A
B
S Fidulity
C
Fidelity
C
OD D
Figure 12: An example question from the user study
20

---

the effectiveness of our approach across diverse CLIP embeddings, as shown in 2 This analysis validates that our method can seamlessly operate within different embedding spaces.
Image
Mask
L/14
L/14@ 336px
B/16
Figure 13: CLIPAway results with different CLIP embedding spaces.
Our projection method is not confined to the OpenCLIP embedding space. As AlphaCLIP's vision transformer and CLIP's vision transformer are trained with aligned objectives, their feature spaces are conceptually similar. Consequently, the projection method can be effectively applied within the AlphaCIIIP embedding space other similar CLIP embedding spaces. To validate this, projection approach on the AlphaCIIIP feature space is evaluated. The outputs of this approach, projection on AlphaCLII space followed by MIP, confirmed that method is applicable beyond the OpenCLIP embedding space.
A.8 Contextual Understanding
As approach leverages SDInpaint its backbone, the inpainting process is constrained to regions delineated by the provided masks, a characteristic shared with other competing methods, such as LAMA, MAT, and ZITS++. Additional experiments incorporating both the primary objects and their associated shadow regions within the input masks were conducted. The results, presented in 14] demonstrate that our method successfully eliminates both objects and their corresponding shadows when the mask encompasses both elements.
Image
Mask
Result
Image
Mask
Result
Figure 14: Shadows can be included in the masks if preferred.
21

---

A.9 Reference-based Background Inpainting
While the primary focus of our work centers on object removal, our framework demonstrates versatil- ity across various tasks, We explore the application of our method reference-based background inpainting, as illustrated in 15 By computing the projected embedding of a reference image that represents the target background, our framework successfully inpaints the background while main- taining the integrity of the foreground. These results highlight the broader potential of our approach addressing general image manipulation and inpainting challenges.
Image-Fa
Mask-Fg
Image-Bg
Mask-Bg
Result
STOP
STOP
10
41
Figure 15: Reference-based background inpainting with CLIPAway.
A.10 LaMa + SDInpainting
To further evaluate the effectiveness of our approach, we compared our method to widely com- munity pipeline for object remova This approach first applies LaMa inpainting as a preprocessing step to the SD-Inpaint pipeline, creating hybrid method intended to capitalize on the strengths of both models. While the integration of SD-Inpaint as post-procces post-processing step enhances overall visual quality metrics, as shown by improved FID scores, it also introduces certain limitations. Specifically, a reduction in object removal effectiveness noted, primarily due to SD-Inpaint's tendency to hallucinate introduce extraneous objects in the inpainted regions. Our CLIPAway model notably surpasses this hybrid approach, achieving superior KID (Kernel Inception Distance) metrics and consistently higher scores in CLIP-based object removal metries. This result is illustrated in Figure 16 where, although LaMa provides a more refined intermediate image for the SD-Inpaint pipeline, the tendency of SD-Inpaint replace or hallucinate objects remains evident.
22

---

Image
Mask
LaMa
Lama w/SD-IInaint
Ours
Figure 16: LaMa + SD-Inpaint comparison with Ours.
23

---

Image
Mask
SDXII-Inpaint == 1.0
Ours (CLIPAway)
0.99
1.0
0.99
Figure 17: SDXIL-Inpaint and SDXI-Innaint+CL.PPAaay results with strength values of 1.0 and 0.99.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Justification: Introduction and Abstract state the limitations of current models and reflect the paper's 's contribution over them. In addition, the paper's 's task of object removal is clearly stated in both the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are mentioned Sec. 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer:
[NA]
Justification: The paper does not include theoretical results. Hence, there is not any proof that needs to be included.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The training details are given in Sec. A.2 of the supplementary material and the algorithm is given step-by- step in Sec. A.1 of the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
26

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The project code can be accessed through the project page url in abstract
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttpcccpppppppppeeeeeeeeeeeeeeee httpa///giiieee/eeeSebiiisssssss /Cc////oddiiisssseeeeeee ssiiiiiiccyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Details are given in Sec. A.2. of the supplementary material. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not provide error bars evaluate pretrained models. Our model is combined with three baselines and consistently improve metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Resources are given in Sec. A.2. of the supplementary material
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics Ethics//ttps:..
Answer: [Yes]
Justification:
The paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Impact is given in Sec. 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: in Sec. 5.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Open release models and datasets are used with proper citations and the licenses are properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: New assets not introduced.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: User study is conducted in a volunteer basis among college students. The details are given in Sec. A.5 of the supplementary material.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There were potential risks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "azYfhuwtYM", "reviewer": "Reviewer_6DqA", "summary": "This paper addresses a commonly observed problem when using pre-trained diffusion models for object removal: in standard inpainting setups, these models often add similar objects in place of the ones to be removed instead of extending the background to the masked area. To address this problem, the authors use CLIPAway, a method that leverages region-focused embeddings from Alpha-CLIP, obtaining further disentangled background embeddings that can then be used to guide the diffusion model to perform high-quality object removal without foreground leakage.", "questions": "- Can the projection also be performed in the AlphaCLIP embedding space (or other CLIP embedding spaces) or does it only work in the adapted OpenCLIP embedding space? This would be especially interesting as it would speak to the generalizability of the proposed method, as being limited to specific CLIP embedding spaces might limit practical applications, such as applying this to a standard unCLIP model (e.g., Karlo https://github.com/kakaobrain/karlo, Stable unCLIP https://huggingface.co/docs/diffusers/en/api/pipelines/stable_unclip).\n- What is the time (wall clock/GPU-hours) required for training the MLP?", "limitations": "The authors have adequately addressed the limitations and societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper addresses a very common and important problem in practical applications of large-scale pretrained T2I diffusion models.\n- While very simple, the proposed method combines existing methods (Alpha-CLIP and T2I-Adapter/T2I diffusion models conditioned on CLIP embeddings) in a novel and very effective manner. The proposed method is demonstrated to work well qualitatively and outperform other approaches when combined with off-the-shelf inpainting methods in a quantiative evaluation.\n- The paper is written clearly and each part of the method is well-motivated and explained both in text and with qualitative examples.", "weaknesses": "The main weakness with the paper is the simplicity of the method. Generally, a simple method that achieves the goal is often better than an over-involved complex method, but most of the parts of this method are relatively obvious (translating from one CLIP embedding space to another seems to primarily be an engineering decision as the pre-trained IP-Adapter does not directly consume Alpha-CLIP embeddings, but it would likely be very simple to just train an adapter on these embeddings; and orthogonalization is a very basic operation, even if it is curious that it works so well in this case) and aspects of transferability of this method, which are relevant for its further impact beyond the implementation presented in this paper (e.g., whether it is reliant on the exact two CLIP embedding spaces used, or whether they can be substituted with other CLIP spaces with similar properties), are not covered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their detailed review.\n\nWe appreciate the reviewer's feedback regarding the simplicity of our method. We share the same preference for simplicity over complex solutions. While our solution might seem obvious now, previous research attempted to address object removal tasks by generating synthetic datasets. For instance, InstInpaint [30], MagicBrush [33], InstructPix2Pix [2], and concurrently ObjectDrop [26] explored different approaches to create training datasets for this purpose. Our method did not require a dataset generation and achieve better results than competing methods which makes it more elegant. \n\nThe reviewer asks about the flexibility of CLIP embedding spaces. This is an excellent point.  Our method is not confined to the CLIP embedding spaces used in our initial experiments. To demonstrate this flexibility, we trained the MLP for AlphaCLIP with model keys VIT-L/14@336px, and VIT-B/16 in addition to the VIT-L/14 that was presented in the paper. After training, we evaluated each model using the same setup as in Table 1 of the manuscript. The qualitative scores for each model are provided in the table below:\n\n| Method | FID  | CMMD | CLIP Dist | CLIP@1 | CLIP@3 | CLIP@5 | \n|-------------------------------------------|------|------|-----------|--------|--------|--------|\n| SD-Inpaint  | 59.21| 0.54 | 0.75 | 70.45  | 57.14  | 49.88  | \n| SD-Inpaint + CLIPAway (VIT L/14 - presented in the paper) | 57.32| 0.53 | 0.81      | 84.82  | 74.42  | 67.76  | \n| SD-Inpaint + CLIPAway (VIT L/14@336px)  | 54.93| 0.48 | 0.80      | 82.36  | 71.68  | 63.28  | \n| SD-Inpaint + CLIPAway (ViT-B/16)  | 55.31| 0.48 | 0.78      | 83.57  | 72.44  | 63.81  | \n\nThe results consistently improve the SD-Inpaint model with different CLIP embeddings. \n\nThe reviewer asks if the projection could be done on the AlphaCLIP space rather than OpenCLIP embedding space. Our projection method is not restricted to the OpenCLIP embedding space. Since AlphaCLIP’s vision transformer is trained with similar objectives as the CLIP vision transformer, their feature spaces are conceptually similar. Therefore, projections can be performed in the AlphaCLIP embedding space or other CLIP embedding spaces with similar properties. To support this, we evaluated the projection method on the AlphaCLIP feature space (projection on AlphaCLIP space followed by MLP) using the same setup as described in Table 1 of the manuscript. The results, shown below, confirm that our projection approach is applicable beyond the OpenCLIP embedding space.\n\n| Model | FID  | CMMD | CLIP Dist | CLIP@1 | CLIP@3 | CLIP@5 |\n|------------------|------|------|-----------|--------|--------|--------|\n| SD-Inpaint  | 59.21| 0.54 | 0.75 | 70.45  | 57.14  | 49.88  | \n| SDInpaint + CLIPAway VIT-L/14 @336px | 54.46| 0.36 | 0.82 | 82.13  | 70.02  | 63.28  | \n|SDInpaint + CLIPAway   VIT-L/14 | 56.15| 0.42 | 0.86 | 85.31  | 74.26  | 68.58  | \n| SDInpaint + CLIPAway  ViT-B/16 | 54.99| 0.41 | 0.84 | 85.08  | 74.79  | 68.35  | \n\nWe will include these results in the revised paper. We believe they will make the paper more comprehensive. We would like to thank the reviewer for these valuable suggestions.\n\nWe also provided qualitative example object removal results in the Rebuttal PDF (Fig. 1), further confirming the effectiveness of our projection approach across different CLIP embedding spaces. These results illustrate our method’s flexibility and robustness, supporting its broader applicability.\n\nTraining the MLP layer for AlphaCLIP with the model key VIT-B/16 using a single Nvidia A40 GPU takes approximately 7 hours which is a negligible GPU time compared to training the diffusion models from scratch.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s1323lXn9F", "reviewer": "Reviewer_hrfv", "summary": "This paper proposes an approach that aims to tackle object removal in stable diffusion models.  The paper utilizes AlphaCLIP embedding (that are trained with an additional alpha layer mask to enable incorporation of regions of interest), and techniques such as IP-Adapter that decouples the diffusion unet cross-attention mechanisms to accept features such as coming from image/text separately. The authors train using image prompts as inputs and controlling the alpha channel to correspond to background or foreground (inverse of background) focus to guide the attention mechanism. Eq 2 aims to calculate embedding that are orthogonal to the foreground embedding, resulting in embedding that are focusing on the background.", "questions": "please see above", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method leverages existing work well, and the proposed method appears to indeed remove objects in a smoother way than competitors\n-It is sometimes difficult to judge from a few examples qualitatively, but it seems that the proposed method  performs relatively better in most cases presented\n- ablation studies show the impact of background/foreground embedding - although in most cases it seems that background and foreground embedding do contain both. The orthogonal embedding do appear to include only information related to the background - although that does not exactly match the background of the image", "weaknesses": "- As with the shadows example mentioned in the limitations, the method does not seem to remove objects based on much context - i.e. region of interest is a stronger control. This may lead to undesirable or unrealistic results - so perhaps a trade-off between strict ROI adherence and context could be employed.\n- The method can be plug and play after training - but the training process itself can make it specific to architectural choices\n- Although mentioned in the paper, this work appears to have a larger computational overhead that simpler inpainting methods, which perhaps can then be improved with simpler post-processing\n- the paper is focused on object removal in a background-consistent manner, which is a limited task, while some compared methods tackle the more general problem if image inpainting", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their detailed feedback. \n\nWe appreciate the reviewer's recognition of our method's ability to leverage existing work effectively and its superior performance in removing objects smoothly compared to competitors. To address the concern regarding the qualitative judgment based on a few examples, we will extend our supplementary material with additional qualitative examples to better demonstrate the effectiveness of our method.  Additionally, we would like to emphasize that the quantitative analysis provided in Table 1 shows we achieve consistent improvements over many baselines.  \n\nWe appreciate the reviewer's comments on our ablation studies showing the impact of background and foreground embeddings. While background and foreground embeddings may sometimes contain overlapping information, our method aims to orthogonalize these embeddings to effectively isolate relevant background information. The orthogonal embeddings are designed to include only information related to the background. Although these embeddings are not expected to exactly match the original background of the image, they are constructed to exclude the foreground object and provide a contextually coherent background representation. To further clarify and validate the effectiveness of our approach, we provide additional examples in the Rebuttal PDF (Fig. 2) which show the behavior of foreground-background, and projected embeddings. We will add more examples in the revised paper. We would like to thank the reviewer for bringing this to our attention.  \n\n**Regarding Contextual Understanding**\\\nSince we use SDInpaint as our backbone, inpainting is carried out only on the regions identified by the provided masks as happens in other competing methods as well, LAMA, MAT, ZITS++. To address the reviewer's concern about the method not removing shadows, we conducted new experiments where both objects of interest and their shadows were included in the corresponding masks.  We share these results in the Rebuttal PDF (Fig. 3). Our method produces highly satisfactory results, effectively eliminating the objects and their shadows if the user prefers to. \n\n**Clarification on \"Plug and Play\" Capability**\\\nWe apologize if the use of the term \"plug and play\" caused any confusion. Our method bridges the AlphaCLIP embedding space to the CLIP embedding space through projections, which requires some architectural decisions during the training process. However, once trained, our CLIPAway module can be integrated into any framework that employs the CLIP embedding space or other CLIP embedding spaces with similar properties — without requiring additional training. As demonstrated in Table 1, adding CLIPAway to SD-Inpaint, Blended Diffusion, and Unipaint frameworks significantly improves their performance on the object removal task. \n\n**Computational Overhead and Efficiency**\\\nWhile diffusion models do have higher computational overhead compared to GAN models, they are currently favored due to the significant improvements they offer. Additionally, there is extensive research focused on optimizing diffusion models and reducing the number of required denoising steps to improve their efficiency. Therefore, we believe it is important to continue improving these models for various tasks. . We are not aware of any straightforward post-processing technique that can elevate simpler inpainting methods to a state-of-the-art level of quality. Reviewer GcHN brought the LaMa + SD-Inpaint baseline to our attention, where the faster LaMa model generates the initial inpainting results and SDInpaint, which is a diffusion-based inpainting method, serves as post-processing. However, this combination is more computationally intensive than our approach and yields worse results.\n\n**Expanding Beyond Object Removal**\\\nWhile our main focus in the paper is on object removal, our framework, can be applied to other tasks. Based on the Reviewer's comment, we investigated the capabilities of our framework  on reference-based background inpainting as given in the Rebuttal PDF (Fig. 5). By computing the projected embedding of a reference image representing the target background, we inpaint the background of the input image while preserving its foreground.  We will revise our submission to include these new results, demonstrating that our method can open directions for image manipulation and general image inpainting problems. Additionally, we would like to emphasize that object removal is a significant task in its own right, and previous research has proposed generating datasets specifically for this purpose.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ObIM8enfu8", "reviewer": "Reviewer_FKYu", "summary": "The paper identifies a common limitation in recent diffusion model-based inpainting methods: unintended hallucinations of the removed object. To address the problem, the paper introduces CLIPAway, a plug-and-play module that does not rely on any specific training. Using vector arithmetic, it successfully obtains an embedding that predominantly focuses on the background. Through comparisons, it is demonstrated that the proposed method outperforms existing methods in object removal.", "questions": "- In lines 192-193, when setting up the baselines, the authors used empty prompt or \"background\" as input. In practice, providing an accurate prompt describing the background can also be useful in removing the object. E.g., when removing a laptop from a table, the prompt can be \"an empty table\". Does this work for the DM-based baselines?", "limitations": "Already addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper identifies a well-known limitation in the current diffusion models and the motivation behind this paper is very strong.\n- The proposed method does not rely on synthetic data or any special annotations during its training, thus having a stronger ability to generalize and more robust performance compared to prior methods (Fig 8).\n- The idea of using AlphaCLIP encoder and doing vector subtraction is simple but effective.\n- Extensive results (Tab 1 and user study) are presented to support the conclusions of the paper, especially on whether the object is correctly removed.", "weaknesses": "- Although the proposed model can effectively remove the object, the quality of the inpainted image is not stable. There are notable artifacts in Fig 1, such as the shadow in the 1st, 2nd rows on the right, and 3rd, 4th rows on the left.\n- The conclusions / observations in Fig 4 lacks enough evidence. E.g., in the bottom-left, it's hard to tell whether the foreground or the background is more prominent. Adding more visual results / quantitative comparisons may help to obtain a more solid/reasonable observation.\n- There have been a lot of evaluations on whether the object is correctly removed (in Tab 1), but should be more on assessing the visual quality, e.g., LPIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their detailed feedback. \n\n**Eliminating Shadows**\\\nWe appreciate the reviewer’s concern about shadows in the inpainted images. Since our method uses SD-Inpaint as its backbone, inpainting is performed only on the masked regions, which ensures that only the areas the user wants to remove are altered, while the unmasked areas are preserved. LaMa, MAT, and ZITS++ follow a similar approach. To address the issue of shadows, we conducted additional experiments where both the objects and their shadows were included in the masks. As shown in the Rebuttal PDF (Fig. 3), this adjustment significantly improves the results by effectively removing both the objects and their shadows, leading to cleaner and more accurate inpainted images. This way, only the areas the user wants to remove are altered, the user can choose to keep the shadow for some artistic designs or choose to remove it. \n\n**Additional Visual Evidence**\\\nWe thank the reviewer for their suggestion to provide more visual evidence. We agree that additional examples would better demonstrate the efficacy of our method. To address this, we have included more visual results in the Rebuttal PDF (Fig. 2) that show the behavior of foreground-background, and projected embeddings.  Specifically, we observe that the foreground embedding is dominated by the foreground object, while the foreground object appears at a smaller scale in the conditional generation results of the background embedding. These new examples highlight the effectiveness of our projection block in removing objects of interest. We will update our submission to include these new results, strengthening our observations and making them more convincing.\n\n**Improving Visual Quality Assessment**\\\nTo provide a more comprehensive assessment of visual quality, we have included Kernel Inception Distance (KID) as an additional photorealism metric as given below (lower is better):\n\n| Method                        | KID    |\n|----------------------------|------|\n| ZITS                          | 0.0208 |\n| MAT                          | 0.0278 |\n| LaMa                        | 0.0195 |\n| Blended                   | 0.0362 |\n| Blended + CLIPAway          | 0.0194 |\n| Unipaint                    | 0.0360 |\n| Unipaint + CLIPAway         | 0.0199 |\n| SDInpaint                     | 0.0145 |\n| SDInpaint + CLIPAway | 0.0108 |\n\nAs shown in the table, KID results align with the photorealism metrics reported previously. We did not use LPIPS because it necessitates reference images with the objects removed, which are not available. With the addition of KID, we now have three metrics to evaluate visual quality, complementing FID and CMMD, which were previously reported in the quantitative comparisons in the main paper.\n\n**Prompt Use in Baseline Comparisons**\\\nWe understand that using a specific prompt, such as \"an empty table\" when removing a laptop, can sometimes improve results. However, we found that this approach does not consistently improve the baselines.  Sample results demonstrating this are included in the Rebuttal PDF (Fig. 4).  For example, \"empty table top\" prompt results in inpainting the region with adding an additional component on top of the table rather than completing it directly in the first row, \"an empty runway\" still adds planes to the image, or \"an empty bed\" adds a pillow to the bed.  Additionally, generating specific prompts for every situation in a large dataset is impractical and time-consuming. Our method circumvents this issue by using vector arithmetic in the CLIP space, eliminating the need for custom prompts. We believe this makes our method more robust and user-friendly. We hope the reviewer finds this perspective agreeable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T0xXUnUFVG", "reviewer": "Reviewer_GcHN", "summary": "This work addresses the object removal problem with a simple and effective embedding arithmetic strategy. This idea is implemented by combining the object semantic understanding ability from alpha-clip and the generative ability from the text-to-image diffusion models. One good property is that the proposed method is a plug-and-play strategy that can be adopted in various Diffusion-based t2i frameworks.\nCompared to the previous GAN-based and Diffusion-based inpaint methods, this method demonstrates better results with suitable background and fewer artifacts.", "questions": "“LaMa + SDinpaint” is required to be compared.\nComparisons on higher resolutions could be demonstrated.", "limitations": "More failure cases could be discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The simple and effective embedding arithmetic strategy is intuitive and suitable for the object removal problem.\n\nThe designed technical framework of the alpha-clip plus t2i diffusion model is flexible and powerful since many powerful conditional image generative models are based on clip embeddings. It would be interesting to see how this technique can be applied to other t2i frameworks.\n\nFigure 5 demonstrates the superiority of the proposed method which performs a good trade-off between FID and CLIP Scores. Figure 6 also shows good inpaint results with fewer artifacts and blurry effects.\n\nThe paper is easy to follow and the algorithm has been illustrated step-by-step.\n\nI highly recommend the authors release code and model for the following works. I believe it can benefit the related CV fields.", "weaknesses": "It seems that the applicable image resolution is limited by the diffusion method. Comparisons on higher resolutions could demonstrate the boundary of this technique since one benefit LaMa-like methods is the applicability to high-resolution images.\n\nOne popular baseline of “LaMa + SDinpaint” should be compared, please check https://github.com/Mikubill/sd-webui-controlnet/discussions/1597\n\nMore failure cases could be presented and analyzed to show the limitations.\n\nI suppose this method can be extended to other frameworks like SDXL, it would be great to perform these extensions to make this work more comprehensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their detailed feedback which help us improve the paper further. \n\n**Applicability to Higher Resolution Images**\\\nWe appreciate the reviewer's insight regarding the applicability of our method to higher resolution images. As noted, the current limitation is inherent to existing diffusion-based image inpainting methods. Our primary goal was to enhance object-removal performance within these existing constraints. Future research could indeed focus on adapting our method for higher resolutions, potentially bridging the gap between diffusion-based techniques and LaMa-like methods. This would involve significant advancements in diffusion model architectures to handle high-resolution images effectively. \n\n**Comparison with “LaMa + SD-Inpaint” Baseline**\\\nThank you for suggesting the comparison with the “LaMa + SD-Inpaint” baseline. This is  a very interesting work. We conducted this comparison using the same setup as in Table 1 of our manuscript. The results are as follows. \n\n| Method            | FID  | CMMD | CLIP Dist | CLIP@1 | CLIP@3 | CLIP@5 | KID    |\n|-------------------|------|------|-----------|--------|--------|--------|--------|\n| SD-Inpaint              | 59.21| 0.54 | 0.75      | 70.45  | 57.14  | 49.88  | 0.0145 |\n| LaMa              | 65.76| 0.81 | 0.66      | 78.34  | 64.42  | 56.85  | 0.0195 |\n| LaMa + SDinpaint  | 51.33| 0.45 | 0.75      | 72.29  | 57.61  | 50.01  | 0.0117 |\n| SD-Inpaint + CLIPAway      | 54.93| 0.48 | 0.80      | 82.36  | 71.68  | 63.28  | 0.0095 |\n\nIncorporating SD-Inpaint as a post-process significantly improved sample quality, as indicated by the improvement in the FID score. However, the object removal metrics reveal a decline in the model’s ability to effectively remove objects. We attribute this decline to SD-Inpaint’s tendency to insert and hallucinate objects, as noted in our paper.  We also would like to mention that in KID, which is the other visual quality metric, CLIPAway model achieves better scores in addition to the significantly better scores on the object removal CLIP based metrics. We appreciate the reviewer’s suggestion, which helped us highlight the trade-offs involved in using SD-Inpaint in combination with LaMa. We will include these results in our manuscript to provide a more comprehensive evaluation of our pipeline.  \n\nWe also add visual results of this comparison in the attached Rebuttal Fig. 6. It can be seen that LaMa fills the erased area in a blurred way, and the SD-Inpaint, followed by LaMa, adds objects again. For example, in the first row, a horse is added to the image.\n\n**Analysis of Failure Cases**\\\nWe appreciate the reviewer's suggestion to present and analyze more failure cases to highlight the limitations of our method. \n\nOne significant limitation of our method is the constraint on applicable image resolution as noted by the Reviewer, which is inherent to diffusion-based inpainting methods. Specifically, the model in a Stable Diffusion (SD) pipeline expects input images and masks to be in the 512x512 pixel resolution, while SD-XL pipelines expect 1024x1024 resolution. Deviating from these resolutions degrades the performance of the UNet in the diffusion pipeline, resulting in less effective inpainting, regardless of the quality of the embeddings provided by the CLIPAway pipeline. We will include a detailed discussion of these limitations in our manuscript to provide a comprehensive evaluation of our method.\n\n**Potential Extensions to Other Frameworks**\\\nWe appreciate the reviewer's suggestion to apply our method to SDXL. After examining the SDXL-Inpaint model, we observed that the issue of object addition is even more pronounced. SDXL recommends setting the strength parameter to 0.99, using the unerased image as input, and introducing noise at this strength (for reference: https://huggingface.co/diffusers/stable-diffusion-xl-1.0-inpainting-0.1). The strength parameter determines how much the masked portion of the reference image is transformed, ranging from 0 to 1. A higher strength adds more noise to the image, requiring more denoising steps. When the strength is set to 1, maximum noise is added, leading to degradation in image quality as noted on the webpage's limitations, with ongoing work to address this issue.\n\nFig. 7 of the Rebuttal PDF illustrates that using the original input image with a strength of 1.0 results in lower output quality. Conversely, a strength of 0.99 often causes the model to regenerate the erased foreground object, as it retains some information about it. This setting yields a CLIP-1 score of 67.22, the lowest among those reported in Table 1 of the main paper. When combined with our CLIPAway method, the SDXL-Inpaint model achieves a CLIP-1 score of 85.91.\n\nWe will include comprehensive experiments with SDXL in the revised manuscript. We believe that eliminating the reliance on unerased images will improve results for both SDXL-Inpaint and our approach.\n\nLastly, we thank the reviewer for recommending that we open-source our code and for finding our work beneficial. We are pleased to announce that the code and model are ready and will be released soon.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to thank all reviewers for their valuable feedback. We have responded to each reviewer's questions in the rebuttal sections, and attached a PDF file with figures for our additional results.

Meta-review: After rebuttal, all the reviewers unanimously vote for acceptance of this work. After checking the rebuttal, the review, and the manuscript, the AC recommends acceptance.
Decision: Accept (poster)