Add-it: Training-Free Object Insertion in Images With Pretrained Diffusion Models
Authors: Yoad Tewel
Rinon Gal
Dvir Samuel
Yuval Atzmon
Lior Wolf
Gal Chechik
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion
Editing
Affordance
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We presents a method to insert objects into existing images using pretrained diffusion models, guided by text prompts, without additional training.

Abstract: Adding Object into images based on text instructions is a challenging task in semantic image editing, requiring a balance between preserving the original scene and seamlessly integrating the new object in a fitting location. Despite extensive efforts, existing models often struggle with this balance, particularly with finding a natural location for adding an object in complex scenes. We introduce Add-it, a training-free approach that extends diffusion models' attention mechanisms to incorporate information from three key sources: the scene image, the text prompt, and the generated image itself. Our weighted extended-attention mechanism maintains structural consistency and fine details while ensuring natural object placement. Without task-specific fine-tuning, Add-it achieves state-of-the-art results on both real and generated image insertion benchmarks, including our newly constructed "Additing Affordance Benchmark" for evaluating object placement plausibility, outperforming supervised methods. Human evaluations show that Add-it is preferred in over 80% of cases, and it also demonstrates improvements in various automated metrics.

Published as a conference paper at ICLR 2025
ADD-IT: TRAINING-FREE OBJECT INSERTION IN IM- AGES WITH PRETRAINED DIFFUSION MODELS
Yoad Tewel
Rinon Gal
Dvir Samuel
NVIDIA, Tel-Aviv University
NVIDIA, Tel-Aviv University
Bar-Ilan University
Yuval Atzmon
Lior Wolf
Gal Chechik NVIDIA
NVIDIA
Tel-Aviv University
Output sign opeling YCL top of the buiding
Source
Output The tenric player wearing headoond
Source
Real
dreep wearing
sitting table
Generated
An empty livin room
A couch
A dag lying on the couch
A parnting of cat saying Do
Step-by-Step
Figure 1: Given an input image (left in each pair), either real (top row) or generated (mid row), along with simple textual prompt describing an object to be added Add-it seamlessly adds the object to the image natural way. Add-it allows the step-by-step creation of complex scenes without the need for optimization pre-training.
ABSTRACT
Adding Object into images based on text instructions is a challenging task in se- mantic image editing, requiring a balance between preserving the original scene and seamlessly integrating the new object in fitting location. Despite exten- sive efforts, existing models often struggle with this balance, particularly with finding a natural location for adding an object in complex scenes. We introduce Add-it, a training-free approach that extends diffusion models' attention mecha- nisms to incorporate information from three key sources: the scene image, the text prompt, and the generated image itself. Our weighted attenddd-attention mecha- nism maintains structural consistency and fine details while ensuring natural ob- ject placement. Without task-specific fine-tuning, Add-it achieves state-of-the- art results on both real and generated image insertion benchmarks, including our newly constructed "Additing Affordance Benchmark" for evaluating object place- ment plausibility, outperforming supervised methods. Human evaluations show that Add-it is preferred in over 80% cases, and it also demonstrates improve- ments in various automated metrics.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Adding objects to images based on textual instructions is a challenging task in image editing, with numerous applications in computer graphics, content creation and synthetic data generation. A creator may want to use text-to-image models to iteratively build a complex visual scene, while autonomous driving researchers may wish to draw pedestrians in new scenarios for training their car-perception system. Despite considerable recent research efforts on text-based editing, this particular task remains a challenge. When adding objects, one needs to preserve the appearance and structure of the original scene closely as possible, while inserting the novel objects in way that appears natural. To do so, one must first understand affordance- the deep semantic knowledge of how people and objects interact, in order to position object in reasonable location. For brevity, we call this task Image Additing.
Several studies (Hertz al. 2022 Meng et et. 2022) tried addressing this task by leveraging modern text-to-image diffusion models. This is a natural choice since these models embody substantial knowledge about arrangements of objects in scenes and support open-world conditioning on text. While these methods perform well for various editing tasks, their success rate for adding objects is disappointingly low, failing align with both the source image and the text prompt. In response, another set of methods took a more direct learning approach (Brooks et al. 2023 Zhang et al. [023] Canberk et al., 2024). They trained deep models on large image editing datasets, pairing images with and without an object to add. However, these often struggle with generalization beyond their training data, falling short of the general nature of the original diffusion model itself. This typically manifests as a failure to insert the new object, the creation of visual artifacts, or more commonly failing to insert the object the correct place, i.e. struggling with affordances. Indeed, we remain far from achieving open-world object insertions from text instructions.
Here we describe an open-worke training-free method that can successfully leverage the knowledge stored in text-to-image foundation models, to naturally add objects into images. As a guiding prin- ciple, we propose that addressing the affordance challenge requires methods to carefully balance between the context of the existing scene and the instructions provided in the prompt. We achieve this by: first, extending the multi-modal attention mechanism (Esser et al. 2024) of recent T2I dif- fusion models also consider tokens from source image; and second, controlling the influence of each multi-modal attention component: the source image, the target image and the text prompt. A main contribution of this paper is a mechanism balance these three sources of attention during gen- eration. We also apply a structure transfer step and introduce a novel subject-guides latent blending mechanism to preserve the fine details of the source image while enabling necessary adjustments, such as shadows or reflections. Our full pipeline is shown at fig. 2
We name our method Add-it.
Image Additing methods typically face three main failure modes: neglect, appearance, and affor- dance. While current CLIP-based evaluation protocols can partially assess neglect and appearance, there is a lack of reliable methods for evaluating affordance. To address this gap, we introduce the "Additing Affordance Benchmark," where manually annotate suitable arcas for object in- sertion in images and propose a new protocol specifically designed to evaluate the plausibility of object placement. Additionally, we introduce metric to capture object neglect. Add-it outper- forms all baselines, improving affordance from 47% to 83%. We also evaluate our method on an existing benchmark (Sheynin al.) 2023) with real images, well as newly proposed Addit- ing Benchmark for generated images. Add-it consistently surpasses previous methods, as reflected by CLIP-based metries, our object inclusion metric, and human preference, where our method is favored in over 80% of cases, even against methods specifically trained for this task.
Our contributions are as follows: (i) We propose a training-free method that achieves state-of-the- art results on the task of object insertion, significantly outperforming previous methods, including supervised ones trained for this task. (ii) We analyze the components of attention in modern diffusion model and introduce a novel mechanism to control their contribution, along with novel Subject Guided Latent Blending and a noise structure transfer. (iii) We introduce an affordance benchmark and a new evaluation protocol to assess the plausibility of object insertion, addressing critical gap in current Image Additing evaluation methods.
2 RELATED WORK
2

---

Published as a conference paper at ICLR 2025
Object Placement and Insertion. Inserting objects into images remains core challenge in image editing. Traditional computer graphics methods often depend on manual object plaeeenet (C. Wang
2014 utilize synthetic data-driven approaches (Fisher et al. 2012). Early computer vision tech- niques employed contextual cues to predict possible object positions (Choi et al., 2012: Lin et al,
2013 Zhao et al.; 201T). With advancements in deep learning, generative models have been trained
to learn object placements. For example, Compositing GAN (Azadi et al., (2200) generates object
composites by refining geometry and appearance, while RelaxedPlacement (Lee et al., 2022) opti-
mizes object placement and sizing based on relationships depicted in scene graphs. OBJect3DIT (Michel et al., 2024) explores 3D-aware object insertion guided by language instructions, primarily
using synthetic data. Despite their effectiveness, these methods often struggle with the complexities of real-world placement scenarios.
Editing with Text-to-Image Diffusion Models. The emergence of high-performing text-to- image diffusion models (Rombach et al. | Saharia et al., 2022, Ramesh et al. [022 Balaji et al.)
2022 Esser et al. 2024) has paved the way for effective text-based image editing techniques. Meth- ods like Prompt-to- Prompt (Hertz et al. 2022) modify attention maps by injecting the input caption's attention into the target caption's 's attention, while SDEdit (Meng et al. 2022) uses a stochastic differential equation to iteratively denoise and enhance the realism of user- user-providee provided pixel edits. For
editing real images, inversion techniques (Mokady et al. 2023] Wallace et al. [022] Pan et al., [2023] Samuel al., 2023 Deutch al. 2024 Huberman-Spp Spienn--pppplpphe glag et al. 2023) first invert an input image to its latent noise representation using given caption, enabling edits via methods like SDEdit or Prompt2Prompt. Cao et al. (2023) further improves real image editing using mutual extended self-attention mechanism. Despite their effectiveness in various tasks, these methods struggle with object addition, often failing to align new objects with both the original image and the text prompt.
To improve editing performance, several methods proposed to directly fine-tune diffusion models. Imagic (Kawar et al.] (023)) fine-tunes, diffusion models to handle complex textual instructions, whereas Text2LIVE (Bar-Tal et al. 2022) and Blended Diffusion (Avrahami et al. 2022) blend edited regions throughout the generation. InstruetPix2Pix (Brooks et al. 2023) introduced an instructable image editing model trained on large synthetic dataset for instruction based edits, while MagicBrush (Zhang et al., 2023) enhances this approach by fine-tuning InstructPix2Pix on manually annotated dataset collected through online editing tool. EmuEdit (Sheynin et al. 2023) trains a diffusion model on a large synthetic dataset to perform different editing tasks given task embedding. EraseDraw (Canberk et al.) 2024) leverages inpainting models to automatically generate high-quality training data for learning object insertion. They show that one can train models to realistically insert diverse objects into images based on language instructions.
Despite advancements in instruction-based image editing, we demonstrate that current methods still face significant challenges in accurately interpreting and executing object addition within images. In this paper, we propose a novel approach addressing the challenging task of object insertion. We show that by controlling the various attention components in the diffusion model, one can add new objects to existing images without further training or fine-tuning of the diffusion model.
METHOD
Our goal is to insert an object into real or generated image using a simple textual prompt, ensuring the result appears natural and consistent with the source image. To achieve this, we leverage pretrained diffusion model without any additional training or optimization. Our solution consists of three core components: (1) weighted extended self-attention mechanism that balances information from the source image, text prompt, and target image, (2) noising approach that preserves the source image's 's structure, and (3) novel Subject-Guided Latent Blending mechanism retain fine background details. For real images, we also introduce an inversion step, detailed below.
3.1 PRELIMINARIES: ATTENTION IN MM-DiT BLOCKS
Modern Diffusion Transformers (DiTs) models, such SD3 (Esser et al.) 2024) and FLUX (Black-
Forest 2024). process concatenated sequences of eectual-prmpp and mmage-patch tokens through unified multi-modal self-attention blocks (MM-DiT blocks). Specifically, FLUX has types of
3

---

Published as a conference paper at ICLR 2025
Tiiee steps DiEEusion
x Thtems) steps
Diéfusion Step
Prange
"A man sitting at the Table'
V
K
7P
Blending
X farget
V
X
Sebject AttenLion
Y
Transfox
X1
XT
Figure 2: Architecture outline: Given a tuple of source noise X text prompt Ptarget. we first apply Structure Transfer inject the source image's structure into the target image. We then extend the self-attention blocks so that Xagget pulls keys and values from both Ptarget and X source- with each source weighted separately. Finally, we use Subject Guided Latent Blending retain fine details from the source image.
source target noise XT targets and
attention blocks: Multi-stream blocks which separate projection matrices (Wx K, Wy: Wo) for text and image tokens, and Single-stream blocks where the same projection matrices are used for both. Both block types compute attention on the concatenated tokens follows:
A softmax (Qpp Qimgl Kp, //mg)///dd)
d+), h = A [Vp, Vingl
(1)
where Qp. Qimg are the textual-pree prompt and the image-patcl queries, respectively. The same applies K and V. Notably, Flux is composed of series of Multi-stream blocks followed by a series of Single-stream blocks.
3.2 WEIGHTED EXTENDED SEL.-ATTENTION
Our approach builds on top of the attention mechanism in MM-DiT blocks. In this attention mech- anism, tokens are drawn from two sources: the image patches X Ximage and the textual prompt P. In prior iitention-based diffusion architectures, it was shown that the appearance of source image can be transferred to a target through an extended self-attention mechanism, where the new image can attend to the tokens of the source. We propose similar extension here, by allowing the multi- modal attention to include another source - the tokens the input image we wish to edit. More formally, we define the three sources of information as: the source image X Xsource- the generated image Xtorget and the textual prompt describing the edit Ptarget- To compute the source image tokens, simply denoise it in parallel to the target image, and concatenate its keys and values to the self-attention blocks, extending eq. (T):
softmax([Q Qtarget] [KKargee]lSoorrer Kpp Ktarget] dvd)), h= A- [V Vosurees Vp Vtarget] (2)
where K Ksource and V Vsource are the keys and value extracted from the source image, and Kp. Vp- K Ktarget. Vtarget are the keys and values from the prompt and target image respectively. When X Xsource is generated image, denoising it in parallel is trivial we simply need to start denoising from the same seed that created X sanree-- Dealing with real image is more complicated, and we will describe our solution the inversion section below.
However, we notice that simply appending the keys and values of the source image to the attention blocks leads to the source image controlling the attention, which turn leads to neglect of the edit prompt, with the final generated image being a simple copy of the source image. We explore the dynamics of this phenomenon in detail in section S To avoid this effect, we can re-balance the contribution of different attention components by weighting their keys. Indeed, by reducing the weight of the source image tokens, we can achieve better balance and allow for more changes. However, if this is not done carefully, then risk upsetting the balance in the opposite fashion and

---

Published as a conference paper at ICLR 2025
seeing alignment with the source image completely ignored. Hence, we can introduce weighting term to each source of information, giving us the following
attention equation:
A = softmax(1p Qearget]l7= Ksource, Tip Kpp Vtt- Ktarget]
dx))
(3)
h A VVource: Vp. Viarget]
where Yes 7pp Yt represent the weighting terms for the source image, the prompt, and the target image, respectively. In section [ we explore the dynamics of the attention distribution across these three sources. In practice, we find that it is necessary to balance two key terms: the first is the attention distributed over the source image Asource explQ, Kuns)) and the second is the attention
distributed over the target image, Aurge expl7-Q, Knge), where Z is the softmax normalization term. To determine 7 we define the function f(v) = Assunce Aurget and use a root-solver algorithm to find y such that f())= =0
3.3 STRUCTURE TRANSFER
The weighted extended- attention mechanism allows to balance between information from the source image and the prompt, but the added objects do not always adhere to the image context (c.g. dog is too big for the chair). We attribute this issue to different seeds dictating specific structures in the generated image, which do not always align with the source image. We show that effect in fig, g where images generated with the same seed produce similar objects with without the extended attention mechanism. To address this problem, we propose to choose seeds with a structural simi- larity the source image. We do so by noising the source latent Nouree to a very high noise level (greerr with randomly sampled noise e ~ NOO 7) following the recitified flow denoising formula X₂ (1 ot T + dyc. When taweer high enough, starting the denoising process from X tmo will result in an image with similar global structure to the source image, while still allowing for changes to image content as demonstrated fig.
3.4 SUBJECT GUIDED LATENT BLENDING
The combination of structure transfer and the weighted attention mechanism ensures that the target image remains consistent with the structure and appearance of the source image, though some fine details, such textures and small background objects, may still change. Our goal is preserve all elements of the source image not affected by the added object. To achieve this, we propose Latent Blending; A naive approach would involve identifying the pixels unaffected by the object insertion and keeping them identical to those in the source image. However, two challenges arise: First, a perfect mask is needed to separate the object from the background to avoid artifacts. Second, we aim to preserve collateral effects from the object insertion, such as shadows and reflections. To address these issues, we propose generating rough mask of the object, which is then refined using SAM-2 (Ravi al. 2024) to obtain a final mask M. We then blend (Avrahami et al. 2022) the source and target noisy latents at timestep Thennd based on this mask.
To extract the rough object mask, we gather the self-attention maps corresponding to the token rep- resenting the object. We achieve this by multiplying the queries from the target image patches, Qtarget, with the key associated with the added object token, kobject- These maps are then aggre- gated across specific timesteps and layers that we identified as generating the most accurate results (further details can be found in the appendix A. We then apply dynamic threshold to the atten- tion maps using the Otsu method (Oisu) 1979) to obtain a rough object mask, M,. Finally, we refine purpose segmentation model, SAM-2. Since SAM-2 operates on im- ages rather than noisy latents, we first estimate an image, Xo, from the model's 's velocity prediction, No, using the formula X, X + (ATyeaat SAM-2 requires a localization prompt in the form of points, bounding box, or an input mask. In our method, we provide input points, as they tend to produce the most accurate masks. To extract these localization points, we iteratively sample local maxima from the attention maps full details of this sampling process are provided in appendix A.I Using these input points, we generate the refined object mask, M. Finally, we apply simple latent blending step timestep Thtend, where we compute Zurger = M @ Zuaget + (1 M) @ Z Zuurce- We present results with and without latent blending, along with the resulting mask M, in fig. 9
this mask using the general-purpose
Tatemee - Ug- addition to an input image,
5

---

Published as a conference paper at ICLR 2025
I-P2P
ED
MB
Pbl
InfEdit
MasaCtrl
SDEdit
P2P
Ours
Affordance
0.276
0.341
0.418
0.311
0.366
0.203
0.397
0.474
0.828
Table 1: Methods comparison based on Affordance score for the Additing Affordance Benchmark.
Ours
Ours
75%
25 %
03
Ours
Ours
84%
16%
50%
50%
Win Rate
Figure 3: User Study results evaluated on the real images from the Emu Edit Benchmark.
Figure 4: User Study results evaluated on the generated images from the Image Additing Benchmark.
3.5 Additing REAL IMAGES AND STEP-BY-STEP GENERATION
In the previous sections, described method for generating edited image by drawing infor- mation from a source image within the same batch. When editing a generated image, this process is straightforward: one can save the source noise, Ésource, that generated the source image and create an input batch containing both Esource and random noise, étarget, used to generate the target image. However, when editing an existing image, Zsource: we do not have access its original noise. A common approach would be to use inversion method to recover the original noise, Esource. that generated X souree. However, our experiments, popular inversion methods, such as DDIM inver- sion (Song et al. (020), do not adequately reconstruct the image using FL.UX. We propose simple solution: instead of recovering the original noise Escurce. we sample a random noise e, At each de- noising step t, produce noisy source latent, X X = X + TrE. We then apply our method as usual, using the input batch at timestep f, [X source Xiarget), where the target image draws information from the source image. This simple technique ensures perfect reconstruction of the source image, since ou = and therefore X source = Xsource-
Our method, applicable to both generated and real images, can be extended for step-by-step gen- eration. Users can start with an initial image from a textual prompt and iteratively modify it with additional prompts, progressively adding elements changes the scene. Examples of step-by- step generation are shown in fig. and fig. 10
Emu Edit
Additing Benchmark
Method
CLIP
CLIP our
CLIP 2m
Inc.
CLIP dir
CLIP out
CLIP:t. im
Inc.
InstruetPix2Pix
0.074
0.312
0.929
34%
0.074
0.244
0.943
55%
Brasedraw
0.088
0.313
0.941
65%
0.117
0.248
0.958
76%
Magicbrush
0.091
0.313
0.927
66%
0.114
0.250
0.925
86%
Paint by Inpaint
0.071
0.316
0.955
58%
0.079
0.246
0.954
68%
InfEdit
0.051
0.321
0.944
53%
0.098
0,250
0.952
54%
MasaCtrl
0.018
0.310
0.890
37%
0.088
0.257
0.890
66%
SDEdit
0.091
0.248
0.955
60%
Prompt2Prompt
0.170
0.280
0.850
97 %
Ours
0.101
0.322
0.929
81 %1%
0.200
0.261
0.968
93%
Table 2: CLIP and Inclusion metric results for EmuEdit and Additing Benchmark.
EXPERIMENTS
compare our method with two classes baselines: (1) Training-Free methods that leverage the existing capabilities of text-to-image models: Prompt-to-Prompl (Hertz) et al. 2022), a method which injects the attention map of the source image into the target image to preserve its structure, and SDEdit (Meng et al.] 2022), method that adds partial noise to existing image and then denoises it. Both methods were re-implemented on the FLUX.1 L.UX.1-ddv model for fair comparison. (2) Pretrained Instruction following models, specifically trained edit and
Evaluation Baselines

---

Published as a conference paper at ICLR 2025
Instruct Pix2Pix
Magic Brush
'A Coyate behind rea-heoded
'A Fork an the left oide of the plate'
bosket on front of bicycles'
Figure 5: Qualitative Results from the Emu-Edt Edit Benchmark. Unlike other methods, which fail to place the object plausible location, our method successfully achieves realistic object insertion.
Source
Ours
Prompt-to-Prompt
SDEdit
toy truck the chid's hands'
"The man is holding a shopping bag"
'A microscope on the counter next to samples'
Figure 6: Qualitative Results from the Additing Benchmark. While Prompt-to Prompt fails to align with the source image, and SDEdit fails to align with the prompt, our method offers Additing that adheres to both prompt and source image.
add objects to existing images: InstructPix2Pix (Brooks et al. 2023) an instruction following model trained large scale of synthetic instruction data, Magicbrush (Zhang et al. 2023) version of InstructPix2Pix fine-tuned on manually annotated editing dataset, and Frasedraw (Canberk et al.) 2024) model trained large dataset constructed using inpainting model. Add-it implementation details can be found in appendix A.I A.I
Metrics We evaluate the results of our method and the baselines using automatic metrics and human evaluations for each source and target image-caption pair. Automatic Metrics: we start by adopting the CLIP (Radford et al. 2021) based metrics proposed in Emu-Edit (Sheynin al. 2023): (i) CLIP dir Gal et al. 2022) measures the agreement between change captions and the change in images. (ii) CLIP img measures similarity between source and target images. (iii) CLIP out measures the target image and caption similarity. We propose two additional metrics: (iv) Inclusion measures the portions of cases the object added to the image, evaluated automatically using the open-
7

---

Published as a conference paper at ICLR 2025
vocabulary detection model Grounding-DDne DINO (Luu et al. 2023 (v) Affordance measures whether the object was added to a plausible location, utilizing Grounding-D - and a manually annotated set of possible locations. Human Evaluations: ask human raters to pick the best Additing output when faced with a source image, instruction and images generated by our method and a competing baseline. Further details in appendix A.8
4.1 EVALUATION RESULTS
Emu-Edit Benchmark Following EraseDraw (Canberk et al. (024) we evaluate our method on a subset of EmuEdit's (Sheynin et al. 2023) validation set with the task class of "Add", designed for insertion instructions. The benchmark consists of sets of images and prompts before and after an edit, and the corresponding instruction. Table [ shows our model outperforms all previous ap- proaches in the CLIP dir. CLIP out and the Inclusion metrics. In the CLIP im metric, which indicates how close the edited image is to the source image, we are second only to Erasedraw. This result is not surprising given that in 35% of the cases Erasedraw did not add object to the image (indi- cated by the Inclusion metric), artificially boosting the image similarity score. Due to the limitations of automatic metrics, especially in assessing the naturalness of edits, conducted a sead-to-head evaluation with human raters against each baseline, as shown in fig. 3 Our method's outputs were preferred in 80% of cases. Finally, we present qualitative comparison other methods using images from the EmuEdit benchmark in fig. 5 Previous methods often produce artifacts, unnatu- ral object placements, or fail to modify the image. In contrast, our method generates high-quality outputs that consider the context of the source image.
Additing Benchmark To evaluate our method against both pre-trained models and zero-shot meth- ods, which tend to perform better on generated images, we created benchmark for the Additing task. We asked ChatGPT to generate 200 sets of source and target prompts along with Additing in- structions. Using Flux, we generated images and filtered 100 sets where the instructions were viable. We report all results in Table 2, Our method outperforms all baselines on the CLIPair dir and CLIP;.. im metrics. Although Prompt-to- Prompt slightly surpasses us on CLIPout and Inclusion, it does so by heavily altering the source image, as shown by its low CLIPim score. As the EmuEdit Bench- mark, we asked human raters to compare our method against the zero-shot baseline. Our method was preferred in 90% of cases against Prompt2Prompt and 83% against SDEdit fig. 4 Finally, fig. 6 shows comparison the Additing Benchmark, where other methods struggle balance object addition, background preservation, and context, while ours produces natural, appealing outputs.
Additing Affordance Benchmark Throughout our experiments we observed that the major short- coming of existing methods is incorrect affordance, namely, objects are added at implausible loca- tions (see the basket in fig. (5). To automatically quantify affordance, we constructed an affordance benchmark. It contains 200 images and prompts, with manually annotated nounding-boxed indi- cating the plausible locations to add objects in each image. Dataset construction and evaluation protocol details are available in appendix A.6 We present the results of all methods in table [] As expected, previous methods perform poorly, with low affordance scores, particularly trained models like InstructPix2Pix. which scored as low 0.276. In contrast, Add-it scores nearly double that of the best tess-performin method, demonstrating its ability to balance information from the source im- age and target prompt. We present additional results of method, including comparisons on the MagicBrush dataset and image quality metrics, in appendix A.2 and appendix A.3
5 ANALYSIS
In this section, we analyze the attention distribution the MM-DIT block and the key components of our method to better justify our design choices. In appendix A.4 we analyze the role of positional encoding in the
mechanism.
MM-DiT Attention Distribution First, we analyze the different attention components in the ex- tended MM-DiT blocks. Recall that in the eatended-attentioo mechanism described in section 3.2 there are three token sets: the source image Xsource, the target image Xtarget and the prompt Ptarget- In our experiments, we notice that simply applying the extended attention mechanism re- sults in the target image closely following the appearance of the source image while neglecting the
8

---

Published as a conference paper at ICLR 2025
(A)
(B)
Auto
Attembien Distribuution (%)
37.6%
37.4%
39.5%
35 35.0%
Nuto
Affordance & Inclusion
22.0%
42.0%
43.6%
26.0%
33.2%
0.6
39.8%
38.5%
27.5%
26.5%
Object incusiion
0.4
Scale=1.0
Auto (Ours)
Scalem1.2
1.00
1.15
Source
Prompt
Target
Scale
Mutl Blocks
Blocks
(C)
Source
Scale=1.0
Auto (Ours)
Scale=1.2
'A ple of loundry the bosket'
Figure 7: (A) Affordance and Object Inclusion scores across weight scale values, with our automatic weight scale achieving good balance between the two. (B) Visualization of the prompt token attention spread across different sources, model blocks, and weight scales, averaged over multiple examples from a small validation set. (C) A representative example demonstrating the effect of varying target weight scales.
Vanilla Generation
Source
Affordance & Inclusion Scores
Ours
Ours
0.8
w/o Structure
0.6
Affordance
Object
0.4
None
1-966
t-933
t-900
t-833
Structure Transfer Step
Figure 8: Ablation over various steps for applying the Structure Transfer mechanism. Applying early misaligns the generated images with the source image's structure while applying it too late causes the output image to neglect the object. Our chosen step strikes a balance between both.
prompt meaning no object is added to the image. We attribute this problem to the way the attention distributed across the three sets of tokens. In particular, find empirically that the target prompt's attention A, exp pp KKource Kp, Krargee)) serves as an effective proxy for balancing the three sources of attention. simple way to control the attention distribution is by introducing scale factor 7p Ttarges so that Ap xx exp((pp - (Keourer 7pp" Kp, "tarree'' Ktan Ktarget)). In practice, we find that using Y = 7p "arget adequate. In fig. (B) we visualize the prompt attention Ap spread across the three token sets. In the standard extended-attention case (y = 1.0). the source image tokens (purple) receive more attention than the target image tokens (orange). preventing the generated image from incorporating the added object. On the other hand when scaling up too much (y = 1.2), the target image tokens overwhelm the source image token, causing the output image to stray away from the source image structure. Finally, when the scaling value balances the attention between X Xoouros and Xtarget (? = Auto), the output image successfully incorporates the added object, while preserving the target image structure and taking into account its context when placing the object. These obser- vations are qualitatively shown in fig. (C) and are also reflected in fig. (A), where the scale that balances the attention offers good balance between affordance and Object Inclusion.

---

Published as a conference paper at ICLR 2025
Affordance
Source
Map
blend
Ours
The grrl is wearing headphores'
water bottle on the bicycle'
Figure 9: Images generated by Add-it with and without the latent blending step, along with the resulting affordance map. The latent blending block helps align fine details from the source image, such as removing the girl's glasses or adjusting the shadows of the bicycles.
Ablation Study Next, we evaluate the impact of different components of our method. First, we demonstrate the effect of the weight scale, % In fig. 7 (A) present a graph showing affordance and Object Inclusion as functions of different weight scales. As the weight scale increases, the added object tends to appear more frequently in the image. However, beyond a certain threshold, the affordance score drops. This decline occurs when the target image ignores the structure of the source image, generating objects in unnatural locations, illustrated in fig. 7 (C). Next, explore the effect of latent blending. In fig. g we show output images with and without the latent blending step, along with the affordance map automatically extracted by our method. Notice how the blending step aligns the fine details of the source image without introducing artifacts. An ablation of the localization mask construction provided in appendix(A.S) Finally, we examine the structure transfer component. In fig. &lwe illustrate the effect of applying the structure transfer step different denoising process stages. When structure transfer is applied too early, the affordance score is low, meaning the target image does not adhere to the structure of the source image. On the other hand, applying it later in the process results in a lower object inclusion metric, indicating that the target image neglects the object. Ultimately, when the structure transfer is applied at 933, we achieve a balance between object inclusion and affordance. A qualitative example is provided fig. g
LIMITATIONS
Add-it shows strong performance across various benchmarks, but it has some limitations. Since the method relies on pretrained diffusion models, it may inherit biases its biases, potentially affect- ing object placement unfamiliar or highly complex scenes, as well as introducing biases such as gender bias (fig. [5p. Additionally, because our method uses target prompts rather than explicit in- structions, users may need to construct more detailed prompts to achieve the same edit. For instance, with image of dog, the prompt "A dog" won't 't add another dog to the scene. Instead, the user would need to provide more specific prompt, such as "A second dog beside the dog in the middle."
CONCLUSION
We introduced Add-it, a training free method for adding objects to images using simple text prompts. We analyzed the attention distribution in MM-DiT blocks and introduced novel mechanisms such as weighted treeddd-attentinn
and Subject-Guided Latent Blending. Additionally, we addressed a critical gap in evaluation by creating the "Additing Affordance Benchmark," which allows for an accurate assessment of object placement plausibility in image Additing methods. Add-it consistently outperforms previous approaches, improving affordance from 47% to 83% and achieving state-of- the-art results on both real and generated image benchmarks. Our work demonstrates that leveraging the knowledge in pretrained diffusion models is a promising direction for tackling challenging tasks like image Additing. As diffusion models continue to evolve, methods like Add-it it have the potential to drive further advancements in semantic image editing and related applications.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
In this work, we acknowledge the ethical considerations associated with image editing technologies. While our method enables advanced object insertion capabilities, it also has the potential for misuse, such as creating misleading or harmful visual content. We strongly encourage the responsible and ethical use of this technology, emphasizing transparency and consent in its applications. Addition- ally, biases present in pretrained models may affect generated outputs, and we recommend further research mitigate such issues future work. Human evaluations were conducted with informed consent.
REPRODUCIBILITY STATEMENT
All necessary information to reproduce Add-it is provided in section Jnnd Appl We provide the proposed "Additing Benchmark" and "Additing Affordance Benchmark" in the supplementary material of our submission.
We will open-source all the code upon publication of the paper.
ACKNOWLEDGMENTS
We thank Assaf Shocher, Lior Hirsch and Omri Kaduri for useful discussions and for providing feedback on an earlier version of this manuscript. This work was completed as part of the first author's PhD thesis Tel-Aviv University.
REFERENCES
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
Azadi, D. Pathak, S. Ebrahimi, and T. Darrell. Compositional gan: Learning image-conditional binary composition. In International Journal of Computer Vision, volume 128, pp. 2642, 2020.
Yogesh Balaji, Scungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. eDiff-I: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01224 2022.
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text- driven layered image and video editing. In ECCV, 2022.
Mikotaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint 1eXxvvv8801..0 01 401, 2018.
Black-Forest Flux: Diffusion models for layered image generation. https://github. com/ 2024. Accessed: 2024-09-24.
black-forest- www.llabsllfi satrrestlaab//flne
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
C. Wang, L. Yang. Scene design by integrating geometry and physics for realistic image synthesis. Computer Graphics Forum, 2014.
Alper Canberk, Maksym Bondarenko, Ege Ozguroglu, Ruoshi Liu, and Carl Vondrick. Erasedraw: Learning to draw step-by-step via erasing objects from images. 2024.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In ICCV, 2023.
W. Choi, Y. W. Chao, C. Pantofaru, and S. Savarese. Context-driven scene understanding from a single image. In ICCV, 2012.
11

---

Published as a conference paper at ICLR 2025
Gilad Deutch, Rinon Gal, Daniel Garibi, Or Patashnik, and Daniel Cohen-Or. Turboedit: Text-based image editing using few-step diffusion models, 2024.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high- high-resolution image synthesis, 2024.
M. Fisher, D. Ritchie, M. Savva, T. Funkhouser, and P. Hanrahan. Example-based synthesis of 3d object arrangements. In ACM Transactions on Graphics (TOG), 2012.
Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-
Or. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions Graphics (TOG), 1-1: 2022.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-promp image editing with cross attention control. arXiv preprint wrXiv:2208.01626 2022.
Inbar Huberman- Spiigellll,, Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. arXiv:2304.0m 06..0011400... 2023.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In CVPR, 2023.
Y. Lee, Z. Tseng, and P. Abbeel. Relaxedplacemennt Learning to synthesize compositional scene layouts with object relations. In Computer Vision and Pattern Recognition (CVPR), 2022.
D. Lin, S. Fidler, and R. Urtasun. Holistic scene understanding for 3d object detection with rgbd cameras. ICCV, 2013.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jic Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint Xiiv:2303.05499 2023.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
O. Michel, A. Bhattad, E. VanderBilt, R. Krishna, A. Kembhavi, and T. Gupta. Object3dit: Language-guin guided 3d-aware image editing. Advances in Neural Information Processing Systems, 36, 2024.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. CVPR, 2023.
Nobuyuki Otsu. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics, 9()) 66, 1979. doi: 10. 101100/M...999.43100000
Zhihong Pan, Riccardo Gherardi, Xiufeng Xie, and Stephen Huang. Effective real image editing with accelerated iterative diffusion inversion. In ICCV, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8768-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXivv2204.061125 2022.
Nikhila Ravi, Valentin Gabeur, Yuan- Soan-TingTan Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.007. 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
12

---

Published as a conference paper at ICLR 2025
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorcalistic text-to-iman -oittee diffusion models with deep language understanding. NeurIPS, 2022.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Dvir Samuel, Barak Meiri, Nir Darshan, Shai Avidan, Gal Chechik, and Rami Ben-Ari. Regularized newton raphson inversion for text-to-image diffusion models, 2023.
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint orXiv:2010.02502, 2020.
Bram Wallace, Akash Gokul, and Nikhil Vijay Naik. EDICT: Exact diffusion inversion via coupled transformations. CVPR, 2022.
Navve Wasserman, Noam Rotstein, Roy Ganz, and Ron Kimmel. Paint by inpaint: Learning to add image objects by removing them first, 2024.
Sihan Xu, Yidong Huang, Jiayi Pan, Ziqiao Ma, and Joyce Chai. Inversion-free image editing with natural language. 2024.
Kai Zhang, Lingho Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: manually annotated dataset for instruction-guided image editing. In NeurlPS, 2023.
W. H. Zhao, J. Jiang, J. Weng. J. He, E. P. Lim, H. Yan, and X. Li. Image-based contextual ad- vertisement recommendation. Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, 2011.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 IMPLEMENTATION DETAILS
Add-it When evaluating Add-ir, we use Istruct 933 for generated images and Estruct 867 for real images and Iilend 500. For the scaling factor y we use the root-finding solver described in section 3.2 on set of validation images and set to 1.05, as it is close to the average result and performs well in practice. We generate the images with 30 denoising steps, building upon the diffusers implementation of the FLUX.I-dey model. We apply the extended attention mechanism until step 670 in the multi-stream blocks, and step = 340 for the single-stream blocks.
Computation Time To compare the computational efficiency of Add-it with the original FLUX model, we generated 200 images using each method and reported the average generation time along with the standard error of the mean. The base FLUX model requires 6.4+0.13 seconds to generate a single image, while Add-it takes 7.2 7.23= seconds per image- an increase of just under one second. This slight increase is primarily due to caching attention maps and the additional computations required for the extended attention mechanism. SAM2 contributes only minimally to this difference, with an average inference time of 0.04 seconds.
Latent Blending Localization To extract refined object mask part of the Subject Guided Latent Blending component, we begin by extracting sub- attention maps. Empirically, we find that the best-performing layers for 13",
ject
this task are:
("cransformer.block "traansformerrbbookkkk.
" www.errrrnnooooooooooooo bloccks...........oooooookkkk... 33"]. To refine the mask from these attention maps, we need identify points use as prompts for SAM-2. To extract points from the attention map, we first select the point with the highest attention value. Then, we exclude the area around the chosen point and select the next highest point. This process repeated until we either identify 4 points or the current maximal point value falls below 0.35 Pmaaas where Pmoz is the initial maximum attention value. Finally, we feed the points to the SAM-2 model to end up with a refined object mask.
standing
horse wearing pirke dress'
is wearing chown!
'A Kright riding the hot se"
in the gorder!
'A yourg gir locking at camera'
"The wearing raincoat
'A heart heart-shaped patch on rancoat"
"A loend is penched on grrto choulder"
Figure 10: Step-by-Step Generation: Add-it can generate images incrementally, allowing it to better adapt user preferences each step.
14

---

Published as a conference paper at ICLR 2025
Source
Output
Source burny citting on the chaik
Output
Source
Output
erompphone shiploying a mapp
The mibble dag weardig het
breaannn of Clowars the boskaet
bottle in the sidee of the back-arkk
woven lookaat with loundry
pormot the maris dhoukker
A person pointing of the exjittion e-imc"..
Rower her hor above her ear
Figure 11: Qualitative results of our method on the Additing Affordance Benchmark show that our method successfully adds objects naturally and in plausible locations.
choraacter holding dariond sward
kright ridiig the
bird pariched pieno
Face paioo the racket
girt walking the Forest
wearthy wuardd
Figure 12: Our method can operate on non-photorealistic images.
A.2 ADDITIONAL RESULTS
In fig. we present step-by-step outputs generated with Add-it. Notice that the scene remains unchanged, while each prompt adds additional "layer" to the final image, resulting in more complex scene.
In fig. we show additional results from the Additing Affordance benchmark. In each case, the object must be added to specific location in the source image. Across all examples, Add-it suc- cessfully places the object in a plausible location, preserving the natural appearance of the image.
In fig. we demonstrate that Add-it can operate on pon-oootorcalistic source images, such as paint- ings and pixel art. Since method requires tuning, we preserve all the generation capabilities of the base model.
In fig. ent initial noise. Our method preserves the diversity of the base model, enabling users to generate multiple variations of the added object until they find the desired one.
show various generation results produced by model, each originating from differ-
15

---

Published as a conference paper at ICLR 2025
In fig. 14 we demonstrate Add-ir's ability to edit existing objects in image. Although our method was originally designed for object Additing, it can also perform edits such as changing a person's hair color, eye color, or his shirt.
In fig. 13 we highlight that biases in the pre-trained diffusion model can also manifest in Add-it. In this figure, we prompt method to add doctor empty hospital room without specify- ing gender or any additional details. We display multiple random generations from different noise initializations and observe that, in all cases, the generated doctor is male. We attribute this bias to underlying biases in FLUX, which our method inherits.
'A dog sitting on the bench"
Source
Variation 1
Variation
Variation
Variation 4
Variation 5
Variation
Variation 7
Figure 13: Our method generates different outputs when given different starting noises. All the outputs remain plausible.
A.3 ADDITIONAL COMPARISONS
To further evaluate Add-it, we compare it against three additional baselines: MasaCtrl (Cao et al, 2023), training-fiee free editing method that conditions
source image using mutual self-attention mechanism; InfEDIT (Xu et al. 2024), which leverages Latent Consistent Models for training- and inversion-free image editing: and Paint by Inpaint (Wasserman et al.) 2024), model trained large inpainting dataset to add objects to images. We present the affordance scores of these baselines in table evaluated the Additing Affordance Benchmark. Consistent with previous evaluations, these methods achieve low scores, demonstrating their inability to consistently insert objects into plausible locations in the scene. We further evaluate the three baselines on both the EmuEdit dataset and the Additing benchmark, with the results shown in table 4 Add-it outperforms all baselines, with the exception of the CLIP metric on the EmuEdit dataset. Notably, the inclusion metric for these baselines lags significantly behind our method, highlighting common failure case where these methods are unable to successfully add the new object to the scene.
We expanded our evaluation by comparing Add-it to existing baselines on the test split of the Mag- icBrush Benchmark, filtering for examples with insertion instructions only. In table 5 we present a quantitative comparison of Add-it against other baselines using CLIP and Inclusion metrics, Notably, our method outperforms all baselines, including the MagicBrush model itself. fig. [6 provides qualitative comparison, showcasing examples where Add-it successfully adds objects like dog and a squirrel plausible locations, seamlessly integrating them into the scene. contrast, other methods struggle with both coherent object generation and proper placement.
Finally, we include two widely used metrics to assess image quality and diversity: KID (Binkkoowkii al. 2018), which measures how closely generated images match the distribution of real images
16

---

Published as a conference paper at ICLR 2025
Source
with ginger heir
wearing pirk shr+
with
blue eyes
A yellw car
sports car
A boot
Figure 14: Add-is can perform editing of existing objects the images, such as changing the man's hair color, switching the car for boat.
Source
doctor standing hospital room
Figure 15: Add-it can suffer from bias of the underlying diffusion model, such as generating mostly male doctors.
(lower better), and Inception Score (Salimnss et al. 2016)) which evaluates both image quality and diversity based on a pre-trained inception model (higher is better). We chose KID over FID as performs more reliably on smaller evaluation sets. While these metrics are traditionally used for image generation, focus is on image Additing. To adapt them for our task, use off- ffff-the-shelf open vocabulary detection model to identify regions containing added objects across the dataset. We then crop these regions to create dataset of object crops and evaluate KID and IS by comparing this dataset against the source images before editing. This approach allows us to assess the quality and diversity of the inserted objects directly. Our method, along with SDEdit and Prompt2Prompt, achieves the lowest KID scores, indicating high quality of the added objects. Additionally. Add-it and Prompt2Prompt attain the highest IS scores, demonstrating that the inserted objects are also diverse.
InfEdit
MasaCtrl 0.203
Paint By Inpaint Ours 0.828
Affordance
0.366
0.311
Table 3: Comparison of methods based on Affordance score for the Additing Affordance Bench- mark.
17

---

Published as a conference paper at ICLR 2025
Emu Edit
Additing Benchmark
Method
CLIP dir
CLIP out
CLIPam 2m
Inc.
CLIPas der
CLIP out
CLIP im
Inc.
InfEdit
0.051
0.321
0.944
53%
0.098
0.250
0.952
54%
MasaCtrl
0.018
0.310
0.890
37%
0.088
0.257
0.890
66%
Paint by Inpaint
0.071
0.316
0.955
58%
0.079
0.246
0.954
68%
Ours
0.101
0.322
0.929
81%
0.200
0.261
0.968
93%
Table 4: CLIP and Inclusion metric results for EmuEdit and Additing Benchmark.
Method
CLIP dàr
CLIP out
CLIP im
Inc.
InstructPix2Pix
0.077
0.297
0.917
45%
EraseDraw
0.117
0.301
0.934
58%
MagicBrush
0.144
0.303
0.905
72%
Paint by Inpaint
0.098
0.300
0.928
62%
Ours
0.124
0.307
0.937
85%
Table 5: CLIP and Inclusion metric results for the MagicBrush Benchmark.
Source
Ours
Erasedraw "A squirrel admines the flower vase"
Instruct Pix2Pix
Magicbrush
Paint by Inpaint
"A dog resting under the bench"
Figure 16: Qualitative results from the MagicBrush benchmark show that our method generates high-quality objects placed correctly within the scene, whereas other methods struggle with both object quality and placement.
Method
KID
IS
InstructPix2Pix
0.034
4.173
EraseDraw
0.027
4.473
MagicBrush
0.040
4.243
Paint by Inpaint
0.030
3.762
SDEdit
0.016
2.783
Prompt2Prompt
0.014
4.901
Ours
0.015
4,949
Table 6: KID and Inception Score metrics used to assess the quality and diversity of edits by Add-it and other baselines.
A.4 THE ROLE OF POSITIONAL ENCODING
Here, we examine the significance of positional encodings in the extended attention mechanism. fig. [7 demonstrates their role through simple experiment: applied method to source image where the positional encoding vectors were shifted down and to the right. This misalignment resulted in mismatch between the positional encoding of the child's head in the source and target
18

---

Published as a conference paper at ICLR 2025
Source Image
w/ PE Shift
w/o PE Shift
Figure 17:
Positional Encoding Analysis: shifting the positional encoding of the source image sults in a corresponding shift in the object's location in the generated image.
images. Consequently, instead of generating headsets at the actual position of the child's head, the model produced them in the area corresponding the "shifted head" position. This outcome demonstrates that the model heavily relies on positional information to transfer features between the source and target images. Despite the target image containing "laptop" features instead of "head" features at the relevant location, the model chose to place the headphones there. This decision was based on the area having the same positional encoding as the "head area" in the source image, rather than on the actual content of the target image at that location. We believe further research on the role of positional encoding vectors an interesting direction for future work in the context of DiT models.
A.5 LATENT BLENDING MASK CONSTRUCTION ABLATION
Here, we examine different methodologies for generating the object mask used as input to the Latent Blending step. We qualitatively display these options in fig. 18 First, compare the results using the coarse attention mask extracted from the attention layers, as described in appendix A.] Notably, the attention maps tend to capture the main parts of the person but often exclude details such legs, resulting in incomplete and messy masks. Next, we consider the mask generated by SAM2 when prompted with the coarse mask input. many cases, the coarse mask proves be an inadequate prompt for SAM2, leading to refined masks that still fail to capture the entire object. Finally, we present our chosen methodology. discussed in appendix A.I where we extract key points from the attention mask and use them as input to SAM2. This approach produces refined mask that captures the entire object, resulting a seamless integration the person into the scene. Additionally, table shows the affordance scores for each methodology, demonstrating that our proposed point-based algorithm yields the best results.
Coarse Attention Mask
Mask Prompt SAM
Ours
Affordance
0.809
0.77
0.828
Table 7: Ablation of blending mask construction methods based Affordance score for the Addit- ing Affordance Benchmark.
A.6 Additing AFFORDANCE BENCHMARK
Affordance in Image Additing In section I we highlight significant gap in current object inser- tion evaluation protocols: the lack of mechanism assess whether an object was added a plau- sible location. Our experiments reveal that existing methods often struggle to find the "right" spot for object placement, which we refer to as affordance. To address this, we introduced the "Additing Affordance Benchmark". The guiding principle behind the benchmark is the need for automated method to evaluate whether an image editing method has inserted an object in a plausible location within the scene. Since developing an automated approach to assess correct object placement is itself an unsolved challenge, we created a dataset consisting of images, insertion instructions, and carefully labeled regions indicating plausible insertion areas for each instruction. This allows
19

---

Published as a conference paper at ICLR 2025
Coarse Attention Mask
Mask Prompt SAM
Ours
Mask Input
Source
Blend Mask
Output
Figure 18: Qualitative ablation of blending mask construction: While the coarse attention mask and the mask-prompted SAM mask may not perfectly segment the added object, our point-based algorithm typically generates precise masks, leading to a seamless integration of the object into the scene.
use an off-the-shelf open-vocabular detector (as described below) to determine whether an added object's bounding box lies within the manually labeled region, thereby indicating whether was placed in plausible location. As shown in section 4 the Additing Affordance Benchmark reveals that all existing image addition methods struggle with this requirement, highlighting a critical gap in the field that needs to be addressed develop effective object insertion methods.
Dataset Construction Here, we provide the details for constructing the Additing Affordance Benchmark dataset. First, we used ChatGPT-4 to generate dataset of tuples, each consisting of source prompt and target prompt, representing an image before and after object insertion, along with an instruction for the transition and a subject token representing the object to be added. The exact prompt is shown in fig. 20 Next, used FLUX. 1-dev to generate the source images from the source prompts in each tuple. We manually filtered out images where the object had no plausible location many possible locations, resulting in dataset of 200 images. Finally, manually annotated bounding boxes for each image, marking the plausible locations where the object could
be added, as shown in fig. 19
Evaluation protocol: Given set of an Additing model output images, use Grounding-DINe DINO to detect the area where new objects were added and set the affordance score of single image to be the fraction of added object that least 0.5 of their area falls inside the GT box.
A.7 PROMPTING WITH Add-it
In contrast to instruction-based editing methods, Add-it operates target prompts that describe the edited image, including the added object. Example prompts appear the qualitative figures throughout the paper. In fig. 21 we demonstrate that LLM, such ChatGPT, easily convert between these representations. Moreover, our EmuEdit evaluation is conducted using automatically converted captions with the exact same prompt showing its effectiveness.
20

---

Published as a conference paper at ICLR 2025
A person porrtig at the expaition cerrec2
bottl in the the brack pack.
The faddy loear hos riad bow tic.
The mádele dog waarinng hef
lourny sitting on the chear
Figure 19:
Visual examples from the Additing Affordance Benchmark. Each image is annotated with bounding boxes highlighting the plausible areas where the object can be added.
Please generate JSON list of 300 sets. Each set consists of: an index, source prompt, instruction, a target prompt, and a subject token.
The source prompt describes source image.
The target prompt describes the source image after an object has been added to it.
The instruction is a description of what needs to be changed to go from the source to the target prompt.
The subject token is the noun that refers the added object, single word that appears in the target prompt. Here are is an example:
"src_ perc_prompt": "A person sitting on a chair", stgt_prompt"; "A scarf wrapped around their neck", "subjectttoken "scarf",
"instruction": "Wrap scarf around the person's s neck." "
Only generate examples where there is clearly
only one possible place for the object to be added, so it can be tagged correctly.
Write it as JSON list yourself.
Please DO NOT include negative examples in your prompts, such as "a man wearing hat" in the source prompt.
DO NOT write code; Return only the JSON list.
Figure 20: The prompt provided to ChatGPT in order to generate the Affordance Benchmark.
A.8 USER STUDY DETAILS
We evaluate the models through an Amazon Mechanical Turk user study using two-alternative forced choice protocol. In the study, raters saw an instruction, a source image, and two edited im- ages, each produced by different approach. They chose the edit that best followed the instruction, taking into account: image quality and realism, instruction following and preservation of the source image. For the evaluation, each head-to head-to-heer to-dadd example rated by two raters. In fig. 22 we show an example of a single trial a rater has seen.
21

---

Published as a conference paper at ICLR 2025
Given an input caption and an editing instruction, generate an output caption that reflects how an input image, corresponding to the input caption, would appear after being edited according to the instruction.
Here are few example:
input_caption: A hotel bed ready for use instruction: Add suitcase to the bed output_caption: A hotel bed with a suitcase on it
input_caption: A table with two plates of breakfast food instruction: Add fork the table between the plates output_captions A table with two plates of breakfast food with a fork between them.
Figure 21: The prompt provided to ChatGPT in order to generate output prompts from instruction- based prompts appearing in benchmark such as EmuEdit.
22

---

Published as a conference paper at ICLR 2025
Select the edited image that best follows insertion instruction below.
Focus on how well the object was inserted while maintaining quality the appearance of the source image.
Instructions
Read these instructions carefully.
will
images
source image
Pick the image that best follows the instructin.
Focus on these factors:
Image quality realism:
qigh-quality
perfectly
distortions.
Instruction accuracy: The image must match the object addition exaclly as described in instruction.
Preservation of the source image: The rest of the image should have only necessary changes.
Choose image.
Instruction: Add football jersey to the skateboarder
Source Image:
Edit
Source Image:
21
Choose
Submit
Figure 22: One trial of the Additing user study.
23


Reviews:
[{"review_id": "u5Zy999BDQ", "reviewer": "Reviewer_rVMn", "summary": "This paper introduces Add-it, a novel, training-free method for inserting objects into images based on text instructions. The approach extends multi-modal self-attention mechanisms in Diffusion Transformer to incorporate a source image as an additional input, reweighting the attention values to effectively balance information between source and target images. Furthermore, the method transfers structural details from the source image and blends the latent representation with guidance from SAM2. To validate Add-it, the authors propose a new benchmark and a custom evaluation metric, demonstrating its effectiveness.", "questions": "- Did the authors experiment with using a coarse mask as an input for SAM2? If so, how does it perform compared to point-based prompts?\n- In the ablation study, the authors claim that setting \\gama = auto yields the best results. However, Figure 7(a) appears to show a peak at approximately 1.07, which surpasses the performance of the auto setting.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Clarity and Presentation.** The paper is well-written and easy to follow. The figures are thoughtfully designed and effectively illustrate the proposed method.\n- **Visual Quality.**  The results are visually compelling; the added objects are integrated seamlessly, with the original image's details preserved well.", "weaknesses": "- **Dataset Size.** The evaluation affordance dataset is relatively small, containing only 200 images, which may limit the generalizability of the findings.\n\n- **Computation Speed.** The need to run additional models (e.g., SAM2) for mask estimation could impact processing speed, especially in comparison to the original FLUX. It would be valuable to provide a direct comparison regarding computational efficiency.\n\n- **Prompt Complexity.** *Add-it* requires text prompts for target images and thus may introduce additional complexity. Previous methods like InstructPix2Pix or MagicBrush may offer a more straightforward user experience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gHP2OnKV4O", "reviewer": "Reviewer_5Sc6", "summary": "Add-it leverages pretrained diffusion models to insert objects into images based on text prompts, without requiring any training. \nThe method extends the multi-modal attention mechanism to incorporate information from the source image, prompt, and generated image, resulting in natural object placement.  \nAdd-it also introduces a novel “Additing Affordance Benchmark” to evaluate object placement plausibility.", "questions": "Please refer to the weakness. Anyway, from my view as a Diffusion Models researcher, adding objects to images is a pretty hard task. Although this paper is having several limitations, I still like the basic idea.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper proposes a novel training-free method called Add-it for adding objects to both real and generated images using simple textual prompts. The method doesn't require fine-tuning or training a separate model for object insertion. Instead, it directly leverages the knowledge embedded within pretrained text-to-image diffusion models. \n\n2. Add-it extends the multi-modal attention mechanism of diffusion models to carefully balance information from three key sources: the source image, the text prompt, and the generated image itself. \n\n3. Add-it introduces a novel Subject-Guided Latent Blending mechanism to preserve the fine details of the source image while enabling necessary adjustments like shadows and reflections.", "weaknesses": "1. Lack of comparison with the existing methods for adding objects [1,2]. Both of them have codes and have been released half a year ago.\n\n2. The paper doesn't fully discuss the potential limitations of relying on SAM-2 for refining the object mask. SAM is a large powerful segmentation model, might not always accurately capture the desired object boundaries. There are also papers using self-attention and cross-attention maps to relieve this requirement [3,4].\n\n3. The paper only briefly mentions potential biases inherited from the pretrained diffusion models. These biases could impact the object placement in unexpected ways. For instance, the model might consistently place objects associated with certain demographics in specific locations based on learned biases, potentially leading to unfair or discriminatory outcomes. \n\n[1] Paint by Inpaint: Learning to Add Image Objects by Removing Them First\n[2] ObjectAdd: Adding Objects into Image via a Training-Free Diffusion Modification Fashion\n[3] Localizing Object-level Shape Variations with Text-to-Image Diffusion Models\n[4] Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vWRoXnjjT9", "reviewer": "Reviewer_mir1", "summary": "This paper introduces Add-it, a training-free method leveraging extended attention mechanisms to seamlessly integrate objects into images from text instructions. Besides, it shows a new benchmark for evaluating object placement plausibility.", "questions": "1. Could you please briefly outline the advantages of the benchmark you've proposed over the existing benchmarks? It would be particularly helpful if you could illustrate these advantages with some statistical data.\n2. Have you experimented with modifying the input to incorporate multiple target objects, or just edit it? eg. in the second line of the fig 6 in your paper, the prompt change to \"The man is holding a shopping bag in the rain.\" or \"The man wears a red shirt\".\n3. Although your paper focuses on the 'Add-it' method, I'm curious to know if this editing approach could also be adapted to perform operations like 'remove something' from the image.\n4. could you please provide more details about Structure Transfer?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing is concise, which makes the paper easy to follow and understand.\n2. The method is straightforward and innovatively.\n3. The experimental results substantiate the effectiveness of the method proposed in this paper.", "weaknesses": "1. The newly introduced benchmark's advantages over other benchmarks are not clearly articulated, and there is a lack of comparison with more existing benchmarks, like MagicBrush. \n2. The paper lacks comparison with traditional image quality metrics such as FID and IS, which are commonly used to evaluate the quality of generated images in terms of their realism and diversity.\n3. Moreover, while a new benchmark is proposed, there seems to be a scarcity of details regarding it in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WaunTdYZ7K", "reviewer": "Reviewer_9iYS", "summary": "The paper proposes a tuning-free method for adding objects into images based on text instructions. They introduce Add-it, a training-free approach that extends the attention mechanisms of diffusion models to incorporate information from three key sources: the scene image, the text prompt, and the generated image itself. The experimental results show that it effectively maintains structural consistency and fine details while ensuring natural object placement compared to existing approaches.", "questions": "The findings in lines 210-212 are interesting. I am curious whether this phenomenon arises from differences in the text encoder or the diffusion model architecture. Could the authors provide insights into the underlying reasons for this observation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a training-free method that achieves state-of-the-art results in object insertion.\n2. The analysis of MM-DiT Attention Distribution is interesting and introduces an effective mechanism to control their contribution.\n3. The paper Introduces a benchmark to assess the plausibility of object insertion.\n4. The writing and presentation are mostly clear and easy to follow. The authors provide hyperparameters and detailed settings used in experiments and method implementation, which is helpful for reproducibility.", "weaknesses": "The proposed techniques are incremental, lacking sufficient novelty to make a significant contribution to the field. The WEIGHTED EXTENDED SELF-ATTENTION, STRUCTURE TRANSFER, and SUBJECT GUIDED LATENT BLENDING methods are commonly used in Unet-based free-tuning image editing [1,2,3,4,5], and the modifications presented here do not represent a substantial advancement.\n\n- In the WEIGHTED EXTENDED SELF-ATTENTION section, the approach of applying a coefficient to the key is similar to existing techniques in the community, such as scaling text embeddings [1]. This makes the proposed contribution appear incremental rather than innovative.\n- In the STRUCTURE TRANSFER section, the proposed technique shows only minor improvements compared to SDEdit [2], without introducing a fundamentally new approach.\n- In the SUBJECT GUIDED LATENT BLENDING section, similar blending operations have been previously demonstrated in [3, 4, 5]. Furthermore, using SAM to assist image blending, while effective, is a straightforward extension and lacks the originality needed to stand out.\n- The selection of tuning-free editing baselines is also inadequate. Methods such as Masactrl [3], Proximal Guidance [4], and InfEdit [5] provide more sophisticated attention or mask control mechanisms. I recommend including a more comprehensive comparison with these existing baselines to better contextualize the contribution of the proposed method.\n\nOther error:\nIncorrect usage of quotation marks on line 237: The word 'choose' is improperly quoted.\n\n[1] https://huggingface.co/docs/diffusers/using-diffusers/weighted_prompts#prompt-weighting\n\n[2] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon, \"SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations,\" International Conference on Learning Representations (ICLR), 2022.\n\n[3] M. Cao, X. Wang, Z. Qi, Y. Shan, X. Qie, and Y. Zheng, \"MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing,\" Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 22560-22570.\n\n[4] L. Han, S. Wen, Q. Chen, Z. Zhang, K. Song, M. Ren, R. Gao, A. Stathopoulos, X. He, Y. Chen, D. Liu, Q. Zhangli, J. Jiang, Z. Xia, A. Srivastava, and D. N. Metaxas, \"ProxEdit: Improving Tuning-Free Real Image Editing with Proximal Guidance,\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2024, pp. 4279-4289\n\n[5] S. Xu, Y. Huang, J. Pan, Z. Ma, and J. Chai, \"Inversion-Free Image Editing with Language-Guided Diffusion Models,\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 9452-9461.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)