Magic Insert: Style-Aware Drag-and-Drop

## Abstract

Abstract We present Magic Insert , a method for dragging-and-dropping subjects from a user-provided image into a target image of a different style in a physically plausible manner while matching the style of the target image. This work formalizes the problem of style-aware drag-and-drop and presents a method for tackling it by addressing two sub-problems: style-aware personalization and realistic object insertion in stylized images . For style-aware personalization, our method first fine-tunes a pretrained text-to-image diffusion model using LoRA and learned text tokens on the subject image, and then infuses it with a CLIP representation of the target style. For object insertion, we use Bootstrapped Domain Adaption to adapt a domain-specific photorealistic object insertion model to the domain of diverse artistic styles. Overall, the method significantly outperforms traditional approaches such as inpainting. Finally, we present a dataset, SubjectPlop, to facilitate evaluation and future progress in this area.

## 1 Introduction

Large text-to-image models have recently made significant progress in generating high-quality images. However, to make these models truly useful, controllability is essential. Users have diverse needs and want to interact with these models in different ways depending on their specific use case. Influential work has been done to enable controllability in these networks, robustly addressing foundational applications and controls such as subject personalization, style learning, layout controls, and semantic controls. Despite this progress, the full potential of these powerful large models has not been fully realized. Some applications that seemed clearly out of reach just a couple of years ago are now possible with careful approaches.

We present one such application: style-aware drag-and-drop. We formalize this problem and introduce Magic Insert, our method to tackle it, which shows strong performance compared to current baselines. One might initially consider addressing style-aware drag-and-drop by trying to inpaint using a stylized subject, for example by combining Dreambooth [], StyleDrop [], and inpainting. We find that approaches of this type are very expensive and achieve subpar results.

In developing Magic Insert, we address two interesting sub-problems: style-aware personalization and realistic object insertion in stylized images. For style-aware personalization, there have been attempts on adjacent problems, such as learning a style and then representing a specific subject in that style [, ], or combining pre-trained custom style and subject models [, ]. Recent style work suggests that fast style learning is possible, but fast learning of a subject, including all the intricacies of identity, is a much harder problem that has arguably not been solved yet [, , , ]. We propose leveraging learnings from both domains and settle on a solution that uses adapter injection of style paired with subject-learning in the embedding and weight space of a diffusion model.

One key idea we propose is to not attempt inpainting directly into an image after achieving style-aware personalization. Instead, for best results, we first generate a high-quality subject and then insert that subject into the target image. To achieve our results, we introduce an innovation called Bootstrap Domain Adaptation, that allows progressive retargeting of a model’s initial distribution to a target distribution. We apply this idea to adapt a subject insertion network that has been trained on real images to perform well on the stylized image domain, enabling the insertion of our generated stylized subject into the background image.

Our method allows the generated output to exhibit strong adherence to the target style while preserving the essence and identity of the subject, and for realistic insertion of the stylized subject into the generated image. The method also provides flexibility in terms of the degree of stylization desired and how closely to adhere to the original subject’s specific details and pose (or allow more novelty in the generation).

In summary, we propose the following contributions:

- •
We propose and formalize the problem of style-aware drag-and-drop, where a subject (a character or object) is dragged from one image into another. Specifically, in our problem formulation the subject reference image and the target image may be in vastly different styles, and the plausibility and realism of the subject insertion is important.
- •
In order to encourage exploration into this new problem, we present SubjectPlop, a dataset of subjects and backgrounds that span widely different styles and overall semantics. We will release this dataset for public use, as well as our evaluation suite.
- •
We propose Magic Insert, a method to tackle the style-aware drag-and-drop problem. Our method is composed of a style-aware personalization component and a style-consistent drag-and-drop component.
- •
For style-aware personalization, we demonstrate strong
and consistent results using subject-learning in the embedding and weight space of a pre-trained diffusion models, along with adapter injection of style.
- •
For drag-and-drop, we propose Bootstrapped Domain Adaptation, a method that allows for progressive retargeting of a model’s initial distribution unto a target distribution. We use this to adapt an object insertion network trained on real images to perform well on the stylized image domain.

## 2 Related Work

#### Text-to-Image Models

Recent text-to-image models such as Imagen [], DALL-E 2 [], Stable Diffusion (SD) [], Muse [] and Parti [] have demonstrated remarkable capabilities in generating high-quality images from text descriptions. They leverage advancements in diffusion models [, , ] and generative transformers. Our work builds on top of SDXL [] and the LDM architecture [].

#### Image Inpainting

The task of filling masked pixels of a target image has been explored using a wide range of approaches: Generative adversarial networks [] e.g. [, , , , , ] and end-to-end learning methods [, , , ]. More recently, diffusion models enabled significant progress [, , , , ]. Such inpainting methods are a precursor to many object insertion approaches.

#### Generative Object Insertion

The problem of inserting an object into an existing scene has been originally explored using Generative Adversarial Networks (GANs) []. [] breaks down the task into two generative modules, one determines where the inserted object mask should be and the other determines what the mask shape and pose. ShadowGAN [] addresses the need to add a shadow cast by the inserted object, leveraging 3D rendering for training data. More recent works use diffusion models. Paint-By-Example [] allows inpainting a masked area of the target image with reference to the object source image, but it only preserves semantic information and has low fidelity to the original object’s identity. Recent work also explores swapping objects in a scene while harmonizing, but focuses on swapping areas of the image which were previously populated []. There also exists an array of work that focuses on inserting subjects or concepts in a scene either by inpainting [, ] or by other means [, ] - these do not handle large style adaptation and inpainting methods usually suffer from problems with insertion such as background removal, incomplete insertion and low quality results. ObjectDrop [] trains a diffusion model for object removal/insertion using a counterfactual dataset captured in the real world. The trained model can insert segmented objects into real images with contextual cues such as shadows and reflections. We build upon this novel and incredibly useful paradigm by tackling the challenging domain of stylized images instead.

#### Personalization, Style Learning and Controllability

Text-to-image models enable users to provide text prompts and sometimes input images as conditioning input, but do not allow for fine-grained control over subject, style, layout, etc. Textual Inversion [] and DreamBooth [] are pioneering works that demonstrated personalization of such models to generate images of specific subjects, given few casual images as input. Textual Inversion [] and follow-up techniques such as P+ [] optimize text embeddings, while DreamBooth optimizes the model weights. This type of work has also been extended to 3D models [], scene completion [] and others. There also exists work on fast subject-driven generation [, , , , ]. Other work allows for conditioning on new modalities such as ControlNet [] and on image features (IP-Adapter []). There is a body of work that dives more deeply into style learning and generating consistent style as well with StyleDrop [] as a pioneer, with newer work that achieves fast stylization [, , , ], or combines subject models with style models like ZipLoRA [] and others []. Our work leverages ideas from Textual Inversion, DreamBooth and IP-Adapter to unlock style-aware personalization prior and combine it with subject insertion.

Figure: Figure 2: Style-Aware Personalization: To generate a subject that fully respects the style of the target image while also conserving the subject’s essence and identity, we (1) personalize a diffusion model in both weight and embedding space, by training LoRA deltas on top of the pre-trained diffusion model and simultaneously training the embedding of two text tokens using the diffusion denoising loss (2) use this personalized diffusion model to generate the style-aware subject by embedding the style of the target image and conducting adapter style-injection into select upsampling layers of the model during denoising.
Refer to caption: extracted/5705895/figure/style_aware_personalization.png

Figure: Figure 3: Subject Insertion: In order to insert the style-aware personalized generation, we (1) copy-paste a segmented version of the subject onto the target image (2) run our subject insertion model on the deshadowed image - this creates context cues and realistically embeds the subject into the image including shadows and reflections.
Refer to caption: extracted/5705895/figure/subject_insertion_inference.png

## 3 Method

### 3.1 Style-Aware Drag-and-Drop Problem Formulation

We formalize the style-aware drag-and-drop problem as follows. Let $\mathcal{I}_{s}$ and $\mathcal{I}_{t}$ denote the space of subject and target images, respectively. The space of subject images consists of images of solely the subject in front of plain backgrounds. Given a subject image $x_{s}\in\mathcal{I}_{s}$ and a target image $x_{t}\in\mathcal{I}_{t}$, our goal is to generate a new image $\hat{x}_{t}\in\mathcal{I}_{t}$ such that:

- 1.
The subject from $x_{s}$ is inserted into $\hat{x}_{t}$ in a semantically consistent and realistic manner, accounting for factors such as occlusion, shadows, and reflections.
- 2.
The inserted subject in $\hat{x}_{t}$ adopts the style characteristics of the target image $x_{t}$ while preserving its essential identity and attributes from $x_{s}$.

Formally, we aim to learn a function $h:\mathcal{I}_{s}\times\mathcal{I}_{t}\rightarrow\mathcal{I}_{t}$ that satisfies:

$$ $h(x_{s},x_{t})=\hat{x}_{t}\quad\text{s.t.}\quad\hat{x}_{t}\sim p(\hat{x}_{t}|x _{t},x_{s})$ (1) $$

where $p(\hat{x}_{t}|x_{t},x_{s})$ represents the conditional distribution of the output image given the subject and target images. This distribution encapsulates the desired properties of semantic consistency, realistic insertion, and style adaptation.
To learn the function $h$, we decompose the problem into two sub-tasks: style-aware personalization and realistic object insertion in stylized images. Style-aware personalization focuses on generating a subject that adheres to the target image’s style while maintaining its identity. Realistic object insertion aims to seamlessly integrate the stylized subject into the target image, accounting for the scene’s geometry and lighting conditions.
By addressing these sub-tasks, we can effectively solve the style-aware drag-and-drop problem and generate visually coherent and compelling results. In the following sections, we present our dataset and the components of our proposed method.

### 3.2 SubjectPlop Dataset

To facilitate the evaluation of the style-aware drag-and-drop problem, we introduce the SubjectPlop dataset and make it publicly available. As this is a novel problem, a dedicated dataset is crucial for enabling the research community to make progress in this area.

SubjectPlop consists of a diverse collection of subjects generated using DALL-E3 [] and backgrounds generated using the open-source SDXL model []. The dataset includes various subject types, such as animals and fantasy characters, and both subjects and backgrounds exhibit a wide range of styles, including 3D, cartoon, anime, realistic, and photographic. The diversity in color hues and lighting conditions ensures comprehensive coverage of different scenarios for evaluation. No real people are represented in the dataset.

The dataset comprises 20 distinct backgrounds and 35 unique subjects, allowing for a total of 700 possible subject-background pairs. The entire dataset is meant for evaluation of the task. This rich set of test cases enables the assessment of performance and generalization capabilities of style-aware drag-and-drop techniques. By introducing SubjectPlop, we aim to provide a standardized benchmark for evaluating and comparing different approaches to the style-aware drag-and-drop problem. We believe this dataset will serve as a valuable resource for researchers and practitioners working in image manipulation and generation, fostering further advancements in this area.

### 3.3 Style-Aware Personalization

Figure: Figure 4: Bootstrapped Domain Adaptation: Surprisingly, a diffusion model trained for subject insertion/removal on data captured in the real world can generalize to images in the wider stylistic domain in a limited fashion. We introduce bootstrapped domain adaptation, where a model’s effective domain can be adapted by using a subset of its own outputs. (left) Specifically, we use a subject removal/insertion model to first remove subjects and shadows from a dataset from our target domain. Then, we filter flawed outputs, and use the filtered set of images to retrain the subject removal/insertion model. (right) We observe that, the initial distribution (blue) changes after training (purple) and initially incorrectly treated images (red samples) are subsequently correctly treated (green). When doing bootstrapped domain adaptation, we train on only the initially correct samples (green).
Refer to caption: extracted/5705895/figure/bootstrap_domain_adaptation.png

Our style-aware personalization approach is illustrated in Figure 2. Let $f_{\theta}$ denote a pre-trained diffusion model with parameters $\theta$. Given a subject image $x_{s}\in\mathcal{I}_{s}$, our method personalizes $f_{\theta}$ on $x_{s}$ in both the weight and embedding space, similar to DreamBooth [] and Textual Inversion [].

In the first step, we train LoRA [] (Low-Rank Adaptation) deltas $\Delta_{\theta}$ to produce an efficiently fine-tuned adapted model $f_{\theta^{\prime}}$ where $\theta^{\prime}=\theta+\Delta_{\theta}$, while preserving the model’s original capabilities. Simultaneously, we learn embeddings $e_{1},e_{2}\in\mathbb{R}^{d}$ for two personalized text tokens, where $d$ is the embedding dimensionality. We use two learned embeddings since we found better performance for both subject preservation and editability in this configuration. The LoRA deltas and and embeddings are jointly trained using the diffusion denoising loss:

$$ $\mathcal{L}_{\text{joint}}=\mathbb{E}_{t,\epsilon}\left[\|\epsilon-\epsilon_{ \theta^{\prime}}(x_{s}^{t},t,[e_{1};e_{2}])\|_{2}^{2}\right]$ (2) $$

where $t\sim\mathcal{U}(0,1)$, $\epsilon\sim\mathcal{N}(0,\mathbf{I})$, $x_{s}^{t}=\sqrt{\bar{\alpha}_{t}}x_{s}+\sqrt{1-\bar{\alpha}_{t}}\epsilon$, and $\epsilon_{\theta^{\prime}}$ is the noise prediction of the adapted model $f_{\theta^{\prime}}$. The joint optimization of $\Delta_{\theta}$, $e_{1}$, and $e_{2}$ is performed using the loss $\mathcal{L}_{\text{joint}}$. These personalized text tokens $[e_{1};e_{2}]$ serve as a compact representation of the subject’s identity. By performing embedding and weight-space learning simultaneously, We find that performing embedding and weight-space learning simultaneously, with two text tokens, captures the subject’s identity more strongly while allowing sufficient editability to introduce the target style.

In the second step, we leverage the personalized diffusion model $f_{\theta^{\prime}}$ to generate the style-aware subject $\hat{x}_{s}$. To infuse the target image $x_{t}$’s style into $\hat{x}_{s}$, we employ style injection. Specifically, we generate a style embedding $e_{t}=\text{CLIP}(x_{t})$ of $x_{t}$ using a frozen CLIP encoder CLIP. We then use a frozen IP-Adapter model v to inject $e_{t}$ into a subset of the UNet blocks of $f_{\theta^{\prime}}$ during inference:

$$ $\hat{x}_{s}=f_{\theta^{\prime}}([e_{1};e_{2}],\textit{v}(e_{t}))$ (3) $$

This approach is similar to InstantStyle [], with injection into the upsample block that is adjacent to the midblock, with some key differences being omitting content/style embedding separation, and injecting into a personalized model. To the best of our knowledge, our central idea of combining adapter injection and personalized models remains unexplored in the published literature.
This ensures that $\hat{x}_{s}$ maintains the subject’s identity while adopting $x_{t}$’s style characteristics.

By combining style-aware personalization with style injection, our method generates subjects that harmoniously blend into the target image while retaining their essential identity, effectively tackling the first challenge of style-aware drag-and-drop and enabling the creation of visually coherent and style-consistent results.

### 3.4 Bootstrapped Domain Adaptation for Subject Insertion

In this section, we address the problem of subject insertion and propose a novel solution using bootstrapped domain adaptation. We formalize the concept of bootstrapped domain adaptation and describe the dataset used for this purpose.
Subject insertion is a crucial component of the style-aware drag-and-drop problem, as it involves seamlessly integrating a stylized subject into a target background image. While diffusion-based inpainting approaches [, , ] can be used for this, they still face challenges such as generating content in smooth regions, producing incomplete figures, erasing objects behind inserted subjects, and having problems with boundary harmonization. We take a simpler and stronger approach, which is to insert the subject by copying and pasting it into the target image, and then subsequently generating contextual cues such as shadows and reflections [] in a second step. Unfortunately, existing subject insertion models are trained on data captured in the real world, severely limiting their ability to generalize to images with diverse artistic styles.

Let $\mathcal{D}_{r}$ denote the distribution of real-world images and $\mathcal{D}_{s}$ denote the distribution of stylized images. Existing subject insertion models are trained on samples from $\mathcal{D}_{r}$, but our goal is to adapt them to perform well on samples from $\mathcal{D}_{s}$. To overcome this limitation, we introduce bootstrapped domain adaptation, a technique that enables a model to adapt its effective domain by leveraging a subset of its own outputs. As illustrated in Figure 4 (left), we employ a subject removal/insertion model $g_{\theta}$ trained on real-data ([] in our case) to first remove subjects and shadows from a dataset $\mathcal{S}\sim\mathcal{D}_{s}$ belonging to our target domain. Subsequently, we filter out flawed outputs and obtain a filtered set of images $\mathcal{S}^{\prime}\subseteq\mathcal{S}$, which we use to retrain the subject removal/insertion model. Filtering can be done using human feedback or automatically given a quality evaluation module.

The bootstrapped domain adaptation process can be formalized as follows:

$$ $\omega=\arg\min_{\omega}\mathbb{E}_{(x,y)\sim\mathcal{S}^{\prime}}\mathcal{L}( g_{\omega}(x),y)$ (4) $$

where $\omega$ denotes the adapted model parameters, $\mathcal{L}$ is the diffusion denoising loss, and $(x,y)$ are pairs of input images and corresponding subject removal/insertion ground truths from the filtered set $\mathcal{S}_{f}$. The concept of bootstrapped domain adaptation is based on the surprising observation that a diffusion model trained for subject insertion/removal on real-world data can generalize to a wider stylistic domain to a limited extent. By retraining the model on its own filtered outputs, we can effectively adapt its domain to better handle stylized images.

Figure: Figure 5: Results Gallery: Examples of our Magic Insert method for different subjects and backgrounds with vastly different styles.
Refer to caption: extracted/5705895/figure/gallery.png

Figure 4 (right) demonstrates the effect of bootstrapped domain adaptation on the model’s distribution. The initial distribution, represented as $p_{\omega}(x)$, evolves after training, becoming $p_{\omega^{*}}(x)$. Images that were initially treated incorrectly, shown as samples from $\mathcal{D}_{s}\setminus\mathcal{S}^{\prime}$, are subsequently handled correctly, as indicated by their inclusion in $\mathcal{S}^{\prime}$. During the bootstrapped domain adaptation process, we train the model only on the initially correct samples from $\mathcal{S}^{\prime}$ to further refine its performance on the target domain. Several steps of bootstrapped domain adaptation can be performed, further enhancing the model’s performance. In our work we find that one step suffices, with a small set of samples (around 50). Figure 7 shows results with and without bootstrap domain adaptation.

To facilitate the bootstrapped domain adaptation process, we curate a dataset $\mathcal{S}$ specifically tailored to this task. The dataset comprises a diverse range of stylized images, selected to represent the target domain $\mathcal{D}_{s}$. In our case, this dataset is constructed by sampling from different text-to-image generative models with diverse prompts that elicit prominent subjects with shadows and reflections in a variety of global styles. By finetuning the subject removal/insertion model on this dataset using the bootstrapped domain adaptation technique, we enable it to effectively handle subject insertion in the context of style-aware drag-and-drop.

Figure: Figure 6: LLM-Guided Affordances: Examples of an LLM-guided pose modification for Magic Insert, with the LLM suggesting plausible poses and environment interactions for areas of the image and Magic Insert generating and inserting the stylized subject with the corresponding pose into the image.
Refer to caption: extracted/5705895/figure/affordances.png

**Table 1: Subject Fidelity Comparisons. We compare our method for subject fidelity (DINO, CLIP-I, CLIP-T Simple, CLIP-T Detailed) across different methods. Our method variants show high subject fidelity.**
| Method | DINO $\uparrow$ | CLIP-I $\uparrow$ | CLIP-T Simple $\uparrow$ | CLIP-T Detailed $\uparrow$ | Overall Mean $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| StyleAlign Prompt | 0.223 | 0.743 | 0.266 | 0.299 | 0.383 |
| StyleAlign ControlNet | 0.414 | 0.808 | 0.289 | 0.294 | 0.451 |
| InstantStyle Prompt | 0.231 | 0.778 | 0.283 | 0.300 | 0.398 |
| InstantStyle ControlNet | 0.446 | 0.806 | 0.281 | 0.283 | 0.454 |
| Ours | 0.295 | 0.829 | 0.276 | 0.293 | 0.423 |
| Ours ControlNet | 0.514 | 0.869 | 0.289 | 0.308 | 0.495 |

## 4 Experiments

**Table 2: Style Fidelity Comparisons. We compare our method for style fidelity (CLIP-I, CSD, CLIP-T). Our method variants show strong style-following.**
| Method | CLIP-I $\uparrow$ | CSD $\uparrow$ | CLIP-T $\uparrow$ | Overall Mean $\uparrow$ |
| --- | --- | --- | --- | --- |
| StyleAlign Prompt | 0.570 | 0.150 | 0.248 | 0.323 |
| StyleAlign ControlNet | 0.575 | 0.188 | 0.274 | 0.345 |
| InstantStyle Prompt | 0.583 | 0.312 | 0.276 | 0.390 |
| InstantStyle ControlNet | 0.588 | 0.334 | 0.279 | 0.400 |
| Ours | 0.560 | 0.243 | 0.268 | 0.357 |
| Ours ControlNet | 0.575 | 0.294 | 0.274 | 0.381 |

**Table 3: ImageReward Metric Comparisons. We compare different methods using the ImageReward metric, which correlates with human preference for aesthetic evaluation. Higher scores indicate better performance. Our variants outperform all benchmarks**
| Method | ImageReward Score $\uparrow$ |
| --- | --- |
| StyleAlign Prompt | -1.1942 |
| StyleAlign ControlNet | -0.5180 |
| InstantStyle Prompt | -0.4638 |
| InstantStyle ControlNet | -0.2759 |
| Ours | -0.2108 |
| Ours ControlNet | -0.1470 |

In this section, we show experiments and applications. Our full method enables insertion of arbitrary subjects into images with diverse styles, with a large expanse of text-guided semantic modifications. Specifically, not only does the subject retain its identity and essence while inheriting the style of the target image, but we can modify key subject characteristics such as the pose and other core attributes such as adding accessories, changing appearance, changing shapes, or even species hybrids (Figure 9). These changes can be integrated with components such as LLMs that allow for automatic affordances and environment interactions (Figure 6).

### 4.1 Style-Aware Drag-and-Drop Results

#### Magic Insert Results

We present a gallery of qualitative results in Figure 5 to highlight the effectiveness and versatility of our method. The examples span a wide range of subjects and target backgrounds with vastly different artistic styles, from photorealistic scenes to cartoons, and paintings. For style-aware personalization we use the SDXL model [], and for subject insertion we use our trained subject insertion model based on a latent diffusion model architecture.

In each case, our method successfully extracts the subject from the source image and blends it into the target background, adapting the subject’s appearance to match the background’s style. Notice how the inserted subjects take on the colors, textures, and stylistic elements of the target images. The coherent shadows and reflections enhance the plausibility of the results.

#### LLM-Guided Affordances

Our proposed style-aware personalization method allows for large changes in character pose, with support from the diffusion model prior. Using and LLM (ChatGPT 4o) we are able to generate LLM-guided affordances for different subjects, by feeding an instruction prompt, the full background image, and the section of the background image in which the character will be positioned. Using these LLM suggestions, we can generate the character following these poses and environment interactions and insert it in the appropriate space. With this, we show in Figure 6 a first attempt at the previously unassailable task of inserting subjects into images realistically with automatic interactions with the scene.

#### Bootstrap Domain Adaptation

We show in Figure 4 a sample case of subject insertion with an insertion model that is trained on real images without adaptation, and on the same model that uses our proposed bootstrap domain adaptation on a small set of 50 samples. Insertion without bootstrap domain adaptation generates subpar results, with problems such as missing shadows, reflections and even added distortions.

#### Semantic Modifications of Subject

Our method inherits all benefits of DreamBooth [] and thus allows for modification of subject characteristics such as pose, adding accessories, changing appearance, shapeshifting and hybrids. We show some examples in Figure 9. The generated subjects can then be inserted into the background image.

#### Editability / Fidelity Tradeoff

Our method (w/o ControlNet) also inherits DreamBooth’s editability / fidelity tradeoff. Specifically, the longer the personalization training, the stronger the subject fidelity but the lesser the editability. This phenomenon is shown in Figure 10. In most cases a sweet spot can be found for different applications. For our work we use 600 iterations with batch size 1, a learning rate of 1e-5 and weight decay of 0.3 for the UNet. We also train the text encoder with a learning rate of 1e-3 and weight decay of 0.1.

**Table 4: User Study. This study evaluates our method against two different baselines (StyleAlign ControlNet and InstantStyle ControlNet) based on subject identity, style fidelity, and realistic insertion. Participants ranked each method by preference.**
| Method | User Preference $\uparrow$ |
| --- | --- |
| Ours over StyleAlign ControlNet | 85% |
| Ours over InstantStyle ControlNet | 80% |

Figure: Figure 7: Bootstrap Domain Adaptation: Inserting a subject with the pre-trained subject insertion module without bootstrap domain adaptation generates subpar results, with failure modes such as missing shadows and reflections, or added distortions and artifacts.
Refer to caption: extracted/5705895/figure/bootstrap_results.png

Figure: Figure 8: Style-Aware Personalization Baseline Comparison: We show some comparisons of our style-aware personalization method with respect to the top performing baselines StyleAlign + ControlNet and InstantStyle + ControlNet. We can see that the baselines can yield decent outputs, but lag behind our style-aware personalization method in overall quality. In particular InstantStyle + ControlNet outputs often appear slightly blurry and don’t capture subject features with good contrast.
Refer to caption: extracted/5705895/figure/comparison_style_personalization.png

Figure: Figure 9: Style-Aware Personalization with Attribute Modification: Our method allows us to modify key attributes for the subject, such as the ones reflected in this figure, while consistently applying our target style over the generations. This allows us to reinvent the character, or add accessories, which gives large flexibility for creative uses. Note that when using ControlNet this capability disappears.
Refer to caption: extracted/5705895/figure/attribute_modification.png

Figure: Figure 10: Editability / Fidelity Tradeoff: We show the phenomenon of editability / fidelity tradeoff by showing generations for different finetuning iterations of the space marine (shown above the images) with the “green ship” stylization and additional text prompting “sitting down on the floor”. When the style-aware personalized model is finetuned for longer on the subject, we get stronger fidelity to the subject but have less flexibility on editing the pose or other semantic properties of the subject. This can also translate to style editability.
Refer to caption: extracted/5705895/figure/slider_space_marine.png

### 4.2 Comparisons

Here we introduce baselines, as well as quantitative and qualitative comparisons, as well as a user study. Specifically, our proposed baselines utilize the StyleAlign [] and InstantStyle [] stylization methods, which can generate images in reference styles given either inversion or embedding of the reference image. We combine these methods with either sufficiently detailed prompting guided by a VLM (ChatGPT 4) or edge-conditioned ControlNet. For prompting we use the VLM to describe the subjects while eliminating style cues, and for edge-conditioning we use Canny edges extracted from the subject reference images to guide the stylized outputs using ControlNet.

#### Baseline Comparisons

We run studies in order to compare the performance of subject stylization for different baselines and our style-aware personalization method. We study the performance of these methods on subject fidelity, style fidelity, and human preference.

For subject fidelity (Table 1), our proposed variants achieve high scores across various subject fidelity metrics (DINO, CLIP-I, CLIP-T Simple, CLIP-T Detailed). DINO and CLIP-I metrics are identical to those presented in DreamBooth [] and CLIP-T Simple / Detailed denotes the CLIP similarity between the output image CLIP embedding and the CLIP embedding of simple and detailed text prompts describing the subject, which are in turn generated by ChatGPT 4.

Regarding style fidelity (Table 2), our proposed variants demonstrate strong style-following performance using CLIP-I [, ], CSD [], CLIP-T [, ] metrics. For style fidelity, InstantStyle ControlNet outperforms our variants using these automatic metrics, although we observe that subject details and contrast is lost in many of these samples as shown in Figure 8. For this, we also compute ImageReward [] scores in Table 3, which correlate strongly with human preference in aesthetic evaluations. We observe that our variants strongly outperform the benchmarks.

Moreover, finding strong quantitative metrics for subject fidelity and for style fidelity is an open problem in the field, and metrics can have strong biases that can make them suboptimal. Again, we show some examples for our proposed style-aware personalization, along with top baseline contenders StyleAlign ControlNet and InstantStyle ControlNet in Figure 8. We observe that the generation quality of our variants is stronger than the benchmarks, especially with both strong stylization performance while still retaining the essence of the subjects. Our Magic Insert + ControlNet variant is powerful given that it exactly follows the outline of the character, and thus has the strongest subject fidelity over all approaches, although it does not have the desirable properties of our method w/o ControlNet which include pose, form and attribute modification of the subject.

#### User Study

Following previous work [, , , ] we perform a robust user study to compare our full method (w/ ControlNet) with the strongest baselines: StyleAlign ControlNet and InstantStyle ControlNet. We recruit a total of 60 users (4 sets of 15 users) to answer 40 evaluation tasks (2 sets of 20 tasks) for each baseline comparison (2 baseline comparisons). We collect a total of 1200 user evaluations. We ask users to rank their preferred methods with respect to subject identity preservation, style fidelity with respect to the background image, and realistic insertion of the subject into the background image. We show the results in Table 4. We observe a strong preference of users for our generated outputs compared to baselines.

## 5 Societal Impact

Magic Insert aims to enhance creativity and self-expression through intuitive image generation. However, it inherits concerns common to similar methods, such as altering sensitive personal characteristics and reproducing biases from pre-trained models. Our experiments have not shown significant differences in bias or harmful content compared to previous work, but ongoing research is crucial. As more powerful tools emerge, developing safeguards and mitigation strategies is essential to address potential societal impacts. This includes reducing bias in training data, developing robust content filtering, and promoting responsible use. Balancing the benefits of creativity with ethical considerations requires continuous dialogue with the broader community.

## 6 Conclusion

In this work, we introduced the problem of style-aware drag-and-drop, a new challenge in the field of image generation that aims to enable the intuitive insertion of subjects into target images while maintaining style consistency. We proposed Magic Insert, a method that addresses this problem through a combination of style-aware personalization and style insertion using bootstrapped domain adaptation. Our approach demonstrates strong results, outperforming baseline methods in terms of both style adherence and insertion realism.

To facilitate further research on this problem, we introduced the SubjectPlop dataset, which consists of subjects and backgrounds spanning a wide range of styles and semantics. We believe that our contributions, including the formalization of the style-aware drag-and-drop problem, the Magic Insert method, and the SubjectPlop dataset, will encourage exploration and advancement in this exciting new area of image generation.

Acknowledgements. We thank Daniel Winter, David Salesin, Yi-Hsuan Tsai, Robin Dua and Jay Yagnik for their invaluable feedback.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: The questions are the discussion to the existing works including stylized object composition and object removal for insertion training.  Besides that I also find some writing have problem. For example "Line 269 to Line 271" same words  have been written in two similar sentences.
soundness: 2
strengths: The key contribution in this paper is forming the combination of object insertion and stylized personal reference object generation.  With the two well studied problem, the proposed method trying define a new settings for image editing especially on image composition.  Besides, this paper demonstrated a unified pipeline for solving the proposed problem.  Especially, the object insertion quality has been greatly improved with high quality data with the proposed filtering schemes.  Experiments also shown many visual plausible results and demonstrated its effectiveness on real applications.
summary: This paper presents a new problem named style aware drag-and-drop, which tries to insert an object to a new stylized image where the insert object is based on a reference concept and the inserted object is also adaptively changed the style from a new background image. The proposed method contains two step: First, the proposed method finetune a personalized text to image diffusion model by LoRA and then leverage the clip embedding to inject the style from a new background to the generated concept for stylized subject generation. Second, the authors proposed a bootstrapped method for selecting high quality foreground object removed results which has along with all the effects such shadows/reflections removals. Then the pure background image and original image will formed as a supervised image pairs for object insertion model training.  To demonstrate the problem, this paper also introduced a new dataset named subjectplot, which contains the   20 subjects and 35 background images.
weaknesses: There are three concerns for the method:
1. The "magic insertion" actually has been studied in a similar manner in previous work [1] and [2][3].  [1] may be a concurrent work but [2][3] have been showing that image composition on stylized images. There methods needs more discussion. 

2. The "Bootstrapped Domain Adaption" is actually a very similar way to object drop [4] but it just in a reversed way.  What's the difference between these two works also  needs to be discussed.

3. The identity shift problem has been seen in most showing cases.  How to deal with it, seems missing in the paper.  Especially, that would be a serious issue for image composition. Why people want  new objects ? It also conflicts with the argument that inpainting is not suitable for the problem due to generating new objects. 

[1]SwapAnything: Enabling Arbitrary Object Swapping in Personalized Visual Editing.
[2]Painterly Image Harmonization. 
[3]Painterly Image Harmonization by Learning from Painterly Objects. 
[4] ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object Removal and Insertion

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: Why do you use 2 tokens to learn the object? Could you show some 1 token example (which is used by many personalized generation papers by default. )
soundness: 3
strengths: This work proposes to solve the problem of object insertion while keeping the subject's identity. Personalized image generation or composition has been widely studied in recent years. However, the task of inserting objects into the background with style harmonization is less explored. This work provides insight into how to adapt the object style to match the background style.
summary: This paper proposes a method that can insert objects into a background with style consistency with the background while keeping the object's identity. The main contributions are the proposed method for this task and an evaluation dataset SubjectPlop.
weaknesses: 1. The biggest concern is identity preservation. This paper shows some results on simple objects like unreal 3D objects, objects without much textures. I would like to see more diverse objects, especially real-world objects with complex textures, scene texts, and logos. Otherwise, the identity preservation of this work will not be convincing. 
2. The comparison methods should include the general object insertion or image composition methods which are not exactly performing style matching but may have similar effects on this task. Please refer to the papers I listed below. 




[1] Song, Y., Zhang, Z., Lin, Z., Cohen, S., Price, B., Zhang, J., ... & Aliaga, D. (2024). Imprint: Generative object compositing by learning identity-preserving representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8048-8058).
[2] Yang, B., Gu, S., Zhang, B., Zhang, T., Chen, X., Sun, X., ... & Wen, F. (2023). Paint by example: Exemplar-based image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 18381-18391).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please see **Weaknesses**.
soundness: 3
strengths: 1. This paper aims to achieve style-aware drag-and-drop, which is an interesting and challenging problem.
2. This paper provides a new dataset consists of subjects and backgrounds that span widely different styles and overall semantics.
3. The generated images of the proposed method seem plausible.
summary: This paper aims to enable the dragging-and-dropping of subjects from a user-provided image into a target image of a different style in a physically plausible manner, while matching the style of the target image. To achieve this goal, this paper proposes a method called Magic Insert which combines style-aware personalization and insertion through bootstrapped domain adaptation.
weaknesses: 1. The so-called 'style-aware drag-and-drop' problem explored in this paper is quite similar to the earlier task of image composition/image blending, which entails seamlessly incorporating the given object into the specific visual context without altering the object’s appearance while ensuring natural transitions [1,2,3]. Therefore, I'm concerned that 'style-aware drag-and-drop' cannot be considered a new problem. Moreover, this paper lacks an introduction and comparison of these highly related methods. \
[1] PrimeComposer: Faster Progressively Combined Diffusion for Image Composition with Attention Steering. ACM MM 2024. \
[2] Paint by Example: Exemplar-based Image Editing with Diffusion Models. CVPR 2023. \
[3] TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Composition. ICCV 2023.

2. The novelty of this paper is insufficient. In style-aware personalization, this paper combines existing style-aware personalization and style injection techniques. In subject insertion, this paper fine-tunes existing subject insertion models. These efforts seem more like an engineering contribution to me.

3. The effect of the proposed bootstrap domain adaptation is quite subtle, as shown in Figure 7. And no quantitative experiments are conducted to further demonstrate its effectiveness.

4. This paper provides limited qualitative results, and there is some redundancy (i.e., some examples appear more than once), raising concerns about potential cherry-picking.

5. No ablation studies are conducted in this paper. The effects of some proposed schemes are not explored and verified. For example, what are the respective roles of the two text tokens? Why using two text tokens is more effective than using just one?

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: Could you elaborate on the robustness of the bootstrapped domain adaptation process? How sensitive is the final model to the selection and quality of initial filtered outputs?
How were the specific subjects and backgrounds for SubjectPlop chosen, and do you plan to expand the dataset to include more styles or scenarios?
The paper mentions that training the style-aware personalization on two learned text tokens improved both subject preservation and editability. Was any systematic study performed to confirm this dual-token approach’s advantages?
soundness: 3
strengths: This paper introduces a two-part solution for style-aware drag-and-drop. The combination of style-aware personalization and domain-adapted insertion is interesting for applications requiring coherence between different visual styles.

 The proposed bootstrapped domain adaptation, which re-trains models using their own filtered outputs, shows practical effectiveness in enhancing insertion realism, with attention to shadows and reflections for a seamless integration.

The SubjectPlop dataset provides a valuable resource for evaluating and comparing models designed for stylized drag-and-drop tasks. This data addition will help future research.

The paper employs various metrics to assess subject fidelity and style alignment, along with a user study to capture subjective preferences. This adds robustness to the evaluation.
summary: The paper introduces Magic Insert, a method for style-aware drag-and-drop, enabling the transfer of subjects from one image into a target image with a different style while preserving the subject’s identity and integrating the target style. This approach combines style-aware personalization and object insertion within stylized contexts.
The method leverages a pre-trained text-to-image diffusion model, fine-tuned with LoRA and CLIP representations, for style-adaptive subject representation and introduces Bootstrapped Domain Adaptation to generalize object insertion from real images to diverse artistic styles.
Additionally, the paper introduces SubjectPlop, a dataset with various subjects and background styles to evaluate and benchmark style-aware drag-and-drop models.
weaknesses: While the paper compares against baselines like StyleAlign and InstantStyle, the effectiveness of the comparisons could be improved. Clarifying the exact metrics used for these baseline models would strengthen the argument, as some performance differences (e.g., style fidelity vs. subject fidelity) are only discussed qualitatively.

The method is very complex as the pipeline involves multiple complex steps, including LoRA training, CLIP embedding, adapter injection, and bootstrapped domain adaptation. Justification of each component  with a clear explanation of how it contributes to the entire network will improve contribution of each component. 

The paper does not fully explore the limitations of its approach, particularly around scenarios where style and subject may conflict heavily (e.g., abstract or minimalistic styles). A more explicit acknowledgment of when and where Magic Insert might struggle would provide a balanced perspective.