$\textit{Bifr\"ost}$: 3D-Aware Image Compositing with Language Instructions
Authors: Lingxiao Li
Kaixiong Gong
Wei-Hong Li
Xili Dai
Tao Chen
Xiaojun Yuan
Xiangyu Yue
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Compositing
Image Generation
Diffusion Model
Primary area: diffusion_based_models

Abstract: This paper introduces $\textit{Bifröst}$, a novel 3D-aware framework that is built upon diffusion models to perform instruction-based image composition. Previous methods concentrate on image compositing at the 2D level, which fall short in handling complex spatial relationships ($\textit{e.g.}$, occlusion). $\textit{Bifröst}$ addresses these issues by training MLLM as a 2.5D location predictor and integrating depth maps as an extra condition during the generation process to bridge the gap between 2D and 3D, which enhances spatial comprehension and supports sophisticated spatial interactions. Our method begins by fine-tuning MLLM with a custom counterfactual dataset to predict 2.5D object locations in complex backgrounds from language instructions. Then, the image-compositing model is uniquely designed to process multiple types of input features, enabling it to perform high-fidelity image compositions that consider occlusion, depth blur, and image harmonization. Extensive qualitative and quantitative evaluations demonstrate that $\textit{Bifröst}$ significantly outperforms existing methods, providing a robust solution for generating realistically composited images in scenarios demanding intricate spatial understanding. This work not only pushes the boundaries of generative image compositing but also reduces reliance on expensive annotated datasets by effectively utilizing existing resources in innovative ways.

BIFRÖST: 3D-Aware Image compositing with Language Instructions
Lingxiao Li
Kaixiong Gong' Weihong Li
Xili Dai²
Tao Chen
Xiaojun Yuan' Xiangyu Yue
MMLab, The Chinese University Hong Kong
The Hong Kong University of Science and Technology (Guangzhou)
Fudan University UUiieersity of Electronic Science and Technology of China Lingxiao-li/BBifrsst li/Bifrost
attps://githur wwt.iitt...o..iieieeeeeeeeeeeeee commlingxiao
"Place the apple behind burger"
"Replace the with the dog"
"Place the backpuck to the left of the vase"
"Place the toy behind the bucket and in front of the box"
Identity Preserved Inpainting
Identity Transfer
Figure 1:
Bifröstt adept at precise, arbitrary object placement and replacement in a background image with a reference object and a language instruction, and achieves 3D-aware high-fidelity harmonized compositing results: Bottom Left: Given coarse mask, Bifrrött can change the pose of the object to follow the shape of the mask; Bottom Right: Our model adapts the identity the reference image to the target image without changing the pose.
Bifröst results on various personalized image compositing tasks. Top:
Abstract
This paper introduces Bifrröst, a novel 3D-aware framework that is built upon diffu- sion models to perform instruction-based image composition. Previous methods concentrate image compositing at the 2D level, which fall short in handling complex spatial relationships (e.g., occlusion). Bifröst addresses these issues by training MLLM as a 2.5D location predictor and integrating depth maps as an extra condition during the generation process bridge the gap between 2D and 3D, which enhances spatial comprehension and supports sophisticated spatial inter-
Corresponding author
38th Conference Neural Information Processing Systems (NeurIPS 2024).

---

actions. Our method begins by fine-tuning MLLM with a custom counterfactual dataset predict 2.5D object locations in complex backgrounds from language instructions. Then, the smage-compositing model is uniquely designed to process multiple types of input features, enabling it to perform high-fidelity image compo- sitions that consider occlusion, depth blur, and image harmonization. Extensive qualitative and quantitative evaluations demonstrate that Bijrrstt significantly out- performs existing methods, providing a robust solution for generating realistically composited images in scenarios demanding intricate spatial understanding. This work not only pushes the boundaries of generative image compositing but also reduces reliance on expensive annotated datasets by effectively utilizing existing resources in innovative ways.
1 Introduction
Image generation has flourished alongside the advancement of diffusion models (Song et al., 2021; Ho et al., 2020; Rombach et al., 2022; Ramesh et al., 2022). Recent works (Saharia et al., 2022; Liu et al., 2023b; Brooks et al., 2023; Zhang et al., 2023b; Huang et al., 2023; Li el al., 2023; Chen et al., 2024; He el al., 2024) add conditional controls, e.g. text prompts, scribbles, skeleton maps to the diffusion models, offering significant potentials for controllable image editing. Among these methods for image editing, generative object-level image compositing (Yang al., 2023; Song et al., 2023; Chen et al., 2024; Song et al., 2024) is a novel yet challenging task that aims to seamlessly inject an outside reference object into given background image with specific location, creating cohesive and realistic image. This ability is significantly required in practical applications including E-commerce, effect-image rendering, poster-making, professional editing, etc.
Achieving arbitrary personalized object-level image compositing necessitates a deep understanding of the visual concept inherent to both the identity of the reference object and spatial relations of the background image. To date, this task has not been well addressed. Paint-by-Exann -ixampe- (Yang et al., 2023) and Objectstitch (Song ef al., 2023) use a target image as the template to edit a specific region of the background image, but they could not generate ID ID-consistent contents, especially for untrained categories. On the other hand, (Chen et al., 2024; Song et al., 2024) generate objects with ID (identity) preserved in the target scene, but they fall short in processing complicated 3D geometry relations (e.g., the occlusion) methods mainly either 1) fail to achieve both ID preservation and background harmony, or 2) do not explicitly take into account the geometry behind the background and fail to accurately composite objects and backgrounds in complex spatial relations.
they only consider 2D-le composition. To sum up, previous
We conclude that the root cause of aforementioned issues is that image composition is conducted at 2D level. Ideally, the composition operation should be done in a 3D space for precise 3D geometry relationships. However, accurately modeling 3D scene with any given image, especially with only one view, is non-trivial and time-consuming (Liu et al., 2023b). To address these challenges, we introduce Bifröst, which offers a 3D-aware framework for image composition without explicit 3D modeling. We achieve this by leveraging depth to indicate the 3D geometry relationship between the object and the background. In detail, our approach leverages a multi-modal large language model (MLLM) as a 2.5D location predictor (i.e. bounding box and depth for the object in the given background image). With the predicted bounding box and depth, method yields depth map for the composited image, which is fed into a diffusion model as guidance. This enables our method to achieve good ID preservation and background harmony simultaneously, as it is now aware of the spatial relations between them, and the conflict at the dimension of depth is eliminated. In addition, MILM enables our method to composite images with text instructions, which enlarges the application scenario of Bifröst. Bifröst achieves significantly better visual results than the previous method, which in turn validates our conclusion.
We divide the training procedure into two stages. In the first stage, we finetune an MLLM (e.g., LLaVA (Liu et al., 2023a, 2024)) for 2.5D predictions of objects in complex scenes with language instructions. In the second stage, we train the image composition model. To composite images with complicated spatial relationships, we introduce depth maps as conditions for image generation. In addition, we leverage an ID extractor to generate diseriminative ID tokens and a frequency-award detail extractor to extract the high- high-frequency 2D shape information (detail maps) for ID preservation. We unify the depth maps, ID tokens, and detail maps to guide a pre-trained coxt-to-mmag diffusion model to generate desired image composition results. This two-stage training paradigm allows
In Norse mythology, a burning rainbow bridge that transports anything between Earth and Asgard.

---

us to utilize a number of existing 2D datasets for common visual tasks e.g., visual segmentation, and detection, avoiding collecting large-volume text-image data specially designed for arbitrary object-level image compositing.
Our main contributions can be summarized as follows: 1) We are the first to embed depth into the image composition pipeline, which improves the ID preservation and background harmony simultaneously. 2) We delicately build a counterfactual dataset and fine-tuned MLLM powerful tool to predict 2.5D location of the object in given background image. Further, the fine-tuned MLLM enables our approach to understand language instructions for image composition. 3) Our approach has demonstrated exceptional performance through comprehensive qualitative assessments and quantitative analyses on image compositing and outperforms other methods, Bifröst allows us to generate images with better control of occlusion, depth blur, and image harmonization.
2
Related Work
2.1 Image compositing with Diffusion Models
Image compositing, an essential technique in image editing, seamlessly integrates a reference object into a background image, aiming for realism and high fidelity. Traditional methods, such image harmonization (Jiang et al., 2021; Xue al., 2022; Guerreiro et al., 2023; Ke al., 2022) and blending (Pérez et al., 2003; Zhang et al., 2020, 2021; Wu et al., 2019) primarily ensure color and lighting consistency but inadequately address geometric discrepancies. The introduction of diffusion models (Ho et al., 2020; Sohl-Dickstein et al., 2015; Song et al., 2021; Rombach et al., 2022) has shifted focus towards comprehensive frameworks that address all facets of image compositing. Methods like (Yang et al., 2023; Song et al., 2023) often use CLiP- adapters to utilize pretrained models, yet they compromise the object's identity preservation, focusing mainly on high-level semantic representations. More recent studies prioritize maintaining the appearance in generative object compositing. Notable developments in this field include AnyDoor (Chen et al., 2024) and ControlCom (Zhang et al., 2023a). AnyDoor integrates DINOv2 (Oquab et al., 2023) with a high-free bigh-frequency filter, while ControlCom introduces local enhancement module, both improving appearance retention. However, these approaches still face challenges spatial correction capabilities. Most recent work IMPRINT (Song et al., 2024) trains an -D-preserving encoder that enhances the visual consistency of the object while maintaining geometry and color harmonization. However, none of the work can deal with composition with occlusion and more complex spatial relations. In contrast, our models novelly propose a 3D-aware generative model that allows more accuracy and complex image compositing while maintaining geometry and color harmonization.
2.2 LLM with Diffusion Models
The open-sourced LlaMA (Touvron et al., 2023; Chiang al., 2023) substantially enhances vision tasks by leveraging Large Language Models (LLMMs). Innovations such as LLaVA and MiniGPT- 4 (Liu et al., 2023a; Zhu et al., 2024) have advanced image-text alignment through instruction-tuniin While many (LLL--baee studies have demonstrated effectiveness in text-generatie human-robot interaction, complex reasoning, and science question answering, GILL (Koh es al., 2023) acts as conduit between MLLMs and diffusion models by enabling LLMs to process and generate coherent images from textual inputs. SEED (Ge et al., 2023b) introduces novel image tokenizer that allows LLMs to handle and concurrently generate images and text, with SEED- 2 (Ge et al.. 2023a) enhancing this process by better aligning generation embeddings with image embeddings from unCLIP-SI), thus improving the preservation of visual semantics and realistic image reconstruction. Emu (Sun et al., 2024), multimodal generalist, is trained on a next-token- prediction model. CM3Leen (Yu et al., 2023a), utilizing the CM3 multimodal architecture and adapted training methods from text-only models, excels in both text-to-image and image-to-text generations. Finally, SmartEdit (Huang et al., 2024) proposes Bidirectional Interaction Module that enables comprehensive bidirectional information interactions between images and the MI.MM output, allowing for complex instruction-basse based image editing. Nevertheless. these methods requires text-image pairs data to train and do not support accurate spatial location prediction and subject-driven image compositing.
tasks like
3

---

Instruction Cy:
Stage1
Place the orange the and behind the apple, output the hounding box and the depth value of the cantr
BEac: [0.10, 0.32, 0.52 0.74] Depli 0.507
MLLM
Ixg
Depth Predictor
Depth Pusion
Diltsion Model
Tal
Stage
Figure 2: Overview of the inference pipeline of Bifrrst. Given background image Lag and text instruction er that indicates the location for object compositing the background, the MLLM first predicts the 2.51) location consists a bounding box and the depth the object. Then a pre-trained depth predictor is applied to estimate the given images' depth. After that. The depth of the reference object scaled to the depth value predicted by MLLM and fused in the predicted location of the background depth. Finally, the masked background image, fused depth, and reference object image used as the imput of the compositing model and generate output image Lout that satisfies spatial relations in the text instruction er and appears visually coherent and natural (e.g., with light and shadow that are consistent with the background image).
Method
The overall pipeline of our Bifrrött is elaborated in Fig. 2. Our method consists of two stages: I) in stage 1, given the input image of object and background, and text instruction that indicates the location for object compositing to the background, the MLLMs are finetuned on our customized dataset for predicting 2.5D location, which provides the bounding box and a depth value of the object in the background; 2) in stage 2, our Bifröst performs 3D-aware image compositing according to the generated 2.5D location, images of object and background and their depth maps estimated by depth predictor. As divide the pipeline into two stages, can adopt the existing benchmarks that have been collected for common vision problems and avoid the demand collected new and task-specific paired data.
We detail our pipeline in the following section. In Sec. 3.1 we discuss building our customized counterfactual dataset and fine-tuning multi-modal large language models predict 2.5D locations given a background image. Following this, in Sec. 3.2, we introduce the 3D-aware image compositing pipeline that uses the spatial location predicted by MLLM seamlessly integrate the reference object into the background image. Finally, we discuss combining two stages Sec. 3.3 and show more application scenarios of our proposed method.
3.1 Finetuning MILMM to Predict 2.5D Location
Given background image Ing and text instruction Cr, which is tokenized as Hr. our goal is obtain the 2.5D coordinate of the reference object we want to place
We indicate the 2.5D coordinate as I which consists of 2D bounding box = [21, Y1, T2, y2] and an estimated depth value d E [0, 1]. During the training stage, the majority of parameters 0 in the LLM are kept frozen, and utilize LoRA (Hu al., 2022) to carry out efficient fine-tuning Subsequently, for a sequence of length L. we minimize the negative log-likelihood of generated text tokens XA. which can be formulated as:
LLL CT) = Σ log ffo (x, Logs CT, Cū XA,C))
Dataset Generation. However, there is a lack of dataset containing image-text data pairs to teach the MLLM predict the reasonable location to place the background image L,g following the instruction Cr and is crucial to create such dataset. Since the model needs to predict both a 2D bounding box and depth value in Z axis, this requires the model to be capable of understanding both the spatial relationship between objects and the size relationship between objects (e.g.. the bounding box of a dog should be smaller than the car if the user wants to place a dog near the car) and the physical laws in the image (e.g., a bottle should be placed the table rather than floating in the air).

---

Inpaint Model
SAM
Depth Predictor
Counterfactual Image
Masked Image
Given Image
Predicted Depth
Instruction: Place the laptop to the left the keyboard,
A Answer:
BB ox: [0.00, 0.42, 0.39, 0.98]
output bounding box and the depth value of the center point.
Depth: 0.57
Figure 3: Overview of the 2.5D counterfactual dataset generation for fine-tuning MLLM. Given scene image T, object o was randomly selected the object want to predict (e.g., the laptop in this figure). The depth of the object is predicted by a pre- pre-trained depth predictor. The selected object is then removed from the given image using the SAM (i.e. mask the object) followed by an SID-based inpainting model (i.e., inpaint the masked hole). The final data pair consists of text instruction, a counterfactual image, and a 2.5D location of the selected object o.
Place the person front of the right the person, output the bounding box and the depth value of the center point.
Instruction:
the
Answer: [0.32, 0.19, 0.56, 0.87], 0.89.
Instruction: Place cat on top refrigerator output the bounding box and the depth value of the center point. Answer: [0.53, 0.36, 0.65, 0.70), 0.30.
Original Image
Depth Map
Counterfactual Image
Figure 4:
Examples of 2.5D counterfactual dataset for fine-tuning MLLM.
Furthermore, the image should not contain the object we want to place (i.e., the MLLM should learn predict the location). To this end, we build a novel counterfactual dataset based on MS COCO dataset (Lin el al., 2014). For a background image & and annotation a, we use a pre-trained depth predictor DPT (Ranftl al., 202 to estimate the depth map d. We randomly select & objects and choose one the target (o,), defining spatial relations with the other k objects (e.g.. left of, above, in front of). GPT-3 (Brown el al., 2020) generates text descriptions D mentioning all objects and their relative positions, shown in Fig. 3. The ground truth includes the bounding box and depth value of os. Using diffusion-based inpainting (Yu el al., 2023b), we remove o, from Ing We collect 30,080 image-nntrructin-aaswwe pairs for training and 855 for testing, with examples in Fig. 4. Further details are in Appendix B.I.
3.2 3D-Aware Image compositing
The training pipeline of Bifrröst's s 3D-aware image compositing module is shown in Fig. 5, Given the reference object, background, and estimated 2.5D location, Bifröst extracts ID Token, detail map. and depth maps to generate high-fidelity, diverse object-scene compositions that respect spatial relations. Unlike previous works using only 2D backgrounds and objects, our key contribution is incorporating depth maps to account for spatial relationships. Additionally, we use large-scale data, including videos and images, to train the model to learn the appearance changes, ensuring high fidelity in generated images. Details of different components are follows.
ID Extractor:
We leverage pre-trained DINO-V2 (Oquab al., 2023) to encode images into visual tokens to preserve more information. We use a single lincar layer to bridge the embedding space of DINOV2 and the pre-trained text-to-image UNet. The final projected II) tokens can be gotten by:
G E, (Iobj);
where& & indicates the ID extractor.
Detail Extractor. The II) tokens inevitably lose the fine details of the reference object due to the information bottleneck. Thus, we need extra guidance for the complementary detail generation. We apply a high-frequency map to represent fine-grained details of the object. We further applied mask shape augmentation to provide more practical guidance of the pose and view of the generated object, which mimics the casual user brush used in practical editing. The results are shown at the bottom of Fig. 1. After obtaining the high-frequend map, we stitch it onto the scene image at the specified locations and pass the collage to the detail extractor:
Ch En (In),
(3)
5

---

Triinble
Depth Prodector
Depth Eseracterr
Fronzs
Masi Shape Angnerisio
Extructar
Refermnee Augneratiion
Ditiusion Model
Frepereyyyccy Filter
Bachba
DONONG
Linzer
ID Tokesa
Figure 5: Overview of training pipeline of Bifröst on image compositing stage. A segmentation
module is first adopted to get the masked image and object without background, followed by ID extractor to obtain its identity information. The high-frequency filter is then applied to extract the detail of the object, stitch the result with the scene at the predicted location, and employ detail extractor complement the ID extractor with texture details. We then estimate the depth of the image and apply a depth extractor to capture the spatial information of the scene. Finally, the II) tokens, detail maps, and depth maps are integrated into pre-trained diffusion model, enabling the target object to seamlessly blend with its surroundings while preserving complex spatial relationships.
depth predictor to
Table I: Statistics of the datasets used in the image compositing stage.
YoUTube
MOSE
VIPSeg
VitonHD
MSRA-10K
HHlickr
SAM (suhset)
Type
Video 4453
Video 1507
Video 3110 /
Image 11647
Image 10000
Image 15572
Image 4833
Image 118287
Image 178976
Samples
Variation
where E, is the detail extractor and с₁ is the extracted high-frequency condition. More details can be found in Appendix B.2
Depth Extractor. As mentioned in Sec. 1, all existing image compositing works can only insert objects as foreground. As Fig. shows, the generated images look wired when inserted object has some overlap with objects in the background. We tackle this problem by proposing a simple yet effective method by adding depth control to the model. By utilizing pre-trained depth predictor DPT (Ranftl et al., 2021), )), we could generate depth-image pairs without demanding extra annotation. Formally. given a target image Lar, we have
Ca = & (DPT(I.ar)),
where & is the depth extractor and ca is the extracted depth condition. In experiment, the detail and depth extractors utilize CostrolNet-style UNet encoders, generating hierarchical detail maps at multiple resolutions.
Feature Injection. Given an image 2o, image diffusion algorithms progressively add noise to the image and produce noisy image zt, where represents the number of times noise is added. Given set of conditions including time step f, object ID condition c, high- high-frequency detail condition C₁. as well as depth condition Ca, image diffusion algorithms learn a UNet to predict the noise added to the noisy image 2 with
Lcomposite Bxx.ttcc..e-- [|]c- co (Zz, t, ci (t)) 1@)
where Cj Ch A Cd and À is hyper parameter weight between two controls. Specifically, the IDD tokens are injected into each UNet layer via cross-attention. The high-frequency detail and depth maps are first added together and then concatenated with the UNet decoder features at each resolution. During training, the pre-trained UNet encoder parameters frozen retain priors, while the decoder is fine-tuned to adapt to the new task.
Classifier-free Guidance. To achieve the trade-off between identity preservation and image harmo- nization, find that classifier-free sampling strategy (Ho and Salimans, 2022) is powerful tool.

---

Bolweeee
Mask Steps Augiommation
Refermace
Object
Pramei
Foumer
Framee
Prome
Target
Horlignuund
Figure 6: Data preparation pipeline of leveraging videos. Given a clip, we first sample two frames, selecting instance from one frame as the reference object and using the corresponding instance from the other frame as the training supervision.
Table Quantitative evaluation results on the accuracy the MILLM's prediction of Bifrist. Note: MiniGPTv2 and LLaVA VA (baseline) do not support depth prediction.
MiniGPTv2
LLaVA (baseline)
Ours
BBxx(MSE) (1)
0.2694
0.0653 0.7567
0.0496
BBox(ToU) (f)
0.0175
0.8515
Depth (MSE) (J)
X
0.0658
Previous work (Tang et al., 2022) found that the classifier-free guidance is actually the combination of both prior and posterior constraints.
logp p (ye c) + 1) log p (c y:) x log (y:) + (log (y: c) log (y:)),
(6)
where denotes the classifier-fre guidance scale. In our experiments, we follow the settings in (Zhang al., 2023b).
6pdd = Euc + 8 (Ec -ue))
where Cpdd. Cues Cos 8 are the model's final output, unconditional output, conditional output, and user-specified weight respectively. In the training process, randomly replace 50% object ID condition C; with empty strings. This approach increases Bifröst'' ability to directly recognize semantics in the input conditions as a replacement for the ID tokens. We further replace 30% of the depth map detail map as blank to increase the robustness of our model. (i.e., our model could generate high-quality images with only one condition, either from detail or depth control).
Dataset Generation. The ideal training data consists of image pairs capturing the same object in different scenes and poses, which existing datasets lack. Previous works (Yang et al., 2023; Song al., 2023) single images with augmentations like rotation and flip, which insufficient for realistic pose and view variants. To address this, we use video datasets to capture frames of the same object as complementary following (Chen el al., 2024; Song al., 2024). As illustrated in Fig. 6, our data preparation pipeline selects two frames from a video and extracts foreground masks. One frame masked and cropped around the object to serve as the reference, while the other frame- with an augmented mask shape-aa acts as the background. The unmasked version of this second frame serves as the training ground truth. The dataset quality another key to better identity preservation and pose variation. The full data used is listed in Tab. 1, which covers a large variety of domains such as nature scenes (SAM (Kirillov el al., 2023), LVIS (Gupta et al., 2019), HFlickr (Cong el al., 2020), DUT (Wang et al., 2017), and MSRA-105 (Borji et al., 2015)), panoptic video segmentation datasets (Youtube VOS (Xu el al., 2018), VIPSeg (Miao el al., 2022), and MOSE (Ding el al., 2023)), and virtual try-on dataset (VitonHD (Choi et al., 2021)).
3.3 Inference
The overall inference pipeline is shown in Fig. 2. Given background image Ing. and text instruction CT. our fine-tuned MLLM predicts the 2.5D location object formulated bounding box b [r1, 3/, 02, V2] and an estimated depth value E [0, 1]. Then, we first scale the depth map of the reference image to the predicted depth and fuse it into the bounding box location of the background depth map. The background image is masked following the bounding box. Finally, one can generate the composited image following the pipeline in Fig. 2. We also support various formats of input (e.g., sser-speeifeed mask and depth), which allows more application scenarios Fig. shows. More details can found the Appendix B.
4 Experiment
4.1 Implementation Details
Hyperparameters. We choose LLaVA (Liu et al., 2023a, 2024) as our method to fine-tune multi- modal large language models and Vicuna (Chiang et al., 2023) the LLM. The learning rate is
7

---

Table 3: Quantitive evaluation results on the performance of image compositing. Bifröst outper- forms all other methods across all metrics.
Paint-by Example
Object-Stitch
TF-ICON
AnyDooo
Ours
DINO-score (f)
72.225
73.108
74.743
76.807
77.746
CLIP-score (f)
84,584
84,836
85.627
88.735
89.722
FID (
22.286
19.489
18.945
15.858
15.025
Bop Sho
AnyDour Figure 7: Qualitative comparison with reference-based image generation methods, including Saint-by-Example (Yang et al., 2023), ObjectStitch (Song el al., 2023), and AnyDoor (Chen er al., 2024), where our Bifröst better preserves the geometry consistency. Note that approaches do not fine-tune the model on the test samples.
Object
Background Fused Depth
Ours
PbE
ObjectStitch
set as 2e and train 15 epochs. We choose Stable Diffusion V2.1 (Rombach et al., 2022) as the base generator for the image compositing model. During training, set the image resolution to 512 x 512. We choose Adam (Kingma and Ba, 2014) optimizer with an initial learning rate le -5. More details can be found in the Appendix A.
Zoom-in Strategy. During inference, given scene image and location box, expand the box into a square using an amplification ratio of 2.0. The square then cropped and resized to 512 x 512 for input into the diffusion model. This approach enables handling scenes with arbitrary aspect ratios and location boxes covering extremely small or large areas.
Fine-tuned MLLM, collect 855 image- instruction-answer pairs for testing as we mentioned in Sec. 3.1. 1. For quantitative results of spatial aware image compositing, we follow the settings in (Chen et al., 2024) that contain 30 new concepts from DreamBooth (Ruiz al., 2023) for the reference images. We manually pick 30 images with boxes in COCO-VE Val (Lin et al., 2014) for the scene image. Thus, we generate 900 images for the object-scene combinations.
Benchmarks. To evaluate the performance of
Evaluation metries. We test the IoU and MSE loss to evaluate the accuracy of the predicted bounding box and depth of our MLLM. For the image compositing model, we evaluate performance on our constructed DreamBooth dataset by following DreamBooth (Ruiz el al., 2023) to calculate the CLIP- Score and DINO-Score, which measure the similarity between the generated region and the target object. We also compute the FID (Heusel er al., 2017) assess realism and compositing quality. Additionally, we conduct user studies with 30 annotators to rate the results based on fidelity, quality, diversity, and 3D awareness.
Table 4: User study on the comparison between our Bifröst and existing alternatives. "Quality", "Fidelity", "Diversity", and "3D Awareness" measure synthesis quality, object identity preservation, object local variation (iee,, across four proposals), and spatial relation awareness (i.e. occlusion) respectively, Each metric is rated from (worst) to 5 (best).
Paint-by by- by-axpl-eeeecce
Object-Stitch
TF-ICON
AnyDoor
Ours (wlo depth)
Ours (w/ depth)
Quality
2.24
2.66
2.75
3.57
3,64
3.96
Fidelity (1)
2.05
2.56
2.63
3.58
3.89
4.03
Diversity (?)
3.87
2.42
2.36
3.57
3.61
3.07
3D Awareness (t)
3.56
3.37
3.42
2.51
2.56
4.21
8

---

Identity Preserved
Inpainting
Identity Transfer
Object Figure 8: Results of other application scenarios of Bifröst.
Background
Ours
Object
Background
Ours
Object
Target Depth
Background
Baseline
CFG
+ HF Filter
Depth
Figure 9: Qualitative ablation study the core components of Bifröst, where the last column is the result of our full model, "HF-Filter" stands the high-frequency filter detail extractor.
Quantitative Evaluation
MLLM Evaluation. To evaluate the effectiveness of our fine-tuned MLLM, we compare our model with two vison-language LLMs: LLaVA (Liu et al., 2023a) and miniGPTv2 (Zhu et al., 2024). However, to our knowledge, none the MLLM support accurate depth prediction. The results are shown in Tab. 2. which indicates that our fine-tuned MLLM outperforms other models significantly on the spatial bounding box prediction task.
Spatial-Awee
Image compositing Evaluation. To demonstrate the effectiveness of our model, test our model and four existing methods Shint-by-Examplee (Yang et al., 2023), ObjectStitch (Song et al., 2023), TF-ICON (Lu et al., 2023)), and AnyDoor (Chen et al., 2024) the dreambooth (Ruiz al., 2023) test sets. The same inputs (a mask and reference object) are used in all models. As shown in Tab. 3, Bifröst consistently outperforms baselines across all metrics, indicating that our model generates images with both realism and fidelity.
4.3 Qualitative Evaluation
To better evaluate the performance of our spatial-aware image compositing model, we qualitatively compare our method against prior methods shown Fig. PbE and ObjectStitch show natural compositing effects, but they fail to preserve the object's 's ID when the object has a complex texture or structure. Although AnyDoor maintains fine-grained texture of the object, it can not handle occlusion with other objects the scene. contrast, our model achieves better ID preservation and demonstrates flexibility in adapting to the background even in complex scenes with occlusion(iee. the poop emoji in the third row seamlessly injected between the bowl and flowerpot with artifact). We also provide more results of other application scenarios Fig.
User Study. We organize user study to compare mhint-by-Examen and our model. The user-study results are listed in Tab. 4. It shows that our model owns evident superiorities for fidelity, quantity, and 3D awareness, especially for 3I) awareness. Without depth control, our model can generate more diverse objects adjusted for the background. Nevertheless, the quality fidelity, and 3D awareness degrade significantly if the depth control is removed. However, as (Yang et al., 2023) only keeps the semantic consistency but our methods preserve the instance identity, they naturally have larger space for the diversity. In this case, Bifröst still gets higher rates than (Song el al., 2023; Lu et al., 2023) and competitive results with (Chen al., 2024), which verifies the effectiveness of our method. This results indicate that introducing the depth information the key for Bifröst to achieve both high fidelity and 3D-aware image compositing. More details are in the Appendix F.
ObjectStitch. TF-ICON, AnyDoor,

---

Object
Background
Depth: 0.35
Depth: 0.50
Depth: 0.65
Depth: 0.80
Figure 10: Ablation study of different depth control from deep to shallow.
Object
Background
Fused Depth
AnyDoor
Instruction: Place the backqack behind bed.
Figure 11: Failure case from the out-of-distribution dataset and comparison with AnyDoor.
4.4
Ablation Study
Tab. 5 shows the effect of components in Bifröst. Performance drops significantly without video data during training, as it cannot adjust poses and views to adapt to unseen scenes with only image data. Fig. 9 visualizes results of removing different components. CFG achieves trade-off between identity preservation and image harmonization, significantly boosting performance. Setting the collage region from the high-frequence map to an all-zero map evaluates the HF Filter's contribution, showing effectively guides the generation of fine structural details. Adding depth control further improves the fidelity and quality generated images, as visualized in Fig. 10, where the red vase moves from behind the table to the front of the white vase as depth increases.
Table 5: Quantitative ablation studies on the core components of the image compositing model of Bifröst. Note that: indicates adding component based on the previous model.
Baseline
Video Data
+CFG
+HF Filter
+Depth
DINO-score (1)
68.693
71.573
75.578
76.372
77.746
CLIP-score (1)
80.458
83.536
86.394
88.634
89.722
FID (.)
20.465
17.168
15.837
15.342
15.025
4.5 Limitations and Future Work
Our Biffröst achieves high-quality, instruction- based image compositing but has limitations. 1) While our fine-tuned MLLM performs well on in-domain test datasets, it struggles with OOD datasets containing untrained objects and more complex scenes. A failure case is illustrated in Fig. 11, where the model intends to position the backpack behind the bed. While MILM predicts the correct location, it overlaps with the chair in the background. Nonetheless, our approach still outperforms prior work (Chen et al., 2024), which fails to handle occlusions between the bed and chair due to insufficient understanding of spatial and depth relationships. This issue can be addressed by increasing the size of the training dataset (e.g., utilizing large-scale, OpenImages (Kuznetsova et al., 2018). 2) The use of depth maps significantly enhances our model's generation capabilities, enabling precise control over object placement and complex spatial relationships. However, this reliance on depth maps restricts the diversity of generated objects, partieularly in terms of novel poses and views. Although we employ classifier-fe free guidance during inference to provide some flexibility, it represents a trade-off between spatial control and object diversity. This can potentially be solved by robust depth control during the training stage. We leave this for future work.
5 Conclusion
In conclusion, Bifrröst represents a significant advancement in object-level image compositing. Lever- aging the capabilities of powerful MLLM, our framework successfully addresses the limitations of existing SID-based image compositing methods, facilitating their transition from research to practical applications. Our experimental results demonstrate that Bifröst not only achieves state-of-the-ar quality and fidelity in instruction based, object-level image compositing but also provides a control- lable generation process that accommodates complex spatial relationships. Future research endeavors encompass the refinement of the MLLM through the utilization of more extensive datasets and extending our framework to support 3D and video-based personalized object compositing tasks, further broadening its applicability and enhancing its performance.
10

---

Acknowledgements This work is partially supported by the National Natural Sci- ence Foundation of Chima (Grant No. 62306261), CUHK Direct Grants (Grant No. 4055190), and The Shun Hing Institute of Advanced Engineering (SHIAB) No. 8115074. We thank Boqing Gong (Google) for feedback an carly draft of this paper and Yuanyuan Zhang (University of Liverpool) for proposing the name Bifröst of our model.
References
Borji, A., Cheng, M.-M., Jiang, H., and Li, (2015). Salient object detection: A benchmark. IEEE Trans. Image Process. 7
Brooks, T., Holynski, A., and Efros, A. (2023). Instructpix2pix: Learning to follow image editing instructions. In IEEE Conf. Comput. Vis. Pattern Recog., pages 18392 18402. 2
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T.,, Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners.
Adv. Neural Inform. Process. Syst., pages 1877-1901. 5, 17
Casanova, A., Carcil, M., Romero- Romero-Soriano, A., Pal. C. J., Verbeek, J., and Drozdzal, M. (2023). Controllable image generation via collage representations. arXiv:2304. 13722. 15
Chen, X., Huang, L.. Liu, Shen, Y.. Zhao, D.,, Zhao, (2024), Anydoor: Zero-shot object-level image 8, 9, 10, 14, 15, 17, 18, 19,
customization. In IEEE Conf. Comput. Vis. Pattern Recog.
Chiang. W.-L.,, Ii, 7.,, Lin, Z., Sheng, Y., Wu, Z.,, Zhang, H., Zheng. I.., Zhuang. S., Zhuang. Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. (2023). Vicuna: An open-source chathot impressing gpt-4 with 90%* * chatgpt quality. 3, 7,
Choi. S.. Park. S., Lee, M., and Choo, (2021). Viton-hd: High-resolution virtual try-on via misalignment- normalization. In JEEE Conf. Comput. Vis. Pattern Recog. 7
Cong, W. Zhang, J. NIu, L., Liu, L., Ling, Li, W., and Zhang, L. (2020). Dovenet: Deep image harmonization via domain verification. In IEEE Conf. Comput. Vis. Pattern Recog. 7
Ding, H., Liu, C., Hc, S., Jiang, X., Torr, P. H., and Bai, S. (2023). Mose: A new dataset for vidco object segmentation in complex scenes. 2rXIv:2302.01872.7
Ge. Y., Zhao, S.. Zeng, Z., Ge, Li. C., Wang. X., and Shan, Y. (2023a). Making llama see and draw with seed tokenizer. arXiv preprint arXiv:2310.01216 3
Y., Ge, Zeng, Z., Wang, X., Shan, (2023b), Planting seed of vision large language model. In Int. Conf. Learn. Represent.
Guerreiro, J. A., Nakazawa, M., and Stenger, B. (2023). Pet-net: Full resolution image harmonization using pixel-wise color transformations. In IEEE Conf. Comput. Vis. Pattern Recog, pages 5917-5926.
Gupta, A., Dollar, P., and Girshick, R. (2019). Lvis: A dataset for large vocabulary instance segmentation. In IEEE Conf. Comput. Vis. Pattern Recog. 7
He. R., Ma. K., Huang, L., Huang. S., Gao. J.. Wei, X.. Dai. J.. Han, J., and Liu. S. (2024). Freeedit: Mask-free reference-based image editing with multi-modal instruction. 2
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017). Gans trained by two time-scale update rule converge to a local nash equilibrium. In Adv. Neural Inform. Process. Syst. 8
Ho, and Salimans. (2022). Classifier-fres diffusion guidance. H. Larochelle, M. Ranzato, R. Hadsell,
M. Balcan, and Lin, editors, NIPS Workshop.
Ho, Jain, and Abbeel, (2020). Denoising diffusion prohabilistic models. H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Adv. Neural Inform. Prucess. Syst., volume 33, pages 6840-6851. 2,3
Hu, J., Shen, Y., Wallis, P.. Allen Z., Li. Y., Wang, S., Wang, L.. and Chen, W. (2022). LoRA: Low-rank adaptation of large language models. Int. Conf. Learn. Represent.
Huang. L., Chen, D., Liu, Y., Shen, Y., Zhao, D., and Zhou, J. (2023). Composer: creative and controllable image synthesis with composable conditions. In Int. Conf. Machine. Learning

---

Huang. Y., Xie. L.. Wang. X., Yuan. Z., Cun, X., Ge, Y.. Zhou. J. Dong, C., Huang, R., Zhang, R., et al. (2024). Smartedit: Exploring complex mnstruction-based image editing with multimodal large language models. In IEEE Conf. Comput. Vis. Pattem Recog. 3
Jiang, Y., Zhang, H., Zhang, J. Wang. Y., Lin, Z., Sunkavalli, K., Chen, S... Amirghodsi. S.,. Kong, S., and Wang. Z. (2021). Ssh: A self-supervised framework for image harmonization. In Int. Conf. Compur. Vis., pages 4832-4841. 3
Kanopoulos, N., Vasanthavada, N., and Baker, R. I. (1988). Design of image edge detection filter using the sobel operator. IEEF Journal of solid-state circuits. 16
Kc, Z. Sun, C., Zhu, L., Xu, K., and Lau, R. W. (2022). Harmonizer; Learning to perform white-bor box image and video harmonization. In Eur. Conf. Comput. Vis., pages 690-706. Springer. 3
Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv:1412.6980. 8, 14
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L.,, Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y,, et al. (2023). Segment anything. arXiv:230.:02643.
Koh, J. Y., Fried, D., and Salakhutdinov. R. (2023). Generating images with multimodal language models. In Adv. Neural Inform. Process. Syst. 3
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M.. Kolesnikov, A., Duerig. T., and Ferrari, V. (2018). The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. Int. J. Comput. Vis. 10
Li, L., Zhang, Y., and Wang. S. (2023). The euclidean space is evil: Hyperbolic attribute editing for few-shot image generation. In Int. Conf. Comput. Vis., pages 227 12-2444..
Lin, T.Y.,, Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). Microsoft coco: Common objects in context. In Eur. Conf. Comput. Vis., pages 740-755. 5, 8, 17
Liu, H., Li, C., Wu, Q., and Lee, Y. J. (2023a). Visual instruction tuning. In Adv. Neural Inform. Process. Syst. 2,
3, 7,
Liu, H., Li. C.,, Li, Y., Lee, J. (2024). Improved baselines with visual instruction tuning. IEEE Conf. Comput. Vis. Pattern Recog. 2,7
Liu, Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., and Vondrick, (2023b), Zero- I-to-- Zero-shot one image to 3d object. In IEEE Conf. Comput. Vis. Puttern Recog., pages 9298-9309. 2
Lu, S., Liu, Y., and Kong, W.-K. (2023). Tf-icon: Diffusion-basse based training-frree cross-domain image composition. In Int. Conf. Comput. Vis., pages 2294-2302 2305. 9,
Miao, J., Wang, X., Wu, Y., Li, W., Zhang, X., Wei, Y., and Yang, Y. (2022). Large-scale video panoptic segmentation in the wild: A henchmark. In IEEF Conf. Comput. Vis. Pottern Recog- pages 21033-21043.
Oquab, M., Darcet, T., Moutakanni, T.. Vo. H. V.. Szafraniec, M., Khalidov. V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Howes, R., Huang. P.-Y., Xu, H., Sharma, V., Li, S.WW., Galuba, W., Rabbat, M., Assran, M., Ballas, N., Synnaeve, G., Misra, I., Jegou. H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, (2023). Dinov2: Learning robust visual features without supervision. 5, 14, 15
Pérez, P., Gangnet, M., and Blake, A. (2003). Poisson image editing. In ACM SIGGRAPH, pages 313-318.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P.. Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In Int. Conf.
Machine. Learning. 14, 15
Ramesh, A., Dhariwal, P.. Nichol, A., Chu, C., and Chen, M. (2022). Hierarchical text-conditional image generation with clip latents. arXiv:2204.06/22
Ranftl, R., Bochkovskiy, A., and Koltun, V. (2021). Vision transformers for dense prediction. In Int, Conf. Comput. Vis., pages 12179-12188. 6
Rombach, R., Blatmann, A., Lorenz, D., Fsser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 10684-10695. 2, 20
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. (2023). Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In IEEE Conf. Comput. Vis. Pattern Recog. pages 22500-2255 -2510 10. 8, 18, 19, 20
12

---

Saharia, C.. Chan. W. Saxena, S., Li, L.. Whang. J. Denton, E. L.. Ghasemipour, K., Gontijo Lopes. R.. Karagol Ayan, B., Salimans, T., et al. (2022). Photorealistic texl-to-image diffusion models with deep language understanding, In Adv. Neural Inform. Process. Syst.
Sarukkai, V., Li, L., Ma, A., Ré, C., and Fatahalian, K. (2024). Collage diffusion. In Win. Conf. App, Comput. Vis. 15
Sohl-Dickstein, J.. Weiss, E., Maheswaranathan, N., and Ganguli. S. (2015). Deep unsupervised learning using nonequilibrium thermodynamics. Int. Conf. Muchine. Learning. pages 2256-2265. PMLR.
Song. Y. Sohl- Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. (2021). casee-based generative modeling through stochastic differential equations. In Int. Conf. Learn. Represent. 2.3
Song, Y., Zhang, Z., Lin, Z., Cohen, S., Price, B., Zhang, J., Kim, S. Y., and Aliaga, D. (2023). Objectstitch: Object compositing with diffusion model. In IEEE Conf. Comput, Vis. Pattern Recog.. pages 18310-183111 2, 3, 8, 9, 15, 17, 19, 20
Song. Y., Zhang, Z., Lin, Z., Cohen, S., Price, B., Zhang, J., Kim, Y., and Aliaga, D. (2024). Imprint: Generative object compositing by learning identity preseerirreeringgg representation. In IEEE Conf. Comput. Vis. Pattern Recog. 2, 3. 7. 15, 18, 20
Sun, Q. Yu, Q. Cui, Y., Zhang, F., Zhang, X., Wang, Y., Gao, H., Liu, J., Huang, T., and Wang, X. (2024). Emu: Generative pretraining multimodality. Int. Conf. Learn. Represent.
Tang, Z. Gu. S., Bao, J., Chen, D., and Wen, E. (2022). Improved vector quantized diffusion models. 2.Xi::2205.16007 7
Touvron, H.. Lavril, T., Izacard, G.. Martinet, X., Lachaux, M.-A,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023). Llama: Open and efficient foundation language models. arXiv preprint M.KIv:330211977 3
Wang. L. Lu, H., Wang. Feng. M.. Wang, D., Yin, B., and Ruan. (2017), Learning to detect salient objects with image-level supervision. In IEEE Conf. Comput. Vis. Puttern Recog.
Wu, H., Zheng. S., Zhang, J., and Huang, K. (2019). Gp-gan: Towards realistic high-resolute image blending.
In Proceedings of the 27th ACM international conference multimedia, pages 2487-2495. 3
Xu, N., Yang, L.,, Fan, Y., Yang. J., Yue, D., Liang, Y., Price, B., Cohen, S., and Huang. T. (2018). Youtube-vos: Sequence- seuuence-to-squecee te-sequence video object segmentation. In Eur: Conf. Comput. Vis., pages 585 601. 7
Xue, B., Ran, S., Chen, Q.. Jia, R., Zhao, B., and Tang, X. (2022). Deef: Deep comprehensible color filter learning framework for high-resolution image harmonization. arXiv preprint mxxiv:2207.04788. 3
Yang. B., Gu, S., Zhang, B., Zhang, T., Chen, X., Sun, X., Chen, D.,, and Wen, F. (2023). Paint by example: Exemplar-based image editing with diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 18381-18391. 2, 3, 7, 8, 9, 15, 17,
Yu, L., Shi, B., Pasunuru, R., Muller, B., Golovneva, O., Wang. T., Babu, A., Tang, B., Karrer, B., Sheynin, S.. et al. (2023a). Sealing autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint arXiv: 209.0223.
Yu, T., Feng, R., Feng, R., Liu, J., Jin, X., Zeng, W., and Chen, Z. (2023b). Inpaint anything: Segment anything meets image inpainting, arXiv preprini arXiv:2304.06799
Zhang, B., Duan, Y., Lan, J., Hong, Y., Zhu, H., Wang, W., and Niu, L. (2023a). Controlcom: Controllable image composition using diffusion model. arXiv preprint arXiv:2308.1 10040.
Zhang, H., Zhang, J., Perazzi, F,, Lin, 7., and Patel, V. M. (2021). Deep image compositing, In Win. Conf. App, Compur. Vis., pages 365-374, 3
Zhang, L.,, Wen, T., and Shi, J. (2020). Deep image blending. In Win. Conf. App. Comput. Vis., pages 231-240. 3
Zhang, L.,, Rao, A., and Agrawala, M. (2023b), Adding conditional control text-to-image diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog.. pages 3813-3824. 2.7
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. (2024). Minigpt-4: Enhancing vision-language understanding with advanced large language models. In Int. Conf. Learn. Represent. 3, 9
13

---

Supplementary Material
Overview
This appendix organized as follows:
Appendix A gives more implementation details of Bifröst. Sec 4.1
Appendix B provides more technical details and mathematical formulae used in Sec 3.1 & Sec 3.2 Appendix C explains more details of our evaluation details in experiments, See 4.1
Appendix D provides more details of creating counterfactual dataset and gives more examples from the constructed dataset. See 3.1
Appendix E shows more visual results and comparison with prior methods. Sec 4.3
Appendix F shows more details of the user study we conducted. Sec 4.3
Appendix G discusses the ethic problems might be caused by Bifrröst and potential positive societal impacts and negative societal impacts.
A Implementation Details
For fine-tuning MLLM, following the base setting in (Liu et al., 2023a), we choose Vicuna (Chiang el al., 2023) as our LLM. We use pre-trained ViT--aree (Radford et al., 2021) as the visual encoder. The learning rate is set at 4e to avoid over-fitting The batch size is set as 16, we train 15 epochs with A800 GPUs, which takes about 5 hours to finish the fine-tuning.
For 3D-aware image compositing model. We use Stable Diffusion 2.1 as the pre-trained diffusion model. The model can also be initialized from the pre-trained weights of (Chen et al., 2024) to better utilize the prior knowledge. We choose Adam (Kingma and Ba, 2014) optimizer with initial learning rate of Le 5 The DINO-V2 (Oquab er al., 2023) ID extractor and the encoder of the U-net frozen during the training. The decoder of the U-net, detail extractor, and depth extractor are fine-tuned during this process. The batch size set as 16, we train 20k steps on 4 A800 GPUs, which takes day to finish the training.
Although training our model requires considerable computing resources as mentioned in Appendix Section A, the runtime cost and resources required for the inference stage are affordable. Our model can run on single NVIDIA RTX 4090 GPU (24GB) thanks to our two-stage training/inference since one does not need to load all models simultaneously. The total inference time for one image composition can be finished in 30 seconds on an RTX 4090 (these include predicting the depth map, running the SAM model to remove the background of the reference image, using MILM to predict the 2.5D location, depth fusion, and image composition, the DDIM sampler is set as 50 steps).
B Method
B.1 MILM Fine Tuning
Given a background image Ib, and text instruction CT, which is tokenized Hr, our goal is to obtain the 2.5D coordinate the reference object we want to place in, indicates I consists of a 2D bounding box b- = [x1,, ,1, y2] and an estimated depth value d [0, 1]. following the base setting in (Liu al., 2023a), As shown in Fig. 3. For an input background image Ing. we first use the pre-trained visual encoder to encode the image, which provides the visual feature Zv = g( Then, we use a simple linear layer to connect image features into the word embedding space. Specifically, we apply trainable projection matrix W to convert Zy into language embedding tokens Hy, which have the same dimensionality as the word embedding space in the language model:
choose Vieuna (Chiang et al., 2023) our LLM fo(-) parameterized by
Hy W. Zv. with Zv g(Ing)
(8)
Thus, we have sequence of visual tokens Hy, which will be sent to the LLM with text tokens Hr- Counterfactual Dataset Generation In practice, we choose the depth value of the central points of the selected object bounding box as the target depth value for MLLM to predict. The reason we do
14

---

30
100
-00
-0.3 -00 6.0
(b)
Figure 12: The distribution of differences between center value and three other choices of the depth values, where (a) is the differences between the max value of depth and center depth value: (b) is the differences between the mean value of depth and center depth value; and (c) is the differences between the median value of depth and center depth value
not use the entire depth value is that the goal of predicting the depth value of the reference object is to estimate the location (in depth dimension) of the object in the background image for the final image composition. To this end, it does not have to be very accurate as long as it is in a reasonable range and a single depth value for the object shown to be sufficient. Though it is possible to estimate the depth value for the entire object (e.g., via DPT). it is more costly and challenging to construct the dataset (pixel-wise depth annotation) and train the MILM for the entire object. We will explore more sufficient strategies for this in the future. However, there are other choices for points to represent the 2.5D location of the selected object besides the center point. There are various options for estimating the depth value, such as the depth value at the center point of the bbox, the maximum, average, and median depth value of the object within the bbox, etc. They all have their pros and cons and result in similar performance. For instance, the median value varies significantly along with the size of the object in the bounding box. The average value may be influenced by extreme values in the bounding box. Furthermore, the maximum value might also be influenced by the extreme value in the bounding box. We conducted an extra experiment in Fig. 12 to compare the differences between different choices. We calculate the differences between different choices of the depth value we want to predict on 5000 examples in the evaluation dataset. The results show that the difference between the maximum depth value and the center point value is small. Most of the differences are less than 0.2. While the differences between the average value and the center point value basically follow a Gaussian distribution with a mean value of -0.05. However, the median value is not reliable compared with other choices. In this work, as we focus on general settings where reference objects are from common categories, we use the depth value of the bbox center point, which works well in our experiments. Optimizing the location for depth value estimation may be helpful for objects from specific categories, which are rare, and we leave this in future work.
predict the depth value
B.2 3D-Aware Image Compositing
ID Extractor
We employ pre-trained visual encoders to extract the target object's identity. Rather than using CLIP (Radford et al., 2021) for object embedding (Yang el al., 2023; Song et al., 2023), prior work (Chen et al., 2024; Song al., 2024) demonstrates that DINO-V2 (Oquab et al., 2023) better captures discriminative features, projecting objects into an augmentation-invarie we use DINO-V2 to encode the image into a global token Tg 1536 and patch tokens 256xx1158 These tokens concatenated retain more information. Following (Chen al., 2024), linear layer bridges the DINO-V2 and pre-trained text-to-image UNet embedding spaces, yielding final ID tokens 2227x1022
feature space. Thus,
Collage Representation. Unlike prior methods (Casanova el al., 2023; Sarukkai et al., 2024; Chen et al. 2024) that use full-color images or found that these approaches often produce objects that resemble the reference image too closely, creating a "ccoy-ppste" effect. To address this, we remove color information from the mF-map in (Chen et al., 2024), enhancing fine- mne-grained details and improving the visual coherence of the generated objects.
maps as complementary information, we
Mask Shape Augmentation. To provide greater user control, we define five levels of coarse masks, including a bounding box mask, as shown in Fig. 14. As the mask level increases (from 1 to 5), the model gains more flexibility in object generation.
15

---

LoRA
LLM
BRox (6,00 0.2,002,,000000 0.32, Depti 0.57
Frosen
H
Vilio Enceddrr
kyoopp keyboadd, coppor
X
Figure 13: Overview of the MILM fine-tuning.
Object
Target
Mask 1
Mask 2
Musk 3
Mask
Mask $
Figure 14: Different types of mask used in the image compositing stage. The generation is constrained in the masked area, so the user-provided mask is able to modify the pose, view, and shape of the subject.
The Detail Extract process is formulated as:
I (Tobj & K2 Lby 69 K.) C Maug:
where K,, and Kg are horizontal and vertical Sobel kernels (Kanopoulos et al., 1988) serving as high-pass filters. Here, @ and @ represent convolution and Hadamard product, respectively. Given an object image lobj. high-frequency regions extracted through these filters, and shape-augmented mask Mang is applied remove information near the outer contour of the object.
B.3 Inference
As discussed Sec. 3, Bifrröst supports multiple types of inference modes. We illustrate different inference modes in Fig. 15.
ID Transfer. To transfer the identity of an object in the background, first employ segmentor to accurately mask the target object while maintaining the original depth background image. The model then generates new image using the reference object, background depth map, and masked background.
ID Preserved Inpainting. To alter the pose view of the reference object, we keep the back- ground image's 's depth unchanged and use the reference object, background depth map, and masked background as inputs to generate new image.
Place. To place object into the background, bounding box and depth value for the desired location. The reference object's depth map scaled accordingly and fused into the background depth map at the specified location, with the mask shape adjusted to reflect this fusion, shown in the third row of Fig. 15. The model then generates new image using the reference object, background depth map, and masked background.
the fine-tuned MLLM to predict the precise
To replace an object in the background with a reference object, we first scale the reference object's i depth map to match the depth value predicted by the MILM and fuse the depth maps. The entire bounding box masked, setting the value to 0 outside the object ensure
Replace.
fully masked. The background image also fully masked within the bounding box. The model then generates a new image using the reference object, background depth map, and masked background.
16

---

ID Transfer
Preserved Inpainting
Place
Replace
Object
Background
Depth
Mask
Generated
Figure 15: Comparison of different inference modes
C
Evaluation Details
To fairly compare the quality of the generated images between different methods, we cropped the generated images the given bounding box area and compared the similarity with the reference objects.
D Counterfactual Dataset
In this section, provide more details building the counterfactual dataset using COCO (Lin et al., 2014) dataset. We take Fig. 16 as an example. For given image with multiple bounding boxes. We randomly select an object as the object we want to predict (i.e., bowl with grape in the red bounding box). Then randomly select multiple objects the objects relative location with (i.e., carrot and spoon with green bounding boxes in this picture). Then we define the relative relations between the selected object and other objects based on their spatial relation. In Fig. 16, the grape bowl is to the left of the spoon and underneath the carrot. The relation definition process also considers the depth information; if the depth values are different between two objects, we will define the relations as "behind" or "in front of" based on the real situation. Then we prompt GPT3 (Brown al., 2020) to generate the instruction text. However, the usage of GPT is not mandatory. The usage of GPT3 is only for sample text instructions given spatial relations and object names, which will not the "reasoning" ability of GPT. This simple task even be done by a naive approach by setting pre-defined instruction templates and filling the blanket with spatial relations and object names. The ground truth answer consists of the bounding box of the selected object and the depth value for the center of the object. Finally, remove the selected object the right image in Fig. 16 shows. We show more examples of our counterfactual dataset Fig. 17.
want to deseribe
E More Qualitative Results
We show more qualitative comparison with Paint-by-Example (Yang et al., 2023). ObjectStitch (Song al., 2023), and AnyDoor (Chen al., 2024) Fig. 18. РbE and ObjectStitch show natural compositing effects, but they fail to preserve the object's ID when the object as lines 2-4 show. Although AnoD maintains fine-grained texture of the object, it can not handle occlusion
17

---

Instruction: Place the bowl to the left of the spoon underneuth the carrot, output the bounding box and the depth value the canter point. (0.60, 0.91],
Figure 16: More examples of 2.5D counterfactual dataset building process
Instruction: Place the person in front of the chair, ourput the bounding box dapth the point.
Answer: [0.09, 0.06, 0.31, 0.90), 0.80.
Instruction: Place the bowl the left of the spoon and undernetth the ourput bounding depth valuo center point. Amswer: [0.60, 0.38, 0.93, 0.91], 0.78.
Instruction: Place the motoreycle front of the person, outpur the bounding and depth value the center point. Answer: (0.00, 0.18, 0.27, 0.94), 0.94.
Instruction: Place the refrigerater under the cal, oulput the boundiig box the depth of center point. Answer: [0:47, 0.60, 0.92, 0.99), 0.73.
Instruction: Place the dining table the left of the bed, output the bounding box and the depth value of the center point. Amswer: (0.00, 0.60, 0.24, 1.00), 0.46.
Instruction: Place sheep in front of the person, oulput bounding box and the depth value of the center point. Answer: (0.08, 0.64, 1.00, 1.00), 0.71.
Original Image
Depth Map
Counterfactual Image
Figure 17: More examples 2.5D counterfactual dataset for Fine-tuning MILM.
with other objects in the scene. In contrast, our model perfectly handles occlusion in complex environments(iee the dogs in lines 1 and 4 behind the hydrant).
We further provide show that seamlessly injects the object into the backgrounds while satisfying the instructions. For instance, the lights and shallows change with the environment. It also perfectly handles the occlusion with other objects.
examples of images generated by Bifrröst given text instructions. The results
Following the same setup in prior works (Ruiz al., 2023; Chen al., 2024: Song et al., 2024), the training and testing data are exclusive for image composition. This has evaluated the OOD ability of the image composition of our method, indicating the good generalization ability of our method. As mentioned in the limitation section our paper, the OOD ability for the MLLM in stage can be affected by the number of object and scene categories in the dataset in stage for predicting 2.5D location (the estimated depth may not be very accurate but still in a reasonable range for OOD objects
18

---

Or scenes). However, as we only require rough depth value for image composition, the effects of the OOD issue for image composition in stage 2 relatively minor.
To further verify this, we report the results of OOD objects and scenes in Fig. 19, though both objects and scenes have not been seen in stage (e.g., piano and church are not included in MS-COCO), the MLLM can still predict reasonable depth values for image composition.
Object
Background
Fused Depth
Ours
PbE
ObjectStitch
AnyDoor
Figure 18: More qualitative comparison with reference-based image generation methods, in- cluding Paint-by-Example (Yang al., 2023), ObjectStitch (Song al., 2023), and AnyDoor (Chen et al., 2024).
Human Evaluation Interface
As mentioned, we conducted an extensive user study with fully randomized survey. Results shown in the main text. Specifically, we compared Bifrröst with four other models Paint-by- Example (Yang et al., 2023), ObjectStitch (Song al., 2023), TF-ICON (Lu el al., 2023), Any- Door (Chen et al., 2024):
1. We chose 30 images from DreamBooth (Ruiz al., 2023) set, and for each image, we then generated 3 variants with different backgrounds using each model, respectively. Overall, there were 30 original images and 90 generated variants in total.
2. For each sample of each model, we present one masked background image, reference object, and the generated image to annotators. We then shuffled the orders for all images.
3. We recruited 30 volunteers from diverse backgrounds and provided detailed guidelines and templates for evaluation. Annotators rated the images on scale of to 5 across four criteria: "Fidelity", "Quality", "Diversity". and "3D Awareness". "Fidelity" evaluates identity preservation, while "Quality" assesses visual harmony, independent of fidelity. "Diversity" measures variation among generated proposals to discourage "copy-paste" style
19

---

Object
Backgrond
Fused Depth
Ous
Instruction: Place the dog the right of the piano.
Instruction: Place the horse front of the church.
Figure 19: Examples that both object and background are from the out-of-distribution dataset.
INPUT
You are given masked backerrund and a referacce objent, We inject the refermnce inje the lackground image with location indicated by
Yourtosk the generatedd image (warst) (hres) concerning
Micelity: the injected object preserves original input ideatity
Background
Object
2) Qualltty: (f the injected object is hammonized in the background
Diversity: If the injected object has nowel
OUTPUT
3D Awwrrnness If injected object handles spatial relationships
Problem 1: Laput the soore for Fidelity
Problem Inget
for Quality
Problam 3: lapul spore for Diversily
Problem 4: Ingot the soom for 3D Awarnnsss
Figure 20: The illustration of the user study interface.
outputs. "3D Awareness" evaluates the object's ability handle spatial relationships (e.g.. occlusion) scamlessly.
The user-study interface is shown in Fig. 20.
G Ethics Discussion
The advancements in image generation (Ruiz al., 2023; Rombach et al., 2022) and composit- ing (Chen el al., 2024; Song et al., 2023, 2024) through our proposed method offer significant positive societal impacts. Our approach enhances practical applications in fields such as e-commerce, professional editing, and digital creation. By enabling precise and realistic image compositing, our method can improve user experience, facilitate creative expression, and streamline workflows in various industries. Additionally, the ability to use text instructions for image compositing makes accessible to non-experts, promoting inclusivity in digital content creation. However, there are potential negative societal impacts to consider. The misuse realistic image compositing technology could lead to the creation of deceptive harmful content, such deepfakes mislead- ing advertisements. This raises ethical concerns regarding privacy, consent, and the potential for misinformation. To mitigate these risks, is crucial to establish guidelines and ethical standards for the use of such technology. Furthermore, transparency in the development and deployment of these models is necessary to build trust and ensure responsible usage. In summary, while our method presents valuable advancements in image compositing, careful consideration and proactive measures essential to address the ethical implications and prevent potential misuse.
technology
20

---

"Replace the plant with the toy"
"Place the sneaker to the right of the horse"
"Place the telephone booth to the left of the train"
"Place the vase in front of the vase"
"Place the dog in front of the truck"
"Replace the bus with the bus"
"Place the backpack to the right of the laptop"
Figure 21: More results generated by Bifröst with language instructions
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
This can be found in the abstract and Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Can be found in Sec. 4.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results violations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specificatton, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
Can be found in Sec. 3 and Appendix
Guidelines:
22

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Can be found in Sec. 4, Appendix A, Appendix and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide the code and dataset with instructions on GitHuu.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
23

---

Please see the NeurIPS code and data submission guidelines (https://nips.c cc/ serriides/oodeSumiiie
for details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nips. wwtpublcc//aeecccccccccccccc wwwcc//ggise/eeeeee (CodeSummissionF
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Can be found in Sec. 4, Appendix F, Appendix A, and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
Can be found in Sec. Appendix F, Appendix A, and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
24

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resourees
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Can be found in Sec. 4 and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : htt////eerippppppe httpppp...pppppppeee wttpccccc/cccceecceccccccccclcct hette//Ehhhhiiiaa cttiiihhhiielleeeee
Answer: [Yes],
Justification: We follows the Code Ethics as required. Can be found in Appendix and Sec. 4, Appendix C, and Appendix A.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss concerns and societal impacts in Appendix G.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
25

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
Can be found in Appendix G.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we follow the rules.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww.coooc/oooooooo datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- e-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does release new assets. Researchers should communicate the details the dataset/code/moodll. as part their submissions via structured templates. This includes details about training, license, limitations, etc.
26

---

The paper should discuss whether and how consent was obtained from people whose used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
The details and screenshots are provided in Appendix F.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
27


Reviews:
[{"review_id": "Hr5JI8YPdC", "reviewer": "Reviewer_M1TS", "summary": "## Summary of the Paper:\n\n*Problem Statement*:\n\n     Given a single-object image I_ref with object O_ref , a background image I_bg containing an object O_bg, and a text prompt P_text, the paper proposes a method for composing O_ref onto I_bg in such a way that the position of O_ref in the resulting image adheres to the text query P_text.\n\n\n*Motivation*:\n\t\n     Existing works on image composition, which mainly relied on processing images in the 2D domain, underperformed in the presence of object occlusion (O_ref in I_ref) . This paper, named BiFrost, argues that image composition algorithms can benefit from processing beyond the 2D domain. A natural choice would be leveraging priors from the 3D domain.  However, due to the complexity of single-image-to-3D analysis, the paper propose to use 2.5D information by incorporating depth information into their pipeline.\n\n*Contributions*:\n\n1)\t Use of depth information in image composing task.\n2)\t A new dataset of image-text pairs for object location prediction task i.e.to predict 2D bbox and depth value of the object. \n3)      Leveraging multi-modal large language models (MLLM) into their pipelines\n\n\n*Dataset used*:\n\n1)\t For predicting 2.5D information: Newly created dataset of image-text pairs based on MS-COCO dataset \n2)\t For image composing task: 9 datasets -- 3 video-based, 6 image-based datasets. (this setting is similar to previous work in this area) \n\n\n*Underlying Modeling Tool*:\n\n     Transformer is the common modeling tool among all involved tasks, including depth map generation (GPT3, DPT from CVPR 2021, SAM and Diffusion-based Inpainting Model), MLLM fine-tuning (using LLaVa), and image composition task (DPT, SAM, DINO_V2 and Stable Diffusion)\n\n\n\n*Learning Mechanism*: \n\n     Strongly supervised\n\n\n*Loss Functions*:\n\n     For fine-tuning MLLM, negative log-likelihood of the generated text tokens\n\n     For training image composing task, standard diffusion loss i.e. L2 loss between ground truth noise added to image at time step t and predicted noise at that step.\n\nQuantitative Metric:\n\n     DINO-score, CLIP-score, FID, User study\n\n\n*Baselines*:\n\n     On the image composition task, four existing methods are compared against: Paint-by-Example, ObjectStitch, TF-ICON, AnyDoor.", "questions": "1)     How do you calculate center point for predicting depth value? In my understanding it is the center of the bbox. Is that correct?\n2)\t Why just a single depth point, why not multiple points or object mask? As mentioned earlier, a single point can be erroneous.\n3)\t Is it assumed that the reference object image I_ref will always have one object? Are there any images during train/test time with multiple objects? \n4)\t When creating the dataset for MLLM finetuning, why use MS-COCO dataset? Why not use a subset of the training data from image composing task itself? How does training on MS-COCO dataset help in predicting better depth values, for, say, YouTubeVOS or HFlickr dataset? \n5)\t Is the model trained on all nine datasets combined OR separately,  and test only on 900 object-scene combination images based on DreamBooth and COCO-Val set? Again, my question is why different datasets for training and test? Are they from same distribution?", "limitations": "Yes, limitations of the work have been discussed.\n\nNo societal impact can be seen from the work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1)\t Well-written paper\n2)\t Incorporating depth information to account for occlusion in text-driven image composition task is an interesting idea\n3)\t A new dataset for predicting bounding box and depth values for a given image and a text prompt  -- such dataset, if made public, will be useful to the research community working on image composing task", "weaknesses": "1)\t It seems that the paper is trying to force-fit large foundation models for a simple task of incorporating depth information. What was the reasoning behind this? What other alternatives were explored?  \n2)\t Almost all of the components are based on large foundation models and have frozen weights. It leaves the paper with a single technical contribution, i.e., the use of 2.5D information, instead of 2D.\n3)\t The proposed pipeline involves fine-tuning a MLLM, which is in-turn, used to predict only a single depth value. Since SAM is used to get object masks, why not use depth values on the entire object? \n4)\t Two-stage training -- I was wondering why stage-1 (where the MLLM is finetuned) cannot be merged with stage-2 to obtain a single end-to-end training pipeline? The reasoning being that since MLLM is tasked to predict only the bounding box and a single depth value \n5)\t Using a single depth value can be erroneous. For e.g., in Figure 3, the star mark on “masked image” shows that the center point of laptop is on the corner of the laptop. There could be cases where the center point is NOT on the object of interest. How does the model perform in such scenarios? \n6)    Due to the many sub-modules involved in the method, starting from training a depth predictor module using GPT3, I suspect the method is not easily extendable, nor replicable within a certain margin error. I want to understand the authors' stand here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 1M1TS\n\nWe thank the reviewer for the positive feedback and insightful comments. We address the questions and comments below.\n\n> Q1: …incorporating depth…reasoning behind this…\n\nR1: Many thanks for acknowledging our novelty in the context of image composition. We achieve 3D-aware image composition by incorporating depth information, which provides the location information in the depth dimension to enable 3D-aware image composition, e.g., avoiding a crash in image composition due to the occlusion issue. We have explored alternative strategies for this problem. We have attempted to generate scenes layer by layer and place the reference object at the target layer according to the language instructions.  However, this does not well solve the problem as decomposing scenes into layers itself is a challenging research problem. Alternatively, one can embed the depth information predicted by [A] directly into the generative models. It fails due to the lack of ability to understand the relations between the reference object and the target background. We will include this in the final version.\n\n\n> Q2: …single technical contribution…\n\nR2: Please refer to the answer of Q1 in the 'Response to All Reviewers'.\n\n> Q3: …depth…entire object…\n\nR3: The goal of predicting the depth value of the reference object is to estimate a location (in depth dimension) of the object in the background image for the final image composition. To this end, it does not have to be very accurate as long as it is in a reasonable range and a single depth value for the object is shown to be sufficient. Though it is possible to estimate the depth map for the entire object (e.g., via DPT), it is more costly and challenging to construct the dataset (pixel-wise depth annotation) and train the MLLM to predict the depth map for the entire object. We will explore more efficient strategies for this in the future.\n\n\n> Q4: …end-to-end training..\n\nR4: Yes, it's possible to merge stage 1 with stage 2. However, merging both stages will inevitably increase training costs and difficulty due to the lack of high-quality paired instruction-original image-compositing image data. Collecting such a dataset for end-to-end training for both stages is complicated and extremely expensive. We formulate our method into 2 stages, making it easier and cheaper for collecting or using existing data for training our method for each stage, as explained in the paper (line 63-66, line 122-124).\n\n> Q5: …center point is NOT on the object of interest…\n\nR5: Please refer to the answer of Q3 in the 'Response to All Reviewers'. In this work, as we focus on general settings where reference objects are from common categories, we use the depth value of the bbox center point, which works well in our experiments. Optimizing the location for depth value estimation may be helpful for objects from specific categories which are rare, and we leave this in future work. We will add this discussion in our final version.\n\n\n> Q6: …using GPT3…not easily extendable…\n\nR6: GPT3 is only used for sampling text instructions given spatial relations and object names for collecting the new dataset at stage 1. And this does not use the “reasoning” ability of GPT. This step can be replaced by a simple approach by setting pre-defined instruction templates and filling the blanket with spatial relations and object names or alternatives. To this end, this is reproducible and scalable. We will release the dataset and scripts for reproducing it.\n\n> Q7: …reference object…have one object? Are there…multiple objects? \n\nR7: We follow exactly the same setting in prior work [C,D], where the reference object only has one object, and the setting of multiple objects is not considered in this work. The method can be simply extended to multiple reference objects by cropping objects and compositing objects into the background image one by one. We leave this in our future work.\n\n> Q8: …why use MS-COCO dataset…How does…YouTubeVOS or HFlickr? \n\nR8: Stage 1 aims to predict 2.5D information indicating the location of the reference object in the background image. This requires the MLLM model in stage 1 to be able to understand different relations (spatial and depth dimensions) between scenes and objects. To this end, we use MS-COCO, which covers diverse common scenes and object categories in daily life. We do not use the dataset used in the image composition in stage 2 because it contains fewer object categories and fewer object-scene/object relations, e.g., many images do not have objects or only one object. \n\nWe thank the reviewer for suggesting two more large datasets that are primarily used for video segmentation. We find that MS-COCO may be better than YTBVOS and HRlickr for stage 1 as MS-COCO covers more object, scene, and relation types. However, it is possible that merging several datasets can further benefit the tasks, and we leave this in the future work.\n\n> Q9: Is the model trained on all nine datasets combined OR separately…\n\nR9: The model was trained on all nine datasets combined, and the training and testing datasets are from different data distributions as the prior works (e.g., AnyDoor, IMPRINT,). We follow exactly the same settings in prior works and use DreamBooth and COCO val as the testing datasets for fair comparisons with previous methods. This setting aims to evaluate image customization or composition methods’ ability in out-of-distribution scenarios [B,C,D]. We will better clarify this in the final version.\n\n[A] Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis, in CVPR, 2023.\n\n[B] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, CVPR, 2023\n\n[C] AnyDoor: Zero-shot Object-level Image Customization, in CVPR, 2024\n\n[D] IMPRINT: Generative Object Compositing by Learning Identity-Preserving Representation, in CVPR, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wYzTwfkIGt", "reviewer": "Reviewer_wo4W", "summary": "This paper deals with a “generative fill” task where the generation model aims to fill the reference object to the target location of a background image in a reasonable flavor. The first difficulty is to figure out the exact target location and the relative depth for the reference object to appear in the generated image. The authors dealt with this problem by fine-tuning a MLLM on a collected counterfactual dataset. The second difficulty is to preserve both general and detailed information of the reference object and condition the diffusion model to produce a plausible result. The proposed method disentangles various visual elements by using existing depth predictor and segmentation networks and recompose those visual representations obtained to a unified generated image through a diffusion model.", "questions": "1. For the MLLM evaluation, how do LLaVa and miniGPTv2 compare to the proposed model? Are they tuned or what? If not tuned, it is almost for certain that they are not comparable to a tuned one (the proposed). \n\n2. The paper mentioned that they chose DINOv2 as an ID extractor since other papers used the same thing. But did the authors try other models? This is a curiosity question and is not related to the final consideration of rating so the authors should feel comfortable not to answer this question if they feel space is a limitation.", "limitations": "The paper proposed a dataset and built the model by using their crafted dataset. It might make sense to check whether the way and the source to build the dataset could lead to data privacy and copy right issues.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors provided access to their codes and dataset, which can be beneficial to the community, if open-sourced afterwards.\n\n2. The proposed model establishes a good result and positions higher than compared methods.", "weaknesses": "1. The proposed method lacks technical novelty. I’m not sure whether this paper is the first to embed depth into the image composition pipeline but it is a common practice in the community to incorporate depth, such as ControlNet, to nominate a popular one. Classifier-free guidance and using videos as training data are seemingly popular techniques and their efficacy is widely proved. Fine-tuning MLLM using LoRA is also common. Therefore, the major techniques are not new, nor offering new knowledge to the community. \n\n2. The proposed method has a decent reliance on various large models, which means that the inference runtime and deployment to restricted devices can be an issue.\n\n3. The method assumes the reference object to have a single depth value across all pixels. This assumption may not work for large objects like trains in a multi-layer image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer wo4W\n\nWe thank the reviewer for the positive feedback and insightful comments. We address the questions and comments below.\n\n> Q1: ...lacks technical novelty…\n\nR1: Many thanks for acknowledging that our method obtains good results than other models and for the contribution of our code and datasets. In this work, we show that our method achieves controllable image generation by incorporating depth as a condition, while prior image composition or text-driven image editing works suffer from object occlusion problems, e.g., existing works fail when injecting an image behind an object in the background images. To the best of our knowledge, we are the first to embed depth information for image composition. We believe that our work and results can inspire future works to consider depth as a valuable source of prior information for image composition. Additionally, the counterfactual dataset collected in this work also enables the MLLM model to predict the 2.5D location of the reference object in a given background image and will also be beneficial beyond this task, such as 3D localization and robotics (navigation, grasping, etc).\n\n> Q2: …inference runtime…\n\nR2: Although training our model requires considerable computing resources as mentioned in Appendix Section A, the runtime cost and resources required for the inference stage are affordable. Our model can run on a single NVIDIA RTX 4090 GPU (24GB) thanks to our two-stage training/inference since one does not need to load all models simultaneously. The total inference time for one image composition can be finished in 20 seconds on an RTX 4090 (these include predicting the depth map, running the SAM model to remove the background of the reference image, using MLLM to predict the 2.5D location, depth fusion, and image composition, the DDIM sampler is set as 50 steps). We will release our code and pre-trained models to the public.\n\n> Q3: …a single depth value across all pixels…\n\nR3: In our method, we provide a hyperparameter to control the range of the depth of the reference objects, and it is set as 0 by default (i.e., all pixels are set to the predicted depth). And we can see that this works well for most cases. In case of compositing large objects like trains in a multi-layer image, one can adjust the hyperparameter (by cross-validation) to a larger value to allow a wider depth range for large objects. The details are provided in our submitted code (line 390 in run_inference.py) and we will clarify this in the final version.\n\n> Q4: …how do LLaVa and miniGPTv2 compare to the proposed model…\n\nR4: LLaVA and MiniGPT2 are not fine-tuned as they are able to predict a location (bounding box) given the text instruction, especially LLaVA. And we show the comparisons between them and our fine-tuned MLLM for demonstrating the effectiveness of our created dataset for understanding the relations of objects in a scene and improving this ability of existing models.  \n\n> Q5: …why DINOv2…\n\nR5: We use DINOv2 as the ID extractor as in the prior works (e.g., AnyDoor and IMPRINT [A, B] ) where the effects of various image encoders such as CLIP, DINOv2, and MLP for ID extractors have been investigated and the analysis shows the superiority of DINOv2. We will better clarify this in the final version.\n\n[A] AnyDoor: Zero-shot Object-level Image Customization, in CVPR, 2024\n\n[B] IMPRINT: Generative Object Compositing by Learning Identity-Preserving Representation, in CVPR, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GvD5TIrgpU", "reviewer": "Reviewer_CdZU", "summary": "This paper presents a 3D-aware image compositing framework to guide the depth-aware placement of objects within images. The method leverages a multi-modal LM which proposes a bounding box and depth of the object to be inserted. This is followed by fusion reference object depth with the background's depth map. Then a diffusion model conditioned on the mask and fused depth map produces the final output. The method aims to address limitations in existing 2D compositing methods by enhancing 3D spatial comprehension. The paper claims significant improvements in usability, visual quality, and realism over previous methods.", "questions": "* How does the generalization compare with related works? \n* How does the method compare with related works quantitatively in terms of 3D awareness?\n* what is runtime cost and resources required for inference?\n* What is the depth value used for the box? center or average value? Center can miss the object completely? (e.g. circular or objects with holes). How is this situation avoided?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 4, "strengths": "* Innovation. The integration of depth maps and the use of a multi-modal large language model (MLLM) for 2.5D location predictions are innovative. These elements are well-justified and represent a significant advancement over existing 2D compositing methods. This basically solves the occlusion issues in previous methods.\n* Qualitative results. Results shown in the paper demonstrate clear improvements in the realism and quality of the composed images. I particularly like the study on varying depth values (Fig. 10).", "weaknesses": "* Clarity. The paper writing is poor and feels rushed. There are several typos and uncommon word choices that distract the reader. Some aspects are inadequately described (particularly depth fusion during inference) and are not entirely clear.\n* Quantitative results. The authors show metrics/user study for quality, fidelity etc but do not present any quantitative results on '3D awareness', which is the main goal of the paper. For example: Since fused depth maps are available, authors can compare object-level depth error compared to previous works like AnyDoor.\n* Generalization: Generalization is an important desired quality of compositing methods. The paper mentions limitations with out-of-domain datasets but does not provide sufficient details or analysis on how significant this issue is. How does the generalization compare with related works? More empirical evidence and discussion on generalization would strengthen the paper.\n* Computational Efficiency: There is limited discussion on the computational cost of the proposed method. Resources required for inference. Runtime? \n\nMinor comments / Typos:\n\n\"Image composing\" is a rather uncommon phrase. A more common term is \"image compositing\" or \"image composition\".\n\"Depth extractor\" can be confused as a model used to predict depth. \"Depth feature extractor\" or \"Depth encoder\" can be more appropriate. Similar for \"detail extractor\".\n\nFig 5 also contains confusing naming. (Depth map -> depth features?) (detail map -> detail features?)\n\nL133: \"...object we want to place in, indicates l consists of a 2D bounding box...\" This sentence doesn't make sense grammatically.\n\nL180: \"cd = Ed (DPT(Itar)), where Ed is the depth extractor and ch is the extracted depth condition.\" ch -> cd\n\nL226: \"During training, we process the image resolution to 512 × 512.\" \"process\" -> \"set\"\n\nL245: \"However, to our knowledge, none of the MLLM support accuracy depth prediction.\" \"accuracy\" ->\"accurate\"?\n\n\nL215: \"We first scale the depth map of the reference image to the predicted depth\" How? \nDepth fusion is vague and inadequately described.\n\nUser study is a quantitative result and should come under Sec 4.2\n\nFig 7 row 1 col 2. Bbox doesn’t match with predictions made by the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer CdZU\n\n\nWe thank the reviewer for the positive feedback and insightful comments. We address the questions and comments below.\n\n> Q1: ...clarity…\n\nR1: Thanks, we will proofread the paper and correct the writing errors. Regarding our depth fusion algorithm, we briefly introduce it, and details are provided in supplementary B3 (our code is attached) due to the space limitation in the main paper. Here, we provide the pseudo-code for depth fusion:\n\n     depth fusion(ref_depth, bg_depth, BBox, depth_range):\n         # def_depth: the depth map of the reference object\n         # bg_depth: the depth map of the background image\n         # BBox: bounding box predicted by the MLLM in format (x, y, w, h)\n         # depth_range: [B-A, B] where A is a hyperparameter set by users, the default value is 0 \n         #  \t\t      B is the target depth predicted by the MLLM\n\t     # resize the depth map of the reference object to the shape of the bounding box\n\t      resized_ref_depth = reshape((bbox[2], bbox[3]), ref_depth)\n\t      scaled_ref_depth = scale(max=B, min=B-A, resized_ref_depth)\n\t      for i in range(w):\n\t\t     for j in range(h):\n\t\t\t     if scaled_ref_depth[i, j] > bg_depth[x+i, y+j]:\n\t\t\t\t     bg_depth[x+i, y+j] = scaled_ref_depth[i, j]\n\t\t\t     end if\n\t\t     end for\n\t     end for\n\t     return bg_depth\n\n\n> Q2: ...quantitative results…3D-awareness…\n\nR2: Prior works for image composition methods are not aware of the geometry relations between the reference objects and scenes (or objects in the scene) and fails to composite images when the relations are complicated to them (e.g., Figure 7 in the main paper, when inject an image behind an object of the background image, existing methods suffer from object occlusions). Unlike them, our method is able to place the object in the correct location in spatial and depth dimensions (Figure 7), which indicates the 3D-awareness of our method. And the object level depth errors of previous works will be much higher than ours, i.e., prior works place the reference images in the wrong location (The last column In Figure 7 in the main paper shows that the prior work, AnyDoor, generated the final image where the object is not behind the bowl as required, leading to higher depth errors than ours).  We can see that some alternative methods including Paint-by-Example (PbE) and ObjectStitch can also handle the occlusion issue to some extent (but they fail to generate the correct reference objects) and we conduct user study for comparisons of quality, fidelity, diversity. We further conduct user study for evaluating the 3D-awareness reported in Table 6 in  the attached pdf in 'Response to All Reviewers', which indicates the 3D-awareness of our method.\n\n> Q3: ...generalization ability…\n\nR3: Following the same setup in prior works[A,B,C], the training and testing data are exclusive for image composition. This has evaluated the OOD ability of the image composition of our method, indicating the good generalization ability of our method. As mentioned in the limitation section in our paper, the OOD ability for the MLLM in our stage 1 can be affected by the number of object and scene categories in the dataset in stage 1 for predicting 2.5D location (the estimated depth may not be very accurate but still in a reasonable range for OOD objects or scenes). However, as we only require a rough depth value for image composition, the effects of OOD issue for image composition in stage 2 are relatively minor. \n\nTo further verify this, we report the results of OOD objects and scenes in the attached pdf in the 'Response to All Reviewers'. As we can see in Figure 20, though both objects and scenes have not been seen in stage 1 (e.g., piano and church are not included in MS-COCO), the MLLM can still predict reasonable depth values for image composition. However, in some cases, like the example in Figure 21 where the method should put the backpack behind the bed, though the location predicted by MLLM is behind the bed, the location overlaps with the chair in the background. However, our method still obtains better performance than prior work [B] in this case, and they fail to deal with the occlusion with the bed and chair in the image due to the lack of understanding of the relations of objects and scenes in spatial and depth dimensions.  We will add this discussion to more clearly clarify the generalization ability of our method in the final version.\n\n> Q4: ...computational effciency…\n\nR4: Please kindly refer to the answer of Q4 in the 'Response to All Reviewers'.\n\n\n> Q5: ...depth value used for the box…\n\nR5: We use the center value for prediction. Please refer to the answer of Q3 in the 'Response to All Reviewers'. Such rare cases will not influence the evaluation performance of our MLLM. The objects in the MS COCO dataset are common. Thus, there is no (or very small number of) such rare cases in the training and evaluation dataset we created. Therefore, there is no such kind of noisy data that degrades the performance of our MLLM. The MLLM learns to predict reasonable 2.5D locations from the training data without seeing the reference image we want to place. As a result, during the inference stage, the MLLM will still predict correct locations that satisfy the spatial relations in the text instructions without considering the shape of the input objects.\n\n\n[A] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, CVPR, 2023\n\n[B] AnyDoor: Zero-shot Object-level Image Customization, in CVPR, 2024\n\n[C] IMPRINT: Generative Object Compositing by Learning Identity-Preserving Representation, in CVPR, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XO0kS6XCGD", "reviewer": "Reviewer_dLCn", "summary": "This paper proposes a 3D-aware framework for generative image compositing. It first fine-tune a MLLM with custom counterfacual dataset consisting of image-instruction-answer triplets to predict object bounding box and depth value. In the second stage, a pretrained diffusion model is fine-tuned with large-scale custom image. Experiments show that the proposed framework presents a great spatial understanding ability.", "questions": "Please see weakness.", "limitations": "The authos have adequately addressed the limitations and societal impact.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The practical design of leveraging MLLM (for 2.5D location prediction) and pretrained diffusion models for 3D-aware object compositing is of great impact. \n- The dataset generation process for finetuning of both MLLM and diffusion models is inspirational. \n- The paper is well written and carefully presented.\n- Experiments are solid and extensive.", "weaknesses": "- How was the image-instruction-answer triplet data generated? Line 150-151 didn't describe the instruction generation process clearly. Also description of how instructions are generated in figure 3 caption is ambiguous. \n- Typo in figure 3 caption: \"Fine-tuning\"->\"fine-tuning\"\n- Font in figure 5 could be larger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer dLCn\n\nWe thank the reviewer for the positive feedback and insightful comments. We address the questions and comments below.\n\n> Q1: ...image-instruction-answer triplet data generation…\n\nAs we described in the Dataset Generation section. We have gotten the names of the selected objects (“laptop” in the example in Figure 3 in the main paper), the names of the objects that have spatial relations with (“keyboard” in the example), and the spatial relations with these objects (“to the left of” in the example). Then, we ask GPT3 to generate a text instruction that includes these key components. An example of the final output can be “Place the laptop to the left of the keyboard, output the bounding box and the depth value of the center point.” However, the usage of GPT is not mandatory. The usage of GPT3 is only for sample text instructions given spatial relations and object names, which will not use the “reasoning” ability of GPT. This simple task can even be done by a naive approach by setting pre-defined instruction templates and filling the blanket with spatial relations and object names. Therefore, this is totally reproducible and scalable. We will release the script to reproduce the dataset.\n\n> Q2: typos\n\nThanks, we will proofread the paper and correct the errors in the revised version.\n\n> Q3: font size\n\nThanks for the suggestion, it would be better if the font size was larger. We will enlarge the font size in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Response to all Reviewers

We thank the reviewers for their valuable time and feedback, and for acknowledging the well writing (Reviewer M1TS, dLCn), interesting ideas (Reviewer M1TS), solid results (Reviewer M1TS, CdZU, dLCn), and novel contribution (Reviewer M1TS, CdZU, dLCn). To our best knowledge, our work is the first attempt to embed depth into image composition and enables 3D-aware compositing images (e.g., our method can understand the spatial and depth relations and generate images with both realism and fidelity). We address the questions and comments below and we will make our code and datasets publicly available for the final version.

> Q1: Novelty and Technical Contributions (To Reviewer M1TS, wo4W)

R1: Many thanks for acknowledging the novelty in the context of image composition. In this work, we show that our method achieves controllable image generation by incorporating depth as condition, while prior image composition or text-driven image editing works suffer from object occlusion problems e.g., existing works fail when injecting an image behind an object in the background images. Additionally, the counterfactual dataset collected in this work also enables the MLLM model to predict the 2.5D location of the reference object in a given background image and will also be beneficial beyond this task, such as 3D localization and robotics (navigation, grasping etc).

> Q2: …predict only a single depth value but not the entire object…  (To Reviewer M1TS, CdZU)

R2: The goal of predicting the depth value of the reference object is to estimate the location (in depth dimension) of the object in the background image for the final image composition. To this end, it does not have to be very accurate as long as it is in a reasonable range and a single depth value for the object is shown to be sufficient. Though it is possible to estimate the depth value for the entire object (e.g., via DPT), it is more costly and challenging to construct the dataset (pixel-wise depth annotation) and train the MLLM to predict the depth value for the entire object. We will explore more efficient strategies for this in the future.

> Q3: Why the center point of the bounding box but not other points?  (To Reviewer M1TS, CdZU) 

R3: As mentioned in Q2 above, the predicted depth value indicates the location in depth dimension of the reference object in the background image and it does not have to be very precise as long as it is in a roughly reasonable range and a single depth value is sufficient. There are various options for estimating the depth value, such as the depth value at the center point of the bbox, the maximum, average, and median depth value of the object within the bbox, etc. They all have their pros and cons and result in similar performance. For instance, the median value varies significantly along with the size of the object in the bounding box. The average value may be influenced by extreme values in the bounding box. Furthermore, the maximum value might also be influenced by the extreme value in the bounding box. We conducted an extra experiment in Figure 19 in the attached PDF file to compare the differences between different choices. We calculate the differences between different choices of the depth value we want to predict on 5000 examples in the evaluation dataset. The results show that the difference between the maximum depth value and the center point value is small. Most of the differences are less than 0.2. While the differences between the average value and the center point value basically follow a Gaussian distribution with a mean value of -0.05. However, the median value is not reliable compared with other choices. In this work, as we focus on general settings where reference objects are from common categories, we use the depth value of the bbox center point, which works well in our experiments. Optimizing the location for depth value estimation may be helpful for objects from specific categories which are rare, and we leave this in future work. We will add this discussion in our final version.


> Q4: Runtime Cost and resources required for inference  (To Reviewer wo4W, CdZU)

R4: Although training our model requires considerable computing resources as mentioned in Appendix Section A, the runtime cost and resources required for the inference stage are affordable. Our model can run on a single NVIDIA RTX 4090 GPU (24GB) thanks to our two-stage training/inference since one does not need to load all models simultaneously. The total inference time for one image composition can be finished in 20 seconds on an RTX 4090 (these include predicting the depth map, running the SAM model to remove the background of the reference image, using MLLM to predict the 2.5D location, depth fusion, and image composition, the DDIM sampler is set as 50 steps). We will release our code and pre-trained models to the public for the final version.

Meta-review: This paper proposes to integrate depth information into instruction-based image composing. The authors also introduce a counterfactual dataset of image-text pairs for the task of object location prediction. Both qualitative and quantitative results demonstrate the effectiveness of the proposed approach. After a rebuttal discussion period, this paper received all positive reviews. The reviewers unanimously agreed that the idea is interesting, the dataset is novel, and the performance is good compared to the state-of-the-art methods. AC recommends that the authors incorporate all reviewers' feedback into the final camera-ready version and congratulates the paper on its acceptance.
Decision: Accept (poster)