$\textit{Bifr\"ost}$: 3D-Aware Image Compositing with Language Instructions

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 08:02:35

Abstract: This paper introduces $\textit{Bifröst}$, a novel 3D-aware framework that is built upon diffusion models to perform instruction-based image composition. Previous methods concentrate on image compositing at the 2D level, which fall short in handling complex spatial relationships ($\textit{e.g.}$, occlusion). $\textit{Bifröst}$ addresses these issues by training MLLM as a 2.5D location predictor and integrating depth maps as an extra condition during the generation process to bridge the gap between 2D and 3D, which enhances spatial comprehension and supports sophisticated spatial interactions. Our method begins by fine-tuning MLLM with a custom counterfactual dataset to predict 2.5D object locations in complex backgrounds from language instructions. Then, the image-compositing model is uniquely designed to process multiple types of input features, enabling it to perform high-fidelity image compositions that consider occlusion, depth blur, and image harmonization. Extensive qualitative and quantitative evaluations demonstrate that $\textit{Bifröst}$ significantly outperforms existing methods, providing a robust solution for generating realistically composited images in scenarios demanding intricate spatial understanding. This work not only pushes the boundaries of generative image compositing but also reduces reliance on expensive annotated datasets by effectively utilizing existing resources in innovative ways.

Main paper: 
  
 Appendix:  


Decision: This paper proposes to integrate depth information into instruction-based image composing. The authors also introduce a counterfactual dataset of image-text pairs for the task of object location prediction. Both qualitative and quantitative results demonstrate the effectiveness of the proposed approach. After a rebuttal discussion period, this paper received all positive reviews. The reviewers unanimously agreed that the idea is interesting, the dataset is novel, and the performance is good compared to the state-of-the-art methods. AC recommends that the authors incorporate all reviewers' feedback into the final camera-ready version and congratulates the paper on its acceptance.

Reviews: review: paper_summary: paper_summary: ## Summary of the Paper:

*Problem Statement*:

     Given a single-object image I_ref with object O_ref , a background image I_bg containing an object O_bg, and a text prompt P_text, the paper proposes a method for composing O_ref onto I_bg in such a way that the position of O_ref in the resulting image adheres to the text query P_text.


*Motivation*:
	
     Existing works on image composition, which mainly relied on processing images in the 2D domain, underperformed in the presence of object occlusion (O_ref in I_ref) . This paper, named BiFrost, argues that image composition algorithms can benefit from processing beyond the 2D domain. A natural choice would be leveraging priors from the 3D domain.  However, due to the complexity of single-image-to-3D analysis, the paper propose to use 2.5D information by incorporating depth information into their pipeline.

*Contributions*:

1)	 Use of depth information in image composing task.
2)	 A new dataset of image-text pairs for object location prediction task i.e.to predict 2D bbox and depth value of the object. 
3)      Leveraging multi-modal large language models (MLLM) into their pipelines


*Dataset used*:

1)	 For predicting 2.5D information: Newly created dataset of image-text pairs based on MS-COCO dataset 
2)	 For image composing task: 9 datasets -- 3 video-based, 6 image-based datasets. (this setting is similar to previous work in this area) 


*Underlying Modeling Tool*:

     Transformer is the common modeling tool among all involved tasks, including depth map generation (GPT3, DPT from CVPR 2021, SAM and Diffusion-based Inpainting Model), MLLM fine-tuning (using LLaVa), and image composition task (DPT, SAM, DINO_V2 and Stable Diffusion)



*Learning Mechanism*: 

     Strongly supervised


*Loss Functions*:

     For fine-tuning MLLM, negative log-likelihood of the generated text tokens

     For training image composing task, standard diffusion loss i.e. L2 loss between ground truth noise added to image at time step t and predicted noise at that step.

Quantitative Metric:

     DINO-score, CLIP-score, FID, User study


*Baselines*:

     On the image composition task, four existing methods are compared against: Paint-by-Example, ObjectStitch, TF-ICON, AnyDoor.
strength_weakness: strength_weakness: 1)	 It seems that the paper is trying to force-fit large foundation models for a simple task of incorporating depth information. What was the reasoning behind this? What other alternatives were explored?  
2)	 Almost all of the components are based on large foundation models and have frozen weights. It leaves the paper with a single technical contribution, i.e., the use of 2.5D information, instead of 2D.
3)	 The proposed pipeline involves fine-tuning a MLLM, which is in-turn, used to predict only a single depth value. Since SAM is used to get object masks, why not use depth values on the entire object? 
4)	 Two-stage training -- I was wondering why stage-1 (where the MLLM is finetuned) cannot be merged with stage-2 to obtain a single end-to-end training pipeline? The reasoning being that since MLLM is tasked to predict only the bounding box and a single depth value 
5)	 Using a single depth value can be erroneous. For e.g., in Figure 3, the star mark on “masked image” shows that the center point of laptop is on the corner of the laptop. There could be cases where the center point is NOT on the object of interest. How does the model perform in such scenarios? 
6)    Due to the many sub-modules involved in the method, starting from training a depth predictor module using GPT3, I suspect the method is not easily extendable, nor replicable within a certain margin error. I want to understand the authors' stand here.
questions: questions: 1)     How do you calculate center point for predicting depth value? In my understanding it is the center of the bbox. Is that correct?
2)	 Why just a single depth point, why not multiple points or object mask? As mentioned earlier, a single point can be erroneous.
3)	 Is it assumed that the reference object image I_ref will always have one object? Are there any images during train/test time with multiple objects? 
4)	 When creating the dataset for MLLM finetuning, why use MS-COCO dataset? Why not use a subset of the training data from image composing task itself? How does training on MS-COCO dataset help in predicting better depth values, for, say, YouTubeVOS or HFlickr dataset? 
5)	 Is the model trained on all nine datasets combined OR separately,  and test only on 900 object-scene combination images based on DreamBooth and COCO-Val set? Again, my question is why different datasets for training and test? Are they from same distribution?
limitations: limitations: Yes, limitations of the work have been discussed.

No societal impact can be seen from the work.
paper_summary: paper_summary: This paper deals with a “generative fill” task where the generation model aims to fill the reference object to the target location of a background image in a reasonable flavor. The first difficulty is to figure out the exact target location and the relative depth for the reference object to appear in the generated image. The authors dealt with this problem by fine-tuning a MLLM on a collected counterfactual dataset. The second difficulty is to preserve both general and detailed information of the reference object and condition the diffusion model to produce a plausible result. The proposed method disentangles various visual elements by using existing depth predictor and segmentation networks and recompose those visual representations obtained to a unified generated image through a diffusion model.
strength_weakness: strength_weakness: 1. The proposed method lacks technical novelty. I’m not sure whether this paper is the first to embed depth into the image composition pipeline but it is a common practice in the community to incorporate depth, such as ControlNet, to nominate a popular one. Classifier-free guidance and using videos as training data are seemingly popular techniques and their efficacy is widely proved. Fine-tuning MLLM using LoRA is also common. Therefore, the major techniques are not new, nor offering new knowledge to the community. 

2. The proposed method has a decent reliance on various large models, which means that the inference runtime and deployment to restricted devices can be an issue.

3. The method assumes the reference object to have a single depth value across all pixels. This assumption may not work for large objects like trains in a multi-layer image.
questions: questions: 1. For the MLLM evaluation, how do LLaVa and miniGPTv2 compare to the proposed model? Are they tuned or what? If not tuned, it is almost for certain that they are not comparable to a tuned one (the proposed). 

2. The paper mentioned that they chose DINOv2 as an ID extractor since other papers used the same thing. But did the authors try other models? This is a curiosity question and is not related to the final consideration of rating so the authors should feel comfortable not to answer this question if they feel space is a limitation.
limitations: limitations: The paper proposed a dataset and built the model by using their crafted dataset. It might make sense to check whether the way and the source to build the dataset could lead to data privacy and copy right issues.
paper_summary: paper_summary: This paper presents a 3D-aware image compositing framework to guide the depth-aware placement of objects within images. The method leverages a multi-modal LM which proposes a bounding box and depth of the object to be inserted. This is followed by fusion reference object depth with the background's depth map. Then a diffusion model conditioned on the mask and fused depth map produces the final output. The method aims to address limitations in existing 2D compositing methods by enhancing 3D spatial comprehension. The paper claims significant improvements in usability, visual quality, and realism over previous methods.
strength_weakness: strength_weakness: * Clarity. The paper writing is poor and feels rushed. There are several typos and uncommon word choices that distract the reader. Some aspects are inadequately described (particularly depth fusion during inference) and are not entirely clear.
* Quantitative results. The authors show metrics/user study for quality, fidelity etc but do not present any quantitative results on '3D awareness', which is the main goal of the paper. For example: Since fused depth maps are available, authors can compare object-level depth error compared to previous works like AnyDoor.
* Generalization: Generalization is an important desired quality of compositing methods. The paper mentions limitations with out-of-domain datasets but does not provide sufficient details or analysis on how significant this issue is. How does the generalization compare with related works? More empirical evidence and discussion on generalization would strengthen the paper.
* Computational Efficiency: There is limited discussion on the computational cost of the proposed method. Resources required for inference. Runtime? 

Minor comments / Typos:

"Image composing" is a rather uncommon phrase. A more common term is "image compositing" or "image composition".
"Depth extractor" can be confused as a model used to predict depth. "Depth feature extractor" or "Depth encoder" can be more appropriate. Similar for "detail extractor".

Fig 5 also contains confusing naming. (Depth map -> depth features?) (detail map -> detail features?)

L133: "...object we want to place in, indicates l consists of a 2D bounding box..." This sentence doesn't make sense grammatically.

L180: "cd = Ed (DPT(Itar)), where Ed is the depth extractor and ch is the extracted depth condition." ch -> cd

L226: "During training, we process the image resolution to 512 × 512." "process" -> "set"

L245: "However, to our knowledge, none of the MLLM support accuracy depth prediction." "accuracy" ->"accurate"?


L215: "We first scale the depth map of the reference image to the predicted depth" How? 
Depth fusion is vague and inadequately described.

User study is a quantitative result and should come under Sec 4.2

Fig 7 row 1 col 2. Bbox doesn’t match with predictions made by the method.
questions: questions: * How does the generalization compare with related works? 
* How does the method compare with related works quantitatively in terms of 3D awareness?
* what is runtime cost and resources required for inference?
* What is the depth value used for the box? center or average value? Center can miss the object completely? (e.g. circular or objects with holes). How is this situation avoided?
limitations: limitations: yes
paper_summary: paper_summary: This paper proposes a 3D-aware framework for generative image compositing. It first fine-tune a MLLM with custom counterfacual dataset consisting of image-instruction-answer triplets to predict object bounding box and depth value. In the second stage, a pretrained diffusion model is fine-tuned with large-scale custom image. Experiments show that the proposed framework presents a great spatial understanding ability.
strength_weakness: strength_weakness: - How was the image-instruction-answer triplet data generated? Line 150-151 didn't describe the instruction generation process clearly. Also description of how instructions are generated in figure 3 caption is ambiguous. 
- Typo in figure 3 caption: "Fine-tuning"->"fine-tuning"
- Font in figure 5 could be larger.
questions: questions: Please see weakness.
limitations: limitations: The authos have adequately addressed the limitations and societal impact.

Comments: title: Official comment by Reviewer wo4W
comment: Dear Reviewer 1M1TS,

We express gratitude for your time spent reviewing our work and your valuable comments. 

In fact, as we discussed in the common response, although the midpoint does not always work, it works most of the time since we focus on general settings where reference objects are from common categories. $\textbf{The rare cases will not influence the performance of the fine-tuned MLLM}$ for predicting reasonable locations that satisfy spatial relations in the language instructions.

There are various options for estimating the depth value, such as the depth value at the center point of the bbox, the maximum, average, and median depth value of the object within the bbox, etc. $\textbf{They all have their pros and cons and result in similar performance}$. For instance, the median value varies significantly along with the size of the object in the bounding box. The average value may be influenced by extreme values in the bounding box. Furthermore, the maximum value might also be influenced by the extreme value in the bounding box. We conducted an extra experiment in Figure 19 in the attached PDF file to compare the differences between different choices. We calculate the differences between different choices of the depth value we want to predict on 5000 examples in the evaluation dataset.

Therefore, $\textbf{we can not find a perfect choice that works well in all cases}$. Optimizing the location for depth value estimation may be helpful for objects from specific categories which are rare, and we leave this in future work. We will add this discussion in our final version.

We will be happy to answer your question if you have any further concerns about our work! 


Best regards,

Authors
Authors,

Thanks for the reminder.

I have read through the rebuttal. I see most of my questions have been answered.

However, for Q5, I do not agree that the midpoint of bbox always lies on the object of interest for common categories. Thing of an office desk with no drawers. The response to Q5 is not convincing, and infact, factually incorrect.

As well, similar questions regarding the viability of predicting a single depth value were raised by other reviewers.

As such, I am not inclined on increasing the score of the paper. It will be. BA from my side.
# Response to Ethics Reviewer ZVAS

We thank the reviewer for the valuable time and suggestions for better clarifying ethics concerns. We have updated the clarification here, and we will include it in the revised version.

> Misuse of Technology

The Bifröst framework, while groundbreaking in its ability to perform 3D-aware image composition based on language instructions, also presents a significant risk of misuse. The advanced capabilities of Bifröst could be exploited to create realistic deepfakes, generate misinformation, or produce other harmful content that can deceive or manipulate people. These potential misuses underscore the need for the development of robust detection tools that can identify and flag synthetic media generated by Bifröst. Additionally, it is crucial to establish and enforce ethical guidelines that govern the use of this technology, ensuring that it is applied only in responsible and constructive ways. Promoting awareness and education about the ethical implications of such technology is equally important to prevent misuse and protect societal integrity.

> Bias and Fairness

Addressing biases in datasets is crucial to ensuring that the outputs generated by Bifröst are fair and unbiased. Biases can manifest in various ways, such as underrepresentation of certain demographic groups, leading to skewed or inaccurate image compositions. To mitigate these biases, we employ several strategies throughout the data preparation and model training processes. For example, data augmentation techniques are used to balance the representation of different demographic groups, ensuring that the model is exposed to a diverse range of scenarios during training. Additionally, further effects, including thorough evaluations of the model's outputs to detect any systematic disparities or biases, are needed. By continuously refining these processes, we strive to minimize biases and promote fairness in the outputs generated by Bifröst.

> Privacy Concerns

The Bifröst framework handles visual data with a strong emphasis on privacy and data protection. Visual data used in Bifröst is collected and processed in strict compliance with data protection regulations, ensuring that individuals' privacy rights are respected at all stages. Consent mechanisms are integral to the data collection process, where explicit consent is obtained from individuals before their data is used. By adhering to these privacy measures, Bifröst ensures that its operations are both ethical and legally compliant.

> Transparency and Reproducibility

While the Bifröst framework is extensively documented, we will take further steps to enhance transparency and reproducibility. We will make the code, datasets, and supplementary materials publicly available, which will enable other researchers to verify and build upon the work and contribute to the broader scientific community. Additionally, the ethical considerations related to the datasets and methodologies used in Bifröst also need to be clearly articulated. This includes transparent documentation of the origins of datasets, any potential biases, and the steps taken to mitigate these issues.
# Response to Ethics Reviewer daJe
We thank the reviewer for the valuable time and suggestions for better clarifying the user study and copyright of using the datasets. We clarify these issues here, and we will include the clarification in the revised version.

> Q1: user study

I appreciate your emphasis on the significance of IRB approvals for user studies in our research. It is indeed crucial to ensure that research involving human subjects adheres to ethical standards and guidelines, and obtaining IRB approval is a key step in this process. 

The Institutional Review Board (IRB) plays a crucial role in upholding research ethics by rigorously evaluating methodologies in studies involving human subjects. The primary goal of these reviews is to protect the well-being of participants, ensuring they are not exposed to harm and that their rights and welfare are fully safeguarded. This is accomplished through a careful assessment of research protocols and related documents to identify and mitigate potential risks.

In the context of this research paper, it's important to highlight that the study design does not present any risks to participants. We utilize a user study methodology where participants are asked to provide feedback on the images generated by our approach compared to those from other methods. To further ensure participant safety, we have implemented strict safety checks within our image generation process to prevent the inclusion of harmful or inappropriate content. Thus, the study adheres to ethical standards by minimizing potential risks and prioritizing the protection of participant rights and welfare.

In this context, we address checklist item 15 as outlined by the regulatory body, as our study does not involve research concerning human subjects. Rather, it may be construed as an online questionnaire administered for human evaluation purposes.


> Q2: the use of YoutubeVOS dataset

The copyright of YotubeVOS dataset is credited to [A] in ECCV 2018 workshop. We obtained the license before using the dataset.

[A] Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang and Thomas Huang, YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark, in ECCV, 2018
Dear Reviewer wo4W,

Thank you for raising your score and for supporting our work and its contributions! We sincerely appreciate the time you dedicated to reviewing our paper, considering the rebuttal, and analyzing the other reviews. We will incorporate additional experiments of hyperparameter adjustment in the revised version.

Best wishes,

Authors
Dear Reviewer CdZU,

Thanks for raising your score and for the support of our work and it's contributions! We greatly appreciate the time you spent reviewing our paper, and going through the rebuttal and the other reviews, as well as your valuable suggestions for improving our work. We will correct the writing errors and incorporate extra experiments in the revised version.

Best wishes,

Authors
Dear Reviewer dLCn,

Thanks for your reply! We appreciate your positive comments and will incorporate your suggestions in the revised version.

Best wishes,

Authors
# Comment

Dear Reviewer 1M1TS,

Thanks again for your time in reviewing our work.

This is a gentle reminder that the Discussion period will end on 13 Aug 11:59 pm AoE. There is only one day remaining. Would you mind checking our responses to your concerns and questions and confirming whether you have any other questions? We are glad to answer any further questions you may have before the deadline.

Best regards,

Authors
Thanks for addressing my concerns. Most of the original concerns are well addressed. For the single depth value issue, I think it might make sense to show the experiments of hyperparameter adjustment as proposed by the authors in the revision. I will raise my score to accept.
I appreciate the authors providing the user study results on '3D awareness' and clarifying other issues. The authors have acknowledged the writing errors and promised to improve them in the final version. Most of my concerns have been addressed, and I recommend accepting this paper.
I have read all the other reviewers' comments and the authors' responses. My concerns have been addressed by the authors. I will not reduce my score.