Title: HOComp: Interaction-Aware Human-Object Composition
Venue: NeurIPS 2025
Accepted: yes
Keywords: Human-Object Interactions; Image Composition
Abstract: While existing image‑guided composition methods may help insert a foreground object onto a user-specified region of a background image, achieving natural blending inside the region with the rest of the image unchanged, we observe that these existing methods often struggle in synthesizing seamless interaction-aware compositions when the task involves human-object interactions.
In this paper, we first propose HOComp, a novel approach for compositing a foreground object onto a human-centric background image, while ensuring harmonious interactions between the foreground object and the background person and their consistent appearances. Our approach includes two key designs: (1) MLLMs-driven Region-based Pose Guidance (MRPG), which utilizes MLLMs to identify the interaction region as well as the interaction type (e.g., holding and lefting) to provide coarse-to-fine constraints to the generated pose for the interaction while incorporating human pose landmarks to track action variations and enforcing fine-grained pose constraints; and (2) Detail-Consistent Appearance Preservation (DCAP), which unifies a shape-aware attention modulation mechanism, a multi-view appearance loss, and a background consistency loss to ensure consistent shapes/textures of the foreground and faithful reproduction of the background human. We then propose the first dataset, named Interaction-aware Human-Object Composition (IHOC), for the task.
Experimental results on our dataset show that HOComp effectively generates harmonious human-object interactions with consistent appearances, and outperforms relevant methods qualitatively and quantitatively.


Peer reviews:
comments: SUMMARY:
The paper introduces HOComp, a diffusion-based framework that composites a foreground object into a human-centric background image. A multimodal large language model is used to choose an interaction type (e.g., holding, kicking) and a plausible spatial region on the person. The authors propose a multi-view appearance loss, and a background consistency loss to ensure consistent shapes/textures of the
foreground and faithful reproduction of the background human, and propose Detail-Consistent Appearance Preservation (DCAP) module for preserving appearance. Also, the authors propose a new dataset, IHOC for the human object composition task. Experimental results show competitive results compared with previous image composition methods.

STRENGTHS AND WEAKNESSES:
Strengh:
(1) The paper is well-written and easy to follow.
(2) The motivation of human-object interaction in customization task is interesting.

Weakness:
(1) Does this model support multi-subject, multi-person composition? For instance, 2 persons and 2 objects. I am willing to see these results in the rebuttal.
(2) The impact of the area of overlap region to the model's performance is not explored in the paper. What if there is large overlap regions between human and subjects? I suggest the authors do such experiments compared with previous methods in the rebuttal.

QUESTIONS:
What is the MLLM's influence on the inference performance? Could you compare using MLLM with other vision-based methods to detect the overlap regions?

LIMITATIONS:
See weakness
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces HOComp, a novel framework for human-object image composition that enables natural and realistic interactions between humans and inserted foreground objects. Unlike prior methods that rely on manually defined regions and text prompts, HOComp autonomously determines the interaction type and region using Multimodal Large Language Models (MLLMs) like GPT-4o, and enforces pose and appearance consistency using several novel loss functions and mechanisms.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. A new approach for interaction-aware human-object composition that ensures realistic, coherent, and visually consistent integrations of foreground objects into human-centric images.\n2. The first large-scale dataset tailored for this task, including paired before/after images, interaction masks, text prompts, and detailed annotations. \n\nWeaknesses:\n\n1. The authors claim that the model can produce physically plausible results, yet no physical constraints are integrated to guarantee that. \n2. There is a lack of comparison of human-object interaction generation methods, making it difficult to evaluate the effect of the proposed method in this domain.\n3.  To more accurately assess the physical plausibility of the generated motions, it may be beneficial to incorporate a more sophisticated metric for measuring interpenetration, such as \"Intersection Volume.\" This metric directly quantifies the volume of penetration between interacting bodies and can provide a more precise evaluation of physical correctness. For reference, the paper \"Physics-aware Hand-object Interaction Denoising\" (CVPR 2024) offers a detailed discussion and implementation of such metrics, which may serve as a useful guideline.\n\nQUESTIONS:\n1. Several previous works, such as CG-HOI and CHOIS, have annotated the BEHAVE dataset with textual descriptions, and datasets like OMOMO have further extended this line of research into 3D human-object interactions with textual annotations. However, the distinction between the proposed dataset and these prior efforts remains unclear. The authors are encouraged to clarify what specific advantages or novel features their dataset introduces. \n\n2. Is affordance estimation explicitly required in the proposed HOI generation method? The paper would benefit from a clarification on this point.\n\nLIMITATIONS:\n1. The paper lacks a comprehensive discussion of recent related work. In particular, several relevant works published in 2024 have not been acknowledged or discussed, including CG-HOI, HOIAnimator, InterFusion, Controllable HOI Synthesis, and F-HOI. These recent methods contribute significantly to the field of human-object interaction generation and text-driven synthesis. \n\n 2. To better understand the generalization ability of the proposed model, it may be helpful to evaluate its performance on interactions involving completely unseen objects that are not included in the BEHAVE or OMOMO datasets. If the dataset is skewed toward certain interaction types, there is a risk that the model may overfit to these patterns.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
1. HOComp present to integrate a foreground object onto a human-centric background image seamlessly while ensuring harmonious interactions and preserving the visual consistency of both the foreground object and the background person.

2. MRPG utilizes MLLMs to identify the interaction region and interaction type to provide coarse-to-fine constraints, while incorporating human pose landmarks to track action variations and enforcing fine-grained pose constraints. DCAP unifies a shape-aware attention modulation mechanism, a multi-view appearance loss, and a background consistency loss to ensure consistency.

3. This paper introduces a Interaction-aware Human-Object Composition (IHOC) dataset to facilitate HOC task

STRENGTHS AND WEAKNESSES:
Strengths 

This paper proposes an effective image editing model for compositing a foreground object onto a human-centric background image. It includes two modules: MLLMs-driven Region-based Pose Guidance to identify the interaction and Detail-Consistent Appearance Preservation to ensure consistency. The paper is well-organized and easy to follow. They also reports sufficient qualitative and quantitative results to support their claims.

Weaknesses

The entire process is quite complex, not only involving multiple processing units to encode the features of the foreground and background, but also employing several loss functions to enforce consistency. I am not very sure about the generalizability of HOComp and the reproducibility of the overall architecture.

QUESTIONS:
The entire framework includes multiple complex modules to ensure the synthesis of seamless interaction-aware compositions. Does the author team plan to open-source the project and the proposed dataset to promote community development?

LIMITATIONS:
yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces HOComp for interaction-aware human-object composition. Existing methods often fail to generate plausible human poses or preserve appearance. HOComp addresses this with two key components. First, MLLM-driven Region-based Pose Guidance (MRPG) automatically determines the interaction type and region to guide pose generation. Second, Detail-Consistent Appearance Preservation (DCAP) uses multiple losses and attention modulation to maintain foreground and background consistency. The authors also introduce the IHOC dataset for this task. Experiments show their method significantly outperforms state-of-the-art approaches in generating harmonious and realistic compositions.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths\n\n1. The proposed MRPG module is a novel use of MLLMs to automate the generation of high-level interaction semantics (type and region), simplifying the user input process.\n\n2. The creation of the IHOC dataset and the HOIBench benchmark is a significant contribution to the community, providing valuable resources for future research in this area.\n\n3. The quantitative and qualitative results, including a user study, are extensive and demonstrate a clear and substantial improvement over numerous state-of-the-art methods.\n\n- Weaknesses\n\n1. The framework's performance heavily relies on an MLLM for initial guidance. While the authors report a high success rate, the system is still susceptible to failures from this non-differentiable module, which can lead to incorrect or nonsensical interactions.\n\n2. The system architecture is highly complex, integrating multiple models (DiT, MLLM, pose estimator, multi-view generator) and a carefully balanced set of loss functions. This complexity may pose challenges for reproducibility, training stability, and tuning.\n\n3. The IHOC training dataset is partially constructed using other generative models for inpainting backgrounds and completing occluded objects. This introduces a potential data bias, where the model may be implicitly learning to reverse the artifacts of the data-generation models rather than learning a more general composition skill.\n\nQUESTIONS:\n1. Regarding the MRPG module's reliance on GPT-4o: You are commendably transparent about the 91.33% success rate. However, a ~9% failure rate at the first step is a significant practical limitation. Could you provide a more detailed analysis of these failure modes? For example, what are the common types of errors (e.g., incorrect body part, illogical interaction type), and how does the rest of the pipeline behave when it receives this incorrect guidance?\n\n2. Regarding the dataset construction: The method of creating paired data by using an inpainting model to generate the \"before interaction\" image raises a critical question about potential model bias. The model might be learning to invert the specific process of the inpainting tool rather than a general interaction generation skill.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper present HOComp, a framework for interaction-aware human-object composition. It leverages MLLM driven region-based pose guidance (MRPG) for constrained human-object interaction, and detail-consistent appearance preservation (DCAP) for maintaining appearance consistency. This paper also introduced the Interaction-aware Human-Object Composition (IHOC) dataset. Extensive experiments demonstrate that HOComp outperforms existing methods in quantitative, qualitative, and subjective evaluations.

STRENGTHS AND WEAKNESSES:
Strengths:
1. This paper propose a new approach for interaction-aware human-object composition, named HOComp, which focuses on seamlessly integrating a foreground object onto a human-centric  background image while ensuring harmonious interactions and preserving the visual consistency of both the foreground object and the background person.
2. HOComp incorporates two innovative designs: MLLMs-driven region-based pose guidance (MRPG) for constraining human-object interaction via a coarse-to-fine strategy,  and detail-consistent appearance preservation (DCAP) for maintaining consistent foreground/background appearances.
3.This paper introduce the Interaction-aware Human-Object Composition (IHOC) dataset which maybe can helpful for others research.
Weaknesses:
1.Ordinary and unremarkable writing
2.The novelty level is not too high
3.Too few bad cases

QUESTIONS:
1.Ordinary and unremarkable writing
2.The novelty level is not too high

LIMITATIONS:
1.hope have more bad cases analysis
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-04