HOComp: Interaction-Aware Human-Object Composition
Authors: Dong Liang
Jinyuan Jia
Yuhao LIU
Rynson W. H. Lau
Venue: NeurIPS 2025 poster ()
Keywords: Human-Object Interactions; Image Composition
Primary area: applications

Abstract: While existing image‑guided composition methods may help insert a foreground object onto a user-specified region of a background image, achieving natural blending inside the region with the rest of the image unchanged, we observe that these existing methods often struggle in synthesizing seamless interaction-aware compositions when the task involves human-object interactions.
In this paper, we first propose HOComp, a novel approach for compositing a foreground object onto a human-centric background image, while ensuring harmonious interactions between the foreground object and the background person and their consistent appearances. Our approach includes two key designs: (1) MLLMs-driven Region-based Pose Guidance (MRPG), which utilizes MLLMs to identify the interaction region as well as the interaction type (e.g., holding and lefting) to provide coarse-to-fine constraints to the generated pose for the interaction while incorporating human pose landmarks to track action variations and enforcing fine-grained pose constraints; and (2) Detail-Consistent Appearance Preservation (DCAP), which unifies a shape-aware attention modulation mechanism, a multi-view appearance loss, and a background consistency loss to ensure consistent shapes/textures of the foreground and faithful reproduction of the background human. We then propose the first dataset, named Interaction-aware Human-Object Composition (IHOC), for the task.
Experimental results on our dataset show that HOComp effectively generates harmonious human-object interactions with consistent appearances, and outperforms relevant methods qualitatively and quantitatively.

HOComp: Interaction-Aware Human-Object Composition
Dong Liang
Jinyuan Jia
Tongji University / CityUHK HKUST(GZ))
Tongji University / HKUST(GGZ)
sse_ Hangdonggtooggii..deee
j jinyuanjiaChkust- -Z edu.
Yuhao Liu' CityUHK yuhaoliu74450gmai..coom com
Rynson W.H. Lau'
CityUHK Rynson.LauSci Lww.cctyuuuccc..o.mmmmm edu. edu. hk
Abstract
While existing image-guided composition methods may help insert foreground object onto a user-specified region of a background image, achieving natural blending inside the region with the rest of the image unchanged, we observe that these existing methods often struggle synthesizing seamless interaction-aware compositions when the task involves human-object interactions. In this paper, we first propose HOCoomp, a novel approach for compositing a foreground object onto human-centric background image, while ensuring harmonious interactions between the foreground object and the background person and their consistent appearances. Our approach includes two key designs: (1) MLLMs driveen Region- based Pose Guidance (MRPG), which utilizes MILMS to identify the interaction region well the interaction type (e.g., holding and lefting) provide coarse- to-fine constraints to the generated pose for the interaction while incorporating human pose landmarks to track action variations and enforcing fine-grained pose constraints; and (2) Retail-Consistent Appearance Preservation (DCAP), which unifies shape-aware attention modulation mechanism, multi-view appearance loss, and a background consistency loss to ensure consistent shapes/textures of the foreground and faithful reproduction of the background human. We then propose the first dataset, named Interaction- aware Human Human-Object Composition (IHOC), for the task. Experimental results on our dataset show that HOComp effectively generates harmonious human-object interactions with consistent appearances, and outperforms relevant methods qualitatively and quantitatively. Project page: httpss //dliang293, gi thub. io/HOComp- project/
Introduction
Considering scenario in which designer aims to create perfume advertisement by compositing the image of a product onto an existing photograph with a human person, as shown in row Fig. [ two critical objectives need to be satisfied in order to produce a visually convincing output. First, the interaction between the person and the perfume bottle should appear natural, such that the bottle may seem be appropriately related (e.g., held by) the person. Second, visual consistency must be maintained, preserving the original identities of both the person (including facial features and makeup) and the perfume bottle (e.g., the logo, color, and shape).
1 Joint corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Inputs: human image & object
(b) GPT-4o
S7
pbe 22
(d) AnyDoor
(c) Ours
AIRI
(a) Inputs: human image & object
(i) Generaled video frames
Figure I: When compositing a foreground object onto human-centric background image, existing methods (b-d) typically rely on manually specifying the target region and text prompt, and often produce unrealistic interactions and inconsistent foreground/backgroundd appearances. In contrast, our proposed HOComp, automatically identifies the target region and generates a suitable text prompt to guide the interaction, resulting in realistic, harmonious and diverse interactions. Note that the text prompts used by the existing methods in the above three examples are: "A model is showing a perfume bottle", "A girl holding hat", and "A woman is lifting a handbag". By integrating with an Image-to- Video (I2V) model, our approach can support applications like human-product demonstration video generation (see results on the bottom region).
Some existing image-guided composition tasks 900 37 may be most relevant to the above task setting. They take a user-supplied foreground exemplar, typically accompanied by a textual prompt and user-defined target region, and aim to synthesize harmonious composition. Within this paradigm, they either incorporate identity- piriity-preeevatioon modules [9] 68) to explicitly retain the original foreground details focus adjusting the colors, shadows, and perspective of the foreground to harmonize it with the background [49
70 92 67), thereby producing photorealistic compositions. Despite the success, when the composition involves human and object interactions, depicted in
Fig. 1 existing methods 9 67 92 struggle to produce satisfactory results.
For our composition task, observe that existing methods tend to fail in or both of the following ways: (I) they may produce inappropriate gestures for the background persons (e.g. most results in Fig.
[ [c.d)); and (2) they may change the contents/identities of the foreground objects (e.g., rows 2 and of Fig. I (b-d) and/or the background persons (e.g., the face in row of Fig. [b), and the clothes in row 2 of Fig. [b.c) and row 3 of Fig. d.d). To address these problems, propose HOComp, an interaction-wwal aware human-object composition framework, to create seamless composited images with harmonious human-object interactions and consistent appearances.
Our HOComp includes two key designs. The first design is the MI-.MM-drriven region-based pose guidance (MRPG), which aims constrain the human-object interaction. By utilizing the capabilities

---

of MLLMs, our method automatically determines suitable interaction types (e.g., holding, eating) and interaction region. Here, we adopt a coarse-to-fine constraint strategy. We first use the interaction region generated by MILMMs as a coarse-level constraint to restrict the region of the background image for the interaction, We then incorporate human pose landmarks as a supervision to capture the variation of the human pose in the interaction, providing a fine-grained constraint on the pose within the interaction region. The second design is the detail-consisten appearance preservation (DCAP), which aims to ensure foreground/background appearance consistency. For the foreground object, we propose a shape-aware attention modulation mechanism to explicitly manipulate attention maps for maintaining a consistent object shape, and multi-viey appearance loss to further preserve the object textures at the semantic level. For the background image, we propose a background consistency loss to retain the details of the background person outside the interaction region.
To train the model, we introduce new dataset called Interaction- aaware Human-Ohject Composition (IHOC) dataset, which includes images of humans before and after interacting with the foreground object, the interaction region, and the corresponding interaction type. We conduct extensive ex- periments on this dataset, and the results demonstrate that our approach can generate accurate and harmonious human-object interactions, resulting in highly realistic and convincing compositions.
The main contributions of this work include:
We propose a new approach for interaction-aware human-object composition, named HO- Comp, which focuses seamlessly integrating foreground object onto a duman-centric background image while ensuring harmonious interactions and preserving the visual consis- tency of both the foreground object and the background person.
2. HOComp incorporates two innovative designs: MI..Ms-drrven region-based pose guid- ance (MRPG) for constraining human-object interaction via a coarse-to-fiee strategy, and detail-consistent appearance preservation (DCAP) for maintaining consistent fore- ground/background appearances.
3. We introduce the Interaction-aware Stuman-Obeec Composition (IHOC) dataset, and con- duet extensive experiments on this dataset to demonstrate the superiority of our method.
2 Related Works
Image-guided Composition. It aims to seamlessly integrate a user-provided foreground exemplar onto designated region of background image, sometimes with textual guidance. Existing methods either focus on appearance harmonization (i.e., adjusting colors, shadows, and perspective) in order
to integrate the foreground onto the background scamlessly [57 98 g 60, 69
61
39
79
17 7 or
emphasize identity preservation by introducing dedicated modules to maintain the identity consistency of the object across scenes g 84 36 99 [66]. However, these methods primarily refine the foreground and often fail to generate natural, realistic human gestures or poses in human-object interactions (HOIs). While DreamFuse [26] adjusts the foreground to adapt the background context, it supports only limited hand actions and struggles with complex HOIs. Recent works 772 74 88 80
2 45 multi-modality methods [86] 52 44), these approaches often unintentionally modify the background human and introduce inconsistencies in the foreground object.
propose unified frameworks integrate multiple image generation/editing tasks. Similar to
Multi-Concept Customization. It aims to generate images that align with both the text prompt and user-specifted concepts, facilitating the creation of personalized content. Tuning- based methods [34 41 typically incorporate new concepts into diffusion models by fine-tuning specific parameters, but each new concept requires separate tuning process. Instead, training- based
1
71
48
47
16,
12 40] train additional modules to extract the identity of a concept and inject it into the denoising network via attention layers. Training-free methods [14] 78 96 (55) incorporate reference-awwre attention mechanisms. These methods typically re-generate both the foreground object and background human, leading to inconsistent background human appearance.
methods [87] (56] [03 777 33 10 42
Human- Interaction (HOI) Generation. It aims to synthesize images that depict plausible and coherent interactions between humans and objects. Recent diffusion-based methods generate HOI
"This interaction type is embedded in the text prompt. For example, "A woman is holding hat", and "A kid is eating donut."
3

---

Telorme Plice
Tradiike Plare
Diftane (hnntbner
MiLLai-diiva Reyies Pose Gullines DIBPC)
Nern
Frypioon
Pouide
Licw
Soparin
MILLAledrive Deglowshaed (MBIF))
Desuk-cnntrnar Promeratio (DCAF)
Arar Hedildion
Froepe
the munuir
Single
Sugb Dourndo
Olger Rt
Made
Kopis teremaaie
Branccia
Madaiaka
No hhewkir
Oiige
Treat
Figure 2: Pipeline of HOComp. Our method includes two core modules: MRPG for constraining hujan-object interaction and DCAP for maintaining appearance consistency. Inference Phase (left): MRPG uses MLLMs to generate text prompt C, object box B, and interaction region B, Among these, B, and C' are encoded and, together with the object ID, detail features, and background features, are used to condition the DiT for final composition generation. Training Phase (right): MRPG constrains the interaction by applying a pose-guided loss Lyuse with keypoint supervision. DCAP enforces appearance consistency via: (1) shape-awan attention modulation to adjust the attention maps to follow the object's shape prior Matape; (2) multi-view appearance loss Cuppearance semantically align synthesized and input foregrounds (multí-views); and (3) background loss Lhackground to preserve original background details.
images/videos by introducing extra cues, such as bounding boxes 20 30 25 or pose structures [01
38 6), reference videos 88 [2], 3D priors [46) and in-context samples of similar interactions 227
102 288. However, all these approaches require additional inputs during inference (e.g. human
poses images describing the interaction). Some works 197 [91] adjust human hand poses during interactions, but this is often insufficient for complex scenarios. Other methods [63] 24 54 15 93
employ relation-awars frameworks to improve HOI generation in subject-driven settings, yet they fail to preserve the background human appearance consistency. Concurrent works, DreamActor- H1 (76) and Hunyuan Video-HOMA 22), explore human interaction in the contexts of human--rodune demonstrations and animated human-object interactions. They incorporate additional modality guidance and exploit the strong multi-modal fusion capabilities of the DiT framework for video generation.
35 94 focus 3D human-object interaction generation, aiming to produce physically plausible 3D human-object interactions. These approaches typically generate 3D models or motion sequences conditioned on textual semantic descriptions of the interaction. However, due to their reliance on 3D information as training references, the range of interaction types they can handle remains limited.
Several methods 13 65
75
In summary, existing methods fall short in addressing the challenge of our interaction- aware human- object composition task, which requires the model to produce harmonious human-object interactions and consistent foreground/background appearances.
3 Method
Given foreground object image Ir and background image In containing human subject, our goal to synthesize a harmoniously composited image Ip that integrates the foreground object onto the human-centric background image. The composited image should exhibit harmonious interactions and maintain appearance consistency between the foreground object and the background human.
To achieve this objective, we propose HOComp, an interaction-ane aware human-object composition framework, as illustrated Fig. 2 Our framework includes two key components: MLLM-driven Region-based Pose Guidunce (MRPG) and Detail-Consisee Consistent Appearance Preservation (DCAP). MRPG leverages Multimodal Large Language Models (MILMMs) and human pose priors to constrain human-object interaction in a coarse-to-fin manner. DCAP preserves the shape and texture of the

---

foreground object while maintaining details of the background human, ensuring faithful and coherent appearance reproduction throughout the composited scene.
In the remainder of this section, we first introduce the preliminaries in Sec. 3.1 We then detail the design of MRPG in Sec. 3.2 followed by DCAP in Sec. [33 Finally, we describe Interaction-
aware Sllman-Objec Composition (IHOC) dataset in Sec. 3.4
3.1 Preliminary
Diffusion Transformer (DiT) is transformer- based diffusion model for image synthesis. Given noisy latent ž, at timestep t, it predicts the denoised output via 2. = DiT t, c), where c denotes a conditioning signal (e.g., text embeddings or visual prompts). Owing to its scalability and strong generative capacity, DiT' serves as a robust backbone for conditional image generation.
Attention Manipulation is key strategy for improving semantic alignment and structural control in diffusion models through attention map editing, external signal injection, or modified attention weight computation. For a standard attention layer defined as = soft QK T /V V, manipulation introduces a structured bias or conditioning modulation: A' = softmax( (QK + MM) +M/vd)j where M E R encodes spatial priors prompt-specific relevance (e.g., object masks).
3.2 MLLM-drriven Region-based Pose Guidance (MRPG)
MRPG adopts a coarse-to-fine strategy to constrain the human-object interaction. At the coarse level, it leverages the reasoning capabilities of MILMs to automatically identify suitable interaction type and corresponding interaction region through a multi-stage querying process. At the fine level, a pose- guided loss is introduced to impose fine-grained constraints human poses within the interaction region, explicitly supervising the predicted image using human pose keypoints.
Generating Interaction Regions and Types. As illustrated in Fig. 2 we employ MLLMs (e.g.. GPT-4o) chain-of- of-thought, a step-by-sted process to generate the interaction type (denoted as a text prompt C') and the interaction region (represented by a bounding-boo B,). While the interaction type specifies what interaction to be performed by the background person on the foreground object (e.g., holding), the interaction region specifies the location in the image that the interaction is to be performed. Specifically, we send the foreground object and the background image to the MILM and query it in three-stage approach: (1) With set of initial prompts the instruction guidance, ask the MLLM to envision a plausible interaction type and return the interaction type in the form of text prompt description C; (2) Conditioned on C, we ask the MLLM to further infer potential region (i.e., bounding box B,) in the background image where the foreground object is to be placed; (3) We ask the MLLM to identify the interaction region B, by considering which human body parts are involved in the interaction. The generated interaction region B, is converted into a mask, encoded via VAE 32 and used alongside text embeddings E,, conditioning inputs the DiT model.
Imposing Fine-grained Pose Guidance. Considering the significant correlation between human- object interactions and body poses, we introduce a pose-guided loss Loose to impose fine-grained constraints on poses within the interaction region. Let P'r and Pired represent the i-th keypoint detected by a pose estimator G, from the ground-truth image IGT and the predicted image /p. respectively. The pose-guided loss Lp is formulated as:
1 Σ Cp
P
Pprea
IEB,
where n denotes the number of pose keypoints located within the interaction region B,. illustrated in Fig. [ This localized pose-guided loss explicitly directs the model's optimization efforts towards accurately capturing human poses involved in the interaction, rather than globally adjusting the entire human pose, thereby enhancing the realism and harmony of the generated interaction.
3.3 Detail-Consistent Appearance Preservation (DCAP)
To ensure fine-grained appearance consistency, for the foreground, we first extract identity and detail information as conditioning inputs for the DiT model. To enforce shape consistency, we introduce a shape-aware attention modulation mechanism to adjust the foreground-relevant attention maps in the
5

---

MM-DiT blocks, guiding the attention maps to align with the foreground object's shape prior better. For texture consistency, we propose a multi-view appearance loss maintain semantic alignment across multiple viewpoints. For the background, we leverage an unchanged region mask to identify unaffected areas and impose background consistency loss to preserve original background details.
Foreground Object Identity and Detail Extraction. We first preprocess the foreground object by removing the background and centering it. To capture the identity information, we then employ the DINOv2-based ID encoder [53], renowned for robust semantic representations, to extract the foreground ID features EID. As the resulting identity tokens have coarse spatial resolution and therefore lack texture details, we extract a high-frequency detail map, desaill as an additional condition: Idstail gray GaussianBlur (ray)) where I gray the grayscale foreground image. A lightweight detail encoder (9 processes Lenail to extract detail features E,, which are then fused with foreground ID features EID to condition the DřT model.
Rerak
Arc.
Shape-aware Attention Modulation. To enhance shape consistency, we modulate foreground-relevant attention maps in the MM-DiT blocks, encouraging the attention maps to align more precisely with the object's shape prior. This design is motivated by the observation that these attention maps highlight object shapes (see Fig. indicating that the model is able to capture structural cues of the foreground objects.
Figure 3: Visualization of attention maps related to the foreground text embeddings Apg ppg-xx x and the identity features AEm- both exhibiting strong alignment with object shape.
Specifically, we compute two foreground-relevant attention maps: one based on the foreground ID features EID, and the other on the foreground text embeddings E with X denoting the target image tokens. Here, E, are extracted from the full text embedding Ec- For instance, if "toy" is annotated as a foreground object in the text prompt C' "A boy is holding a toy", , E the sub-embedding aligned with "toy" from E.. The attention maps are computed as:
Qx QxK K Fg
QxKEB Vd
Agg - = softmax
AEID = softmax
Vd
where Qx E TR. Nxd are queries from the target image tokens, and KE KEDD E Mxd are keys projected from E and EID. respectively.
To obtain the shape prior, shown in Fig. 2 extract foreground object mask Masape from the ground-truth image. We aim to enhance the attention within the object region while suppressing distractions outside it. Considering that directly modifying the attention maps may potentially compromise the image quality of the pre-trained model [31,, we adopt a residual- based modulation strategy over the extracted attention maps A Agt+x -x and A - incorporate shape priors while preserving the original attention distribution. The modulation is defined as:
A' = A a -MMshope (Amax A) - (1 Mahape) (A (A-44 (41-))m))) ))
(3)
where A E [Agt-xx AEDD-x) Amax and Amin are the per-query maximum and minimum values computed row-wise. The scalar a E R controls the modulation strength. The modulated attention map is then integrated into the DiT model to encourage shape-aware feature learning.
Multi-view Appearance Loss. To address texture inconsistencies caused by changes in viewpoint during interactions, encourage the predicted foreground object maintain consistent semantic appearance with the ground truth across diverse views. Specifically, we synthesis multí-view images for both the predicted result and the input foreground, and measure their semantic similarity.
As shown in Fig. 2 we first segment the predicted foreground object from Ip. Given the segmented output and the input foreground image I,, we apply multi-view generator G to synthesize k views:
V pred
(V()) } 1=1 G(Segment(L,)) VGr
(v(+) ) G(T
We then extract CLIP 58 features from each synthesized view: Flo = CLIP((V(1 ( CLIP(V ).) The multi-view appearance loss is then formulated as:
7(0)
pred s GT =
1 F(i)
Lappearance
pred
(5)
k
pred

---

which encourages semantic alignment of the predicted object with the ground truth across multi-views.
Background Consistency Loss. To preserve the appearance of the background human during the process, we utilize an unchanged region mask Munchnged, which is provided by our dataset and indicates the region that remains unaffected throughout the interaction. By constraining the generated image to match the ground-truth image in this unchanged region, we enforce consistency with the original background appearance. The background consistency loss Lo is defined as:
Lbockground = MMinchangel C ||xET - xpreall
(6)
iEI
where XGT and Xpred denote the pixel values of the ground-truth image Ior and of the predicted image Ip. respectively.
Overall Training Objective. The model is optimized with the composite loss:
Low = Lannoising + auLp + agCo + azCa.
where Ldemoising is the standard denoising loss. L,, Las, faa are the pose-guided, background consis- tency, and multi-view appearance losses. 01, 02; 0g the coefficients of the corresponding loss terms.
3.4 Dataset Preparation
We introduce the Interaction-awarr Human-Object Composition (IHOCC dataset to address the lack of paired pre- and post-interaction data crucial for modeling realistic and coherent human- object compositions. IHOC includes six components: (1) background human images (without the object); (2) foreground object images; (3) composited images with harmonious interactions and consistent appearances; (4) text prompts describing the interaction type; (5) interaction regions; and (6) unchanged region masks to indicate unaffected background areas.
Our dataset is constructed through the following stages: O Composited Images: To enhance data diversity, we adopt the 117 human-object interaction types defined by HICO-DET (S) and include both real and synthetic samples. For real data, we manually select 50 images per type (5,850 total) from HICO-DET, To ensure the quality of our dataset and to reduce bias, we exclude images that (1) contain multiple persons, (2) lack clearly visible persons (e.g., only a hand is shown), or (3) have large parts of the foreground objects occluded or not visible (e.g., only one wheel of a bicycle is visible), making it difficult to identify them. The final selection emphasizes diversity in object type, scale, and human pose across diverse scenes. For synthetic data, we use GPT-4o to generate 50 prompts per type and synthesize 5,850 images using FLUX.1 [dev] [3]. These synthetic samples help complement the real data by introducing a wider range of human appearances, poses, viewpoints, and visual styles (e.g., cartoon, sketches). In total, we have collected 1 11,700 composited interaction examples. Forreground Object Images: Foreground objects are segmented from the composite images using SAM (59). To address occlusions caused by human-object interactions, we use GPT-4o to infer and complete missing regions, producing plausible and visually consistent object appearances. @ Background Human Images & Unchanged Region Masks: We manually inpaint composite images using FLUX.1 Fill [dev] 4 to remove interacting objects and recover plausible human poses without the interactions. An inpainting mask denotes an interaction-altered region; its inverse produces the unchanged region mask, highlighting the area unaffected by the interaction. O Text Prompts & Interaction Regions: For real images, we use GPT-4o to generate text prompts. For synthetic images, we reuse the generation prompts. In addition, we use GPT-4o to annotate each prompt with foreground object tokens, indicating which words correspond the foreground objects. The interaction regions are derived by inverting the unchanged region masks. More information on our dataset, including statistics and visualizations, can found in Sec. of the Appendix.
Experiments
Implementation Details. We adopt FLUX.1 [dev] 3 as the base model and fine-tune it using
LoRA (23) with rank 16, applied to the attention layers. All training images are resized to 512x512 resolution. The model is trained for 10,000 steps with a batch size of 2, using AdamW and learning
rate of le-5. Training takes approximately 20 hours on 2 100 GPUs We employ DWPose (95] for
pose estimation, Zero123+ [64] for multi-view generation and GPT-4o 52 as MLLM in MRPG.
7

---

Table 1: Quantitative comparison of our method with nine SOTA methods. The user study reports the averaged rank (lower is better) of nine methods in image quality (QQ), interaction harmonization (IH), and appearance preservation (AP). The best and second-bes results shown in bold and underlined. respectively. Training or tuning-based methods without released training codes are marked with a
Cuteguey
Metrics
AnyODor
Pht 22
IneoCoomp
1 Froox Coston
[4]
PrimeCoom
GenAm:
[+]
Lnn om 74
GPTT-4
Thes
FID C1 CL.IPSomm
18.57
15.91
22.55
18.57
17.45
12.13
14.52
11.55
0.95
9.27
12.65
29.03
27.56
18.45
78.11
20.77
70.
75:78
70:
30.29
Automatic
HOL Score
25.09
02.53
59.91
57.39
59333
SSIMBBG)
90.71
82.00
[Q2
9:22
7.47
K.20
9:13
2.0
6.25
3.10
1.37
User study
[E
8.18
9.23
R.46
6.72
6.68
5.25
4.88
2.57
2.61
1.14
AP
2.44
5.41
6884
7.33
6.07
4.73
6.54
5.36
587
1,11
Iapur image object
GPT-4e 52
(c) GienArt,
Omniition 88
(e) AnyDear
UniCam
Cturs
Figure 4: Qualitative comparison with six top performing SOTA methods from Table 16 The prompts for the above four examples are: "A girl is reading a magic book", "A woman is holding an ornate folding fan", "A woman opening a gift box", and "A puppet-style old man playing guitar".
Evaluation Metries. We use FID 19 to assess the overall quality of the generated images, where lower score indicates better alignment with real images. To evaluate how well generated image depicts the specified human-object interaction (i.e., HOI Alignment), we compute the HOI-Score using a pre-trained HOI detector (e.g., UPT (0O)). which measures the accuracy of the interaction in the generated image. Additionally, we employ the CLIP-Score 18] to evaluate the global semantic alignment between the generated image and the text prompt. Subsequently, we use the DINO-Score to assess how well the foreground object appearance is preserved, where higher score indicates better appearance consistency to the input foreground object. Finally, background consistency is evaluated by computing the Structural Similarity Index (SSIM) 81 over the area outside the interaction region, where higher SSIM(BG) score indicates a better retention of the original background.
Benchmark. We introduce a new benchmark, HOIBench, to evaluate the quality the human-object interaction task. We begin by collecting 30 images, each with human person, from the internet. The humans in these images cover diverse appearances, including different poses and clothes. Half of these images feature the upper body, while the other half depict the full body. To ensure a broad range of interaction types, we adopt the 117 interaction types defined in the HICO-DET [22] We prompt GPT-4o with each type to infer a plausible foreground object (e.g., playing + guitar). A concise textual description of each object is then used to retrieve a representative image from the internet, yielding 117 interaction
image pairs. Finally, for each human image, we randomly sample 20 interaction-object pairs from generated set, producing total of 600 human-object interaction instances (20 interactions 30 human images) for evaluation.
8

---

Table Ablation study on removing one of the key components from our full model (left table) and adding one of the key components to our base model (right table). Lp. L.. L., and SAAM denote the pose-guided loss, background consistency loss, multi-view appearance loss, and shape-aware attention modulation, respectively. Best performances are marked in bold.
FID
C1JP1
SSTM(RC))
69.
28.42
59.72
47.32
29.37
(a) Input image object
(b) Only faemsting
cc) W/o
(6) W/6
(c) W/o 4.
W/o SAAM
(E) With
S S
2
20
Figure 5: Visual comparison of the ablation study in Table 2
4.1 Comparison with State-of-the-Art Methods
We compare HOComp with SOTA methods: AnyDoor 9 Paint by Example 92], FreeCom- pose [II] FreeCustom [14], OmniGen [88] GenArtist (8), PrimeComposer 779. UniCombine [74
and GPT-4o 12 All methods with public training code are retrained or fine-tuned on our dataset.
Quantitative Comparison. Table 16 compares the performances of our method against the nine existing methods. The results in the top part of the table show that our method consistently outper- forms all these baselines across all evaluation metrics. Specifically, it achieves the highest HOI-Score (87.39), surpassing GPT-4o by 12.17 and OmniGen by 25. 25.06, underscoring its strong ability to model accurate and coherent human-oke object interactions. In terms of visual consistency, our method achieves the lowest FID (9.27) and the highest CLIP-Score (30.29), demonstrating superior realism and semantic alignment ability. Our DINO-Score (78.21) significantly outperforms AnyDoor by 19.38 and GPT-4o by 13.0, indicating improved foreground appearance consistency. Further, our model produces the most consistent background details with the highest SSIM(BG) score (96.57), outperforming AnyDoor by 5.86.
Qualitative Comparison. Fig. 4 visually compares the results of our method and those of the six pop-erforrming methods from Table Rows 3-4 of Fig. (b) show that although GPT-4o can synthesize plausible human-object interactions, it fails to maintain foreground appearance consistency. Meanwhile, its generated backgrounds exhibit substantial variations, as shown in rows 1-3 of Fig. b). Similar to GPT-4,, GenArtist and OmniGen also suffer from foreground baroouuuuubb inconsistency. In addition, methods in Fig. ee-g) produce suboptimal or implausible hand poses. In contrast, our method effectively constrains the generated human poses as well as the shapes/textures of the foreground objects. As result, the images produced by our method exhibit superior appearance consistency with harmonious human-object interactions.
User Study. We have also conducted a user study to compare our method with all 9 existing methods. We recruit a total of 75 student participants for the subjective assessment. Each participant presented with 10 sets of cases, where each set contains an input human image, a foreground object, text prompt to describe the interaction, and ten randomly shuffled results from HOComp and the 9 competing methods. Participants rank the images based on three criteria: image quality (IQ), interaction harmonization (IH), and appearance preservation (AP). We collect ranking scores from all participants and compute the average ranking for each of the three aspects, as shown in bottom part of Table [6 These results show that our approach ranks first in all three aspects: image quality (1.37), interaction harmonization (1.14), and appearance preservation (1.11), highlighting it being the most preferred method by all participants.

---

4.2 Ablation Study
Component Analysis. We conduct an ablation study on HOComp by systematically removing one key component from our full model (Table ( or by adding one key component to our base model (Table (right)). Fig. visualizes some results of the ablation study. Based on these results, draw SIx key conclusions: O Pose constraint (Lp) is essential for ensuring proper human pose generation during interactions, When removed, the result in Fig. ((c) exhibits a distorted and incongruous interaction, leading to the lowest CLIP and HOI scores shown in row of Table (left). Its absence also lowers the SSIM(BG) score from 96.57 to 94.91, showing a mild but noticeable loss of background consistency. ? Background consistency loss (La) helps prevent unintended modifications of non-interaction region of the background image. Without it, the person as well as the background scene may undergo significant changes (Fig. ())). resulting in the worst FID score shown in row 2 of Table 2 (left). As a result, the SSIM(BG) score plummets to 49,49, the largest drop among all settings, causing the most severe background degradation. @ Multi-view appearance loss (Ca) ensures consistency in the texture/appearance of the foreground object in the generated image. Removing it leads noticeable color and texture shifts of the object (e.g.. the balloons in Fig. 5(e)) and the lowest DINO score shown in row 3 of Table 2
(left). 4 Shape-aware attention modulation (SAAM) plays a crucial role in preserving object shape consistency. As shown row 4 of Table 2 (left), removing SAAM leads to inconsistent shape transformations and appearance variations, with the DINO score dropping significantly from 78.21 to 66.52. @ Finally, by integrating all key components, our proposed method achieves the best performance, as shown in row 5 of (left) @ (right shows that each component individually enhances a specific aspect of the model. Lp helps improve interaction quality, as reflected in higher HOI and CLIP scores. Lb improves background consistency, evident from the SSIM(BG) score. L. and SAAM help maintain foreground appearance consistency, leading to improved DINO performances.
Conclusion
In this paper, we have presented HOComp, framework for Iaput image ubject
Incorrect
Correct B,,
interaction-aware human-object composition. It leverages MLLM- driven region-based pose guidance (MRPG) for constrained human-object interaction, and detaill-onsissent appearance preser- vation (DCAP) for maintaining appearance consistency. To support HOComp training. we have also introduced the Interaction-awart Human- Object Composition (IHOC) dataset. Extensive experi- ments demonstrate that HOComp outperforms existing methods in quantitative, qualitative, and subjective evaluations.
Figure 6: An example failure case of HOComp. The red boxes indicate the interaction regions.
HOComp does have limitations. Although MLLMs correctly identify the interaction region in 91.33% of the samples in our benchmark, HOIBench, incorrect predictions may still affect the quality of the generated interactions, as shown in Fig. 6 As a future work, we would like to consider incorporating human pose priors into predicting the interaction region.
Acknowledgements
This work is supported in part by Ant Group and General Program of Foundation of China (NSFC, No. 6207071897).
National Natural Science
References
[1] Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a- scene: Extracting multiple concepts from a single image. In ACM SIGGRAPH Asia, pages 1-12, 2023.
[2] Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, et al. Flux. kontext: Flow matching for in-context image generation and editing latent space. arXiv e-prints, pages arXiv-2506, 2025.
10

---

[3] Black Forest Labs. FLUX.I-dev: A 12B Parameter Rectified Flow Transformer for Text- to-Image Generation. httpss//agggiigaaae.o///p/p/
1abs/FLUX
1 d-dev 2024.
[4] Black Forest Labs. FLUX.I-Fill-dey. A 12B Parameter Rectified Flow Transformer for Inpainting and Outpainting. http:://hwggggggga.a..cobbbaaeae
1 1-Fill-dev/ 2024.
[5] Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-ohject interactions. Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), pages 381-389, 2018.
[6] Binghui Chen, Chongyang Zhong, Wangmeng Xiang, Yifeng Geng, and Xuansong Xic. Virtualmodel: Generating object-id-retentive human-object interaction image by diffusion model for e-commerce marketing. arXIv:2405.09993, 2024.
[7] Jiaxuan Chen, Bo Zhang, Qingdong He, Jinlong Peng, and Li Niu. Murcobjectsticch: Multi- reference image composition. arXiv:2411.07462, 2024.
[8] Xi Chen, Yutong Feng, Mengting Chen, Yiyang Wang, Shilong Zhang, Yu Liu, Yujun Shen, and Hengshuang Zhao. Zero-shot image editing with reference imitation. In NeurIPS, volume 37, pages 14010-84033 2024.
[9] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. In CVPR, pages 6593 6602, 2024.
[10] Xi Chen, Zhifei Zhang, He Zhang, Yuqian Zhou, Soo Ye Kim, Qing Liu, Yijun Li, Jianming Zhang, Nanxuan Zhao, Yilin Wang. al. Unireal: Universal image generation and editing via learning real-world dynamics. arXiv:2412.07774, 2024.
[11] Zhekai Chen, Wen Wang, Zhen Yang. Zeqing Yuan, Hao Chen, and Chunhua Shen. Freecom- pose: Generic zero-shot image composition with diffusion prior. In ECCV, pages 70-87. Springer, 2024.
[12] Yufan Deng, Xun Guo, Yizhi Wang, Jacob Zhiyuan Fang, Angtian Wang, Shenghai Yuan, Yiding Yang, Bo Liu, Haibin Huang, and Chongyang Ma. Cinema: Coherent multi-subject video generation via milm-based guidance. arXiv:2503. 10391, 2025.
[13] Christian Diller and Angela Dai. Cg-hoi: Contact-guided 3d human-object interaction genera- tion. In CVPR, pages 19888-19901, 2024.
[14] Ganggui Ding, Canyu Zhao, Wen Wang, Zhen Yang. Zide Liu, Hao Chen, and Chunhua Shen. Freecustom: Tuning-fres free customized image generation for multi-concee composition. In CVPR. pages 9089-9098. 2024.
[15] Oran Gafni and Lior Wolf. Wish you were here: Context-aware human generation. In CVPR, pages 7840-7849, 2020.
[16] Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, and Weijia Wu. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models, In NeurlPS, volume 36, pages 15890-15902, 2023.
[17] Jixuan He, Wanhua Li, Ye Liu, Junsik Kim, Donglai Wei, and Hanspeter Pfister. Affordance- aware object insertion via mask-aware dual diffusion. 200iv:24/2./446633 2024.
[18] Jack Hessel, Ari Holtzman, Maxwell Forbes, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In EMNLP, 2021.
[19] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter: Gans trained by two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017.
11

---

[20] Jiun Tian Hoe, Xudong Jiang, Chee Seng Chan, Yap-Peng Tan, and Weipeng Hu. Interactd- iffusion: Interaction control text-to-image diffusion models. In CVPR, pages 0180-6189 2024.
[21] Lukas Höllein, Aljaž Božič, Norman Müller, David Novotny, Hung- Yu Tseng, Christian Richardt, Michael Zollhöfer, and Matthias Nießner. Viewdiff: 3d-consistent image generation with text-to-image models. In CVPR, pages 5043-5052, 2024.
[22] Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao. Visual compositional learning for human-object interaction detection. In ECCV, pages 584-600. Springer, 2020.
[23] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[24] Xinting Hu, Haoran Wang, Jan Eric Lenssen, and Bernt Schicle. Personahoi: Effortlessly improving personalized face with human-object interaction generation. In CVPR, 2025.
[25] Tianyu Hua, Hongdong Zheng, Yalong Bai, Wei Zhang. Xiao-Ping Zhang. and Tao Mei. Exploiting relationship for complex-scene image generation. In AAAI, volume 35, pages 1594-1592, 2021.
[26] Junjia Huang, Pengxiang Yan, Jiyang Liu, Jie Wu, Zhao Wang, Yitong Wang, Liang Lin, and Guanbin Li. Dreamfuse: Adaptive image fusion with diffusion transformer. arXiv:2504.0822 082991, 2025.
[27] Siteng Huang, Biao Gong, Yutong Feng. Xi Chen, Yuqian Fu, Yu Liu, and Donglin Wang. Learning disentangled identifiers for action-customizzd text-to-image generation. CVPR, pages 7797-7806, 2024.
[28] Ziqi Huang, Tianxing Wu, Yuming Jiang, Kelvin CK Chan, and Ziwei Liu. Reversion: Diffusion-based relation inversion from images. In SIGGRAPH Asia, pages 1-11, 2024.
[29] Ziyao Huang, Zixiang Zhou, Juan Cao, Yifeng Ma, Yi Chen, Zejing Rao, Zhiyong Xu, Hongmei Wang, Qin Lin, Yuan Zhou, et al. Hunyuanvideo-homa: Generic human-object interaction in multimodal driven human animation. arXiv:2506.0879) 2025.
[30] Jian-Yo Jinn Kang-Yan Yang Huang, Ling Lo, Yi-Ning Huang, Terence Lin, Jhih-Ciang Wu, Hong-Han Shuai, and Wen-Huang Cheng. Record: Reasoning and correcting diffusion for hoi generation. In ACM MM, pages 9465-9474, 2024.
[31] Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Was Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. In ICCV, pages 7701-7711, 2023.
[32] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 2013.
[33] Zhe Kong, Yong Zhang, Tianyu Yang. Tao Wang, Kaihao Zhang, Bizhu Wu, Guanying Chen, Wei Liu, and Wenhan Luo. Omg: Seclusion-friendly personalized multi-concept generation in diffusion models. In ECCV, pages 253-270. Springer, 2024.
[34] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Zhu. Multi- concept customization of text-to-image diffusion. In CVPR, pages 1931-1941, 2023.
[35] Jiaman Li, Alexander Clegg, Roozbeh Mottaghi, Jiajun Wu, Xavier Puig, and C Karen Liu. Controllable human-object interaction synthesis. In ECCV, pages 54-72. Springer, 2024.
[36] Lingxiao Li, Kaixiong Gong, Wei-Hong Li, Tao Chen, Xiaojun Yuan, and Xiangyu Yue. Bifrost: 3d-aware image compositing with language instructions. In NeurIPS, volume 37, pages 129480-129506, 2024.
[37] Pengzhi Li, Qiang Nie. Ying Chen, Xi Jiang, Kai Wu, Yuhuan Lin, Yong Liu, Jinlong Peng, Chengjie Wang, and Feng Zheng. Tuning-free image customization with image and text guidance. In ECCV, pages 233-250 Springer, 2024.
12

---

[38] Yuheng Li. Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In CVPR, pages 22511-22521 2023.
[39] Dong Liang, Jinyuan Jia, Yuhao Liu, Zhanghan Ke, Hongbo Fu, and Rynson WH Lau. Vodiff: Controlling object visibility order in text-to-image generation. In CVPR, pages 18379-18389, 2025.
[40] Feng Liang, Haoyu Ma, Zecheng He, Tingbo Hou, Ji Hou, Kunpeng Li, Xiaoliang Dai, Felix luefei-Xu, Samaneh Azadi, Animesh Sinha, et al. Movie weaver: Tuning-free free multi-concept video personalization with anchored prompts. arXiv:2502.07802, 2025.
[41] Wang Lin, Jingyuan Chen, Jiaxin Shi, Yichen Zhu, Chen Liang, Junzhong Miao, Tao Jin, Zhou Zhao, Fei Wu, Shuicheng Yan, et al. Non-confusing generation of customized concepts in diffusion models. arXiv:2405.06914 2024.
[42] Zhe Lin, Zhifei Zhang, He Zhang, Andrew Gilbert, John Philip Collomosse, and Soo Ye Kim. Multitwine: Multi-object compositing with text and layout control. In CVPR, 2025.
[43] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv:2303.05499 2023.
[44] Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi I. Liao, Yingming Wang, Honghao Fu, Chunrui Han, et al. Steplx-edit: A practical framework for general image editing. 1711: 2025.
[45] Yuhao Liu, Zhanghan Ke, Fang Liu, Nanxuan Zhao, and Rynson WH Lau. Diff-plugin, Revitalizing details for liffusion-basee low-level tasks. CVPR, pages 4197-4208, 2024.
[46] Yuhao Liu, Tengfei Wang, Fang Liu, Zhenwei Wang, and Rynson WH Lau. Shape-for-motion: Precise and consistent video editing with 3d proxy. arXiv:2506.22 22432, 2025.
[47] Zhiheng Liu, Yifei Zhang, Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones 2: Customizable image synthesis with multiple subjects. In NeurIPS, pages 57500-57519, 2023.
[48] Zhiheng Liu, Yifei Zhang, Yujun Shen, Kecheng Zheng. Kai Zhu, Ruili Feng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Customizable image synthesis with multiple subjects. In NeurIPS, volume 36, pages 57500-57519, 2023.
[49] Shilin Lu, Yanzhu Liu, and Adams Wai-Kin Kong. Tf-icon: Diffusion- based training- cross-domain image composition. In /CCV, pages
2023.
[50] MidJourney. Midjourney official website, 2025.
[51] Pouyan Navard, Amin Karimi Monsefi, Mengxi Zhou, Wei-Lun Chao, Alper Yilmaz, and Rajiv Ramnath. Knobgen: Controlling the sophistication of artwork in fketch-based diffusion models. arXiv:2410.01595, 2024.
[52] OpenAl. ChatGPT (model 4o). https://chaal www.ai..com.. 2025.
[53] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, al. Dinov2: Learning robust visual features without supervision. TMLR.
[54] Rishubh Parihar, Harsh Gupta, Sachidanand VS, and R Venkatesh Babu. Text2place: Affordance-awarre text guided human placement. In ECCV, pages 57-77. Springer, 2024.
[55] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. ACM SIGGRAPH, pages 1-11, 2023.
[56] Maitreya Patel, Sangmin Jung, Chitta Baral, and Yezhou Yang. X-eclipse: Multi-concept personalized text-to-image diffusion models by leveraging clip latent space. TMLR, 2024.
13

---

[57] Kien T Pham, Jingye Chen, and Qifeng Chen. Tale: Training-free cross-domain image composition via adaptive latent manipulation and energy- geggygg optimization. In ACM MM, pages 31 3160-3169, 2024.
[58] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. ICML., pages 9748-8763, 2021.
[59] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, al. Sam 2: Segment anything in images and videos. arXiv:2408.007 2024.
[60] Mengwei Ren, Wei Xiong, Jae Shin Yoon, Zhixin Shu. Jianming Zhang, HyunJoon Jung, Guido Gerig, and He Zhang. Relightful harmonization: Lighting-aware portrait background replacement. In CVPR, pages 6452-6462, 2024.
[61] Nataniel Ruiz, Yuanzhen Li, Neal Wadhwa, Yael Pritch, Michael Rubinstein, David E Jacobs, and Shlomi Fruchter. Magic insert: Style-aware drag-and-dropg arXiv:2407.02488, 2024.
[62] Mchdi Saface, Aryan Mikaeili, Or Patashnik, Daniel Cohen-Or, and Ali Mahdavi-Amiri, Clic: Concept learning in context. In CVPR, pages 6924-6933, 2024.
[63] Qingyu Shi, Lu Qi, Jianzong Wu, Jinbin Bai, Jingbo Wang, Yunhai Tong, and Xiangtai Li. Dreamrelation: Bridging customization and relation generation. In CVPR, 2025.
[64] Ruoxi Shi. Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero 12: a single image to consistent multi-view diffusion base model. arXiv:2310. 1511100000010,,, 2023.
[65] Wenfeng Song, Xinyu Zhang, Shuai Li, Yang Gao, Aimin Hao, Xia Hou, Chenglizhao Chen, Ning Li, and Hong Qin. Hoianimator: Generating text-prompt human-object animations using novel perceptive diffusion models. In CVPR, pages 811 811-820, 2024.
[66] Wensong Song, Hong Jiang, Zongxing Yang, Ruijie Quan, and Yi Yang. Insert anything: Image insertion via in-context editing in dit. arXiv:2504. (00::2504.1009,, 2025.
[67] Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang, Soo Ye Kim, and Daniel Aliaga. Objectstitch: Object compositing with diffusion model. In CVPR, pages 18310-18319, 2023.
[68] Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang, Soo Ye Kim, He Zhang, Wei Xiong, and Daniel Aliaga. Imprint: Generative object compositing by learning identity-preserving representation. In CVPR, pages
2024.
[69] Weijing Tao, Xiaofeng Yang, Miaomiao Cui, and Guosheng Lin. Motioncom: Automatic and motion-awar image composition with Ilm and video diffusion prior. arXiv:2409. 200,0:9000000000 2024.
[70] Gemma Canet Tarrés, Zhe Lin, Zhifei Zhang, Jianming Zhang, Yizhi Song, Dan Ruta, Andrew Gilbert, John Collomosse, and Soo Ye Kim. Thinking outside the bbox: Unconstrained generative object compositing, arXiv:2409.00 04559, 2024.
[71] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personalization. In ACM SIGGRAPH, pages 1-11, 2023.
[72] Xueyun Tian, Wei Li, Bingbing Xu, Yige Yuan, Yuanzhuo Wang, and Huawei Shen. Mige: A unified framework for multimodal instruction-based image generation and edit- ing. arXiv:2502.21291, 2025.
[73] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from single image using latent video diffusion. In ECCV, pages 439- Springer, 2024.
14

---

[74] Haoxuan Wang, Jinlong Peng. Qingdong He, Hao Yang, Ying Jin, Jiafu Wu, Xiaobin Hu, Yanjie Pan, Zhenye Gan, Mingmin Chi, al. Unicombine: Unified multi-conditional combination with diffusion transformer. 2rX1:2250..09277, 2025.
[75] Li Wang, Xinyu Zhang, Baowei Xv, Jinzhao Zhang, Rong Fu, Xiaoyu Wang, Lei Zhu, Haibing Ren, Pingping Lu, Jun Li, et al. Interfusion: Interaction pleraattoonbased 4d radar and lidar fusion for 3d object detection. In IROS, pages 12247-12253, IEEE, 2022.
[76] Lizhen Wang, Zhurong Xia, Tianshu Hu, Pengrui Wang, Pengfei Wang, Zerong Zheng, and Ming Zhou. Dreamactor-hl: High-fidelity human-product demonstration video generation via motion-designed diffusion transformers. arXiv:2506. 10568, 2025.
[77] Xierui Wang, Siming Fu, Qihan Huang, Wanggui He, and Hao Jiang. Ms-diffusion: Multi- subject zero-shot image personalization with layout guidance. In ICLR, 2025.
[78] Yibin Wang, Weizhong Zhang, and Cheng Jin. Magicface: Training-free free universal-style human image customized synthesis. arXiv:2408.0744 2024.
[79] Yibin Wang, Weizhong Zhang, Jianwei Zheng, and Cheng Jin. Primecomposer: Faster progressively combined diffusion for image composition with attention steering. In ACM MM, pages 10824-10000 10832, 2024.
[80] Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu. Genartist: Multimodal Ilm as an agent for unified image generation and editing. In NeurlPS, volume 37, pages 128374-128395. 2024.
[81] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From enorr visibility structural similarity. IEEE TIP, 13(4):600 6116000061122000 2004.
[82] Yujie Wei, Shiwei Zhang, Hangjie Yuan, Biao Gong, Longxiang Tang, Xiang Wang, Haonan Qiu, Hengjia Li, Shuai Tan, Yingya Zhang, et al. Dreamrelation: Relation-centric video customization. 250%iv22503.000 07602, 2025.
[83] Daniel Winter, Matan Cohen, Shlomi Fruchter, Yael Pritch, Alex Rav-Acha, and Yedid Hoshen. Objectdrop: Bootstrapping counterfactuals for photorealistic object removal and insertion. In ECCV, pages 112-129, 129. Springer, 2024.
[84] Daniel Winter, Asaf Shul, Matan Cohen, Dana Berman, Yael Pritch, Alex Rav-Acha, and Yedid Hoshen. Objectmate: recurrence prior for object insertion and subject-driven generation. 1xXiv:4422086645 2024.
[85] Young Beom Woo and Sun Eung Kim. Flipconcept: Tuning-free multi-concept personalization for text-to-image generation. arXiv:2502.15203, 2025.
[86] xAI. Grok 3: The age of reasoning agents. httg:s///..a//neew///rroo- 2025.
[87] Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han. Fastcom- poser: Tuning-free multi-subject image generation with localized attention. IJCV, pages 1-20, 2024.
[88] Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li. Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. In CVPR, 2025.
[89] Ziyi Xu, Ziyao Huang, Juan Cao, Yong Zhang, Xiaodong Cun, Qing Shuai, Yuchen Wang, Linchao Bao, Jintao Li, and Fan Tang. Anchorerafter: Animate cyberanchors saling your products via human-object interacting video generation. arXiv:2411.17338 2024.
[90] Ben Xue, Shenghui Ran, Quan Chen, Rongfei Jia, Binqiang Zhao, and Xing Tang. Decf: Deep comprehensible color filter learning framework for high-resolution image harmonization. In ECCV, pages 300-316. Springer, 2022.
[91] Zihui Sherry Xue, Romy Luo, Changan Chen, and Kristen Grauman. Hoi-swap: Swapping objects in videos with hand-object interaction awareness. In NewrIPS, volume 37, pages 77132-77164, 2024.
15

---

[92] Binxin Yang. Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. In CVPR, pages 18381-18391, 2023.
[93] ChangHee Yang, ChanHee Kang, Kyeongbo Kong, Hanni Oh, and Suk-Ju Kang. Person in place: Generating associative skeleton guidance maps for human-object interaction image editing. In CVPR, pages 8164-8175 8175, 2024.
[94] Jie Yang, Xuesong Niu, Nan Jiang, Ruimao Zhang, and Siyuan Huang. F-hoi: Toward fine- grained semantic-aligned 3d human-object interactions. In ECCV, pages 91-110. Springer, 2024.
[95] Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effective whole-body pose estimation with two-stages distillation. In ICCV, pages 4210-4220, 2023.
[96] Zebin Yao, Lei Ren, Huixing Jiang, Chen Wei, Xiaojie Wang, Ruifan Li, and Fangxiang Feng. Freegraftor: Training- free cross-image feature grafting for subject-driven text-to-image generation. 25th:25504195558 2025.
[97] Yufei Ye, Xucting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song, Shubham Tulsiani, and Sifci Liu. Affordance diffusion: Synthesizing hand-object interactions. In CVPR, pages 22479-22489, 2023.
[98] Yongsheng Yu, Ziyun Zeng, Haitian Zheng, and Jíebo Luo. Omnipaint: Mastering object- oriented editing via disentangled insertion-removal inpainting. arXiv:2503.0860 2025.
[99] Bo Zhang, Yuxuan Duan, Jun Lan, Yan Hong, Huijia Zhu, Weiqiang Wang, and Li Niu. Controlcom: Controllable image composition using diffusion model. S.4Iv:2338..00444 2023.
[100] Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer. In CVPR, pages 20104- 201 12, 2022.
[101] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pages 9836-3847 2023.
[102] Yuxin Zhang, Fan Tang, Nisha Huang, Haibin Huang, Chongyang Ma, Weiming Dong, and Changsheng Xu. Motionerafter: One-shot motion customization of diffusion models. arXiv:2312.052883 2023.
[103] Yuxuan Zhang, Yiren Song, Jiaming Liu, Rui Wang, Jinpeng Yu, Hao Tang, Huaxia Li, Xu Tang, Yao Hu, Han Pan, et al. Ssr-encoder: Encoding selective subject representation for subject-driven generation. In CVPR, pages 8069-8078, 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
It is sure we have make it clear in abstract and introduction about contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have included limitations in conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have disclosed all the information needed to reproduce the main results in implementation details and appendix.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
We have opensourced our github repository will soon release all the codes and dataset.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/ppp cc/public/guu wetggiddddeeeeeebbiiiinnnn htt.SSbbbiiiiooooiiiiiooooi for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have disclosed all the information in main paper and appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We do not report error bars due to limited computing resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have offered this information in experiment part.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer:
[Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed these problems in Section J of Supp.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We have discussed these problems in Section J of Supp.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have discussed these problems in Section J of Supp.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have provided this information the appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cood/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We have discussed these problems in Section J of Supp.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
We have discussed these problems in Section J of Supp.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: We used MLLMs to assist in identifying interaction regions and interaction types, as well as in generating the dataset. The methodology has been described in detail in the Methods section.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy lattps://neurij: 2tte for what should or should not be described.
23

---

HOComp: Interaction-Aware
Human-Object Composition
Appendix
A Overview
In this appendix, we provide additional implementation details, ablation analyses, and extended evaluations to further support and expand upon the findings presented in the main paper.
Specifically, we address the following key aspects in our appendix: (1) Presenting detailed statistical analyses and the construction procedure of our IHOC dataset (Sec. B (2) Offering additional
clarifications on our approach, including experimental configurations and supplementary ablation
analyses (Sec. G H): (3) Presenting additional experiments to validate our method, including further comparisons with state-of-the art approaches and more results of our method [Sec.[] Kj. (4) Discussing on ethical considerations, data governance procedures, and responsible use safeguards implemented in the development of our method. (Sec.
B Extended Details on IHOC dataset
B.1 Dataset Construction
Composited
Retism
Type
Coopusted
GPT-4a
Nertin
heppning Olacaaly Anmoaaad
Comporace Imago
Bacigreadd Hora lrm
Emage RICO
HUULI
Mixually picod bo
Real world Compeined beager
Unchange Ragio Malia
Rajion
Makic Invanwan
Obriining Prompts
Tear abosinl
plcnng
Stage Extrocting Ferground Objjett Images
3cd przzzi
Compaiiiod keage
Real wild Congeriee beiges
You Phrnga fes
Parggnenn
Objict Emagu
GPT-4o
Figure 7: Overview the construction process of our Interaction-aware Human- Human-Object Composition (IIOCC dataset. It involves four stages: (1) collecting synthesized and real-world composited images, (2) obtaining corresponding text prompts, (3) extracting foreground object images, and (4) getting background human images, unchanged region masks, and interaction regions.
In Sec. 3.4 of the main paper, we briefly discuss our Internation-awwrre Human-Object Composition (IHOC) dataset, which includes six components: (1) background human images (without the object); (2) foreground object images; (3) composited images with harmonious interactions and consistent appearances; (4) text prompts describing the interaction type: (5) interaction regions; and (6) un- changed region masks to indicate unaffected background areas. As shown in Fig. 7 our IHOC dataset construction comprises four stages.
Stage 1: Collecting synthesized and real composited images. To ensure data diversity, we adopt the 1 17 human-object interaction categories from HICO-DET(22), comprising both real and synthetic samples. For real images, we manually selected 50 images per category, resulting in total of 5,850 from HICO-DET, excluding those that (1) contain multiple people, (2) lack clearly visible humans, or (3) lack clearly visible objects, which impair recognizability. The final set emphasizes diversity in

---

object type, scale, and human pose across scenes. For synthetic images. we use GPT-4o to generate 50 text prompts per category and synthesize 5,850 samples using FLUX.1 [dev] These images complement the real data by introducing broader variations in human appearance, pose, viewpoint, and visual style (e.g., cartoons, sketches). In total, we collect 11,700 composited images.
Stage 2:
Generating text prompts. For real images, we use GPT-4o to generate descriptive prompts. For synthetic images, we reuse the prompts originally used for generation.
Stage 3: Extracting foreground objects. We segment foreground objects from composited images using SAM [59]. To address occlusions caused by human-object interactions, GPT-4o infers and fills missing regions, producing complete and visually consistent objects.
Stage 4: Getting background images, unchanged region masks, and interaction regions. We manually annotate inpainting masks and use FLUX.1 FILL [dev] 4 to remove interacting objects and reconstruct plausible human poses without interactions. The inpainting masks define interaction- affected regions; their inverse yields the unchanged region masks. Interaction regions are computed by extracting the minimal bounding box of the interaction area within the unchanged region mask.
B.2 Dataset Statistics
As shown in Fig- [ our dataset consists six components: (1) background human images (without the object); (2) foreground object images; (3) composited images with harmonious interactions and consistent appearances; (4) unchanged region masks to indicate unaffected background areas; (5)
interaction regions and (6) text prompts describing the interaction type;
Composited Images
Background Images
Foreground Images
Unchanged Region Mask and Interaation Region
"A woman holding a bog.""
"A gärl eating a domut.""
"A man is standing suitcuve."
"A mon playing guitar..
girl is reading book.."
Text Prompt
Figure 8: Visualization of our Interaction- Human- -bijctt Composition (IHOC) Dataset.
Our dataset consists of 11,700 composited images, with half sourced from real-world data and the other half generated synthetically. Our dataset comprises total of 117 types of interaction types and 342 distinct foreground object categories. To highlight the diversity our dataset, we analyze its statistical properties across six dimensions, as illustrated in Fig. a--f)
Our dataset includes four distinct human viewpoints, categorized by body visibility and camera angle: full-body frontal, full-body side, upper-body frontal, and upper- body side (see Fig. a)). Upper-body frontal is the most common (42.4%), followed by full-body frontal (27.5%), upper-body side (15.7%), and full-body side (14.5%). This distribution is reasonable, as frontal views typically support wider range of interaction types and are more frequently used in practice.
(1) Human Viewpoint:
25

---

Human Viewpoint Distribution
Human Pose Distribution
8030
7315
4936
(61.750
7000
Number of Semples
Samples
6000
4300
321.3 (27.5%9
5000
1697 (14.5%)
(15.7%)
2020
130
1000
500 (4.1%)
110
(3 0.8%4
Standing
Siiting
Squatting
Sample distribution across human viewpoint
Sample distribution across human
Interaction Body Distribution
Foreground Object Distri Distribution
(54.356)
6000
5183 (44.3%%
6000
5020
Number of Samples
5000
Number of Sample
4000
3432
(29,24)
(26.5%)
3000
1752
2000
(15.0%)
1289 (11.0%)
1465 (12.5%)
B43
1000
(7.2%)
Hand//rm
Fooblleg
Torso
Headiface
Multiple Parts
Small:=10%)
Medium110% 30%)
Large-377)
c) Sample distribution across part involved the interaction process.
Sample distribution foreground object
Image Style Distribution
Background Type Distribution
4000
.369 (27.9%)
3411
7000
Num of Samples
6000
3000
2295 (10.6%)
(23
2722
2500
of
4000
2000
3000
1500
1000
1000 (8.5%)
1000
1000 (8.5%)
(8.5%)
(8.5%)
1000
Photoc realitiic
Carboon
Sketch like
Artatic
Digital Art
Smole Indcor
Complex Indoor
Simple Outdbor Complex Outdbor
Sample distribution image style
Sample distribution background type.
Figure 9: Statistical analysis our Interaction-ane aware Human- Human-Object Composition (IHOC) dataset
across six dimensions: (a) human viewpoint, (b) human pose, (c) interaction body part, (d) foreground object size, (e) image style, and (f) background scene type. These statisties demonstrate the dataset's diversity in visual appearance, interaction types, and contextual complexity.
(2) Human Pose: Our dataset covers five major categories of human pose: standing, sitting, lying, squatting, and other (e.g.. jumping on skateboard) (see Fig. (b)) Standing the most prevalent (61.7%), followed by sitting (21.3%) squatting (10.1 1%), lying (4.3%), and other (2.6%). This distribution demonstrates that our dataset includes both common and less frequent poses.
(3) Interaction Body Part: We categorize the interactions our dataset into five body regions based on which part of the body changes position before and after the interaction: hand/arm, foot/leg, torso, head/face, and multiple parts (see Fig. c)). Hand/arm interactions are the most dominant (54.3%), other interactions involve foot/leg (15.0%), multiple parts (12.5%), torso (11.0)), and head/face (7.2%). This distribution highlights the diversity interaction types and the involved body regions in dataset.
(4) Foreground Object Size: Our dataset includes foreground objects of varying sizes. Based on the ratio of foreground object area the entire image area, classify them into three categories: small
26

---

(<10%). medium (10-30%), and large (>30%) (see Fig. d)). Medium objects are the most common (44.3%), followed by small (29.2%) and large (26.5%). This distribution indicates that our dataset captures diverse range of object sizes, which is essential for evaluating interaction robustness across different foreground scales.
(5) Image Style:
Our dataset spans five distinct image styles: photo-realistic, cartoon, sketch-like, artistic, and digital (see Fig. e)). Photo-realistic images comprise the majority (65.8%), while the remaining styles each account for 8.5%. This diversity supports our method in handling images from different visual domains.
(6) Background Scene Type: Our dataset includes images with diverse background scenes, which use GPT-4o to judge the complexity of background scene: simple indoor, complex indoor, simple outdoor, and complex outdoor (see Fig. f)). The distribution is relatively balanced: complex indoor (29.25 (29.2%) simple indoor (27.9%), complex outdoor (23.3%), and simple outdoor (19.6%) ensuring broad coverage across varied scene contexts.
C Generalizability of HOComp.
To address the generalizability concern of our model, have evaluated HOComp under challenging and diverse conditions.
We first validated our method's performance under three extreme or special cases: (1) multi-subjer and multi-person compositions, (2) extremely large interaction regions, and (3) interactions involving completely occluded target objects.
Multi-subject and multi-person compositions:
We have tested HOComp on inputs with multiple persons and objects, focusing on cases where multiple persons interact with several objects simultaneously, Since HOIBench lacks multi-persons data, we have constructed a new benchmark with two scenarios: (1) two persons with two objects and (2) three persons with three objects.
Test Set Construction:
We have collected 40 diverse images (20 with two persons, with three persons) from the internet, and generated 117 Eeeeactiio-foregrouue image pairs following the same procedure of constructing HOlBench described Sec.
For Two persons, two objects: Each person in an image is randomly assigned a unique interac- tion-object pair, ensuring wide variety of combinations. For each of the 20 two-person images, we repeat this assignment process 10 times with different pairs, resulting in 200 two-person, two-object test cases.
For Three persons, three objects: Similarly, each person is assigned a unique interaction-objee pair, randomly sampled from the 117 pairs described above. This process is repeated for each seree-peroon image, resulting in 200 three-person, three-object cases.
We have evaluated our model using the new test set of 400 cases (200scennrio) and compared its performance with the top three methods from the main paper. We compute the HOI-Score and DINO-Score for each individual person-object interaction pair, and then take the average of all these scores across all interaction pairs. As shown in the following table, our approach achieves the best results in both scenarios and on all metrics.
Table 3: Quantitative results of our method on multi-subject and
compositions.
Category
Method
FID.
CLIP-Score
HOI-Score
DINO-Score
SSIM(BG)
9nyDoor9
22.38
26.86
23.12
55.46
83.19
OmniGen, 88
15.98
29.03
51.45
38.66
75.91
2 persons & objects
GPT 52
11.13
29.18
73.12
63.17
25.50
Ours
10.25
29.87
83.61
73.48
93.25
AnyDoor 9
24.17
26.03
22,98
52.07
80.11
OmniGen
18.25
28.14
49.01
35.85
72.14
3 persons & objects
GPT-4o 52
13.46
28.85
70.88
55.14
19.80
Ours
10.43
29.76
82.98
73.09
91.52
2. Extremely large interaction regions:
27

---

We have also evaluated HOComp's performance on cases with exceptionally large interaction regions, such those illustrated in Fig. (e.g.. "a man playing guitar," where the interaction covers over 70% of the image). We have identified 82 images in HOIBench where the interaction region occupies more than 70% of the total image area, and compared HOComp with previous methods on this subset. As shown in the following table, HOComp achieves substantially better results than other baselines across all metrics in large interaction region cases.
Table 4: Quantitative results of our method on extremely large interactions region cases.
Method
FID
HOI-Score" CLIP-Score
DINO-Score
SSIM(BG)
Any Door 9
21.80
26.93
22.96
44.62
85.78
OmniGen. 88
15.31
29.14
55,41
38.45
72.83
UniCom. 74
13.55
29.17
54.22
49.37
84.02
GPT--4o 52
10.03
29.12
74.01
63.39
32.91
Ours
9.44
30.26
85.11
77.48
95.43
Interactions involving completely occluded target objects:
Some actions may cause the interacting object to become entirely occluded in the resulting image, e.g.. food being eaten (fully inside the mouth) or an object blocked by car. To evaluate the model's generalization in such cases, we have curated 50 test instances where the target object is fully occluded after the interaction. We append "the object is occluded because of the human action" to the MLLM-geneeatee descriptions, forming dedicated evaluation subset.
Standard metrics requiring object visibility (e.g., DINO-Score, HOI-Score) inapplicable here. Instead, we manually assess the plausibility of the depicted interactions ("Occlusion Accuracy"), focusing on whether the generated images realistically reflect the intended occlusion. The results in the table below clearly demonstrate the superiority of our method over existing works.
Table 5: Quantitative results of our method on cases involving completely occluded target objects.
Method
FID
CLIP-Scoret
Occlusion Accuracyt
SSIM(BG)
AnyDoor[9
22.95
26.85
0.08
83.52
OmniGen. [88
15.83
29.07
0.42
71.89
UniCom. 74
15.58
29.06
0.28
83.62
GPT-4o 52
10.52
29.31
0.84
25.28
Ours
9.82
29.87
0.86
94.35
Extension the User Study: To further demonstrate the generalization of our method to diverse inputs, we have expanded the user study in the main paper from 10 to 50 input cases, covering broader range of human poses, viewpoints, and object categories. 45 participants have completed the evaluation. As shown below, our method consistently outperforms all baselines across all metrics, indicating strong user preferences for our approach.
Table 6: Extension user study results of our method.
Metries Any 9
pbЕ 92
FreeC [II]
FreeCu, [1
Prime [79
Omni. 88
GenArt. [80
UniC. 77
GPT-4o 52
Ours
IQ
4
8.28
7.64
9.66
9.16
3.14
2.49
4.07
3.29
1.24
IH L
8.65
8.90
8.35
5.85
6.15
5.01
5.34
2.91
2.69
1.15
AP
3.05
5.04
7.20
6.92
6.25
4.93
6.65
8.05
5.74
1.17
Our framework can also be adapted to different foundation models. In our original experiments, we used FLUX. [[dev] as the base DiT model. additionally evaluated HOComp with FLUX.1 Kontext[dev [ as the base model. As shown below, HOComp attains superior performance on FLUUX.1 Kontext[dev], further demonstrating the adaptability of our approach across base model variants.
28

---

Table 7: Results on different foundation models.
Base DiT Models
FID +
CLIP-Scoret
HOI-Score*
DINO-Scoret
SSIM (BG)1
[
9.27
30.29
87.39 89.21
78.21
96.57
FLUX.1 Kontext[dev][2
9.19
30.32
81.13
97.34
D
More Discussions on Limitations of Our Method
Our MRPG module adopts coarse-to-fine strategy for constraining human-object interactions. At the coarse level, it leverages MI.LMs to automatically identify suitable interaction types/regions via multi-stage querying. In some cases, MILLMs may fail to accurately predict the interaction regions. We provide a detailed analysis of these failure modes and potential solutions below.
In our HOIBench experiments, MLLMs (e.g., GPT-4o) identified reasonable interaction types in 100% of samples. The corresponding interaction region was predicted correctly in 91.33% cases (548/600). For the remaining 8.67% (52/600), two main failure modes are observed:
1. Mismatch Between Interaction Regions and Types (6% of cases): When multiple plausible interaction types exist, MILMs may misassign interaction regions. For example, if the object sunglasses and the interaction type "hold", the model may incorrectly assign the region around the eyes (i.e., "wear" action). While the generated image depicts person wearing sunglasses with harmonious interactions and consistent appearance, the predicted interaction type is inconsistent.
2. Incorrect Interaction Region Size (2.67% % of cases): In these cases, the predicted interac- tion region does fully cover the area, including the object and the human body part, for modification. As shown in Fig. 6 this can hinder generation of correct interactions.
To address these two issues, explored the following solutions:
Additional Input Conditions: To alleviate the mismatching issue, we can further incor- porate human pose priors (from pose estimator) GPT-4o, the region prediction accuracy increased from 91.33% to 96.5% This improve- ment can be attributed to the introduction of explicit keypoint information, which provides precise localization of body parts such as the face and hands.
additional prior MLLMs. For
Training with Noisy Data: To mitigate the impact of incorrect interaction region sizes, we introduce noisy data during training. We first generated 1,000 accurate input data samples. In accordance with the observed error rate, we then moved the bounding boxes of 700 samples to create mismatches with the interaction types (e.g., moving the bounding box of "soccer ball" with "kick" action cover the hand, which may correspond action). Additionally, we reduced the bounding boxes of 300 samples so that they no longer fully covered the interaction object or the required body movement. We fine-tuned the previously pretrained HOComp model for 3,000 steps using these noisy samples alongside the original IHOC dataset. After retraining, we re-evaluated our method using the 52 failure cases. As shown in the following table, training with noisy data substantially improved the model's performance even when MLLMs mispredicted the interaction region.
"hold"
Table Results of our method using different training strategies.
Training Strategy
FIDI
CLIP-Scoret
HOI-Scoret
DINO-Scoret
SSIM (BG)1
Without Noisy Data
11.58 10.03
29.55 29.82
38.62 77.89
70.02 74.29
62.01
With Noisy Data
88,93
E Effectiveness of Residual-based Modulation Strategy
As discussed in Sec. 3.3 of the main paper, our shape-aware attention modulation employs residual- based strategy to adjust the attention maps. This design is motivated by the concern that directly
29

---

modifying attention maps may degrade the visual quality of the generated images, suggested by previous work BT.
We define our modulation as:
A' A a- (Maaae (Amax - A) shaps) (A Amin))
where A is the original attention map. Mahape is the ground-truth shape mask, a is modulation strength, Amax and Amin denote the maximum and minimum attention values per query. The terms (Amax - A) and (A - Amin) serve as residuals, which helps constrain the modulation within the original attention range. This ensures that the updated attention map A' does not deviate excessively, thereby preserving the pretrained model's attention distribution. For comparison, we also evaluate naive modulation strategy without residual constraints, formulated as:
A' = A + a (Mahape - (1 (1- MMshape)
We conduct an ablation study on the HOIBench to compare the effectiveness of the residual-based strategy versus the non-residual version. As shown in Fig. [o] Table. removing the residual leads notable drop in FIID and DINO scores, indicating degraded image quality and reduced consistency of the generated foreground objects. Other metrics also show minor decreases. Visually, the generated shapes deviate more from the input guidance, confirming the importance of the residual design.
Table 9: Ablation study attention modulation strategies.
Modulation Strategy
FID
CLIP
HOI
DINO f SSIM(BG)
Non-residual Strategy
10.89
30.07
84.32
69.72 78.21
95.58
Residual Strategy
9.27
30.29
87.39
96.57
(a)Input image object
(b)Residual Strategy
Nonn--siiia Strategy
Figure 10: Visual results of ablation study on attention modulation strategies Table 9
F Effect of Coefficients
We evaluate the impact of four coefficients in the overall training loss and the shape-aware attention modulation HOlBench. Specifically, an, 02, and 03 are the coefficients of the pose-guided loss, background consistency loss, and multi-view appearance loss, respectively. a denotes modulation strength used in the shape-aware attention modulation.
As shown in Table. 10 O Increasing on from 1.5 (Rows 2) improves HOI score (87.39 88.01) and CLIP score (30.29 + 30.31), indicating better pose alignment. However, this comes at the cost of image quality and consistency, with FID increasing (9.27 10.65), and both DINO and SSIM(BG) decreasing (78.21 - 73.32, 96.57 + 94.33). @Raising da from 0.5 to 1.0 (Rows 1 vs. 3) improves SSIM(BG) (96.57 + 96.92), reflecting better background preservation, but significantly degrades other metrics including FID, CLIP, HOI, and DINO suggesting that excessive emphasis on background stability impairs semantic and visual coherence. @Inereasing 0g from 0.8 to (Rows 1 vs. 4) slightly improves DINO (78.21 - 78.58), indicating enhanced shape alignment, but at the cost of higher FID (12.92) and lower SSIM(BG) (94.88), showing trade-off between appearance consistency and image quality. OFinally, increasing modulation strength from 1.0 to 1.5 (Rows
30

---

1 vs. 5) causes moderate declines in FID (9.27 + 10.87), DINO (78.21 + 77.63). and (SIM (96.57 - 95.48), this effect may arise due the destabilization of the pretrained attention distribution caused by excessively aggressive attention modulation.
Table 10: Quantitative comparison of different coefficient combinations. at ag, and 0g are the coefficients the pose-guided loss, background consistency loss, and multi-view appearance loss, respectively. denotes the modulation strength used in the shape-aware attention modulation.
Coefficients (411, a2, d3s (x)
FID
CLIP
HOI 1
DINO
SSIM(BG)
0,= 0.50.5 03 a=1
9.27
30.29
87.39
78.21
96.57
a a,=1.5, 0g= a=1
10.65
30.31
88.01
73.32
94.33
6 11=1, ag= a=1
11.29
29.88
82.16
74.10
96.92
a,=1, 025 13 03-1, a=1
12.92
29.71
85.75
78.58
94.88
a==1, 0250.5 0.8..8. a=1.5
10.87
30.25
86.11
77.63
95.48
G Extended Details on Using MLLMs to Identify Interaction Types and Regions
In Sec. 3.2 of the main paper, we briefly described the use of MLLMs infer interaction types and interaction regions via multi-turn querying. Here, we detail the full process.
Given background human image I and foreground object image 1,. we iteratively use MLLM to extract: (1) a text prompt describing the interaction, (2) the object bounding box B., and (3) the interaction region the human By. The multi-tuny procedure proceeds as follows:
1. Interaction Prompt Generation. The MILM is queried with I and In using the instruction: "Please analyze and describe suitable type of interaction between them and generate a simple prompt for this interaction." The model outputs text prompt C? describing the interaction type.
Object Box Prediction. Using 1,, T, and C, we query the MILM with: "Please describe the position of the foreground object and give bounding box coordinates that it aligns with the specified interaction." " The model returns the object bounding box B.-
Interaction Region Prediction. Given 1,, Ib. C. and Bo, we ask: "Based on the images and interaction prompt, and assuming the object is at B,, identify the regions on the person that would be affected during the interaction and return their bounding box." " The MLLM then predicts the interaction region box B,,
H Additional Ablation studies
H.1 Multi-View Generators and View Numbers
We evaluate the impact of the number of views used in the multi-viww appearance loss (Fig. Table. (let)). Using only single view leads noticeable inconsistencies object appearance. As the number of views increases, performance improves steadily across all metrics, confirming the value of richer multi-vier supervision.
We further evaluate different multi-view generation methods (Fig. [2 Table. (right). Without multi- view supervision, the model fails to maintain appearance consistency under significant viewpoint changes. Incorporating multiple generated views into the CLIP loss enhances coherence across varying poses and backgrounds. Among the methods, Zero 23+(55) achieves the best results, while SV3D(73) and View (21) also outperform the no multi-view baseline, underscoring the importance of high-fidelity multi-view supervision. H.2 LoRA Ranks
Table 12: Ablation study LoRA Ranks
Table 12 presents the results of varying the LoRA rank (8, 16, 32, 64) across five evaluation metrics. Rank 16 consistently achieves the best overall performance, yielding the lowest FID (9.27) and the
5.51
24.95
14.12
74.12
16.12
75.31
16.37
77.12
W::1
31

---

Inpet image object
Views
Figure 11: Visual results ablation study on view numbers used in multi-view appearance loss.
Input image & objoct
No Multi-view
SV3D
ViewDiff H
Zero123+ e
Figure 12: Visual results of ablation study on multi-view generators.
highest scores CLIP (30.29), HOI (87.39), DINO (78.21), and SSIM(BG) (96.57). When the rank is too low (e.g.. 8), the model underperforms across all metrics, indicating insufficient capacity to model human-object interactions and maintain consistent appearances. However, higher ranks (32, 64) yield marginal no improvements (e.g., DINO drops to 77.26 and 77.12), suggesting possible overfitting.
H.3 ID Encoder Backbone
As discussed in Sec. 3.3 of the main paper, we adopt DINOv2 as the backbone for extracting object identity features. Here, we conduct an ablation study comparing different backbones: VAE [5]], CLIP 162), and DINOv2 [43]. To ensure fair evaluation, we additionally report CLIP-I [58), which measures the CLIP similarity between the synthesized foreground object and the input foreground object.
As shown in Table. 13 DINOv2 consistently outperforms other ID encoder backbones across all evaluated metrics. As shown Fig. 13 using DINOv2 as the ID encoder backbone yields the most consistent foreground object.
Input image & object
VAB 32
CLIP
DINOv2 53
Figure 13: Ablation study on different backbones for foreground ID encoders.
H.4
Guidance Scale
To study the impact the guidance scale on our model, we evaluate performance under six different inference- time guidance scales: 1, 2, 3.5, 4, and 5.
As shown in Table, 14 and Fig. 14 guidance scale 3.5 achieves the best overall performance (FID 9.27, CLIP 30.29, HOI 87.39, DINO 78.21, SBIM(BG) = 96.57). Correspondingly. the visual results at this setting exhibit the most faithful preser- vation of the foreground object's appearance. In con- trast, lower guidance scales (gs 1.0 or 2.0) lead to diminished semantic alignment, particularly evident
Guidonce Scale
FID
CLIP 1
HOI
DINO
SSIM:BG)
g 1.0
10.11
20.42
30.01
62.33
95.25
9.48
- 3.5
9.27
30.29
37.39
78.21
96.57
p4-40
9.99
36.19
83.91
77.56
95.89
Table 14: Performance of our model under dif- ferent guidance scales during inference. The model is trained with guidance scale of 1.
32

---

Table 11: Ablation on different numbers of views (left) and multi-view generators (right).
Views
FID
HO1
DINO
SSIM(R))
Method
FID
CLIP
HOI
DINO
SSIM(BG)
I(No multi-view)
11.55
29.52
K1.32
68.83
95.83
29.55
No multi-view
11.55
29.52
81.32
68.83
95.83
10.19
29.81
85.08
70.26
95.87
Zero123+
9.27
30.29
87.39
78.21
96.57
SV3D
9.89
29.85
$1.98
75.26
96.01
5
9.29
30.23
$6.07
74.19
96.21
ViewDir 20
10.30
29.99
86.19
74.63
95.98
Table 13: Ablation study on different ID encoder backbones
Backbone
FID
CLIP
HOL 1
DINO 1
CLIP-11
SSIM/BG)
51
29.72
CLIP
62
30.17
85,24
75.72
87.79
96.53
DINO
43
9.27
30.29
87.39
78.21
90.25
96.57
in the foreground regions, as reflected by lower DINO
scores. Increasing the scale beyond 3.5 (e.g. gs = 4.0 or 5.0) results in subtle declines in both quantitative scores and foreground object consistency.
Iaput human and objoct
1.0
2.0
g 3.0
gs 3.5
4.0
5.0
Figure 14: Ablation study on different guidance scales (denoted as gs) during inference.
I Comparison with Multi-Modality Models
We compare our method with recent state-of-the-ar multi-modality models, including GPT-4o [52] Gr6k3[86], and MidJourney V7 [50]_ All models receive identical inputs: foreground object, background human image, a designated interaction region, and a corresponding text prompt.
Qualitative results reveal clear limitations in existing models. GPT-4o and MidJourney V7 frequently fail to generate consistent foreground objects (e.g., Row 2(b), Rows 2-3(d) in Fig. 15p. Grok3 and Midlourney V7 struggle to preserve the background human and scene details (Rows 1-3(cdd)). In addition, GPT-4o may struggle to accurately model interactions under complex scenarios (see Row 1(b)).
Quantitatively, our method outperforms all baselines across five key metries, achieves the lowest FID (9.27), highest CLIP score (30.29), HOI score (87.39), DINO score (78.21) and SSIM(BG) score (96.57). This demonstrate that our method delivers more harmonious human-object interactions and consistent appearances.
J Additional Comparison with Image Composition Methods
In addition to the nine methods compared in the main paper, we conducted further comparisons with five additional state-of-the-art image composition methods: DreamFuse [26), InsertAnything (66).
MimicBrush (8). Bifrost (36] and DreamRelation (63]. For fairness, all methods with publicly available training code were retrained or fine-tuned on our dataset.
Fig. 17 shows qualitative comparisons. DreamFuse and InsertAnything generate visually faithful foreground objects, but often fail to model realistic human-object interactions (see Rows 2-4 in -ig()7 DreamRelation produces interaction-like gestures, yet struggles to preserve the visual consistency of the foreground object and background human (Rows 1-4 Fig I7)). MinicBrush and Bifrost, on the other hand, produce neither convincing interactions nor accurate object appear- ances (Fig. d-e).. In contrast, our method generates diverse and harmonious interactions while maintaining the consistent appearance of both the foreground and the background.
33

---

Table 15: Qualitative comparison with recent state-of-the-ar art molti-modality models.
Method
FID4
CLIP
HOH
DINO
SSIM(BG)1
Grok3
86
13.27
29.07
65.03
57.02
58.25
GPT-4o
52
9.98
29.35
75.22
65.23
47.22
MidJourney V7 [50]
10.85
29.87
73.45
60.18
41.34
Ours
9.27
30.29
87.39
78.21
96.57
(a)Input image & object
(b)GPT-44 52
(c)Grok3 [86
(d)MidJoumey V7
(e)Ours
Quantitative comparison with recent state-of-the-art multi-modality models. The prompts for the above three cases are: "A woman is riding horse", "A girl is holding stack of books", "A model is presenting a skincare bottle".
Figure 15:
Table, provides quantitative results. Our method achieves the best FID (9.27), CLIP-Score (30.29), HOI-Score (87.39), and DINO-Score (78.21), indicating superior image quality, semantic alignment, interaction quality and appearance consistency. User study results further validate our approach, ranking it highest in image quality (IQ), interaction harmonization (IH), and appearance preservation (AP), with all scores significantly outperforming other methods.
K
Additional Results of HOComp
Fig. shows additional qualitative results of our method, Each example includes: (1) Top: the final composited image, (2) Bottom: the input background human and foreground object. These results demonstrate that our method produces natural and plausible human-object interactions while maintaining visual consistency of both the foreground object and the background human.
L Ethical Considerations
Human Subjects and Informed Consent
Our institution currently does not maintain a formal Institutional Review Board (IRB) or equivalent ethics committee. To ensure compliance with international ethical norms, we conducted an internal review modeled on IRB standards and assessed the study along four key dimensions: privacy, informed consent, participant protection, and data security.
34

---

(n)input image object
(bjDreamFuu (e)
Md)Mi X
(e)Rifroat (DDreamRe
OerOurs
Figure 16: Additional qualitative comparisons our HOComp with 5 SOTA methods. The prompts for the above four examples are: "A boy is holding mickey mouse toy", "A girl is showing perfume bottle", "A woman is lifting a bag", and "A sitting man is holding a balloon".
Table 16: Additional quantitative comparison of method with 5 SOTA methods. The best and second-bes best results are highlighted bold and underline, respectively. Training or tuning-bast based methods without released training codes are marked with a
Category Metrics
DreamFuse' 26
InsertAnything
66
MimicBrush
Bifrost 36
DrcamRelation
63
Ours
FID J.
13.35
10.72
15.88
16.21
15.85
9.27
CLIP-Score
29.53
29.76
28.62 78.62
28.
28.55
30.29
Automatic
HOI-Score
63.75
36.04
38.98
52.66
DINO Score f
44.89
37.07
SSIM(BG)
93.23
92.19
84.56
88.11
25.19
96.57
1Q4
3.10
3.85
User study
IH
2.28
2.43
6.00
5:95
3.27
1.07
AP
2.89
2.43
4.33
4.44
5.90
1.01
The user study was conducted via anonymous online questionnaires, in which participants were asked only rank composite images by visual quality and realism. No identifiable demographic data were collected. All participants were adult volunteers who provided explicit informed consent before participation and were compensated with an equivalent of US$8 for approximately 15 minutes of participation-exceedin the local minimum wage. Participation was voluntary, and participants could withdraw at any time without penalty. All data were stored on secure, password-protecter institutional servers and will be permanently deleted after completion of the analysis.
Data Licensing and Source Transparency
Our data usage strictly adheres to the licenses and terms of all sources involved:
The HICO-DET(S) dataset is distributed under the MIT License, permitting unrestricted research and redistribution use.
The HOIBench dataset is constructed using images from Internet, whose licenses allow editing, derivative works, and non-restrictive reuse of -nngeuinnncluinn those depicting people-without requiring additional permissions.
We employ these resources solely for academic research. To safeguard privacy, any external user wishing to access our dataset must sign agreement restricting use to non-commercial research and explicitly prohibiting any misuse, defamation, or violation of depicted individuals' rights. We will also publish clear data-use policy ensure transparency and traceability of all derivative works.
35

---

Privacy Protection and Consent Risks
Because the datasets include human figures, we recognize potential privacy and consent issues, especially regarding secondary use of images beyond their original intent. To mitigate such risks, we (1) use all images under licenses that allow derivative use, (2) avoid any manual modification that changes the identity or context of individuals in an offensive or misleading way, and (3) provide clear disclaimers our documentation that prohibit using HOComp for identity-related, defamatory, or deceptive content.
Misuse Risks and Responsible Deployment
We acknowledge that HOComp, like other generative models, carries potential misuse risks such as creating deceptive content or inappropriate composites involving real individuals. To address these risks, we adopt the following safeguards:
Embedding invisible watermarks or provenance metadata in generated images to support detection and accountability.
Releasing the model and code only for academic research under license that explicitly prohibits deceptive, defamatory, privacy-violating applications.
Implementing content filters to prevent generation sensitive or harmful scenarios (e.g.. involving weapons, religion, polities).
Including explicit user guidelines and documentation that highlight responsible use principles and ethical restrictions.
Environmental Considerations
Our method fine-tunes existing pre-trained models rather than training from scratch, substantially reducing energy consumption and the carbon footprint associated with large-scale model training. We estimate our total GPU usage to be less than 15% of that required for comparable baseline models trained from scratch.
Commitment to Responsible Al Research
We are committed to transparency, accountability, and the responsible use of generative technologies. Our future work will continue emphasize ethical risk assessment, dataset documentation, and open research practices consistent with the broader goals of trustworthy and socially beneficial AI development.
36

---

Figure 17: Additional qualitative results of HOComp. Each example includes: (1) Top: the final composited image, (2) Bottom: the input background human and foreground object.
37


Reviews:
[{"review_id": "lHM675K4zb", "reviewer": "Reviewer_dYr4", "summary": "This paper present HOComp, a framework for interaction-aware human-object composition. It leverages MLLM driven region-based pose guidance (MRPG) for constrained human-object interaction, and detail-consistent appearance preservation (DCAP) for maintaining appearance consistency. This paper also introduced the Interaction-aware Human-Object Composition (IHOC) dataset. Extensive experiments demonstrate that HOComp outperforms existing methods in quantitative, qualitative, and subjective evaluations.", "questions": "1.Ordinary and unremarkable writing\n2.The novelty level is not too high", "limitations": "1.hope have more bad cases analysis", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper propose a new approach for interaction-aware human-object composition, named HOComp, which focuses on seamlessly integrating a foreground object onto a human-centric  background image while ensuring harmonious interactions and preserving the visual consistency of both the foreground object and the background person.\n2. HOComp incorporates two innovative designs: MLLMs-driven region-based pose guidance (MRPG) for constraining human-object interaction via a coarse-to-fine strategy,  and detail-consistent appearance preservation (DCAP) for maintaining consistent foreground/background appearances.\n3.This paper introduce the Interaction-aware Human-Object Composition (IHOC) dataset which maybe can helpful for others research.\nWeaknesses:\n1.Ordinary and unremarkable writing\n2.The novelty level is not too high\n3.Too few bad cases", "was_revised": true, "final_justification": "I appreciate the author's rebuttal which addressed some of my concerns, so I raise my original score to 4 from 3.", "rebuttal": "We thank this reviewer for valuable comments. We appreciate the recognition of our IHOC dataset. We address the main concerns and questions below. (Abbreviations used: Q/W/L refer to Question/Weakness/Limitation.)\n\n### Q1&W1: ''Ordinary and unremarkable writing''\n\nWe would like to briefly summarize the logical structure of our paper:\n\n- **Background and Motivation:** We start with the practical need for compositing product images with humans in advertising scenarios (e.g., perfume ads), highlighting two essential objectives: (a) achieving natural and plausible human-object interactions, and (b) maintaining strict appearance consistency for both the person and the object.\n- **Task Definition:** We define the problem as *interaction-aware human-object composition* and explicitly state the main challenges: existing approaches often fail to deliver either natural interactions (such as human pose and gesture) or visual consistency (identity and appearance preservation).\n- **Technical Contributions:** We introduce **HOComp** to address these challenges, proposing two novel modules:\n    - **MLLMs-driven region-based pose guidance (MRPG):** A coarse-to-fine mechanism for constraining and guiding human-object interactions.\n    - **Detail-consistent appearance preservation (DCAP):** A method to preserve both the details and identities of the foreground object and the background person.\n- **Dataset Contribution:** To enable training, we introduce the **IHOC** dataset, the first large-scale dataset specifically designed for interaction-aware human-object composition.\n- **Experiments:** We conduct comprehensive experiments on HOIBench. Results demonstrate the advantages of HOComp in generating realistic interactions and visually consistent human-object compositions.\n\nWe would be happy to refine our presentation if this reviewer can provide more specific information of which part (e.g., section/discussion) that we may improve on.\n\n---\n\n### Q2&W2: ''The novelty level is not too high''\n\nOur work has three main categories of contribution.\n\n**Idea Contribution.** In this paper, we introduce **HOComp**, the first **interaction-aware human-object composition** approach to seamlessly integrate a foreground object into a human-centric background while ensuring **harmonious interactions** and maintaining **consistent appearance** between the foreground object and the background person. Our method can be applied in areas such as automated advertising generation and digital content creation.\n\nCompared to existing methods, HOComp addresses the following shortcomings:  \n(1) **Failure in natural human-object interactions:** Many image-guided composition methods fail to generate realistic human gestures/poses during human-object interaction, resulting in unnatural outcomes.  \n(2) **Inconsistent appearance of humans or objects:** Existing multi-concept customization or HOI-based image generation methods regenerate both the foreground object and the background human, leading to inconsistencies in the background human's appearance.\n\n**Technical Contributions.** **HOComp** introduces two key technical contributions:\n1. *MLLMs-driven Region-based Pose Guidance (MRPG),* which utilizes MLLMs to identify the interaction region and interaction type (e.g., holding and lifting) to provide coarse-to-fine constraints to the generated pose for the interaction while incorporating human pose landmarks to track action variations and enforcing fine-grained pose constraints.\n2. *Detail-consistent Appearance Preservation (DCAP),* which unifies a shape-aware attention modulation mechanism, a multi-view appearance loss, and a background consistency loss to ensure consistent shapes/textures of the foreground and faithful reproduction of the background human.\n\n**Dataset/Benchmark Contributions.**\n1. To facilitate research in interaction-aware human-object composition, we introduce the **IHOC dataset**, specifically designed for this task. It is the first dataset to contain paired pre-/post-interaction data, which is crucial for modeling realistic and coherent human-object compositions.\n2. We also present **HOIBench**, a benchmark for comprehensive evaluation of human-object composition methods.\n\nIf this reviewer can provide more specific information regarding this novelty issue, we would be happy to further discuss.\n\n---\n\n### W3&L1: ''Too few Bad Cases'' and More Analysis\n\nWe appreciate the reviewer’s suggestion to provide more failure cases and more analysis.\n\nOur MRPG module adopts a coarse-to-fine strategy for constraining human-object interactions. At the coarse level, it leverages MLLMs to automatically identify suitable interaction types/regions via multi-stage querying. In some rare cases, MLLMs may fail to accurately predict the interaction regions. We provide a detailed analysis of these failure modes and potential solutions below.\n\nIn our HOIBench experiments, MLLMs (e.g., GPT-4o) identified reasonable interaction types in **100%** of samples. The corresponding interaction region was predicted correctly in **91.33%** of cases (548/600). For the remaining **8.67%** (52/600), two main failure modes are observed:\n\n1. **Mismatch Between Interaction Regions and Types (6% of cases):** When multiple plausible interaction types exist, MLLMs may misassign interaction regions. For example, if the object is sunglasses and the interaction type is ''hold'', the model may incorrectly assign the region around the eyes (i.e., ''wear'' action). While the generated image depicts a person wearing sunglasses with harmonious interactions and consistent appearance, the predicted interaction type is inconsistent.\n\n2. **Incorrect Interaction Region Size (2.67% of cases):** In these cases, the predicted interaction region does not fully cover the area, including the object and the human body part, for modification. As shown in Fig.6 (main paper), this can hinder generation of correct interactions.\n\nTo address these two issues, we explored the following solutions:\n\n- **Additional Input Conditions:** To alleviate the mismatching issue, we can further incorporate human pose priors (from a pose estimator) as an additional prior to MLLMs.  \nFor GPT-4o, the region prediction accuracy increased from **91.33%** to **96.5%**.  This improvement can be attributed to the introduction of explicit keypoint information, which provides precise localization of body parts such as the face and hands.\n- **Training with Noisy Data:** To mitigate the impact of incorrect interaction region sizes, we introduce noisy data during training.  \nWe first generated 1,000 accurate input data samples. In accordance with the observed error rate, we then moved the bounding boxes of 700 samples to create mismatches with the interaction types (e.g., moving the bounding box of a ''soccer ball'' with a ''kick'' action to cover the hand, which may correspond to a ''hold'' action). Additionally, we reduced the bounding boxes of 300 samples so that they no longer fully covered the interaction object or the required body movement. We fine-tuned the previously pretrained HOComp model for 3,000 steps using these noisy samples alongside the original IHOC dataset. After retraining, we re-evaluated our method using the 52 failure cases. As shown in the following tables, training with noisy data substantially improved the model's performance even when MLLMs mispredicted the interaction region.\n\n| Training Strategy      | FID↓ | CLIP-Score↑ | HOI-Score↑ | DINO-Score↑ | SSIM (BG)↑ |\n|-----------------------|-----------------|----------------------|---------------------|----------------------|---------------------|\n| Without Noisy Data    | 11.58           | 29.55                | 38.62               | 70.02                | 62.01               |\n| With Noisy Data       | **10.03**       | **29.82**            | **77.89**           | **74.29**            | **88.93**           |\n\nQualitative results and further experimental details will be included in the supplemental materials."}, {"review_id": "uUCXp3y0cD", "reviewer": "Reviewer_gQQw", "summary": "This paper introduces HOComp for interaction-aware human-object composition. Existing methods often fail to generate plausible human poses or preserve appearance. HOComp addresses this with two key components. First, MLLM-driven Region-based Pose Guidance (MRPG) automatically determines the interaction type and region to guide pose generation. Second, Detail-Consistent Appearance Preservation (DCAP) uses multiple losses and attention modulation to maintain foreground and background consistency. The authors also introduce the IHOC dataset for this task. Experiments show their method significantly outperforms state-of-the-art approaches in generating harmonious and realistic compositions.", "questions": "1. Regarding the MRPG module's reliance on GPT-4o: You are commendably transparent about the 91.33% success rate. However, a ~9% failure rate at the first step is a significant practical limitation. Could you provide a more detailed analysis of these failure modes? For example, what are the common types of errors (e.g., incorrect body part, illogical interaction type), and how does the rest of the pipeline behave when it receives this incorrect guidance?\n\n2. Regarding the dataset construction: The method of creating paired data by using an inpainting model to generate the \"before interaction\" image raises a critical question about potential model bias. The model might be learning to invert the specific process of the inpainting tool rather than a general interaction generation skill.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- Strengths\n\n1. The proposed MRPG module is a novel use of MLLMs to automate the generation of high-level interaction semantics (type and region), simplifying the user input process.\n\n2. The creation of the IHOC dataset and the HOIBench benchmark is a significant contribution to the community, providing valuable resources for future research in this area.\n\n3. The quantitative and qualitative results, including a user study, are extensive and demonstrate a clear and substantial improvement over numerous state-of-the-art methods.\n\n- Weaknesses\n\n1. The framework's performance heavily relies on an MLLM for initial guidance. While the authors report a high success rate, the system is still susceptible to failures from this non-differentiable module, which can lead to incorrect or nonsensical interactions.\n\n2. The system architecture is highly complex, integrating multiple models (DiT, MLLM, pose estimator, multi-view generator) and a carefully balanced set of loss functions. This complexity may pose challenges for reproducibility, training stability, and tuning.\n\n3. The IHOC training dataset is partially constructed using other generative models for inpainting backgrounds and completing occluded objects. This introduces a potential data bias, where the model may be implicitly learning to reverse the artifacts of the data-generation models rather than learning a more general composition skill.", "was_revised": true, "final_justification": "I appreciate the author's rebuttal which addressed some of my concerns, so I maintain my original score.", "rebuttal": "We thank this reviewer for the positive review and insightful questions. We appreciate the recognition of the novelty of our MRPG module, the value of the IHOC dataset and HOIBench benchmark, and the appreciation of our experimental results. We address each of the questions below. (Abbreviations used: Q/W refer to Question/Weakness.)\n\n---\n\n## Q1&W1: Concerns Regarding MRPG's Reliance on MLLMs' Predictions\n\nTo address this concern, we provide a detailed analysis of the failure modes and potential solutions.\n\nIn our HOIBench experiments, MLLMs (e.g., GPT-4o) identified reasonable interaction types in **100%** of samples. The corresponding interaction region was predicted correctly in **91.33%** of cases (548/600). For the remaining **8.67%** (52/600), two main failure modes are observed:\n\n1. **Mismatch Between Interaction Regions and Types (6% of cases):**  \n   When multiple plausible interaction types exist, MLLMs may misassign interaction regions. For example, if the object is sunglasses and the interaction type is \"hold\", the model may incorrectly assign the region around the eyes (i.e., \"wear\" action). While the generated image depicts a person wearing sunglasses with harmonious interactions and consistent appearance, the predicted interaction type is inconsistent.\n\n2. **Incorrect Interaction Region Size (2.67% of cases):**  \n   In these cases, the predicted interaction region does not fully cover the area, including the object and the human body part, for modification. As shown in Fig.6 (main paper), this can hinder generation of correct interactions.\n\nWe have explored the following potential solutions to mitigate these two issues:\n\n- **Additional Input Conditions:**  \nTo alleviate the mismatching issue, we can further incorporate human pose priors (from a pose estimator) as an additional prior to MLLMs. \n\n  For GPT-4o, the region prediction accuracy increased from **91.33%** to **96.5%**. This improvement can be attributed to the introduction of explicit keypoint information, which provides precise localization of body parts such as the face and hands.\n\n- **Training with Noisy Data:**  \n  To mitigate the impact of incorrect interaction region sizes, we introduce noisy data during training. \n\n  We first generated 1,000 accurate input data samples. In accordance with the observed error rate, we then moved the bounding boxes of 700 samples to create mismatches with the interaction types (e.g., moving the bounding box of a \"soccer ball\" with a \"kick\" action to cover the hand, which may correspond to a \"hold\" action). Additionally, we reduced the bounding boxes of 300 samples so that they no longer fully covered the interaction object or the required body movement. We fine-tuned the previously pretrained HOComp model for 3,000 steps using these noisy samples alongside the original IHOC dataset. After retraining, we re-evaluated our method using the 52 failure cases. As shown in the following table, training with noisy data substantially improved the model's performance even when MLLMs mispredicted the interaction region.\n\n| **Training Strategy**  | FID↓  | CLIP-Score↑ | HOI-Score↑ | DINO-Score↑ | SSIM (BG)↑ |\n|-----------------------|-------|-------------|------------|-------------|------------|\n| Without Noisy Data    | 11.58 | 29.55       | 38.62      | 70.02       | 62.01      |\n| With Noisy Data       | **10.03** | **29.82**   | **77.89**  | **74.29**   | **88.93**  |\n\nQualitative results and further experimental details will be included in the Supplemental.\n\n---\n\n## W2: Concerns on Framework Complexity and Reproducibility\n\nAlthough our framework comprises multiple components, each component of HOComp fulfills a distinct, essential function: Our framework builds on state-of-the-art diffusion transformers, introduces MLLMs for coarse-level interaction constraints, and reduces user input requirements. The pose estimator is incorporated during training to optimize the generated human poses, enabling more harmonious interactions. The multi-view generator ensures appearance consistency and semantic alignment across viewpoints.\n\nWe will release all codes, the IHOC dataset, and HOIBench upon acceptance, to facilitate reproducibility.\n\n---\n\n## Q2&W3: Potential Data Bias in the IHOC Dataset\n\nWe understand and appreciate the reviewer’s concern about potential bias introduced by the use of inpainting models to generate pre-interaction images in IHOC dataset.\n\nFirst, our IHOC dataset is a mixture of real-world data collected from the internet and AIGC-generated content. During training, inpainting is employed to generate pre-interaction images, similar to previous methods [1,2]. In contrast, the ground-truth post-interaction images depict natural human-object interactions sourced from both real-world images and AIGC-generated interaction images.\n\nSecond, the vast majority of input images used in our evaluation are real-world photographs, not inpainted data (e.g., rows 1–3 in Fig.1 (main paper), rows 2–3 in Fig.4 (main paper), and rows 1–3 in Fig.9 (supplementary)). The strong performance of our model on these real images demonstrates that it does not rely on or reproduce inpainting-specific biases.\n\nFinally, to further alleviate this concern, we will provide additional results on more real-world test cases, and in future work, we plan to expand our dataset by extracting pre- and post-interaction pairs directly from video footage to ensure even greater realism and diversity.\n\n---\n\n### References\n\n1. Learning Object Placement by Inpainting for Compositional Data Augmentation. ECCV'20.  \n2. ObjectMate: A Recurrence Prior for Object Insertion and Subject‑Driven Generation, ICCV'25."}, {"review_id": "CvCM9z1Tkg", "reviewer": "Reviewer_2Em8", "summary": "1. HOComp present to integrate a foreground object onto a human-centric background image seamlessly while ensuring harmonious interactions and preserving the visual consistency of both the foreground object and the background person.\n\n2. MRPG utilizes MLLMs to identify the interaction region and interaction type to provide coarse-to-fine constraints, while incorporating human pose landmarks to track action variations and enforcing fine-grained pose constraints. DCAP unifies a shape-aware attention modulation mechanism, a multi-view appearance loss, and a background consistency loss to ensure consistency.\n\n3. This paper introduces a Interaction-aware Human-Object Composition (IHOC) dataset to facilitate HOC task", "questions": "The entire framework includes multiple complex modules to ensure the synthesis of seamless interaction-aware compositions. Does the author team plan to open-source the project and the proposed dataset to promote community development?", "limitations": "yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n\nThis paper proposes an effective image editing model for compositing a foreground object onto a human-centric background image. It includes two modules: MLLMs-driven Region-based Pose Guidance to identify the interaction and Detail-Consistent Appearance Preservation to ensure consistency. The paper is well-organized and easy to follow. They also reports sufficient qualitative and quantitative results to support their claims.\n\nWeaknesses\n\nThe entire process is quite complex, not only involving multiple processing units to encode the features of the foreground and background, but also employing several loss functions to enforce consistency. I am not very sure about the generalizability of HOComp and the reproducibility of the overall architecture.", "was_revised": true, "final_justification": "I would like to express my gratitude to the author for their rebuttal, which effectively addressed the concerns I had raised.", "rebuttal": "We thank this reviewer for the positive review and constructive comments. We appreciate the recognition of paper organization and clarity, as well as the sufficiency of our qualitative/quantitative results. We address each of the questions below. (Abbreviations used: Q/W refer to Question/Weakness.)\n\n---\n\n### Q&W: Concerns about the Complexity of the HOComp Framework.\n\nAlthough our framework contains multiple components, each component of HOComp fulfills a distinct, essential function: the framework builds on latest diffusion transformers, integrates MLLMs to enforce coarse-level interaction constraints and reduce user input, incorporates a pose estimator during training to optimize human pose generation for harmonious interactions, and employs a multi-view generator to ensure appearance consistency and semantic alignment across viewpoints.\n\n---\n\n### Q&W: Generalizability of HOComp.\n\nTo address the generalizability concern of our model, we have evaluated HOComp under challenging and diverse conditions.\n\nWe first validated our method's performance under three extreme or special cases:  \n(1) multi-subject and multi-person compositions,  \n(2) extremely large interaction regions, and  \n(3) interactions involving completely occluded target objects.\n\n---\n\n#### 1. Multi-subject and multi-person compositions:\n\nTo evaluate our model’s ability to handle multi-object and multi-person compositions, we have constructed a dedicated test set, since the current HOIBench does not include such cases. Our new benchmark covers two scenarios: (1) two persons with two objects and (2) three persons with three objects.\n\n**Test Set Construction:**  \nWe have collected 40 diverse images (20 with two persons, 20 with three persons) from the internet, and generated 117 interaction–foreground image pairs following the same procedure of constructing HOIBench described in Section 4 (main paper).\n\nFor **Two persons, two objects:** Each person in an image is randomly assigned a unique interaction–object pair, ensuring a wide variety of combinations. For each of the 20 two-person images, we repeat this assignment process 10 times with different pairs, resulting in 200 two-person, two-object test cases.\n\nFor **Three persons, three objects:** Similarly, each person is assigned a unique interaction–object pair, randomly sampled from the 117 pairs described above. This process is repeated for each three-person image, resulting in 200 three-person, three-object cases.\n\nWe have evaluated our model using the new test set of 400 cases (200/scenario) and compared its performance with the top three methods from the main paper. We compute the HOI-Score and DINO-Score for each individual person-object interaction pair, and then take the average of all these scores across all interaction pairs. As shown in the following table, our approach achieves the best results in both scenarios and on all metrics.\n\n\n| Category               | Method     | FID↓   | CLIP-Score↑ | HOI-Score↑ | DINO-Score↑ | SSIM(BG)↑ |\n|------------------------|-----------|--------|-------------|------------|-------------|-----------|\n| 2 persons & 2 objects  | AnyDoor   | 22.38  | 26.86       | 23.12      | 55.46       | 83.19     |\n|                        | OmniGen.  | 15.98  | 29.03       | 51.45      | 38.66       | 75.91     |\n|                        | GPT-4o    | 11.13  | 29.18       | 73.12      | 63.17       | 25.50     |\n|                        | **Ours**  | **10.25** | **29.87** | **83.61**  | **73.48**   | **93.25** |\n| 3 persons & 3 objects  | AnyDoor   | 24.17  | 26.03       | 22.98      | 52.07       | 80.11     |\n|                        | OmniGen.  | 18.25  | 28.14       | 49.01      | 35.85       | 72.14     |\n|                        | GPT-4o    | 13.46  | 28.85       | 70.88      | 55.14       | 19.80     |\n|                        | **Ours**  | **10.43** | **29.76** | **82.98**  | **73.09**   | **91.52** |\n\n---\n\n#### 2. Extremely large interaction regions:\n\nWe have also evaluated HOComp's performance on cases with exceptionally large interaction regions, such as those illustrated in Fig.2 (Supplementary) (e.g., \"a man is playing a guitar,\" where the interaction covers over 70% of the image). We have identified 82 images in HOIBench where the interaction region occupies more than 70% of the total image area, and compared HOComp with previous methods on this subset. As shown in the following table, HOComp achieves substantially better results than other baselines across all metrics in large interaction region cases.\n\n\n| Method    | FID↓   | CLIP-Score↑ | HOI-Score↑ | DINO-Score↑ | SSIM(BG)↑ |\n|-----------|--------|-------------|------------|-------------|-----------|\n| AnyDoor   | 21.80  | 26.93       | 22.96      | 44.62       | 85.78     |\n| OmniGen.  | 15.31  | 29.14       | 55.41      | 38.45       | 72.83     |\n| UniCom.   | 13.55  | 29.17       | 54.22      | 49.37       | 84.02     |\n| GPT-4o    | 10.03  | 29.12       | 74.01      | 63.39       | 32.91     |\n| **Ours**  | **9.44** | **30.26**  | **85.11**  | **77.48**   | **95.43** |\n\n---\n\n#### 3. Interactions involving completely occluded target objects:\n\nSome actions may cause the interacting object to become entirely occluded in the resulting image, e.g., food being **eaten** (fully inside the mouth) or an object **blocked** by a hand. To evaluate the model’s generalization in such cases, we have curated 50 test instances where the target object is fully occluded after the interaction. We append “the object is occluded because of the human action” to the MLLM-generated descriptions, forming a dedicated evaluation subset.\n\nStandard metrics requiring object visibility (e.g., DINO-Score, HOI-Score) are inapplicable here. Instead, we manually assess the plausibility of the depicted interactions (“Occlusion Accuracy”), focusing on whether the generated images realistically reflect the intended occlusion. The results in the table below clearly demonstrate the superiority of our method over existing works.\n\n\n| Method    | FID↓   | CLIP-Score↑ | Occlusion Accuracy↑ | SSIM(BG)↑ |\n|-----------|--------|-------------|---------------------|-----------|\n| AnyDoor   | 22.95  | 26.85       | 0.08                | 83.52     |\n| OmniGen.  | 15.83  | 29.07       | 0.42                | 71.89     |\n| UniCom.   | 15.58  | 29.06       | 0.28                | 83.62     |\n| GPT-4o    | 10.52  | 29.31       | 0.84                | 25.28     |\n| **Ours**  | **9.82** | **29.87**  | **0.86**            | **94.35** |\n\nWe will include these qualitative results in our paper.\n\n---\n\n**Extension to the User Study:**  \nTo further demonstrate the generalization of our method to diverse inputs, we have expanded the user study in the main paper from 10 to 50 input cases, covering a broader range of human poses, viewpoints, and object categories. 45 participants have completed the evaluation. As shown below, our method consistently outperforms all baselines across all metrics, indicating strong user preferences for our approach.\n\n\n| Metrics        | AnyDoor | PbE   | FreeComp. | FreeCustom | PrimeComp. | OmniGen. | GenArt. | UniCom. | GPT-4o | Ours   |\n|---------------|---------|-------|-----------|------------|------------|----------|---------|---------|--------|--------|\n| IQ ↓          | 8.28    | 7.64  | 9.66      | 9.16       | 3.14       | _2.49_   | 6.03    | 4.07    | 3.29   | **1.24** |\n| IH ↓          | 8.65    | 8.90  | 8.35      | 5.85       | 6.15       | 5.01     | 5.34    | 2.91    | _2.69_ | **1.15** |\n| AP ↓          | _3.05_  | 5.04  | 7.20      | 6.92       | 6.25       | 4.93     | 6.65    | 8.05    | 5.74   | **1.17** |\n\n---\n\nOur framework can also be adapted to different foundation models.  \nIn our original experiments, we used FLUX.1[dev] as the base DiT model. We additionally evaluated HOComp with FLUX.1 Kontext[dev] [1] as the base model. As shown below, HOComp attains superior performance on FLUX.1 Kontext[dev], further demonstrating the adaptability of our approach across base model variants.\n\n\n| Base DiT Models      | FID↓   | CLIP-Score↑ | HOI-Score↑ | DINO-Score↑ | SSIM (BG)↑ |\n|----------------------|--------|-------------|------------|-------------|------------|\n| FLUX.1[dev]          | 9.27   | 30.29       | 87.39      | 78.21       | 96.57      |\n| FLUX.1 Kontext[dev]  | **9.19** | **30.32**  | **89.21**  | **81.13**   | **97.34**  |\n\n---\n\n### Q&W: Reproducibility of our HOComp.\n\nWe will release all codes, the IHOC dataset, and HOIBench upon acceptance, to facilitate reproducibility.\n\n---\n\n### References\n\n[1] FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space, arXiv 2506.15742."}, {"review_id": "VBvNkVluuQ", "reviewer": "Reviewer_7QgL", "summary": "This paper introduces HOComp, a novel framework for human-object image composition that enables natural and realistic interactions between humans and inserted foreground objects. Unlike prior methods that rely on manually defined regions and text prompts, HOComp autonomously determines the interaction type and region using Multimodal Large Language Models (MLLMs) like GPT-4o, and enforces pose and appearance consistency using several novel loss functions and mechanisms.", "questions": "1. Several previous works, such as CG-HOI and CHOIS, have annotated the BEHAVE dataset with textual descriptions, and datasets like OMOMO have further extended this line of research into 3D human-object interactions with textual annotations. However, the distinction between the proposed dataset and these prior efforts remains unclear. The authors are encouraged to clarify what specific advantages or novel features their dataset introduces. \n\n2. Is affordance estimation explicitly required in the proposed HOI generation method? The paper would benefit from a clarification on this point.", "limitations": "1. The paper lacks a comprehensive discussion of recent related work. In particular, several relevant works published in 2024 have not been acknowledged or discussed, including CG-HOI, HOIAnimator, InterFusion, Controllable HOI Synthesis, and F-HOI. These recent methods contribute significantly to the field of human-object interaction generation and text-driven synthesis. \n\n 2. To better understand the generalization ability of the proposed model, it may be helpful to evaluate its performance on interactions involving completely unseen objects that are not included in the BEHAVE or OMOMO datasets. If the dataset is skewed toward certain interaction types, there is a risk that the model may overfit to these patterns.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. A new approach for interaction-aware human-object composition that ensures realistic, coherent, and visually consistent integrations of foreground objects into human-centric images.\n2. The first large-scale dataset tailored for this task, including paired before/after images, interaction masks, text prompts, and detailed annotations. \n\nWeaknesses:\n\n1. The authors claim that the model can produce physically plausible results, yet no physical constraints are integrated to guarantee that. \n2. There is a lack of comparison of human-object interaction generation methods, making it difficult to evaluate the effect of the proposed method in this domain.\n3.  To more accurately assess the physical plausibility of the generated motions, it may be beneficial to incorporate a more sophisticated metric for measuring interpenetration, such as \"Intersection Volume.\" This metric directly quantifies the volume of penetration between interacting bodies and can provide a more precise evaluation of physical correctness. For reference, the paper \"Physics-aware Hand-object Interaction Denoising\" (CVPR 2024) offers a detailed discussion and implementation of such metrics, which may serve as a useful guideline.", "was_revised": true, "final_justification": "After considering the rebuttal and discussions, I maintain my original evaluation with a few clarifications. \n\nThe authors have satisfactorily addressed several concerns. \n\nHowever, one key issue remains partially unresolved: the lack of direct comparison with recent 3D-based HOI synthesis methods such as CG-HOI and HOIAnimator. To further strengthen the work, I encourage the authors to include such comparisons in future revisions.", "rebuttal": "We thank this reviewer for the positive review and useful comments. We appreciate your recognition of our IHOC dataset. We address each of the questions below. (Abbreviations used: Q/W/L refer to Question/Weakness/Limitation.)\n\n---\n\n## W1: Physical Constraints in Our Model\n\nWe would like to clarify that our approach utilizes pose-guided supervision to enforce physical constraints in the generated images. Specifically, during the training of HOComp, we impose human poses constraints within the interaction regions, using pose keypoints as explicit supervision. This helps the model learn physical priors that ensure realistic interactions between the foreground object and the background person.\n\nWhile it is possible to employ reconstruction methods[2] to generate 3D hand meshes and apply stronger physical constraints on specific body parts (e.g., hands), our model is not limited to hand interactions alone. As shown in Fig.3(c) (Supplemental), samples involving hand interactions alone comprise 54.3% of our training data, with other body parts (e.g., legs) also playing a significant role in the interactions in the remaining samples.\n\nIf we were to impose 3D constraints on the entire body, it would have introduced additional modeling errors and substantially increased computational complexity during training. Instead, we use DWPose[1], a robust pose estimator to capture landmarks across the entire body. DWPose predicts multiple hand keypoints and key points for major joints (e.g., each finger contains four keypoints from fingertip to base, and each joint has one keypoint). This approach allows our model to accurately capture human body movements across a variety of interaction types while maintaining computational efficiency.\n\n---\n\n## W2&L1: Lack of Discussion and Comparison of Recent Related Works\n\nWe thank the reviewer for pointing out relevant works on human-object interaction generation. We would like to clarify that the methods mentioned (CG-HOI, HOIAnimator, InterFusion, Controllable HOI Synthesis, and F-HOI) primarily focus on 3D human-object interaction generation, which aims to generate physically plausible 3D human-object interactions. These methods typically generate 3D models or sequences based on the interaction description, differing significantly from our approach, which focuses on generating 2D composited images. \n\nOur work addresses a distinct task of generating 2D composited images with not only harmonious human-object interactions, but also consistent foreground object and the background person appearances. Existing methods fall short in addressing our challenges. Although human-object interaction image generation methods can generate images with good human-object interactions, which is discussed in Section2 (main paper). One key distinction is that most of the referenced methods do not specify particular reference humans and objects (except for DreamRelation[3] and PersonaHOI[4]). Further, these methods do not emphasize the need to maintain consistent appearances of both humans and objects, in addition to generating harmonious interactions, which is a core aspect of our approach. Because PersonaHOI has not made their official code available for direct comparison, we have included comparisons with DreamRelation in Tab.8 and Fig.10 of the Supplemental.\n\nWe will incorporate the above discussions in our revision.\n\n---\n\n## W3: Incorporate a More Sophisticated Metric to Assess the Physical Plausibility of the Generated Motions\n\nAs suggested, we have thoroughly examined the six metrics, MPJPE and MPVPE (for hand pose/shape reconstruction accuracy), IV and PD (for physical plausibility and penetration degree), Contact IoU (for contact area accuracy), and Plausible Rate (for overall plausibility and manipulation credibility), used in ``Physics-aware Hand-object Interaction Denoising'' (CVPR'24).\n\nHowever, all of these metrics require the generated 3D mesh for computation. Except for \"Intersection Volume'', the other metrics additionally require access to the corresponding GT 3D mesh for evaluation. Our method produces only 2D composited images as output and does not include corresponding 3D GT during inference. While it is theoretically possible to reconstruct 3D meshes from 2D images and then compute Intersection Volume without the need for 3D GT, such reconstruction can introduce significant errors, especially given the diverse interaction types and occlusions present in our task.\nNevertheless, we will explore if there are more appropriate metrics for such an evaluation.\n\n---\n\n## Q1: Difference from Prior Human-object Interaction Datasets\n\nThe key distinctions and advantages of our dataset compared to BEHAVE and OMOMO are as follows:\n\n**Task and Modality:** Our dataset is the first large-scale resource specifically tailored for the interaction-aware human-object composition task, providing paired pre-/post-interaction image data with comprehensive annotations. In contrast, BEHAVE and OMOMO focus primarily on 3D human-object interaction generation, providing 3D meshes of humans and objects. While BEHAVE contains some RGB images during interaction, it lacks the crucial pre-interaction data required for modeling before/after composition.\n\n**Scale and Diversity:** Our dataset covers a significantly broader range of human-object interactions, actions, and scenes. Compared to BEHAVE (8 subjects, 20 objects) and OMOMO (17 subjects, 15 objects), our dataset includes 11,700 unique samples, each with a different human subject. We provide 117 types of interactions and 342 distinct foreground object categories, encompassing diverse human poses, viewpoints, both indoor and outdoor environments, and a wide variety of object categories, sizes, and interaction body parts. In addition, our dataset features a broad spectrum of image styles. Comprehensive statistics and visualizations can be found in Fig.3 (Supplemental).\n\nIn summary, our dataset uniquely enables research on image-based, interaction-aware human-object composition with a level of scale, diversity, and paired pre/post-interaction coverage not available in prior datasets. We will add this discussion in our revision.\n\n---\n\n## Q2: \"Is affordance estimation explicitly required in the proposed method?\"\n\nOur method does not explicitly involve affordance estimation. Instead, we rely on the MLLM to infer the appropriate human-object interaction based on the visual input of a background human image and a foreground object image. The MLLM performs reasoning on these inputs to predict the interaction type, without the need for explicit affordance estimation. In our HOIBench evaluation, MLLMs (e.g., GPT-4o) correctly predicted plausible interaction types in 100% of the tested cases.\n\n---\n\n## L2: Generalization to Interactions with Out-of-Distribution Objects\n\nOur method performs well on objects not included in the BEHAVE or OMOMO datasets. For example, all examples shown in Fig.1 in the main paper, as well as those in Fig.4 (rows 1, 2, and 4), involve object categories that are absent from both BEHAVE and OMOMO. Nevertheless, our method is able to generate satisfactory results on these cases.\n\nWe would like to clarify that, when constructing our dataset, we did **not** select object types based on those present in BEHAVE or OMOMO. Specifically, BEHAVE contains only 20 objects and OMOMO contains 15 objects. In contrast, the object types in our training set were determined using an MLLM, which inferred potential object categories according to 117 interaction types (such as ''hold'', ''play''). These pairings (e.g., ''hold-bag'', ''play-guitar'') were then manually verified for plausibility, resulting in a total of **342 distinct object categories** in our dataset.\n\nIn our **HOIBench** evaluation, we cover all 117 interaction types and 108 object categories. While some categories may overlap with those in the training set, we ensured that **both the human subjects and the object images in the test set are completely disjoint from those in the training set**. We will provide more visual results in the supplementary materials.\n\n---\n\n## Ethical Concerns on Data Privacy, Copyright, and Consent\n\nAll images in our training set are sourced from the public HICO-DET dataset, images generated by the T2I model FLUX.1 [dev], or openly licensed free image websites such as Unsplash and Pexels. The HOIBench also exclusively uses images from freely available public websites. Therefore, there are no ethical issues for our dataset. We will release our dataset upon acceptance of the paper.\n\n\n---\n\n\n## References\n\n[1] Effective Whole-body Pose Estimation with Two-stages Distillation, ICCV'23.  \n[2] Reconstructing Hands in 3D with Transformers, CVPR'24.  \n[3] Dreamrelation: Bridging customization and relation generation, CVPR'25.  \n[4] PersonaHOI: Effortlessly Improving Personalized Face with Human-Object Interaction Generation, CVPR'25."}, {"review_id": "yxkwdRhiC3", "reviewer": "Reviewer_zMtM", "summary": "The paper introduces HOComp, a diffusion-based framework that composites a foreground object into a human-centric background image. A multimodal large language model is used to choose an interaction type (e.g., holding, kicking) and a plausible spatial region on the person. The authors propose a multi-view appearance loss, and a background consistency loss to ensure consistent shapes/textures of the\nforeground and faithful reproduction of the background human, and propose Detail-Consistent Appearance Preservation (DCAP) module for preserving appearance. Also, the authors propose a new dataset, IHOC for the human object composition task. Experimental results show competitive results compared with previous image composition methods.", "questions": "What is the MLLM's influence on the inference performance? Could you compare using MLLM with other vision-based methods to detect the overlap regions?", "limitations": "See weakness", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengh:\n(1) The paper is well-written and easy to follow.\n(2) The motivation of human-object interaction in customization task is interesting.\n\nWeakness:\n(1) Does this model support multi-subject, multi-person composition? For instance, 2 persons and 2 objects. I am willing to see these results in the rebuttal.\n(2) The impact of the area of overlap region to the model's performance is not explored in the paper. What if there is large overlap regions between human and subjects? I suggest the authors do such experiments compared with previous methods in the rebuttal.", "was_revised": true, "final_justification": "The authors partially address my concerns, so I remain my initial positive rating.", "rebuttal": "We thank this reviewer for the positive evaluation and appreciate the recognition in the paper’s clarity and the motivation for the customization of the human-object interaction task. We address each of the questions below. (Abbreviations used: Q/W refer to Question/Weakness.)\n\n---\n\n## W1: Support for Multi-object and Multi-person Composition\n\nTo evaluate our model’s ability to handle multi-object and multi-person compositions, we have constructed a dedicated test set, since the current HOIBench does not include such cases. Our new benchmark covers two scenarios: (1) two persons with two objects and (2) three persons with three objects.\n\n**Test Set Construction:**  \nWe first collected 40 images from the internet containing multiple persons: 20 images with two persons, and 20 images with three persons. These images were selected to ensure diversity in human appearance, pose, and clothing.\n\nFor **Two Persons – Two Objects:**  To broadly generate interaction-object pairs, we have adopted the 117 interaction types defined in HICO-DET. For each interaction type, we prompt GPT-4o to infer a plausible foreground object (e.g., “playing”→“guitar”). A concise textual description is then used to retrieve a representative object image from the internet, resulting in 117 interaction–object pairs. For the two-person scenario, each person in an image is randomly assigned a unique interaction–object pair from this pool, ensuring a wide variety of combinations. For each of the 20 two-person images, we repeat this assignment process 10 times with different pairs, resulting in 200 two-person, two-object test cases.\n\nFor **Three Persons – Three Objects:**  Similarly, for three-person images, each person is assigned a unique interaction–object pair, randomly sampled from the 117 pairs described above. This process is repeated for each three-person image, resulting in 200 three-person, three-object cases.\n\n**Test Results:**  \nWe have evaluated our model using the new test set of 400 cases (200/scenario) and compared its performance with the top three methods from the main paper. We compute the HOI-Score and DINO-Score for each individual person-object interaction pair, and then take the average of all these scores across all interaction pairs.\n\nAs shown in the following table, our approach achieves the best results in both scenarios and on all metrics.\n\n\n| Category               | Methods     | FID↓   | CLIP-Score↑ | HOI-Score↑ | DINO-Score↑ | SSIM(BG)↑ |\n|------------------------|------------|--------|-------------|------------|-------------|-----------|\n| 2 persons & 2 objects  | AnyDoor    | 22.38  | 26.86       | 23.12      | 55.46       | 83.19     |\n|                        | OmniGen.   | 15.98  | 29.03       | 51.45      | 38.66       | 75.91     |\n|                        | GPT-4o     | 11.13  | 29.18       | 73.12      | 63.17       | 25.50     |\n|                        | **Ours**   | **10.25** | **29.87** | **83.61** | **73.48**   | **93.25** |\n| 3 persons & 3 objects  | AnyDoor    | 24.17  | 26.03       | 22.98      | 52.07       | 80.11     |\n|                        | OmniGen.   | 18.25  | 28.14       | 49.01      | 35.85       | 72.14     |\n|                        | GPT-4o     | 13.46  | 28.85       | 70.88      | 55.14       | 19.80     |\n|                        | **Ours**   | **10.43** | **29.76** | **82.98** | **73.09**   | **91.52** |\n\n---\n\n## W2: The Impact of the Area of Interaction Region\n\nWe evaluated HOComp's performance on cases with exceptionally large interaction regions, such as those illustrated in Fig.2 (Supplementary) (e.g., \"a man is playing a guitar,\" where the interaction covers over 70% of the image). We have identified 82 images in HOIBench where the interaction region occupies more than 70% of the total image area, and compared HOComp with previous methods on this subset. As shown in the following table, HOComp achieves substantially better results than other baselines across all metrics in large interaction region cases.\n\n| Method    | FID↓   | CLIP-Score↑ | HOI-Score↑ | DINO-Score↑ | SSIM(BG)↑ |\n|-----------|--------|-------------|------------|-------------|-----------|\n| AnyDoor   | 21.80  | 26.93       | 22.96      | 44.62       | 85.78     |\n| OmniGen.  | 15.31  | 29.14       | 55.41      | 38.45       | 72.83     |\n| UniCom.   | 13.55  | 29.17       | 54.22      | 49.37       | 84.02     |\n| GPT-4o    | 10.03  | 29.12       | 74.01      | 63.39       | 32.91     |\n| **Ours**  | **9.44** | **30.26** | **85.11** | **77.48**   | **95.43** |\n\n---\n\n## Q1 & Q2: MLLMs' Influence on Inference Performance and Comparison with Vision-Based Methods to Detect Interaction Regions\n\nAt MPRG's coarse level, it leverages MLLMs to automatically identify suitable interaction types/regions via multi-stage querying. In some rare cases, MLLMs may fail to accurately predict the interaction regions. We provide a detailed analysis of these failure modes and potential solutions below.\n\nIn our HOIBench experiments, MLLMs (e.g., GPT-4o) identified reasonable interaction types in **100%** of samples. The corresponding interaction region was predicted correctly in **91.33%** of cases (548/600). For the remaining **8.67%** (52/600), two main failure modes are observed:\n1. **Mismatch Between Interaction Regions and Types (6% of cases):** When multiple plausible interaction types exist, MLLMs may misassign interaction regions. For example, if the object is sunglasses and the interaction type is \"hold\", the model may incorrectly assign the region around the eyes (i.e., \"wear\" action). While the generated image depicts a person wearing sunglasses with harmonious interactions and consistent appearance, the predicted interaction type is inconsistent.\n2. **Incorrect Interaction Region Size (2.67% of cases):** In these cases, the predicted interaction region does not fully cover the area, including the object and the human body part, for modification. As shown in Fig.6 (main paper), this can hinder generation of correct interactions.\n\nTo address these two issues, we explored the following solutions:\n- **Additional Input Conditions:** To alleviate the mismatching issue, we can further incorporate human pose priors (from a pose estimator) as an additional prior to MLLMs. For GPT-4o, the region prediction accuracy increased from **91.33%** to **96.5%**. This improvement can be attributed to the introduction of explicit keypoint information, which provides precise localization of body parts such as the face and hands.\n- **Training with Noisy Data:** To mitigate the impact of incorrect interaction region sizes, we introduce noisy data during training.  \n  We first generated 1,000 accurate input data samples. In accordance with the observed error rate, we then moved the bounding boxes of 700 samples to create mismatches with the interaction types (e.g., moving the bounding box of a “soccer ball” with a “kick” action to cover the hand, which may correspond to a “hold” action). Additionally, we reduced the bounding boxes of 300 samples so that they no longer fully covered the interaction object or the required body movement. We fine-tuned the previously pretrained HOComp model for 3,000 steps using these noisy samples alongside the original IHOC dataset. After retraining, we re-evaluated our method using the 52 failure cases. As shown in the following tables, training with noisy data substantially improved the model's performance even when MLLMs mispredicted the interaction region.\n\n| Training Strategy     | FID↓   | CLIP-Score↑ | HOI-Score↑ | DINO-Score↑ | SSIM(BG)↑ |\n|----------------------|--------|-------------|------------|-------------|-----------|\n| Without Noisy Data   | 11.58  | 29.55       | 38.62      | 70.02       | 62.01     |\n| With Noisy Data      | **10.03** | **29.82** | **77.89** | **74.29**   | **88.93** |\n\nQualitative results and further experimental details will be included in the Supplemental.\n\n**Comparison using MLLM with other vision-based methods to detect interaction regions**: Vision-based methods face significant challenges in this task, as they not only need to localize humans and objects but also infer plausible interaction types and predict specific body parts/regions involved in the interaction.\n\nTo the best of our knowledge, existing vision-based approaches [1-3] are limited: they typically operate on images where the interaction already exists (e.g., our composited results), and can only detect existing relationships or involved regions. They generally lack the ability to perform open-ended reasoning to predict plausible interaction types/regions for images where no interaction has yet occurred, given only an isolated person and a reference object.\n\nWe have also compared the performance of different MLLMs, as shown in the following table. The accuracy of interaction type/region predictions is computed through a user study involving 20 participants, who evaluate the correctness of MLLM predictions. The correct predictions are counted, and the average accuracy per participant is computed.\n\n| MLLMs        | Interaction Type Prediction Accuracy | Interaction Region Prediction Accuracy |\n|--------------|-------------------------------------|---------------------------------------|\n| GPT-4o       | **100%**                            | **91.33%**                            |\n| Gemini2.5Pro | 99.67%                              | 86.83%                                |\n| Grok4        | 99.5%                               | 88.67%                                |\n\n---\n\n## References\n\n[1] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection, ECCV'24  \n[2] Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models, NeurIPS'24.  \n[3] OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling, NeurIPS'24."}]

Author rebuttal:

Meta-review: The paper presents an interesting problem formulation of editing the hand object composition in an image with a generative model. Reviewers acknowledged that this paper proposes a complex yet reasonable framework along with a new dataset, and the experimental results show its effectiveness. 

Initially, reviewers raise concerns about the complexity of the method, the robustness of the complex pipeline, various generalization capabilities, the evaluation of physical plausibility, and potential data bias. During the rebuttal, authors provide extensive experimental results and clarifications to address the concerns and promise to release the code. Reviewers are satisfied with this response; however, there are several remaining issues. For example, the comparison with the 3D HOI method and the lack of physically plausible measures.

Given that all technical reviewers lean toward borderline acceptance, the contribution of this paper, and the authors have demonstrated willingness and ability to address most concerns, AC recommends accepting this paper with an expectation that the issues (including ethics issues) can be fully resolved in revision.
Decision: Accept (poster)