Learning to Generate Human-Human-Object Interactions from Textual Descriptions
Authors: Jeonghyeon Na
Sangwon Baik
Inhee Lee
Junyoung Lee
Hanbyul Joo
Venue: NeurIPS 2025 poster ()
Keywords: Generative Model
Human-Human Interactions
Human-Object Interactions
Primary area: applications

Abstract: The way humans interact with each other, including interpersonal distances, spatial configuration, and motion, varies significantly across different situations. To enable machines to understand such complex, context-dependent behaviors, it is essential to model multiple people in relation to the surrounding scene context.
In this paper, we present a novel research problem to model the correlations between two people engaged in a shared interaction involving an object. We refer to this formulation as Human-Human-Object Interactions (HHOIs).
To overcome the lack of dedicated datasets for HHOIs, we present a newly captured HHOIs dataset and a method to synthesize HHOI data by leveraging image generative models. As an intermediary, we obtain individual human-object interaction (HOIs) and human-human interaction (HHIs) from the HHOIs, and with these data, we train an text-to-HOI and text-to-HHI model using score-based diffusion model. Finally, we present a unified generative framework that integrates the two individual model, capable of synthesizing complete HHOIs in a single advanced sampling process. Our method extends HHOI generation to multi-human settings, enabling interactions involving more than two individuals.
Experimental results show that our method generates realistic HHOIs conditioned on textual descriptions, outperforming previous approaches that focus only on single-human HOIs. Furthermore, we introduce multi-human motion generation involving objects as an application of our framework.

Learning to Generate Human-Human-Objert Interactions from Textual Descriptions
Jeonghyeon Na, Sangwon Baik, Inhee Lee, Junyoung Lee, Hanbyul Joo
Scoul National University
Equal Contribution *Corresponding Author
{prom317, baw1907, ininin0516, fanp:s1,,jnncoo juncong, hbjoo) (aajoo@@auu..... aamu.ac.kk kr.
Abstract
The way humans interact with each other, including interpersonal distances, spatial configuration, and motion, varies significantly across different situations. To enable machines understand such complex, context-dependent behaviors, it is essential to model multiple people in relation to the surrounding scene context. In this paper, present novel research problem model the correlations between two people engaged in shared interaction involving an object. We refer to this formulation as Hoiman-Human--ae Object Interactions (HHOIs). To overcome the lack of dedicated datasets for HHOIs, we present a newly captured HHOIs dataset and a method to synthesize HHOI data by leveraging image generative models. As an intermediary, we obtain individual human-object interaction (HOIs) and human numan-human human interac- tion (HHIs) from the HHOIs, and with these data, we train an text-to-HOI and text-to-HHl HHI model using score-based diffusion model. Finally, we present unified generative framework that integrates the two individual model, capable of synthe- sizing complete HHOIs in a single advanced sampling process. Our method extends HHOI generation to multi-human settings, enabling interactions involving more than two individuals. Experimental results show that our method generates realistic HHOIs conditioned on textual descriptions, outperforming previous approaches that focus only on single-human HOIs. Furthermore, we introduce multi-human motion generation involving objects as an application of our framework.
Introduction
Human behavior in real-world environments is inherently social and context-dependent. People naturally interact with one another through structured patterns of interpersonal distance, spatial configuration, and motion, which are intuitively understood by humans vary significantly across different situations. Understanding these nuanced, multi-human interactions is critical for Al systems that aim to interpret, simulate, or engage naturally in human-centric environments. While Human- Object Interactions (HOIs) [66, 57, 9, 32, 65, 63, 45, 26] and Human- Interactions (HHIs) [67, 23, 10, 35, 42, 10, 43, 54, 50, 36, 56] have been extensively studied in isolation, modeling interactions involving both multiple people and shared objects remains underexplored. In particular, dyadic interactions, where two individuals engage coordinated activity around a common object, are prevalent in everyday life, but have received relatively little attention. Examples include sitting together on a sofa, sharing an umbrella, or standing side by side at a whiteboard for discussion. In this paper, present novel research problem: modeling the coordinated behavior of two individuals interacting around a shared object. We refer to this formulation as Human-Fluman-Objeer Interactions (HHOIs). A core challenge in studying HHOIs is the absence of dedicated datasets. Existing HOI datasets [52, 15, 68, 16, 61] primarily feature single-human-objee interactions, while
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

"Two people are moving a box together."
"Three people are sitting on bench"
Figure 1: Results of our HHOI generation given object instances and text- text-prompt descriptions. Multiple humans in action are generated by jointly enforcing scene-level consistency across human- object interactions (HOls) and human- huuman-humnn interactions (HHIss.
HHI datasets [30, 12, 14, 35, 55, 13. 38, 42] typically lack object context, often limited dyadic conversational scenarios in standing poses.
To address the lack of diverse HHOI data, introduce newly collected 3D dataset captured using a multi-camera system, specifically designed to support the training and evaluation of HHOI models. In addition, we present synthetic HHOI dataset generation pipeline that leverages pretrained image diffusion models [41] to complement real-world data, particularly for scenarios that are challenging capture in studio environments. These diverse data sources are unified through score- based diffusion model, enabling realistic generation of dyadic human numann-object interactions. Our model is conditioned on textual descriptions, enabling semantically grounded generation of HHOIs. Importantly, we further demonstrate that our framework can be extended beyond dyadic interactions to accommodate multi-human interactions, offering scalable solution for modeling increasingly complex social behaviors. Experimental results show that our method produces more realistic and coherent interactions compared existing baselines that model only single-human HOIs. As potential application of our model, we present multi-human motion generation via SarassonnNois--ppmizzztiiii [25], with our output HHOI as constraint.
Our contributions are summarized as follows: (1) Curated datasets for HHOI, along with method- ologies for constructing them: (2) Score-based HHOI model that jointly captures both individuals' interactions with a shared object and their interaction with each other, conditioned on textual de- scription; (3) Extension to multi-human HHOIs. enabling generation interactions beyond dyadic settings; (4) Application to object interaction-avee multi-huma motion generation.
2 Related Work
Human-Objec Interaction Human-objess interaction (HOI) aims to understand how humans interact with objects in the environment. This line of research is crucial for enabling machines to interpret and mimic human behaviors, thereby supporting the development of embodied AI agents and realistic digital human modeling with natural object manipulation. There have been considerable efforts to collect and scale up 3D HOI dataset, aiming to pursue data-driven approach in this direction. Early work tries to capture the 3D HOI scenes in a controlled setup using marker [22], IMU [52, 15, 68, 16], multi-view capture system [18, 19, 61, 3), or hybridizing them [11, 28, 64, 63, 37J. In addition to real-world capture, synthetic datasets have also been introduced using game engines [6] and physies simulation [4]. More recently, automated pipelines have emerged generate 3D HOI scenes from pre- pre-trained 2D image models, significantly reducing capture effort and expanding scenario

---

Training
Lecore
Rever OTE Proes
{o 2
V
4 Ineaiiany Tos
Colliia
1=
Penatee HOU FEET Suraphe
V *=
Score
Text Enbhadnng
Three Stap
Made Sebecx
Yoe price diilig book?
Rto
Inference
Reveess CDC Trecest
laads
gz
people (a) Training/ Inference Process for HHl and HOI
sumplny Roads
(b) Advanced HHO1 Sampling
Figure 2: Method Overview. (a) The training and inference process of the HOI/HHI part. (b) The advanced HHOI sampling process by introducing inconsistency loss and collision loss.
diversity [26, 17, 33, 27J. Various models [57, 9, 32, 65, 45, 58, 29, 60, 21] have been proposed to learn from the presented datasets, including models manipulating articulated objects [11, 28], and multiple objects once [63, 28]. However, the majority of existing methods are limited to single-human interaction scenarios, while scenarios involving multiple humans with objects remain largely unexplored. Core4D [37] takes step toward this direction by collecting HHOI data for collaborative tasks involving two people, but its scale and diversity are still limited.
Human-Huuman Interaction Modeling the interaction between individuals is essential to capture cooperative behaviors and social dynamics [24], which is crucial for developing embodied Al agents capable of natural human interaction. Previous works have primarily focused on dyadic human- human interactions, aiming either to reconstruct plausible interactions from images [12, 38], or to generate natural motion [30, 35, 55, 13, 42]. Recently, several methods have been proposed to extend the interaction modeling to more than two individuals [67, 23, 10, 43]. Many of these approaches are conditioned by contextual signals such as text [35, 42, 10, 43], music [44, 34], or predefined action-reate rettionreaction roles [54, 50, 36, 56]. yet they often struggle to capture interactions that are tightly coupled with specific target object. MAMMOS [36] explores scene-conditioned multi-human motion generation, yet remains limited scope and lacks complex interactions.
Score-based Generative Models. Score-based generative models [46, 47] estimate gradients of the data distribution for generative modeling, by introducing noise conditional score networks to learn score functions at multiple noise levels. Later work [48] generalized this approach using stochastic differential equations, providing a continuous-tima formulation. This framework has proven highly effective for various generation tasks such as object rearrangement [53], object pose estimation [62], scene-graph generation [49] and human pose estimation [8]. Recent extensions apply score-based models interactive settings, such human-object interaction [27], and object-object interaction [1].
3 Method
Our HHOI model is structured as a combination of HOI model and HHI model. We first introduce how each component HOI model and HHI model is independently represented (Sec. 3.1). Specifically, we model HOI and HHI using score- based diffusion models [46, 48]. We then describe the training and inference procedures for each component (Sec. 3.2). Finally, present guided sampling method that integrates each component with inconsistency and collision constraints during the inference process to generate coherent and plausible HHOI configurations (Sec. 3.3).
3.1 HHOI Formulation
Modeling Human-Object Interaction (HOI). For HOI, we adopt an object-centric coordinate frame
in which the object instance mesh M is centered at the origin. Our goal is to model the distribution of plausible human spatial and postural configurations relative the object, covering a variety of HOI scenarios. Formally, we define HOI as the rotation Rx HR translation tx R3 scale
3

---

Sy. E R2, and body pose embedding ON E R of a human H. These are conditioned on an object mesh M and a textual description describing the HOL. We denote the HOI distribution as pe and a corresponding HOI sample as H
JHOI
Pe M
& (Ry, t7e, 87, 0z)-
(I)
We use SMPL-X [39] for the human model and 6D representation [70] for Rx. Given an HOI sample df human mesh H can be obtained as follows:
H = St. R RxH + tH
(2)
H X smplx(dec((x))
where cle(-)) is the body pose decoder (detailed below) that maps the body pose embedding 0H to SMPL-X pose E 2 & The function smplx(0) returns the SMPL-X mesh for pose 8 in its canonical frame. We by R2 both the 6D representation and the corresponding SO(3) rotation, since there is a straightforward one-to-one mapping between them.
Modeling Human-Human Interaction (HHI). Given a text prompt e that describes the interaction between two humans, denoted Hi and H2. we model the HHI using their body poses along with the relative rotation and translation of H2 with respect to Hi. We assume both humans share the same scale. Specifically, we define the HHI distribution as Pe H-H and the HHI sample as dHI!l
IMI
H -
(0H1: RH 192 022)
(3)
The human H2 can be reconstructed from the HOI of H₁ and the HHI:
H2 SH1 RHH R 722 + SH1 RHG 1722 + 1711
(4)
H (mmplx(de((xzz)
Body Pose Embedding, We find that modeling the human pose distribution using a low-dimensional embedding 0x is more effective than using the full 126D (=21 6D representation for each joint rotation) body pose 9 directly. Therefore, we train body pose encoder and decoder to obtain an embedding vector of the body pose, enabling to model HHOI the latent space of body poses. To this end, we process 922K human body pose data from [18, 20, 52], and train body pose encoder and decoder, each implemented as a 4-layer MLP. In experiments, we embed 126D human body poses in 10D space, that is, H = 10, which results in of E R220 and HHI E IR
3.2 Score-based HHOI Diffusion Model
We model HHOI using score-based diffusion, similar to how poses and scales of objects are modeled in [1. 62]. Let us denote our HHOI diffusion model as Vee. parameterized by é. Then, Ve represents the noised score function of the HOI, HHI distribution time step t:
V spar log Pe PM (OHOT)
= HOI
o (0i, t/c, 2)
V mmn log pe
H- (H (!!!)..
2 HHI
where o.()) is noised HOI or HHI sample at time step f, E (HOI, HHI) represents mode selector, and @HÖI @HIIl = θ COI HHIl = 0. For simplicity, we do not use a mesh instance M as input when modeling HOI; rather, we assume a fixed M is provided for each scenario.
Training, As HHOI is formulated by decomposing into HOI and HHI Sec. 3.1, we also train the acore-based diffusion in decoupled manner. Each mode is trained with the following objective function presented in Denoising Score MDtching(DSM) [51]:
| | (of, 6² of
L (O*) EE.i((E.1)
XeE of
(6)
(t)
where is minimal noise level, of N( (2(() amin A is regularization term.
anus
is variance factor, and den
The training process is shown in Fig. 2(a). First, we generate perturbed samples of as defined. We - and the estimated score from the HHOI diffusion model to
then compute the target score -()

---

evaluate the score loss in Eq. (6), We use the CLIP text encoder [40] to obtain text embedding from text prompt. For simplicity, we use с to denote both the text prompt and its corresponding
embedding. Additionally, we adopt the text prompt augmentation strategy from [1].
Inference. To sample HOI and HHI instances independently, we solve the following Probability Flow(PF) ODE [48] in reverse time, i.e., from t = 1 to =
कं NO. I.)
d - ()) di
We solve the ODE using external library [7], which is fully supported to run on the GPU. Fig. 2(a) shows this process. However, independently sampling HOI and HHI leads to incoherent posture and spatial configuration in each sampling output, and does not give us the desired HHOL. To address this issue, we propose advanced guided sampling technique to obtain HHOI in Sec. 3.3.
3.3 Advanced Guided Sampling for HHOI
We adopt the PF ODE augmentation paradigm [1, 27], introducing additional terms that enforce the appropriate combination of HOI and HHI samples during HHOI sampling. Specifically, we incorporate an inconsistency loss to enforce consistency across samples and collision loss to prevent human collisions. Consider the HHOI involving N humans(i.e., N HOIs), along with M HHIs:
HOLHH PeHon; & HOI,H, (RH, tH. SH, Oz.);
(8)
1 III He HH ~ H427 Φ I Hy 7, = (0Ht RH, *H, HH, OH.):
where 1, N, and K(, 1 siskkss
+ k)] M NIN-1) Note that the N(N-1) The graph formed by connecting the number of possible HHIs is at most CC2 = 2
M human pairs should be directed acyclic graph (DAG). For example. given three HHIs (H2 > Hi, Hss > H2, Hi > Hs}, this is not a valid HHI set because it has a circular dependency.
Inconsistency Loss. To generate unified HHOI sample from inconsistency HOI/HHI samples set, we introduce inconsistency loss Line enforcing coherence between the human representations derived by each sample. In nutshell, Line is the role of enforcing consistency between humans obtained from Eq. 2 and humans obtained from Eq. 4. Specifically, it penalizes discrepancies in scale, body pose, rotation, and translation of HOI and HHI samples at time step l during sampling, as follows:
Cine (n)) = ( + Lvar,e + Lear, (ear.R(R) (R) + (t),,e(t),
(9)
where tb denotes union of N HOI samples (o) (O. } and M HHI samples {
HHI,H, H, at time d%
step f. Each term in Eq. (9) minimizes the variance of each component, thereby enhancing overall consistency as detailed below.
The scale variance loss, (sar.s()) penalizes deviations in human scale across HOI samples since the HHT model assumes equal scales for paired humans:
Lvar, (s) N Var((sH, N
(10)
The body pose variance loss Lear, (f) enforces consistency in body poses for the same human:
Lear. o (0)
Σ No Var((On..n)*
(11)
where N; denotes total occurrences of human H. across HOT and HHI samples. More precisely, N; -1 is the number of HHIS where Hi appears. The rotation and translation variance losses, L I (R)
and Lear, (t), ensure consistency among human rotation and translation across interactions:
N
Lear, Fear,I(()) (R) Σ
N/ Var((Ru, (RH,.. Ru.-HH No
(12)
N
Lvar,((t) = Σ Ny. Var((tH.) U (sH, RHH,. tH,-HHm + tze I=1)
(13)
1=0
5

---

where N represents the number of HHIs where human H4 appears as target(i.e., in pairs of the form H4 - Hj). Note that Var( =
Collision Loss. To ensure plausible interactions among multiple humans, it is essential to avoid physically implausible scenarios such as unintended body intersections. In particular, for human pairs that are not explicitly connected by HHI, assume "no collision" constraint that is, the absence of collision serves as their implicit HHI. For instance, if the HHI set is {M2 - N1, Ha - H2), then we enforce that H41 and My do not collide.
However, computing accurate inter-human collisions using full SMPL-X meshes during ODE-based sampling is computationally expensive. To address this, we approximate collisions between two humans in following steps: (1) Compute joint positions via forward kinematics over body joints using each human pose; (2) Construct a 24-capsule proxy from these joints with predefined radius factors to approximate each human as a capsule-based model; (3) Compute the sum of overlaps over all capsule pairs for two humans as the collision loss. Each overlap is computed simply the sum of the two radii minus the distance between the capsules' axis segments. Accordingly, the collision loss Lool (b,) is computed as
21
21
Ceal =
E
Σ
Σ
max(0, r + Try H, dHH.. )
(14)
24
(E
c,=1 cy=1
where may is the set of non-adjacent human pairs, rHH is the radius of the c,-th capsule of human Hi H, is the distance between the axis segment of the c--th capsule of human Hi and the axis and de...
segment of the ey-th capsule of human Hy. Note that in Eq. 14, if the sum of the two capsule radii is less than the distance between their axis segments, there is no collision and the value is clipped to zero. See Supp. Mat. for a process to approximate a human with a 24-capsule proxy.
Guided HHOI Sampling, Using Eq. and Eq. 14, we augment the PF ODE in Eq. 7 as follows to sample HHOI:
$ ~ NO, o max I,),
dø... - (0) _
(15)
A o Line (T+) + A2 Lant (o:),
dt
where of is in HOI(: = HOI) sample or HHI( = HHI) sample at time step t, X and X2 are weight terms. In Fig. 2(b), we provide an intuitive overview of our advanced sampling process through
concrete HHOI example in which three humans sit on a bench.
Datasets
In this section, we present the motivation and data collection procedure for our dataset. To align with our model architecture, the distribution of body poses in the HOI dataset must be consistent with that of the HHI dataset. For instance, HOI dataset with action "sitting body pose with head pose fixed to the front, assuming only one human is in the scene, while the HHI dataset may contain a broader range of head orientations more representative of real-world variability. In this case, our model will be unable to generate these natural variations during the HHOI synthesis.
bench", may consist only of
The most straightforward solution is to collect data from scenes that include both multiple humans and interacting objects. While several individual datasets that contain either multi-human poses or human-object interactions have been released, there are limited number of datasets that capture both multi-human poses and human-object interactions. CORE4D [37] is one of them, providing high-quality motion capture data of two individuals interacting with single object. It includes object pose annotations and human SMPL-X parameters for six object categorics. However, the range of interaction types in CORE4D is relatively narrow, focusing primarily collaborative actions such as passing an object or moving it together. In contrast, weal-world interactions are more diverse. Notably, there are scenarios where no direct contact occurs between humans and objects, yet the mere presence of the object influences spatial relationships (proxemics) and interaction dynamics between individuals. Such implicit object effects underrepresented in existing datasets.
To address the lack of diverse human- naman-human-ohject interactions, we collect dyadic human and object poses using multi- camera system. Specifically, we record two people interacting with various

---

objects from 36 synchronized RGB cameras. We apply an off- the-she-eel 2D pose detector [59] and 3D optimization method [69] to recover the SMPL-X parameters for both individuals in each camera frame, followed by manual post-filtering The collected HHOI dataset is divided into HOI and HHT subsets, which are used to train the respective score-based models. The same data processing and training procedure is applied to the CORE4D dataset. Our dataset consists of 5,078 frames spanning 11 object categories, and is expressed in metric units. Refer to Supp. Mat. for more details.
While the aforementioned datasets provide high-quality HHOI data, their content is limited to scenarios that can be captured in controlled studio environments. As a result, interactions involving large objects or outdoor scenarios, such as "two people riding motorcycle together", are difficult to capture with our multi-view camera systems. To address this limitation, we leverage the knowledge of 2D image diffusion model on HOI and HHI. Recent models such as Flux [31] can generate realistic human-object interaction images from textual prompts while preserving crucial social eues, including joint body poses and interpersonal distances that reflect real-world dynamics.
To generate HOI data, we utilize ComA [26], which takes as input an object instance and a textual description of an HOI scenario, and outputs corresponding 3D human-object interactions represented via SMPL-X parameters. For HHI data, we generate images of dyadic human interactions involving an object using textual prompts and the Flux diffusion model. Subsequently, we apply a Human Mesh Recovery method [2] to reconstruct 3D human meshes from the generated images.
5 Experiments
5.1 Baselines and Metries
Baselines. Previous research on human-object interaction has primarily focused on generating single human pose with respect to a specific object. As there exists no comparable method for generating multiple humans interacting with an object as in our setting, we develop two approaches for this novel task: (1) by extending GenZI [33], and (2) by lifting separately generated humans using a depth optimization strategy, referred to as Depth Opt.
First, we modify GenZI, which originally renders a target object in multi-view and inpaints a single human based on a text prompt. We revise the prompt and inpainting process to synthesize images with multiple people, and follow GenZI's pipeline by detecting 2D body poses and optimizing SMPL-X parameters using multi-view joint positions. For Depth Opt., we inpaint single-view object rendering using a diffusion- braded model [41], extract 3D human meshes via Human Mesh Recovery [2], and align them to the object using depth optimization with Depth-Pro [5]. See Supp. Mat. for further details.
Metrics. To evaluate how realistically our model generates body poses and interpersonal distances, we compare the distributions of generated results against the test set in CORF4D and our collected dataset. For body pose, we compute the Fréchet Distance (FD) between the embedded pose distributions which are obtained via our body pose encoder. For interpersonal distance, we calculate FD on the per-human global translation differences.
We additionally employ CLIP-score [40] to assess semantic alignment between the generated outputs and the input text prompts. The HHOI outputs are rendered from multi-view, and CLIP-score is computed by averaging image-text cosine similarities across these views. For cases involving more than three humans, we report the success rate to evaluate the robustness of our model in multi-human settings. We acccount generation success fully generating desired number of people in 3D.
Physical plausibility of the generated HHOI is also an important factor, so we evaluate it using two metrics perriictonnnttattioonn ratio and contact distance. Penetration ration measures the ratio of mesh vertices that lie inside another mesh against the total mesh vertices. We count human vertices inside another human the object and report human-human and human-object separately. Contact distance is measured only for categories requiring contact, split into hand-contact (board, box, bucket, chair, desk) and hip-contact (bench). For each, we randomly sample 10 vertices from the relevant body parts and compute their mean distance to the object mesh
Finally, we conduct a user study to further validate the realism and text coherence of our method. Participants compare outputs from our model and baselines and select the most realistic and faithful to the prompt.
7

---

Depth Opt.
GenZI
Ours
"Poople holding the edge of the board, and moving
"People sitting next to campfire."
"People sharing umbrella.
"People are moving bucket together."
"People helping another person in wheelchair.
"People are having discussion in front of whiteboard.
Figure 3: HHOI generation result of dyadic, and multiple humans in action with model and baselines. multiple HHOI, number of humans ranges from 5. Empty result represents cases where generation failed in 10 trials. Our model can generate complex HHOIs with varying number of humans the scene, while preserving the natural social cues.
5.2 Results
Qualitative results on our model output, along with baseline outputs is shown in Fig. Since both baseline models leverage the knowledge of 2D image diffusion model to generate plausible HOIs, their performance depends heavily on the inpainting output. In scenes where the object actively used and in direct contact with human, inpainting quality degrades; the target object is frequently missing from the output, which turn leads to poor optimization quality. Additionally, as the number of people increases, the percentage of inpainting where it fails generate the whole N number of

---

Table 1: Quantitative comparison on text-guided dyadic HHOI generation. Our model shows robust performance compared baseline models in all the metrics.
Method
Body Pose FD
Distance FD +
CLIP Score t
User Study (%)
Depth Opt.
0.6834
0.4180
0.2647
20.8
GenZI
1.3500
0.3542
0.2633
18.4
Ours
0.1755
0.0689
0.2695
60.9
Table 2: Quantitative comparison of text-guided multi-human generation. Our model consistently outperforms baseline methods across all evaluation metrics, demonstrating robust performance.
Success Rate (%)
Method
CLIP Score f
3 human
4 human
5 human
Depth Opt.
0.2510
33.3
18
13
GenZI
0.2584
67.3
41.4
22
Ours
0.2685
100.0
100.0
100.0
people also increases. The multi-view consistency of each human also further deteriorates as the number of people increases.
Aside from inpainting quality, both models suffer from lack of 3D HOI and HHI knowledge. Depth Opt. generates implausible human position relative to the object, due to depth estimation error. The multi-view SMPL-X parameter optimization in GenZI sensitive to consistent body pose in each view, and can lead to poor body pose output quality. On the contrary, our generation results show high quality HHOIs in various object categories in both static and dynamic scenes.
Tab. shows quantitative results evaluating the realism on generating two people in action with object. Compared to baseline models, model achieves significantly higher and distance FD. implying our model can produce more realistic and natural HHOIs, that resemble those in real world environment. Note that due to background scene rendering, images with different HHOI generation output may have similar CLIP embeddings, which may be the cause of all 3 models producing CLIP scores in close range. This suggests FD of body pose and interpersonal distance is more reliable method in evaluating how realistic each output is. Nevertheless, our model beats the baseline model in CLIP score by slight margin. Our model also outperforms baseline models in multi-human generation. Tab. 2 shows that our model achieves higher scores in CLIP score and success rate when generating more than people, implying a more realistic and robust generation.
in body pose
We show the physical plausibility metrics of our model output against the baselines in Tab. 3. Our model outperforms baseline models contact distance metric, implying our model output achieves more precise contact with the object compared to baseline. For the numan-humaa and human- object penetration ratios, our model demonstrates superior performance in dyadic human generation, whereas Depth Opt. achieves comparable results when generating three or more people. However, this apparent performance stems from Depth Opt.'s tendency to place humans at excessively large distances from objects, a consequence of the instability in depth estimation. When combined with Human Mesh Recovery, this results low penetration metries but unrealistic spatial arrangements in HHOI scenarios. The poor contact distance scores of Depth Opt. further substantiate this observation.
5.3 Applications
The capability of our model to generate plausible human-human-obieee interactions (HHOIs) involv- ing multiple individuals from textual prompts provides foundation for range of future research directions. One promising avenue is generating multiple human motion in the presence of objects in the scene. In this way, we leverage sampled human configurations conditioning inputs for motion-in-betteennin tasks.
(DNO) [25] performs various motion-related editing tasks, using existing motion-diffusion models as a motion prior. The output of naive motion generation compared with the condition to calculate joint loss. By changing the sampling process to ODE,

---

"Two people vending
"These people togatarr
peogll
discaovion."
paople walking discuesing painting.
Figure 4: Motion in be-beeeeenigg outputs from DNO and InterGen, given naive standing pose the start frame constraint and our HHOI generation output as the end frame constraint.
Table Quantitative comparison on physical plausibility metries of our model and baseline outputs.
Metric
Method
Human
Human
4 Human
Human
Human- Human Penetration Ratio
Depth Opt.
13.91
14.06
19.35
26.16
GenZI
13.91
24.60
21.55
49.06
(x 1000)
Ours
7.90
15.68
19.82
25.69
Human-Object Penetration Ratio +
Depth Opt.
11.48
4.15
9.41
1.20
GenZI
60.71
15.64
11.93
8,91
(x 1000)
Ours
5,49
7.74
7.19
10.18
Depth Opt.
0.666
1.992
4,415
2.736
Contact Distance +
Gen7I
0.103
0.126
0.537
0.389
(m)
Ours
0.029
0.031
0.028
0.026
can obtain a motion sample from latent noise deterministically and retrieve the latent noise from the motion output. Since this process is deterministic, inversion process and optimize the latent noise to minimize the joint loss.
backpropagate the joint loss to the
Human samples generated by our model are utilized as conditional inputs to guide the optimization of the motion diffusion model InterGen [35]. By integrating object interaction cues from our sampled HHOIs with motion priors from diffusion-based models, we enable the synthesis of natural multi- human motions that exhibit plausible interactions with objects in the environment. As illustrated in Fig, 4, our method allows precise positioning of generated humans and object-specific poses that are not observed in existing models.
6 Discussion
In this paper, we propose method to model Human-Human--Ohe Interactions (HHOIs) using score-based generative models. By combining separately trained HOI and HHI models, we introduce novel sampling strategy that enables the generation of an arbitrary number of people interacting with an object. To train and evaluate our model, we construct new HHOI dataset by capturing additional samples, together with synthetic data generated by pipeline. We demonstrate that our method can synthesize realistic multi-he sunii--umann interaction with diverse objects. As shown in our applications, this capability enables downstream tasks such interaction and provides foundation for future research multi-agent embodied intelligence.
motion generation
As limitation, our score-based model cannot directly learn HHOOs from datasets dedicated solely to HOI or HHI, due distributional discrepancies human configurations. Extending our framework to effectively leverage such datasets remains open research question.
10

---

Acknowledgements
This work was supported by NRF grant funded by the Korean government (MSIT) [No. RS-2022- NR070498 and RS-2025-25396 144), and IITP grant funded by the Korea government (MSIT) [No. RS-2024-00449854,, No. RS-2025-25441833, No. RS-202I-1211143, No. RS-2025-25442338, and No.2022-0-00156]. H. Joo is the corresponding author.
References
[1] S. Baik, H. Kim, and H. Joo. Learning 3d object spatial relationships from pre-trained 2d diffusion models. ICCV, 2025, 3, 4,5
[2] F. Baradel M. Armando, S. Galaaoui, R. Brégier, P. Weinzaepfel, G. Rogez, and T. Lucas Multi-hmr: Multi-person whole-hody human mesh recovery in single shot. ECCV, 2024. 7
[3] B. L. Bhatnagar, X. Xie, I. A. Petrov, C. Sminchisescu, C. Theobalt, and G. Pons-Moll. Behave: Dataset and method for tracking human object interactions. CVPR, 2022. 2
[4] J. Black, Patel. Tesch, and Yang. Bedlam: A synthetic dataset bodies exhibiting detailed lifelike animated motion. CVPR, 2023. 2
[5] A. Bochkovskii, A. Delaunoy, H. Germain, Santos, Zhou, S. R. Richter, Koltun. Depth pro: Sharp monocular metric depth in less than a second. ICI,, 2025.
Z. Cao, H. Gao, K. Mangalam, Q.-z Cai, M. Vo, and J. Malik. Long-term human motion prediction with scene context. ECCV, 2020.
[7] R. T. Q. Chen. torchdiffeq, 2018. URL
5
[8] H. Ci, M. Wu, W. Zhu, X. Ma, H. Dong, F. Zhong, and Y. Wang. Gfpose: Learning 3d human pose prior with gradient fields. CVPR, 2023. 3
[9] C. Diller and A. Dai. Cg. Cg-hoi: Contact-guided 3d luman-object interaction generation. CVPR, 2024. 1,
[10] K. Fan, J. Tang, W. Cao, unified framework number-free text-to-motion synthesis. ECCV, 2024. 1,3
Li, J. Gong, Zhang, Y. Wang, C. Wang, and Ma. Freemotion:
[11] Z. Fan, O. Taheri, D. Tzionas, M. Kocabas, M. Kaufmann, M. J. Black, and O. Hilliges. ARCTIC: dataset dexterous bimanual hand-object manipulation. CVPR, 2023. 2.
[12] M. Fieraru, M. Zanfir, E. Oneata, A.-I. Popa, V. Olaru, and C. Sminchisescu. Three-dimensional recon- struction of human interactions. CVPR, 2020. 2,3
[13] A. Ghosh, R. Dabral, V. Golyanik, C. Theobalt, and P. Slusallek. Remos: 3d motion-conditioned reaction synthesis for two-person interactions. ECCV, 2024. 2,
[14] W. Guo, X. Bic, X. Alameda-Pineda, and F. Moreno-Noguer. Multi-person extreme motion prediction. CVPR, 2022. 2
[15] V. Guzov, A. Mir. T. Sattler, and G. Pons-Moll. Human poseitioning system (hps): 3d human pose estimation and lelfllocalization in large scenes from body-mounted sensors. CVPR, 2021. 1,2
[16] V. Guzov, Chibane, R. Marin, Y. He. Saracoglu, T. Sautler, and Pons-Moll. Interaction replica: Tracking human-object interaction and scene changes from human motion. 3DV, 2024. 1,2
[17] S. Han Joo. Learning canonicalized 3d human-object spatial relations from unbounded synthesized images. ICCV, 2023.
[18] C.-H. P. Huang, H. Yi, M. Höschle, M. Safroshkin, Alexiadis, S. Polikovsky, D. Scharstein, and M. J. Black. Capturing and inferring dense full-body human-scene contact. CVPR, 2022. 2, 4
[19] Y. Huang, O. Taheri, M. J. Black, and D. Tzionas. Intercap: Joint markerless 3d tracking of humans and objects in interaction. GCPR, 2022.
[20] C. Jonescu, D. Papava, V. Olaru, and C. Sminchiscscu. Human3.6m 6.6:: Large scale datasets and predictive methods for 3d human sensing in natural environments. TPAMI, 2008. 4
[21] N. Jiang, Z. He, Z. Wang, H. Li, Y. Chen, S. Huang, and Y. Zhu. Autonomous character interaction synthesis from text instruction. SIGGRAPH Asia, 2024.

---

[22] N. Jiang, Z. Zhang, H. Li. X. Ma. Z. Wang. Y. Chen, T. Liu, Y. Zhu, and S. Huang. Scaling up dynamic human-scene interaction modeling. CVPR, 2024. 2
[23] H. Joo, H. Liu, L. Tan, L. Gui, B. Nabbe, I. Matthews, T. Kanade, S. Nobuhara, and Y. Sheikh. Panoptic studio: A massively multiview system for social motion capture. CVPR, 2015. 1,3
[24] H. Joo, T. Simon, M. Cikara, and Y. Sheikh. Towards social artificial intelligence: Nonverbal social signal prediction a triadic interaction. CVPR, 2019.
[25] K. Karunratanakul, K. Preechakul, E. Aksan, T. Beeler, S. Suwajanakorn, and S. Tang. Optimizing diffusion noise can serve as universal motion priors. CVPR, 2024. 2, 9
[26] H. Kim, S. Han, P. Kwon, and H. Joo. Beyond the contact: Discovering comprehensive affordance for 3d objects from pre-trained 2d diffusion models. ECCV, 2024. 1, 1,3,7
[27] H. Kim, S. Baik, and Joo. David: Modeling dynamic affordance of objects using pre-trained video diffuusinn models. ICCV, 2025. 3,5
[28] J. Kim, J. Kim, J. Na, and H. Joo. Parahome: Parameterizing everyday home activitics towards 3d generative modeling of human-object interactions. CVPR, 2025. 2,
[29] N. Kulkarni, D. Rempe, K. Genova, A. Kundu, J. Johnson, D. Fouhey, and L. Guibas. Nifty: Neural object interaction fields for guided human motion synthesis. CVPR, 2024. 3
[30] J. N. Kundu, H. Buckchash, P. Mandikal, A. Jamkhandi, V. B. Radhakrishnan, et al. Cross-conditineed recurrent networks for long-term synthesis of inter-person human motion interactions. WACV, 2020. 2, 3
[31] B. F. Labs. Flux. https: g github...... cwwccooo///aaa.. http//lackfforst--aaaae worest-laastEtttt labs/flux, 2024.7 7
[32] J. Li, A. Clegg, R. Mottaghi, J. Wu, X. Puig, and C. K. Liu. Controllable human-ohject interaction synthesis. ECCV, 2024. 1,3
[33] L. Li and A. Dai. GenZI: Zero-shot 3D human-scenc interaction generation. CVPR, 2024. 3,
[34] R. Li, Y. Zhang, Y. Zhang, Y. Zhang, M. Su, J. Guo, Z. Liu, Y. Liu, and X. Li. Interdance: Reactive 3d dance generation with realistic duet interactions. arXiv preprint 2rXi::2412.1690 2024. 3
[35] H. Liang, W. Zhang, W. Li, J. Yu, and L. Xu. Intergen: Diffusion-based multi-human motion generation under complex interactions. IJCV, 2024. 1, 10
[36] D. Lim, C. Jeong, and Y. M. Kim. Mammos: Mapping multiple human motion with scene understanding and natural interactions. ICCVW, 2023. 1.3
[37] Y. Liu, C. Zhang, R. Xing, B. Tang, B. Yang, and L. Yi. Core4d: A 4d human-object-humaa interaction dataset for collaborative object rearrangement. arXiv preprint arXiv:2406. 19353, 2024. 3,
[38] L. Müller, V. Ye, G. Pavlakos, M. Black, and A. Kanazawa, Generative proxemics: prior for 3d social interaction from images. CVPR. 2024. 2.3
[39] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black. Expressive body capture: 3D hands, face, and body from a single image. CVPR, 2019.
[40] A. Radford, W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, et al. Learning transferable visual models from natural language supervision. ICMI, 2021. 5,
[41] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. CVPR, 2022. 2, 7
[42] Y. Shafir, G. Tevet, R. Kapon, and A. H. Bermano. Human motion diffusion as a generative prior. ICLR, 2024. 2,
[43] M. Shan, L. Dong. Y. Han, Y. Yao, T. Liu, L Nwogu, G.-J. Qi, and M. Hill. Towards open domain text-driven synthesis of multi-perse motions. ECCV, 2024. 1,
[44] L. Siyao, T. Gu, Z. Yang, Z. Lin, Z. Liu, H. Ding, L. Yang, and C. C. Loy. Duolando: Follower gpt with off-policy reinforcement learning for dance accompaniment. ICLR, 2024. 3
[45] W. Song, X. Zhang, S. Li, Y. Gao, A. Hao, X. Hou, C. Chen, N. Li, and H. Qin. Hoianimator: Generating text-prompt human-object animations using novel perceptive diffusion models. CVPR, 2024. 1.3
12

---

[46] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. NearIPS, 2019. 3
[47] Y. Song and Ermon. Improved techniques for training score-based generative models. NewrIPS, 2020.
[48] Y. Song. J. Sohl-Dickstein, D. P. Kingma, A. Kumar. S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2021. 3.5
[49] M. Suhail, A. Mittal, B. Siddiquie, C. Broaddus, J. Eledath, G. Medioni, and L. Sigal. Energy-based learning for scene graph generation. CVPR, 2021.
[50] M. Tanaka and K. Fujiwara. Role-aware interaction generation from textual description. ICCV, 2023. 1,3
[51] P. Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7): 1661-1674, 2011. 4
[52] T. von Marcard, R. Henschel, M. Black, B. Rosenhahn, and G. Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. E.CV,, 2018. 1, 2,4
[53] M. Wu, F. Zhong, Y. Xia, and H. Dong. TarGF; Learning target gradient field for object rearrangement. NewrIPS, 2022. 3
[54] L. Xu, Z. Song. D. Wang. J. Su, Z. Fang, C. Ding, W. Gan, Y. Yan, X. Jin, X. Yang. et al. Actformer: A gan- gan-based transformer towards general action condiiiond human motion generation. ICCV, 2023, 1,3
[55] L. Xu, X. Lv, Y. Yan, X. Jin, S. Wu, C. Xu, Y. Liu, Y. Zhou, F. Rao, X. Sheng, et al. Inter-x: Towards versatile human- interaction analysis. CVPR, 2024. 2,
[56] L. Xu, Y. Zhou, Y. Yan, X. Jin, W. Zhu, F. Rao, X. Yang, and W. Zeng. Regennet: Towards human action-reaction synthesis. CVPR, 2024. 1,3
[57] S. Xu, Z. Li, Y-X. Wang, and L.-Y. Gui. Interdiff: Generating 3d human-object interactions with physics-informed diffusion. ICCV, 2023. 1,3
[58] S. Xu, Z. Wang, Y-X. Wang, and L.-Y. Gui. Interdreamer: Zero-shot text to 3d dynamic human-object interaction. NeurlPS, 2024. 3
[59] Z. Yang, A. Zeng, C. Yuan, and Li. Effective whole-body pose estimation with two-stages distillation. ICCV, 2023. 7
[60] H. Yi, Thies, M. J. Black, X. B. Peng, and D. Rempe, Generating human interaction motions in scenes with text control. arXiv preprint arXiv:2404. 10685, 2024. 3
[61] Zhang, H. Luo, H. Yang, X. Xu, Q. Wu, Y. Shi, J. Yu, L. Xu, and Wang. Neuraldome: A neural modeling pipeline on multi-view human-object interactions. CVPR, 2023. 1,2
[62] J. Zhang, M. Wu, and H. Dong. Generative category-level object pose estimation via diffusion models. NeurIPS, 2024. 3, 4
[63] Zhang, J. Zhang, 7. Song, 7. Shi, C. Zhao, Shi, Yu, I.. Xu, and J. Wang. Hoi-m" 3: Capture multiple humans and objects interaction within contextual environment. CVPR, 2024. 2,
[64] X. Zhang, B. L. Bhatnagar, S. Starke, I. Petrov, V. Guzov, H. Dhamo, E. Pérez-Pellitero, and G. Pons- Moll. Force: Dataset and method for intuitive physics guided human-object interaction. arXiv preprint arXiv:2403.11233 2024. 2
[65] Y. Zhang, H. Yang, C. Luo. Peng. Y. Wang, and Z. Zhang. Ood-hoi: Text-driven 3d whole-body human-ohject interactions generation beyond training domains. arXiv preprint 1..1v02411.18660, 2024. 1,
[66] K. Zhao, S. Wang, Y. Zhang, T. Beeler, and S. Tang. COINS: Compositional human-scene interaction synthesis with semantic control. ECCV, 2022.
[67] Y. Zheng, R. Shao, Zhang, T. Yu, 7. Zheng, Q. Dai, and Y. Liu. Deepmulticap: Performance capture of multiple characters using sparse multiview cameras. ICCV, 2021. 1,3
[68] Y. Zheng, Y. Yang, K. Mo, J. Li. T. Yu, Liu, C. K. Liu, and L. J. Guibas. Gimo: Gaze-informed human motion prediction in context. ECCV, 2022. 1,2
[69] Z. Zheng, Yu. Y. Liu, and Q Dai. Pamir: Parametric conellconditioned implicit representation for image-based human reconstruction. TPAMI, 2021. 7
[70] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the continuity of rotation representations in neural networks. CVPR, 2019. 4
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made the abstract and introduction accurately reflect core contributions of the paper. which are also clearly summarized in the final paragraph of the introduction and supported by the experimental results.
Guidelines:
The answer NA mcans that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be pereeived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals motivation long it is clear that these goals attained by the paper.
2.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: briefly mention limitations Sec. 6, and provide detailed information supplementary material regarding training setup, data, and evaluation, enabling readers to assess potential assumptions and limitations of our approach.
Guidelines:
The answer NA mcans that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what implications would
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance the approach. example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicablle, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their hest judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Question:
Answer: [NA]
Justification: The paper does not include theoretical results or formal proofs, focuses on empirical methodology and evaluation.
Guidelines:
14

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should he numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
Tormal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data provided or not)?
Answer: [Yes]
Justification: We provide detailed descriptions of our datasct generation process, model architecture, and evaluation procedure in the supplementary material ensure the reproducibility our main results.
Guidelines:
The answer means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the contribution is a dataset and/or model, authors should describe the steps taken make
their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g.. with open- dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed- source models, it may be that access to the model is limited in some way (e.g.. registered users). but should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We intend to release the code and dataset publicly after further polishing and preparation. However, they are not included at submission time, and we do not currently provide access to ensure the release meets quality and usability standards.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines htttps://nips.cc/pbbll guiides/Codeeuubmieessooooo/ for more details.
15

---

While we encourage the release of code and data, we understand that this might not be possible. "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g.. for a new open-soure benchmark).
The instructions should contain the exact command and environment needed the results. See the NeurlPS code and data submission guidelines (httpe: : /wwps/ips...oe.o cc cc/pubicc/ Stles///oooeSuuiiiiiiooooo
reproduce
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should releasc anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify training and test details (e-g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all necessary training and evaluation details-inchleee including data splits, dataset statistics, hyperparameters, and optimizer settings in the main paper and supplementary material,
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: We do not report error bars or confidence intervals, as we did not repeat our main experiments multiple times. However, evaluate model on a fixed test set composed of thousands of samples and report averaged results with detailed evaluation settings.
Guidelines:
answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error hars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error har than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
16

---

Answer: [Yes]
Justification: We provide detailed specifications our compute resources-includeee GPU model, CPU, and memory-as well as training and inference time in the supplementary material, to ensure transparency.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, every respect, with NeurIPS Code of Ethics https:/ //pp::///eurip.....
Answer: [Yes]
Justification: Our research adheres to the NeurIPS Code of Ethics. We ensured respect for privacy and dignity. did not involve human subjects sensitive data, and considered potential societal impacts and harms. No ethical concerns arose during the development evaluation of our method.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special considcration duc laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts the work performed?
Answer: [NA]
Justification: Our paper focuses on generating plausible interacting motion between humans and objects. The generated data is synthetic, does not include any private or identifiable information, and is not directly usable for disinformation, surveillance, impersonation. Since the work does not involve downstream deployment or decision-making, and there clear path misuse societal harm, we believe the broader societal impact is minimal.
Guidelines:
The answer means that there is no societal impact of work performed. the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point out. For example, the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorith for optimizing neural networks could enable people train models that generate Deepfakes faster.
legitimate to point out that an improvement
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
17

---

11. Safeguards
Does the paper describe safeguards that been put in place responsible release data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not involve models or datasets with high risk for misuse. Our datasct is collected and processed preserve anonymity and does not include personally identifiable sensitive content.
Guidelines:
The answer means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We credit all external assets used in the paper the supplementary material, including citation of sources and license information for datasets and models.
Guidelines:
The answer NA means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
authors should state which version
asset is and, possible, include URL.
The name of the license (e.g.. c,, CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. com/ datassta has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce new model and dataset, and provide preprocessing steps data structure details the supplementary material. Although we plan to publicly release the dataset after further preparation, usage guidelines and license information are not included at submission time due to time constraints.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/codemmodel as part of their sub- missions via structured templates. This includes details about training. license. limitations,
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
18

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [Yes]
Justification: We use a publicly available survey platform human preference evaluation study and provide full instructions, interface screenshots, and compensation details in the supplementary material.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage in country data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our study does not involve interventions or risks that require IRB approval. The human subject evaluation is limited to voluntary, anonymized preference ratings using publicly available platform.
Guidelincs:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial suhmissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
19


Reviews:
[{"review_id": "rmUOkXYS3G", "reviewer": "Reviewer_KKEh", "summary": "The paper primarily focuses on generating interactions between multiple humans and a single object, implemented within a simulator. The core framework relies on score-based diffusion models and ODE sampling. \nThe approach first processes human-object interactions (HOI) and human-human interactions (HHI) separately, then integrates them into a unified framework for generating human-human-object interactions (HHOI). The model design incorporates considerations for datasets, collision avoidance, and interactive dynamics.", "questions": "In Figure 1, human feet are embedded in the floor. Is this because the scene is not used as input information? What role does the complete scene play besides decoration?\n\nIn Figure 3, humans and wheelchairs show model penetration. Does this indicate issues with the initial HOI module or the integrated HHOI optimization? The paper mentions collision handling only for human-human interactions, suggesting a stronger focus on human-human rather than human-object interactions.\n\nLine 279 claims high-quality results in dynamic scenes, but according to the description in Figure 4, HHOI only generates single final frames, with continuous frames processed by other methods, failing to demonstrate the dynamic capability of this method. It seems that objects are merely treated as fixed elements-—what significant differences does this approach have from ordinary human-scene generation methods like Narrator (ICCV 2023)?\n\nThe paper mentions object inputs ambiguously: \"object category\" in lines 40-41, \"object instance mesh\" in line 100, and \"object context\" in line 304. Is the model trained for each \"object category\", with \"object instance mesh\" input during testing? The pipeline should clarify input/output definitions.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \nThe authors have expanded upon HOI and HHI, demonstrating in-depth thinking on the HHOI problem by considering factors such as indirect implicit interactions, physical collision constraints, inconsistency constraints, and motion priors. The dataset design is comprehensive, incorporating two types: multi-view laboratory HHOI data and diversely generated HOI/HHI data from images. The experimental design is rational, and the results align with expectations.\n\nWeaknesses:\nThis paper appears to overstate its claims, particularly regarding high-quality results in dynamic scenes. While the proposed HHOI method generates single final frames, the processing of continuous frames relies on auxiliary techniques, raising concerns about its dynamic capability. Additionally, the treatment of objects as static elements rather than dynamically interacting entities further limits the perceived innovation. To strengthen this paper, it would be valuable to explicitly highlight the key distinctions between this approach and prior human-scene generation methods, such as Narrator (ICCV 2023), particularly in terms of temporal coherence and object-scene interaction.", "was_revised": true, "final_justification": "The authors have addressed most of my previous concerns. Hence, I will keep my rating.", "rebuttal": "**Q1) Clarification on the use of the term \"dynamic\"**\n\nRegarding the use of the term \"dynamic\" in Line 279, we intended it to refer to single-frame scenes where the object is actively being used and manipulated by humans, and is thus in direct contact with them. Line 270 describes our original intention. However, since \"dynamic\" is often interpreted as motion across continuous frames, we will revise the wording in the final version to better reflect our intended meaning.\n\nRegarding the motion generation component, we clarify that the result shown is conditioned on our model's output and produced by utilizing Diffusion-Noise-Optimization (CVPR 2024). This is presented as a potential downstream application rather than a core contribution of our paper. We will update the summary of contributions accordingly to avoid misunderstanding.\n\n**Q2) Differences between our work and Narrator (ICCV 2023)**\n\nNarrator generates a single frame of human-scene interaction, taking a text description and a scene mesh as input. It is trained on the single-person human-scene interaction dataset PROX (ICCV 2019), but can produce multi-person scenes via collision-based optimization.\n\nOur model produces a similar type of output, that is a single frame of multiple people interacting in the presence of an object. However, the key difference lies in the nature of the interactions: specifically, our model captures human-human interactions, while Narrator does not.\n\nBecause Narrator is trained on a single-person dataset, it can place multiple people in the scene with appropriate spacing, but it lacks the ability to generate meaningful interactions between them. For example, it may generate two people standing in front of a whiteboard, but it cannot depict them actively engaging with each other (e.g., talking, gesturing, or making eye contact). In contrast, our model can synthesize plausible configurations of body pose, orientation, and interpersonal distance that correspond to genuine human-human interactions, which is difficult to achieve when training only on single-person pose data.\n\n**Q3) Human-Object penetration**\n\n*Table 1. The ratio (×1000) of SMPL-X human vertices that penetrate the object. Higher values indicate more penetration.*\n\n| Method          | 1 Human | 2 Human | 3 Human | 4 Human | 5 Human |\n|-----------------|---------|---------|---------|---------|---------|\n| HOI             | 5.39 | -       | -       | -       | -       |\n| HHOI            | -       | 5.49 | 7.74 | 7.19 | 10.18 |\n| HHOI (Refine)   | -       | **2.78** | **2.77** | **3.67** | **4.55** |\n\n---\n\nWe perform additional experiments to analyze the cause of the human-object penetration.\n\n*Table 1* reports a quantitative evaluation of human-object penetration: for each (method, human count) combination, we randomly selected 160 scenes (16 categories × 10 scenes each) and measured the fraction of human mesh vertices that penetrate the object (scaled by ×1000). The penetration ratio for each scene is measured as the average of each human-object penetration in the scene.\n\nAs shown in *Table 1*, the penetration ratio increases as the number of humans sampled in HHOI sampling grows. This is because, in our advanced PF ODE procedure (Eq. (16) in the main paper), additional terms such as $\\mathcal{L}\\_\\text{inc}$ and $\\mathcal{L}\\_\\text{col}$ can slightly hinder the recovery of the original HOI and HHI distributions.\n\nTo address this, we introduce an additional post-processing step after sampling that optimizes translations to reduce inter-mesh penetration. The optimization objective is as follows:\n\n$\\min\\_{\\\\{t_i\\\\}\\_{i=1}^N} \\mathcal{L}\\_\\text{pen}(\\\\{t_i\\\\}\\_{i=1}^N) + \\lambda\\sum\\_{i=1}^N ||t\\_i||\\_2^2$,\n\nwhere $t\\_i$ is translation vector of human $i$, $\\mathcal{L}\\_\\text{pen}(\\\\{t_i\\\\}\\_{i=1}^N)$ is total penetration between all meshes, and $\\lambda$ is weight factor for regularization term.\n\nThe last row of *Table 1* shows that the human-object penetration is greatly improved after this optimization.\n\n**Q4) Role of the background scenes**\n\nOur model considers the relative configuration to the target object, and other parts of the scene, including the floor, are not considered. The surrounding environment is shown purely for aesthetic purposes to provide contextual understanding of the scene.\n\n**Q5) Clarification regarding the role of objects in the model input/output**\n\nOur model’s input is text description only, and object meshes are not provided as input. However, because the training data (both HOI data and HHI data) is organized per specific mesh instance, we wrote “object instance mesh is provided” on Line 101. We will revise the inputs/outputs description for clarity."}, {"review_id": "rRE4WzAZzG", "reviewer": "Reviewer_YRDq", "summary": "This work introduces a novel framework for generating Human-Human-Object Interactions (HHOIs), addressing the scarcity of diverse multi-person interaction data. The authors contribute a new 3D dataset captured with a multi-camera system, as well as a synthetic HHOI dataset generated using pretrained image diffusion models, especially targeting scenes that are difficult to capture in real life.\n\nA score-based diffusion model unifies both real and synthetic data, enabling semantically grounded, realistic generation of dyadic and multi-person HOIs, conditioned on object categories and text descriptions. The approach is scalable to more complex social scenarios involving multiple people and shared objects.", "questions": "1. Why is the pose embedding dimension set to 126? So, where is the human root translation represented in the model?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### **Strength**  \n1. Overall, the paper is well-structured and easy to follow.\n2. The proposed method supports the generation of interactions involving 2 to 5 humans, and demonstrates good generalization capabilities to unseen scenes and textual descriptions.\n3. Both the qualitative and quantitative results appear to outperform the baselines reasonably well.\n\n\n\n### **Weakness** \n1. While the visual results are promising on semantic-level, Figure 1 reveals noticeable interpenetration between humans, objects, and the ground.\n2. There is no evaluation metric provided to quantify human-object interpenetration or contact accuracy, which are crucial for assessing the physical plausibility of the generated interactions.\n3. Although the interaction categories in the CORE4D dataset are relatively limited, it contains dynamic interaction sequences, whereas the dataset used in this work focuses on static interactions. As a result, the two datasets differ in terms of difficulty and research contribution. It would be beneficial to include a comparative table that provides a statistical overview of the datasets, which could help clarify their respective scopes and highlight the novelty of the proposed dataset.", "was_revised": true, "final_justification": "Thank you to the authors for the detailed responses. Most of my concerns have been addressed, and I will raise my rating to BA.", "rebuttal": "*Table 1. The ratio (×1000) of interpenetrating vertices among SMPL-X humans. Lower values indicate less penetration.*\n\n| Method        | 2 Human | 3 Human | 4 Human | 5 Human |\n|---------------|--------:|--------:|--------:|--------:|\n| Ours          |   7.90  |   15.68  |   19.82  |  25.69  |\n| Ours+Refine |   **3.22**  |  **6.94**  |   **7.37**  |   **8.25**  |\n| GenZI |   13.91  |   24.60  |   21.55  |   49.06  |\n| Depth Opt. |   13.91  |   14.06  |   19.35  |   26.16  |\n\n---\n\n*Table 2. The ratio (×1000) of SMPL-X human vertices that penetrate the object. Lower values indicate less penetration.*\n\n| Method        | 2 Human | 3 Human | 4 Human | 5 Human |\n|---------------|--------:|--------:|--------:|--------:|\n| Ours          |   5.49  |   7.74  |   7.19  |  10.18  |\n| Ours+Refine |   **2.78**  |  **2.77**  |  **3.67**  |   4.55  |\n| GenZI |   60.71  |   15.64  |   11.93  |   8.91  |\n| Depth Opt. |   11.48  |   4.15  |   9.41  |   **1.20**  |\n\n---\n\n*Table 3. The contact distance (m) between a SMPL-X human and an object. Lower values indicate more precise contact.*\n\n| Method        | 2 Human | 3 Human | 4 Human | 5 Human |\n|---------------|--------:|--------:|--------:|--------:|\n| Ours          |   **0.029**  |   **0.031**  |   **0.028**  |  **0.026**  |\n| Ours+Refine |   0.032  |   0.038  |   0.041  |   0.038  |\n| GenZI |   0.103  |   0.126  |   0.537  |   0.389  |\n| Depth Opt. |   0.666  |   1.992  |   4.415  |   2.736  |\n\n---\n\n**Q1) Evaluation for physical plausibility**\n\nOur evaluation primarily focuses on human-human coordinated interactions, assessing how realistic the generated body poses and configurations are. However, to address concerns about the absence of explicit physical-plausibility evaluation, we define two additional metrics, *penetration ratio* and *contact distance*, and conduct further experiments.\n\nFor human-human penetration, we evaluate each human mesh by measuring the ratio of its vertices that lie inside another human mesh. This ratio is then averaged over all individuals in the scene. Similarly, we evaluate each human mesh by measuring the ratio of its vertices that lie inside the object for human-object penetration. *Table 1* reports the human-human penetration ratio, and *Table 2* reports the human-object penetration ratio.\n\nFor the contact distance, we evaluate six categories: five hand-contact categories (board, box, bucket, chair, and desk) and one hip-contact category (bench). For each case, we randomly sample 10 vertices from the corresponding body region and compute their average distance to the object mesh. The final contact distance is obtained by averaging these per-case distances. *Table 3* reports the contact distance.\n\nWe randomly sampled 10 scenes per (category, human-count) combination.\n\n*Table 1, 2, 3* show that our model outperforms competitors in terms of penetration and contact even before refinement (cover this in **Q2**). Although the penetration ratio for Depth Optimization looks favorable when sampling three or more humans, this is an illusion: Depth Optimization often places humans far from the object, thereby lowering the penetration ratio. This becomes clear in the contact distance results in *Table 3*. Especially, for the 5 human case, Depth Optimization’s contact distance is about 100 times larger than that of our method.\n\n**Q2) Human-Ground penetration**\n\nWe would like to note that the surrounding environment, including the ground, is not used as input to our model, and it is used as decorative content for rendering. We'll revise this to avoid confusion.\n\n**Q3) Human-Object penetration**\n\nRegarding human-object penetration, the additional terms $\\mathcal{L}\\_\\text{inc}$ and $\\mathcal{L}\\_\\text{col}$ in our advanced PF ODE procedure (Eq. (16) in the main paper) may slightly hinder the recovery of the original HOI and HHI distributions. To address this, we introduce an additional simple post-processing step after HHOI sampling that optimizes translations to reduce inter-mesh penetration. The optimization objective is as follows:\n\n$\\min\\_{\\\\{t_i\\\\}\\_{i=1}^N} \\mathcal{L}\\_\\text{pen}(\\\\{t_i\\\\}\\_{i=1}^N) + \\lambda\\sum\\_{i=1}^N ||t\\_i||\\_2^2$,\n\nwhere $t\\_i$ is translation vector of human $i$, $\\mathcal{L}\\_\\text{pen}(\\\\{t_i\\\\}\\_{i=1}^N)$ is total penetration between all meshes, and $\\lambda$ is weight factor for regularization term.\n\nAs shown in *Tables 1* and *Table 2*, the post-processing noticeably reduces penetration. The contact distance in *Table 3* increases slightly, but the change is very small and therefore negligible.\n\n**Q4) Dataset comparison & statistics**\n\n*Table 4. Comparison of CORE-4D and our own collected dataset.*\n\n| Dataset       | Frame type | # of categories | Non-contact interaction objects |\n|---------------|--------:|--------:|--------:|\n| CORE-4D [36] |   Sequence  |  6  |  X  |\n| Capture (ours) |   Single  | 11  |   O  |\n| Synthetic (ours) |   Single  | 3  |   O  |\n\n---\n\n*Table 4* compares the features of our newly collected data with those of CORE-4D. \"Frame type\" refers to whether the dataset is composed of single frame or sequential motion. \"# of categories\" refers to how many categories of object are present in the dataset. \"Non-contact interaction objects\" refer to items such as \"painting\", which are involved in the scene context but do not require physical contact during the interaction.\n\nWe captured our own dataset primarily due to the limited object categories in CORE-4D, which includes only a few manipulative objects such as \"box\" or \"board\". In contrast, our captured dataset incorporates a wider variety of non-contact objects such as \"campfire\" and \"whiteboard\". For objects that are difficult to capture in an indoor environment, we employed a synthetic data generation pipeline.\n\n**Q5) Regarding human pose representation**\n\nThe pose embedding is 10D, not 126D (see Line 126). The 126D refers to the original dimensionality of the body pose, which comes from the 6D rotation representation of 21 body joints (see Line 121). The root orientation and translation of human $\\mathcal{H}$ are represented by $\\mathbf{R}\\_{\\mathcal{H}}$ and $\\mathbf{t}\\_\\mathcal{H}$ as in Eq. (1) of the main paper."}, {"review_id": "roQKhNAgnd", "reviewer": "Reviewer_w3tN", "summary": "This paper proposes a method to generate human-human-object interactions (HHOIs), which is a novel and challenging task in the community. The authors decompose the task into human-human interaction (HHI) and human-object interaction (HOI), and propose an advanced sampling strategy with an inconsistency loss and a collision loss to combine the results. To train the model to generate more natural HHOIs, the authors also build a new dataset captured using multi-view systems. The qualitative results demonstrate that the method can produce plausible HHOIs.", "questions": "Please address the concerns raised in the weaknesses section, particularly those related to the sampling strategy, loss functions, training process, dataset, and evaluation metrics.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Decomposing HHOI into HOI and HHI reduces the training difficulty and enables the generation of arbitrary individuals.\n- The proposed Advanced Sampling strategy considers how to integrate multiple results and constrains their interrelations to form a coherent HHOI.\n- The proposed dataset may be valuable for future research on HHOI.\n\n\nWeaknesses\n- The summary of contributions may need to be reorganized. This paper indeed has meaningful contributions, such as the decomposition of HHOI, the joint sampling of HOI and HHI, and the proposed dataset. However, the current summary does not accurately reflect these contributions. For example, Text-to-HOI and Text-to-HHI should not be claimed as contributions of this paper, as the authors did not make any technical improvements in this area.\n- What is the fundamental difference between HHOI and prior HHI tasks? The authors should clearly articulate this in the introduction. Since HHOI is a relatively new task, clarifying the distinctions would help readers better understand its importance and highlight the contributions of this work.\n- Why not use VPoser [A] for pose embedding? Both approaches essentially aim to achieve the same goal.\n\n     [A] \"Expressive body capture: 3d hands, face, and body from a single image.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n- Why is scale in the human model introduced? Why not normalize all input objects to a unified scale instead? This could simplify the problem and better align with real-world applications.\n- What is the exact formulation of the variance loss? A clear mathematical description is necessary.\n- The proposed method samples N + N(N–1) humans. How is the final result selected from these candidates?\n- How is the directed acyclic graph (DAG) constructed? How does the model determine which individuals are involved in HHI with each other?\n- Does combining HOI and HHI to form HHOI require further training? What datasets are used to train the HOI and HHI components respectively?\n- Collision loss cannot capture fine-grained interactions. Since some interactions may not involve mesh-level penetration but could still result in overlapping bounding boxes, could the collision loss penalize these correct interactions?\n- The constructed real-world dataset contains only 5,078 frames, which is relatively small in scale. Moreover, current reconstruction methods cannot accurately model collaborative interaction scenarios, which may result in low-quality generated data. How much does this dataset actually contribute to the improvement of the proposed method?\n- The current evaluation metric, Fréchet Distance (FD), is insufficient to assess the quality of interactions.", "was_revised": true, "final_justification": "I believe most of concerns from other reviewers are also addressed. The dataset might be useful for the community.", "rebuttal": "**Q1) FD is insufficient to assess the quality of interactions**\n\nOur evaluation primarily focused on human-human coordinated interactions, assessing how realistic the generated body poses and configurations are. However, to address explicit physical-plausibility evaluation, we define two additional metrics, *penetration ratio* and *contact distance*, and conduct further experiments.\n\nFor human-human penetration, we evaluate each human mesh by measuring the ratio of its vertices that lie inside another human mesh. This ratio is then averaged over all individuals in the scene. Similarly, we evaluate each human mesh by measuring the ratio of its vertices that lie inside the object for human-object penetration.\n\nFor the contact distance, we evaluate six categories: five hand-contact categories (board, box, bucket, chair, and desk) and one hip-contact category (bench). For each case, we randomly sample 10 vertices from the corresponding body region and compute their average distance to the object mesh. The final contact distance is obtained by averaging these per-case distances.\n\nDue to space constraints, please refer to the *Table 1, 2, 3* shown in the rebuttal for **Reviewer YRDq** for evaluation results on additional metrics.\n\n**Q2) Whether the collision loss penalizes correct interactions**\n\n*Table 1. The ratio (×1000) of interpenetrating vertices among SMPL-X humans. Lower values indicate less penetration.*\n\n| Method        | 3 Human | 4 Human | 5 Human |\n|---------------|--------:|--------:|--------:|\n| AABB-based (orignal version) |  15.68  |   **19.82**  |  **25.69**  |\n| Capsule-based |   **15.23**  |  25.66  |   28.72  |\n\n---\n\n*Table 2. The ratio (×1000) of SMPL-X human vertices that penetrate the object. Lower values indicate less penetration.*\n\n| Method        | 3 Human | 4 Human | 5 Human |\n|---------------|--------:|--------:|--------:|\n| AABB-based (orignal version) |  **7.74**  |   **7.19**  |  10.18  |\n| Capsule-based |   7.77  |  8.41  |   **8.37**  |\n\n---\n\n*Table 3. The contact distance (m) between a SMPL-X human and an object. Lower values indicate more precise contact.*\n\n| Method        | 3 Human | 4 Human | 5 Human |\n|---------------|--------:|--------:|--------:|\n| AABB-based (orignal version) |  0.031  |   **0.028**  |  **0.026**  |\n| Capsule-based |   **0.026**  |  0.028  |   0.030  |\n\n---\n\nAs mentioned in Lines 179-182, the collision loss is applied only between humans who are not explicitly specified to have an HHI (that is, only between those without \"fine-grained interactions\"), so its impact is minimal. However, to address reviewer’s concern and to eliminate even this minor influence, we test a new collision loss that approximates each human as a set of capsules and computes collisions accordingly. The process is as follows:\n\nPreliminary: The mathematical definition of a capsule is the set of all points equidistant from a line segment. Specifically, a capsule consists of a cylinder with radius $r$ around the central axis segment and two hemispheres with radius $r$ attached to the ends of the cylinder’s top and bottom circular faces.\n\n1) We define the edges formed by the 21 SMPL-X joints as the central axis segments of each capsule. Additional edges are introduced to represent the hands and head. There are a total of 24 edges.\n\n2) We optimize the radius $r_i$ for each capsule $C_i$. We use the same dataset (mentioned in Line 125) employed when training the pose encoder/decoder as ground truth, and use Chamfer Distance as the objective function.\n\n3) Given $(\\mathbf{R}, \\mathbf{t}, \\mathbf{s}, \\theta)$, we can now compute the capsule-based human representation with minimal computation.\n\n4) The collision between two capsules $C\\_i$ and $C\\_j$ is approximated as $\\min(0, r\\_i + r\\_j - d\\_{i,j})$, where $d\\_{i,j}$ is the distance between their central axis segments.\n\n5) The collision between two humans $\\mathcal{H}\\_1$ and $\\mathcal{H}\\_2$ is defined as the sum of pairwise capsule collisions between all capsules from each human.\n\n---\n\nTo compare our original AABB (axis‑aligned bounding‑box)–based collision loss with the newly proposed capsule‑based collision loss, we sampled HHOI scenes with each method and evaluated them in terms of penetration and contact quality.\n*Table 1* reports the average human-human penetration ratio, *Table 2* reports the average human-object penetration ratio, and *Table 3* reports the average human-object contact distance.\n\nThe results show negligible differences between the two losses, indicating that, contrary to reviewer’s concern, the AABB‑based collision loss does not heavily penalize correct interactions.\n\n**Q3) Does the proposed dataset contribute to the improvement?**\n\n*Table 4. Comparison of our collected dataset.*\n\n| Dataset | # of categories | # of samples | Non-contact interaction objects |\n|---------------|--------:|--------:|--------:|\n| CORE-4D |   5  |  8160  |  X  |\n| Capture |   11  | 18860  |   O  |\n| Synthetic |   3  | 299  |   O  |\n\n---\n\nFirst, to avoid confusion, we clarify that all samples belonging to each object category are collected from only one source — either CORE‑4D, our own capture system or synthetic generation.\n\nBecause prior work on HHOI is scarce, publicly available datasets are almost nonexistent. CORE‑4D focuses mainly on scenarios where two people carry a single object together. However, we captured a new diverse dataset including scenarios in which HHOI occurs even without direct contact. Its significance lies not in simply boosting HHOI generation performance on models, but rather in enabling the modeling of HHOI scenarios that existing datasets cannot.\n\n*Table 4* compares the features of each collection pipeline. \"# of categories\" refers to how many categories of object are present in the dataset. \"# of samples\" refers to the total number of samples, including both HOI and HHI samples. \"Non-contact interaction objects\" refer to items such as \"painting\", which are involved in the scene context but do not require physical contact during the interaction. For objects that are difficult to capture in an indoor environment, we employed a synthetic data generation pipeline.\n\nIn addition, to address the limited size of our current dataset, we plan to expand it by collecting more categories and samples in the future.\n\n**Q4) Does combining HOI and HHI to form HHOI require further training?**\n\nWe emphasize that our advanced sampling enables sampling from the joint distribution of two different score-based models without additional training.\n\n**Q5) What datasets are used to train the HOI and HHI components respectively?**\n\nIn brief, we train the HOI diffusion and HHI diffusion models using a combination of existing datasets (e.g., CORE-4D), data captured with our own system, and synthetic data generated using off-the-shelf models. Detailed information is provided in Sec. 4 of the main paper and Sec. A of the supplementary material.\n\n**Q6) Exact formulation of the variance loss**\n \nIt follows the standard definition of variance: $\\mathrm{Var}(\\\\{x\\_i\\\\}\\_{i=1}^N) = \\sum\\_{i=1}^N \\frac{||x\\_i - \\mu||^2\\_2}{N}, x\\_i \\in \\mathbb{R}^n$.\n\n**Q7) How is the final result selected from $N + N(N–1)$ candidates?**\n\nExactly $N+2M$ humans are sampled (where $M$ is the number of HHI pairs; see Lines 155-156 of the paper for the precise formulation). Since we mitigate the discrepancy between duplicate humans using the inconsistency loss, we finally take the average of the duplicate humans' $(\\mathbf{R}, \\mathbf{t}, \\mathbf{s}, \\theta)$.\n\n**Q8) How is the DAG constructed?**\n\nWe first create a pickle file for HHOI sampling. The pickle generation program takes as input the number of humans, text prompts, and the HHI pairs (i.e., explicitly specifying that human $\\mathcal{H}_i$ and $\\mathcal{H}_j$ are involved in an HHI). It automatically verifies whether the specified HHI pairs form a valid DAG via a topological sort.\n\n**Q9) Why not use VPoser for pose embedding?** \n\nThere are two main reasons why we trained a new body pose embedding network:\n\n1) We want to keep the dimensionality of the body pose embedding low.\n\n2) We aim to maintain a simple network architecture.\n\nIn our experience, when the dimensionality of the body pose embedding exceeds 20D, HHOI diffusion shows a performance drop (VPoser uses a 32D embedding). Moreover, keeping the pose embedding network simple prevents a performance drop during the PF ODE process of our advanced HHOI sampling.\n\n**Q10) Why adopt a human scale instead of normalizing all objects to a unified scale?** \n\nThank you for the valuable feedback! In fact, we consider handling humans of different scales as part of our future work, which is why we introduced the concept of human scale. However, regardless of whether human scale is used, reviewer’s suggestion to normalize object scales into a unified scale appears to be a promising approach for real-world applications. Since there is no fundamental difference, switching to that version would not be difficult and is worth trying.\n\n**Q11) Text-to-HOI and Text-to-HHI should not be claimed as contributions?**\n\nWe emphasize that our contribution also lies in modeling HOI and HHI using a score-based diffusion approach, since these models serve as the components enabling coherent HHOI sampling. While previous methods have addressed these two tasks independently, merging the two individual models into a single model has been a challenge due to each model's unique structure.\n\n**Q12) What is the fundamental difference between HHOI and prior HHI tasks?**\n\nCompared to prior HHI tasks, HHOI explicitly incorporates objects, making the interaction scenarios more contextually concrete. For instance, a generic \"two people talking\" interaction can vary significantly in terms of interpersonal distance, body orientation, and poses, depending on whether they are sitting face-to-face at a table or side-by-side on a couch. By conditioning on the object context, our HHOI reduces such ambiguity."}, {"review_id": "12TSi28Q2f", "reviewer": "Reviewer_hSov", "summary": "This paper introduces Human-Human-Object Interactions (HHOI) to model people interacting around an object. It introduces a new dataset and a score-based diffusion model to generate realistic multi-person scenes from object and text inputs, outperforming single-person HOI methods and extending to multi-person interactions.", "questions": "n/a", "limitations": "I believe HHOI has unique value for multiple people interaction in complex scenarios, especially in embodiment AI. Although it raises concerns about potential biases in synthetic data, the novel HHOI framework and generative approach warrant recognition.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The proposal of HHOI generation is novel.\n\n2. Develops a novel real-world HHOIs dataset and employs image generative models for synthetic data, enabling coverage of dyadic to multi-person interaction scenarios.\n\n3. Utilizes a score-based diffusion framework to merge HOI and HHI modalities, allowing end-to-end generation of interactions from object inputs and text descriptions, scalable from 2 to multiple humans.\n\nMajor Weaknesses:   \n\n1. Since HHOI is composed of HOI and HHI, it is essential to respectively compare the HOI and HHI results generated by HHOI with those from dedicated HOI and HHI methods. For HHI, previous approaches suitable for comparison include diffusion-based methods such as ComMDM (ICLR 2024), InterGen (IJCV 2024), in2IN (CVPRW 2024), and the transformer-based InterMask (ICLR 2025). For HOI, InterDiff(ICCV 2023), HOI-Diff(CVPRW 2025), CHOIS(ECCV 2024) can be selected for comparation. Metrics about generated motion quality such as R-precision, MM-Dist, MModality and Diversity can be evaluated. Interaction quality metrics mentioned in CHOIS also can be used for HHOI for evaluation. Additionally, can the HOI diffusion and HHI diffusion be replaced with other generative models? For example, will replacing the generated sequence from HHI diffusion with InterMask or ComMDM improve quality?\n\n2. Although the advanced sampling technique includes collision loss to minimize human-body intersections, in figure 3, meshes still exhibit unnatural interaction, particularly in hand movements.\n\nMinor Weaknesses:   \n1. Synthetic data may introduce bias due to differences from real-world environments, as pretrained models may ignore spatial coherence or collision constraints in 3D space. The real-world dataset is predominantly dyadic, with multi-person synthetic data (3–5 people) relying on generated priors. This may affect robustness in complex scenarios.  \n\n2. The paper does not explicitly state the proportion of synthetic data to real data in the dataset.  \n\n3. When multiple people interact simultaneously, such as three people stacking their hands together when they touch the object, can HHOI generate such complex scenarios?", "was_revised": true, "final_justification": "The authors have addressed most of my concerns, I recommend borderline accept.", "rebuttal": "**Q1) Compare our HOI and HHI results with dedicated HOI and HHI methods**\n\nWe clarify that our primary contribution of our paper lies in generating static HHOI poses, not dynamic HHOI motion generation. However, since we proposed HHOI motion generation as an application in Sec. 5, it is also meaningful to compare it with the motion generation methods. We perform additional experiments and evaluations on motion generation following the reviewer's guidance.\n\nWe generate HHOI motions using the method proposed in Sec. 5 and then split each sequence into an HHI motion and HOI motions. Because a single HHOI motion can yield two HOI motions, we randomly select one of them. Then, we compare them with ComMDM and InterGen for HHI, and HOI-Diff and InterDiff for HOI.\n\nWe report the *Diversity*, *MM-Dist*, and *MModality* as evaluation metrics. *Diversity* measures the total diversity of the generated motions by computing the average distance between 300 random sampled motion pairs. *MM-Dist* is the average Euclidean distance between the input text embedding and the output motion embedding.  *MModality* similarly measures diversity between motions generated from the same text prompt.\n\nFor HHI motion, we evaluate on 16 object categories. For HOI motion, we evaluate on 4 object categories—box, chair, desk, and monitor—because these categories are shared by our dataset and the BEHAVE dataset on which HOI-Diff and InterDiff were trained. In both cases, we generated 10 scenes per object category to compute metrics.\n\n*Table 1. Quantitative evaluation of HHI motion against existing methods.*\n\n| Method   |  Diversity ($\\uparrow$) | MM-Dist ($\\downarrow$) | MModality ($\\uparrow$) |\n|---------------|--------:|--------:|--------:|\n| ComMDM | 1.70   | 4.10  | **1.26** |\n| InterGen |   **1.82** |  **3.82**   | 1.02 |\n| Ours | 1.73 |  4.00 | 1.26 |\n\n---\n\n*Table 2. Quantitative evaluation of HOI motion against existing methods.*\n\n| Method       |  Diversity ($\\uparrow$) | MM-Dist ($\\downarrow$) | MModality ($\\uparrow$) |\n|---------------|--------:|--------:|--------:|\n| HOI-diff |  **5.09**   | **16.06**  | **3.39** |\n| InterDiff |  4.84  |  16.68  | 3.08 |\n| Ours | 3.34 | 16.12  | 2.89 |\n\n---\n\nOur results are comparable to those of existing motion generation methods. For HHI motion generation, our method outperforms ComMDM in Diversity and MM-Dist and surpasses InterGen in MModality. For HOI motion generation, we achieve a better MM-Dist than InterDiff.\n\n**Q2) Can the HOI diffusion and HHI diffusion be replaced with other generative models?**\n\nThey cannot be replaced with other types of models. We emphasize that our contribution also lies in modeling HOI and HHI using a score-based diffusion approach, since these models serve as the components enabling coherent HHOI sampling. While previous methods have addressed these two tasks independently, merging the two individual models into a single model has been a challenge due to each model's unique structure.\n\n**Q3) Unnatural hand movements**\n\nOur work focuses on coordinated interaction among multiple humans for a single object, rather than precise hand-object contact. So, we only model SMPL-X body poses and do not include hand poses. The unnatural hand movements stem from this design choice. As future work, we plan to incorporate hand modeling to address this issue.\n\n**Q4) Regarding the phrase \"multi-person synthetic data (3–5 people)\"**\n\nWe emphasize that all our datasets (including the captured data, CORE-4D, and synthetic data) consist solely of dyadic scenarios. The ability of our HHOI diffusion model to generate multi-human (>2) interactions does not stem from the presence of multi-human data, but rather from our advanced sampling strategy.\n\n**Q5) Synthetic data may introduce bias**\n\nOur use of synthetic data is limited to scenarios that are difficult to capture in studio settings (e.g., two people riding a motorcycle together), as mentioned in Sec. 4.\n\n**Q6) The proportion of synthetic data to real data**\n\nDataset we used in training consists of data from CORE-4D, multi-view capture system, and from our synthetic pipeline. A total of 4080 HOI and 4080 HHI samples are used from CORE-4D, 9430 HOI and 9430 HHI samples from our multi-view capture system, and 159 HOI and 140 HHI samples from our synthetic pipeline. That is, our dataset consists of 99% real data and 1% synthetic data.\n\n**Q7) Can HHOI diffusion generate scenarios where multiple people interact simultaneously?**\n\nIt is possible, but as mentioned in **Q3**, precise hand poses are not modeled."}]

Author rebuttal:

Meta-review: The reviewers agreed that the problem addressed in the paper is meaningful, the proposed method and dataset are valuable, and the results are encouraging. The rebuttal satisfactorily resolved most of the reviewers’ concerns, and acceptance was recommended by all. The authors are encouraged to enhance the final version of the paper by taking into account the reviewers’ suggestions.
Decision: Accept (poster)