Title: HOI-Dyn: Learning Interaction Dynamics for Human-Object Motion Diffusion
Venue: NeurIPS 2025
Accepted: yes
Keywords: Human-object motion generation

Interaction dynamics

Causal consistency
Abstract: Generating realistic 3D human-object interactions (HOIs) remains a challenging task due to the difficulty of modeling detailed interaction dynamics. Existing methods treat human and object motions independently, resulting in physically implausible and causally inconsistent behaviors. In this work, we present HOI-Dyn, a novel framework that formulates HOI generation as a driver-responder system, where human actions drive object responses. At the core of our method is a lightweight transformer-based interaction dynamics model that explicitly predicts how objects should react to human motion. To further enforce consistency, we introduce a residual-based dynamics loss that mitigates the impact of dynamics prediction errors and prevents misleading optimization signals. The dynamics model is used only during training, preserving inference efficiency. Through extensive qualitative and quantitative experiments, we demonstrate that our approach not only enhances the quality of HOI generation but also establishes a feasible metric for evaluating the quality of generated interactions. Project website: https://wulin97.github.io/hoi-dyn


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel framework that models 3D human-object interactions as a driver-responder system using a lightweight transformer-based dynamics model to predict object reactions to human motions, combined with a residual-based dynamics loss for consistency.\n\nSTRENGTHS AND WEAKNESSES:\n#### Strength\n1. One notable advantage is the innovative concept of modeling humans as \"drivers\" and objects as \"responders,\" which proves superior to simultaneous generation of human and object motions. This driver-responder framework better captures the causal dependency in human-object interactions, enabling more logically consistent and physically plausible motion generation by explicitly defining how human actions drive object reactions.\n\n2. Quantitative comparisons demonstrate that the proposed method outperforms previous baselines, with significant improvements observed in key metrics with statistical significance, which validates the effectiveness and superiority of the approach.\n\n#### Weakness\n\n1. The absence of an ablation study on the impact of using the interaction dynamics loss during training. \n\n2. Given that physics-based approaches are widely adopted for capturing physical plausibility in human-object interactions, the paper should give comparison or discussion on using physics simulators to model object reactions to human actions.\n\n3. In the demonstration video, noticeable foot sliding artifacts are observed. Additionally, there are instances where the object exhibits premature movement before human contact—for example, the cases starting at 3s and 11s.\n\nQUESTIONS:\n1. Unclear writing, especially regarding the workflow in Figure 1(b), which is inadequately described in the paper. Can you explain more about \"causality alignment\"?\n\n2. What is error feedback in L125?\n\nLIMITATIONS:\nWeakness #2 should be disscussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a transformer-based dynamics model that predicts object movements induced from human movements during interaction. The dynamics model is used to calculate dynamics loss when training a conditional diffusion-based generative model which outputs human object interaction from object geometry/text/waypoint conditional input. The paper demonstrates the dynamics-based approach improves human-object interaction quality (consistency, contact, natural and plausible human/object movements, etc).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Defining interaction dynamics and consistency as object movement induced by human movement is innovative and intuitive. The integration of dynamics loss into CHOIS's HOI learning diffusion model is clever, introducing no additional inference overhead while enabling conditional generative models to learn interaction dynamics.\n\n2. Defining dynamics as residuals is effective and the paper demonstrates that such formulation mitigates possible errors from the dynamics model, making the approach plausible and robust.\n\n3. The method shows improved results compared to baselines, with extensive experiments conducted across various settings for the dynamics model.\n\nWeaknesses\n\n1. The result videos exhibit common artifacts where objects drift when not in contact with humans (at demo video 00:02, 00:12, 00:16). A contact-aware term that prevents object moving when human-object interaction has not yet occurred would improve realism.\n\n2. While L181 states that the dynamics model is locally smooth and temporally homogeneous, the local smoothness assumption requires more explanation. The claim that \"local smoothness reflects the continuity of physical motion\" needs more detail on how the dynamics model ensures continuity.\n\n3. It would be better if performance improvement is analyzed per action type (e.g., \"put,\" \"kick\") in Table 1. In the video, the CHOIS baseline appears effective for hand-based interaction (carry, push, etc) but degrades significantly for foot-based interactions compared to the paper's method, and it would be clear for readers if this difference is demonstrated numerically. \n\n(minor weakness for better clarity)\n\n4. Please include input text in videos so readers can evaluate whether the output motion is semantically aligned with the given text.\n\nQUESTIONS:\nThis paper introduces interaction dynamics through a clever and novel approach using dynamics models and residual-based dynamics loss. The method achieves improved HOI motion synthesis results compared to SOTA baselines, which is demonstrated through comprehensive experiments. Despite the presence of motion artifacts (Weakness 1), the work represents a valuable advancement.\nHowever, it would be better if: (1) more clarification that the assumptions made in using the dynamics model for calculating dynamics loss (Weakness 2), and (2) action-specific baseline comparisons to more clearly demonstrate the advantages of the dynamics-based approach (Weakness 3).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a novel driver–responder perspective for human-object interaction (HOI) generation, where human actions drive object responses. Central to this approach is the concept of interaction dynamics, which governs how objects naturally react to human motion—implicitly handling contact and ensuring physically plausible interactions.\n\nSTRENGTHS AND WEAKNESSES:\n### **1. Strength:**\n- The entire article is basically complete and the experiments are adequate.\n- The overall qualitative and quantitative results are satisfactory, showing clear improvements over CHOIS across several metrics and visual examples.\n- Interaction Dynamics modeling perform well on the object pose correction.\n\n### **2. Weakness:**\n- The related work section is somewhat rough and lacks precision in categorizing and positioning recent HOI generation methods. For instance, HOI-Diff and CG-HOI are concurrent works, so the statement that CG-HOI is further enhanced is inaccurate and potentially misleading. Additionally, in Section 2.3, I would not categorize CHOIS under joint human-object prediction. Similar to OMOMO, CHOIS relies heavily on predefined object trajectories, effectively simplifying the task. A more appropriate categorization would be to group methods like HOI-Diff, CG-HOI, InterDreamer [1], THOR, HIMO [2], and ChainHOI [3] together, as they focus on jointly modeling human-object interactions without access to future object motion. OMOMO, CHOIS and [4] should be categorized into Object trajectory-guided HOI generation.\n\n\n[1] InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction. Sirui Xu et al.\n\n[2] HIMO: A New Benchmark for Full-Body Human Interacting with Multiple Objects. Liang Xu et al.\n\n[3] ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generation. Ling-An Zeng\n\n[4] Human-Object Interaction from Human-Level Instructions. Zeng Wu et al.\n\nQUESTIONS:\n1. One point of curiosity is how the model would perform if the object waypoint constraints were removed. Since the current framework relies on predefined waypoints to guide HOI generation, it would be insightful to evaluate whether the interaction modeling still holds up without such guidance. This could better demonstrate the model’s true capacity for reasoning about physical causality and interaction dynamics.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces HOI-Dyn, a framework for generating more realistic 3D human-object interactions. It models interactions as a \"driver-responder\" system, where human actions drive object reactions. The core uses a lightweight Transformer model to predict object responses and incorporates the loss of a residual dynamic for consistency. This dynamics model is only used during training for efficient inference. Experiments show that HOI-Dyn improves HOI generation quality and provides a new evaluation metric.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The idea of modeling HOI generation as a reactive system is interesting and offers a novel perspective on the problem.\n2. The assumption of an active-passive system is very reasonable because objects shouldn't move without contact.\n\nWeaknesses:\n1. Can this modeling approach be extended to settings with multiple objects or multiple people?\n2. Should the absolute position and pose of the object be considered separately? It seems their reactive behaviors would follow different patterns. If they're mixed, could this lead to overfitting?\n3. What's the fundamental advantage of this method compared to prior approaches that model human, object, and interaction separately? Intuitively, this modeling approach seems more suited for a single human and a single object. In scenarios with more people and objects, this method might limit scalability and significantly increase problem complexity.\n\nQUESTIONS:\n1. Can this modeling approach be extended to settings with multiple objects or multiple people?\n2. Should the absolute position and pose of the object be considered separately? It seems their reactive behaviors would follow different patterns. If they're mixed, could this lead to overfitting?\n3. What's the fundamental advantage of this method compared to prior approaches that model human, object, and interaction separately? Intuitively, this modeling approach seems more suited for a single human and a single object. In scenarios with more people and objects, this method might limit scalability and significantly increase problem complexity.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}