Human-Object Interaction via Automatically Designed VLM-Guided Motion Policy
Authors: Zekai Deng
Ye Shi
Kaiyang Ji
Lan Xu
Shaoli Huang
Jingya Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Human-Object interaction
Character animation
Human motion generation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a unified physics-based HOI framework that leverages VLM-guided spatio-temporal reasoning to automatically generate goal states and reward functions, enabling long-horizon interactions with diverse object types.

Abstract: Human-object interaction (HOI) synthesis is crucial for applications in animation, simulation, and robotics. However, existing approaches either rely on expensive motion capture data or require manual reward engineering, limiting their scalability and generalizability. In this work, we introduce the first unified physics-based HOI framework that leverages Vision-Language Models (VLMs) to enable long-horizon interactions with diverse object types — including static, dynamic, and articulated objects. We introduce VLM-Guided Relative Movement Dynamics (RMD), a fine-grained spatio-temporal bipartite representation that automatically constructs goal states and reward functions for reinforcement learning. By encoding structured relationships between human and object parts, RMD enables VLMs to generate semantically grounded, interaction-aware motion guidance without manual reward tuning. To support our methodology, we present Interplay, a novel dataset with thousands of long-horizon static and dynamic interaction plans. Extensive experiments demonstrate that our framework outperforms existing methods in synthesizing natural, human-like motions across both simple single-task and complex multi-task scenarios. For more details, please refer to our project webpage: https://vlm-rmd.github.io/.

Published as a conference paper at ICLR 2026
HUMAN-OBIECT INTERACTION VIA AUTOMATICALLY DESIGNED VLM-GUIDED MOTION POLICY
Zekai Deng'
Shi Kaiyang Ji ShanghaiTech University ?AgiBot
Lan Xu'
Shaoli Huang²
Jingya Wang
{dengzk2023, shiye, jiky2024, xulan1, wangsiagyaaeennnghattee..... shaolthuengeaggboo com
ABSTRACT
Human-object interaction (HOI) synthesis is crucial for applications in animation, simulation, and roboties. However, existing approaches either rely on expensive motion capture data or require manual reward engineering, limiting their scalabil- ity and generalizability. In this work, we introduce the first unified physics-basee HOI framework that leverages Vision-Language Models (VLMs) enable long- horizon interactions with diverse object types including static, dynamic, and articulated objects. We introduce VLM-Guided Relative Movement Dynamics (RMD), a fine-grained spatio-temporal bipartite representation that automatically constructs goal states and reward functions for reinforcement learning. By encod- ing structured relationships between human and object parts, RMD enables VLMs motion guidance without
to generate semantically grounded, interaction-awant manual reward tuning. To support our methodology, we present Interplay, a novel dataset with thousands of mang-horizon static and dynamic interaction plans. Ex- tensive experiments demonstrate that framework outperforms existing meth- ods in synthesizing natural, human-like motions across both simple single-task and complex multi-task scenarios. For more details, please refer to our project webpage: https: //s:://vlwe ///l/mmm- rmd. github.ion.. io/
INTRODUCTION
Sluman-Objede Interaction (HOI) understanding is fundamental to advancing embodied Al systems in simulation, animation, and robotics. Existing approaches to HOI learning generally fall into two categories. The first paradigm employs motion tracking policies to imitate reference demon- strations (Wang et al. 2023 2024a Xu et al. 2025). While these methods can reproduce specific interaction patterns, they face critical limitations: Their heavy reliance on high-quality motion cap- ture data creates scalability bottlenecks, and their strict adherence to reference trajectories inherently restricts their ability to generate novel interactions beyond the training distribution.
The second paradigm adopts a task-centric perspective, developing specialized policies for individ- ual interactions like sitting (Chao et al., 2021 Zhang et al., 2022) carrying (Xu et al., 2024b Szot et al.] 2021 Deitke et al.] 2022] Gao et al. 2024). However, these methods encounter two fundamental challenges. First, they require labor-intensive reward engineering by domain experts a particularly demanding requirement given the complex dynamics and contact-rich nature of re- alistic HOI scenarios (Sutton & Barto 2018). This manual reward design process inherently limits generalizability across interaction types, creating need for automated objective formulation. Sec- ond, constrained by artificially designed, single-objective reward mechanisms, trained policies often overfit to specific behavioral patterns. Producing task-compliant but biomechanically unrealistic motions that violate natural human kinematics.
Recent efforts such as Eureka (Ma al. 2023) and Grove (Cui et al. 2025) leverage large language models (LLMs) to automatically generate reward function code. However, they rely on iterative search paradigms that are often sample-inefffcient and computationally costly. Xiao et al. (2024) ad- vances this line of work by unifying reward design through "chain-of-contact abstraction, which models interaction discrete sequence of contact events. While conceptually elegant, this
"Corresponding author.

---

Published as a conference paper at ICLR 2026
Li and
Se the dest work
Fish Peeieaa of
Ples huncry babidit of theherre mustr.
COME
Figure I: Our framework automatically constructs both goal states and reward functions for diverse interaction tasks in reinforcement learning. By leveraging VLM guidance, the learned motion pol- icy drives physics-based characters to perform coherent, long-horizon interactions with static and dynamic objects, producing natural and task-consistent behaviors.
formulation overlooks the motion dynamics interactions and lacks the capacity to model whole- body coordination or accommodate dynamic objects. Consequently, it struggles handle dynamic interactions and often produces jittery, suboptimal behaviors even in static interaction scenarios.
To address these limitations, propose physics-based HOI framework that leverages VLMs to automatically construct goal states and reward functions, guiding the motion policy to perform long- term interactions with diverse objects. Specifically, we aim empower VLM to provide motion- level, interaction-aware guidance by leveraging its abilities in motion imagination and semantic reasoning. Drawing inspiration from the classical notion of relative motion (Zanstra 1924), we introduce Relative Movement Dynamics (RMD), structured representation that encodes the fine- grained spatio-temporal relationships between human parts and object parts across an interaction sub-sequence. For example, in the task of lifting box, the relative spatial configuration between the hands and the box remains stable, providing a natural constraint that captures both discrete interaction goals (e.g., contact) and continuous dynamics (e.g., coordinated movement). RMD is designed to prompt VL.Ms to imagine interaction dynamics while effectively grounding their high- level reasoning in motion-level patterns. This formulation enables VLMMs move beyond symbolic planning and participate in dynamic skill composition, thereby supporting more expressive and gen- eralizable HOI motion policy learning. Building on RMD, our framework automatically constructs unified goal states that support both static and dynamic interactions, along with automatically de- signed reward functions that guide motion patterns consistent with the plan.
Our framework supports long-horizon interactions with diverse object types. However, existing datasets typically focus either on static interactions (Xiao al., 2024) or on object rearrange- ment (Xu et al. (2024b), and thus fail to cover this setting. To address this gap, we construct new dataset, Interplay, which includes long-horizon static and dynamic interaction tasks across var- ied scene contexts. This dataset enables systematic evaluation of our framework in long-horizon multi-task scenario. Experimental results demonstrate that our method achieves promising quantita- tive performance and generates natural, human-liks motions across a diverse range of HOI tasks.
In summary, our contributions include:
We propose the first unified phaseds-based HOI synthesis framework for long-horizon human-objee interactions leveraging the powerful world knowledge of VLMs, supporting a diverse range of ob- jects, including static, dynamic, and articulated ones.
We introduce VLMdGuided Relative Movement Dynamics (RMD), a fine- fine-grained bipartite diagram to automatically construct goal states and reward functions for reinforcement leamn ing. This approach bypasses manual reward engineering while supporting diverse HOI, including both static and dynamic interactions.
2

---

Published as a conference paper at ICLR 2026
Table 1: Comparative analysis of key features between ours and other methods.
Medand
Acsistacc Objoc
Asterzzee Rovand Doize Dyname Object Ley--sotto Transiso
HSH Grszanze
Mottizak
Hibbleed Massiu
United Palay
X02
Ooase
Rule bacel FSM
Coase
Unitt
Comsa
We introduce Interplay, a novel dataset comprising thousands of interaction plans that cover long- horizon static and dynamic interaction tasks. Extensive simulation results demonstrate the effective- ness of our approach on both single-task and long-horizon multi-task scenarios.
2 RELATED WORK
Human Motion Synthesis with Static Scenes. In the field of character animation and motion synthesis, most research has focused on human motion interacting with static 3D scenes (Li et al.
2019 Zhang et al., 2020; Xuan et al., 2023] Zhao et al. 2022 Lee & Joo 2023 Wang et al,
2022b 2024b) Starke et al. 2019: /hao et al. 2023a). Typically, researchers break down complex
instructions into several key static poses within the scene (Hassan al.) 2021 Wang et al. 2022a), and then use motion inpainting and optimization techniques to generate transitions between these key poses (Wang et al. 2020). However, this approach often leads to mediocre and inconsistent motions due to the limited expressiveness of the intermediate sequences. Recently, ciftusiio-bbaee methods like (Ho et al.) 2020; 2020) Huang al.) 2023) Tang et al., 2024) Yi al, 2025 Zhao a 2024] Tevet et al.) 2024) have achieved better results in human-scene synthesis. However, these data- driven approaches are limited to generating short-term motions due to constraints in the dataset, and the physical plausibility of the generated motions is not guaranteed. Apart from data-driven kinematic approaches, some studies have explored the problem within reinforcement lcarning framework. For instance, (Zhao et al. 2023b Zhang & Tang 2022) achieved long-term goal- oriented behaviors by leveraging motion primitives with specific reward designs. Xiao et al. (2024) decomposes instructions into a sequence of point-reaching control tasks under unified formulation. However, it addresses only the spatial constraints of static scenes while overlooking the temporal dynamics of objects, rendering it inadequate for managing complex dynamic interactions.
Kinematic-based Human Motion Synthesis with Dynamic Objects. Research in human motion synthesis has increasingly focused on modeling interactions with dynamic objects (Starke et al. 2020 Jiang al.] [2022] Zhang al. [2022] 2024). Diffusion-basee frameworks (Li et al. [023b a Pi et al. 2023] Peng et al.] 2023] Ji et al. 2025 Zhang et al. 2025). guide motion generation with object trajectories but often lack realism due to predefined object paths that fail account for physical plausibility. In contrast, InterDreemmrr (Xu et al. (2024a) first generates human motion and subsequently uses a pre-trained world model to produce object trajectories, though this approach is limited by the simplicity of the world model, leading to inaccuracies in trajectory prediction. Some studies attempt to address these limitations by jointly modeling human-object interactions with supplementary guidance techniques, including relation intervention (Wu et al. 2024a). contact prediction (Diller & Dail 2024), and affordance estimation (Peng et al. (202). Truman (Jiang et al.) 2024b) and Lingo (Jiang et al.) 2024a) trained a high-quality human-scene interaction (HST) dataset, achieves dynamic stability through an auto-regressive diffusion model guided by action labels text. Nevertheless, kinematic models continue to face challenges such penetration, sliding issues, and difficulties in generating long-term motion, requiring extensive annotations.
Physics-based Human Motion Synthesis with Dynamic Objects. To generate physically plau- sible motions, reinforcement learning methods have been shown to effectively train HOI skills using motion capture data (Chao et al. 2021 Merel et al.) 2020, Peng et al., 2018 Xie et al.) 2023) Xu et al. 2025 Wu et al. 2024b). AMP (Peng et al. 2021 introduced an adversarial motion prior framework for realistic motion synthesis. InterPhys (Hassan et al. 2023) further extended this framework by incorporating an HOI motion prior, achieving success in tasks such as sitting, lying, and carrying. Further advancements have led to successful applications in sports activities, including
basketball (Wang et al. 2023 2024a). tennis (Zhang et al. (023). skating (Liu & Hodgins 2017).
and soccer (Luo al. 2024 .il
2022
Xie al. 2022). TokenHSI (Pan al. 2025) integrates
3

---

Published as a conference paper at ICLR 2026
Refermer etample
VLM-Gulded Motion Pollcy
Soeee-contest understanding
9
"For should lpt the loundry basker
concept delination
Idea outline
Specific plan Instance
object larget
Fat
Jaaadty basket
Destcatine State
Destitation
Plan
Fii
reghtaad
30D Cogha
R
Heght Nap
Oiges Statio
Mileel
kasbit right
R KAFA 15
exp 401
Rlwardd
- cap (-1at of
Environment Context
You eesaaas leata seked lecuus aut luget, auaio, bubked
Action dt
duacaiod
"raf" (1%) (hooort bandy "Tadie oite
Instruction
"Tidy up dirty clorhes."
foot "aadii slor
GPT-4V
VLM OUTPUT
VLM-Guider RMD Planner
Section 3.1
Section 3.2
An overview of our architecture. Receiving instruction and environment context as input, the VLM-Guided RMD Planner generates a multi-step interaction plan the form of RMD. Based on this plan, framework automatically designs both goal states and reward functions, enabling the VLM-Guided Motion Policy to execute the interaction step by step.
Figure 2:
multiple skills into a single policy via task tokenizer. These approaches often depend on heuristic designs and struggle to generalize to longer, multi-round scenarios. This limitation highlights the need for a principled representation that can bridge short-horizon rules and long- morizoo interac- tion dynamies. Our method seeks to address these limitations by introducing Relative Movement Dynamics representation, enabling the model to capture both spatial and temporal dynamics ef- fectively. This results in physically plausible and temporally coherent interactions, eliminating the need for manual annotations and enhancing realism in dynamic, long-term HOI tasks. The detailed comparisons of key features listed in Tab.
METHOD
Our method comprises two tightly coupled components. First, describe how VLM is leveraged to translate high-level instruction and environment context into a sequence of structured interaction plans. This process is grounded in our proposed concept of RMD, which models fine inneggrined spatio- temporal relationships between human and object parts and serves as the foundation for interaction planning (Sec. B.T). Then, we present VLM-guided motion policy learning framework, where each RMD plan is automatically transformed into goal state and a corresponding reward function, enabling the agent to learn executable motion policies without manual specification (Sec. 3.2
3.1 VLM--UIDED RMD PLANNER
We employ GPT-4V Achiam et al. 2023) our VLM planner to bridge high-level task instructions
and low-level interaction execution. As illustrated Fig. 2 the planner takes three inputs: a high- level textual instruction I, top-view contextual image of the environment C. and a set of modular prompts designed to guide the decomposition of the task into detailed interaction plans.
Effectively leveraging the motion imagination capabilities of VLMs for HOI planning necessitates a principled formulation of the interaction process. Simplistic abstractions, such as the chain-of- contacts (CoC) proposed by UniHSI (Xiao et al., 2024), is inadequate: by imposing only transient point-contact constraints that are discarded immediately upon satisfaction, CoC fails to capture the evolving spatio-temporal relationships critical for modeling dynamic and coordinated interactions.
4

---

Published as a conference paper at ICLR 2026
In contrast, we propose novel concept, Relative Movement Dynamics (RMD), which encodes fineggrained spatial and temporal relationships between human body parts and object components throughout the course of an interaction. This structured formulation enables the VLM to reason more effectively about the physical organization of human-object interactions and to produce structured, executable plans aligned with underlying motion dynamies. Our key insight is that human-object interactions can be abstracted as the relative motion between two sets of rigid bodies; human body parts and object parts, evolving over time. We formalize this abstraction as a bipartite graph B that encodes part-level motion dynamics. Let
PH {Ph, + Phz...... Ph... ). Po = {Por Poz. Po..
(1)
denote the sets of human and object parts, respectively. Then the graph B is defined as,
B = EV)
== PHUPP E x Po. w E - {0. 1,2.
(2)
Each edge Cij = (Ph. Pay) E connects a human part Ph. E Pи to an object part Po, E Po. with an associated weight wiz E {0, 1, 2, 3} that characterizes their relative-motion pattern: Wij =
indicates stationary contact, Wij = 1 denotes approaching motion, Wij 2 represents separating motion, and Wky = implies no consistent relative trend.
To facilitate the planner's 's ability to infer such dynamics, we adopt modular prompt design strategy. Each prompt segment is crafted to trigger a specific reasoning ability, such as environmental parsing, part-level object understanding. motion dynamic inference, symbolic representation generation. Through this mechanism, the VLM is guided to produce both the RMD graph B and two spatial anchors: the target location for the human root Ti and the target location for the object root To, which together define the global interaction objective.
Finally, the planner outputs an interaction plan 'D as sequence of N structured steps,
D= (G1 -.. 92,....9N)):
(3)
where each step G. is triplet that specifies the spatial goals and dynamic pattern,
G B).
(4)
3.2 AUTOMATIC POLICY LEARNING VIA VLM-GUIDED RMD
Upon receiving the plan generated by the "LMdguided RMD Planner, the policy produces joint- level torques actuate the agent and execute the plan. Our physics-based HOI policy learning is formulated within the framework of goal-conditioned reinforcement learning. At each timestep I, the agent samples an action from its policy π (a+ 8t, g:), conditioned on the current state 8; and the goal state gr- After executing the action, the environment transitions to the next state St41, and the agent receives two types of rewards: task reward y (st. gt, $141), which incentivizes behaviors aligned with achieving the specified goal g:, and a style reward y (St, (t,,)) which encourages the agent to produce natural behaviors as proposed in |eng et al. (2021) I). Unlike AMP and most prior methods that rely on manually specifying aask-specffic goal states and handcrafted reward terms, framework supports the automatic construction of goal states and reward functions. In the following, detail how our framework automatically generates both goal states and reward functions from VLM output, enabling the agent to learn behaviors aligned with the intended objectives.
To provide the policy with fine- grained spatio- temporal guidance during the interaction, we encode the i-th planned interaction step G, (Tn, To}. generated by the VLM, with both kinematic and dynamic information.
Automatic Goal State Construction via VLM-Guided RMD.
For each edge eij E, we extract from the simulator the absolute position-velocity pair of the human joint, denoted by Př P% and that of the nearest surface point on the object part, P P.j. We then compute their relative quantities in the agent-centric coordinate frame.
Pij = P5, Ph.- Vij = PP; Ph
(5)
The edge weight Wiy {0, 2, is encoded as one-hot vector w/ E {0,1)* and concatenated with the relative position and velocity to form the feature tuple (Pij. Vay. w'y) at timestep t. Stacking these features across all edges yields the RMD state,
s
RMD
conCat(..)jEE [Pij, Vij, why] E R (3+(3++++1
(6)
5

---

Published as a conference paper at ICLR 2026
Both Th and To generated by VLM are in the form ob jejttt (spatial-relattonnsii) e.g.,
armchair (front). For every referenced object we approximate its geometry by an axis-aligned bounding box with edge lengths Lye ly. l and center Cobj- We map each spatial--e token d to a displacement vector (q( in the object's local frame. For instance, front + (0.7 0, 0).
The absolute targets for the human root and the object root are defined as,
Piar Cohj + (6()))
Piar = Cobj + Aq
The destinations proposed by the VLM, originally in the global coordinate system, are transformed into the agent-centric frame to produce the final destination state,
d = dy).
To improve local perception and obstacle avoidance, we attach a fixed-resolution heightmap h, E R that records the elevation of nearby geometry in the root frame (Tessler al. 2024 Xiao
et aL 2024). In parallel, we describe object dynamics by the axis- bounding box vertices 1233 the rotation 0, E R2. the linear velocity V, e R2, and the angular velocity w, ERS
V b0x
(Vio, R., Vi, Wi}.
0: =
By concatenating the RMD state sMMD, the destination d,, the heightmap h,, and the object state O.. we construct the complete goal representation,
g: {s RMD d. h. o.}.
(10)
To guide buman-object interaction in align- ment with the i-th planned interaction step G; = {Tn, B, To} produced by the VLM, we design a composite reward function that promotes three key objectives: (i) guiding the human root trajectory towards the spatial target The (ii) guiding the object root trajectory towards To, and (iii) enforcing the relative motion patterns prescribed by the RMD graph B.
Automatic Reward Design via VLM-Guided RMD.
To realize these objectives, we define three corresponding reward terms. The term r encourages the human root position 5 to approach the designated target d', and rg encourages the object root position of to move toward its designated target de,
r exp (-1|=) db11"),
r2 = exp (-1les -1d1"
(11)
The term FRMD encourages each edge ei B, representing (human-part, object-part) pair, to follow the relative motion dynamics specified by its associated edge weight Wij. as planned by the VLM. Specifically, FRMD is computed by summing the individual alignment reward قار () each of which measures how well the pair (ph, Po,) follows the motion pattern defined by Wij, weighted by coefficient Àij.
TRMD
Σ (i.))E
Xij . (Pij, Vaj. Why)
(12)
Further implementation details of At (.) are provided in the appendix. Combining these terms, the overall task reward at time t is given by,
(Sr, S1+1) = ARMD FRMD + An -r + No .v.
(13)
Once the task reward G which is bounded within the range [0, 1], exceeds 0.9, the policy transitions the next planned interaction step Gi+1 at the following timestep. To balance the contributions of different reward terms, we adopt adaptive weighting strategies as proposed by Xiao et al. (2024), which dynamically adjust the value of in Eq. 12 and Eq. [3] This design enables the current reward to be evaluated in balanced manner and facilitates accurate assessment of progress within the current stage, thereby ensuring the appropriate timing for transitioning to the next goal.
The style reward S (st, St+1) is computed by discriminator (05:-10z)) based on window of 10 consecutive states. Combining the task reward G and the style reward SS, the total reward at timestep t is defined as,
F1 Clask G (8t, Be: Sp+1) ) + Ossyle 8 (St, 81+1)-
(14)

---

Published as a conference paper at ICLR 2026
Table 2: Comparison with baselines in long-horizon multi-task scenario.
Mecoods
Sotic Intrnctionn
Compittte Rase [5]
Dyyomi: Insricicon
Statz Intirattinn
Salvsaip Completion (1))
Dysume Interaction
Llyndd
Static Intemction
Sthwarp U'rciiinn (cm))
Dl Inmenctionn
Hytee
InterPass
47.8
22.5
61.
5411
13.8
16.9
Theel
24
16.
65.7
60.1
15.1
16.5
16.4
Dailtst Stame
17.2
61.3
10.2
Our
$4.3
718
EXPERIMENT
We structure our experiments into two main parts: evaluating HOI skill learning in complex long- horizon multi-task scenario (Sec. 4.1 and in a simple single-task scenario (Sec. 4.2)
InterPlay Dataset. To evaluate our framework in long-horizon multi-task scenario, we construct a novel dataset that includes both static and dynamic interaction tasks under diverse indoor scene contexts. Each sample comprises four components: (i) set of processed 3D assets that can be directly deployed in simulation, (ii) the corresponding 3D scene layout, (iii) a top-view rendered image, and (iv) natural-languar instruction specifying the target task. We curate high-quality 3D object assets from established resources, including PartNet (Mo et al.) (019), 3D-FRONT (Fu et al.) 2021 SAMP (Hassan et al.) 202I). CIRCLE (Araújo et al.) (023), and PartNet-Mobility (Xiang et al. 2020). Each scene contains at least two interactive objects and no fewer than two furniture assets, providing richer contextual diversity and increased HOI complexity. Additional details the dataset construction are provided in the appendix.
Implementation Details. To ensure our agent interacts with objects in a natural way, we collect motion clips from SAMP (Hassan et al. 2023), OMOMO (Li et al.) 2023a). CIRCLE (Araújo al., 2023). We conduct experiments in parallel simulated environments within Isaac Gym (Makoviychuk al. 202T), using PyTorch for neural network implementation. In line with previous studies (Has- san et al.] [2023] Xiao et al. 2024). our physical humanoid model consists of 15 rigid body parts and 28 joints, all controlled by a PD controller. We utilize Proximal Policy Optimization (PPO) (Schul- man et al. 2017) to train the policy network on an NVIDIA RTX 3090 GPU.
4.1 EVALUATION IN A LONG-HHORZEE MULTI-TASE SCENARIO
Real-world applications require agents to perform multiple tasks concurrently or sequentially, re- quiring advanced coordination and planning. In this context, the agent must interact with multiple objects in sequence according to the plan, which poses significant challenges for ensuring the ro- bustness of VLM VLM-guided motion policy learning and for managing long- long-horizon transitions.
Settings. We split our dataset into three categories: interiiittraa,oon, cymamic-interaction and hybrid setups. In the static-only setup, sequentially interact with at least two static objects as planned in the dataset. In the dynamic-only setup, we interact with at least two dynamic objects articulated objects. In the hybrid setup, select at least three objects, ensuring that at least one dynamic human-object interaction task is included in the overall plan. To ensure fair compar- isons under this taxonomy, we align each baseline with the subset of settings it can natively sup- port. UniHSI (Xiao et al. 2024) supports only sequential interactions with static objects. We adapt its released prompt to generate plans based on scenarios from our static-only dataset. Both Inter- Phys (Hassan et al. 2023) and TokenHSI (Pan et al.) (2055) support interactions with both static and dynamic objects, but they only cover a subset of HOI tasks. For methods without native multi-task planning, we introduce a standardized adaptation protocol before evaluation. To enable multi-task evaluation, we re-implement their methods by training individual skills in parallel across different environments, manually designing goal states and corresponding reward functions for each. A rule- based finite-state machine is employed to switch between tasks during exccution. This adaptation requires substantial manual effort.
Metrics. We report the Completion Rate, which measures the percentage of trials in which the character sequentially completes all sub-step tasks and returns to natural standing pose, with the
7

---

Published as a conference paper at ICLR 2026
Table 3: Comparison with baselines in single-task scenario.
Methods
Completion (R)
Success
1
Precision (em)
Camy
Push
Open
Sit
Reach
Caryy
Open
Curry
Open
Reach
AMP
AMP-PPnns.c Peng cl.l.
1,4
14,9
70.3
77.4
17.3
18.1
Interttsss [Ea5san cE
2023
67.8
47.1
83.2
23.2
2:7
95.3
88.2
78.1
90.1
88.1
76.4
95.9
12.3
24.7
10.1
10.3
14.8
3.4
TokonHSi Irin ml 2p22
71.2
49.3
81.1
27.8
8.9
95.7
92.2
79.5
90.8
92.8
78.2
95.9
11.2
22.5
X.7
9.8
13.2
1.5
UniHSI Xalo
58.9
23.2
97.1
94,3
81.5
97.5
5.6
12.8
2.1
Ours
88.3
84.1
91.2
92.6
62.0
97.5
93.3
90.1
95.1
95.6
87.0
97.8
10.1
18.2
5.2
4.9
9,1
1.9
mulli-ome one
75.1
71.3
11.7
59.2
65.8
28.1
97.2
14.3
7.2
w.o. Pu
71.7
73.8
77.0
72.9
36.8
96.8
88.9
87.8
90.3
91.8
79.8
96.8
10.9
19.7
10.1
5.6
12.8
2.9
w.o. Vy
78.2
76.7
75.2
76.1
42.1
96.9
90.1
84.2
87.8
92.1
79.2
96.9
11.3
20.0
9.9
6.1
14.2
2.5
w.o. udy
69.1
68.4
62.1
67.0
30.3
95.1
82.3
82.3
86.2
90.3
78.9
95.2
14.1
23.9
9.2
7.6
18.6
2.2
character's ' root within 20 cm of the target position. We introduce the Sub-step Completion Ratio, which measures the ratio of completed sub-steps the total number of planned sub-steps. This metric provides more granular view of the agent's ability to handle complex tasks with multiple sub-goals. Additionally, we use Sub-step Precision to measure the average distance between specific human part and its target, or between the object root and its target.
Result Analysis. As shown in Tab. 2 our approach achieves state-of-the-ard performance in both individual sub-step execution and long-horizon transitions, reflecting its robustness in handling in- tricate sequences of actions. Our method benefits from a unified formulation of interaction and fine-grained spatio-temporal guidance for all human body parts, enabling seamless transitions to multi-task setup. This design not only streamlines the training process by maintaining consistent representations across various tasks but also significantly improves the model's ability to general- ize across different interaction scenarios. In the absence of unified formulation, approaches that naively combine multiple tasks, such as InterPhys and TokenHSI, often encounter difficulties in smoothly switching between tasks. In contrast, our integrated method preserves critical spatial and temporal information throughout the pipeline. While UniHSI excels at approaching and interacting with static objects, it struggles with actions such getting up after interaction, which notably limits its performance in morgoooizon scenarios.
Ablation. While both VLMs and LLMs possess extensive world knowledge, VLMs exhibit supe- rior spatial awareness and motion imagination thanks to their integration of visual and textual inputs. To evaluate this, we replaced top-view images with textual descriptions and assessed an LL---based planner under purely textual conditions. The resulting performance drop highlights the essential role of VLMs in our framework. Pretrained on diverse vision-language data, the VLM excels at gener- ating semantically mcaningful and spatially grounded plans capabilities critical for skill learning in complex, multi-object environments. Without VLM guidance, the system struggles to produce coherent long-horizon behaviors, leading lower task success and degraded motion quality.
4.2 EVALUATION IN A SINGLE-TASK SCENARIO
Settings. We select set of interaction tasks from our InterPlay dataset, including both tasks com- monly used in prior work and several novel tasks not previously explored. These include three static HOI tasks: reaching, sitting, and lying down, and three dynamic HOI tasks: carrying, pushing, and opening. This setup lets us evaluate not only whether an interaction is achieved, but also whether agents transition reliably across heterogeneous tasks. While previous works (Hassan al. 2023 Xiao et al. 2024) primarily focus on approaching and interacting with objects, they often overlook critical aspect: after interacting, the agent must return neutral state facilitate subsequent interactions. For instance, in the sitting task, previous methods consider the task complete once the agent seated. However, this disregards the need for the agent return to standing in order to transition smoothly to the next task. Therefore, we redefine the task completion criteria by in- troducing a "leaving" step: after interacting with an object, the agent must stand up and walk to a designated position. This step ensures that the agent returns to neutral pose, which is essential for smooth transitions between tasks. For each episode, objects are initialized with a random orientation (ranging from 0 to 2TT), random distance (from 4 to 10 meters), and random scale (from 0.8 to 1.2). The finish position is then randomly sampled at a point 3 meters away from the object. To ensure fair comparison with previous state-of-the-ee methods, rigorously followed their estab-
8

---

Published as a conference paper at ICLR 2026
Push(ours]
Ueloum)
Push(InterPhys)
Lle(UniH5I)
Visualization for qualitative comparison. Other methods exhibit unnatural motion (In- terPhys) or incomplete interactions (UniHSS), whereas our method demonstrates human-like motion quality in qualitative assessments. More qualitative visualization videos can be found in the supple- mentary materials.
Figure 3:
lished implementation protocols and re-implemented AMP (Peng et al. 2021), InterPhys (Hassan ct al., 2023), and TokenHSI (Pan et al. (025). modifying goal states and reward functions only as necessary to adapt them experimental setup.
Metrics. We first introduce a new metric, Completion Rate, to evaluate the percentage of tri- als in which the character only completes the interaction with the object but also returns neutral standing pose, with the root located within 20 cm of the designated finish position. This metric is designed to reflect the agent's ability recover from interaction and complete the full motion sequence, offering more comprehensive evaluation of control quality. In line with prior works (Hassan al. 2023 Xiao al. 2024), we also report the Success Rate, which measures the percentage of trials which the character successfully interacts with the object. A trial is consid- ered successful if the distance between the object's root and a specific humanoid part is less than 20 cm (for tasks like reaching, sitting, or lying down), or if the distance between the object's root and the target location less than 20 cm (for tasks like carrying, pushing, or opening). Furthermore, we report Precision to quantify the average distance between specific human body part and its cor- responding target position over the course of the entire interaction in successfully completed trials. All metrics evaluated over 4096 trials.
Result Analysis. As shown in Tab. our method achieves higher or comparable performance across all these metrics compared the baselines. Our method outperforms tasks involving the recovery process of static human-object interactions, such as getting up after sitting or lying down, as the Relative Movement Dynamics guides each body part to move away from the object. In contrast, as shown in Fig. 3 UniHSI (Xiao et al. (022)) struggles to get up because it treats the human-objed interaction sequence independent spatial reaching tasks, neglecting the temporal dynamics that coordinate the movements of different body parts. Similarly, InterPhys (Hassan et al.) (023)) produces unnatural and unstable motions to complete tasks, such as abrupt kicking or thrusting spatio-temporal guidance for coordinating the movements of
forward, due to its lack of fine-graine different body parts. Instead, our design ensures that the agent keeps both hands and arms relatively stationary against the back of the sofa.
Ablation. We conduct an ablation study on key components to evaluate the contribution of the pro- posed VLM-guided RMD. In the multi-one setup, simplify object representation by removing the division into distinct parts and treating the object as single entity. Representing the object only by its root pose and kinematic data leads to a slight performance drop, the model loses fine- fine-graing -naagrdided geometric information necessary for accurate interaction modeling, In the one-one setup, we restrict the VT-Megudedd RMD Planner to model only the dynamics of a single human part interacting with one object part sub-step. This disrupts whole-body coordination and often traps optimization in local optima, resulting a noticeable decline in performance. These results highlight the effective- ness of the multi-multi formulation in handling complex interactions. Moreover, removing either kinematic relation encoding (p.". o.") or dynamic encoding (w(²/) further degrades performance, as both are crucial for capturing the spatio-temport relationships between human and object. Overall, the ablation results confirm the central role of VLM-guided RMD in framework.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
In this paper, we present a physics-based framework for synthesizing diverse Human--Obees Inter- actions, guided by VLMs, through our proposed Relative Movement Dynamics - a spatio-temporal representation that structures part-level relationships between human and object. RMD automates the generation of goal states and reward functions for reinforcement learning, eliminating manual engineering while supporting statie, dynamic, and articulated interactions. Our method bridges high- level semantic reasoning with low-level physics- control, producing natural motions across diverse interactions that outperform task-specific and LLMM-based baselines in realism and adapt- ability. Beyond performance gains, RMD offers a unified representation that may be useful for extending broader HOI settings. To facilitate evaluation, we introduce the Interplay dataset for long-horizon HOI tasks. Extensive results demonstrate the effectiveness of our approach on both single-task task and long-horizan multi-task scenarios.
6 LIMITATIONS, EXTENSIONS, AND FUTURE WORK
Limitations. While our framework advances dynamic human--objec interaction modeling, it fo- cuses on single-agent scenarios and does not address multi-agent interactions (e.g. collaborative tasks involving multiple agents) or social dynamics (e.g. social impact considerations in shared environments). This restricts its applicability to real-world settings requiring coordination among multiple agents, such as collaborative kitchens assistive care scenarios. Additionally, the cur- rent VLM-based planner may struggle with long-horizon tasks that demand temporal reasoning and hierarchical planning. For instance, tasks like cooking multi-step meals or tidying interconnected household spaces remain challenging due to the lack of explicit task decomposition mechanisms.
Extending to Real-Workd Applications. We view RMD mid-level relational abstraction that bridges semantic intent and executable control by explicitly modeling the spatio-temporal relation- ships among interacting rigid bodies. This graph formulation is naturally extensible: by enriching the vertex and edge sets, RMD can encode additional entities and constraints (e.g.. multiple agents, articulated objects, tool use, and self-interaction), while preserving the same downstream policy- learning interface. In simulation, privileged part-level human/object states are available, allowing our framework automatically propose goal states and construct rewards for policy training, yield- ing strong controllers without manual engineering. During deployment in real-world or partially observed settings where these signals are inaccurate, the trained controllers can serve as teachers in a dtuillation/imitationn pipeline from egocentric sensory inputs, followed by lightweight adaptation (e.g., residual RL or teacher-student distillation). Finally, robustness can be improved by strength- ening knowledge grounding and multi-step reasoning (e.g., retrieval-augmented planning and struc- tured deliberation) and by adding a dedicated verification stage that filters or repairs infeasible plans before policy execution.
Future Work. Beyond these directions, we will explore integrating diffusion-basee generative pri- to synthesize more diverse and natural interaction motions, which may further improve erage of rare but realistic behaviors. We are also interested in joint training paradigms that more tightly couple the VLM planner and the physics-based controller (e.g.. via alternating optimiza- tion or planner-controller consistency objectives), with the goal of reducing planner-controller mis- match and improving both generalization and control fidelity complex, hong-horizon interactions. In addition, we plan to extend policy learning to multi-agent settings with learned coordination mechanisms, such as adaptive communication and dynamic role allocation, to support collaborative household activities at scale.
ACKNOWLEDGEMENTS
This work was supported by National Natural Science Foundation of China (62406195, 62303319), Shanghai Local College Capacity Building Program (23010503 100), ShanghaiTech A14S Initiative SHTAI4S202404, HPC Platform of ShanghaiTech University, Core Facility Platform of Computer Science and Communication of ShanghaiTech University, and MoE Key Laboratory of Intelligent Perception and Human-Machine Collaboration (ShanghaiTech University), and Shanghai Engineer- ing Research Center of Intelligent Vision and Imaging.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv: 08774, 2023.
Joao Pedro Araújo, Jiaman Li, Karthik Vetrivel, Rishi Agarwal, Jiajun Wu, Deepak Gopinath, Alexander William Clegg, and Karen Liu. Circle: Capture in rich contextual environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
21211-21221, 2023.
Yu- Wei Chao, Jimei Yang, Weifeng Chen, and Jia Deng. Learning to sit: Symthesizing human- chair interactions via hierarchical control. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 5887-5895, 2021.
Jieming Cui. Tengyu Liu. Ziyu Meng, Jiale Yu, Ran Song, Wei Zhang, Yixin Zhu, and Siyuan Huang. Grove: A generalized reward for learning open-vocabulary physical skill. In CVPR, 2025.
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. Advances Neural Information Processing Systems, 35:5982-
5994, 2022.
Christian Diller and Angela Dai. Cg-hoi: Contact-guidedt 3d human-object interaction generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
19888-19901, 2024.
Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiamingg Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and seman- tics. Proceedings of the JEEE/CV International Conference on Computer Vision, pp. 10933- 10942, 2021.
Jiawei Gao, Ziqin Wang, Zeqi Xiao, Jingho Wang, Tai Wang, Jinkun Cao, Xiaolin Hu, Si Liu, Jifeng Dai, and Jiangmiao Pang. Coohoi: Learning cooperative human-object interaction with manipulated object dynamics. arXiv preprint arXiv:2406. 14558, 2024.
Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J Black. Stochastic scene-aware motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11374-11384 2021.
Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, and Xue Bin Peng- Synthesizing physical character-scene interactions. In ACM SIGGRAPH 2023 Conference Pro- ceedings. pp. 1-9, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851. 2020.
Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song- Chun Zhu. Diffusion-based generation, optimization, and planning in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16750-16761, 2023.
Kaiyang Ji, Ye Shi, Zichen Jin, Kangyi Chen, Lan Xu, Yuexin Ma, Jingyi Yu, and Jingya Wang. Towards immersive human-x interaction: A real-time framework for physically plausible motion synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 10173-10183, 2025.
Nan Jiang, Tengyu Liu, Zhexuan Cao, Jieming Cui, Yixin Chen, He Wang. Yixin Zhu, and Siyuan Huang. Chairs: Towards full-body articulated human-objec interaction. arXiv preprint arXiv:2212. 10621, 3, 2022.
11

---

Published as a conference paper at ICLR 2026
Nan Jiang, Zimo He, Zi Wang, Hongjie Li, Yixin Chen, Siyuan Huang, and Yixin Zhu. Autonomous character-scene interaction synthesis from text instruction. In SIGGRAPH Asia 2024 Conference Papers, pp. 1-11, 2024a.
Nan Jiang, Zhiyuan Zhang. Hongjie Li, Xiaoxuan Ma, Zan Wang. Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human scene interaction modeling. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1737-1747, 2024b.
Jiye Lee and Hanbyul Joo. recomotion-actionn aitimotoo-accioommmmpplaatoooooo Synthesizing human-scene interac- tions in complex 3d environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Jiaman Li, Alexander Clegg, Roozbeh Mottaghi, Jiajun Wu, Xavier Puig, and C Karen Liu. Con- trollable human-object interaction synthesis. arXiv preprint arXiv:2312.03913, 2023a.
Jiaman Li, Jiajum Wu, and C Karen Liu. Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG), 42(6):1-11, 2023b.
Xucting Li, Sifei Liu, Kihwan Kim, Xiaolong Wang. Ming- Hsunn Yang. and Jan Kautz. Putting hu- mans in a scene: Learning affordance in 3d indoor environments. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12368-12376 2019.
Libin Liu and Jessica Hodgins. Learning to schedule control fragments for physics- based characters using deep q-learning. ACM Transactions on Graphics (TOG), 36(3):1-14, 2017.
Siqi Liu, Guy Lever, Zhe Wang. Josh Merel, SM Ali Eslami, Daniel Hennes, Wojeiech M Czarnecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmalcki, et al. From motor control to team play in simulated humanoid football. Science Robotics, 7669):cbb00235, 2022.
Zhengyi Luo, Jiashun Wang, Kangni Liu, Haotian Zhang, Chen Tessler, Jingbo Wang. Ye Yuan, Jinkun Cao, Zihui Lin, Fengyi Wang, et al. Smplolympics: Sports environments for physically simulated humanoids. arXiv preprint arXiv:2407.00// 87, 2024.
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayara- man, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv: 1rx1--2310.12931, 2023.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gau-based physics simulation for robot learning. arXiv preprint urXiv:2108. 10470, 2021.
Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. Catch & carry: reusable neural controllers for vision- guided whole-body tasks. ACM Transactions on Graphics (TOG), 39(4)39-- 2020.
Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
Liang Pan, Jingbo Wang. Buzhen Huang, Junyu Zhang, Haofan Wang, Xu Tang, and Yangang Wang- Synthesizing physically plausible human motions in 3d scenes. In 2024 International Conference on 3D Vision (3DV), pp. 1498-1507 IEFE, 2024.
Liang Pan, Zeshi Yang, Zhiyang Dou, Wenjia Wang, Buzhen Huang, Bo Dai, Taku Komura, and Jingbo Wang. Tokenhsi: Unified synthesis of physical human-scena interactions through task tokenization. In CVPR, 2025.
Xiaogang Peng, Yiming Xie, Zizhao Wu, Varun Jampani, Deqing Sun, and Huaizu Jiang. Hoi-diff: Text-driven synthesis of 3d hujan- interactions using diffusion models. arXiv preprint arXiv:2312.0655 06555,,, 2023.
12

---

Published as a conference paper at ICLR 2026
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 37(4):1-14, 2018.
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics- character control. ACM Transactions on Graphics (ToG), 4024):1-20, 2021.
Huaijin Pi, Sida Peng. Minghui Yang, Xiaowei Zhou, and Hujun Bao. Hierarchical generation of human-object interactions with diffusion probabilistic models. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 15061-15073, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. Neural state machine for character-scene interactions. ACM Transactions on Graphics, 38(6):178, 2019.
Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Zaman. Local motion phases for learning multi-contact character movements. ACM Transactions Graphics (TOG), 39(4):54-1, 2020 2020.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Bradford
Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turnrr, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in neural information processing systems, 34:251-266, 2021.
Janngan Tang, Jingya Wang, Kaiyang Ji, Lan Xu, Jingyi Yu, and Ye Shi. A unified diffusion framework for scene-aware human motion estimation from sparse signals. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 21251-21266 2024.
Chen Tessler, Yunrong Guo. Ofir Nabati, Gal Chechik, and Xue Bin Peng. Maskedmimic: Uni- fied physics-based character control through masked motion inpainting. ACM Transactions on Graphics (TOG), 2024.
Guy Tevet, Sigal Raab, Setareh Cohan, Daniele Reda, Zhengyi Luo, Xue Bin Peng, Amit H Bermano, and Michiel van de Panne. Closd: Closing the loop between simulation and diffu- sion for multi-task character control. arXiv preprint arXiv:2410.03441 2024.
Jiashun Wang. Huazhe Xu, Jingwei Xu, Sifei Liu, and Xiaolong Wang. Synthesizing long-term 3d human motion and interaction in 3d scenes, 2020.
Jingbo Wang, Yu Rong, Jingyuan Liu, Sijie Yan, Dahua Lin, and Bo Dai. Towards diverse and natural scene-aware 3d human motion synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20460-20469, 2022a.
Yinhuai Wang, Jing Lin, Ailing Zeng. Zhengyi Luo, Jian Zhang, and Lei Zhang. Physhoi: Physics- based imitation of dynamic human-object interaction. arXiv preprint arXiv:2312.04393, 2023.
Yinhuai Wang, Qihan Zhao, Runyi Yu, Ailing Zeng, Jing Lin, Zhengyi Luo, Hok Wai Tsui, Jiwen Yu, Xiu Li, Qifeng Chen, Jian Zhang, Lei Zhang, and Tan Ping. Skillmimic: Learning reusable basketball skills from demonstrations. arXiv preprint arXiv:2408. 15270, 2024a.
Zan Wang. Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Humanise: Language-conditioned human motion generation in 3d scenes. Advances in Neural Information Processing Systems, 35:14959-14971. 2022b.
Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Move you say, interact you Language-guin guiddd human motion generation with scene affordance. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024b.
13

---

Published as a conference paper at ICLR 2026
Qianyang Wu, Ye Shi, Xiaoshui Huang, Jingyi Yu, Lan Xu, and Jingya Wang. Thor: Text human- object interaction diffusion via relation intervention. arXiv preprint arXiv:2403. 2024a.
Zhen Wu, Jiaman Li, Pei Xu, and C. Karen Liu. Hbmen-object interaction from human-leve in- structions, 2024b. URL hatp:::ts:::/aas htt.pps:/aaxx...or///aaaa
www. abs/ wttp:o/gg.ab//......oo
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang. Yifu Yuan, He Wang. Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated part-based interactive environment. In The IEEE Conference Computer Vision and Patlern Recognition (CVPR), June 2020.
Zeqi Xiao, Tai Wang, Jingho Wang, Jinkun Cao, Wenwei Zhang, Bo Dai, Dahua Lin, and Jiang- miao Pang. Unified human-scene interaction via prompted chain-of-contacts. In The Twelfth International Conference Learning Representations, 2024.
Zhaoming Xie, Sebastian Starke, Hung Yu Ling, and Michiel van de Panne. Learning soccer juggling skills with layer-wise mixture-of-experts, In ACM SIGGRAPH 2022 Conference Proceedings, pp- 1-9, 2022.
Zhaoming Xie, Jonathan Tseng, Sebastian Starke, Michiel van de Panne, and C Karen Liu. Hier- archical planning and control for box loco-manipulation. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 6(3):1-18, 2023.
Sirui Xu, Ziyin Wang, Yu-Xiong Wang, and Liang- Yan Gui. Interdreamer: Zero-shot text to 3d dynamic human-object interaction. arXiv preprint arXiv:2403. 2024a.
Sirui Xu, Hung Yu Ling, Yu-Xiong Wang, and Liangyan Gui. Intermimic: Towards universal whole-
body control for physics-based hojen-object interactions. In CVPR, 2025.
Xinyu Xu, Yizheng Zhang, Yong-Lu Li, Lei Han, and Cewu Lu. Humanvla: Towards vision- language directed object rearrangement by physical humanoid. arXiv preprint arXiv:2406. 19972, 2024b.
Haibiao Xuan, Xiongzheng Li, Jinsong Zhang. Hongwen Zhang, Yebin Liu, and Kun Li. Narrator: Towards natural control of human-scene interaction generation via relationship reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 22268- 22278, October 2023.
Hongwei Yi, Justus Thies, Michael J Black, Xue Bin Peng, and Davis Rempe. Generating human interaction motions in scenes with text control. In European Conference on Computer Vision, pp. 246-263. Springer, 2025.
Herman Zanstra. A study of relative motion in connection with classical mechanics. Physical Review, 23(4);528, 1924.
Chengwen Zhang, Yun Liu, Ruofan Xing, Bingda Tang, and Li Yi. Core4d: A 4d human- object-human interaction dataset for collaborative object rearrangement. arXiv preprint arXiv:2406.1935. 2024.
Haotian Zhang, Ye Yuan, Viktor Makoviychuk, Yunrong Guo, Sanja Fidler, Xue Bin Peng, and Kayvon Fatahalian. Learning physically simulated tennis skills from broadcast videos. ACM Transactions on Graphics (TOG), 1244 2023.
Siwei Zhang, Yan Zhang, Qianli Ma, Michael J Black, and Siyu Tang. Generating person-scene interactions in 3d scenes. In International Conference on 3D Vision (3DV), volume 2, 2020.
Xiaohan Zhang, Bharat Lal Bhatnagar, Sebastian Starke, Vladimir Guzov, and Gerard Pons-Moll. Couch: Towards controllable human-chai interactions. In European Conference on Computer Vision, pp. 518-535. Springer, 2022.
Yan Zhang and Siyu Tang. The wanderings of odysseus in 3d scenes. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 20481-20491, 2022.
14

---

Published as a conference paper at ICLR 2026
Zhenhao Zhang, Ye Shi, Lingxiao Yang. Suting Ni, Qi Ye, and Jingya Wang. Openhoi: Open world hand-object interaction synthesis with multimodal large language model. In The Thirty-ninth Anmual Conference on Neural Information Processing Systems, 2025.
Kaifeng Zhao, Shaofei Wang, Yan Zhang, Thabo Beeler, and Siyu Tang. Compositional human- scene interaction synthesis with semantic control. In European conference on computer vision (ECCV), 2022.
Kaifeng Zhao, Yan Zhang, Shaofei Wang. Thabo Beeler, and Siyu Tang. DIMOS: Synthesizing diverse human motions in 3d indoor scenes. In International conference computer vision (ICCV), 2023a.
Kaifeng Zhao, Yan Zhang, Shaofei Wang. Thabo Beeler, and Siyu Tang. Synthesizing diverse human motions in 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14738-14749, 2023b.
Kaifeng Zhao, Gen Li, and Siyu Tang. Dart: A diffusion-based autoregressive motion model for real-time text-driven motion control. arXiv preprint arXiv:2410.05660, 2024.
15

---

Published as a conference paper at ICLR 2026
APPENDICES
Details of VLM Planner
16
InterPlay Dataset
19
Additional Details of Reinforcement Learning
19
Visualizations in Supplementary Material
26
Ablation study on the VLM Planner
26
User Study
27
G Statements
28
Text Instruction
Scene top-view
human, root, target: laundry basketffiont)
object root target: laundrybaskke/ceeee
reletive mow.moooeetttt mynamics::
"Take away clothes
[Cpellis", "root" "1").
("orra", "reot" "11).
("head", 'root', '1'). ("eff_upper_ng) "basket_lef_sside
(Teft Joower_arm", "Sasker side" "1").
('lef_hand", "basket Jeft side "1").
Prompt sections
GPT-4v
("right_upper_ng "ouskerLright_sii sightrssstsdeeee '1')
('right. hoott//owrr.mmtt arm"", "basket righttssis
(right_hand) "bassee_right_iiee '1").
RMD
(Tett, thigh", "root", 137)
understanding
defination
(1ett shin", "rooe": 13%)
foor" "rooe" 13%)
Spedificplan
("right_thigh", "root", "1").
Instance
Idea outline
("right shin", 'roor" 19%)
('rightffoor" "root', "3")
Reference example
Figure 4: An overview of VLM-guided RMD Planner pipeline.
A DETAILS OF VLM PLANNER
As shown in Fig. given top-view image of the surrounding scene and textual instruction, the VLM-guided RMD Planner generates sequence of sub-step plans in the form of structured JSON. This output format facilitates direct downstream processing via Python scripts.
To ensure that the planner performs intended, we design prompts that explicitly guide it to un- derstand the concept of Relative Movement Dynamics (RMD), perceive the contextual scene infor- mation, decompose high-level instructions, and produce outputs in a structured format consistent with our framework. For clarity and modularity, we organize the prompt into distinct sections, each fulfilling a specific functional role as illustrated in Fig. and Fig. The content of each section is detailed below.
Scene-context Understanding. This section the prompt helps the planner interpret the scene context and determine the target destination using simplified relative spatial relationship. Specifi- cally, we reduce spatial relations to seven canonical types: center, forward, back, left, right, up, and down.
16

---

Published as a conference paper at ICLR 2026
Prompl sectionl: scene-conlext underslanding
Prompt section2: RMD concept definition
To effectively plan sequential interactions between hunancid and objects within scene, recognize the type of each object, understand the spatial relationships arong they, and identify the most suitable obfect for interaction at each sub sub-step, based 's top-down view. simplify the spatial objects
Relattive Novement Dynanics represents novesent trend between each huran body part the correspording object part over a period of time, This trand is nodeled as a bipartite groph, where one subset of nodes represents huran body parts, and the other subset represents parts an object.
relative
essential
types: center, forward, back, left, right, up, and down. for exarple, you can use the tem "box [front)" to describe the position that His in front of the box.
relationships
The subset representing huran body parts consists of fifteen rodes correspunding specific regions of the
body:
peluis
torso
head right hand
right perr_uppen__r aam
righh Jowen_arm
Prompt section:: specific plan instance
left_upper_oo
leftt_lver__am
left_hand
right_thigs
rrigh_shinn
right foot
leftt thigh
-left_ shin
leftt foot
haman root, target: box:front), aesoctt carget: bou(conter), reletive, pwweeeess dynemic::
The subset representing the object parts contains x nodes, where x depends on the specific object and instruction. You expactad infer reasonable partitioning the object into parts. Note that your partition should reflect comron divisions of objects as they are understood in everyday life.
("peivss", "roat", "1"),
("sorse", "reet", "2"),
("heae", Seett_upper__r "box wot.lsstt_s000 stge" "9"),
xide", (9"),
("left hand", "sos left xide". *er).
("right_upper uppee_arre "box_ right_stse"" (9(),
1.e. (human (hunan_body__aane body_part, abjoct_body_ body_part, adgo woight)
("right Jower_are" "box_ right stset" (9"),
("right hand", "bou.righ, sida", ter),
("lett_shin', "hhx _idd"" 121)
("left_thigh" "reat". "3").
2, or 3, where:
Each edge
graph can
weight 1,
("left_foot" "box Stde"" '27),
indicates that the body part and object part connected this edge renain relatively stationary with respect to each other over tine.
("right_ thigh", ("right "bourrihh_ssdd"" ('right_foot''
"""),
vide",
indicates that the distance between the body part and the object part connected by this edge steadily decreasing over time.
This instance describes the relative noveseet dynanics during scenario where huranoid bends down to grab a box with both hands and picks it up.
indicates that the distance between the body part and the object part connected by this edge steadily increasing over time.
The "pelvis", "torso", and "head" have an odge weight of 1 with "rost" because the distance between the pelvis, torso, head and the box center decreases as the box is gradually lifted.
indicates that the relative novenent trend between the huraa part and the object part is unstable unclear over time.
The "left uppar. have an edge weight of with "box left side", , indicating that these body parts remain relatively stationary with respect to the box side during this time.
"left
am"" "Ieft hand"
Prompt section plan
The "Jeft "left_shin", "Ieft_ foot" and "box _eft_side" side" have edge weight of body.
1. All instructions pertain to the interaction task between a humanoid robot, nodeled with relative simplicity and equipped with a ball rather than a finger on its hand, and object. This interaction nay dynaric static.
is leaving with lower
The rest of the weight can be obtained in the save way.
The humanoid does not need to sove during interaction, therefore the husan_root porgetareet carrespond the position in front of box.
2. Each plan nay involve sequential interactions with nultiple objects. for each object, you should design a plan comprising least five steps, including approaching and departing from the abject.
The object does not need to be unved in sy plane during the interaction, therefore the root_ target corresponds to the position at the center of the box.
3. Your return about relative palative_novenett. oyyanins_dynanics
should be formatted into sequence of [(hunan body part, object body part, edge Neight), (huran body_part, object body_part,
Prompt section::: idea outline
edge_ weight), ...l, 15 edges for one sequence.
Complex huran-obfect interaction (HOI) tasks can be deconposed into several sub-sequences, where each hunen part naintains a consistent relative sovment trend with its corresponding obfect part until the transition to the next suh- sequenee... Therefare, universally D, with
4. If you confused about which part of the object you should select, you can select the root to represent the whole.
define the objactive the interaction the forsulation,
5. relatively sensitive
judgnent
= (6, fy -..6g]
relstive novenent dynenics, if it is not stable(e-g. when holding box and walking towards destination, alternatey approach and
where 6. denotos the 1-th plan interaction step. will provide you with instruction and top-viow image depicting the surrourding scene context. your task break
feet and the nove away fron each other for period of tine), just set the weight to 3.
the instruction sequences represented relative sovesent dynanics, based on the top-viee image. Please note that for different Hot tasks, the granularity of an object's part decorposition ma vary. This granularity is determined the context and the instruction itself.
sub-
6. Your thought process may unfold as follows: parcetve the surrounding scone context and identify the relevant object, decorpose the instruction into sequence huran-objact interactions, inagine corresponding notions, and represent each sub- sequence in the form of BYD.
Figure 5: Details of prompt section.
17

---

Published as a conference paper at ICLR 2026
Prompt sectiop6: refereuce example
stwol: close lundry basket.
Instruction:
Take away clothes and have a rest.
hasen roct target: laundry brscut(bick),, asject reot target: loundry tasket(cester), reletive soweeene dyyneeess
Top-view image:
("turso' "rast" '1"),
("pelvis", "ront" ":"),
("head", "noot" "1"),
Sopt__upe__am arn', Festet_lert_side (17).
["left] Towe
"nasket_ left side" "3")
("left "leff_hhedd', hand" "basket left xide", "1");
("right, uppor_ore", "bosket rigit
("right, Jower_ore", wwssict_rggtt_teee "1),
("right_hand"" Tbasket_right_sste '17)
("left_thip""
"3"))
('left. Foot", ("right, thigh" "reot" 12%),
("right_onin shin"" "noot", "3"),
("right_/oon foot", "noot",
Prompt seciinii: reference example
stop2: bends cown to grob the box with both hands and picks it up.
steol: welking to the eashing nathine while holding the Laundry basket.
Tuman, target: Laundry (bskkt()
humarroot_tae washing frcnt)e((rront),
object targec: laundry basket(cansar), reletive novoooootttttccmmmm dynanies::
Deject_noot_aare target: washing machine(frart). reletive poveeett pepp_eoweerr__mmmeee.
("pelvtt") "neot", '17),
("pelvis", "noot", 'g"),
("torso", "root", '8"),
("head" "rous".
("head",
("Ieft
"bastet left sice", "0"),
("Ieft_uppe__nm
left_side",
("loft Jower. "basiet sice" '8"),
("Ieft ("Ieftthand", "bosket
left '0"),
("left_hand' hand", "hesket left vide Te").
sidt
("right _rs',, "basket (Tright Jower "banket right vide" '0"), (Tright hand", "basket right side", "0"), ("left Sheeft_thihh", "root", "3"), ("lett ('lett_shin', shin", "hesket_lef" Jeft Jeft_stde" 137), ("let ('left_font" "hestat left side", '2")) Crright Trighi_thighs thigh" "rast", "s"),
xide". '6"),
("right Bright_lowen_oree (Tright "basket Sarptrrrr________ 187).
197)
rigtt_terrtts (9")
("left. Thigt" "reet", ("left_anin", "root". 19"). ("Ieft, foot", "root" 19%),
("right, thight Treot" +3%
("rightiahtn'' "basket, basket_righ dight_ste"" (2"), ("righttfoot"", "basket_ Seaee__rggi__seeeeeeeee
("right shin", traet", "s"),
("right Root", "roet", "s")
Prompt sectionfe reference example
stops: put down the laundry basket.
steps:: get
rontt torget: washing rachinet(roott) peooct_roctttotte voshing (oronne( relutiow roverntt dynamica:
Larget: chuir(frent),
reletive thair(canter),
("palvis", "root", '2").
("pelvix", "raet". '1")
("torso",
("torsu", "rout" "2")
("baad" "roos". '1")
("left
left
("left_inper_ee
Troot". '1"),
("left
left side"
("left ("left_inerr ara"" Troat". '1")>
("left uppar
("left_Sand" "reat", "3").
(right upper arm..
"raes" '1").
("right_ are', "hasket right side". ("right hand"
"onorrtlooorrmmm "root" '17),
"basket right side" '0"),
("on""" "root", "1"),
("left thigh" "ruot". '59).
("left_thigh thigh", "reot" '3"),
("lfft shin". "baskut left side", "1"),
("left shinn. "roat", "3"),
("lett_ foot", "basket Jefftt_Jeff__o Jeft__stee stse", "1"), "reat", '3"),
("Iof(Fooo"" "reot", "2"),
("night_ "basket Banket-iight -ige" side", '17),
("right_thigh"" "noot", "3").
("right ("right_eii"", siin"
("right_sntn", "root". ("right_foot""
("right_foat" foot", "basket - right side", '1")
Prompt sectioni;: reference example
stepf:: sit down
step7: have rest.
Isean root target: chair(canter), 0bsect root target: chair(centar), reletive relotive_evccee www.ovve.oootttttccoooooo dynarics:
pwwprrtttttteeeeeeee tergets their(center), ubject, rost Larget: chair(cuntee), reletive Setetttv_eeeerrr___yyaat..
humen root
("pelvis', "seat, support', "1"),
("torso", "root" "s"), ("haad", "root", "3"),
("pelvis", "seat_ peat_supprtt", '8").
supoort"
("head", "head ptllow
("left_inpee_ne _am""
"root" ("Ieft_loer" arn", "jeft_arr_sofo", ("Ieft_hand", "reat" '3"), ("right upper are". "root", "3"),
"3"),
hand" upper, trright_ooweoone orw", "root". bardght_aand" "roat". "3"),
("right, lower.ore". are". piggttomm_oors.. (1"),
('eet ("ief-_ thigh" tehate foeef, 187).
(iett shin', "chair: Foot" 'gt),
foats, "chain.foot" trright thigh", "chair foot". 18").
(6"),
("rdght treet", "3"),
("left "chotr_root"" 187),
fooe", "chain foot", "e"),
ontn",, "chate Toot", (g").
("right thigh", "chain foot". "6"),
(tright_shin" tehair_ Fooe", -8%),
("right shia", "chain_foot", 181),
rost"" "ehate_ fooe", "0")
("right_Food Foot", "chain_foot",, '6")
Figure 6: Details of prompt section.
18

---

Published as a conference paper at ICLR 2026
These relative positions are defined with respect to local coordinate system centered on the refer- ence object.
RMD Concept Defination. This section of the prompt helps the planner understand the concept of RMD, clarifies its intended semantics, introduces a method for instantiating it, and provides formal definition of its instantiated form.
Specific Plan Instance.
This section presents a templated instance of an AMD-based plan, illus- trating the correspondence between the template elements and RMD components, well how such an instance aligns with a real-world interaction scenario. This helps the planner internalize the concept and encourages outputs that conform to the desired format.
Idea Outline.
This section offers a high-level overview of our approach, specifying the expected input and output of the interaction planner. Human-object interaction tasks are decomposed into sequence of sub-steps, each characterized by consistent movement dynamics.
Plan Rules. This section enumerates the rules that the planner must follow, emphasizing their signif- icance in prompt engineering. These rules define task interaction boundaries, output formatting, and criteria for determining movement dynamics. By adhering these rules, the planner can effectively decompose complex interactions into manageable and coherent sub-sequences.
Reference Example. Empirical findings from the NLP community indicate that one-shot prompt- ing often yields significantly better results than zero-shot prompting. Guided by this insight, we include a complete input-output example in the prompt to help the planner better understand the task expectations and output format.
B INTERPLAY DATASET
To evaluate our framework long-horizon, multi-task scenarios, we construct a novel dataset, In- terPlay, which encompasses both static and dynamic interaction tasks under diverse indoor scene contexts. Each data sample includes: (i) a list of processed 3D assets ready for direct use in sim- ulation, (ii) the corresponding 3D scene layout, (iii top-view rendered image, and (iv) natural language instruction describing the intended task.
In building the InterPlay dataset, incorporate high-quality 3D object assets from several exist- ing datasets, including PartNet (Mo et al. 2019), 3D-FRONT (Fu et al., 2021), SAMP (Hassan eet al.) 202T), CIRCLE (Araújo et al. 2023), and Part sources, we carefully select 117 assets spanning categories such as bed, chair, sofa, washing ma- chine, box, door, window, and wardrobe. Each asset is normalized, segmented into functional parts, and associated with part-level point clouds, based either on existing annotations or manual labeling.
(Xiang al. (020). From these
To generate scene layouts, we apply a set of heuristic rules that ensure: (i) well-distributed ar- rangement without object collisions, (ii) semantically meaningful placements aligned with typical household settings, and (iii) a minimum of two interactive objects and at least two furniture items per scene, thereby enriching the contextual complexity of human-obipe interaction. We then populate these layout templates by randomly sampling from the processed asset pool.
For each completed layout, we render a top-view image and employ GPT-4V Achiam et al. (2023) to generate corresponding task instruction grounded in the scene context. Finally, we apply our VI.-guided RMD Planner to produce interaction plans, which are manually reviewed and refnndd before inclusion in the dataset.
In total, InterPlay contains 1,210 interaction plans, covering wide range of HOI behaviors involv- ing static, dynamic, and articulated objects with varying temporal lengths, all situated in realistic, houschold-ins -insiirdddd--iiedddddee environments.
C ADDITIONAL DETAILS OF REINFORCEMENT LEARNING
Our physics-bast based animation framework is built upon coal-conditioned reinforcement learning. At each time step the agent samples action from its policy <(a st, g:) based the current state s, and the goal state g., After executing the action, the environment transitions to the next state 8111, and the agent receives task reward G (s, gr, 814). Further details provided below.
19

---

Published as a conference paper at ICLR 2026
yincs Sik deg
Jost
borater Toureooten plve
Top
Irtraio Teusmistnnd flaes
Figure Visualization of long-term interaction with objects in an indoor home setting (part 1).
20

---

Published as a conference paper at ICLR 2026
Forare ferdderr
Top baps
Tetcaturs 56 la datto
Intoata:: le Alm
Top stago
Crggi Hagh"
Figure Visualization of long-term interaction with objects in an indoor home setting (part 2).
21

---

Published as a conference paper at ICLR 2026
hodeee Teedd Kid
Tap-sin baags
motel nn anliittiii
Incratis un
Figure Visualization of long-term interaction with objects in an indoor home setting (part 3).
22

---

Published as a conference paper at ICLR 2026
hoteike Thocka marmedyy Kiy five body.
bateekie Thedde diddiii on macy. big bim dty.
haage
Inbru be dedles cnrey.y cole at
Try coe Inage
Figure 10: Visualization of long-term interaction with objects an indoor home setting (part 4).
23

---

Published as a conference paper at ICLR 2026
Porerdee The nenyy, kin the ki
Topaoee bagge
dho pinee conet Eie Entoeeeeed bug Rrm bo
Topaar Lmage
Eatecciv oolast ond bog Rmm bo
Figure II: Visualization of long-term interaction with objects in an indoor home setting (part 5).
24

---

Published as a conference paper at ICLR 2026
Action. We use simulated humanoid character with 28 degrees of freedom (DoF), represented as a kinematic structure composed of 15 rigid bodies. The action space is defined as the target joint positions for 28 proportional proporiioall-derivtivee (PD) controllers. At each timestep, the predicted actions specify these targets, which are then translated by the PD controllers into joint torques to drive the character's motion.
The proprioceptive state s, is a 233-dimensional feature vector that encodes the character's physical state. It includes the positions, rotations, linear velocities, and angular velocities of all rigid bodies, expressed in the character's 's local coordinate frame. The only exception is the root height, which is represented the world coordinate frame preserve global elevation information.
Proprioception.
Task Reward. Recall that the term TRMD encourages each edge ety E B, representing (human- part, object-part) pair, to follow the relative motion dynamics specified by its associated edge weight Waj- as planned by the VLM. Specifically, frmd is computed by summing the individual alignment reward r (-), each of which measures how well the pair (Ph.-Po) follows the motion pattern defined by Waj, weighted by a coefficient Aej-
TRMD
Σ (i.))E
Aaj
"Tnnd (Pij, Vizs wiy)
(15)
Specifically, we formally define r'md (Pij, Vig. Whs) as,
exp
(Piy v(s)")
Wij = 0,
2
1
exp
(-lip,, l12)
+ 1
exp
(vr
P
Pi.
+ +0²))
W/s 1,
Tmud (Paj, Vay, mxy)
(16)
Pij [P: ll
2 exp
exp
Vig
1²
M/ 2,
Wij 3.
The behavior of each reward term varies depending on the value of way!
When Wij 0, the reward promotes either vanishing relative velocity orthogonal orienta- tion between the relative position and velocity, thereby minimizing directional alignment.
For w.y 1, the goal is to maintain spatial proximity while aligning the relative velocity with the direction of the relative position. In particular, the velocity projection is encouraged to approach a predefined target x" m/s.
When Wij = 2, the reward favors increasing separation between parts, while ensuring that the leaving speed along the direction of displacement remains close v*.
Finally, way = denotes an unstable or unpredictable relative motion trend. Here, the reward is set to neutral constant value of avoid imposing misleading gradients.
Training details in a single-task scenario. Our policy network is divided into a critic network and an actor network, each starting with three multilayer perceptron (MLP) layers configured with [1024, 1024, 512] units. The discriminator network follows a similar architecture, also consisting of three MLP layers with [1024, 1024, 512] units. The hyperparameters used during the training process are detailed Tab, 4 Some baseline methods either did not implement certain HOI tasks (e.g., open, push) or failed to account for post-interaction transitions with static objects (e.g., sit, lie). To enable fair and thorough comparison, we re-implemented these baselines with carefully hand-crafted goal states and reward functions for each task. This ensures that we can comprehensively evaluate the full interaction cycle: approaching the object, interacting with it, and departing afterward, across range of individual HOI tasks. All methods were trained using a shared set of hyperparameters to maintain consistency.
Training details in long-horizon multi-task scenario. Our policy consists of separate actor and critic networks, each implemented as a three-layer multilayer perceptron (MLP) with hidden dimensions of 1024, 1024, 512]. The discriminator network adopts the same architecture. Both our method and the baselines trained using two-stage procedure:
25

---

Published as a conference paper at ICLR 2026
Table 4: Hyperparameters for RMD.
Hyperparumeter
Value
Hyperparameter
Value
Hyperparameter
Value
Num. envs
4096
Max episode length
450
Num. epochs
25,000
Discount factor 7
0.99
GAE parameter À
0.95
Minibatch size
16384
Optimizer
Adam
Learning rate
2c-5
Gradient clip norm
1.0
PPO borizon length
32
PPO clip
0.2
PPO miniepochs
6
Actor loss weight
Critic loss weight
5
Discriminator loss weight
5
AMP batch size
512
AMP sequence length 1*
10
AMP grad. penalty wop
5
Task reward weight Orack
0.5
Style reward weight Osylle
0.5
Bounds loss coefticient
10
(i) Single-task pre-training. We divide the parallel simulation environments into distinct subsets according to task type and difficulty, allowing the agent to learn basic interaction skills in a focused manner.
(ii) Long-horizon multi-task post-training. For each environment instance, tasks are randomly sam- pled from our InterPlay dataset, promoting the agent's ability to handle long-horizon human-object interaction tasks in complex and diverse scenes.
To extend InterPhys multi-task setup, concatenate all possible goal states and use a one-hot encoding to indicate the task identity. Additionally, we implement script to translate the sequential plans in dataset into the format required by other baselines (i.e., sequential goal states), thereby unifying the training pipeline. During execution, a base- finite-state machine is used to manage task transitions in both InterPhys and TokenHSI.
D VISUALIZATIONS IN SUPPLEMENTARY MATERIAL
To complement the qualitative results presented in the main paper, we provide a demonstration video that combines the key aspects of our method. This video offers detailed visualizations show- casing the effectiveness of our framework in various real-world scenarios and compares with competing approaches. It includes an overview of the motivation, challenges, main pipeline, and both quantitative and qualitative results. Additionally, the video highlights long-horizon interac- tions in both single-task and multi-task scenarios, involving static, dynamic, and articulated objects within a realistic indoor setting. In the single-task scenario, we present a comparison our method with UniHSI (Xiao et al. 2024) for the sitting task and InterPhys (Hassan et al., 2023) for the door-opening task. While UniHSI struggles with high-frequency jitter and unrealistic motion, our method ensures smooth, human-like transitions. Similarly, in the door-opening task, approach uses coordinated hand movements for a more lifelike interaction, in contrast to InterPhys, which relies solely body movement and produces unnatural results. The multi-task scenario illustrates how our method excels by leveraging human-like motion, a unified objective design across different tasks, and the capabilities of Vision-Language Models generate contextually relevant action plans. Additionally, we provide detailed visualizations alongside the Planner's 's output, as shown in Fig. 7
Fig. X
Fig. 10 and Fig.
E ABLATION STUDY ON THE VLM PLANNER
Model and prompt variants. To assess the robustness of our approach, we performed ablation experiments by (i) replacing GPT-4V with other models such as LLaVA-1.6 and Qwen- VL-Max. and (ii) simplifying the prompt. As shown in Tab. 5 our framework remains effective across different VLMs and prompt designs, with only moderate performance differences. This is possible because our framework is modular, and the VLM planner functions as a plug-in component that can be replaced by any vision-language model with basic visual understanding and reasoning capabilities.
Planner label accuracy on InterPlay. To further quantify planner reliability, we manually anno- tated ideal RMD labels for key human-object pairs across all tasks in each subset (static, dynamic, and hybrid) and compared them with the planner's 's outputs. The empirical rate of contradictory la- bels is 8.9%, 11. 11.2%, and 14.7 4.7% for static, dynamic, and hybrid tasks, respectively, with higher error
26

---

Published as a conference paper at ICLR 2026
Table 5: Comparison of VL.M choice and prompt designs.
VLM
Complction Rate (%) f Sub-step Completion Ratio (%)
Sub-step Precision (em)
LLaVA-1.6
67.8
Owen-VL-Maa.
47.7
68.3
12.7
GPT-4V (Simplified Prompt)
45.3
66.
12.9
GPT-VV
53.8
71.8
11.2
on hybrid tasks due to their complex multi-stage nature. Together with the ablations in Tab. 5 this analysis indicates that our framework robust moderate planner imperfections.
Part decomposition novel objects. To further evaluate the planner's behavior unseen assets, we additionally construct 100 new HOI tasks involving 50 objects that do not appear in the InterPlay dataset and manually inspect the VLM planner's part decompositions. Only 7 of 100 cases exhibit clearly unreasonable part splits, suggesting that the planner generalizes reasonably well to unfamiliar but semantically common objects, likely due its web-scale pretraining.
Stageetransition threshold. To evaluate the sensitivity of method to the stage-transt stage-transition threshold, we perform an ablation study on the AiterPlay-Hybrits subset by sweeping the thresh- old from 0.80 to 0.95. As shown in Tab. 6 performance varies smoothly between 0.80 and 0.90 and peaks at 0.90, while only the extreme setting 0.95 leads to a clear drop in completion and preci- sion. Based on these results, we keep 0.90 the default threshold, since offers the best trade-off between completion, sub-step ratio, and precision while the range [0.80, 0.90) is generally stable.
Table 6: Comparison of threshold choice.
Threshold
Completion Rate (%)
Sub-step Completion Ratio (%)
Sub-step Precision (cm) 4
0.80
47.9
65.6
13.5
0.85
51.0
67.2
12.9
0.90
53.8
71.8
11.2
0.95
32.8
47.2
16.7
F
USER STUDY
To evaluate the visual quality and goal alignment of the generated motions from a human-centric perspective, we conducted a user study with 20 participants. Each participant watched motion se- quences, each paired with its corresponding task description, and rated them two criteria: Motion Realism (assessing whether the interaction appears natural and physically plausible) and Task Con- sistency (measuring how well the motion aligns with the task objective). Each criterion rated on a scale from 0 to 5, where higher scores indicate better performance. The results, summarized in Tab. show that our method achieves superior performance in both realism and task alignment.
Table 7: Comparison of motion realism and task consistency.
Method
Motion Realism 1
Task Consistency 1
InterPhys
3.2 + 0.7
3.5 + 0.5
TokenHSI
3.4 0.4
3.7 0.6
UniHSI
3.1 + 0.5
3.7 + 0.4
Ours
4.0 0.4
4.1 0.3
27

---

Published as a conference paper at ICLR 2026
G
STATEMENTS
Ethics Statement. This work complies with the ICLR Code Ethics. The research involves no human subjects sensitive personal data. All datasets are publicly available, and their use strictly follows the corresponding licenses and established ethical standards.
Reproducibility Statement. This work complies with the ICLR standards of reproducibility. We provide detailed descriptions of the experimental settings, model configurations, and evaluation pro- locols in the main text and appendix, ensuring that our results can be independently verified.
LLM Usage Statement. We disclose that large language models (LLMs) used solely for grammar checking and minor text polishing. All aspects of the research design, experimentation, analysis, and results remain entirely the responsibility of the authors.
28


Reviews:
[{"review_id": "fDRRC30Ehs", "reviewer": "Reviewer_WtPe", "summary": "This paper introduces a novel framework for synthesizing physics-based human-object interactions (HOI) by leveraging Vision-Language Models (VLMs) to automatically generate goal states and reward functions. The core contribution is the Relative Movement Dynamics (RMD) representation—a fine-grained spatio-temporal bipartite graph that models part-level relationships between human and object components during interactions. This allows the VLM to reason about motion dynamics and generate semantically grounded, interaction-aware plans without manual reward engineering. The authors also contribute the InterPlay dataset, which includes long-horizon static and dynamic interaction tasks in diverse indoor scenes. Experiments in both single-task and multi-task settings demonstrate that the proposed method outperforms existing approaches in terms of completion rate, motion naturalness, and generalization.", "questions": "1. The paper claims the contribution of being the \"first unified physics-based HOI synthesis framework leveraging the powerful world knowledge of VLMs,\" but it may not actually be the first work to use VLMs for physics-based HOI. This statement could be somewhat biased.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe introduction of Relative Movement Dynamics is a conceptually elegant and technically sound way to bridge high-level task instructions with low-level motion control. By modeling interactions as a bipartite graph of human and object parts with explicit motion trends, the method captures both spatial and temporal aspects of interaction in a unified manner.\n2.\tThe method achieves state-of-the-art performance across multiple metrics (completion rate, sub-step precision) in both single-task and long-horizon multi-task scenarios. The improvements are especially notable in dynamic and hybrid interaction settings, where prior methods often fail.", "weaknesses": "1. The InterPlay dataset introduced in the paper includes articulated objects not found in current interaction datasets, but it does not specify what types of articulated objects are included or what kinds of interactions are involved.\n2. The 6D pose of objects is crucial. How does the method prevent issues like unintended object rotation caused by inaccurate 6D pose estimation during interaction?\n3. The construction of Graph B seems to be only briefly discussed in the paper, yet I believe the connectivity of this graph and the selection of edge weights would directly impact the generated motion outcomes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RvKeZcI7Ny", "reviewer": "Reviewer_xkXT", "summary": "The paper proposes a unified physics-based HOI framework that uses a VLM (GPT-4V) to automatically construct both goal states and reward functions from a structured representation called Relative Movement Dynamics (RMD). RMD is a part-to-part bipartite graph over human and object components with discrete relative-motion labels (e.g., approach, separate, stationary ). The VLM planner outputs a sequence of RMD steps plus spatial anchors; the RL policy (PPO) then executes them with a composite task reward (human/object destination + RMD consistency) and a style prior. A new InterPlay dataset supports long-horizon static, dynamic, and articulated interactions. Experiments show large gains in completion, sub-step ratio, and precision, with ablations that isolate the value of VLMs, multi-part modeling, and the RMD features.", "questions": "- What is the empirical error rate of the VLM planner on InterPlay, and how does task performance degrade?\n- The stage transition uses a fixed and hand-picked 0.9 threshold. How sensitive are completion and precision to this value, and did you try progress-based or confidence-weighted switching that might be less brittle in cluttered or dynamic cases?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The RMD abstraction gives the VLM a concrete “language” to plan in, and gives RL a direct mapping from that plan to goals and rewards. This avoids per-task reward tuning and makes long-horizon transitions feel principled instead of different parts being glued together. This is a meaningful step toward building more general-purpose simulated agents.\n- I appreciate the completion definition (do the interaction and return to a neutral position). It addresses what usually breaks in HOI: recovery and chaining. The method’s advantage in hybrid / multi-task settings (Table 2) is meaningful.\n- The Interplay dataset, including both static and dynamic objects and requiring multi-object sequences, is a welcome addition to the community.", "weaknesses": "- In the demo video opening door scene, it is actually very hard to see the agent using its hand/stump.\n- The system leans heavily on GPT-4V with prompts.  What is the failure case for GPT-4V that may impact training and learning agents?\n- There are two compute costs here: (i) planner inference (VLM) and (ii) training/execution. It would help to see end-to-end wall-clock and per-episode runtime (planning + control), and how that compares to baselines.\n- The naturalness of the human motion ultimately leans on known motion-style priors (AMP-like discriminator), which means out-of-distribution motion may not be feasible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jBiD30dYKE", "reviewer": "Reviewer_7XuJ", "summary": "This paper addresses the limitations of existing Human-Object Interaction (HOI) synthesis methods—such as reliance on expensive motion capture data, manual reward engineering, and poor support for long-horizon/dynamic interactions—by proposing a unified physics-based framework. The core innovation is VLM-Guided Relative Movement Dynamics (RMD), a fine-grained spatio-temporal bipartite graph representation that encodes part-level relationships between humans and objects (e.g., stationary contact, approaching/separating motion). RMD enables Vision-Language Models (VLMs, e.g., GPT-4V) to automatically generate goal states and reward functions for reinforcement learning (RL), eliminating manual tuning. The framework supports interactions with static, dynamic, and articulated objects, and the authors introduce InterPlay, a novel dataset of thousands of long-horizon HOI plans. Experiments demonstrate that the method outperforms baselines (e.g., InterPhys, TokenHSI, UniHSI) in both single-task and long-horizon multi-task scenarios, producing more natural, task-aligned motions as validated by quantitative metrics and a user study.", "questions": "1. The edge weight categories (0=stationary, 1=approaching, 2=separating, 3=unstable) are heuristic—what empirical or theoretical basis supports this granularity? Could a learned weight space or more granular categories (e.g., varying speed of approach) improve interaction naturalness?\n\n2. How does the VLM handle part decomposition for novel objects with ambiguous functional parts (e.g., a multi-purpose tool)? Is there a fallback mechanism, and how is decomposition quality evaluated?\n\n3. For tasks more complex than those in InterPlay (e.g., \"tidy the living room, wash dishes, and prepare coffee\"), how does the framework scale? Would integrating chain-of-thought prompting (mentioned in future work) quantitatively improve plan coherence and task completion?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Captures fine-grained spatio-temporal dynamics between human and object parts, addressing the limitations of coarse contact-based (e.g., chain-of-contacts) or kinematic-only representations. This enables modeling of continuous interaction dynamics (e.g., carrying an object) rather than discrete events.\n\n2.  Leverages VLMs to translate high-level instructions and scene context into structured RMD plans, which are directly mapped to goal states and composite rewards. This eliminates labor-intensive manual reward engineering, a major bottleneck for scalable HOI.\n\n3. Uniquely supports static, dynamic, and articulated objects, as well as long-horizon multi-task sequences (e.g., pick up → carry → place → sit). Most prior methods are limited to single-task or static-only interactions.\n\n4.  Fills a critical gap in existing HOI datasets by providing long-horizon, context-rich interaction plans, enabling systematic evaluation of multi-task HOI synthesis.\n\n3. Outperforms baselines across key metrics (completion rate, sub-step precision) in both single and multi-task settings. The user study further confirms superior motion naturalness and task alignment.", "weaknesses": "1. The framework only supports single-agent interactions, ignoring multi-agent collaboration (e.g., two people moving a sofa) or social dynamics—key for real-world applications like assistive robotics or collaborative environments.\n\n2. The VLM planner may struggle with extremely complex long-horizon tasks requiring deep hierarchical planning (e.g., multi-step cooking with ingredient prep, cooking, and serving), as noted in the paper’s future work.\n\n3. Object part decomposition depends on the VLM’s implicit judgment, with no explicit evaluation of decomposition accuracy or robustness to novel/unfamiliar objects (e.g., specialized tools).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gfP19jTIzp", "reviewer": "Reviewer_iGBJ", "summary": "For HOI synthesis, this paper’s main contributions are twofold: (1) using a VLM as a high-level planner and (2) proposing RMD as a mid-level interface between planning and control. The method requires neither task-specific motion-capture demonstrations nor manual reward engineering, and supports long-horizon, multi-stage interactions with static objects, dynamic objects, and articulated objects. The authors also introduce InterPlay, a dataset containing thousands of long-horizon interaction plans for systematic evaluation. Experiments show consistent gains over strong baselines (e.g., AMP, InterPhys, UniHSI, TokenHSI) in completion rate and precision across single-task and multi-task settings. Ablations further demonstrate that VLM visual input and the fine-grained RMD representation are key to the performance.", "questions": "* Physical causal reasoning. The method relies on VLM-imagined motion trends. In highly dynamic settings (e.g., projectile interactions or multiple moving objects), can GPT-4V reason causally? Can RMD be extended to handle exogenous object motion (not initiated by the agent)? If objects move autonomously or unexpected events occur (e.g., a rolling ball), can the framework revise the plan online?\n\n* Planner stability. Did you observe VLM hallucinations (e.g., incorrect part segmentation, anomalous RMD weights)? What filtering or correction mechanisms were used? How sensitive are plans to prompt phrasing? Please provide observations and mitigation strategies for planner stability.\n\n* Open resources. Please confirm the release timeline for the InterPlay dataset and the codebase (including prompt templates and exemplar plans).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Comprehensive ablations and analysis. The paper isolates component contributions via ablations and alternative VLMs. Removing visual input or coarsening RMD degrades performance, underscoring the importance of VLM spatial grounding and fine-grained RMD guidance. A user study in the appendix aligns with quantitative metrics, indicating more natural and plausible motions.\n\n* Thorough empirical validation. InterPlay spans diverse scenes and task types. Metrics (e.g., Completion Rate, Precision) are appropriate, and comparisons against representative baselines cover static, dynamic, and multi-task scenarios. Results show sizable improvements (often tens of percentage points in completion) and higher motion fidelity, supporting generalization and robustness.", "weaknesses": "* Planner reliability. While GPT-4V outputs are structured, the planner remains a black box. Erroneous sub-plans could hinder training or misguide rewards (e.g., incorrect reasoning about dynamic object trends). The paper does not quantify or correct such errors. Moreover, cross-phase physical causality is only implicitly encoded via RMD; the VLM may be pattern-matching rather than reasoning about outcomes, which could break in highly dynamic tasks (e.g., throwing/catching).\n\n* RMD’s bipartite modeling misses self-contact and intra-object contact. By construction, RMD only encodes relations between human parts and object parts. Many everyday interactions require human–human self-contact and articulated object self-contact (e.g., folding a clamshell laptop). Such intra-agent and intra-object constraints are not representable with a strictly bipartite graph, which can lead to inaccurate guidance or reward shaping for these tasks.\n\n* Task scope. Extremely dynamic, high-precision, or fast-response scenarios are not covered; it is unclear whether RMD + GPT-4V planning scales to these regimes.\n\n* Writing. The related-work narrative around “using a planner” vs. “the representation that the planner outputs” is not sufficiently disentangled; these are distinct axes and should be contrasted more clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)