OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model
Authors: Zhenhao Zhang
Ye Shi
Lingxiao Yang
Suting Ni
Qi Ye
Jingya Wang
Venue: NeurIPS 2025 oral ()
Keywords: Hand-Object Interaction;Open-World;Large Language Models
Primary area: applications
TL;DR: Introduce the first Open-World Hand-Object Interaction (HOI) Synthesis framework that can generate Long-horizon HOI sequences of Unseen Objects from Open-vocabulary instruction with 3D Multimodal Large Language Model.

Abstract: Understanding and synthesizing realistic 3D hand-object interactions (HOI) is critical for applications ranging from immersive AR/VR to dexterous robotics. Existing methods struggle with generalization, performing well on closed-set objects and predefined tasks but failing to handle unseen objects or open-vocabulary instructions. We introduce OpenHOI, the first framework for open-world HOI synthesis, capable of generating long-horizon manipulation sequences for novel objects guided by free-form language commands. Our approach integrates a 3D Multimodal Large Language Model (MLLM) fine-tuned for joint affordance grounding and semantic task decomposition, enabling precise localization of interaction regions (e.g., handles, buttons) and breakdown of complex instructions (e.g., “Find a water bottle and take a sip”) into executable sub-tasks. To synthesize physically plausible interactions, we propose an affordance-driven diffusion model paired with a training-free physics refinement stage that minimizes penetration and optimizes affordance alignment.
Evaluations across diverse scenarios demonstrate OpenHOI’s superiority over state-of-the-art methods in generalizing to novel object categories, multi-stage tasks, and complex language instructions.

OpenHOl: Open- Hand-Object Interaction Synthesis with Multimodal Large Language Model
Zhenhao Zhang', Ye Shi"; Lingxiao Yang', Suting Ni', Qi Ye', Jingya Wang
'ShanghaiTech University "Zhejiang University
(zhangzhh2024, shiye, yanglx2023, nist2024, sangjiangahehhangaaateed...dd cn
qi yedzju. edu.
Abstract
Understanding and synthesizing realistic 3D hand-object interactions (HOI) is critical for applications ranging from immersive AR/VR to dexterous robotics. Existing methods struggle with generalization, performing well closed-set ob- jects and predefined tasks but failing to handle unseen objects or open-vocabulary instructions. We introduce OpenHO1, the first framework for open-world HOI synthesis, capable of generating long-horizon manipulation sequences for novel objects guided by free-form language commands. Our approach integrates 3D Multimodal Large Language Model (MLLM) fine-tuned for joint affordance grounding and semantic task decomposition, enabling precise localization of in- teraction regions (e.g.. handles, buttons) and breakdown of complex instructions (c.g., "Find a water bottle and take sip") into executable sub-tasks. To synthesize physically plausible interactions, we propose an affordance-dre -driivvveee diffusion model paired with a training-free physics refinement stage that minimizes penetration and optimizes affordance alignment. Evaluations across diverse scenarios demonstrate OpenHOl's superiority over state-of-the-art methods in generalizing to novel object categories, multi-stage tasks, and complex language instructions. Our project page at httiss://ppenhoii github.io
Introduction
Hand-object interaction (HOI) involves jointly modeling hand articulation and object dynamics
to generate and interpret realistic manipulation sequences [9] 17 25 26
[3 47 5]. This reflects
one of the most pervasive human behaviors, deeply embedded in daily activities. Generating and understanding 3D HOI sequences is critical for advancing machine capabilities in human-centric applications. In augmented/virtuul reality (AR/VR), realistic HOI modeling enables immersive digital experiences, allowing users to manipulate virtual objects naturally. For robotics, it provides the foundation for dexterous, feedback-driven manipulation in unstructured environments.
Generating HOI sequences from natural language instructions remains significant challenge interaction research. While traditional methods [19 40 rely on handerafted motion priors, recent diffusion- basedd approaches 22 3 directly map text to action sequences, climinating the need for manual motion design. However, due limited data and modeling capacity, these models only deal with closed datasets and struggle to generalize to unseen objects and open- apen-vocabulary instructions.
Recent advances in Large Language Models (LLMs) have significantly enhanced joint vision- -ssiun-language understanding. Building on this progress, concurrent work like HOIGPT (13) proposes an LLMM-based architecture for aligning textual instructions with HOI sequences. However, these approaches face
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Closed-Set HOI
Open-World HOI
(a) Interact with Unseen Objects
*Use headphones with both hands."
"Imm feeling bit thirsty, could you pour using your right hand?"
want to relax and read, with both hands read it.
book
Seen
"We need can you take with your left hand, please?"
(b) Open-Vocabulary High Level Instruction
"Use with right hand."
"I'm feeling thirsty, you find
bottle and sip?"
0%
"I'm bit dehydrated, please open the water bottle cap with both hands, then drink bottle water using your right hand."
Unseen
Timestamp
Figure 1: Motivation: OpenHOl introduces an open-world framework for generating HOI sequences that demonstrates strong generalization across and unseen objects, high-level instructions, and long-horizon tasks.
fundamental generalization challenges in unseen shapes and complex 3D interaction scenarios due to the lack of 3D knowledge LLM. Fig. Iillustrates the comparison between conventional Closed-Set HOI and Open- Woold HOI.
The growth of 3D datasets has advanced 3D multimodal large language models (MLLMs) with strong capabilities understanding and grounding 3D geometric, semantic, and functional relationships. Critically, fine- mhe-grained part-level grounding and affordance grounding [34 4 can identify actionable object regions and their interaction possibilities, providing strong priors for synthesizing physically consistent hand-object interactions (HOI). Building this insight, propose OpenHOI, the first open-world HOI synthesis framework capable of generating long-horizon manipulation sequences for unseen objects guided by open-vocabulary instructions. Our approach fine-tunes 3D MILMM endowed with comprehensive affordance reasoning priors, enabling two core capabilities: 1) Gen- eralizable Affordance Prediction: Precise localization of interaction regions (c.g.. handles, buttons) for both known and novel objects. 2) Instruction Decomposition: Breaking down open-vocabulary commands (e.g., "I' feeling thirsty, could you find water bottle and take a sip"") into executable sub-tasks grounded in object affordances. By integrating these priors with a diffusion- based inter- action generation and physics-awe refinement, OpenHOI achieves unprecedented generalization across object categories, task horizons, and linguistic complexity.
In summary, contributions
follows:
We introduce OpenHOI, the first open- world hand-object interaction synthesis framework capable of generating long-horizon manipulation sequences for unseen objects guided by open-vocabulary instructions. Unlike prior closed-set HOI methods, OpenHOI generalizes to novel objects and complex, linguistically diverse commands.
OpenHOI involves fine-tuning 3D MLLM that jointly learns geometric affordance priors and semantic task decomposition. Subsequently, affordance-drrven HOI Diffusion with Physical Refinement developed generate realistic HOI sequences for each task.
Our experiments demonstrate state-of-the-art performance, outperforming existing meth- ods with a large margin. Notably, OpenHOI generalizes robustly to unseen objects and open-vocabulary instructions, achieving strong compositional generalization across diverse scenarios.

---

Table 1: Comparative analysis of our method versus existing HOI synthesis approaches.
Methods
Open-world
3D MLLM
Open-vocabulary
Long-horizon
Planning
Motion In-between
HandDiffuse
Text2hoi
HOIGPT
Ours
2
Related Work
Hand-Object Interaction Synthesis. Synthesizing realistic and diverse hand-object interactions (HOI) have gained significant attention, with numerous approaches exploring this problem under various settings. These include the creation of large-scale datasets like GigaHands (7) and OAKINK2 [53]. methods for affordance learning [21,56] 32 45). Others have focused on synthesizing two- physics-aware synthesis and dexterous grasping [44] 46 57). Several key works have specifically tackled text-guided or semantically rich HOI synthesis. Tex12HO1 ( introduced pioneering approach to generate 3D Hand-Objec Interaction Sequences from text by decomposing the task into contact prediction and motion generation using diffusion model, but it can struggle with fine- meneggr control from low-level text and tends to produce short HOI interaction sequences. HOIGPT [13] adopts an LLM- -Lsed sequential model predict hand-object trajectories from text for long-horizon sequences, yet it lacks explicit affordance- segidddeegededd mechanisms and cannot ensure smooth motion in-between multiple sub-sequences in complex interactions. SemGrasp (20) focuses semantic-aware grasp generation, but it discretizes motion space and is limited to static grasps. Grasp as You Say [46] addresses category-level language-guided grasping, but it requires hand-crafted hand-object contact templates and lacks dynamics, While these methods have advanced the field, generating fine- grre-graind, diverse, and oong-horizon hand-object interaction sequences with high-level text on unseen objects cohesively remains a significant challenge.
hand interactions like InterHandGen 19 and HandDiffuse [23).
Multimodal Large Language Model. The advent of Large Language Models (LLMs) has rev- olutionized natural language understanding. and their influence gradually expanding into 3D perception and interaction tasks. Some recent works leverage pure LLM architectures handle motion generation from language. For instance, HOIGPT (13) and MotionGPT (T6) adopt VQ-VAE tokenization and GPT-style architectures to achieve bidirectional generation between natural language and HOI systhesis. However, these models operate primarily on discrete motion tokens, and lack grounding in actual 3D perception reasoning, they cannot perceive objects or environments in 3D nor reason about spatial affordances or constraints, thus limiting their ability to generate context- aware interactions. Conversely, Multimodal Large Language Models (MLLMs) have emerged as powerful tools that extend the success of LLMs into various domains, aiming to bridge the gap between language and other modalities such images, videos, and 3D representations, enabling tasks like visual question answering like ShapeLLLM (34), 3D object generation like Point-E (29) and CLip-Forge (37. and affordance understanding like LASO (22), DAG[42]and AffordanceLLM (35]
These MLLMS demonstrate strong capabilities in static 3D understanding. especially in segmenting or identifying affordances from language prompts. Other MI..M-based approaches like SeqAfford (52
and 3DD-Affordancel..... 4 further extend this to sequential or compositional affordance reasoning. However, these approaches remain limited to perception tasks and do not address the synthesis of continuous hand-object interactions or motion dynamics. In contrast, while LLM-based methods focus on generating motion from text without grounding 3D environments, and MLLM-bai -ased methods enable 3D spatial understanding without supporting interaction generation, none of the exist- ing approaches bridge both capabilities. Our work addresses this gap by leveraging an MLLM that trained to jointly model language, 3D perception, and interaction dynamics. This enables our model to both understand object affordances from language and synthesize long-horizon, task-consistent hand-object interactions with unseen objects from apen-vocabulle instructions. By unifying reason- ing and generation, our approach extends 3D MI.LMs toward fine-grained, dynamic, and open- world HOI synthesis.
3

---

(I)Affordance and Sub-tasks Extraction
(2) Afffforraneedrriven HOI Diffusion
Conditional Inpur
failing thiney, card you a water bottle and like cipp!"
Object Point Cloud
(a) (a)anniing
Clip Encoder
Pretrained 3D Visson Encoder
Cleaifier-fro Guidance
Tiist the bottl cap both hands."
"Then drink the bottle using your right
<AF 1o <AFF
Task-l
Tisk-2
Affondance
Sequence
Multimodal-LLM
X;
token
feuture
Conditoral Deasise
Diffusion Model
Transformer
Linear
Xg
Original Sequence
Lineer
Large Language
LoRA
(>Trrinnigg-ree free Refinement
Generated HOI Sequenee Siart
Took-1 Soq
Affeed Aligament
Taski
Complete
"I'm delyydrood, please open the water buttle
Diffusion
Matton hetwooe Aliamne
cap HAHF 1- with both hands, then drink the bottle
Sequence
<ATI using
Sub-tasks
ascc
f
Affordance
Affordance Decoder
Sog
Peoetraiin
Aligrneent
End
Figure 2: Pipeline: Our framework comprises two sequential components. First, a multimodal large language model (3D MLLM) ingests high-level instructions and object point clouds to generate sequential affordance maps and decompose the high-level task into sequence of sub-tasks. Second, the diffusion model takes the affordance map and the decomposed task sequence as conditions to synthesize realistic hand-object interaction sequences.
Method
We propose OpenHOI, novel framework for generating long-horizon hand-object interaction sequences guided by high-level open-vocabulary instructions. OpenHOI generalizes to unseen object categories by leveraging semantic reasoning from a 3D MILM paired with 3D affordance prior. Our framework consists of two core components:
3.1 Instruction Decomposition and Affordance Reasoning via 3D MLLM
Given a 3D object point cloud and a high-level.free-fooe instruction (e.g., "I am thirsty"), the 3D MLLM semantically grounds the instruction into object-centric affordances by bridging the gap between abstract intent and actionable object functionality. This process yields two structured outputs: (1) A spatial affordance map that identifies geometrically plausible interaction regions (c.g.. highlighting the cap of a water bottle for opening and the body for grasping).
(2) A temporally decomposed sub-task sequence that translates the instruction into executable atomic actions (c.g., 1) grasp bottle cap with both hands > 2) twist counterclockwise to open - 3) lift bottle to mouth with right hand").
Network structure of 3D MILM. Recent advances in 3D multi-modal language modeling have significantly enhanced open-world understanding of 3D objects. In particular, ShapeLLM (34] has been pretrained to capture wide range of embodied interactions. Consequently, we adopt ShapeLLM as our backbone. Its point-cloud encoder ReCon++ pretrained via multi-view distillation from ReCon (33]. and its language component is initialized from LLaMa [41]. Prior approaches to 3D affordance prediction have typically relied on standalone 3D backbones [5T] or on separate point-to- language encoders [22). which often lack robust reasoning and open- world generalization capabilities.
By leveraging a unified 3D MILM rather than exclusively using pure LLMs or conventional visual architectures, we achieve both enhanced generalization to unseen objects and affordances and the intrinsic integration of affordance perception into natural language representations, thereby facilitating subsequent affordance reasoning.

---

In-context 3D Affordance Reasoning. Although 3D MLLMs effectively align 3D representations with natural language. they are predominantly tailored for object-centric text generation and thus lack inherent support for dense 3D prediction tasks such as fine-grained affordance segmentation. To address this limitation, we augment the MLLM vocabulary with dedicated segmentation token <AFF>, following the design of Lisa (18), thereby enabling the model to represent and reason about segmentation outputs within its language-based framework. Formally, given a point-cloud input Foej and instruction text Tins expressing user intent over candidate objects, the 3D MLLM jointly encodes both modalities to produce sub-task sequence:
sub sub_tasks tasks
MLLM abj. Tim).
which contains S occurrences of the segmentation token <AFF>, each marking a predicted affordance region. We then collect the corresponding last-layer embeddings { (h 1 and pass each through an affordance decoder:
Jimo
n Decoder (F
(Fanj. barr)
i= 0. i= 1-1.
(2)
Coarse-to-Fine Affordance Tuning. Our training pipeline comprises two sequential stages. First,
we fine-tune the model on large-scale, coarse-grained, object-centric static affordance datasets [35
(2]. to instill strong affordance priors, thereby enhancing its generalization and open-vocabulary capabilities. Second, fine-tune the stage model on smaller dynamic hand-object interaction dataset, which can get fine-grae aineggranee affordance maps by using the voxel-based method, to generate more precise affordance Maps and task decomposition sequences. We leverage an auto-regressive cross-entropy loss Lunk to supervise sub-tasks generation, complemented by Dice loss and Binary Cross-Entropy loss Lair guide affordance prediction.
L = Arask (Y "uu__ sub_uakks Lasks
sub Sub_rssks + LatrLan(AAas Aut)):
where the weights Atakk Aaff are utilized to balance the different loss items.
3.2 Affordance-driven HOI Diffusion with Physical Refinement
To generate long-horizon, natural, and physically plausible hand-object interaction sequences X based Interaction Prior C from Sec 3.1 which include affordance prior, sub-tasks, and object pointcloud as follows,
C= S mskks) Fobil:
We propose Affordance-driven HOI Diffusion, which decomposes this challenging task into tractable components. During training, we improve the alignment of the affordance map via classifier- free guidance [1] 30 [2). During inference, we propose training-free refinement method that generates natural, physically plausible, and long-horizon HOI sequences.
Training HOI Diffusion with Affordance Prior from MLLM. During training, given a condi- tioning pair [X, CJ, we first add noise to X using the forward diffusion process:
p(X, Xo) ~ (Vo, (1 - T)
(5)
where Xo denotes the original HOI sequence, X, is its noisy version at diffusion timestep t, and ag controls the noise schedule. We then train transformer- based neural network X, to directly predict the HOI sequence Xo from the noisy input. The training objective for HOI generation is defined as:
Thai train (xa, C) Tnno dirl (Xo, C) + Thoi distance (Xo) ) Thoo orient TAnoi_orient (xo):
(6)
- (X,, t, C is the denoising loss of the diffusion process for HOI reconstruction. Thoi distance (8)) donates the HOI distance map loss, which encourages precise surface contact and enhances the physical plausibility of the contact area by penalizing eтrors more strongly when the hand is near the object. Lnoiooriem (xx)) represents the HOI relative orientation loss, which aligns predicted orientations with ground truth to ensure accurate rotational poses for tasks like grasping and manipulation by modeling hand-objec relative rotation, respectively. To better align the generated HOI sequences with the Interaction Prior C, we adopt the classifier-free guidance strategy:
where Looi diff( Xo. C)= =
XX,,,, t, C)= Xo(,,, t, 0) + s (Xo C) -x((X,,,0))
(7)
5

---

where s is the guidance scale. To enable this strategy, we randomly mask 10% of the conditional t.
inputs C during training train the unconditional (8) model (X,,,,t, 0) alongside the conditional one. During sampling, we gradually denoise X, to Xo via posterior distribution pl X Xx):
X Me + 02e, (0, I),
(8)
where fı vor(1-au- X + Vor 1 Xj(Xe,t, t, C)2 and or 1-a:
1-8--1 Be.
Training-free HOI Diffusion with Physical Refinement. While the vanilla diffusion sampling process in Equation & models the joint conditional distribution of HOI sequences, it suffers from two key limitations: 1) Lack of Physical Constraints. The sampling process does not inherently enforce physical plausibility (28). such as preventing interpenetration ensuring stable contact. While prior work addresses this by training discriminative physical refiners, such methods often deviate from the underlying joint HOI distribution, introducing artifacts due distribution shift. Temporal Incoherence. The generated sequences exhibit discontinuous hand motions, particularly during transitions between suu-eeeueesss, leading to unrealistic motion dynamics.
To address these limitations, we propose three key refinement objectives during the diffusion sampling process: 1) affordance refinement for precise contact, 2) physical constraint refinement prevent penetration, and 3) temporal coherence refinement for smooth transitions, all achieved without additional training distribution-shiffing..
a) Affordance Refinement: Since direct affordance-based HOI generation often fails to produce precise hand-object contact, we propose an affordance-aware loss to guarantee geometric consistency:
lai- = lea [iaand,
obj ) + Il right [[d([thamd, 5tjoiit )|?
where d(...) is the Euclidean distance between hand joints (Jmand, Sthand) and closest affordance prjoint).
region (Fligint pb)
b) Penetration Refinement: To mitigate interpenetration artifacts, we propose penetration loss: Ipenetration en [ld( [id(Vinand obj )1 I right []d(( Vitandd poj
(10)
where .((..)) is the Euclidean distance between hand joints (Viandd Vetaan) and closest affordance
region (Plvar ; )
c) Motion In-between Refinement:
For seamless transitions between HOI sub-sequences, synthesize natural hand motion vor bridging the end pose VT of the preceding sequence and the start pose after of the subsequent sequence. The transition loss can be defined as:
Iransition
trans
VT
11
1 trans
after
2 2
(11)
where
00 trans trans denote the predicted start and end frames of the transition motion.
While gradient descent after each denoising step could minimize these losses, risks introducing artifacts and distribution shifts [9). Recently, a training-free conditional diffusion model named DSG (50) offers larger, adaptive step sizes the loss function to achieve better alignment with the constraints while preserving the original distribution learned by the diffusion model. Inspired by DSG. we also introduce the Spherical Gaussian Constraint during the sampling stage to preserve the original distribution, thus mitigating the Distribution-shifiin problem. We utilize the analytical solution to enforce steepest gradient descent to enhance alignment:
p"=-Vdor V x, (Xx((X L, c)),
(12)
where d represents the data dimensions and denotes the loss in Eq 9 TO II for refinement stage. To enhance the sample quality, we utilize a mixture of deterministic steepest gradient descent direction and random sampling direction:
Dmix D saappl + w (D* Dsample),
(13)
Dunx
X,-1 14 Vdo,
(14)
[[Dmnix]
where Dsample JeEe is the random sampling direction, and represents the guidance rate.

---

Experiments
Our framework integrates two core stages: 1) fine-tuning 3D MLLM to predict object affordance maps and decomposing open-vocabuee vocabulary instructions into concrete sub-tasks, and 2) synthesizing HOI sequences with the condition output stage We evaluate our method using diverse datasets and metrics, demonstrating its capability to generate long-ho horizoo HOI sequences with high-level instructions of both seen and unseen objects. Comparative experiments and ablations validate our design choices. Our experiments were conducted on NVIDIA A100 GPU.
4.1 Dataset
For our experiments, we utilize two prominent hand-object interaction datasets: GRAB 388 which provides comprehensive full-body motion data of subjects interacting with 51 everyday objects, and ARCTIC (6, large-scale dataset specializing in bi-manual interactions with articulated objects and dense 3D annotations. We preprocess both two datasets with unified pipeline, which involves initial geometric operations on the object point clouds specificall upsampling to support detailed affor- dance map generation via our inference model, followed by downsampling to ensure computational efficiency during training. Crucially, we enrich the original annotations by employing a multimodal large language model (MILLM) to convert low-level HOI motion descriptions into open- apen-vocabulary. intent-centric language instructions. These semantic labels enhance the expressiveness of the data and allow model to better generalize to unseen scenarios, supporting open-world HOI synthe- sis with long-horizon sequences and diverse objects. For both GRAB (38] and ARCTIC (6], we follow standard protocol by partitioning each dataset into 80% for training and 20% for unseen testing, ensuring reliable evaluation of our model's ' generalization capabilities. This setup allows us rigorously assess performance unseen objects, motions, and interaction intents, validating the robustness of our MLLM-guided generation framework across both single-hand and bi-manual interaction scenarios.
4.2 Evaluation metric
To comprehensively evaluate the quality, diversity, realism, and physical plausibility of our generated hand-object interaction (HOI) sequences, employ multi-faceted set of quantitative and qualitative We roughly divide the evaluation indicators
metrics inspired by prior works 2 13 24
40 14].
into three categories
Motion Accuracy. We evaluate geometric precision using the Mean Per-Joint Position Error (MPJPE;) computed over hand joints, and assess object placement with the Final Object Location Error (FOL;), defined as the Euclidean distance between the predicted object center and the target location at the final frame.
Generation Realism. We measure realism via the Fréchet Inception Distance (FID;) between real and synthesized motions in pre-trained motion feature space, capturing distributional alignment and perceptual fidelity.
Diversity & Multi-modality, Diversity quantifies across-prompt variability of generated outputs, while MModality captures within-prompt variability across multiple samples. Both are computed from pairwise distances or variance statistics in the motion feature space.
Lower MPJPE, FOL, and FID indicate higher accuracy and fidelity, while higher multi-modality and closer to GT diversity reflect stronger generative expressiveness.
4.3 Main Results
Comparison with SOTA Methods. We evaluate our method under the widely adopted seen / unseen split protocol and compare it against saae-ooftthe-ar methods (i.e., MDM (39], TM2T (10). MotionGPT [16). Text2HOI (2) both GRAB (38) and ARCTIC (6) datasets. As shown in Table and Table our method consistently outperforms all baselines across both seen and unseen objects.
These results firmly establish our method as stat-oof-the-at, demonstrating the strong generalization ability of our MLLL--uuided Affordance Reasoning and Affordance-driver HOI diffusion with Physical Refinement generation, which validates the effectiveness of our Open-
Open-World World HOI synthesis
7

---

framework in generating instructions.
HOI sequences of unseen objects from Open-vocabulary
Table 2: Main Results on GRAB.
Method
MPJPE4
FOL.
FID 4
Diversity
MModality
GT
4.66
MDM 39
24.92+2.25
0.62++002
1..37+1.56
0.18
12.77++0:45
Seen
TM2T TO
59.27+1.19 0.46+0.06
57.41+2.30
3.60+0.07
21.28++0.82
MotionGPT
63.94+2.56 0.43+0001
52.03++1.22
3.61=0.08
20.26++0.51
Text2Hr
56.29+2.13
0.44+0.03
33.72+1.27
3.41=0.16
17.71+0..87
Ours
1..6
0.26.10.02
26.43
3.69-0.27
24.59-12.01
MDM 39
1.89
0.69
7..59
3.07+10.11
11.15:10.85
Unseen
TM2T TO
1..07++1.34
0.55+00.2
1..63-11.66
3.37+00.07
14.03-10.67
MotionGPT
6..26+1.99
0.51+0.01
56.49+1.98
2.852+0.007
16.36+0055
Text2HOr
60.67+1.80
0.41+0.02
36.96+0..77
1.80+00005
10.98+0.444
Ours
51.34.10.85
0.27=10.11
28.2911.62
3.61-0.09
19.91-10.63
Table 3: Main Results on ARCTIC.
Method
MPJPE4
FOL4
FID
Diversity
MModality t
GT
3.39
MDM
39
72.67=0.63
0.60
33.66:t0.19
2.35
8.20+10.00
Seen
TM2T
10
54.39=0.64
0.41
34.12+0.49
1.67
13.60=#0117
MotionGPT
16
60.17=0.72
0.41+0.03
31.58::0.46
1,89:00002
13.23+0.09
Text2HO
52.16:10.41
0.33
23.35++0.33
2.43
11.21
Ours
0.25
1..74
2.05
11.25
MDM
39
86.75+1.35
0.64
41.53++1.77
1.58+0004
7.13++0.63
Unseen
TM2T
10
55.57=1.26
0.53
37.22+
1.54
11.23++0.44
MotionGPT
T6
0.741=0.73
0.43+0.04
33.99++2.43
1.50+0.09
11.08
Tex12HO12
2
57.83=1.61 0.39
25.22:L0.59
1.61
7.11
Ours
07.25-10.99
0.28
20.05:10.80
2.49
12.66:10.71
Qualitative results. We present the generated HOI sequence results. Open-world capability en- ables the generation of long-horizon HOI sequences under both unseen-object and open-vocabulary conditions. In this section, long-horizon HOI results are shown in Fig. 3 Experimental findings demonstrate physical realism and coherence of the generated sequences.
4.4 Ablation Study
We conduct systematic ablations on the GRAB 38 and ARCTIC [6] dataset validate the necessity of our core components through controlled experiments. Each component is carefully analyzed to demonstrate its contribution robust hand-object interaction generation, the details can be found in Table 5
Affordance Awareness (w/o Affordance). We remove the part of obtaining accurate affordance maps by a well-trained MI.LM, which significantly degrades interaction quality on each evaluation metric. Without this component, the model loses its ability focus on functionally critical object regions, leading to unnatural hand placements and increased penetrations. This confirms that explicit affordance grounding is essential for semantically meaningful interactions.
Classifier-Free Guidance Diffusson (w/o CFG). Classifier- Free Guidance(CFG) plays critical role in ensuring that the generated hand-object interaction (HOI) sequences remain aligned with the input conditions C. Without CFG, the model often fails to adhere to the conditioning signal, leading to semantic misalignment and affordance mismatch.
Loss-guided Physical Refinement (w/o penetrattoon w/o lax). Experimental result demonstrates the critical importance of our loss- guidance strategy for temporal coherence and physical plausibility.

---

Seen object
C: I'm feeling hungry, could you apple so can eat?
Sure, please hand the apple with your right hand, then eat using your right hand.
Unseen object
Timestamp
<: I'd like some milk, could you help me with that?
T Of course, please open the milk container with both hands, then drink the milk using your right hand.
Multi-un bulti-objects
Timestamp
feel like playing with some toys, can you bring me something fun?
Alright, please hand over the duck with your left hand, then fly the airplane using your left hand.
Timestamp
Figure 3: Qualitative result: The visualization results showcase three types of long- horizon sequences sequences-seen--oeee -beeno.ject,, unseen-object, and multi-object. The experiments demonstrate that our method exhibits strong generalization on both unseen objects and open-vocabulary instructions, enabling open-world HOI sequence synthesis.
Without this guidance, the diffusion model exhibits two key failure modes: first, it fails generate complete long-horizon HOI sequences, particularly in complex bimanual interactions where discrete actions remain disjointed rather than forming fluid motions; second, produces physically unrealistic results with frequent object penetrations (lpnnetrtion)) and unnatural affordance patterns (latt), most noticeable during precision manipulation tasks. Through direct optimization of these constraints via our loss aasiduddeece framework, we achieve significant improvements in generating physically valid and temporally coherent hand-object interactions.
Table 4: Ablation Study GRAB
Method
MPJPEL
FOLL
FID.
Diversity
MModality
GT
4.66
w/o Affordance
56.04 + 1.22
0.36 - 0.03
31.60 0.93
3,47 0.05
16.89 1.50
Seen
w/o CFG
51.54 + 0.80
0.41 1.110.00
29.52 0.90
3.39 - 0.18
20.55 + 1.80
w/o penetraatio
53.42 + 0.37
0.01
28.43
0.74
3.31 - 0.21
18.59 1.74
w/o laff
49.17 + 0.36
0.36
0.43 1.3+000 0.02
29.20 0.85
3.41 0.21
19.25 + 1.90
Ours
47.64+1.03
0.26+0.02
26.43:1.777
3.6910.27
24.59+2.01
w/o Affordance
60.37 + 1.69
0.40 - 0.08
36.82
3.04
15.72 + 1.50
Unseen
w/o CFG w/o Ipenetration
55.04 + 0.83
0.41
37.05 1.38
3.24
18.46 1.13
56.68 = 1.6
0.10
35.69
3.36
18.78 + 0.38
w/o laff
54.56 0.47
0.19
36.13
3.28
18.29 0.38
Ours
51.34+00.5
0.27+0.01
28.29 0.62
3.61+0.09
19.91

---

Table 5: Ablation Study on ARCTIC
Method
MPJPEL
FOL
FID
Diversity
MModality
GT
3.39
w/o Affordance
53.03 + 2.76
0.40 = 0.03
26.21 + 0.54
2.03 + 0.19
13.68 + 0.37
Seen
w/o CFG
52.79 +: 0.95
0.38 0.02
25.44
+ 0.29
3.39 + 0.18
14.05 2.72
w/o penetration
51.66 + 0.88
0.37 - 0.01
26.73
+ 0.37
2,45 0.28
13.99 + 0.58
w/o laff
46.35 + 1.13
0.39
25.08
+ 0.20
2.31 + 0.07
13.06 1.34
Ours
45.15:0.94
0.25+0.04
19.74
2.6510.16
15.25+1.44
w/o Affordance
57.29 2.33
0.43 0.03
30.05 0.71
1.97 0.36
11.22 0.60
Unseen
w/o CFG
56.25 0.95
0.45 = 0.02
2.25 1 0.24
11.45 + 1.03
w/o penetration
55.66 + 0.14
0.41 = 0.03
27.51 1.04
2.13 0.10
10.58 0.39
w/o laff
49.18 + 1.13
0.42 = 0.02
27.36 0.25
2.08 + 0.13
11.01 + 1.02
Ours
07.25:10.99
0.28=0.03
20.05=0.80
2.49-10.08
12.66+0.711
Failure cases and error analysis
Given a scenario, there is a row of cabinets, and the instruction "Open the cabinet", our model can open one but cannot target a specific one (e.g.. "Open the second cabinet"). This limitation arises because the model has not been trained on large-scale 3D QA dataset, resulting in reduced logical reasoning capabilities. Due to the accumulation of model errors, performance typically degrades after more than three consecutive actions(mort more than 450 frames).
5 Conclusion
We present OpenHOI, the first spendworld framework for synthesizing long-horizon 3D hand- object interaction (HOI) sequences guided by open-vocabulary instructions. By fine-tuning 3D) multimodal large language model (MLLM) to jointly model geometric affordances and decompose semantic instructions, method achieves strong generalization to unseen objects and linguistically complex tasks. The integration of affordance- diffusion-basee generation and physics-aware refinement enables physically consistent manipulation sequences, advancing beyond closed-set HOI synthesis methods. Extensive evaluations demonstrate OpenHOl's superiority in handling novel object categories, multi-stage tasks, and open-ended language commands, bridging critical gaps in human-centri Al applications. While OpenHOI represents significant step toward open- open-world HOI synthesis, several challenges remain: Although our physics-aware refinement improves interaction plausibility. fine- mne-grained dynamics (c.g.. fluid simulation for pouring tasks) remain challenging. Hybrid neuro-symbolic physics models may enhance realism. While OpenHOI supports multi-stage stage tasks, handling compositional cong-horizon sequences like "cook meal" remains challenging due to limitations in hierarchical task decomposition. Future work could explore thain-of-thousee reasoning to better model such complex action hierarchies.
Acknowledgement
This work was supported by National Natural Science Foundation of China (62406195, 62303319), Shanghai Local College Capacity Building Program (23010503100) Shanghai'Tech A14S Initiative SHTAI4S202404, HPC Platform of ShanghaiTech University, and MoE Key Laboratory Intelligent Perception and Fuuann-achine Collaboration (ShanghaiTech University). and Shanghai Engineering Research Center of Intelligent Vision and Imaging. This work was also supported in part by computational resources provided by Feloud CO., LTD.
10

---

References
[1] German Barquero, Sergio Escalera, and Cristina Palmero. "Seamless Human Motion Compo- sition with Blended Positional Encodings", In: (2024).
[2] Junuk Cha al. "Tex12H01: Text-guided 3D Motion Generation for Hand-Object Interaction". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
2024. pp. 1577-1585.
[3] Sammy Christen et al. "DiffH20: Diffusion-based synthesis of hand-object interactions from textual descriptions". In: SIGGRAPII Asia 2024 Conference Pupers. 2024.
[4] Hengshuo Chu et al. AffD-AAffrdanceel LLM: Harnessing Large Language Models for Open- Vocabulary Affordance Detection 3D Worlds". In: arXiv preprint arXiv:2502.2004 (2025).
[5] Zekai Deng et al. Human-Object Interaction via Automatically Designed VLM- Motion Policy. 2025. arviv: 2503 18349 [cs. Cv] URL: URL: hel::htts://arre arwiiv .rgg btrgg/abs/2555 /ttbo///550.... 18349
[6] Zicong Fan et al. "ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation". In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2023.
[7] Rao Fu al. "GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities". In:
(2025).
[8] Alexandros Graikos, Srikar Yellapragada, and Dimitris Samaras. Conditional Generation from Unconditional Diffusion Models using Denoiser Representations. 2023. arXiv: 2306 01900 org/ abs/ 2306. 01900.
[ URL:
[9] Kristen Grauman et al. Exo4Ex Understanding Skilled Human Activity from First- and Third-Person Perspectives. 2024. arXiv: 2311. 18259 [cs.CV] URL: hattps://axx:: org/
abs/2311. 18259
[10] Chuan Guo et al. "TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts". In: ECCV. 2022.
[11] Jonathan Ho, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models", In: Advances in neural information processing systems 33 (2020), pp. 6840-6851.
[12] Jonathan Ho and Tim Salimans. Classifier-Free Diffussion Guidance. 2022. arXiv: 220:,2220..v 12598
[GG URL: https: htt///xxivvorrr/ awaaaa//2220) .2598
[13] Mingzhen Huang et al. "HOIGPT: Learning Long Sequence Hand-Object Interaction with
Language Models" In: IEEE Conference on Computer Vision and Puttern Recognition (CVPR). Nashville, USA, 2025.
[14] Chaofan Huo, Ye Shi, and Jingya Wang. "Monocular Human-Object Reconstruction in the Wild". In: Proceedings of the 32nd ACM International Conference on Multimedia. 2024, pp. 5547-5555.
[15] Kaiyang Ji et al. Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesis. 2025. arXiv: 2508 02106 [cs.CV]
URL: https
//arxiv... org/
02106
[16] Biao Jiang et al. "MotionGPT: Human Motion Foreign Language". In: Advances in Neural Information Processing Systems 36 (2024).
[17] Taein Kwon et al. 1120: Two Hands Manipulating Objects for First Person Interaction Recog- nition. 2021. arXiv: 2104. 11181 [cs. CV] URL: hattss://rrxt:: htttps///aaxx..or//abb////// http//bb/2100.... 11181
[18] Xin Lai al. "Lisa: Reasoning segmentation via large language model". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 1579-9589.
[19] Jihyun Lee et al. "InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion". In: The IEEE Conference on Computer Vision and Patlern Recognition (CVPR). 2024.
[20] Kailin Li al. "Semgrasp: Semantic grasp generation via language aligned discretization". In: European Conference on Computer Vision. Springer. 2024, pp. 127.
[21] Peiming Li et al. "ClickDiff: Click to Induce Semantic Contact Map for Controllable Grasp Generation with Diffusion Models". In: Proceedings of the 32nd ACM International Con-
ference on Multimedia, MM '24. ACM, Oct. 2024, pp. 273 281. DOI: 10. 1 1145/3664647 3680597 URL: http: / /xx doi org/10. 1145/ 3664647 36805597 [22] Yicong Li et al. "LASO:
sagguage-guides Affordance Segmentation on 3D Object". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 14251-14260.
11

---

[23] Pei Lin. "HfudDiffuse: generative controllers for two-hand interactions via diffusion models". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. 5. 2025, pp. 5280- 5288.
[24] Xueyi Liu and Li Yi. "GencOH Diffusion: Towards Generalizable Hand-Object Interaction Denoising via Denoising Diffusion". In: The Twelfth International Conference on Learning
Representations. 2024.
[25] Yun Liu et al. TACO: Benchmarking Generalizable Bimanual Tool-ACiionn-Oieeer Understand-
ing. 2024. arXiv: 2 [cs.cV] URL:
08399
[26] Yunze Liu et al. HO14D: A 4D Egocentric Dataset for Category-Level Human-Object Interac-
tion. 2024. arXiv: 2203. 01577 [cs. CV] URL: http::tttt::///te pttbppp://arxiv..om wwwp://gg//aaaaaeeooooooo hbtp///a.2222... 01577
[27] Haowen Luo, Yunze Liu, and Li Yi. "Physics-aware Hand-object Interaction Denoising". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. 2024,
pp. 2341 1350..350.
[28] Rundong Luo et al. PhysPurt: Physically Plausible Part Completion for Inteructable Objects.
2025, arxiv: 2408 13724 [cs.CV]
URL: https://aarree aww.vv.arvvvvvv.m wrg/ ab//abb////4 /408 13724
[29] Alex Nichol et al. "Point-e: A system for generating 3d point clouds from complex prompts".
In: arXiv preprint arXiv:2212.087
(2022).
[30] Alexander Quinn Nichol and Prafulla Dhariwal. "Improved denoising diffusion probabilistic
models". In: International Conference on Machine Learning. PMLR. 2021, pp. 8162-8171.
[31] Mokai Pan al. UniDB++: Fast Sampling of Unified Diffusion Bridge. 2025, arXiv: 2505
21528
URL: pettts://arxiiy org/ absa 2505/ 21528
[32] Aditya Prakash et al. "How Do Do That? Synthesizing 3D) Hand Motion and Contacts for Everyday Interactions". In: Computer Vision and Pattern Recognition (CVPR). 2025.
[33] Zekun Qi et al. "Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining". In: International Conference on Machine Learning. PMLR. 2023,
pp. 19223-28243.
[34] Zekun Qi et al. "Shapellm: Universal 3d object understanding for embodied interaction". In:
arXiv preprint arXiv:2402. 17766 (2024).
[35] Shengyi Qian al. " "Affordancellm: Grounding affordance from vision language models". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 7587-7597.
[36] Zhongwei Ren et al. "Pixellm: Pixel reasoning with large multimodal model". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 26374 26383.
[37] Aditya Sanghi et al. "Clip- Towards zero-shot text-to-shape generation". In: arXiv preprint arXiv:21100 21..2122222244 (2021).
[38] Omid Taheri et al. "GRAB: A Dataset of Whole-Body Body Human Grasping of Objects". In: European Conference on Computer Vision (ECCV). 2020. URL: https: /gra//rrbb....
tue tue
mpg.de de
[39]
Guy Tevet al. "Human Motion Diffusion Model". In: The Eleventh International Confer- ence on Learning Representations. 2023. URL: https: http://ppppeevieeee wwt/ ? idd=
SJ1kSy02jwu
[40] Jie Tian et al. "Gaze-guided Hand-Object Interaction Synthesis: Dataset and Method". In: arXiv preprint arXiv:2403. 16169 (2024).
[41] Hugo Touvron et al. "Llama: Open and efficient foundation language models". In: arXiv preprint arXiv:2302. 13971 (2023).
[42] Hanqing Wang et al. DAG: Unleash the Potential of Diffusion Model for Open- Vocabulay
3D Affordance Grounding. 2025, arXiv: 2508. 01651 [cs. CV] URL:: httpt:tttss:::t/iise
hhas/2508.018
[43] Hanqing Wang et al. SDEval: Safety Dynamic Evaluation for Multimodal Large Language Models. 2025. arXiv: 2508 06142 [cs.CV]
URL: https: //t//axivvv./. org abs 2508
06142
[44] Wenbo Wang et al. "UniGraspTransfomeer Simplified Policy Distillation for Scalable Dexter- ous Robotic Grasping". In: arXiv preprint arXiv:2412.02699 (2024).
[45] Yi-Lin Wei et al. "Afforddexgrasp: Open-set language-guided dexterous grasp with generalizableiinstre affordance". In: arXiv preprint arXhv:2503.0736 (2025).
12

---

[46] Yi-Lin Wei et al. "Grasp you say: Language -uiddeeeeggdddddddee dexterous grasp generation". In: arXiv preprint 19X2 (2024).
[47] Qianyang Wu et al. THOR: Text to Staman-Objer Interaction Diffusion via Relation Interven-
tion. 2024. arXiv: 2403. 11208 [cs URL: https:///tatti:ii: httss://axxiv org/ abaaab//249400 2403. 11208
[48] Shijie Wu al. "AffordDP:: Generalizable Diffusion Policy with Transferable Affordance". In:
arXiv preprint arXXi:24122.03144 (2024).
[49] Kaustubh Yadav. A Comprehensive Study on Optimization Strategies for Gradient Descent
Deep Learning. 2021, arXiv: 2101 02397 [cs.LG]
URL: atttps://arxvv. org/ abs/
2101.00 .23997
[50]
Lingxiao Yang al. "Guidance with Spherical Gaussian Constraint for Conditional Diffusion". In: Forty-first International Conference on Machine Learning. 2024.
[51] Yuhang Yang et al. "Grounding 3d object affordance from 2d interactions in images". In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023, pp. 10905- 10915.
[52]
Chunlin Yu et al. "SeqAfford: Sequential 3D Affordance Reasoning via Multimodal Large Language Model". In: arXiv preprint arXiv:2412.01550 (2024).
[53] Xinyu Zhan et al. "Oakink2: A dataset of bimanual hands-object manipulation in complex task completion". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 445-456.
[54] Hui Zhang et al. ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulation. 2024. arXiv: 2309 03891 [cs.RO] URL: https: axxiv. org/ abs
2309. 03891
[55]
Hui Zhang et al. GraspXL: Generating Grasping Motions for Diverse Objects at Scale. 2024. Xiv: 2403. 19649 [cs URL: hhttss://arxi:: org/ /403. 19649
[56] Wanyue Zhang et al. BimArt: A Unified Approach for the Syuthesis of 3D Bimanual Interaction
with Articulated Objects. 2025. arXiv: 2412. 05066 [cs.CV]
CRL: https://arxiv. org/
2b1/ 05066
[57] Yiming Zhong et al. "DexGrasp Anything: Towards Universal Robotic Dexterous Grasping with Physics Awareness". In: arXiv preprint arXiv:2503.0822 08257 (2025).
13

---

OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model Appendix
Appendix:
Implementation Details
Instruction Decomposition and Affordance Reasoning via 3D MLLMM
B.1
Affordance Reasoning
B.2
Instruction Decomposition
Dillusion Process
4
Loss-guided Physical Refinement
Additional Experiments
Results on Extreme- Case: Completely Unseen Datasets
C.2
Evaluation for physical realism
3D
MLLM tunnng
Sensitivity Analysis
Ablation Study Multi <AFF>
C.6
Ablation Study on Motion In- In-between Refinement
Visualization Affordance
6
Qualitative results Compare with SOTA
C.9
Statistically Insignificant
Code and Datasel
D
Code
8
D.2 Dataset
Discussion
Generalizability ability in HOI: Affordance as key
E,2
How to use 3D MLLM in Emboided AI: Choose Powerful Foundation Model and Coarse-to-fine tuning
10
E.3 Future of Work: Real- World Applications, about AR/VR and Robotics
10
A
Implementation Details
3D MLLM. We initialize our model from the ShapeLLM-7B checkpoint, freezing its 3D encoder and augmenting the visual backbone with Uni3D for robust dense 3D prediction, while the projection head is implemented as a shallow MLP, and LoRA is applied to streamline fine-tuning. Training unfolds in two stages: first, we optimize for seven epochs with AdamW (learning rate 10-4 zero weight decay) under cosine-annealing schedule and 2% linear warm-up; then, we continue for three additional epochs with AdamW (learning rate 10-4, zero weight decay) under the same cosine schedule but a 1% warm-up.

---

Diffusion Model. We employ T = 1000-step noising process with cosine noise schedule, and inject positional information at both the frame- and agent-levels using sinusoidal encodings. During sampling, we apply classifier-free guidance by randomly substituting 10% of conditioning inputs with unconditional noise while retaining 90% of the original conditions, and use guidance scale of 2.5 to steer the denoising trajectory.
B Instruction Decomposition and Affordance Reasoning via 3D MLLM
B.1 Affordance Reasoning
3D Object Point Cloud Encoding. We take input point cloud of object, sampled N points. The backhone is a ReCon++ (4 network (or a similar architecture) that processes these points and produces per-point feature representations
Fobj IR NxC.
which capture both local geometric details and the overall global context.
Multi- Token Fusion Mechanism. Rather than generating a single <AFF> segmentation token, PixcILM defines, each visual scale f, a segmentation codebook comprising N learnable <AFF> tokens. After encoding the textual prompt, the model sequentially outputs the N tokens, each associated with hidden vector h4. A linear projection & then aggregates these vectors into a unified representation
h =M,
which is concatenated with the scale-specific image features and fed into the pixel decoder to produce the final segmentation mask. Experiments on the MUSE validation set indicate that increasing N from to 3 improves cloU, demonstrating that the multi-token fusion mechanism captures more nuanced semantic details and significantly enhances fine-grained segmentation.
B.2 Instruction Decomposition
OpenHOI decomposes single high-level instruction into an ordered sequence of actionable affor- dance steps. Each step is marked by special token and then grounded spatially the 3D point cloud.
Instruction Text Encoding. We take a natural-language instruction Tins as the model input. The backbone is LLaMA-style Transformer that produces token-wise hidden states (hx ( and aggre- gates them via pooling operation into a single embedding
hols E ER.
which is then used for downstream tasks.
Segmentation Token Injection. Extend the MLLM's vocabulary by adding special marker <AFF>, which explicitly denotes the boundary of each sub-task in the generated sequence.
Conditioned Autoregressive Generation. Given the fused 3D point features Fobj and the instruc- tion embedding hass the Transformer predicts an interleaved stream of action words and <AFF> tokens, for example:
Pick
<AFF>
Twist
<AFF>
Lift
<AFF>
Let S be the total number of <AFF> tokens generated.
Boundary Localization Hidden-State Extraction. Record the positions 11,...,44 where <APF> appears. For each 1,..., S., extract the corresponding last-layer hidden vector
h4. E RD.
which encodes the full context immediately preceding the end of sub-task i.

---

Sequential Mask Decoding. For each step i, use the query E, to perform aross-attention over the point features Fabj and decode per-point mask:
M(()) = or (Decoder(FFp, =))
- lo....
where F are the fused features and o is the sigmoid activation. Collect the ordered set {M, ,,,..., Ms3 } obtain the final sequence of affordance masks, each aligned with its cor- responding sub-task.
B.3 Diffussin Process
Our framework employs diffusion models to learn the conditional distribution p(X|C), of hand-object interaction (HOI) sequences, where the conditioning signal combines:
Object affordance prior Aobj
Sub-task embedding folip (T sub_tusks)
Object point cloud features Fobj
Forward Process. The diffusion process gradually corrupts the input data through the forward process with fixed noise schedule on [o,
P(X, Xo) (Vva,Xo, (1 T.)
(15)
where Xn is the original HOI sequence, X, represents its noisy version at timestep / and Ö₁ I Cty This forward process progressively transforms the data distribution into a tractable Gaussian distribution NN0, I).
Loss Function. Like VAEs, the diffusion model can be optimized by maximizing the ELBO:
/ po(No.r (C)
log (Xx C) = log
(16)
Po( (C)Xxr())((X1TTAAXM C) dX
log
(17)
(XxiTIXo:C C)
(o
log Ep(Xur
(18)
pl (011111 Xo, C)
po Po(oo
EptXir (X,,c) (),
log
(19)
p(X 1:7 Xo: C)
By Simplification, Eq.
19
can be reduced to the following:
Po (C)
+ arg min ā a1)
arg max E
1 Xo.c)
log
p(X1.T No. C)
20 (1- | C) - -Xol
(20)
where o,
1-6r-1 Br After removing constant terms, we obtain the denoising loss in diffusion T-a;
models:
Lnoi (Xo. C) Ex Xx-x(x (x)
Xo C)112
(21)
We also introduce geometric loss, including distance map loss Luoi distance and relative orientation loss Lboi orient for physical plausibility. To enable classifier free guidance, we randomly mask 10% of the condition to train an unconditional model Xo(X,, I, 0). Since the unconditional model captures the natural HOI sequence, it is then utilized prior to generate seamless transitions between different HOI sequences.
Sampling Process. During sampling, employ classifier-fre guidance enhance alignment with the conditioning input C. This approach demonstrates superior performance compared to using only the conditional model Xx(X,,,, C):
(X,,, C) Xo(X,,t, t, 0) + (Xo(X,, C) Xo(XX,, t,0,, 0)),,
(22)
3

---

where s 2 controls the guidance strength. We generate samples through an iterative denoising
process using the reverse diffusion posterior:
p(Xi-1 (o.xxx Xt) =
(23)
((X,(xn) N (XXivvaX - a4 N va--iv -x-1)1)
(24)
N (Xi: vā (1 a1
- -N( var(1 1 f + vāa-1 04 ox)
(1 - a1) -
(25)
-
1- āi
~ N/ Xt-1; varll a + va-- (x)1(11a)xxxXAp1
(1 - at)(1 (
(26)
dq
1 āe
B.4 Loss-guided Physical Refinement
Loss guidance is a technique that minimizes the off-the-shelf loss function L(Xo, y) during the sampling time:
min L(Xo. y) X
(27)
s.t. X, M
where y is the conditioning input, M denotes the conditional data manifold that follows the con- ditional distribution p(Xo learned by the diffusion model. In this work, propose novel loss-guided sampling strategy that explicitly enforces physical constraints during the denoising process to achieve more realistic hand-objes interactions.
C Additional Experiments
C.1 Results on Extreme-Casee Completely Unseen Datasets
Result on H20. We further subject our model to extreme-case testing to stress its generalization under the most challenging conditions. All objects and instructions in the H20 dataset entirely novel to models trained on GRAB and ARCTIC, making evaluation on H2O a particularly stringent test of generalization. Despite this extreme distribution shift, our experiments demonstrate that the proposed model nonetheless delivers robust and state-of-the-as performance (shown in Table AI).
Table Al: Unseen Results on H20. Training on GRAB / ARCTIC and evaluation on H2O.
Method
MPJPEL
FOL.
FID
Diversity
MModality
GT
3.43
MDM 39
95.1243.21
0.64
70.54
2.36+1.111
12.50=10.50
GRAB
TM2T 10
40.45+4.50
0.6810.03
65.47+3.20
2.51
14.45=10.60
Motion
16
85.08+44.00
0.61+0.02
60.12+2.80
2.73+00.13
15.93+00770
Text2HO1
80.25+3.80
0.63+00025
55.23+2..50
1.90++0.14
+7.00+0080
Ours
4..68
0.52+
51.33++3.41
3.07+0.27
18.15
MDM 39
105.32+5.00
0.80+0.04
75.89+3.50
1.90
11.27=40.35
ARCTIC
TM2T 10
98.47=4.88
0.82+0.03
72.55+3.30
2.10
13.05=00:45
MotionGP
16
93.15
0.74
65.3743.00
2.30
12.56
Texi2HO1
88.02-14.30
0.71+0.025
60.28+2.80
1.50+10.1
14.78
Ours
81.36
0.63
55.78
1.693
1..44
C.2 Evaluation for physical realism
We supplemented our experiments by evaluating Physical Realism and IV metrics against the closest baseline, Tex (HOIGPT's code not publicly available) in A2 and conducted ablation studies on Physical Refinement module in A3

---

Table A2: Comparison with Tex12HOI
Method
Physical realism 1
IV /
Seen
Tex12HO1
0.87=10.03 0.93:10.02
11.74:1
Ours
9.25:10.73
Unseen
Tex12HO1
0.79-+0.05 0.89110011
14.63-t:107
Ours
10.35:1082
Table A3: Ablation Study on Physical Refinement
Method
Physical realism
IV 4
Seen
w/o Physical Refinement
0.897 0.93:00.02
10.75=10.80
Ours
0.75
Unseen
w/o Physical Refinement
0.84
12.27=00.48
Ours
0.89++0.01
10.35:10.82
C.3 3D MLLM Fine-tuning
We fine-tune the MILM on the Affordance dataset [22 and the HOI dataset [38 6]. We first perform coarse gaarsneg fine- mhe-tuning on the Affordance dataset to instill strong affordance priors, and then carry fine-grained tuning on the HOI dataset produce final model [43 The results shown in
Table, A4
Table A4: MLLM Coarse-to-Fine Affordance Tuning
Method
AUC 1
w/o Fine-tuning
68.77
Coarse searse-grained tuming
84.65
Coarse-to-Fine Tuning (full model)
87.02
C.4 Sensitivity Analysis
We conducted a sensitivity analysis on the guidance rate, and the results are as follows(shown in Table A5 and Table A6). Our experimental results demonstrate that the proposed model maintains robust performance even under these challenging conditions.
C.5 Ablation Study on Multi <AFF>
The additional ablation study results are as follows
C.6 Ablation Study on Motion In-between Refinement
Motion In-between Metric. To the best of our knowledge, previous work has defined an evaluation metric for motion in-between hand-object interaction. We thus introduce a simple yet effective measure, the "Smooth Rate," to quantify the temporal continuity of interpolated motion segments as follows,
dFID dt
SmoothRate
(28)
where dt is the derivative of time. The results are shown in Table T8ble Table A9
5

---

Table A5: Guidance Rate on GRAB
Guidance Rate
MPJPFL
F
FIDD +
Diversity
MModality
GT
4.66
0.5
58.0810.87
0.3840.01
34.40:1..57
3.35-10.06
18.21:0.31
Seen
2.0
5..6610.62
0.29+0.03
27.45=1.13
3.63=0.02
23.35=:0:66
2.5
1..64+1.03
0.26+:0.2
26.43:10.77
3.69=0.27
24.59-12.01
3,0
1.08
0.32++0.03
26.75=#0.0
3.57=0.10
23.86=0077
5.0
58.92+1.28
0.33++0.22
34.29:+0.47
3.40=0.08
23.55+00.51
0.5
61.39+2.45
0.40+0002
35.44+1.45
3.32#0.11
13.34+0.47
Unseen
2.0
54.95+1.30
0.30+0.00
29.21
3.55=0.07
18.70=0.79
2.5
0.83
0.27++0.01
2..29+0062
3.61
-9.91+0.63
3.0
14.62+1.61
0.33++0.01
28.61+0081
3.50+0.33
5.0
1.99
0.35
37.25+0082
3.34
18.81 +1.41
Table A6: Guidance Rate ARCTIC
Guidance Rate
MPJPEL
FOLL
FID |
Diversity + MModality
GT
3.39
0.5
52.23 1.06
0.40
11.05
1.97:10.19
12.77-10.45
Seen
2.0
1..94
0.28
20.38
2.62-10.04
14.96-10.61
2.5
0.91
0.25:10.04
1.17
2.65++0.03
15.25+11.44
3.0
1.75
0.27+0002
21.03:#0.67
2.68
15.03+1.75
5.0
53.2512.077
0.38+0.02
32.85+2.04
3.40=0.06
12.86-1..07
0.5
11.67
0.35+0.02
31.5440.68
2.07=0.05
10.1810.26
Unseen
2.0
07.70
0.30+0002
20.
2.46100.7
12.36:10.99
2.5
07.25
0.28+0.03
20.05
2.49
12.6611.771
3.0
07.76+0066
0.29+0001
21.07
2.51
12.50
5.0
54.1910.89
0.34
27.3
2.21
10.53+0.69
To determine the most suitable window size the following comparative experiment
for the motion in-between algorithm, conducted
C.7
Visualization Affordance
In this subsection, we present visualizations of open-world affordances and unseen objects in
Fig. AT
C.8
Qualitative results Compare with SOTA
This section presents additional visual comparisons between approach and existing oftte-of-theeart (SOTA) methods.
Seen Objects.
Qualitative results on seen objects in Fig. A2
Unseen Objects.
Qualitative results on unseen objects in Fig. A3
C.9
Statistically Insignificant
For the ablation study, we performed paired two-sample t-tests, repeating each test five times and reporting the mean p-value. As summarized in Table the proposed method is significant the 95% confidence level for the majority of metrics(P-value $0.05).

---

Unseen Object Affordance
want to play computer games
hungry
thirsty
want listen to Joy Chou
Template-free Generalizability
C go shopping, can you pick up these bags?
Sure! pick with these parts!
Figure Al: Visualization on Affordance
am thirsty, could you help me ? course, please use the bowl drink some water with both hands. Timestep
MDM
Tox
Ours
Figure A2: Qualitative results on seen object

---

Table A7: Ablation on Affordance Configuration (Single vs. Multi)
Method
MPJPE 4
FOL +
FID +
Diversity +
MModality 1
Seen
Single <AFF>
5..05=0.82 1.06
0.33:10.01
29.31
3.30:1..12 3.69:1.27
21.05
Multi <AFF>
0.26:10.02
26.43:10.77
24.59
Unseen
Single <AFF>
56.48
0.39+10.02 0.27
34.15::1.08 28.2910.62
3.40:H0.22 3.61
17.03++1.25
Multi <AFF>
0.83
19.91
Table A8: Ablation Study of Motion In-between Results on GRAB
Setting
Method
SmoothRate
w/o Motion In-between
38.18 + 6.75
Seen
Ours
2.98 + 0.43
w/o Motion In-between
35.25 + 4.91
Unseen
Ours
3.70 + 0.61
D
Code and Dataset
D.1
Code
We will release our code possible. GitHub OpenHOl
D.2
Dataset
This dataset contains 571,645 synchronized multi-view RGB-D frames captured with five Kineet sensors in three indoor scenes. Each frame includes 3D poses for both hands, 6-DoF object poses, and verb-noun action labels (36 classes). Split into training (344,645), validation (73,380), and test (153,620) frames, H2O supports egocentric interaction recognition and manipulation benchmarks.
H2O.
want to read that interesting story again!
Reading storybook is great choice. Please cradle the volume with one hand on each cover and bring it to comfortable viewing angle and distance. Timestep
MDM
Text2HOI
Ours
Figure A3: Qualitative results on unseen object

---

Table A9: Ablation Study of Motion In-between Results on ARCTIC
Setting
Method
SmoothRate
w/o Motion In-between
41.37 + 5.98 6.77 + 2.17
Seen
Ours
w/o Motion In-between
44.05 + 5.63
Unseen
Ours
6.04 + 3.25
Table A10: Window Size Compare on GRAB
Size
SmoothRate (Seen) + SmoothRate (Unseen)
4.77 + 0.68
5.26 + 2.47
3.59 0.81
4.58 + 1.05
5
2.98 + 0.43
3.70 + 0.61
10
3.24 = 0.65
4.08 = 0.82
20
3.30 = 0.57
4.19 = 0.74
Dataset annotation. For both the GRAB 38 and ARCTIC (6) datasets, we preprocess the datasets and annotate the semantics. After that, the object point clouds are first upsampled to gain accurate affordance maps using our inference model while ensuring fine geometric details in the meantime. Our model then processes the upsampled data to infer accurate affordance maps, which are subsequently downsampled match the original resolution for efficient computation.
We preprocess both datasets and get their semantic annotations. First, we upsample the object point clouds enhance geometrie details and generate accurate affordance maps using our inference model. The upsampled data is then processed to infer affordance maps, which are downsampled back to the original resolution for computational efficiency.
To enhance the semantic alignment between language and interaction, we employ a large language model (L.LM) to refine the original hand-object interaction (HOI) descriptions. The LLM generates high-level, natural language annotations that better capture the intent and dynamics of HOI.
E
Discussion
E.1
Generalizability ability in HOI: Affordance as a key
Affordance is powerful and explicit prior for interaction that can guide complex and fine-grained HOI synthesis. Our model achieves strong generalization by using affordance middleware layer.
Open-Worke Affordance Grounding: We first employ coarse-to-fie tuning strategy to equip the model with strong affordance reasoning capabilities, enabling to generate open-world affordance grounding. This enables the synthesis of realistic HOI sequences, demonstrating strong template-free generalization capabilities.
Affordance serves as a crucial condition: The open-world affordance grounding serves as a crucial condition for the affordance-driven HOI Diffusion. We incorporate affordance not only during training but also in the loss-guidance applied during inference.
Affordance-basse Refinement: we design an improved refinement strategy based on affor- dance: for the interaction between the hand and the object, optimize based affordance grounding rather than the conventional closest-uurfac-peeee approach.
A Template-free Example: Our 3D MILM has learned from wide variety of cups, it can still generate accurate affordance grounding for completely unseen mug. Accurate affordance grounding will guide the synthesis of realistic HOI sequences.

---

Table A11: Two-test Statistically Insignificant
Metric
w/o Affordance
w/o CFG
w/o penetratton
w/o laff
MPJPE (Seen)
3.1 10-6
2.0 10-4
7.5 x 10
2.6 x 10
MPJPE (Unseen)
4.4 x 10-5
1.2 x 10-4
5.5 x 10
2.6 x 10
FOL (Seen)
1.7 x 10-3
6.8 10-5
1.7 10
3.1 x 10-3
FOL (Unseen)
2.2 x 10-2
2.0 x0-3
4.9 x 10-2
3.0 x 10-1
FID (Seen)
2,4 x 10-5
x 10-3
1.0 x 10
2.0 X 10-3
FID (Unseen)
2.2 10
8.7 10
2.0 10
2.5 X 10
Diversity (Seen)
1.5 10-1
1.0x 10
4.4 x 10
1.4 10-1
Diversity (Unseen)
1,8 x 10
3.0 10
4.6 10
4.8 X 10
MModality (Seen)
2.6 10
2.5 10
5.0 X 10
1.1 10
MModality (Unseen)
1.0 10-3
x.7 10
2.2 x 10
2.2 X 10
E.2
How to use 3D MLLM in Emboided AI: Choose Powerful Foundation Model and Coarse-to-fine tuning
The 3D multimodal large model has been widely applied in HOI and embodied intelligence, and it is very important to choose basic model suitable for downstream tasks. In OpenHOI, chose ShapeLLM as our base model, ShapeLLM is powerful 3D foundation model that performs exceptionally well multiple downstream tasks (e.g., Embodied Visual Grounding, Visual Question Answering, and Scene Understanding), making it highly suitable for HOI tasks.After research, we found that ShapeLLMM has the following advantages and disadvantages
Advantages: ShapeLLM is trained on a large amount 3D embodied interaction data and achieves state-of-the-art performance across various downstream tasks. It possesses strong priors in 3D interaction and demonstrates impressive zero-shot 3D representation capabilities.
Disad vantages: ShapeLLM has not been trained on part-level object annotations, which limits its reasoning capabilities for fine-grained object understanding.
In order to make the selected 3D base model as suitable as possible for our task, we need to use data to fine tune the model. We adopt coarse-to- to-sse-t--fie tuning strategy: first pre-train the model object-centric affordance dataset to enable to acquire strong affordance priors. Then, we fine-tune the model on HOI datasets to better align the semantics with the target domain. This allows the model to learn highly effective affordance representations.
E.3 Future of Work: Real-World Applications, about AR/VR and Robotics
OpenHOI can be extended to wide range of future work in other fields. We have listed several noteworthy areas and provided preliminary solutions for the challenges in future applications
Robotics Manipularion: OpenHOI can be integrated into real-world robotic manipulation sys- tems, including industrial robot arms and service robots, to enable more flexible and human-like
interactions 48 54 55)
Open-World World Affordance Grounding as Powerful Guidance for Robots:
Leveraging Open- HOl's 3D MLLM, our method performs open-world affordance grounding facilitate the identification of feasible grasping, pushing, and tool-use regions on novel objects, thereby significantly improving success rates for pick-and aad--ndd-ppcee assembly, and tool- tool-handling tasks.
Realistic HOI sequences synthesis for Robot Manipulation: The HOI sequences generated by OpenHOI can be adapted into robotic manipulation sequences. First, we can use an extraction algorithm to obtain the object's 's 6-DoF pose. Then, inverse kinematics are applied to the wrist parameters to compute the robot arm's pose. Finally, a retargeting algorithm transfers the human hand motions onto various robotic hand configurations for manipulation. This approach ensures smooth, precise, and robust manipulation behaviors real-world deployments.
10

---

Virtual Reality Vision:
By synthesizing realistic 3D hand hand-object interaction sequences, OpenHOl enables users to manipulate virtual objects naturally, for example, by picking up. twisting, or pouring items, thereby enhancing immersion in training simulators, gaming, and virtual prototyping.
Challenges: Robotic manipulation tasks typically demand rapid inference. We plan to employ DPM-Solver to accelerate the diffusion inference process, which is an important direction for our future w3rk[3]]

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As stated in the abstract and introduction, this paper introduce the first Open- World Hand-Object Interaction (HOI) Synthesis framework that generate Long-horizon HOI sequences of Unseen Objects from Open-vocabulary instruction with 3D Multimodal
Large Language Model.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The analysis of limitations is provided in Section 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
12

---

Answer: [NA]
Justification:
Our work dose not include theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
These details are provided in the Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
13

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will release the code once the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iiss.cc/pill wttpcccpppppppppeeeeeeeeeeeeeeee httpa////giieee/eeeSSbiiiiisssss (Cooeooomiissssseeeeeeee sww.iissisiiiiiiiico for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
These details are provided in Section and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The standard deviation in our experiment results (Table 2 and Table 3) shows the statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
14

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information on the computer resources provided See and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpps:/neurips. http::://eerrip..ccpbbbtt/tttte
Answer: [Yes]
Justification:
Our research conformed with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper primarily focuses on theoretical research in how to employ the diffusion model in affordance learning, with no consideration of societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
15

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our experiment poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The implementation is cited properly in the Appendix
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
16

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
17

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
We do not involve LLMs as any important, original, or non-standard compo- nents.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standarr components.
Please refer to our LLM policy ptttps://neuriiee / for what should or should be described.
18


Reviews:
[{"review_id": "3tWxu6yGzj", "reviewer": "Reviewer_7v2M", "summary": "OpenHOI is the first open-world hand-object interaction (HOI) synthesis framework that generates long-horizon manipulation sequences for unseen objects guided by open-vocabulary instructions. It fine-tunes a 3D multimodal large language model (MLLM) to support two key abilities: (1) Affordance prediction, localizing actionable regions on novel objects; and (2) Instruction decomposition, breaking complex commands into executable sub-tasks. The framework combines affordance reasoning with diffusion-based interaction generation and physics-aware refinement, achieving state-of-the-art performance and strong generalization across object categories, task structures, and language complexity.", "questions": "1. In line 193, shouldn't the term \"hand joints\" actually refer to hand surface vertices? If so, how did you recover the hand's surface?\n2. In the affordance refinement, why is the distance computed to the nearest point on the entire object surface, rather than to the nearest point within the predicted contact region?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strength**\n- The paper is well-structured and easy to follow overall.\n- Open-world HOI generation is a highly valuable and challenging direction. The paper effectively leverages the strengths of 3D MLLMs to provide affordance priors and integrates them with HOI generation, generating interactions under high-level instruction.\n- Both the qualitative and quantitative results are promising, and the accuracy of the affordance prediction is also reasonably good.\n\n\n\n### **Weakness**\n- The visualizations provided for affordance prediction and hand-object interaction are somewhat limited, particularly for unseen objects. More qualitative examples would help better illustrate the model’s generalization capability.", "was_revised": true, "final_justification": "The paper makes a meaningful contribution to addressing HandOI, with well-motivated objectives and comprehensive experiments. The rebuttal has also resolved my concerns, so I am inclined to recommend acceptance.", "rebuttal": "We greatly appreciate Reviewer 7v2M's valuable comments and insightful questions. We will address your questions in detail below.\n\n>Q1:The visualizations provided for affordance prediction and hand-object interaction are somewhat limited, particularly for unseen objects. More qualitative examples would help better illustrate the model’s generalization capability.\n\nA1: Thanks for your suggestion. We included additional visualizations of HOI sequences for unseen objects in the **Appendix(Figure A2)**. More visualizations of the affordance results will be added in the final version, particularly highlighting comparisons on unseen objects. \n\n\n>Q2:In line 193, shouldn't the term \"hand joints\" actually refer to hand surface vertices? If so, how did you recover the hand's surface? \n\nA2: Yes, you are right. The \"hand joints\" actually mean \"hand surface vertices\". We recover the hand's surface information to the MANO model using **$L_{hoi\\_train}$ (Eq. (6))**. \n\n\n>Q3: In the affordance refinement, why is the distance computed to the nearest point on the entire object surface, rather than to the nearest point within the predicted contact region? \n\nA3: Thank you for raising the concern. We are sorry for the typo error in our description and will revise the explanation accordingly in the final version of the paper.  Yes, aligning the hand keypoints more closely with the affordance region yields superior performance. In our method, we explicitly guide the hand joints toward the **affordance region**, which is why we refer to this process as **\"Affordance Refinement\"**."}, {"review_id": "aiMv7oBBga", "reviewer": "Reviewer_7WnN", "summary": "This paper introduces OpenHOI, the first framework for open-world hand-object interaction (HOI) synthesis. It enables the generation of long-horizon HOI sequences for novel objects conditioned on language instructions. The core technical idea is to leverage the embedded knowledge of a 3D Multimodal Large Language Model (MLLM) to enhance the generalizability of HOI generation. Specifically, the approach fine-tunes a pre-trained 3D MLLM for affordance grounding and semantic task decomposition. These outputs are then used in a diffusion-based HOI synthesis and a physics-based refinement stage to produce the final HOI sequences. IIn experiments, OpenHOI achieves new state-of-the-art performance on HOI generation for both seen and unseen objects, outperforming existing state-of-the-art methods.", "questions": "Please see the Weaknesses section.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\n\n1. Writing quality\n\nOverall, the paper is easy to read, and the motivation and technical contributions are clearly presented.\n\n2. Good motivation\n\nI agree that enhancing the generalizability of HOI generation to unseen object categories is an important research direction, as this remains a known limitation of existing HOI generation methods. In this context, leveraging the embedded knowledge of a pre-trained 3D MLLM is a reasonable and promising approach.\n\n3. Strong method novelty\n\nThe proposed method is novel and interesting. This is one of the first works to leverage a large language model for HOI generation, and the approach aligns well with the high-level goal of improving generalizability, as noted in the previous point.\n\n\n[Weaknesses]\n\n1. Questionable generalizability of the diffusion-based HOI generation module\n\nThe proposed framework consists of two sequential stages: (1) 3D MLLM-based instruction decomposition and affordance reasoning, and (2) diffusion-based HOI generation conditioned on the outputs of the 3D MLLM. While the first stage may offer high generalizability by leveraging the pre-trained 3D MLLM, it remains unclear how the second, diffusion-based module achieves strong generalizability. In the experiments, the performance gap relative to the existing state-of-the-art baseline (Text2HOI) does not noticeably differ between seen and unseen objects. This raises the question of whether the improved performance is primarily due to stronger representation capacity rather than actual improvements in generalization, which is the main claim of the paper.\n\n2. Missing evaluation for physical realism\n\nOne of the claimed technical contributions is the introduction of a physics-based refinement stage to enhance the physical realism of the generated HOI sequences. However, empirical validation for this component is missing, despite the fact that the closest baselines (e.g., Text2HOI, HOIGPT) report evaluation metrics specifically designed to assess physical realism (e.g., physical realism, interpenetration volume).\n\n3. Questionable performance on long-horizon generation\n\nAnother core contribution claimed by the paper is the ability to generate long-horizon HOI sequences. However, I have concerns about this claim due to the method design. Methodologically, the approach performs in-between motion refinement by enforcing smoothness only between adjacent frames (Eq. 11), which contrasts with existing long-horizon synthesis methods (e.g., [R1]) that consider much longer transition windows. Clarifying these points would be helpful.\n\n[R1] Barquero et al., FlowMDM: Seamless Human Motion Composition with Blended Positional Encodings, CVPR 2024.", "was_revised": true, "final_justification": "During the author–reviewer discussion, most of my concerns were addressed, so I have decided to maintain my positive rating.", "rebuttal": "We appreciate Reviewer 7WnN's detailed feedback and the opportunity to clarify our work. We provide comprehensive responses to your concerns in the following.\n\n>Q1: Questionable generalizability of the diffusion-based HOI generation module.The proposed framework consists of two sequential stages: (1) 3D MLLM-based instruction decomposition and affordance reasoning, and (2) diffusion-based HOI generation conditioned on the outputs of the 3D MLLM. While the first stage may offer high generalizability by leveraging the pre-trained 3D MLLM, it remains unclear how the second, diffusion-based module achieves strong generalizability. In the experiments, the performance gap relative to the existing state-of-the-art baseline (Text2HOI) does not noticeably differ between seen and unseen objects. This raises the question of whether the improved performance is primarily due to stronger representation capacity rather than actual improvements in generalization, which is the main claim of the paper.\n\nA1: Thanks for your question. Affordance is a **powerful and explicit prior** for interaction that can guide complex and fine-grained HOI synthesis. Our model achieves strong generalization by using affordance as a **middleware layer**. \n- **Open-World Affordance Grounding:** We first employ a coarse-to-fine tuning strategy to equip the model with strong **affordance reasoning capabilities**, enabling it to generate **open-world** affordance grounding. This enables the synthesis of realistic HOI sequences, demonstrating strong **template-free** generalization capabilities. \n- **Affordance serves as a crucial condition:** The open-world affordance grounding serves as a crucial condition for the affordance-driven HOI Diffusion.We incorporate affordance not only during training but also in the loss-guidance applied during inference. \n- **Affordance-based Refinement:** we design an improved refinement strategy based on **affordance**: for the interaction between the hand and the object, we optimize based on **affordance grounding** rather than the conventional closest-surface-point approach. \n- **A Template-free Example:** Our 3D MLLM has learned from a wide variety of cups, it can still generate accurate affordance grounding for a completely unseen mug. **Accurate affordance grounding** will guide the synthesis of realistic HOI sequences.\n\n\n\n\n\n>Q2: Missing evaluation for physical realism. One of the claimed technical contributions is the introduction of a physics-based refinement stage to enhance the physical realism of the generated HOI sequences. However, empirical validation for this component is missing, despite the fact that the closest baselines (e.g., Text2HOI, HOIGPT) report evaluation metrics specifically designed to assess physical realism (e.g., physical realism, interpenetration volume).\n\nA2: We supplemented our experiments by evaluating **Physical Realism** and **IV** metrics against the **closest baseline**, Text2HOI (HOIGPT’s code is not publicly available), and conducted ablation studies on our **Physical Refinement** module. We will include these two metrics in the final version of the paper.\n\n\nCompare with the closest baselines:\n| Method| Physical realism $\\uparrow$| IV $\\downarrow$|\n|:-------:|:-------:|:-------:|\n| Text2HOI(Seen) |0.87±0.03|11.74±1.22|\n| Ours(Seen) |**0.93±0.02**|**9.25±0.73**|\n| Text2HOI(Unseen) |0.79±0.05|14.63±1.07|\n| Ours(Unseen) |**0.89±0.01**|**10.35±0.82**|\n\nAblation Studies on Physical Realism and IV:\n| Method| Physical realism $\\uparrow$| IV $\\downarrow$|\n|:-------:|:-------:|:-------:|\n| w/o Physical Refinement(Seen)|0.89±0.07|10.75±0.80|\n| Ours(Seen) |**0.93±0.02**|**9.25±0.73**|\n| w/o Physical Refinement(Unseen) |0.84±0.03|12.27±0.48|\n| Ours(Unseen) |**0.89±0.01**|**10.35±0.82**|\n\n\n>Q3: Questionable performance on long-horizon generation. Another core contribution claimed by the paper is the ability to generate long-horizon HOI sequences. However, I have concerns about this claim due to the method design. Methodologically, the approach performs in-between motion refinement by enforcing smoothness only between adjacent frames (Eq. 11), which contrasts with existing long-horizon synthesis methods (e.g., FlowMDM) that consider much longer transition windows. Clarifying these points would be helpful.\n\nA3: Thank you for your suggestion. Motivated by FlowMDM's longer transition windows, we conducted comparative experiments using **transition windows of different sizes**, and the results are shown below. We will include this experiment in the final version of the paper and revise our Motion In-between method accordingly. We sincerely appreciate your feedback, which has helped improve our approach. \n\n\n|Size| SmoothRate(Seen) $\\downarrow$| SmoothRate(Unseen) $\\downarrow$| \n|:-------:|:-------:|:-------:|\n| 1| 4.77 ± 0.68    |5.26 ± 2.47    | |\n| 3 | 3.59 ± 0.81   |4.58±1.05 |\n| 5 |**2.98 ± 0.43**   | **3.70 ± 0.61**|\n| 10 |3.24 ± 0.65    |4.08±0.82 |\n| 20 |3.30 ± 0.57    |4.19±0.74|"}, {"review_id": "fVXvogKvDV", "reviewer": "Reviewer_wJWx", "summary": "The paper proposes OpenHOI, the first open-world hand-object interaction (HOI) synthesis framework capable of generating long-horizon manipulation sequences for novel objects guided by open-vocabulary instructions. The paper's contributions are significant and address a challenging problem in the field of 3D interaction research. The proposed approach combines a 3D multimodal large language model (MLLM) with an affordance-driven diffusion model and physics-aware refinement, achieving strong generalization capabilities across object categories, task horizons, and linguistic complexity. Experimental results demonstrate the superiority of the method over existing state-of-the-art approaches.", "questions": "1. The paper adopts ShapeLLM as the backbone of the 3D MLLM. What are the advantages and disadvantages of this choice compared to the other 3D MLLM models? How does ShapeLLM's performance in the context of OpenHOI align with its original design goals and capabilities?\n2. What are the potential applications of the proposed OpenHOI framework in real-world scenarios, such as robotics and virtual reality? What challenges might arise in practical applications, and how can they be addressed?", "limitations": "Please find in weakness and questions.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. The paper presents the first open-world hand-object interaction (HOI) synthesis framework, OpenHOI, which can generate long-horizon manipulation sequences for novel objects guided by open-vocabulary instructions. This represents a significant advancement in the field of 3D interaction research, addressing a previously unresolved challenge.\n2. The proposed method demonstrates strong generalization capabilities across object categories, task horizons, and linguistic complexity. It outperforms existing state-of-the-art methods in handling unseen objects and complex, linguistically diverse commands.\n3. The paper is well-organized and clearly written. The authors provide a detailed introduction to the research background and motivation, comprehensively review related work, meticulously describe the proposed method, and thoroughly evaluate the experimental results.\nWeakness\n1. Hierarchical Task Decomposition Limitations: The paper notes that while OpenHOI supports multi-stage tasks, it struggles with compositional long-horizon sequences like \"cook a meal\" due to limitations in hierarchical task decomposition. This suggests that the method may not be suitable for highly complex and hierarchical tasks.\n2. Limitations in Fine-Grained Dynamics: Although the physics-aware refinement improves interaction plausibility, the paper acknowledges challenges in handling fine-grained dynamics, such as fluid simulation for pouring tasks. This indicates that the proposed method may not fully capture the complexities of certain interactions.", "was_revised": true, "final_justification": "After reading the rebuttal, I keep my acceptance score for this paper.", "rebuttal": "We sincerely thank Reviewer wJWx for his/her comments and questions. We offer point-by-point clarifications to the issues you raised.\n\n> Q1-1: The paper adopts ShapeLLM as the backbone of the 3D MLLM. What are the advantages and disadvantages of this choice compared to the other 3D MLLM models? \n\nA1-1：Thank you for your question. We enumerate the advantages and disadvantages below.\n\n**Advantages:** ShapeLLM is trained on a large amount of 3D embodied interaction data and achieves state-of-the-art performance across various downstream tasks. It possesses strong priors in 3D interaction and demonstrates impressive zero-shot 3D representation capabilities.\n\n**Disadvantages:** ShapeLLM has not been trained on part‑level object annotations, which limits its reasoning capabilities for fine‑grained object understanding.\n\n\n\n>Q1-2: How does ShapeLLM's performance in the context of OpenHOI align with its original design goals and capabilities?\n\nA1-2：First, ShapeLLM is a powerful 3D foundation model that performs exceptionally well on multiple downstream tasks (e.g., Embodied Visual Grounding, Visual Question Answering, and Scene Understanding), making it highly suitable for HOI tasks.\n\nSecond, we adopt a coarse-to-fine tuning strategy: we first pre-train the model on an object-centric affordance dataset to enable it to acquire strong affordance priors. Then, we fine-tune the model on HOI datasets to better align the semantics with the target domain. This allows the model to learn highly effective affordance representations.\n\n\n\n>Q2: What are the potential applications of the proposed OpenHOI framework in real-world scenarios, such as robotics and virtual reality? What challenges might arise in practical applications, and how can they be addressed?\n\n \nA2: Thank you for your suggestion. Applying OpenHOI to real‑world scenarios is both a challenging and compelling endeavor.\n\n**Robotics Application:** OpenHOI can be integrated into real‑world robotic manipulation systems, including industrial robot arms and service robots, to enable more flexible and human‑like interactions. \n- **Open-World Affordance Grounding as Powerful Guidance for Robots**:Leveraging OpenHOI's 3D MLLM, our method performs open-world affordance grounding to facilitate the identification of feasible grasping, pushing, and tool-use regions on **novel objects**, thereby significantly improving success rates for pick-and-place, assembly, and tool-handling tasks. \n- **Realistic HOI sequences synthesis for Robot Manipulation:** The HOI sequences generated by OpenHOI can be adapted into robotic manipulation sequences. First, we can use an extraction algorithm to obtain the object's **6-DoF pose**. Then, **inverse kinematics** are applied to the wrist parameters to compute the robot arm's pose. Finally, a **retargeting** algorithm transfers the human hand motions onto various robotic hand configurations for manipulation. This approach ensures smooth, precise and robust manipulation behaviors in real‑world deployments. \n\n**Virtual Reality Application:** By synthesizing realistic 3D hand–object interaction sequences, OpenHOI enables users to **manipulate virtual objects** naturally, for example by picking up, twisting, or pouring items, thereby enhancing immersion in training simulators, gaming, and virtual prototyping.\n\n\n**Challenges:** Robotic manipulation tasks typically demand rapid inference. We plan to employ DPM‑Solver to accelerate the diffusion inference process important direction for our future work."}, {"review_id": "MEzoDDlbzC", "reviewer": "Reviewer_juyL", "summary": "The paper proposes a method for 3D hand-object interactions leveraging an MLLM. The use of MLLM allows for long-horizon manipulation. The framework leverages a pre-trained MLLM and diffusion model, and minimal training using LoRA adapters. In addition, several refinements during and post-training are applied to account for the geometrical shortcomings of LLMs.", "questions": "The paper is well written. Most questions are regarding error analysis, as discussed more extensively above, but the key things to address are:\n\n- Clarity on the \\<aff\\> tokens is necessary\n- Some results appear statistically insignificant; it would be best to show the importance of each component\n- Failure cases, error analysis and limitations are missing", "limitations": "The authors have not discussed limitations and societal impact, both of which should be included in the paper.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I have little experience in the HOI task (which is reflected in the confidence of this review). As such, the comments are primarily regarding the MLLM and the completeness of the experiment section.\n- is the method using a single \\<Aff\\> token? it appears as if multiple are used in Fig 2, but only one is mentioned in the methodology section. If multiple are used, these need to be made explicit in the method section, and relevant experiments should be included in the ablation studies.\n- Furthermore, all results presented would benefit from statistical significance testing, as mean and standard deviation are provided. This is important, as some results in the ablation (eg, MPJPE and Diversity) appear comparable, and the importance of each component may be undermined (eg, affordance loss).\n- The method should include failure cases and an error analysis. \n- How long is the horizon before performance deteriorates? What instructions/samples does the method fail to address?\n- Are there any limitations to the method?", "was_revised": true, "final_justification": "The authors have addressed most of my and other reviewers' concerns. As such, I will maintain my original positive rating.\nThe only thing that remains a bit of a concern is the claim for long-horizon manipulation; in the rebuttal, the authors admit performance degrading after three actions, which does not seem very long.", "rebuttal": "We sincerely value Reviewer juyL's comments, which are helpful. A point-by-point reply to your comments is provided below.\n\n\n>Q1：Is the method using a single \\<Aff\\> token? It appears as if multiple are used in Fig 2, but only one is mentioned in the methodology section. If multiple are used, these need to be made explicit in the method section, and relevant experiments should be included in the ablation studies.\n    \nA1: Yes, our method uses multi \\<AFF\\> tokens. Lines 139-141 and Formula (6) show that we use **S** occurrences of the segmentation token \\<AFF\\>. The additional ablation study results are as follows:\n\n\n| Method|MPJPE $\\downarrow$|FOL $\\downarrow$|FID $\\downarrow$|Diversity $\\rightarrow$|MModality $\\uparrow$| \n|:------------:|:-----:|:----:|:----:|:---------:|:---------:|\n|Single \\<AFF\\>(Seen)|52.05±0.82|0.33±0.01|29.31±0.76|3.50±0.12|21.05±1.47|\n|Multi \\<AFF\\>(Seen)| **47.64±1.03**|**0.26±0.02**|**26.43±0.77**|**3.69±0.27**|**24.59±2.01**| \n|Single \\<AFF\\>(Unseen)|56.48±1.06|0.39±0.02|34.15±1.08|3.40±0.22|17.03±1.25|\n|Multi \\<AFF\\>(Unseen)|**51.34±0.85**|**0.27±0.01**|**28.29±0.62**|**3.61±0.09**|**19.91±0.63**|\n    \n\n\n>Q2: Furthermore, all results presented would benefit from statistical significance testing, as mean and standard deviation are provided. This is important, as some results in the ablation (eg, MPJPE and Diversity) appear comparable, and the importance of each component may be undermined (eg, affordance loss).\n    \nA2: Thank you for your suggestion. For the ablation study, we performed paired two‑sample t‑tests, repeating each test five times and reporting the mean p‑value. As summarized in Table , the proposed method is significant at the 95% confidence level for the majority of metrics (P-value $\\leqslant$ 0.05).\n\n| Metric               | w/o Affordance | w/o CFG     | w/o $l_{penetration}$ | w/o $l_{aff}$    |\n|:---------------------:|:---------------:|:------------:|:------------------:|:-------------:|\n| MPJPE (Seen)         | 3.1×10⁻⁶       | 2.0×10⁻⁴    | 7.5×10⁻⁵          | 2.6×10⁻²     |\n| MPJPE (Unseen)       | 4.4×10⁻⁵       | 1.2×10⁻⁴    | 5.5×10⁻⁴          | 2.6×10⁻⁴     |\n| FOL (Seen)           | 1.7×10⁻³       | 6.8×10⁻⁵    | 1.7×10⁻⁵          | 3.1×10⁻⁵     |\n| FOL (Unseen)         | 2.2×10⁻²       | 2.0×10⁻³    | 4.9×10⁻²          | 3.0×10⁻¹     |\n| FID (Seen)           | 2.4×10⁻⁵       | 1.5×10⁻³    | 1.0×10⁻²          | 2.0×10⁻³     |\n| FID (Unseen)         | 2.2×10⁻⁵       | 8.7×10⁻⁵    | 2.0×10⁻⁴          | 2.5×10⁻⁴     |\n| Diversity (Seen)     | 1.5×10⁻¹       | 1.0×10⁻¹    | 4.4×10⁻²          | 1.4×10⁻¹     |\n| Diversity (Unseen)   | 1.8×10⁻³       | 3.0×10⁻⁴    | 4.6×10⁻²          | 4.8×10⁻²     |\n| MModality (Seen)     | 2.6×10⁻⁴       | 2.5×10⁻²    | 5.0×10⁻³          | 1.1×10⁻²     |\n| MModality (Unseen)   | 1.0×10⁻³       | 4.7×10⁻²    | 2.2×10⁻²          | 2.2×10⁻²     |\n    \n\n                                               \n>Q3: Failure cases, error analysis and limitations are missing\n    \nA3: Thanks for your question. We enumerate the failure cases, error analysis and limitations below.\n\n**Failure Cases and Error Analysis:** Given a scenario, there is a row of cabinets, and the instruction \"Open the cabinet\", our model can open one but cannot target a specific one (e.g., \"Open the second cabinet\"). This limitation arises because the model has not been trained on a large-scale 3D QA dataset, resulting in reduced logical reasoning capabilities.\n\n**Limitations:** As stated in the Conclusion section, we have outlined our limitations and future work (lines 292-297). We acknowledge that our model currently struggles with (1) highly complex, scene‑level, long‑horizon interactions (e.g., \"cooking a meal\") and (2) extremely fine‑grained dynamics on articulated objects (e.g., \"fluid simulation in pouring tasks\"). Addressing these challenges will be the focus of our future work. \n    \n\n    \n>Q4: How long is the horizon before performance deteriorates? \n\nA4: Due to the accumulation of model errors, performance typically degrades after more than three consecutive actions. \n\n\n    \n>Q5: Diversity arrows on tables seem incorrect.\n    \nA5: Our diversity arrows on tables are correct. In fact, in the HOI domain, sequence diversity is deemed better the more closely it matches the ground‑truth distribution of the original dataset; this is exactly how Text2HOI and HOIGPT report their results."}]

Author rebuttal:

Meta-review: This paper introduces a novel framework for open-world 3D hand-object interaction (HOI) synthesis. Its core contribution is the integration of a fine-tuned 3D MLLM for affordance grounding and semantic task decomposition with an affordance-driven diffusion model for motion generation. This enables the system to generate long-horizon, physically plausible manipulation sequences for novel, unseen objects, addressing a significant limitation of prior closed-set methods. 

This paper moves the field beyond closed-set constraints by creatively integrating the latest advances in 3D MLLMs with generative models. The proposed framework is novel, technically sophisticated, and delivers compelling empirical results that validate its core claims. This paper has the potential to significantly influence research in AR/VR, robotics, and embodied AI.
Decision: Accept (oral)