Title: OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model
Venue: NeurIPS 2025
Accepted: yes
Keywords: Hand-Object Interaction;Open-World;Large Language Models
Abstract: Understanding and synthesizing realistic 3D hand-object interactions (HOI) is critical for applications ranging from immersive AR/VR to dexterous robotics. Existing methods struggle with generalization, performing well on closed-set objects and predefined tasks but failing to handle unseen objects or open-vocabulary instructions. We introduce OpenHOI, the first framework for open-world HOI synthesis, capable of generating long-horizon manipulation sequences for novel objects guided by free-form language commands. Our approach integrates a 3D Multimodal Large Language Model (MLLM) fine-tuned for joint affordance grounding and semantic task decomposition, enabling precise localization of interaction regions (e.g., handles, buttons) and breakdown of complex instructions (e.g., “Find a water bottle and take a sip”) into executable sub-tasks. To synthesize physically plausible interactions, we propose an affordance-driven diffusion model paired with a training-free physics refinement stage that minimizes penetration and optimizes affordance alignment.
Evaluations across diverse scenarios demonstrate OpenHOI’s superiority over state-of-the-art methods in generalizing to novel object categories, multi-stage tasks, and complex language instructions.

OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model

Abstract

Understanding and synthesizing realistic 3D hand-object interactions (HOI) is critical for applications ranging from immersive AR/VR to dexterous robotics. Existing methods struggle with generalization, performing well on closed-set objects and predefined tasks but failing to handle unseen objects or open-vocabulary instructions. We introduce OpenHOI, the first framework for open-world HOI synthesis, capable of generating long-horizon manipulation sequences for novel objects guided by free-form language commands. Our approach integrates a 3D Multimodal Large Language Model (MLLM) fine-tuned for joint affordance grounding and semantic task decomposition, enabling precise localization of interaction regions (e.g., handles, buttons) and breakdown of complex instructions (e.g., 'Find a water bottle and take a sip') into executable sub-tasks. To synthesize physically plausible interactions, we propose an affordance-driven diffusion model paired with a training-free physics refinement stage that minimizes penetration and optimizes affordance alignment. Evaluations across diverse scenarios demonstrate OpenHOI's superiority over state-of-the-art methods in generalizing to novel object categories, multi-stage tasks, and complex language instructions. Our project page at https://openhoi.github.io

1 Introduction

Hand-object interaction (HOI) involves jointly modeling hand articulation and object dynamics to generate and interpret realistic manipulation sequences [9, 17, 25, 26, 15, 47, 5]. This reflects one of the most pervasive human behaviors, deeply embedded in daily activities. Generating and understanding 3D HOI sequences is critical for advancing machine capabilities in human-centric applications. In augmented/virtual reality (AR/VR), realistic HOI modeling enables immersive digital experiences, allowing users to manipulate virtual objects naturally. For robotics, it provides the foundation for dexterous, feedback-driven manipulation in unstructured environments.

Generating HOI sequences from natural language instructions remains a significant challenge in 3D interaction research. While traditional methods [19, 40] rely on handcrafted motion priors, recent diffusion-based approaches [2, 3] directly map text to action sequences, eliminating the need for manual motion design. However, due to limited data and modeling capacity, these models only deal with closed datasets and struggle to generalize to unseen objects and open-vocabulary instructions.

Recent advances in Large Language Models (LLMs) have significantly enhanced joint vision-language understanding. Building on this progress, concurrent work like HOIGPT [13] proposes an LLM-based architecture for aligning textual instructions with HOI sequences. However, these approaches face

GLYPH(cmap:d83d)

:'I'm feeling thirsty, could you find a water bottle and take a sip?'

GLYPH(cmap:d83e)

:'I'm a bit dehydrated,  please open the water bottle cap with both hands,

then drink the bottle water using your right hand.'

'I want to relax and read,

go ahead and hold the book

with both hands to read it.'

'We need the cocoa,

can you take it out with your left hand, please?'

'I'm feeling a bit thirsty,

could you pour some milk

using your right hand?'

(a) Interact with Unseen Objects

Closed-Set HOI

(b) Open-Vocabulary High Level Instruction

Open-World HOI

Seen

Unseen

'Use headphones with both hands.'

'Use spray with right hand.'

Timestamp

fundamental generalization challenges in unseen shapes and complex 3D interaction scenarios due to the lack of 3D knowledge in LLM. Fig. 1 illustrates the comparison between conventional Closed-Set HOI and Open-World HOI.

The growth of 3D datasets has advanced 3D multimodal large language models (MLLMs) with strong capabilities in understanding and grounding 3D geometric, semantic, and functional relationships. Critically, fine-grained part-level grounding and affordance grounding [34] [4] can identify actionable object regions and their interaction possibilities, providing strong priors for synthesizing physically consistent hand-object interactions (HOI). Building on this insight, we propose OpenHOI, the first open-world HOI synthesis framework capable of generating long-horizon manipulation sequences for unseen objects guided by open-vocabulary instructions. Our approach fine-tunes a 3D MLLM endowed with comprehensive affordance reasoning priors, enabling two core capabilities: 1) Generalizable Affordance Prediction: Precise localization of interaction regions (e.g., handles, buttons) for both known and novel objects. 2) Instruction Decomposition: Breaking down open-vocabulary commands (e.g., 'I'm feeling thirsty, could you find a water bottle and take a sip') into executable sub-tasks grounded in object affordances. By integrating these priors with a diffusion-based interaction generation and physics-aware refinement, OpenHOI achieves unprecedented generalization across object categories, task horizons, and linguistic complexity.

In summary, our contributions are as follows:

We introduce OpenHOI, the first open-world hand-object interaction synthesis framework capable of generating long-horizon manipulation sequences for unseen objects guided by open-vocabulary instructions. Unlike prior closed-set HOI methods, OpenHOI generalizes to novel objects and complex, linguistically diverse commands.

OpenHOI involves fine-tuning a 3D MLLM that jointly learns geometric affordance priors and semantic task decomposition. Subsequently, affordance-driven HOI Diffusion with Physical Refinement is developed to generate realistic HOI sequences for each task.

Our experiments demonstrate state-of-the-art performance, outperforming existing methods with a large margin. Notably, OpenHOI generalizes robustly to unseen objects and open-vocabulary instructions, achieving strong compositional generalization across diverse scenarios.

2 Related Work

Hand-Object Interaction Synthesis. Synthesizing realistic and diverse hand-object interactions (HOI) have gained significant attention, with numerous approaches exploring this problem under various settings. These include the creation of large-scale datasets like GigaHands [7] and OAKINK2 [53], methods for affordance learning [21, 56, 32, 45]. Others have focused on synthesizing twohand interactions like InterHandGen [19] and HandDiffuse [23], or on physics-aware synthesis and dexterous grasping [44, 46, 57]. Several key works have specifically tackled text-guided or semantically rich HOI synthesis. Text2HOI [2] introduced a pioneering approach to generate 3D Hand-Object Interaction Sequences from text by decomposing the task into contact prediction and motion generation using a diffusion model, but it can struggle with fine-grained control from low-level text and tends to produce short HOI interaction sequences. HOIGPT [13] adopts an LLM-based sequential model to predict hand-object trajectories from text for long-horizon sequences, yet it lacks explicit affordance-guided mechanisms and cannot ensure smooth motion in-between multiple sub-sequences in complex interactions. SemGrasp [20] focuses on semantic-aware grasp generation, but it discretizes motion space and is limited to static grasps. Grasp as You Say [46] addresses category-level language-guided grasping, but it requires hand-crafted hand-object contact templates and lacks dynamics. While these methods have advanced the field, generating fine-grained, diverse, and long-horizon hand-object interaction sequences with high-level text on unseen objects cohesively remains a significant challenge.

Multimodal Large Language Model. The advent of Large Language Models (LLMs) has revolutionized natural language understanding, and their influence is gradually expanding into 3D perception and interaction tasks. Some recent works leverage pure LLM architectures to handle motion generation from language. For instance, HOIGPT [13] and MotionGPT [16] adopt VQ-VAE tokenization and GPT-style architectures to achieve bidirectional generation between natural language and HOI systhesis. However, these models operate primarily on discrete motion tokens, and lack grounding in actual 3D perception or reasoning, they cannot perceive objects or environments in 3D nor reason about spatial affordances or constraints, thus limiting their ability to generate contextaware interactions. Conversely, Multimodal Large Language Models (MLLMs) have emerged as powerful tools that extend the success of LLMs into various domains, aiming to bridge the gap between language and other modalities such as images, videos, and 3D representations, enabling tasks like visual question answering like ShapeLLM [34], 3D object generation like Point-E [29] and CLIP-Forge [37], and affordance understanding like LASO [22], DAG[42]and AffordanceLLM [35]. These MLLMs demonstrate strong capabilities in static 3D understanding, especially in segmenting or identifying affordances from language prompts. Other MLLM-based approaches like SeqAfford [52] and 3D-AffordanceLLM [4] further extend this to sequential or compositional affordance reasoning. However, these approaches remain limited to perception tasks and do not address the synthesis of continuous hand-object interactions or motion dynamics. In contrast, while LLM-based methods focus on generating motion from text without grounding in 3D environments, and MLLM-based methods enable 3D spatial understanding without supporting interaction generation, none of the existing approaches bridge both capabilities. Our work addresses this gap by leveraging an MLLM that is trained to jointly model language, 3D perception, and interaction dynamics. This enables our model to both understand object affordances from language and synthesize long-horizon, task-consistent hand-object interactions with unseen objects from open-vocabulary instructions. By unifying reasoning and generation, our approach extends 3D MLLMs toward fine-grained, dynamic, and open-world HOI synthesis.

Clip

Encoder

'I'm feeling thirsty, could you find a

water bottle and take a sip?'

Pretrained 3D

Vision Encoder

(1)Affordance and Sub-tasks Extraction

Object Point Cloud

LoRA

point feature

text token

3D Multimodal-LLM

Affordance

Decoder

Affordance

(2)Affordance-driven HOI Diffusion

Transformer

Encoder

Diffusion

'I'm a bit dehydrated,  please open the water bottle

cap <AFF 1> with both hands, then drink the bottle

water  <AFF 2> using your right hand.'

c

<AFF 2>

<AFF 1>

Sub-tasks

Model

Affordance

'First open the

bottle cap with

both hands.'

'Then drink the

bottle water using

your right hand.'

Task-1

Task-2

c

<AFF 1> <AFF 2>

Linear

Conditional Input

Classifier-free

Guidance

•

•

•

Linear

Denoise

Noisy

Sequence

Original

Sequence

Diffusion

Model

Task-i

Afford-

ance

Penetration

Alignment

Afford

Alignment

Task-1

Seq

Task-2

Seq

Motion In-between

Alignment

Generated HOI

Sequence

Complete

HOI

Sequence

•

•

•

Conditional

(b)Training-free Refinement

(a)Training

End

Start

μ

μ

3 Method

We propose OpenHOI, a novel framework for generating long-horizon hand-object interaction sequences guided by high-level open-vocabulary instructions. OpenHOI generalizes to unseen object categories by leveraging semantic reasoning from a 3D MLLM paired with 3D affordance prior. Our framework consists of two core components:

3.1 Instruction Decomposition and Affordance Reasoning via 3D MLLM

Given a 3D object point cloud and a high-level,free-form instruction (e.g., "I am thirsty"), the 3D MLLM semantically grounds the instruction into object-centric affordances by bridging the gap between abstract intent and actionable object functionality. This process yields two structured outputs: (1) A spatial affordance map that identifies geometrically plausible interaction regions (e.g., highlighting the cap of a water bottle for opening and the body for grasping). (2) A temporally decomposed sub-task sequence that translates the instruction into executable atomic actions (e.g., 1) grasp bottle cap with both hands → 2) twist counterclockwise to open → 3) lift bottle to mouth with right hand").

Network structure of 3D MLLM. Recent advances in 3D multi-modal language modeling have significantly enhanced open-world understanding of 3D objects. In particular, ShapeLLM [34] has been pretrained to capture a wide range of embodied interactions. Consequently, we adopt ShapeLLM as our backbone. Its point-cloud encoder ReCon++ is pretrained via multi-view distillation from ReCon [33], and its language component is initialized from LLaMa [41]. Prior approaches to 3D affordance prediction have typically relied on standalone 3D backbones [51] or on separate point-tolanguage encoders [22], which often lack robust reasoning and open-world generalization capabilities. By leveraging a unified 3D MLLM rather than exclusively using pure LLMs or conventional visual architectures, we achieve both enhanced generalization to unseen objects and affordances and the intrinsic integration of affordance perception into natural language representations, thereby facilitating subsequent affordance reasoning.

In-context 3D Affordance Reasoning. Although 3D MLLMs effectively align 3D representations with natural language, they are predominantly tailored for object-centric text generation and thus lack inherent support for dense 3D prediction tasks such as fine-grained affordance segmentation. To address this limitation, we augment the MLLM vocabulary with a dedicated segmentation token <AFF>, following the design of Lisa [18], thereby enabling the model to represent and reason about segmentation outputs within its language-based framework. Formally, given a point-cloud input F obj and instruction text T ins expressing user intent over candidate objects, the 3D MLLM jointly encodes both modalities to produce a sub-task sequence:

which contains S occurrences of the segmentation token <AFF> , each marking a predicted affordance region. We then collect the corresponding last-layer embeddings { h ( i ) aff } S -1 i =0 and pass each through an affordance decoder:

Coarse-to-Fine Affordance Tuning. Our training pipeline comprises two sequential stages. First, we fine-tune the model on large-scale, coarse-grained, object-centric static affordance datasets [35, 52], to instill strong affordance priors, thereby enhancing its generalization and open-vocabulary capabilities. Second, we fine-tune the stage 1 model on a smaller dynamic hand-object interaction dataset, which can get fine-grained affordance maps by using the voxel-based method, to generate more precise affordance Maps and task decomposition sequences. We leverage an auto-regressive cross-entropy loss L task to supervise sub-tasks generation, complemented by Dice loss and Binary Cross-Entropy loss L aff to guide affordance prediction.

where the weights λ task , λ aff are utilized to balance the different loss items.

3.2 Affordance-driven HOI Diffusion with Physical Refinement

To generate long-horizon, natural, and physically plausible hand-object interaction sequences X based on Interaction Prior C from Sec 3.1, which include affordance prior, sub-tasks, and object pointcloud as follows,

We propose Affordance-driven HOI Diffusion , which decomposes this challenging task into tractable components. During training, we improve the alignment of the affordance map via classifierfree guidance [11, 30, 12]. During inference, we propose a training-free refinement method that generates natural, physically plausible, and long-horizon HOI sequences.

Training HOI Diffusion with Affordance Prior from MLLM. During training, given a conditioning pair [ X, C ] , we first add noise to X using the forward diffusion process:

where X 0 denotes the original HOI sequence, X t is its noisy version at diffusion timestep t , and α t controls the noise schedule. We then train a transformer-based neural network ˆ X θ to directly predict the HOI sequence X 0 from the noisy input. The training objective for HOI generation is defined as:

where L hoi_diff ( ˆ X θ , C ) = E X 0 ∼ p ( X 0 ) ,X t ∼ p ( X t | X 0 ) ,t ∼ [1 ,T ] || X 0 -X θ ( X t , t, C ) || 2 2 is the denoising loss of the diffusion process for HOI reconstruction. L hoi_distance ( ˆ X θ ) donates the HOI distance map loss, which encourages precise surface contact and enhances the physical plausibility of the contact area by penalizing errors more strongly when the hand is near the object. L hoi_orient ( ˆ X θ ) represents the HOI relative orientation loss, which aligns predicted orientations with ground truth to ensure accurate rotational poses for tasks like grasping and manipulation by modeling hand-object relative rotation, respectively. To better align the generated HOI sequences with the Interaction Prior C , we adopt the classifier-free guidance strategy:

where s is the guidance scale. To enable this strategy, we randomly mask 10% of the conditional inputs C during training to train the unconditional [8] model X θ ( X t , t, ∅ ) alongside the conditional one. During sampling, we gradually denoise X t to X 0 via posterior distribution p ( X t -1 | X θ , X t ) :

Training-free HOI Diffusion with Physical Refinement. While the vanilla diffusion sampling process in Equation 8 models the joint conditional distribution of HOI sequences, it suffers from two key limitations: 1) Lack of Physical Constraints. The sampling process does not inherently enforce physical plausibility [28], such as preventing interpenetration or ensuring stable contact. While prior work addresses this by training discriminative physical refiners, such methods often deviate from the underlying joint HOI distribution, introducing artifacts due to distribution shift. 2) Temporal Incoherence. The generated sequences exhibit discontinuous hand motions, particularly during transitions between sub-sequences, leading to unrealistic motion dynamics.

To address these limitations, we propose three key refinement objectives during the diffusion sampling process: 1) affordance refinement for precise contact, 2) physical constraint refinement to prevent penetration, and 3) temporal coherence refinement for smooth transitions, all achieved without additional training or distribution-shifting.

a) Affordance Refinement: Since direct affordance-based HOI generation often fails to produce precise hand-object contact, we propose an affordance-aware loss to guarantee geometric consistency:

where d ( · , · ) is the Euclidean distance between hand joints ( ˆ J lhand , ˆ J rhand ) and closest affordance region ( ˜ P ljoint obj , ˜ P rjoint obj ) .

b) Penetration Refinement: To mitigate interpenetration artifacts, we propose a penetration loss:

where d ( · , · ) is the Euclidean distance between hand joints ( ˆ V lhand , ˆ V rhand ) and closest affordance region ( ˜ P lvert obj , ˜ P rvert obj ) .

c) Motion In-between Refinement:

For seamless transitions between HOI sub-sequences, we synthesize natural hand motion ˆ V 0: T trans bridging the end pose V T pre of the preceding sequence and the start pose V 0 after of the subsequent sequence. The transition loss can be defined as:

where ˆ V 0 trans , ˆ V T trans denote the predicted start and end frames of the transition motion.

While gradient descent after each denoising step could minimize these losses, it risks introducing artifacts and distribution shifts [49]. Recently, a training-free conditional diffusion model named DSG [50] offers larger, adaptive step sizes to the loss function to achieve better alignment with the constraints while preserving the original distribution learned by the diffusion model. Inspired by DSG, we also introduce the Spherical Gaussian Constraint during the sampling stage to preserve the original distribution, thus mitigating the distribution-shifting problem. We utilize the analytical solution to enforce steepest gradient descent to enhance alignment:

where d represents the data dimensions and l denotes the loss in Eq 9, 10, 11 for refinement stage. To enhance the sample quality, we utilize a mixture of deterministic steepest gradient descent direction and random sampling direction:

where D sample = σ t ϵ t is the random sampling direction, and w represents the guidance rate.

4 Experiments

Our framework integrates two core stages: 1) fine-tuning a 3D MLLM to predict object affordance maps and decomposing open-vocabulary instructions into concrete sub-tasks, and 2) synthesizing HOI sequences with the condition output of stage 1. We evaluate our method using diverse datasets and metrics, demonstrating its capability to generate long-horizon HOI sequences with high-level instructions of both seen and unseen objects. Comparative experiments and ablations validate our design choices. Our experiments were conducted on NVIDIA A100 GPU.

4.1 Dataset

For our experiments, we utilize two prominent hand-object interaction datasets: GRAB [38], which provides comprehensive full-body motion data of subjects interacting with 51 everyday objects, and ARCTIC [6], a large-scale dataset specializing in bi-manual interactions with articulated objects and dense 3D annotations. We preprocess both two datasets with a unified pipeline, which involves initial geometric operations on the object point clouds-specifically, upsampling to support detailed affordance map generation via our inference model, followed by downsampling to ensure computational efficiency during training. Crucially, we enrich the original annotations by employing a multimodal large language model (MLLM) to convert low-level HOI motion descriptions into open-vocabulary, intent-centric language instructions. These semantic labels enhance the expressiveness of the data and allow our model to better generalize to unseen scenarios, supporting open-world HOI synthesis with long-horizon sequences and diverse objects. For both GRAB [38] and ARCTIC [6], we follow a standard protocol by partitioning each dataset into 80% for training and 20% for unseen testing, ensuring reliable evaluation of our model's generalization capabilities. This setup allows us to rigorously assess performance on unseen objects, motions, and interaction intents, validating the robustness of our MLLM-guided generation framework across both single-hand and bi-manual interaction scenarios.

4.2 Evaluation metric

To comprehensively evaluate the quality, diversity, realism, and physical plausibility of our generated hand-object interaction (HOI) sequences, we employ a multi-faceted set of quantitative and qualitative metrics inspired by prior works [2, 13, 24, 27, 40, 14]. We roughly divide the evaluation indicators into three categories

Motion Accuracy. We evaluate geometric precision using the Mean Per-Joint Position Error ( MPJPE ;) computed over hand joints, and assess object placement with the Final Object Location Error ( FOL ;), defined as the Euclidean distance between the predicted object center and the target location at the final frame.

Generation Realism. We measure realism via the Fréchet Inception Distance ( FID ;) between real and synthesized motions in a pre-trained motion feature space, capturing distributional alignment and perceptual fidelity.

Diversity & Multi-modality. Diversity quantifies across-prompt variability of generated outputs, while MModality captures within-prompt variability across multiple samples. Both are computed from pairwise distances or variance statistics in the motion feature space.

Lower MPJPE, FOL, and FID indicate higher accuracy and fidelity, while higher multi-modality and closer to GT diversity reflect stronger generative expressiveness.

4.3 Main Results

Comparison with SOTA Methods. We evaluate our method under the widely adopted seen / unseen split protocol and compare it against state-of-the-art methods (i.e., MDM [39], TM2T [10], MotionGPT [16], Text2HOI [2] on both GRAB [38] and ARCTIC [6] datasets. As shown in Table 2 and Table 3, our method consistently outperforms all baselines across both seen and unseen objects.

These results firmly establish our method as state-of-the-art, demonstrating the strong generalization ability of our MLLM-guided Affordance Reasoning and Affordance-driven HOI diffusion with Physical Refinement generation, which validates the effectiveness of our Open-World HOI synthesis

framework in generating long-horizon HOI sequences of unseen objects from Open-vocabulary instructions.

Qualitative results. We present the generated HOI sequence results. Open-world capability enables the generation of long-horizon HOI sequences under both unseen-object and open-vocabulary conditions. In this section, long-horizon HOI results are shown in Fig. 3. Experimental findings demonstrate physical realism and coherence of the generated sequences.

4.4 Ablation Study

We conduct systematic ablations on the GRAB [38] and ARCTIC [6] dataset to validate the necessity of our core components through controlled experiments. Each component is carefully analyzed to demonstrate its contribution to robust hand-object interaction generation, the details can be found in Table 4 and Table 5.

Affordance Awareness (w/o Affordance). We remove the part of obtaining accurate affordance maps by a well-trained MLLM, which significantly degrades interaction quality on each evaluation metric. Without this component, the model loses its ability to focus on functionally critical object regions, leading to unnatural hand placements and increased penetrations. This confirms that explicit affordance grounding is essential for semantically meaningful interactions.

Classifier-Free Guidance Diffusion (w/o CFG). Classifier-Free Guidance(CFG) plays a critical role in ensuring that the generated hand-object interaction (HOI) sequences remain aligned with the input conditions C . Without CFG, the model often fails to adhere to the conditioning signal, leading to semantic misalignment and affordance mismatch.

Loss-guided Physical Refinement (w/o l penetration,w/o l aff ). Experimental result demonstrates the critical importance of our loss-guidance strategy for temporal coherence and physical plausibility.

GLYPH(cmap:d83d)

: I'm feeling hungry, could you pass me an apple so I can eat?

GLYPH(cmap:d83e)

: Sure, please hand over the apple with your right hand, then eat it using your right hand.

GLYPH(cmap:d83d)

: I'd like some milk, could you help me with that?

GLYPH(cmap:d83e)

: Of course, please open the milk container with both hands, then drink the milk using your right hand.

GLYPH(cmap:d83d) :  I feel like playing with some toys, can you bring me something fun?

GLYPH(cmap:d83e)

: Alright, please hand over the duck with your left hand, then fly the airplane using your left hand.

Timestamp

Timestamp

Timestamp

Seen object

Unseenobject

Multi-objects

Without this guidance, the diffusion model exhibits two key failure modes: first, it fails to generate complete long-horizon HOI sequences, particularly in complex bimanual interactions where discrete actions remain disjointed rather than forming fluid motions; second, it produces physically unrealistic results with frequent object penetrations ( l penetration) and unnatural affordance patterns ( l aff), most noticeable during precision manipulation tasks. Through direct optimization of these constraints via our loss-guidance framework, we achieve significant improvements in generating physically valid and temporally coherent hand-object interactions.

4.5 Failure cases and error analysis

Given a scenario, there is a row of cabinets, and the instruction "Open the cabinet", our model can open one but cannot target a specific one (e.g., "Open the second cabinet"). This limitation arises because the model has not been trained on a large-scale 3D QA dataset, resulting in reduced logical reasoning capabilities. Due to the accumulation of model errors, performance typically degrades after more than three consecutive actions(more than 450 frames).

5 Conclusion

We present OpenHOI, the first open-world framework for synthesizing long-horizon 3D handobject interaction (HOI) sequences guided by open-vocabulary instructions. By fine-tuning a 3D multimodal large language model (MLLM) to jointly model geometric affordances and decompose semantic instructions, our method achieves strong generalization to unseen objects and linguistically complex tasks. The integration of affordance-driven diffusion-based generation and physics-aware refinement enables physically consistent manipulation sequences, advancing beyond closed-set HOI synthesis methods. Extensive evaluations demonstrate OpenHOI's superiority in handling novel object categories, multi-stage tasks, and open-ended language commands, bridging critical gaps in human-centric AI applications. While OpenHOI represents a significant step toward open-world HOI synthesis, several challenges remain: Although our physics-aware refinement improves interaction plausibility, fine-grained dynamics (e.g., fluid simulation for pouring tasks) remain challenging. Hybrid neuro-symbolic physics models may enhance realism. While OpenHOI supports multi-stage tasks, handling compositional long-horizon sequences like 'cook a meal' remains challenging due to limitations in hierarchical task decomposition. Future work could explore chain-of-thought reasoning to better model such complex action hierarchies.

References

German Barquero, Sergio Escalera, and Cristina Palmero. 'Seamless Human Motion Composition with Blended Positional Encodings'. In: (2024).

Junuk Cha et al. 'Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2024, pp. 1577-1585.

Sammy Christen et al. 'DiffH2O: Diffusion-based synthesis of hand-object interactions from textual descriptions'. In: SIGGRAPH Asia 2024 Conference Papers . 2024.

Hengshuo Chu et al. '3D-AffordanceLLM: Harnessing Large Language Models for OpenVocabulary Affordance Detection in 3D Worlds'. In: arXiv preprint arXiv:2502.20041 (2025).

Zekai Deng et al. Human-Object Interaction via Automatically Designed VLM-Guided Motion Policy . 2025. arXiv: 2503.18349 [cs.CV] . URL: https://arxiv.org/abs/2503.18349 .

Zicong Fan et al. 'ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation'. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 2023.

Rao Fu et al. 'GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities'. In: (2025).

Alexandros Graikos, Srikar Yellapragada, and Dimitris Samaras. Conditional Generation from Unconditional Diffusion Models using Denoiser Representations . 2023. arXiv: 2306.01900 [cs.CV] . URL: https://arxiv.org/abs/2306.01900 .

Kristen Grauman et al. Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives . 2024. arXiv: 2311.18259 [cs.CV] . URL: https://arxiv.org/ abs/2311.18259 .

Chuan Guo et al. 'TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts'. In: ECCV . 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. 'Denoising diffusion probabilistic models'. In: Advances in neural information processing systems 33 (2020), pp. 6840-6851.

Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance . 2022. arXiv: 2207.12598 [cs.LG] . URL: https://arxiv.org/abs/2207.12598 .

Mingzhen Huang et al. 'HOIGPT: Learning Long Sequence Hand-Object Interaction with Language Models'. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Nashville, USA, 2025.

Chaofan Huo, Ye Shi, and Jingya Wang. 'Monocular Human-Object Reconstruction in the Wild'. In: Proceedings of the 32nd ACM International Conference on Multimedia . 2024, pp. 5547-5555.

Kaiyang Ji et al. Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesis . 2025. arXiv: 2508.02106 [cs.CV] . URL: https: //arxiv.org/abs/2508.02106 .

Biao Jiang et al. 'MotionGPT: Human Motion as a Foreign Language'. In: Advances in Neural Information Processing Systems 36 (2024).

Taein Kwon et al. H2O: Two Hands Manipulating Objects for First Person Interaction Recognition . 2021. arXiv: 2104.11181 [cs.CV] . URL: https://arxiv.org/abs/2104.11181 .

Xin Lai et al. 'Lisa: Reasoning segmentation via large language model'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2024, pp. 9579-9589.

Jihyun Lee et al. 'InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion'. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 2024.

Kailin Li et al. 'Semgrasp: Semantic grasp generation via language aligned discretization'. In: European Conference on Computer Vision . Springer. 2024, pp. 109-127.

Peiming Li et al. 'ClickDiff: Click to Induce Semantic Contact Map for Controllable Grasp Generation with Diffusion Models'. In: Proceedings of the 32nd ACM International Conference on Multimedia . MM'24. ACM, Oct. 2024, pp. 273-281. DOI: 10.1145/3664647. 3680597 . URL: http://dx.doi.org/10.1145/3664647.3680597 .

Yicong Li et al. 'LASO: Language-guided Affordance Segmentation on 3D Object'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2024, pp. 14251-14260.

Pei Lin. 'HandDiffuse: generative controllers for two-hand interactions via diffusion models'. In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 39. 5. 2025, pp. 52805288.

Xueyi Liu and Li Yi. 'GeneOH Diffusion: Towards Generalizable Hand-Object Interaction Denoising via Denoising Diffusion'. In: The Twelfth International Conference on Learning Representations . 2024.

Yun Liu et al. TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding . 2024. arXiv: 2401.08399 [cs.CV] . URL: https://arxiv.org/abs/2401.08399 .

Yunze Liu et al. HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction . 2024. arXiv: 2203.01577 [cs.CV] . URL: https://arxiv.org/abs/2203.01577 .

Haowen Luo, Yunze Liu, and Li Yi. 'Physics-aware Hand-object Interaction Denoising'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2024, pp. 2341-2350.

Rundong Luo et al. PhysPart: Physically Plausible Part Completion for Interactable Objects . 2025. arXiv: 2408.13724 [cs.CV] . URL: https://arxiv.org/abs/2408.13724 .

Alex Nichol et al. 'Point-e: A system for generating 3d point clouds from complex prompts'. In: arXiv preprint arXiv:2212.08751 (2022).

Alexander Quinn Nichol and Prafulla Dhariwal. 'Improved denoising diffusion probabilistic models'. In: International Conference on Machine Learning . PMLR. 2021, pp. 8162-8171.

Mokai Pan et al. UniDB++: Fast Sampling of Unified Diffusion Bridge . 2025. arXiv: 2505. 21528 [cs.CV] . URL: https://arxiv.org/abs/2505.21528 .

Aditya Prakash et al. 'How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions'. In: Computer Vision and Pattern Recognition (CVPR) . 2025.

Zekun Qi et al. 'Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining'. In: International Conference on Machine Learning . PMLR. 2023, pp. 28223-28243.

Zekun Qi et al. 'Shapellm: Universal 3d object understanding for embodied interaction'. In: arXiv preprint arXiv:2402.17766 (2024).

Shengyi Qian et al. 'Affordancellm: Grounding affordance from vision language models'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2024, pp. 7587-7597.

Zhongwei Ren et al. 'Pixellm: Pixel reasoning with large multimodal model'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2024, pp. 2637426383.

Aditya Sanghi et al. 'Clip-forge: Towards zero-shot text-to-shape generation'. In: arXiv preprint arXiv:2110.02624 (2021).

Omid Taheri et al. 'GRAB: A Dataset of Whole-Body Human Grasping of Objects'. In: European Conference on Computer Vision (ECCV) . 2020. URL: https://grab.is.tue. mpg.de .

Guy Tevet et al. 'Human Motion Diffusion Model'. In: The Eleventh International Conference on Learning Representations . 2023. URL: https://openreview.net/forum?id= SJ1kSyO2jwu .

Jie Tian et al. 'Gaze-guided Hand-Object Interaction Synthesis: Dataset and Method'. In: arXiv preprint arXiv:2403.16169 (2024).

Hugo Touvron et al. 'Llama: Open and efficient foundation language models'. In: arXiv preprint arXiv:2302.13971 (2023).

Hanqing Wang et al. DAG: Unleash the Potential of Diffusion Model for Open-Vocabulary 3D Affordance Grounding . 2025. arXiv: 2508.01651 [cs.CV] . URL: https://arxiv.org/ abs/2508.01651 .

Hanqing Wang et al. SDEval: Safety Dynamic Evaluation for Multimodal Large Language Models . 2025. arXiv: 2508.06142 [cs.CV] . URL: https://arxiv.org/abs/2508. 06142 .

Wenbo Wang et al. 'UniGraspTransformer: Simplified Policy Distillation for Scalable Dexterous Robotic Grasping'. In: arXiv preprint arXiv:2412.02699 (2024).

Yi-Lin Wei et al. 'Afforddexgrasp: Open-set language-guided dexterous grasp with generalizable-instructive affordance'. In: arXiv preprint arXiv:2503.07360 (2025).

Yi-Lin Wei et al. 'Grasp as you say: Language-guided dexterous grasp generation'. In: arXiv preprint arXiv:2405.19291 (2024).

Qianyang Wu et al. THOR: Text to Human-Object Interaction Diffusion via Relation Intervention . 2024. arXiv: 2403.11208 [cs.CV] . URL: https://arxiv.org/abs/2403.11208 .

Shijie Wu et al. 'AffordDP: Generalizable Diffusion Policy with Transferable Affordance'. In: arXiv preprint arXiv:2412.03142 (2024).

Kaustubh Yadav. A Comprehensive Study on Optimization Strategies for Gradient Descent In Deep Learning . 2021. arXiv: 2101.02397 [cs.LG] . URL: https://arxiv.org/abs/ 2101.02397 .

Lingxiao Yang et al. 'Guidance with Spherical Gaussian Constraint for Conditional Diffusion'. In: Forty-first International Conference on Machine Learning . 2024.

Yuhang Yang et al. 'Grounding 3d object affordance from 2d interactions in images'. In: Proceedings of the IEEE/CVF International Conference on Computer Vision . 2023, pp. 1090510915.

Chunlin Yu et al. 'SeqAfford: Sequential 3D Affordance Reasoning via Multimodal Large Language Model'. In: arXiv preprint arXiv:2412.01550 (2024).

Xinyu Zhan et al. 'Oakink2: A dataset of bimanual hands-object manipulation in complex task completion'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2024, pp. 445-456.

Hui Zhang et al. ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulation . 2024. arXiv: 2309.03891 [cs.RO] . URL: https://arxiv.org/abs/ 2309.03891 .

Hui Zhang et al. GraspXL: Generating Grasping Motions for Diverse Objects at Scale . 2024. arXiv: 2403.19649 [cs.RO] . URL: https://arxiv.org/abs/2403.19649 .

Wanyue Zhang et al. BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects . 2025. arXiv: 2412.05066 [cs.CV] . URL: https://arxiv.org/ abs/2412.05066 .

Yiming Zhong et al. 'DexGrasp Anything: Towards Universal Robotic Dexterous Grasping with Physics Awareness'. In: arXiv preprint arXiv:2503.08257 (2025).

OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model Appendix

Appendix:

3D MLLM. We initialize our model from the ShapeLLM-7B checkpoint, freezing its 3D encoder and augmenting the visual backbone with Uni3D for robust dense 3D prediction, while the projection head is implemented as a shallow MLP, and LoRA is applied to streamline fine-tuning. Training unfolds in two stages: first, we optimize for seven epochs with AdamW (learning rate 2 · 10 -4 , zero weight decay) under a cosine-annealing schedule and a 2% linear warm-up; then, we continue for three additional epochs with AdamW (learning rate 5 · 10 -4 , zero weight decay) under the same cosine schedule but a 1% warm-up.

Diffusion Model. We employ a T = 1000-step noising process with a cosine noise schedule, and inject positional information at both the frame- and agent-levels using sinusoidal encodings. During sampling, we apply classifier-free guidance by randomly substituting 10% of conditioning inputs with unconditional noise while retaining 90% of the original conditions, and use a guidance scale of 2.5 to steer the denoising trajectory.

B Instruction Decomposition and Affordance Reasoning via 3D MLLM

B.1 Affordance Reasoning

3D Object Point Cloud Encoding. We take as input a point cloud of an object, sampled to N points. The backbone is a ReCon++ [34] network (or a similar architecture) that processes these points and produces per-point feature representations

which capture both local geometric details and the overall global context.

Multi-Token Fusion Mechanism. Rather than generating a single <AFF> segmentation token, PixelLM[36] defines, at each visual scale ℓ , a segmentation codebook comprising N learnable <AFF> tokens. After encoding the textual prompt, the model sequentially outputs the N tokens, each associated with a hidden vector h ℓ i . A linear projection ϕ then aggregates these vectors into a unified representation

which is concatenated with the scale-specific image features and fed into the pixel decoder to produce the final segmentation mask. Experiments on the MUSE validation set indicate that increasing N from 1 to 3 improves cIoU, demonstrating that the multi-token fusion mechanism captures more nuanced semantic details and significantly enhances fine-grained segmentation.

B.2 Instruction Decomposition

OpenHOI decomposes a single high-level instruction into an ordered sequence of actionable affordance steps. Each step is marked by a special token and then grounded spatially in the 3D point cloud.

Instruction Text Encoding. We take a natural-language instruction T ins as the model input. The backbone is a LLaMA-style Transformer that produces token-wise hidden states { h t } T t =1 and aggregates them via a pooling operation into a single embedding

which is then used for downstream tasks.

Segmentation Token Injection. Extend the MLLM's vocabulary by adding a special marker <AFF> , which explicitly denotes the boundary of each sub-task in the generated sequence.

Conditioned Autoregressive Generation. Given the fused 3D point features F obj and the instruction embedding h cls , the Transformer predicts an interleaved stream of action words and <AFF> tokens, for example:

Let S be the total number of <AFF> tokens generated.

Boundary Localization & Hidden-State Extraction. Record the positions t 1 , . . . , t S where <AFF> appears. For each i = 1 , . . . , S , extract the corresponding last-layer hidden vector

which encodes the full context immediately preceding the end of sub-task i .

Sequential Mask Decoding. For each step i , use the query E i to perform cross-attention over the point features F obj and decode a per-point mask:

where ˜ F are the fused features and σ is the sigmoid activation. Collect the ordered set { M 1 , M 2 , . . . , M S } to obtain the final sequence of affordance masks, each aligned with its corresponding sub-task.

B.3 Diffusion Process

Our framework employs diffusion models to learn the conditional distribution p ( X | C ) , of hand-object interaction (HOI) sequences, where the conditioning signal C combines:

Object affordance prior ˜ A obj

Sub-task embedding f clip ( ˜ T sub_tasks )

Object point cloud features F obj

Forward Process. The diffusion process gradually corrupts the input data through the forward process with a fixed noise schedule α t ∈ [0 , T ]

where X 0 is the original HOI sequence, X t represents its noisy version at timestep t and ¯ α t = ∏ t i =1 α t . This forward process progressively transforms the data distribution into a tractable Gaussian distribution N (0 , I ) .

Loss Function. Like VAEs, the diffusion model can be optimized by maximizing the ELBO:

By Simplification, Eq. 19 can be reduced to the following:

where σ t = 1 -¯ α t -1 1 -¯ α t β t . After removing constant terms, we obtain the denoising loss in diffusion models:

We also introduce geometric loss, including distance map loss L hoi_distance and relative orientation loss L hoi_orient for physical plausibility. To enable classifier-free guidance, we randomly mask 10% of the condition to train an unconditional model X θ ( X t , t, ∅ ) . Since the unconditional model captures the natural HOI sequence, it is then utilized as a prior to generate seamless transitions between different HOI sequences.

Sampling Process. During sampling, we employ classifier-free guidance to enhance alignment with the conditioning input C . This approach demonstrates superior performance compared to using only the conditional model X θ ( X t , t, C ) :

where s ≥ 1 controls the guidance strength. We generate samples through an iterative denoising process using the reverse diffusion posterior:

B.4 Loss-guided Physical Refinement

Loss guidance is a technique that minimizes the off-the-shelf loss function L ( X 0 , y ) during the sampling time:

where y is the conditioning input, M denotes the conditional data manifold that follows the conditional distribution p ( X 0 | C ) learned by the diffusion model. In this work, we propose a novel loss-guided sampling strategy that explicitly enforces physical constraints during the denoising process to achieve more realistic hand-object interactions.

C Additional Experiments

C.1 Results on Extreme-Case: Completely Unseen Datasets

Result on H2O. We further subject our model to extreme-case testing to stress its generalization under the most challenging conditions. All objects and instructions in the H2O dataset are entirely novel to models trained on GRAB and ARCTIC, making evaluation on H2O a particularly stringent test of generalization. Despite this extreme distribution shift, our experiments demonstrate that the proposed model nonetheless delivers robust and state-of-the-art performance (shown in Table A1).

C.2 Evaluation for physical realism

We supplemented our experiments by evaluating Physical Realism and IV metrics against the closest baseline, Text2HOI (HOIGPT's code is not publicly available) in A2, and conducted ablation studies on our Physical Refinement module in A3.

C.3 3D MLLM Fine-tuning

We fine-tune the MLLM on the Affordance dataset [52] and the HOI dataset [38, 6], We first perform coarse-grained fine-tuning on the Affordance dataset to instill strong affordance priors, and then carry out fine-grained tuning on the HOI dataset to produce our final model [43]. The results shown in Table. A4.

C.4 Sensitivity Analysis

We conducted a sensitivity analysis on the guidance rate, and the results are as follows(shown in Table A5 and Table A6). Our experimental results demonstrate that the proposed model maintains robust performance even under these challenging conditions.

C.5 Ablation Study on Multi <AFF>

The additional ablation study results are as follows A7.

C.6 Ablation Study on Motion In-between Refinement

Motion In-between Metric. To the best of our knowledge, no previous work has defined an evaluation metric for motion in-between hand-object interaction. We thus introduce a simple yet effective measure, the 'Smooth Rate,' to quantify the temporal continuity of interpolated motion segments as follows,

where d t is the derivative of time. The results are shown in Table A8 and Table A9.

To determine the most suitable window size [1] for the motion in-between algorithm, we conducted the following comparative experimentsA10.

C.7 Visualization on Affordance

In this subsection, we present visualizations of open-world affordances on seen and unseen objects in Fig. A1.

C.8 Qualitative results Compare with SOTA

This section presents additional visual comparisons between our approach and existing state-of-the-art (SOTA) methods.

Seen Objects. Qualitative results on seen objects in Fig. A2.

Unseen Objects. Qualitative results on unseen objects in Fig. A3.

C.9 Statistically Insignificant

For the ablation study, we performed paired two-sample t-tests, repeating each test five times and reporting the mean p-value. As summarized in Table A11, the proposed method is significant at the 95% confidence level for the majority of metrics(P-value ≤ 0.05).

Unseen Object Affordance

I want to play computer games

I am hungry

I am thirsty

I want to listen to Jay Chou

Template-free Generalizability

I want to go shopping, can you tell me how to pick up these bags?

Sure! You can pick them up with these parts!

I am thirsty, could you help me ?

GLYPH(cmap:d83d)

:

MDM

Text2HOI

Ours

GLYPH(cmap:d83e)

:

Of course, please use the bowl to drink some water with both hands.

Timestep

D Code and Dataset

D.1 Code

We will release our code as soon as possible. GitHub is OpenHOI

D.2 Dataset

H2O. This dataset contains 571,645 synchronized multi-view RGB-D frames captured with five Kinect sensors in three indoor scenes. Each frame includes 3D poses for both hands, 6-DoF object poses, and verb-noun action labels (36 classes). Split into training (344,645), validation (73,380), and test (153,620) frames, H2O supports egocentric interaction recognition and manipulation benchmarks.

GLYPH(cmap:d83d) : I want to read that interesting story again!

GLYPH(cmap:d83e) : Reading a storybook is a great choice. Please cradle the volume with one hand on each cover and bring it to a comfortable viewing angle and distance.

MDM

Text2HOI

Ours

Timestep

Dataset annotation. For both the GRAB [38] and ARCTIC [6] datasets, we preprocess the datasets and annotate the semantics. After that, the object point clouds are first upsampled to gain accurate affordance maps using our inference model while ensuring fine geometric details in the meantime. Our model then processes the upsampled data to infer accurate affordance maps, which are subsequently downsampled to match the original resolution for efficient computation.

We preprocess both datasets and get their semantic annotations. First, we upsample the object point clouds to enhance geometric details and generate accurate affordance maps using our inference model. The upsampled data is then processed to infer affordance maps, which are downsampled back to the original resolution for computational efficiency.

To enhance the semantic alignment between language and interaction, we employ a large language model (LLM) to refine the original hand-object interaction (HOI) descriptions. The LLM generates high-level, natural language annotations that better capture the intent and dynamics of HOI.

E Discussion

E.1 Generalizability ability in HOI: Affordance as a key

Affordance is a powerful and explicit prior for interaction that can guide complex and fine-grained HOI synthesis. Our model achieves strong generalization by using affordance as a middleware layer.

Open-World Affordance Grounding: We first employ a coarse-to-fine tuning strategy to equip the model with strong affordance reasoning capabilities, enabling it to generate open-world affordance grounding. This enables the synthesis of realistic HOI sequences, demonstrating strong template-free generalization capabilities.

Affordance serves as a crucial condition: The open-world affordance grounding serves as a crucial condition for the affordance-driven HOI Diffusion.We incorporate affordance not only during training but also in the loss-guidance applied during inference.

Affordance-based Refinement: we design an improved refinement strategy based on affordance: for the interaction between the hand and the object, we optimize based on affordance grounding rather than the conventional closest-surface-point approach.

A Template-free Example: Our 3D MLLM has learned from a wide variety of cups, it can still generate accurate affordance grounding for a completely unseen mug. Accurate affordance grounding will guide the synthesis of realistic HOI sequences.

E.2 How to use 3D MLLM in Emboided AI: Choose Powerful Foundation Model and Coarse-to-fine tuning

The 3D multimodal large model has been widely applied in HOI and embodied intelligence, and it is very important to choose a basic model suitable for downstream tasks. In OpenHOI, we chose ShapeLLM as our base model, ShapeLLM is a powerful 3D foundation model that performs exceptionally well on multiple downstream tasks (e.g., Embodied Visual Grounding, Visual Question Answering, and Scene Understanding), making it highly suitable for HOI tasks.After research, we found that ShapeLLM has the following advantages and disadvantages

Advantages: ShapeLLM is trained on a large amount of 3D embodied interaction data and achieves state-of-the-art performance across various downstream tasks. It possesses strong priors in 3D interaction and demonstrates impressive zero-shot 3D representation capabilities.

Disadvantages: ShapeLLM has not been trained on part-level object annotations, which limits its reasoning capabilities for fine-grained object understanding.

In order to make the selected 3D base model as suitable as possible for our task, we need to use data to fine tune the model. We adopt a coarse-to-fine tuning strategy: we first pre-train the model on an object-centric affordance dataset to enable it to acquire strong affordance priors. Then, we fine-tune the model on HOI datasets to better align the semantics with the target domain. This allows the model to learn highly effective affordance representations.

E.3 Future of Work: Real-World Applications, about AR/VR and Robotics

OpenHOI can be extended to a wide range of future work in other fields. We have listed several noteworthy areas and provided preliminary solutions for the challenges in future applications

Robotics Manipularion: OpenHOI can be integrated into real-world robotic manipulation systems, including industrial robot arms and service robots, to enable more flexible and human-like interactions[48, 54, 55].

Open-World Affordance Grounding as Powerful Guidance for Robots: Leveraging OpenHOI's 3D MLLM, our method performs open-world affordance grounding to facilitate the identification of feasible grasping, pushing, and tool-use regions on novel objects, thereby significantly improving success rates for pick-and-place, assembly, and tool-handling tasks.

Realistic HOI sequences synthesis for Robot Manipulation: The HOI sequences generated by OpenHOI can be adapted into robotic manipulation sequences. First, we can use an extraction algorithm to obtain the object's 6-DoF pose. Then, inverse kinematics are applied to the wrist parameters to compute the robot arm's pose. Finally, a retargeting algorithm transfers the human hand motions onto various robotic hand configurations for manipulation. This approach ensures smooth, precise, and robust manipulation behaviors in real-world deployments.

Virtual Reality Vision: By synthesizing realistic 3D hand-object interaction sequences, OpenHOI enables users to manipulate virtual objects naturally, for example, by picking up, twisting, or pouring items, thereby enhancing immersion in training simulators, gaming, and virtual prototyping.

Challenges: Robotic manipulation tasks typically demand rapid inference. We plan to employ DPM-Solver to accelerate the diffusion inference process, which is an important direction for our future work[31].

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: As stated in the abstract and introduction, this paper introduce the first OpenWorld Hand-Object Interaction (HOI) Synthesis framework that can generate Long-horizon HOI sequences of Unseen Objects from Open-vocabulary instruction with 3D Multimodal Large Language Model.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The analysis of limitations is provided in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our work dose not include theoretical result.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: These details are provided in the Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will release the code once the paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: These details are provided in Section 4 and Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The standard deviation in our experiment results (Table 2 and Table 3) shows the statistical significance.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The information on the computer resources is provided in Sec 4 and Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conformed with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our paper primarily focuses on theoretical research in how to employ the diffusion model in affordance learning, with no consideration of societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our experiment poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The implementation is cited properly in the Appendix

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not introduce new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: We do not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for 3D hand-object interactions leveraging an MLLM. The use of MLLM allows for long-horizon manipulation. The framework leverages a pre-trained MLLM and diffusion model, and minimal training using LoRA adapters. In addition, several refinements during and post-training are applied to account for the geometrical shortcomings of LLMs.\n\nSTRENGTHS AND WEAKNESSES:\nI have little experience in the HOI task (which is reflected in the confidence of this review). As such, the comments are primarily regarding the MLLM and the completeness of the experiment section.\n- is the method using a single \\<Aff\\> token? it appears as if multiple are used in Fig 2, but only one is mentioned in the methodology section. If multiple are used, these need to be made explicit in the method section, and relevant experiments should be included in the ablation studies.\n- Furthermore, all results presented would benefit from statistical significance testing, as mean and standard deviation are provided. This is important, as some results in the ablation (eg, MPJPE and Diversity) appear comparable, and the importance of each component may be undermined (eg, affordance loss).\n- The method should include failure cases and an error analysis. \n- How long is the horizon before performance deteriorates? What instructions/samples does the method fail to address?\n- Are there any limitations to the method?\n\nQUESTIONS:\nThe paper is well written. Most questions are regarding error analysis, as discussed more extensively above, but the key things to address are:\n\n- Clarity on the \\<aff\\> tokens is necessary\n- Some results appear statistically insignificant; it would be best to show the importance of each component\n- Failure cases, error analysis and limitations are missing\n\nLIMITATIONS:\nThe authors have not discussed limitations and societal impact, both of which should be included in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes OpenHOI, the first open-world hand-object interaction (HOI) synthesis framework capable of generating long-horizon manipulation sequences for novel objects guided by open-vocabulary instructions. The paper's contributions are significant and address a challenging problem in the field of 3D interaction research. The proposed approach combines a 3D multimodal large language model (MLLM) with an affordance-driven diffusion model and physics-aware refinement, achieving strong generalization capabilities across object categories, task horizons, and linguistic complexity. Experimental results demonstrate the superiority of the method over existing state-of-the-art approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper presents the first open-world hand-object interaction (HOI) synthesis framework, OpenHOI, which can generate long-horizon manipulation sequences for novel objects guided by open-vocabulary instructions. This represents a significant advancement in the field of 3D interaction research, addressing a previously unresolved challenge.\n2. The proposed method demonstrates strong generalization capabilities across object categories, task horizons, and linguistic complexity. It outperforms existing state-of-the-art methods in handling unseen objects and complex, linguistically diverse commands.\n3. The paper is well-organized and clearly written. The authors provide a detailed introduction to the research background and motivation, comprehensively review related work, meticulously describe the proposed method, and thoroughly evaluate the experimental results.\nWeakness\n1. Hierarchical Task Decomposition Limitations: The paper notes that while OpenHOI supports multi-stage tasks, it struggles with compositional long-horizon sequences like \"cook a meal\" due to limitations in hierarchical task decomposition. This suggests that the method may not be suitable for highly complex and hierarchical tasks.\n2. Limitations in Fine-Grained Dynamics: Although the physics-aware refinement improves interaction plausibility, the paper acknowledges challenges in handling fine-grained dynamics, such as fluid simulation for pouring tasks. This indicates that the proposed method may not fully capture the complexities of certain interactions.\n\nQUESTIONS:\n1. The paper adopts ShapeLLM as the backbone of the 3D MLLM. What are the advantages and disadvantages of this choice compared to the other 3D MLLM models? How does ShapeLLM's performance in the context of OpenHOI align with its original design goals and capabilities?\n2. What are the potential applications of the proposed OpenHOI framework in real-world scenarios, such as robotics and virtual reality? What challenges might arise in practical applications, and how can they be addressed?\n\nLIMITATIONS:\nPlease find in weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces OpenHOI, the first framework for open-world hand-object interaction (HOI) synthesis. It enables the generation of long-horizon HOI sequences for novel objects conditioned on language instructions. The core technical idea is to leverage the embedded knowledge of a 3D Multimodal Large Language Model (MLLM) to enhance the generalizability of HOI generation. Specifically, the approach fine-tunes a pre-trained 3D MLLM for affordance grounding and semantic task decomposition. These outputs are then used in a diffusion-based HOI synthesis and a physics-based refinement stage to produce the final HOI sequences. IIn experiments, OpenHOI achieves new state-of-the-art performance on HOI generation for both seen and unseen objects, outperforming existing state-of-the-art methods.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n\n1. Writing quality\n\nOverall, the paper is easy to read, and the motivation and technical contributions are clearly presented.\n\n2. Good motivation\n\nI agree that enhancing the generalizability of HOI generation to unseen object categories is an important research direction, as this remains a known limitation of existing HOI generation methods. In this context, leveraging the embedded knowledge of a pre-trained 3D MLLM is a reasonable and promising approach.\n\n3. Strong method novelty\n\nThe proposed method is novel and interesting. This is one of the first works to leverage a large language model for HOI generation, and the approach aligns well with the high-level goal of improving generalizability, as noted in the previous point.\n\n\n[Weaknesses]\n\n1. Questionable generalizability of the diffusion-based HOI generation module\n\nThe proposed framework consists of two sequential stages: (1) 3D MLLM-based instruction decomposition and affordance reasoning, and (2) diffusion-based HOI generation conditioned on the outputs of the 3D MLLM. While the first stage may offer high generalizability by leveraging the pre-trained 3D MLLM, it remains unclear how the second, diffusion-based module achieves strong generalizability. In the experiments, the performance gap relative to the existing state-of-the-art baseline (Text2HOI) does not noticeably differ between seen and unseen objects. This raises the question of whether the improved performance is primarily due to stronger representation capacity rather than actual improvements in generalization, which is the main claim of the paper.\n\n2. Missing evaluation for physical realism\n\nOne of the claimed technical contributions is the introduction of a physics-based refinement stage to enhance the physical realism of the generated HOI sequences. However, empirical validation for this component is missing, despite the fact that the closest baselines (e.g., Text2HOI, HOIGPT) report evaluation metrics specifically designed to assess physical realism (e.g., physical realism, interpenetration volume).\n\n3. Questionable performance on long-horizon generation\n\nAnother core contribution claimed by the paper is the ability to generate long-horizon HOI sequences. However, I have concerns about this claim due to the method design. Methodologically, the approach performs in-between motion refinement by enforcing smoothness only between adjacent frames (Eq. 11), which contrasts with existing long-horizon synthesis methods (e.g., [R1]) that consider much longer transition windows. Clarifying these points would be helpful.\n\n[R1] Barquero et al., FlowMDM: Seamless Human Motion Composition with Blended Positional Encodings, CVPR 2024.\n\nQUESTIONS:\nPlease see the Weaknesses section.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOpenHOI is the first open-world hand-object interaction (HOI) synthesis framework that generates long-horizon manipulation sequences for unseen objects guided by open-vocabulary instructions. It fine-tunes a 3D multimodal large language model (MLLM) to support two key abilities: (1) Affordance prediction, localizing actionable regions on novel objects; and (2) Instruction decomposition, breaking complex commands into executable sub-tasks. The framework combines affordance reasoning with diffusion-based interaction generation and physics-aware refinement, achieving state-of-the-art performance and strong generalization across object categories, task structures, and language complexity.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strength**\n- The paper is well-structured and easy to follow overall.\n- Open-world HOI generation is a highly valuable and challenging direction. The paper effectively leverages the strengths of 3D MLLMs to provide affordance priors and integrates them with HOI generation, generating interactions under high-level instruction.\n- Both the qualitative and quantitative results are promising, and the accuracy of the affordance prediction is also reasonably good.\n\n\n\n### **Weakness**\n- The visualizations provided for affordance prediction and hand-object interaction are somewhat limited, particularly for unseen objects. More qualitative examples would help better illustrate the model’s generalization capability.\n\nQUESTIONS:\n1. In line 193, shouldn't the term \"hand joints\" actually refer to hand surface vertices? If so, how did you recover the hand's surface?\n2. In the affordance refinement, why is the distance computed to the nearest point on the entire object surface, rather than to the nearest point within the predicted contact region?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}