MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation
Authors: Bohan Zhou
Yi Zhan
Zhongbin Zhang
Zongqing Lu
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Generation
Hand-Object Interaction
Primary area: deep_learning
TL;DR: MEgoHand is the starting point for generating high-quality motion sequences of hand-object interactions, conditioned on egocentric RGB images, textual instructions, and given initial MANO hand parameters.

Abstract: Egocentric hand-object motion generation is crucial for immersive AR/VR and robotic imitation but remains challenging due to unstable viewpoints, self-occlusions, perspective distortion, and noisy ego-motion. Existing methods rely on predefined 3D object priors, limiting generalization to novel objects, which restricts their generalizability to novel objects. Meanwhile, recent multimodal approaches suffer from ambiguous generation from abstract textual cues, intricate pipelines for modeling 3D hand-object correlation, and compounding errors in open-loop prediction. We propose **MEgoHand**, a multimodal framework that synthesizes physically plausible hand-object interactions from egocentric RGB, text, and initial hand pose. MEgoHand introduces a bi-level architecture: a high-level “cerebrum” leverages a vision language model (VLM) to infer motion priors from visual-textual context and a monocular depth estimator for object-agnostic spatial reasoning, while a low-level DiT-based flow-matching policy generates fine-grained trajectories with temporal orthogonal filtering to enhance stability. To address dataset inconsistency, we design a dataset curation paradigm with an Inverse MANO Retargeting Network and Virtual RGB-D Renderer, curating a unified dataset of **3.35M** RGB-D frames, **24K** interactions, and **1.2K** objects. Extensive experiments across **five** in-domain and **two** cross-domain datasets demonstrate the effectiveness of MEgoHand, achieving substantial reductions in wrist translation error (**86.9%**) and joint rotation error (**34.1%**), highlighting its capacity to accurately model fine-grained hand joint structures and generalize robustly across diverse scenarios.

MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation
Bohan Zhou' Yi Zhan' Zhongbin Zhang?
Zongqing Lu Lu 131
School of Computer Science, Peking University
Department Automation, Tsinghua University BeingBeyond
Abstract
Egocentric hand-object motion generation is crucial for immersive AR/VR and robotic imitation but remains challenging due to unstable viewpoints, self- occlusions, perspective distortion, and noisy ego-motion. Existing methods rely predefined 3D object priors, limiting generalization to novel objects, which restricts their generalizability novel objects. Meanwhile, recent multimodal approaches suffer from ambiguous generation from abstract textual cues, intricate pipelines for modeling 3D hand-object correlation, and compounding errors in open-loop predic- tion. We propose MEgoHand, multimodal framework that synthesizes physically plausible hand-object interactions from egocentric RGB, text, and initial hand pose. MEgoHand introduces bi-level architecture: a high-level "cerebrum" leverages a vision language model (VI.M) to infer motion priors from visual-textual context and monocular depth estimator for object-agnostic spatial reasoning, while a low-level DiT--ased flow-matching policy generates fine-grained trajectories with temporal orthogonal filtering to enhance stability. To address dataset inconsistency, we design a dataset curation paradigm with an Inverse MANO Retargeting Net- work and Virtual RGB-D Renderer, curating unified dataset of 3.35M RGB-D frames, 24K interactions, and 1.2K objects. Extensive experiments across five in- nomaiiin and two cross cross-domain datasets demonstrate the effectiveness of MEgo- Hand, achieving substantial reductions wrist translation error (86.9% %) and joint rotation eтror (34.1% %) ), highlighting its capacity to accurately model fine finengrained hand joint structures and generalize robustly across diverse scenarios.
Introduction
The egocentric perspective is humanity's native mode of interaction, directly reflecting how perceive and engage with the world [7, 17, 48]. (8.. It provides rich contextual cues that are often lost in third-person observations, such the alignment of gaze and hand motion and the real-time visual- motor feedback that guides manipulation [39]. Generating hand-object motions from first-person views is fundamental for many applications. In AR/VR, it enables precise virtual-real alignment [17,30, 40, 37, 36), while in robot learning, it facilitates natural imitation from human demonstrations [27, 43, 46]. Despite its significant potential, predicting hand-object interactions from egocentric perspectives remains highly challenging [20]. First, continuous camera motion from head-mounted setups causes unstable and shifting viewpoints, disrupting spatial consistency. Second, frequent self-occlusions by the user's own body often hide the hands or objects, leading to missing visual information. Third, the close distance from the camera introduces strong perspective distortion and rapid scale changes, complicating spatial perception. Finally, distinguishing intentional hand
"Equal contribution
i Corresponning author <zongging.nicc.. @ @ku.edu..n>>
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

GARRNI
APD
HOND
ABSTC
METIC
HILO
MEgoHand
3.35M frames 1.2K objects 24K trajectories
SOTA: 86.9% 34.1% % error reductions in hand translation and joint rotation
Figure I: MEgoHand stands the starting point for generating high-quality motion sequences of hand-object interactions, conditioned on egocentric RGB images, textual instructions, and given initial MANO hand parameters.
movements from head-induee induced ego-motion requires models to reason under partial observations and sparse visual cues.
Existing methods largely ignore the challenges of first-person perspectives because they typically rely on predefined 3D object attributes (e.g., mass, geometry) to model hand-object interactions (HOI) [49, 34, 8, 6]. However, their performance significantly degrades when dealing with novel or unknown objects. Recent efforts have begun to address these limitations of first-person vision and reduce dependence on explicit object models. SIGHT-Fuuion [16] mitigates occlusion by segmenting hands and objects in egocentric RGB inputs, but the lack of textual guidance often leads to ambiguous action generation. In contrast, the multimodal LatentAct [3 [31] incorporates visual and textual information along with 3D contact points, enabling more context-award modeling. Nevertheless, the complex pipeline required for contact map generation limits its feasibility in real- world applications. Furthermore, these methods adopt an open-loop prediction strategy based solely on the first frame. Without corrective feedback, errors caused by viewpoint shifts and occlusions accumulate over time, ultimately leading to cascading failures in interaction prediction.
To address the aforementioned challenges, we propose MEgoHand, a Multimodal Egocentric Hand- Object Motion Generation approach. Given a textual instruction, RGB image, and initial hand pose, MEgoHand synthesizes high-fidelity, physically plausible motion sequences applicable to real- world scenarios. It adopts bi-level architecture. The High-Level module leverages Eision-languago model (VLM) to infer motion priors from visual perception, task understanding, and alignment. Furthermore, to enhance spatial understanding of hand-object relationships without relying on object-specific priors, we incorporate a monocular depth estimator that encodes RGB images into a dense depth representation. The Low-Level module generates fine-grained hand trajectories via a DiT-based flow-matching policy, effectively modeling temporal uncertainty and ensuring motion continuity. To mitigate observation noise induced by egocentric camera motion, the model performs frame-wise prediction of trajectories over the next / frames, followed by Temporal Orthogonal Filtering (TOF) decoding strategy enhance temporal coherence and stability.
Additionally, we notice that despite the abundance egocentric hand-object interaction datasets, inconsistencies in language instructions, annotation quality, and pose representations pose significant challenges to unified training. To address these issues, we develop a standardized preprocessing pipeline comprising Inverse MANO Retargeting Network for pose normalization and Vir- tual RGB-D Renderer for generating aligned depth maps. Based on this framework, we curate a multimodal dataset containing 3.35M RGB-D frames, 24K interaction trajectories, and 1.2K objects.
Our experiments show that our MEg achieves SOTA performance across five in-domain and two cross-domain datasets, reducing hand translation and joint rotation errors by 86.9% and 34.1 1%, respectively. After Procrustes alignment, joint and mesh vertex emoor decrease 0.424 em and 0.409 em, corresponding to 71.2% and 71.9% relative improvements over strong baselines. These results

---

highlight MEgoHand'ss 's effectiveness in modeling fine-gr teneggraied hand joint structures and its strong generalization capability across domains.
The main contributions of this paper are as follows: (1) We propose MEgoHand, the first framework to leverage vision-language models for motion prior inference in egocentric hand-object interactions, augmented with a monocular depth module for object-agnostic spatial reasoning. (2) To address dataset inconsistencies, we design standardized pipeline with an Inverse MANO Retargeting Network and a Virtual RGB-D Renderer to unity poses and generate aligned depth maps, producing a 3.35M-frrame multimodal dataset with 24K interactions and 1.2K objects for unified training and evaluation. (3) MEgoHand outperforms baselines on five in-domain and two cross- dross-domain benchmarks, substantially reducing hand translation and joint rotation errors, demonstrating its effectiveness in fine-grt ome-grained articulation and robust generalization.
2
Related Work
2.1 Hand Object Interaction Prediction
Recent advances in hand-object interaction prediction have explored diverse input modalities and generative paradigms to model semantically meaningful and physically plausible behaviors. Object- centric methods such as GEARS [49] and MACS [34] improve motion realism by explicitly modeling physical properties, such as mass and geometry. Similarly, DifiH20 [8] and Tex12HO1 [6] condition motion prediction on textual descriptions, but still require object-specific information. This reliance on predefined object parameters limits their ability to generalize to novel or unseen objects. In contrast, umage-based methods avoid explicit object modeling by learning from visual cues. For example, SIGHT-Fusion [16] predicts hand motion from egocentric images using contact-guided diffusion, demonstrating resilience to occlusions, though it still requires accurate object detection. Moreover, multimodal fusion approaches have shown notable advantages in modeling hand-object interactions. Representative works such as GROOT [4] integrate vision, language, and action modalities for human- like control, while HandsOn [3] leverages textual semantics and visual grounding for hand motion planning. Among them, LatentAct [31] is most relevant our goal of 3D hand motion generation. It jointly predicts 3D hand trajectories and contact maps from a monocular RGB image, textual action descriptions, and 3D contact points. However, its reliance on an intricately tailored process to generate contact maps hinders its practicality in real-world applications.
2.2 Hand Pose Reconstruction
3D hand pose estimation serves as a critical foundation for hand-object interaction prediction by providing accurate 6D pose initialization, enabling physically plausible motion forecasting through spatiotemporal constraints. Current approaches can be broadly classified into image-based and video-based methods. Image-based techniques include HaMeR [28]. which scales transformer architectures for pose estimation; Hamba [10], which employs Mamba-based state-space modeling to capture joint dependencies; and SimHand [22], which reduces annotation dependence through contrastive similarity learning. Video-based methods, by contrast, enhance temporal coherence by incorporating motion priors-for example, HMP [11] applies VAE-regularized latent optimization with AMASS priors, while Deformer [14] leverages part-aware deformation-F field transformers for dynamic modeling. Beyond isolated hand reconstruction, Hold [13] is the first category agnosic method for joint articulated hand-object reconstruction from monocular videos, using a compositional
implicit model with hand-object constraints.
In addition, egocentric videos, captured from head- or body cameras, pose distinct challenges for 3D hand pose estimation due to dynamic viewpoints, frequent self-occlusions, and ego-motion artifacts, factors that distinguish them from third-person perspectives. Recent solutions, such as HaWor [45], which employs SLAM-based motion decoupling, and HaPTIC [42], which leverages 4D cross-view attention to ensure temporal consistency, address these difficulties by modeling the complex spatiotemporal dependencies inherent in first-person views. In our study, we focus on egacentric video scenarios where accurate first-frame 3D hand pose estimation serves as the kinematic anchor for subsequent interaction prediction.
3

---

3 Methodology
3.1 Problem Formulation
We represent the hand using the MANO model [32], which is parameterized by hand shape parameters
(shape feature and finger rotations 0) and wrist pose parameters (rotation and translation t). To ensure smooth and continuous rotation modeling, we adopt the 6D rotation representation [50] for ! and A MANO hand is represented
IR 15x6 B E R Rix6, E IR'S Given task description T, visual observation Vk. and the initial l2s the goal is to predict a sequence of future MANO parameter sequences over time steps. V/ consists of possible RGB frame It and depth frame DR-
(0; B; r; #] IR 109 where
HA (h&+1, h842. b MEgoHand Vks hx)-
1. Text Encoding
2. Visual Encoding
Hand Motion Generation
DOOD
Vision-Language Model
Tokenizer Embedding
TDC
Depth Encoder
RGBD Fusion
Eagle Vision Encoder
{gloxe , )
h%g Current Motion Sequence
Noise
T
Prompt Task Description coneeneeeeeeee
D Estimated Depth
I
RGB Frame
Diffusion Transformer (DiT)
intall Isken, generate
seguence
UniDepth
(0 , , (ta) ha-!
following hand motion descripion."
DepthAnything
"Pick the cup on the tablee
H, Future Motion Sequence
During inference, the system prompt and task instruction are encoded using frozen VLM tokenizer. At each timestep, an RGB image is processed by a pretrained depth estimator to obtain a metric depth map. The RGB and depth images are then combined and encoded into a visual embedding, which together with the text embedding- motion generator receives this multimodal representation along with the initial hand parameters to predict relative future hand motion. During training, the depth encoder, VLM vision encoder, and DiT head are finetuned.
Figure 2:
input the frozen VLM. A DiT-based
3.2 Cerebrum: Multimodal Perception & Understanding
Robust hand motion generation in hand-object interaction demands recognition of target objects in cluttered scenes and reasoning about contact. Traditional approaches depend explicit object models [49, 34, 8, 6] or intricate modeling of hand-object relationships [31], which can limit adaptability and scalability. In this work, utilize VLMs for their strong generalizability. enabling automatic extraction task-relevant semantics and interaction patterns directly from visual observations and textual instructions-witheee the limited 3D spatial understanding of VLMs, we further incorporate a monocular metric depth estimation module that supplies geometric context essential for interaction planning.
relying on predefined object models. And additionally. to complement
VLM Backbone. The core of MEgoHand is built upon Eagle-2 [21], VLM that integrates SmolLM2 [1] language backbone with SigLIP-2 [38] vision encoder, both pre-trained large-scale Internet data. The text tokenizer and transformer blocks are frozen, and the vision encoder trainable.
3D Spatial Understanding. Pre-trained visual encoders are usually effective at 2D semantic under- standing while struggling with 3D spatial understanding, To address this, we first incorporate depth into multimodal hand motion generation framework. Specifically, we adopt the monocular metric depth estimator UniDepth V2 [29] estimate depth map from an input RGB image. Once the estimated depth map is obtained, we need to encode its spatial features. To the best of our knowledge, there is no existing depth encoder pretrained on large-scale depth datasets. Therefore, we adopt ResNet-50 [18] encoder pretrained on ImageNet [9]. Although trained on RGB data, we observe that its low-level priors (e.g., edges and textures) transfer effectively to depth inputs. To meet the 3-channel input requirement of ResNet, we replicate the single-channel depth map across channels. To finetune the depth encoder, we use mean squared error (MSE) to align the representation of predicted and ground- truth depth maps. Finally, an additive fusion module combines visual features

---

x² and depth features x= into a unified representation x xT, which interacts with semantic features xJ via cross-modal attention the Eagle-2 LLM. The resulting output 2 TTD captures both the hand-object correlation and the task requirement, enabling holistic understanding and action planning.
3.3 Cerebellum: Hand Motion Generation via Flow Matching
Conditioned Hand Motion Generation. After the VLM encodes the task instruction T. RGB frame Iz. and a depth map Da into a latent representation z 2TD a DiT-based motion generator is employed produce future motion sequence Hr of length a trunk of MANO parameters. This generation is conditioned on the initial MANO parameters h& because providing an initial hand configuration reduces ambiguity related to hand shape and dexterity, resulting in more realistic and intention contissenntn motion sequences. The predicted motion trunk is supervised using conditional flow matching loss [23, 24]:
L (0) E (PP
[116 h,, 2 (HG)))
(2)
where u(H[[H++) c- He. c NN I). The subscript denotes motion timestep, and the superscript E [0. 1] denotes flow matching timestep which is sampled from beta distribution biased toward lower (noisier) values during training. During inference, hand motions are generated by integrating the learned vector field from to r = 1, starting from Gaussian noise H. N/ I). The integration follows the forward Euler method:
H456 H% 8 vo (H), hg, 2k TDD),
where 5 is the integration step size. Please refer to Appendix A.3 for more details. In practice, all transformations are computed in the camera frame that we can conveniently estimate initial hand using modern detectors [28, 45]. Additionally, we predict the relative wrist transformation to the initial pose and repeat the initial shape parameter part of the output for each timestep.
Smooth Decoding Strategy. To ensure temporal coherence in the generated motion sequences, propose Temporal Orthogonal Filtering (TOF), a training-fres decoding strategy denoise predicted rotation sequences. At each timestep, we query the motion generator to produce overlapping motion chunks. Let f2z 0j denote the wrist rotation matrix and translation vector at timestep k generated during the query at timestep convolution with uniform weights aggregates all rotation and translation estimates corresponding to the same timestep k. The resulting translation is given by l = =' 2 /L. The resulting convolved rotation Rs is then projected onto the closest valid SO(3) manifold via Singular Value Decomposition (SVD), producing the smooth output Rg. The process of TOF is formalized in Equation (4). We can freely adjust the frequency of the query to balance inference speed and generation quality.
To suppress
jitter, a temporal
R4 arg min || Ralls UVT, where USV SVD(R,), R4
Σ
Ak
RESO(3)
4 Datasets Integration
Despite the abundance of egocentric hand-object interaction datasets, inconsistencies in language instructions, annotation quality, and hand pose representations hinder unified training. To address these discrepancies, we systematically integrate and preprocess large-scale public datasets into a unified and standardized training corpus.
Inverse MANO Retargeting: Some early datasets, such FPHA [15], only provide 3D hand joint positions j2 captured using wearable sensors instead of MANO parameters. The world-frame coordinates of 21 hand keypoints correspond to the output of the MANO model and cannot be directly used as inputs for MEgoHand or as supervision signals for motion generation. To utilize these datasets, we introduce an Inverse MANO Retargeting Network é, which recovers the MANO parameters from joint coordinates. A straightforward approach would be to employ end-to-end supervised learning. However, this method fails completely in practice, as even minor deviations in hand shape- particulaalllllll for the shape vector 3 3-can can lead to severely distorted hand geometry and huge reconstruction errors. To overcome this, we propose novel two-stage iterative training strategy, along with relfssecontttuttionn loss. As shown in Figure 3a, first prioritize optimizing the hand
5

---

(b) Summary of curated HOI datasets. We split the first 6 for training and the last for testing.
Dataset
Frame
Trajectory
Object
Mesh
RGBD
OakInk2 [44]
600K
2.5K
75
X
Refore
Prioritize
Prioritize
HOT3D [2]
400K
3K
33
X
Optimization
Hand Shape
Wrist Pose
HOI4D [26]
400K
3K
TACO [25]
2.2K
(a) Green dots denote the ground-truth hand joints, while purple dots represent the joint positions obtained by retargeting hand shape and wrist pose using our In- verse MANO Retargeting Network and re- constructing them via the MANO model.
H20 [19]
100K
IK
FPHA [15]
100K
1.3K
26
ARCTIC [12]
250K
IK
12
X
HOLO [39]
1200K
40
Total
3.35M
24K
1.2K
shape, and upon convergence, shift focus to refining the wrist pose. A gating signal a is used to switch between the two loss functions Li and L2, where indicates that Li has converged, and O otherwise. The objective is in Equation (5):
L1 Cahupe 0.3) + ferecon (M))))
C2 W2Cpose (t)), + Frewon (M)NO(d(i)..).
Linv 01 (1-)) L2.
We pretrain the é using 10K paired samples of MANO parameters and joint coordinates covering diverse hand motions, sourced from TACO [25] and OakInk2 [44]. This enables to reuse the Inverse MANO Retargeting Network to annotate datasets that only provide hand joint positions. For more details the pretraining process, please refer to Appendix A.2.
Virtual RGB-D Rendering. Effective training of the depth encoder requires paired RGB-D data captured from the same egocentric viewpoint. However, many existing datasets, such as ARCTIC [12], HOT3D [2], and OakInk2 [44], provide only RGB image sequences without corresponding depth maps. To address this limitation,
design a Virtual RGB-D Renderer to synthesize depth images aligned with the available RGB frames. Given the intrinsic matrix K, extrinsic transformation Tews and object points P.. E R x4 in homogencous world coordinates, we render the depth map D in the camera view by first transforming points to the camera frame: P. = Tow PT them to pixel coordinates via Pue
up then projecting (K (P. @ Ze)), where Ze (P.) and T(-) denotes taking the first two components and rounding to the nearest integers. For each pixel (u, v) puc()) within
the image bounds and with z > 0, the depth map is updated as D(u, u| = min(D(e: u. u|] z!!) to retain the closest surface. We render depth maps for both objects and hands (if visible), capturing accurate hand-object spatial relationships, which are shown to be important Section 5.
By integrating inverse MANO retargeting and virtual RGB-D rendering, we curate unified mul- timodal dataset consisting of 3.35M RGB-D frames, 24K interaction trajectories, covering 1.2K objects. We only consider right hand motion in this submission.
5 Experiments
5.1 Experiment Setups
Datasets. We include 6 training datasets: TACO, FPHA, HOI4D, H20, HOT3D, and OakInk2. Among them, FPHA was re-annotated using the inverse MANO network and is exclusive for evaluation. For the other five datasets, we hold out 10% of the data from each in-domain evaluation sets, ensuring no overlap with the training data in terms of action or object categories. Additionally, to assess generalization unseen domains, evaluate two cross-domain test sets: full ARCTIC dataset and a 10% partition of the HOLO dataset as mentioned in [31].
Metrics. (1) MPJPE [31]: Mean Per Joint Position Error, the average Euclidean distance between predicted and ground-truth 3D hand joint positions over all timesteps. (2) MPJPE-PA [31]: Procrustes Aligned MPJPE, the MPJPE after applying single transformation (scale, rotation, translation) to align the predicted and ground-truth joint trajectories. (3) MPVE: Mean Per Vertex Error, the average Euclidean distance between predicted and ground-truth mesh vertices of the MANO model. (4)

---

Procrustes Aligned MPVE, the MPVE after applying Procrustes alignment to remove global scale and pose differences. (5) MWTE: Mean Wrist Translation Error, the average Euclidean distance between predicted and ground-truth 3D wrist translation vectors. (6) MRE: Mean Rotation Error, the average angular difference between predicted and ground-truth joint rotations for 0 and Given rotation matrices R1 R2 E 16xx3x3 for all 16 joints (1 wrist and 15 finger joints), MRE
MPVE-PA:
is defined MRE 1 16 2j=1 cos ccoss[((taaaae '[(trace( R, R2,) 1)/2], where each R1,j R2, E 33x3 represents the rotation matrix of the j-th joint. MRE provides continuous measure of rotational emorr within the range [0, n] radians.
Baselines. Among existing approaches, object-centric representation methods [6, 8] are not appli- cable in our setting because we do not have access to 3D object models. Similarly, hand-centric representations [35, 41] are unsuitable since the hand is not always visible. Recent work, LatentAct [31], which predicts 3D hand poses and contact maps from a textual action description, a single RGB image, and 3D hand-object contact points, serves as a strong baseline for our task. It does not rely on object models and can operate even when the hand is not visible in the input image. In comparison, MEgoHand takes only text description and visual observation as inputs. To evaluate the effectiveness of our approach, we compare Transformer-bart variant LatentAct-Diff. Furthermore, to analyze different usages of 3D information, we also compare variants LatentAct without contact maps and MEgoHand without depth input.
LatentAct and its diffusion-bass
Modalities, To comprehensively evaluate the effects of incorporating different modalities, we explore several input configurations: (1) MEgoHand- only takes textual descriptions; (2) MEgoHand-1 only takes RGB images; (3) MEgoHand- Hand-ID takes RGB images with depth estimation; (4) MEgoHand-TT takes text and RGB images; (5) MEgoHand (ours) incorporates text, RGB images, and depth maps predicted by a foundation depth estimator.
5.2 Evaluation on In-Domaie Datasets
Table I: Average metrics of in-domain evaluation across 5 datasets: TACO, HOI4D, H2O. HOT3D, and OakInk2. The unit MRE is radians, and the remaining metrics are measured in centimeters.
Method
MPJPE
MPJPE-PAJ
MPVE
MPVE-PAL
MWTE
MREJ
LatentAct
7.726
1.478
7.696
1.453
7.221
0.937
no concat map
8.523
1,481
8.476
1.464
7.813
0.947
LatentAct-Diff
7.819
1.498
7.787
1.483
7.322
0.941
no concat map
8.802
1.582
8.752
1.564
8.051
0.950
MEgoHand-T
8.328
0.477
8.282
0,460
7.637
0.145
MEgoHand-I
6.269
0.480
6.120
0.457
5.521
0.143
MEgoHand-ID
5,969
0.470
5,920
0.453
5.213
0.137
MEgoHand-TI
5,683
0.476
5.632
0.459
4.889
0.136
MEgoHand (ours)
5,425
0.425
5.381
0.409
4.756
0.123
Evaluation against Baselines. shown in Table 1, our method consistently and significantly outperforms the baseline across all evaluation metrics. Notably, it achieves an 86.9% reduction in mean MRE, resulting in an average rotational deviation of approximately degrees. In contrast, LatentAct struggles to generate accurate finger joints, likely due to its reliance on single-view RGB image and a single 3D contact point-conssin 3D hand-object relationship. Our approach addresses these limitations by: (1) leveraging VLMs for richer contextual understanding, (2) integrating 3D depth estimation to better capture hand-object contact features, and (3) employing closed-loop prediction with TOF decoding strategy to ensure temporally consistent and stable forecasting, Notably, after applying global Procrustes alignment, our method achieves further reductions in joint (MPJPE-PA) and mesh vertex (MPVE-PA) emrrrs to 0.424 and 0.409, corresponding to relative improvements of 71 71.2% and 71.9% over LatentAct, respectively. These results demonstrate the superior capability of our approach in modeling hand morphology and fine-grained pose prediction.
that severely limit its ability to model the intricate
Evaluating the Modality Flexibility of Our Model. We analyze the performance of four variants: text instructions alone, RGB images alone, RGB images predicted depth maps, text instructions + RGB images. As shown in the green section of Table 1, text-only inputs produce the weakest results.

---

The absence of visual guidance increases translation error by 61% compared to MEgoHand. RGB- only inputs mitigate this but suffer from ambiguous action patterns, converging to average behaviors due to insufficient 3D spatial cues. MEgoHand-ID resolves these limitations by integrating depth maps, enhancing 3D spatial reasoning, and improving all metries, achieving a 50% lower MPJPE than LatentAct. To investigate the approaches of incorporating 3D information, we evaluate LatentAct without its contact map against our MEgoHand-TT TI variant. Crucially, removing the contact map from LatentAct degrades its performance (10.3% MPJPE increase), while MEgoHand-TT achieves 50% lower error than LatentAct. This indicates that LctentAct critically depends on contact maps, making it less practical for
scenarios.
Evaluation from different datasets. As illustrated in Figure 4, MEgoHand displays a performance hierarchy: it achieves the strongest results on H2O and OAKINK2, with MPJPE deviations con- strained to approximately em, while exhibiting the weakest performance on HOI4D. We analyze that HO14D's 800 object instances across 610 scenes account for its generalization challenge. H2O and OAKINK2 feature structured tasks with consistent interaction regions, such as handles and edges, where stable spatial correlations can be learned to enhance generation.
5.3
Zero-Shot Transfer on Cross-domain Datasets
MEgoHand
MEgoHand-TI
LatentAct
LatentAct no concat map
14
8
4
2
0
H2O HOI4D
HOT3D
OAKINK2
TACO
ARCTIC
HOLO
Figure 4: The evaluation of our two methods and two baseline variants on five in-domain (H20, HOI4D, HOT3D, OAKINK2, TACO) and two cross-domain datasets (ARCTIC, HOLO), using MPJPE as metric (unit: cm, lower is better).
To evaluate our method's generalization capacity, we conduct a comprehensive analysis of its zero- shot transfer performance across two cross-domain datasets, spinning object diversity, task complexity, and scene changes. ARCTIC poses greater challenges through complex dynamic coupling between articulated objects and hand configurations (e.g., scissor-cutting requiring coordinated finger- blade kinematics), exposing limitations from our rigid-object training data. Conversely, HOLO features clearer task segmentation and semantically grounded instructions (e.g., "rotate the screwdriver counterclockwise"). which help narrow the action search space and partially mitigate domain shift. Notably, from Table 2, our method achieves SOTA performance with 33.9% and 29.8% MPJPE improvements over the strongest baselines ARCTIC and HOLO, respectively, demonstrating robust cross-domain transfer capabilities in HOI modeling.
5.4 Ablations around Depth
a) Does a pretrained depth estimator make a difference? From Table 3, MEggHand compatible with various pretrained modern depth estimators like DepthAnythingV2 and UniDepth, achieving comparable results across metrics. This suggests free plug-and-play use of diverse depth estimators.
b) Do we need real depth supervision? Comparative analysis with the deptthsuupervision-abllaee variant, here the depth encoder updates solely through motion prediction loss, reveals significant performance degradation in Table 3. This empirically underscores the insufficiency of final motion loss alone to learn -aware representations, necessitating real depth supervision.
c) Metric depth or relative depth? We also compare accurate metric depth and relative depth inputs for MagoHand. The results vary in Table 3. We observe in in-domain scenarios, metric depth can provide consistent scale and distance cues, enhancing 3D spatial understanding. However, in
8

---

Table 2: Average metrics of out-of-domain evaluation across 2 datasets: ARCTIC and HOLO. The unit for MRE radians, and the remaining metrics are measured in centimeters.
Dataset
Method
MPJPEL
MPJPE-PA
MPVEL
MPVE-PA
MWTEJ
MRE
LatentAct
11.65
1.975
11.58
1.942
9.920
1.577
nо concat map
12.04
2.023
11.96
1.990
10.25
1.590
LatentAct-Dilf
10.98
1.905
10.90
1.870
9.642
1.543
concal map
12.27
2.033
12.19
1.999
10.83
1.559
ARCTIC
MEgoHand-T
10.17
1.318
10.10
1.306
8.872
0.489
MEgoHand-I
8,964
1.218
8.826
1.204
6,985
0.456
MEgoHandIID
8,316
1.161
8.226
1.144
6.689
0,384
MEgoHand TI
8,305
1.173
8.194
1.126
6.313
0,452
MEgoHand (ours)
7.358
1.161
7.268
1.106
5.958
0.398
LatentAct
8.341
8,303
1.606
8.051
1.112
- no concat map
8.682
1.658
8.650
1.635
8,303
1.133
LatentAct-
8.235
1.605
8.196
1.582
7.973
1.101
no concat map
8,492
1.631
8,453
1.609
8.172
1.118
HOLO
MEgoHand-T T
8.605
0.879
8.572
0.860
8.204
0.499
MEgoHand-1
7.525
0.841
7.484
0.819
6.871
0.416
MEgoHand-ID
6.525
6,484
0.790
5.871
0.321
MHgoHand-T1
6.054
0.772
6.011
0.750
5,485
0.298
Mago (ours)
5.775
0.697
5.747
0.673
5.437
0.271
cross-domain scenarios, metric depth is more sensitive to variations in drastic camera parameters alternation, leading performance drop.
Table 3: The ablation studies of MEgoHand variants across evaluation datasets and test datasets.
Dataset
Method
MPJPE
MPJPE-PAJ
MPVEJ
MPVE-PAJ
MWTEJ
MREJ
MEgoHHand
5.425
0,425
5.381
0.409
4.756
0.123
Evaluation
depthanythingy2
5.671
0.475
5.621
0.457
4.895
0.137
depth supervision
5.725
0.492
4,900
0.142
relative depth
5.610
5.564
0.128
MEgoHand
7.358
1.161
7.268
1.106
5.958
0.398
depthanythingy2
8.240
1.220
8.141
1.203
6.287
0,544
ARCTIC
depth supervision
8.174
1.140
1.092
6.608
0.436
relative depth
7.564
6.082
0.473
MEgoHand
5.775
0.697
5.747
0.673
5.437
0.271
depthanything'
6.094
0,895
6.055
0.873
5.512
0,331
HOLO
depth supervision
6.434
5.889
0.473
relative depth
S.879
0.663
5.841
0.643
5.418
0.280
5.5
Visualization
TACO
OAKINK2
ARCTIC
HOLG
Ground
Truth
MEgoHand
LatentAct
Figure We present visualizations across in-domain (green) and cross-domain (blue) datasets. The misalignments of ground-truth annotations attributed to labeling noise and camera calibration errors. For fair comparison with LatentAct, we provide the initial hand pose and align the motion predictions of LatentAct to the first frame in chunk.

---

TACO
OAKINKZ
Ground Truth
MEgoHand
LatentAct
HO14D
Ground Truth
MEgoHand
LatentAct
ARCTIC
HOLO
Ground Truth
MEgoHand
LatentAct
Additional visualizations of LatentAct and MEgoHand. Green part is sampled from training sets. Blue part is sampled from evaluation sets. The Yellow part is sampled from testing sets.
Figure 6:
We decode the generated MANO parameters into hand mesh vertices and visualize the projections overlaid on the original RGB videos. As illustrated in Figure 5, MEgoHand consistently outperforms LatentAct with more accurate hand poses and finer geometric aligmment, particularly in wrist pose and finger joint rotations. We analyze that metric depth inputs play important role in the generation of higher precision. Besides, we observe that no hand is visible in the initial frame, LatentAct struggles predict precise shape parameters B. This emphasizes the significance of initial hand parameters. Please refer to Appendix B.2 for more visualizations.
Conclusion & Limitation
We introduce MEgoHand, multimodal framework for egocentric hand motion generation that integrates initial hand parameters, textual instructions, and RGB images to predict realistic hand-object interaction motion sequences. The hierarchical design combines vision-language model and depth estimation for semantic understanding and 3D reasoning. A DiT-based motion generator conducts closed-loop prediction, enhanced by Temporal Orthogonal Filtering for temporal consistency. To address data scarcity, we curate million-scale HOI dataset by leveraging inverse MANO retargeting and virtual RGB-D rendering. for motion generation, MEgoHand demonstrates strong generalization, achieving SOTA results on five in-domain and two cross-domain datasets.
initial attempt to unify vision language models with reasoning
Some limitations can be addressed in future research. Utilizing our pretrained inverse MANO retargeting network to annotate broader range HOI datasets or adopting modern hand pose detectors [28, 45] to label in-the-wild human videos can further improve data scale, which is promising towards better results.
Limitation.
10

---

Acknowledgements
This work was supported by NSFC under Grant 62450001 and 62476008. The authors would like to thank the reviewers for their valuable comments and advice.
References
[1] Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarin, Vaibhav Srivastav, al. Smollm2: When smol goes big-data-centric training of small language model. arXiv preprint arXiv:2502.02737, 2025.
[2] Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, Jakob Julian Engel, and Tomas Hodan. Hot3d: Hand and object tracking in 3d from egocentric multi-view videos. CVPR, 2025.
[3] Chen Bao, Jiarui Xu, Xiaolong Wang, Abhinav Gupta, and Homanga Bharadhwaj. Hand- sonvlm: Vision-language models for hand-object interaction prediction. arXiv preprint arXiv:2412.13187, 2024.
[4] Johan Bjorck, Fernando Castañeda. Nikita Chermiadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu. Spencer Huang, et al. Gr0Ot nl: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503. 14734, 2025.
[5] Remi Cadene, Simon Alibert, Alexander Soare, Quentin Gallouedec, Adil Zouitine, and Thomas Wolf. Lerobot: State-of thetthrrrtt machine learning for real-world robotics in pytorch, 2024.
[6] Junuk Cha, Jihyeon Kim, Jac Shin Yoon, and Seungryul Back. Text2hoi: Text-guided 3d motion generation for hand-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1577-1585, 2024.
[7] Ning Cheng, Qiqi Dong, Zhen Zhang, Li Wang, Xiaojing Chen, and Cheng Wang. Egocentric processing of items in spines, dendrites, and somas in the retrosplenial cortex. Neuron, 112(4): 646-660 6666660,222 2024.
[8] Sammy Christen, Shreyas Hampali, Fadime Sener, Edoardo Remelli, Tomas Hodan, Eric Sauser, Shugao Ma, and Bugra Tekin. Diffh2o: Diffusion- based synthesis of hand-object interactions from textual descriptions. In SIGGRAPH Asia 2024 Conference Papers, pages 1-11, 2024.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-28 255. Ieee, 2009.
[10] Haoye Dong, Aviral Chharia, Wenbo Gou, Francisco Vicente Carrasco, and Fernando De la Torre. Hamba: Single-view 3d hand reconstruction with graph-guided bi-scanning mamba. Advances Neural Information Processing Systems, 37:2127 2160, 2024.
[11] Enes Duran, Muhammed Kocabas, Vasileios Choutas, Zicong Fan, and Michael J Black. Hmp: Hand motion priors for pose and shape estimation from video. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages
2024.
[12] Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. Arctic: A dataset for dexterous bimanual hand-object manipulation. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[13] Zicong Fan, Maria Parelli, Maria Eleni Kadoglou, Xu Chen, Muhammed Kocabas, Michael Black, and Otmar Hilliges. Hold: Category-agnostic 3d reconstruction of interacting hands and objects from video. In Proceedings of the IEEF//VF Conference on Computer Vision and Pattern Recognition, pages 494-504, 2024.

---

[14] Qichen Fu, Xingyu Liu. Ran Xu, Juan Carlos Niebles, and Kris M Kitani. Deformer: Dy- namic fusion transformer for robust hand pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23600-23611, 2023.
[15] Guillermo
Shanxin Yuan, Seungryul Back, and Tae- Kue-Kyun Kim. First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. In Proceedings of Computer Vision and Puttern Recognition (CVPR), 2018.
[16] Alexey Gavryushin, Florian Redhardt, Gaia Di Lorenzo, Luc Van Gool, Mare Pollefeys, Kaichun Mo, and Xi Wang. Sight: lingle-image conditioned generation of hand trajectories for hand- object interaction. arXiv preprint
2025.
[17] Kristen Grauman, Andrew Westbury, Eugene Byrne Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18995-19012 2022.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[19] Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2o: manipulating objects for first person interaction recognition. Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 10138-10148, October 2021.
Two hands
[20] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Ego-exo: Transferring visual representations from third-person to first-person videos. In CVPR, pages 6943-6953, 2022.
[21] Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang. Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, et al. Eagle 2: Building post-training data strategies from scratch for frontier vision-language models. arXiv preprint WrXiv:2501.148/1 2025.
[22] Nie Lin, Takehiko Ohkawa, Yifei Huang, Mingfang Zhang, Minjie Cai, Ming Li, Ryosuke Furuta, and Yoichi Sato. Simhand: Mining similar hands for large-scale 3d hand pose pre- training. arXiv preprint (.Xi::2502.15251, 2025.
[23] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.0274; 2022.
[24] Qiang Liu. Rectified flow: A marginal preserving approach to optimal transport. arXiv preprint 220.::2709914577, 2022.
[25] Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu, and Li Yi. Taco: Benchmarking generalizable bimanual oool-action-obeect understanding. arXiv preprint arXiv:2401. 08399, 2024.
[26] Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: 4d egocentric dataset for category- hbben- interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 21013-21022, June 2022.
[27] Hao Luo, Bohan Zhou, and Zongqing Lu. Pre-trained visual dynamics representations for efficient policy learning. In European Conference on Computer Vision, pages 249-267. Springer, 2024.
(28) Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9826.9836, 2024.
[29] Luigi Piccinelli, Christos Sakaridis, Yung-Hsu Yang, Mattia Segu, Siyuan Li, Wim Abbeloos, and Lue Van Gool. Unidepthv2: Universal monocular metrie depth estimation made simpler. arXiv preprint arXi:: 2025.
12

---

[30] Chiara Plizzari, Gabriele Goletto, Antonino Furnari, Siddhant Bansal, Francesco Ragusa. Giovanni Maria Farinella, Dima Damen, and Tatiana Tommasi. An outlook into the future of egocentric vision. International Journal of Computer Vision, 132(1 1 1:4880-4936, 2024.
[31] Aayush Prakash, Benjamin Lundell, Dmitry Andreychuk, David Forsyth. Saurabh Gupta, and Harpreet Sawhney. How do do that? synthesizing 3d hand motion and contacts for everyday interactions. arXiv preprint arXiv:2504. 12284, 2025. Preprint.
[32] Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint (rXXi:22011026110 2022.
[33] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop. Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1874-1880 2016.
[34] Soshi Shimada, Franziska Mueller, Jan Bednarik, Bardia Doosti, Bernd Bickel, Danhang Tang, Vladislav Golyanik, Jonathan Taylor, Christian Theobalt, and Thabo Beeler. Macs: Mass conditioned 3d hand and object motion synthesis. In 2024 International Conference on 3D Vision (3DV), pages 1082-1091. IEEE, 2024.
[35] Omid Taheri, Yi Zhou, Dimitrios Tzionas, Yang Zhou, Duygu Ceylan, Soren Pirk, and Michael J Black. Grip: Generating interaction poses using spatial cues and latent consistency. 2024 International Conference on 3D Vision (3DV), pages 933-943. IEEE, 2024.
[36] Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, et al. Towards general computer control: A multimodal agent for red dead redemption ii as a case study. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024.
[37] Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, et al. Cradle: Empowering foundation agents towards general computer control. arXiv preprint ar arXiv:2403.033 03340000.36,,.. 2024.
[38] Michael Tschannen, Alexey Gritsenko, Xiao Wang. Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv: /4786, 2025.
[39] Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, et al. Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 20270-20281, 2023.
[40] Fang Xu, Tianyu Zhou, Hengxu You, and Jing Du. Improving indoor wayfinding with ar-enabled egocentric cues: A comparative study. Advanced Engineering Informatics, 59:102265, 2024.
[41] Yufei Ye, Abhinav Gupta, Kris Kitani, and Shubham Tulsiani. G-hop: Generative hand-object prior for interaction reconstruction and grasp synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[42] Yufei Ye, Yao Feng, Omid Taheri, Haiwen Feng, Shubham Tulsiani, and Michael J Black. Predicting 4d hand trajectory from monocular videos. arXiv preprint arXiv:2501.0833 2025.
[43] Haoqi Yuan, Bohan Zhou, Yuhui Fu, and Zongqing Lu. Cross-embodiment dexterous grasping with reinforcement learning. arXiv preprint 24X.v:2410.02479, 2024.
[44] Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. In Proceedings of the JEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 445-456, 2024.
13

---

[45] Jinglei Zhang, Jiankang Deng, Chao Ma, and Rolandos Alexandros Potamias. Hawor: World- space hand motion reconstruction from egocentric videos. arXiv preprint arXiv:2501.02973. 2025.
[46] Bohan Zhou, Ke Li, Jiechuan Jiang, and Zongqing Lu. Learning from visual observation via offline pretrained state-to-go transformer. Advances Neural Information Processing Systems, 36:59585-59600 2023.
[47] Bohan Zhou, Haoqi Yuan, Yuhui Fu, and Zongqing Lu. Learning diverse bimanual dexterous manipulation skills from human demonstrations. arXiv preprint urXiv:2410.02477, 2024.
[48] Bohan Zhou, Zhongbin Zhang, Jiangxing Wang, and Zongqing Lu. Nolo: Navigate only look once. arXiv preprint arXiv:2408. 01384, 2024.
[49] Keyang Zhou, Bharat Lal Bhatnagar, Jan Eric Lenssen, and Gerard Pons-Moll. Gears: Local geometry-aware hand-object interaction synthesis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 00634-20643,, 2024.
[50] Yi Zhou, Connelly Barns,, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5745-5753, 2019.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction (sce Section 1) accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of method in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We fully disclose all the necessary information required to reproduce the main experimental results. Details regarding model architecture, training settings, and evaluation protocols provided throughout the paper. Please refer to Section 5.1 and Appendix A for implementation details and experimental setups.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
16

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
All codes and documents are included in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
Sogllic/guidess/oeeeSubmmii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. wwtppcccccgcceeecccccccccc wwtlcc/guee///ee httpgeeeoooooedddmiiieeeeeeeeeee wwweeesubmssiiooool...o
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test details are provided in Section 5.1, including data splits and evaluation protocols. The hyper-parameters and their selection are detailed in Appendix
A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed evaluation results in Section 5.2 and Section 5.3, including error bars shown in Figure 4, assess the robustness and statistical reliability of our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix A.6.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htai::::///uuriiieee nnwrrps....pp// cc
ines??
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix D.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We introduce the original datasets in Appendix A.5.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
All datasets and codebases used in this study is open-sourced.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy fttpp://neerip..tt httpr/pps..c/ooteeeeccee////2222 for what should or should not be described.
21

---

Appendices
A Implementation Details
A.1 Hyperparameters
We report important hyperparameters used for MEgoHand training in Table 1.
Table 1: Hyperparameters of MEgoHand Training.
Hyperparameter
Value
Prediction Trunk Size I
16
Integration Step Size 6
0.1
Gradient steps
50,000
Batch size
64
Learning Rate
3e-4
Optimizer
AdamW
Adam 31
0.95
Adam 32
0.999
Adam
1e-8
LR scheduler
cosine
Weight Decay
le-5
Warmup Ratio
0.05
VLM text tokenizer
frozen
VLM vision encoder
unfrozen
DiT
unfrozen
A.2 Inverse MANO Pretraining
Architecture. The model architecture of the Inverse MANO Retargeting Network consists of PointNet encoder with 3-layer MLPs.
Training Parameters. We set w = 4.0 and we 5.0 for Li and Le respectively. Lahape and Lrecom both L1 loss, supervising shape feature B, translation rotation in 6D representation 0,
Visualization. Figure shows using Inverse MANO Retargeting Network & to label FPHA dataset.
RTS
TEF
We forward the MANO model convert the outputs of Inverse MANO Retargeting Network @ to hand meshes, which are projected to the original frames in FPHA with the help of camera intrinsics and extrinsics.
Figure 1:
22

---

A.3 Flow Matching
Recent work tigh-resolution image and video synthesis has shown that flow matching can achieve linear-Gaussinn Gaussian (or optimal transport)
strong empirical performance when combined with simple linear-
probability path, given by:
af (HHTHH) = (a, (1 -1-T) I))
In practice, the network is trained by sampling random noise e ~ (N,,I), computing the "noisy actions" 7 7H2 (1 - 7)c, and then training the network outputs vo (HG h&s 2 22 match the denoising vector field:
u u(HTHH))
Hk-
The action expert uses a full bidirectional attention mask, so that all action tokens attend to each other During training, we sample the flow matching timestep J from a beta distribution that emphasizes lower (noisier) timesteps. At inference time, we generate actions by integrating the learned vector field from to t 1, starting with random noise H. (O, We use the forward Euler integration rule:
Hg = HT + Hoo,H, hy, h DT
where 5 is the integration step size, We use 10 integration steps (corresponding to = 0.1) in our experiments. Note that inference can be implemented efficiently by caching the attention keys and values for the prefix h2s 2k each integration step.
TDDI and only recomputing the suffix corresponding to the hand motion for
A.4 Vision Language Model
Visual inputs are resized to 224 x 224 and encoded by SigLIP-2 with pixel shuffle [33], producing 64 spatially-aware visual tokens per frame, denoted as -², In parallel, textual instructions are processed by SmoILM2 to extract semantic representations z², facilitating cross cross-modal alignment.
A.5
HOI Datasets
Resources. We utilize variety of publicly available egocentric hand-object interaction datasets in our experiments. Below a brief description of each dataset along with its official website for reference:
H20: A large-scale egocentric dataset featuring hand-ohject interactions with both RGB and depth modalities. https: http:::///aeiiwooo/ com/ com/proj acts/h2o/
HOI4D: A dataset of hojan-object interactions, capturing fine-grained manipulation across various tasks. https: hithhhhii...gghhubbboo.. io/
HOT3D: A dataset for hand-object tracking and manipulation with accurate annotations.
gi thub. 1 ///tttd////
OAKINK2: A comprehensive benchmark for large-scale egocentric manipulation with articulated object models. https: / /ww.kkkknkkkk// net/v2/
TACA: A task-oriented dataset for contact-aware human-ohject interaction analysis. https: //taco2024. gi thub. .oo/
ARCTIC: A richly annotated dataset for tracking hand-object contact and motion in ego- centric scenarios. https: bttps:://aree //acctic. is tue .pg..pg.................. de/
HOLO: large-scale dataset of household manipulation tasks captured in real-world environments. https: http://hholass...cooooo gi thub. 10 Ao/#HoloAssist
Format. Our training corpora are built upon the LeRobot [5] dataset format, a widely used standard
in the open-source robotics community and interaction learning community. Developed by Hugging Face, LeRobot is designed to make it easier to work with demonstration-Be based learning by offering unified structure for storing, sharing, and utilizing demonstration data. popularity stems from its adaptability and the rich ecosystem of pretrained models and datasets available on the Hugging Face hub. The LeRobot format combines several well-established file types ensure efficient storage and accessibility:
23

---

States, actions, and metadata are stored in Parquet files, which provide compact columnar storage and rapid access. This structure supports fast filtering and slicing slicing-criticall critical for training modern machine learning models.
Tabular Data:
Visual Data: Observations in the form of videos (MP4) or image sequences (PNG) are referenced in the Parquet files, significantly reducing storage requirements while preserving accessibility.
Metadata:
Supplementary information such as dataset statistics and episode indexing is stored in JSON format, allowing structured, machine-readable access dataset characteris- tics.
Demonstration sequences are organized into episodes, where each frame captures synchronized observations and corresponding actions. Observations typically include visual inputs (e.g.. observation. images. *) and internal states (e.g., observation. state), while actions encode control directives. This episodic structure supports a wide range of learning paradigms. For imitation learning, the data enables supervised prediction of actions from observations. For reinforcement learning, facilitates evaluation and optimization of decision-making strategies under varied state- action contexts. This standardized data format not only enhances reproducibility and interoperability across learning systems but also lowers the barrier to entry for researchers by providing clean interface to high-quality interaction datasets.
While the LeRobot format provides a solid foundation, our work introduces several extensions to accommodate richer modality integration. We augment the standard format with the following components:
Modality Configuration: A modality. json file is introduced within the meta directory to explicitly define the structure of the initial state and action vectors. This configuration maps each vector component to its semantic meaning and includes additional metadata relevant to each modality.
Fine-Grained Semantic Decomposition: Departing from the monolithic vector approach of the original format, we decompose both state (initial hand state) and action (future hand motion trunk) vectors into semantically interpretable components-such as 0, B, r, and t-each annotated with its own data type, valid range, and transformation rules.
Multi-Annotation Integration: The dataset format is extended to support multiple forms of annotations, such as task descriptions, validity indicators, and success labels. These annotations follow the LeRobot practice of storing indices in the Parquet files, with the corresponding content stored in auxiliary JSON files.
Rotation Representation Specification: To ensure correct processing of rotational com- ponents during training, we require explicit declaration of the rotation representation used (e.g., quaternion, Fuler angles, or axis-angle) for each relevant field.
These enhancements collectively enable more structured learning from complex demonstration data, with explicit modality definitions and robust support for multimodal supervision.
Preprocessing. For FPHA, we pretrain Inverse MANO Retargeting Network to label MANO parameters. For ARCTIC, HOT3D and OAKINK2, we adopt virtual RGB-D rendering to produce high-quality metric depth images in advance. All RGB and depth images are resized to 256 x 256. is worth noting that split longer sequences short clips (<500 steps) with the same task instruction for training and testing.
A.6 Computation
Resources.
MEgoHand is trained using 8x80GB NVIDIA A800 GPUs over approximately 24 hours. All evaluations and visualizations are performed on a single 80GB A800 GPU for around three hours.
Efficiency. We evaluated the end-to-end inference performance for generating 16-frame sequence on a single RTX 4090 GPU. MEgoHand over 2x faster and uses nearly 50% less VRAM than the strong LatentAct baseline. This superior efficiency is direct result of our novel architectural design, which eliminates the need for the expensive online contact map computation. Instead, MEgo leverages a lightweight, pre-trained depth estimation module that implicitly captures geometric cues
24

---

while operating at fraction of the computational cost. This design not only reduces inference latency and memory footprint but also enhances robustness across diverse interaction scenarios, making well-suited for real-time AR/VR and robotic applications.
Table 2: End-to-end inference performance efficiency for generating a 16-frame sequence on a single RTX 4090 GPU.
Method
Inference Time!
FPS
VRAM Usage!
LatentAct
156ms
6.4
10.8GB.
MEgoHand (Ours)
74ms
13.5
5.8GB
A.7 Smooth Decoding
t=0
t=1
t=2
t=3
t=4
t=5 t=6
Figure 2: Illustration for smoothing predicted transformations.
Decoding Strategy. As illustrated in Figure 2, 1 0 MEgoHand receives initial hand MANO parameters, a egocentric RGB observation, and a depth map to predict trunk t = 1. The predicted wrist pose is relative to the initial hand pose and the predicted 3 is repeated from initial B. Then at t= I. similarly, the predicted wrist pose - + is relative to the wrist pose predicted at and so on. After converting all relative transformations to absolute transformations, we average all predictions at the same timestep to get smoother transformations.
Visualization. From Figure 3 we can see that smooth decoding stategy is effective in mitigating jitter.
t=60
t=80
t=100
MEgoHand without Smooth Decoding
MEgoHand
Figure 3: Frames randomly sampled from task "Stir the bowl with spatula" of TACO. Without decoding strategy, the predicted trajectory exhibits more fluctuations.
25

---

B Additional Visualizations
B.1 Zero-Shot Depth Estimation & Virtual Depth Rendering
In Figure 4, visualize the zero-shot depth estimation of Vni [29] and the virtual depth rendered from object models. Three datasets (OAKINK2.HOTSDAARCTT)) are involved, as there real depth frames in these datasets.
AGB
OAKINK2
AGB
Depth Rensering
HOT3D
Zero Shot Depth Extimation
Witual Deptit Rendaring
ARCTIC
Figure 4: Colorbars indicate the absolute depth values (unit: m). The depth values of all depth frames fall within [0, 2].
B.2 HOT hand motion Generation
We visualize more clips of policy inference in Figures 6 and 5. MEgoHand is superior to baseline LatentAct in most cases.
C Empirical Results
We report the average metrics of MEgoHand in each dataset in Table 3.
Table 3: Average metrics across evaluation (TACO, HOI4D, H2O, HOT3D, OakInk2) and testing datasets (ARCTIC, HOLO). The unit for MRE is radians; the remaining metries measured in centimeters.
Dataset
MPJPE
MPJPE-PA
MPVE
MPVE-PA
MWTE
MRE
H2O
3.013
0.352
2.969
0.334
2.450
0.099
HOI4D
8.958
0.856
8.933
0.826
8.462
0.213
HOT3D
6.437
0.236
6.352
0.228
5.045
0.086
OAKINK2
3.424
0.217
3.380
0.205
2.837
0.071
TACO
4.936
0.358
4.899
0.346
4.465
0.131
ARCTIC
7.358
1.161
7.268
1.106
5.958
0.398
HOLO
5.775
0.697
5.747
0.673
5.437
0.271
26

---

TACO
OAKINK2
Ground Truth
MEgoHand
LatentAct
H20
HOT3D
Ground Truth
MEgoHand
LatentAct
ARCTIC
HOLO
Ground Truth
MEgoHand
LatentAct
Figure 5: Additional visualizations of LatentAct and MEgoHand. Green part is sampled from training sets, Blue part is sampled from evaluation sets. The Yellow part sampled from testing sets.
D Social Impact
MEgoHand forwards an important step toward universal hand-object motion generation from multiple modalities including task instruction, RGB observation, depth image, and initial conditions. There
are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
27


Reviews:
[{"review_id": "CgRBsD95zB", "reviewer": "Reviewer_mXQm", "summary": "This paper proposed MEgoHand, a multi-modal framework for egocentric hand-object interaction synthesis. Text, RGB, and depth map are taken as input conditions. VLM is used to improve the generalization of MEgoHand. MEgoHand shows its effectiveness by outperforming baselines on five in-domain and two cross-domain benchmarks.", "questions": "1. To evaluate the zero-shot transfer and cross-domain generalization, ARCTIC and HOLO are used. Although the authors explain the differences of ARCTIC, HOLO, and the training datasets, we don't have a clear understanding of the motion distribtion of these datasets. For example, what the object categories, are there any overlap? What's the action categories and anhy overlap there? Without thoses statistics, we can't evaluate the generalization of the model.\n2. \"physically plausible\" is claimed in the abstract, but I don't see any physics involved in the methodology.\n3. The predictions from the model seems not well-aligned with the model. I guess this is from the VLM model, which is dificult to preserve the pixel-aligned information. Existing pure vision-based hand motion estimation methods can estiamte very accurate hand motions from videos. Since RGB videos always used in the evaluation settings, comparisons with vision-based estimation methods are necessary.\n4. MEgoHand is call a \"Motion Generation\" method. But with RGB image, depth, it is more like an estimation method for me. The capability of pure generation is not shown in this paper. For example, is the model able to generate interaction, even unseen interactions since there is a VLM involved, given only a text description?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "+ Contribute a multi-modal model with VLM for ego-centric hand-objection interaction.\n+ The architecture with a high-level “cerebrum” is developed to leverage VLM.\n+ Contribute a large human-objection interaction dataset, with 3.35M RGB-D frames, 24K interactions, and 1.2K objects.\n\n- Lack of conprehensive evaluation of generalization to unseen task and objects.\n- The prediction is not well-aligned with the image.", "was_revised": true, "final_justification": "The authors' detailed response resolved my concerns. I updated my rating to borderline accept.", "rebuttal": "We sincerely thank the reviewer for the detailed comments and insightful review. We are willing to provide a point-by-point response to address your concern below.\n\n---\n\n## **Q1: Unseen Task and Objects Evaluation**\nWe evaluate our method on the full ARCTIC dataset and a 10% stratified HOLO subset. Below we present comparative statistics between the aggregated training dataset (TACO, FPHA, HOI4D, H2O, HOT3D, OakInk2) and OOD dataset ARCTIC and HOLO.\n\n| Feature | **Aggregated Training Datasets** | **ARCTIC** | **HOLO** |\n| :--- | :--- | :--- | :--- |\n| **Object Categories** | **Rigid Objects** *(e.g., containers: bowl, tools: spatula)*  | **Articulated Objects** *(e.g., scissors, laptop)* | **Assembly Components** *(e.g., furniture parts)* |\n| **Task Categories** | **Atomic & Direct Manipulation**      *(e.g., pour a ball into a bowl)* | **Continuous & Coordinated Control**      *(e.g., cutting with scissors, typing)* | **Long-Horizon & Instruction-Driven**      *(e.g., assembling furniture via verbal guidance)* |\n| **Task Complexity** | **Short-Horizon & Single-Goal** | **Mid-Horizon & Physically Coupled** | **Long-Horizon, Multi-Step & Contextual** |\n| **Object Overlap with Training Set** | N/A | **Minimal.** *Although object category like 'laptop' exists in HOI4D, the interaction mode is fundamentally different.* | **Minimal.** *While electronics (GoPro, DSLR) overlap by category, they are situated within a novel procedural task context.* |\n| **Task Overlap with Training Set** | N/A | **None.** *ARCTIC's actions are continuous processes requiring fine-grained, coordinated finger control, while the training set's open/close actions are discrete state changes.* | **None.** *Training tasks are self-initiated and atomic. HOLO introduces a hierarchical task structure driven by external, semantic guidance.* |\n| **Key Generalization Challenges** | N/A | **Physics Generalization.**      *A method demands understanding the dynamic coupling between fine-grained hand motion and the internal mechanics of articulated objects.* | **Task & Language Generalization.**      *A method demands understanding long-horizon, goal-oriented tasks grounded in real-time natural language instructions.* |\n\nBriefly, the OOD datasets represent two key generalization challenges:\n- **ARCTIC:** Rigid → Articulated object interactions\n- **HOLO:** Atomic actions + objects → Procedural, language-grounded tasks\n\nGiven the absence of scene and task overlap between the OOD datasets and training sets, we conduct a focused analysis of MEgoHand's generalization capability on subsets containing entirely novel objects. Statistical evaluation reveals that over 80% of test samples involve objects unseen during training. We compare LatentAct and MEgoHand across these unseen categories (see table below), reporting all metrics in centimeters (lower values indicate superior performance). MEgoHand achieves consistent error reductions of approximately 30% in translation and 70% in orientation relative to LatentAct, demonstrating significantly improved pose estimation accuracy and motion fidelity.\n\n\n| Benchmark | Method | Unseen Object Proportion | MPJPE↓ | MPJPE-PA↓ | MPVE↓ | MPVE-PA↓ | MWTE↓ | MRE↓ |\n|-----------|--------|:----------------------:|:------:|----------:|:-----:|:--------:|:-----:|:----:|\n| ARCTIC | LatentAct | 10 / 12 (83.3%) | 12.156 | 2.075 | 12.084 | 2.042 | 10.420 | 1.677 |\n| ARCTIC | MEgoHand | 10 / 12 (83.3%) | 8.358 | 1.361 | 8.268 | 1.306 | 6.958 | 0.498 |\n| 10% HOLO | LatentAct | 99 / 120 (82.5%) | 8.841 | 1.729 | 8.803 | 1.706 | 8.551 | 1.212 |\n| 10% HOLO | MEgoHand | 99 / 120 (82.5%) | 6.275 | 0.797 | 6.247 | 0.773 | 5.937 | 0.371 |\n\n---\n\n## **Q2: Physical Evaluation**\nThank you for pointing out the potential ambiguity of the term \"physically plausible.\" We wish to clarify that we use this term within the context of data-driven generation, not physics-based simulation. We define 'physically plausible' motion as satisfying two key properties learned implicitly from real-world data: (a.) **Kinematic realism**, where generated poses adhere to MANO's biomechanical constraints (e.g., no backward-bending fingers), and (b.) **Interaction coherence**, ensuring motion trajectories and contact patterns (e.g., grasps) align with observed human interactions. Crucially, these constraints emerge from data-driven learning rather than physics simulation. To improve precision, we will replace 'physically plausible' with 'kinematically realistic and temporally coherent' in the latest version of MEgoHand.\n\nTo quantitatively measure physical quality, two metrics are adopted:\n1. **Contact Accuracy (CA, %):** CA = $\\frac{1}{|V_H|}\\sum_{v \\in V_H} \\mathbf{1}(\\min_{o \\in P_O} \\|v - o\\| \\leq \\delta)$, where $V_H$ is the set of hand-mesh vertices, $P_O$ is the object point cloud, $\\delta=2 cm$. It measures the percentage of hand-mesh vertices that are correctly positioned within a small distance threshold (δ=2cm) of the object surface. **A higher CA is better.**\n   \n2. **Penetration Depth (PD, cm):** PD = $\\frac{1}{|V_H^{in}|}\\sum_{v \\in V_H^{in}} |d(v)|$, $d(v) = \\min_{f \\in M_O} \\text{sdist}(v,f)$, where $V_H^{in} = \\{v : d(v) < 0\\}$, $M_O$ the object mesh, and $\\text{sdist}(v,f)$ the signed distance from vertex $v$ to mesh face $f$. It calculates the average depth of penetration for all hand vertices that are illegally inside the object mesh. **A lower PD is better.**\n\nWe evaluated MEgoHand against the strong LatentAct baseline on both an in-domain (ARCTIC) and a challenging cross-domain (HOLO) dataset.\n\n| Method | Dataset | **Contact Accuracy (CA) ↑** | **Penetration Depth (PD) ↓** |\n| :--- | :--- | :--- | :--- |\n| LatentAct | ARCTIC | 68.3% | 4.2 cm |\n| **MEgoHand (Ours)** | ARCTIC | **82.5% (+20.8% rel.)** | **2.3 cm (-45.2% rel.)** |\n| LatentAct | HOLO | 65.1% | 5.8 cm |\n| **MEgoHand (Ours)** | HOLO | **78.4% (+20.4% rel.)** | **2.7 cm (-53.4% rel.)** |\n\nThe results provide powerful, quantitative proof of MEgoHand's superior physical plausibility. Our method generates motions with **substantially higher contact accuracy** and **less than half the object penetration depth** compared to the baseline. \n\n---\n\n\n## **Q3: Image Alignment**\n\n\nThe reviewer's intuition is correct: a standard Vision-Language Model, on its own, would indeed struggle with precise spatial alignment.  However, this is not a weakness of our work; on the contrary, it is the core technical challenge that our novel bi-level architecture was explicitly designed to overcome.\n\nTo address the inherent lack of spatial acuity in traditional VLMs, our key innovation is to **integrate a metric depth estimator directly into our high-level \"cerebrum.\"** This synergy is the central mechanism of our model: the VLM provides the high-level semantic intent (*what* to do), while the depth module provides the precise, object-agnostic 3D geometric grounding (*where* to do it).\n\nRegarding the minor visual discrepancies in our visualizations, we must clarify that these artifacts often originate not from our model's predictions, but from **inherent noise in the ground-truth annotations of the datasets themselves.**\n*   As noted in our **Figure 5 caption**, and further evidenced in the \"Ground Truth\" rows of **Figures 5–6 in the Appendix**, the provided ground-truth hand poses and camera calibrations are not always perfectly aligned with the RGB video.\n*   Therefore, the most rigorous way to evaluate performance is through **relative error metrics**. On this fair and standard basis, our quantitative results are unequivocal: **MEgoHand significantly outperforms LatentAct**, demonstrating a superior ability to generate motions.\n\n---\n\n\n## **Q4: Motion Generation rather than Estimation**\n\nThis critique contains a fundamental misunderstanding of our task's nature and misinterprets the very contribution of our architecture. Comparing our method to \"vision-based estimation methods\" is fundamentally inappropriate, as it overlooks the essential difference in task formulation and objective.\n\n*   **Pose Estimation is Reconstruction, Not Generation:** Pose estimation methods are given the full video sequence as input to reconstruct poses for each frame. They solve a reconstructive problem with full access to visual information at every timestep.\n*   **Our Task is Future Motion Generation:** As defined by Equation (1), MEgoHand performs a significantly harder and more practical task: generative forecasting. Given only an initial state at time `t` (an instruction, RGB frame, and MANO state), it must generate a sequence of future motions for times `t+1` through `t+l` with zero access to any future visual input. This is a predictive problem.\n\n---\n\n## **Q5: Text-Only Generation**\nIn Section 2.1, we investigated that current text-only generation methods like DiffH2O [1] and Text2HOI [2] rely on additional object-specific inputs (i.e. object mesh) which are unavailable in MEgoHand. Notably, the reviewer asks whether our model can generate motion from *only* a text description. The answer is clearly **yes**, and this capability is both demonstrated and quantified in our ablation studies.\n\n* As shown in **Table 1 (Page 7)**, the `MEgoHand-T` variant performs exactly this: it generates full 3D hand motion sequences **conditioned solely on text**, without any visual input.\n* The strong performance and semantic coherence of `MEgoHand-T` provide direct evidence of our model’s ability to perform pure generation, validating its strength as a true generative framework.\n\n\n---\n**References:**\n\n[1] DiffH2O: Diffusion-based synthesis of hand-object interactions from textual descriptions. SIGGRAPH 2024.\n\n[2] Text2HOI: Text-guided 3D motion generation for hand-object interaction. CVPR 2024.\n\n---\n\nOnce again, thank you for your careful review. If most of your concerns are addressed, would you please consider increasing the recommendation of the paper?"}, {"review_id": "5mmap1iQSg", "reviewer": "Reviewer_5Uyw", "summary": "MEgoHand is a multi-modal framework for egocentric hand-object motion generation from text, RGB image, and initial hand pose. It uses a bi-level architecture: a high-level vision-language model infers motion priors and depth, while a low-level DiT-based policy generates fine-grained trajectories with temporal filtering. A standardized preprocessing pipeline ensures data consistency across modalities. MEgoHand achieves state-of-the-art results on 7 datasets, significantly reducing translation and rotation errors, and demonstrates strong generalization.", "questions": "Please see weakness", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### **Strength**\n- Overall, the paper is well-structured and easy to follow.\n- The use of a vision-language model to reason about affordances and task intent, coupled with a diffusion-based low-level generator, is effective.\n- The method demonstrates strong performance across five in-domain and two cross-domain datasets, with ablation studies on multimodal components.\n- The effort to standardize and unify multiple HOI datasets, producing a 3.35M-frame multimodal dataset, is a substantial and impactful contribution to the community.\n\n### **Weakness**\n- Although the paper introduces the Temporal Orthogonal Filtering (TOF) strategy, it lacks comparisons with existing smoothing techniques such as Kalman filtering or Gaussian smoothing. Without ablation studies or baseline comparisons, the necessity and effectiveness of TOF remain insufficiently justified.\n- There is no evaluation metric provided to quantify contact accuracy, which is crucial for assessing the physical plausibility of the generated hand-object interactions.\n- The appendix section appears to be half-finished.", "was_revised": true, "final_justification": "The authors have provided extensive additional experiments in the rebuttal, which have addressed most of my concerns. I am inclined to recommend acceptance.", "rebuttal": "We sincerely thank your careful review and valuable feedback. We are particularly honored by your recognition of our data curation efforts and architectural design. Your insightful critiques regarding the technical justifications are exactly the kind of guidance that helps us strengthen the rigor and clarity of our work. In response, we have carefully addressed each of your comments with detailed, data-supported revisions, as outlined below.\n\n---\n\n## **Q1: On Justifying Temporal Orthogonal Filtering (TOF) vs. Baselines**\n\n**Our Response:**\nThis is an excellent and critical point. We have now conducted the requested comparison to demonstrate TOF is effective in our hand motion generation task. Standard methods like Gaussian or Kalman filtering are suboptimal for two key reasons:\n\n*  **Geometric Invalidity:** These filters operate on vectors. Applying them naively to rotation representations (e.g., 6D vectors or quaternions) and averaging them does not guarantee the result remains on the valid SO(3) manifold. This can lead to geometrically invalid, distorted rotations that break the kinematic chain.\n*  **Dynamic Mismatch:** These filters often assume linear dynamics (Kalman) or simple Gaussian noise, which is a poor match for the highly non-linear, bursty, and aperiodic nature of human hand-object interactions. A hand might move slowly and then suddenly accelerate, a pattern these filters struggle to model without aggressive, accuracy-damaging smoothing.\n\nOur proposed Temporal Orthogonal Filtering (TOF) is architecturally superior because it is a **geometrically-aware decoding strategy**. It operates directly on rotation matrices and, crucially, incorporates an SVD-based projection that explicitly ensures the final, smoothed rotation is always a valid point on the SO(3) manifold. This maintains geometric integrity and produces kinematically sound results.\n\nTo provide empirical proof, we compared TOF against Gaussian Smoothing and a well-tuned Kalman Filter on the challenging ARCTIC dataset.\n\n| Smoothing Method                   | MPJPE↓ | MPJPE-PA↓ | MPVE↓ | MPVE-PA↓ | MWTE↓ | MRE↓ |\n|------------------------------------|:------:|:---------:|:-----:|:--------:|:-----:|:----:|\n| Gaussian Smoothing                 | 7.85   | 1.28      | 7.82  | 1.16     | 6.92  | 0.47 |\n| Kalman Filtering                   | 7.38   | 1.16      | 7.45  | 1.12     | 6.08  | 0.41 |\n| **Temporal Orthogonal Filtering (Ours)** | **7.36**   | **1.16**     | **7.37**  | **1.11**     | **5.96**  | **0.40** |\n\nThe results clearly highlight a critical trade-off: traditional filters improve temporal smoothness at the expense of accuracy. Both Gaussian and Kalman filters introduce artifacts that compromise the geometric fidelity of the poses. In contrast, TOF effectively enhances temporal stability and suppresses jitter while preserving the geometric precision of the original predictions.\n\n---\n\n\n## **Q2: On Quantifying Contact Accuracy and Physical Plausibility**\n\n\n**Our Response:**\nThis is a critical and highly insightful suggestion. We agree completely that quantifying contact is essential for a rigorous assessment of physical plausibility, and we thank you for pushing us to strengthen our evaluation in this regard.\n\n*   **The Potential Ambiguity of \"Physically Plausible.\":** We define 'physically plausible' motion as satisfying two key properties learned implicitly from real-world data: (a.) **Kinematic realism**, where generated poses adhere to MANO's biomechanical constraints (e.g., no backward-bending fingers), and (b.) **Interaction coherence**, ensuring motion trajectories and contact patterns (e.g., grasps) align with observed human interactions. Crucially, these constraints emerge from data-driven learning rather than physics simulation. To improve precision, we will replace 'physically plausible' with 'kinematically realistic and temporally coherent' in the latest version of MEgoHand.\n\nImportantly, to directly address your point, we have designed and implemented two new metrics to explicitly measure the quality of the physical interaction between the generated hand and the object:\n\n\nTo quantitatively measure physical quality, two metrics are adopted:\n1. **Contact Accuracy (CA, %):** CA = $\\frac{1}{|V_H|}\\sum_{v \\in V_H} \\mathbf{1}(\\min_{o \\in P_O} \\|v - o\\| \\leq \\delta)$, where $V_H$ is the set of hand-mesh vertices, $P_O$ is the object point cloud, $\\delta=2 cm$. It measures the percentage of hand-mesh vertices that are correctly positioned within a small distance threshold (δ=2cm) of the object surface. It quantifies the model's ability to generate plausible contact patterns. **A higher CA is better.**\n   \n2. **Penetration Depth (PD, cm):** PD = $\\frac{1}{|V_H^{in}|}\\sum_{v \\in V_H^{in}} |d(v)|$, $d(v) = \\min_{f \\in M_O} \\text{sdist}(v,f)$, where $V_H^{in} = \\{v : d(v) < 0\\}$, $M_O$ the object mesh, and $\\text{sdist}(v,f)$ the signed distance from vertex $v$ to mesh face $f$. It calculates the average depth of penetration for all hand vertices that are illegally inside the object mesh. It directly measures the severity of non-physical collisions. **A lower PD is better.**\n\nWe evaluated MEgoHand against the strong LatentAct baseline on both an in-domain (ARCTIC) and a challenging cross-domain (HOLO) dataset. The results, which highlight the relative improvement of our method, are presented below.\n\n| Method | Dataset | **Contact Accuracy (CA) ↑** | **Penetration Depth (PD) ↓** |\n| :--- | :--- | :--- | :--- |\n| LatentAct | ARCTIC | 68.3% | 4.2 cm |\n| **MEgoHand (Ours)** | ARCTIC | **82.5% (+20.8% rel.)** | **2.3 cm (-45.2% rel.)** |\n| LatentAct | HOLO | 65.1% | 5.8 cm |\n| **MEgoHand (Ours)** | HOLO | **78.4% (+20.4% rel.)** | **2.7 cm (-53.4% rel.)** |\n\nThe results provide powerful, quantitative proof of MEgoHand's superior physical plausibility. Our method generates motions with **substantially higher contact accuracy** and **less than half the object penetration depth** compared to the baseline, across all settings. This directly validates that our bi-level architecture, particularly the synergy of the VLM and depth estimator, learns a much more accurate and physically coherent model of hand-object interaction.\n\nWe will add the formal definitions of these new metrics and this compelling results table to our experiments section in the final manuscript.\n\n---\n\n## **Q3: On the Supplementary Materials**\n\n**Our Response:**\nWe sincerely apologize for this unacceptable oversight. The incomplete appendix in our initial submission is not at all reflective of the high standards we hold for our work. We thank you for pointing it out.\n\nIn response to your valuable feedback and that of other reviewers, we have not only completed but also **significantly expanded** the appendix to provide a much richer and more comprehensive set of supporting analyses. The now complete and polished Appendix includes:\n\n*   **Detailed Ablation Studies:**\n    *   **Sec [A.X]: Justification for Temporal Orthogonal Filtering (TOF):** Includes the new comparative results against baselines and a discussion on manifold geometry.\n    *   **Sec [A.Y]: Analysis of Prediction Trunk Size `l`:** Provides the empirical data justifying our choice of `l=16`.\n\n*   **New Quantitative Evaluations:**\n    *   **Sec [B.X]: Contact Plausibility Metrics:** Formal definitions and results for our new **Contact Accuracy (CA)** and **Penetration Depth (PD)** metrics.\n    *   **Sec [B.Y]: Computational Latency and Efficiency:** A detailed performance benchmark, including FPS and VRAM usage.\n    *   **Sec [B.Z]: Quantitative Analysis of Dataset Distribution:** A new table detailing the minimal object and action overlap between our training and test sets to rigorously support our generalization claims.\n\n*   **Expanded Qualitative Results:**\n    *   **Sec [C.X]: Visual Demonstrations:** Additional high-quality visualizations and a link to supplementary videos that further showcase the fluidity and accuracy of our generated motions.\n\nWe are confident that this greatly expanded appendix now provides the necessary depth and detail to fully support the claims made in the main paper. We assure you that the final camera-ready version will be accompanied by this comprehensive and thoroughly proofread supplementary material.\n\n\n\n---\n\n\nOnce again, thank you for your careful and highly constructive review. Your feedback has been instrumental in helping us strengthen the paper's technical rigor. We believe that with the addition of the new TOF comparison, the new contact plausibility metrics, and a completed appendix, we have fully addressed your concerns and significantly improved the paper. If most of your concerns are addressed, would you please consider increasing the recommendation of the paper?"}, {"review_id": "KUZMNdohPn", "reviewer": "Reviewer_6C5K", "summary": "The paper introduces a two-step system that converts a single RGB frame, text instruction, and MANO hand pose into a full interaction sequence. For the first (cerebrum) part, the authors use a VLM with a monocular encoder to infer semantics of the scene. The second (cerebellum) part then uses a DiT-based flow-matching policy with a temporal filter applied at test time to keep the motion stable.", "questions": "— What is the end-to-end inference speed on a single GPU, and how does that compare with LatentAct?\n\n— I request the authors fix the poorly written sentence in the abstract: “Existing methods rely on predefined 3D object priors, limiting generalization to novel objects, which restricts their generalizability to novel objects”.\n\n— Is there any attempt to improve cross-domain generalization? It seems performance drops on MPJPE.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n— The two-stage design separating high-level reasoning from low-level components is novel. Also the addition of the monocular metric depth estimation module to the VLM is new to my knowledge.\n\n— The large dataset (built via inverse-MANO retargeting and virtual depth rendering) is valuable on its own.\n\n— The paper shows pretty convincing gains over LatentAct in both position and rotation metrics.\n\n**Weaknesses:**\n\n— There’s no report of inference speed or memory (would like to see it given added depth prediction and flow-matching steps).\n\n— There’s no failure analysis. I would like to see what happens when there are depth errors or occlusions.", "was_revised": true, "final_justification": "The paper is overall sound but the proposed TOF's advantage over Kalman fitler is still not supported by quantitative evidence. I lean towards weak accept.", "rebuttal": "We are sincerely grateful for your positive and insightful review. We are particularly encouraged that you recognized the novelty and value of our core contributions, including the \"well-motivated design of separating high-level reasoning from low-level components,\" the \"addition of the monocular metric depth estimation module,\" and our large-scale HOI dataset.\n\nYour feedback is excellent, and we have conducted the requested analyses to address your concerns. We are pleased to provide a point-by-point response below.\n\n---\n\n## **Q1: On Inference Speed and Memory**\n\n**Our Response:**\nThis is a critical point for evaluating the practical viability of our method, and we thank you for prompting us to provide these benchmarks. We have now measured the end-to-end inference performance for generating a 16-frame sequence on a single RTX 4090 GPU.\n\n| Method | **Inference Time↓** | **FPS ↑** | **VRAM Usage↓** | **Key Architectural Advantage** |\n| :--- | :--- | :--- | :--- | :--- |\n| LatentAct | 156ms | 6.4 | 10.8GB | Relies on a computationally **expensive online contact map pipeline**. |\n| **MEgoHand (Ours)** | **74ms** | **13.5** | **5.8GB** | **Replaces the contact map bottleneck** with a highly efficient, pre-trained depth estimator. |\n\nThe results are striking: **MEgoHand is over 2x faster and uses nearly 50% less VRAM** than the strong LatentAct baseline. This superior efficiency is a direct result of our novel architectural design. By replacing the intricate and slow online contact map generation with a fast, object-agnostic depth module, we have created a framework that is not only more accurate but also significantly more practical for real-time applications in AR/VR and robotics. We will add this compelling analysis to our experiments section.\n\n---\n\n## **Q2: On Failure Analysis**\n\n**Our Response:**\nThis is an excellent suggestion that provides deeper insight into the model's behavior. We have now performed this failure analysis and will include a dedicated subsection with qualitative examples in the **Appendix**. Our key findings reveal how our bi-level architecture leads to degradation:\n\n*   **Under Severe Depth Errors:** In challenging scenarios (e.g., highly reflective surfaces), the depth estimator can occasionally fail. When this happens, the generated motion's spatial grounding may be affected (e.g., reaching slightly too far). However, importantly, the high-level **semantic correctness of the action is typically preserved by the VLM \"cerebrum.\"** The model still knows *what* to do (e.g., form a grasp), even if its knowledge of *where* is slightly compromised.\n*   **Under Heavy Occlusion:** When the target object is heavily occluded, the model intelligently defaults to relying more on the textual instruction and its learned semantic priors from the VLM. This may result in a more \"canonical\" or averaged motion, as it lacks precise visual cues.\n\nThis analysis demonstrates a key strength: our model does not fail catastrophically. Its components provide a degree of built-in robustness, with the VLM ensuring semantic plausibility even when low-level visual information is imperfect.\n\n---\n\n## **Q3: On Cross-Domain Performance Drop**\n\n**Our Response:**\nThis is a crucial point. A performance drop on out-of-distribution (OOD) data is not only **expected but is also a testament to the genuine difficulty of the cross-domain challenge** we are tackling. The key insight is not that performance drops, but that MEgoHand's drop is significantly smaller than the baseline's, and it remains state-of-the-art in this challenging zero-shot setting.\n\n**1. The \"Performance Drop\" Validates the Challenge:**\nYou are correct to observe a drop in absolute MPJPE. The reason for this, as you astutely noted, is that our OOD test sets, ARCTIC and HOLO, are exceptionally challenging. They introduce not just unseen objects, but fundamentally **new interaction paradigms**:\n*   **ARCTIC:** Introduces complex **articulated object physics**.\n*   **HOLO:** Introduces long-horizon, **procedurally-guided tasks**.\nThese are not simple variations; they are zero-shot challenges that require true generalization. The performance drop across all methods is a testament to the difficulty of this task.\n\n**2. Our Method is a Successful Attempt to Improve Generalization.**\nThe entire architectural design of MEgoHand represents our primary and most successful attempt to improve cross-domain generalization.\n*   **Architectural Innovation:** By fusing a generalist, pre-trained Vision-Language Model (VLM) with an object-agnostic depth estimator, we deliberately designed a system that reasons about semantics and 3D space without overfitting to specific object geometries seen during training. This is a departure from prior methods that are more tightly coupled to their training data distribution.\n*   **State-of-the-Art Validation:** The success of our design is validated by our results. While all models' performance drops, MEgoHand's performance drops significantly less, and it consistently and substantially outperforms the strongest baseline in these zero-shot settings.\n    *   On **ARCTIC**, we achieve a **33.9% relative reduction in MPJPE** over LatentAct.\n    *   On **HOLO**, we achieve a **29.8% relative reduction in MPJPE**.\n    \nThis demonstrates that our method is already the most effective at generalizing across these difficult domain gaps.\n\n**3. Future Avenues to Further Bridge the Gap:**\nWhile MEgoHand sets a new SOTA, we agree that there is still room for improvement. Building on our strong foundation, we are actively exploring next-generation techniques to narrow the OOD gap even further. As you suggested, promising directions include:\n*   **(a) Large-Scale Synthetic Data Augmentation:** Leveraging domain-randomized virtual rendering to expose the model to a near-infinite variety of scenes and object configurations.\n*   **(b) Advanced Self-Supervised Learning:** Employing auxiliary objectives, such as contrastive losses on hand-object feature representations, to learn even more transferable and disentangled features.\n\nIn summary, the performance drop is a feature of the challenging problem, not a bug in our method. Our model's core design is our successful attempt to address generalization, and its state-of-the-art performance on these OOD datasets is the strongest evidence of its effectiveness.\n\n---\n\n## **Q4: On Improving the Abstract Wording**\n\n\n**Our Response:**\nMany thanks for your attentive reading. We completely agree the sentence was redundant and poorly phrased, and we apologize for the oversight. We have revised it to be more concise and clear: *\"Existing methods often rely on predefined 3D object priors, limiting their generalization to novel objects.\"*\n\n---\n\nWe truly hope that the additions and clarifications, particularly the new performance benchmarks and failure analysis, have addressed your concerns and helped to highlight the novelty and practical value of our work. We are sincerely grateful for your thoughtful and constructive feedback, which has greatly improved the quality of the paper. If you feel that your concerns have been fully resolved, would you please consider increasing the recommendation of the paper?"}, {"review_id": "au0GuCmK0d", "reviewer": "Reviewer_ULTh", "summary": "This paper introduces a framework named MEgoHand for generating realistic hand-object interaction motion sequences from an egocentric perspective. The generation is conditioned on multimodal inputs: a textual instruction, an RGB image, and an initial hand pose. A key capability of the proposed framework is its ability to generate motions for novel objects without relying on predefined 3D object models. Its main contributions include a high-level \"Cerebrum\" module for scene understanding and intent reasoning, a low-level \"Cerebellum\" module for fine-grained trajectory synthesis, and a dataset curation pipeline.", "questions": "1) Could the authors clarify how the prediction trunk size l=16 was determined? Specifically, how was the trade-off between prediction length, motion prediction performance, and computational cost evaluated?\n2) I would like to ask about the Inverse MANO Retargeting network. Compared to an end-to-end network, why can the two-stage training method avoid severely distorted hand geometry and huge reconstruction errors? Is it necessary to strictly follow the sequence of converging on the shape parameters first, and then the pose parameters?", "limitations": "No, the authors have not adequately addressed the limitations and potential negative societal impact of their work. While they briefly mention future work directions, a more explicit and comprehensive discussion is needed.\n\n1) *Single-Hand Interaction*: The model is currently limited to single, right-hand motion, which restricts its applicability for the many real-world tasks that are bimanual.\n2) *Computational Latency*: A discussion on the inference speed of the diffusion-based model would be valuable, as latency is a critical factor for real-time applications in AR/VR and robotics.\n3) *Sensitivity to Language Quality*: The model's performance likely depends on the clarity and style of the input text. An acknowledgment of its potential sensitivity to ambiguous or complex natural language would be appropriate.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**\n1) *Quality*: The bi-level Cerebrum/Cerebellum architecture thoughtfully separates high-level planning from low-level control, offering a robust framework for motion generation. The evaluation is thorough, spanning seven datasets (including two for zero-shot generalization).   Ablation studies validate the contribution of each component.\n2) *Clarity*: The paper is well-written and easy to follow. The introduction clearly motivates the problem and outlines the limitations of prior work. The cited references include the latest seminal methods in the field. The experiments are conducted extensively and the results support the claims convincingly.  \n3) *Significance*: The paper tackles a vital problem in computer vision, robotics, and AR/VR by enabling realistic hand motion generation for novel objects without predefined 3D models. This capability is essential for developing versatile interactive agents and immersive virtual environments.  \n4) *Originality*: The core architectural design is original. The synthesis of a VLM (for high-level semantics) with a depth estimator (as a proxy for object-agnostic 3D geometry) to create a motion prior is novel. The introduction of the Temporal Orthogonal Filtering (TOF) is a simple yet novel training-free decoding strategy that effectively addresses the common issue of jitter in generated motion sequences. The data curation pipeline, unifying diverse hand-object interaction (HOI) datasets into a 3.35M-frame resource, is also novel.    \n\n**Weaknesses**\n1) The paper doesn't model the object poses/shape, which limits the overall significance of the paper. Without explicit object modeling, the generated hand trajectories may collide or penetrate the manipulated objects in practice.\n2) The current framework is limited to single, right-hand interactions, as explicitly stated by the authors (\"We only consider right hand motion in this submission,\" Line 218). This is another constraint, as many real-world manipulation tasks require bimanual coordination.  \n3) There is a lack of discussion or ablation studies on the choice of Prediction Trunk Size parameter l.  \n4) The paper does not provide a specific method for obtaining the text instructions, which might limit the scalability and flexibility of the proposed method on new datasets that do not have textual instructions.", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns. While my concern regarding the physical plausibility (lack of modeling and evlauation for collision avoidance) remains, I would maintain my rating of borderline accept.", "rebuttal": "We sincerely thank the reviewer for the detailed comments and insightful review, especially for the convincing \"robust architecture for motion generation\" and novel design of \"VLM + depth estimator\". We are willing to provide a point-by-point response to address your concern.\n\n---\n\n## **Q1: Regarding Object Modeling (and Potential Collisions)**\n\n**Our Response:**\nThank you for this valuable suggestion. We agree that explicitly modeling object geometry is an important research direction. In this paper, we made a **strategic choice** to prioritize what we consider a more fundamental and unsolved challenge in the field: **achieving robust generalization to novel objects without any prior 3D models.**\n\n*   **Breaking the Dependency Bottleneck:** The reliance on predefined object meshes has been a major bottleneck limiting the scalability and real-world applicability of prior work. Our primary contribution is demonstrating that this dependency can be broken. MEgoHand successfully generates high-quality interactions by uniquely fusing a VLM's semantic understanding of *affordances* with a depth estimator's object-agnostic understanding of *3D geometry*.\n*   **Implicit vs. Explicit Modeling:** While we don't model objects explicitly, our approach learns an **implicit model of object surfaces and volumes** from the depth channel. The state-of-the-art accuracy we achieve (e.g., a **71.9% reduction in MPVE-PA**) is strong evidence that our generated hand motions are already well-aligned with the scene geometry, inherently minimizing significant penetrations.\n\n**Importantly, thank you for your valuable advice on incorporating object modeling.** You are absolutely right that for applications where object models do exist. It is a good idea to add an auxiliary loss to supervise the object poses or shape with explicit object meshes. Given the time constraints of the rebuttal period, we cannot integrate this feature now, but we will revise our discussion to explicitly acknowledge this limitation and highlight your suggestion as a key method for improving performance in our future direction.\n\n---\n\n## **Q2: Regarding Single, Right-Hand Interactions**\n**Our Response:**\nWe really thank you for raising this important point. Our initial focus on single, right-hand motion was a deliberate and strategic decision designed to rigorously tackle the foundational challenges of this complex problem before scaling to bimanual scenarios. We fully agree that bimanual interaction is a critical and exciting frontier. Our current work does not just acknowledge this; it actively paves the way for it.\n*   **Architectural Extensibility:** Our framework is inherently extensible to the bimanual case. The extension is conceptually straightforward: it would involve adding a parallel decoder for the second hand and introducing a mechanism—such as cross-attention within our DiT-based policy—to model the complex dependencies and coordinated roles between the two hands. This represents a well-defined and highly promising research path, not a fundamental redesign.\n*   **Data-Readiness:** Crucially, we designed our data curation pipeline with this future in mind. As you may have noted from our dataset statistics, **our 3.35M-frame curated dataset already contains a rich set of bimanual interactions**. We have already done the hard work of collecting and unifying this data, making it a ready-to-use resource for the community (and our own future work) to build directly upon our single-hand foundation.\n\nIn our revised manuscript, we will expand the Limitations section to frame this not just as a limitation, but as a strategic choice with a clear path forward. We will explicitly state:\n\"Our current study is intentionally focused on single-hand motion to establish a robust baseline for object-agnostic generation. We recognize that bimanual tasks represent a vital next step. Our framework is designed for extensibility to this scenario, and our curated dataset already includes the necessary bimanual data to support this future research, which we believe is a significant and exciting direction for the field.\"\n\n---\n## **Q3: Regarding Prediction Horizon (`l=16`)**\n\n**Our Response:**\nThis is an excellent question regarding a key hyperparameter. The choice of `l=16` was determined empirically to find the optimal balance between prediction quality, temporal coverage, and real-time performance. We have conducted an ablation study on the challenging ARCTIC dataset to illustrate this trade-off.\n\n| Prediction Horizon | MPJPE↓ | MPJPE-PA↓ | MPVE↓ | MPVE-PA↓ | MWTE↓ | MRE↓ | Inference Time (ms)↓ |\n|-------------------|--------|-----------|-------|----------|-------|------|---------------------|\n| 4                 | 8.204  | 1.312     | 8.142 | 1.276    | 6.781 | 0.421| 42ms               |\n| 8                 | 7.722  | 1.201     | 7.653 | 1.151    | 6.218 | 0.405| 54ms               |\n| **16**            | **7.358**  | **1.161**     | **7.268** | **1.106**    | **5.958** | **0.398**| **74ms**           |\n| 32                | 7.289  | 1.154     | 7.193 | 1.083    | 5.825 | 0.380| 112ms              |\n\nAs the table shows, increasing the horizon from 8 to 16 yields a notable improvement in motion quality across all metrics. However, extending it further to 32 offers only marginal gains while significantly reducing the inference speed below a practical threshold for many interactive applications (~10 FPS). Therefore, `l=16` represents the best trade-off, capturing a meaningful half-second of interaction while maintaining interactive frame rates. We will add this table and analysis to the Appendix.\n\n---\n\n## **Q4: Regarding Obtaining Text Instructions**\n**Our Response:**\nThis is a practical point about the model's deployment. Our framework assumes a high-level task instruction is available, which is a standard paradigm for VLM-driven agents. For our experiments on datasets that lacked explicit instructions (e.g., parts of H2O), we used a simple yet effective standardized template: *\"Interact with a {object} using the {side} hand.\"* This demonstrates the model's robustness even with minimal textual guidance.\n\nFor broader application to unlabeled videos, the pipeline is straightforward: instructions can be automatically generated by leveraging modern vision-language models, either through lightweight video-to-caption models or by prompting powerful foundation models. This modularity is a strength, allowing MEgoHand to be integrated into larger, fully automated systems. We will clarify this in our limitations section.\n\n---\n\n## **Q5: Regarding the Two-Stage Optimization for Inverse MANO**\n**Our Response:**\nThank you for this question. The two-stage training is critical for achieving stable and accurate results. Prioritizing shape (`β`) over pose (`θ`) is indeed an established practice in human motion retargeting (e.g., OmniH2O [1]), and we have validated its importance in the context of hands. The rationale is twofold:\n1.  **High Entanglement:** The MANO shape and pose parameters are highly entangled. An end-to-end optimization often falls into poor local minima, where an error in the shape estimate is compensated by an extreme, physically impossible pose, leading to severe geometric distortions.\n2.  **Decoupling the Problem:** Our two-stage approach decouples this complex optimization. By first converging on the hand's intrinsic shape, we establish a robust geometric foundation. With a stable shape model, the subsequent optimization for pose parameters becomes a much better-constrained problem, leading to significantly higher fidelity and realism. Our experiments confirmed that this sequence was essential for avoiding the very distortions the reviewer mentioned.\n\n---\n\n## **Q6: Regarding Computational Latency**\n**Our Response:**\nLatency is indeed a critical factor for AR/VR and robotics. We benchmarked our model's end-to-end inference efficiency and found that MEgoHand is not only more accurate but also significantly more efficient than the strong LatentAct baseline. The comparison was run on an RTX 4090 (24GB) GPU for a single sample (generating a 16-frame sequence).\n\n| Method | **Inference Time** | **FPS** | **GPU-Utilization** |**VRAM Usage** |**Key Reason for Efficiency** |\n| :--- | :--- | :--- | :--- | :--- |:--- |\n| LatentAct | 156ms | 6.4 | 45% | 10.8GB | Requires costly online **contact map computation** |\n| **MEgoHand (Ours)** | **74ms** | **13.5** | **24%** | **5.8GB** | **Object-agnostic design** bypasses this bottleneck |\n\nMEgoHand is over 2x faster and uses nearly half the VRAM. This is a direct benefit of our novel object-agnostic design, which replaces the computationally expensive contact map generation with an efficient depth estimator. This superior efficiency makes MEgoHand far more suitable for practical, real-time applications.\n\n---\n\n## **Q7: Regarding Sensitivity to Language Quality**\n\n**Our Response:**\nThank you for raising this important and practical point. We agree that a model's sensitivity to the quality of natural language is a critical factor for real-world deployment. While any VLM-based system has some inherent dependence on language, we designed MEgoHand with specific architectural choices to maximize its robustness to linguistic variation, and our results empirically validate this design. As shown in Tables 1, 2, the performance gap between our vision-and-depth-only variant `MEgoHand-ID` and the full, text-conditioned model `MEgoHand` is remarkably small. This is a crucial finding: it quantitatively demonstrates that while text provides valuable semantic direction, the model's accuracy and fine-grained realism are primarily driven by its strong visual grounding.\n\n---\n**References:**\n\n[1] OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning. CoRL 2024.\n\n---\nOnce again, thank you for your careful review. If most of your concerns are addressed, would you please consider increasing the recommendation of the paper?"}]

Author rebuttal:

Meta-review: This paper received four borderline accepts as final ratings. Initially, reviewers praised the paper for the large dataset, strong performance of the method, and novelty of the proposed model architecture. However, they had concerns regarding the failure analysis; poor alignment between images and predictions; comparisons with the Temporal Orthogonal Filtering method; and physical plausibility. The rebuttal stage cleared up many of the reviewers' concerns with additional results and definitions with all reviewers ending up on a positive final rating. The AC saw no reason to overturn the reviewers and agreed with their sentiments towards the paper and thus recommends acceptance.

The AC reminds the authors to address the changes brought up by the review process and add in the promised changes during the rebuttal stage.
Decision: Accept (poster)