HandsOnVLM: Vision-Language Models for Hand-Object Interaction Prediction
Authors: Chen Bao
Jiarui Xu
Xiaolong Wang
Abhinav Gupta
Homanga Bharadhwaj
Venue: Accepted by TMLR ()

Abstract: How can we predict future interaction trajectories of human hands in a scene given high-level colloquial task specifications in the form of natural language? In this paper, we extend the classic hand trajectory prediction task to several tasks involving explicit and implicit language queries. Our proposed tasks require extensive understanding of human daily activities and reasoning abilities about what is happening next given cues from the current scene. We also develop new benchmarks to evaluate the proposed two tasks, Vanilla Hand Prediction (VHP) and Reasoning-Based Hand Prediction (RBHP). We enable solving these tasks by integrating high-level world knowledge and reasoning capabilities of Vision-Language Models (VLMs) with the auto-regressive nature of low-level ego-centric hand trajectories. Our model, HandsOnVLM is a novel VLM that can generate textual responses and produce future hand trajectories through natural-language conversations. Our experiments show that HandsOnVLM outperforms existing task-specific methods and other VLM baselines on proposed tasks, and demonstrates its ability to effectively utilize world knowledge for reasoning about low-level human hand trajectories based on the provided context. More details can be found at https://www.chenbao.tech/handsonvlm/.

Published in Transactions on Machine Learning Research (08/2025)
HandsOnVLM: Vision-Language Models for Hand-Object Interaction Prediction
Chen Bao Carnegie Mellon University
chenbao@cmu.cdt edu
Jiarui Xu UC San Diego
jia026 edu
Xiaolong Wang!
riu0120uusdd edu
UC San Diego
Abhinav Guptal
abhinaug@cs.cmm. edu
Carnegie Mellon University
Homanga Bharadhwaj
homangosaackhole@@@@mm.dooo
Carnegie Mellon University
Reviewed on OpenReview: https: // openreview. net/ forum? id= ehhMF denotes equal advising.
Abstract
How can we predict future interaction trajectories of human hands a scene given high- level colloquial task specifications in the form of natural language? In this paper, we extend the classic hand trajectory prediction task to several tasks involving explicit and implicit language queries. Our proposed tasks require extensive understanding of human daily activitics and reasoning abilitics about what is happening next given cues from the cur- rent. scene. We also develop new benchmarks to evaluate the proposed two tasks, Vanilla Baased Hand Prediction (VHP) and Reasoning- Based Hand Prediction (RBHP). We enable solving these tasks by integrating high-level world knowledge and reasoning capabilitics of Vision- Language Models (VLMs) with the auto-regressive nature of low-level ego-centric hand trajectories. Our model, HandsOn VLM is novel VL.M that can generate textual responses and produce future hand trajectories through natural-language conversations. Our experi- ments show that Hands VLM outperforms existing task-specific methods and other VLM baselines on proposed tasks, and demonstrates its ability to effectively utilize world knowl- edge for reasoning about low-level human hand trajectorics based on the provided context. More details can found at https:://// //www/wwwc/ooo//// chenbao. tech/handsonvlm/.
1 Introduction
Humans interact with the everyday world and express themselves with informal and oftentimes vague lan- guage descriptions. Consider the example in Fig. 1 when try to open the jar, might think, "Ah, need something to help open this slippery jar more casily.' We are uncertain about what we exactly want as well as about. how to come up with a solution. Building computational system addressing this need would require a good understanding of what tools we have lying around (visual scene understanding). gen- eral apriori experience opening jars (reasoning ability and world knowlodge priors), and the ability actually execute the necessary actions for opening the jar (low-level trajectory). this paper, we develop language-conditiones prediction tasks for tackling this problem, propose benchmarks for evaluating progress

---

Published in Transactions on Machine Learning Research (08/2025)
Xh. rieed somehhing help open this slippery jar more casily...
HandsOnVLLM Assistant
To open slippery you can use the cloth nearby.
plausible
mnjeectoy your right hand can take overlaid on scene:
<HANDS.... <HAND>
Figure 1: HandsOn VLM forecasts low-level actions the form of hand trajectories in the user's egocentric view of a scene when queried with a question via natural language. It also capable of handling indirect queries that require reasoning about what object interact with and how perform the interaction. [Right] We show glimpse of left and right hand trajectory predictions from Hands VLM across diverse real-world scenarios.
on these tasks, and build a vision-language model (VLM) for predicting low-level hand trajectorics in user's egocentric view of a scene given colloquial language queries.
Towards a similar goal, some prior works have focused on identifying human intentions based on egocentric human vidcos of daily activitics (high-level intentions of the form "cutting pepper', "washing plates") (Kr- ishna et al., 2017; Grauman el al., 2022; Kahalapiliya et al., 2024), while others have focused on predicting low-level actions such as hand trajectories given human action clips (Liu et al., 2022; Zhang et al., 2024b) without conditioning the predictions these scenarios are a bit restrictive since for most everyday tasks (e.g. in Fig. 1) we need a combination of high-level reasoning of what to do in a scene and low-level understanding of how to interact with the relevant objects in the scene.
detailed language descriptions of the task to be performed. Both
By drawing on the recent successes of VLMs for high-level reasoning (Liu al., 2024: Lai et al., 2024; Cheng et al., 2024) and advancements in hand reconstructions from generic web videos (Shan et al., 2020b; Rong et al., 2020; Pavlakos et al., 2024), develop system for future hand trajectory prodiction given conversation-style language instructions. Current best multinodal VLMs are good at predicting semantic actions in the form of what is happening at a certain point in a video (Maaz et al. (2023); Huang et al. (2024)), interpreting what objects are in a scene (Achiam et al., 2023) and natively support free-form language conversations for conditioning. However, they are not good at directly predicting low-leved actions (in the future) of the form of hand-object trajectories. At the same time, recovering low-level interactions in videos, like hand meshes (Pavlakos al., 2024), object meshes (Fan et al., 2024), and regions of interactions (Shan et al., 2020b; Goyal al., 2022) have independently become very reliable in recent years. Our key insight is to fine-tune a pre-trained VLM with auto-regressive trajectory predictions of human hand positions, given a few seconds of context video and a language description of the task.
Our approach VLLd casts hand trajectory prediction as an auto-regressive next token prediction conditioned on fused video and language tokens. We develop HandsOn VLM as an interactive chat assistant that we can query with informal instructions of the form, "Where should my hand move if want to open the refrigerator?" and video (or an image) of scene, and obtain outputs of the form, "To open the refrigerator, the predicted hand trajectory is <HAND> ..... <HAND> The HondsOn VLM model first converts the RGB video context to visual tokens and fuses them with the language tokens through flow-fast pooling (Huang et al., 2024) for capturing temporal information from the context video at fine resolution. We extend the vocabulary to add a new <HAND> token, and output a sequence of text and hand Lokens. We finally have a trajectory decoder to convert the hand tokens to a sequence of positions of the left and right hands over the prediction horizon, projected on the image frame.
In summary, our paper has the following contributions:
2

---

Published in Transactions on Machine Learning Research (08/2025)
We develop HandsOnVLM, novel VLM that generate textual responses and produce future hand trajectories through conversations by expanding the original vocabulary with hand tokens and having iterative position encodings for auto-regressive predictions during inference.
We extend existing traditional hand prediction tasks to new lasks, including Vanilla Hand Prediction (VHP) and bassoning-based Hand Prediction (RBHP), to predict hand trajectorics from ego-centric human videos conditioned on language queries of different forms.
We develop benchmarks for evaluating progress on the VHP and RBHP tasks which we open-source to the community, in addition to our trained models on the respective benchmarks.
Our results on diverse real-world datasets of human videos and zero-shot evaluations on completely unseen datasets demonstrate strong generalization and reasoning capabilities of HandsOn VLM for hand trajectory prediction given colloquial language instructions. Furthermore, the model outperforms most baselines on the Reasoning- based Hand Prediction (RBHP) task, showcasing its capability to reason and leverage world knowledge of VLMs.
2
Related Work
We discuss prior works on human motion reconstruction and forecasting, developments in multimodal large language models and action understanding from human videos.
2.1 Human Motion Prediction
Several prior works have attempted to recover hand meshes and full body meshes from human videos (Rong et al., 2020; Pavlakos et al., 2024). Going beyond reconstruction, other works have also investigated fore- casting motions of humans in the future. Early works used RNNs (Bütepage et al., 2017; 2018; Honda et al., 2020) for anticipating future human poses, and recent approaches include Transformer architectures for more diverse and plausible future predictions Ding et al. (2023). More directly related to our work, some approaches predict egocentric hand-trajectoric in the form of 2D waypoints (Liu al., 2020), and others also predict object affordances jointly with hand trajectories (Liu et al., 2022; Bharadhwaj et al., 2024b). Some predict hand trajectories in 3D space conditioned et al., 2023). Architectures for such egocentric predictions have ranged from transformers (Liu et al., 2022; Bao et al., 2023) to diffusion models (Ma et al., 2024b;a) trained specifically for this prediction task. Our work extends this line of low-level egocentric trajectory prediction by enabling reasoning capabilities through augmentation and joint training with a pre-trained VLM.
few RGB observations from egocentric view (Bao
2.2 Multimodal Large Language Models
Our work enabled by developments in multimodal Large Language Models that augment vision and language reasoning in a unified model. Such models like LLaVA (Liu el al., 2024) and Video-ChatGPT (Maaz et al., 2023) have enabled large-scale video understanding and localization of temporal events (semantic actions) in videos (Huang al., 2024). Adjacently, other works have sought to make the inputs to the VLMs more flexible and informal through automatic segmentations of language instructions (Lai et al., 2024; Yang et al., 2023) and visual grounding allowing flexibility to process both image and region inputs (Rasheed et al., 2024). Recent works have extended the capabilities of VLMs to diverse domains including robotic navigation (Zhang et al., 2024a), robotic manipulation (Kim et al., 2024; Zitkovich et al., 2023), spatial reasoning Cheng et al. (2024), and reasoning about 3D human poses from images and text Feng et. al. (2024). While these approaches are orthogonal to our task of egocentric hand trajectory prediction, they serve evidence of the potential of VLMs for downstream applications.
2.3 Action Recognition and Prediction from Videos
Understanding actions in the form of what is happening video segment has a long history in computer vision (Sigurdsson et al., 2017; Liu et al., 2021; Kovashka & Grauman, 2010; Feichtenhofer et al., 2019).
3

---

Published in Transactions on Machine Learning Research (08/2025)
H
<images> Where should my hand move to if want to gently mix the contents of the cooking vessel?
Large Language Model J
Sure! the predieted hand
trajectory is HAND>>
Xg
<HADD> HAND
HAND
Visual Backbone
Trajectory Decoder
h
X
F Fenc
dec
Figure 2: Overview of the HondsOn VLM architecture, where and denote trainable and frozen modules separately. HandsOn VLM casts hand trajectory prediction as an auto-regressive next token prediction conditioned on fused video and language tokens. The architecture augments pre-trained VL.M with additional hand token in the vocabulary. We use tively.
and to represent text and <HAND> tokens respec-
Several benchmarks and datasets containing human videos and action labels for tasks have also been proposed for related problems (Grauman et al., 2022; Caba Heilbron et al., 2015; Goyal et al., 2017; Xiong et al., 2023). Our work leverages such datasets and goes beyond recognition of actions in videos to prediction of low-level actions in the future by first reasoning about future high-level actions through a VLM. As such our work can have potential applications in robotics for learning motion from web videos for manipulation by complementing prior works in this space (Bharadhwaj al., 2024a; Bahl al., 2023; Nair al., 2022; Bharadhwaj et al., 2025).
3 Approach
HandsOn VLM is video-based VLM with the capability of predicting future hand trajectories given video context and language instructions. There are three key components of HandsOn VLM's architecture: (1) SlowFast tokens to capture temporal information at fine temporal resolution, (2) hand represcntation using an augmented vocabulary of <HAND> token, and (3) iterative hand decoding to enable auto-regressive trajectory training and inference. In the training stage, we fine-tune a pre-trained VLM by combining next-token prediction loss and trajectory loss.
3.1 Architecture
We show an overview of the HandsOn VLM model architecture Fig 2. HandsOn VLM takes a sequence of T frames X2 and language instruction X, as input and predicts future hand trajectories where N is the future horizon. At each future time step T the future hand location hr+: consists of the 2D location of the center of the left. and right hands projected to the last observation frame X, [-1]. The key components of the architecture include a visual backbone Fanes a pision-to-langunn projection layer f. a Large Language Model(LLM) F and a trajectory decoder Facc-
+hr++)1 \
SlowFase Token Compression. To obtain a capable video-conditioned VLM we need to be able interpret temporal information a fine resolution. Following Huang et al. (2024), given X,, we embed them into T x M visual lokens using a visual backbone, where M is the number of tokens in each frame. Then apply slow-fast pooling to get T + M visual tokens. In the fast path, average all the tokens within each frame to get T tokens overall. We also uniformly select frames among all T frames and perform x spatial average pooling to get M slow frames in total. These slow tokens will help preserve spatial information during the encoding process. Then we embed and align T + M visual tokens to the language space through a lision-to-language projector f(-).
Hand as Embedding. To represent hand in the language space, we extend the existing vocabulary with a < HAND> token. However, typical embedding layer would encode each <HAND> token identically,
4

---

Published in Transactions on Machine Learning Research (08/2025)
resulting in individual <HAND> token being indistinguishable from another. To overcome this limi- tation, we embed ground truth hand positions into the <HAND> tokens during the tokenization process. We feed them into the Large Language Model backbone and get the embedding of the last layer H, where = = (Xee)(XX))).
Iterative Hand Decoding. For i-th token the sequence, let H, be the last-layer embedding of this token from the Large Language Model. HandsOn VLM decode it to predict the 1)-th token LLMs do. When (i + 1)-th token is < <HAND> token, we imput H; into hand trajectory decoder Fdec to prediet the hand position of the (i+ 1) token h4+1 Faee (H.). During inference, this decoded position is then encoded into the corresponding < <HAN>> token embedding for following prediction rounds. In this way, we ensure that each subsequent prediction is conditioned on all previously predicted hand positions, maintaining temporal consistency and spatial awareness throughout the inference process and mitigating compounding errors.
3.2 Training Objectives
The model is trained end-to-end using a text generation loss Lix and a hand trajectory prediction loss Lhand The overall objective is the weighted sum of both losses, determined by Atxs and Aband'
L = Aixe Ahand Chand
(1)
Specifically, Lixt is the auto-regressive cross-entropy loss for text generation, and Lnand is the hand prediction loss, which encourages the model to generate high-quality hand trajectories as well. Following Liu et al. (2022), we employ a reconstruction loss over future timesteps and a KL-Divergence Regularization loss Chand:
N
Lhand Σ
Lranon (1rstshTte)) + Lui (un:on)
(2)
We employ CVAE (Sohn et al., 2015) as the hand trajectory decoder in this work (although the method is not tied to it). Thus, Lrecon is the MSE loss over valid hand positions, and μh, σy here are the mean and the standard deviation that regularizes the latent z-space to be close to the normal distribution.
4 Reasoning and Predicting Hand Trajectories
In this section, we introduce two tasks: the Vanilla Hand Prediction (VHP) task, which extends the classic hand motion prediction (Liu et al., 2022), and the proposed Reasoning- based Hand Prediction (RBHP) task. Finally, we describe two-step ionotation-generatin pipeline to build the corresponding RBHP datasct.
4.1 Vanilla Hand Prediction Task
In this task, explicit action narration is required to predict the next hand motion. Here explicit means the action narration directly specifies the action and the target object without ambiguity, such as "cut the paper" or "open the microwave". We choose Epic-Kitchen (Damen et al., 2018; 2022), H2O (Kwon et al., 2021) and FPHA (Garcia-Hernande) et al., 2018) as datasets for this task. To generate the hand labels for all the datascts, following (Liu al., 2022), we first run an off-tho-shelf active hand-object detector (Shan et al., 2020a) to get the hand bounding box in each frame. To get the ground truth of each future hand jrajectory, first compute pairwise homographies by matching SURF (Bay, 2006) features of masked regions through RANSAC and project each future hand position into the last observation frame. Then, we apply cubic Hermite spline interpolation to smooth the projected trajectories and fill any missing points. Finally, we filter the resulting tra trajectories with multiple criteria, including confidence thresholds, highest-score detection selection, feature matching thresholds, trajectory completeness checks, and boundary constraints.
To reformat these datasets for visual question answering, we structure them in a question-anse ansser format using the following template:
5

---

Published in Transactions on Machine Learning Research (08/2025)
"USER:<images>, you give me the future hand trajectory for ferplicit action narration ASSISTANT: Sure, it i<HHAND>..<HNNDD..
where < <images> a placeholder of visual tokens of the input frames. Note that the action is optional because we can also generate general templates without specifying the action, and in this case the task reduces to that prior works (Liu et al., 2022; Bao et al., 2023; Ma al., 2024b).
4.2 Reasoning-basee Hand Prediction Task
In addition to the Vanilla Hand Prediction Task, we introduce the Reasoning- based Hand Prediction (RBHP) task. Instead of utilizing explicit instructions to directly predict the hand motion, here the system required to reason about it with implicit instructions. We define implicit instructions as colloquial language instruc- tions that provide sufficient information for inferring the intended human hand action through reasoning, without explicitly naming the target object action.
To construct a dataset for this lask, we implement a two-slep cennolationggneraalln pipeline (Fig. 3) powered by GPT-4 (Achiam et al., 2023). This pipeline extracts implicit instructions from the Epic-Kitchenn-100 dataset (Damen et al., 2022). Prompt templates for these two steps are provided in the Appendix.
Classic Hand Trajectory Dataset
Action: Get the dough
GPT-4 Action-aware Image Description
In the image, there is countertop with various kitchen items. A hite mixing bowl containing dough is positioned on the left, while a black baking tray sits in the center. The person's hand, covered in dough, reaching towards the tray, suggesting they are about transfer the dough. To the right, there frying pan with some cooked meat, and blender is visible in the background. The overall scene depicts busy kitchen environment, focused food preparation.
Implicit Action Generation
Visual Question-Answering Conversion
<imagey> What is the recommended hand movement for accessing the mixture in the white container?
To accessing the mixture in the white container, the recommended hand trajectory <HAND--HHAND-CHHDD>>-HNAA
Figure 3: Illustration of the annotation pipeline for the RBHP task. By using GPT-4 on human video datasets we extract implicit language instructions for visual question-answering, The red and blue lines respectively show trajectories for the right and left hands.
Action-aware Image Description. To get the implicit instructions, the first step is to generate a detailed description of the scene including all the objects in the foreground. We prompt GPT-4 with the ground truth action to capture action-related information, such as the physical properties of the target object or the spatial relationship with other objects.
Implicit Action Generation. Using the action-aware description of the scene, are able generate the implicit instructions using GPT-4 in text template follows:
"USER:<images>; can you give me the future hand trajectory for {action implicil description]? ASSISTANT: Sure, il is <HAND>.<HAND>>>
We choose Epic-Kitchen and Ego4D (Grauman al., 2022) as datascts for this task. Through the annotation- generating pipeline, we generate 7.5k question-answering pairs from Epic-Kitchen and 8k pairs from Ego4D for zero-shot evaluation.

---

Published in Transactions on Machine Learning Research (08/2025)
On Validation Split
Zero-shot
EK55
EK100
H2O
FPHA FDE 4 WDE 4
Approach
BBox Input
ADE 4. FDE 4
WDE
ADE
4 FDE 4 WDE 4
ADE &
FDE & WDE J ADE &
KF OCT
0.392
0.386
0.199
0.317
0.318
0.168
0,216
0.199
0.105
0.209
0.187
0.102
OCT-global
0.232
0.2188
0.115
0.216
0.193
0.105
LA
0.156
0.139
0.076
0.254
0.224
0.124
0.150
0.121
0.032
0.214
0.189
0.043
L.AVA-TTaa
0.126
0.142
0.073
0.201
0.191
0.103
0.133
0.130
0.031
0.167
0.041
HandsOn VLM
0.136
0.106
0.062
0.194
0.157
0.090
0.135
0.108
0.028
0.
0.151
0.034
Table 1: Comparison of VHP task with different baselines. We reported the performance on the validation split of Spic-Kitchee dataset. For the RBHP baselines, we also evaluate them on two unseen datasets, H2O and FPHA.
5
Experiments
We perform experiments for both the proposed tasks in order to answer the following research questions:
How plausible are the hand trajectories produced by HandsOn VLM?
Does Hands On VLM exhibit reasoning abilities for implicit language queries?
Does Hands VLM generalize zero-shot to unseen scenes from new datasets?
5.1 Experiment Details
Architecture. Following LITA's 's architecture, We use CLIP-I-14 (Radford et al., 2021) as the visual encoder and Vicuna (Chiang et al., 2023) as the LLM module. We adapt the vision-language projector from LLaVA (Liu et al., 2024) and have CVAE (Sohn et al., 2015) as trajectory decoder.
Datasets. For VHP and RBHP datasets, we sample 10 frames and prodict the hand position in next 4 frames at FPS = 4. In addition to our proposed datasets, HandsOn VLM are also trained on a few additional datasets for five different tasks, namely Ceelvvttyee-Caaption (Krishna et al., 2017)and YouCook2 (Zhou et al., 2018) for dense video captioning and event localization, NExT-QA (Xiao et al., 2021) for video question answering, LLAVA-150K (Liu et al., 2024) for image instruction tuning, ActivityNet-RTI (Huang et al., 2024) for reasoning temporal localization. We co-train with these additional tasks to help with visual understanding and reasoning, and this is enabled by the flexible modeling of HandsoO VLM that allows training on generic QA datascts.
Implementation Details. For HandsOn VLM and other VLM-based baselines, in each epoch we select 24K samples from the Epic- Siii-Kitchenn-IUU VHP dataset. For HandsOn VLM', in each epoch we randomly select 6K samples in Epic-Kitchens: 100 VHP dataset, 6K Epic-Kitchens- 100 RBHP dataset and another 12K that are uniformly distributed among all other 5 tasks. We use a batch size of 128, learning rate of 2e-5 and train for 40 epochs. The total wall-clock time for training is around 18 hours for the 7B models while using 8 H100 GPUs. The LLM and vision-language projector are initialized with the LLaVA-1.3 pre-trained weights. During training, we freeze the visual backbone and fully fine-tune other modules.
5.2 Metrics and Baselines
Following previous works (Liu et al., 2022; Ma et al., 2024b) use Average Displacement Error (ADE). Final Displacement Error (FDE) and Weighted Displacement Error (WDE) as metrics to evaluate VHP and RBHP tasks.
Vanilla Hand Prediction. For the VHP task, we choose Kalman filter(KF). and Object-centric Trans- former( (OCT) (Liu et al., 2022) as the basclincs. Since OCT still requires the bounding box feature of the hand and object as input, to get a fairer comparison with other end-to-end methods, we implement version without the requirement of the bounding box, which we call OCT-global.

---

Published in Transactions on Machine Learning Research (08/2025)
RBHP (Epic-K)
RBHP (Ego4D)
Approach
ADE y FDDE LWDE
LADE FFDE LWDE
Kling 1.5
0.31
0.35
0.19
0.27
0.41
0.18
LumaLabs
0.29
0.37
0.18
0.21
0.28
0.13
LLaVA-P2S
0.27
0.24
0.13
0.31
0.28
0.14
LLaVA-T
0.19
0.18
0.10
0.38
0.35
0.17
HandsOn VLM
0.19
0.16
0.09
0.22
0.19
0.10
HondsOn VLM
0.18
0.15
0.08
0.22
0.18
0.09
Table 2: Comparison of HandsOn VLM the RBHP task with different basclines. fmeans fine-tuned
the RBHP dataset.
Reasoning-based Hand Prediction. To evaluate FandsOnVLM'ss performance on the RBHP task, we perform baseline comparisons with several VLM-based methods. We describe these basleines below:
LLLVA-Traj. Note that the hand trajectories are a sequence pixel positions, we can represent them in text directly. In this case, we can directly fine-tune the LLaVA without any modification.
LLaVA-Pixel2Seq. An alternative approach to representing hand positions involves quantizing the image into discrete spatial bins (Chen et al., 2021), each corresponding to unique token. We can extend the existing vocabulary with those discrete tokens.
Language-condditoond Image-to- Video Models. We also compare model to baselines of the langnage-conditioned image-to-video generation followed by hand-tracking. We use commercial state- of-the-art language-conditioned image-to-video systems such as LumaLabs (LumaLabs, 2024), Kling 1.5 (KlingAI, 2024) and generate videos conditioned on the last observation frame and the language description. Following the hand label generation process in Sec. 4.1, we track and extract the hand trajectories of the generated video.
5.3 Comparisons with Baselines
We evaluate VLM both the VHP task and the proposed RBHP task and report the results and comparisons with baselines in Table 1 and Table 2 respectively. All models except HandsOn VLM are trained on VHP datascts. HandsOn VLMt is trained on all available datasets (Data Combo Table 3).
VHP Task. We evaluate all the baselines on the VHP datasets as described in section 5.1. Here, the FPHA and H2O datasets serve unseen datasets test zero-shot generalization capabilities. Among all the VHP datasets, HandsOn VLM outperforms both the task-speeific methods as well as the VLM-based methods, which demonstrates its strong ability to produce plausible trajectories corresponding to how a real human hand would move given explicit instructions. We also find that HandsOn VLM can generalize to completely unseen scenes (for example scenes from H20 and FPHA datasets), which demonstrates it can effectively leverage the world knowledge the pre-trained VLM.
RBHP Task. For evaluations on the RBHP task shown in Table 2, HandsOn VLM achieves state-of- the-art performance in all three metrics. This suggests that HandsOn VLM is able to reason based on implicit cues of the scene and be applied to complicated scenarios involving everyday natural language conversations. However, observe that LumaLabs (LumaLabs, 2024) achieves the smallest ADE in the Ego4D RBHP benchmark but relatively higher FDE and WDE. This may be because the commercial text- conditioned image-to-video generation models have realistic video generation capabilities but cannot un- derstand reasoning- based language prompts which is necessary for generating plausible videos maintaining temporal consistency. Since the training datasct compositions of these vidco models are not disclosed, there may also be some data leakage issues of the evaluation datasets in this paper being part of their training corpora.
8

---

Published in Transactions on Machine Learning Research (08/2025)
Epic-K Kitchen
Data Combos
55
100
LITA data
RBHP data
ADE4
FDE4
WDE
0.206
0.195
0.101
2
0.197
0.165
0.094
3
0.199
0.163
0.094
4
0.187
0.156
0.089
Table 3: Analysis of the impact of training data the performance of HandsOnVIM. We can that performance increases with additional data of VHP (first two rows), even with datasets of other tasks (third row), but the highest gains come from the proposed RBHP dataset (last rows).
Num of Generations
ADE
FDE
WDE
1
0.187
0.156
0.089
4
0,184
0.152
0.087
0.182
0.151
0.086
0.182
0.150
0.086
Table 4: Analysis of test-time computations for HandsOn VLM in the form of stochastic decoding with self- consistency (Wang al., 2023).
5.4 Ablation Study
In this section, we conduct broad study of the different components of our model. All experiments this section are evaluated on the RBHP task.
Effects of Different Sources of Dataset. In Table 3, show the contribution of each type of dataset to the performance of HandsOnVLM. LITA dataset denotes the different datasets for 5 additional tasks (Huang et al., 2024) described in Section 5.1 ranging from dense video captioning to reasoning about temporal localization. While increasing the scale the VHP dataset (first two rows) can bring some improvement, we find that fin-tuning with the reasoning dataset (last two rows) can significantly boost the performance, even when tun-tuning with tasks that are not directly related hand trajectory prediction. This demonstrates that HandsOn VLM can leverage world knowledge learned by other tasks reason about predicting plausible hand trajectories.
Test-time Computation. Recent works (Snell et al., 2024; OpenAI, 2024) have shown that using more test-time computation is critical step for LLMs to improve their performance, especially on reasoning tasks. Motivated by these, we also invostigate such propertics can enhance the performance of HandsOn VLM pre- dictions. We report the performance using different numbers generations during the stochastic decoding with self-consistency (Wang al., 2023) Table The main idea to sample a diverse set of reasoning paths instead of just one and then select the most consistent output through marginalization. To obtain the self-consistency result in our context, generate multiple answers for each inquiry and then average the predicted hand trajectory. We find that increasing the test-time computation in this form can robustly improve the performance of HandsOn VLMas seen by the lower metrics from top to bottom in Table 4.
Effect of Different Observation Frames. Table 7, we investigate the performance of our approach and the baselines when conditioned on only one observation frame instead of an observation video. Here we have four comparisons: OCT-last-im, DCT-globballast-im, HandsOn VLM-last-im, HandsOn last-m which respectively correspond to versions of our baselines in Sec. 5.2 but only conditioned on the last frame of the input video context. We find that the results in this evaluation scenario comparable to the setting where the context is a video, indicating that HandsOnVL.M can be flexibly conditioned on just one image when a video context is not available.
Effect of Fine-tuning Visual Backbone. We follow the standard practice training the Vision languaee Model, freezing the visual backbone during training. Many previous works Liu et al. (2024); Feng et al. (2024); Huang et al. (2024) have demonstrated the effectiveness of this method. Due to computational resource linitations, conduct the experiments for thsi ablation study relatively small scale, which
9

---

Published in Transactions on Machine Learning Research (08/2025)
Prompt
Input
LLaVA-Pixel2Seq
HandsOn VL.M
GT
Where should my hand move to want to remove debris from round, metallic object the sink?
Can you provide the hand trajectory for blending dieed vegetables in the pan?
Where should my hand move if'l want gently mix the contents the cooking vessel?
Can you provide the hand trajectory for closing the bottle of milk?
Can you provide the hand trajectory for taking out espresso?
Figure 4: Qualitative results for different samples from the validation split of our RBHP dataset (top in blue) and zero-shot evaluations on completely unseen datasets FPHA and H2O (bottom in pink). The left-hand trajectory is visualized blue and the right-hand trajectory is in red. The arrows denote the direction of each trajectory. GT trajectories are provided for reference.
Task
Methods
ADE
FDE
WDEL
OCT
0.209
0.187
0.102
OCT-last-im
0.213
0.191
0.104
OCT-global
0.216
0.193
0.105
VHP
OCT-global-lasse last-iim
0.212
0.189
0.103
HandsOn VLM
0.194
0.157
0.090
Hands VLM-last-im
0.197
0.165
0.094
HandsOn VLM
0.197
0.165
0.094
HondsOn --M-last-im
0.197
0.163
0.093
RBHP
HandsOn. VLM
0.187
0.156
0.089
HandsOn VLMiilast-mm
0.187
0.155
0.088
Table 5: Analysis of the # of observation frames during inforence.
is named VHP-Small in Table 6. We find that fine-tuning the visual backbone degrades performance. This observation is aligned with some previous works in the VLM space Karamcheti et al. (2024).
Effect of Slow-fast Tokens. Many previous works have suggested that slow-fast tokens are beneficial for video understanding. In workFeichtenhofer et al. (2019), the authors have demonstrated the contribution of slow-fast tokens to video action classification and detection tasks. A recent work Huang al. (2024) implemented similar design in video question answering problems and achieved state-of-the-art performance
10

---

Published in Transactions on Machine Learning Research (08/2025)
Task
Methods
ADE
FDEL
WDEL
Fine-fine Visual Backbone
0.192
0.178
0.100
VHP-Small
Frozen Visual Backbone
0.174
0.154
0.089
Table 6: Analysis of fine-tuning visual backbone during training.
Task
Methods
ADE
FDE
WDEL
Fast. Token Only
0.316
0.236
0.144
VHP-Small
Slow Token Only
0.200
0.174
0.100
Slow-fast Tokens
0.174
0.154
0.089
Table 7: Analysis of slow-fast tokens during training.
in video understanding and answering tasks. We find that among all the options, slow-fast tokens work the best, proving the effectiveness of our design choice.
5.5 Qualitative Results
Fig. we show qualitative results for Hands VLM and the strongest baseline LLa LLaVA-Pixel2Seg The section above the horizontal line shows visualization from the validation split of RHBP datascts, while the section below the line shows zero-shol results on scenarios from completely unseen datasets.
In the second row, observe that Hands VLM generates a trajectory where the left hand stably holds the pan while the right hand performs the blending action. In contrast, LLaVA-Pixel2Seg fails to correctly predict the act of holding the pan. The third row results demonstrates HandsOn VLM's ability to reason about multimodal solutions for the same task. While the ground truth shows that the right hand moving the pot, HandsOn VLM chooses to use the left hand to execute the same action, illustrating its multimodal reasoning capability.
5.6 Human Evaluation
Going beyond automated metrics, to determine plausibility of the generated hand trajectories for various scenarios, we also perform human evaluations. We create a simple user interface that contains the context video along with language description of the task, and present predicted hand trajectories (for two different methods). We ask the users to pick which prediction appears more plausible given the context video and task description. We randomize the relative orders across different instances, and after the user clicks on a prediction, the page refreshes showing the the next scenario. Overall, we perform this study with 100 examples each for the VHP and RBHP settings, and recruit 20 participants for the study. Results Table 8 show that the predictions from HandsOn VLM (the model fine-tuned on reasoning tasks) more plausible for both VHP and RBHP evaluations, suggesting that model is able to effectively leverage world knowledge from other reasoning tasks to reason about low-level hand-object interaction predictions in diverse scenarios.
separate videos corresponding to
6 Conclusion
In this work, we propose HandsOn VLM, novel video-based VLM to predict future hand motion from ego-centric vidco contexts. We also propose different prediction tasks, including Vanilla Hand Prediction
Method
VHP 30
RВНР 28 5%
Hands On VLM
3%
HandsOn VLM
3%
5%
Table 8: Human study showing the % mean and SE of trials where participants consider hand trajectory predictions from one method more plausible than the other.
11

---

Published in Transactions on Machine Learning Research (08/2025)
(VHP) and Reasoning-based Hand Prediction (RBHP) to benchmark low-level trajectory prediction and reasoning. We demonstrate the effectiveness of our approach through extensive evaluations in diverse real- world scenarios. We believe this research represents a promising initial step towards integrating egocentric hand-object trajectory predictions with the powerful reasoning capabilities of VLMs. As the ground-trute hand reconstruction from videos gets belter due to wide community interest, the capabilities of our prediction model can further improve in scenarios like occlusion, and be practically deployable for in-context predictions in wearable devices that are becoming ubiquitous. An exciting direction of future work would be adapt our model for long-horizon predictions for activities like "making coffee" which would consist of several steps and require reasoning over extended period. Since video clips on the web have significant. camera motion over time, a viable strategy for this could be chaining the model sequentially for different sub-tasks.
Acknowledgements
We thank Gaurav Parmar and Jinkun Cao for feedback on the paper, and thank Yufei Ye, Ruihan Yang, Unnat Jain, Mohan Kumar Srirama, Shubham Tulsiani, and many others from CMU and UCSD for helpful discussions. This work used Bridges-2 at Pittsburgh Supercomputing Center from the Advanced Cyber- infrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants 2138259, 2138286, 2138307, 2137603, and 2138296.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint XrXi::2003.08 08774, 2023.
Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos as versatile representation for robotics. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13790, 2023.
Wentao Bao, Lele Chen, Libing Zeng, Zhong Li, Yi Xu, Junsong Yuan, and Yu Kong. Uncertainty-aware state space transformer for egocentric 3d hand trajectory forocasting, 2023. URL https://arxiv.org aba/2307.0820
Herbert Bay. Surf: Speeded up robust features. Computer Vision ECCV, 2006.
Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation. arXiv preprint arXiv:240916625 2024a.
Homanga Bharadhwajj, Abhinav Gupta, Vikash Kumar, and Shubham Tulsiani. Towards generalizable zero- shot manipulation via translating human interaction plans. In 2024 IEEE International Conference Robolics and Aulomation (TCRA), pp. 6904-699 6911. TEEE, 2024b.
Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2act: Predicting point tracks from internet videas enables generalizable robot manipulation. In European Conference on Springer, 2025.
Computer Vision, pp.
Judith Bütepage, Michael J. Black, Danica Kragic, and Hedvig Kjellström. Deep representation learning for human motion prediction and classification. In 2017 TEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1591 1599, 2017. doi: D0.1109/CVPR.2010
Judith Bütepage, Hedvig Kjellström, and Danica Kragic. Anticipating many futures: Online human motion prediction and generation for human--obed interaction. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 4563-455 4570, 2018.
doi: N..1109ICCR..0015
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Nicbles. Activitynet: A large- scale video benchmark for human activity understanding. In Proceedings of the iees conference on computer vision and pattern recognition, pp. 961 1970 2015.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. arXie preprint arXiv:2109. 10852, 2021.
An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language model. arXiv preprint arXiv:2,66.0111 01584, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: tttps://Imsys.. org/blog// blog/ 30-vicuna/
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic- kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), pp. 720 736, 2018.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, al. Rescaling egocentric vision: Collec- tion, pipeline and challenges for epic-kitchens-1000 International Journal of Computer Vision, pp. 1-23, 2022.
Pengxiang Ding, Qiongjie Cui, Min Zhang, Mengyuan Liu, Haofan Wang, and Donglin Wang. Expressive forecasting of 3d whole-body human motions. ar preprint drXiv:2312. 2023.
Zicong Fan, Maria Parelli, Maria Eleni Kadoglou, Xu Chen, Muhammed Kocabas, Michael J Black, and Otmar Hilliges. Hold: Category-agnostic 3d reconstruction of interacting hands and objects from video. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 494-504, 2024.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast nelworks for video recog- nition. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 6202 6202-6211, 2019.
Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J. Black. ChatPose: Chatting about 3d human pose. In CVPR, 2024.
Guillermo Garcia- Hernand, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 409-419, 2018.
Mohit Goyal, Sahil Modi, Rishabh Goyal, and Saurabh Gupta. Human hands as probes for interactive object understanding. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3293 3303, 2022.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, He- una Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freiie et al. The' something something' video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 18995-19012, 2022.
Yutaro Honda, Rei Kawakami, and Takeshi Naemura. considering interaction between players. In BMVC, 2020.
motion prediction in competitive fencing
13

---

Published in Transactions on Machine Learning Research (08/2025)
De-An Huang, Shijia Liao, Subhashree Radhakrishhan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed lemporal-localliztion assistant. arXiv preprinl arXim:2403.1000 19046, 2024.
Kumara Kahatapitiya, Amurag Arnab, Arsha Nagrani, and Michael S Ryoo. Victr: Video-conditioned text representations for activity recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 18547 18558, 2024.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh, Prismatic vlms: Investigating the design space of visually-conditioned language models. In Forty-first International Conference on Machine Learning. PMLR, 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafacl Rafailov, Ethan Foster, Grace Lam, Pannag Sankeli, et al. Openvla: An open-source vision-language- action model. arXiv preprint arXiv:2406.092: 2024.
KlingAl. Kling ai. https://klingaat com// 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural informalion processing systems, 35:22199 22213, 2022.
Adriana Kovashka and Kristen Grauman. Learning hierarchy of discriminative space-time neighborhood features for human action recognition. In 2010 IEEE computer socicly conference on compuler vision and pattern recognition, pp. 2046 2053. IEEE, 2010.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Denso-captioning events in videos. Proceedings of the IEEE international conference on compuler vision, pp. 706 715, 2017.
Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Mare Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10138 10148, October 2021.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 9579-9588, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Miao Liu, Siyu Tang, Yin Li, and James M Rehg. Forecasting human-object interaction: joint prediction of motor attention and actions in first person video. In Computer Eision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pp. 704-721. Springer, 2020.
Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognátion (CVPR), 2022.
Xin Liu, Silvia Pintea, Fatemeh Karimi Nejadasl, Olaf Booij, and Jan C Van Gemert. No frame left behind: Full vidco action recognition. In Proceedings of the IEEE/CVE conference computer vision and pottern recognition, pp. 14892-14901, 2021.
LumaLabs. Dream machine.
ai /ed/ddrrammmcacnn 2024.
Junyi Ma, Xieyuanli Chen, Wentao Bao, Jingyi Xu, and Hesheng Wang. Madiff: Motion-aware mamba diffusion models for hand trajectory prediction on egocentric videos. arXiv preprint 0rXiv:2409.02638 2024a.
Junyi Ma, Jingyi Xu, Xicyuanli Chen, and Hesheng Wang. Diff-ip2d: Diffusion-based hand-object interaction prediction on egocentric videos. Xixiv preprint or 04432445504370, 2024b.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. aiv preprint or arx::23366.04224 2023.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupla. R3m: A universal
visual representation for robot manipulation. arXiv preprint Akt0:2203.12600 2022.
OpenAL. Introducing OpenAI 01 preview, 2024.
URL /ttps:://ppeai.com//
Georgios Pavlakos, Dandan Shan, Nija. Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9826-9836 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Eric Xing, Ming-Hsuuan Yang, and Fahad S. Khan. Glamm: Pixel grounding large multi- modal model. The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmocap: Fast monocular 3d hand and body motion capture by regression and inlegration. arviv preprint urXiv:2008.08322 2020.
Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 9869 987899977.4888 2020a.
Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding humnn hands in contact at internet scale. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 9869-886777 2020b.
Gunnar A Sigurdsson, Olga Russakovsky, and Abhinav Gupta. What actions are needed for understanding human actions in videos? In Proceedings of the IEEE international conference on computer vision, pp. 2137 2146, 2017.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint XrXi::2408. 03314, 2024.
Kihyuk Sohn, Honglak Lee, and Xinchon Yan. Learning structured output representation using deep con- ditional generative models. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc., 2015.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
Jason Wei, Xuczhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural in information processing systems, 35:24824 24837, 2022.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answerigg to explaining temporal actions. Tn Proceedings of the IEFE/CVE conference on compuler vision and pattern recognition, pp. 9777 0777-9786, 2021.
Haoyu Xiong, Haoyuan Fu, Jieyi Zhang, Chen Bao, Qiang Zhang, Yongxi Huang, Wenqiang Xu, Animesh Garg, and Cewu Lu. Robotube: Learning houschold manipulation from human videos with simulated twin environments. In Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pp. 1-10. PMLR, 2023.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.12 172400 2023.
Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang. Qi Wu, Zhizheng Zhang, and He Wang. Navid: Video-based vlm plans the next step for vision-and-langunge navigation. arXiv preprint arXi::2402.1585 15852, 2024a.
Zichen Zhang, Hongchen Luo, Wei Zhai, Yang Cao, and Yu Kang. Pear: Phrase-basee hand-object interaction anticipation. arXiv preprint K.i::2477211100 2024b.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instruc- tional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, et al. RI-2: Vision-language-actionn models transfer web knowledge to robotic control. In Conference on Robot Learning, pp. 2163 2183. PMLR, 2023.
16

---

Published in Transactions on Machine Learning Research (08/2025)
A Appendix
Here we provide additional details of the model implementation, dataset curation, and more qualitative results.
A.1 Dataset Details
Statistics. Table shows the statistics of all datascts used in our tasks. Note that H2O, FPHA and Ego4D are only used for zero-shot evaluation so there are no training samples.
Task
Dalaset
Training Samples
Validation Samples
Epic- Kitchen-55
8523
1894
Eitc-K0tc
24148
3513
VHP
H2O
503
FPHA
501
Epic-Kitchen-100
4018
3513
RBHP
Ego4D
8673
Table Data Statistios of VHP and RBHP task.
A.2 Pipeline Details
Here provide the illustrations of the training pipeline and the inference pipeline.
Linear
CVAE
Output Token Embeddings
Predicted Position
Token Probabilities
Large Language Model
Lhand
L
Hand Position Embed
Ground Truth Position
LLM Positional Embed
Input Token Embeddings
Actual Next Token
The
trajectory
is
<HANDD> <HANDD>
Figure Illustration training pipeline.
A.3 Other Ablation Studies
Scaling Model Improves the Prediction. To evaluate the scaling ability of our model, we use LLaVA- V1.5-7B and LLaVA-V1.5-13h as the LLM backbone of our model. We refer them as HandsOn VLM-7B and HandsOn VLM-13B. We show the performance both models in Fig. 7.
Zero-shot Chain-of-thoughtt We also conduct an ablation study on the zero-shot chain-of-though (Wei al., 2022; Kojima et al., 2022) prompting, shown in Fig. 8. We add "Let's think step by step" in the
17

---

Published in Transactions on Machine Learning Research (08/2025)
Linear
CVAE
Output Token Embeddings
Prodicted Position
Token Probabilities
Large Language Model
Sampling Strategy
Hand Position Embed
Predicted Position
LLM Positional Embed
Input Token Embeddings
Predicted Next Token
The
trajectory
is
<HAND> <HAND>>
Current Token
Figure 6: Illustration of inference pipeline.
front of the answer generated the inference stage. Contrary to our expectations, this approach yielded poorer results. This unexpected outcome may attributed to the limited diversity of our datasets.
Approach
ADE
FDEL
Reasoning Method
ADE]
FDE.
HandsOn VLM-7B 0.197
0.165
Direct Answer
0.197
0.165
HandsOw On VLM-13B
0.183
0.149
Chain-of Thought
0.220
0.191
Figure 7: Ablation study on the LLM backbone size.
Figure 8: Comparison of direct answer and chain-of- thought reasoning methods.
We evaluate them on the RBHP task.
A.4 More Visualizations
Failure Cases. We show some failure cases in Fig. 9. We observe failures when (1) there are someone's hands in the video, (2) the hands are occluded by objects, and (3) the target object in the instruction is not found in the frame.
Figure 9: Failure cases of the model: (left) multiple hands in the video, (middle) occlusions, and (right) the target trash can out of view.
More Qualitative Results. We provide more visualizations in Fig. 10.
18

---

Published in Transactions on Machine Learning Research (08/2025)
A.5 Prompt for VHP and RBHP Dataset Generation
We provide the GPT4 prompls for the RBHP dataset generalion pipeline mentioned in Section 4.2 in Table 10 and Table 11.
19

---

Published in Transactions on Machine Learning Research (08/2025)
GPT4 Prompt for Action-aware Image Description
You are a system generating descriptions for ego-centric human images. Human is doing household activities.
Provided with an image and a action narration of what is happening next, such as "use the scissor', you will describe the main item that you in the image, giving details but staying concise.
You can describe unambiguously what the item is, its color or relative position if clearly identifiable. You should also give out overall description of the scene, the environment where the action is taking place.
Table 10: GPT4 prompt for action-aware image description.
GPT4 Prompt for Implicit Action Generation
You are tasked with creating specific, indirect questions and instructions that human could use to identify and interact with objects basod on their names or detailed descriptions provided by users.
You will be given an action phrase which the human is going to do next, such as "use the scissor'.
Based on the descriptions, you must formulate responses that precisely hint at the action phrase without naming it directly. The aim is to enable the agent to deduce the correct action through these indirect cues, enhancing its ability to understand and execute tasks involving the object.
Please formal your generated respouse as a hand trajectory question, some templates are provided below for reference:
"Where should my hand move to I want to (implicit description)'
"Can you provide the hand trajectory for {implicit description "What is the recommended hand movement for fimplicit description)?"
Table 11: GPT4 prompt for implicit action generation.
Question Templates to Build VHP Datasets.
"Can you provide the hand trajectory?'
"What is the recommended hand movement?"
"What is the future hand trajectory in this vidco?"
"What is the predicted hand trajectory given current observations?"
WWhere should my hand move to I want to (explicit action)??
"Can you provide the hand trajectory for (explicit action)?"
"What is the recommended hand movement for fexplicit action}?"
Table 12: Question Templates to build VHP datasets.
Answer Templates to build VHP and RBHP datascts.
"Sure! Here is the hand trajectory {hand token sequence)."
"Based on the video, the hand trajectory is as follows: (hand token sequence]."
"The predicted hand trajectory is as follows: (hand token sequence)."
"Certainly! The hand trajectory for {action instruction] is as follows: (hand token sequence).'
"To {action instruction), the recommended hand trajectory is: {hand token sequence)."
Table 13: Answer Templates to build VHP and RBHP datascts.
20

---

Published in Transactions on Machine Learning Research (08/2025)
Prompt
Input
LLaVA-Pixel2Seq
WLLd
GT
What the recommended hand movement for transferring bottlc and its lid into their respective containers?
Where should my hand move to if want to transfer delicious pizza from its parchment paper to decorative dish?.......
Where should my hand move to if want to place a long, cylindrical baking tool into the wooden drawer?
Where should my hand move to if'l want to transfer the diced pieces of eggplant from the cutting board to the cooking vessel?
Where should my hand move to if'l want transfer delicious pizza from its parchment paper to decorative dish?
Where should my hand move to if want to add savory filling to the round pieces of dough?
What is the recommended hand movement for taking out espresso?
What the recommended hand movement for opening milk box?
Where should my hand move if want clean glasses?
What is the recommended hand movement for opening letter?
Figure 10: More Qualitative results for different samples from the validation split of our RBHP dataset (top in blue) and zero-shot evaluations on completely unseen datascts FPHA and H2O (bottom in pink). GT trajectories are provided for reference.
21


Reviews:
[{"review_id": "n8ScutNUUX", "reviewer": "Reviewer_tiy7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n++ The new extended tasks VHP and RBHP enable hand trajectory prediction from ego-centric videos based on different forms of language queries.\n\n++ The method demonstrates strong generalization and reasoning abilities on diverse real-world datasets.\n\n\nWeaknesses:\n\n-- \"... we open-source to the community ...\": Where can we find the mentioned open-source resources?\n\n-- Section 3.1: The explanation regarding the extraction of slow tokens is inaccurate.\n\n-- The visual backbone is frozen for this task. How would the model perform if the visual backbone is also fine-tuned?\n\n-- How are the hand trajectories tokenized? Are they tokenized using the employed LLM's native tokenizer?\n\n-- \"plauisble\" in Figure 1 is a typo.\n\n-- page 2: \"and and hand\"\n\n-- The notation of lamba_{txt} in page 4 is inconsistent with that in Eq. (1).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes HandsOnVLM, a video-based VLM for generating textual responses and predicting future hand motion from ego-centric video contexts given language prompts. The model casts hand trajectory prediction as auto-regressive next token prediction conditioned on fused video and language tokens. To consider different forms of language conditions, the traditional hand prediction tasks are extended to two new tasks: Vanilla Hand Prediction (VHP) and Reasoning-based Hand Prediction (RBHP). Experiments show that HandsOnVLM outperforms baselines on both VHP and RBHP tasks, demonstrating strong generalization and reasoning capabilities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Add the mentioned open-source resources.\n\n- Revise the above writing issues.\n\n- Results for fine-tuning also the visual backbone.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ft5R0Raa0w", "reviewer": "Reviewer_pB3k", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The paper is clearly and cleanly written, with intuitive figures and a well-organized appendix.\n- It shows that integrating multiple off-the-shelf components yields strong end-to-end hand-trajectory forecasting on both explicit and reasoning tasks\n\n## Weakness\n- Incremental novelty, all the components are off-the-shelf.\n- Niche task. The paper and methodology of approach seems very limited and ineffective to similar poblems.\n- Missing ablation study/motivation for fast tokens. (Maybe its not needed?)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### The paper presents HandsOnVLM, a unified vision-and-language model that forecasts hand trajectories from short video clips and natural-language instructions. \n---\nFor example, given the prompt “open a can,” a human’s steps might be:\n\n-- Grasp the can with the left hand\n\n-- Pick up the can opener with the right hand\n\n-- Position the opener’s blade on the lid\n\n-- Rotate the can while holding the opener steady\n\n---\n\nHandsOnVLM learns to predict exactly those motions in order, alongside any explanatory text.\n\n## Visual Tokenization\n\nA visual backbone processes T successive frames into T×M patch embeddings.\n\n---\n\nWe have two types of tokens to consider\n- Fast tokens – the per-frame average of M embeddings (one “fast” vector per frame) [For clarification It will be a single avergaed value over a frame's M token]\n- Slow tokens – M spatial tokens obtained by uniformly sampling s frames and applying 2D average-pooling (SxS) over their M×M grids\n\n---\n\n## Vision-to-Language Fusion\n\nBoth fast & slow tokens are projected into the LLM’s embedding space and fused with the text query (“open a can”).\nThe combined embeddings serve as the context for autoregressive decoding.\n\n## Autoregressive Decoding + CVAE Trajectory Head\n\nDuring generation the model alternates between predicting words and a special \"HAND\" token.\n\nWhenever \"HAND\" is emitted, its last-layer hidden state is fed into a lightweight CVAE decoder, which outputs the next 2D hand coordinate (e.g. the pixel location where the opener contacts the lid).\nThat coordinate is then re-embedded into the \"HAND\" slot so that all future predictions remain conditioned on the full motion history, preserving temporal and spatial consistency.\n\n## Evaluation Benchmarks\n\nPaper defines two new metric for the task.\n\n---\n\nVanilla Hand Prediction (VHP): explicit “how-to” commands (e.g. “open a can”) → trajectory forecasting\n\nReasoning-based Hand Prediction (RBHP): implicit, commonsense queries (e.g. “How would I reach to turn on the light switch?”) → plausible hand motions\n\n---\n\n\nPaper uses existing methods and models to approach this unique problem statement.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Need additional ablation study to justiy novel contribution (slow + fast token). If it's well establish method, please share the resource as i am not aware.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Iot9jod5yj", "reviewer": "Reviewer_EfpJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. A novel framework for hand position prediction.\n2. Proposed datasets are good.\n\n\n**Weaknesses**\n1. I didn't see clear difference between the newly extended tasks and the corresponding original tasks. Please state them clearly.\n2. The paper writing is somehow bad, making it hard to read.\n3. This work is more likely to be a technical work, especially the design of the model framework, which mostly involve the existing techniques.\n4. What is the future horizon? I did not see explicit definition in the paper.\n5. How did the authors extend the existing vocabulary with a new <HAND> token?\n6. Regarding the establishment of the proposed datasets, the authors use some existing models to obtain the 'pseudo-GT', how can the authors prove that such obtained info can be used as GT. The last sentence of the first paragraph in Sec 4.1 is not clear enough for me.\n7. In Fig, 4, I did not think the performance of the proposed model is better than the existing work, or in other words, we say close to GT. In the third scene, why did the proposed model predict one more trajectory?\n8. The human study in Tab. 6 cannot convince me. I highly suggest the authors can provide explicit user study with clear numerical results instead of giving similar numbers with similar positive and negative ranges. Besides, please state the difference between the ones w/ and w/o mark. Furthermore, the use study without comparing to the existing works does not make sense to me.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a new framework called HandsOnVLM. It leverages the inputs of videos and text prompts to predict the hand position in the next frame. The author also proposed new datasets and tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the **Weaknesses**.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)