HandsOnVLM: Vision-Language Models for Hand-Object Interaction Prediction

Authors: Chen Bao, Jiarui Xu, Xiaolong Wang, Abhinav Gupta, Homanga Bharadhwaj
Primary area: foundation or frontier models, including LLMs
Keywords: vision-language model, hand-object interaction

Abstract:
How can we predict future interaction trajectories of human hands in a scene given high-level colloquial task specifications in the form of natural language? In this paper, we extend the classic hand trajectory prediction task to two tasks involving explicit or implicit language queries. Our proposed tasks require extensive understanding of human daily activities and reasoning abilities about what is happening next given cues from the current scene. We also develop new benchmarks to evaluate the proposed two tasks, Vanilla Hand Prediction (VHP) and Reasoning-Based Hand Prediction (RBHP). We enable solving these tasks by integrating high-level world knowledge and reasoning capabilities of Vision-Language Models (VLMs) with the auto-regressive nature of low-level ego-centric hand trajectories. Our
model, HandsOnVLM is a novel VLM that can generate textual responses and produce future hand trajectories through natural-language conversations. Our experiments show that HandsOnVLM outperforms existing task-specific methods and other VLM baselines on proposed tasks, and demonstrates its ability to effectively utilize world knowledge for reasoning about low-level human hand trajectories based on the provided context.

Decision: reject
Reviewer scores: 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730694521484

thread_type:
official_review

<thread_object>
creator: reviewer_9wrp

<content>
summary:
The authors propose training a video-language model which can reason about hand trajectories (curves, not poses) given videos and user queries as language input. Two associated benchmarks, Vanilla Hand Prediction (VHP) and Reasoning-Based Hand Prediction (RBHP) are introduced. VHP consists of predicting hand trajectories given an input video segment and a clear description of the object to be manipulated and the action to be performed. RHBP consists of predicting hand trajectories given less straightforward language input for which more complex reasoning must be performed. The authors curate datasets for both benchmarks, and promise to release these to the community. Evaluation on kitchen settings, as well as zero-shot evaluation on not (fully) kitchen-related datasets for the VHP benchmark shows the superiority of the method against state-of-the-art baselines.

strengths:
The paper is well-written.

The proposed method has useful applications in research and industry.

The authors promise to release two new, relevant benchmarks to the community.

The proposed method is evaluated on multiple datasets and makes qualitatively sound predictions on unseen datasets.

The proposed method outperforms most state-of-the-art architectures.

weaknesses:
The RBHP benchmark only includes kitchen scenes, and numerical validation is hence performed on kitchen settings only.

A comparison with a static-frame version of the proposed architecture would be appropriate to ensure fairness in Tabs. 1 and 2, as the baselines do not have access to the context provided by the observation history. Additionally, the need to use videos is a limitation, as in many settings we do not have a history of frames available.

Typing errors in lines 208 (l_hand) and 325 (FPGA).

The prediction is limited to curves. A version involving hand poses would be very useful for many settings.

questions:
Did you test your method with different numbers of input frames? What happens when you use only one frame? How well does your model handle the case of using a different number of tokens than what was used during training? Maybe the weakness I listed does not apply if the method performs well even when operating on a single frame.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702568657

thread_type:
official_review

<thread_object>
creator: reviewer_lkjo

<content>
summary:
The authors propose HandsOnVLM, a VLM-based framework to reasoning hand activities and predicting hand motions. In this framework, the hand trajectories  The HandsOnVLM achieves SOTA on the proposed benchmark for Vanilla Hand Prediction (VHP) and Reasoning-based Hand Prediction (RBHP) tasks.

strengths:
1. The proposed task appears engaging, as egocentric hand activities and motion prediction present challenging problems.
2.  The idea that encodes the hand as embedding is novel

weaknesses:
1.  The authors only compare their method with naive baselines and traditional methods for hand motion prediction. One potential additional baseline would be using foundation models for video prediction and then track the motion as predictions.

questions:
1. The authors report the results on Epic-Kitchen dataset which includes lots of ego motion. The egomotion would make the hand motions change significantly but they are barely unpredictable. How did the authors handle this problem?
2. How the <HAND> token is decoded to hand motions is not clear to me.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722184325

thread_type:
official_review

<thread_object>
creator: reviewer_179j

<content>
summary:
To avoid previous works which predict hand trajectories based on high-level language task command, this paper aims to extend the classic hand trajectory prediction task to two tasks involving explicit or implicit language queries. By developing new benchmarks to evaluate the proposed two tasks named Vanilla Hand Prediction (VHP) and Reasoning-based Hand Prediction (RBHP), this paper requires the model to acquire extensive understanding of human daily activities and reasoning abilities about what is happening next given cues from the current scene. To be specific, this paper propose a model named HandsOnVLM which generates textual responses and produce future hand trajectories through natural-language conversations. The experiments validate that the model outperforms existing methods on the proposed tasks.

strengths:
(1) This paper iseasy to follow and well-written. (2) The tasks proposed in this paper sound interesting and I do believe they play an important in egocentric-relavent tasks, such as VR/AR, robotics.

weaknesses:
Although the proposed tasks are interesting, I find that some critical details are either missing or require further elaboration in this paper, such as:
(1) Definition of Hand Pose: It remains unclear how the authors define the hand pose—whether it is based on joint positions, bbox or another representation. The paper merely includes two red and blue curves on images as qualitative results, which is ambiguous and lacks clarity.
(2) Details of H2O and FPHA Datasets: The paper provides insufficient information about the specifics of the H2O and FPHA datasets, particularly regarding the labels used in the study. This omission makes it challenging to fully understand the data, leaving readers to infer these details from the training objectives alone.
(3) Experiments: it's difficult to observe stable improvement from the experiments (such as in Table.1)

questions:
In A.1, the training set of VHP and RBHP only contains Epic-Kitchen, as far as I know, this dataset does not contain hand pose labels. So where does the authors obtain the hand pose labels?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732049441716

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response and appreciation

comment:
We thank all the reviewers for their detailed and thoughtful comments.
We are glad the reviewers found the paper easy to follow, well-written (179J), the proposed egocentric vision tasks relevant to VR/AR and robotics (179J, Lkjo,9WrP), and the experimental evaluations performant (9WrP). 

We respond to the comments of individual reviewers below and summarize common clarifications and new results that we have added in order to clarify some of the questions. We have also revised the paper to incorporate these clarifications and results (please find our modifications in blue on the pdf)

**Summary of revisions**: We summarize changes to our manuscript below; these changes have also been highlighted (blue) in the new version of the paper.
- We have added comparisons with video prediction followed by hand-tracking baselines suggested by reviewer Lkjo.
- We have added new experimental comparisons on single-image-based future hand trajectory prediction as suggested by reviewer 9WrP.
- We have added a new Ego4D RBHP dataset for zero-shot evaluation on the non-kitchen environments. 
- We identified a bug in our computation of the metrics (ADE, FDE, WDE) and have now revised the results of our method and the baselines. The bug in the metrics had made our approach look much weaker earlier, and after the resolution, we see that our method significantly outperforms almost all the baselines on all the metrics. 

Again, we thank the reviewers for their constructive feedback. We believe we have addressed all the comments and questions, but are happy to address any further clarifications from the reviewers.

Thank you,

Authors of HandsOnVLM (submission 960)
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a method for predicting future hand trajectories, based on language query inputs.

The reviewers agree the setting of the task is interesting, and thinks it may be beneficial for ego-centric related tasks in AR/VR and robotics.

The reviewers also raise several weaknesses, the key one being the lack of qualified baselines. 

After the discussion period, the reviewers are mixed, ranging from borderline reject (5), borderline accept (6) and weak accept (8).  Some points from the reviewers are not well-addressed, e.g. the point about the hand embedding.

Having read through the paper, author responses and reviews, the AC recommends to reject the paper at this time.  The AC agrees with reviewers on the limited and naive baselines, though from a different perspective, as outlined below:

Currently, the hand prediction is based only on the scene and observed trajectory, along with some cues from the text prompt.  This doesn't guarantee the claims about "reasoning". Any boost in performance may come from the increased capacity of the vlms etc. A proper ablation would consider no informative instruction at all, then only object name, then full instructions etc.  In a similar line of thought, the videos should be curated to have similar context but proceed in different ways (linked to different instructions), again, to reinforce the claims on reasoning.  

In addition, another purpose of the natural language query relates to affordance grounding and intentions.  The authors instead shift the setting into the prediction of hand trajectories, thereby bypassing several major line of work.  The included baselines are instead either too trivial or not meaningful to compare, e.g. raw video prediction / generation. 

A second weakness linked to the previous point is clarity in the task setting, whether or not it is meaningful for downstream settings or tasks and whether or not it should be in 2D or 3D.  As one reviewer points out it would be better to have hand poses - this is perhaps too nuanced for the current seting.  Without hand-poses, though, we are reduced only to bounding boxes and spatial regions to form the trajectory.  Yet, the concept of predicting entire future trajectories in 2D (vs. key landmarks in a scene, e.g. where the hand might be placed) is a bit puzzling - the trajectory is not physically grounded nor meaningful without 3D knowledge.