Title: Toward Human Deictic Gesture Target Estimation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Deictic Gesture

Gesture Target Estimation

Social Interaction

Social Artificial Intelligence
Abstract: Humans have a remarkable ability to use co-speech deictic gestures, such as pointing and showing, to enrich verbal communication and support social interaction. These gestures are so fundamental that infants begin to use them even before they acquire spoken language, which highlights their central role in human communication. Understanding the intended targets of another individual's deictic gestures enables inference of their intentions, comprehension of their current actions, and prediction of upcoming behaviors. Despite its significance, gesture target estimation remains an underexplored task within the computer vision community. In this paper, we introduce GestureTarget, a novel task designed specifically for comprehensive evaluation of social deictic gesture semantic target estimation. To address this task, we propose TransGesture, a set of Transformer-based gesture target prediction models. Given an input image and the spatial location of a person, our models predict the intended target of their gesture within the scene. Critically, our gaze-aware joint cross attention fusion model demonstrates how incorporating gaze-following cues significantly improves gesture target mask prediction IoU by 6% and gesture existence prediction accuracy by 10%. Our results underscore the complexity and importance of integrating gaze cues into deictic gesture intention understanding, advocating for increased research attention to this emerging area. All data, code will be made publicly available upon acceptance. Code of TransGesture is available at GitHub.com/IrohXu/TransGesture.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose dataset called GestureTarget which contains over 19000 annotated instances of source pointing, reaching, showing and giving (i.e. deictic) gestures with corresponding image masks for the target. The motivation for this dataset lies comes from the general unavailability of laerge deictic gesture datasets and hence blocking important research in the domain of deictic gesture prediction. The authors also propose TransGesture, a Transformer based model family (with cross attention between gaze source gesture) that integrates both human gestures and gaze to infer gesture targets in a complex scene. With ablation studies, the authors demonstrate the added value of gaze in predicting the target of the deictic gesture. Through various pretrained visual backbones, fusion methodologies, ablation studies, and comparison with SOTA VLMs, the authors study their design choices and validate the need for a Cross Attention module (i.e. JCA fusion) as well as the proposed joint loss function.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The proposed dataset contains valuable annotations of source gesture and target masks for furthering research in the Gesture Generation/Detection and Human Behavior modeling communities. The authors combine multiple multi-interaction datasets to build a larger dataset with over 19,000 annotations.\n- The authors measure that SOTA MLLMs models such as GPT 4-0, Claude and VLMs like GLaMM, OMG-LLaVA perform poorly on the proposed dataset, hence motivating the need for training dedicated models for gesture prediction\n- In table 1, the authors validate that gaze is a relevant conditioning variable to make better target predictions.  More specifically, Cross Attention based fusion (i.e. JCA) with gaze results in clear wins on top of Gesture-only baseline\n\n\n### Weaknesses\n-  Gestures are typically a temporal sequence, rather than just one image. The dataset is constructed using other Social Interaction datasets with videos. The construction of the dataset in this paper with images instead feels like a missed opportunity. Additionally, deictic gestures in single images may account for false positives (e.g. the person might not be pointing at someone, but performing beat gestures)\n- The choice of \"Accuracy of Gesture Existing\" as a metric might potentially not correctly represent the model's performance if the distribution of Gesture Existing and not-exisiting in the train/val dataset is not equal. It would benefit the community if the distribution of existence of gestures in the dataset is available to the readers and potentially using metrics such as F1-score or reporting both Precision and Recall would add value.\n\nQUESTIONS:\n* What is the distribution of existence of gestures in the dataset?\n* Would it be possible to share Existence F1-Score?\n* How were \"wrong-labeled frames during verification\" (l116) detected? Were the annotators shown the video to make that determination?\n* Could the authors share more detailed information (e.g. guidelines, annotation webpage screenshots, inter-annotator correlation) on the annotation process?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel task and accompanying benchmark dataset, GestureTarget, designed for evaluating the semantic target estimation of social deictic gestures. To tackle this task, the authors propose TransGesture, a model that incorporates a pretrained gaze target estimation module to extract gaze direction cues. By integrating gaze information, TransGesture aims to infer the actual intended target of a gesture, rather than relying solely on gesture features. Experimental results validate that gaze cues significantly enhance the model’s ability to accurately identify the object or person being pointed at.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The task definition is clearly stated and well-motivated.\n2. The proposed method is targeted and appears effective in addressing the defined problem.\n3. The introduced dataset is well-aligned with the task and has the potential to benefit related tasks in the community, such as interactive generation.\n4. The experiments are comprehensive, and comparisons across multiple pretrained models provide useful insights for future work on visual feature extraction.\n5. The paper is generally well-written and easy to follow.\n\n**Weaknesses**\n1. The paper lacks subjective evaluations (e.g., user studies) that could help assess whether the predicted targets align with human perception.\n2. While the authors emphasize the role of gaze in the model design, there is no clear ablation study isolating the contribution of gaze. Including such experiments would help substantiate its importance.\n3. Inference efficiency is not discussed. Highlighting runtime would help assess its applicability in real-world or interactive settings.\n4. Figure 3 is not referenced in the main text, which may confuse readers.\n\nQUESTIONS:\nSee \"Weaknesses\" in the review.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe study presents a new dataset, GestureTarget, by re-purposing four existing datasets related to gestures and social interaction to gesture target estimation. The authors then introduced TransGesture, a family of transformer models that fuse gesture and gaze cues. This fusion showed improvement over the gesture-only baseline.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- I find the task itself interesting (as well as the dataset). This type of information can guide models to calibrate to human attention and interactions. \n\n- The results make sense (e.g., gaze gives some prediction improvements), and the baselines, as well as the backbone variety, are good. The ablation for the current study is complete. Even though I would have liked to see a kinematic model (i.e., based on body models such as MediaPipe or SMLP-X).\n\n**Weaknesses:**\n\n- I find the choice of using RGB-based representations (especially for gestures) unintuitive. I believe that a kinematic module might be a better choice, either alone or in addition to the RGB raw data. \n\n- The evaluation is not cross-validated properly. The fixed 80 / 20 split is convenient but risks optimistic results if adjacent video frames land in both sets. K-fold cross-validation or at least a video-level split would strengthen the evidence.\n\n- No human-performance benchmark is reported. An annotation study (e.g. randomly sampled frames) would establish how close the best model is to human agreement.\n\nQUESTIONS:\n- What are the practical aspects of this work?\n- What is the difference between Y_exist and Y_mask? Is the Y_exist just a one-dimensional binary value? This is not clear.\n\n- Is there a human baseline for the target estimation performance? If so, what is the upper bound?\n\nLIMITATIONS:\n- Human-based evaluation (baseline)\n- A kinematic model that captures gestures effectively", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}