Toward Human Deictic Gesture Target Estimation
Authors: Xu Cao
Pranav Virupaksha
Sangmin Lee
Bolin Lai
Wenqi Jia
Jintai Chen
James Matthew Rehg
Venue: NeurIPS 2025 poster ()
Keywords: Deictic Gesture
Gesture Target Estimation
Social Interaction
Social Artificial Intelligence
Primary area: applications
TL;DR: Proposed the task of human deictic gesture target estimation and the first model for this task.

Abstract: Humans have a remarkable ability to use co-speech deictic gestures, such as pointing and showing, to enrich verbal communication and support social interaction. These gestures are so fundamental that infants begin to use them even before they acquire spoken language, which highlights their central role in human communication. Understanding the intended targets of another individual's deictic gestures enables inference of their intentions, comprehension of their current actions, and prediction of upcoming behaviors. Despite its significance, gesture target estimation remains an underexplored task within the computer vision community. In this paper, we introduce GestureTarget, a novel task designed specifically for comprehensive evaluation of social deictic gesture semantic target estimation. To address this task, we propose TransGesture, a set of Transformer-based gesture target prediction models. Given an input image and the spatial location of a person, our models predict the intended target of their gesture within the scene. Critically, our gaze-aware joint cross attention fusion model demonstrates how incorporating gaze-following cues significantly improves gesture target mask prediction IoU by 6% and gesture existence prediction accuracy by 10%. Our results underscore the complexity and importance of integrating gaze cues into deictic gesture intention understanding, advocating for increased research attention to this emerging area. All data, code will be made publicly available upon acceptance. Code of TransGesture is available at GitHub.com/IrohXu/TransGesture.

Toward Human Deictic Gesture Target Estimation
Xu Cao', Pranav Virupaksha", Sangmin Lee", Bolin Lai', Wenqi Jia',
Jintai Chen', James M. Rehg'
University of Illinois Urbana-Champar Champaann
Geergia Institute of Technology
KKorea University
"The Hong Kong University of Science and Technology (Guangzhou) aillimiss.....iiii...mmmmmmmmi edu
Abstract
Humans have remarkable ability to use co-speech deictic gestures, such point- ing and showing, to enrich verbal communication and support social interaction. These gestures
fundamental that infants begin to them even before they acquire spoken language, which highlights their central role in human communica- tion. Understanding the intended targets of another individual's 's deictic gestures enables inference of their intentions, comprehension of their current actions, and prediction of upcoming behaviors. Despite its significance, gesture target estima- tion remains an underexplored task within the computer vision community. In this paper, introduce Gesture comprehensive evaluation of social deictic gesture semantic target estimation. To gesture
novel task designed specifically for
address this task, we propose TransGesture, set of Transformer--aase target prediction models. Given an input image and the spatial location of person, our models predict the intended target of their gesture within the scene. Critically, our gaze-aware joint cross attention fusion model demonstrates how incorporating faze-following cues significantly improves gesture target mask prediction IoU by 6% and gesture existence prediction accuracy by 10%. Our results underscore the complexity and importance of integrating gaze cues into deictic gesture inten- tion understanding, advocating for increased research attention to this emerging area. All data, code will be made publicly available upon acceptance. Code TransGesture is available at htttt//co.oooooooo/o/aoneeeee
Introduction
Human social communication is profoundly multidimensional and multimodal. Beyond speech, gestures and body language play an important role in conveying intent, directing attention, and facilitating social interaction [ 2 3. For example, simple pointing gesture can silently direct someone's 's gaze toward a distant object, implying the intention of "look at that"; simple giving gesture can pass business card to someone, implying the intention of "remember me". Such deictic gestures (e.g., pointing with a finger or transferring an object with hands) enable people to share focus and coordinate actions without speaking. The ability to interpret these gestures is fundamental to human social capability we can easily infer what another person is referring to when they present deictic gestures 4 (5 Endowing machines with similar understanding of human gestures would significantly advance human-Al communication and embodied intelligence, enabling applications from assistive robotics to mental monitoring tools where AI agents must respond to non-verbal gesturing cues
g
However, despite decades of progress in object detection and action recognition, the specific task of estimating gesture targets inferring what a person is gesturing to has remained largely unexplored.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

SIRINS
Suul ject
Target
Target
Sublecc
Subbject
Targer
betect soson a searmn praffe Luler und bedaet pansensu clrh olavell soleshed eoanng susdoan building win greenery in the bacignound recaking on orange from another porss..
Suls parsen io who meo bltch saceet gleen appaars or crobeeniieg gain..
Sabeet persss the Highligmeed image long They ussted 199is, genuring with their honds as eedlaing or d douuiing somwthing cwrmgegnappatil
politing le engaged setiity
Figure 1: Gesture Target Estimation: the first row is the input image and the location of subject person; the second row is the groundtruth mask of the target; the third row visualizes the gesture interaction between subject person and target. The fourth row the text description of subject person.
Prior work has primarily focused on gesture classification [12 14 or synthesis [ 16 17 [8]
rather than on interpreting their referential intent. Additionally, gesture target estimation is particularly challenging. Unlike standard object detection or segmentation. it requires reasoning across multiple pieces of information, including body pose, gaze direction, spatial context, and occlusions. A pointing gesture might refer to a nearby object or a distant landmark, and the intended referent may be partially hidden. Compounding the challenge, there exist public datasets for this task, limiting progress in the field.
In this paper, we address the previously mentioned gaps by introducing GestureTaareet novel benchmark dataset for deictic gesture target estimation, and a Transformer-baser gaze-aware gesture model, TransGesture, that learn to predict the targets of human gestures by leveraging multimodal cues. GestureTarget is, to our knowledge, the first large-scale dataset focused on images of people engaged in deictic gestures (such pointing, reaching, showing object) with annotations of the intended target of each gesture. Each data instance in GestureTaare Target consists of an image containing a specified person-of- of-iterrettt who is performing a gesture, along with bounding box and segmentation mask annotation indicating the target of that gesture. By collecting and curating this benchmark, we aim to facilitate the quantitative evaluation of models on the gesture target estimation task and drive progress on this underexplored problem.
Building on this dataset, we propose TransGesture, group of Transformer- baseline models designed for gesture target prediction. Inspired by the psycholinguistic and cognitive theory in the relationship between gaze and deictic gesture 199 20 2022) we incorporate gaze target prediction to guide the estimation of gesture target. By fusing gaze direction via multi-layer joint cross attention, the model can disambiguate the intended gesture target even in complex scenes. We term TransGesture as gaaze-awar gesture models" because they have a pre- pre-trained trained gaze target estimation branch (on the GazeFollow [23] dataset). They are then finetuned with large amount of gestural data from the Gesture Target dataset. TransGesture models are designed to be adaptable, serving as a basis for various deictic gesture understanding tasks. Experimental results show that our approach of incorporating gaze cues significantly improves performance in identifying deictic gesture targets, validating the intuition that "where they look" helps convey "what they point at." Our contributions can be summarized as:
(1) We propose TransGisturr,, a group of Transformer models that integrates human gesture and gaze social cues through large-scale frozen visual encoder and applys joint cross attention fusion mechanisms to accurately infer gesture targets in complex visual scenes.
(2) We demonstrate that incorporating gaze target estimation an auxiliary modality can signifi- cantly improve understanding of deictic gesture targets, highlighting the importance of gaze as a critical cue in understanding nonverbal human communication.

---

(3) We introduce Gesture Target, a new task and dataset designed for deictic gesture target estimation, containing over 20K annotated instances of pointing, reaching, showing, and giving gestures with corresponding target mask annotations.
2 Related Works
Hand and Mind Gestures are a core channel of human communication, continuously shaping and enriching spoken language. Developmental studies show that infants' early spontaneous gesticulation can reliably predict the timing of later linguistic milestones [24, 25 26 27). McNeill argues that gestures externalize imagery that words alone cannot always capture, so speech and gesture must be analyzed together to expose the workings of thought 28 29). To distinguish different gestures, Kendon situates gesture functional continuum that runs from spontaneous co-speech gesticulation, through language- like gestures, to pantomimic emblems, and finally fully conventionalised sign languages (30 31. Moving rightward along Kendon's 's scale: (1) Dependence accompanying speech diminishes. (2) Linguistic structure hecomes more explicit. (3) Idiosyncratic movements give way to socially regulated symbols [32] Most computer vision research has focused the latter
two categories - emblematic gesture 33
and sign language recognition [36] 37 38]. In contrast, spontaneous co-speech gestures, especially the deictic gestures, remain largely unexplored, leaving crucial gap in the computational understanding of human communication.
34
13 35
Gesture & Gaze Modeling Gesture recognition and understanding is a well-know and widely explored task in the computer vision community, with many introduced datasets and models. Notable datasets for gesture recognition include LD-ConGR, featuring 542 RGB-D videos representing 10 gestures [12,, HaGRID, which focuses on high-resolution image based gesture recognition across 18 hand gesture classes [13], and EgoGesture, with over 2 million frames from 50 subjects, mcant for egocentric interactions with wearable devices [39]. Recently in social gesture understanding, SocialGesture introduces a large-scale multi-person gesture dataset with more than 42,000 instances of social gestures [40]. Almost all gesture models focus on classification, including CNN-based
models [41 42 and Transformer-bant
models
Similar to gesture analysis, interpreting gaze is important for human behavior understanding. Gaze- Follow 233, Atdetonttetioonaaare (46) and ChildPlay (47 introduce the gaze-following task, where the task requires models to predict the location in a provided scene that target person is looking at. Most approaches, including the original GazeFollow model, have taken fusion-based approach, processing elements such depth, pose, head position and head keypoints separately and then combining them to yield a gaze prediction [48] 49 50 51 (2). More recently, Gaze-LLE [53] moved away from multi-branch designs by utilizing frozen, pretrained visual encoder and decoder condi- tioned with head position information to make aodividual-speciff gaze predictions. However, there is limited exploration of the relationship between gesture and gaze in model development [54 55].
3 Gesture Target Estimation
We introduce GestureTarget a new benchmark under CC BY-NC 3.0 License for simultaneous localization and recognition of deictic gesture targets. To overcome the sparsity of deictic gestures in natural images, we repurpose four existing multi-person interaction datasets-VCR [56). Werewolf Among Us (57), SocialGesture 40) and Social-IQ (58 each of which already provide human and object bounding box annotations. The annotation process is described as follows: (i) Frame selection. We manually identify representative frames containing clear deictic gestures from videos. (ii) Pseudo-annotatiin
We run YOLOvII's segmentation and a human keypoint detector [59] to produce initial masks and bounding boxes. (iii) Human verification. Our author-annotaton review each image, confirm or correct the subject person's 's body and head boxes, and select the correct semantic mask of the gesture target. Images with incorrect human boxes or keypoints are discarded. (iv) Dataset finalization. After removing roughly 30% of wrone-labeled frames during verification, Gesture Target comprises 20K instances. Each instance includes: Body and head bounding boxes of the gesturing subject; Bounding box and semantic segmentation mask for the gesture target; A paired textual description linking the subject person to their target. Our annotation pipeline leverages existing resources to efficiently create rich dataset for deictic gesture target estimation.
3

---

Gaze Decoder
Subject Person
2 gaze
Trapel Cormur
Head Patch Encoding
jjste
Greas Affert on
Atteri Tranf Block
Creas Ather Rigk
Mask Head
DINOv2
mask
Ieep
Body Encoding
Body Subject
Troett
Blcck
Exist Head
Trisif
pesture
Y Texist
gesture
Gesture Decoder
We introduce TransGesture, a new framework for gesture target estimation under the guidance of human gaze. We first use frozen DINOv2 to extract scene tokens and then project them to Zgaze and Zgstture with two linear layers. After adding head or body patch encodings, Zgaze and Zegesture updated by separate transformer-bant based decoders. The outputs fused by multi- block joint cross attention transformer modules and finally upsample and decode by convolution mask head.
Figure 2:
Figure I iillustrates example annotation from Gesture Given input RGB image X E R3xH XTV our annotations include: the bounding box of the subject's body (Xsubject E R²); eye
keypoints (left eye Xeyee E 2R22, right eye Xeye RR);; hand keypoints (left hand Xihand E IR2 right hand Xyhand E IR²); and textual description (Xtext) detailing the subject's appearance, location, and action. For gesture target annotations, we provide a bounding box (Ytarget E I),, a semantic segmentation mask mask & HW),, ), location, and categorical existence label (Yexist) indicating whether the gesture target present, absent, or out of frame. The goal of the task is predicting Yexist and Ymak-
text description (Ytext) including the target's attributes and
How Do SOTA Vision-Language Models Perform on PesstreeTargett Additionally, to demon- strate the utility of GestureTaareet, we evaluate the zero-shot performance of state-of-the-art Multi- modal Large Language Models (MLLMs) on predicting the existence of deictic gestures. GPT-4o (60) achieves deictic gesture existence accuracy of 70.21 %. while Claude-3.1 3.7 [61] obtains 68.48% -fate-of-the-art open- vocabbacyy segmentation VLMs (GLaMM (62) OMG-LLaVA 63) all achieve <5.0% IoU for the gesture target mask estimation task. These results indicate that even current SOTA MLLMs and VLMs struggle with reliably identifying deictic gestures and their target, highlighting the importance of designing new foundation model for this task.
4 Method
4.1 Model Architecture
Figure 2 illustrates our transformer-based. gaze-aware gesture target estimation model TransGesture. The framework consists large-scale frozen visual encoder and two transformer-basee decoders dedicated to gaze and gesture representation learning, respectively. Their outputs are subsequently fused through gesture-gaze joint cross-attention module produce a segmentation mask of the gesture target and a binary existence prediction.
Shared Backbone. Inspired by recent advances VLMs and gaze estimation [66 53], we utilize visual representations from a large-scale, frozen pretrained feature extractor as input to our downstream gesture target estimation model. While such extractors encode rich semantic information, gesture target estimation fundamentally relies fine- fine-grained
and prior work on human pose

---

spatial reasoning across instances. We empirically found that DINOv2 67 based encoders in preserving intra-instance token coherence, leading more accurate gesture target estimation. While CLIP excels at semantic discrimination, its token embeddings are less spatially consistent across instances in our task setting.
outperforms CLIP-
Specifically, for an imput RGB image X E R RHax Wixx the frozen visual encoder provide the visual feature Z (X()), where Z kixW Xdume, H and W is the height and width of the output feature map, defined by H Hin/ and W = WWn/P, where P is the patch size of the token.
Head and Body Patch Encoding. To incorporate the head position and body positions of a subject person who perform gestures, we construct two downsampled, binarized masks Masad E R x and Mandy "x While Maody is extracted by the subject's person bounding box Xsubjecct RetinaFace-RessNe55 668 predicts Mhead and uses the eye key points Xleye and Xraye to match the head to extracted subject bounding boxes. Two learnable patch encoding PE dgaee and PEaddy E dgesture are used to enhance selected region for downstream gaze and gesture decoders. Consequently, have:
Zgaze = + Mhead PEhcad
Zgesture Z +
Mbody PEbody
(2)
where Zaaze E R Wxdw and Zgesture E Nx Xdgoum are feature maps projected from Z, addi- tional absolute 2D sinusoidal position embeddings (69 are added to each visual feature before adding the position encoding.
Transformer-based Gaze Decoder. We employ frozen gaze decoder, pre-trained on the GazeFol- low dataset 233. to extract feature representations of gaze targets. The decoder consists of stack of Transformer layers designed to model spatial dependencies relevant to gaze estimation. takes as input the head-conditioned feature map Zeazes which encodes the positional and contextual visual in- formation centered on the subject's head. Leveraging self-attention, the decoder captures fine-grained relationships between different spatial regions in Zgaze enabling it to infer the likely direction and location of the person's 's gaze. We freeze the decoder to preserve its pre-learned self-attention patterns and prevent overfitting during training on our gesture dataset.
Transformer-based Gesture Decoder. To refine the feature representation for gesture understand- ing, introduce other lightweight learnable Transformer layers that capture spatial dependencies critical for gesture existence classification and target mask prediction. The input Zgesture is first reshaped from H x W x dgesture
HW dzesture where HW is the length of visual tokens and dyesture is smaller feature dimension. Then learnable gesture existence token lexist R. is concatenated to gessuure
Zeesture
[fxxists L1 11,12, 12, [. E
R HW x ddaam
Zyynee
geeauure is used as the input of the gesture transformer decoder. A 2-layer MLP takes fexist of the last layer as the input and output the existence score of deictic gesture.
Gesture-Gaze Joint Cross Attention (JCA). To enhance the feature integration of gaze and gesture, we apply joint cross attention layer to fuse tokens. Assume Fgaee E I HW X and Fgesture E R HW xdycaue are the output of the Transformer-based gaze and gesture decoder. A projection function f is used to alignment dimension of Fyuee to Fgesture- Thus, the gesture-gazz joint feature defined by:
Pjoint
[81, 83, WWWW, , 11, b2 ba H.W E IR 2HW x Xdgotuc
(4)
(F(p)))
Fgaum
Then, the module performs cross-attention between Egesture and F fointi
Q F gesture Wa.
K Fjjiin Wks ls
= =-jiann W.:
(5)
= Softmax (QKT/dggsure))
JointCrossAttention/ (Fgesture: Fjoint) AV
5

---

Gaze
Gaze
Input Image
Visual Encoder
Com
Decoder
Decoder
Come
Decoder
Gesture
Gesture
Decoder
Decoder
(a) Gesture Only
(b) Direct-Fusion
(c) Conv-Fusion
Figure 3: Baseline models for gesture target estimation.
Moreover, as self-attention, we also use multiple heads when we apply gesture- gaze joint cross attention in the Transformer model. Layer Normalization and feed- forward network is also connected after the joint cross attention. The design simplifies the fusion process without weakening the interaction between gaze tokens and gesture tokens.
Joint Learning Objective. We train our model using a joint multitask objective that combines pixel- wise binary cross-entropy loss for gesture target segmentation and focal loss for gesture existence prediction. The ground truth for the segmentation task is binary mask Yaakk E IR # while the gesture existence supervised with binary label Yexist- We use focal loss address class imbalance in the existence prediction task. The overall training loss is defined as:
Latal (1 - 3)) LBae(Ymask mask) + B. Lrocal (Yexist; Texist)
4.2 Model Training
Stage 1: Pre-training for Gaze Decoder. We first pre-train the gaze decoder on the GazeFollow [23 to precisely extract gaze-related features. Specifically. modify the original model architecture by removing all gesture-related branches and appending a dedicated gaze prediction head after the gaze decoder. The training objective is formulated minimizing the pixel-wise binary cross-entropy loss for gaze target heatmap prediction. An auxiliary head regression loss [52] for 2D spatial guidance is also used. During pre-training, we freeze the visual encoder parameters and exclusively optimize all weights associated with the gaze decoder, including the head positional encoding 53
Stage 2: Fine-tuning End-to-End. In this stage, we always keep the visual encoder weights and gaze decoder frozen, and continue update the weights in the gesture decoder and fusion modules. The training objective is the joint multitask loss function defined in Formula
4.3 Baselines
Apart from the gesture-gaze joint cross attention fusion module, we also introduce several baseline fusion strategies designed for gesture target estimation. These baseline models serve initial benchmarks and can be considered as different varient of TransGesture, providing insights into the effectiveness of incorporating multimodal information such gaze direction and visual context. Each baseline employs a shared frozen visual encoder and separate decoders for gaze and gesture targets, but they differ in their approach fusing gesture and gaze cues.
Direct Fusion. Direct fusion integrates gaze and gesture visual features via straightforward concate- nation. Specifically, the visual encoder extracts visual features, and the gaze decoder provides gaze features. These two feature vectors are concatenated along the channel dimension and then passed to gesture decoder. Formally, the fusion can described as: Frasion = Concat (Fgurey F Fgesuur)..
This simple concatenation facilitates the basic integration of multimodal information but does not explicitly model interactions between modalities.
Conv Fusion. Deep convolution fusion method employs more sophisticated convolution based fusion strategy designed to capture richer multimodal interactions. After obtaining visual features and gaze features these features are concatenated and subsequently processed through convolutional layers (denoted as Conv) in residual blocks (ResBlock) [70] better integrate and refine multimodal interactions. The single step fusion operation can be formally represented as:

---

Flusion ResBlock( CessBlock(Concatii Paaees F gestture)) +
gesture
The Conv Fusion approach thus explicitly learns interactions between gaze and visual contexts, enhancing the ability of the model to accurately predict gesture targets in complex scenes.
Experiments and Results
5.1 Implementation Details
All models trained and evaluated the proposed Gesture'Target dataset, using 80:20 split for training and testing. Following practices from PASCAL VOC Action Recognition [71. where ground-truth bounding boxes for subjects are provided during both training and evaluation, similarly assume availability of the subject person's 's bounding box at both train and test time. For gaze decoder pre-training, we the official GazeFollow training dataset 23
Evaluation Metries We evaluate the gesture target estimation performance by two main metrics: Existence Accuracy, whether the intended target is correctly detected as present in the scene, and IoU. the localization overlap with the true target.
Technical Details All models in our study generate gesture target masks with resolution of 128 x 128. We freeze the visual encoder during training, using its default image and patch sizes. For the gesture-gazz gaze fusion module, we employ three Transformer layers featuring joint cross- atssetttten,on, each configured with 16 attention heads and a dizeesinnn MLP. To enhance model robustness and generalization, we apply diverse augmentation techniques during training, including head/body bounding box jittering, color jittering, random resizing and cropping, random horizontal flipping, random rotations, and random masking of scene patches. All models are trained for 25 epochs using the Adam optimizer and cosine learning rate scheduler with an initial rate of le-3 and batch size 32, followed by an additional 5 epochs with a reduced learning rate of 1e-5, All experiments are conducted with I NVIDIA H100 GPU.
5.2 Main Results
Table compares comppres the performance of different fusion strategies and visual encoders for gesture target estimation, reporting Existence Accuracy, which indicates whether the intended target is correctly detected as present in the scene, and IoU, which measures the localization overlap with the true target. We analyze the effectiveness of the proposed Gesture-Gaze Joint Cross-Attention (JCA) fusion module across various visual backbones. JCA fusion consistently improves performance for all tested image encoders, outperforming other fusion strategies in both existence accuracy and IoU. For instance, with the CLip- encoder, the baseline (gesture-only, no gaze) achieves an existence accuracy of 79.21 % and an IoU of 54.37% %. Incorporating JCA improves them to 87 87.66% and 56.65%, respectively an absolute gain of +8.5% in accuracy and +2.3% in IoU. similar trend is observed with the SigLIP-Bast Base model. Notably, the DINOv2-Bard backbone, which starts with relatively low-pe oww-prforming baseline, sees the largest improvement: JCA raises its existence accuracy from 79.19% to 89.01% % and IoU from 52. 14% to 58.48% (a +9.8% and +6.3% jump, respectively). Across all encoders, JCA yields the best performance among the fusion modules. In comparison, convolution-basee fusion techniques Direct Fusion and Conv Fusion offer only modest gains, and can even degrade target estimation performance if not carefully designed, These results highlight the effectiveness of the joint cross-attention module in integrating gaze and gesture cues, leading to higher accuracy and more precise localization across all visual encoders.
In addition, we also recruited 10 volunteer participants and asked them to perform two tasks on randomly selected subset of 50 images from our test set: (1) Gesture existence detection (binary classification); determining whether the subject in the image is performing deictic gesture; and (2) Target identification: a gesture is present, identifying the intended target in the image by clicking on the corresponding object or person mask. For the gesture existence detection task, the average human performance was 87. $7.00%, with a maximum of 90.00%. For the target estimation task, we measured performance by the IoU between the region selected by each participant and the ground-truth target
7

---

TransGesture series Models
Visual Encoder
#arrmss
Resolation/Patch Size
Fusion Method
Gaze Guidance
Exist Ace (%)
IoU (%)
Gesture Only
79.21
54.37
Direct Fusion
83.23
53.70
Carge-arge**
Conv Fusion
85.19
56.07
JCA Fusion
87.66
56.65
Gesture-Only
8
75.96
54,58
Simple-Fusion
83.52
54.24
SigLIP-Base
86.0M
512/16
84.32
JCA Fusion
84.48
57.54
Gosture-Ouly
x
76.70
55.34
Fusion
SigLIP2-Base
86.0M
512/16
Conv Fusion
84 64
54,78
JCA Fusion
85.13
56.13
Gesture-Only
79.19
52.14
Direct Fusion
80.19
53.66
DINOv2:
Conv Fusion
88, 10
54.63
JCA Fusion
89.01
58.48
Human Baselines
Human Groups
Exist Ace (%) 1
ToU (%)
Average of subjects
66.40
Maximum of all subjects
90.00
80.00
Table Exploring the influence CLIP/SigLIP-Dase based visual encoder and DINOv2 in gesture target estimation with different gaze and gesture feature fusion strategies. In human baselines, we compare the average human performance and the maximum human performance.
Original Image
Sig.
Sigl
Buse
Figure 4: Compared with CLIP, SigLIP, SigLIP2, DINOV2 in token-wise wise affinity map visualization.
mask; the average human Target IoU was 66.40, with a maximum of 80.00%. This human study provides upper bound achievable performance.
Comparison of Different Visual Backbones. Table I compares the performance of several pre- trained visual encoders, including CLIP (72.. SigLIP 7 SigLIP2 774,, and DINOv2 [67], all trained on large-scale datasets. Among them, DINOv2 achieves the best performance on our task, demonstrating strong
feature extraction capabilities. As shown the token affinity visualizations in Figure 4 DINOv2 preserves notably stronger intra instance token affinity, resulting in more coherent and spatially structured representations. contrast, CLIP- and SigLIP-basee models exhibit weaker spatial coherence, indicating limitations in modeling spatial relationships in complex social scenes involving multiple humans.
Ablation Study
Impact of Model Scale. To assess how backbone size influences gesture target estimation, we evaluated visual encoders with progressively larger parameter counts, as shown in Table Increasing the encoder size leads to modest but consistent improvements in both existence accuracy and mask IoU. However, this trend plateaus DINOv2-Larg

---

Image Encoder
#Params
Exist Ace (%) f
IoU (%) f
Exist Acc (%) t
IoU (%)
SigLIP2-Bass Pase
86.0M
85.13
56.13
0.01
84.74
56.24
SigLle2-Large 303M
88.36
57.27
0.1
86.44
58.62
DINOv2-Small
22.1M IM
87.52
56.77
0.5
89.01
58.48
DINOv2-Base
86.6M
89.01
58.48
0.9
87.36
55.28
DINOv2-Large 304M
89.85
58.26
0.99
85.50
56.49
Table 2: Ablation of different scale of pretrained Vision Table 3: Ablation of 3 in the loss function Transformer Encoder.
(use DINOv2-Bas. image encoder).
Impact of Loss Hyperparameter. In Table 3 we study the impact of hyperparameter 3 in the total loss function (Formula 6) We observe that both decreasing and increasing 3 lets the model perform poorly in the existence classification and target mask prediction tasks, thus demonstrating the complementary nature of these two training objectives.
Impact of Freezing Individual Modules. We ablated TransGesture by selectively freezing its constituent components; the results are summarised in Table Finetuning the DINOv2 visual backbone yields only marginal uplift in existence accuracy and measurable gain in target mask loU, yet inflates the number of trainable parameters by an order magnitude and lengthens training time substantially. These findings suggest that DINOv2's pretrained representations are already near-optimal for gesture-target estimation, and further finetuning offers a poor cost-benefit trade-off.
Frozen Image Encoder
Frozen Gaze Decoder
Frozen Gesture Decoder
Trainable AParams
Exist Ace (%)
IoU (9)
X
X
94.21M
91.07
57.22
X
7.63M
87.48
57.92
X
11.55M
85.56
57.52
5.07M
89.01
58.48
Table 4: Ablation experiment on frozen modules in the proposed TransGesture framework.
Inference Efficiency. We also measured the inference runtime of the TransGesture model (using DINOv2 encoder) compare the computational cost of its different fusion architectures 518 x 518 images. The average runtimes were 0.0108s (Gesture-Only) 0.01 (Direct Fusion), 0.0131s (Conv Fusion), and 0.0110s (JCA Fusion). This demonstrates that incorporating JCA fusion adds negligible overhead of only 0.0002s compared to the gesture-only baseline. In practice, the computations for the gaze branch, which primarily involve few additional cross-attention layers, are lightweight relative to the overall cost of the Transformer-ban visual encoder.
5.4 Visualization
Figure contrasts three variants of our system: the gesture-only bascline, Conv Fusion, and the proposed gesture-gaze JCA Fusion. For gesture-only baseline, predictions frequently blur across large image regions or drift to distracting objects, and breakdowns are common when scenes contain more than three people. For conv fusion, the additional convolutional fusion layer suppresses some noise, yet masks remain coarse and often bleed beyond true object boundaries- coundariss-especiallll especially in cluttered, interactions. For proposed method that can jointly aligning gaze and gesture pinpoints the intended target with compact masks that closely trace the ground-truth silhouettes while ignoring irrelevant actors, even under occlusion, low light, or dense layouts. Failures arise only in the most complex scenes that demand high-level spatial reasoning (the last row). These qualitative results highlight our methods' clear advantage in precise gesture target estimation.
5.5 Demo
We implement an MILMM workflow that uses TransGesture as tool to support complex text queries from users and can integrate with different other social Al models. Details this agent showed in Appendix

---

Original Image
Target Ground-truth
No Fusion (Gesture-only)
Conv Fusion
JCA Fusion
Figure 5: Qualitative examples of gesture target estimation under different fusion strategies. Green bounding boxes indicate the gesture initiator, and red masks show the predicted target person.
6
Discussion
The practical applications of deictic gesture target estimation are diverse and impactful. In health- care, this technology can significantly enhance the analysis of non- non-verbal cues in clinical settings, particularly for early autism diagnosis. Since individuals with autism spectrum disorder often display differences in gesture use and interpretation, an accurate model for detecting these deictic cues could assist clinicians in identifying subtle communication deficits in young children, enabling earlier inter- ventions 75). Beyond the clinical domain, this work is crucial for improving human-Al interaction. Because deictic gestures are fundamental to natural communication, enhancing a machine's 's ability to understand them facilitates smoother, more intuitive collaborations [76). This is especially valuable for assistive robotics, where robots must interpret human intent provide effective support, and for virtual assistants operating in complex social or professional environments.
As with many activity recognition tasks, our work carries potential privacy implications. Therefore, while our dataset and models will be publicly released for research purposes, careful ethical consid- eration is essential-especiine especially when utilizing these resources in contexts involving accusations or consequential decision-making more detailed discussion is provided in the Appendix D
Conclusion
We investigated a novel problem: detecting deictic gestures and associating them with their intended targets. To address this challenge, we introduced a new Transformer- based architecture featuring joint cross-attention between gesture and gaze cues. Additionally, we contributed large-scale annotated dataset, providing images with labeled bounding boxes for subject persons, corresponding body locations, and semantic masks of gesture targets. Through extensive experiments, demonstrated the effectiveness and importance of our approach across two key tasks: deictic gesture existence prediction and gesture target estimation.
Acknowledgments
We acknowledge partial support from the Health Care Engineering Systems Center (Grainger College of Engineering, UTUC). We also extend our thanks Prof. Jianguo Cao (Shenzhen Children's Hospital) for his valuable discussion regarding clinical practice.
10

---

References
[1] Michael Arbib. How the brain got language: The mirror system hypothesis, volume 16. Oxford University Press, 2012.
[2] Holger Diessel. Where does language come from? some reflections on the role of deictic gesture and demonstratives in the evolution of language. Language and Cognition, 5(2-3):239-249, 2013.
[3] Sangmin Lee, Minzhi Li, Bolin Lai, Wenqi Jia, Fiona Ryan, Xu Cao, Ozgur Kara, Bikram Boote, Weiyan Shi, Diyi Yang. et al. Towards social ai: A survey on understanding social interactions. arXiv preprint arXiv:2409.1533 2024.
[4] Adam Kendon. Do gestures communicate? review. Research on language and social interaction, 27(3):155-200 1994.
[5] Oliver Herbort and Wilfried Kunde. How to point and interpret pointing gestures? instructions can reduce pointer-observer misunderstandings. Psychological research, 82(2):395-406, 2018.
[6] Allison Sauppé and Bilge Mutlu. Robot deictics: How gesture and context shape referential communication. In Proceedings of the 2014 ACM/IEEE international conference on Human- robot interaction, pages 342-349, 2014.
[7] Qi Wu, Cheng-Ju Wu, Yixin Zhu, and Jungseock Joo. Communicative learning with natural ges- tures for embodied navigation agents with human-in-the-scene. In 202/ IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4095-4102. IEEE, 2021.
[8] Martha W Alibali, Rebecca Boncoddo, and Autumn Hostetter. Gesture in reasoning: An embodied perspective. In The Routledge handbook of embodied cognition, pages 233-245. Routledge, 2024.
[9] Hannah VanderHoeven, Nathaniel Blanchard, and Nikhil Krishnaswamy. Point target detection for multimodal communication. International Conference on Human-Computer Interaction. pages 356-373. Springer, 2024.
[10] Yihao Liu, Xu Cao, Tingting Chen, Yankai Jiang, Junjie You, Minghua Wu, Xiaosong Wang, Mengling Feng, Yaochu Jin, and Jintai Chen. From screens to scenes: A survey of embodied ai in healthcare. Information Fusion, 119: 103033, 2025.
[11] Abdul Mohaimen Al Radi, Xu Cao, Fanyang Yu, Yuyuan Liu, Fengbei Liu, Chong Wang, Yuanhong Chen, Jintai Chen, Hu Wang, Yanda Meng, et al. Agentic arge-language-modde systems in medicine: A systematic review and taxonomy. Authorea Preprints, 2025.
[12] Dan Liu, Libo Zhang, and Yanjun Wu. Ld-congr: large rgb-d video dataset for long- long-distance continuous gesture recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3304-3312, 2022.
[13] Alexander Kapitanov, Karina Kvanchiani, Alexander Nagaev, Roman Kraynov, and Andrei Makhliarchuk. Hagrid-hand gesture recognition image dataset. In Proceedings of the IEEE/CVF Winter Conference on Applications Computer Vision (WACV), pages 4572-4581, 2024.
[14] Anton Nuzhdin, Alexander Nagaev, Alexander Sautin, Alexander Kapitanov, and Karina Kvanchiani. Hagridv2: Im images for static and dynamic hand gesture recognition. arXiv preprint arXiv:2412.01508, 2024.
[15] Zeyi Zhang, Tenglong Ao, Yuyao Zhang, Qingzhe Gao, Chuan Lin, Baoquan Chen, and Libin Liu. Semantic gesticulator: Semantics-awan co-speech gesture synthesis. ACM Transactions on Graphics (TOG), 43(4):1-17, 2024.
[16] Xingqun Qi, Hengyuan Zhang, Yatian Wang, Jiahao Pan, Chen Liu, Peng Li, Xiaowei Chi, Mengfei Li, Wei Xue, Shanghang Zhang, et al. Cocogesture: Toward coherent co-speech 3d gesture generation in the wild. arXiv preprint 46874 2024.

---

[17] Supreeth Narasimhaswamyy Uttaran Bhattacharya, Xiang Chen, Ishita Dasgupta, Saayan Mitra, and Minh Hoai. Handiffuser: Text-to-image generation with realistic hand appearances. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2468-2479, 2024.
[18] Haozhou Pang, Tianwei Ding, Lanshan He, Ming Tao, Lu Zhang, and Qi Gan. Llm gesticulator: leveraging large language models for scalable and controllable co-speech gesture synthesis. In Eighth International Conference on Computer Graphics and Virtuality (ICCGV 2025), volume 13557, page 1355702. SPIE, 2025.
[19] David McNeill. Gesture, gaze, and ground. In International workshop machine learning for multimodal interaction, pages 1-14. Springer, 2005.
[20] Kristine Lund. The importance of gaze and gesture in interactive multimodal explanation. Language Resources and Evaluation, 41:289-303, 2007.
[21] Marlou Rasenberg, Asli Özyürek, and Mark Dingemanse. Alignment in multimodal interaction: An integrative framework. Cognitive science, 44(1 (H(11)e11291) 2020.
[22] Gozdem Arikan, Peter Boddy, and Kenny R Coventry. The relative importance of language. gaze, and gesture in deictic reference. Journal of Experimental Psychology: Learning, Memory, and Cognition, 2025.
[23] Adria Recasens, Aditya Khosla, Carl Vondrick, and Antonio Torralba. Where are they looking? Advances in neural information processing systems, 28, 2015.
[24] Daniel S Messinger and Alan Fogel. Give and take: The development of conventional infant gestures. Palrer Quarterly (1982-), pages 566-590, 1998.
[25] Amanda Woodward and Jose Guajardo. Infants' understanding of the point gesture as an object-directed action. Cognitive Development, 17(1): 2002.
[26] Colwyn Trevarthen. Form, significance and psychological potential of hand gestures of infants.
In The Biological Foundations of Gesture, pages 149-202. Psychology Press, 2014.
[27] Irene Guevara, Cintia Rodríguez, and María Núñez. Developing gestures in the infant classroom: from showing and giving to pointing. European Journal of Psychology of Education, 39(4):4671-
4702, 2024.
[28] David McNeill. Hand and mind: What gestures reveal about thought. University of Chicago press, 1992.
[29] David McNeill. Gesture and thought. University of Chicago press, 2019.
[30] Adam Kendon. How gestures can become like words. In This paper is a revision of a paper presented to the American Anthropological Association, Chicago, Dec 1983. Hogrefe Huber Publishers, 1988.
[31] Adam Kendon. Gesture: Visible action as ufterance. Cambridge University Press, 2004.
[32] Susan Goldin Meedio-Meadowwaao and Diane Brentari. Gesture, sign, and language: The coming of age of sign language and gesture studies. Behavioral and brain sciences, 40:e46, 2017.
[33] Jun Wan, Yibing Zhao, Shuai Zhou, Isabelle Guyon, Sergio Escalera, and Stan Z. Li. Chalearn looking at people rgb-d isolated and continuous datasets for gesture recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 56-64, 2016.
[34] Liang Zhang, Guangming Zhu, Lin Mei, Peiyi Shen, Syed Afaq Ali Shah, and Mohammed Bennamoun. Attention in convolutional Istm for gesture recognition. Advances neural information processing systems, 31, 2018.
[35] Xin Zeng, Xiaoyu Wang, Tengxiang Zhang, Chun Yu, Shengdong Zhao and Yiqiang Chen. Gesturegpt: Toward zero-shot free-form hand gesture understanding with large language model agents. Proceedings of the ACM Human- Computer Interaction, 8(ISS):42-4499 2024.
12

---

[36] Lionel Pigou, Sander Dieleman, Pieter-Jan Kindermans, and Benjamin Schrauwen. Sign language recognition using convolutional neural networks. In Computer Fision-ECCV 2014 Workshops: Zurich, Switzerland, September 6-7 and 12, 2014, Proceedings, Part I 13, pages 572-578, Springer, 2015.
[37] Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transformers: Joint end-to-end sign language recognition and translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10023-10033, 2020.
[38] Ronglai Zuo, Fangyun Wei, and Brian Mak. Natural aunuage--ssisstdd sign language recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14890-14900, 2023.
[39] Yifan Zhang, Congqi Cao, Jian Cheng, and Hanqing Lu. Egogesture: A new dataset and bench- mark for egocentric hand gesture recognition. IEEE Transactions on Multimedia, 20(5): 1038- 1050, 2018.
[40] Xu Cao. Pranav Virupaksha, Wenqi Jia, Bolin Lai, Fiona Ryan, Sangmin Lee, and James M Rehg. Socialgesture: Delving into multi-person gesture understanding. In Pruceedings of the Computer Vision and Pattern Recognition Conference, pages 19509-19519, 2025.
[41] Pavlo Molchanov, Shalini Gupta, Kihwan Kim, and Jan Kautz. Hand gesture recognition with 3d convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 1-7, 2015.
[42] Pradyumna Narayana, Ross Beveridge, and Bruce A Draper. Gesture recognition: Focus on the hands. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5235-5244, 2018.
[43] Andrea D' Eusamio, Alessandro Simoni, Stefano Pini, Guido Borghi, Roberto Vezzani, and Rita Cucchiara. A transformer-based network for dynamic hand gesture recognition. In 2020 International Conference on 3D Vision (3DV), pages 632-632. IEEE, 2020.
[44] Mallika Garg, Debashis Ghosh, and Pyari Mohan Pradhan. Gestformer: Multiscale wavelet pooling transformer network for dynamic hand gesture recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2473-2483, 2024.
[45] Sindhu B Hegde, KR Prajwal, Taein Kwon, and Andrew Zisserman. Understanding co-speech gestures wn-the-wild. arXiv preprint arXiv:2503. 22668, 2025.
[46] Eunji Chong, Yongxin Wang, Nataniel Ruiz, and James M Rehg. Detecting attended visual targets in video. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 5396-5406, 2020.
[47] Samy Tafasca, Anshul Gupta, and Jean-Mare Odobez. Childplay: A new benchmark for understanding children's gaze behaviour. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[48] Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. Appearance- gaze estimation in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 451 1-4500, 2015.
[49] Yihua Cheng and Feng Lu. Gaze estimation using transformer. In 2022 26th International Conference on Pattern Recognition (ICPR), pages 3341-3347. IEEE, 2022.
[50] Mingfang Zhang, Yunfei Liu, and Feng Lu. Gazeonce: Real-time multi-person gaze estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 4197-4206, 2022.
[51] Samy Tafasca, Anshul Gupta, Victor Bros, and Jean-Mare Odobez. Toward semantic gaze target detection. Advances in Neural Information Processing Systems, 37:121422-12144 2024.
[52] Yuehao Song, Xinggang Wang, Jingfeng Yao, Wenyu Liu, Jinglin Zhang, and Xiangmin Xu. Vitgaze: gaze following with interaction features in vision transformers. Visual Intelligence, 2(1).1-15, 2024.
13

---

[53] Fiona Ryan, Ajay Bati, Sangmin Lee, Daniel Bolya, Judy Hoffman, and James M Rehg. Gaze-lle: Gaze target estimation via large-scale learned encoders. 2025.
[54] Anshul Gupta, Pierre Vuillecard, Arya Farkhondeh, and Jean-Mare Odobez. Exploring the zero-shot capabilities of vision-language models for improving gaze following. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 615-624, 2024.
[55] Anshul Gupta, Samy Tafasca, Arya Farkhondeh, Pierre Vuillecard, and Jean-marc Odobez. Mtgs: A novel framework for multi-person temporal gaze following and social gaze prediction. Advances in Neural Information Processing Systems, 37:15646-15677 2024.
[56] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720-6731, 2019.
[57] Bolin Lai, Hongxin Zhang, Miao Liu, Aryan Pariani, Fiona Ryan, Wenqi Jia. Shirley Anugrah Hayati, James M Rehg, and Diyi Yang. Werewolf among us: A multimodal dataset for modeling persuasion behaviors social deduction games. arXiv preprint arXiv:2212.08279, 2022.
[58] Amir Zadeh, Michael Chan. Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social- iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8807-8817, 2019.
[59] Glenn Jocher, Jing Qiu, and Ayush Chaurasia. Ultralytics YOLO, January 2023.
[60] OpenAl. Gpt-4o system card. htttps://openti hotpa/aaa...emeee///oooo-o---//-
2025.
[61]
Anthropic. Introducing the next generation of claude. claude-3- fammily 2024.
[62] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages I 13009-13018, 2024.
[63] Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shumping Ji, Chen Change Loy, and Shuicheng Yan. Omg-Ilava: Bridging image-level. object-level, pixel-level reasoning and understanding. Advances in Neural Information Processing Systems, 37:71737-71761 2024.
[64] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[65] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
[66] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in neural information processing systems, 35:38571-38584, 2022.
[67] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint S1t::2304.00193.. 2023.
[68] Jiankang Deng, Jia Guo, Yuxiang Zhou, Jinke Yu, Irene Kotsia, and Stefanos Zafeiriou. Reti- naface: Single-stage dense face localisation in the wild. arXiv preprint 2019.
[69] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint 2010. 2020.
14

---

[70] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[71] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-338 2010.
[72] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748-8763 8763. PMLR, 2021.
[73] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. Proceedings of the IEEE/CVF international conference on computer vision, pages 1975-11986, 2023.
[74] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025.
2:
[75] Şeyda Özęalışkan, Lauren B Adamson, and Nevena Dimitrova. Early deictic but not other gestures predict later vocabulary in both typical development and autism. Autism, 20(6):754-765 2016.
[76] Cynthia Matuszek, Liefeng Bo, Luke Zettlemoyer, and Dieter Fox. Learning from unscripted de- ictic gesture and language for human-robot interactions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, 2014.
[77] Samy Tafasca, Anshul Gupta, and Jean-Mare Odobez. Sharingan: A transformer architecture for multi-person gaze following. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 2008-2017, 2024.
[78] Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro Werra, and Erik Kaunismäki. 'smolagents": a smol library to build great agentic systems, hhttps://gith: honm//uugiinggaacc///nnggggg 2025.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, and see abstract and Section 1
We claim to design new task for human deictic gesture target estimation and propose the first model for this task.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This is a task-driven application baased paper, thus we do not have any theoreti- cal results or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The code and a subset of Gesture dataset is available the supplementary material. They will be made publicly available upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We will provide access the data and the code upon acceptance, in accordance to the NeurlPS code and data submission guidelines. For paper review, we attach our code and subset of the Gesture' Target in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
y /nips. /nips.cc///pp http:/ccccccccguueeeeeeeeeeeeee/ htwbbiiccgu.ddes/ddduubbiii
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
See Section 5.1 Technical Details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We run the experiments multiple times, with different train-validation splits, and take the average.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 5.1 Technical Details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttp:://eeuripg: wwtppppbic./cchccccuuu..d.o.odd.
Answer: [Yes]
Justification: All authors in the paper reviewed and strictly followed NeurIPS Code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Positive impacts are discussed in Section Introduction. Negative impacts are discussed in Section 7 Conclusion. We will include more detailed discussion of these impacts in the supplementary material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks, as all data used was publicly available.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We use the cc-by-nc-3.0 license for our work. (Code and Dataset)
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: We provide sufficient documentation along with our new assets (dataset, code and models).
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not have crowdsourcing component in the data annotation process.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The dataset has IRB approval. We will include this approval and any necessary description in our supplementary material.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We do not use LLMs to help in our methodology. The extent to which we use them is zero-shot evaluations
dataset underscore the impact of our dataset.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neur)) httt:p///nerrpssccc/oooooeeeeeee cct.e//eeeeeeeeee/225555555 for what should or should not be described.
22

---

A Notations
Data and Indices
Hi
Height of imput image
Win
Width of input image
X E <R
Input RGB image
Xsubject E R
Bounding box of subject's body
X Xiyee E IR2
Left eye keypoints
reee E I?
Right eye keypoints
X Ihand 1g2
Left hand keypoints
X rhand
E TR
Right hand keypoints
X text
Text describing subject's appearance, location, action Bounding box target
Y target
R
Y mask
E HRWW
Target semantic segmentation mask
text
Target text describing subject's appearance, location, action Existence label
Yexist:
Embeddings and Positional Encodings
P
Patch size of the token
H Hm/PP
Height of feature map
W Win/P /P
Width of feature map
Z RR xdmasa
Feature map outputted by visual encoder Downsampled binary mask for head
Mhead RRHW
Mbody E RH
Downsampled binary mask for body
PEnand E dgaze
Patch encoding for head Patch encoding for body
PE E digesture
Zyane E R xW ddqax
Feature map adding gaze positional embeddings to Z
Thren TR. HW Xdheed
Feature map adding head positional embeddings Z
Attention Mechanism and Transformer Components
texist Rdyam
Learnable gesture existence token
Q.K. K,
Query, key, value matrices
F Fazz H. dage
Output of the gaze decoder
F geesture R.HW xdea
Output of the gesture decoder
Fjoint E 2HW x Agouur
Joint feature used for joint cross attention
JointCrossAttentionn))
Joint cross-attention
Loss Functions and Parameters
LBCE
Binary crossentropy loss
Liecal Listal
Focal loss
Total loss
3
Loss weighting parameter
B Additional Experiment
Results on Stage - Pre-training for Gaze Decoder model training Stage we freeze the visual encoder to focus solely on pre-training the gaze decoder. Table 6 presents the evaluation outcomes on the GazeFollow test set, highlighting the effectiveness of our pre-trained gaze decoder. Our approach demonstrates highly competitive performance, achieving results comparable to state- of-the-art gaze estimation models. Specifically, we report an AUC of 0.959, Average L.2 distance of 0.096, and Minimum L2 distance of 0.045. These metrics place our method closely alongside leading models such as Sharingan [77] (AUC: 0.938, Avg L2: 0. 108, Min L2: 0.054), Gaze-LLE (53) (AUC: 0.958, Avg L.2: 0.099, Min L.2: 0.041), and ViTGaze (52) (AUC: 0.949, Avg L2: 0.105, Min L.2:
0.047). These results validate the strength of our gaze decoder pre-training strategy effectively capturing precise gaze targets and maintaining competitive accuracy.
23

---

Image Encoder
#Params
Learnable #Params
AUC t
Avg 1.2 4
Min 1.2 4
Carge
303M
2.93M
0.9550
0.0965
0.0455
SigLIP-Base
86.0M
2.86M
0.9484
0.1119
0.0559
SigLIP2-Bass Base
86.0M
2.86M
0.9529
0.1052
0.0508
2-Nav2-Small
22. IM
2.76M
0.9502
0.1 146
0.0583
DINOv2- Base
86.6M
2.86M
0.9562
0.1021
0.0484
MNNOv2-Large
304M
2.93M
0.9591
0.0957
0.0446
Table 6: Performance on Gazefollow in the gaze decoder pretraining stage.
Head Bounding Box
Exist Acc (%) f
IoU (%) f
89.76
58.92
90.04
50.57
Table 7: Comparison in the test set: existence of head bounding box.
Impact of Head Bounding Box (Gaze Guidance) To understand the influence of head bounding boxes form of gaze guidance, conducted ablation study using TransGesture with the DINOv2-Large visual encoder, summarized in Table We divided the test dataset into two subsets based the visibility of head bounding boxes. The first subset includes samples where the subject's head bounding box is visible, providing explicit gaze cues, while the second subset consists of cases without visible head bounding box, typically due occlusion or the subject's head being out of the camera's 's field of view.
Our findings reveal significant performance disparity between these two subsets. Specifically, when head bounding boxes are present, the loU scores for gesture target localization improve substantially compared cases lacking explicit head gaze cues. This indicates that head bounding boxes (gaze guidance) serve as critical visual signals, enabling more accurate gesture targets estimation by effectively aligning gaze intention with the subject's gestures.
C Application: Integrate TransGesture into LLM Toolbox
To demonstrate the utility of TransGesture in downstream applications, we integrate the models part of larger toolbox for an LLM--bssed agent supported by GPT-4o. In our experiment, we implement social gesture comprehension-focut gesture target estimation), object detection model, head detection model, and a gaze target estimation model. Figure shows the proposed workflow for the agent.
agent by smolagent (78), with access to TransGesture (for
The ideal query consists of least one image along with a social gesture comprehension question posed by the user.
For stability, the tools designed be self-containe that is, they depend minimally processed raw inputs and compute any other required inputs such as head or subject bounding box information.
We define the inputs the tools to be an image from the user's query, well as an optional description of a human subject to be analyzed. This description is parsed from the user's 's query and is returned only if present. Optionally, if a user inputs image of the target person instead of a text description, a vision-land Laniuae model can be used to generate the description automatically.
If a description is included, for the TransGesture tool specifically, we first use YOLOv11 to detect persons in the image and yield respective bounding boxes. We then pass cropped regions of the image corresponding to individuals to large dision-language model, which determines which region (if any) corresponds to the described target person. This then used to extract head bounding box information via RetinaFace. The tool now has all the required inputs for our TransGesture model. The output of our model is overlaid on the original image, and returned the L.L.M Figure shows example integration of TransGesture tool.
24

---

User
TransGesture
Gaze Target Estimation
Output
Reasoning
Agent Response
Figure 6: Proposed Agent Workflow.
As shown in Figure 7 TransGesture can be easily integrated as part of an agent's toolbox. will publicly release all APIs of the agent demo.
D Broader Impacts
D.1 Positive Societal Impacts
The ability estimate human deictic gesture targets has promising benefits for embodied intel- ligence and human-AI interaction. By empowering Al systems to understand where a person is pointing/eachhngg/hooinngg TransGesture can facilitate more natural, intuitive communication be- tween humans and embodied AI agents such as robots or AR assistants. This is because deictic gesture can direct attention to relevant objects and aiding collaboration. Likewise, gaze cues are fundamental part of non-verbal communication, and combining both modalities can substantially improve an Al's understanding of human intent. In assistive technology contexts, this capability could enable point-and-look interfaces for people with speech or mobility impairments, allowing them to refer objects or locations and receive assistance without complex verbal commands. Overall, our transformer-batt TransGesture model (trained on the GestureTarge dataset) can enhance embodied Al systems by making them more context-aware and responsive natural human signals, potentially improving case-of-use and inclusivity in everyday human-Al interactions.
D.2 Negative Societal Impacts
However, recognize that this technology also raises important ethical and societal concerns. First, any system that relies on continuous visual and gaze monitoring inherently implicates privacy even seemingly innocuous gaze data can reveal far more personal information than users intend (including aspects of identity, health, or interests). Without safeguards, a gesture-target estimator could be misused for unwarranted surveillance without their consent. This calls for careful dataset curation, and inclusive design. We stress the importance deploying this technology with user consent, transparency, and context-approprriat limitations. By proactively addressing privacy (e.g. on-device processing, data anonymization), bias mitigation, and clear use policies, we aim to ensure that the benefits of our system are realized responsibly, without compromising individual rights or societal trust.
attention tracking, monitoring where people look and point
25

---

2:31
USER:
pointing at?
Generated description: "young adult male with dark skin, short black hair, and glasses"
<TransGesture Tool Call>
Extracted Matched Subject + Head Boxes
2:31
TransGesture Tool
Output
Agent Output: "The described person is pointing at young adult female with blonde hair and checkered shirt and a smiling young with short dark hair and light blue shirt." "
Figure 7: Example Agent Query and Response. Green marks content visible the user. Yellow marks general internal agent outputs. Orange indicates tool internal outputs.
26


Reviews:
[{"review_id": "j5PItN410V", "reviewer": "Reviewer_BGdf", "summary": "The study presents a new dataset, GestureTarget, by re-purposing four existing datasets related to gestures and social interaction to gesture target estimation. The authors then introduced TransGesture, a family of transformer models that fuse gesture and gaze cues. This fusion showed improvement over the gesture-only baseline.", "questions": "- What are the practical aspects of this work?\n- What is the difference between Y_exist and Y_mask? Is the Y_exist just a one-dimensional binary value? This is not clear.\n\n- Is there a human baseline for the target estimation performance? If so, what is the upper bound?", "limitations": "- Human-based evaluation (baseline)\n- A kinematic model that captures gestures effectively", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- I find the task itself interesting (as well as the dataset). This type of information can guide models to calibrate to human attention and interactions. \n\n- The results make sense (e.g., gaze gives some prediction improvements), and the baselines, as well as the backbone variety, are good. The ablation for the current study is complete. Even though I would have liked to see a kinematic model (i.e., based on body models such as MediaPipe or SMLP-X).\n\n**Weaknesses:**\n\n- I find the choice of using RGB-based representations (especially for gestures) unintuitive. I believe that a kinematic module might be a better choice, either alone or in addition to the RGB raw data. \n\n- The evaluation is not cross-validated properly. The fixed 80 / 20 split is convenient but risks optimistic results if adjacent video frames land in both sets. K-fold cross-validation or at least a video-level split would strengthen the evidence.\n\n- No human-performance benchmark is reported. An annotation study (e.g. randomly sampled frames) would establish how close the best model is to human agreement.", "was_revised": true, "final_justification": "I am happy with the authors rebuttal. Therefore, I increased my ratings", "rebuttal": "We thank Reviewer BGdf for recognizing the value of our task definition and dataset, and for providing thoughtful, constructive feedback. Several of the stated concerns appear to arise from misunderstandings; we clarify these points and address each weakness and question in detail below.\n\n>**Weakness 1: Use of RGB vs. Kinematic/Keypoint Inputs**\n\nWe appreciate the reviewer’s suggestion to explore kinematic-based inputs. Our choice of using raw RGB images was driven by the nature of in-the-wild data and the limitations of kinematic feature (e.g. keypoint) extraction in such settings. **In our dataset, images come from diverse, uncontrolled sources (movies, social interactions, etc.), where obtaining accurate 2D/3D skeletons or keypoints for every person is challenging – automated keypoint detectors can fail under occlusions, extreme poses, or low lighting.** Relying solely on keypoints could thus introduce significant noise or miss subtle contextual cues (e.g. finger orientation or held-object context) that the RGB modality provides.\n\nThat said, we agree that incorporating explicit pose representations or specialized human keypoint-centric features is a promising direction. To quantify the potential benefit of an explicit pose-centric backbone, we replaced our DINOv2-Large encoder with Meta's Sapiens-0.3B—pretrained model [1] specifically for human pose understanding—while keeping all other components and hyper-parameters unchanged. We chose Meta’s Sapiens-0.3B because its model size is the same as a ViT-Large, making for a fair comparison.\n\n| Visual Encoder | \\#Params | Exist Acc | Target IoU |\n| :---- | :---- | :---- | :---- |\n| DINOv2-Large | 304M  | 89.85 | 58.26 |\n| Sapiens-0.3B | 304M  | 70.13 | 19.80 |\n\nSapiens performs markedly worse than DINOv2-Large. We believe there are two main factors:\n\n- Pre-training bias. Sapiens is trained on Humans-300 M, which provides single-person key-point supervision but no labels for inter-person pointing relationships. Consequently, it learns body-pose geometry but not the relational cues that are essential for identifying a deictic target.\n\n- Frozen-backbone protocol. In our ablation we kept the visual encoder frozen, so Sapiens could not adapt its single-person priors to our multi-person task, further widening the gap.\n\nThese findings reinforce our hypothesis that full RGB imagery—capturing both fine-grained pose details and rich scene context—should remain the dominant signal for gesture-target estimation. A promising next step is to add a lightweight kinematic/key-point branch or decoder that complements, rather than replaces, the RGB stream, allowing the model to exploit pose cues where key-point extraction is reliable while still relying on RGB for holistic context. Thank you again for the insightful suggestion!\n\n**Reference:**      \n[1] Khirodkar, Rawal, Timur Bagautdinov, Julieta Martinez, Su Zhaoen, Austin James, Peter Selednik, Stuart Anderson, and Shunsuke Saito. \"Sapiens: Foundation for human vision models.\" In European Conference on Computer Vision, pp. 206-228. Cham: Springer Nature Switzerland, 2024.\n\n\n>**Weakness 2: Dataset Splitting and Cross-Validation**\n\nWe understand the reviewer’s concern about potential overlap or leakage in the dataset split. To clarify, we performed a **source-level split** for training and testing. This means that all images originating from the same video clip or the same source dataset are placed entirely in either the train or test set, but not both. This ensures that near-duplicate frames or highly similar contexts from a single source cannot appear in both sets, preventing the model from simply memorizing scene-specific details. In the initial submission we used an 80/20 split on this source-stratified basis.\n\nTo further verify the robustness of our results, we have now conducted a **5-fold cross-validation** on the GestureTarget dataset (maintaining source-consistent splits in each fold). The performance was **highly consistent across folds**. For instance, the gesture existence accuracy across the 5 folds ranged  from **≈ 87% to 89%**, and the mean IoU for target masks varied from **≈ 56% to 57%**. The standard deviation was low, indicating no single split was an outlier. These results give us confidence that our reported performance is stable and not an artifact of a particular data split. We will include a brief summary of the cross-validation experiment in the revised paper to emphasize the reliability of our evaluation.\n\n|  | Exist Acc (%) | IoU (%) |\n| :---- | :---- | :---- |\n| Previous Test Set | 89.01 | 58.48 |\n| 5-fold cross-validation | 88.75 ± 0.25 | 57.42 ± 0.45 |\n\n>**Weakness 3: Human Study \\- Gesture Existence and Target Identification**\n|  | Exist Acc | Target IoU |\n| :---- | :---- | :---- |\n| GPT-4o | 70.21 | \\- |\n| Claude 3.7 | 68.48 | \\- |\n| OMG-LLaVA | \\- | 3.26 |\n| DINOv2-Base (Gesture-only) | 79.19 | 52.14 |\n| DINOv2-Base (JCA Fusion) | 89.01 | 58.48 |\n| DINOv2-Large (JCA Fusion) | 89.85 | 58.26 |\n| Human (Avg) | 87.00 | 66.40 |\n| Human (Max) | **90.00** | **80.00** |\n\nThank you for your valuable suggestion. We are pleased to report that we have conducted a human evaluation study and can now include these results. Due to the time constraints of the rebuttal period, we recruited 10 volunteer participants and asked them to perform two tasks on a randomly selected subset of 50 images from our test set: (1) Gesture existence detection (binary classification): determining whether the subject in the image is performing a deictic gesture; and (2) Target identification: if a gesture is present, identifying the intended target in the image by clicking on the corresponding object or person mask. Please note that this human study is different to the annotation process because the human study participants could not see the whole source video or additional image annotation information. We plan to further expand the size of the test population in the final version of the paper.\n\nFor the gesture existence detection task, the average human performance was 87.00%, with a maximum of 90.00%. For the target estimation task, we measured performance by the IoU between the region selected by each participant and the ground-truth target mask; the average human Target IoU was 66.40, with a maximum of 80.00%. This human study provides an upper bound on achievable performance.\n\nWe observe that our proposed model has reached human-level performance in gesture existence detection. However, a gap remains in target estimation accuracy. We will incorporate these findings into the paper to highlight that, while there is still room for improvement—particularly in target localization—our approach is closing the gap toward human-level agreement.\n\n>**Question 1: Practical aspects of this work**\n\nThe practical applications of deictic gesture target estimation are diverse and impactful, spanning multiple domains:\n\n- Non-verbal communication analysis in healthcare: Deictic gesture target estimation can greatly enhance our understanding of non-verbal cues in clinical settings, particularly for early autism diagnosis [2]. Individuals with autism spectrum disorder often display differences in both the use and interpretation of gestures. An accurate model for detecting and interpreting these gestures could assist clinicians and therapists in identifying subtle communication deficits in young children, enabling earlier and more effective interventions.\n\n- Human–AI interaction: Deictic gestures play a crucial role in natural communication. Improving machine understanding of these gestures will facilitate smoother, more intuitive human–AI interactions [3]. This is particularly valuable for assistive robotics—where robots must interpret human intent to provide effective support—and for virtual assistants operating in social or professional contexts.\n\nFurthermore, our paper observes that the co-occurrence of gaze and gesture can enhance gesture prediction. These non-verbal features remain underexplored in current Multimodal LLMs and VLMs. Our proposed model could be incorporated as a specialized tool in the toolkit of LLM agents. (We present a developmental experiment on this in the supplementary material.)\n\n**Reference**:      \n[2] Özçalışkan, Şeyda, Lauren B. Adamson, and Nevena Dimitrova. \"Early deictic but not other gestures predict later vocabulary in both typical development and autism.\" Autism 20, no. 6 (2016): 754-763.     \n[3] Matuszek, Cynthia, Liefeng Bo, Luke Zettlemoyer, and Dieter Fox. \"Learning from unscripted deictic gesture and language for human-robot interactions.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 28, no. 1. 2014.\n\n\n>**Question 2: What is the difference between Y\\_exist and Y\\_mask? Is the Y\\_exist just a one-dimensional binary value?**\n\nThanks for your question. \nY_exist is a binary value in {0, 1}, so we use binary cross-entropy (BCE) loss for this prediction.\nY_mask is a binary mask used for semantic segmentation.\n\n>**Question 3: Is there a human baseline for the target estimation performance? If so, what is the upper bound?**\n\nPlease see our response to **Weakness 3: Human Study – Gesture Existence and Target Identification** for details."}, {"review_id": "dl6RYLSJWW", "reviewer": "Reviewer_iuYx", "summary": "This paper introduces a novel task and accompanying benchmark dataset, GestureTarget, designed for evaluating the semantic target estimation of social deictic gestures. To tackle this task, the authors propose TransGesture, a model that incorporates a pretrained gaze target estimation module to extract gaze direction cues. By integrating gaze information, TransGesture aims to infer the actual intended target of a gesture, rather than relying solely on gesture features. Experimental results validate that gaze cues significantly enhance the model’s ability to accurately identify the object or person being pointed at.", "questions": "See \"Weaknesses\" in the review.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The task definition is clearly stated and well-motivated.\n2. The proposed method is targeted and appears effective in addressing the defined problem.\n3. The introduced dataset is well-aligned with the task and has the potential to benefit related tasks in the community, such as interactive generation.\n4. The experiments are comprehensive, and comparisons across multiple pretrained models provide useful insights for future work on visual feature extraction.\n5. The paper is generally well-written and easy to follow.\n\n**Weaknesses**\n1. The paper lacks subjective evaluations (e.g., user studies) that could help assess whether the predicted targets align with human perception.\n2. While the authors emphasize the role of gaze in the model design, there is no clear ablation study isolating the contribution of gaze. Including such experiments would help substantiate its importance.\n3. Inference efficiency is not discussed. Highlighting runtime would help assess its applicability in real-world or interactive settings.\n4. Figure 3 is not referenced in the main text, which may confuse readers.", "was_revised": true, "final_justification": "I have carefully read the authors's rebuttal. Most of my concerns have been addressed by the authors. Therefore, I would like to raise my rating.", "rebuttal": "We thank you for recognizing the value of our task definition and our proposed model. Thank you for the thorough review and helpful feedback. We address each concern below, and we outline the improvements we will make in the revision in response to your comments.\n\n\n\n>**1\\. The paper lacks subjective evaluations (e.g., user studies) that could help assess whether the predicted targets align with human perception.**\n\n|  | Exist Acc | Target IoU |\n| :---- | :---- | :---- |\n| GPT-4o | 70.21 | \\- |\n| Claude 3.7 | 68.48 | \\- |\n| OMG-LLaVA | \\- | 3.26 |\n| DINOv2-Base (Gesture-only) | 79.19 | 52.14 |\n| DINOv2-Base (JCA Fusion) | 89.01 | 58.48 |\n| DINOv2-Large (JCA Fusion) | 89.85 | 58.26 |\n| Human (Avg) | 87.00 | 66.40 |\n| Human (Max) | **90.00** | **80.00** |\n\nThank you for your valuable suggestion. We are pleased to report that we have conducted a human evaluation study and can now include these results. Due to the time constraints of the rebuttal period, we recruited 10 volunteer participants and asked them to perform two tasks on a randomly selected subset of 50 images from our test set: (1) Gesture existence detection (binary classification): determining whether the subject in the image is performing a deictic gesture; and (2) Target identification: if a gesture is present, identifying the intended target in the image by clicking on the corresponding object or person mask. Please note that this human study is different to the annotation process because the human study participants could not see the whole source video or additional image annotation information. We plan to further expand the size of the test population in the final version of the paper.\n\nFor the gesture existence detection task, the average human performance was 87.00%, with a maximum of 90.00%. For the target estimation task, we measured performance by the IoU between the region selected by each participant and the ground-truth target mask; the average human Target IoU was 66.40, with a maximum of 80.00%. This human study provides an upper bound on achievable performance.\n\nWe observe that our proposed model has reached human-level performance in gesture existence detection. However, a gap remains in target estimation accuracy. We will incorporate these findings into the paper to highlight that, while there is still room for improvement—particularly in target localization—our approach is closing the gap toward human-level agreement.\n\n>**2\\. While the authors emphasize the role of gaze in the model design, there is no clear ablation study isolating the contribution of gaze. Including such experiments would help substantiate its importance.**\n\nWe apologize for the confusion regarding the contribution of gaze cues. In the original Table 1 of the paper, we did include an ablation analysis isolating the effect of gaze in the main table, but we did not highlight it as an additional table for ablation study. Specifically, we compare our full model (with gaze input and joint cross-attention fusion) against a variant that uses gesture cues only (no gaze). \n\n| Backbone | Gaze Fusion | Exist Accuracy | Traget IoU |\n| :---- | :---- | :---- | :---- |\n| CLIP-Large | &cross; | 79.21 | 54.37 |\n| CLIP-Large | &check; | 87.66 | 56.65 |\n| SigLIP2-Base | &cross; | 76.70 | 55.34 |\n| SigLIP2-Base | &check; | 85.13 | 56.13 |\n| DINOv2-Base | &cross; | 79.19 | 52.14 |\n| DINOv2-Base | &check; | 89.01 | 58.48 |\n\nThe ablation shows a clear benefit from incorporating gaze: for example, on the CLIP-Large encoder, the gesture-only model achieves 79.2% existence accuracy and 54.4 IoU, whereas adding gaze cues via JCA Fusion raises performance to 87.7% and 56.7 IoU. Thank you for noting this omission; we will ensure the gaze contribution ablation is clearly presented in the next final version of our paper.\n\n>**3\\. Inference efficiency is not discussed. Highlighting runtime would help assess its applicability in real-world or interactive settings.**\n\nWe thank the reviewer for the suggestion. Our model uses a frozen visual encoder (e.g., CLIP or DINO backbone) followed by a lightweight transformer fusion module, so the inference computational cost is largely in the backbone forward pass. **We have measured the inference runtime of our method and compared it across different backbone architectures to address this concern.** The table below summarizes the average runtime per image (single 518x518 image input) for various model variants, measured on a single NVIDIA H100 GPU:\n\n| Visual Encoder | Fusion Method | Average run time per image (s) |\n| :---- | :---- | :---- |\n| SigLIP2-Base | Gesture-Only | 0.0094 |\n| SigLIP2-Base | Direct Fusion | 0.0106 |\n| SigLIP2-Base | Conv Fusion | 0.0134 |\n| SigLIP2-Base | JCA Fusion | 0.0133 |\n| DINOv2-Base | Gesture-Only | 0.0108 |\n| DINOv2-Base | Direct Fusion | 0.0111 |\n| DINOv2-Base | Conv Fusion | 0.0131 |\n| DINOv2-Base | JCA Fusion | 0.0110 |\n\nAs shown, incorporating our full JCA fusion adds only a minimal overhead of 0.001 to 0.003 seconds compared to the gesture-only baseline. In practice, the gaze branch computations—primarily a few additional transformer layers for cross-attention—are lightweight relative to the overall cost of the Transformer-based visual encoder. We will include this runtime analysis in the final version's appendix to assure readers of the model’s efficiency. Overall, our method’s inference speed is comparable to standard ViT-based image understanding models, and the added gaze branch provides clear benefits at a negligible computational cost.\n\n>**4\\. Figure 3 is not referenced in the main text, which may confuse readers.**\n\nThanks for pointing out this missing reference problem, We will add it to Section 4.3 Baselines."}, {"review_id": "qqNQJjwE7s", "reviewer": "Reviewer_yXKj", "summary": "The authors propose dataset called GestureTarget which contains over 19000 annotated instances of source pointing, reaching, showing and giving (i.e. deictic) gestures with corresponding image masks for the target. The motivation for this dataset lies comes from the general unavailability of laerge deictic gesture datasets and hence blocking important research in the domain of deictic gesture prediction. The authors also propose TransGesture, a Transformer based model family (with cross attention between gaze source gesture) that integrates both human gestures and gaze to infer gesture targets in a complex scene. With ablation studies, the authors demonstrate the added value of gaze in predicting the target of the deictic gesture. Through various pretrained visual backbones, fusion methodologies, ablation studies, and comparison with SOTA VLMs, the authors study their design choices and validate the need for a Cross Attention module (i.e. JCA fusion) as well as the proposed joint loss function.", "questions": "* What is the distribution of existence of gestures in the dataset?\n* Would it be possible to share Existence F1-Score?\n* How were \"wrong-labeled frames during verification\" (l116) detected? Were the annotators shown the video to make that determination?\n* Could the authors share more detailed information (e.g. guidelines, annotation webpage screenshots, inter-annotator correlation) on the annotation process?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- The proposed dataset contains valuable annotations of source gesture and target masks for furthering research in the Gesture Generation/Detection and Human Behavior modeling communities. The authors combine multiple multi-interaction datasets to build a larger dataset with over 19,000 annotations.\n- The authors measure that SOTA MLLMs models such as GPT 4-0, Claude and VLMs like GLaMM, OMG-LLaVA perform poorly on the proposed dataset, hence motivating the need for training dedicated models for gesture prediction\n- In table 1, the authors validate that gaze is a relevant conditioning variable to make better target predictions.  More specifically, Cross Attention based fusion (i.e. JCA) with gaze results in clear wins on top of Gesture-only baseline\n\n\n### Weaknesses\n-  Gestures are typically a temporal sequence, rather than just one image. The dataset is constructed using other Social Interaction datasets with videos. The construction of the dataset in this paper with images instead feels like a missed opportunity. Additionally, deictic gestures in single images may account for false positives (e.g. the person might not be pointing at someone, but performing beat gestures)\n- The choice of \"Accuracy of Gesture Existing\" as a metric might potentially not correctly represent the model's performance if the distribution of Gesture Existing and not-exisiting in the train/val dataset is not equal. It would benefit the community if the distribution of existence of gestures in the dataset is available to the readers and potentially using metrics such as F1-score or reporting both Precision and Recall would add value.", "was_revised": true, "final_justification": "Thank you to the authors for providing a detailed response to all the reviewers questions.\n\n1. The positive/negative examples ratio is 3:5 as mentioned by the authors. That coupled by F1 scores add more validation to the metrics.\n2. Thank you for sharing the Dataset collection methodology. I would recommend adding them to the final paper/appendix for the benefit of the readers\n2. Thank you for sharing a human study as well. The best acc metrics by a human is close to 90% percent which is close enough to the best Model (DINOv2-Large (JCA Fusion)). While there is not a huge gap to cover in predicting the existence vs non-existence, there seems to be a performance gap in locating the gesture location in the image. Intuitively, this makes sense as locating the target of the location is a harder problem, especially in 2D images, and potentially could benefit from a lot more context (e.g. scene, video history before arriving at the pointing gesture etc).\n\nBased on the additional information, I have updated my score", "rebuttal": "We sincerely thank you for your thoughtful and supportive review, and for recognizing the strengths and impact of our work. We greatly appreciate the suggestion to release detailed dataset annotation information and to develop video-based annotations in the future.\n\n>**Weakness 1: Gestures are typically a temporal sequence, rather than just one image. The dataset is constructed using other Social Interaction datasets with videos. The construction of the dataset in this paper with images instead feels like a missed opportunity. Additionally, deictic gestures in single images may account for false positives (e.g. the person might not be pointing at someone, but performing beat gestures)**\n\nWe thank the reviewer for raising this point. We agree that temporal information can be very useful for disambiguating gestures—for example, continuous motion can help differentiate a deictic point from a brief beat gesture. Adapting to video input is also our ultimate goal for gesture target estimation.\n\nOur decision to build the GestureTarget dataset using still images was primarily motivated by annotation feasibility and data availability. **Similar to the development of gaze-estimation tasks (which progressed from image-based datasets like GazeFollow to video datasets such as VideoGaze and ChildPlay), constructing a large-scale video dataset with frame-by-frame target masks would have been prohibitively costly.** Instead, we curated a large number of representative frames containing clear gestures. Many of the source datasets we leveraged (e.g., VCR, SocialGesture) were originally image- or frame-based, with metadata such as bounding boxes available per frame, which we could repurpose.\n\nWe took several steps to mitigate the ambiguities that can arise in single images. During dataset creation, we manually filtered out ambiguous cases, such as conversational beat gestures or poses where it was unclear if pointing was occurring. Our annotation guidelines explicitly defined positive deictic gesture instances as those where the target is clear to the subject. They are opposed to gestures without a clear target (such as a generic hand wave or beat).\n\nNonetheless, we agree that temporal context and other modalities, such as depth or speech, would further enrich the task. We are actively working on a video sequence extension (VideoGestureTarget), where the bounding box and mask of each gesture target is annotated across short video clips, allowing models to utilize motion cues and temporal consistency.\n\n>**Weakness 2: The choice of \"Accuracy of Gesture Existing\" as a metric might potentially not correctly represent the model's performance if the distribution of Gesture Existing and not-existing in the train/val dataset is not equal. It would benefit the community if the distribution of existence of gestures in the dataset is available to the readers and potentially using metrics such as F1-score or reporting both Precision and Recall would add value.**\n\nThank you for the constructive feedback. In our dataset, the ratio of gesture existence is approximately 3:5 (exist:non-exist) in both the training and test sets. To provide a more comprehensive evaluation, we have computed the F1-score for the gesture existence detection task and updated our results table accordingly. The backbone model used for these results is DINOv2-Base.\n\n| Fusion Method | Exist Acc (%) | Exist F1 (%) |\n| :---- | :---- | :---- |\n| Gesture-Only | 79.78 | 80.18 |\n| Direct Fusion | 80.19 | 80.64 |\n| Conv Fusion | 88.10 | 87.62 |\n| JCA Fusion | 89.01 | 88.73 |\n\n>**Question 1: What is the distribution of existence of gestures in the dataset?**\n\nPlease see our response to **Weakness 2**. The distribution of gesture existence is approximately 3:5 (exist:non-exist) in both the training and test sets.\n\n>**Question 2: Would it be possible to share the existence F1-score?**\n\nYes, we report the F1-score in the table for **Weakness 2** above.\n\n>**Question 3: How were \"wrong-labeled frames during verification\" (l116) detected? Were the annotators shown the video to make that determination?**\n\nThank you for raising this important point. The 30% \"wrong-labeled\" rate refers to incorrect pseudo-annotated masks generated by YOLOv11 instance segmetnation model. Potential \"wrong-labeled\" situations are the mask only cover half of the target human; two or more human's mask regions are overlapped. We manually reviewed all target mask pseudo-annotations, and if the mask region deviated from the intended target, those data were removed from the dataset.\n\nFor the deictic gesture key frame selection, annotators needed to refer adjacent frames within the video sequence if the source data is a video.\n\n>**Question 4: Could the authors share more detailed information (e.g., guidelines, annotation webpage screenshots, inter-annotator correlation) on the annotation process?**\nWe appreciate the reviewer’s interest in our annotation process, as it allows us to highlight the efforts we took to ensure data quality:\n\nAnnotation Verification: Every annotation in GestureTarget underwent manual verification. Our process involves:\n\n- Key frame selection (identifying frames in the video that clearly contained deictic gestures),\n\n- An automatic pseudo-annotation step (using YOLOv11 instance segmetnation model to segment all human and object masks), and\n\n- Manual selection of gesture target masks by the author team.\n\nDuring key frame selection, we employed a three-step methodology (annotation, verification, and quality assurance), measuring inter-annotator agreement using metrics such as Cohen's kappa. For ambiguous or edge cases, final decisions were made through a three-person consensus. During the manual mask selection stage, one of the authors selected the target in each image, confirmed the subject person’s bounding box and head position, and verified that the mask generated by YOLOv11 was correct. Any image with incorrect or ambiguous masks was discarded. In fact, approximately 30% of candidate frames were removed during verification due to issues such as overlapping or missing masks.\n\nWe are committed to open research and plan to release the GestureTarget dataset on the Hugging Face Hub, accompanied by comprehensive documentation. This documentation will include a data card detailing dataset composition, annotation protocol, class definitions, and ethical considerations. We believe this level of transparency is important for any new dataset, and we thank the reviewer for highlighting this point. We will ensure that our final version clearly states that the dataset and its detailed documentation will be made publicly available (with a link or DOI)."}]

Author rebuttal:

Meta-review: The reviewers are unanimously positive about the paper. The paper introduces an underexplored problem and provides a valuable dataset to the community. The core methodology is sound, and the experimental results are comprehensive and convincing. The rebuttal addresses the reviewers concerns. The ratings are two accepts and a borderline accept. Therefore, our decision is to accept the paper.
Decision: Accept (poster)