Title: Doodle to Detect: A Goofy but Powerful Approach to Skeleton-based Hand Gesture Recognition
Venue: NeurIPS 2025
Accepted: yes
Keywords: Hand Gesture Recognition

Skeleton based Action Recognition

Online Recognition

Modality Transform

Vision Transformer
Abstract: Skeleton-based hand gesture recognition plays a crucial role in enabling intuitive human–computer interaction. Traditional methods have primarily relied on hand-crafted features—such as distances between joints or positional changes across frames—to alleviate issues from viewpoint variation or body proportion differences. However, these hand-crafted features often fail to capture the full spatio-temporal information in raw skeleton data, exhibit poor interpretability, and depend heavily on dataset-specific preprocessing, limiting generalization. In addition, normalization strategies in traditional methods, which rely on training data, can introduce domain gaps between training and testing environments, further hindering robustness in diverse real-world settings. To overcome these challenges, we exclude traditional hand-crafted features and propose Skeleton Kinematics Extraction Through Coordinated grapH (SKETCH), a novel framework that directly utilizes raw four-dimensional (time, x, y, and z) skeleton sequences and transforms them into intuitive visual graph representations. The proposed framework incorporates a novel learnable Dynamic Range Embedding (DRE) to preserve axis-wise motion magnitudes lost during normalization and visual graph representations, enabling richer and more discriminative feature learning. This approach produces a graph image that richly captures the raw data’s inherent information and provides interpretable visual attention cues. Furthermore, SKETCH applies independent min–max normalization on fixed-length temporal windows in real time, mitigating degradation from absolute coordinate fluctuations caused by varying sensor viewpoints or differences in individual body proportions. Through these designs, our approach becomes inherently topology-agnostic, avoiding fragile dependencies on dataset- or sensor-specific skeleton definitions. By leveraging pre-trained vision backbones, SKETCH achieves efficient convergence and superior recognition accuracy. Experimental results on SHREC’19 and SHREC’22 benchmarks show that it outperforms state-of-the-art methods in both robustness and generalization, establishing a new paradigm for skeleton-based hand gesture recognition. The code is available at https://github.com/capableofanything/SKETCH.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a hand gesture recognition algorithm using 3d skeleton sequences as input. It works by transforming the collection of sequences (x,y,z)_t for all the joints of the hand into three graph plots {x(t),y(t),z(t)}, where for each coordinate, all the joints are represented within the same plot, with different colours, and using min-max normalisation. To retrieve the different relative dynamic ranges between coordinates, the same data are fed to a module that outputs a range embedding added to each plot. The three resulting plots - for a fixed-sized temporal window - are vertically stacked to form an image that is submitted to a swin-transformer network with one classification head that performs the gesture recognition, and two regression heads that respectively locate the start and end times of the gesture.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well written and easy to read. The authors made a remarkable effort in the graphical presentation of their paper and the figures are excellent.\nThe major problem I have in this paper lies precisely in its visual representation of the skeletal data. I find it not only \"goofy\" as confessed in the title (and only there curiously), I find it scientifically shocking. The argument of \"reduction\" from 4d to 2d data does not make sense to me, since it's about transforming a light collection of time series into a high definition image with sparse relevant information (that lose topological information between joints BTW). And I don't think that good results are sufficient to make it acceptable. The authors should at least compare their approach with a more natural method using the native time series, and be able to explain why this anthropomorphic stacked-plots-as-image classification method possibly works better.  \nI also do not agree with some concluding claims saying that \"These results suggested that skeleton-based hand gesture recognition can play a key role\", and that \"the SKETCH module opened up a new perspective in hand gesture recognition through an interpretable data representation.\" I consider that interpretability comes from the pose estimation and the skeletal input, not from your method.\n\nQUESTIONS:\nThe role of the Dynamic Range Embedding is not clear. Additional explanations are needed. And why learn an embedding range and not simply compute it from the sequences?\nOne important point of gesture recognition methods working on pose data is their robustness to pose estimation errors. How does your method deal with that?\n\nLIMITATIONS:\nThere is a short section on limitations, but I consider that it does not address the main limitations mentioned above: 1) what is the fundamental contribution of the proposed visual representation of skeletal time series? 2) what is the robustness of the method with respect to pose estimation errors?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SKETCH (Skeleton Kinematics Extraction Through Coordinated grapH), a novel approach for real-time hand gesture recognition using raw 3D skeleton data. Unlike conventional methods that rely on hand-crafted features, SKETCH transforms raw four-dimensional skeleton data (time, x, y, z) into interpretable graph images. These visual representations are processed by pretrained vision backbones (ViT/Swin Transformer) for gesture classification and boundary localization.\n\nA central innovation is the Dynamic Range Embedding (DRE), which compensates for the loss of axis-relative scale caused by per-axis min–max normalization. The method also handles domain shifts—due to sensor viewpoints or user-specific kinematics—by applying normalization independently per window at inference time. SKETCH is evaluated on SHREC’19 and SHREC’22 benchmarks, achieving state-of-the-art accuracy and interpretability, while operating in real-time.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. Original and Clever Formulation. The conversion of skeleton sequences into visual plots is both novel and intuitive. By leveraging pretrained vision backbones, the method sidesteps the need for complex temporal-spatial modeling architectures.\n\n2. High Practical Impact. The method is highly relevant for real-time systems such as VR/AR, robotics, and HCI. Its streaming-friendly window-based architecture is well-suited for deployment in online gesture detection settings.\n\n3. Strong Experiment Results. The method consistently outperforms prior art on two major gesture benchmarks across multiple metrics (DR, FP, JI, FPS). Ablations are provided for normalization, DRE, loss components, etc.\n\n**Weaknesses**:\n\n1. Potential Performance Bottlenecks in Latency-Sensitive Scenarios. Despite real-time FPS claims, the use of vision transformers on graph images may still be computationally heavier than lightweight GCNs or 1D CNNs, particularly for mobile applications.\n\n2. No Formal Analysis of Robustness. Although SKETCH handles viewpoint and subject variation empirically, there is no robustness analysis under noise, occlusion, or missing joints—common in real-world sensors.\n\nQUESTIONS:\n1. Have the authors tried other backbones besides ViT and Swin Transformer, perhaps more lightweight ones? Because this paper addresses online recognition, efficiency is a critical factor that should be taken into account. And based on Table 1, this method doesn’t outperform existing methods in terms of time and FPS. So it would be helpful to provide some further analysis on efficiency.\n\n2. Could the proposed SKETCH be applied to human body skeletons? The method doesn’t seem to restrict itself to any specific type of skeleton topology (unlike graph convolution networks, which require a fixed spatial topology to create an adjacency matrix), so I think this method could generalize well to body skeletons.\n\n3. Can the visual representation handle missing data or occlusions robustly?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a pipeline for online skeleton-based hand gesture recognition. The authors observe that existing approaches often suffer from a “geometric shortcut,” stemming from Euclidean-space-based representations or from using hand-crafted features that are difficult to interpret and generalize. Inspired by ViTST, the paper introduces a graph plot-based representation, which effectively converts windowed skeleton coordinates into line charts that plot normalized local joint movement per axis. This representation is more interpretable and can be processed by any vision-based model. To address the information loss regarding global motion, the authors propose Dynamic Range Embedding, which embeds z-normalized axis movement across a window. Experiments on SHREC’19 and SHREC’22 demonstrate that the proposed method operates in real time and outperforms previous approaches in both detection rate and false positive rate.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper is exceptionally well-written and easy to follow—arguably the clearest paper I’ve read in this NeurIPS cycle.\n\n- The idea of converting skeleton coordinates to graph charts is novel and interesting. The proposed method demonstrates strong effectiveness compared to previous hand-crafted features and other baselines.\n\n- The authors provide thorough ablation studies and experiments on all components, demonstrating the effectiveness of the plotting strategy, the presented feature, and the normalization technique.\n\nWeaknesses\n\n- The paper does not systematically analyze how the proposed feature performs under global rotations or changes in camera viewpoint. It remains unclear how robust the method is in these scenarios.\n\n- Both SHREC’19 and SHREC’22 are relatively simple datasets for gesture recognition by today’s standards. The paper could be further strengthened by evaluating on more challenging gesture-language datasets, such as BOT57M.\n\nQUESTIONS:\n* Does the method use sliding windows or non-overlapping windows for segmenting the data?\n* How does the method perform if vectorized representations are used directly instead of converting features into pixel-based plots? For example, what if the normalized $N_f \\times J$ feature is fed directly, rather than through image plotting?\n* What is the final resolution of the generated graph images?\n\nLIMITATIONS:\n* As mentioned in the limitation section, information from preceding windows is not utilized. Future work could consider adopting video processing models to leverage longer temporal context.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SKETCH (Skeleton Kinematics Extraction Through Coordinated grapH), a novel method that transforms 4D skeleton data (time, x, y, z) into a visually interpretable representation. This transformation enables the application of advanced visual recognition architectures such as SWIN and ViT for skeleton-based hand gesture recognition. The proposed approach seeks to make gesture recognition more intuitive and effective by converting temporal motion patterns into a visual domain.\n\nWhile the visual idea is interesting and the method shows promise, the motivation lacks depth in positioning the contribution relative to the rich body of existing gesture recognition work. Further, model complexity details are absent, and the evaluation is limited to only two datasets, which restricts insight into the generalizability of the approach.\n\nSTRENGTHS AND WEAKNESSES:\nTransforming 4D skeleton data into a visual format is creative and facilitates the use of powerful visual transformer architectures.\nThe integration of vision transformer-based models (e.g., SWIN, ViT) for gesture recognition through the visual encoding is novel and bridges spatial-temporal modeling with image-based classification.\nThe SKETCH framework is conceptually aligned with the nature of the task (hand gestures), which makes the overall presentation engaging and easier to follow.\n\nThe paper does not sufficiently highlight the limitations of existing gesture recognition approaches or clearly justify why a transformation to a visual representation is necessary or superior.\n\n\nDetails such as the number of parameters, computational cost (GFLOPs), or inference time are not provided. These are essential when using large models like SWIN or ViT on a task typically solved with lighter architectures.\n\n\nGiven the task focuses on keypoint dynamics, the use of heavy models like ViT and SWIN-T seems excessive unless justified through clear comparative analysis or accuracy-performance trade-offs.\n\n\nThe experiments are conducted on only two datasets. Evaluating on more diverse and dynamic datasets would strengthen the generalizability claims.\n\nQUESTIONS:\nProvide a more detailed discussion of what limitations exist in current gesture recognition methods and how SKETCH directly addresses them.\nExplain why transformer-based models (SWIN, ViT) are necessary for this problem. Could lighter models suffice, especially given the reduced input complexity (keypoints instead of raw video)?\nReport parameter count, FLOPs, and runtime comparisons to provide readers with a sense of computational efficiency and feasibility.\nConsider evaluating on additional gesture datasets, especially those with varying motion complexity and user diversity, to assess the robustness and generalizability of SKETCH.\n\nAn ablation showing the impact of the SKETCH transformation versus raw skeleton input directly into the models would help isolate the contribution of your method.\n\nLIMITATIONS:\nAs discussed above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}