Doodle to Detect: A Goofy but Powerful Approach to Skeleton-based Hand Gesture Recognition
Authors: Sang Hoon Han
Seonho Lee
Hyeok Nam
Jae Hyeon Park
Min Hee Cha
Min Geol Kim
Hyunse Lee
Sangyeon Ahn
Chae moon ju
Sung In Cho
Venue: NeurIPS 2025 poster ()
Keywords: Hand Gesture Recognition
Skeleton based Action Recognition
Online Recognition
Modality Transform
Vision Transformer
Primary area: deep_learning
TL;DR: We proposed novel SKETCH method that extracts high level feature transforming a raw four-dimensional online skeleton point coordinates data to graph images.

Abstract: Skeleton-based hand gesture recognition plays a crucial role in enabling intuitive human–computer interaction. Traditional methods have primarily relied on hand-crafted features—such as distances between joints or positional changes across frames—to alleviate issues from viewpoint variation or body proportion differences. However, these hand-crafted features often fail to capture the full spatio-temporal information in raw skeleton data, exhibit poor interpretability, and depend heavily on dataset-specific preprocessing, limiting generalization. In addition, normalization strategies in traditional methods, which rely on training data, can introduce domain gaps between training and testing environments, further hindering robustness in diverse real-world settings. To overcome these challenges, we exclude traditional hand-crafted features and propose Skeleton Kinematics Extraction Through Coordinated grapH (SKETCH), a novel framework that directly utilizes raw four-dimensional (time, x, y, and z) skeleton sequences and transforms them into intuitive visual graph representations. The proposed framework incorporates a novel learnable Dynamic Range Embedding (DRE) to preserve axis-wise motion magnitudes lost during normalization and visual graph representations, enabling richer and more discriminative feature learning. This approach produces a graph image that richly captures the raw data’s inherent information and provides interpretable visual attention cues. Furthermore, SKETCH applies independent min–max normalization on fixed-length temporal windows in real time, mitigating degradation from absolute coordinate fluctuations caused by varying sensor viewpoints or differences in individual body proportions. Through these designs, our approach becomes inherently topology-agnostic, avoiding fragile dependencies on dataset- or sensor-specific skeleton definitions. By leveraging pre-trained vision backbones, SKETCH achieves efficient convergence and superior recognition accuracy. Experimental results on SHREC’19 and SHREC’22 benchmarks show that it outperforms state-of-the-art methods in both robustness and generalization, establishing a new paradigm for skeleton-based hand gesture recognition. The code is available at https://github.com/capableofanything/SKETCH.

Doodle to Detect: A Goofy but Powerful Approach to Skeleton-based Hand Gesture Recognition
SangHoon Han MinHee Cha?,
Seonho Lte MinGeol Kim
Hyeok Nam I-t HyunSe Lee?, Sung In Cho
JaeHyeon Park?, SangYeon Ahn?,
MoonJu Chae?.
Department Artificial Intelligence at Sogang University 35, Backbeom-ro, Mapo-gu, gu, Scoul 04107, Republic of Korea (1eo4102, sunhozizi, skagur10, rlaalsrjf6, csi2267) Gsogang ac kr
2 Department of Computer Science and Artificial Intelligence at Dongguk University
30, Pildong-ro I-gil, Jung-gu, Scoul 04620, Republic of Korea {pjh0011, rninicha, sae4394, sy990607, fulldoor) @dgu. ac.kr
Abstract
preleton-based hand gesture recognition plays crucial role in enabling intu- itive muman-compuuc interaction. Traditional methods have primarily relied hand-crafted features-St such as distances between joints or positional changes across frames to alleviate issues from viewpoint variation or body proportion differences. However, these hand-crafted features often fail to capture the full spatio-temporal information in skeleton data, exhibit poor interpretability, and depend heavily on dataset-specific preprocessing, limiting generalization. In addition, normalization strategies in traditional methods, which rely on training data, can introduce domain gaps between training and testing environments, further hindering robustness diverse real-world settings. To overcome these challenges, we exclude traditional hand-crafted features and propose Skeleton Kinematics Ex- traction Through Coordinated grapH (SKETCH), a novel framework that directly utilizes raw Nour-dimensiona (time, x, and 2) skeleton sequences and transforms them into intuitive visual graph representations. The proposed framework incorpo- rates novel learnable Dynamic Range Embedding (DRE) preserve axis-wise motion magnitudes lost during normalization and visual graph representations, enabling richer and more discriminative feature learning. This approach produces a graph image that richly captures the raw data's inherent information and provides interpretable visual attention cues. Furthermore, SKETCH applies independent min-max normalization on fixed-length temporal windows real time, mitigat- ing degradation from absolute coordinate fluctuations caused by varying sensor viewpoints or differences in individual body proportions. Through these designs, approach becomes inherently topology-agnoss aggostic, avoiding fragile dependencies on dataset- or sensor-specific skeleton definitions. By leveraging pre- pre-trained vi- sion backbones, SKETCH achieves efficient convergence and superior recognition accuracy. Experimental results on SHREC' 19 and SHREC'22 22 benchmarks show that it outperforms state-of-the-art methods in both robustness and generalization, establishing new paradigm for skeleton- baased hand gesture recognition. The code is available http://///hlocooocc/cpccoooilii/
"Corresponding author. These authors contributed equally.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Traditional hand-crafted feature extraction
Our feature extraction
Joint collection
Frame diffirrncee vectors
Raw data
dis-lances
Raw data
Coordinate (4
Joint adjacency matrix
anvernatnnn www.omooenn..................... www.mnoeeen.....................
Online normalization
Invariant, Non intuitivvnnsss
Gap
Training distrbutton
distribution
N (MMraaiay Oreiin) * N (Httess Ttest)
Our graph image features
Figure 1: The left shows an example of the conventional methods dependent on the training data. The right shows the proposed feature extraction approach, which converts raw data into visual representation and performs normalization based on real-time input data, with Dynamic Range Embedding.
Introduction
Hand gesture recognition using skeleton data has emerged as key component in realizing sophisti- cated human-comppter interaction
various fields, including intelligent roboties, mixed reality (MR), and human-robot interaction [21]. Hand gesture recogmition via skeletal tracking contributes to the implementation of intuitive interfaces in virtual reality/augmmentdd reality (VR/AR) environments and greatly assists robots in perceiving and interpreting human activities (7,31] In such environments, various sensors-such as RGB cameras, depth sensors, and LiDAR Lipturc human motion and the skeleton data extracted from these sensors (i.e., the 2D/3D coordinates of key joints) provides concise representation of human posture that is less affected by changes in illumination. Unlike RGB video frames [13], skeleton data focuses on the structure of the body [19, offering higher processing efficiency and robustness against variations in background and lighting conditions. However, directly utilizing such raw skeleton data entails several inherent challenges for reliable gesture understanding real-worke environments.
This challenge is further compounded by the "geometric shortcut" phenomenon, as identified in Sonata (50). where directly inputting raw coordinate data lead models to overfit to low-level spatial patterms-such simply memorizing hand-height values-rather than extracting high-level features, ultimately resulting representation collapse 6 52 Furthermore, raw skeletal coordinates are highly sensitive to sensor viewpoint and subject position, leading to substantial variation in observed sequences for identical actions.
result, models trained under fxxe-viewper assumptions may suffer degraded recognition accuracy during testing due to discrepancies in absolute coordinates In addition to these viewpoint-related issues [16), further variations may arise from individual- -seecffic factors, resulting in distribution shifts between training and testing data. Such factors include differences in users' physical proportions (c.g., hand size and finger proportions) and individual variations in movement style and execution speed, all of which can significantly affect the resulting skeleton data.
caused by viewpoint shifts 23 30
To address these issues, conventional skeleton- based action recognition approaches 32 33 generally do not primarily rely on using raw joint data directly (although there have been prior attempts to do so, their generalization performance is limited by the geometric shortcut; see Appendix AA for details); instead, they focus on hand-crafted features designed to alleviate variations, which are extracted via rule-based methods from the raw data. As shown in Fig. [ representations such joint
collection distances 54 frame difference vectors g 60), joint adjacency matrices
(60 24 and
reference-joint difference vectors 660 24 are used to encode skeletal motion in time series through

---

rule-based processing, which directly extracts high-level features to mitigate geometric shortcuts and avoid issues related absolute coordinate variability caused by differing viewpoints. Among them, adjacency matrices are employed to explicitly represent the structural relationships among joints for topology modeling, and the reference-joint difference vectors encode each joint in a sequence as a vector relative to designated reference joint defined in the first frame.
However, existing hand-crafted feature extraction methods share common drawbacks: they are heavily dependent on domain expert knowledge, often lack interpretability, and struggle to fully capture the rich spatio-temporal information contained in raw skeleton data, Among them, methods based on reference-join difference vectors suffer from additional inherent limitations. Their performance is highly sensitive to the choice of reference joint, requiring prior topology information. In practice, the same anatomical joint may be indexed differently across datasets, or may not be defined at all, leading to reliance on dataset- or sensor- sensor-specific topology definitions and limiting broader applicability. Furthermore, even with an optimal reference joint, these approaches remain fragile, as missing or noisy joints in real-world scenarios can severely degrade generalization performance, resulting in strong reference-joint dependency. Beyond these limitations, subject-specifie variations such as differences in body proportions or motion styles remain unresolved. Due to the inherent coordinate variations caused by differences in subjects' physical proportions, the hand-crafted features extracted from raw data inevitably retain these variations. This necessitates normalization based on global statistics computed from the entire training dataset to reduce distribution shifts arising from subject- specific body scales. Notably, window-wise (i.e., fixed-length segments-wist per-axis normalization is generally avoided in this context, as it may distort the extracted features and undermine the intended viewpoint-invariance mitigation of the representation. However global normalization can suppress fine-grained differences in joint movements due to the influence of joint coordinates with large variance, and it inherently assumes that the test data distribution will closely resemble that of the training set-an assumption often violated in real-world scenarios, potentially degrading model performance at inference time.
To overcome these limitations, this study-drawing inspiration from VITST 22 proposes a novel hand gesture recogmition framework termed Skeleton Kinematics Extraction Through Coordinated grapH (SKETCH). Unlike conventional hand-crafted approaches, shown in Fig. [ the proposed SKETCH directly utilizes four-dimensional skeleton coordinate data, thereby eliminating the re- liance domain expert knowledge, enhancing interpretability, and preserving the rich information inherent in the raw data. To further address the domain gap between training and testing data caused by differences in subject-specific variations-isse
that even hand-crafted features struggle to mitigate-SKETCCH employs a window-bass axis-wise min-max normalization strategy (see Ap- pendix E.4 for ablation analysis). However, normalization alone is insufficient resolve the inherent limitations of raw coordinate inputs, particularly the "geometric shortcut" phenomenon (as further detailed in Appendix A To tackle this, we leverage plotting technique as an intermediate medium to transform normalized coordinates into graph image representation that alleviates viewpoint and subject variations, enabling more robust and semantically meaningful feature learning. Beyond its role in feature extraction, this design is inherently topology-agnostic allowing the proposed SKETCH method to generalize across diverse skeleton structures and remain resilient to pose estimation errors (see Appendix (E.6). The resulting visual graph also enhances interpretability by making it possible to analyze which spatiotemporal patterns or movement cues the model focuses on. This image-based rep- resentation further enables the use of
on image classification datasets such as ImageNet-Ik Ik (37 and -22k -t to extract high-level features. By leveraging a pretrained backbone, our approach converges in fewer epochs for training than existing methods and achieves higher hand gesture classification accuracy. While the proposed window-bard based axis-wise min max normalization effectively reduces domain shifts, it inevitably normalizes all coordinate axes to the same [o, 1] range, resulting in the loss of relative movement scale across axes. To address this issue, we introduce Dynamic Range Embedding (DRE), which compressively encodes the movement range variability of each axis into a scalar and injects it into the corresponding partition of the image. Consequently, the SKETCH cramework-combining visual representation construction, hindow-based axis-wise normalization, and axis-wise ORE-enables robust recognition even under varying absolute scales or sensor viewpoints 233. Furthermore, since each windowed input unit is processed independently, the model exhibits strong adaptation to environmental variability. This design is particularly well-suited for online (streaming) methods that process data on a per-frame basis for action recognition.
The main contributions of this study are as follows:
3

---

Direct utilization of raw data: Uses raw joint coordinates without complex preprocessing. preventing information loss due to hand-crafted feature extraction and simplifying the data gee-procesinng pipeline.
A simple and intuitive image graphing method-Skeleton Kinematics Extraction Through Coordinated grapH (SKETCH): SKETCH is a novel plotting technique that converts time-series skeleton data into images, effectively mitigating the geometric short- cut problem [50], where models tend to overfit to low-level spatial cues when using raw coordinates. By applying window-wise per-axis min-max normalization and incorporating Dynamic Range Embedding (DRE) to encode each axis's movement range and variability, the method achieves stable real-time recognition despite variations in viewpoint, scale, user body proportions, and domain gaps between training and testing data. is also inherently aypology-agnosiie enabling robust generalization across diverse skeleton structures and real-world conditions.
Integration with vision models and interpretable image features: By combining with powerful vision models such as ViT [T] and Swin Transformer [27]. the proposed approach attains superior classification and detection accuracy compared conventional methods and introduces a new wwradigm-distinet distinct from existing approaches-aa as an interpretable feature in the field of skeleton- based hand gesture recognition.
2 Related Work
Skeleton-based hand gesture recognition techniques primarily utilize sequence data composed of coordinates and frames. Depending on the characteristics of the dataset, existing methods can be broadly categorized into offline approaches [9] and online approaches [2] [2].
2.1 Offline Classification Methods
As shown in Fig. TO'a), offline datasets annotated such that each sequence corresponds to single gesture class. Offline hand gesture recognition methods can be broadly divided into two approaches. First, CNN and LSTM-based methods [ 25 46 47 utilize the current frame along with adjacent frame information to recognize gestures. This approach is effective for data with short sequence lengths but struggles to fully exploit information from frames that temporally distant. Second, to simultaneously consider fine-graieed gesture details and temporal information beyond what the first approach offers, recent studies have actively investigated architecture incorporating structures such as GCN and Transformer [55
voxelize the joints and then apply 3D CNN to simultaneously learn spatial and temporal information. DDNet [54] extracts joint collection distances (JCD) and employs ID CNN architecture incorporate temporal information between frames for gesture classification. However, these CNN-based techniques [25] [4] do not define the relationships between joints, making it difficult to fully capture the geometric characteristics of joint configurations. Duc to the issue of not defining the relationships between joints, GCN-based methods 11
CNN-based methods in 25
46
60 that consider joint connectivity have recently been actively studied. Notably, methods such ST-GCN 53
43
53
and MS-ISTGCN [43] reflect the structural characteristics of the skeleton in a graph, modeling the interactions between adjacent joints using techniques like adjacency matrix. However, since GCN-based approaches rely on connected adjacent information, they struggle to capture correlations between frames that are temporally distant. To address this, methodologies applying Transformer architectures which can leverage long sequence time-series data-have also been proposed [3 57). Nevertheless, applying Transformer architecture requires large amount of training data and extensive
training 48 49 57 58).
Offline recognition methods achieve high classification accuracy by generating a single prediction for an entire sequence labeled with one gesture class. However, they are unsuitable for real-world applications requiring frame-we wise recognition in continuous input streams.
2.2 Continuous (Online) Classification
In real-world environments, hand gesture recognition implemented using online hand gesture recognition methods shown in Fig. bO. A segment is defined as unit consisting of consecutive

---

frames that share the same label. In practical settings, gesture recognition results must be predicted every frame from a streaming data sequence. Moreover, the input sequence contains multiple types of gestures, with the gesture's ' start and end points as well as intervening non-gesture segments. Therefore, online classification techniques must simultaneously perform the following two tasks: (1) determining the type of gesture at each frame, and (2) detecting the start and end points of gestures. Online recognition methods [88 43 57 can be broadly categorized into two approaches: two-stage and one-stage methods. Two-stage methods [18] first determine whether gesture is present within segment and then decide the gesture class for that segment. This approach has the drawback that its overall performance heavily depends the effectiveness of the gesture presence detector. On the other hand, one-stage methods receive real-time input data and simultaneously predict the gesture's s start frame and end frame as well as the gesture class. RNN- based models (20 30] tend to forget past frames when the sequence length becomes long: to mitigate this, LST models [5] 36] have been proposed. However, since these methods classify frames based sequential processing. parallel processing is challenging and training speeds are slow. Transformer-base models (57] enable parallel processing, but they face difficulties in achieving high performance when there is insufficient training data.
To overcome these issues, OO-dMVMT [8. which utilizes sliding window technique, has been pro- posed and provides state-of-the-an (SOTA) performance. O0-dMVMT integrates spatial and temporal information by employing features such as joint collection distances (JCD) and the difference vectors for the same joint between consecutive frames. However, since the conversion from coordinates to JCD relies the Euclidean distance between joints, some of the fine- aheeggriied spatial information contained in the raw coordinate data is lost. Moreover, the normalization statistics computed from the training dataset are also applied during testing, based on the assumption that the test data follows similar distribution an assumption that may not hold in real-world scenarios.
2.3
VITST
To handle irregularly sampled time-series data, VITST 22 converts time-series data into line graph images and then employs pretrained vision transformer for classification. Specifically, each feature within the time-series data represented as an individual line graph. In each line graph, the horizontal axis represents time, while the vertical axis denotes the observed values of the corresponding feature. The converted image is then fed into a vision transformer. When ViT is used as the backbone, the image representing the graph is divided into fixed-size patches, and global relationships are captured by comparing the similarities between these patches. In contrast, the Swin Transformer divides the input graph image into local Swin windows, performs self-attentio within each Swin window, and adopts a hierarchical structure by shifting the Swin windows to merge information between adjacent Swin windows. However, because VITST employs a fixed per-feature normalization scheme, applying it directly to coordinate data that exhibit significant positional variability over time can obscure subtle variations in movement trajectories. Additionally, data that belong different absolute coordinate systems but share the same local coordinate system may not be recognized as the same class. Moreover, when features correspond to observed coordinate which can vary widely over time-their line graph representations struggle to convey the differing scales of multiple feature axes within a single image. ViTST, which does not incorporate dynamic range embedding, therefore fails to capture this variability.
3 Proposed Method
Figure illustrates the overall flow of the proposed training method, divided into three stages. As shown on the left, the SKETCH (Skeleton Kinematics Extraction Through grapH) module transforms raw four-dimensional skeleton data (time, x, and 2) into a visual eepresentation-enablii high- level feature extraction from the raw data without prior topology knowledge-then applies our novel Dynamic Range Embedding (DRE) to encode axis-wise relative range information, which is subsequently added via element-wiss broadcasting to the corresponding partition of the image. Here, F; represents the frame corresponding to the skeleton information at time t, and a window W, is defined by grouping Nr frames (where Nr denotes the number of frames per window), which is then converted into single graph image. In the middle stage, the graph image generated by SKETCH used as input to the backbone, where each image is divided into patches; relationships between patches are captured via self-attention within Swin windows, and subsequently, 'patch merging' [27
5

---

Skeleton Kinematies Extraction Through Coordinated grapit (SKETCH)
Backbone Network
Classification Regression heads
Fir
(Swin transformer)
Cls. head
Swin block
Lat
Stari Cramic head
Dynamic Range Embekling (DRE)
Swin block
Wio
Latart
Wy
(N
End frame
N,
Swin
Sliding window
Wa.Wey Way
... W. :W... We
Visual Represeacation
Graph image
Swiin window
Figure 2: Overall architecture the proposed method. First, the skeleton data are partitioned into windows (W,). The SKETCH module then normalizes each window per-axis, generates visual representation, and applies Dynamic Range Embedding to produce the final graph image. The transformed graph image is then passed through backbone, followed by two separate heads: one for gesture classification and another for start and end frame regression, enabling simultaneous gesture recognition and segment boundary detection. indicates element-wise broadcasting addition.
integrates local and global information. On the right, a classification head is responsible for gesture classification per window, and two separate regression heads estimate the start and end frames of the gesture within the window. Finally, the classification head and regression head are optimized using cross entropy loss and mean squared error (MSE) loss, respectively.
3.1 Transforming the Online Task into a Window Problem
As shown in Fig. II in this study, the continuous stream of input frames in an online environment grouped into window (W), thereby converting the real-time frame-based classification task into a window-based problem. The window W is represented as W, E Nexxx using time t. Each window, composed of NF frames, is represented corresponding to the numbers frames (Np), joints (j), and coordinates (c), effectively capturing the nempool characteristics of the skeleton data. Within each window, the inherent frame-level labels (Ne labels) are aggregated, and the most frequent label is assigned as the window-level label, strategy applied during both training and testing. A window may also be labeled non-gesture frames are dominant. In particular, when both gesture frames and non-gesture frames are mixed within a window, the gesture's start and end frames are also labeled to regress the boundaries of the gesture, where these annotations are normalized to a [o, 1] range by mapping the first and last frames of the window to 0 and 1, respectively. Aggregating frame-level labels in this way enables the assignment of a reliable representative label to each window, helping to mitigate the effects of noisy as transient frame-level mismatches between the actual gesture transition
a three-dimensional array with dimensions
'non-gesture' if
or inconsistent frames and the incorrectly labeled frames. Moreover, by converting the problem into a window-based one, leveraging the independent contextual information of each window and performing normalization within each window can reduce the domain gap between training and testing data.
3.2 Skeleton Kinematics Extraction Through Coordinated grapH (SKETCH) Module
Figure 3 shows the graph plotting, normalization, and Dynamic Range Embedding (DRE) employed in the proposed SKETCH module. The raw data is divided into windows (W), and min-max nor- malization is performed on each coordinate axis (x, and z) within each window. Through this process, the relative patterns and subtle differences in the data are emphasized without relying on the absolute coordinates. The normalized data is then converted into 2D image through graph plotting. Specifically, as shown on the right side of Fig. 3 the horizontal axis represents time (t), while the vertical axis represents each coordinate value of (x, y, and 2), and three subplots are vertically stacked to form a single graph image. each subplot, the coordinates as discrete points, which are connected by lines represent continuous movement. To preserve cross-axis relative scales that may be lost this transformation, we introduce our novel DRE to encode axis-wise relative range information. Specifically, for each axis, coordinate values across
joints are displayed

---

Max
Min
x1
Graph image
Input
(Nr
DRE
Output
subpio
Crssture start frame
Input
(Np
DRE,
Nens
TWO
Input
x (Nr--)
DRE;
Dynamic Range Embedding (DRE)
Frame 4
Framm 5
Frame
Figure 3: Process of the SKETCH module. Raw skeleton sequences are windowed and min-max normalized per coordinate axis (I, y, and 2). In each subplot, 26 lines corresponding to the joint coordinates are displayed. The dynamic range embedding module shares weights across all coordinate axes. Frames 4, 5, and 6 show skeleton data aligned with the time-axis points of the graph image.
Np frames and j joints undergo Z-normalization 35 and are then reshaped into vector of length R. x(Nx))), This vector is processed by two successive fully connected (FC) layers each followed by BatchNorm (BN) and ReLU activation and finally by a third FC layer to produce single scalar DRE value per axis, denoted as DRF,, DRF.. and DRF... These scalar values, which compactly represent the movement range or variability of each axis, are broadcast [4 to all pixel channels of the corresponding axis subplot and added to the visual representation, as illustrated in Fig. The entire process is termed Skeleton Kinematics Extraction Through Coordinated grapH (SKETCH).
Unlike VITST, as described in Section 2.3 the proposed SKETCH module represents the interactions among the joint lines within a single subplot while assigning separate subplots for each coordinate axis (I, y. and 2), as shown in Fig. 3 This joint-wise co-location preserves the relative spatial configuration between joints, enabling the model directly perceive their spatial dependencies. Moreover, by vertically stacking the subplots. temporal information is aligned along the horizontal axis, providing a coherent temporal flow that effectively preserves the continuity of motion, thereby aiding the model in identifying precise gesture boundaries such the start and end frames. In contrast, VITST's feature-wise isolated line graphs fragment the spatial context and obscure inter-joint dependencies. making it difficult for the model to capture spatial relationships and temporal continuity simultaneously (further details are provided in Appendix [E.I]. Furthermore, each joint consistently assigned fixed color, ensuring consistent joint representation. Unlike conventional methods that rely either on band-crafted features or on dataset- and sensor-specific topology definitions, our approach represents the time, graph image implicitly incorporates features typically extracted through rule-basee alle-based methods while enabling the model to autonomously learn richer representations from the raw data, and it is inherently copology-agnostie ensuring robust generalization across diverse skeleton structures. In this way. by representing raw data window basis and performing normalization for each coordinate within the window, the distribution differences (domain gap) between the training and testing data can be mitigated. Through the SKETCH module, converting four-dimensional data into wwo-dimensional images enables the effective utilization of powerful pre-trained classification models.
space, and joints of the data in single graph. This
3.3 Backbone Models
In this study, we adopt Vision Transformer (ViT) or Swin Transformer as backbone models, which tokenize input into patches and utilize Transformer architectures for joint gesture classification and boundary localization [1] 27 Swin Transformer further captures both local and global dependencies via hierarchical patch merging and Swin window shifting, enabling the model learn intra-subplot relationships (among joints sharing the same coordinate axis) and inter-subplot relationships (across
7

---

different coordinates and time steps). To support this, each backbone is equipped with classification head and two regression heads immllmmeeeeeeeeed single linear layers- for predicting gesture types and temporal boundaries between gesture and non-gesture segments, respectively.
3.4 Loss Function
The model employs a combined loss function consisting of classification and regression terms. The classification loss (Lets) uses cross entropy as in [8] to classify the type of gesture, while the regression losses (Lstart, Lend) mean squared error (MSE) in [8] predict normalized start and end frame positions within each window. This formulation enables the model to simultaneously optimize gesture classification and boundary prediction. The total loss defined
L = Lets A2hstart Aahend-
(I)
3.5 Online Inference Process
In the inference phase, the output from the regression head is not used; only the results from the classification head are utilized. Continuous window- predictions are post-processed following Algorithm [ to classify frames into certain gesture. Briefly, the input frames are grouped into windows of fixed length (Np), and for each window, class prediction value (j)) is obtained using model (f). These predictions are then accumulated
queue structure of fixed size (Q), and y is determined by applying majority voting method to the class prediction values stored in the queue, enabling noise-robust frame-wise classification. However, waiting for the final detection after collecting Q prediction values introduces certain delay. Therefore, appropriate value for Q must be applied, taking into account both noise reduction and delay.
Experiments
All experiments were conducted using PyTTorch 34 on a single NVIDIA RTX A6000 or NVIDIA 100 GPU, and the model was trained and tested using the SHREC'19 and SHREC'22 datasets (details provided in Appendix B). Appendix reports the impact of various ablation versions of the SKETCH module, loss configurations, and related factors, etc. Training strategies and implementation details are provided in Appendix G
4.1 SHREC' 19 Results
Table I: Comparison of benchmark and proposed methods SHREC' 19. DR and FP denote detection rate and false positive, respectively. Values in parentheses indicate backbone configurations, where backbone type: V = VIT, S Swin; model size: L Large, B = Base, S Small; patch size; Swin window size (for Swin only); and image resolution. JCD, FD, Adj, and Ref indicate the use of joint collection distances, frame difference vectors, adjacency matrices, and reference-joint difference vectors features, respectively. Plot denotes the use graph image as input. 'Aug.' indicates the use of additional data augmentation strategies, which are detailed in Appendix G
Method
DR
FPI
Time (s)
FPS
JCD
FD
Adj
Ref
Plot
PSUMNet
0.64
0.22
0.0250
40
MS-G3D
0.69
0.25
0.0303
SeS-GCN 38
0.75
0.12
0.0020
500
SW 3-cent
0.76
0.19
0.0030
333
DS-GCN 51
DSTA 41
0.81
0.08
0.0088
114
DG-STA [
0.81
0.07
0.0042
238
DDNet S4
0.82
0,10
455
DDN YE
0.82
0.10
0.0022
455
BlockGCN 60
0.83
0.04
DDeePGRU
0.85
0.10
0.0030
333
ProtoGCN
24
0.86
0.05
0.0334
30
00-dMVMT
0.05
0.0058
172
SKETCH (V-L-16-384)
0.90
0.03
0.0176
SKETCH (S-S-4-7-224)
0.88
0.04
0.0039
256
SKETCH (S-B-4-12-384)
0.91
0.03
0.0091
110
SKETCH (S-1-4-12-384)
0.92
0.02
0.0142
SKETCH (S-L-4-12-384) + Aug.
0.93
0.01
0.0142
70
8

---

As presented in Tab. D and illustrated in Fig. 12 the proposed SKETCH outperforms previous SOTA methods in gesture- wise detection accuracy across diverse categories, while also ensuring real-time operation with throughput of at least 30 frames per second (FPS). This performance is particularly noteworthy given the challenging evaluation protocol of the SHREC' 19 dataset [4, which involves clear domain shift, as the test subjects were entirely different from those used in training. Despite this domain discrepancy, the SKETCH module exhibits robust generalization ability, suggesting its effectiveness in mitigating the domain gap-more more so than existing methods.
4.2 SHREC'22 Results
Table 2: Recognition performance the SHREC'22. JI denotes Jaccard index. Configurations follow the same notation as deseribed in Tab. I
Method
DRT
FP1
J11
Delay (ft.)
Time
JCD
FD
Adj
Plot
DeepGRU
30
0.26
0:25
0.21
8.0
0.0031
DG-STA
5
0.51
0.32
0.40
8.0
0,0042
SeS-GCN
0.60
0.16
0.53
8.0
0.0018
PSUMNet
41
0.62
0.24
0.52
8.0
0.0244
MS-G3D
0.21
0.57
0.0293
Stronger
0.72
0.34
0.59
14.8
0,1000
DSTA
0.0092
2ST-GON+5F
I2
0.74
0.23
0.61
13.3
0.0021
TN-ISM+JD
12
0.77
0.23
0.63
10.0
0.0046
Causal TCN
0,80
0.29
0.68
19.0
0.0280
DDNet [54]
0.88
0.16
0.78
8.0
0,0022
SKETCH (S-B-4-12-384)
0.91
0.06
0.86
0.0097
SKETCH (S-L-4-12-384)
0.07
0.87
0.0124
SKETCH (5-.844-1 + Aug.
0.95
0.06
0.91
8.0
0,0124
presents the classification results of our method on the SHREC'22 dataset 12) demonstrating competitive performance against SOTA approaches. Unlike conventional methods that rely on hand-crafted features, approach leverages SKETCH-basee visual plots for a more interpretable representation. As illustrated in Figs. 7 13, and 14 our method effectively distinguishes visually similar dynamic gestures such Circle, V. and Cross-task. where hand-crafted feature-based methods often struggle due their limited use raw skeletal data.
Peddiciiontionnt truth
Sturt
framnes
Noa frunes
Method
Three
Four
Cross
Grab
Deny
Knoh
PPG
TheTTe
Non
FouFFeu
FoutFon
V/V
Non
CrerCro
GraiGea
Non
Non
Deai
Koo
Gnd CAM
Cred.
NonFFo NonFau
Nen
Figure 4: Grad-CAM visualizations illustrating how boundary regression loss enhances the model's ability to localize gesture boundaries. Each Grad-CAM image corresponds to single input window. The red and blue lines represent the start and end frames of the gestures within each window, respectively. For simplicity, abbreviated gesture names (e.g. Three + Thr) are used in the figure.
Figure 4 shows the results of Grad-CAM [40] to visualize the impact of the regression losses (Estort and Lend) on the classification process. By detecting the start and end frames of a gesture, the model learns localized boundaries rather than performing global classification. This enables refined discrimination even in graphs where gestures and non- gestures are mixed, making it difficult judge single classification based solely on the overall gesture. In contrast, models trained without regression loss exhibit tendency to attend to broader temporal regions, overlooking the actual boundaries between different consecutive gestures. This often leads to ambiguities in temporal localization and increases the likelihood misclassifying gestures, particularly when gesture transitions are subtle or occur in quick succession. Such boundary-awand modeling allows for more adaptive and precise

---

Ok
Deny
Figure 5: Visualization of joint-level attention for gestures in the SHREC'22 dataset. The color intensity indicates the magnitude of attention assigned by the model to each joint, with attention increasing progressively from blue to red (see Fig. 15 for extended visualizations).
predictions compared to classification-onlyy approaches, In addition, it opens up new perspectives on the interpretability of features, thereby enhancing the reliability of gesture recognition.
Humans distinguish hand gestures by focusing on key joints or finger movements that uniquely define each gesture. To examine whether the model attends to gestures in similar way, applied Grad-CAM to the SKETCH-base graph images and mapped the resulting attention scores back to the 3D skeleton coordinates for quantitative evaluation. As illustrated in Figs. 5 and 15 attention maps reveal that the model focuses on task-relevant joints in a human-like manner, suggesting that it captures high-level semantic representations rather than overfitting geometric biases. This supports the interpretability and intuitiveness of our SKETCH-base graph transformation. For further details on the attention mapping process, qualitative interpretation, see the description in Fig. [5] In addition, gesture-specific attention patterns and their semantic alignment are thoroughly analyzed in Fig.
5 Conclusion
In this study, we proposed the Skeleton Kinematics Extraction Through Coordinated grapH (SKETCH) module to overcome the limitations of conventional hand-crafted feature-based approaches in skeleton-bar based hand gesture recognition. The proposed method converted four-dimensional (time, I, y, and 2) skeleton raw data into simple and intuitive visual graphs, with Dynamic Range Embed- ding (DRE) further preserving axis-wise relative scale during this transformation, demonstrating that it could fully leverage the rich information that is easily overlooked by conventional methods. In particular, while existing approaches provided non- non-intuitive and uninterpretable representations using a sequence of data, the SKETCH module opened up new perspective in hand gesture recog- nition through an interpretable data representation. Moreover, by applying an online, independent min-max normalization technique, it effectively mitigated the domain gap caused by changes in sen- sor viewpoints and individual body proportion differences, thereby achieving consistent performance across various environments. Through these designs, the framework becomes aggnosy-agnostic,, ensuring robust generalization across diverse skeleton structures, Experimental results showed that the proposed SKETCH method outperformed existing approaches in terms of high detection and classification performance on major benchmarks such as SHREC' 19 and SHREC'22. These results suggested that skeleton-basee hand gesture recognition can play a key role in realizing intuitive human- computr interaction in fields such as intelligent robotics and mixed reality systems, and our approach demonstrated the potential for extension to various future applications.
5.1 Limitations and Future Work
Although we demonstrated that high-level features can be extracted directly from raw skeleton data with our novel representation, the current architecture still makes each decision independently and therefore ignores information from preceding windows. In future work, we plan to overcome this limitation by moving beyond fixed-length chunks an adaptive scheme that captures longer-ran range temporal context, ultimately enabling a unified architecture suitable for both online and offline 4D action recognition tasks. Moreover, while our evaluation focused on skeleton-based hand gesture recognition, the topology-agnostie design of SKETCH suggests its potential applicability to broader domains such as skeleton-based human action recognition, which we leave as promising directions for future research.
10

---

Acknowledgements
This research was supported by the National Research Foundation (NRF) funded by the Korean gov- emment (MSIT) under Grant RS2023-00208763, and R.S-2025-0263361 (Intelligent Semiconductor Technology Development Program).
References
[1] Danilo Avola, Marco Bernardi, Luigi Cinque, Gian Luca Foresti, and Cristiano Massaroni. Exploiting recurrent neural networks and leap motion controller for the recognition of sign language and semaphoric hand gestures. IEEE Transactions on Multimedia, 21(1):234-222 2434,245, 2019. doi: 10. 10.1 1.2556999
[2] Ariel Caputo. Andrea Giachetti, Simone Soso, Deborah Pintani, Andrea D' Eusanio, Stefano Pini, Guido Borghi, Alessandro Simoni, Roberto Vezzani, Rita Cucchiara, et al. Shree 2021: Skeleton-based hand gesture recognition in the wild. Computers & Graphics, 99:201-211, 2021.
[3] Fabio M Caputo, Pietro Prebianca, Alessandro Carcangiu, Lucio D Spano, and Andrea Giachetti. Comparing 3d trajectories for simple mid-air gesture recognition. Computers & Graphics, 73: 17-25, 2018.
[4] Fabio Marco Caputo, S Burato, Gianni Pavan, Théo Voillemin, Hazem Wannous, Jean-Philippe Vandeborre, Mehran Maghoumi, EM Taranta, Alaleh Razmjoo, Joseph J La Viola Jr, et Shree 2019 track: online gesture recognition. In Eurographics Workshop on 3D Object Retrieval, volume 160897, pages 93-102. The Eurographics Association, 2019.
[5] Yuxiao Chen, Long Zhao, Xi Peng, Jianbo Yuan, and Dimitris N Metaxas. Construct dy- namic graphs for hand gesture recognition via spatial-temporal attention. arXiv preprint arXiv: 2019.
[6] Zijun Cheng and Yiguo Wang. Partial contrastive point cloud self-supervised representation learning. Scientific Reports, 15:16182, 2025. doi: 10.1038/s41198 198--025-98521-; URL https
//www. www.ra.tuee..mm www./articeee///1111111 00000-002599882552 98521-y
[7] Hyung-gun Chi, Myoung Hoon Ha, Seunggeun Chi, Sang Wan Lee, Qixing Huang, and Karthik Ramani, Infogen: Representation learning for human skeleton- action recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 20186-20196, June 2022.
[8] Federico Cunico, Federico Girella, Andrea Avogaro, Marco Emporio, Andrea Giachetti, and Marco Cristani. Oo-dmvmt: A deep multi-view multi-task classification framework for real-time 3d hand gesture classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2745-2754, 2023.
[9] Quentin De Smedt, Hazem Wannous, Jean-Philippe Vandeborre, Joris Guerry, Bertrand Le Saux. and David Filliat. Shrec' 17 track: 3d hand gesture recognition using depth and skeletal dataset. In 3DOR- 1Oth Eurographics Workshop on 3D Object Retrieval, pages 1-6, 2017.
[10] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Icce, 2009.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint urXiv20110.19929 2020.
[12] Marco Emporio, Ariel Caputo, Andrea Giachetti, Marco Cristani, Guido Borghi, Andrea D'Eusanio, Minh-Quan Le, Hai-Dang Nguyen, Minh-Triet Tran, Felix Ambellan, et al. Shrec 2022 track on online detection of heterogeneous gestures. Computers Graphics, 107:241 2022.
11

---

[13] Mallika Garg, Debashis Ghosh, and Pyari Mohan Pradhan. Gestformer: Multiscale wavelet pooling transformer network for dynamic hand gesture recognition. In Proceedings of the IFEE/CVF Conference on Computer Vision and Putlern Recognition (CVPR) Workshops, pages 2483-2483, June 2024.
[14] Priya Goyal, Piotr Dollár, Ross Girshick, Picter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangging Jia, and Kaiming He. Accurate large minibatch sgd: Training imagenet in hour, 2018. URL.httpss//arxiv.. http:/aaxvvvv....aae.. hts///bs///0 wttps//70.00277777
[15] Alexander Kapitanov, Karina Kvanchiani, Alexander Nagaev, Roman Kraynov, and Andrei Makhliarchuk. Hagrid hand gesture recognition image dataset. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4572-4581 January 2024.
[16] Seong-heum Kim and Donghyeon Cho. Viewpoint-awart action recognition using skeleton- based features from still images. Electronics, 10(9), 2021. ISSN 2079-9292. doi: 10.3390/ electromics (0091 18. URL BtLLhttps://// //ttps::/wwww.ee http://wwwwwpppp..cmmm
[17] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint MKKiv:660.0090000 2016.
[18] Okan Köpükküü Ahmet Gunduz, Neslihan Kose, and Gerhard Rigoll. Real-tin time hand gesture de- tection and classification using convolutional neural networks. In 2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019), pages 1-8. IEEE, 2019.
[19] Chenyang Li, Xin Zhang, Lufan Liao, Lianwen Jin, and Weixin Yang. Skeleton based gesture recognition using several fully connected layers with path signature features and temporal transformer module. Proceedings of the AAAI Conference on Artificial Intelligence, 33(1)): 8585-8593, Jul. 2019. doi: 10. 1609/aaaiiiivv htt00///aaav3331.33333 333000.330011188. https:ts::///g aaai org/
index. php/ /aa////AA///// wwwaaarreeeeeeeeeeeee.
[20] Chuankun Li, Shuai Li, Yanbo Gao, Xiang Zhang, and Wanqing Li. A two-stream neural network for pose-based hand gesture recognition. IEEE Transactions on Cognitive and Developmental Systems, 14(4):1594-1603, 2021.
[21] Rui Li, Hongyu Wang, and Zhenyu Liu. Survey mapping human hand motion to robotic hands for teleoperation. JEEE Transactions on Circuits and Systems for Video Technology. 32 (5):2647-2665, 2021.
[22] Zekun Li, Shiyang Li, and Xifeng Yan. Time series as images: Vision transformer for irregularly sampled time series. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 49 49187-49204, Curran Associates, Inc., 2023, URL / httttppp//proocediiggs. neurips. cc lalpppppp___ll_ wwwrre-ooffeeenn.cmmm .ddf
[23] Hanchao Liu, Yujiang Li, Tai-Jiang Mu, and Shi-Min Hu. Recovering complete ac- tions for cross-dataset skeleton action recognition. A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 22055-92083 Curran Associates, Inc., 2024. httphttss:// httttss:/proceeednnng www.eeeddddinnnnnnenneeeeeeeeeee wwtpuurps..ccpaapaeeeeee hopppp www.fflllppppae 1illes/aaaer/222//iila a78f 77554222228111183377777777000000
Conference .dff
[24] Hongda Liu, Yunfan Liu, Min Ren, Hao Wang, Yunlong Wang, and Zhenan Sun. Revealing key details to see differences: A novel prototypical perspective for Akeleton-basee action recognition. In Proceedings of the Computer Vision and Puttern Recognition Conference, pages 29248- 29257, 2025.
[25] Jianbo Liu, Yongcheng Liu, Ying Wang, Veronique Prinet, Shiming Xiang, and Chunhong Pan. Decoupled representation learning for skeleton-based gesture recognition. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5751-5760, 2020.
12

---

[26] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transfor- mation for multi-view 3d object detection. In Shai Avidan, Gabriel Brostow. Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022, pages 531-548, Cham, 2022. Springer Nature Switzerland.
[27] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[28] Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, and Wanli Ouyang, Disentangling and unifying graph convolutions for skeleton-base based action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 143-152, 2020.
[29] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https //arxiv. owg//ogg/////.ooooooooo.ooo.o /ww.///........
[30] Mehran Maghoumi and Joseph J LaViol.. Deepgru: Deep gesture recognition utility. In Advances in Visual Computing: 14th International Symposium on Visual Computing, ISVC2019 Lake Tahoe, NV, USA, October 7-9, 2019, Proceedings, Part I 14, pages 16-31. Springer, 2019.
[31] Yunyao Mao, Jiajun Deng, Wengang Zhou, Yao Fang, Wanli Ouyang, and Houqiang Li. Masked motion predictors are strong 3d action representation learners. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 10181-10191, October 2023.
[32] Eshed Ohn-Bar and Mohan Manubhai Trivedi. Hand gesture recognition in real time for automotive interfaces: A multimodal vision-based approach and evaluations. IEEE transactions on intelligent transportation systems, 15(6))2368- 2377, 2014.
[33] Omar Oreifej and Zicheng Liu. Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 716-723, 2013.
[34] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, perforerfore deep learning library. Advances in neural information processing systems, 32, 2019.
[35] Karl Pearson. Contributions to the mathematical theory of evolution. iii standard proportion the and standard deviation. Philosophical Transactions the Royal Society of London. Series A. 1 185:325 414,,,444111 1894, doi: 10. 1098/rsta. 1ttp:/sta.189..00000 1894.009.
[36] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by mack-propagating errors. nature, 523(6888) 1986.
[37] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 11:2211-22,, 2015.
[38] Alessio Sampieri, Guido Maria D' Amely di Melendugno, Andrea Avogaro, Federico Cunico, Francesco Setti, Geri Skenderi, Marco Cristani, and Fabio Galasso. Pose forecasting in industrial human-robot collaboration. In European Conference on Computer Vision, pages 51-69. Springer, 2022.
[39] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
[40] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
13

---

[41] Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Decoupled spatial- attention network for exeleton-based settonggesture recognition. In Proceedings of the Asian conference on computer vision, 2020.
[42] Changyong Shu, Jiajun Deng, Fisher Yu, and Yifan Liu. 3dppe: 3d point positional encoding for cunsformer-basee multi-camera 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3580-3589, October 2023.
[43] Jae-Hun Song, Kyeongbo Kong, and Suk-Ju Kang. Dynamic hand gesture recognition using improved spatio-temporal graph convolutional network. IEEE Transactions Circuits and Systems for Video Technology, 32(9):6227-6233 2022.
[44] Neel Trivedi and Ravi Kiran Sarvadevabhatla. Psumnet: Unified modality part streams are all you need for efficient pose-based action recognition. In European Conference on Computer Vision, pages 21 1-227. Springer, 2022.
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[46] Peng-Shuai Wang. Octformer: Octree-based transformers for 3d point clouds. ACM Truns. Graph., 42(4), July 2023. ISSN 0730-0301. doi: 10.1 10.1145/3592131. Utt.:ttps://do org/ 10.1145/3592131
[47] Peng-Shuee Wang, Yang Liu, Yu-Xiao Guo, Chun-Ya Yu Sun, and Xin Tong. O-cnn: octree- based convolutional neural networks for 3d shape analysis. ACM Trans. Graph., 36(4), July 2017. ISSN 0730-0301. 5072959.3073603
doi: 10. (0.114453072999330777 3073399
wttp:///ooo..r/////// 111111....11. 1145/
[48] Xiaoyang Wu, Yixing Lao, Li Jiang. Xihui Liu, and Hengshuang Zhao Point trans- former v2: Grouped vector attention and partition-based pooling. In S. Koyejo, S. Mo- hamed, Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 33330-33342. Curran Associates, http://eeeeeaauuuuiiieepeaappppa cc wwaeeeeffiessppppeeeeeeee paaiee///paer//////iee/// 7788c666133331466619968686886186 how.://0ee..00eeeeoooooooooooono
Inc., 2022. URL
Paper Conference pdf
[49] Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler faster stronger. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4840 4851, June 2024.
[50] Xiaoyang Wu, Daniel DeTone, Duncan Frost, Tianwei Shen, Chris Xie, Nan Yang, Jakob Engel, Richard Newcombe, Hengshuang Zhao, and Julian Straub. Sonata: Self-supervised learning of reliable point representations, 2025. URL WtRLsstts:///// hnttps://arxiv. org/ abaaabas/255/ 16429
[51] Jianyang Xie, Yanda Meng, Yitian Zhao, Anh Nguyen, Xiaoyun Yang, and Yalin Zheng. Dynamic semantic-based spatial graph convolution network for skeleton-based human action recognition. In Proceedingy of the AAAI conference on artificial intelligence, volume 38, pages 6225-6233 2024.
[52] Saining Xie, Jiatao Gu, Demi Guo, Charles R. Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020, pages 574-591, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58580-88
[53] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018
[54] Fan Yang, Yang WU, Sakriani Sakti, and Satoshi Nakamura. Make skeleton-based action recognition model smaller, faster and better. In Proceedings of the 1st ACM International Conference Multimedia in Asia, pages 1-6, 2019.
14

---

[55] Fan Yang, Lin Guo, Zhi Chen, and Wenbing Tao. One-inlier is first: Towards ef- ficient position encoding for point cloud registration. In S. Koyejo, S. Mohamed. A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Infor- mation Processing Systems, volume 35, pages 1982-6995. Curran Associates, Inc., 2022. URL https::/pproeedddngg neurips. cc/
wttpppssccpppp.pppeeeeeep pwpppprperr__r ciles/fiil//eee @1154400caa317/72277166662282822 PawpseeeedddaPee--ooooeoe Conferecen pdf
[56] Xumin Yu, Lulu Tang. Yongming Rao, Tiejun Huang. Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19313- 19322, June 2022.
[57] Dongdong Zhao, Hongli Li, and Shi Yan. Spatial-temporal synchronous transformer for skeleton- based hand gesture recognition. IEEE Transactions on Circuits and Systems for Video Technology, (13):1403-14122 2023.
[58] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip H.S. Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 19259-16268, October 2021.
[59] Jianqiao Zheng, Xueqian Li, Sameera Ramasinghe, and Simon Lucey. Robust point cloud processing through positional embedding. In 2024 International Conference 3D Vision (3DV), pages 1403-1412, 2024. doi: 10.10033D6644..044000111...
[60] Yuxuan Zhou, Xudong Yan, Zii-Qi Cheng, Yan Yan, Qi Dai, and Xian-Sheng Hua. Blockgen: Redefine topology awareness for skeleton-based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 2049-2058,
June 2024.
15

---

Appendix
Table of Contents
A Overcoming Geometric Shortcut and Coordinate Bias via SKETCH
17
Dataset Description
18
Evaluation Metrics
19
Gesture-Specific Attention Patterns and Interpretability Analysis
20
Ablation Studies
21
E.1
Ablation Study: Plotting Methods
21
E.2
Ablation Study: Impact of Regression Loss, Loss Weights, and Window Length
22
E.3
Ablation Study: Conventional Hand-Crafted and Proposed Features
23
E.4
Ablation Study: Normalization Strategies
24
E.5
Ablation Study: Dynamic Range Embedding
24
E.6
Ahlation Study: Robustness on Pose Estimation Errors
25
E.7
Ablation Study: Regression Head and Dynamic Range Embedding
25
Additional Visualizations
26
Training Strategies and Implementation Details
31
Toy Example Evaluation of Geometric Shortcut
32
Impact of Spatial Emphasis in Skeleton-Basd Bassdd Gesture Recognition
16

---

A Overcoming Geometric Shortcut and Coordinate Bias via SKETCH
In prior 3D-based studies, there have been continued efforts to encourage models to autonomously learn rich representations directly from unprocessed raw point data, as discussed in [56]. In particular, goordinate-basee 3D inputs such as skeletal joints and point clouds are commonly processed by feeding normalized coordinate values directly into the model, without any intermediate representa- tional transformation. However, this approach often leads to an excessive reliance on the absolute coordinates of the training data, causing the model to overfit to low-level geometric cues [6] 50 52 rather than learning semantically meaningful representations.
This phenomenon has also been clearly identified in recent work such as Sonata 50 which highlights
the geometric shorteut as a key limitation in 3D self-supervised learning 550 52 56 The geometric shortcut refers to the model's Iss tendency to overly rely on low-level spatial hints such as surface normal vectors, point heights, and relative positions, rather than learning meaningful semantic features. This issue is particularly pronounced in 3D point cloud data, which is inherently sparse and non-grid in structure, making more likely for coordinate values to be misinterpreted as semantic cues. For example, in environments where objects like the floor, desk, and ceiling can be distinguished solely based on their y- coooddnaate (height), models may memorize these positional cues instead of learning abstract, high-level representations. As result, the learned features tend to collapse into simple geometric patterns, ultimately degrading the model's s generalization ability.
(4) Origiual space R?))
(b) 1.2 ncenulization
(c) Min-ttax nomuliatioon (fo,1) RR')
(d) Z-score nonmall isation (standardized space)
sphere)
Uhster
Chster2
Chuke3
2+
O
Orinc III
Detan Vin
Figure 6: Comparison of raw and normalized 3D spaces using synthetic point clusters, demonstrating that the relative spatial relationships between clusters persist across different normalization methods. Black boundaries indicate each normalization space: (a) original space with three color-coded clusters in Octants I, III. and VIII: (b) L2 normalization on the unit sphere; (c) min-max normalization in the [0, 11s cube; (d) Z-score normalization in standardized 330 cube.
Furthermore, Fig. 6 illustrates that simple normalization alone is insufficient to fundamentally climinate coordinate bias (56). The figure shows red, blue, and green point clusters located in Octant I (æ > 0, > 0, > 0), Octant III (Ir < 0, y < 0, 2 > 0), and Octant VIII (x > 0, y < 0, < 0) of the original dineediimnsional space, respectively, and demonstrates that these clusters retain their relative spatial configuration even after normalization. Whether L.2 normalization [39], min-man scaling, or Z-score normalization (35] is applied, the directional and positional relationships between clusters remain preserved, thereby allowing the model to continue learning features based on coordinate- dependent spatial cues. Therefore, applying normalization to the raw data still fails to resolve the geometric shorteut problem.
For example, in the case of an arbitrary spatial movement over time (c.g.. Octant I-+ III + VIII), the same dynamic gesture may be observed as different coordinate trajectories (c.g., Octant III + I + VIII) depending on the viewpoint. Although the underlying motion remains identical, the resulting coordinate trajectory differs due to changes in the observation angle. In other words, the same gesture can be represented by entirely different coordinate sequences depending on the viewpoint. Even after normalization, the relative arrangement and orientation of the points are preserved, allowing the model to learn low-level representations based on such spatial cues. As result, normalization alone is insufficient to eliminate viewpoint-inducar variation, and it fails to prevent the model from overfitting to geometric shortcuts based on specific positions or directions. A similar phenomenon can also be observed in Static Gestures (SG). Even when SG are performed within a single octant, the orientation
17

---

of the hand may vary depending on the sensor's viewpoint. As a result, the same gesture may exhibit instance, the 2-coordinate of the index finger may appear greater or smaller than that of the thumb. Such variation arises not only from changes in viewpoint also from individual differences in body proportions. This indicates that simple normalization insufficient for achieving viewpoint-agnostic or subject-invariant representation learning.
different inter-joint spatial relationships-
Such issues have also been observed in prior studies hand gesture recognition. For example, models like deepGRU (30), which rely (solely) on raw coordinate inputs, have exhibited significantly degraded performance due to representation collapse.
To address these challenges, this study proposes a plotting- based intermediate medium (SKETCH) that goes beyond simple coordinate normalization and transforms normalized coordinates into representations that alleviate sensor viewpoint and subject-specific variations. Specifically, raw point joint data converted into graph images, which are then processed by pretrained image backbones (e.g.. ViT or Swin Transformer) to effectively extract high-level semantic features in the image domain. This approach enables more effective utilization of the representational potential of raw 3D data, addressing the geometric shortcut problem inherent in direct coordinate input while providing robust pathway for expressive representation learning.
B Dataset Description
The datasets used for training and evaluation were both online datasets: SHREC'2015 2019 4 and SHREC'2022 [12]. The SHREC 2019 dataset captures five distinct hand gestures Cross (X), Circle (O), V-mark (V), Caret (*), and Square (I 100-perfomeed by 13 users using LeapMotion sensor 0.05-second intervals. Each frame records 3D positions (x, y. and 2) and quaternions (r, y, 2, and w) from 16 hand joints, though only the positional data is used in our approach for gesture classification and gesture boundary prediction. Data from four users are used for training, and data from the remaining nine users are used for testing. One sequence missing annotation was excluded from performance evaluation.
The 2022C'2022 dataset comprises 144 training sequences and 144 testing sequences captured with the Hololens2 device [12), with each sequence consisting of 3D coordinates (x, y, and z) for 26 joints across frames of varying lengths. It contains 16 types of gestures, as depicted in the Fig. 7 which are further classified into four categories: Static (SG: One, Two, Three, Four, Ok, and Menu), Dynamic (DG: Left, Right, Circle, V, and Cross), Fine-grained Dynamic (FG-DG: Grab and Pinch), and Dynamic-periodis gestures (D-PG: Deny, Wave, and Knob).
The SHREC' 19 online-gesture dataset is publicly available for non-commereial academic use (no explicit license is specified by the organizers), whereas the SHREC'22 dataset is released under the Creative Commons Atrrburion-oonCCmmevaaa--DD 4.0) license.
4.0 International (CC BY-NC-ND
ONE
TWO
THREE
FOUR
OK
MENU
LEFT
RIGHT
CIRCLE
CROSS
GRAB
PINCH
DENY
WAVE
KNOB
Figure 7: Illustration of the 16 gesture classes in the THREC'2022 dataset 12 categorized into Static, Dynamic, Fine- grained Dynamic, and Dynamic-periodic gestures.
18

---

C
Evaluation Metrics
To quantitatively evaluate the performance of gesture recognition, adopted four evaluation metrics: detection rate (DR), false positive (FP) rate, Jaccard index (JI), and delay. These metrics were computed for each gesture class and averaged across all classes to obtain the final results. The evaluation protocol strictly follows the official metrics defined in the SHREC'22 [12] online gesture recognition track. A predicted gesture segment considered correct detection if it satisfied the following conditions: The predicted label matches the ground-truth label. The predicted segment length does exceed twice the ground-truth segment length. The temporal (frame) overlap ratio between the predicted segment [fp. Ip] and the ground-truth segment [ is greater than or equal 0.5:
Detection criterion A predicted gesture segment is considered a correct detection if satisfies all of the following conditions:
1. The predicted label matches the ground-truth label.
2. The predicted segment length does not exceed twice the ground-truth segment length.
3. The temporal (frame) overlap ratio between the predicted segment [ff. fpl and the ground- truth segment [fğ, fý] is at least 0.5:
overlap min(Jz fp) max (fg. f8)
f fa
Evaluation metrics
Based on the above matching rule, we define the following metrics:
Detection rate (DR):
TP
DR GT
where TP is the number of correctly detected gesture segments, and GT is the total number of ground-truth gesture segments.
False positive rate (FP rate):
FP FPeount
FP-
GT
where FP count is the number of unmatched predicted gestures.
Jaccard index (JI):
TP JI
FN+
where FN is the number of missed gesture segments.
Delaa:
Q
1, if Q is even,
2
Delay
Q
2 if Q is odd.
where Q is the queue size in Algorithm
metrics were calculated for each gesture class and the overall performance was computed as mean value across all gesture classes. This evaluation protocol enables comprehensive assessment of detection accuracy, robustness to false positives, segmentation quality, and temporal prediction latency.
19

---

D Gesture-Specific Attention Patterns and Interpretability Analysis
Luet
Righ
888 88 88 R
Menu
Circle
Pinch
Cross
Figure 8: Visualization of model's attention during gesture prediction for the Left, Right, Menu, Circle, Pinch, and Cross gestures using Grad-CAM 40 All examples were extracted from different sequences.
Figure 8 illustrates Grad-CAM 40 result on the hand gesture recognition process through the SKETCH. demonstrates that the unique motion patterns of each gesture align with the regions that the model focuses on, thereby showcasing the interpretability of our proposed feethod-enablee by its use of raw data-in contrast to conventional, uninterpretable hand-crafted features. In this manner, the transformation process using the SKETCH technique reflects the intrinsic characteristics of each gesture and confirms that forms an interpretable structure that consistently aligns with the regions on which the model focuses.
For instance, the Left and Right gestures are characterized by the hand's overall movement toward the left or right direction while the hand shape itself remains fixed; during the SKETCH transformation, linearly increasing decreasing graph is formed in the z-coordinate subplot. Accordingly, the model's 's attention is observed to align closely with these linear patterns in the graph. The Menu gesture, defined by a static open-handed posture, exhibits pattern of widely spaced lines distributed across the entire image, suggesting that capturing the overall structural pattern plays more critical role in prediction than relying solely localized information-anl observation further supported by the model's is behavior. The Circle gesture involves circular movement of the hand without altering its shape. This results in waveform patterns appearing on both the z- and y-coordinate subplots after the SKETCH transformation. Thus, it is confirmed that the model focuses on the regions where these waveforms persist. The Pinch gesture distinguished by the convergence the thumb and index finger coordinates, and the SKETCH transformation results in pattern where the lines in the æ and y subplots gradually widen. The model's emphasis on the widening region indicates that this is critical element in distinguishing the Pinch gesture.
20

---

E
Ablation Studies
E.1
Ablation Study: Plotting Methods
Table 3: Quantitative comparison on the SHREC'22 dataset of gesture classification error rates (%) across four gesture categories- Gestures (SG), Dynamic Gestures (DG), Fine- grained Dynamic Gestures (FG-DG), and synamic-ppriodee Gestures (D-PG)- under different ablation settings of the proposed SKETCH module, including removal of color, ViTST-style joint coordinate arrangement, and addition of markers. The values parentheses "()" indicate the error rate increase compared to the standard SKETCH module.
Gestures
SKETCH
(a) Black
(b) VITST
(c) Marker
SG
6.02
15.74 (+9.72)
29.63 (+23.61)
8.80 (+2.78)
DG
5.00
8.89 (+3.89)
7.78 (+2.78)
7.78 (+2.78)
FG-DG
12.50
15.28 (+2.78)
30.56 (+18.06)
15.28 (+2.78)
D-PG
13.89
20.37 (+6.48)
42.59 (+28.70)
19.44 (+5.55)
Statie gestures
Dynamic-periodic gestures
One
Four
OK
Menu
Deay
Wave
Knob
60 Blach
Mccled
60 Marker (6s VITES
Figure 9: Visualization of graph image representations generated by different ablation versions of the SKETCH module for both static gestures (left) and Denamicc-priodii gestures (right). Each row corresponds to a different visualization method: the standard SKETCH design (SKETCH), a version without color differentiation (Black), version adopting ViTST-style joint arrangement (ViTST), and a version with added markers (Marker). For dynamic-periodee gestures three temporally consecutive and non-ovellapping images shown for each class better capture and visualize the underlying periodic patterns. Each graph image corresponds to a single input window.
Table 3 presents quantitative comparison of eтror rates across four gesture categories (SG, DG, FG-DG, D-PG) under various ablation settings of the proposed SKETCH module. The experiments were conducted under three ablation configurations: (a) removing color differentiation by rendering all joints the same black color, (b) replacing the joint coordinate arrangement of SKETČH with the VITST 22 layout (i.e., representing each joint as an individual line graph), and (c) adding markers at each observation point to emphasize the observed data. It should be noted that setting (b), the full plotting method of ViTST, as described in Section 2.3 was not adopted. Instead, the joint coordinates were rearranged into ViTST-style individual line graphs by treating each joint coordinate as an independent feature, while still applying our proposed normalization method. The group-wise gesture classification error rate (Group Error Rate) is computed based on the prediction performance of all gesture classes belonging to the group (Static Gestures (SG), Dynamic Gestures (DG), Fine- grained Dynamic Gestures (FG-DG), and Dynamic-periodic Gestures (D-PG)) and defined as follows:
Σ TP:
GroupErrorRateg
IEG
Seg,
ieG
21

---

Here, G' denotes specific gesture group (e.g., FG-DG) and i refers to the i-th gesture class (e.g.. Grab) belonging to group G. TP, represents the number of correctly predicted segments (true positives) for class i, while Seg, denotes the total number of ground-truth segments for that class. In other words, the group error rate is calculated based on the overall prediction accuracy aggregated across all gesture classes within the group.
Based on the analysis, ablation setting (a) resulted in the largest error-rate increase for SG, followed by D-PG, while setting (b) caused the greatest increase for D-PG, followed by SG. These results are closely related to the inherent characteristics of SG and D-PG. Gesture classes belonging to SG involve maintaining a specific hand shape over a period of time, with little to no noticeable movement over time. As shown in the SKETCH visualizations in Fig. []] the graph images of SG classes (e.g.. One, Four, OK, and Menu) produced by the SKETCH module display joint coordinates as horizontally aligned parallel lines. This visual pattern reflects the static nature of these gestures, with minimal variation along the time axis.
Therefore, the distinction between different SG classes relies not on temporal dynamics, but rather on the relative spatial relationships between joint coordinates captured in the graph image. In this context, the SKETCH module's strategy of consistently assigning unique color to each joint plays a critical role in differentiating the joints. The relative spatial configuration-suchh as a particular colored joint appearing higher than others within the same coordinate subplot, or two joints being positioned closely or far apart serves as a key visual cue for identifying SG gesture classes. In this context, ablated version (a), which renders all joints in the same color (black), eliminates the ability to distinguish between joint coordinates, ultimately hindering the model's 's ability to effectively learn the unique spatial arrangement (permutation) of joints characteristic of each SG class. In contrast, ablated version (b) modifies the joint coordinate layout of SKETCH to follow the VITST approach, where each joint is represented as an individual line graph. This deviates from the standard SKETCH format, in which multiple joint coordinates sharing the same coordinate axis are visualized together within a single subplot. As result, it becomes more difficult to capture the relative positions of joints within localized region of the image. Instead of directly perceiving spatial relationships among joint coordinates, the model is forced to infer them indirectly based on the positional patterns within each individual line graph. Consequently, compared to the original method that visualizes multiple joints on the same coordinate axis within a single subplot, this configuration introduces limitations in effectively recognizing the spatial relationships among joints.
defining characteristic of gestures in the D-PG category (e.g., Deny, Wave, and Knob) is their distinct periodicity, as clearly visualized in Fig. 9 Jusing the SKETCH module. Effectively recognizing such periodic gestures requires that the shape and repetition of each cycle along the time axis be visualized with sufficiently high resolution. However, in ablated version (b), the joint coordinate layout modified according to the ViTST format, where each joint is represented as an individual line graph arranged in grid within single graph image. This configuration leads to physical compression of the time axis (in terms of pixel space), as each line graph with the same time length is constrained within the limited image area. This becomes especially problematic for D-PG gestures, as subtle temporal variations and periodic patterns along the time axis are crucial for accurate gesture discrimination. If these patterns are visualized at low temporal resolution, their ane-grained structures may become distorted or lost, ultimately hindering the model's 's ability to correctly recognize D-PG gestures.
While (c) caused a slight performance drop for all four categories due to markers subtly distorting the fine shape of the lines and the relationships between joints. These results highlight the effectiveness of SKETCH in preserving critical spatio-tempont and structural cues through carefully designed visual representations, enabling more robust gesture recognition across diverse gesture categories.
E.2 Ablation Study: Impact of Regression Loss, Loss Weights, and Window Length
Table 4: Quantitative comparison on the SHREC'22 dataset of gesture recognition performance with and without regression loss for gesture boundary localization.
Method
DR1
FPJ
JI
w/ regression
0.92
0.07
0.87
w/o regression
0.88
0.09
0.82
22

---

Table 5: Influence of loss function weights (A1, 22 and Aς) on overall model performance on the SHREC'22 dataset.
X
22
À
DR
FP4
JI
1.0
0.5
0.5
0.90
0.07
0.85
1.0
1.0
1.0
0.92
0.07
0.87
1.0
1.5
1.5
0.88
0.06
0.84
Table 6: Effect of window length (Nr) recognition performance the SHREC'22 dataset.
NF
DR
FP4
JI1
8
0.84
0.10
0.77
12
0.90
0.10
0.82
0.92
0.07
0.87
20
0.89
0.08
0.83
Based on the qualitative analysis of Fig. in Section 4.2 we quantitatively compared the impact of regression loss the classification process as shown in Tab. The experimental results showed that the detection rate (DR) increased by 0.04, the false positive (FP) rate decreased by 0.02, and the Jaccard index (JI) improved by 0.05, confirming that the regression loss contributes enhancing the model's 's overall prediction performance across all evaluation metrics. Furthermore, the experimental results in Tab. iidiicaee that optimal performance is achieved when regression loss and classification loss are balanced.
In Tab. 6 the hindow-based approach is useful for online tasks as it mitigates noise and preserves spatio-temport context. Analysis of performance across varying window sizes confirmed that an appropriate window size improves real-time classification performance.
E.3 Ablation Study: Conventional Hand-Crafted and Proposed Features
Table 7: Quantitative comparison of plotting-based gesture recognition performance on SHREC'22 using different input features.
Plotting features
DR1
FP..
JIt
Hand-crafted
Raw
JCD
0.80
0.14
0.73
FD
0.87
0.06
0.83
SKETCH (ours)
0.92
0.07
0.87
All features were visualized using the same SKETCH-based plotting and model pipeline to ensure fair comparison. The quantitative results are summarized in Tab, 7 While JCD and FD are traditional hand-crafted features commonly used in prior work, their performance falls short when compared to directly plotted raw joint coordinates, indicating that hand-crafted preprocessing may lead to information loss even under identical visualization and modeling setups. This result highlights fundamental limitation of hand-craftes representations, which rely predefined rules and may fail to capture full complexity of skeletal motion. contrast, by directly utilizing raw coordinate data, our approach enables the model to autonomously extract discriminative features from rich, unprocessed inputs, thereby achieving superior recognition performance that is not solely attributable to the visualization method.
23

---

E.4
Ablation Study: Normalization Strategies
Table 8: Impact of normalization strategies on gesture recognition performance SHREC' 19.
Target
Axis-wise Z-norm Min-max norm
DR
FP
0.85 0.01
Window
0.84
0.02
Window (ours)
0.92 0.02
Axis-wise normalization, which treats x, y, and dimensions independently, consistently outperforms joint normalization across all coordinates, it better preserves directional variations specific to each axis. The superiority of min-max over Z-score normalization lies in its ability to preserve the relative spatial and temporal variations of joint movements within each window. Specifically, min-man max normalization maps each coordinate value to a fixed [0, 1] interval based on the observed minimum and maximum values along each axis (x, y, and z), thereby maintaining the relative displacement between joints at a given frame (inter-joint spatial structure) as well as the variability in a joint's s trajectory over time (intra-joint temporal dynamics). Therefore, commonly practiced in prior work ViTST (22), Z-score normalization is often accompanied by outlier removal techniques such as interquartile range (IQR), as used in [22], mitigate the influence of extreme values. In our experiments, for the Z-norm condition, outliers were removed using the IQR method, where the interquartile range was defined as IQR Q3 QI, and values outside the range of [Q1 1.5 x IQR, Q3 1.5 IQR) were excluded. After removing outliers, missing joint values were reconstructed using an interpolation process to ensure continuity in the plotted line graphs (i.e. graph image). However, skeleton-basee hand gesture recognition, where the number of joints is limited and each joint trajectory may contain critical semantic cues, removing even a few outlier joint values can disrupt the temporal continuity and degrade the model's is ability to recognize subtle and continuous motion patterns essential to accurate gesture interpretation. These results empirically validate our claim that window-level, axis- axis, and min-max normalization provides increased robustness to changes in viewpoint, user scale, and sensor positioning, while serving as critical component in achieving stable recognition in diverse environments.
E.5
Ablation Study: Dynamic Range Embedding
Table 9: Ablation study on the effects of including Dynamic Range Embedding (DRE) and alternative plotting methods the SHREC' 19 benchmark dataset.
Method
DR1
FPJ
(a) Black
0.91
0.04
(b) VITST
0.75
0.03
(c) Marker
0.9
0.03
(d) SKETCH w/o DRE (ours)
0.89
0.03
(e) SKETCH w/ DRE (ours)
0.92
0.02
Inspired by the customized positional encoding schemes diverse 3D domains to capture intrinsic spatial information, we devised a novel method that embeds the unique characteristics of each coordinate axis into visual representation. Table 2 ablation study on the SHREC' 19 dataset, evaluating both plotting methods and the Dynamic Range Embedding (DRE) module across five configurations: (a) ViTST baseline applied directly raw joont-coordinate sequences; (b) SKETCH- black, in which all joints are rendered in uniform black; (c) CKETCH-marken attaching markers to each joint coordinate at every observation point (reference plots for variants (a)-(c) using the same method are shown in Fig. 9 (d) SKETCH without DRE; and (e) SKETCH with DRE. Variants (b) do not utilize DRE. The results indicate that converting each skeleton sequence into visual representation and then applying the DRE module more effectively encodes axis-wise movement range than the version without DRE, enabling SKETCH to extract. higher-level features.
26
42
46
48
49
55
58
developed in
presents an
24

---

E.6
Ablation Study: Robustness on Pose Estimation Errors
Table 10: Impact of perturbations and missing rates SHREC' 19 gesture recognition performance.
Perturbations Missing Rate
DR
FP4
0.92
0.02
Sequence
30%
0.91
0.02
Sequence
50%
0.89
0.02
Sequence
70%
0,85
0.02
Sequence
90%
0.73
0.34
Frame
30%
0.90
0.02
Frame
50%
0.78
0.20
Frame
70%
0.06
0.00
Table 11: Robustness on SHREC' 19 under noise perturbations with varying noise rates (std.).
Perturbations Std.
DR
FP]
Noise
0.02
0.89
0.04
Noise
0.03
0.89
0.04
Noise
0.05
0.76
0.07
To further assess the robustness of SKETCH pose estimation errors, conducted additional evaluations under missing-data perturbations on SHREC' 19, introduced only at test time to examine generalization under degraded pose inputs. Specifically, we considered three types of perturbations. First, missing-joint robustness was tested by simulating joint-sensor failure through random deletion of joints, and second, occlusion robustness was examined by masking joint coordinates across both temporal and spatial dimensions to simulate partial marker occlusions. The results of these two settings are summarized in Tab. [O] Finally, test noise robustness was evaluated by injecting white Gaussian noise into the training joint coordinates and subsequently testing the model on sequences corrupted with the same noise distribution, as shown in Tab. 11
As summarized Tabs. 10 and 11 we report detection rate (DR) and false positives (FP) under all three perturbation settings. Despite these synthetic degradations, SKETCH consistently demonstrated stable performance by capturing the spatio-temporal flow of joint kinematics, rather than relying on individual joint positions, thereby highlighting its resilience pose estimation errors.
E.7
Ablation Study: Regression Head and Dynamic Range Embedding
Table 12: Ablation study of our proposed modules on the SHREC' 19 benchmark dataset.
Method
Regression
DRE
DR1
FP
0.84
0.04
SKETCH
0.89
0.03
0.90
0.04
SKETCH (ours)
0.92
0.02
As detailed in Sections 2.2 and :.1 this section we conducted an ablation study the SHREC' dataset to evaluate the individual contributions of the regression head and Dynamic Range Embedding (DRE). We demonstrate that incorporating regression loss (see Equation [T into the feature extraction process during training vital for effective hand gesture recognition, while our visual representation, enhanced by DRE, effectively encodes the relative movement range and variability of joint coordinates. By enabling the model distinguish whether a motion ongoing or has ended, and to represent the dynamic behavior of joint coordinate trajectories, SKETCH is able extract high-level spatiotemporal
25

---

features directly from raw data. Accordingly, Tab. reports performance with and without each component: the full highest accuracy and the lowest false positive rate.
both the regression head and consissssty achieves the
F Additional Visualizations
Not-gesture
Gesture
Ciesture 2
Start frome
(N)
frame
segiment
segment
segmmnnt
segmen:
segmeet
(b)
Figure 10: Comparison of offline and online datasets. (a) In the offline dataset, all frames within sequence correspond to single gesture. (b) In the online dataset, non-gesture segments exist, and each individual frames may belong different gesture classes within the same sequence.
As illustrated Fig. 10 and discussed in Sections 2.1 and 2.2 Fig. 10 a) represents an offline
dataset setting, in which each entire sequence corresponds to single gesture label. This setup is formulated sequence-level classification task, where the model receives the full frame sequence and predicts a single class. In contrast, Fig. [T]b) depicts online (real-time) setting, where non- gesture segments are present and frame-wise labels may vary within a sequence. The model must therefore simultaneously perform gesture segmentation and classification in streaming manner. This online scenario presents greater challenges, as the system is required to identify gesture boundaries (i.e., start and end frames) and assign the correct class labels to each detected segment. Additionally, because the number of gestures per sequence is unknown in advance, evaluation must consider both detection and localization performance irffomance-incorppraatng metrics such false positive rate and the Jaccard index.
W
Wi11
W; MONINE
Num
Now-20
Non-5
Non-
New geiuure
Non
Pinc
Piec
Finch
Non-
Non-
Non-gesture
Figure 11: Window-based problem formulation for online gesture recognition. A continuous frame stream is partitioned into fixed-length windows (W,), where aggregated frame labels determine the window label by majority voting. For windows containing both gesture and non-gesture frames, gesture boundaries additionally labeled for localization.
As shown in Fig. and discussed in Section 3.1 is partitioned into fixed-length windows W, xxjxx, where each window contains Np frames of skeleton data with j joints and e coordinate dimensions. window-level label determined by majority voting over the frame-level annotations within that window. In the case of "mixed" windows-i.e. windows containing more than one gesture-related label, such as both gesture and non-gesture frames the gesture's start and end positions are additionally annotated to facilitate boundary localization. For such windows, the start index initially set and the end index to NF 1, under the assumption that the gesture spans the entire window. Then, the label sequence is
an online setting, the incoming frame stream
26

---

traversed in temporal order to identify the actual transition points: when frame label switches from non-gesture to a gesture class, the corresponding index updated as the gesture start: conversely, when the label changes from gesture class to non-gesture label, the index just before the transition assigned as the gesture end. no transitions are found (i.e., the window contains only gesture or only non-gesture), the default start and end indices remain unchanged. Finally, the start and end indices are normalized to the range [0,1] I] by dividing by Ne resulting in two real-valued outputs, Is and Je. for boundary regression.
False Positive (FP)
0.25
0.55
DC
Oc devorr
StETTET cown
Detoction Rate
0.65
0.10
0.30
0.05
0.75
MS-05D
PSUIMNET
BW Paprst
Bes-OCC
DiNa
DDayORU
DSTA
DO-STA
00-MMVMT
0.70
Spare
Cooss
Gesture Classes
Chile
Viak(V)
Method
(a)
(b)
Figure 12:
Performance comparison on the SHREC'19 dataset. (a) Per-class detection rate (DR)
for the Square, Cross, Caret, Circle, and V-Mark gestures. (b) Total number of false positives (FP)
aggregated across all gestures.
As shown in Fig. 12 the SHREC" 19 hataset-whert the training and test sets are drawn from entirely different groups of subjects, with only four subjects the training set nine unseen subjects test set-presents substantial domain gap. This challenging setting demonstrates that our SKETCH method outperforms other benchmark approaches both detection rate and false positive rate. These results indicate that our method robustly extracts high-level features that generalize well to unseen users and new domains.
False Positive (FP)
0.3
Detection Rate
0.2
0.7
0.6
DRNs
64
Simweger
DO
Camal TEN
BesoRU
PSUMNET
DNTA
WETOO sF
TN-FS56-21 up
MS0DD
Sas-OO
DrNet
DO-IMVM
SKETCH (lme)
TWD
mans
MBRU
CTRCLP
creas
GRA6
potrai DRn
Kntoa
Gesture Classes (8)
Method
(b)
Figure 13: Performance comparison on the SHREC'22 dataset. (a) Detection rate (DR) for each of the 16 gesture classes. (b) Overall false positive (FP) aggregated across all classes.
Figure 13 illustrates that on the SHREC'22 benchmark which features larger set gesture classes than SHREC' 19 and, unlike SHREC'19 19, does not clarify whether the training and test subjects differ-our SKETCH method achieves detection rate comparable to the state-of-the-art 00-dMVMT model, while significantly reducing the false positive rate. This reduction is particularly important for online gesture recognition. These results position SKETCH as the new ofte-of-the-ane approach in the field.
27

---

TWO
THREE
30
FOUR
OK-
36
25
MENU
54
LEFT
3G
True Class
34
GRAB
MINCH
-10
DENY
0
WAVE
0
1
1
KNDB
NONE
0
ONE
TWD
THREE
FOUR
MERU
LEFT
RIGHT
CROSS
GRAS
PRICH
DENY
WAVE
HONE
Predicted
Figure 14: Confusion matrix (17 17) on the SHREC'22 dataset. Each row indicates ground-truth and each column shows predicted class, including the additional NONE class to reflect non-gesture.
Figure 14 visualizes the confusion matrix for SHREC'22. 22, where each of the 16 gesture classes is evenly represented by instances, and shows that our method achieves uniformly high performance across all classes.
28

---

Two
Ok
Menu
Circle
V
Grab
Pinch
Deny
Knob
t
Figure 15: Extended visualization of joint-level attention (corresponding to Fig. for gestures in the SHREC'22 dataset.
To investigate whether the model distinguishes gestures based meaningful semantic cues, applied Grad-CAM to the SKETCH-bbaed 2D graph images and visualized attention scores for each joint. The Grad-CAM activations were first extracted from the 2D image space, and each joint's attention value was mapped back to its physical 3D coordinates (z, and 2). Attention values across axes were then averaged obtain scalar score per joint. The attention maps (Fig [5) show that for the Ok gesture, the model concentrated on the thumb and index finger well as the fingertips of the remaining three fingers; for the Grab gesture, high attention was uniformly distributed across all five fingers; and for the Deny gesture, the focus was exclusively on the index finger. These findings demonstrate that the model relies on critical joints for gesture discrimination, closely mimicking human perception. This behavior further provides evidence that the model captures high-level semantic structure and avoids overfitting low-level geometric features, validating the SKETCH-BE -asdd representation in terms of both interpretability and effecctiveness.
29

---

skip Algorithm Online Gesture Recognition Algorithm
Require: Model J. total number of gesture classes K, non-gesture label Cnon. window size Nr, queue size Q, threshold value т (frames; for minimum gesture length 19REC'19).
Input: Stream of predictions, W,- {X1, .N, where t is the current time index, X xx and where j is the number of joints.
Output: Gesture start frame, end frame, and class label.
1: 1 1, start + 1, end -1
2: prev_g g <- Cnon
Initialize previous majority vote result
Histogram of gesture class occurrences 4: Queue + {Cnon: Onon...., Cnon E RIXQ >> Sliding window (c.g.. using deque for efficient updates)
hist + {0, .......... E Rlxk
5: while receiving new predictions W, do
6: if size of Queue Q then
7:
Remove oldest element from Queue end
8:
9: Di. < f(W,)
Predict label from model
Append Dt Queue
11:
9r f- arg max (iinnnonuuuuuueee)) Majority vote in the window In case of tie, choose the
label with the smallest index
12:
if 9i & Onon then
Gesture detected D Gesture start detected
13:
prev_g 9- = Cnon then start <- max(1,t (Nr/2])
14:
15:
end
16:
hist[gr] +- hist[gd +1 else
17:
Non-gesture detected Gesture end detected
18:
if prev_g g + Cnon then end <- max(1, [Nr/2])
19:
chosen label +- arg max(hist) & Final gesture label; tie-breaking similar above. Only one gesture in a sequence Longer than threshold
21:
if SHREC' 19 then
if (end - start) then
23:
Output (start, end, chosen_label
break
End processing for single-gese geggtur sequence
25:
else
Too short; do not output
27:
end if
28:
else
For diverse gestures in a sequence
29:
Output (slart, end, caoosen_labell
30:
end
31:
Reset: start +- -1, end -1, hist + 00,0,....0
32:
end
33:
end if
34:
prev 4-
> Update previous vote
35:
t t
36:
end while
30

---

G Training Strategies and Implementation Details
All experiments were conducted using the A optimizer 29 with a weight decay of 10-4 a batch size of 32 trained over 20 epochs, and loss weights \ = X2 X = 1. We applied linear warm-up [4 from the target learning rate over the first 30% of training steps, followed by cosine-annealle decay schedule.
For the SHREC' 19 dataset, the initial learning rate the queue length Q was set to nine, and the threshold T (see in Algorithm [ was set to 21, resulting in delay of four frames in Swin 27 Large model.
8 10-5, the window length Nr 40,
For 22 we retained these core settings but reduced the imitial learning rate to x 10 10-8 and set NE to 16, and the Q was set to 18, resulting in delay of cight frames in our Swin Large model. To improve label reliability while training, we retained only those training windows in which the majority gesture occupied least 70% of the Np frames; windows lacking such dominant class were discarded because their mixed labels reduced annotation confidence.
For the input resolution of 224 224 used in the pretrained backbones (Tabs. D generated 225 225 images through SKETCH and then applied center cropping to obtain the final 224 x 224 inputs.
and 2). we first
For both SHREC' 19 and SHREC'22, we applied the same data augmentation strategy during training. Specifically, augmentation was applied only to the training set, while the test set remained unchanged. rigid rotation, where the Buler angles (Tz, Ty. r=) were sampled from uniform distribution U (-66.0 0) with 0 = 0.3 radians. The sampled rotation was applied consistently to all frames within the window, preserving temporal dynamics and skeletal topology while introducing viewpoint variations. For each original window, two additional augmented windows were generated, resulting in a threefold increase the training data. In Tabs. [ 2 the notation 'Aug." was used exclusively for our method to indicate that the proposed aug- mentation was applied, whereas other methods were reported with their original training protocols. For our method, configurations without this notation ('Aug,") explicitly indicate training without augmentation.
Each temporal window was augmented using a random 3D
31

---

H Toy Example Evaluation of Geometric Shortcut
Training data
Testing data
ONE
TWO
First-layer output features
Last-layer output features
First-layer output features
Last-layer
THREE
oulput features
OK
Toy example of OO BAAaNN NMT
MENU
LEFT
RIGHT
CIRCLE
CROSS
GRAB
PINCH
(b)
(c)
(d)
DENY
WAVE
KNOB
SKETCH (ours)
(c)
(f)
(g)
(h)
Figure 16: t-distributed stochastic neighbor embedding (1-SNE) visualizations of model representa- tions on SHREC'22. (a-d) toy example of 00-dMVMT; (e-h) our SKETCH. (a, e) and (b, f) illustrate t-SNE visualizations the first- and last-layer output features on the training data, respectively, while (c, g) and (d, h) present the corresponding features on the testing data. Each point is colored by its gesture class as shown the legend.
This experiment was conducted to demonstrate the geometric shortcut phenomenon that arises when raw skeleton data is used as input. We implemented toy example using the O0-dMVMT architecture-1th state-of-the-arr model-replacing its hand-crafted features with raw skeleton coordinates. We evaluated this toy example the SHREC'22 22 benchmark dataset using only pure windows-those in which all (Nr) frames share the same label-excluding mixed label windows avoid confusing feature clusters. Figure depictss t-distributed stochastic neighbor embedding (t-SNE) plots of every feature representation for pure windows from both the training and testing data. For each of the raw-input 00-dMVMT toy example and our SKETCH model, we applied t-SNE the features extracted by the first and the last layers of the model. Specifically, for SKETCH the first-layer the zeroth Swin Transformer block, whereas for the 00-dMVM' toy example they correspond to the features at which multi scale features are concatenated, (see (8) which introduces this process). In both models, the output of the final layer corresponds to the high-level feature representation just before it is passed to the classifier head. Under these conditions, the 00-dMVMT toy example achieved 96. 22% window-level training accuracy and 59.82% window-level test accuracy before post-processing. whereas our proposed SKETCH model attained 99.00% and 92.58%, respectively.
Despite its reasonably high window-level accuracy, the toy example of 00-IMVMT still falls into the geometric shortcut phenomenon. This can be observed through the comparison in Fig. d6) and (h), which present (-SNE visualizations of the last-layer output feature representations for 00-dMVMT and SKETCH, respectively. Compared to SKETCH, the toy example of 00-dMVMT produces features that are merely hyperplane-separee fail to form clearly clustered distributions by class, highlighting the lack of semantically meaningful structure. This reflects the toy example of 00-dMVMT's tendency to exploit geometric shortcut-i.e.. relying lewel,vel, coordinate-basee cues that lead to non- non-gennrallizblee decision boundaries rather rather than learning semantically rich, well-separated class clusters. This indicates that feeding raw skeleton coordinates causes the 00-dMVMT model to overfit trivial coordinate cues extractiin only
ie.. sufficient for linear classifier distinguish but
32

---

low-level features instead of semantically meaningful, high-level representations and thus provides evidence of the geometric shorteut effect.
I Impact of Spatial Emphasis in Skeleton-Based Gesture Recognition
Gesture class
Two
Three
Menu
(a)
(b)
xy
XZ
yz
PCA
Lefl
Right
Wave
Deny
Pinch
Circle
Cross
Knob
Grab
Figure 17: Visualization of 14 representative gesture classes from the SHREC'22 dataset using the ShapeProj plotting method. Each block corresponds to gesture class. Pink section (a) shows (i) the 3D skeleton and (ii) the reference image. Green section (b) presents four projection views of the joint trajectories: projections onto the zy, æz, and yz planes, and view generated via principal component analysis (PCA). This layout highlights the spatial configurations captured by ShapeProj across multiple perspectives.
Table 13: Classification performance comparison between ShapeProj and SKETCH on the SHREC'22 dataset. Backbone configurations (e.g., S-B-4-12-384) follow the same notation as described Tab.
Method
DR
FP
JIf
ShapeProj (S-B-4-12-384)
0.82
0.79
ShapeProj (S-L-166-38))
0.85
0.05
0.81
SKETCH (S S--B-4-12-38
0.91
0.03
0.86
SKETCH (S-L-16-384)
0.92
0.02
0.87
33

---

Certain hand gestures in 3D space exhibit highly distinctive and discriminative shapes, such that the gesture class can be identified solely based on the spatial configuration of joints. As shown in Fig. a7).. gestures such as Two, Ok, and Pinch display clear morphological distinctions: Two involves extension of only the index and middle fingers; Ok is formed by creating a circle with the thumb and index finger while the remaining fingers are extended; and Pinch is characterized by the thumb and index finger closing together with the other fingers naturally curled inward. These clearly defined hand shapes serve as intuitive cues for human perception and can also act as effective features for gesture recognition models.
Building on this observation, this section investigates how an alternative plotting strategy-referred referred to hereafter as ShapeProj- affects model performance, with a particular focus on preserving the natural spatial structure of the human hand in 3D space through orthogonal projection. While both ShapeProj and SKETCH approaches utilize raw coordinate data without relying on rule-based feature engineering (i.e., hand-crafted features), the proposed method adopts rendering technique designed to maintain the geometric integrity of the hand. We aim to assess whether visualizations that preserve the physical configuration of the hand (as shown in Fig. 17 b)) can serve as effective representations for gesture recognition.
ShapeProj projects the 3D skeleton sequence (E R R xjxš) onto four views: the æ2, xy, and yz planes, and an additional axis on which the variance is maximized when the 3D skeleton sequence is projected (derived by principal component analysis (PCA)). The use of PCA aims to depict the hand shape from perspective that most clearly separates joint positions, regardless of the absolute location of the gesture in 3D space, and with minimal loss of information. For visualization, the resulting 2D joint coordinates obtained from each orthogonal projection and PCA are assigned the four quadrants of a single image. Prior to plotting, min-man normalization is applied independently to each 2D axis of the data (E xjx3) to ensure scale consistency. Each joint is assigned a unique color, and the temporal trajectory of each joint is illustrated by connecting its positions across frames using colored lines. Markers are added to indicate observed joint positions, and for the first frame in each quadrant, black edges are drawn to connect joints based on semantic hand structure, enabling direct rendering of the hand shape.
Figure b) shows the visualization results for the SHREC'22 dataset using ShapeProj. Compared to SKETCH, this approach more clearly represents the spatial configuration of the hand joints in 3D space, but sacrifices temporal resolution as a trade-off. As shown in Tab. [3] the classification performance of the hand shape-preserving visualization (ShapeProj) is sufficiently robust to sug- gest that emphasizing spatial configuration can provide meaningful discriminative cues for gesture recognition.
However, comparison with SKETCH reveals that SKETCH-based representations consistently out- perform those generated by ShapeProj. This suggests that for an online, streaming based gesture recogmition task, the model relies more heavily on relative motion patterns between joints over time, rather than on spatial configurations in which temporal dynamics are less prominently expressed. Such tendency is also observed in Tab. 7 where we compare hand-crafted features by plotting joint collection distances (ICD) and frame difference vectors (FD). JCD captures static spatial structure by computing distances between joints within the same frame, while FD encodes temporal dynamics by measuring displacement across frames. Despite both being hand-crafted, the FD-based images yield superior performance compared to JCD, highlighting the importance of temporal information. In summary, while the spatial configuration of the hand provides a strong and informative basis for classification, our experiments indicate that characteristic motion patterns between joints over time play a more decisive role in effectively solving the gesture recognition task.
34

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In Abstract and Section [ we point out the limitations existing methods and propose corresponding solutions. These contributions summarized in the final paragraph of the Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 5. introduces the current limitations of our method and explores possible avenues for future enhancements."
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken low lighting. Or specch-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
35

---

Answer: [NA]
Justification: Our work is not strictly a purely theoretical work, we provide more of an empirical formula.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide information to ensure that the experiments reported in Section and Appendix G can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which
5. Open access to data and code
36

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We publicly-accessibee data SHREC' 19 4 and SHREC'22 [12]. We
upload the codes and instructions to recover the results. Once the blind review period is finished, we'll open-source all codes, instructions, and model checkpoints.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guider PoodeSummissiooooliee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips. wwtpp:/pppwpiggiccee/c//////c/e/ pooodeSubmissiooPooo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide details the optimizer type, hyperparameter settings, batch size, number of epochs, and weight decay in Section G The hyperparameter configurations were determined empirically, demonstrated in Section
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report eтror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yss]
Justification: We ensured fairness by conducting thorough ablation studies and extensive syper-parameter tuning across multiple experimental settings incgluiing A100 and A6000 GPUs as noted in Section 4 with full details provided in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
37

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We specify the type of computational resources used in Section 4 the main text, and provide additional details in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
We have reviewed the NeurlPS Code of Ethics and confirm full compliance with its guidelines, including transparency (see Appendix G), and privacy protection (see
Justification:
Section B)
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section I indirectly demonstrates that research on hand-gesture recognition plays vital role in enhancing quality of life.
38

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. dep.oymepl of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work uses only publicly available skeleton gesture datasets (SHREC' 19 [4]. SHREC'22 (12] and does not release any new high-capacity generative models or sensitive scraped data; therefore no special safeguards are required.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We two public skeleton-gesture datasets: SHREC'19 4 (no explicit license; free for academic use) and SHREC'22 [2 (CC BY-NC-ND 4.0).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
39

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will upload the codes and instructions recover the results. Once the blind review period is finished, we'll open-source all codes, instructions, and model checkpoints.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codc/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The study relies on publicly available skeleton-based gesture datasets (SHREC' 19 (4, SHREC'22 (12) and does not involve any new data collection, crowdsoure- ing tasks, or experiments with human subjects; therefore this item is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
40

---

Justification: The study uses only publicly available skeleton-basee based gesture datasets (e.g.. SHREC' 19 4 and SHREC'22 [12). No new data were collected from human subjects, and no crowdsourcing or user studies were conducted; therefore IRB approval is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: Large language models (e.g., ChatGPT) were employed solely for minor english proofreading and wording suggestions during paper preparation.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
41


Reviews:
[{"review_id": "7HR1WeFzPg", "reviewer": "Reviewer_JPRk", "summary": "The paper introduces SKETCH (Skeleton Kinematics Extraction Through Coordinated grapH), a novel method that transforms 4D skeleton data (time, x, y, z) into a visually interpretable representation. This transformation enables the application of advanced visual recognition architectures such as SWIN and ViT for skeleton-based hand gesture recognition. The proposed approach seeks to make gesture recognition more intuitive and effective by converting temporal motion patterns into a visual domain.\n\nWhile the visual idea is interesting and the method shows promise, the motivation lacks depth in positioning the contribution relative to the rich body of existing gesture recognition work. Further, model complexity details are absent, and the evaluation is limited to only two datasets, which restricts insight into the generalizability of the approach.", "questions": "Provide a more detailed discussion of what limitations exist in current gesture recognition methods and how SKETCH directly addresses them.\nExplain why transformer-based models (SWIN, ViT) are necessary for this problem. Could lighter models suffice, especially given the reduced input complexity (keypoints instead of raw video)?\nReport parameter count, FLOPs, and runtime comparisons to provide readers with a sense of computational efficiency and feasibility.\nConsider evaluating on additional gesture datasets, especially those with varying motion complexity and user diversity, to assess the robustness and generalizability of SKETCH.\n\nAn ablation showing the impact of the SKETCH transformation versus raw skeleton input directly into the models would help isolate the contribution of your method.", "limitations": "As discussed above", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Transforming 4D skeleton data into a visual format is creative and facilitates the use of powerful visual transformer architectures.\nThe integration of vision transformer-based models (e.g., SWIN, ViT) for gesture recognition through the visual encoding is novel and bridges spatial-temporal modeling with image-based classification.\nThe SKETCH framework is conceptually aligned with the nature of the task (hand gestures), which makes the overall presentation engaging and easier to follow.\n\nThe paper does not sufficiently highlight the limitations of existing gesture recognition approaches or clearly justify why a transformation to a visual representation is necessary or superior.\n\n\nDetails such as the number of parameters, computational cost (GFLOPs), or inference time are not provided. These are essential when using large models like SWIN or ViT on a task typically solved with lighter architectures.\n\n\nGiven the task focuses on keypoint dynamics, the use of heavy models like ViT and SWIN-T seems excessive unless justified through clear comparative analysis or accuracy-performance trade-offs.\n\n\nThe experiments are conducted on only two datasets. Evaluating on more diverse and dynamic datasets would strengthen the generalizability claims.", "was_revised": false, "final_justification": "", "rebuttal": "We will incorporate the additional comments into the paper. Due to space constraints, the references use the same alphabetical identifiers as those listed at the bottom of Reviewer bPvY’s rebuttal.\n\n**Limitations of Prior Skeleton Methods and SKETCH’s Solutions**\n- Prior work and limitations\n   - Skeleton-based gesture recognition has traditionally started from approaches that directly utilize **① raw joint coordinates**. However, this direction presents several challenges:\n     1. **Geometric shortcuts** [49]: Overfitting to low-dimensional cues like absolute coordinates and distances.\n     2. **Coordinate bias** [54]: Sensitivity to changes in viewpoint and scale.\n     3. **Inter subject differences**: Difficulty in generalizing due to variations in hand size and range of motion.\n   - To address these challenges, **② hand-crafted features** have been used [8, 58, b], but they face limitations:\n     4. **Expert knowledge**: Dependence on specific skeleton formats or datasets.\n     5. **Interpretability and raw data richness**: Limited interpretability regarding which joints or temporal segments influence decisions, and inability to fully exploit the richness of raw data.\n     6. **Local inductive biases and long‑range dependencies** [a]: 3D/4D CNNs methods [45] constrained by local inductive biases, limiting their ability to model long‑range dependencies across space and time.\n - Why visualization is needed and how SKETCH addresses existing limitations \n   -  Addresses **① raw data** challenges:\n      1. By transforming raw coordinates into images, SKETCH uses pretrained visual backbones to extract high-level features, preventing overfitting to **geometric cues and reducing reliance on shortcuts**.\n      2. **Coordinate bias** is mitigated through the pretrained backbone’s invariance to spatial transformations, enabling gesture recognition despite changes in viewpoint, scale, or location.\n      3. **Inter-subject variability** is addressed through per-window normalization, using relative joint positions in image space.\n   -  Overcomes Limitations of **② Hand-Crafted Features**:\n      4. Our representation eliminates the need for **expert design**, adopting a transformation similar to how humans interpret visual charts, making it generalizable and intuitive.\n      5. SKETCH enables **interpretability** by converting raw skeletons into visual images, allowing inspection of joint and temporal segment contributions to classification (**Figs. 4, 7 (Appendix D) , and 14 (Appendix F)**). \n      6. Once mapped to the image domain, our data can be readily processed by global self-attention backbones, enabling explicit modeling of **long-range dependencies** between joints.\n - Contributions of the work\n   - The proposed SKETCH method leverages raw skeleton data to enhance interpretability and preserve rich spatial-temporal information, resulting in improved recognition performance compared to prior approaches (**Tabs. 1 and 2**). Under these conditions, SKETCH shows fast convergence in small number of epochs ($≤ 20$ epochs) (**Tab. B**) and high generalization (**Tabs. C and D**).\n\n**Model Complexity and Efficiency**\n - We conducted additional experiments on model complexity (**Tab. A**), which will be incorporated into the main paper. As noted, SKETCH’s computational cost was high with Transformer architecture backbones. To address this, we evaluated the method using lighter ResNet-18 and 34 architectures and plan to explore reducing computational costs through architectural simplification or model compression.\nAdditionally, since our framework transforms each window  into a graph image of fixed spatial resolution, throughput and inference speed remain constant, unlike competing methods, where computational load increases with sequence  length or joint count.\n\n**Table A: Results of computational costs with state-of-the-art methods for action recognition.**\n|Methods|Venue|Params. (M)|FLOPs (G)|FPS|\n|-|-:|-:|-:|-:|\n|DC-GCN+ADG|ECCV'20|4.90|1.83|-|\n|EfficientGCN-B4|TPAMI'22|2.00|15.20|-|\n|MS-G3D|CVPR'20|6.36|3.26|33|\n|DSTA|ACCV'20|13.72|3.24|114|\n|MST-GCN|AAAI'21|12.00|-|-|\n|OO-dMVMT|CVPRW'23|0.20|0.001|172|\n|BlockGCN|CVPR'24|5.20|6.52|-|\n|ProtoGCN (joint)|CVPR'25|4.12|15.38|30|\n|SKETCH (ResNet-18)||12.00|5.36|345|\n|SKETCH (ResNet-34)||22.11|10.81|196|\n|SKETCH (ViT-Large)||303.94|174.82|57|\n|SKETCH (Swin-Small)||49.63|8.55|256|\n|SKETCH (Swin-Base)||87.52|44.66|110|\n|SKETCH (Swin-Large)||195.76|100.29|70|\n\n**Necessity of Transformers Compared to Lightweight Backbones**\n - We understand the reviewer’s concern that using heavy models like ViT or Swin may be excessive for graph image input. To address this, we include a comparison (**Tab. B**) evaluating SKETCH with both lightweight CNN backbones and transformer models. As shown, while CNN backbones have lower computational costs, their performance lags behind transformer models.\nThis gap is mainly due to ViT and Swin’s ability to capture long-range dependencies across the entire temporal image window, which is essential for understanding joint kinematics in gesture sequences (**Appendix E.1**). In contrast, CNNs rely on local receptive fields and inductive biases that may not fully capture global temporal-spatial patterns [a].\n\n**Table B: Results on SHREC’19: benchmarks vs. our ResNet models (224 and 384 denotes input resolution $H=W$).**\n|Methods|DR↑|FP↓|Params. (M)|FLOPs (G)|FPS|Epochs ($\\times$10)|\n|-|-:|-:|-:|-:|-:|-:|\n|DSTA [40]|0.81|0.08|13.72|3.24|114|48|\n|DG-STA [5]|0.81|0.07|0.27|0.17|238|30|\n|OO-dMVMT [8]|0.88|0.05|0.20|0.001|172|10|\n|ProtoGCN (joint) [b]|0.86|0.05|4.12|15.38|30|15|\n|SKETCH (ResNet-18-224)|0.84|0.20|12.00|5.36|345|2|\n|SKETCH (ResNet-34-224)|0.89|0.02|22.11|10.81|196|2|\n|SKETCH (ViT-Large-384)|0.90|0.03|303.94|174.82|57|2|\n|SKETCH (Swin-Small-224)|0.88|0.04|49.63|8.55|256|2|\n|SKETCH (Swin-Base-384)|0.91|0.03|87.52|44.66|110|2|\n|SKETCH (Swin-Large-384)|0.92|0.02|195.76|100.29|70|2|\n\n**Why Raw Keypoints Should Not Be Used Directly**\n - We appreciate the valuable comment. Directly using raw keypoint coordinates introduces challenges such as geometric shortcut and coordinate bias, which hinder generalization and semantic learning. The analysis is described in **Appendix A**, **Supp. B**, and the additional experiments in **Tab. E**. To address this, we proposed SKETCH as an intermediate medium that transforms raw keypoints into structured images—making it feasible to leverage pretrained vision backbones.\n\n**Evaluation on the Generalization and Robustness**\n - Cross-dataset evaluation on challenging offline benchmarks:\n   - Although SKETCH is designed for **online hand-gesture recognition**, we evaluated it across on/offline scenarios and human-action recognition tasks. While our results are slightly below those of models specifically tailored to each dataset, they remain competitive. Note that due to the short rebuttal period, we couldn't fully optimize hyperparameters for all three datasets, but we expect stronger results after further tuning. We compared SKETCH against benchmark methods specialized for each dataset, as shown in **Tabs. C and D**.\n      - **NW-UCLA** [c]: An **offline multi-view 3D human-action** benchmark with sequences captured from **different camera viewpoints**.\n      - **SHREC’17** [d]: An **offline 3D hand-gesture** benchmark requiring distinction between single-finger configurations and whole-hand shapes.\n\n**Table C: Results on the NW-UCLA dataset for offline human action recognition (refer to [58] for modality abbreviations, J+B+JM+BM=M).**\n|Methods|Venue|Modalities|Acc.|\n|-|-|:-:|-:|\n|4S-Shift-GCN|CVPR'20|M|94.6|\n|DC-GCN+ADG|ECCV'20|M|95.3|\n|CTR-GCN|ICCV'23|M|96.5|\n|InfoGCN|CVPR'22|M|96.6|\n|BlockGCN|CVPR'24|M|96.9|\n|GAP|ICCV'23|J|94.0|\n|TD-GCN|IEEE Trans. Multimed.'23|J|94.8|\n|BlockGCN|CVPR'24|J|95.5|\n|SKETCH (Ours)||SKETCH|95.5|\n\n**Table D: Results on legacy benchmark methods—no recent baselines are available—on the SHREC’17 dataset (14 gestures) for offline hand gesture recognition.**\n|Methods|Venue|Acc.|\n|-|-|:-:|\n|CNN+LSTM|PR'18|91.8|\n|ParallelCNN|RFAI'18|93.3|\n|STA-Res-TCN|Gesture'18|94.5|\n|MFANet|Sensor'19|94.6|\n|DDNet|MMAsia'19|94.6|\n|SKETCH (Ours)||94.6|\n\n**Ablation Study of Sketch versus Raw Skeleton Inputs to the Model**\n- As the reviewer noted, isolating the contribution of the SKETCH transformation ideally requires comparing the same backbone architecture with and without SKETCH. However, our vision transformer backbones are pretrained on image inputs and cannot directly accept raw skeleton coordinates, and methods like OO-dMVMT [8] are designed for hand-crafted features and are incompatible with raw joint inputs due to dimensional mismatches.\n- To address this, we conducted a **proxy comparison** using SHREC’19 as benchmarks (**Tab. E**):\n   - (i) vs. (ii): We modified OO-dMVMT to accept raw skeleton data and compared it to the original version with hand-crafted features (**Supp. B**).\n   - (iii) vs. (iv): We compared a vanilla transformer (similar to ViT) with raw inputs against our SKETCH-transformed input fed into a pretrained Swin-T backbone.\n\n- In these comparisons, raw-input variants ((i) and (iii)) failed to learn meaningful representations due to coordinate bias and geometric shortcuts (**Appendix A**). t-SNE visualizations in **Supp. B** show that SKETCH embeddings form semantically coherent clusters, while raw inputs collapsed into low-level groupings. Further comparisons with DeepGRU [9] on raw input (**Tab. 2**) support this observation.\n\n- These results empirically validate SKETCH’s role in generating a more structured and discriminative input space, improving model effectiveness and generalization.\n\n**Table E: Impact of SKETCH Transformation vs. Raw Input on SHREC’19 Performance**\n|Index|Methods|Backbone|Raw|Hand-crafted|SKETCH|DR↑|FP↓|\n|-|-|:-:|:-:|:-:|:-:|-:|-:|\n|(i)|OO‑dMVMT|-|✓|||0.32|0.13|\n|(ii)|OO‑dMVMT|-||✓||0.88|0.05|\n|(iii)|Ours|Vanilla-T|✓|||0.41|0.14|\n|(iv)|Ours|Swin-T|||✓|0.92|0.02|"}, {"review_id": "cAOCzK6Gfb", "reviewer": "Reviewer_bPvY", "summary": "This paper presents a pipeline for online skeleton-based hand gesture recognition. The authors observe that existing approaches often suffer from a “geometric shortcut,” stemming from Euclidean-space-based representations or from using hand-crafted features that are difficult to interpret and generalize. Inspired by ViTST, the paper introduces a graph plot-based representation, which effectively converts windowed skeleton coordinates into line charts that plot normalized local joint movement per axis. This representation is more interpretable and can be processed by any vision-based model. To address the information loss regarding global motion, the authors propose Dynamic Range Embedding, which embeds z-normalized axis movement across a window. Experiments on SHREC’19 and SHREC’22 demonstrate that the proposed method operates in real time and outperforms previous approaches in both detection rate and false positive rate.", "questions": "* Does the method use sliding windows or non-overlapping windows for segmenting the data?\n* How does the method perform if vectorized representations are used directly instead of converting features into pixel-based plots? For example, what if the normalized $N_f \\times J$ feature is fed directly, rather than through image plotting?\n* What is the final resolution of the generated graph images?", "limitations": "* As mentioned in the limitation section, information from preceding windows is not utilized. Future work could consider adopting video processing models to leverage longer temporal context.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The paper is exceptionally well-written and easy to follow—arguably the clearest paper I’ve read in this NeurIPS cycle.\n\n- The idea of converting skeleton coordinates to graph charts is novel and interesting. The proposed method demonstrates strong effectiveness compared to previous hand-crafted features and other baselines.\n\n- The authors provide thorough ablation studies and experiments on all components, demonstrating the effectiveness of the plotting strategy, the presented feature, and the normalization technique.\n\nWeaknesses\n\n- The paper does not systematically analyze how the proposed feature performs under global rotations or changes in camera viewpoint. It remains unclear how robust the method is in these scenarios.\n\n- Both SHREC’19 and SHREC’22 are relatively simple datasets for gesture recognition by today’s standards. The paper could be further strengthened by evaluating on more challenging gesture-language datasets, such as BOT57M.", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1**\n- Thank you for this insightful comment. To address it, we have conducted additional experiments on two benchmarks involving **global rotations and challenging camera viewpoints**:\n\t- **Northwestern-UCLA (NW-UCLA) Multiview Action3D** [c], which provides offline multiview 3D skeletons for human action recognition under varied **global rotations**, where the testing dataset is captured from **camera viewpoints distinct** from those used in the training dataset.\n\t- **Joint annotated Human Motion DataBase (JHMDB)** [e], an offline human action dataset, which offers 2D skeletons extracted from movie and web clips and encompasses **challenges such as camera motion, occlusion, motion blur, and significant viewpoint variation**.\n- Although SKETCH is primarily designed for online hand gesture recognition, we deliberately evaluated it beyond that scope–across both online and offline settings, human action recognition tasks, and 2D/3D skeleton modalities. Despite not being tailored to each dataset, our method achieved competitive performance, only marginally below heavily optimized baselines (**Tabs. A and B**). We believe this is due to SKETCH’s topology-agnostic design, which, unlike GCN-based methods that rely on predefined adjacency structures, enables robust generalization across varied skeleton representations and camera viewpoints. We also kindly note that further hyperparameter tuning, which was constrained by the limited rebuttal period, is likely to yield even stronger results. We compared SKETCH against benchmark methods specialized for each dataset, as shown in **Tabs. A and B**.\n\n**Table A: Results on benchmark and proposed methods on the JHMDB dataset for offline 2D human action recognition.**\n|Methods|Venue|Acc.|\n|-|-|-:|\n|TwoBranchesLSTM|Sensors'22|61.24|\n|EHPI|ITSC'19|65.50|\n|JMRN|WACV'22|68.55|\n|DDNet|MMAsia'19|77.20|\n|TDNet|IET Comput. Vis.'22|79.30|\n|ST-GCN|AAAI'18|62.69|\n|AA-GCN|CVPR'18|67.46|\n|MSG3D|CVPR'20|77.52|\n|CTR-GCN|ICCV'21|73.02|\n|X3D|CVPR'22|79.14|\n|C3D|CVPR'22|80.20|\n|4s-AGCN|WACV'24|77.82|\n|SKETCH (Ours)||78.40|\n\n**Table B: Results on benchmark and proposed methods on the NW-UCLA dataset for offline multi-view 3D human action recognition (refer to [58] for modality abbreviations: J = Joint, B = Bone, JM = Joint Motion, and BM = Bone Motion).**\n|Methods|Venue|Modalities|Acc.|\n|-|-|:-:|-:|\n|EnsembleTS-LSTM|ICCV'17|J+B|89.2|\n|2S-AGC-LSTM|CVPR'19|J+B+JM+BM|93.3|\n|4S-Shift-GCN|CVPR'20|J+B+JM+BM|94.6|\n|DC-GCN+ADG|ECCV'20|J+B+JM+BM|95.3|\n|CTR-GCN|ICCV'23|J+B+JM+BM|96.5|\n|InfoGCN|CVPR'22|J+B+JM+BM|96.6|\n|BlockGCN|CVPR'24|J+B+JM+BM|96.9|\n|GAP|ICCV'23|J|94.0|\n|TD-GCN|IEEE Trans. Multimed'23|J|94.8|\n|InfoGCN|CVPR'22|J|96.5|\n|BlockGCN|CVPR'24|J|95.5|\n|SKETCH (Ours)||SKETCH|95.5|\n\n**Weakness 2** \n- We sincerely appreciate the reviewer’s valuable suggestion to evaluate our method on more challenging gesture-language datasets, such as BOT57M. As suggested, we have initiated the request for access to the BOT57M dataset through its official site; however, we have not yet received the data. Once access is granted, we plan to incorporate experiments on this dataset in our future revision to further validate the robustness and generalizability of SKETCH. \nIn the meantime, to further demonstrate robustness beyond SHREC’19 and SHREC’22, we have also applied our approach to three additional benchmarks: \nIn response to **Weakness 1** and alongside evaluations on the **NW-UCLA and JHMDB datasets**, as shown in **Tabs. C and D**, **SHREC'17** [d], an **offline hand‑gesture** benchmark requiring the distinction of single‑finger configurations from whole‑hand shapes.\n\n**Table C: Results on legacy benchmark methods–no recent baselines are available–on the SHREC’17 dataset (14 gestures) for offline hand gesture recognition.**\n|Methods|Venue|Acc.|\n|-|-|-:|\n|Dynamic Hand|CVPRW'16|88.2|\n|Key-frameCNN|3DOR'17|92.9|\n|3Cent|STAG'17|88.6|\n|CNN+LSTM|PR'18|91.8|\n|ParallelCNN|RFAI'18|93.3|\n|STA-Res-TCN|Gesture'18|94.5|\n|MFANet|Sensor'19|94.6|\n|DDNet|MMAsia'19|94.6|\n|SKETCH (Ours)||94.6|\n\n\n**Table D: Results on benchmark and proposed methods on SHREC'17 (28 gestures)**\n|Methods|Venue|Acc.|\n|-|:-:|-:|\n|PLSTM-base|CVPR'20|87.6|\n|PLSTM-early|CVPR'20|93.5|\n|PLSTM-PSS|CVPR'20|93.1|\n|PLSTM-middle|CVPR'20|94.7|\n|PLSTM-late|CVPR'20|93.5|\n|P4Transformer (30 Epochs)|CVPR'21|87.5|\n|P4Transformer (50 Epochs)|CVPR'21|91.2|\n|Kinet|CVPR'22|95.2|\n|PointCMP|CVPR'23|93.3|\n|P4Transformer+MaST-Pre (30 Epochs)|ICCV'23|90.2|\n|P4Transformer+MaST-Pre (50 Epochs)|ICCV'23|92.4|\n|SKETCH (Ours)||91.4|\n\n- Although **SKETCH was originally designed for online data and we were unable to perform full hyperparameter tuning due to the limited rebuttal period**, our method nonetheless delivers competitive performance on NW‑UCLA, JHMDB, and SHREC’17–falling only marginally below current state‑of‑the‑art results.\n\n**Question 1**\n- We apologize for any confusion caused. We acknowledge that the manuscript lacks a detailed description of this aspect. To clarify, the sliding window technique used in our experiments assumes a fixed window size of $W$, with a stride of 1, which effectively results in an overlap of $W-1$ frames. For better understanding, we kindly refer the reviewer to **Figs. 2 and 10 (Appendix F)**, which illustrates this window-based formulation for online gesture recognition. We will revise the manuscript to describe this setting more clearly and explicitly.\n\n**Question 2**\n- Thank you for the valuable question. We agree that directly using normalized $N_f \\times J$ feature matrices–rather than converting them into pixel-based image plots–could serve as a useful comparison to isolate the impact of our SKETCH transformation.\nHowever, our backbone architectures are pretrained on image data and thus are not directly compatible with vectorized representations. To approximate the reviewer’s intent, we conducted two proxy experiments on SHREC’19 and SHREC’22 (**Tabs. E and F**):\n\t- (i) vs. (ii): We modified the input layer of OO-dMVMT [8] to accept raw joint sequences and compared its performance to the original version using hand-crafted features (**Supp. B**).\n\t- (iii) vs. (iv): We compared a variant using raw joint data with a vanilla transformer (architecturally similar to ViT) against our proposed method using SKETCH-transformed images with a pretrained Swin-T backbone.\n\n- Across both comparisons, models using raw or vectorized skeleton input ((i) and (iii)) struggled to learn high-level representations–often overfitting to geometric shortcuts or coordinate bias (**Appendix A**). Furthermore, t-SNE visualizations in **Supp. B** showed that SKETCH embeddings formed semantically coherent clusters, while raw-input versions collapsed into low-level geometric groupings. Additional comparisons with DeepGRU [9] on raw input (**Tab. 2**) further support this observation.\nThese results empirically validate the SKETCH transformation’s role in producing a more structured and discriminative input space, thereby enhancing model effectiveness and generalization.\n\n**Table E: Impact of SKETCH Transformation vs. Raw Input on SHREC’19 Performance**\n|Index|Methods|Backbone|Raw|Hand-crafted|SKETCH|DR↑|FP↓|\n|-|-|:-:|:-:|:-:|:-:|-:|-:|\n|(i)|OO‑dMVMT [8]|-|✓|||0.32|0.13|\n|(ii)|OO‑dMVMT [8]|-||✓||0.88|0.05|\n|(iii)|Ours|Vanilla-T|✓|||0.41|0.14|\n|(iv)|Ours|Swin-T|||✓|0.92|0.02|\n\n**Table F: Impact of SKETCH Transformation vs. Raw Input on SHREC’22 Performance**\n|Index|Methods|Backbone|Raw|Hand-crafted|SKETCH|DR↑|FP↓|JI↑|\n|-|-|:-:|:-:|:-:|:-:|-:|-:|-:|\n|-|DeepGRU|-|✓|||0.26|0.25|0.21|\n|(i)|OO‑dMVMT [8]|-|✓|||0.71|0.13|0.64|\n|(ii)|OO‑dMVMT [8]|-||✓||0.92|0.09|0.85|\n|(iii)|Ours|Vanilla-T|✓|||0.68|0.16|0.61|\n|(iv)|Ours|Swin-T|||✓|0.92|0.07|0.87|\n\n**Question 3**\n- We acknowledge that the resolution details in the caption of **Tab. 1** may not have been as clear as intended, which could have caused some confusion for readers. To enhance clarity, we will revise the main text to state explicitly that the input spatial resolution is $224 \\times 224$ for Swin Small and $384 \\times 384$ for ViT Large, Swin Base, and ViT Large. In the additional experiments conducted during the rebuttal phase, the input size for ResNet-18 and ResNet-34 was also set to $224 \\times 224$.\n\n**Limitations**\n- As mentioned in **Sec. 5.1 (Limitations and Future Work)**, we acknowledge that, as the reviewer pointed out, our current approach does not incorporate information exchange between previous windows, making it difficult to treat the model as one that fully understands the entire sequence. In future work, we plan to explore video and time-series processing methods [f] to address this limitation and improve our model's ability to capture longer temporal contexts.\n\n[a] Lou, Meng, et al. \"OverLoCK: An Overview-first-Look-Closely-next ConvNet with Context-Mixing Dynamic Kernels.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[b] Liu, Hongda, et al. \"Revealing key details to see differences: A novel prototypical perspective for skeleton-based action recognition.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[c] Wang, et al. \"Cross-view action modeling, learning and recognition.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2014.\n\n[d] De Smedt, et al. \"Shrec'17 track: 3d hand gesture recognition using a depth and skeletal dataset.\" 3DOR-10th Eurographics workshop on 3D object retrieval. 2017.\n\n[e] Jhuang, et al. \"Towards understanding action recognition.\" Proceedings of the IEEE international conference on computer vision. 2013.\n\n[f] Pang, et al. \"Context-enhanced memory-refined transformer for online action detection.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025."}, {"review_id": "HVpMooArat", "reviewer": "Reviewer_D9gv", "summary": "This paper introduces SKETCH (Skeleton Kinematics Extraction Through Coordinated grapH), a novel approach for real-time hand gesture recognition using raw 3D skeleton data. Unlike conventional methods that rely on hand-crafted features, SKETCH transforms raw four-dimensional skeleton data (time, x, y, z) into interpretable graph images. These visual representations are processed by pretrained vision backbones (ViT/Swin Transformer) for gesture classification and boundary localization.\n\nA central innovation is the Dynamic Range Embedding (DRE), which compensates for the loss of axis-relative scale caused by per-axis min–max normalization. The method also handles domain shifts—due to sensor viewpoints or user-specific kinematics—by applying normalization independently per window at inference time. SKETCH is evaluated on SHREC’19 and SHREC’22 benchmarks, achieving state-of-the-art accuracy and interpretability, while operating in real-time.", "questions": "1. Have the authors tried other backbones besides ViT and Swin Transformer, perhaps more lightweight ones? Because this paper addresses online recognition, efficiency is a critical factor that should be taken into account. And based on Table 1, this method doesn’t outperform existing methods in terms of time and FPS. So it would be helpful to provide some further analysis on efficiency.\n\n2. Could the proposed SKETCH be applied to human body skeletons? The method doesn’t seem to restrict itself to any specific type of skeleton topology (unlike graph convolution networks, which require a fixed spatial topology to create an adjacency matrix), so I think this method could generalize well to body skeletons.\n\n3. Can the visual representation handle missing data or occlusions robustly?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1. Original and Clever Formulation. The conversion of skeleton sequences into visual plots is both novel and intuitive. By leveraging pretrained vision backbones, the method sidesteps the need for complex temporal-spatial modeling architectures.\n\n2. High Practical Impact. The method is highly relevant for real-time systems such as VR/AR, robotics, and HCI. Its streaming-friendly window-based architecture is well-suited for deployment in online gesture detection settings.\n\n3. Strong Experiment Results. The method consistently outperforms prior art on two major gesture benchmarks across multiple metrics (DR, FP, JI, FPS). Ablations are provided for normalization, DRE, loss components, etc.\n\n**Weaknesses**:\n\n1. Potential Performance Bottlenecks in Latency-Sensitive Scenarios. Despite real-time FPS claims, the use of vision transformers on graph images may still be computationally heavier than lightweight GCNs or 1D CNNs, particularly for mobile applications.\n\n2. No Formal Analysis of Robustness. Although SKETCH handles viewpoint and subject variation empirically, there is no robustness analysis under noise, occlusion, or missing joints—common in real-world sensors.", "was_revised": true, "final_justification": "Thanks to the authors for addressing my concerns beyond my anticipations. I believe all my concerns have been addressed. Therefore, I've changed my score to accept.", "rebuttal": "Due to space constraints, the references use the same alphabetical identifiers as those listed at the bottom of Reviewer bPvY’s rebuttal, with the venue details also provided extensively by bPvY.\n\n**Weakness 1**\n- We appreciate the reviewer’s comment on the **computational cost** of using vision transformers on graph images. We agree that such models may be less optimal for mobile application scenarios due to their higher overhead compared to lightweight baselines like GCNs or 1D CNNs. Nonetheless, as shown in **Tabs. A and B**, we believe the accuracy improvements of hand gesture recognition our method offers—particularly in interpretability and recognition performance—provide meaningful advantages that justify its use within our framework. \n- Furthermore, although transformer-based models are more demanding, our framework **still achieves real-time FPS (≥ 30)** performance in practice. This suggests that, despite the increased cost, the method remains applicable in latency-sensitive settings while maintaining a favorable balance between accuracy and efficiency. We also recognize the need for further improving computational efficiency. As such, we plan to explore future directions such as architectural simplification and model compression techniques to better accommodate resource-constrained environments of a model.\n\n**Question 1**\n- We agree that efficiency is crucial for online recognition. To address this, we explored **lightweight alternatives** to ViT and Swin (Tabs. A and B). While convolution-based models are computationally efficient, they underperform compared to transformer-based models.\n- The reason we primarily utilized transformer-based models is their ability to capture long-range dependencies across the entire temporal sequence of the input images. In contrast, convolution-based models, which rely on local receptive fields and spatial inductive biases [a], are less effective at globally understanding the overall flow of joint kinematics in gesture sequences. As shown in our comparison **Tabs. A and B**, while lightweight CNN models are faster and more computationally efficient, they do not perform as well as ViT-based models in capturing the global motion dynamics essential for accurate gesture recognition.\n\n - **Training and scalability analysis**: Many prior works require significantly more training epochs due to training from scratch and rely on multi-stream fusion frameworks, which independently train and infer multiple models. For example, BlockGCN [58] uses four streams and ProtoGCN [b] uses six. As shown in **Tabs. A and B**, traditional methods typically train for more than three times as many epochs compared to ours. \"In contrast, our method uses pretrained backbones, converging within 20 epochs without multi-stream fusion, thus avoiding the added overhead.\nFurthermore, since our input image resolution remains fixed regardless of the number of input frames or joints, our model maintains constant computational load and throughput. In contrast, many prior approaches incur increasing complexity as frame count or joint count grows.\n\n**Table A: Results on SHREC’19: benchmarks vs. our ResNet-18/34 models (224 and 384 denotes input resolution $H=W$).**\n|Methods|DR↑|FP↓|Parameters (M)|FLOPs (G)|FPS|Epochs ($\\times 10$)|\n|-|-|-|-:|-:|-:|-:|\n|MS-G3D [27]|0.69|0.25|6.34|3.26|33|16|\n|DSTA [40]|0.81|0.08|13.72|3.24|114|48|\n|DDNet [52]|0.82|0.10|1.80|0.02|455|60|\n|OO-dMVMT [8]|0.88|0.05|0.20|0.001|172|10|\n|ProtoGCN (joint) [b]|0.86|0.05|4.12|15.38|30|15|\n|SKETCH (ResNet-18-224)|0.84|0.20|12.00|5.36|345|2|\n|SKETCH (ResNet-34-224)|0.89|0.02|22.11|10.81|196|2|\n|SKETCH (ViT-Large-384)|0.90|0.03|303.94|174.82|57|2|\n|SKETCH (Swin-Small-224)|0.88|0.04|49.63|8.55|256|2|\n|SKETCH(Swin-Base-384)|0.91|0.03|87.52|44.66|110|2|\n|SKETCH(Swin-Large-384)|0.92|0.02|195.76|100.29|70|2|\n\n**Table B: Results on SHREC’22: benchmarks vs. our ResNet-18/34 models.**\n|Backbone|DR↑|FP↓|JI↑|Parameters (M)|FLOPs (G)|FPS|Epochs ($\\times 10$)|\n|-|-:|-:|-:|-:|-:|-:|-:|\n|DG-STA [5]|0.51|0.32|0.40|0.27|0.11|238|30|\n|PSUMNet [43]|0.62|0.24|0.52|-|-|41|20|\n|MS-G3D [27]|0.68|0.21|0.57|6.36|2.62|34|16|\n|DSTA [40]|0.73|0.24|0.61|13.72|2.12|109|48|\n|DDNet [52]|0.88|0.16|0.78|1.80|0.01|455|60|\n|OO-dMVMT [8]|0.92|0.09|0.85|0.21|0.001|224|10|\n|SKETCH(ResNet-18-224)|0.87|0.04|0.84|11.53|5.36|345|2|\n|SKETCH(ResNet-34-224)|0.86|0.07|0.81|21.64|10.81|196|2|\n|SKETCH(Swin-Base-384)|0.91|0.06|0.86|87.05|44.66|110|2|\n|SKETCH(Swin-Large-384)|0.92|0.07|0.06|195.28|100.28|70|2|\n\n - **Efficiency and throughput analysis**: We carefully profiled our method’s computational efficiency by measuring throughput, normalization time (preprocessing), and the time required for both sequential and parallel image transformation (i.e., rendering; **Tab. C**). All measurements were conducted under the same computational environment. As reported, the combined preprocessing and model inference time ensures real-time FPS performance in practice.\n\n**Table C: Inference time (ms) breakdown of our method for different backbones**\n|Backbone|Preprocessing|Sequential Rendering|Parallel Rendering|Model Inference|\n|-:|:-:|:-:|:-:|:-:|\n|ResNet-18-224|0.1|4.3|0.1|2.9|\n|ResNet-34-224|0.1|4.3|0.1|5.1|\n|ViT-Large-384|0.1|5.0|0.1|17.6|\n|Swin-Small-224|0.1|4.3|0.1|3.9|\n|Swin-Base-384|0.1|5.0|0.1|9.1|\n|Swin-Large-384|0.1|5.0|0.1|14.1|\n\n**Question 2**\n- **Evaluation on human body skeletons** and challenging offline benchmarks\nAlthough SKETCH is primarily intended for online hand gesture recognition, we deliberately evaluated it far beyond that setting—across on/offline scenarios, human‑action recognition tasks, and both 2D and 3D skeleton modalities. While our scores are only marginally below those of models specially tailored to each dataset, they nevertheless remain highly competitive. Please also bear in mind that, owing to the **short rebuttal window, we could not fully optimize hyper‑parameters for all three additional datasets**; we anticipate even stronger results after proper tuning. We compared SKETCH against benchmark methods specialized for each dataset.\n   - **NW‑UCLA** [c] – an offline, multi‑view 3D human‑action benchmark in which training and test sequences are captured from different camera viewpoints.\n   - **JHMDB** [e] – an offline 2D human‑action dataset composed of movie and web clips that feature camera motion, occlusion, and viewpoint shifts.\n   - **SHREC’17** [d] – an offline 3D hand‑gesture benchmark that requires distinguishing single‑finger configurations from whole‑hand shapes.\n- Although SKETCH is primarily designed for online hand gesture recognition, we deliberately evaluated it beyond that scope—across both online and offline settings, human action recognition tasks, and 2D/3D skeleton modalities. Despite not being tailored to each dataset, our method achieved competitive performance, only marginally below heavily optimized baselines (**Tabs. D, E, and F**). We fully agree with the reviewer’s insight that, unlike graph convolutional networks which require a fixed spatial topology, our topology-agnostic design enables SKETCH to generalize well to diverse skeleton structures, including full-body representations. This flexibility allows SKETCH to perform robustly across various skeleton modalities and camera viewpoints. We also kindly note that further hyperparameter tuning, which was constrained by the limited rebuttal period, is likely to yield even stronger results.\n\n**Table D: Results on the NW-UCLA dataset for offline human action recognition (refer to [58] for modality abbreviations).**\n|Methods|Publication|Modalities|Acc.|\n|-|:-|:-:|-:|\n|DC-GCN+ADG|ECCV'20|J+B+JM+BM|95.3|\n|CTR-GCN|ICCV'23|J+B+JM+BM|96.5|\n|BlockGCN|CVPR'24|J+B+JM+BM|96.9|\n|GAP|ICCV'23|J|94.0|\n|TD-GCN|IEEE TMM'23|J|94.8|\n|InfoGCN|CVPR'22|J|96.5|\n|BlockGCN|CVPR'24|J|95.5|\n|SKETCH (Ours)||J|95.5|\n\n**Table E: Results on the JHMDB dataset for offline 2D human action recognition.**\n|Methods|Acc.|\n|-|-|\n|DDNet|77.20|\n|TDNet|79.30|\n|MSG3D|77.52|\n|CTR-GCN|73.02|\n|X3D|79.14|\n|C3D|80.20|\n|4s-AGCN|77.82|\n|SKETCH (Ours)|78.40|\n\n**Table F: Results on the SHREC'17 dataset (14 gestures)**\n|Methods|Acc.|\n|:-|:-:|\n|Key-frameCNN|92.9|\n|ParallelCNN|93.3|\n|STA-Res-TCN|94.5|\n|MFANet|94.6|\n|DDNet|94.6|\n|SKETCH (Ours)|94.6|\n\n**Question 3 & Weakness 2**\n- We conducted robustness experiments on the SHREC’19 dataset by applying missing-data perturbations—augmentations not used during training—to evaluate model generalization.\n   - **Missing-joint robustness**: Simulated joint-sensor failure by randomly deleting joints and measuring performance.\n   - **Occlusion robustness**: Simulated partial marker occlusions by masking joint coordinates over time and joints, then measuring performance.\n    - **Test noise robustness**: We injected white Gaussian noise directly into the joint coordinates of the training data and evaluated the model on test sequences corrupted with the same noise.\n- As shown in **Tabs. G and H**, we evaluated DR and FP under these three perturbations. Despite the synthetic perturbations, SKETCH remained robust by capturing the spatio-temporal flow of joint kinematics, rather than relying on individual joint trajectories. This ability to model long-range dependencies enabled SKETCH to handle missing joints effectively, without the need for additional augmentations of training, demonstrating its robustness in challenging conditions.\n\n**Table G: Robustness of our method under various perturbations of missing joints**\n|Perturbations|Missing Rate|DR↑|FP↓|\n|-|-|-|-|\n|-|-|0.92|0.02|\n|Sequence|30%|0.91|0.02|\n|Sequence|50%|0.89|0.02|\n|Sequence|70%|0.85|0.02|\n|Sequence|90%|0.73|0.34|\n|Frame|30%|0.90|0.02|\n|Frame|50%|0.78|0.20|\n|Frame|70%|0.06|0.00|\n\n**Table H:  Robustness of our method under noise perturbations with varying noise rates (std.)**\n|Perturbations|Std.|DR↑|FP↓|\n|-|-|-|-|\n|Noise|0.02|0.89|0.04|\n|Noise|0.03|0.89|0.04|\n|Noise|0.05|0.76|0.07|"}, {"review_id": "ojs5B5k4MA", "reviewer": "Reviewer_sCbL", "summary": "This paper presents a hand gesture recognition algorithm using 3d skeleton sequences as input. It works by transforming the collection of sequences (x,y,z)_t for all the joints of the hand into three graph plots {x(t),y(t),z(t)}, where for each coordinate, all the joints are represented within the same plot, with different colours, and using min-max normalisation. To retrieve the different relative dynamic ranges between coordinates, the same data are fed to a module that outputs a range embedding added to each plot. The three resulting plots - for a fixed-sized temporal window - are vertically stacked to form an image that is submitted to a swin-transformer network with one classification head that performs the gesture recognition, and two regression heads that respectively locate the start and end times of the gesture.", "questions": "The role of the Dynamic Range Embedding is not clear. Additional explanations are needed. And why learn an embedding range and not simply compute it from the sequences?\nOne important point of gesture recognition methods working on pose data is their robustness to pose estimation errors. How does your method deal with that?", "limitations": "There is a short section on limitations, but I consider that it does not address the main limitations mentioned above: 1) what is the fundamental contribution of the proposed visual representation of skeletal time series? 2) what is the robustness of the method with respect to pose estimation errors?", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper is well written and easy to read. The authors made a remarkable effort in the graphical presentation of their paper and the figures are excellent.\nThe major problem I have in this paper lies precisely in its visual representation of the skeletal data. I find it not only \"goofy\" as confessed in the title (and only there curiously), I find it scientifically shocking. The argument of \"reduction\" from 4d to 2d data does not make sense to me, since it's about transforming a light collection of time series into a high definition image with sparse relevant information (that lose topological information between joints BTW). And I don't think that good results are sufficient to make it acceptable. The authors should at least compare their approach with a more natural method using the native time series, and be able to explain why this anthropomorphic stacked-plots-as-image classification method possibly works better.  \nI also do not agree with some concluding claims saying that \"These results suggested that skeleton-based hand gesture recognition can play a key role\", and that \"the SKETCH module opened up a new perspective in hand gesture recognition through an interpretable data representation.\" I consider that interpretability comes from the pose estimation and the skeletal input, not from your method.", "was_revised": true, "final_justification": "I have read the reviews of my colleagues and the author's rebuttal. \n\nAfter consideration of the comprehensive point-to-point anwsers that have been provided by the authors, and although I'm still reserved about the method, and I find that some arguments - like those about the limitations of the analytical normalisation - are a little weak, I recognise the considerable efforts that have been made by the authors to improve their paper, and my opinion is significantly more positive. \n\nI therefore increase my score to \"borderline accept\".\n\nBest regards,", "rebuttal": "To ensure a complete and thoughtful response within the space constraints, **we kindly refer the Reviewer to tables cited in responses to other Reviewers**, as several analyses are shared across comments. **We sincerely appreciate your understanding.**\n\n**Disagreement with the 4D-to-2D Reduction Perspective**\n- We appreciate the Reviewer’s comment and agree that our method is not a true dimensionality reduction. Rather, it aims to unlock and structurally reorganize the rich spatio-temporal patterns embedded in the original 4D skeleton data—patterns that are often underutilized by conventional hand-crafted feature approaches. \n- As detailed in **Appendix A**, directly using raw joint coordinates risks overfitting due to geometric shortcuts and coordinate bias, which normalization alone cannot fully resolve. SKETCH addresses this by transforming skeleton sequences into 2D graph images that highlight joint trajectories and temporal structure, enabling the model to learn more robust motion features. \n- While this incurs some additional computational cost (**Tab. A** in our response to **Reviewer JPRk**), we believe the improved accuracy and real-time performance justify the trade-off.\n\n**Fundamental contribution of the proposed visual representation**\n- Prior work and limitations\n   - Please refer to our response to **Reviewer JPRk** under “Limitations of Prior Skeleton Methods and SKETCH’s Solutions” for details on related work and challenges.\n - Why visualization is needed and how SKETCH addresses existing limitations \n   -  Addresses **① raw data** challenges:\n      1. By transforming raw coordinates into images, SKETCH uses pretrained visual backbones to extract high-level features, preventing overfitting to geometric cues and reducing reliance on shortcuts.\n      2. Coordinate bias is mitigated through the pretrained backbone’s invariance to spatial transformations, enabling gesture recognition despite changes in viewpoint, scale, or location.\n      3. Inter-subject variability is addressed through per-window normalization, using relative joint positions in image space.\n   -  Overcomes Limitations of **② Hand-Crafted Features**:\n      4. Our representation eliminates the need for expert design, adopting a transformation similar to how humans interpret visual charts, making it generalizable and intuitive.\n      5. SKETCH enables interpretability by converting raw skeletons into visual images, allowing inspection of joint and temporal segment contributions to classification (**Figs. 4, 7 (Appendix D), and 14 (Appendix F)**).\n      6. Once mapped to the image domain, our data can be readily processed by global self-attention backbones, enabling explicit modeling of long-range dependencies between joints.\n - Contributions of the work\n   - The proposed SKETCH method leverages raw skeleton data to enhance interpretability and preserve rich spatio-temporal information, resulting in improved recognition performance compared to prior approaches (**Tabs. 1 and 2**). Under these conditions, SKETCH shows fast convergence in small number of epochs ($≤ 20$ epochs) (**Tab. B** in our response to **Reviewer JPRk**) and high generalization (**Tabs. C and D** in our response to **Reviewer JPRk**).\n\n**Concern About Loss of Joint Topology**\n- While we agree that topology is an important cue, SKETCH deliberately omits explicit 3D topological structure, it was deliberately designed to emphasize temporal joint dynamics via structured, time-aligned graph images.\n- As shown in **Appendix E.3** and **Supp. C**, we compared spatial (e.g., JCD) and temporal (e.g., FD) features under the same SKETCH pipeline; temporal ones consistently outperformed spatial (**Tab. 7 (Appendix E.3)**), suggesting motion cues are more discriminative in our setting. \n- For further analysis, we developed ShapeProj—a topology-preserving visualization designed purely for analysis—and found SKETCH yielded better accuracy (**Tab. 11 (Supp. C)**). This suggests temporal kinematics play a more decisive role in solving gesture recognition tasks.\n- Additionally, SKETCH’s topology-agnostic design simplifies deployment across diverse skeleton datasets requiring no prior structural knowledge, as further discussed in our response to **Reviewer D9gv's Question 2**.\n\n**Comparison with Native Time-Series Methods:**\n- Thank you for pointing this out. We clarify the model types used in **Tabs. A and B**, and now provide explicit comparisons to native time-series baselines (RNN, TCN, and Transformer).\n\n**Table A: Results on SHREC'19 compared to native time series methods**\n|Methods|Architecture|DR↑|FP↓|\n|-|:-:|-|-|\n|DSTA [40]|Transformer|0.81|0.08|\n|DG-STA [5]|Transformer|0.81|0.07|\n|uDeepGRU [4]|GRU (RNN)|0.85|0.10|\n|Ours|SKETCH|0.92|0.02|\n\n**Table B: Results on SHREC'22 compared to native time series methods**\n|Methods|Architecture|DR↑|FP↓|JI↑|\n|-|:-:|-|-|-|\n|DeepGRU [29]|GRU (RNN)|0.26|0.25|0.21|\n|DG-STA [5]|Transformer|0.51|0.32|0.40|\n|DSTA [40]|Transformer|0.73|0.24|0.61|\n|TN-FSM+JD [12]|Transformer|0.77|0.23|0.63|\n|CausalTCN [12]|TCN|0.80|0.29|0.68|\n|Ours|SKETCH|0.92|0.07|0.87|\n\n- This highlights SKETCH’s strength in capturing temporal dynamics over native time-series methods.\n\n**Advantages of the Stacked Plots as Image Representation:**\n- As discussed in **Appendix A**, raw joint sequences often suffer from coordinate bias and geometric shortcuts, even with normalization. SKETCH mitigates these issues by converting sequences into structured graph images that emphasize consistent joint trajectories and temporal dynamics.\n-  This allows pretrained vision transformers to better capture motion patterns and focus on relevant attention cues, yielding more robust and interpretable learning than approaches relying on hand-crafted features.\n\n**Clarifying the Source of Interpretability**\n- We strongly agree that interpretability stems from raw skeletal inputs and apologize for the unclear phrasing.\n- SKETCH enables intuitive joint-level attention by preserving alignment with original joint movements. In contrast, when applied to hand-crafted features, interpretability diminishes, as these features do not preserve joint-specific semantics and often result in unintuitive attention patterns.\n- SKETCH mitigates geometric shortcuts caused by using raw coordinates directly by transforming sequences into temporally structured graph images.\n- Our intention in referring to SKETCH as “interpretable” was to highlight that it maintains the transparency of raw skeleton inputs while enabling effective training in otherwise failure-prone settings.\n\n**Clarifying the Role of Dynamic Range Embedding**\n- We clarify that the Dynamic Range Embedding (DRE) is designed to address a limitation of the axis-wise min–max normalization used in SKETCH. While normalization improves consistency across subjects and viewpoints, it removes the relative magnitude of joint motion across axes—information that can be critical for distinguishing between gestures.\n- To restore this, DRE injects a scalar encoding of the original per-axis dynamic range into each subplot of the graph image. Unlike fixed statistical encodings (e.g., variance or max–min), DRE is learnable, allowing the model to adaptively capture semantically meaningful motion scales during training. This idea draws inspiration from prior work showing that learnable positional embeddings—used in place of sinusoidal encodings in Transformers—improve pattern modeling and flexibility during fine-tuning (**Appendix E.5**) [g].\n- We also conducted ablation studies (**Tabs. C and D**) comparing DRE to fixed encodings. DRE achieved better performance, likely due to its robustness to outliers and its ability to optimize jointly with the model. This highlights DRE’s value in enhancing discriminative capacity while maintaining model robustness.\n\n**Table C: Effectiveness of DRE vs. statistical encodings (Std. / Max–Min) on SHREC'19**\n|Ablation|Std.|Max‑Min|DRE|DR↑|FP↓|\n|-|:-:|:-:|:-:|-:|-:|\n|SKETCH (w/o DRE)|✓|||0.89|0.02|\n|SKETCH (w/o DRE)||✓||0.89|0.03|\n|SKETCH (w/ DRE)|||✓|0.92|0.02|\n\n**Table D: Effectiveness of DRE vs. statistical encodings (Std. / Max–Min) on SHREC'22**\n|Ablation|Std.|Max‑Min|DRE|DR↑|FP↓|JI↑|\n|-|:-:|:-:|:-:|-:|-:|-:|\n|SKETCH (w/o DRE)|✓|||0.90|0.08|0.84|\n|SKETCH (w/o DRE)||✓||0.89|0.06|0.84|\n|SKETCH (w/ DRE)|||✓|0.92|0.07|0.87|\n\n**Robustness to Pose Estimation Errors**\n- To assess SKETCH’s robustness to pose estimation errors, we additionally evaluated it under missing-data perturbations on SHREC’19—introduced only at test time—to examine generalization under degraded pose inputs.\n    - **Missing-joint robustness**: Simulated joint-sensor failure by randomly deleting joints and measuring performance.\n    - **Occlusion robustness**: Simulated partial marker occlusions by masking joint coordinates over time and joints, then measuring performance.\n    - **Test noise robustness**: We injected white Gaussian noise on training joint coordinates and evaluated the model on test sequences corrupted with the same noise.\n- As shown in **Tabs. E and F**, we evaluated detection rate (DR) and false positives (FP) under these three perturbations. Despite the synthetic perturbations, SKETCH remained robust by capturing the spatio-temporal flow of joint kinematics, rather than relying on individual joint trajectories. \n\n**Table E: Robustness of our method under various perturbations of missing-joints**\n|Perturbations|Missing Rate|DR↑|FP↓|\n|-|-|-|-|\n|-|-|0.92|0.02|\n|Sequence|30%|0.91|0.02|\n|Sequence|50%|0.89|0.02|\n|Sequence|70%|0.85|0.02|\n|Sequence|90%|0.73|0.34|\n|Frame|30%|0.90|0.02|\n|Frame|50%|0.78|0.20|\n|Frame|70%|0.06|0.00|\n\n**Table F:  Robustness of our method under noise perturbations with varying noise rates (std.)**\n|Perturbations|Std.|DR↑|FP↓|\n|-|-|-|-|\n|Noise|0.02|0.89|0.04|\n|Noise|0.03|0.89|0.04|\n|Noise|0.05|0.76|0.07|\n\n[g] Yu et al., \"Lape: Layer-adaptive position embedding for vision transformers with independent layer normalization.\", ICCV 2023."}]

Author rebuttal:

Meta-review: The paper proposes SKETCH – a lightweight pipeline that converts 4‑D skeleton streams into a compact, interpretable visual representation and feeds the resulting images to pre‑trained vision transformers (ViT/Swin). By embedding a Dynamic Range Embedding (DRE) to preserve relative motion scale, the method achieves real‑time, state‑of‑the‑art gesture detection on the SHREC’19/22 benchmarks while remaining highly interpretable.
In the rebuttal, the authors have successfully addressed most of the reviewers' concerns, resulting in two "accept" and two "borderline accept" ratings for the paper.
Given the strong novelty, practical relevance, and the authors’ thorough responses to reviewer concerns, we recommend acceptance of the paper. The contributions are clear, the experiments robust, and the method has immediate applicability to real‑world gesture‑recognition systems.
Decision: Accept (poster)