# IPN HandS: Efficient Annotation Tool and Dataset for Skeleton-Based Hand Gesture Recognition
## 1 Introduction  
Hand gesture recognition (HGR) plays a crucial role in various applications, including human-computer interaction, sign language recognition, augmented reality, and robotics [1, 2]. Unlike traditional input methods like the mouse and keyboard, vision-based interfaces offer a practical alternative due to the intuitive nature of hand gestures [3]. In some scenarios, static hand gestures might suffice for capturing simple commands through spatial cues. On the other hand, more complex dynamic gestures require precise motion interpretation. For example, touchless screen manipulation has gained significant importance in our hygiene-conscious world, reducing the need for physical contact [4, 5]. Among the different approaches to HGR, skeleton-based recognition has gained attention for its ability to capture hand dynamics efficiently while being robust to background, lighting, and appearance variations [2]. However, despite the growing interest and recent progress in this field [6, 7, 8, 9, 10, 11], advancement is still limited by the lack of large-scaledatasets containing dense, frame-level annotations of hand landmarks. This limitation is particularly evident in continuous gesture sequences captured under realistic conditions, where challenges such as motion blur, occlusion, and spontaneous hand motion frequently occur. Building reliable gesture recognition models for these settings demands annotations that are not only spatially precise but also temporally consistent. Yet, producing such annotations remains a significant bottleneck due to the manual effort required to label landmarks frame by frame. Recent deep learning models for hand pose estimation [12; 13] offer fast automatic detection of hand landmarks that can serve as a valuable starting point for annotation. Nevertheless, their outputs often suffer from inconsistencies across frames, as well as incomplete or missing detections, especially in complex poses or low-quality frames. As a result, manual refinement is crucial to address these issues and ensure both temporal and spatial coherence. Without such refinement, HGR models trained on noisy or incomplete annotations may struggle to accurately capture the true temporal dynamics of gestures, ultimately compromising recognition performance. To address the mentioned challenges, we introduce IPN HandS, an extension of our IPN Hand dataset [5] enriched with high-quality hand skeleton and temporal gesture annotations. Alongside this dataset, we present a novel annotation tool designed to ease the annotation process. By integrating automatic landmark detection, inter-frame interpolation, copy-paste functionalities, and manual refinement capabilities, our tool reduces the average annotation time from 70 to 27 min in videos of about 4000 frames long, enabling large-scale annotation with improved efficiency. Figure 1 illustrates the enhancement in annotation quality achieved through our refined IPN HandS annotations. It compares our annotations to the outputs of commonly used hand pose estimators, such as MediaPipe Hands [12] and MediaPipe H holistic [13]. The comparison focuses on a single frame that exhibits motion blur, which is a frequent and recurrent issue in video-based HGR applications. Further examples of annotation errors can be found in Section 4.3. We showcase the quality of our refined annotations by training a simple yet effective LSTM-based HGR model on the IPN HandS dataset. This model is evaluated using both our refined data and the noisier annotations generated by MediaPipe estimators. The results indicate accuracy improvements of about 12% and 8% when compared to models trained solely on MediaPipe Hands [12] and Holistic [13] annotations, respectively. These findings underscore the importance of high-quality annotations in advancing skeleton-based HGR methods. The main contributions of this paper can be summarized as follows:  
Figure 1: Comparison of hand skeleton annotations for a single frame with motion blur. (**a**) MediaPipe Hands [12], showing inaccurate finger landmarks.