# IPN HandS: Efficient Annotation Tool and Dataset for Skeleton-Based Hand Gesture Recognition
## 3 IPN HandS Annotation Tool  
The purpose of the IPN HandS annotation tool is to provide a practical and efficient environment for annotating hand skeletons in video sequences, with a particular focus on datasets used for hand gesture recognition (HGR). Unlike general-purpose annotation tools, which often require frame-by-frame manual labeling and lack specialized support for skeletal structures, our tool combines automatic landmark detection with user-friendly refinement mechanisms to significantly reduce annotation time and improve consistency. The tool was designed specifically to overcome the limitations of existing software by supporting high-resolution landmark editing, interpolated corrections across frames, and seamless visualization of temporal gesture segments. These features make it particularly well-suited for refining noisy automatic detections (e.g., from MediaPipe), ensuring the annotated skeletons are reliable and coherent enough to be used as training data in skeleton-based HGR models. The graphical user interface of the annotation tool is shown in Figure 2. At the center of the interface is the video display panel, where annotated frames and, when available, the estimated hand and body skeleton (toggled by the checkbox) are visualized. The top of the interface includes a traditional menu bar offering access to diverse functions. Below the display panel, a set of navigation buttons allows the user to move frame by frame or play through the sequence, while buttons labeled MP Hands and MP Holistic trigger single-frame landmark estimation using the corresponding MediaPipe models. Beneath the control buttons, a horizontal slider enables quick navigation through the video sequence, allowing annotators to efficiently jump across frames. This is followed by a color-coded annotation timeline, which shows the temporal segmentation of gesture classes as defined by the original IPN Hand dataset. Each color corresponds to a gesture label, helping annotators identify and focus on gesture-specific segments. Finally, at the bottom of the interface, a confidence slider is used to adjust the detection threshold for the single-frame estimation tools, providing flexibility in handling uncertain or partially visible gestures. In general, the annotation process focuses on the efficient refinement of hand skeletons over long video sequences that contain multiple instances of gestures. To initialize the annotations, the MediaPipe Holistic model [13] is applied to all frames of a given video. This generates preliminary hand and body landmark detections, which are automatically saved and serve as a first-pass estimation for each frame. Annotators then navigate through the sequence by referencing the temporal segmentation of gesture classes, allowing them to focus on relevant gesture intervals. During this process, frames can be efficiently reviewed one by one, and if the initial detections are inaccurate, hand landmarks can be re-estimated using alternative models (MP Hands [12] and MP Holistic [13]) or by manually adjusting hand poses. The annotation process is supported by interpolation and copy-paste functionalities, which reduce the need for frame-by-frame correction and improve efficiency when refining sequences that exhibit consistent motion or contain static gestures. Once all necessary refinements are complete, the final annotations are saved for use in training hand gesture recognition models. To facilitate the correction of inaccurate detections, the tool provides a set of interactive editing capabilities. As illustrated in Figure 3, the annotator can perform fine-grained modifications to the hand skeletons by manually selecting and dragging individual landmarks, adjusting entire fingers, or repositioning the full hand. These operations enable intuitive corrections to spatial inaccuracies commonly introduced by automatic estimators, particularly in challenging frames with motion blur or occlusion. In addition to direct manipulation,  
Figure 2: The user interface of the IPN HandS annotation tool.