# IPN HandS: Efficient Annotation Tool and Dataset for Skeleton-Based Hand Gesture Recognition
## 5 Hand Gesture Recognition Evaluation
### Experimental Results  
Table 6 summarizes the classification accuracy and average recall of models trained and evaluated using a single annotation source. The model trained and evaluated on IPN HandS annotations achieved the best performance overall, with an accuracy of 91.2% and an average recall of 78%. In contrast, models trained and tested using MediaPipe-based annotations yielded significantly lower recall values, 62.7% for MP Hands and 66.9% for MP Holistic, despite achieving relatively high overall accuracy. This discrepancy stems from the class imbalance in the dataset, where a large proportion of instances belong to pointing gestures P1F and P2F (more than a 10\(\times\) difference, as shown in Table 4). These classes tend to dominate the predictions, inflating the overall accuracy even when less frequent gestures are poorly recognized. Average recall, by contrast, treats all classes equally and, therefore, more accurately reflects per-class consistency and generalization. The average recall gap between the IPN HandS model and MediaPipe highlights the importance of high-quality  
spatial and temporal annotations for improving the recognition of all gesture categories, not just the most common ones. To further explore the generalization capacity of models trained on different annotation types, Table 7 presents the cross-annotation results. Specifically, we tested models trained on IPN HandS against MediaPipe annotations, and vice versa. When trained on IPN HandS and tested on MediaPipe annotations, performance remains strong: 87% accuracy and 74.5% recall on MP Hands, and 89.6% accuracy with 74.7% recall on MP Holistic. These results reflect substantial improvements of about 12% and 8% in average recall compared to training and testing on MP Hands and MP Holistic annotations, respectively. This improvement reflects the model's ability to generalize to noisy test annotations, thanks to the high-quality spatial and temporal consistency it learned during training. Conversely, the models trained on MediaPipe annotations perform worse when evaluated on IPN HandS. For example, a model trained on MP Hands yields only 61.1% recall on IPN HandS data, underscoring that models trained on noisier annotations fail to capture the nuanced patterns needed for reliable recognition. Even the more stable MP Holistic variant achieves just 67.1% recall when tested on refined annotations. In summary, training on the temporally consistent and spatially accurate annotations provided by IPN HandS not only improves in-domain performance but also significantly enhances model robustness under cross-annotation evaluation. These findings confirm the value of manual refinement for building models that generalize beyond their training distribution and are less sensitive to the noise and inconsistencies typically found in automatic pose estimators. Figure 8 shows the normalized confusion matrices for three key evaluation scenarios: training and testing on IPN HandS annotations, training and testing on MP Holistic annotations, and training on IPN HandS with testing on MP Holistic. The improvements enabled by our refined annotations are particularly evident in gestures with fine spatial differences, such as distinguishing between one-finger (C1F) and two-finger clicking (C2F), or between zoom-out (ZOT) and grab gestures (GRB). Classes like throwing-left (TLT) and throwing-right (TRT), which are frequently misclassified when trained with MediaPipe outputs, show more concentrated and accurate predictions when trained with IPN HandS. One notable exception is the 2C1F class, which shows reduced performance in the IPN HandS model compared to those trained with automatic annotations. This may result \begin{table}
\begin{tabular}{l c c} \hline \hline
**Annotations** & **Accuracy (\(\sigma\))** & **Average Recall (\(\sigma\))** \\ \hline IPN HandS & 91.2 (1.4) & 78.0 (1.2) \\ MP Hands & 87.5 (1.0) & 62.7 (2.0) \\ MP Holistic & 89.2 (0.6) & 66.9 (0.7) \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Gesture classification results.