# FlickPose: A Hand Tracking-Based Text Input System for Mobile Users Wearing Smart Glasses
## 4 Machine Learning Model to Estimate Hand Pose
### Data Collection  
Hand pose classification methods include rule-based and machine learning models. However, the accuracy of rule-based methods is insufficient, and it is difficult to accommodate the users' diverse hand shapes and poses. Therefore, this study uses machine learning models to classify hand poses.  
To build a machine learning model to determine the five hand poses described above, the hand data were collected.  
Here, FlickPose uses right-hand key selection and left-hand hand poses to input characters, but a pattern in which the left and right hands are interchanged is also possible. However, including such a pattern would result in too many experimental comparisons, and so the role of each hand is fixed in this study. Therefore, only the hand data necessary to estimate the hand pose of the left hand is collected.  
Since it was not necessary to acquire dynamic changes to determine hand poses, data on the coordinates and orientation of each joint point, as described below, were collected.  
In addition to the five types of hand poses used in the system, data from other hand poses that do not belong to the five types should be collected to prevent misjudgment during hand pose switching. Then, in the implementation of the system, the current hand pose is updated when the hand pose is classified in another way.  
The \"other\" class includes specific hand shapes that do not fall under Poses 1-5. In particular, it covers 14 patterns, such as those shown in Figure 7, where each finger is moderately bent in a relaxed position. These poses are expected to occur frequently during typical user behavior.  
Figure 8 shows the data collection scene, which includes a text display screen and a hand data collection start button. The text screen is positioned in front of the user, while the start button is located near the user's hand and initially appears blue. To collect data,the user touches the start button with their right hand while maintaining a specific hand pose with the left hand. This records the coordinates of 24 joints and the orientation of the wrist base at intervals of 0.1 s.  
Since the \"other\" class includes many types of hand poses, collecting an equal number of samples for each pose would result in bias toward the \"other\" class. To avoid this, we collected 1000 samples for Pose 1 (thumb tucked in), Pose 5 (all fingers extended except the little finger), and poses in the \"other\" class, while collecting 5000 samples for each of the remaining poses. The number of samples collected per person for each class is shown in Table 1.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Class** & **Pose 1** & **Pose 2** & **Pose 3** & **Pose 4** & **Pose 5** & **Other** & **Total** \\ \hline Amount of data & 6000 & 5000 & 5000 & 5000 & 6000 & 14,000 & 41,000 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Amount of data to be collected per person.  
Figure 8: Scene of hand data collection.  
Figure 7: Hand pose with each finger moderately bent.  
Finally, the data collection system was used to conduct experiments to collect hand data from the subjects. The subjects were five males between the ages of 22 and 25, and the collection was conducted after explaining the experiment had been explained and the equipment tested. In addition, all five subjects performed the experiment at the same location. The data collected were split into training, validation, and test data at a ratio of 8:1:1.