# FlickPose: A Hand Tracking-Based Text Input System for Mobile Users Wearing Smart Glasses
## 4 Machine Learning Model to Estimate Hand Pose
### Classification Using Neural Network  
Machine learning is performed using the data obtained by preprocessing. The dataset consists of a set of three-dimensional coordinates for 24 joint points, i.e., 72-dimensional numerical data, and the total number of datapoints is 205,000 because data were collected from five persons. The training data in this study were preprocessed hand data. These were divided 8:1:1 and used as training data, validation data, and test data, respectively.  
Since FlickPose aims to run in stand-alone manner, it uses Unity Barracuda version 3.0.1 ([[https://[EMAIL_ADDRESS]/manual/index.html](https://[EMAIL_ADDRESS]/manual/index.html)]([https://[EMAIL_ADDRESS]/manual/index.html](https://[EMAIL_ADDRESS]/manual/index.html)), accessed on 23 June 2025) for the operation of the machine learning model. In addition, since the load placed on the device by the machine learning model needs to be reduced, it is necessary to choose a model that can run on the Unity Barracuda and is lightweight. Therefore, this study compares MLP, LightGBM, Random Forest, and Rule-Based methods as candidates. The models were built and tested on Google Colab using Keras with Tensorflow as the backend. Since the hand poses to be classified in this study deal with the bending and stretching of fingers with large joint changes and the number of patterns is small, the model uses the original architecture with a simple structure.  
First, training is performed in a way that uses all collected joint point positions as inputs. We constructed the machine learning model of MLP [29]. The architecture of the model consists of three dense layers as shown in Figure 9. The input of the model has 72 dimensions and outputs SoftMax values for 6 classes of poses 1-5 and Other. Note that Softmax is used as the activation function in the output layer, while ReLU is used in the other two layers. The other hyperparameters used are Adam (Adaptive Moment Estimation) for the optimizer, Sparse Categorical Crossentropy for the loss function, and accuracy for the evaluation function. The training results with the preprocessed hand data are demonstrated in Table 2. The dataset, as mentioned above, consists of the five varieties of hand data collected, divided 8:1:1 into training data, validation data, and test data.  
Note that accuracy, precision, recall, and Macro-F1 are used as evaluation indices. As shown in Table 2, high values exceeding 0.99 were obtained for all the evaluation indices, indicating that classification with high accuracy can be expected.  
Since FlickPose is intended for use in situations where text input is performed by other applications, the memory it occupies must be minimized. Therefore, we reduced the number of joint points used for inputs and conducted training with a model that uses as input a total of 15-dimensional data points from the joint points of the fingertips, which are assumed to be the most important in the estimation of the hand poses adopted in this study. The training results for this model are demonstrated in Table 3. The dataset is the same as when the model was trained using all the joint points demonstrated in Table 2.  
Comparing the evaluations in Tables 2 and 3, we see that the scores of the model with only the fingertips as input are slightly lower for each evaluation index, but the results for both models show that each value is very high, exceeding 0.99, indicating that even with only the fingertips as input, the accuracy is expected to be sufficient for character input. Therefore, in this study, only the joint point position of the fingertip was used as input to the model.