# Handwriting Recognition Based on 3D Accelerometer Data by Deep Learning
## 4 Experiments and Numerical Evaluation
### Evaluation and Results  
The three proposed neural network architectures were evaluated and extensively compared. For this purpose, the dense layer and the Adam optimization algorithm remained operating without modifications in the stochastic objective functions during the training. The database was divided into ten disjoint subsets using a different number of samples per class. Furthermore, each subset \(C_{i}\) was built with 40, 80, 120, 160, 200, 240, 280, 320, 360, and 399 samples per class. The performance evaluation was conducted using a k-fold cross-validation technique with \(k=10\). Each subset was subdivided to complete the evaluation; 90% of data was used for training, and the remainder, 10%, was used for testing.  
Training and test stages were processed on a PC using an Intel(r) Core(tm) [EMAIL_ADDRESS] GHz, 16 GB RAM, and a GPU NVIDIA GTX 1080 with 8 GB of DDR5 memory, and 2560 CUDA cores.  
Figure 5 shows the distribution of classes after applying a dimensionality reduction using the t-Distributed Stochastic Neighbor Embedding (t-SNE) algorithm [37].  
In order to have a clear idea of how the classes are distributed in the original feature space, the t-SNE method was applied. That method explicitly shows the intrinsic difficulty of classifying each class. The t-SNE method only was used to visualize the data distribution  
Figure 5: Distribution of the features’ space. (**Above**) For X, Y, and Z axes. (**Below**) For the three axes.  
after reducing the dimension of the original feature space. The feature space dimension was reduced from \(\mathbb{R}^{116}\) to \(\mathbb{R}^{2}\), and each color represents each of the 36 classes used to classify in Figure 5.  
Additionally, the distributions of the feature space obtained from the 3D accelerometer data for all classes in the X, Y, and Z axes are shown in Figure 5. In this paper, the three axes were used in the recognition process. According to Figure 5, clearly separated clusters can be observed for the X axis, whereas for the Y and Z axes, clusters look mostly overlapped. The numerical results are shown in Figure 6. As it can be observed, the three implemented architectures have a comparable performance for more than 120 samples per class.  
For example, in Figure 6, it was observed that the LSTM architecture presented the best performance using only a few samples. However, when 120 samples are used, the three architectures exhibit almost the same performance. It is important to notice that the three architectures present nearly the same performance for the complete database (i.e., 399 samples per class).  
The loss function and the Area Under the Curve (AUC) combined with the Receiver Operating Characteristic (ROC), forming the AUC-ROC parameter, were used to evaluate the performance of the three models in detail. The loss function was used to calculate the model errors during the optimization process, and the ROC curve expresses the ability of a model to classify. The numerical results for the loss function and ROC curve for the proposed CNN, LSTM, and CNN-LSTM architectures are shown in Figure 7, respectively. The numerical results of the AUC-ROC for the three last proposed architectures are 0.99464, 0.99679, and 0.9950.  
The loss function used in each proposed architecture was the Cross-Entropy or Log loss. This function measures the performance of a model in classification tasks, where the output is a probability value between 0 and 1. The Cross-Entropy loss function is widely used in Deep Neural Networks [38; 39].  
Figure 6: Accuracy of three proposed architectures (CNN, LSTM, and CNN-LSTM) for different numbers of samples per class.