# A Dynamic Gesture Recognition Interface for Smart Home Control based on Croatian Sign Language
## 4 Experiments
### Performance Evaluation of the Sign Recognition Module  
In this work, a sign recognition module is realized as a hierarchical architecture of two identical parallel subnetworks, each operating on different input modality. For the training process, we firstly initialize each of two subnetworks with Nvidia hand gesture dataset using respective RGB and Depth data. For model initialization, a total of five training-validation runs are performed. The pre-trained subnetworks are selected based on max reported validation accuracy of 62.3% and 65.6%, respectively, for RGB and depth modality. After obtaining the validation accuracy, the model was updated with the rest of the 306 validation samples to finish the initialization process. The following fine-tuning process to the SHSL dataset started with a learning rate of 0.005, and it was controlled by scheduler that adjusted in order to reduce the learning rate by factor 5 if the cost function did not improve by 10%. For this experiment, we compared the performance of each single-modality subnetwork, and modality-fusion model concerning the number of input frames M, to determine the influence of a particular input modality on the recognition result. The sign recognition module was evaluated using a leave-one-subject-out (LOSO) cross-validation scheme on the SHSL dataset. As explained in Section 3.4, the SHLS dataset contains video data of 40 users where every user was recorded performing each of 25 sign language gestures twice. In the proposed validation scheme, analyzed networks are trained with video data collected from 39 subjects, and tested on a single user. The average classification results of 40 users for LOSO are shown in Table 2. From the results, it is visible that network accuracy depends on the length of input sequence M, were using longer input achieves a higher performance. Also, we can inspect that employing Depth modality network performs better in comparison with RGB data, while a significant performance improvement was reported when modality fusion was introduced.  
To improve the recognition for personalized use, we implemented an additional fine-tuning process in which we retrain dense layers with the first version of users' trials containing each of 25 gestures. Similarly, as in the LOSO scheme, performance evaluation was made by training each model with data of 39 users (1950 videos) together with the first 25 corresponding videos of the current  
\begin{table}
\begin{tabular}{l c c c} \hline \hline  & **8-Frames** & **16-Frames** & **32 Frames** \\ \hline SHSL–Depth & 60.2 & 66.8 & 67.4 \\ SHSL–RGB & 59.8 & 62.2 & 63.4 \\ SHSL–Fusion & 60.9 & 69.8 & 70.1 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Sign language recognition—leave-one-subject-out (LOSO).  
Figure 5: The accuracy of Wake-up model during training.  
signer. Results in Table 3 show an average accuracy of 40 users from which it is evident that the user adaptation technique can additionally increase performance.  
Performance per user is given in Figure 6. where it is visible that signer #5 achieved the lowest accuracy of 58.8% while signer #2 achieved the best accuracy of 80.9%. Also, Figure 6 enables us to inspect accuracy improvement per user where it is visible that most notable improvement of 7.8% is achieved for signer #21, while there are few cases where this procedure decreases the results as with singer #12 (\(-\)3.3%); in the average subject, adaptation delivers an improvement of 2.1%.