# Automatic Recognition of Mexican Sign Language Using a Depth Camera and Recurrent Neural Networks
## 4 Results
### System Robustness to Noise  
To evaluate the system's robustness to noise, we created five additional testing sets with different levels of Gaussian noise in the keypoints coordinates. The noise levels went from zero to 50 cm of standard deviation in increasing steps of 10 cm. Table 5 shows the testing accuracies of the best model variation from each architecture evaluated on different testing sets. The models ending with _aug_ are the ones augmented during training with Gaussian noise on the inputs as described in Section 3.3.  
From the results of Table 5 we can conclude that the LSTM model is more robust to noise than the RNN and GRU models. Additionally, adding Gaussian noise during training helped significantly. Due to this, we selected a small LSTM architecture with 32 units on the first layer and 16 units on the second layer and trained multiple models with increasing levels of Gaussian noise augmentation to the inputs going from 0 to 100 cm of standard deviation. Table 6 shows the accuracy of these LSTM models evaluated on multiple testing noise levels. This table indicates that a Gaussian noise augmentation of 30 cm of standard deviation gave the best results for increasing noise levels on the test data.  
Figure 7 shows the precision and recall curves for these different models evaluated on the testing set with Gaussian noise of 40 cm of standard deviation. The figure demonstrates that the best model corresponds to the one trained with Gaussian noise of 30 cm of standard deviation.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**LSTM** & \multicolumn{6}{c}{**Testing Noise**} \\ \cline{2-7}
**Model** & **No-Noise** & **10 cm** & **20 cm** & **30 cm** & **40 cm** & **50 cm** \\ \hline
0 cm & 92.44 & 66.22 & 65.33 & 63.11 & 67.77 & 62.66 \\
10 cm & **96.44** & 74.22 & 74.66 & 74.44 & 75.33 & 72.66 \\
20 cm & 94.88 & 84.22 & 82.44 & 79.55 & 83.11 & 84.22 \\
30 cm & 93.11 & **86.0** & 87.33 & **85.11** & **87.11** & **87.11** \\
40 cm & 81.55 & 80.66 & 81.77 & 79.55 & 83.33 & 82.88 \\
50 cm & 82.44 & 79.77 & 78.22 & 79.55 & 80.66 & 79.77 \\
60 cm & 68.44 & 68.88 & 71.77 & 70.0 & 71.33 & 72.44 \\
70 cm & 70.66 & 71.55 & 71.33 & 70.0 & 72.88 & 69.55 \\
80 cm & 59.33 & 58.22 & 56.44 & 57.11 & 57.55 & 59.33 \\
90 cm & 61.11 & 57.77 & 58.22 & 59.55 & 58.22 & 60.22 \\
100 cm & 61.33 & 58.66 & 60.22 & 57.55 & 58.44 & 60.22 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Classification accuracy of multiple LSTM models trained with different levels of noise augmentation to the inputs evaluated on noisy testing sets. The best result on each column is highlighted in bold.  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Best** & \multicolumn{6}{c}{**Testing Noise**} \\ \cline{2-7}
**Model** & **No-Noise** & **10 cm** & **20 cm** & **30 cm** & **40 cm** & **50 cm** \\ \hline RNN & 92.44 & 45.11 & 45.33 & 46.44 & 46.44 & 46.88 \\ RNN _aug_ & 63.55 & 60.44 & 58.44 & 60.0 & 59.33 & 59.33 \\ LSTM & 96.66 & 66.22 & 65.33 & 63.11 & 67.77 & 62.44 \\ LSTM _aug_ & 95.55 & **89.33** & **90.44** & **89.11** & **90.44** & **88.88** \\ GRU & **97.11** & 48.22 & 50.66 & 51.11 & 46.44 & 46.66 \\ GRU _aug_ & 96.22 & 69.11 & 69.33 & 68.66 & 68.44 & 67.33 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Classification accuracy of the best model variation of each architecture evaluated on noisy testing data. The names ending with _aug_ refer to a model augmented during training with Gaussian noise on the inputs of zero mean and standard deviation of 30 cm. Every column represents a test set with a different noise level. The best result on each column is highlighted in bold.  
In the next section, we report the results of a set of ablation studies aimed to study the performance of our classifier at different model depths and a different set of input features to understand the contribution to the overall system.