# Deep Forest-Based Monocular Visual Sign Language Recognition
## 4 Experiments and Results  
We use the public dataset ASLIVD and a private testing set in the experiments. The ASLIVD consists of more than 3300 American Sign Language (ASL) signs in video clips, including nouns, verbs, adjectives, and pronouns. Each sign is illustrated by 1-6 native ASL signers. In total, there are more than 9800 clips of signs. This dataset includes multiple synchronized videos showing the signs from different viewpoints. We only explore the front view of the monocular camera. Body joint features and sign recognition result examples are shown in Figure 4.  
The classifiers are trained with the annotated images. In our experiments, four classifiers, the state-of-the-art deep forest and the standard classifiers including support vector machine, decision tree, and logistic regression have been used and compared in this experiment. We list the performance of all four classifiers, while only the SVM and deep forest are chosen as the typical classifiers to make an in-depth analysis. In the experiment, there are 10 steps, and 10 signs are randomly chosen at the first step. Each sign includes around 3 video clips, and each clip consists of between 40 and 500 frames. All frames are labeled with their corresponding signs. Then, 20 signs are selected in the second step, 30 signs in the third step, and so on. Lastly, 104 signs are used to train the classifiers (we add 4 extra signs in that these signs can make up some common sentences). All of the experiments are conducted on a workstation with an Intel Xeon E5-1620 CPU, with 16 GB RAM and a Nvidia GTX 1080 Ti GPU. The average training time of the deep forest classifier is about 5143 s.  
The samples of 104 signs (about 24,385 samples) are randomly split into 80% training data (19,508 samples) and 20% testing data (4877 samples). We adopt precision, recall, and F1 score to evaluate the performance of multi-sign recognition. The F1 score is the harmonic average of the precision and recall, where an F1 score reaches between 0 and 1. The F1 score of the four classifiers in 10 experiments is shown in Figure 5. It is defined as  
\[\text{Precision} = \frac{TP}{TP+FP}\] \[\text{Recall} = \frac{TP}{TP+FN}\] \[\text{F}_{1} = 2\cdot\frac{\text{Precision}\cdot\text{Recall}}{\text{Precision} +\text{Recall}}, \tag{14}\]  
where \(TP\), \(FP\), and \(FN\) refer to the number of true positive, false positive, and false negative samples. In the random 10-sign experiments, all classifiers show good performance with F1 scores over 97%, as shown in Figure 5. However, with the increase in signs, the F1 score of the standard classifiers appears to decrease. The F1 score of the standard classifiers declines below 90% for 100 signs, while the deep forest classifier still has an F1 score of 97.7%. This can be explained as the standard classifiers are more useful for binary classification than multiple classification. Figure 4: Visualized features and extracted vectors of sign recognition from the American Sign Language Lexicon Video Dataset (ASLLVD). The performances of the two typical classifiers on the top 20 signs in the testing data are illustrated in the form of box-plots in Figure 6. The names of signs, the number of frames, precision, recall, and F1 score are listed in the columns \"Signs\", \"Number\", \"Precision\", \"Recall\", and \"F1\". In each classifier, the 20 signs with top number of test samples are listed in Table 2.  
Figure 5: Performance of the support vector machine (SVM) classifier and the deep forest classifier. Figure 5.0 Boxplot comparison between four classifiers. As shown in Table 2, SVM and deep forest have average F1 scores of 86% and 98%, respectively. The performances of these two classifiers are quite different on some signs. For example, for the sign _hot_, SVM gets 0.00 for precision, recall, and F1 score, while deep forest gets 1.00, 0.92, and 0.96, respectively. Moreover, for the sign _milk_, SVM gets precision, recall, and F1 scores of 0.48, 0.76, and 0.58, respectively, while deep forest gets 1.00 for all scores.