# Deep Forest-Based Monocular Visual Sign Language Recognition  
[PERSON]  
1 School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China; [EMAIL_ADDRESS] (Q.X. ); [EMAIL_ADDRESS] (D.W.)1  
[PERSON]  
1 School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China; [EMAIL_ADDRESS] (Q.X. ); [EMAIL_ADDRESS] (D.W.)1  
[PERSON]  
1 School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China; [EMAIL_ADDRESS] (Q.X. ); [EMAIL_ADDRESS] (D.W.)1  
[PERSON]  
1 School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China; [EMAIL_ADDRESS] (Q.X. ); [EMAIL_ADDRESS] (D.W.)1  
Footnote 1: email: {[EMAIL_ADDRESS], [EMAIL_ADDRESS], [EMAIL_ADDRESS], xue_qifan@fox  
The visual SLR method using depth/stereo cameras, such as Kinect(r), can generate outstanding results using a depth vision feature [8; 9]. Kinect(r) is a motion-sensing device produced by Microsoft that is based on a time-of-flight camera. These cameras have a large overhead of hardware, and they are sensitive to environments like variant illumination. Compared with the depth/stereo camera, the monocular camera has a lighter structure and is less expensive, but requires robust recognition algorithms. Since the monocular camera only produces RGB images without depth information, it requires a powerful algorithm to achieve a promising result compared with depth/stereo cameras [2]. In the field of semantic recognition, there are two tendencies: the traditional methods, including hidden Markov models (HMMs) and dynamic time warping (DTW) [10; 11], and the machine learning methods such as support vector machine (SVM) and deep neural networks (DNNs) [12]. The HMM model was mainly used in the field of speech recognition in the early days [13; 14]. Although the HMM has achieved great success in speech recognition, its performance in SLR is not satisfactory. The reason is that the traditional HMM method needs to establish HMM models for each gesture separately, affecting the real-time performance of the system. In contrast, the DTW method is simple and effective. The optimal dynamic programming matching algorithm can be used to improve the accuracy rate of SLR [15]. Since DTW is based on a template matching algorithm, it is difficult for it to learn from data, which limits its robustness. Machine learning methods have the characteristics of high parallelism, adaptability, and certain learning capabilities [16; 17]. In particular, DNNs normally have various network models, which satisfy different application requirements. The accuracy of the DNNs depends on the number of training samples. In the field of sign language, as shown in Table 1, there is no public sign language dataset that has a large number of various signs and meanwhile has a sufficient volume of samples for each sign. It is hard for DNNs to achieve good performance if there are only between one and three samples for a sign. In addition, the computational overhead is also an obstacle in practical applications. This paper introduces a novel SLR method for dynamic gestures that has high robustness and a strong generalization performance. We propose a combined joint model with both hand and arm joints to represent a human's pose. Considering the characteristics of joints, we employ the extracted joints as the body features to explain the sign language. Overall, our SLR model involves two steps: In the first step, a visual skeleton extraction method is used to encode the body joint information via the OpenPose detector [18]. In the second step, a small sample data is used to train a classifier based on the deep forest model [19], which is compared with the SVM-based one. The paper is organized as follows: In Section 2, the paper introduces relevant research from the aspect of skeleton detection. In Section 3, the basic modeling formulation is presented with the visual skeleton extraction method, the joint feature re-encoding method, the semantic classification, and the voting mechanism.