# Gesture Recognition Based on 3D Human Pose Estimation and Body Part Segmentation for RGB Data Input
## 2 Related Works  
In this section, we briefly described the previous methods relevant to our work for gesture recognition. Although significant improvement on gesture recognition has been reported, but new challenges appear with different input modalities and restrictions. In early years, gesture or action recognition problem has been dealt with classical machine learning methods such as support vector machine (SVM) [1], bag-of-features [2], and hidden Markov model (HMM) [3]. In these methods, gestures are classified based on the features extracted by a hand-engineered extractor. [PERSON] et al. [1] presented a discriminative descriptor for action classification based on the covariance matrix for 3D skeleton joint locations. [PERSON] [2] similarly used the SVM classifier to identify gesture classes, but via a bag-of-words vector mapped from key points extracted by scale-invariant feature transform (SIFT). [PERSON] [3] presented a method using HMM based on the likelihood threshold estimation of the input pattern. The gesture recognition task has been tackled by conventional machine learning methods, but there are significant limitations in these approaches. For instance, the parameters of the model depend on experience, and the system is sensitive to noise. Recently, due to the revolution of deep learning, gesture recognition approaches have been presented with impressive performances compared to the traditional methods. The deep learning-based methods became popular due to its capability to extract spatiotemporal features from the raw video input automatically. A convolutional neural network (CNN) was initially used for extracting spatial features for static images, but it has been extended to deal with different input types or to extract different types of features. Various approaches [4; 5; 15; 16] have utilized CNN to treat sequential frames of a video as multi-channel inputs for the purpose of video classification. [PERSON] [4] incorporated a two-stream network with separate ConvNets for RGB images and optical flow images to extract motion information for gesture classification. [PERSON] [15] proposed data level fusion to combine RGB and optical flow modalities with static images to extract action features. CNN can be incorporated with RNN or LSTM to learn both spatial and temporal features of a video for action classification. [PERSON] [8] deployed long-term recurrent convolutional network (LRCN), in which CNN is used to extract spatial features of images and LSTM is applied to capture temporal dependencies in the sequence of such features. For dynamic gesture recognition, [7; 17; 18] applied a 3D convolutional neural network to capture discriminative features along both spatial and temporal dimensions due to 3D convolutions and 3D pooling. [PERSON] [17] used spatiotemporal features extracted by 3 DCNN to classify gesture with SVM classifier. [PERSON] [16] proposed recurrent 3D convolutional neural networks (R3 DCNN) to recognize gestures online, where 3 DCNN is used as a feature extractor. The gesture recognition based on a human pose has also achieved impressive results. Utilizing RNN or LSTM to capture temporal features from human skeleton joints for gesture classification is gaining popularity [19; 20; 21], in which skeleton joints are extracted by depth information with a ToF camera, but it becomes more challenging when only RGB data input is used. In other works, multiple data modalities or multiple deep learning models are combined to achieve better performance in gesture classification. [PERSON] [9] presented a convolutional two-stream consensus voting network based on 3 DCNN for RGB and depth channels to identify the gesture classes in a video input. [PERSON] [10] also proposed a multi-stream model based on RNN with hand location information extracted from RGB-D input.