# Gesture Recognition Based on 3D Human Pose Estimation and Body Part Segmentation for RGB Data Input
## 3 Proposed Method
Figure 5 shows the example of temporal 3D skeleton joints of video frames extracted by the Pose_3D network. Figure 4: Example of color-encoded 12 body parts segmentation from RGB images sequence by the RefineNet model. * Two-stream 3D_ResNet networks for RGB and color body part segmentation modalities: The residual 3D_CNN network based on ResNet architecture [23] was applied to benefit from two data modalities: the background subtracted RGB images and color-coded 12 body parts segmentation images for gesture classification. For the input of 3D_CNN of the RGB image branch, we subtracted the background of the RGB image by color body part segmentation and fed it into the network. Due to spatiotemporal feature learning by 3D convolution and 3D pooling, the 3D_CNN network is known as one of the essential frameworks for video classification. Residual 3D_CNN could significantly improve the classification performance of basic 3D_CNN framework. The 3D_ResNet is one of the current residual 3D_CNN versions. Various ResNet-based architectures with 3D convolutions were studied, but the 3D_ResNet network based on ResNeXt-101 was employed because of the quality performance for the proposed method. Different from other original bottleneck blocks with a standard convolutional layer, the ResNeXt block employs a group convolution layer with its capacity to divide the feature maps into small groups. The single 3D_ResNet stream modality network in our proposed method included five ResNeXt blocks. The structure of each ResNeXt block consisted of convolution layers (group convolution layer), a batch normalization (BatchNorm) layer, and a rectified-linear unit (ReLu) layer, as shown in Figure 6. The input of each modality stream network was a fixed number of \(T\) sampled frames of a video: \(V_{c}=\{x_{c1},\,x_{c2},\,\ldots,\,x_{cT}\}\) for RGB modality and \(V_{ps}=\{x_{ps1},\,x_{ps2},\,\ldots,\,x_{psT}\}\) for color body part segmentation modality. The operation function of these stream-networks was \(\Theta_{c}\)(.) and \(\Theta_{ps}\)(.) respectively. Then, the prediction probabilities of the stream networks for RGB input and color body part segmentation for \(i\) classes can be described as:  
\[P_{c}l{p_{1}}|V_{c},\,p_{2}|V_{c},\,\ldots,\,p_{i}|V_{c}]=\Theta_{c}(V_{c}) \tag{1}\]  
\[P_{ps}l{p_{1}}|V_{ps},\,p_{2}|V_{ps},\,\ldots,\,p_{i}|V_{ps}]=\Theta_{ps}(V_{ps}) \tag{2}\]  
where \(p_{i}\) is the prediction probability of the video belonging to the class \(i\)th, and \(\rm{P_{c}}\) and \(\rm{P_{ps}}\) denote the network outputs, which are the vectors or class prediction probabilities. Figure 5: Example of skeleton joints extraction from RGB images sequence by temporal video 3D human pose estimation-Pose_3D. * The LSTM network for 3D skeleton joints modality: The LSTM network was proposed as a submodel for gesture recognition to benefit from the extracted 3D skeleton joints data. The 3D skeleton data provides useful information about the temporal features such as the localization of the relevant body joints over a time series to recognize the performed action. The LSTM networks are utilized to capture the contextual information of a temporal sequence for long periods by the gates and memory cells. In an LSTM network, the operation of gates and memory cells by time can be described as follows: \[\left\{ \begin{array}{l}i_{t}=\sigma(W_{i}[x_{t},h_{t-1}]+b_{i}),\\ f_{t}=\sigma\big{(}W_{f}[x_{t},h_{t-1}]+b_{f}\big{)},\\ \alpha_{t}=\sigma(W_{o}[x_{t},h_{t-1}]+b_{o}),\\ \overline{c_{t}}=\tan h(W_{c}[x_{t},h_{t-1}]+b_{c}),\\ c_{t}=f_{t}*c_{t-1}+i_{t}*\overline{c_{t}},\\ h_{t}=\tan h(c_{t})*\alpha_{t}\end{array} \right.\] (3) where \(i,f,\) and \(o\) are the vectors of the input gate, the forget gate and the output gate, respectively. \(\overline{c_{t}}\) and \(c_{t}\) denote the \"candidate\" hidden state and internal memory of the unit respectively. ht is the output hidden state and \(\sigma(. )\) is a sigmoid function while \(W\) and \(b\) represent connected weights matrix and bias vectors, respectively.