# Experimenting the Automatic Recognition of Non-Conventionalized Units in Sign Language
## 2 Materials and Methods
### Proposal for a Generalizable and Compact Continuous Sign Language Recognition Framework
#### 2.3.1 Signer Representation  
Since available training data are limited in quantity, we have decided to partly rely on pre-trained models for signer representation, with a separate processing of upper body, face and hands--which are usually dealt with in very specific ways, whether in SL-specific or non-SL-specific models. Upper Body: \(\mathtt{Image}\to\mathtt{2D}\), \(\mathtt{Image}\to\mathtt{3D}\), \(\mathtt{Image}\to\mathtt{2D}\to\mathtt{3D}\)  
CNNs have emerged as a very effective tool to get relevant features from images. OpenPose (OP) [47; 48] is a powerful open source library, with real-time capability for estimating 2D body pose. Widely used in the gesture recognition community, its Image\(\rightarrow\) 2D body pose estimation module is fast--close to real-time for 25 frames per second (fps) videos on a modestly powerful desktop computer--is easy to use and works well even when part of the body is missing from the image--we only kept the 14 upper body keypoints and leave out the leg keypoints. This is a great benefit, as most SLs videos only show the upper body. Many other pose estimation models we have experimented on do not offer this feature. Although direct Image\(\rightarrow\) 3D models do exist (for instance [49]), we were not able to find one fitting our requirements. Indeed, as for Image\(\rightarrow\) 2D models, prediction usually fails when part of the body is missing from the image, or when the person is not centered with respect to the image. Another type of issue is related to the training data of these models. As they were not trained with SL data, our experience is that they do not perform well when fed with SL images. Fortunately, the 2D OP estimates have proven robust even on SL videos. Therefore, we decided to rely on OP in order to get good 2D estimates, then train a 2D\(\rightarrow\) 3D Deep Neural Network (DNN), reproducing the architecture from [50]. In the end, a Image\(\rightarrow\) 2D\(\rightarrow\) 3D model was thus obtained, aiming to learn the function \(f\) that estimates the third coordinate for each landmark of the signer in frame \(t\), that is, with \(n\) as the number of landmarks (\(n=14\) in our case):  
\[f: \begin{array}{rcl}\mathds{R}^{2n}&\longrightarrow&\mathds{R}^{n}\\ \left[(\widehat{x}_{1t},\widehat{y}_{1t}),(\widehat{x}_{2t},\widehat{y}_{2t}), \ldots,(\widehat{x}_{nt},\widehat{y}_{nt})\right]&\longmapsto&[\widehat{z}_{1t},\widehat{z}_{2t},\ldots,\widehat{z}_{nt}]\end{array}  \tag{17}\]  
where \(\widehat{x}_{it}\), \(\widehat{y}_{it}\) and \(\widehat{z}_{it}\) are a standardized version of the original coordinates \(x_{it}\), \(y_{it}\) and \(z_{it}\), with respect to the whole training dataset. The training loss is defined as the Euclidean distance between predictions and ground-truth data. The training data we decided to use for training consisted of motion capture data from the LSF corpus MOCAP1 [51]. These data have been particularly valuable since it contains high precision 3D landmarks recording of LSF, from four different signers. In order to increase model generalizability, data augmentation techniques were used during training. In detail, the 3D data from MOCAP1 were randomly rotated at each training epoch, with added pan \(\Delta\theta_{p}\in[-45^{\circ},+45^{\circ}]\), added tilt \(\Delta\theta_{t}\in[-20^{\circ},+20^{\circ}]\) and added roll \(\Delta\theta_{r}\in[-5^{\circ},+5^{\circ}]\). The proposed DNN is implemented with Keras [52] on top of TensorFlow [53]. All hidden layers use Rectified Linear Unit activation [54], with Dropout to prevent overfitting [55]. RMSProp was used as the gradient optimizer [56]. Six neuron layers were stacked, with sizes [28; 28; 28; 28; 28; 14]. The proposed Image\(\rightarrow\) 2D\(\rightarrow\) 3D pipeline for processing the 3D upper body pose from signers in RGB frames is shown in Figure 6.  
Hands  
Hands are obviously one of the main articulators in SL.