# Experimenting the Automatic Recognition of Non-Conventionalized Units in Sign Language
## 2 Materials and Methods
### Proposal for a Generalizable and Compact Continuous Sign Language Recognition Framework
#### 2.3.1 Signer Representation
Although linguists do not all share a common ground for the description and linguistic role of sub-units for the hands, three important parameters have been identified. More specifically--at least from an articulatory point of view--the location, shape and orientation of both hands are known to be critical, along with the dynamics of these three variables, that is: hand trajectory, shape deformation and hand rotation. In addition to the body pose feature, the OP library also includes a \(\mathtt{Image}\to\mathtt{2D}\) hand pose estimation module, with RGB images as input [58]. From our experience, this module is quite sensitive to the image resolution, and even more to the video frame rate. Indeed, very poor results are obtained on blurred images, which is often the case for the hands with 25-30 fps videos in standard resolution. This is mostly due to the fact that hands can move fast in SL production, causing motion blur around the hands and forearms. Moreover, the hand shapes used in SL can be very sophisticated, and somehow never used in the daily life of non-signing people. Therefore, in all likelihood, the data that were used to train the OP models did not include such hand configurations, which sometimes makes predictions unreliable. That being said, the OP hand module can still be seen as a good and light proxy for hand representation. Let us note that although \(\mathtt{Image}\to\mathtt{3D}\) hand pose models have been developed (see for instance [49]), we have not found any that was able to provide a reliable estimate on hand pose on real-life 25 fps SL videos. Indeed, these models are even more sensitive to the issues encountered by the \(\mathtt{Image}\to\mathtt{2D}\) estimators. An alternative direction is to extract global features from hand crops. Focusing on hand shape--thus setting aside location and orientation--a SL-specific model was developed in [18]. This CNN model classifies cropped hand images into 61 predefined hand shapes classes and was trained on more than a million frames, including motion blurred images. Three SL corpora of different types were compiled (Danish Sign Language (DTS), New Zealand Sign Language (NZSL), German Sign Language (DGS)). Even though the hand shapes frequency of occurrence is very likely to vary between different SLs, we have made the assumption that SLs other than DTS, NZSL and DGS could still be dealt with without retraining the model. Indeed, many hand shapes are obviously shared across most SLs, since they are used to depict salient and/or primary forms (flat, round, square, etc.). The trained prediction model 1-miohands-v2 is publicly available ([[https://www-i6.informatik.rwth-aachen.de/~koller/1](https://www-i6.informatik.rwth-aachen.de/~koller/1) miohands/]([https://www-i6.informatik.rwth-aachen.de/~koller/1](https://www-i6.informatik.rwth-aachen.de/~koller/1) miohands/)), under the Caffe architecture [59]. A simplified scheme is presented in Figure 7. The input of the model is a cropped hand image, which was processed by  
Figure 6: Proposed \(\mathtt{Image}\to\mathtt{2D}\to\mathtt{3D}\) pipeline for the upper body pose, applied to a random frame from the French Sign Language corpus LS-COLIN [57]. OpenPose enables to get 2D estimates, then a DNN model was used to estimate the missing third coordinate of each landmark. several CNN layers. The final layer is of Fully Connected (FC) type, with 61 neurons, one for each class. The model outputs the most probable class with a softmax operation. However, we chose to extract the output of the last fully-connected layer and thus get a much more informative representation vector of size 61, for each hand, instead of the unique value corresponding to the most probable class. Face and Head Pose  
Similarly to body pose and to hand pose, the OP library makes it possible to get a 70-keypoint \(\texttt{Image}\rightarrow\texttt{2D}\) face pose estimate.