# A Real-Time Dynamic Gesture Variability Recognition Method Based on Convolutional Neural Networks
###### Abstract
While these studies offer innovative systems for sign language gesture recognition, they are not without limitations. For example, using EMG signals is one of the systems [12] that may require invasive procedures, which may limit its applicability in real conditions. Another system [13] uses data from smartwatches, which may not capture all the nuances of sign language gestures. The use of Leap Motion technology in one of the systems [16] can also create problems with predicting the position of fingers. Leap Motion controllers also have problems with detecting some hand positions (e.g., vertical, in which one finger is covered by another). Finally, there is a problem in considering all the variations and the complexity of SL for a more accurate and reliable interpretation of SL in general. The authors of [16] present a system that uses data from a Leap Motion (LM) device and the HMC algorithm to learn gestures, achieving an average gesture recognition accuracy of 86.1% with a standard deviation of 8.2%. The system was tested on ASL gestures and was also able to recognize them with a typing speed of 3.09 words per minute. However, the authors pointed out some issues with using LM technology for gesture recognition. For instance, when the user's fingers are not visible to the infrared (IR) cameras, LM may make mistakes in predicting their position. In such cases, the hand may be depicted with folded fingers when they are actually extended. Additionally, the position of the thumb when pressed against the palm or between other fingers is poorly defined, making it difficult to reliably identify the gesture. The authors took a step forward in their study [6; 7] by including information about the movements of the human body and head together with the movements of the hands for a multimodal analysis of human movements. By recording changes in a person's posture, 3D head movement, and hand movement in a vector matrix, the authors strive to obtain a total picture of the dynamic gesture being performed. Approaches to gesture recognition [21; 22; 23; 24] based on wearable electronic input devices, which are gloves worn on the hand and containing various electronic sensors that track the movements of the hand, demonstrate fairly good gesture recognition results. However, these approaches have significant disadvantages, such as the size and material of the gloves and the complexity of connection and configuration, making them unsuitable, for example, for the elderly or for people with skin diseases. The authors of [25] have developed a multilingual approach in which hand movement modeling is also carried out using data that is independent of the target sign language through derivation of subunits of hand movement. The scientists tested the proposed approach by researching Swiss, German, and Turkish sign languages and demonstrated that SL recognition systems could be effectively developed using multilingual sign language resources. By integrating information from different modalities, the system can better understand the nuances of SL and accurately translate them into written language, greatly improving the accuracy and efficiency of automatic SL interpretation systems and making them more accessible to a wide range of users. Continuing the idea proposed in reference [25], in this paper, the authors propose a methodology for improving the prediction of categorical responses in SL recognition tasks by automatically collecting the spatiotemporal characteristics of gestures and grouping them by their form using the method of principal components, then selecting the optimal architecture that is suitable for all groups of gestures. This proposed method is also aimed at reducing the time spent on data preparation by automating data collection. The proposed method was tested on the popular datasets, including the Ankara University Turkish Sign Language Dataset (AUTSL) [26] and the Dataset for Argentinian Sign Language (LSA64) [27]. The experiment confirmed the system's effectiveness and multilingualism.