# Korean Sign Language Recognition Using Transformer-Based Deep Neural Network
###### Abstract  
Sign language recognition (SLR) is one of the crucial applications of the hand gesture recognition and computer vision research domain. There are many researchers who have been working to develop a hand gesture-based SLR application for English, Turkey, Arabic, and other sign languages. However, few studies have been conducted on Korean sign language classification because few KSL datasets are publicly available. In addition, the existing Korean sign language recognition work still faces challenges in being conducted efficiently because light illumination and background complexity are the major problems in this field. In the last decade, researchers successfully applied a vision-based transformer for recognizing sign language by extracting long-range dependency within the image. Moreover, there is a significant gap between the CNN and transformer in terms of the performance and efficiency of the model. In addition, we have not found a combination of CNN and transformer-based Korean sign language recognition models yet. To overcome the challenges, we proposed a convolution and transformer-based multi-branch network aiming to take advantage of the long-range dependencies computation of the transformer and local feature calculation of the CNN for sign language recognition. We extracted initial features with the grained model and then parallelly extracted features from the transformer and CNN. After concatenating the local and long-range dependencies features, a new classification module was applied for the classification. We evaluated the proposed model with a KSL benchmark dataset and our lab dataset, where our model achieved 89.00% accuracy for 77 label KSL dataset and 98.30% accuracy for the lab dataset. The higher performance proves that the proposed model can achieve a generalized property with considerably less computational cost. korean sign language (KSL); transformer; convolutional neural network (CNN); sign language recognition (SLR); hand gesture recognition (HGR) +
Footnote †: journal:  
+++ ==WARNING: Truncated because of repetitions==
same because it is not a motion representation like spoken language. The overall situation is unsuitable for the DHH communities' people to learn sign language, and they often face difficulties in communication due to this variation. Because of this difficulty, there are many obstacles for the DHH community to access their daily requirements, such as public information, social engagement, medical healthcare, education, etc. To ease their lives, we have been lowering the language barrier between the DHH and hearing people community.Many researchers have been working to develop automatic sign language recognition systems with advanced computer vision, robotics and natural language processing to ease their life [2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13]. Some researchers worked to direct a translation system from a sign language into a spoken language [14; 15]. Sensor-based [16] and vision-based approaches [17] are the most common domains for developing automated sign language systems [18]. In the last decade, CNNs have made an extraordinary contribution to the vision-based sign language recognition approaches to recording one-handed and two-handed statistical or dynamical movement [4; 5; 19; 20; 21; 22]. In the meantime, self-attention-based architectures have shown an excellent capability of capturing long-distance relationships and have become the de facto most popular approach in natural language processing (NLP) [23]. Inspired by NLP successes, multiple works combine CNN-like architectures with self-attention [24; 25], some replacing the convolutions entirely [26; 27]. Besides the NLP domain, the transformer model has been applied in different research domains for segmentation, detection, and classification by following the success in NLP [28]. In many works, the CNN architecture is completely replaced by the transformer. Recently, many researchers have proposed a transformer-based method to recognize sign language and classification [29].