# Korean Sign Language Recognition Using Transformer-Based Deep Neural Network
###### Abstract
For the image processing task, a pure transformer (Vit) is used to replace the CNN backbone [30]. The main concept of transformer-based image processing is firstly to split the original image into a discrete non-overlapping patch by following the word token of the NLP. Then the extracted patches are fed into the transformer for extracting the global relation and feature for the image classification. Inspiration by ViT, many methods have been developed based on the same concept, such as IPT and SETR [31; 32; 33]. The technique of the ViT is good for the image-based classification because researchers can directly apply the transformer to the sequence of the patches the same as the NLP sequence. However, some basic themes exist between the image-based vision tasks and the NLP sequence. Moreover, due to the fixed size of the patch, it is difficult to extract multi-scale feature maps and low-resolution features. High computational complexity is a well-known property of the transformer as well. To overcome the problem, researchers recently sequentially combined the transformer feature and the convolution layer, focusing on the computer vision-based task instead of the NLP sequence system CNN meet transformer (CMT) [28]. They employed four stages of the transformer and the CNN, where they mainly considered stage-wise design for extracting multi-scale features, reducing the resolution and increasing the dimension flexibility. However, the CMT method plays a crucial role in overcoming the problem of the pure transformer in the image processing task. However, their main concern is they combined CNN with a transformer sequentially and used it four times in four stages, increasing the time and computational complexity exponentially. In addition, few researchers have extracted CNN and transformer-based features in parallel concepts, and no work has been performed for Korean sign language recognition for the deaf and mute communities. To overcome the challenges, we proposed a convolutional layer-based transformer and CNN-based multi-branch model to recognize sign language recognition to overcome existing challenges. In the method, we parallelly extracted CNN and transformer-based features from the sign image and fed the concatenated feature into a classification module for classifying sign language. To overcome existing patches sequence problems [33], we employed a grain architecture module that firstly employed two-three \(\times\) three convolutions with a stride, then included a three \(\times\) three convolution with a stride of 1 and an output channel of 32 to reduce the size of input images for extra tracking and better local information extraction. The grain module mainly consists of a convolution and a layer normalization (LN), which is employed before the first stage to reduce the size of the intermediate feature (2\(\times\) down-sampling of resolution) and project it to a larger dimension (2\(\times\) enlargement of dimension). The convolutional layer-based transformer can extract short-range and long-range dependencies. Lastly, we employed a classification module with a global average pooling layer, a fully connected (FC) layer and an n-way classification layer with softmax. In addition, we added position embedding, fed into the attention-based deep learning model. The main contribution of the study is given below. * We present a deep learning adaptable Korean sign language (KSL) video dataset. * We proposed a multi-branch attention and CNN-based neural network to recognize sign language, which followed two stages to generate feature maps of different stages. The first stage included two parallel branches to generate feature maps of different scales, which are important for the dense prediction task. The second stage is considered a classification module. * A grain module was employed to reduce the size of the original image and solve the existing patched processing problems. Then, lightweight and CNN features were concatenated in the first stage and fed into the classification module in the second stage.