# Korean Sign Language Recognition Using Transformer-Based Deep Neural Network
## 4 Proposed Methodology  
In the study, we proposed a convolutional layer-based transformer and CNN as a multi-branch hybrid network to include the transformer and convolutional neural network (CNN) feature by following the architecture of CMT [28], ResNet-50 [22], and DeiT [33, 78]. Most of the existing transformer-based systems directly split the input image into multiple patches, and the main drawback of the patch system is that it can be poorly modeled with a linear projection. In the study, though we followed the patches concept, we employed a new architecture, grain architecture, to implement the patches idea instead of splitting the image. In the grain architecture, firstly, we included two 3 \(\times\) 3 convolutions with stride one. Then we used a 3 \(\times\) 3 convolution layer with a 2-stride layer where 32 was used as an output channel for reducing the size of the image to establish the patch concept of the transformer and modern CNNs [5, 22]. Our model has three stages--initial grain, feature extraction, and classification module, shown in Figure 3a. The initial stage is used to perform the patches mechanism; the second stage is a feature extraction consisting of two parallel branches for generating different scales of feature maps to achieve dense prediction tasks. Finally, the stage is considered a classification module. After concatenating the two branches of features, we produced the final features fed into the classification module. In the convolutional layer-based transformer, we included an initial module, convolutional layer, layer normalization and lightweight multi-head attention layer and MLP convolution. In the initial module, we combined a convolutional layer with a normalization layer to produce the hierarchical representation of the grained features [5, 33]. Convolutional layer-based transformer extracted short-range and long-range dependencies among the pixels. Lastly, we employed a classification module, including some convolutional layer with GELU activation and then a global average pooling layer, a fully connected (FC) layer and an n-way classification layer with a SoftMax activation function. Algorithm 1 shows the overall working flow where we included every step we followed in our work.  
Figure 2: Proposed Korean sign language word.  
```
Input: Set of Input Dataset \(P_{i}\in P(n)\)  Number of Samples: N, 70% for Training and 30% for Test Output: Set of vector \(s_{i}\) define Model(input=InputLayer, outputs=ClassificationLayer): \(GrainFeature\gets GrainModule(D)\) \(FirstBranchFeature\gets ConvolutionalLayerBasedTransformer(GrainFeature)\) \(SecondBranchFeature\gets CNN(GrainFeature)\) \(FinalFeature\gets Concatenate(FirstBranchFeature,SecondBranch-Feature)\) \(PredictedClass\gets Classification-Module(Final-Feature)\) return PredictedClass while\(i\
eq NumEpochs\)do // For Training while\(Batch\
eq NumberBatchTraining\)do \(PredictedClass\gets Model(Batch)\) \(Loss\gets Criterion(PredictedClass,TrainClass)\) \(UpdateInteloss\gets Loss.backward(),Optimizer.Step()\) // For Testing while\(Batch\
eq NumberBatchTesting\)do \(PredictedClass\gets Model(Batch)\) \(Output\gets CPerformanceMatrix(PredictedClass,TestClass)\)
```  
**Algorithm 1** Pseudocode of the proposed system.