## 5 Experimental Results  
We implemented our networks using PyTorch [43], which is an open source machine learning library for Python. The Adam optimizer [44] was used to train the network weights and biases for 50 epochs with an initial learning rate 0.001. During the training, we changed the learning rate every 20 epochs by the exponential decay scheduler with discount factor 0.5. We also used the dropout regularization with a probability of 0.8 and the gradient clipping with a threshold 5. Note that the dropout regularization is necessarily high as the size and the variation of the dataset is small compared to other datasets specialized for deep learning training. For the sequence-to-sequence models including the vanilla seq2 seq model and two attention-based models, the dimension of hidden states was 256. For the Transformer model, we used the dimension for input and output (\(d_{\text{model}}\) in [42]) of 256. The other hyper-parameters used for the Transformer were the same as in the original model including the scheduled Adam optimizer in their own setting. Moreover, the batch size was 128, the augmentation factor was 100, the number of chosen frames was 50, and the object 2D normalization was used unless otherwise specified. As our dataset is annotated in Korean, which is an agglutinative language, the morphological analysis on the annotated sentences should be performed because the size of dictionary can be arbitrarily large if we split sentences into words simply by white-spaces in such languages. For this reason, we used the Kkma part-of-speech (POS) tagger in the KoNLPy package, which is a Python package developed for natural language processing of the Korean language to tokenize the sentences into the POS level [45]. To evaluate the performance of our translation model, we basically calculated \"accuracy\", which means the ratio of correctly translated words and sentences. It should be mentioned that the accuracy was calculated only for comparing sentence-level annotation and gloss-level annotation in Table 3 since it is not suitable to compare two cases by accuracy since we had five ground truth sentences for each sing video. In this experiment, we trained our model with a single ground truth sentence or a single sequence of glosses. Besides, we also utilized four types of metrics that are commonly used for measuring the performance of machine translation models: BLEU [46], ROUGE-L [47], METEOR [48], and CIDEr [49] scores. **Sentence-level vs. Gloss-level training. ** As in [23], we conducted an experiment to compare the translation performance depending on the type of annotations. Because each sign corresponds to a unique sequence of glosses as well as to multiple natural language sentences, it is easily predictable that the gloss-level translation shows better performance. Indeed, we can confirm the anticipation from the summary of results provided in Table 3.  
This also leads us to the future work for translating sequences of glosses into natural language sentences. We expect that the sign language translation can be a more feasible task by separating the task of annotating sign videos with natural language sentences by two sub-tasks where we annotate sign videos with glosses and annotate each sequence of glosses with natural language sentences. **Comparison with CNN-based approaches. ** In Table 4, we compare our approach to the classical methods based on CNN features extracted from well-known architectures such as ResNet [50] and VGGNet [51]. Since the size of sign video frames (\(1920\times 1080\)) is different from the size of input of CNN models (\(224\times 224\)), we first cropped the central area of frames in \(1080\times 1080\) and resize the frames to \(224\times 224\). The experimental results show that ResNet-152 exhibits the best translation performance on the validation set and the VGGNet-16 demonstrates the best performance on the test set.