To process a sequence, there have been several interesting variants of recurrent neural networks (RNNs) proposed including long short-term memory (LSTM) [3] and gated recurrent units (GRUs) [4]. These architectures have been successfully employed to resolve many problems involving the process of sequential data such as machine translation and image captioning [5; 6; 7; 8]. Moreover, many researchers working on the field of image and video understanding have raised the level that seemed infeasible even a few years ago by learning their neural networks with a massive amount of training data. Recently, many neural network models based on convolutional neural network (CNNs) exhibited excellent performances in various visual tasks such as image classification [9; 10], object detection [11; 12], semantic segmentation [13; 14], and action recognition [15; 16].  
Understanding sign languages requires a high level of spatial and temporal understanding and, therefore, is regarded as very difficult with the current level of computer vision and machine learning technology [1; 17; 18; 19; 20; 21; 22]. It should be noted that sign languages are different from hand (finger) languages as the hand languages only represent each letter in an alphabet with the shape of a single hand [23] while the linguistic meaning of each sign is determined by subtle difference of shape and movement of body, hands, and sometimes by facial expression of the signer [2]. More importantly, the main difficulty comes from the lack of dataset for training neural networks. Many sign languages represent different words and sentences of spoken languages with temporal sequences of gestures comprising continuous pose of hands and facial expressions. This implies that there are uncountably many combinations of the cases even to describe a single human intention with the sign language.  
Hence, we restrict ourselves to the task of translating sign language in various emergency situations. We construct the first Korean sign language dataset collected from fourteen professional signers who are actually hearing-impaired people and named it the KETI sign language dataset. The KETI sign language dataset consists of 14,672 high-resolution videos that recorded the Korean signs corresponding to 419 words and 105 sentences related to various emergency situations. Using the KETI sign language dataset, we present our sign language translation model based on the well-known off-the-shelf human keypoint detector and the sequence-to-sequence translation model. To the best of our knowledge, this paper is the first to exploit the human keypoints for the sign language translation problem. Due to the inherent complexity of the dataset and the problem, we present an effective normalization technique for the extracted human keypoints to be used in the sign language translation. We implement the proposed ideas and conduct various experiments to verify the performance of the ideas with the test dataset.  
The main contributions of this paper are highlighted as follows:  
1. We introduce the first large-scale Korean sign language dataset for sign language translation.
2. We propose a sign language translation system based on the 2D coordinates of human keypoints estimated from sign videos.
3. We present an effective normalization technique for preprocessing the 2D coordinates of human keypoints.
4. We verify the proposed idea by conducting various experiments with the sign language dataset.