Taiwan Sign Language Recognition System Using LC-KSVD Sparse Coding Method

Sign language, for deaf-impaired people, plays an important role in communication. In this paper, we devise a Taiwan Sign Language recognition system. We use the Kinect2 sensor to get data from 94 sign morphemes shown once by 4 people, and extract hand shape features and trajectory features from depth images and joints of the body skeleton. Finally, we have each sign morpheme dictionary trained by label consistent K-SVD (LC-KSVD) sparse coding algorithm for recognition. Experiments show our system performs well and the accuracy achieves 99.47% in close test. Introduction Sign language, a kind of languages not via the oral vocal system, combines hand gestures with body movements and even more facial expressions to show its meaning. For many deaf-impaired people, sign language is mother language [1]. Representing manner is not only one type but also many kinds for the same meaning of sign language, such as the character of Chinese and English among spoken languages. Hundreds of sign languages are in use around the world and also are the cores of local deaf cultures, like Taiwan Sign Language. According to previous sign language recognition research [2], sign language recognition is identified to four blocks of sign language: hand shape, hand position, trajectory orientation and movement. With four blocks in mind, hand gestures can be classified as hand postures and spatiotemporal gestures [3]; hand postures are combinations of hand shapes and orientations, and spatiotemporal gestures refers to where the hand is placed relative to the body and hand movement traces out a trajectory in space. Taiwan Sign Language (TSL) is exactly a kind of sign language, and the key points of lexical meaning resolution, e.g., hand shapes, positions, actions, directions, not hand motions, contact point, relationship of hands, repeat movements, and posture intensity, is observed from sign morphemes, rules of structure and restriction [4]. Sign language recognition system can be constructed by the foregoing factors. However, sign language recognition system for extracted methods of features can be distinguished between image-based and sensor-based approaches [5]. The advantage of image-based systems for signers uses easy equipment, but needs a lot of computing of pre-processing. This paper belongs in the image-based sign language recognition system. A classical image-based system has five processing step: image obtained, pre-processing, segmentation, features extracted and classify [6], and so does the sign language recognition system roughly in this paper. In the previous sign language works, S. Mo et al. [8] get hand shapes with color skin; but in their experiment result, selecting color skin for usage cannot catch hand shapes possibly, if there are similar color objects. In [9], A. -M. Cretu et al. use depth information and the threshold of distance to capture hand shapes; nevertheless, it is hard to catch hand shapes when the hand is near body. In [10], X. Wu et al. use depth joints features to draw hand location and trajectory; although, trajectory features do not used with hand shapes for gestures recognition. The above-mentioned problems are considered as the opinions of features choice. Depth images and joints are the target used to extract features in this paper, and are illustrated their characteristic in the chapter 3. 6th International Conference on Machinery, Materials, Environment, Biotechnology and Computer (MMEBC 2016) © 2016. The authors Published by Atlantis Press 519 In this paper, the Histogram of Oriented Gradients (HOG) [12] algorithm referred from [11] is used to extract the hand features. In the HOG algorithm, the local oriented gradients are computed from depth image of the hands, and do normalization respectively. The normalized values are considered as hand shape features. However, the dimension of hand features computed by HOG is so large that it has to do dimension reduction via the Principal Component Analysis (PCA) algorithm to get the lower dimensional and useful hand features. In spite of effectively reduced dimensions of hand feature via PCA, the data quantity of the sign language is still quite big and needs more time cost did classification directly. For the better effect, the method, the improved sparse coding: Label Consistent K-SVD (LC-KSVD) [14,15], is applied in the dimension reduction and the classification. We introduce Kinect Sensor and LC-KSVD in section 2, proposed method is given in section 3. Last section is experiment and conclusion. A Sign Data Stream Depth Skeleton Depth Image Compute each vector of pair of joints Get hands shape With HOG + Input Data Extracted Features Features Sparse Code Test Data Discriminative Dictionaries