Towards Speech to Sign Language Translation

Sign Language (SL) is the primary means of communication for a majority of the hearing-impaired community. Current computational approaches in this research area have focused specifically on Sign Language Recognition [1] (SLR) and Sign Language Translation (from SL to text) [2] (SLT). However, the reverse problem of translating from spoken language to sign language has so far been unexplored. The goal of our ongoing project is to make the audio track content from online videos available to people with hearing disabilities, by automatically generating a video-based speech to sign language translation. In this paper, we will point out the shortcomings that limit the advances of this research area and propose first steps towards this end. Most existing researches in Sign Language Recognition focused on automatic recognition [3] and classification [4] of signs. SLR approaches have used handcrafted intermediate representations [5, 6] and the temporal information of these features have been designed using classical graph based approaches, such as Hidden Markov Models [7], Conditional Random Fields [8] or template-based methods [9, 10]. Nowadays, with the advance of deep learning, researchers have adopted Convolutional Neural Networks for manual [11, 12] or non-manual [13] feature representation, and Recurrent Neural Networks for temporal modeling [14]. However, most of these systems treat the problem as a simple recognition task ignoring the rich grammatical and linguistic structures of sign language that differs from spoken language [15]. It is also interesting to note that sign languages have their own linguistic rules [16], and spoken languages are not translated into sign languages on a word-by-word basis. Previous studies [17] have proposed composing sentences by recognizing an isolated set of signs without taking into account the special linguistic structure of sign language. In contrast, [2] addressed the problem of SLT in the framework of Neural Machine Translation and formulated the task as a sequence-to-sequence problem, resulting in the first end-to-end system to translate sign language into text. However, both these approaches use sign-language as their inputs to generate natural language text as output. Here, we propose to use a similar translationbased end-to-end model but for speech-to-sign language translation. To the best of our knowledge this venue is not yet explored but it would be the necessary methodology to enable a real-time speech to sign language translation. Nonetheless, in order to build an end-to-end speech-to-signs system, it is necessary to be able to combine components for speech recognition, machine translation and sign language/video synthesis. Furthermore, a large dataset that 2 Amanda Duarte, Gorkem Camli, Jordi Torres, and Xavier Giro-i-Nieto includes the speech signal and its respective interpretation in sign language is needed. Although there are Sign Language datasets available [18, 17, 19–22, 22– 24] they are usually weakly annotated. Moreover, most of them are recorded in controlled environments with limited vocabulary which inhibits the end goal of SLT. In Table 1 we list some of the most common sign language datasets along with their language ids, segmentation level, public availability and principal content provided. All datasets presented here contain videos including single or multiple signers interpreting general or specific subjects. These content are segmented by sentence-level, word-level or just letter. The content fall into sentence-level category when the continuous sings interpretation is provided, in the word-level when the dataset provides just isolated signs or just letters(or finger spelling) when it contains just letters, numbers or specific signs. We show in the table that some datasets also provide gloss-level sign-by-sign written information along with notations to account the facial and body grammar that goes with the signs. Some of them also provide the correspondent text translation of the respective spoken sign language. We would like to bring the readers attention to the fact that none of these datasets have a speech component included. In our opinion, this is one of the major challenges and one reason for lack of prior work in the area of Sign Language Translation from speech to sign language. Table 1. Sign Language standard datasets: DGS and ASL stands for German Sign Language and American Sign Language respectively. Trans designates the translation/transcription of the content into the respective language. X* sign indicates that dataset is available through contact the authors. Dataset Name Language ID Segmentation Public? Content