ISLTranslate: Dataset for Translating Indian Sign Language
Year: 2023

Abstract:
Sign languages are the primary means of communication for many hard-of-hearing people worldwide. Recently, to bridge the communication gap between the hard-of-hearing community and the rest of the population, several sign language translation datasets have been proposed to enable the development of statistical sign language translation systems. However, there is a dearth of sign language resources for the Indian sign language. This resource paper introduces ISLTranslate, a translation dataset for continuous Indian Sign Language (ISL) consisting of 31k ISL-English sentence/phrase pairs. To the best of our knowledge, it is the largest translation dataset for continuous Indian Sign Language. We provide a detailed analysis of the dataset. To validate the performance of existing end-to-end Sign language to spoken language translation systems, we benchmark the created dataset with a transformer-based model for ISL translation.

heading: 1 Introduction
text: There are about 430 million hard-of-hearing people worldwide1 of which 63 million are in India2. Sign Language is a primary mode of communication for the hard-of-hearing community. Although natural language processing techniques have shown tremendous improvements in the last five years, primarily, due to the availability of annotated resources and large language models (Tunstall et al., 2022), languages with bodily modalities like sign languages still lack efficient language-processing systems. Recently, research in sign languages has started attracting attention in the NLP community (Yin et al., 2021; Koller et al., 2015; Sincan and Keles, 2020; Xu et al., 2022; Albanie et al., 2020; Jiang et al., 2021; Moryossef et al., 2020; Joshi
1https://www.who.int/news-room/fact-sheets/ detail/deafness-and-hearing-loss
2https://nhm.gov.in/index1.php?lang=1&level=2& sublinkid=1051&lid=606
et al., 2022). The availability of translation datasets has improved the study and development of NLP systems for sign languages like ASL (American Sign Language) (Li et al., 2020), BSL (British Sign Language) (Albanie et al., 2021), and DGS (Deutsche Gebärdensprache) (Camgoz et al., 2018). On the other hand, there is less amount of work focused on Indian Sign Language. The primary reason is the unavailability of large annotated datasets for Indian Sign Language (ISL). ISL being a communication medium for a large, diverse population in India, still faces the deficiency of certified translators (only 300 certified sign language interpreters in India3), making the gap between spoken and sign language more prominent in India. This paper aims to bridge this gap by curating a new translation dataset for Indian Sign Language: ISLTranslate, having 31,222 ISL-English pairs. Due to fewer certified sign language translators for ISL, there is a dearth of educational material for the hard-of-hearing community. Many government and non-government organizations in India have recently started bridging this gap by creating standardized educational content in ISL. The created content helps build basic vocabulary for hard-of-hearing children and helps people use spoken languages to learn and teach ISL to children. Considering the standardized representations and simplicity in the vocabulary, we choose these con-
3The statistic is as per the Indian government organization Indian Sign Language Research and Training Centre (ISLRTC): http://islrtc.nic.in/
10466
tents for curating an ISL-English translation dataset. We choose the content specific to education material that is standardized and used across India for primary-level education. Consequently, the vocabulary used in the created content covers diverse topics (e.g., Maths, Science, English) using common daily words.
ISL is a low-resource language, and the presence of bodily modality for communication makes it more resource hungry from the point of view of training machine learning models. Annotating sign languages at the gesture level (grouping similar gestures in different sign sentences) is challenging and not scalable. Moreover, in the past, researchers have tried translating signs into gloss representation and gloss to written language translation (Sign2Gloss2Text (Camgoz et al., 2018)). A gloss is a text label given to a signed gesture. The presence of gloss labels for sign sentences in a dataset helps translation systems to work at a granular level of sign translation. However, generating gloss representation for a signed sentence is an additional challenge for data annotation. For ISLTranslate, we propose the task of end-to-end ISL to English translation. Figure 1 shows an example of an ISL sign video from ISLTranslate. The example shows a translation for the sentence “Let’s discuss”, where the signer does the sign for the word “discuss” by circularly moving the hands with a frown face simultaneously followed by palm lifted upwards for conveying “let’s.” The continuity present in the sign video makes it more challenging when compared to the text-to-text translation task, as building a tokenized representation for the movement is a challenging problem. Overall, in this resource paper, we make the following contributions:
• We create a large ISL-English translation dataset with more than 31,222 ISL-English pair sentences/phrases. The dataset covers a wide range of daily communication words with a vocabulary size of 11,655. We believe making this dataset available for the NLP community will facilitate future research in sign languages with a significant societal impact. Moreover, though not attempted in this paper, we hope that ISLTranslate could also be useful for sign language generation research. The dataset is made available at: https://github.com/ Exploration-Lab/ISLTranslate.
• We propose a baseline model for end-to-end ISL-English translation inspired by sign language transformer (Camgoz et al., 2020).

heading: 2 Related Work
text: In contrast to spoken natural languages, sign languages use bodily modalities, which include hand shapes and locations, head movements (like nodding/shaking), eye gazes, finger-spelling, and facial expressions. As features from hand, eye, head, and facial expressions go in parallel, it becomes richer when compared to spoken languages, where a continuous spoken sentence can be seen as a concatenated version of the sound articulated units. Moreover, translating from a continuous movement in 3 dimensions makes sign language translation more challenging and exciting from a linguistic and research perspective. Sign Language Translation Datasets: Various datasets for sign language translation have been proposed in recent years (Yin et al., 2021). Specifically for American Sign Language (ASL), there have been some early works on creating datasets (Martinez et al., 2002; Dreuw et al., 2007), where the datasets were collected in the studio by asking native signers to sign content. Other datasets have been proposed for Chinese sign language (Zhou et al., 2021), Korean sign language (Ko et al., 2018), Swiss German Sign Language - Deutschschweizer Gebardensprache (DSGS) and Flemish Sign Language - Vlaamse Gebarentaal (VGT) (Camgöz et al., 2021). In this work, we specifically target Indian Sign Language and propose a dataset with ISL videos-English translation pairs. End-to-End Sign Language Translation Systems: Most of the existing approaches for sign language translation (Camgoz et al., 2018; De Coster and Dambre, 2022; De Coster et al., 2021) depend on intermediate gloss labels for translations. As glosses are aligned to video segments, they provide fine one-to-one mapping that facilitates supervised learning in learning effective video representations. Previous work (Camgoz et al., 2018) has reported a drop of about 10.0 in BLEU-4 scores without gloss labels. However, considering the annotation cost of gloss-level annotations, it becomes imperative to consider gloss-free sing language translation approaches. Moreover, the gloss mapping in continuous sign language might remove the grammatical aspects from the sign language. Other recent works on Sign language translation include Voskou et al.
(2021); Yin and Read (2020), which try to remove the requirement for a glossing sequence for training and proposes a transformer-based architecture for end-to-end translations. We also follow a gloss-free approach for ISL translation.

heading: 3 ISLTranslate
text: ISLTranslate is a dataset created from publicly available educational videos produced by the ISLRTC organization and made available over YouTube. These videos were created to provide school-level education to hard-of-hearing children. The videos cover the NCERT4 standardized En-
4https://en.wikipedia.org/wiki/National_ Council_of_Educational_Research_and_Training
glish educational content in ISL. As the targeted viewers for these videos are school children and parents, the range of words covered in the videos is beginner-level. Hence, it provides a good platform for building communication skills in ISL. The videos cover various NCERT educational books for subjects like science, social science, and literature. A single video (about 15-30 minutes) usually covers one chapter of a book and simultaneously provides an audio voice-over (in English) conveying the same content. Apart from ISLRTC’s educational sign videos which make up a significant portion of ISLTranslate, we also use another resource from Deaf Enabled Foundations (DEF) (https://def.org.in/). DEF videos consist of words with respective descriptions and example sentences for the same words, along with the text transcriptions available in the descriptions5. We split the DEF Sign videos into multiple segments using visual heuristics for separating segments corresponding to words, descriptions, and examples. In total, ISLTranslate consists of 2685 videos (8.6%) from DEF, and the remaining 28537 videos (91.4%) are from ISLRTC. ISLTranslate Creation: We use the audio voiceover (by analyzing the speech and silence parts) to split the videos into multiple segments. Further, these segments are passed through the SOTA speech-to-text model (Radford et al., 2022) to generate the corresponding text. As the generated text is the same as present in the book chapters’ text, verifying the generated sample was easy and was done by manually matching them with the textbook. In general, we found automatically transcribed text to be of high quality; nevertheless incorrectly generated text was manually corrected with the help of
5Example video: https://www.youtube.com/watch?v= 429wv1kvK_c
content in the books. Figure 2 shows an example (from ISLTranslate) of a long sentence and its translation in ISL. The frames in the figure are grouped into English words and depict the continuous communication in ISL. Notice the similar words in the sentence, “sign/signs” and “language.” (also see a visual representation of Sign Language6). As common to other natural languages, representations (characters/gestures) of different length are required for communicating different words. In ISLTranslate, we restrict to the sentence/phrase level translations. The dataset is divided into a train, validation, and test splits (Details in App. A). App. Figure 3 shows the distribution of number of samples in various splits.
6https://www.youtube.com/watch?v=SInKhy-06qA
Comparison with Other Continuous SignLanguage Datasets: We primarily compare with video-based datasets containing paired continuous signing videos and the corresponding translation in written languages in Table 1. To the best of our knowledge, we propose the largest dataset for ISL.
Data Cleaning and Preprocessing: The videos (e.g., App. Fig. 4) contain the pictures corresponding book pages. We crop the signer out of the video by considering the face location as the reference point and removing the remaining background in the videos.
Noise Removal in ISLTranslate: As the ISLTranslate consists of videos clipped from a longer video using pauses in the available audio signal, there are multiple ways in which the noises in the dataset might creep in. While translating the text in the audio, a Signer may use different signs that may not be the word-to-word translation of the respective English sentence. Moreover, though the audio in the background is aligned with the corresponding signs in the video, it could happen in a few cases that the audio was fast compared to the corresponding sign representation and may miss a few words at the beginning or the end of the sentence. We also found a few cases where while narrating a story, the person in the audio takes the character role by modifying speech to sound like
the designated character speaking the sentence. For example, in a story where a mouse is talking, instead of saying the sentence followed by the “said the mouse” statement, the speakers may change their voice and increase the pitch to simulate dialogue spoken by the mouse. In contrast, in the sign language video, a person may or may not take the role of the mouse while translating the sentence to ISL. ISLTranslate Validation: To verify the reliability of the sentence/phrase ISL-English pairs present in the dataset, we take the help of a certified ISL signer. Due to the limited availability of certified ISL Signers, we could only use a small randomly selected sign-text pairs sample (291 pairs) for human translation and validation. We ask an ISL instructor to translate the videos (details in App. C). Each video is provided with one reference translation by the signers. Table 2 shows a sample of sentences created by the ISL instructor. To quantitatively estimate the reliability of the translations in the dataset, we compare the English translation text present in the dataset with the ones provided by the ISL instructor. Table 3 shows the translation scores for 291 sentences in ISLTranslate. Overall, the BLEU-4 score is 48.94, ROUGE-L (Lin, 2004) is 60.44, and WER (Word Error Rate) is 61.88. To provide a reference for comparison, for text-totext translations BLEU score of human translations ranges from 30-50 (as reported by Papineni et al. (2002), on a test corpus of about 500 sentences from 40 general news stories, a human translator scored 34.68 against four references). We speculate high reliability over the translations present in the ISLTranslate with a BLEU score of 48.93 compared against the reference translations provided by certified ISL Signer. Ideally, it would be better to have multiple reference translations available for the same signed sentence in a video; however, the high annotation effort along with the lower availability of certified ISL signers makes it a challenging task.

heading: 4 Baseline Results
text: Given a sign video for a sentence, the task of sign language translation is to translate it into a spoken language sentence (English in our case). For benchmarking ISLTranslate, we create a baseline architecture for ISL-to-English translation. We propose an ISL-pose to English translation baseline (referred to as Pose-SLT) inspired by Sign Lan-
guage Transformer (SLT) (Camgoz et al., 2020). Sign language transformer uses image features with transformers for generating text translations from a signed video. However, considering the faster realtime inference of pose estimation models (Selvaraj et al., 2022), we use pose instead of images as input. We use the Mediapipe pose estimation pipeline7. A similar SLT-based pose-to-Text approach was used by Saunders et al. (2020), which proposes Progressive Transformers for End-to-End Sign Language Production and uses SLT-based pose-to-text for validating the generated key points via back translation (generated pose key points to text translations). Though the pose-based approaches are faster to process, they often perform less than the image-based methods. For the choice of holistic key points, we follow Selvaraj et al. (2022), which returns the 3D coordinates of 75 key points (excluding the face mesh). Further, we normalize every frame’s key points by placing the midpoint of shoulder key points to the center and scaling the key points using the distance between the nose key point and the shoulders midpoint. We use standard BLEU and ROUGE scores to evaluate the obtained English translations (model hyperparameter details in App.D). Table 4 shows the results obtained for the proposed architecture. Poor BLEU-4 result highlights the challenging nature of the ISL translation task. The results motivate incorporating ISL linguistic priors into data-driven models to develop better sign language translation systems.

heading: 5 Conclusion
text: We propose ISLTranslate, a dataset of 31k ISLEnglish pairs for ISL. We provide a detailed insight into the proposed dataset and benchmark them using a sign language transformer-based ISL-poseto-English architecture. Our experiments highlight the poor performance of the baseline model, pointing towards a significant scope for improvement for end-to-end Indian sign language translation systems. We hope that ISLTranslate will create excitement in the sign language research community and have a significant societal impact.
7https://ai.googleblog.com/2020/12/ mediapipe-holistic-simultaneous-face.html
Limitations
This resource paper proposes a new dataset and experiments with a baseline model only. We do not focus on creating new models and architectures. In the future, we plan to create models that perform much better on the ISLTranslate dataset. Moreover, the dataset has only 31K video-sentence pairs, and we plan to extend this to enable more reliable data-driven model development. In the future, we would also like to incorporate ISL linguistic knowledge in data-driven models.
Ethical Concerns
We create a dataset from publicly available resources without violating copyright. We are not aware of any ethical concerns regarding our dataset. Moreover, the dataset involves people of Indian origin and is created mainly for Indian Sign Language translation. The annotator involved in the dataset validation is a hard-of-hearing person and an ISL instructor, and they performed the validation voluntarily.

heading: Acknowledgements
text: We want to thank anonymous reviewers for their insightful comments. We want to thank Dr. Andesha Mangla (https://islrtc.nic.in/ dr-andesha-mangla) for helping in translating and validating a subset of the ISLTranslate dataset.

heading: A Data Splits
text: Data splits for ISLTranslate are shown in Table 5.
B ISLTranslate Word Distribution

heading: C Annotation Details
text: We asked a certified ISL instructor to translate and validate a random subset from the dataset. The instructor is a hard-of-hearing person and uses ISL for communication; hence they are aware of the subtitles of ISL. Moreover, the instructor is an assistant professor of sign language linguistics. The instructor is employed with ISLRTC, the organization involved in creating the sign language content; however, the instructor did not participate in videos in ISLTranslate. The instructor performed the validation voluntarily. It took the instructor about 3 hours to validate 100 sentences. They generated the English translations by looking at the video.

heading: D Hyperparameters and Training
text: We follow the code base of SLT (Camgoz et al., 2020) to train and develop the proposed SLT-based pose-to-text architecture by modifying the input features to be sign-pose sequences generated by the
mediapipe. The model architecture is a transformerbased encoder-decoder consisting of 3 transformer layers each for both encoder and decoder. We use the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 0.0001, β = (0.9,0.999) and weight decay of 0.0001 for training the proposed baseline with a batch size of 32.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations section after the Conclusion

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statement Section

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Introduction Section
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 and 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: We provide the details in Github page for the paper

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We provide the details in Github page for the paper
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3 and Appendix
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3 and Appendix
C 3 Did you run computational experiments? Section 3 and 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 3 and 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3 and 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3 and 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3 and 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3 and Appendix

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section 3 and Appendix

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 3 and Appendix

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section 3
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In the future, we plan to create models that perform much better on the ISLTranslate dataset. Moreover, the dataset has only 31K video-sentence pairs, and we plan to extend this to enable more reliable data-driven model development. In the future, we would also like to incorporate ISL linguistic knowledge in data-driven models. Ethical Concerns
We create a dataset from publicly available resources without violating copyright. We are not aware of any ethical concerns regarding our dataset. Moreover, the dataset involves people of Indian origin and is created mainly for Indian Sign Language translation. The annotator involved in the dataset validation is a hard-of-hearing person and an ISL instructor, and they performed the validation voluntarily