Introducing Deep Learning with Data Augmentation and Corpus Construction for LIS
Authors: Manuela Marchisio, Alessandro Mazzei, Dario Sammaruga (Università di Torino, Orbyta Tech S.r.l., 2023)
Topic: Multimodal

Abstract
The development of home video recording has had a big impact in the development of video documents containing Italian Sign Language (LIS) sentences. LIS2SPEECH is an ongoing project by Orbyta Tech s.r.l. to build a complete translation chain from LIS to speech. The idea is to build a free software framework to transform video containing LIS sentences into Italian vocal sentences. In this way, LIS signers can indirectly produce Italian vocal sentences. In this paper, we describe two milestones for LIS2SPEECH, which are: i. the development of some deep neural models trained by using data augmentation techniques, and ii. the construction of a new dataset for LIS to Italian. Referring to the first point, a number of deep learning models were developed and tested. Then, data augmentation was performed by using some geometric transformations on the videos belonging to the original training set. With reference to the second point, we constructed the TGLIS-227 dataset by using video and audio segmentation techniques, starting from a corpus of RAI newscasts. This dataset is a novelty in the current research panorama as there are no public datasets for LIS with sentence-level granularity.

Introduction
In this paper, we present the architecture of a real-time translation system from Italian Sign Language (henceforth LIS) to Italian speech. An ideal system platform for this task should be composed of three main modules: - a first module which, starting from an input video, returns the glosses contained in the video. So, this module performs a Sign Language Recognition Task. - a second module that translates the glosses into the Italian language. So, this module performs a Sign Language Translation task. - a third module for the text-to-speech system, that is for pronouncing the sentences in Italian. So, this module performs a Text to Speech task. The task of Sign Language Recognition (SLR) is a classification task that allows to automatically obtain the glosses corresponding to the signs performed by a signer in a video. In general, the SLR is approached as a multiclass classification problem, i.e. a type of supervised learning which, on the basis of a (statistical) model, can associate the correct gloss among the k, where k is the cardinality of the LIS dictionary considered. In the specific settings of the LIS2SPEECH project, the input is a sequence of signs encoded in a video while the labels are the corresponding glosses. So, this is a case of supervised learning, where the models are trained on a dataset containing numerous examples (signs) labeled with the relative class (gloss). Considering the module for translation from LIS to Italian, this implements a task where it is true that the rule "more data is better data". In this paper, we follow this prescription in two distinct directions. In Section 2, we describe a number of experiments with deep neural models trained by using a data augmentation technique. We enlarge the possibility given by a relatively small initial dataset by using a number of geometrical transformations to the original videos. We describe these transformations and experiment their impact on the performances of the Isolated Sign Language Recognition (ISLR) task, i.e. when each video contains a single sign. Moreover, in Section 3, we describe the initial steps toward the release of a new dataset for LIS in the news domain. We provide a description of the algorithmic process used to provide a sentence-level segmentation of the original videos. In Section 4, we summarize the contributions of this paper and provide a brief description of the ongoing work.

Conclusion
In this work, we have presented two main results obtained in the LIS2SPEECH project. First, we have described the application of a number of data augmentation techniques to some deep neural models in the task of ISLR. We proved with experiments that some of these transformations have a strong impact on the final performance of the classification task. Second, we built the TGLIS-227, a new sentence-level dataset for LIS, applying a new procedure for the automatic segmentation of the newscasts. In future work, we intend to develop the following two ideas: 1. to annotate TGLIS-227 video with the glosses that they contain; 2. to develop a system like Common Voice [17] to collect more data to build an open-source dataset for LIS; Moreover, a more challenging development could be to encode additional two video features that are: the lips and facial expressions. Finally, we noted that very often in the news the signers "read" the gloss by using their lips and, moreover, express an emotion related to the gloss by using their facial expression.