Did somebody say `Gest-IT'? A pilot exploration of multimodal data management
Authors: Ludovica Pannitto, Lorenzo Albanesi, Laura Marion, Federica Maria Martines, Carmelo Caruso, Claudia S. Bianchini, Francesca Masini, Caterina Mauri (Università di Bologna, University of Poitiers, 2024)
Topic: Multimodal

Abstract
The paper presents a pilot exploration of the construction, management and analysis of a multimodal corpus. Through a three-layer annotation that provides orthographic, prosodic, and gestural transcriptions, the Gest-IT resource allows to investigate the variation of gesture-making patterns in conversations between sighted people and people with visual impairment. After discussing the transcription methods and technical procedures employed in our study, we propose a unified CoNLL-U corpus and indicate our future steps.

Introduction
Corpora represent the main tool for linguists to observe language in its real use and verify its general trends on both a quantitative and qualitative basis AUTHOR. Today, written language corpora are the most used, thanks to the greater availability of written data and the ease of processing. However, in speech, speakers appeal to numerous semiotic sources (e.g., spoken channel, gestures, proxemics, facial expressions, etc.) to create and convey meaning, and written corpora fail to account for this richness of modalities. To effectively study how language works, one should observe these different semiotic sources independently of each other and take their interactions into account AUTHOR. To capture this complexity, it is necessary to go beyond written data and use multimodal corpora, namely collections of audio-visual linguistic data that allow to both hear and see linguistic productions. Multimodal corpora can be used to analyze a wide variety of linguistic phenomena, especially those related to the use of body in human communication and interaction, and to the way bodily communication and spoken language interact to generate meaning. They are, therefore, the primary sources for the analysis of co-speech gestures AUTHOR and Sign Languages. Following AUTHOR, we define a multimodal corpus as `an annotated collection of coordinated content on communication channels including speech, gaze, hand gesture and body language'. Multimodal corpora come in various shapes, depending on the nature of the communication channel captured by the resource. For the sake of this article, we specifically restrict our attention to resources including both a video and audio recording of linguistic content). For a collection of available multimodal resources see \url{https://www.clarin.eu/resource-families/ multimodal-corpora} (spoken language) and (sign language), while a list of audio-only resources of spoken language can be found at }. The long tradition of analyzing written and spoken data has, over time, led to the development of transcription systems (such as \textsc{IPA}, orthography, and various prosodic conventions), which have become recognized standards within the linguists' community. These systems allow for an ``objective'' description of speech, independently of any considerations on the functions or the meanings of the described elements. For gestural data, some transcription systems have also been proposed, such as the Linguistic Annotation System for Gestures (LASG; AUTHOR), but none of them has attained enough acceptance to qualify as a standard AUTHOR. In the absence of an effective transcription system, hybrid solutions are often employed, situated between transcription and annotation, where the choices for describing gestural forms reflect their attributed functions (e.g.: a ``shrug'' is a shoulder movement, but this label is often used to refer to any pragmatic gesture of epistemic denial performed by moving the shoulders, without specifying either the characteristics of the shoulder movement or the movements of other body parts that may have contributed to the execution of the gesture). Similar challenges arise in studies on Sign Languages. Although there is a larger number of transcription systems for the latter (see, for instance, the review in AUTHOR), none of them has achieved the status of a universal standard. Additionally, attempts to adapt these systems to the transcription of gestures have so far been limited and not particularly successful. The lack of a transcription standard for gestures, that describes them independently of their function or meaning, hinders the ability to precisely investigate the relationship between speech and gesture. Another aspect concerns the nature of language data captured in multimodal resources: as the collection and standardization process for this kind of linguistic data is, by its very nature, much more complex, resources are often tailored to specific purposes and therefore involve task-oriented interactions (e.g., describing objects as in the NM-MoCap-Corpus AUTHOR; spatial comunication tasks as in the SaGA Corpus AUTHOR), thus capturing interactions that may be naturalistic but are inherently non-ecological, i.e. not naturally-occurring AUTHOR. Often, participants are asked to wear special devices such as headsets or trackers during the recordings AUTHOR, clearly altering the spontaneity of the interaction. The aim of the Gest-IT project is to build a multimodal corpus of ecological data, allowing for the integrated analysis of verbal and gestural communication in spontaneous interactions. In this paper, we will focus on the protocol of multimodal data management that we tested for this resource. We will first discuss the main existing multimodal resources (Section~sec:related), showing how, as of today, there doesn't seem to be any ecological, accessible, multimodal corpus for Italian. We will then introduce the Gest-IT pilot resource and present its main features with respect to existing resources (Section~sec:gestit). Section~sec:schema outlines the main design choices taken for the creation of our resource and Section~sec:future describes the path ahead.

Conclusion
The aim of this paper is to share with the scientific community the protocol developed to build a multimodal resource for the Italian language in terms of data collection (design, ethic issues, practicalities); data management and curation; data transcription, annotation and analysis. In doing so, we contribute to the debate on multimodal resource building, which is still lacking an established standard. In particular, our contribution in this respect is twofold. Firstly, our study suggests to adopt a three-layer transcription where the three layers (i.e., the orthographic transcription, the prosodic/interactional transcription, and the gestural transcription) align to each other, by using ELAN as a tool for transcribing and CoNLL-X as an interoperable output format. This has the advantage of grounding gestures as an integrated semiotic source within verbal conversation and ultimately allows to unveil gesture-speech regularities. Secondly, we propose an innovative approach for the annotation of gesture data. By relying on common practices in the field of sign languages, we suggest that gesture transcription should follow the same rationale of phonetic transcription, with a method that describes `objective' aspects that characterize the `form' of the gesture, thus allowing for an interpretation-independent annotation. Clearly, the project is still at a very preliminary stage. Next steps will include the complete orthographic, prosodic and gesture transcription of the recordings; a thorough revision and pseudoanymization.