Constructing a Multimodal, Multilingual Translation and Interpreting Corpus: A Modular Pipeline and an Evaluation of ASR for Verbatim Transcription
Authors: Alice Fedotova, Adriano Ferraresi, Maja Miličević Petrović, Alberto Barrón-Cedeño (Università di Bologna, 2024)
Topic: Machine Translation

Abstract
This paper presents a novel pipeline for constructing multimodal and multilingual parallel corpora, with a focus on evaluating state-of-the-art automatic speech recognition tools for verbatim transcription. The pipeline was developed during the process of updating the European Parliament Translation and Interpreting Corpus (EPTIC), leveraging recent NLP advancements to automate challenging tasks like multilingual alignment and speech recognition. Our findings indicate that current technologies can streamline corpus construction, with fine-tuning showing promising results in terms of transcription quality compared to out-of-the-box Whisper models. The lowest overall WER achieved for English was 0.180, using a fine-tuned Whisper-small model. As for Italian, the lowest WER (0.152) was obtained by the Whisper Large-v2 model, with the fine-tuned Whisper-small model still outperforming the baseline (0.201 vs. 0.219).

Introduction
The present paper introduces a pipeline for the construction of multimodal and multilingual parallel corpora that could be used for translation and interpreting studies (TIS), among others. The construction of such resources has been acknowledged as a ``formidable task'' AUTHOR, which if automated ---as we propose--- involves a number of subtasks such as automatic speech recognition (ASR), multilingual sentence alignment, and forced alignment, each of which poses its own challenges. Yet tackling these subtasks also offers a unique way to evaluate state-of-the-art natural language processing (NLP) tools against a unique, multilingual benchmark. In this paper we discuss the development of a modular pipeline adaptable for each of these subtasks and address the issue of whether performing ASR with OpenAI’s Whisper AUTHOR could be suitable for verbatim transcription. We showcase the utility of this pipeline by expanding the European Parliament Translation and Interpreting Corpus (EPTIC), a multimodal parallel corpus comprising speeches delivered at the European Parliament along with their official interpretations and translations AUTHOR. The transcription conventions adopted for the compilation of EPTIC were developed ad hoc and aim at reproducing minimal prosodic features, but can still be considered an instance of verbatim transcription AUTHOR; the issue of what truly constitutes verbatimness is still an object of debate and will be further discussed. There is fairly widespread agreement on the statement that every transcription system reflects a certain methodological approach AUTHOR, and that by ``choosing not to transcribe a particular dimension, the researcher has implicitly decided that the dimension plays no role in the phenomenon in question'' AUTHOR. To investigate the characteristics of Whisper's AUTHOR transcriptions in English and Italian, we formulate the following two research questions: RQ1~Is it possible to use fine-tuning to adapt the transcription style to the one of an expert annotator? RQ2~What is the impact of speech type (native, non-native, interpreted) on transcription quality? We find that satisfactory results can be achieved with automatic speech recognition, although challenges remain, especially with regards to the verbatimness of the transcription ---a crucial factor in corpora intended for TIS\@. Fine-tuning Whisper-small on English data obtains a lower word error rate (WER) of 0.180 compared to Whisper-large v2 (0.194), potentially indicating that fine-tuning Whisper models holds promise for improving their performance in terms of adhering to a certain transcription style. However, this was not the case when considering the experiments based on Italian. In the Italian scenario, Whisper-large-v2 obtained a WER of 0.152 compared to a WER of 0.201 obtained by the fine-tuned Whisper-small model. It should be noted, however, that this constituted an improvement over the baseline Whisper-small model, which obtained a higher WER of 0.219. A significant limitation in the case of fine-tuning in Italian was constituted by the smaller amount of data available for tuning compared to English. Lastly, we find that sentence alignment can be facilitated through state-of-the-art embedding-based tools, whereas forced alignment can be considered a largely solved problem. This makes the construction of corpora such as EPTIC more streamlined and requiring less human intervention, with wider implications for multilingual corpus construction in the field of TIS and beyond.

Conclusion
This paper presented a novel pipeline for constructing multimodal and multilingual parallel corpora, with a focus on evaluating state-of-the-art automatic speech recognition tools for verbatim transcription. Experiments with Whisper models on EPTIC revealed robust performance across languages and speech types, particularly for English and Italian. However, some limitations remain regarding ASR performance and achieving verbatim transcriptions. Fine-tuning Whisper showed promising reductions in WER, particularly for English, indicating the potential of adapting the model to use a more verbatim style. Yet qualitative analysis revealed inconsistencies in handling disfluencies, truncations, and discourse markers. Furthermore, higher WERs for non-native and interpreted speech underscore remaining challenges. Future research efforts could explore incorporating additional metrics beyond WER to better capture the degree of verbatimness in the transcriptions, and expanding the Italian dataset to potentially improve the performance of the fine-tuned model. Another avenue for research could include augmenting the dataset with external data containing pairs of audio and verbatim transcripts, most notably the Switchboard corpus introduced in AUTHOR. Other methods besides fine-tuning could be explored to enhance the quality of transcriptions, for instance by leveraging the official verbatim reports on the European Parliament’s website. Lastly, a model could be developed for detecting the metadata item relative to the speech type, i.e.\ impromptu, read out, or mixed, based on textual or multimodal features. The work of A. Fedotova is supported by the NextGeneration EU programme, ALMArie CURIE 2021 - Linea SUpER, Ref. CUPJ45F21001470005. \appendix