An experiment in error analysis of real-time speech machine translation using the example of the European Parliament’s Innovation Partnership
Authors: Elisa Di Nuovo (Università di Torino, 2023)
Topic: Multimodal

Abstract
In recent years, technological progress has made Machine Translation (MT) a reality. Significant improvements have been obtained using deep learning models as opposed to rule-based and statistical MT models. Human evaluation still remains under-explored. In 2019, the European Parliament (EP) started an innovation partnership with commercial operators, with the purpose of developing a tool exploiting state-of-the-art, real-time Automatic Speech Recognition (ASR) and MT technologies to make parliamentary plenary sessions accessible to D/deaf and hard of hearing. In this paper, we present a quantitative and qualitative error analysis carried out on a test set consisting of 78 short speeches delivered by Members of the EP in 19 languages deployed in the EP prototype by November 2022. The taxonomy used for ASR and MT is adapted from the Multidimensional Quality Metrics framework. Results show that sentence segmentation is the biggest issue in the ASR output—not considered using automatic metrics—which often affects the MT output.

Introduction
In recent years, the landscape of language translation has been fundamentally transformed by remarkable technological advancements. Machine Translation (MT), once an ambitious aspiration, has now become a tangible reality. This transformation has been primarily fueled by the advent of deep learning models, specifically neural machine translation and transformers. These cutting-edge models have ushered in a new era of translation, eclipsing the limitations of traditional rule-based and statistical MT methods. Deep learning models use the mechanism called attention to improve the performance and have been usually evaluated on offline written translation tasks involving a few language pairs. Very recently, research expanded its focus also on speech machine translation, tackled as a concatenation of Automatic Speech Recognition (ASR) and MT, or as an end-to-end task (i.e. direct translation of speech in language A into text in language B). In the last evaluation campaign of the 19th International Conference on Spoken Language Translation (IWSLT 2022), one of the eight shared tasks focused on real-time speech translation, addressed as translation of ASR output or directly from the audio source and involving English to German, English to Japanese, and English to Mandarin Chinese. A novelty of this year’s campaign is the addition of manual evaluation of real-time outputs. Like many natural language processing tasks, MT is difficult to evaluate. One of the reasons for this is the nondeterministic nature of translation, i.e. there is more than one correct way to translate from one language into another. Evaluation in shared tasks is usually carried out by means of automatic metrics, BLEU (Bilingual Evaluation Understudy) being the standard for MT evaluation. This metric tries to overcome the nondeterministic nature of translation using multiple references. However, automatic metrics have several limitations. On the other hand, human evaluation, if carried out using fine-grained guidelines to limit subjectivity, can give a clearer indication of the MT output quality. However, being resource-expensive (i.e., it is hard to find skilled evaluators; skilled evaluators have a high cost), it has been used limitedly and in small studies. To avoid these limitations, crowdsourced annotators have been used. Unfortunately, crowdsourced annotators are frequently inexperienced. As affirm, crowdsourced human evaluation can be used when MT quality is poor, because it can still provide a useful indication; but, as quality improves, it becomes unfit and leads to erroneous claims. In 2019 the European Parliament (EP) started an Innovation Partnership with commercial operators, with the purpose of developing a tool that can perform real-time ASR and MT from and into all the 24 official languages of the European Union (EU). This partnership has the aim of making parliamentary plenary sessions accessible in near-real time to D/deaf and hard of hearing persons. The challenges faced by this project are manifold: the high degree of multilingualism which is highly ambitious considering the technical limits of current MT particularly in a number of low-resource languages, the presence of non-native accents, the large variety of vocabulary/EU jargon required by the numerous specific domains tackled in the plenaries, the low latency constraints to have transcriptions and translations in near-real time, and the required high quality of the output. By November 2022, 19 language models have been developed and made available via a demo interface. These languages are English (EN), French (FR), German (DE), Spanish (ES), Italian (IT), Polish (PL), Greek (EL), Romanian (RO), Dutch (NL), Portuguese (PT), Bulgarian (BG), Czech (CS), Slovak (SK), Croatian (HR), Lithuanian (LT), Finnish (FI), Hungarian (HU), Swedish (SV), and Slovenian (SL). In this paper, we present a quantitative and qualitative study—using Word Error Rate (WER) metric and manual human evaluation—on a test set consisting of short speeches delivered by members of the EP in the 19 languages already deployed in the prototype. The aim of this study is to evaluate the quality of both ASR and MT output and to reflect on the different insights of the same text given by different annotators. The manual human evaluation is based on an error taxonomy adapted from the Multidimensional Quality Metrics (MQM) framework and applied to part of the test set covering 6 languages (EN, FR, ES, IT, RO, DE). The MQM framework, developed in the EU QTLaunchPad and QT21 projects, provides a hierarchy of translation errors that can be adapted according to the application. We devised our taxonomy consisting of different error categories for ASR and MT and 3 severity levels (i.e., neutral, minor, and major). We decided to exclude critical errors as in [10]. The remainder of the paper is organized as follows: in Section 2 we describe the methodology applied for automatic and human evaluation; in Section 3 we report the results quantitatively and qualitatively analyzed per language and per annotator; Section 4 concludes the paper.

Conclusion
We presented a quantitative and qualitative evaluation of the tool developed in the context of the European Parliament’s (EP) Innovation Partnership. The evaluation focused on both Automatic Speech Recognition (ASR) and Machine Translation (MT). The quality of ASR was assessed using the Word Error Rate (WER) score, while MT quality was evaluated solely through human manual evaluation. The average WER across a multilingual test set consisting of 19 languages deployed by November 2022 was found to be 6.43%, a very low value. However, it is important to note that WER does not take segmentation issues into account. Human evaluation revealed that sentence segmentation, particularly in languages like Romanian (RO) and Italian (IT), requires refinement. This suggests that WER alone does not provide a comprehensive picture of transcription quality. Human evaluation, while essential, remains subjective. The evaluation process involves categorizing errors, including those related to sentence segmentation. The background of the annotators significantly affects how errors are perceived and identified, and this influence should be further explored. We also found, as noted in previous studies, that the depth of error annotation by different annotators was a key cause of disagreement, especially when they attempted to assess the consequences of the errors. The qualitative results of the human evaluation, focusing on ASR output and its translation into Italian (IT), showed that Spanish (ES) produced the best output, followed by Italian (IT), French (FR), and English (EN), while Romanian (RO) yielded the poorest results. In general, annotators rated ASR output more negatively than MT output. This may be due to annotators’ tendency to emphasize the provided reference transcription of the speech, especially regarding punctuation, which represents only one possible accepted version of the transcribed text. Qualitative findings further indicated that different annotators offered varying assessments of the same speeches, highlighting the necessity for a second round of annotations to reduce disagreement. This second round would help clarify the use of certain error categories, such as "unintelligible," which is often misapplied.