FBK@IWSLT Test Suites Task: Gender Bias evaluation with MuST-SHE (2024)

Abstract: This paper presents the FBK contribution to the IWSLT-2024 “Test suites” shared subtask, part of the Offline Speech Translation Task. Our contribution consists of the MuSTSHE benchmark evaluation, designed to assess gender bias in speech translation. By focusing on the en-de language pair, we rely on a newly created test suite to investigate systems’ ability to correctly translate feminine and masculine gender. Our results indicate that – under realistic conditions – current ST systems achieve reasonable and comparable performance in correctly translating both feminine and masculine forms when contextual gender information is available. For ambiguous references to the speaker, however, we attest a consistent preference towards masculine gender, thus calling for future endeavours on the topic. Towards this goal we make MuST-SHE freely available at: https://mt.fbk.eu/must-she/

heading: 1 Introduction
text: In today’s interconnected world, speech translation technology stands as a cornerstone of global communication, facilitating seamless interactions across linguistic barriers. Indeed, the last few years have seen notable advancements for the task of speech-to-text translation (ST), which has made strides in generic performance (Bentivogli et al., 2021; Anastasopoulos et al., 2021, 2022; Agarwal et al., 2023). Also, the emergence massively multilingual solutions has greatly expanded the language coverage of competitive “one-model-fits-all” speech models (Radford et al., 2022; Communication et al., 2023; Peng et al., 2024; Pratap et al., 2024).
Amid such advancements, there arise the increasing need to pair traditional overall quality assessments of ST with more fine-grained analyses by accounting for relevant aspects of translations. It is within this context that the IWSLT Test Suites
shared task emerges, aiming to provide a dedicated evaluation framework for specific dimensions of the ST output, which are otherwise overlooked with generic test sets and holistic metrics.
In light of the above, our contribution is dedicated to the critical themes of gender bias in automatic translation (Costa-jussà, 2019; Savoldi, 2023; Vanmassenhove, 2024).1 Given the largescale deployment of ST, biased translations are not only relevant from a technical perspective, where gender-related errors negatively impact the accuracy of automatic translation. Rather, biased and non-inclusive systems can pose the concrete risk of under/misrepresenting gender minorities by overproducing masculine forms and reinforcing gendered stereotypes (Blodgett et al., 2020; Sun et al., 2019). Indeed, gendered linguistic expressions affect the representation and perception of individuals (Stahlberg et al., 2007; Corbett, 2013; Gygax et al., 2019), and are actively used as a tool to negotiate the social, personal, and political reality of gender (Hellinger and Motschenbacher, 2015). A such, models that systematically favor masculine over feminine forms fail to properly recognize women, can reduce feminine visibility, and offer an unequal service quality (Crawford, 2017).
This paper presents the FBK participation in the Test Suites shared task by conducting evaluations on the MuST-SHEIWSLT24 en-de dataset. It represents the newly created speech-to-text extension of the English→German textual-only portion of MuST-SHE (Savoldi et al., 2023), a multilingual gender bias benchmark (Bentivogli et al., 2020).
In the hereby presented evaluations, we obtained translations of our test suites by systems that are part of the Offline Speech Translation Task of the 21st International Conference on Spoken Language
1Its relevance is also attested by the creation of dedicated workshops on theme of gender bias and inclusivity, such as GeBNLP (Hardmeier et al., 2022) and GITT (Vanmassenhove et al., 2023).
113
Translation (IWSLT 2024). Specifically, we evaluated 13 systems for MuST-SHEIWSLT24 en-de.
2 MuST-SHEIWSLT24
MuST-SHEIWSLT24 is a test suite designed to evaluate the ability of ST systems to correctly translate gender. It is composed of 200 segments that require the translation of – at least – one English gender-neutral word into the corresponding masculine or feminine target word(s) in German.2 The test suite is created as an extension of MuST-SHE, a multilingual, natural benchmark built on TED talks data (Bentivogli et al., 2020). The original corpus comprises ∼3,000 (audio, transcript, translation) triplets annotated with qualitatively differentiated gender-related phenomena for thee language pairs: English→ French/Italian/Spanish. Recently, MuST-SHE was also extended to English→ German for the MT task – i.e. MuST-SHEWMT23 (Savoldi et al., 2023). However, since it only consists of a textual portion (transcript, translation), it does not allow for the evaluation of ST models.
Here, we introduce the expansion of MuST-SHE English→ German for the ST task, by incorporating the additional speech input portion so as to obtain (audio, transcript, translation) triplets.

heading: 2.1 Audio Portion Creation
text: To ensure conformity, the dataset audio portion was obtained by following the same automatic procedures used for MuST-SHE and other TED-based
2See §5 for a discussion on the use of (binary) gender as a variable.
resources, as reported in (Cattoni et al., 2021). Accordingly, from the official TED website we downloaded the videos of the talks included in the textual portion of MuST-SHE English→ German. On this basis, i) audio tracks were extracted from the videos, and ii) an alignment procedure was applied to split talks into segments and generate aligned (audio, transcript, translation) triplets. Since this automatic procedure generates 90% of properly aligned triples on average (Cattoni et al., 2021), we performed qualitative checks. Two evaluators – both students proficient in the German language and with a background in Applied Linguistics3 – reviewed all the extracted audios and corrected any audio-text misalignment.4 Hence, we ensured the quality of all audio segments included in MuSTSHEIWSLT24, and the exact alignment of each (audio, transcript, translation) triplet.

heading: 2.2 Dataset Features
text: MuST-SHE is designed to evaluate the translation of a source English neutral word into its corresponding target gender-marked one(s) in the context of human referents, e.g. en: the good friend, de: der/die gute Freund/in. To allow for finegrained analyses, each segment in MuST-SHE is enriched with the following annotations: · GENDER, which allows to distinguish results for Feminine (Fem) and Masculine (Masc) forms, thus revealing a potential gender gap. · CATEGORY, which differentiates between CAT1
3Their work was carried out during an internship at FBK. 4We relied on the ELAN annotation tool: https://
archive.mpi.nl/tla/elan.
– first-person references to be translated according to the speakers’ linguistic expression of gender5 (e.g. I am a teacher) – and CAT2 – references to any participant, to be translated in agreement with gender information available in the sentence (e.g. He/she is a teacher). These categories allow analysing models’ behaviour across unambiguous and ambiguous gender translation instances.6
· GENDER-SWAPPED WORDS, providing, for each target gender-marked word annotated in MuSTSHE reference translations, a corresponding wrong form swapped in the opposite gender (e.g. en: she is a friend; de: Sie ist eine<ein> Freundin<Freund>). As described in §3.2, such pairs of annotated target gender-marked words are a key feature of MuST-SHE, which enables genderfocused evaluations.
All above-mentioned dimensions are already provided with the textual portion of MuST-SHE English→ German, and are consequently also included in MuST-SHEIWSLT24. In Table 1, we show examples of annotated (transcript, translation) segments from the corpus. Overall dataset statistics are provided in Table 2.

heading: 3 Experimental Settings

heading: 3.1 Models
text: The test suite evaluation is carried out on the systems that were submitted to the IWSLT Offline Speech Translation tasks. Overall, four different participants – i.e. HW-TSC, CMU, NYA, and KIT – submitted a total of 13 models. Of those, six models were presented as primary system submission, while the other 7 models are additional, contrastive models. All systems contributions are built upon
5Speaker’s gender information is provided for each segment. Note that gender has been labeled based on the personal pronouns the speakers used to describe themselves in their publicly available personal TED section.
6For direct ST solutions that directly translate from the audio input without intermediate textual representations, CAT1 can also reveal whether such models leverage speakers’ voice as an unwanted cue to translate gender. See Gaido et al. (2020).
cascade architectures, which resolve the ST task as pipelined ASR+MT solutions.
Since the participants (with the only exception of NYA) segmented the sentences before generating the outputs, we isolated the predicted translation for each reference sentence by means of the mWERSegmenter tool (Matusov et al., 2005). This procedure mirrors what is done in the standard evaluation of the offline task (Agarwal et al., 2023).

heading: 3.2 Evaluation
text: Following the original MuST-SHE evaluation protocol described in Gaido et al. (2020), MuSTSHEIWSLT24 evaluation allows to focus on the gender realization of the target gender-marked forms, which are annotated in the reference translations together with their wrong, gender-swapped form (see Table 1). The evaluation is carried out in two steps, and by matching the annotated (correct/wrong) gender-marked words against the ST output. Accordingly, we first calculate the Term Coverage as the proportion of gender-marked words annotated in the MuST-SHE references (either in the correct or wrong form) that are actually generated by the system, on which the accuracy of gender realization is therefore measurable. Then, we define Gender Accuracy as the proportion of correct gender realizations among the words on which it is measurable. This evaluation method7 has several advantages. On one side, term coverage unveils the precise amount of words on which systems’ gender realization is measurable. On the other, gender accuracy directly informs about systems’ performance on gender translation and related gender bias: scores below 50% indicate that the system produces the wrong gender more often than the correct one, thus signalling a particularly strong biased behaviour.

heading: 4 Results
text: In Table 3 we present the MuST-SHEIWSLT24 results of the 13 IWSLT Offline ST cascade models. Starting from coverage scores (All-Cov), all models achieve overall positive results, which range from ∼70% (HW-TSC_CONSTRAINED-wLLM.primary) to 74.79% (HW-TSC_CONSTRAINED.primary). Hence, these models produce a good amount of
7The evaluation script is publicly available at: https://github.com/hlt-mt/FBK-fairseq/blob/ master/examples/speech_to_text/scripts/gender/ mustshe_gender_accuracy.py.
gender-marked words that can be evaluated with regards to the accuracy of their gender realization.
Moving onto the overall accuracy scores (AllAcc), we can see that – while there is still room for improvement – all of the evaluated ST systems achieve reasonable results, by being able to correctly translate gender with an accuracy of at least 77.42% (KIT.contrastive1) up to 84.44% for HW-TSC_CONSTRAINED.primary. Similar accuracy ranges are attested also by disaggregating results across feminine (F-ACC) and masculine (M-Acc) genders. Interestingly, such results show that none of the models exhibit perfectly equal performance across both genders. Still, the divide is fairly limited, with i) a comparable number of ST systems achieving slightly higher results on either the feminine or masculine set of MuST-SHE, and ii) little variation in scores across the 13 models, as attested in terms of standard deviation. If we go
more fine-grained into disaggregated results, however, we unveil a higher degree of variation.
In Figure 1, we report results across categories for masculine (1M and 2M) and feminine gender realizations (1F and 2F). On the one hand, for unambiguous gender translation from CAT2, systems are slightly better in performing feminine gender translation. Instead, results on CAT1 unveil a wide gender gap, where feminine accuracy is consistently lower compared to its masculine counterpart. In fact, most models tend to generate the correct feminine form in less than 50% of the cases, namely below random chance. The ST model HW-TSC_CONSTRAINED-wLLM.primary, which overall emerges as the best system for gender translation, still remains at 68.18%.
To conclude, our results show that – when confronted with ambiguous source sentences – current ST models tend to favour the generation of mas-
culine forms in the German target language. We acknowledge that the phenomena subject to our analysis (gender bias) are not currently accounted for in the design of ST systems, which are rather designed with the goal of optimizing overall translation quality. Towards the creation of fairer ST technology, however, we hope that our evaluation will raise awareness in the community, and encourage the development of capable models, which can equally accommodate feminine and masculine language.

heading: 5 Conclusion
text: This paper summarizes the results of our IWSLT2024 Test Suites evaluation, which focused on gender bias in translation. To this aim, we have introduced the speech expansion of the en-de MuST-SHE test set. Overall, results on MuSTSHEIWSLT24 show that the evaluated ST systems are reasonably good at translating gender under realistic conditions, achieving comparable results across feminine and masculine gender translation. Also, all models are quite robust, and show a similar behaviour for translation of unambiguous gender phenomena, where they can rely on contextual gender information. However, for ambiguous cases where the input sentence does not inform about the gender form to be used in translation, we confirm a strong skew where all systems favour masculine generation almost by default. This finding calls for further research endeavours and evaluation initiatives to counter gender bias in ST and measure future advances.
Limitations
The main limitation of this work concerns the limited size of data points (i.e. gender-marked words) available for evaluation. As such, even in the case of gender performance parity, the dataset does not allow to make conclusive statements about the absence of bias in the assessed models. Despite its restricted size, however, MuST-SHEIWSLT24 provides a first glimpse into understanding and monitoring en-de systems’ behaviour with respect to gender bias and translation.
Ethics Statement
The use of gender as a variable in this paper warrants some reflections. Namely, when working on the evaluation of speaker-related gender translation for MuST-SHE (i.e. Category 1) we solely focus
on the rendering of their reported linguistic gender expressions. No assumptions about speakers’ self determined identity (GLAAD, 2007) – which cannot be directly mapped from pronoun usage (Cao and Daumé III, 2020; Ackerman, 2019) – has been made.
Also, in our diagnosis of gender bias we only account for feminine and masculine linguistic forms, which are those traditionally in use and the only represented in the used data. However, we stress that – by working on binary forms – we do not imply or impose a binary vision on the extra-linguistic reality of gender, which is rather a spectrum (D’Ignazio and Klein, 2020). Also, we acknowledge the current challenges faced for grammatical gender languages like German in fully implementing neutral language (Paolucci et al., 2023), and support the rise of both non-binary language (Shroy, 2016; Gabriel et al., 2018; Conrod, 2020) and translation technologies (Lauscher et al., 2023; Gromann et al., 2023).

heading: Acknowledgements
text: The work presented in this paper is funded by the European Union’s Horizon research and innovation programme under grant agreement No 101135798, project Meetween (My Personal AI Mediator for Virtual MEETtings BetWEEN People) and the PNRR project FAIR - Future AI Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU. Also, we would like to thank the 2022 FBK internship students Sabrina Raus and Abess Benissmail from the University of Bolzano: the creation of MuSTSHEIWSLT24 was made possible by their work.

heading: Future Work
text: The work presented in this paper is funded by the European Union’s Horizon research and innovation programme under grant agreement No 101135798, project Meetween (My Personal AI Mediator for Virtual MEETtings BetWEEN People) and the PNRR project FAIR - Future AI Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU. Also, we would like to thank the 2022 FBK internship students Sabrina Raus and Abess Benissmail from the University of Bolzano: the creation of MuSTSHEIWSLT24 was made possible by their work