Overview of the Shared Task on Machine Translation Gender Bias Evaluation with Multilingual Holistic Bias (2024)

Abstract: We describe the details of the Shared Task of the 5th ACL Workshop on Gender Bias in Natural Language Processing (GeBNLP 2024). The task uses Multilingual HolisticBias dataset to investigate the quality of Machine Translation systems on a particular case of gender robustness. We report baseline results as well as the results of the first participants. The shared task will be permanently available in the Dynabench platform.

heading: 1 Introduction
text: Gender bias poses challenges across various aspects of automatic translation. These challenges include preserving correct pronouns, understanding the correct gendered context, and relating adjectives and professions to the proper gender. The issue becomes even more complex when considering multilingual translation, especially for lowresource languages. The GeBNLP 2024 workshop aims to raise awareness of these challenges by introducing a dedicated shared task for investigating translation quality using the Multilingual HolisticBias dataset (Costa-jussà et al., 2023). This initiative seeks to foster a community-driven effort and long-term solutions toward improving gender representation in machine translation. We encourage researchers to contribute their expertise, not just for the workshop but for the ongoing pursuit of advancements in this field.

heading: 2 Motivation
text: The development of gender (Stanovsky et al., 2019; Renduchintala et al., 2021; Levy et al., 2021; Costajussà et al., 2022; Renduchintala and Williams, 2022; Savoldi et al., 2021; Alhafni et al., 2022; Attanasio et al., 2023) or demographic-specific (Costa-jussà et al., 2023) datasets has raised the interest in evaluating Natural Language Processing (NLP) models beyond standard quality terms.
In Machine Translation (MT), gender bias is observed when translations show errors in linguistic gender determination despite the fact that there are sufficient gender clues in the source content for a system to infer the correct gendered forms. To illustrate this phenomenon, sentence (1) in Table 1 does not contain enough linguistic clues for a translation system to decide which gendered form should be used when translating into a language where the word for doctor is gendered. Sentence (2) in Table 1, however, includes a gendered pronoun which most likely has the word doctor as its antecedent. Sentence (3) in Table 1 shows two variations of the exact sentence differing only in the gender inflection.
(1) I didn’t feel well, so I made an appointment with my doctor.
(2) My doctor is very attentive to her patients’ needs.
(3) Mi amiga es una ama de casa. Mi amigo es un amo de casa. [English: My friend is a homemaker.]
Table 1: Gender phenomenons’ examples
Gender bias is observed when an MT system produces the wrong gendered form when translating sentence (2) into a language that uses distinct gendered forms for the word doctor. On the contrary, a single error in the translation of an utterance such as sentence (1) would not be sufficient to conclude that gender bias exists in the model; doing so would take consistently observing one linguistic gender over another. Finally, a lack of robustness would be shown if the translation quality differed for the two sentences in (3). It has previously been hypothesized that one possible source of gender bias in MT is gender representation imbalance in large training and evaluation data sets, e.g., Costa-jussà
399
et al. (2022); Qian et al. (2022). Our task goes beyond previous gender bias MT evaluation efforts, such as Stanovsky et al. (2019); Renduchintala et al. (2021); Levy et al. (2021); Costa-jussà et al. (2022); Renduchintala and Williams (2022); Savoldi et al. (2021); Alhafni et al. (2022); Attanasio et al. (2023), to name a few, mainly by increasing the number of languages and fairly comparing three main gender MT issues which are gender-specific, gender robustness, and unambiguous gender (see Section 4).

heading: 3 Goals
text: The goals of the Multilingual HolisticBias task as part of the 5th ACL Workshop on Gender Bias in Natural Language Processing are:
• To investigate the quality of MT systems on a particular case of gender preservation for tens of languages.
• To examine and understand special gender challenges in translating in different language families.
• To investigate the performance of gender translation of low-resource, morphologically rich languages.
• To open to the community the first challenge of this kind.
• To generate up-to-date performance numbers in order to provide a basis for comparison in future research.
• To investigate the usefulness of multilingual and language resources.
• To encourage beginners and established research groups to participate and interchange discussions.

heading: 4 Multilingual HolisticBias Task
text: We propose to evaluate three cases of gender bias: gender-specific, gender robustness, and unambiguous gender translation.

heading: 4.1 Task 1: Gender-specific
text: In the English-to-X translation direction, we evaluate the capacity of MT systems to generate genderspecific translations from English neutral inputs (e.g., I didn’t feel well, so I made an appointment with my doctor). This can be illustrated by the fact that MT models systematically translate neutral source sentences into masculine or feminine
depending on the stereotypical usage of the word (e.g., homemakers into amas de casa, which is the feminine form in Spanish and doctors into médicos, which is the masculine form in Spanish).

heading: 4.2 Task 2: Gender Robustness
text: In the X-to-English translation direction, we compare the robustness of the model when the source input only differs in gender (masculine or feminine), e.g., Spanish Mi amiga es una ama de casa and Mi amigo es un amo de casa (My friend is a homemaker).

heading: 4.3 Task 3: Unambiguous Gender
text: In the X-to-X translation direction, we evaluate the unambiguous gender translation across languages and without being English-centric, e.g, Spanish-toCatalan: Mi amiga es una ama de casa is translated into La meva amiga és una mestressa de casa.

heading: 4.4 Submission details
text: Data This task is based on Multilingual HolisticBias (Costa-jussà et al., 2023) – the first multilingual extension of HolisticBias (Smith et al., 2022) which covers tens of languages.
X Languages In addition to English, our challenge covers 26 languages: Modern Standard Arabic, Belarusian, Bulgarian, Catalan, Czech, Danish, German, French, Italian, Lithuanian, Standard Latvian, Marathi, Dutch, Portuguese, Romanian, Russian, Slovak, Slovenian, Spanish, Swedish, Tamil, Thai, Ukrainian, Urdu
Evaluation The challenge is evaluated using automatic metrics: BLASER (Chen et al., 2022) and ChrF (Popović, 2015). Evaluation criteria are in terms of overall translation quality and difference in performance for masculine (m) and feminine (f) sets. Leaderboard ranking will be made using the following combination of BLASER and ChrF:
GES = 20× average(BLASERm,BLASERf ) 1 + |ChrFm − ChrFf |
where BLASERm/f and ChrFm/f use masculine or feminine references. The metric is a percentage and it should be maximized. The numerator evaluates the semantic quality, and the denominator evaluates the difference in ChrF between using masculine or feminine references. We call this metric Gender Equity Score (GES).
Submission platform We use the Dynabench platform for all tasks.
Baseline systems We use open-source NLLB models: NLLB-600M and NLLB-3.3B (NLLB Team et al., 2022).
Participants This edition of the shared task received only one submission. The participants expanded the DAMA framework (Debiasing Algorithm through Model Adaptation, Limisiewicz et al. (2024)) to be applicable in the multilingual translation task. DAMA proposes a method for identifying and mitigating gender bias in language models. In the original paper, the researchers discovered that specific layers of LLaMA (Touvron et al., 2023) are responsible for gender bias and intervened on these layers by modifying their weights to nullify their effect. The shared task participants replicated the same intervention on ALMA-R (Xu et al., 2024), an MT-specific LLM that performs better than previous LLMs, including GPT-3.5. The findings showed that DAMA could reduce gender bias in translation without compromising quality in the overall domain. However, the suggested approach is susceptible to the introduction of bias in the prompts.

heading: 5 Results
text: This section reports results for the two baselines and the submitted system. We provide results only for Task 2 (gender robustness), which received the submission.
Table 2 shows the results. We can notice that NLLB-3.3B performs better in terms of translation quality and GES. Note that in this case, higher GES shows that there is less translation quality variation when gender varies in the input. We observe that the difference across models differs across languages, with larger discrepancies in languages like Arabic or Thai and smaller in languages like German or Spanish. For a few languages, e.g., Catalan or Romanian, GES is higher for NLLB-600M. On average, NLLB-3.3B scores higher in GES by more than 0.5. The result is coherent with previous research that shows that by just increasing the translation quality of the model, gender robustness increases (Communication et al., 2023).
Finally, Table 3 shows the participant entry compared to the best baseline. We observe that the strongest baseline surpasses DAMA models in terms of translation quality (absolute ChrF or
BLASER) and GES.

heading: 6 Final Remarks
text: This paper introduces the Multilingual HolisticBias Dynabench task1 which has been launched in the context of the 5th ACL Worskhop on Gender Bias in NLP2. This task will remain open for participation. At the moment of the preparation of this paper, we have received a single participation which evaluates the mitigation strategy of DAMA (Limisiewicz et al., 2024) for gender robustness. We are also reporting strong baseline results with NLLB models for this particular task. However, we do not include baselines for gender-specification and unambiguous gender, which is left as further work.
We are looking forward to receiving more submissions in the near future. Also notice that an extension of the Multilingual HolisticBias dataset is currently going on and released (Tan et al., 2024).
Limitations
Our shared task shares the same limitations as the Multilingual HolisticBias dataset on which it is based (Costa-jussà et al., 2023).

heading: Future Work
text: We are looking forward to receiving more submissions in the near future. Also notice that an extension of the Multilingual HolisticBias dataset is currently going on and released (Tan et al. , 2024). Limitations
Our shared task shares the same limitations as the Multilingual HolisticBias dataset on which it is based (Costa-jussà et al. , 2023)