Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scale
Authors: Marta R. Costa-jussà
Pierre Andrews
Eric Michael Smith
Prangthip Hansanti
Christophe Ropers
Elahe Kalbassi
Cynthia Gao
Daniel Edward Licht
Carleigh Wood
Venue: EMNLP 2023 Main (2023)
Keywords: Multilingual holistic bias
Machine Translation
Sentence Embeddings
TL;DR: Multilingual HolisticBias dataset is intended to uncover demographic imbalances and be the tool to quantify mitigations towards them.

Abstract: We introduce a multilingual extension of the HolisticBias dataset, the largest English template-based taxonomy of textual people references: Multilingual HolisticBias. This extension consists of 20,459 sentences in 50 languages distributed across 13 demographic axes. Source sentences are built from combinations of 118 demographic descriptors and three patterns, excluding nonsensical combinations. Multilingual translations include alternatives for gendered languages that cover gendered translations when there is ambiguity in English. Our dataset is intended to uncover demographic imbalances and be the tool to quantify mitigations towards them. Our initial findings show that translation quality for EN-to-XX translations is an average of almost 8 spBLEU better when evaluating with the masculine human reference compared to feminine. In the opposite direction, XX-to-EN, we compare the robustness of the model when the source input only differs in gender (masculine or feminine) and masculine translations are an average of almost 4 spBLEU better than feminine. When embedding sentences to a joint multilingual sentence representations space, we find that for most languages masculine translations are significantly closer to the English neutral sentences when embedded.

Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scale
Marta R. Costa-jussà, Pierre Andrews, Eric Smith, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cynthia Gao, Daniel Licht, Carleigh Wood
iFAIR, Meta
cesstajussa,mortiine ems, prangthippaasaant.
chrisr chrisropers, ekalbassi, cynthiagao dlinsaao,lllceee dlicht, carleighwood com
Abstract
Sentence 1: eng Latn I'm an alcoholic. 
 
 spa Latn
Soy alcohólico. Masculine Ref: Yo soy un alcohólico.
alco holic 

We introduce multilingual extension of the HolisticBias dataset, the largest English cemplate-bseed taxonomy of textual people ref- erences: Multilingual HolisticBias. This ex- tension consists of 20,459 sentences in 50 languages distributed across 13 demographic axes. Source sentences are built from combina- tions of 118 demographic descriptors and three patterns, excluding nonsensical combinations. Multilingual translations include alternatives for gendered languages that cover gendered translations when there ambiguity in English. Our dataset is intended to uncover demographic imbalances and be the tool to quantify mitiga- tions towards them.
Feminine Ref: soy una alcohólica.
Sentence 2: eng Lain have friends who homem ak

 <SD spa Lam Tengo amigos que son amas de casa.
Masculine Ref: Tengo amigos que amos de
Feminine Ref: Tengo amigas que amas de casa.
Sentence 3: Lat_Latn Latn m homem aker 
 <>>
spa Latn Soy una ama de casa
Masculine Ref: Soy amo de casa.
Feminine Ref: Soy ama de casa.
Our initial findings show that translation qual- ity for EN-to-XX translations is an average of almost 8 spBLEU better when evaluating with the masculine human reference compared to feminine. In the opposite direction, XX-to-EIi -NNE, we compare robustness of the model when the source input only differs in gender (mas- culine or feminine) and masculine translations are an average almost 4 spBLEU better than feminine. When embedding sentences to joint multilingual sentence representations space, we find that for most languages masculine trans- lations are significantly closer to the English neutral sentences when embedded.
Sentence Spa_Latn Soy amo casa 
 
 eng Lain I'm the master of the house.
Spa_Latn Soy ama de housewife.
casa.


eng
Ref: I'm homemaker
Figure I: Pathological examples (in sentence piece) of Multilingual HolisticBias source and NLLB translation. with Spanish masculine/feminine references. Input attri- butions (Ferrando et al., 2022) of sentence piece in bold are shown in red gradients. (1) English is translated into Spanish, and we observe masculine form, which could be overgeneralisation or stereotype. (2) English is translated into Spanish masculine for "amigos", which can be seen overgeneralisation, but into feminine for "amas casa", which a stereotype. (3) is translated into feminine, which given the lack of translations into feminine, we assume is a stereotypical translation. (4) Spanish masculineffeminine human translations used source, NLLB translations and English as reference. These examples illustrate the lack of gender robustness. supported by similar input attributions.
WARNING: the current paper contains examples that may be offensive.
1 Introduction
Demographic biases are relatively infrequent phe- nomena but present a very important problem. The development of datasets in this area has raised the interest in evaluating Natural Language Processing (NLP) models beyond standard quality terms.
This can be illustrated by the fact that machine translation (MT) models systematically translate neutral source sentences into masculine or feminine depending on the stereotypical usage of the word
(e.g. "homemakers" into "amas de casa", which is the feminine form in Spanish and "doctors" into "médicos", which is the masculine form in Spanish).

---

While gender is one aspect of demographic biases, we can further explore abilities, nationalities, races or religion and observe other generalizations of the models that may perpetuate or amplify stereotypes and inequalities in society. Quantifying and evalu- ating these biases is not straightforward because of the lack of datasets and evaluation metrics. Proper evaluation will enable mitigation of these biases.
tion when evaluating on the feminine reference set compared to masculine. This showcases the pref- erence towards masculine translations. Among the 13 demographic axes of HolisticBias, the quality of translation averaged across languages is highest for the nationality axis and lowest for the cultural axis, Results when translating to English show that the masculine set has almost 4 spBLEU improvement compared to the feminine set. When embedding sentences to a joint multilingual sentence represen- tations space which is the core tool of multilingual data mining, we find that for most languages, there is a significant difference in the similarity between the masculine translations and the feminine one. Masculine translations are significantly closer to the English sentence when embedded, even if this difference remains small and do not yet know the effect on the mining algorithm.
Related work HolisticBiaa (Smith et al., 2022) is English dataset built from templated sentences that can elicit enough examples in various contexts to analyze and draw actionable conclusions: when measuring toxicity after translating HolisticBias prompts (Costa-jussā et al., 2023); when measur- ing the relative perplexity of different sentences as a function of gendered noun or descriptor (Smith et al., 2022); when looking at skews of the usages of different descriptors in the training data, etc. Other datasets consisting of slotting terms into tem- plates were introduced by (Kurita al., 2019; May et al., 2019; Sheng et al., 2019; Brown al., 2020; Webster et al., 2020), to name a few. The advan- tage of templates is that terms can be swapped in and out to measure different forms of social biases, such as stereotypical associations. Other strate- gies for creating bias datasets include careful hand- crafting of grammars (Renduchintala and Williams, 2022), collecting prompts from the beginnings of existing text sentences (Dhamala al., 2021), and swapping demographic terms existing text, ei- ther heuristically (Papakipos and Bitton, 2022) or using trained neural language models (Qian et al.. 2022). Most of these alternatives cover few lan- guages or they are limited in the bias scope (e.g. only gender (Stanovsky et al., 2019; Renduchintala al., 2021; Levy al., 2021; Costa-jussà al., 2022; Renduchintala and Williams, 2022; Savoldi et al., 2021: ?)).
Background: HolisticBias
HolisticBias is composed of 26 templates, more than 600 descriptors (covering 13 demographic axes) and 30 nouns. Overall, this dataset consists of 472k English sentences used in the context of two-person conversation. Sentences are typically created from combining a sentence template (e.g., am a [NOUN PHRASEJ."), a noun (e.g., parent), and a descriptor (e.g., disabled). The list of nearly 600 descriptors covers 13 demographic axes such as ability, race/ethnicity, gender/sex. The noun can imply certain gender (e.g. woman, man) or avoid gender references (e.g. child, kid). Sentence templates allow for both singular and plural forms of the descriptor/noun phrase.
Experiments in MT with the NLLB model us- ing the full initial (English only) version of the HolisticBias dataset, as reported in (Costa-jussà et al., 2023), show that the percentage of true added toxicity also relatively low (from 0.004% in Chi- nese to 1.07% in Kinyarwanda) but that the number of examples in absolute value is much greater (20 in Chinese, 4,951 in Kinyarwanda) due to the fact that HolisticBias is composed of roughly 230 times more sentences than the FLORES- 200 dev and de- vtest sets put together. The templated nature of HolisticBias also makes it possible to observe dif- ferent translation behaviors for the same lexical items in different contexts. Even if it is ideal for prompting English language models and MT from English to other languages, the main shortcom- ings of the HolisticBias dataset are that we cannot
Contributions Our work approaches this prob- lem by carefully translating a subset of the HolisticBias dataset into 50 languages (see ap- pendix A for a complete list), covering 13 demo- graphic axes. As an extension of HolisticBias, we will invite additions and amendments to the dataset, in order to contribute to its establishment standardized method for evaluating bias for highly multilingual NLP models. We use the pro- posed dataset experiment on MT and sentence representation. Results when translating from En- glish show an average of almost spBLEU reduc-

---

evaluate how the quality varies for this particular domain; and we cannot study biases in variety of languages, which affects multilingual NLP applica- tions.
anced"); (7) overlap with another term in the dataset ("American Indian" vs. "Native American"),
3.2 Data Statistics
HolisticBias successfully combines sentence pat- terns, descriptors and nouns. However, when con- sidering its extension to multiple languages, did not consider translation of patterns, descrip- tors and nouns because languages don't have the same, universal lexical, syntactic or morphological schemes, and not all languages allow for the use of templating/sloting like was done for English. In- stead, we did a full translation of selected sentences in HolisticBias as described in the next section.
Altogether, our initial English dataset consists of 325 sentences. Figure shows the number of trans- lations for each gender (masculine, feminine, neu- tral and generic). There are 15 languages' for which we only have the generic human transla- tion. Those languages do not show feminine and masculine inflections for the patterns that we have chosen. Among the other languages where have several translations, the number of sentences for each gender varies. For the languages in which we have gender inflections, Multilingual Bols keeps separated sets: one for each gender represen- tation (masculine, feminine, neutral and generic).
3 Multilingual HolisticBias
HolisticBias is a large dataset and is unfeasible to translate putence-by-sentene to a large multilin- gual set of languages. In this section, we describe the data selection criteria and we report the statis- ties of the final Multilingual HolisticBias dataset. Translation guidelines are reported in appendix B.
Machine Translation Evaluation
In this section we use Multilingual HolisticBias to evaluate the quality of translations and compare performances across the gendered sets and we do demographic analysis.
3.1 HolisticBias sentence selection
We selected total of 118 descriptors and three templates. Descriptor terms were selected based on the 44 most toxic ones in NLLB translation (Costa-jussà et al., 2023), 45 descriptors that are not robust in translation (excluding descriptors with hyphens due to their complexity). and 29 random ones. These descriptors cover 13 demographic axes shown in Table in appendix C.
4.1 Implementation details
We limit our comparison to the performance of the translation of masculine and feminine sentences. We exclude multiple comparisons with neutral and generic cases, which we leave for further work. As can be seen in Figure 2, not all languages have the same number of masculine and feminine transla- tion, which makes it impossible to compare transla- tion quality. In order to do the experiments with the same amount of sentences accross all languages, exclude from analysis those languages that have less than one hundred masculine translations (which include the 15 languages that we mentioned in section 3.2 that only have generic human trans- lations and nine others?). This means that we keep 26 languages for the following MT analysis. For these languages, when there is no masculine nor feminine translation, we replace it by the neutral translation if available, otherwise the generic one; this ensures that we have 325 sentences to translate and compare for each case and language.
We use three templates that refer to people in different ways, people, person and children: have friends who are [descriptor] people.", "I'm (a/an) [descriptor] person", and "What do you think about [descriptor] children?" We prioritize neutral nouns referring to people because these tend to generate stereotypical translations when translating into gen- dered languages like Romance ones. We choose a mixture of singular and plural. We manually ex- cluded any descriptors that: (1) do not make sense with child(ren), eg. "I'm senior-citizen child", "I'm an unmarried child", replacing the noun with "veteran(s)"; ; (2) focus on beauty/ugliness because of being demographically uninteresting eg. "dirty- blonde", (3) have a tendency to be always pejora- tive ("trailer trash"); (4) are US-specific ("Cuban- American"), (5) are English specific (e.g. "blicket", a purposefully nonsense term following English phonological rules); (6) are relatively rare ("affi-
CChnesse (simplified), Estonian, Finish, Irish, Hungarian, Indonesian, Japanese. Georgian, Halh Mongolian, Persian, Swahili, Turkish. Northen Uzbeck, Vietnamese, Yue Chinese (traditional)
Gandaa, Assamese, Central Kurdish, Bengali, Kyrgyz, Wolsh, Eastern Panjabi, Polish, Maltese and Hindi

---

neutral
fominine
massuline
650
550
800
400
380
250
200
50
guages (see Figure 3), except Thai, when evalu- ating with the feminine reference, the translation quality is lower. We observe that the highest dif- ferences are with Spanish (23.0), Slovak (17.6) and Catalan (16.5). We know of one linguistic feature in Thai, which may have some bearing on this result. The Thai first-person pronoun has two forms: a generic (or underspecified) pronoun and a male-specific pronoun, but no femalc-specific form. Both females and males can choose to use the un- derspecified pronoun to refer to themselves in the first person. The direct consequence of this phe- nomenon that the underspeeified pronoun, which also the only first-person pronoun used by female speakers, is likely by far the more frequently used first-person pronoun. When averaging the transla- tion results from English to the other languages, the biggest difference comes when using either the masculine or the feminine translation as reference (see Figure 6 (bottom)). There is an average drop of 7.9 spBLEU when using feminine compared to masculine references. This shows that the MT system has strong preference towards generat- ing the masculine form. Finally, we observe that scores are higher when using the two translations as references (multi, both masculine and feminine translations as references at the same time). How- ever, when using these multiple references, there improvement (+0.4) compared an Arde
sen Ueng bet Cyr Bong bu Oyn cal Lan ows Lats elo vnt cmn Hore
cmn Lain
des Lain all Grak
est Lain
(in, tain
Lan
gi Lain
hin Deva
hun Lain
ind Lain
lia Lain
ppn, Joan
Aal Geer
Cyn
her Lyyl
Lain
log, Lain
los Lain
mar Deva
mit Lain
nid Lain
pan Guru
pés, Amb
pal Lain
pot Lain
Lain
rus Gyr
alk Lain
Latn
spa Laan
Lain
swft Lain
tan Tami
tha Tha
lir Lein
und Ard
um Lan
ve Lan you Hard
Language
Figure 2: Number of human translations per language and gender (masculine, feminine, neutral and generic).
The translation system is the open-sourced NLLB-200 model with 3 billion parameters avail- able from HuggingFace? We follow the standard setting (beam search with beam size 5, limiting the translation length to 100 tokens). We use the sacrebleu implementation of spBLEU (Goyal et al., 2022) to compute the translation quality with add - k = smoothing. We use ALTT+ implemen- tation 4 to compute the input attributions of Figure 1.
4.2 EN-to-XX translation outputs
We perform our analysis using the masculine, the feminine or both human translations as ref- erence. For this analysis the source is English (EN) HolssticBias, which is a set of unique sen- tences with ambiguous gender. We translate the English set into the all other languages from Multilingual HolisticBias selected from section 4.1). For these languages, when an English source sentence does not have a masculine or feminine reference translation, we use the neutral or generic translation as reference. Figure 3 shows the scores per target languages and Figure 6 (bottom) in ap- pendix D shows the average scores over all sources (eng_Latn) We found that for all pairs of lan-
http://hngiiaaaaeeewtiiccotdobbd 600M

---

mescaline source with meeculine reference
feminine source with feminne reference
spéleu with both referencass
45
wylku acone
5
Bebarusion
Tamil
Unraman
Romanian
Nanthi
Urdu
Russann
Labian
German
Dukh
Pertigueen
Italean
Anblic
Bugaran
Slovaraan
Danish
Swedleh
Lituanaan
Cobiion
Slovak
Spanieh
Language
Figure 3: spBLEU for EN-to-XX using unique English from Multilingual HolisticBias as source and XX human translations from Multilingual HolisticBias (masculine, feminine and both) reference.
only using the masculine reference. We believe that this improvement comes from stereotyped feminine cases, see sentences 2 and 3 in Figure 1. Input at- tributions shown by ALTI+ (Ferrando et al., 2022) show that when translating into feminine we tend get more attention from the source sentence.
has a considerable higher quality when translating the feminine human translation (+4.2).
We observe that the average translation quality from any language to English is 3.8 spBLEU points higher when translating masculine sentences than the feminine ones (see Figure 6 (top)). This shows that for the same sentence pattern which only varies in gender (masculine or feminine), the quality sig- nificantly varies, which confirms a gender bias the system. We give examples in Figure (sentence 4) that show the lack of gender robustness and the wrong translation meaning in the case of translating "amo" to "master" in this particular context for the particular of "I'm homemaker" translated from Spanish. In this case, input attributions are similar.
4.3 XX-to-EN translation outputs
We interested see the quality of translation when starting from a gendered sentence and trans- lating to English where the sentence should be in an ambiguous language. To evaluate this, we use either the masculine or the feminine human trans- lations from Multilingual HolisticBias as source input and the unique English sentences without gender alternatives. reference. Note that because using templated approach, the source in- put only varies gender, which means that we are comparing the robustness of the model in terms of gender.
4.4 XX- translation outputs
While we have seen how the model behaves when dealing with English, the NLLB model is built to be multilingual, so we want to understand how be- haves when translating to and from other languages than English.
Similarly to what we have observed when trans- lating from English, when translating to English from a different language, the model quality is bet- ter for masculine cases. Figure 4 shows results per source language and Figure (top) in appendix D shows the average quality for all sources towards English. We observe that the highest differences be- tween masculine and feminine are with Arabie (24 spBLEU difference), Latvian (14.9) and Swedish (13.4). We observe that there is only Lituanian that
We observe similar trend as in the previous sec- tion, where the translation quality is better when translating from masculine sentence and with masculine reference. Figure 5 in appendix D shows spBLEU differences when using the maseu- line source with masculine reference vs the femi- nine source with feminine reference per language

---

masculine source
feminine
55
45
40
35
30
20
15
Lithuatian
Thai
Czech
Dulch
German
French
Catalon
Portugueeo
Slovenian
Belarusia
Slovak
Tamil
Graek
Italiian
Undu
Ukrainian
Romanian
Spanish
Marathi
Russian
Danish
Bulgarian
Swadeh
Lalvian
Arabic
Source Language
Figure 4: spBLEU for XX-Io-EN translations using XX human masculine or feminine translations as source set and English reference.
pair. Among the highest differences we find cases involving English (e.g. English-Spanish) but also other translation directions such as Thai-to-Arabic and Arabic-to-Swedish. In general, the differences vary with translation direction, which means that may have high difference between Arabic-to- Swedish and so high on the inverse direction. This asymetry makes sense because the MT system is more prone to errors (e.g. overgeneralisation to a single gender) when going from a source that does not specify gender to target that needs to specify it. Whereas going from specified gender towards a unspecified gender tends to be safer (except for cases where we find lack of robustness).
cases are evaluated against the generic sentence and like for English, the model does better in this condi- tion. Of the other languages, with the same number of masculine and feminine reference translations, translations involving Lithuanian are interesting is the only language that shows more translation bias when being used in the target than when it used as the translation source. We can see lan- guages with difference in biases depending they are in the source or in the target.
4.5 Demographic Analysis
The system has tendency output the masculine, except for strongly stereotyped translations. For example, the source sentence I have friends who are one-percenters. is translated into the masculine Tengo amigos que son los ur-percenters. But the source sentence I have friends who have been wid- owed. is translated into the feminine Tengo amigas que han quedado viudas.
As pointed in Table 4 in appendix D, different languages follow different patterns depending if they are used source or target. For 17 languages, when used as source or as target, there no dif- ference the gap in spBLEU when translating masculine vs feminine sentences. As we have dis- cussed in the previous section, translation involving English seems to show less bias when English is used as a target as it means that gendered sentences are translated towards the same generic sentence. Thai special case it does not have spe- cific feminine pronoun, but instead uses a generic (underspecified) pronoun, which means that when evaluating translation towards Thai, the feminine
Table shows mean spBLEU the sentence level on Multilingual HolisticBias axis translations from English, averaged over descriptors, templates, languages, and masculine vs. feminine references We observe that the axes with the lowest quality are the cultural, body-type and socioeconomic ones.
*We exclude the descriptor "queer", an outlier because it falls in both the gender/sex and sexual orientation axes.

---

and the axes with the highest quality are the na- tionality, age, and sexual orientation ones. We see that translation quality scores from feminine ref- erences are lower on average across all axes than from masculine ones. Higher differences in quality between masculine and feminine may indicate axes with higher biases. If we compare among descrip- tors with similar number of samples (>9k), ability has higher bias than body type; comparing axes with between 2.5k and 9k samples, age is the axis with the highest difference, compared with religion, race, characteristics and gender/sex; and the sexual orientation axis is above the socioeconomic, cul- tural, and nationality axes for a lower number of samples.
create training datasets for multilingual translation models (see (NLLB Team al., 2022)). With the
encoders, we can compute a common, language- independent representation of the semantics of a sentence. This is the case for LASER (Heffernan et al., 2022) and LaBSE (Feng et al., 2022). Ide- ally. the encoders should be able to encode the am- biguous English sentences that they are equidis- tant from the gendered versions in the gendered languages. Thus, we should expect "I'm a hand- icapped person" be at the same distance in the embedding space as "Je suis handicapé" (masculine French) and "Je suis handicapée" (feminine French) as they would both be expressed the same in En- glish. The Multilingual HolisticBias dataset lets us test this assumption, because we have the gendered annotation for each marker and its translation in different templates.
Axis
Masc
Fem
Mulii
Avg
Count
Cultural
26.0
22.0
26.3
24.8
1050
Body type
26.0
23.0
26.5
25.2
11250
Sociocomomic class
29.4
26.5
29.8
28.6
1200
Gender and sex
30.5
27.4
30.8
29.6
5400
Religion
Ability
33.1
29.6
33.5
32,1
9900
Racolechnicity
33.5
29.1
33.9
32.2
2700
Characteristes stics
35.9
31.1
36.8
34,6
5700
5.1
Methodology and Implementation details
Nationality
36.0
31.8
36.7
34,8
900
Age
38.3
32.4
38.9
36.5
2700
Sexual orientation
39.0
33.8
39.3
37.4
2100
For LASER implementation (Heffernan et al., 2022) and for each language, we encode each sen- tence and its masculine and feminine translations. If there is a custom encoder for the language, we use this one, and some languages also have a cus- tom sentence piece model (Kudo and Richardson, 2018). Otherwise, (Schwenk and Douze, 2017). We then compute the cosine similarity between the English source and both versions of the translation (when available). We can do a paired t-test to compare the two sets of distances, the null hypothesis being that there is no difference between the similarities and the alternate hypothesis corresponding to the mascu- line being more similar than the feminine reference (hypothesis that there is a bias towards masculine representation). For LaBSE (Feng et al., 2022). we follow similar procedure, only changing the encoders. For our analysis we use the same lan- guages as selected for the MT analysis in section 4.1, that is the ones with more than hundred mas- culine/feminine translations, however, we do not need the same number of samples per language to do the analysis. Therefore, we do not do any replacements like done in the MT section but use only the available, aligned masculine/feminine human translations. This means that we exclude Thai from this analysis as it has enough masculine translations, but no feminine ones.
the mean per-axis spBLEU on trans- lations from English, averaged over descriptor, tem- plate, and language, for masculine references ("Masc")) feminine references ("Fem"); both references com- bined ("Multi"); the average the first 3 columns ("Avg"): x and the total number of measurements across descriptors, templates, languages, and reference types ("Count').
Table 1: Columns:
use the base LASER encoder
Descriptors with the lowest spBLEU, aver- aged over language, template. and masc vs. fem, are mostly in the body type axis: barrel- chested, chunky, kinky-haired, goateed, gangly. balding, and chubby, with the exceptions being one-percenter (socioeconomic axis), nonagenarian (age axis), and "with lisp" (ability axis). Descrip- tors with the highest mean SpBLEU belong to more variable demographic axes: 55-year-old, 40-year- old, 50-year-old, teenage (age); refugee, orphan (characteristics); transscxual (gender and scx); het- erosexual, bisexual (sexual orientation): and Mor- mon (religion). These two sets of descriptors have similar mean percentage biases towards masculine outputs (15.4% and 16.2%, respectively). See com- plete details in Tables and 6 in Appendix E.
5 Multilingual Sentence Embeddings
Sentence representations are used, among others, to compute data mining of multilingual sentences and

---

5.2 Results
one is significant, with a maximum mean differ- ence of 0.09: Modern Standard Arabic, Italian, Spanish, Danish, Marathi, Portuguese, Belarusian, Urdu, Dutch, French, Catalan, German, Standard Latvian, Ukrainian, Russian, Bulgarian, Slovak, Czech, Slovenian and Greek. See Figure 7 (bottom) in appendix F for examples of Spanish, Swedish and Romanian.
Languages where we cannot exclude the null hy- pothesis. There six languages for which the p-value is over 0.05: Tamil, German, Lithuanian, Slovenian, Czech and Urdu; hence we cannot ex- clude the null hypothesis (the difference between the two populations zero). For these languages, the mean difference between the masculine ref- erence and the feminine reference similarities is small (<0.01). Figure 7 (top) in appendix F shows example of Urdu, which has many samples with masculine and feminine translations but similarity scores that are very close between both conditions.
Conclusions
We present multilingual extension of the HolisticBias dataset of approximately 20,500 new sentences. This Multilingual HolisticBias dataset includes the translations 18 descriptors in 50 languages. For each language, we have one or two references, depending on if there is gender inflection in the language. Each translated sentence includes the masculine/neutral translation and second translation with the fem- inine alternative if it exists. Our dataset is meant to be used evaluate translation quality with de- mographic details and study demographic repre- sentations in data. Other potential uses include prompting on multilingual language models.
different patterns and
Languages where we exclude the null hypothe- sis. There are 18 languages for which the p- p-value is <0.01: Spanish, Danish, Portuguese, Bulgar- ian, Dutch, Swedish, French, Standard Latvian, Marathi, Romanian, Belarusian, Ukrainian, Italian, Catalan, Modern Standard Arabic, Slovak, Greek and Russian. For these languages, the difference between the masculine and feminine semantic dis- tance the neutral English equivalent is signifi- cantly different. That is, the feminine translation is always considered to be further away by the LASER semantic space than the masculine one. In reality there should not be significant differences in meaning, so the LASER embedding has bias for these languages. See Figure 7 (top) in appendix F for examples of Spanish and Swedish. However, it is not clear how this would affect the mining pro- cess described in (NLLB Team et al., 2022), as can select multiple sentences based on the margin score. Because of the small difference between the two representations (max 0.04), the rest of the neighbors used in the mining might end up with a worse margin score. This is something to be tested in mining.
We use this new dataset to quantify biases of gender across demographic axes for MT and sen- tence representations and showcase several gender pathologies (e.g. overgeneralisation masculine, gendered stereotypes, lack of gender robustness and wrong meaning). MT has higher performance for masculine sets than for feminine. For EN-to -t- XX translations, performance increases over 8 sp- BLEU. For XX-to-EN, which tests the robustness of the MT model to gender, performance increases almost 4 spBLEU. In terms of demographics, we lower performance for those axis where there seems to be higher masculine stereotype. e.g. so- cioeconomic status ("one ponee-eerenterr".. Multilingual embeddings show that they can be a source of bias, because for most languages, there a significant (p < 0.01) difference among neutral English set and masculine or feminine target set.
LaBSE LaBSE (Feng al., 2022) is similar to the LASER encoder, in that it "produces similar representations exclusively for bilingual sentence pairs that are translations of each other.". We there- fore have the same expectations for LaBSE when comes to embedding the Multilingual HolisticBias dataset. However, we see similar bias in the co- sine distance between the English source and the masculine/feminine translations. LaBSE has four languages for which we cannot exclude the null hy- pothesis: Romanian, Lithuanian, Swedish, Tamil. There are 20 languages where the difference be- tween the masculine translation and the feminine
Limitations
In the current approach to build the dataset, hu- man translators use the English source translate to the corresponding language, thus, the English- centric sentence fragments lack complete cor- respondence across languages. the translators had access to the machine translations provided to other languages they could guarantee parallel

---

translation across languages. However, this is not the case, and we have observed that we have cases such as "Tengo amigos/amigas" (I have friends, extended to both masculine and feminine) being used in Spanish but "Tinc amistats" ("I have friend- ships") being used in Catalan. While this case has been corrected to "Tine amics/amigues" ("I have friends", extended to both masculine and feminine) in Catalan, there may be other cases that are not corrected.
doing Figure 1 and also Mikel Artetxe and Hila Gonen for enriching discussions in early steps of this project.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
The word "friends" in one of our three sentence patterns could mean: multiple friends of mixed gender, multiple female friends or multiple male friends. Most romance languages, for example, will still have the ambiguity that "friends" can rep- resent mixed set of friends, and this, historically, has taken the form of the masculine plural noun. Recently, there are trends that may change this at least for some languages that tend to include both masculine and feminine nouns even in plural. How- ever, one could argue the preference towards the masculine noun in the translation might represent a preference towards the "ncutral/mixed" case, which could well be the most represented case in the data. It would be interesting to see if observe the same behaviors when we exclude the friends sam- ples, (however, in this case we'd have a lot less data).
Marta R. Costa-jussà, Carlos Escolano, Christine Basta, Javier Ferrando, Roser Batlle, and Ksenia Kharitonova. 2022. Gender bias in multilingual neu- machine translation: The architecture matters.
Marta R. Costa-jussà, Eric Smith, Christophe Ropers, Daniel Licht, Jean Maillard, Javier Ferrando, and Car- los Escolano. 2023. Toxicity in multilingual machine translation scale.
Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open ended language genera- tion. In Proceedings of the 2021 ACM Conference Fairness, Accountability. and Transparency, pages 862-872.
Fangxiaoyu Feng. Yinfei Yang. Daniel Cer, Naveen Ari- vazhagan, and Wei Wang. 2022. Language-agnoste BERT sentence embedding. In Proceedings of the 60th Anmual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 878-891, Dublin, Ireland. Association for Computa- tional Linguistics.
While we have translated huge amount of sentences, over 20k, our Multilingual HolisticBias dataset may be quite small in relation to standard MT benchmarks.
The best alternative would be to consider ex- tending the Multilingual HolisticBias dataset ei- ther with more human translations or by artificially extending what we have.
Javier Ferrando, Gerard I. Gállego, Belen Alastruey, Carlos Escolano, and Marta R. Costa-jussà. 2022. Towards opening the black box of neural machine translation: Source and target interpretations of the transformer. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Process- ing (EMNLP).
Note that our extension is limited few hun- dred sentences in each language, so we cannot per- form the toxicity analysis for each language as it was done in previous work (Costa-jussà et al., 2023).
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju. Sanjana Kr- ishnan, Mare' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores- 101 evaluation benchmark for low-resource and multilingual ma- chine translation. Transactions of the Association for Computational Linguistics, 10:522-538.
Our analysis in the current paper is limited to comparing masculine and feminine performance. We exclude multiple comparisons with neutral and generic cases, which we leave for further work. Examples from Figure are explicitly chosen to show what kind of challenges the MT model shows.
Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022. Bitext mining using distilled sentence repre- sentations for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2101-2112, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Acknowledgments
The authors want to thank Belen Alastruey and Javier Ferrando for their help and guidance for

---

Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Adithya Renduchintala and Adina Williams. 2022. In- vestigating failures of automatic translationin the case of unambiguous gender. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 3454-344 1454,3469, Dublin, Ireland. Association for Compu- tational Linguistics.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contex- tualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166-172.
Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Mat- teo Negri, and Marco Turchi. 2021. Gender bias in machine translation. Transactions of the Association for Computational Linguistics, 9:845-888 874.
Holger Schwenk and Matthijs Douze. 2017. Learn- ing joint multilingual sentence representations with neural machine translation. In Proceedings of the 2nd Workshop on Representation Learning for NLP. pages 157-167, Vancouver, Canada. Association for Computational Linguistics.
Shahar Levy, Koren Lazar, and Gabriel Stanovsky. 2021. Collecting large-scale gender bias dataset for coref- erence resolution and machine translation. In Find- ings of the Association for Computational Linguis- fics: EMNLP 202., pages 2470-2480, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Emily Sheng, Kai- Wei Chang, Prem Natarajan, and Nanyun Peng. 2019. The woman worked as babysit- ter: On biases in language generation. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-UCCNLP", pages 3407-3412.
Chandler May, Alex Wang. Shikha Bordia, Samuel Bow- man, and Rachel Rudinger: 2019. On measuring so- cial biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers). pages 622-628.
Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. 2022. "Im sorry to hear that": Finding new biases in language models with a holistic descriptor dataset. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9180-9211, 1, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
NLLB Team, Marta R. Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad. Kenneth Heafield, Kevin Hef- feman, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loie Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami. Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human- centered machine translation.
Gabriel Stanovsky, Noah A. Smith, and Luke Zettle- moyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 1679-1684, Florence, Italy, Association for Computational Linguistics.
Kellie Webster, Xuezhi Wang. Ian Tenney, Alex Beu- tel, Emily Pitler, Ellic Pavlick, Jilin Chen, and Slav Petrov, 2020. Measuring and reducing gendered cor- relations in pre-trained models.
Zoe Papakipos and Joanna Bitton. 2022. Augly: Data augmentations for robustness, arXiv preprint arXiv:2201.00 06494.
A List of languages
Rebecca Qian, Candace Ross, Jude Fernandes, Eric Smith, Douwe Kiela, and Adina Williams. 2022. Per- turbation augmentation for fairer nlp. arXiv preprint arXiv:2205. /2586.
The languages included in this study represent 13 families and 13 scripts, as shown in Table 2.
Adithya Renduchintala, Denise Diaz, Kenneth Heafield, Xian Li, and Mona Diab. 2021. Gender bias ampli- fication during speed-quality optimization in neural machine translation. In Proceedings of the 59th An- nual Meeting of the Association for Computational I Linguistics and the 1 1hh International Joint Confer- ence on Natural Language Processing (Volume 2: Short Pupers), pages 99-109, Online. Association for Computational Linguistics.
B Translation Guidelines
The objective of our dataset is to have gold- standard human translations from professional lin- guists that are accurately faithful the source sen- tences. The additional challenge of the bias data set is that the source sentences generated via the templated approach are vague and disconnected

---

arb_Arab
Modern Standard Arabic
(especially descriptors), given that personal expe- rience is where bias may exist; instead, conduct lexical research through credible sources of infor- mation, such as unilingual dictionaries or encyclo- pedias, and provide information as to the source being used and the rationale for selecting one trans- lation over another; (4) remain faithful to the source (see below for further details on faithfulness to the source).
asm Beng
Assamese
be_Cyrl
Belarusian
Beng
Bengali
bul Cyrl
Bulgarian
cat Latn
Catalan
ces Latn
Czech
ckb_Arab
Central Kurdish
emn Hans
Mandarin Chinese (simplified script)
cym_Late Latn
Welsh
dan Latn
Danish
deu Latn
German
ell Grek
Greek
Being faithful to the source is north-star prin- ciple in any translation exercise, which can some- times conflict with other guidance frequently given to translators, such as the need to produce fluent or natural- sounding translations. The two principles are complementary when the source material con- tains naturally produced, fluent language. However, due to the templated nature of the source material in our particular casc, some source sentences may appear lacking in fluency (especially when using the nouns people or person). The question there- fore arose whether these nouns should be translated or omitted. The general guidance given to linguists was that (1) they should bear in mind that the source sentences may not necessarily sound fluent or natu- ral to native speakers of the source language (here, English) and they should strive to remain faithful to the source, and (2) they should feel free to omit such nouns if they feel that the resulting transla- tion sounds unacceptable in their respective native languages.
est Latn
Estonian
fin Latn
Finmish
fra Latn
French
Lle_Latn
Irish
hin Deva
Hindi
hun Latn
Hungarian
ind Latn
Indonesian
ita Lain
Italian
jpn Jpan
Japanese
kat Geor
Georgian
khk Cyrl
Halh Mongolian
kir_ Cyrl
Kyrgyz
Latn
Lithuanian
lug Latn
Ganda
Lat_
Standard Latvian
mar Deva
Marathi
mit Latn
Maltese
nid Latn
Dutch
pan Guru
Eastern Panjabi
nes_Arab
Western Persian
pol_Late Latn
Polish
por_Latm
Portuguese
ron Latn
Romanian
rus rus_Cye Cyrl
Russian
slv Lain
Slovak
Slovenian
spa Latn
Spanish
swe Latn
Swedish
swh__aa Lam Swahili
Additionally, we established a peer-review pro- cess, which involved reviewers from different ven- dors. This added an extra layer of quality checks to ensure accuracy and consistency in the final transla- tion. This process was similar to translation quality checks in which two reviewers provided by dif- ferent vendors assigned to work together review, refute the translation from the translating vendor, and suggest the most uppropriate one, if necessary. All research and discussions by review- ers were documented to ensure transparency and accountability. This crucial step helped us track the changes made to the original translation and identify issues that may arise during the translation process. The reviewed translation is considered the final one.
tam Taml Tamil
tha Thai Thai
tur Latn Turkish
ukr_ Cyrl
Ukrainian
urd Arab
Undu
uzn Latn
Northern Uzbek
vie Latn
Vietnamese
yue Hant
Yue Chinese (traditional script)
Table 2: The 50 languages analyzed in this work, subse- lected from the 200 NLLB languages.
from any disambiguating context. Therefore, lin- guists needed to make sure that their translations were both accurate enough to not include bias and generic enough to be used in most possible con- texts. Linguists were asked to: (1) provide ac- curate and culturally appropriate translations; (2) provide separate translations for each noun class or grammatical gender for languages that make use noun classes or grammatical genders; (3) avoid relying on their personal experience to translate
C List of descriptors
Table shows the complete list of descriptors as reported in section 3.

---

Axis
Deseriptors
Ablliy
aphasse, disabled, dyslexic. hundicapped, hearing-inpaired. low-vision. mute, neurodiverse (reworded to "woundivvegen""" neuroeypical, paralyzed, quadriplegic, who incured rraumatic hrain injury, who is an amputee, who is deaf, who is visually impaired,
cognilve disubility, developmentaal disability, with lisp, with stulter, with an intellectual disability
disability.
Age
40 year 50 year balding, harrel. chested, black haired, hranette, hurly, chubby, chunky, curly haired, dark haired, dark skinned, four eyed. gungly, ginger, gosteed, graving (rewrded "going gray" kinky -hiredd mildly overweight, morbidll obese, mustachiond. overweight, phump, ponytailad, semifty, alighfly overweight, toned, underweight
year adolescent, nocagemaria, oclogenarin, leenage, thirty something
Body tye
Choracteristies
Reserve, ncn (reworded "imumigrant" ) orplun, part timer. refugec, undocumented. widowed, with criminal background with a gambling problem, with domentia, with no kids
born, addict, alcoholic. homemaker..
Millonnial, hilbbily
Gender and sox
agender, nous, disgender, gender non-conforming, gendertiai, genderqueer, bemmpphoddie, intersex, trans,
femme, queet
tranesexual
Gender und sot Sexual orientation
Nationality
Dominican, Salvadoran
othnicity
Aboriginal, Black, Hispanic, Indigenous, Native
Religion
Bahã'i, Confucionist, Evangelical, Hasithe, Mormon, Rastafarian, Shintoist, Sikh. Spiritualist, Unitarian, Wiccan,
Secual orientation
ascuaal. bisexual, demisexual, gay. heterosekual, pansexual, polyarneeus one see-perventer, wealihy,
dogree
Table 3: List of complete descriptors classified by demographic axes for Multilingual HolisticBias.
D XX-to-XX Analysis Details
In this section we show the complete results re- ported in section 4.4. We show the heatmap of the XX-to-XX differences between spBLEU when us- ing the masculine (feminine) source with the mas- culine (feminine) reference in Figure 5; followed by spBLEU average in Figure 6; and Table 4 that report the exact differences.
E Demographic Analysis Details
Tables 5 and 6 present the details of the demo- graphic analysis from section 4.5. For the top 10 and bottom 10 HolisticBias descriptors ranked by mean spBLEU, translation quality scores are higher on average from masculine references than from feminine references. Translation quality is highest on average for the "I have friends who are [descriptor] people." template, roughly spBLEU higher than for "I' m (a/an) [descriptor] person." " and have friends who are [descriptor] people." templates.
F Multilingual Sentence Embeddings Details
Figure 7 show examples of similarity distributions among genders with LASER (top) and LaBSE (bot- tom).

---

Abbb
bel Cyrl
byl_Cyel
Latn
Lain
Latn
Lain
Lain
25
Ro. Lain
20
spateu mascuinng menneeeenn rerteminii
Latn
15
Ivs Lain
-10
Devo
-5
Ild_aan
-0
sor_Latn
manlatn
nss_Cyr
sk
du Lain
spa Lain
wmw. Labn
Taml
tho, That
urd Arab
bel Ove
oa Lato
dan Latn
deu Latr
ang Latn
Ro Latn
Nr. Leto
Davs
NN Latn
por Lat
mn Jahn
ske Lam
m0e Latn
the Thei
und Ano
al
no
IR
all
spa
tan
Targer
Figure 5: XX-to-XX differences between spBLEU when using the masculine source with masculine reference vs the feminine source with feminine reference.

---

masculine
faminine
mascutine
fominino source with both references
20
16
10
mar Deve
non Latn
bal Cyt
tam, Taml
und Anb
ue Dyn
Jawh
nus Gyn
eng Lao
bul Oyr
eat Latn
Ine Lain
alv Lain
dan Lain
akk sain
ava
Target Language
masculine source with masculine reforence
faminine scurco with fominino roference faminine raferances
masculine source with both references
30
15
Lihuman
German
Crook
Buggarinn
Tamii
Calaonn
Portigueess
Dolch
Undu
Romenion
Slovenian
Marathi
Roseden
French
Spanish
Swadiih
Danish
Labiion
Englih
Language
Figure 6: spBLEU average for XX-to-XX translations, averaged per target language (top) and source language (bottom). For both, we show averages with masculine (feminine) human translations as source with masculine (feminine) or both (masculine and feminine) as references.

---

Juste posteepettltl nethy
22
a: Spanish LASER
b: Swedish LASER
Urdu LASER
33-
6.0
342
6.00
082
d: Spanish LaBSE
Swedish LaBSE
f: Urdu LaBSE
Figure Example similarity distributions among genders when using LASER (top) and LaBSE (bottom) encoders. Urdu and Spanish show different behaviors in LASER and LaBSE.

---

Language
Lang-XX
XX-Lang
Masc
Fem.
Masc
Fem
Multi
Count
Ref.
Ref.
barel-chestedd
18.1
18.2
17.2
one nee-pereenter
18.1
15.6
18.1
17.3
450
Arabic*
11.5
3.8
325
168
kinky-haired
19.8
17.2
19.9
19.0
450
Belarusian
0.8
0.5
134
134
nonagenarian
16.8
20.2
19.1
300
goaleed
20.1
17.1
20.2
19.1
300
Bulgarian
1.7
4.5
137
138
gangly
21.0
18.0
21.1
20.0
450
Catalan
2.1
4.6
224
224
a lisp
21.2
21.6
20.5
balding
22.0
18.4
22.0
20.88
450
Czech
1.6
0.1
179
179
chubby
Danish
5.8
6.1
117
117
bisexual
43.0
36.8
43.3
41.0
300
Dutch
2.8
2.0
119
119
Ieenage
43.2
36.0
44.2
41.1
transecuaal
43.2
38.0
43.5
41.6
450
Englisht
8.0
3.8
NA
NA
orphan
44.2
37.3
44.5
42.0
450
French
3.6
5.1
157
157
heterosexual
refugee
48.3
37.2
48.9
44.8
450
German
1.0
2.9
151
161
50-yeur-old
41.6
48.5
46.0
300
Greek
1.6
4.9
144
145
40-ycar-old
42.1
49.4
46.7
55-ycar-old
51.8
45.3
52.7
50.0
Italian
1.7
2.6
212
213
Latvian
5.8
4.1
325
148
the mean per-descriptor spBLEU translations from English, averaged over template and language. Only top and bottom desriptors are shown. Columns in Table 1.
Table 5: Columns:
Lithuanian
-0.3
10.7
165
165
Marathi
3.4
-13
128
128
Portuguese
2.5
0.9
131
131
Romanian
3.1
0.1
183
183
Russian
3.5
3.7
106
106
Slovak
3.8
7.8
156
156
Slovenian
3.3
5.7
186
186
Spanish
4.5
5.5
225
229
Swedish
5.7
7.5
124
124
Tamil
1.8
0.5
162
161
Thaiş
5.2
-0.5
238
Ukrainian
2.1
2.3
138
137
Urdu
3.0
1.6
320
267
Table 4: XX-to-XX differences between spBLEU when using the masculine source with masculine reference the feminine source with feminine reference, av- eraged over all targets all sources. The last two columns show the number of reference translation in each case. Some notable cases: Englishi doesn't 't have masculine/feminine references, Thaiţ has zero feminine translation as a generic (underspecified) pronoun is used instead, * has difference between mas- culine/feminine cases when used as source but a big difference when used as target. Arabic follows inverse trend.
Axis
Mac
Fem
Multi 29.7
Avg 28.7
Count
"What think about [descripto]] children""
29.7
26.6
13338
"I'm (alan) [descriptor) person."
29.3
28.5
30.4
29.4
17700
have friends who are [dsecripoor] people."
35.7
28.3
36.1
33.3
17700
Table 6: Columns: the mean per-template spBLEU on translations from English, averaged over axis, descriptor, and language. Columns are as in Table


Reviews:
[{"review_id": "SpykpN9p4u", "reviewer": "Reviewer_Uryd", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments!\n\nAbout the storyline, nice point! We now corrected some sentences that did not specify “translation bias”. We are not analising language bias but translation biases for a large scale of translation directions. We clarified when it was not clear.\nAbout the missing citations, thanks for pointing out the survey of Savoldi. We do a small review of state-of-the-art in the area in the introduction. Since there is a huge amount of literature in studying gender bias in MT, we focus mainly on the works that develop datasets for this topic, which is closer to our work. Given the limitations in space, we are not covering exhaustively even this part. However, it makes sense to cite the survey  you mention.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- a large-scale multilingual dataset that is manually translated. This will be a valuable resources for evaluating multilingual models on biases. \n- The paper proposes a practical framework to quantify the gender biases in MT using the dataset and provides interesting analysis and insights.", "reasons_to_reject": "- The storyline could be a bit more precise regarding the gender biases across languages. Given that the dataset focuses specifically on toxic descriptors, I think the lessons here from the experiments seem to be that there is a strong association with masculinity and toxicity in many languages rather than in general these language has masculinity biases. \n\n- gender bias in machine translation is not something new. The author(s) might want to better position the work with regard to other studies that investigate biases in machine translation such as \"Gender Bias in Machine Translation\" Savoldi · 2021\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents Multilingual Holistic Bias, a multilingual extension of the English HolisticBias dataset covering 50 languages. This dataset is then used to particularly quantify the gender bias in machine translation in this study. The key finding is that MT models tend to favour masculine forms across languages (I.e. MT models tend to generate more masculine translations and masculine source sentences tend to be translated with better quality compared with feminine counterparts). This gender bias is also confirmed by examining the multilingual embedding space where masculine translations are usually significantly closer to the English source sentence than feminine translations. The key contributions for the paper are a new and extensive multilingual dataset with a specific focus on gender and demographic biases, and interesting analysis to show that machine translations are biased towards the masculine translations. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 203: Among the other languages where have -> Among the other languages where there are\n"}, "extra_text": {"Missing_References": "Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, Marco Turchi; Gender Bias in Machine Translation. Transactions of the Association for Computational Linguistics 2021;\nand the studies referenced in this survey study. "}}, {"review_id": "hGsn8sCFsv", "reviewer": "Reviewer_QhKn", "summary": "", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The reviewer is totally right, figure 4 should only report feminine source and masculine source. There is no need to report “both” references since the English is neutral. As you can observe from the figure, masculine with one or two references is exactly the same. The same happens with feminine. We are changing the figure accordingly and clarifying in the text.\n\nWe have worked on reordering the figures in the paper, specially, moving earlier in the sections the examples. Also, we have added the missing references now.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces a valuable multilingual resource that can be used to study and analyze bias in multilingual MT models. The studies that are introduced in the paper demonstrate well how the dataset could be used and its viability. Most of the paper was written clearly and was easy to follow.\n", "reasons_to_reject": "I found some parts of the paper to be hard to follow:\n* In Figure 4, the authors present results on XX-to-EN translations. I am confused about the results that refer to gendered references in English. The three English templates in the corpus contain descriptors that refer to: 1) third person (as in the first and third templates); 2) first person (as in the second template). Unless the descriptors are gendered, then it will not be possible to tell if the English reference includes a masculine or a feminine reference. So what does the legend of the table stand for here when describing masculine vs feminine English references?\n\n* The figures in the paper were not ordered and do not align with the overall story. I would urge the authors to fix the order of the figures to make the paper easier to read.", "questions_to_authors": "See above.", "paper_topic_and_main_contributions": "In this paper, the authors introduce MultilingualHolisticBias, a multilingual extension of the HolisticBias dataset. HolisticBias is a dataset of 472K English sentences that are created through templates. These templates are filled with descriptors that cover various demographical axes such as race, gender, and race.\n\nThe author selected a subset of 325 sentences from HolisticBias bias which includes 118 demographic descriptors covering 13 demographic axes, and three templates. They manually introduced the target translations of the 325 sentences for 50 languages. The translations include gender-rewritten (or gender-neutral) alternatives for gender-marking languages when there's ambiguity in English. Therefore, each English sentence will have one or two references depending if the target language is gender-marking or not. In total, MultilingualHolisticBias consists of ~20.5K multilingual parallel sentences.\n\nFurthermore,  the authors demonstrate the usage of MultilingualHolisticBias by analyzing and quantifying MT biases in different translation directions: EN-to-XX, XX-to-EN, and XX-to-XX. For their MT model, they use the open-source NLLB model. The authors found that: 1) EN-to-XX translations are almost 8 BLEU points better when evaluating with masculine references compared with feminine translations; 2) XX-to-EN translations are 4 BLEU better when the source is in the masculine form; 3) XX-to-XX translations vary based on which language was used for source and target. But overall, the translation quality is better when translating from a masculine source to a masculine target.\n\nMoreover, when embedding sentences into a joint multilingual space, they found that masculine targets are closer to the English ambiguous (non-gender) source sentences.\n\n\n\n ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "There has been some work on reporting gender bias in MT when translating from English to a gender-marking language, specifically Arabic. I find these papers to be relevant:\n\n1) https://aclanthology.org/2022.naacl-main.46.pdf\n2) https://aclanthology.org/W19-3822.pdf\n3) https://aclanthology.org/2022.lrec-1.199.pdf\n"}}, {"review_id": "q7ypi2EEU1", "reviewer": "Reviewer_seQ7", "summary": "", "questions": "A. For the languages with only generic translations, e.g., Chinese, are these data useful for demographic bias measurement.\n\nB. Why are only 3 templates selected? In addition to the ‘people, person and children’ reason, are there any other reasons for choosing these 3 templates?\n\nC. What is add-k=1 smoothing?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments! \n\nQA. Yes. we can study the quality of the translations in each demographic axis. However, results are not strictly comparable since different demographics use different vocabulary. But still, it is useful!\n\nQB. We limited to 3 patterns because of budget constraints. Translations are professionally created with strict quality checks.\n\nQC. This is the smoothing option that we choose available in SacreBleu. We add this detail for completeness.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper proposes a multilingual bias measurement dataset covering 50 languages, which is valuable for measuring demographic biases in a variety of languages.\n- This paper investigates the demographic biases in both multilingual machine translation systems and multilingual sentence encoders. The comprehensive results show that demographic biases exists in these multilingual models. ", "reasons_to_reject": "- For each language, the number of sentences is from 325 to 650 , which is limited and the results may be affected by a few samples.", "questions_to_authors": "A. For the languages with only generic translations, e.g., Chinese, are these data useful for demographic bias measurement.\n\nB. Why are only 3 templates selected? In addition to the ‘people, person and children’ reason, are there any other reasons for choosing these 3 templates?\n\nC. What is add-k=1 smoothing?", "paper_topic_and_main_contributions": "This paper aims to measure demographic bias in multilingual machine translation systems, e.g., NLLB, and multilingual sentence encoders, e.g., LASER and LaBSE, in a variety of languages.\n\nFirstly, it extends the English-only bias measurement dataset, HolisticBias, to MultilingualHolisticBias covering 50 typologically-diverse languages with professional linguists, translation guidelines, and quality check. The dataset has 20k sentences in total and covers 3 demographic axes.\n\nUsing MultilingualHolisticBias, it conducts some comprehensive analysis on multilingual NLP applications. For multilingual translation models, results on NLLB shows that the system has a tendency to output the masculine, except for strongly stereotyped translations. For multilingual sentence encoders, this paper find that for most languages masculine translations are significantly closer to the English neutral sentences when embedded.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a multilingual extension of the English HolisticBias dataset covering 50 languages, created by manual translation. This dataset is then used to quantify the gender bias in machine translation, and the authors find that MT models tend to favour masculine forms across languages.

The reviewers find that the proposed resource will be valuable for studying and analyzing demographic biases in multilingual machine translation systems and multilingual sentence encoders. Most of the paper was found to be clearly written and easy to follow. Reviewers remarked, however, that some crucial references on biases in MT were missing, and that some parts of the paper could benefit from additional clarifications. The requested changes should be easy to accommodate in the camera-ready version.
Decision: Accept-Main