A Rose by Any Other Name would not Smell as Sweet: Social Bias in Names Mistranslation
Authors: Sandra Camille Sandoval
Jieyu Zhao
Marine Carpuat
Hal Daumé III
Venue: EMNLP 2023 Main (2023)
Keywords: fairness and bias
NLP-related harms
name translation
evaluation
TL;DR: We develop an approach to measure disparities in the machine translation of first names across race/ethnicity and gender on translation accuracy and fluency; we find significant negative effects for Black and Hispanic female-associated names.

Abstract: We ask the question: Are there widespread disparities in machine translations of names across race/ethnicity, and gender? We hypothesize that the translation quality of names and surrounding context will be lower for names associated with US racial and ethnic minorities due to these systems’ tendencies to standardize language to predominant language patterns. We develop a dataset of names that are strongly demographically aligned and propose a translation evaluation procedure based on round-trip translation. We analyze the effect of name demographics on translation quality using generalized linear mixed effects models and find that the ability of translation systems to correctly translate female-associated names is significantly lower than male-associated names. This effect is particularly pronounced for female-associated names that are also associated with racial (Black) and ethnic (Hispanic) minorities. This disparity in translation quality between social groups for something as personal as someone's name has significant implications for people's professional, personal, and cultural identities, self-worth and ease of communication. Our findings suggest that more MT research is needed to improve the translation of names and to provide high-quality service for users regardless of gender, race, and ethnicity.

A Rose by Any Other Name would not Smell as Sweet: Social Bias in Name Mistranslations
Sandra Sandoval
Jieyu Zhao
University of Maryland
University of Southern California jieyuzêusc. edu
sandracs@umd. edu
Marine Carpuat
Hal Daumé III
University of Maryland marine@umd. edu
University of Maryland Microsoft Research hal hal3@umd. edu
Abstract
Source:
Journee es una pocta británica de fuerza, clar- idad oficio honesto.
We ask the question: Are there widespread disparities machine translations of names across race/ethnicity, and gender? We hypoth- esize that the translation quality of names and surrounding context will be lower for names associated with US racial and ethnic minorities due to these systems' tendencies to standard- ize language predominant language patterns. We develop dataset of names that are strongly demographically aligned and propose transla- tion evaluation procedure based round-trip translation. We analyze the effect of name de- mographics on translation quality using gener- alized linear mixed effects models and find that the ability of translation systems to correctly translate female-associated names is signifi- cantly lower than mule-associated names. This effect particularly pronounced female- associated names that are also associated with racial (Black) and ethnic (Hispanic) minorities. This disparity translation quality between so- cial groups for something as personal some- one's 's name has significant implications for peo- ple's professional, personal and cultural iden- tities, self-worte and case of communication. Our findings suggest that more MT research is needed to improve the translation of names and to provide high-quality service for users regardless of gender, race, and ethnicity.
Translation:
Journee is British poct of force, clarity and honest craft.
MT Output:
Girls are British poet of strength, clarity and honest craft.
Error(s):
Name completely mistranslated
Source:
Прижтио с нами поSнERоMPTECH AMHA.
Translation:
Pleased to meet you Amiyah.
MT Output:
I'm happy to meet you mom.
Erron(s):
Name translated as common noun
Figure I: Two hypothetical input sentences, originally in Spanish and Russian, together with correct English translations well the output of MT systems from our study; both mistranslate person's 's given name.
frequent in online settings, whether via Linkedln. professional email, Twitter, Slack, any other place where people address others or addressed by name. In multilingual personal or workplace settings, displaying names incorrectly due to mis- translations as Figure 1, whether in greetings or in the context of larger statements related to a person's 's name, can misrepresent a person's 's work, make professional communications more difficult, or erode their sense of identity and self-worth This can lead to both allocational harms and represen- tational harms (Crawford, 2017: Barocas et al., 2017), especially when rates of name mistransla- tion are spread disparately across social groups (Blodgett et al., 2020) (see section 2).
1 Introduction
When people see their names incorrectly displayed by the technologies they use, this creates an un- necessary burden for them. Over time, the regu- lar experience of an AI system getting person's s name wrong have insidious detrimental ef- fects such as the erosion of cultural identity and self-worth, similar effects from racial microa- gressions such as name mispronunciations expe- rienced in the classroom (Kohli and Solórzano, 2012). Such experiences have the potential to be
Unfortunately, the subtle but crucial detail of having correct name translations for everyone seems to have been overlooked. For instance,
prominent survey paper on gender bias in machine translation (MT) (Savoldi al., 2021) presents a range of work that, while detailing underepresen- tation of social groups language, narrowly de- fines the problem as relating to misreprescntation of women with respect to linguistic expressions

---

about them (for example, incorrect pronoun usage) or the frequent mistranslation of female named en- tities into male entities. This reflects the tendency of research on gender bias in NLP to be focused almost exclusively on personal pronouns. A major exception is work in named entity recognition and coreference resolution, where various research has found that the accuracy of name recognition tends deteriorate across gender as well as race and ethnicity (Mishra et al., 2020).
Name mistranslation also gives rise significant representational harms, including, least, harms related to quality of service, denial of self-identity, and erasure and alienation of both the name and its associated culture (Katzman et al., 2023).
offeeiceo harms resulting from name mistranslation clear-ift a machine translation system works less well for one person's name, or names strongly associated with marginalized social groups, than others, those people and those groups experience a less effective machine translation sys- tem. This is particularly problematic when the user is less likely to be able to identify translation errors, such as when the target language uses a different al- phabet than the source. This type of harm parallels those studied by Dyal-Chanl (2021) in the context of incorrect auto-complete for some names over others, where names are corrected to more angli- cized versions or translated to nouns. Dyal-Chand (2021) emphasizes that to mitigate the danger of reinforcement of structural racism through tech- nologies, we must acknowledge the role that they have on all individuals stating that everyone has" "... right to full and equal use of these technologies".
In this paper, we ask: Are there disparities in the accuracy of machine translations of names across race, ethnicity, and gender? We define and evalu- ate particular form of bias through incquality in the machine translations of names. To achieve this, we constructed a dataset, Diverse Names in Con- text (DNIC), of English sentences, by combining templates with names that are known to be strongly associated with particular social groups (section 4). Using this dataset, we develop procedure for ana- lyzing the quality of machine translation systems' ability to correctly translate names and their con- texts based on round-trip translation from and to English, through Spanish, Russian, Arabic and Chi- nese (section 5). We find that name translation errors across three state-of-ther -f-th-rrtt machine transla- tions systems are incquitably impacting those with ethnic minority names, particularly Black (odds ra- tio=0.69) and Hispanic females (odds ratio=0.62). at a much higher rate than White males (section 6).
Denial of self-identity (the inability to express oneself as desired), erasure (a system's is routine re- moval of traces of some social groups), and alien- ation (a system marking some social groups as other) arise due to the close ties between one's s name and one's culture and community. Kohli and Solórzano (2012) emphasize how racial mieroag- gressions, such as teachers' mispronunciation anglicization of of names, often leave people of color feeling diminished, feeling like they or their culture aren't 't valued, and presure them to take re- sponsiblity to make it easier for others to pronounce their names. When viewed through the lens of Al systems, this pressure leads to form of "language standardization" in technology (Pérez-Quiñone Quiñones and Salas, 2021), form of language ideology that can be used to justify social hierarchies (c.g., Alim et al., 2016; Rosa and Flores, 2017).
2 Sociolinguistic Background
Given names tend be highly personal, and also reflective of a person's position in social groups, of- ten with strong associations to a particular gender, a particular race or ethnicity, a social class, a religion, and more (Pharr, 1993). Beyond simply marking somcone's identity, given names also signal and reinforce individuality and the social position of the person being referenced (Jeshion, 2009).
Due to the importance of given names, their reg- ular mistranslation can potentially have substantial negative impacts on a person's life. Allocative harms (Crawford, 2017; Barocas et al., 2017) can arise either because 1) some people are experienc- ing worse system behavior despite paying the same for the use of system, or 2) when professional communication (e.g.. in job-seeking or recognition of their work) made more difficult due to lower quality translations of their names and sentences written about them.
Because names can code for a multiplicity of social axes (gender, race, ethmicity, religion, social class, birth generation, etc.), it is important to study the impact of name mistranslation not just on a sin- gle social axis, but also based on how axes intersect. Intersectionality is core concept in Black femi- nism, introduced in the Combahee River Collective Statements 1977; 1983: "Because the intersectional experience is greater than the sum of racism and

---

Templates
My name
Journee poeta británica de fuerza, claridad oficio honesto.
Where are you from
MT (->ssp)
Roundtrip Translation
MT (->eng)
is British poet of force, clarity and honest craft.
Journee British poet of force. clarity and honest craft.
Girls British poet of strangth, clarity and honest
Baby Names
Aaliyah
Black
Evaluation
Journee
Name Match False Bleu #82.42
Zachariah
White
Social Group Black, Female
Evaluation approach: A template is combined with a baby name (with an associated social group) to produce an English sentence. This sentence is round-trij translated through a pivot language this case, Spanish) back into English. We measure (a) whether the exact input name (Journee) appears verbatim in the output and (b) the BLEU score, which get stored with the associated social group.
Figure 2:
sexism, any analysis that does not take intersection- ality into account cannot sufficiently address the particular manner in which Black women are sub- ordinated." " Intersectionality was applied in legal setting by Crenshaw (1989) to analyze the ways in which U.S. antidiserimination law fails Black women. In our context, we analyze name mistrans- lation across a combined race/cthmicity axis (White non-Hispanic, Black non-Hispanic, Hispanic, and Asian Pacific Islander), and binary gender axis (Male, Female), with the concomitant limitations arising from that coarse categorization.
significance of our measures (of the correctness of name translations, and of sentence translation qual- ity as whole). We used mixed effects statistical analysis to quantify these effects.
We describe each of these components in the context of the dataset we developed next.
Diverse Names in Context Dataset
To study the effect of mistranslating names on women and ethnic minority populations in the United States, we construct the DNIC dataset by collecting diverse set of names and English tence templates where matters. Instantiating these templates with different names let us isolate the effect of changing just the name. This builds on many existing datasets that seek to measure the potential harm caused by NLP systems across different social groups using tem- plates instantiated with different names, pronouns, or other variables (Levesque et al., 2011; Garg al., 2018: Cho al., 2019; Rudinger et al., 2018).
3 Approach Overview
translation particularly
In Figure 2, we provide a visual depiction of process flow for an example English sentence (tem- plate plus name) the context of our contributions. Our goal was to assess names get mistranslated at different rates across race/ethnicity and gender groups in realistic contexts. Thus, our steps were: 1) Names: We collected list of names strongly as- sociated with race/ethnicity and gender; 2) Insian- tiated English Sentences (Names plus Templates): We then situated the names in sentence contexts (i.e., templates) that we thought would be found in the "real world"; 3) Round-trip Translations (RT): Then we translated the complete English sentences (template + name) into four languages and back to English. We used RT translation to see if the final name in the translation output would be the same the original name, well whether the template portion remained in tact; Finally, 4) Translation Evaluation Approach: with our resulting RT trans- lations, we could ascertain the impact of the com- binations of race/cthnicity and gender of the names on the translations by assessing the strength and
4.1 Name Selection
We select names from birth records that are strongly associated with race/ethnicity and gender, as mea- sured by Weight of Evidence (Good, 1985).
We started with baby names list from the New York City Open Data "Popular Baby Names" web- site, which contained a record for all baby names by race/ethnicity and gender for births in 2019. This list gave us a total of 1, 935 first name records where each had the following attributes (fields): the gender of the baby from birth records (Female
'https: / ///ata/daaaaaa dityyffeeworr us /s/Heallle th/Popular Nabby-Name//2th-aaee

---

Male), the birth parent's² race/ethnicity (White, Black Non-Hispanic, Asian and Pacific Islander, Hispanic), and the number of babies for that name, gender and race/ethnicity combination.
Social Group
Gender
Race/Ethnicity
Name
WoE(E
Inaaya
17.77
AAPI
Tenzin
17.77
Since the same name can be given to babies of different gender and racc/ethnicity, we determine how strongly associated each name is with each social group by computing its weight of evidence (WoE) favor of a race/ethnicity and gender group. Good (1985) defines the WoE as how strong the ev- idence is in favor of a hypothesis. In our setting, the evidence is the name, and the hypothesis is the com- bination of race/cthnicity and gender and we ask how much evidence the name provides for social group (race/ethnicity and gender combination):
Joanna
17.73
Fatoumata
18.21
Black
Wynter
17.87
Reign
17.83
Female
Alaia
18.09
Hispanic
Valerie
17.40
Alexa
17.36
Chaya
18.31
White
Miriam
18.23
Rivka
17.89
Ayaan
18.64
AAPI
Mohammad
18.44
Arham
18.33
Nasir
18.29
Black
Malachi
17.95
Kenzo
17.91
Pnnam group)
Male
Angel
18.31
WoE(group name) A >log
P(nae -group)
Hispanic
Gael
18,27
Thiago
18.02
Moshe
18.41
This log-odds interpretation the weight evi- dence (Alvarez-Melis et al. 2021) is positive when the probability of the name evidence is higher under a specific group than under all other groups. Fol- lowing Good (1985), we associate any name with social group whenever WoE(group: name) 2 2. Three example names with highest WoE are shown for each social group in Table and overall statis- ties for the baby names dataset (including our WoF- filtered version) are given in Table 2
White
Chaim
17.89
Mordechai
17.73
Table 1: A sample of the names with the highest WoE for each race/ethnicity and gender group.
Unique Names
# Final Names
# Babies
F
M
F
M
F
M
AAPI
3370
4695
157
175
45
57
Black
3246
4418
172
168
94
66
4.2 Template Selection
Hispanic
7832
10566
254
264
113
117
White
12372
14197
383
362
231
211
Each template consists of an English sentence with placeholder to be replaced by a first name. We se- lect 16 templates (Table 3) where any of the names in our list can be inserted (i.e., lexical syntac- tic gender does not code for a specific name), and which represent contexts where names and their mistranslations matter: 1) everyday interpersonal interactions, 2) professional biographies.
Table 2: High-level statistics for the original baby names data as well as with Weight-of-Evidence filtering for values of two and greater (WoE >> 2).
as the personal identity and sense of importance of the named participants.
For personal interactions, we use 5 simple sen- tences from English-Italle Italian tourist phrasebook (DK, 2017) as well tence as another simple source of comparison. In these contexts, mistranslating a name has the poten- tial to negatively impact communication, as well
We use English sentences from the professional biographies data set associated with (De-Arteaga et al., 2019). In these contexts, mistranslating a name negatively impact professional commu- nication, as well as the named person's recognition and lead to socioeconomic harms.
manually created sen-
'The original names data terms this the "Mother's ethnic- ity:: we assume that their use "Mother" is equivalent the "birth parent" and reference as such.
Finally, the DNIC dataset is constructed by in- stantiating each of the 16 templates with the 1,935 name records selected. yielding a total of 179,328 English sentences containing name in context, where each name is strongly associated with a com- bination of gender and race/ethnicity.
"The number of babies in the original names list data was thresholded such that only those with a count of 10 or greater (for name, gender and race/cthnicity combination) were in- cluded, which is why there appear to be far more male than female babies (apparently female bahies have rarer names).

---

Template
were selected represent diffffernt language fam- ilies, writing systems and typology. They are all high resource languages, for which we expect MT quality to be good enough on average to be use- ful, as indicated by BL.UU scores on the FLORES benchmark (Figure 4).
1 They never go anywhere without
2 Pleased to meet you
3 My name is ----
4 Where are you from
5 What would you like to
6 is an Associate Professor of Development Stud- ies at Nelson Mandela University in South Africa. is reputable dentist with over 20 years of ex- perience helping patients of all ages care for their smiles through top-quality treatment.
7
5.2 Evaluating Name Translation Quality
Evaluating the quality of name translation raises some specific challenges in addition to all the dif- ficulties that come with MT evaluation in general settings. Different people might have different cri- teria for defining what constitutes acceptable translation of their name. Some might use the ex- act same name in the two languages. Others might want to see specific diacrities in Spanish but not English, might expect their Chinese name to use specific characters, or might even use an entirely different first name in the target language. As a result, name translation quality cannot be directly evaluated by checking whether MT outputs match reference translations written by a third party. In- stead, we propose round trip translation approach to estimate name translation quality automatically, without collecting first person judgments.
8
is Software Engineer by passion and profes- sion.
underground magician whose technical sleight of hand prowess is weseee-respected by ma- gicians in the inner circle.
10
Orthopedic Surgeon (Hand Surgeon) in Laredo, TX.
11
painter who approaches the medium formal exercise.
a British poet of foree, clarity and honest craft. --- is a Jungian psychotherapist with nearly 30 years clinical experience.
an interior designer who has specialized historic restorations and consultation.
is a behavioral nutritionist spcializing the carb and keto lifestyle.
is Registered Dietitian with a graduate degree nutrition and wellness.
Table 3:
The 16 templates chosen through manual re- view to have selection with varying lengths and lan- guage usage. Phrase I was manually created, phrases 2-5 were from the tourist phrase book, and phrases 6- 16 came from the bios dataset.
Round Trip Translation. We evaluate the qual- ity of each MT output on the DNIC dataset by translating it back into English using the exact same system used for the forward translation pass, and by comparing the resulting Round Trip (RT) output with the original input using different metries.
5
Methodology
We now describe how we use MT to translate the DNIC dataset (Section 5.1), and introduce our ap- proach to automatically evaluate the resulting name translations (Section 5.2) and conduct statistical analysis of the impact of social groups name translation quality (Section 5.3).
While RT translation is rightly considered be reliable approach to evaluate translation qual- ity the general case (Somers, 2005), it is well suited to our specific use case for several reasons. First, although roundtrip translation does not en- able us to detect in which direction an eтror was introduced, any error that shows up in the round trip must have been caused by (at least) one error in one direction the other. Thus, errors detected through RT translation represent lower bound on the actual error rate. On the other hand, a name comes back correctly in the round trip, it could be that it was correctly translated in both directions, or it could be that two errors conspired to lead to no error. So, while we may not pinpoint where any translation errors may occur with the roundtrip evaluation approach,
5.1 Machine Translation Settings
To get representative sample of real-world MT quality, we use three different MT systems: two widely used online translation services (Google T and Microsoft Translator") and the OPUS MT open m based on Marian MT?.
We translate the English sentences from the DNIC dataset into four diverse languages: Arabic, Chinese, Russian and Spanish. These languages
rence/rest htttss//ww...irrrsttt...//r/rrrs
are able to deduce the minimum error rate of a name.
usiness/translator-app com/Helsiinke PLPppupsM
/https://github. hwt.ttuuu..oo/..eeeeeeeeee
Finally, the computation of translation accuracy and quality in the final translation output relative
thttps://githhhe cato.ccom//aaaaa / ian- intariiaananttmiannnnnn

---

the template would be straightforward and reasonably accurate. not requiring knowledge of non-English languages nor human review of each translation. In essence, round-trip evaluation allowed us to provide the closest approximation of an error rate possible for name translations using MT. Manual evaluation of name translations in each direction, on the other hand, would require significant amount of expertise and labor.
Odds Ratio
Coefficient
B
p
(Intercept)
3.92
1.37
0.012
9
IsFemale
0.92
-0.08
0.000
22
IsAAPI
1.20
0.18
0.000
23
IsBlack
0.93
-0.07
0.004*
24 IsHisp
1.21
0.19
0.000
d1 IsFemale A ISAAPI
0.87
-0.13
0.001
d2 IsFemale A IsBlack
0.69
-0.37
0.000
Name Translation Accuracy. We count a name as translated correctly if and only if the RT transla- tion contains a token that is a case-sensitive match to the original name from the names list.
d3 IsFemale A IsHisp
0.62
-0.47
0.000
Table 4: Results of Logistic Mixed Effects Regression with Name-Exact-Nee Exact-Match Outcome Variable. Fixed ef- fects shown. We also controlled for the random effects of template id, translation language, and MT system.
BLEU To capture the impact of name mistrans- lations on the rest of the sentence, we computed Sacrebleu's (Post, 2018) Mantence-BLEU using de- fault settings to score the transkttanllsions against the original English sentences.
Here, capitalized letters indicate the indicator vari- ables and lowercase variables represent estimated parameters. is a fixed intercept, and the outcome is either name exact match (in which case the link function f is the logistic function) or sentence- BLEU (for which f is the identity function).
5.3 Measuring the Impact of Social Groups with Mixed Effects Models
We aim to estimate the effect of the social group (gender and race/ethnicity) associated with a name on our relevant outcome variables-name accu- racy and sentence- LontenceBB..FU-WWA - controlling for all other effects: MT system, pivot language, and tem- plate. We adopt a parametric approach using gener- alized linear mixed effects models for this analysis, where we model the outcome as a linear combi- nation of variables passed through a link function f. We represent each input variable as a binary indicator of gender (female or not), race/ethnicity (AAPI or not, Black or not, Hispanic or not), MT system, pivot language, and template id as":
In our analysis, we treat the social group vari- ables (gender g. race/cthnicity 2; and interactions d,) as fixed effects (colored purple) and the inciden- tal variables (MT system 8;, pivot language I; and template id f4) random effects (colored green). This choice is based on the definition for fixed and random effects by Searle, Casella, and McCul- loch that "effects are fixed if they are interesting in themselves or random if there interest the underlying population" (Searle et al., 1992). our setting, we were most interested in the effects of gender and race/ethnicity on our outcome variables and therefore consider these our fixed effects. We considered all other variables random effects since expected them randomly impact transla- tion quality and accuracy and wanted control for their effects in our model.
y [(b +
gG
22 + Z3 + 24 Z4
Baby Gender
Birth Parant's Etriclly
+ di GxZz + Gx + dg GxG2
Interactions betweon Gorder and Eltniciticcty
S2S2 S3 + L2 + LzLg +
4L4
Results
MT Systems
Pivat Languages
+ T2 + + T16Ti6
To answer our primary research question, "Are there widespread disparities in machine transla- tions of names across race/ethnicity, and gender?". we analyzed our results to determine if names are mistranslated at higher rates for certain groups, and investigated any accompanying translation quality degradation of the entire sentence.
Template
"except "effective order =True" which was necessary centence-BLEUU calculations versus corpus-BLEU "To avoid a redundant encoding, our reference (or "base") variables were: Male for Gender, White Non-Hispamic for race/cthnicity, Google for MT system, Spanish for language, and template for template id. Effects are relative to these.

---

Odds
(IsFeealee A IsAAPI). This adjustment shown in Table 5. Here, we see that Black and Female- associated names will be mistranslated about 11% more frequently than baseline, 31% percent more for Hispanic=Femalleasssiiaae names, and 3% more for AAPL+fcmalc-assooiin names.
Full Effeet
Ratio
B
% Diff
Black Female
0.59
-0.53
-40.91
AAPI Female
0.97
-0.03
-3.14
Hispanic Female
0.69
-0.36
-30.51
The fill effect takes into account the combined effect of each contributing coefficient for a race/ethnicity and gender combination and is case- case-sensitive match. % Difference is calculated relative to White males (Odds 1). the reference gender and race/ethnicity group.
Table 5:
6.2 Overall Translation Quality
Beyond the (mis unis)translation the name alone, we wanted to see how overall sentence translation qual- ity varied for the dataset records corresponding to each social group, based on the name and its com- bination with the name attributes of gender and race/ethnicity (as measured by sentence- (ettence-BBBU),. We show some example sentences in Appendix Ta- ble of degraded translation quality for sentences with Black female-associated names. The results of the linear mixed effects regression are shown in Table Here, we see similar patterns as in the name translation accuracy, but with smaller effect sizes (as expected, since BLEU is average over many words, mostly correctly translated).
Coefficient
B
p
b (Intercept)
22.81
0.467
9 IsFemale
0.10
0.469
22 IsAAPI
0.25
0.225
23 IsBlack
-0.25
0.204
24 IsHisp
0.32
0.047*
d1 IsFemale A IsAAPI
-0.22
0,464
d2 IsFemale A IsBlack
-0.83
0.001 *
d3 IsFemale A IsHisp
-0.77
0.001 *
Table 6: Results of Linear Mixed Effects Regression with Sentence-BLen BBEU Outcome Variable. We show the coefficients the fixed effects including gender, race/ethnicity, and the interactions between them. We also controlled for the random effects of template id, translation language, and MT system.
We significant (p 0.05) effects for Hispanic-associated names (BLEU increase of 0.32, p = 0.047). as well as for Female+ associated names (BLEU decrease 0.83, p 0.001) and for Femmle-+Hispanic-assoiine names (BLEU decrease of 0.77, p = 0.001). Similar to the adjustment in the case of name translation, the over- all effect for Female and Black-associater names is
6.1 Correctness of Name Translation
Table 4 shows the estimated odds ratios (and corre- sponding coefficients and p-values) for each of the fixed effects in the logistic mixed regression anal- ysis of name translation accuracy. Here, we see that female-associated names (g) had significant negative effects (Oddsratio = 0.92,3 = -0.08, p- 0.00) on the odds of having a correct name translation in the round-trip translation; in terms of name
1.08 BLEU points, and for Female and Hispanic- associated names 00.45 BLEU points.
6.3 Average BLEU and Name Translation Accuracy Across Systems
In analysis consider the machine transla- tion system (as well the pivot language and the template) be random effects, and therefore do not present results related to the quality of each machine translation system on our task. Nonethe- less, it is of potential interest how well each of these systems performs on our data. In terms of name translation accuracy, Google Translate has an average accuracy of 73%, Marian has 70%, and Microsoft Translator has 78%. In terms of BLEU, Google Translate has a score of 55.92. Marian has 40.72, and Microsoft Translator has $5.20 (com- pare to 41.01, 41.01, 33.28, and 37.71 respectively on the FLORES benchmark Appendix Figure 4).
odds ratio, the odds of mistranslation is eight percent greater than that of male-associated names. Furthermore, the odds of an A Allaassociated name being translated cor- rectly is 20% higher than baseline (Odds Ratio 1.2,3 = 0.18, p >= 0.00).
The largest effects we see are the intersectional effects, which are exceptionally pronounced for Black and female-associated names and Hispanic and female-associated names. However, these have to be adjusted for the fact that, for instance, the effect of an AAPI- and female-asson associated name will be the sum of g (IsFemale), 21 (IsAAPI), and

---

6.4 Qualitative Insights related to Gender and Ethnicity in Roundtrip Translations
Mistranslation Rate
95% Cis (n=75 by Group)
066
Hispanic Female
Based on the results above, we investigate what types of errors there are at a more fine-grained level than just names translated correctly versus not. To perform this analysis, manually coded a randomly chosen sample of 300 names, split be- tween our different social groups. Two were cho- sen to be "hard" for machine translation systems based on the previous results-Blas Female- and Hispanic Female-associated names-and two were chosen to be "easy" Hispanic Male- and AAPI- associated names. In Figure 3, we show estimates of the prevalence of each type of mistranslation for Black Female and Hispanic Female-associated names (the name types that were more difficult to translate) with rate fluctuation ranges based on a 95% confidence level. We do not observe a system- atic difference between the "hard" and "easy" to translate social groups in terms of the distribution of types of errors, thus we show only the former.
Q4
Black Female
8o
Variant
Shared Sounds
Common Noun
Case Error
Missing Name
Other
Category
Figure 3: The mistranslation rate a rough estimate based on the number of translation emrrr manually coded as a given category (x-axis) divided by the to- tal number of randomly chosen mistranslated names for the social group (only the Black Female and Hispanic Female shown here since other categories had similar distributions of error types).
do Prince + "What princes do you want to be", and "Miracle is a software engineer..." + "The miracle is a software engineer...').
A plurarility of the errors is variant of name cases when the name was translated to slightly dif- ferently spelled variant of, arguably, the same name (e.g., Hanna/Hannah, Mohammed/Mohamad, Ami- rah/Amira). For these, either only a few characters were different, or it was labeled as such based on our cultural knowledge of name vo
A similar percentage of errors are case-errors (but not common nouns). This is group in which the name missing entirely (e.g.. "My name is Eve" - "I'm summer night"). Occasionally the output contains non-Latin characters, again, likely due to a pivot through language which uses a non-Latin script.
The next most common type of error is names that were translated as other names with some sounds in common, but which we did not judge to be common name variation (e.g.. Kamiyah/Camia, Brielle/Preel). These likely occur largely because of transliteration into non-Latin scripts with different phonetics.
Conclusion
We explored how race/ethnicity and gender im- pact both the odds of obtaining a correctly trans- lated name as well as the translation quality of a sentence. To accomplish this, we introduced a roundtrip-translation- based approach to evaluate the machine translation of names in context, as well as a dataset of Diverse Names in Context based on names highly associated with birth-derived gender and birth parent race/ethnicity.
A smaller category of errors includes words that are translated into common nouns, which often has small spillover effects into the translation of the of the sentence (e.g.. "What would you like to
IIIn our results, most common type of error is a name variant, which could potentially be perceived as unimpor- tant. There is substantial evidence that even small name vari- ations (including misspellings) can cause harm. Freedberg (2002) observes names don't 't have accents or the tildes, they are not spelled correctly". Lieberson and Bell (1992) note that "chonsing a name an inherently social pracess" influenced by a wide range of personal considerations, and "(name) choices are increasingly free to be driven by factors of the choices will also reflect differences in taste be- tween subsets of the population". Spelling of names is also often very closely tied to social groups; for example, Camp- bell (2023) observes that "Given names used by Black people are often invented or rreatively-spelled variants of more tra- ditional names." Overall, these translation errors are harmful and socially contingent and important to correct.
Our results showed that female-associated names are likely to be mistranslated at significantly higher rates than assecaasoceated names. Our intersec- tional analysis further demonstrates that Black Female- and Hispanic Femalc-associated names are mistranslated 30-40% more frequently than White
names. Furthermore, Black and Hispanic Female-associated names asso- ciated with significantly worse overall translation quality as measured by BLEU.
These results hold, even when controlling for

---

the template, MT system, and pivot language. We have not performed a root cause analysis to determine from where the name translation dispar- ities we observed arise, but expect that they may arise in part due to names of certain social groups being underrepresented in the training data upon which machine translation and large language mod- els are trained (Weidinger et al., 2021). In addition, proper names, cross- proos-iinguisticallyy may have (for instance) inflectional forms by gender, or derivative forms originating from verbs; the way these forms are reflected in the training data may also influence mistranslation.
The gender categories (only male/female) and racc/ethnicity categories are significantly limiting. Furthermore, by focusing on names babies born in the United States, in addition to a US- -enttrii analysis, the analysis here erases the experience of immigrants. We expect that if anything name translation errors are higher for immigrant's names (unless they choose an American name) due to the social trends of anglicization of names in the US per Pinsker (2019). "in general, the names immigrants give their children go through three stages; from names in the original language to universal names, and finally to names in the destination-country language."
This work highlights the need mitigate the bias that vulnerable social groups experience both per- sonally and economically by seeing their names in- correctly displayed in translations. Despite the po- tential for harm arising from mistranslating names, well as the fact that names are often translated differently from other word types, there has been relatively little work in machine translation that fo- cuses specifically on names. (Maurel et al., 2007) is one example, which focuses specifically on the automated translation of rare proper names. Rec- ognizing that some names should be translated and some should be transliterated, (Hermjakob et al., 2008) build classifier to predict which is which, and to apply a different model for the two types.
The second major limitation is that both evalua- tion measures we have are at best proxies for real harms incurred by people. Name translation error and BLEU score capture important intrinsic prop- erties of the translation quality, but do not directly speak to the allocational or representation harms suffered as result of those mistranslations.
Finally, it is worth noting that the templates do not fully represent the full variety of sentences that
appear in real-world contexts. This is a standard limitation of template-bas based datasets. We chose our sentence templates in light of our harms, and specifically to align to scenarios where names mat- ter and to be independent from the demographic variables considered, including ensuring gender neutrality of the sentences. Our statistical analysis through mixed-effects model, where the template is treated as a random variable, also reflects the nology has exacerbated harms for vulnerable popu- lations (e.g., Weidinger et al., 2021; Bender et al., 2021; Blodgett et al., 2020; Savoldi et al., 2021), suggest that lower translation accuracy for names of people from vulnerable social groups risks their experiencing both allocative harms (Crawford, 2017; Barocas et al., 2017)-sud as poorer quality MT system service or poorer professional com- munications when MT systems incorrectly translate their names (or sentences including their names) and representational harms such as erasure of the name and its associated culture (Katzman et al., 2023).
We hope that this work highlights the danger that name mistranslations could have resulting in long- term harms (section 2). Given our findings, and in light of historical evidence that language tech-
fact that this is a non-exhaustive, non-random set of possible templates.
Ethics Statement
In this paper, we emphasize the importance of ethi- cal machine translation that considers intersection- ality, i.e., social groups defined but not limited by the characteristics of gender and race/ethnicity. We highlight the fact that MT technologies may rein- force structural racism if a community we do not acknowlede wedge and attempt to mitigate harms such as lesser quality of service, denial of self identity, and erasure of a name and its associated culture. We find that the use of mate-of-the-an MT tools as seen in the mistranslations of names and their contexts incquitably impacts vulnerable social groups With respect to data considerations, we note that our first (given) names list was made public by the City of New York and therefore is open access, as
Limitations
Our analysis is limited by the nature of the baby names data we used: birth parent's ethnicity is not the same as child's ethnicity, and gender assigned at birth is not the same as the child's gender.

---

are the biographies we utilized for our templates; however, the tourist phrases were sourced from tourist phrase book but do not contain sensitive in- formation. Our names list is a reasonably diverse representation of the population of New York City, but is not representive of the U.S. nor world popula- tion whole. Finally, for a name to be included in the names list, it must have been both first name well as been given to ten more babies for that name, race/ethnicity and gender combi- nation. Therefore, there is little risk of revealing personally identifiable information.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in nip. arXiv preprint 2005:20005.40500
Michael D. Campbell. 2023. Behind the name: the etymology and history of first names. https: / /ww w. behinnaeename.... com/. Accessed: 21 October 2023).
Won Ik Cho. Ji Won Kim, Scok Min Kim, and Nam Soo Kim. 2019. On measuring gender bias in translation of gender-neutral pronouns. In Proceedings of the First Workshop on Gender Bias in Natural Longuage Processing, pages 173-181, Florence, Italy. Associa- tion for Computational Linguistics.
Acknowledgments
Combahee River Collective. 1977. The Combahee River Collective Statement. Retrieved from the Li- brary of Congress.
First, this work was supported in part by the NSF Fairness in AI Grant 2147292, by the Google Award for Inclusion Research Program, and an NSF Computing Innovation Fellowship (latter for Jieyu Zhao). We thank these organizations for making work such ours in faimess/bias research possible.
Combahee River Collective. 1983. The Combahee River Collective Statement. Home Girls: A Black Feminist Anthology, 1:264-274.
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint wrXiv:2207.06672
We greatly appreciate the time and care our re- viewers, program chairs and areas chairs put into reading our paper and providing constructive feed- back. In addition, we would like to thank Dr. Peter Rankel for his helpful feedback in the early stages of our development of our methodology, as well as the University of Maryland's Computational Lin- guistics and Information Processing Lab (CLIP) for feedback on our paper draft.
Kate Crawford. 2017. The trouble with bias. Con- ference on Neural Information Processing Systems (NeurIPS).
Kimberlé Williams Crenshaw. 1989. Demarginalizing the Intersection of Race and Sex: Black Femi- nist Critique of Antidiscrimination Doctrine, Fem- inist Theory and Antiracist Politics. University of Chicago.
References
H. Samy Alim, John R. Rickford, and Ametha F. Ball. 2016. Raciolinguistics: How Language Shapes Our Ideas About Race, Oxford University Press.
Maria De-Arteaga, Alexey Romanov, Hanna M. Wal- lach, Jennifer T. Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Cem Geyik, Krishnaram Ken- thapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in high-stakes setting. Proceedings of the Conference on Fairness, Accountability, and Transparency.
David Alvarez-Melis Harmanpreet Kaur, Hal Daumé III. Hanna M. Wallach, and Jennifer Wortman Vaughan. 2021. From human explanation to model interpretabilityy: A framework based on weight of ev- idence. In AAAI Conference on Human Computation & Crowdsourcing.
DK. 2017. Eyewitess Travel Phrase Book Italian. DK Publishing, London.
Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Al- locative versus representational harms in machine learning, In Proceedings of SIGCIS, Philadelphia, PA. The Special Interest Group for Computing. Infor- mation and Society.
Rashmi Dyal-Chand. 2021, Autocorrecting for white- ness. Boston University Law Review, 101:191.
Louis Freedberg 2002. Claim your name. https: //ww w. www.sstte.oo/ppiiiiaeo. wtt.m//ppiniionaatttlle//eerraoa spective-cllaim-oorr-eee 2764-m2452222222.... php.
Emily M Bender, Timnit Gebru, Angelina McMillan-
Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2027 ACM confer- ence on fairness, accountability, and transparency, pages 610 610-623.
Sahaj Garg. Vincent Perot, Nicole Limtiaco, Ankur Taly, Fdd H. Chi, and Alex Beutel. 2018. Counterfactual faimess in text classification through robustness. Pro- ceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society.

---

LJ Good. 1985. Bayesian Statistics 2, Weight of Evi- dence, A Brief Survey. Elsevier Science Publishers B.V. (North-Holland), Cambridge, UK.
Rachel Rudinger, Jason Naradowsky, Brian Leonard. and Benjamin Van Durme. 2018. Gender bias coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Pupers), pages 8-14, New Orleans, Louisiana. Association for Computational Linguistics.
Ulf Hermjakob, Kevin Knight, and Hal Daumé III. 2008. Name translation in statistical machine translation- learning when to transliterate. In Proceedings of ACL-08: HLT, pages 389-397.
Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Mat- teo Negri, and Marco Turchi. 2021. Gender bias in machine translation. Transactions of the Association for Computational Linguistics, 9:845-874.
Robin Jeshion. 2009. The significance of names. Mind & Language, 24:370-403.
Jared Katzman, Angelina Wang, Morgan Klaus Scheuer- man, Su Lin Blodgett, Kristen Laird, Hanna M. Wal- lach, and Solon Barocas. 2023. Taxonomizing and measuring representational harms: A look at image tagging. ArXiv, 2tb//2355.0176..
S.R. Searle, G. Casella, and C.E. McCulloch. 1992. Variance Components. Wiley, New York.
Harold Somers. 2005. Round-trij trip translation: What is good for? Proceedings of the Austrulasian Lan- guage Technology Workshop, pages 17,-133 Syd- ney, Australia.
Rita Kohli and Daniel G Solórzano. 2012. Teachers, please learn our names!: Racial microugressions and the K-12 classroom. Race Ethnicity and Education, 15(4):411 462.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Ballc, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models, arXiv preprint arX:v:2112.04359
Hector J. Levesque, Ernest Davis, and L. Morgenstem. 2011. The Winograd Schema Challenge. In Interna- fional Conference on Principles of Knowledge Rep- resentation and Reasoning.
Stanley Lieberson and Eleanor O Bell. 1992. Children's first names: An empirical study of social taste, Amer- ican Journal of sociology, 98(3):511-554
Denis Maurel, Duško Vitas, Cvetana Krstev, and Svetla Koeva. 2007. Prolex: a lexical model for translation of proper names. application to french, serbian and bulgarian. Bulag-Bulletin Linguistique Appliquée et Générale, pages 55-72.
Shubhanshu Mishra, Sijun He, and Luca Belli. 2020. Assessing demographic bias in named entity recogni- tion. ArXiv, abs/2008 03415.
Manuel A. Pérez-Quifhones and Consuelo Carr Salas. 2021. How the ideology of monolingualism drives us to monolingual interaction. Interactions, 28:66- 69.
Paul Pharr. 1993. Onomastic divergence: study of given-name trends among african americans. Ameri- can Speech, 68:400-409.
Joe Pinsker. 2019. American immigrants and the dilemma of white-sounding" names. The Atlantic.
Matt Post. 2018. A call for clarity in reporting BLEU In Proceedings of the Third Conference Machine Translation: Research Papers, pages 186- 191. Brussels, Belgium. Association for Computa- tional Linguistics.
Jonathan Rosa and Nelson Flores. 2017. Unsettling race language: Toward raciolinguistic perspective. Language in Society, 46(5):621-64)

---

eng-spa MT Quality against FLORES 200 Reference Translations
Sentence- Seinnc--iecce Piee- BLEU[Corpus-Leve)
Google
Marian
Microsoft
NLLB-200
ara-eng eng-ara arb
eng-rus
rus-eng
spa-eng
eng-zho
zho-eng
rus
spa
zho
Translation Direction
Figure 4: Translation quality of MT systems considered on the the FLORES 200 benchmark, as measured by Sacrebleu's sentence-piece corpus-level BLEU.
A Benchmarking MT Systems on the FLORES 200 Reference Translations
We ran Sacrebleu's sentence-piece corpus-level BLEU against Meta Al's FLORES 200 human-translated benchmark dataset (Figure 4) to gain insights into the variation sentence-piec BLEU scores by NMT system and language (Costa-jussà et al., 2022). Similar to the benchmark sentence-piece BLEU scores, with our data we generally saw that Google outperformed Microsoft and Marian, with Marian machine translation having the lowest sentence- BLEU scores. The ranges for the benchmark sentence-piece BLEU scores were about 33 for Marian 41 for Google, whereas sentence- sentence-BLEU scores ranged from about 41 for Marian 56 for Google.
Why Mixed Effects Models: Additional Explanation
We utilized mixed effects models for couple of reasons. Most critically, believed that the variation in sentence-BLEy and name translation accuracy would be distinct for different groups (ex. for White females versus Black males) and explained by both fixed and random effects variables. In addition, we could not sure that our observations were independent; in particular, we had multiple observations for each template, our unit of analysis. For example, "Camila is painter who approaches the medium as formal exercise" was the template for multiple observations where the observations differed in that they each had a unique combination of translation language, NMT system, and template. Therefore, the different values of sentence-BLER and the "name exact match" outcome measures for each template could be correlated!"
Model specification
Both models shared the same list of fixed effects and random effects, with the fixed effects being race/ethnicity and gender and the random effects being the MT system, language, and template. Our analysis dataset for input our models consisted of the outcome variables (the evaluation metrics) and the indicators for each our predictors, where each was category the variable (dimension). For example, the gender variable was 1 for a name with a female attribute and for a name with a male attribute.
hhtttps://githbb. com/ htttss////tthuu...oo/sshheeeeess M-MixedLM

---

Input Sentence
Round-trip Translation
Mistranslation Type
Pleased to meet you Amiyah
I'm happy to meet you Amiya is respected dentist with over 20 years of experience helping patients of all ages take of their smiles with high qual- ity treatments.
Translated common noun Variant of name
Amiyah is reputable dentist with over 20 years of experience helping patients of all ages for their smiles through top- quality treatment.
Amiyah is an underground magician whose technical sleight of hand prowess is widely- respected by magicians in the inner circle. Brielle behavioral nutritionist specializ- ing in low-carb and keto lifestyle.
Amiya is underground mage whose tech- nical sleight of hand is highly respected by mages from the inner circle.
Variant of
Precl expert behavioural nutrition
Name with shared sounds
specializing in low-carbohydrate kito lifestyle.
Brielle a British poet force, clarity honest craft.
Briel a British poet of strength, clarity and honest craftsmanship.
Variant of name
Where are you from Jewel
Where are you from? Who's who?
Missing name entirely
Jewel a behavioral nutritionist specializ- ing in low-carb and keto lifestyle.
Jewell behavioral nutritionist specializ- ing in the low carb and keto lifestyle.
Variant of name
Journee a British poet of force, clarity and honest craft.
Girls are a British poet of strength, clarity,
Translated a common noun
and honest craft.
Pleased to meet you Journce
Nice to meet you Journyy
Variant of name
Table 7: Examples of Degraded Translation Quality for
associated Names
D Model and Implementation Details
Binomial Mixed Effects Logistic Regression. For this model, we regressed our binary outcome variable for translation accuracy, the "name exact match", on the fixed effects and random effects explanatory variables as listed. The name exact match variable had value of "0" for non-exact match of the original name to the roundtrip translation and a "1" for an exact match of the name.
Linear Mixed Effects Regression. For this model, regressed our continuous outcome variable for translation quality, sentence- BLEU (ranging from 0-100) scoring the round-trip translation relative our reference (the original English sentence template) on the same set of fixed effects and random effects explanatory variables as above.
Execution of our Models To execute the models, we utilized the linear mixed effects regression package (]mer4) and the generalized mixed effects regression (glmer) package (for the logistic mixed effects regression) from the statistical computing software R. For both, we were able specify random effects variables with non-nested (non-bierarchical) relationships with each other, specifying that intercepts should vary (be random) amongst each random factor. This was important given how we developed our data set; most variables were crossed and non non-nested nested for each template, our unit of analysis.


Reviews:
[{"review_id": "IpfyYMBxnB", "reviewer": "Reviewer_QCvK", "summary": "", "questions": "- Question A: Are name variant errors equivalent to translation errors?\n- Question B: How are the race categories determined in the NYC baby names data? Do certain categories have more internal diversity?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, Area Chairs, Senior Area Chairs and Program Chairs,\n\nThank you for the careful review and constructive feedback. We address the main comments and questions below.\n\nReviewer QCvK (your paraphrased comments below followed by our responses):\n\nComment 1 (“Issues with error measurement”): \nReviewer noted that most translation errors were variants of the same name and wondered (since they are potentially linguistic artifacts) if name variants should be considered errors (reviewer question A); as such, the reviewer also recommended considering non-exact match errors (eg. phonetic/edit distance based).\n\nResponse to Comment 1 and Question A: Our position is that name variant errors are equivalent to translation errors. We chose to evaluate errors relative to the exact translation of United States (US)-assigned baby names from birth records since exact spellings are frequently associated with cultural identities of racial/ethnic subgroups and possibly personal preference. We acknowledge that different people may accept different spelling variations but we have no way of knowing these differing preferences or the desired translation system behaviors without conducting a human study to gather these preferences. Your questions and the suggestion to include non-name exact matches are helpful; we agree that such a measure will be a useful comparison for our later version of the paper.\n\nComment 2(“Non-uniqueness of roundtrip mapping”): The reviewer basically asked why round trip machine translation (MT) is suited to our evaluation versus translation of source to target language questioning specifically why the use of alternative acceptance criteria for name translations in target languages would not also pose an issue with round trip MT. They also suggested we narrow to a couple of languages to explore those languages’ conventions for first-name translations, pointing us to additional references.\n\nResponse 2: We utilized a round-trip evaluation approach because it allowed us to provide the closest approximation of an error rate possible for name translations using automatic translation, i.e. MT. Manual evaluation of name translations in each direction would require a significant amount of expertise and labor. While we note (see Section 5.2 Round Trip Translation) that it is difficult to pinpoint where any translation errors may occur with this approach (in the forward pass or the backward pass of the round trip translation), since we use a mixed effects regression approach to assess translation accuracy and translation quality, we are able to focus primarily on gender and race/ethnicity (our fixed effects) on name translation errors. In particular, we control for the error contributed by template, pivot language, and MT system, so that we can isolate the fixed effects and their intersections (interactions) on name translation. However, your suggestion to explore conventions for name translations in other languages (which potentially impact the overall roundtrip translation error rate) might shed light on the causes of the errors, and thus assist with future mitigation of name mistranslations. We thank you for the additional suggestion and references. \n\nReviewer Question B: How are the race categories determined in the NYC baby names data? Do certain categories have more internal diversity?\n\nResponse to Reviewer Question B: Race/ethnicity for a baby name is based on that documented for the birth parent from NYC governmental birth records. We are not clear on what specific information you seek in the second question, but please reference Section 4.1 Table 2 in case you are looking for the distribution (count) of names by race/ethnicity and gender categories. \n\nMissing References: We will include the recommended references. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Good statistical modeling and analysis: This quantitative methodology for mixed effects modeling, both classification and regression, provides a good example of measurement of bias across related conditions/system for MT models .\n- New dataset for names and contexts to measure MT system biases: The dataset is generated from a New York City baby names dataset, with demographics derived from log-odds associated with the conditional likelihood of a name belonging to a group. Sentence templates are selected from scenarios that are unlikely to have demographic dependence (interaction and biographies) making this a well-designed dataset.", "reasons_to_reject": "- Issues with error measurement: The qualitative analysis of errors (sec 6.3) indicates that a majority of errors are due to a variant/spelling of the same name (eg. Hanna → Hannah). It’s unclear whether these should qualify as an error since this may be an artifact of different phonemic natures of the orthography of English and the intermediate language. The authors should consider using non-exact match errors (eg. phonetic/edit distance based) as the focus is on the name translation errors.\n- Non-uniqueness of round trip mapping: Related to the error measurement, the authors claim that name translation quality cannot be directly evaluated by checking source→target MT as target languages may use alternative acceptance criteria for name translations. It is unclear why this same critique does not extend to the round trip task that is the focus of the paper. The authors may consider reducing the variety of target languages and consider the delving deeper into language specific conventions for translations of proper names (eg Hermjakob et al. (2008), Maurel et al. (2007)).", "questions_to_authors": "- Question A: Are name variant errors equivalent to translation errors?\n- Question B: How are the race categories determined in the NYC baby names data? Do certain categories have more internal diversity?", "paper_topic_and_main_contributions": "This paper carries out an in-depth analysis of the biases in translation quality of contexts and names associated with different gender and racial/ethnic groups. Specifically, the authors use a dataset of baby names and associated demographic data, and embed each name into different sentence context templates. They then quantify the disparity of error rates in round-trip (English → Intermediate Language → English) translations of each sentence while controlling for intermediate language, translation method, and sentence template. They provide quantitate (mixed-effect model coefficient) and qualitative error analysis of error rates in absolute match and BLEU scores associated with name gender, racial/ethnic groups.\n\n**Contributions:**\n\n- A dataset for testing demographic biases in machine translation named DNIC (Demographic Names in Context).\n- A mixed-effects model setup for quantifying the demographic biases across MT systems.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Hermjakob, Ulf, Kevin Knight, and Hal Daumé III. \"Name translation in statistical machine translation-learning when to transliterate.\" *Proceedings of ACL-08: HLT*. 2008.\n- Maurel, Denis, et al. \"Prolex: a lexical model for translation of proper names. Application to French, Serbian and Bulgarian.\" *Bulag-Bulletin de Linguistique Appliquée et Générale* (2007): 55-72."}}, {"review_id": "HrQNgVT2PT", "reviewer": "Reviewer_EGZQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, Area Chairs, Senior Area Chairs and Program Chairs,\n\nThank you for the careful review and constructive feedback. We address the main comments and questions below.\n\nReview 2: Reviewer EGZQ (your comments below followed by our responses)\n\nComment 1: The paper says that these mistranslations are due to \"the systems' tendencies to standardize language to predominant language patterns.\" This is a very strong causal interpretation of the results. Is that really justified by the evidence presented? It could be that these mistranslations are due to the relative frequencies of these patterns in the MT system training data.\n\nResponse to Comment 1: Thank you for describing your interpretation of our statement. Our reading of your description of the likely reason for the translation errors is actually quite similar to the intended meaning of our statement, but we understand that you feel it is strongly worded. As background, subword segmentation such as Byte-Pair Encoding will produce representations or embeddings of the most frequently found patterns in the parallel corpora used to train neural MT systems. To clarify our statement, by “standardize language” we mean that name-related subword representations from the English training corpora are more likely to be from the majority social groups represented, so the roundtrip translations may have a tendency to output these more common names erroneously in cases where the original names were associated with less-represented social groups in the training data. As stated by (Weidinger et al., 2021) “...training data can be biased because some communities are better represented in the training data than others. As a result, LMs (language models) trained on such data often model speech that fails to represent the language of those who are marginalized, excluded, or less often recorded”. \n\nComment 2: The paper also says these mistranslations have \"significant implications for people’s professional, personal and cultural identities, self-worth and ease of communication.\" That is another very strong causal interpretation of the results. Is that really justified by the evidence presented?\n\nResponse to Comment 2:  \nTo clarify, we do not argue that our Results section nor Conclusion show that these harms are directly having these effects on a specific group of technology users; in order to conclusively show that, we would need to conduct a before-and-after longitudinal study following a representative sample of people from different social groups over time to understand the impacts of their experiences with MT system name translations.\nIn the particular statement you reference from our abstract, we are instead suggesting, given our results, that name mistranslations could have long-term adverse impacts in the form of harms as outlined in Section 2 Socio(linguistic) Background. To recap, in the Results section, we report lower translation accuracy and translation quality scores for names associated with females particularly Black and Hispanic females. Given these results and in light of historical evidence that language (including MT) technology has exacerbated harms for vulnerable populations (Weidinger, et al., (2021), Bender, et al., (2021), Blodgett et al., (2020), Savoldi, et al., (2021)), we suggest that lower translation accuracy and quality for vulnerable social groups risk their experiencing allocative harms (Crawford, 2017; Barocas et al., 2017), such as poorer quality of MT system service or poorer professional communications when MT systems incorrectly translate their names (or sentences including their names), or representational harms (Katzman et al., 2023) such as erasure of the name and its associated culture.\n\nComment 3: The paper has an unjustified focus on US gender and racial issues in the current time period. Machine translation systems are about languages, not about specific countries at specific time periods.\n\nResponse to Comment 3: \nWe appreciate your alternative perspective and concern. Now that MT systems are routinely used in the real world, they inevitably impact people in many ways (both positive and negative). In line with a growing body of work on understanding and measuring these impacts (Weidinger, et al., (2021), Bender, et al., (2021), Blodgett et al., (2020), Savoldi, et al., (2021)), our study needs to be situated in specific real-world data. To investigate fairness and inclusivity of name translation, where names are a meaningful aspect of languages, we use a dataset of names from the US (and the associated social group distinctions) because it lets us use English as a source language and is publicly available. As discussed in our Limitations and Ethics sections, we do not claim that our results generalize beyond the settings tested.\n\nComment 4: \nThe abstract seems to oversell the results. There are no differences in translation accuracy for female, black, AAPI, or female AAPI names for one of the outcome measures. They have to get into a few of the specific interactions to find significant effects. This is not explained.\n\nResponse to Comment 4: This statement does not accurately represent our results. We did find significant differences in translation accuracy for Female, Black, AAPI, and Female AAPI-associated names – please reference Section 6.1 “Correctness of Translation” (these results can be seen in Tables 4 and 5 of our paper; significant effects are denoted with an asterisk after each relevant p-value). We describe in Section 6.1 how translation accuracy results are lower across both female-associated names as a whole, as well as the intersectional combinations (e.g. female-AAPI-associated names) across the board (see odds ratios for differences for each social group in comparison to the baseline of an odds ratio = 1). Name translation quality results as measured by sentence-BLEU can be seen in Section 6.3 Table 6. Our results for translation quality reinforce our finding that Black and Hispanic-female-associated names have lower translation accuracy; specifically, we observe degraded translation quality for sentences including these names. We observed that the translation quality effect sizes have smaller effect sizes than the translation accuracy, but this was expected since BLEU is an average over many words, most of which will probably be correctly translated.\n\nComment 5:\nLogistic regression has strong statistical assumptions for inference that are not discussed. Construction of standard errors are not discussed.\n\nResponse to Comment 5: \nWe utilized a logistic mixed effects model which is distinct from a standard logistic regression therefore we chose to express the statistical assumptions related to mixed effects models in Appendix B; these apply to both our logistic and linear mixed effects models. For more information on the logistic mixed effects model regression and the binary outcome variable we used for it (name accuracy where an exact match of the name existed between the roundtrip translation and the original name), please see Section 5.2 specifically “Name Translation Accuracy”, our overall model specification detailed in Section 5.3 and Appendix C (predictors were the same for both logistic and linear mixed effects models, only the outcome variables differed where the one for logistic regression was a binary value of 1 for match and 0 for non-match), and finally the model and implementation details further listed in “BinomialMixed Effects Logistic Regression” under Appendix D. With respect to standard errors, we chose not to include these since we were most interested in determining whether the odds ratios (and corresponding coefficients) for the logistic mixed effects regression were significant and by what degree for a given fixed effect factor (for example, gender*race_ethnicity=AAPI). We would be happy to provide standard errors upon request for the fixed effects factors but these may not be easily interpretable given that all of our predictors are binary variables.\n\nCitations\n\nSolon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Allocative versus representational harms in machine learning. In Proceedings of SIGCIS, Philadelphia, PA. The Special Interest Group for Computing, Information and Society.\n\nEmily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT '21). Association for Computing Machinery, New York, NY, USA, 610–623. https://doi.org/10.1145/3442188.3445922\n\nSu Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (Technology is Power: A Critical Survey of “Bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476, Online. Association for Computational Linguistics.\n\nKate Crawford. 2017. The trouble with bias. Conference on Neural Information Processing Systems (NeurIPS).\nBeatrice Savoldi, Marco Gaido, Luisa Bentivogli, Mateo Negri, and Marco Turchi. 2021. Gender bias in machine translation. Transactions of the Association for Computational Linguistics, 9:845–874.\n\nWeidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P. S., ... & Gabriel, I. (2021). Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Shows biases along race and gender lines in machine translation, where there are more errors for some disadvantaged groups when you interact multiple membership together.  \n\n", "reasons_to_reject": "The paper says that these mistranslations are due to \"the systems' tendencies to standardize language to predominant language patterns.\" This is a very strong causal interpretation of the results. Is that really justified by the evidence presented? It could be that these mistranslations are due to relative frequencies of these patterns in the MT system training data.\n\nThe paper also says these mistranslations have \"significant implications for people’s professional, personal and cultural identities, self-worth and ease of communication.\" That is another very strong causal interpretation of the results. Is that really justified by the evidence presented?\n\nThe paper has an unjustified focus on US gender and racial issues in the current time period. Machine translation systems are about languages, not about specific countries at specific time periods. \n\nThe abstract seems to oversell the results. There are no differences in translation accuracy for female, black, AAPI, or female AAPI names for one of the outcome measures. They have to get into a few of the specific interactions to find significant effects. This is not explained.\n\nLogistic regression has strong statistical assumptions for inference that are not discussed. Construction of standard errors are not discussed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates the idea that translation quality for names and surrounding context will be lower for names associated with US racial and ethnic minorities. They develop a dataset of demographically aligned names and propose a translation evaluation procedure based on round-trip translation. The study finds that translation systems are significantly less capable of correctly translating female-associated names, especially those associated with racial (Black) and ethnic (Hispanic) minorities.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ABO6rtJquw", "reviewer": "Reviewer_ndtx", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, Area Chairs, Senior Area Chairs and Program Chairs,\n\nThank you for the careful review and constructive feedback. We address the main comment and question below.\n\nReview 3: Reviewer ndtx\n\nReviewer Comment: The 16 template sentences were chosen with reasons, but it is possible that these sentences do not completely represent the wide variety of sentences with names in them that appear in \"real world\" contexts. The authors also discuss the US-centric dataset used in their Limitations section.\n\nOur Response: \nThank you for the observation; yes the templates do not fully represent the full variety of sentences that appear in real-world contexts. This is a standard limitation of template-based datasets. We chose our sentence templates in light of our harms, and specifically to align to scenarios where names matter and to be independent from the demographic variables considered, including ensuring gender neutrality of the sentences. We will add a note about this in the Limitations section in a future version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "If mistranslation of names has not been studied yet, then this paper brings up an important new type of social bias encoded in language models.", "reasons_to_reject": "The 16 template sentences were chosen with reasons, but it is possible that these sentences do not completely represent the wide variety of sentences with names in them which appear in \"real world\" contexts. The authors also discuss the US-centric dataset used in their Limitations section.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors measure the difference in how often different names are mistranslated. They use round-trip machine translation for 16 template sentences filled in with about a thousand names (categorized by gender and race/ethnicity). They find that female-associated names and names associated with racial and ethnic minorities are mistranslated more often.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an analysis with the objective of comprehending biases within machine translation systems concerning names across various dimensions, including gender and racial/ethnic groups. The authors have also contributed a dataset to facilitate the examination of this issue. The paper offers an intriguing perspective on biases in machine translation systems, supported by a comprehensive experimental framework.

The authors have provided robust responses to the reviewers' critiques, particularly in addressing the concerns raised by Reviewer 2. I concur with Reviewer 1's point that name variant errors should not be equated with translation errors, even though both aspects are crucial to assess. I strongly encourage the authors to incorporate this suggestion into the paper, as articulated in their response to the reviewers.

In summary, this work contributes to our understanding of biases in machine translation systems. By addressing the reviewers' feedback, particularly the suggestion from Reviewer 1, the paper can further enhance its clarity and impact.
Decision: Accept-Main