Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpus
Authors: Andrea Piergentili
Beatrice Savoldi
Dennis Fucci
Matteo Negri
Luisa Bentivogli
Venue: EMNLP 2023 Main (2023)
Keywords: inclusivity
machine translation
gender
non-binary
evaluation
benchmark

Abstract: Gender inequality is embedded in our communication practices and perpetuated in translation technologies. This becomes particularly apparent when translating into grammatical gender languages, where machine translation (MT) often defaults to masculine and stereotypical representations by making undue binary gender assumptions. Our work addresses the rising demand for inclusive language by focusing head-on on gender-neutral translation from English to Italian. We start from the essentials: proposing a dedicated benchmark and exploring automated evaluation methods. First, we introduce GeNTE, a natural, bilingual test set for gender-neutral translation, whose creation was informed by a survey on the perception and use of neutral language. Based on GeNTE, we then overview existing reference-based evaluation approaches, highlight their limits, and propose a reference-free method more suitable to assess gender-neutral translation.

Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpus
Andrea Piergentili, 1,2+ Beatrice Savoldi,
Dennis Fucci, 1.2 Matteo Negri', Luisa Bentivogli*
University of Trento
Fondazione Bruno Kessler
{apiergentili, bsavoldi if dfucci, negri, benti be
Abstract
neutral strategies have emerged and increas- ingly adopted in academia (APA, 2020), institu- tions (Höglund and Flinkfeldt, 2023), and industry alike (Langston, 2020). These strategies aim to overcome marked forms that treat the masculine gender as the conceptually generic, default human prototype (e.g.. humankind vs. mankind) (Silveira, 1980; Bailey et al., 2022). Thus, they challenge gender norms and embrace all gender identities by avoiding gendered terms when unnecessary (e.g. chair vs. chairmanlchairwoman (Hord, 2016).
Gender inequality is embedded in our commu- nication practices and perpetuated in translation technologies. This becomes particularly ap- parent when translating into grammatical gen- der languages, where machine translation (MT) ofien defaults to masculine and stereotypical representations by making undue binary gen- der assumptions. Our work addresses the ris- ing demand for inclusive language by focusing head-on on genter-neutral translation from En- glish Italian. We start from the essentials: proposing a dedicated benchmark and explor- ing automated evaluation methods. First, we introduce GeNTE, natural, bilingual test set for gender-neutre neutral translation, whose creation was informed by survey on the perception and use of neutral language. Based on GeNTE, we then overview existing reference-based eval- uation approaches, highlight their limits, and propose a reference-free method more suitable assess gender-neutral translation.
English, being the forefront of inclusive lan- guage changes and with its limited gendered gram- mar (Ackerman, 2019), has faced fewer obstacles in adapting to neutral forms, which have already been modeled into monolingual generative tasks (Sun et al., 2021; Vanmassenhove et al., 2021). As recently underscored by Amrhein al. (2023), however, the resources and approaches made avail- able for English are not portable grammatical gender languages. Such need for dedicated ef- forts is exemplified in Italian, where neutral so- lutions must navigate the extensive encoding of masculine/feminine marking (e.g. the doctors are qualified -> it: i/le dottori/esse sono qualificati/e) through synonymy or more complex rephrasing (Papadimoulis,, 2018) (e.g- - il personale medico [the medical staff]). While indeed more chal- lenging, pursuing inclusivity in Italian is relevant exactly because sexist attitudes are more visible and impactful in grammatical gender languages (Wasserman and Weseley, 2009). Nonetheless, the implementation of neutral language in MT remains to date basically uncharted territory, despite the desirability of neutral outputs under several cireum- stances where gender is ambiguous irrelevant.
1 Introduction
Societal gender asymmetries and inequalities are reflected and perpetuated through language (Stahlberg et al., 2007; Menegatti and Rubini, 2017). Such awareness has grown also within the Natural Language Processing (NLP) field (Blodgett al., 2020), where extensive research has high- lighted how several applications suffer from gender bias (Sun et al., 2019; Sheng et al., 2021). As also noted by MT users themselves (Olson, 2018; Dev et al., 2021), among these applications are trans- lation systems used at large scale, which pose the concrete risk of misrepresenting gender minorities by over-producing masculine forms, while reinforc- ing binary gendered expectations and stereotypes (Savoldi et al., 2021; Lardelli and Gromann, 2022). To foster greater inclusivity and break free from the constraints of masculine/feminine language,
In light of the above, by focusing on English-Italian as an exemplary and representa- tive translation pair and direction, we hereby lay the groundwork toward gender-neutre MT. Starting from a survey aimed to understand the challenges
"The authors contributed equally.

---

of neutral translation in cross-lingual settings, we provide the necessary tools and resources to foster research on the topic by estimating gender neutral translation in MT. Hence, our main contributions are: (1) A study on the feasibility of neutral trans- lation, by surveying the potential trade-off among fluency, adequacy, and neutrality; (2) The creation of GeNTE,' the first natural, parallel corpus de- signed to test MT systems' ability to generate neu- tral translations; (3) A comprehensive analysis of the (un)suitabilliy of existing automatic metries to evaluate neutral translation. As an inherent bench- mark component, we indicate an alternative solu- tion capable to better assess the task.
gender languages like Arabic (Alhafni et al., 2022), Italian (Vanmassenhove and Monti, 2021). Spanish, or French (Rarrick et al., 2023) entail offering two possible translation outputs, still constrained to bi- nary gender forms (e.g., it: Ho chiamato il dotooe MASC vs. la dottoressa FEM).
As exception within the current MT land- scape, Cho al. (2019) and Ghosh and Caliskan (2023) investigate the preservation of gender- ambiguous pronouns for Korean/Bengali Baaia//BBenalllinggiiisi >English, Since English can already boast the well- established neutral pronoun they, their study does not face the additional challenges of preserving such unmarked vagueness into grammatical gen- der languages. Such challenges are exemplified by Saunders et al. (2020), who created paral- lel test and fine-tuning data develop MT sys- tems able to generate non-binary translations for However, their target sentences are artificial - created by replacing gen- dered morphemes and articles with synthetic place- holders - thus serving only as a proof-of-concepl. To the best of our knowledge, Piergentili et al. (2023) are the first to advocate the use of target gender-neutral rephrasings and synonyms as a vi- able paradigm toward more inclusive MT when gender is unknown or simply irrelevant. Despite this call to action, no concrete steps have been taken yet to actually facilitate research in this direction, not even toward suitable benchmarks to recognize the neutral forms occasionally generated by current systems (Savoldi et al., 2022).
We make the GeNTE dataset freely available at https: //mt. fbk. eu/gente and release the eval- uation code under Apache License 2.0 at https: httpp//....o.o..ttttttt/ettttt--
2 Background
Emerging research has highlighted the importance of reshaping gender in NLP technologies in more inclusive manner (Dev et al., 2021), also through the representation of non-binary identities and lan- guage (Wagner and ZarrieB, 2022; Lauscher et al., 2022; Ovalle et al., 2023). Foundational works in this area have included several applications, such as coreference resolution systems (Cao and Daumé III, 2020; Brandl et al., 2022), intra- aitra-lingual fair rewrit- ers (Amrhein et al., 2023), and automatic classifica- tion of gender-neutral text (Attanasio et al., 2021). In MT, the research agenda has mainly focused on the improvement of masculine/feminine gender translation. Along this line, different mitigation methods have been devised to ensure that unam- biguous gendered referents (c.g. he/she is doctor) are properly resolved in the target language (Costa- jussà and de Jorge, 2020; Choubey et al., 2021; Saunders et al., 2022). These methods are often tested on synthetic template-based datasets such as WinoMT (Stanovsky et al., 2019) or Simple- GEN (Renduchintala and Williams, 2022). As also stressed by Saunders and Olsen (2023), however, in realistic scenarios MT systems are also confronted with ambiguous input sentences that do not convey any gender distinction (e.g., en: I called the doctor). Nonetheless, to date the resources and solutions en- visioned for resolving such cases into grammatical
In light of the above. the path toward gender- neutral translation in MT is bottlenecked by the lack of dedicated datasets and automated evalua- tions. Here, we fill this gap so to guide and allow research on this novel topic. To this aim, we start in $3 by first ensuring that gender-neutral language can enable acceptable translations, not being per- ceived inappropriate intrusive.
3 Surveying Gender-Neutral Translation
Neutralization is a form of linguistic gender in- clusivity that relies on the retooling of established forms and grammar (Gabriel et al., 2018). Accord- ing to the review of several gender-inclusive public guidelines by Piergentili et al. (2023), these can range from i) simple word changes, like omissions or article/noun replacements with epicene alterna-
'Gender-Neutral Translation Evaluation. In Italian, gente means folks, a term used for inclusive grectings in lieu of "guys".
"Such doublc-outputs are currently offered for short, am- biguous queries also by Google Translate and Bing.

---

QUESTIONNAIRE EXAMPLE SENTENCES
Eq. NT GT
for. responses
56.3 42.5
Always
Sometimes
Soms metaal may he toxic mare.
39.6
50 10.4
GT metalli possono essere
per l'uomo.
2omaa
NT Certi metalli [..] per gli esseri umani (tamrr))
Use
incormal
Does anyone wish speak against the propesal?
54.7 31.6 13.7
Accep
GT Qualcuno desidera intervenire contro proposia?
NT CI sono Interventl ther sueectes contro [--]
Accep
Incormal
GT 1. Indomesia ha milione di profught...
Iudonesic is dealing with one million refugees...
40
23.2 36.8
0%
25%
50%
75%
NT L Inddonesia ha an milione di canfit
Figure 1: Frequency of use and acceptance of neutral language in formal vs. informal communication.
Table I: Questionnaire example of English sentences with translation alternaties. For each example, partic- ipants responses GT and NT are equivalent, preferrable, GT is preferrable- are shown (percentage).
them to be equivalent (see Table 1). Then, in part (ii) we asked direct questions gauge participants use and attitude toward gender-neutral language. The questionnaire was distributed online and re- ceived 98 responses by eligible participants. While all details are provided in Appendix A, here we summarize our main insights.
tives (e.g. il maestro vs. l'insegnante)) ), to ii) more complex reformulations, which might involve al- tering the sentence structure (e.g. i miei colleghi vs. le persone cui lavoro).* As such, to ensure neutrality, these solutions might have an effect in terms of brevity or perceived fluency.
First, the linguistic acceptability of gender- neutral translations was positively judged, perhaps at the higher rate than our own expectations. In fact, overall results indicate that in the majority of cases the NT was deemed preferable (42.5%) rather than equivalent to the GT (36.5%), where only a minor- ity favoured gendered translations (21%). Possibly, such trend is explained in light of participants ide- ological preference for inclusive language rather than by purely linguistic factors. As shown in Ta- ble I, this seems to be confirmed by disaggregating responses for each translated example sentence, where we can extract more qualitative insights. In- deed, example A attests one of the strongest prefer- ences for NT, signalling a negative attitude toward the propagation of default, masculine forms in GT. Then, concerming the use of more of complex neu- tral rephrasings (example B), we found that slightly longer and sentence- altering neutralization strate- gies were still considered largely acceptable. In- stead, literal NT with limited changes, which how- ever sacrificed more the source meaning or altered its tone (example C), were comparatively penalized. This trend was also confirmed in part (ii) of the sur- vey (see Appendix A). Finally, as shown in Figure 1, participants' responses to direct questions attest that neutralization strategies
While widespread in monolingual, institutional contexts (Papadimoulis, 2018), the use of neutral forms in cross- tross-lingual settings requires to weigh additional non-negligible factors. First, translations are bounded to source text, whose meaning must be properly rendered in the target language. Thus, more creative reformulations might collide with this instrine constraint. Also, it might be not al- ways clear-cut when neutral translations ought to be performed. This is the case of masculine gener- ics in the source language (e.g. All firemen:: while they do not neatly fall under the idea of "ambigu- ous" input, their propagation to a target language clashes with the goal of inclusive MT itself.
These issues stand unaddressed: the study by Lardelli and Gromann (2023) represents the only empirical investigation on the feasibility of gender- neutral translation, but it is concerned with the cog- nitive effort that its realization poses to post-editors. Therefore, to better understand the implications of gender-neutral translation for a wider range of stakeholders, we carried out preliminary analysis on English-+Italian by surveying the opinions of potential MT end-users.
Questionnaire. Our survey was structured into two main parts. In part (i), we indirectly assessed linguistic acceptability: given a source English sentence paired with both a gendered (GT) and a neutral (NT) translation, we asked participants to indicate whether they had a preference found
accepted/used dif- ferently depending on the speech situation, with a preference for their use in formal communicative situations. Having confirmed the feasibility and overall acceptability of neutral translations, we em- bed the gathered qualitative considerations the design of the GeNTE corpus ($4).
Pen: "the teacher".
'en: "my collcagues" vs. "the people work with".
$i.c. equally adequate and fluent.

---

4 The GeNTE corpus
targeted Set-G segments by matching source En- glish sentences that contained explicit gender cues, e.g. lexically gendered words (sister, woman), ti- tles (Mr, Mrs) and marked pronouns (him, her). Set-N, instead, was populated by matching several word classes that do not convey any gender dis- tinction in English (e.g- you, cilizens, wen(), but typically correspond to masculinc/feminine expres- sions in the target language. Also, we searched for masculine terms used generically, such man and its derived compounds (e.g., chairman, layman). In fact, masculine generies are unreliable gender cues and, following the survey findings ($3), should not be propagated in MT.
GeNTE is the first test set designed to evaluate MT models' ability to perform gender-neutral transla- tions, but only under desirable circumstances. In fact, when referents' gender is unknown or irrele- vant, undue gender inferences should not be made and translation should be neutral. However, neu- tralization should not be always enforced; for in- stance, when a referent's gender is relevant and known, MT should not over-generalize to neu- tral translations. The corpus hence consists of 1,500 English-Italian parallel sentences with men- tions to human referents that equally represent two translation scenarios: 1) Set-N, featuring gender- ambiguous source sentences that require to be neu- trally rendered in translation; 2) Set-G, featuring gender- sunidrruunambuuuuus source sentences, which shall be properly rendered with gendered (masculine or feminine) forms translation. Altogether, these sets allow to benchmark whether systems are able to perform gender-neutral translation, and if they do so when appropriate.
Sentence editing, On the collected material, a first intervention was carried out to streamline the evaluation of gender-neutral translation. In fact, some of the source sentences contained men- tions of multiple referents, which required the combination of different forms in translation (i.e. neut/masc/femm). In those cases, the parallel sen- tences were manually edited so as to ensure that they only include referents that require the same type of (either neutral or gendered) forms. this way, each sentence pair can be handled as a whole coherent unit, thus avoiding the complex- ities of evaluating intricate combinations of phe- nomena. To ensure balanced distribution of in- stances from both Set-N and Set-G, second inter- vention was required compensate for the under- representation of unambiguous cases.? Although these edits slightly reduce the naturalness of the data, they allow for a simpler and sound evalua- tion, crucial to shed light on a complex task such as gender-neutral MT. Instead, other edits were made to enhance the quality of the corpus; all of them reported in Appendix SB.1. Once the editing phase concluded, all sentence pairs were anno- tated as N Set-N, and as M in Set-G. In the annotation process, it was verified that the initial pool of - automatically extracted - candidate sen- tences were correctly assigned to Set-N and Set-G by accounting for the sentence context. In this way, we could differentiate between the use of gendered words as either masculine generies (e.g. It is up to accused employer to prove his innocence identified as N) as informative of referent's gender (e.g. I would like to thank Commissioner Byrne for his cooperation. identified as G).
We build GeNTE on naturally occurring in- stances of both scenarios retrieved from Europarl (Koehn, 2005). Besides being widely popular and high-quality MT resource, we chose this cor- pus inasmuch it represents formal communicative situations from the adminnstrativv/nssittttoaall do- main. Accordingly, it reflects the context for which gender-neutral forms are traditionally intended, also in line with the stakeholders' preference high- lighted in $3. Also, as examined by Saunders (2022), Europarl exhibits large amount of gender- ambiguous cases that although translated with gendered forms in the original references of the corpus - lend themselves as suitable candidates for neutralization. As explained in the forthcom- ing paragraphs ($4.2), for each of these original Europarl gendered target sentences, we create an additional gender-neutee reference translation.
4.1 Data selection and annotation
Data extraction. To retrieve Europar! segments representing our two translation scenarios of inter- est, we crafted regular expressions to: i) identify source sentences containing mentions to human referents, ii) maximize the variability of linguistic phenomena included in the corpus, and tii) ensure a balanced distribution of both unambiguous and am- biguous gender translation cases. To this aim, we
"Such lack confirms the vast representation of generic and unknown referents in Europari, as found in Saunders (2022).
h/ttps://www..r stwtst..ttttmm wwaeinttorrg/eeeeeee mtggeurroaar/arcchieee html

---

SRC
Lalong all my colleagues,
welcome [..]
REF-G
Insieme a tutti i miei colleghi, desidero esprimere mio compiacimento per quesia [...]
i-N N
REF-N
Insieme agli altri membri watera). desidero esprimere compiacinento questa
Insieme ogni collega colloaue): desidero exprmere mio compiacimento per questa [..] Insieme tutle le persone cun cui lavorogal
REF-N3 3
desidero espeimere il mio compiacimento per quesia [..]
SRC
I welcome this excellent report from my colleague Mr Skinner.
REF-G
Valuto positivamente la relazione del collega. oncrevale Skinner.
R M
REF-N
Valuto positvamentee la relazionc dell' oonoeeole collegtiot of the nonorabie callaggue Skinner.
REF-N
Valuto positivamente la relazione oele'oneeeel collega Skinner.
REF-N3
posiivamente relazione dell' onnorvlle collega Skinner.
SRC
Mrs Ana de Palacio Vallelersundi has sister who Commissiuner [...] La omerevole [..] ha una sorella, quale una Comissaria [...]
REF-G
W1-F
REF-N
N.A.
L'anorevole [..] ha uno stretto legame parentelagi, clossty relared] con un membro della Commixsione; water of the cowisstts)) N.A.
REF-N
Table 2: Examples of entries in the COMMON-SET, REF-G indicates the gendered references, REF-N I, 2, 3 indicate the neutralized references produced by Translator 1, 2. and 3 respectively. Words in bold are mentions of human referents; underlined words are linguistic cues informing about the referents's gender.
4.2 Creation of gender-neutral references
In Appendix B.2, we provide qualitative insights regarding revisions and supervision of the linguist.
As confirmation of the predominant use of gen- dered forms when translating into grammatical gen- der languages, worth remarking that almost all (97.2%) segments collected from Europarl have gendered references in Italian. Inspired by the de- sign of natural (binary) gender bias benchmarks such as MuST-SHE (Bentivogli et al., 2020) and MT-GenEval (Currey et al., 2022), we thus created second translation, so to allow for reference- based contrastive evaluation of gender-neutral MT (see $5.2). To this aim, for each sentence pair, we created an additional Italian reference, which dif- fers from the original one only in that it refers to the human entities with neutral expressions. This makes it possible to isolate gender-related linguistic elements as the only source of variation in the score of system outputs when evaluated against both the gendered and the neutral references. As neutraliza- tion is an open-ended task that entails a high degree of variability in the possible solutions, we wanted such variability accounted for in the neutral refer- ences. Therefore, their creation was assigned to three professional translators hired via a translation agency. Each of them was assigned a different portion of the collected Italian references, to be post-edited to only replace gendered terms with neutral formulations. An expert linguist native speaker of Italian prepared detailed instructions drawing from existing guidelines for the institu- tional domain. After an initial training session, the linguist supported the translators throughout the process and finally checked all the neutralizations.
GeNTE COMMON-SET. Whereas each transla- tor was in charge of post-editing one given portion of the corpus, we also selected a common set of 200 references to be neutralized by all translators (henceforth referred to the COMMON- SET); 100 were taken from the gendered set (COMMON-SET- G), and 100 from the neutral one (COMMON-SET- N). Thus, we obtained 200 source sentences, each paired with one (original) gendered reference and three (post-edited) neutral references. The creation of a COMMON-SED was primarily motivated by the goal of having subset of the corpus that could be used to test the robustness of evaluation proto- cols and metrics across the three different neutral references (see $5.2). Orthogonally, it allowed us to measure línguistic variability among the neu- tral and gendered references (sce Appendix B.3). Table shows examples from the COMMON-SET, which confirm the findings of our preliminary sur- vey ($3). Example i is representative of the vari- ability that is inherent to the neutralization task. Example ii, instead shows rare situation where all translators used the same neutralization device and produced an identical sentence. Finally, iii shows a gendered term, whose neutralization re- quires verbose periphrases that compromise the original text's fluency and style. This case was signaled as particularly difficult to neutralize: two translators out of three did not create a neutral ref- erence. Overall, based manual analysis of the COMMON-SET the translators produced three identical gender-neutral references in 13.57% of the cases, while an additional 8% of translations exhibited a high degree of similarity (c.g.. the same neutral words are used, but in a different order).
"The cost paid to the agency of 60 euros/hour, for total of 14 hours of work for each translator. "The linguist one of the authors the paper. 10 RRelcased together with the GeNTE corpus.

---

GENTE
Source
REF-G
REF-N
sentenoss 750
Avg. longth
sentenocs
Avg. length
semenecs
Avg length
G-words
Sat-N Set-6
25.67 26.51
750
24.66
750
26.95
1,922
750
25.26
750
26.55
2,148
COMMON-SET
Source
REF4
REF-N
A scmiences
Avg length
senienzes
Avg lengih
sentences
Avg lengih
# G-words
Set-N
100
27.45
100
27.00
300
28.87
300
Set-G
100)
26.99
100
26.34
300
27:57
299
Table 3: Corpus statistics for GENTE and its subset COMMON-SET. Both sets requiring gendered translations (Set-G) are equally balanced between and M sentences. Average lengths are calculated ignoring punctuation. In the last column, we provide the number gendered words the REF-Gs that had be neutralized in the REF-Ns.
These statistics are positive: they show that the GeNTE COMMON-STT exhibits a good level of variability (~79%), which is desirable to test open- ended generation tasks like MT. Also and espe- cially in light of the fact that the translators worked independently - the 21 of identical/similar neu- tralizations suggests that neutralizing translation is a challenging but feasible task.
late" and DeepL. 12 However, manual inspection showed an almost complete lack of representation of gender-neutral translations in the outputs: gen- dered translations were generated for all but of the COMMON-SET-N inputs." This result re- vealed the unsuitability of such outputs for inves- tigating the automated evaluation neutral trans- lation itself. Accordingly, to obtain neutral (MT- like) outputs to be included in the test-bed, we re- sorted to manually post-editing the 100 COMMON- SET-N translations generated with undue gender assignments. To do so, we leveraged our manually- created neutral references ($4.2): we substituted the neutral forms produced by the three profes- sional translators to the gendered forms in the MT outputs, so as to make them neutral without alter- ing the rest of the sentence. 14 For each system, we thus obtained three sets of neutral output sen- tences (one per translator), so to account for the robustness of different evaluation methods the linguistic variability expressed in the inventory of neutralization strategies potentially applicable by humans and machines.
To conclude, relevant statistics for GeNTE and its COMMON-SEE SET are provided in Table 3.
5 Gender-neutral Evaluation Protocols
We complement our benchmark creation effort with a study on the possible approaches for us- ing GeNTE to conduct automated evaluations of neutral MT. To this aim, we first define sound test-bed conditions ()5.1). On this basis, we then experiment with contrastive, reference-based protocol to inspect the effectiveness of standard MT metrics to assess neutral translation ($5.2). Then, overcome the limitations encountered with the reference-based approach, we implement a reference- free protocol ($5.3), which shows promise in advancing the task's 's evaluation.
5.2 Reference-based Evaluation
5.2.1 Setting
5.1 Test-bed
In this evaluation protocol aim to verify whether common reference-based MT metries can be effec- tively used to identify gendered and neutral trans- lations. The protocol is based on the idea that a system generates a gendered translation, its output will be rewarded when evaluated against a gendered reference and penalized when evaluated
To ensure a sound comparison between different automatic evaluation protocols, we built test-bed based on the GeNTE COMMON-SET ($4.2, Table 3). Our test-bed includes relevant instances in relation to our task, namely gendered and gender-neutral automatic translations in equal proportion. On this basis, the analyzed evaluation approaches can be compared in their ability to reward systems that generate neutralized outputs only when due.
"https://aws amazon. cwt...co//eeeeeeeee.o con/translate/ Phttpss//www deepl. comm/e//translee 13 provides a glimpse into the shortcomings of inclu- sivity within current MT landscape.
The automatic Italian translations of the COMMON-SEE SET sources were generated with two leading commercial MT systems: Amazon Trans-
"On average. 12% of the words present in the systems' output were substituted through post-editing, thus these edits have a minimal and circumscribed impact that does not alter the original output sentence.

---

COM-
COMMON-SUTA
Metric
Deepl.
Amazon
Deepl.
Amazon
REF-G
REE-N
REF-
3%%
RKF-N
REE-
REF-N
REF-G
Â
BLEU
34.95
21.97
19.98
35.20
28.12
20.11
24.91
22.82
8.39
24.44
22.44
8.19
chriF
64.18
58.52
8.82
64.01
58.32
8.90
55.49
55.81
-0.59
$5.54
55.76
-0.40
TER J
52.18
59.68
14.38
53.54
61.35
14.59
66.52
70.99
6.73
66.68
71.32
6.97
METEOR
62.10
54,26
12.63
60.90
52.99
13,00
48.34
47.37
2,00
47.79
46.90
1.86
BERTscore
58.34
86.
2.47
85.79
2.52
84.25
84.36
-0.13
84.13
84.20
-0.08
COMET
$7.89
86.08
2,06
87.
85.50
2.13
84.89
$5.06
-0.20
84.69
84.92
-0.27
BLEURT
80.50
79.67
76.79
Table 4: Corpus-level scores for DeepL and Amazon Translate, and percentage gains (A%, with sign changed for TER) with respect to the correct references. COMMON-SET-S the original MT output is evaluated against each of the three available references, resulting scores are averaged. COMMON-SET-N. each of the three edited MT outputs is evaluated against the two references not used neutralize it, all resulting scores are averaged.
against a neutral onc. On the contrary, if system produces a neutral translation, this is expected to be rewarded when compared to neutral reference and penalized when compared gendered one.
assign higher scores to systems evaluated against the neutral references.
These results show that, compared neural met- rics, n-gram overlap metrics appear more suitable for assessing gender-neutrality The lower effec- tiveness of the neural metrics could be attributed to the lower frequency of neutral expressions the training data of these models, Icading to lower probabilities assignment. Also, neural metrics are sensitive to semantic variations, but robust to sur- face lexical morphological differences. Thus, since gender neutralizations preserve the essen- tial semantics of their gendered equivalents, neural metrics are unable to properly frame their differ- ences in this contrastive setting. This is evident the consistently higher A percentages observed in COMMON-SETE for n-gram overlap metrics (all above 8%) compared to the lower percentages ob- tained with neural metrics (all below 5%).
Contrastive Protocol. Given a system output and reference- basenncce-eeedddeeeee metric, we compute corpus- level scores against both the gendered and the neu- tral references provided in COMMON-SET Then, for COMMON-SET-G the metric is effective if the scores are higher when computed against the gen- dered translations than the neutral ones; vice versa for COMMON-SET-N.
Metrics. We study the effectiveness of a set of widely used metrics. These can be categorized as i) n-gram overlap metrics: BLEU'S (Papineni et al..
2002), chrF (Popović, 2015), TER (Snover et al., 2006), and METEOR (Banerjee and Lavie, 2005), which are sensitive surface form differences be- tween outputs and references (Glushkova al., 2023); üi) neural model-basee metries: BERTScore (Zhang et al., 2020), BLEURT (Sellam et al., 2020), and COMET (Rei et al., 2020), which compare semantic representations based on the respective underlying models.
We further experimented with BLEU, TER, and METEOR to investigate their ability to provide more fine- mne-grained ovaluations. We thus tested them on the same data and the same contrastive princi- ple, but at the sentence level (i.e. neutral output sentence should obtain higher scores the neu- tral reference compared to the gendered reference, and vice versa for gendered output translation). In this way, the A obtained with the contrastive reference-based evaluation protocol that relies on BLEU, TER and METEOR can be used to cate- gorize each sentence as either gendered or gender- neutral: higher BLEU on the neutral reference wrt. the gendered reference - output sentence clas- sified neutral; higher BLEU on the gendered reference wrt. the neutral reference - output tence classified as gendered. By distinguishing the sentences belonging COMMON-SET-N and COMMON-SET-C in advance, we can thus calculate an overall accuracy that represents the proportion of sentences correctly categorized. The results are
5.2.2 Results
Table 4 reports the results computed with each metric our test-bed. First, results are consis- tent between DeepL and Amazon Translate. With respect to COMMON-SET-G, all the metries cor- rectly give higher scores for gendered references than for neutral ones: positive percentage differ- ences thus indicate that the metrics correctly re- ward systems' gendered translations. However, in COMMON-SET-N there is divergence in perfor- mance between n-gram overlap metrics and neu- ral metrics. Only three metrics based on n-gram overlap BLEU, TER, and METEOR correctly
>BLEE|]: 1 cc::mieddeeeeee e: tak: |::ps: s:exp|v:2.3 2..33

---

presented in Table 5. For COMMON-SET-G, the per- formance rather promising for all the three met- rics, with accuracy scores always above 90% for the outputs of both Amazon Translate and DeepL. This is in line with the corpus-level results for the same set. Interestingly, though, for COMMON-SET N the accuracy scores are very low, worse than or close to random choice for METEOR and BLEU. Only TER-based evaluations for Amazon Translate, 65.83% for DeepL).
Metric
Deepl.
Amazon
Set-G
Set-N
All
Set-G
Set-N
All
BLEU
92.00
52.00
72.00
93.33
52.66
73.08
TER
90.33
65.83
78.08
91.67
65.17
78.42
METEOR
94.67
42.71
68.69
94.67
41.43
68.05
Classifier
91.00
88.67
89.83
87.00
87.33
87.17
Table 5: Accuracy scores for reference- based (BLEU, TER, and METEOR) and reference-fress (classifier) eval- uation protocols.
higher (65.17%
In conclusion, through this closer inspection, we find that none of the n-gram overlap metrics is ac- tually reliable for the evaluation of gender-neutral translation. This is possibly due to the fact that our reference- based evaluation approach, just like the metrics it is based on, is heavily dependent on the reference sentences, Outputs that deviate from the reference, even if they are acceptable translations, may therefore be penalized. This issue becomes particularly critical when evaluating gender-neutral translations, as periphrasis or synonymy are among the most common and accepted techniques used for achieving neutrality ($3). These strategies are inherently penalized by n-gram overlap metrics and do not seem to entail significant differences in meaning according to neural metrics, thus advocat- ing for alternative, reference-free solutions.
triplets of sentences, which only differ for the in- serted (neut/masc/feem) seedword. This resulted in ~~60,000 sentences with very low level of noise, but featuring rather simple and repetitive syntactic structure. Therefore, we finally carried out sec- ond generation round, prompting GPT to rewrite each triplet adding context increase sentence variability and length. This led to final synthetic corpus of ~~80,000 sentences, equally distributed across neut/masc/fem instances, and with varied structures to favor generalization. 16
Gender-Neutral Classifier. To implement the classification model, we leveraged UmBERTo," 17 Roberta-base based language model (LM) fine-tuned on the Italian section of the web corpus OSCAR (Or- tiz Suárez et al., 2019). In the survey by Tamburini (2020), UmBERTo was proven to be one of the best- performing LMs for Italian. Given sequence of tokens, UmBERTo returns a contextualized vector for each token, including the special [CLS] token placed at the beginning of the sentence. As sug- gested by Devlin et (2019), we added linear layer on top of the [CLS] vector to predict the neu- tral or gendered class. We trained the parameters of both the linear layer and UmBERTo on the classifi- cation task using the synthetic corpus labeled with neutral or gendered for feminine and masculine tags. Since this solution yielded the best results, our final classifier was trained in unbalanced data conditions, by making use of all synthetic gendered sentences (e.g. 1/3 fem and 1/3 masc) and all neu- tral sentences (1/3 neut). For complete details on the training setup see Appendix C.2.
5.3
Reference-free Evaluation
5.3.1 Setting
Our reference-free protocol for the evaluation of gender-neutral translation explores classification- based approach. We cast the problem a bi- nary task to classify if attomattiallyyttrnslttee sen- tences are gendered neutral. Implementing this procedure requires i) generating training data, and ii) training the classifier on the collected data. Then, results are computed on our test-bed in terms of classification accuracy.
Synthetic Data Generation. Confronted with the lack of Italian corpora featuring gender-neutral language, we resorted to synthetic data generation by prompting GPT (gpt-3.5-turbo). To do so, we devised a three-step approach that allowed for more controlled generation procedure with reduced risk of noise (for full details, see Appendix C. First, similarly Attanasio et al. (2021), we manu- ally created 800 triplets of neutral, masculine. and feminine referents (e.g. the neighbours: il vic- inato i vicini vicine). Then, we used such triplets as seedwords to prompt GPT and generate
5.3.2 Results
Compared the accuracy scores obtained via sentence-level contrastive evaluation based on BLEU,, TER, and METEOR (first three rows of
1he synthetic corpus is released with evaluation model.
"httpss:/huugiiggfaal ho/Musixmatchh umberto- uwmertt-ccoomnneeeeee cottoooccooooocalllaaeee---

---

Table 5), it is evident that the scores achieved by the trained classifier (row 4) are notably higher. These discrepancies primarily arise from the per- formance on the neutral outputs (Set-N), where the classifier outperforms the best n-gram overlap metric (TER) by a margin of up to 22.67 points for DeepL. However, for gendered outputs, the classi- fier demonstrates slightly lower results compared the three reference-based metries (except for DeepL. where it outperforms TER). As result, for our reference-free approach, the gap between the scores obtained on gendered and neutral oulputs is small (especially for Amazon Translate), attesting a balanced performance across the two classes.
($4 & $5.3) and modeling ($5.3) for the specific English-<Italian language pair. Thus, except for the GPT prompts written in English, the released GeNTE neutral references and the trained classifier cannot be directly used for other target languages. However, Italian was chosen as a highly represen- tative example of the challenges faced in cross- lingual transfer from English. Accordingly, we believe that our design considerations, the method- ology for the creation of GeNTE, as well as the presented evaluation protocols broadly apply other target grammatical gender languages, too.
In the experiments, we relied on different closed- source models: Amazon Translate, DeepL and GPT (gpt-3.5-turbo). This has reproducibility conse- quences, since these models are regularly updated, thus potentially yielding future results that differ from those reported in this paper. Also, their access via APT (paid in US dollars) might not be affordable for all instituiion//rescacherss.
All in all, the proposed reference-free evaluation protocol appears promising evaluation method to accompany the release of GeNTE and favour its future utilization as benchmark for gender-neutral MT. It proves to be robust approach, capable of handling the linguistic variability associated with gender neutralization strategies, and overcoming the limitations of the reference-based approaches.
Finally, due to the inability of current MT mod- els to generate gender- neutral translations, the out- put sentences used to test different MT metrics and evaluation protocols ($5) were partially edst-edited. Indeed, this solution does not completely reflect standard evaluation practices conducted on fully MT generated output. However, this post-editing process only targeted gender-related aspects of the output sentences, thus still vastly preserving the MT generation and offering controlled, realistic scenario. It should be recognized though that, by design, we enacted evaluation conditions where the MT models succeeded in generating the expected (either neutral or gendered) output translation. In- stead, since the models' outputs did not exhibit cases where MT failed at generating the expected (gendered) translation, we could not test the busteness of our evaluation protocols for such a scenario.
6 Conclusions
In this work, we investigated gender-neutral trans- lation as a path for inclusive MT. To this aim, we focused on English-+Italine representative of the challenges of implementing neutral forms into grammatical gender languages, As a novel area of inquiry, we started from the fun- damentals. First, we conducted a survey on the acceptability of gender-neutral translation, which highlighted the openness of potential MT end-users, especially in formal communicative situations. Sec- ond, informed by the survey, we built GeNTE, the first natural benchmark for evaluating gender- neutral translation in MT. Third, investigated the (un suitabiibyy of existing automatic evaluation protocols to assess gender-neutral translation, and thus proposing an alternative, reference- -ree solu- tion. Having taken the first steps toward gender- neutral MT, our resources and evaluation method are made available to foster and inform the future development of more inclusive MT.
pair that is highly
Ethics Statement
By addressing inclusivity in MT, this work presents an inherent ethical component. It builds from con- cerns toward the socictal impact of widespread translation technologies that reflect and propagate discriminatory and exclusionary language. Con- cretely, by potentially feeding into existing stereo- types, reinforcing male- call--grouned visibility, and perpetuating the erasure of non-binary gender iden-
Limitations
Naturally, this work presents some limitations. In the paper we took the very first steps to enable evaluation and research on the task of gender- neutral translation for inclusive MT into grammati- cal gender languages. To do so, we provided data

---

tities. 18 Still, our work is not without risks ei- ther and thus warrants some ethical considera- tions. In particular, we propose the use of gender- neutralization strategies that avoid the use of un- necessary gendered terms via the retooling of estab-
Seattle, United States, Association for Computational Linguistics.
Chantal Amrhein, Florian Schottmann, Rico Sennrich, and Samuel Läubli. 2023. Exploiting biased models to de-bias text: A gender-fair rewriting model. In Proceedings of the 61st Anmual Meeting of the As- sociation for Computational Linguistics (Volume 1: Toronto, Canada.
lished forms and grammars. These strategies can be considered as form of Indirect Non-binary Lan- guage (INL) (Attig and López, 2020), which are in- tended as we do in this paper - equally elicit all gender identities in language and prevent misgen- dering by excluding any kind of gender assumption (Strengers et al., 2020). Instead, Direct Non-binary Language (Attig and López, 2020) emerging via grassroots efforts and more predominately in on- line social medias (Lauscher et al., 2022) - resort to the creation of neologisms, neopronouns or even neomorphology to typically enhance the visibility of non-binary individuals.
Long Papers), pages Association for Computational Linguistics.
APA. 2020. Publication Manual of the American Psy- chological Association, 7th edition. American Psy- chological Association.
Giuseppe Attanasio, Salvatore Greco, Moreno La Qua- tra, Luca Cagliero, Michela Tonti, Tania Cerquitelli, and Rachele Raus. 2021. E-mimic: Empowering multilingual inclusive communication. In 202/ IEEE International Conference on Big Data (Big Data), pages 4227-4234 IEEE
In light of the above, several, concurring forms can serve different inclusive language needs (Co- mandini, 2021: Knisely, 2020). Thus, it should be stressed that the ncutralization strategies incor- porated in our MT work are not prescrivetely in- tended. Rather, they are orthogonal to other at- tempts and non-binary expressions for inclusive language (technologies) (Lauscher et al., 2023; Ginel and Theroine, 2022).
Remy Attig and Ártemis López. 2020. Queer Commu- nity Input in Gender-Inclusee Translations. Linguis- ric Society of America (Blog).
April H Bailey, Adina Williams, and Andrei Cimpian. 2022. Based on billions of words the internet. people= men. Science Advances, x(33)mabm24633
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Sunnarization, pages 65-72, Ann Arbor, Michigan, Association for Computational Linguis- tics.
Acknowledgements
This work is part of the project "Bias Mitiga- tion and Gender Neutralization Techniques for Automatic Translation", which is financially sup- ported by an Amazon Research Award AWS AT grant. Moreover, we acknowledge the support of the PNRR project FAIR- Future AI Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU.
Luisa Bentivogli, Beatrice Savoldi, Matteo Negri, Mat- tia A. Di Gangi, Roldano Cattoni, and Marco Turchi. 2020. Gender in Danger? Evaluating Speech Trans- lation Technology on the MuST-SHE Corpus. In Proceedings the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6923- 6933, Online. Association for Computational Lin- guistics.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey "bias" in NLP. Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 5454- 5476, Online, Association for Computational Lin- guistics.
References
Lauren Ackerman. 2019. Syntactic and cognitive issues in investigating gendered coreference. Glossa: journal of general linguistics, 4(1).
Bashar Alhafni, Nizar Habash, and Houda Bouamor. 2022. User-centric gender rewriting. Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguis- ties: Human Language Technologies, pages 618-631,
Stephanie Brandl, Ruixiang Cui, and Anders Søgaard. 2022. How conservative are language models? adapt- ing to the introduction of gender-neutral pronouns. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1624-3630, Seattle, United States. Association for Computational Linguistics.
"We use non-binary as an umbrella term to encompass all gender identities that do not conform to the Western mascu- line/feminine binary, and that in particular are not linguisti- cally represented hy binary gendered expressions.

---

Yang Trista Cao and Hal Daumé III. 2020. Toward gender-inclue inclusive coreference resolution. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 4568-4595, On- line. Association for Computational Linguistics.
Sourojit Ghosh and Aylin Caliskan. 2023. Chatgpt per- petuates gender bias in machine translation and ig- nores non-gendered promouns: Findings across ben- gali and five other low-resource languages.
María Isabel Rivas Ginel and Sarah Theroine. 2022. Neutralising for equality: All-inclusive games ma- chine translation. In Proceedings of New Trends in Translation and Technology, pages 125-133. NeTTT.
Won Ik Cho, Ji Won Kim, Seok Min Kim, and Nam Sao Kim. 2019. On Measuring Gender bias Trans- lation of Gender-neutral Pronouns. In Proceedings of the First Workshop Gender Bias Natural Language Processing, pages 173-181, Florence, IT. Association for Computational Linguistics.
Taisiya Glushkova, Chrysoula Zerva, and André F. T. Martins. 2023, Bleu meets comet: Combining lexical and neural metrics towards robust machine transla- tion evaluation.
Prafulla Kumar Choubey, Anna Currey, Prashant Mathur, and Georgiana Dinu. 2021. Improving gen- der translation accuracy with filtered self-training.
Frida Höglund and Marie Flinkfeldt. 2023. De- gendering parents: Gender inclusion and standard- ised language in screen-level bureaucracy. Interna- tional Journal of Social Welfare.
arXiv preprint arXiv:2104.0760
Gloria Comandini. 2021. Salve tutto, tutt*, tuttu, tutix e tu l'uso delle strategie di neutralizzazione di genere nella comunità queer online. Indagine su un corpus di italiano scritto informale sul web. Testo Senso, 23:43-64 64.
Levi C. R. Hord. 2016. Bucking the linguistic binary: Gender neutral language English, Swedish, French, and German. Western Papers in Linguistics/aahies linguistiques de Western, 3(1):4.
Marta R. Costa-jussi and Adrià de Jorge. 2020. Fine- tuning neural machine translation on cender-balaneed datasets. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 26-34, Barcelona, Spain (Online). Association for Computational Linguistics.
Kris Aric Knisely, 2020. Le français non non- binaire: Lin- guistic forms used by non-binary speakers of French. Foreign Language Annals, 53(4):850-876
Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Proceedings of the tenth Machine Translation Summit, pages 79 86, Phuket, TH. AAMT.
Anna Currey, Maria Nadejde, Raghavendra Reddy Pap- pagari, Mia Mayer, Stanislas Lauly, Xing Niu, Ben- jamin Hsu, and Georgiana Dinu. 2022. MT-GenEval: counterfactual and contextual dataset for evaluating gender accuracy in machine translation. In Proceed- ings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 4287-4299, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Jennifer Langston. 2020. New AI tools help writers be more clear, concise and inclusive in Office and across the Web, https://blog:. www.ssootttoooooooooooooooooo.oc com/ htt//mcccooofttoooomooooooooooot 365- tools/. Accessed: 2021- 02-25.
Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Ar- jun Subramonian, Jeff Phillips. and Kai-Wei Chang. 2021. Harms of gender exclusivity and challenges in non-binary representation in language technologies. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 1968-1994, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Manuel Lardelli and Dagmar Gromann. 2022. Gender- fair (machine) translation. In Proceedings of New Trends in Translation and Technology, pages 166- 177. NeTTT.
Manuel Lardelli and Dagmar Gromann. 2023. Gender- fair post-editing: A case study beyond the binary. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 251-260, Tampere, Finland.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186 Minncapolis, Minnesota. Association for Computational Linguistics.
Anne Lauscher, Archie Crowley, and Dirk Hovy. 2022. Welcome to the modern world of pronouns: Identity- inclusive natural language processing beyond gen- der. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 1221- 1232, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Ute Gabriel, Pascal M. Gygax, and Elisabeth A. Kuhn. 2018. Neutralising linguistic sexism: Promising but cumbersome? Group Processes Intergroup Rela- tions, 21(5):844-858
Anne Lauscher, Debora Nozza, Archie Crowley, Ehm Miltersen, and Dirk Hovy. 2023. What about em? how commercial machine translation fails to handle (neo-)pronouns.

---

Michela Menegatti and Monica Rubini. 2017. Gender Bias and Sexism in Language. In Oxford Research Encyclopedia of Communication vol. 1, pages 451- 468. Oxford University Press, New York, USA.
Danielle Saunders. 2022. Domain adaptation for neu- ral machine translation. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 9-10, Ghent, Belgium. European Association for Machine Translation.
Parmy Olson. 2018. The Algorithm That Helped
Google Translate Become Sexist. https://bit. ly/ hawn_loooll_____eeeeeeee Accessed: 2023-06-20.
Danielle Saunders and Katrina Olsen. 2023. Gender, names and other mysteries: Towards the ambiguous for gender-inclusie translation.
Pedro Javier Ortiz Suárez, Benoît Sagot, and Lau- rent Romary. 2019. Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Re- source Infrastructures. In 7th Workshop on the Chal- lenges in the Management of Large Corpora (CMLC- 7). Cardiff, United Kingdom. Leibniz-Instin für Deutsche Sprache.
Danielle Saunders. Rosie Sallis, and Bill Byme. 2020. Neural machine translation doesn't 't translate gender coreference right unless you make it. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 35-43, Bareclona, Spain (Online). Association for Computational Linguistics.
Anaclia Ovalle, Palash Goyal, Jwala Dhamala, Zachary Jaggers, Kai- Wei Chang, Aram Galstyan, Richard 7emel, and Rahul Gupta. 2023. "i'm fully who i am": Towards centering transgender and non-binary voices measure biases in open language genera- tion. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 1246 1266.
Danielle Saunders, Rosie Sallis, and Bill Byrne 2022. First the worst: Finding better gender translations dur- ing beam search. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3814- 3823, Dublin, Ireland, Association for Computational Linguistics.
Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Mat- teo Negri, and Marco Turchi. 2021. Gender bias in machine translation. Transactions of the Association for Computational Linguistics, 9:845-88 874.
Dimitrios Papadimoulis. 2018. GENDER-NEUTRAN LANGUAGE in the European Parliament. European Parliament 2018
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Mat- teo Negri, and Marco Turchi. 2022. Under the mor- phosyntactic lens: A multifaceted evaluation of gen- der bias in speech translation. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 1807-1824, Dublin, Ireland. Association for Compu- tational Linguistics.
Andrea Piergentili, Dennis Fucci, Beatrice Savoldi, Luisa Bentivogli, and Matteo Negri. 2023. Gender neutralization for an inclusive machine translation: from theoretical foundations to open challenges.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 7881-7892, Online. Association for Computational Linguisties.
Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
Emill Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4275-4293, Online. Association for Computational Linguistics.
Spencer Rarrick, Ranjita Naik, Varun Mathur, Sundar Poudel, and Vishal Chowdhary. 2023. Gate: A chal- lenge set for gender-ambiiuous translation examples.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685-2702, Online. Association for Computational Linguistics.
Jeanette Silveira, 1980. Generic Masculine Words and Thinking. Women's Studies International Quarterly, 3(2-3):165-11
Adithya Renduchintala and Adina Williams. 2022. In- vestigating failures of automatic translationin the case of unambiguous gender. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3454- 3469, Dublin, Ireland. Association for Computational L Linguistics.
Matthew Snover, Bonnie DOrT, Richard Schwartz, Lin- nea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human An- notation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Amer- icas,, pages 223-231, Cambridge. Association for Machine Translation in the Americas.

---

Dagmar Stahlberg, Friederike Braun, Lisa Irmen, and Sabine Sezesny. 2007. Representation of the Sexes in Language, Social commumication, pages 163-187.
Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- rie Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush, 2020. Trans- formers: State-of -hee-----hhhaaeee art natural language processing. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing: System Demonstrations, pages 38 38-45, Online, Association for Computational Linguisties.
Gabriel Stanovsky, Noah A. Smith, and Luke Zettle- moyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 1679-1684, Florence, Italy. Association for Computational Linguistics.
Yolande Strengers, Lizhen Qu, Qiongkai Xu, and Jar- rod Knibbe. 2020. Adhering, steering, and queering: Treatment of gender in natural language generation. In Proceedings of the 2020 CHI Conference on Hu- man Factors in Computing Systems, CHI '20, page 1-14, New York, NY, USA. Association for Comput- ing Machinery.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q- Weinherger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with bert.
A Gender-Neutral Translation Survey
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai BlSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 1630-1640, Florence, Italy. Association for Computational Linguistics.
The questionnaire was released online in April 2023. We distributed it via targeted emails and social media posts, with request to share with rele- vant groups. Participation in this survey was volun- tary, uncompensated and anonymous, as no identi- fying or personal information about the participant was collected. Also, participants were free to with- draw at any time without penalty or consequence. An anonymized version of the survey is accessible here: htttss:/forrs.gge wttpe//////////-////PP...
Tony Sun, Kellie Wehster, Apu Shah, William Yang Wang, and Melvin Johnson. 2021. They, Them, Theirs: Rewriting with Gender-Neutral English.
arXiv preprint arXiv:2102.06788.
Fabio Tamburini. 2020. How "bertology" changed the state-of the-art also for italian NLP. In Proceedings of the Seventh Italian Conference on Computational Linguistics, CLiC-it 2020, Bologna, Italy, March 1-3, 2021, volume 2769 of CEUR Workshop Proceedings. CEUR-WS,Co
Note that we did not target individuals who use MT as professional tool. Rather, generie stake- holders that might have used MT directly or indi- rectly (e.g. being offered translations of web pages), They were made aware that results of the question- naire would have been put to use for research on inclusive MT.
Eva Vanmassenhove, Chris Emmery, and Dimitar Shteri- onov. 2021. NeuTral Rewriter: A ruse-based and neu- ral approach to automatic rewriting into gender neu- tral alternatives. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 8940-8948, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Screening questions. As the survey required judging English- Italian translations, only partici- pants with high competence of both Italian (C1 or higher) and English (B2 or higher) were cligible to take part in the survey. Accordingly, screen- ing questions aimed to verify such language skills were placed at the beginning of the survey. Of the 101 received responses, 98 were from cligi- ble participants and thus included in our analysis. The screening questions also excluded participants under 18.
Eva Vanmassenhove and Johanna Monti. 2021. gENderr-
IT: An annotated English-Italle parallel challenge set for cross- linguisii natural gender phenomena. In Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing, pages 1-7, Online. Association for Computational Linguistics.
Jonas Wagner and Sina ZarrieB. 2022. Do gender neu- tral aftixes naturally reduce gender bias in static word embeddings? In Proceedings of the 18th Conference on Natural Language Processing (KONVENS 2022), pages 88-97, Potsdam, Germany, KONVENS 2022 Organizers.
Sociodemographic information. To gain so- ciodemographic information about our participant pools, the survey consisted of a short section asking for background information (e.g., educational level and field), as well as age and self-reported gender information. Overall, our pool of participants was quite homogeneous in terms of education levels
Benjamin D Wasserman and Allyson J Weseley, 2009. à qué? quoi? do languages with grammatical gender promote sexist attitudes? Sex Roles, 61))10)):33

---

cisgender
Why the noutral tun tuniiation less preforablet
Temale
55
polse th quaetree
cis male
male
deverano pentoosere cohool telowilon
man
29
iimero che dhiddee editvio
lad
Stado he depuatic le arrá fralemizuate chi marifortave
non-binary transgender
trans man
hot at
Somewhat
I don't 't define myself
meeting
rulon Lutbl
Table 6: Open responses to the question: How do you identify?
0
0
Somwrting
0
0
stile off
compromione
(i.e., the majority of respondents had master's de- gree) and age (with 24-35 being most represented age range). This was expected, because of the channels through which the survey was distributed, but especially in light of the high English compe- tence required to take part in the survey. In terms of gender, the breakdown in Figure 6 shows an higher representation of the feminine segment of the population in the survey. Since participation to the survey was voluntary, it might have attracted individuals more interested in the topic.
Other masore applicabii they can writion in wither Einlish taelass)
ta0 roposto
Figure 2: Questionnaire: follow-up questions linguis- tic acceptability.
preferred neutral translation from 4 different op- tions.
We not consider this homogeneity as lim- itation per se. Rather, it allowed us to gauge the opinions of relevant, interested stakeholders, which are mostly affected by discriminatory language.
Use and attitude. The last portion of the survey directly investigates users' attitude and perception toward gender-neutral language. For instance, in Figure 3, we attest that participants seem inclined to sacrifice brevity in favor of neutrality. Note that, since these questions focus gender-neutral language rather than translation they concep- tually preceded the section on linguistic accept- ability. However, we placed them afterwards, so as to avoid influencing participants' responses on gender-neutre translations.
Linguistic acceptability. The pairs of source En- glish sentences aligned with an Italian GT and NT were created by professional linguist with prior experience gender-inclusive language. The original source and GT parallel sentences were re- trieved from the administrraivvellgislavve domain of EU multilingual documents. The linguist then created the second NT.
From a methodological perspective, we decided to pair the GT and NT alternatives so to allow for fixed comparative term. Otherwise, different judgments of different NT translations alone would have not allowed for the isolation of gender-related factors from other aspects of the translations that could have influenced participants' perception of acceptability.
Always
Sometimes
Brevily
Style
fuency
0%
25%
50%
75%
Figure 3: Willingness sacrifice different communica- tive aspects to ensure neutrality.
Though not shown in the paper due to space con- strains, for each example sentence in the survey the participants were directed to follow up questions, motivate their choice and provide more in- sights on the limits of the offered NT (see Figure 2). Overall, in this section, total of 7 example translation were shown. Additionally, for 3 source English sentences, participants were asked to pick
B GeNTE corpus details
B.1 Data editing report
In our data editing process, we performed two types of interventions: (i) editing which was functional to the creation and the optimal use the corpus

---

(motivation described in $4.1); (ii) editing aimed improve the overall quality of corpus data.
but were particularly challenging as they required the use of neutral strategies for unequivocally gen- dered terms.
Functional interventions (i) include two proce- dures: (A) the editing of source and reference sen- tences as to have them only include referents that require the same type of either neutral or gen- dered forms (203 entries total); (B) the duplication of gendered entries, in which, then, the gendered words were replaced with equivalents of the op- posite gender - thus we produced 126 masculine entries and 247 feminine ones (373 entries total). Some of the entries underwent both procedures. These procedures were performed total of 576 entries.
Less problematic and systematic difficulties in- volved specific terms which the translators strug- gled with, such as "deputato" (deputy). This is possibly due to the fact that some domain-specific terms and their translations, like "deputy-deputato" are established and rooted in the language the point where producing gender-neutral translation counter-intuitive and challenging. all cases, the linguist intervened and proposed a solution c.g.. "persona deputata' (lit. deputed person).
B.3 Linguistic diversity in GeNTE's gender-neutral references
With the second type of interventions (ii) we im- proved the quality of the corpus data. We did so by correcting translation errors in the original refer- ences and removing extra elements in both source and reference sentences. For example, from the source sentence "EN) would like, in particular, to thank Mrs Van den Burg, Dutch Social Democrat who worked particularly hard on Article 25." we removed the segment "EN y". We performed these corrections on a total of 89 corpus entries. More- over, to improve the variability within the data, we replaced the most frequent noun which entailed a gendered translation, rapporteur, with other terms linguistic do-
Table 7 reports our evaluation of the linguistic vari- ability within the references of the COMMON-SET. To perform such evaluation we computed BLEU matching every reference of each entry with the other two references of that entry. The scores show how there is noticeable variability, which attributable to the different neutralization strate- gies employed by the three translators. On one hand, the rather high BLEU scores indicate that the references are very similar as expected, since they share all the original content of the gendered reference, except for the gender-related terms. On the other hand, their distance from score of 100 BLEU points indicates that they are not perfectly identical. The variability appears coherent among the sets: the scores of the neutral references evalu- ated against other neutral references are especially similar in COMMON-SETE the lowest scores differ by less than 1 BLEU point, possibly indicating that the neutralization strategies employed in this set were indeed different, but had very similar impact on the original sentences.
from the instiatiiona//admiiitraan main (e.g., spokesperson, delegate, deputy). We performed this operation on 70 corpus entries.
Overall, we edited 314 original source sentences and 393 original reference sentences.
B.2 Challenges in the creation of gender-neutral references.
where the highest and
From qualitative perspective, two main type of challenges were identified the creation of the neutral references:
Articles: in 11 instances, the translators pro- duced partial neutralizations, as they overlooked masculine articles. This possibly suggests that, in Italian, articles may be perceived as secondary in expressing gender compared to nouns, adjectives, and verbs, even by native speakers. Regardless, all errors were spotted by the linguist and corrected.
Classifier Training
C.1 Generation of Synthetic Training Data
Seed words. The generation process began with the creation of 200 unique triples of seed words (e.g.. il personale impiegato [neutral] -attralJJ/iiiieeen l'impiegato [masculine] l'impiegata [feminine]). Half of them were sourced from Europarl training data by means of keyword extraction, 19 the other 100 were instead created from scratch. We then manually augmented this initial list of triplets by re-generating them with various inflectional morphology, which is relevant Phttess//bypp.oo//ppjje///aaaa
Lexical gender: translators were unable pro- duce neutralization for instances of lexically gendered nouns such as 'sorella' (sister) and 'figlia' (daughter). Such cases all concerned the creation neutral references for the SET-G which served the purpose the contrastive evaluation (Sec. 5)

---

1. REF
CAND
Reference
Reference 2 75.14
Reference
REF-G
Reference
77.65
74.14
Reference
75.14
75.09
72.08
Reference
77.59
75.03
74.89
REF-G
74.04
71.98
74.82
COMMON-SET-N
REF
CAND -
Reference
Reference 2 76.88
Reference 3
REF-G
Reference
76.27 76.15
75.89
Reference
76.91
73.36
Reference
76.28
76.14 73.26
73.02
REF-G
75.78
72.92
Table 7: BLEU scores representing the linguistic variability in COMMON-SET's references.
distinguish for the task of neutralization (e.g., make them plural, use indefinite article etc.). Ac- cordingly, we obtained ~00 triplcts of seedwords.
C.2 Training Setup
We trained the parameters of both the linear layer and UmBERTo on the classification task for 2 epoch, with learning rate of 5e-5, batch size of 64 and maximum sequence length of 64, p3. p3.2xlarge instance AWS (featuring NVIDIA V100 GPU). The code for finetuning re- lies on Huggingface transformers library (Wolf et al., 2020).
Generation: first round. We prompted the model from the GPT LLM fam- ily20 to generate triplets of sentences given the triplets of seed words. We used few-shot ap- proach with given examples the task to be per- formed (see Figure 4.). We access the model via OpenAl paid API and setting a temperature of 0.5.21 21 In total, approximately 60,000 sentences were generated. A random sample of 100 sentences manually inspected, revealing that noise was very low (10%), but that the sentences exhibited simple structure, consistently placing the subject the beginning.
Generation: second round. To enhance the qual- ity and textual context of the generated sentences, second round of generation lower temperature of 0.3 (see Figure 5). Each triplet of sentences was rewritten multiple times in different forms. This process resulted the gener- ation of approximately 320,000 sentences, which had a higher occurrence incorrect alternatives for the seedwords, estimated to be around 40% based on the inspection of 100 randomly selected sentences. The final synthetic corpus consists of approximately 380,000 sentences, featuring varied sentence structures. Specifically, one-third of the sentences contain masculine seedword, another third contain a feminine seedword. and the remain- ing third contain a neutral seedword.
performed using
Overall, cost of $13.12 USD was estimated.
30 gpt-3-5 21 hhttps://platfrr openai. www.ap..co//eaaeeee com/docs/ api-reference
openai. wwwpppp.woe/ooooooooooopo com/doccs/odels/

---

sood-wordd
Generate the sentences based on these instrctions:
Curpur groups
per
trpie.
Use word at each serianes.
Expected outbut example:
duga encorn patiaennarr, commission hanro diccso persegure stataita inamiononae. A seguto oagil incorn periamentar, commessaar fatto deciso perseguie stodell atamaahonadee seguto dagi incorin patiamentari, commiasane hanno dedso pareegare s3GD88 untemaoonaas
commisiion non di deddono, dozbiamo runnic.
3e commosan non si deciono, do28kamo nunfti
conrenisue decakono, dutissmoo musiit
Figure 4: Prompt template for the generation of triplet of sentences from (NEUT/FEM/MASC) seed words.
follwing
Change the structuee of the sontenocs, make them longer and with bi Make the semence longr. 1) Use in style fist resembbee the bengagg from Europen Porisement
Abox rewntfon verbereze achere
consioents.
remithe
conten meccutio hemnew occupahorel contean fomenane occapalorvll nouns.
the rowitbin
contan masosmne occupational ncuns, sowitton Fwedidioe must contain fammane occupational nouns, docs not contain masculine occupational
Thank step step
Accondin to the iedructions and comstratn cutputt 6 groups three sentences per each trplet of seed.
in seguito collogu partamentrr, le commisioon hanno deliberatt persegaire stabatta internaionate delta oftadirance penegulte
in colloga
le commissonn perengano decisone, doviemno procedere convocare peivergano dovemnn commessete peverngno
dall dafooocceoee
sighor
udtenze stateilo
Figure 5: Prompt Template for the rewriting of triplet of (NEUT/FEM/MASE seed sentences.


Reviews:
[{"review_id": "HqA9jA2kG3", "reviewer": "Reviewer_yjUc", "summary": "", "questions": "A. Can you quantify how post-editing the neutral translations that are generated from Amazon and DeepL affect their naturalness?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**CLASSIFIER AND OVERALL TRANSLATION QUALITY**. \nOur reference-free evaluation metric (i.e. classifier) was purposefully designed to focus only on the assessment of gender-neutral translation. In line with current practices in MT evaluation (e.g. those based on test suites), this new metric does not aspire to also gauge overall translation quality and is rather intended to complement established metrics for generic performance. For this reason, a human-evaluation study on the matter was not conducted.  \nIn the paper we compare the results obtained from the classifier with those obtained from a reference-based evaluation protocol that relies on standard MT metrics (Table 5). However,  the protocol does not exploit those metrics to measure overall translation quality but to make them informative about gender. Specifically, it proposes a contrastive method computed on the double set of GeNTE references (described in Sec. 5.1) that, when applied at the sentence level, relies on BLEU, TER, and METEOR to function as a classifier of neutral vs. gendered output translations. As such, the comparison between the two protocols is possible and shows that the classifier outperforms the reference-based method in correctly classifying neutral vs. gendered output translations.\n\n\n**NATURALNESS OF POST-EDITED MT OUTPUTS**.\nWe confined the post-editing process to gender-related words only, so as to avoid altering the ‘naturalness’ (intended as the original level of adequacy/fluency) of the MT-generated outputs. Further small adjustments were only occasionally needed to preserve the (original) grammaticality of the sentence: e.g., to ensure number agreement as in “the neighbors are” [i vicini sono - G/Plur.] vs. “the neighborhood is” [il vicinato è - N/Sing.]. On average, 12% of the words present in the systems’ output were substituted through post-editing, suggesting that these edits have a minimal and circumscribed impact on the original sentences.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A thorough review of existing research in neutralization and gendered translations in MT. \n2. A carefully constructed parallel corpus containing sentences that facilitate evaluation in both gender-ambiguous and gender-neutral scenarios. \n3. A review of how existing reference-based metrics fare in detecting the quality of neutral translations, and their particular shortcomings. For example, the fact that lower frequencies of neutral expressions affect the neural metrics more than the n-gram overlap-based ones, and the final finding that even n-gram metrics are not fully reliable at the sentence level. \n4. A new, reference-free evaluation metric that accounts for the shortcomings of the reference-based ones.", "reasons_to_reject": "The reasons for a new metric are well-motivated, but it is unclear how good the new metric is at gauging the quality of the resulting translation, aside from detecting whether a translation is gendered or not. This new classification metric perhaps needs to be validated with the human study to make sure it's measuring the quality of translations as well. Because of this reason, it is unclear what it means when the authors say \"the classifier outperforms\" (line 614). \n\nThis is something I might have misunderstood, in which case I welcome the authors to clarify further. Although, I really appreciate the authors being honest about the thoughtful limitations section.", "questions_to_authors": "A. Can you quantify how post-editing the neutral translations that are generated from Amazon and DeepL affect their naturalness?", "paper_topic_and_main_contributions": "In this paper, the authors investigate gender-neutral translation for inclusive Machine Translation, English to Italian as the testbed. The authors provide a thorough overview of existing research on gender-neutral/gender-aware translation and identify the research gap clearly. Beginning with a human study to gauge the feasibility of gender-neutral translation, the authors create a parallel corpus containing both gender-ambiguous source sentences (where neutralization may be preferred) and gender-unambiguous source sentences (where the appropriate gender is desired in the target language) to foster research in this field. Finally, the authors study the effectiveness of traditional reference-based evaluation metrics and their shortcomings, and propose an evaluation metric to address those gaps.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos - \n\n1. Grammer: \"Ambiguos\" -> ambiguous [Line 189, Line 281]"}, "extra_text": {}}, {"review_id": "DLMyO1oIbf", "reviewer": "Reviewer_qfqY", "summary": "", "questions": "How would you integrate the proposed evaluation metric with other measures that also attempt to capture other aspects of translation goodness, i.e., fluency/adequacy?\n\nCan you supply judgements of the translators who created the eval set? How often did the translators create gender-neutral translations they were unhappy with? I have some concern that without being very attentive to this concern, we could be at risk of imposing (largely non-inflecting) English-based concepts of language onto other languages where gender has extensive grammatical function, where in English it's largely semantic. A more robust description of the edited gender-neutral translations and their naturalness and appropriateness for native speakers would help allay this concern.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**METHOD TO COLLECT TEST DATA**.\nThe regular expressions were used to create initial pools of – automatically extracted – candidate sentences which contained our phenomena of interest. The actual sentence selection, however, was carried out manually by a professional linguist who carefully evaluated and annotated each sentence. During this process, the sentence context was indeed used to differentiate between the use of gendered words as either masculine generics (e.g., “If this directive were adopted, it would then be up to the accused employer to prove *his* innocence…” – identified as N) or as informative of a referent’s gender (e.g. “I should like to mention that the President of the Republic will be accompanied, as is usually the case, by *his* delegation” – identified as G). The differentiation of such cases represents a key aspect of the GeNTE creation process (as already mentioned at lines 305-310), which we will discuss in higher detail in the camera-ready version of the paper.\n\n\n**DIFFICULTIES IN CREATING NATURAL GENDER-NEUTRAL REFERENCES**.\nWe did not add this information due to space reasons. While producing the gender-neutral references, there were four instances where two of the translators reported difficulties in neutralizing lexically gendered terms, i.e. ‘figlia’ (daughter) and ‘sorella’ (sister), as shown in Table 2, example *iii*. It should be noted that such words are not supposed to be translated as gender-neutral and we only tasked the translators with doing so in order to produce the neutral references necessary to enable the reference-based contrastive evaluation (Sec. 5.1.) on the double set of G vs N references. Less problematic difficulties involved specific terms which the translators struggled with, such as ‘deputato’ (deputy). This is possibly due to the fact that some domain-specific terms and their translations, like ‘deputy-deputato’, are established and rooted in the language to the point where producing a gender-neutral translation is counterintuitive and challenging. In all cases, the linguist intervened and proposed a solution (e.g., ‘persona deputata’, lit. ‘deputed person’). Moreover, there was one common pattern of errors: in 11 instances they overlooked masculine articles, thus only producing partial and incomplete neutralizations. This possibly suggests that, in Italian, articles and prepositions could be perceived as secondary in expressing gender compared to nouns, adjectives, and verbs, even by native speakers. Regardless, the linguist highlighted such errors and the translators were able to produce a different neutralization. \nAs these examples show, English linguistic strategies for gender-neutralization cannot – and were not – directly transposed and applied to the Italian language. Rather, in the creation process of GeNTE we had a strong focus on the features of grammatical gender languages as to avoid being primed on English strategies. For this reason, the creation of GeNTE was (1) motivated and guided by recent literature on gender-neutral translation in Italian (see [1] and [2]); (2) informed by the survey described in Sec. 3, which analyzes the native speakers’ perception of gender-neutral translation; (3) entrusted to professional translators, native speakers of Italian. Overall, given the small number of problematic instances mentioned-above, we believe that that neutralization task is challenging but feasible. \n\n[1] Lardelli, Manuel & Gromann, Dagmar. (2023). [Gender-Fair (Machine) Translation](https://www.researchgate.net/publication/369948882_Gender-Fair_Machine_Translation). \n\n[2] Piergentili et al. (2023). [From inclusive language to gender-neutral machine translation](https://arxiv.org/abs/2301.10075).\n\n**(REFERENCE-FREE) GENDER ASSESSMENT AND OVERALL TRANSLATION QUALITY**. \nThis work maintains a distinction between the evaluation of gender neutrality (as a discrete binary measure) and other continuous measures of overall translation quality (i.e., fluency/adequacy). Hereby, such a distinction was deemed preferable to present a first, pinpointed assessment of novel and specific gender-related aspects. However, integrating these aspects could prove valuable, yielding comprehensive and multifaceted evaluations. This comes with the non-negligible challenge posed by the combination of different ‘scores’ that measure different aspects. To achieve this in future work, in the wake of studies such as [1] and [2], instruction-tuned models could be used to develop multidimensional evaluations based on detailed schemes with the potential to inform scalar quality scores. Also, upon the availability of the created synthetic gender-neutral data (Sec. 5.2), neural metrics and QE could be retrained and adapted for evaluating gender-neutral translation, similarly to [3]. \n\n[1] Fernandes et al., (2023). [The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation](https://arxiv.org/abs/2308.07286).\n\n[2] Lai et al., (2023).  [Multidimensional Evaluation for Text Style Transfer Using ChatGPT](https://arxiv.org/abs/2304.13462).\n\n[3] Sharami et al., (2023). [Tailoring Domain Adaptation for Machine Translation Quality Estimation](https://arxiv.org/abs/2304.08891).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main strengths of this paper are: \n1. It presents the first test set and benchmark for measuring how well MT systems do at appropriately generating gender-neutral language. This will be a valuable resource for understanding this dimension of the translation task.\n2. Before launching the evaluation task, they ran a study assessing the naturalness of language that had been edited to be gender neutral and how speakers reacted to the edited language. This was an important step in justifying the subsequent work, showing that people typically appreciated the neutral language, and that it was not overly contorted.\n3. They show that conventional MT metrics don't capture this dimension of the MT task well, and that other methods are needed.\n4. They train a classifier that is effective at predicting the use of gender-neutral language.\n", "reasons_to_reject": "The main weaknesses are:\n1. The methods used to collect the test set data may not have been ideal. E.g., they used a regex to find instances of gender-unambiguous language, including gendered pronouns. However, in English, gendered pronouns are sometimes used in what should be gender-neutral circumstances (\"To each his own\").\n2. They don't supply statistics about how difficult/natural/realistic it was to create the eval set. I.e., we don't see how often the translators tasked with creating the eval set had difficulty in creating natural gender-neutral edited language, only that one example was so difficult to make gender neutral that only one of the translators attempted it. \n\nMinor:\n1. To create the benchmark set, they post-edited MT outputs to use more gender-neutral language, thus not a fully realistic scenario. However, they discuss this in the limitations section, and it seems like a warranted compromise, given the extreme bias of MT systems to use gendered language.\n2. The reference-free gender classifier doesn’t account for how good the translation is overall, and thus would not penalize translations that are incorrect other than their handling of gender. ", "questions_to_authors": "How would you integrate the proposed evaluation metric with other measures that also attempt to capture other aspects of translation goodness, i.e., fluency/adequacy?\n\nCan you supply judgements of the translators who created the eval set? How often did the translators create gender-neutral translations they were unhappy with? I have some concern that without being very attentive to this concern, we could be at risk of imposing (largely non-inflecting) English-based concepts of language onto other languages where gender has extensive grammatical function, where in English it's largely semantic. A more robust description of the edited gender-neutral translations and their naturalness and appropriateness for native speakers would help allay this concern.", "paper_topic_and_main_contributions": "This paper addresses the challenge of correctly handling gender in machine translation, and particularly making sure that gender-neutral language is used when appropriate. To this end, they start by conducting a study to understand how people react to language that has been changed to be gender neutral when the actual gender of the referent is unknown or irrelevant, and they find that people largely preferred the gender-neutral edited language, particularly in formal settings. \nThey then created a test set of gender neutral and gendered language for EN-IT, as well as a test bed of MT outputs, some of which were post-edited to use correct gender-neutral language, and experimented with the goodness of existing reference-based metrics for evaluating the correctness of whether translation outputs correctly output gendered or gender-neutral forms. They find that existing metrics have little utility for this task, particularly neural metrics.\nFinally, they proposed a classifier for predicting whether a sentence contains gender-neutral or gendered language, and find that it vastly outperforms generic MT metrics not intended for this task.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1VGyJYsna9", "reviewer": "Reviewer_VboY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments, which give us the chance to further elaborate on the goals of our work as well as to improve on the final version of the paper. We are glad to clarify all of the raised concerns – especially those that question the soundness of our work and present overly stringent justifications for paper rejection. We believe that the major problems reported in the review do not subsist and hope to solve remaining doubts surrounding our submission. \n\n\n\n**GENDER NEUTRALITY ASSUMPTION IN THE SOURCE LANGUAGE**.\n\n**(1) Context.** While the data extraction was carried out automatically, all extracted segments were manually checked, selected and annotated as either G (gendered) or N (neutral) by a professional linguist. In doing so, context was definitely taken into account. For instance, “They are police officers…” is to be annotated as N (to be neutralized in translation, since no specific information is given in the source sentence), whereas a sentence like “She is a police officer…” is considered G (to be rendered as gendered in translation, given the presence of the disambiguating gendered cue ‘she’, even though ‘police officer’ is lexically gender-neutral). Further examples for the word ‘colleague’ are also given in Table 2. We stress that the identification of contextual information as gender-specific cues (e.g., ‘she’, but also ‘Mr./Madam’ etc.) was a key portion of the selection/annotation process, as – perhaps too swiftly – mentioned at lines 297-300. Granted the additional page in the camera-ready version of the paper, we will include a more detailed description of this context-aware selection process, which was carried out at the sentence level. Given i) the novelty of the task represented by our corpus, and ii) the fact that current MT landscape predominantly consists of SOTA systems that perform translation at the sentence level, we decided not to account for a larger, document-level context: it would have added additional complexity to an already challenging – and yet to be truly explored – task.   \n**(2) Survey.** To the best of our understanding of the question, the survey did not include *direct* questions on source gender-neutrality (i.e., is this English sentence neutral?). The example sentences included in the survey were selected and created by an expert on the topic of gender-inclusive language to inform i) when and ii) how neutralization should be performed in translation (for more details see also Appendix A). Our approach was therefore *indirect*: by asking participants if they found the neutral translation acceptable compared to a gendered one, we also indirectly assessed the ‘neutrality’ of the source sentence (i.e., if the participants considered it an example that required a neutral translation). \n\n\n**OVER-NEUTRALIZATION**. \nOur work is motivated by the intent to avoid using gendered terms in translation when unnecessary. Such is the case for a source sentence like example C in Table 1. Lower scores for the Neutral Translation here (though we stress that the majority of respondents found the Gendered Translation and the Neutral Translation to be equally acceptable) are to be attributed to the type of target neutralization strategy used, which – as described at lines 235-237 – was found by some respondents to sacrifice adequacy for neutralization. Indeed, we were aware of potential trade-offs to consider in performing neutral translation, and the survey served also to investigate them. Accordingly, it is not that source example C *should not* require a neutral translation; rather, the survey informed us that a different linguistic strategy for neutralization should be employed to better preserve the meaning of the source sentence while still ensuring inclusivity in the target language. In fact, example C is given as a negative example not to follow in the guidelines provided to the translators who created GeNTE’s neutral references (see the supplementary materials, example 7).  \nExamples II and III in Table 2, instead, are cases from the G-SET of the corpus, i.e., sentences that should *not* be neutralized in translation. As described in Sec 4.1 and 5.1., the creation of alternative neutral references for II and III serves the purposes of the contrastive, reference-based evaluation protocol. Please refer to our answer below for more on this point (see *REFERENCE-FREE EVALUATION*). We thus reject the argument that they represent wrongfully created instances of over-neutralization. \n\n\n**MULTIPLE NEUTRAL REFERENCES**. \nBased on a manual analysis of the common-set, the translators produced three identical gender-neutral references in 13.57% of the cases, while an additional 8% of translations exhibited a high degree of similarity (e.g., the same neutral words are used, but in a different order). These statistics are positive: they show that our common-set exhibits a good level of variability (~80%), which is desirable to test open-ended generation tasks like MT. Also – and especially in light of the fact that the translators worked independently – the ~20% of identical/similar neutralizations suggests that neutralizing translation is a challenging but feasible task.  \nWe did not perform a multi-reference evaluation since each entry in the common-set contains 1 gendered reference only vs. 3 neutral references. This imbalance would have compromised the soundness of our experiments with the risk of skewing the results. Granted the extra-space of the camera-ready, we will include the above statistics and these considerations. \n\n**SENTENCE VS DOCUMENT LEVEL GENDER ANNOTATION**.\nThe Europarl corpus consists of parliamentary speeches annotated with the gender of the associated speaker [1]. For GeNTE, however, we did not rely on such document-level annotations, as our corpus represents several types of referents beside the speaker to ensure more variability. As described in our previous answer (see *Gender neutrality assumption in the source language*), the N vs G tags were carefully assigned to each sentence by a professional linguist who accounted for the sentence-level context.  \n\n[1] Vanmassenhove et al. (2018). [Europarl Datasets with Demographic Speaker Information](https://aclanthology.org/2018.eamt-main.59/). \n\n**REFERENCE-FREE GENDER-NEUTRAL MT EVALUATION**.  Our reference-free evaluation metric (i.e. classifier) was purposefully designed to focus only on the assessment of gender-neutral translation and is not intended to also provide information on overall translation quality.  Although integrating these two aspects in a single metric could indeed prove valuable, this comes with the non-negligible challenge posed by the combination of different ‘scores’ that measure different aspects. For this reason -  and in line with current practices in MT evaluation (e.g. those based on test suites) - keeping the two aspects separated was deemed preferable to present a first, pinpointed assessment of novel and specific gender-related aspects.  \nIn Table 5 we compare the results obtained from the classifier with those obtained from a reference-based evaluation protocol that relies on standard MT metrics. This protocol, however, does not exploit those metrics to measure overall translation quality but to make them informative about gender. As described in Sec. 5.1., the proposed contrastive approach – computed on the double set of GeNTE references (Neutral vs Gendered) – assumes that a NEUTRAL output translation should obtain higher scores on the N-reference compared to the G-reference, and vice versa for a GENDERED output translation. In this way, when applied at the sentence level, the contrastive reference-based evaluation protocol that relies on BLEU, TER and METEOR functions as a classifier (e.g. higher BLEU on the N-reference wrt. the G-reference ⇒ output sentence classified as NEUTRAL). Accordingly, in Table 5 we can i) directly and soundly compare the results of the reference-based evaluation with the reference-free one, and ii) show that the latter is more accurate in correctly classifying NEUTRAL vs GENDERED output translations, thus emerging as a first, promising step toward enabling the assessment of inclusivity in MT. \n\n**ASSESSMENT OF GPT-GENERATED DATA**. \nThe description of the generation of GPT-data is reported in Appendix C.1., where we also explain how the data have been assessed by means of a manual analysis on a random sample of 100 triplets (i.e. 300 sentences) for each generation round. \n\n\n**MISSING REFERENCES**. \nWe thank the reviewer for pointing out the study on “Gender suffix in German” by Wagner & Zarrieß (n.4).  It is indeed relevant to our work, and we will make sure to include it in the final version of our paper. Concerning the other references, however, we would like to stress that to consider them *missing* is partially inaccurate and seems overly penalizing. In fact, the work by Savoldi et al., (2021 - n.3) and Cho et al., (2019 -n.5) are both already referenced in the paper – respectively, at lines 37 and 131. Also, paper n.2 is not highly relevant, since it refers to the English, monolingual GAP corpus for coreference resolution on binary gender, and paper n.1 was published on May 29th, 2023, and it is thus concurrent to our submission.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper focuses on a special case of gender bias across translations -- when the source gender is ambiguous or ungiven whereas the translation defaults to specific genders. \n\nAn open-source survey is conducted on 100 participants to assess how acceptable and preferable are the gender-neutral translations for the general audience. \nThe demographical details of the participants are illustrated in the Appendix.\n\nThe paper points out that 97.2% of segments collected from Europarl are biased toward gendered references in Italian.\nThus, GeNTE devotes the effort to manually correcting referent gendered and neutral sentences.", "reasons_to_reject": "Some major concerns:\n- Gender neutrality assumption in the source language. Though some nouns are lexically gender-neutral in English, document context could render these nouns gender-specific. Relevant questions: 1) is context taken into consideration while filtering gender-neutral English sources? 2) did the survey include questions regarding source gender neutrality? \n- Over-neutralization. For specific nouns as in example C of Table 1, much less participants prefer the gender-neural alternative in Italian. Is it necessary for these target translations to be gender-neutral? Also in Table 2, why would we need to neutralize ii and iii for translations if the source English is already gendered? \n- On multiple N references. How frequently do the three translations overlap or are identical? Have you considered multi-reference evaluations? \n- Sentence vs. document level. How are the sentence-level N-vs-G tags migrated, are they inherited directly from the document level? \n- Problematic reference-free gender-neutral MT evaluation. The binary N-G classification accuracy only evaluates the scenario when a gender-neutral noun is translated into Italian. It does not provide any information on the overall translation quality of the documents/sentences thus not linearly dependent and cannot be compared proportionally to standard MT metrics, BLEU, TER, and METEOR. Unfortunately, the much higher \"classifier\" number in Table 5 does not lead to the conclusion that the reference-free evaluation is \"promising\" (line 625).\n\nMinor clarifications are necessary:\n- There is no concrete accuracy assessment on the two rounds of GPT-generated training data;\n- \"Grammatical gender languages\" is a confusing term in this paper. Only concepts that have M-F-N cognates are examined in this paper, but not other nouns that have inherent gender in Italian, such as desks, beds, etc.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces GeNTE, an English-->Italian test set balanced between Neutral (N) and Gendered (G) English nouns, to evaluate faithfulness when translating gender-neutral nouns from English to Italian, a language in which masculine is the default gender. \nIn preparation for the data and evaluations, they conducted a survey regarding N vs. G Italian translations in formal and informal settings. \nThey also evaluate reference-based n-gram and neural evaluation metrics on gender-neutral MT and suggest a reference-free accuracy evaluation on such translations.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "\"ambiguos\" --> ambiguous"}, "extra_text": {"Missing_References": "- A newer Italian Language model: BureauBERTo. https://colinglab.fileli.unipi.it/wp-content/uploads/2023/04/Ital_IA_2023_BureauBERTo.pdf\n- References to other gender datasets such as GAP (https://aclanthology.org/Q18-1042.pdf)\n- More on gender bias in translation: https://aclanthology.org/2021.tacl-1.51.pdf\n- Gender suffix in German: https://aclanthology.org/2022.konvens-1.10.pdf\n- \"In some sense, a well-biased translator is a well-performing translator that reflects\nthe inter-cultural difference.\" from https://aclanthology.org/W19-3824.pdf"}}, {"review_id": "U2gOwybJgt", "reviewer": "Reviewer_sWdu", "summary": "", "questions": "Did you consider revisiting the survey done in section 3 but asking respondents to judge the created data itself?   \n    ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**QUALITY OF THE DATASET & IAA**. \nAs an open-ended task, translation allows for multiple acceptable alternatives rather than a single correct option. For this scenario, inter-annotator agreement does not offer a reliable indication of quality, as it risks penalizing different and yet equally good translations. To assess consistency with a less stringent approach, we instead manually analyzed all the references in the common-set and found that the translators produced identical gender-neutral references in 13.57% of the cases, while an additional 8% exhibited a high degree of similarity (e.g., the same neutral words are used, but in a different order). These statistics are positive: they show that our common-set presents a good level of variability (~80%), which is desirable to test open-ended generation tasks like MT. Also – and especially in light of the fact that the translators worked independently – the ~20% of identical/similar neutralizations suggests that neutralizing translation is a challenging but feasible task.  \n\nTo ensure the good quality of the final data, we also pipelined their creation methodology: i) first, the linguist annotated the gendered words of interest in each sentence, ii) following detailed instructions, the professional translators worked to neutralize the already identified gendered expressions, and finally, iii) the linguist revised the neutral translations produced. From a qualitative perspective, two main type of challenges were identified in the creation of the neutral references:  \n\n **(A) Articles**: in 11 instances, the translators produced partial neutralizations, as they overlooked masculine articles. This possibly suggests that, in Italian, articles may be perceived as secondary in expressing gender compared to nouns, adjectives, and verbs, even by native speakers. Regardless, all errors were spotted by the linguist and corrected.\n\n**(B) Lexical gender**: translators were unable to produce a neutralization for 4 instances of lexically gendered nouns such as ‘sorella’ (sister) and ‘figlia’ (daughter). Such cases all concerned the creation of neutral references for the SET-G – which served the purpose of the contrastive evaluation (sec. 5.1) – but were particularly challenging as they required the use of neutral strategies for unequivocally gendered terms. \n\nWe appreciate the reviewer’s interest in these aspects of the creation of the corpus and we will consider including this discussion and data in the camera-ready version of the paper.\n\n**ADDITIONAL SURVEY TO JUDGE THE CREATED DATA**. \nWe deliberately placed the survey at the outset of our research, so as to gather valuable insights head-on and provide direction for the subsequent development of the GeNTE corpus, drawing on valuable feedback from end-users. Given such feedback and insights, we then entrusted the expert linguist and professional translators to successfully implement them in the creation of the corpus. While we believe that the employed methodology represents the soundest and most natural sequence of steps to ensure the good quality of our data, closing the whole process with a second round of end-users’ feedback would indeed be interesting and could further validate our corpus and task. We thank the reviewer for the suggestion, which we will consider for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Very useful test set.   Excellent introduction and motivation for the problem of gender neutral translation.  The background and survey are really nicely presented, as is the small-scale study surveying translator's preferences wrt neutral translations.    Overall,  a very strong case is made for working on the problem and for the GeNTE corpus.   The work is likely to be influential.", "reasons_to_reject": "As the authors note in their limitations section,  the collection focuses on English->Italian translation,   and so the data and classifier models can't be used directly for other target languages.    Personally,  I'm ok with this,  as I think these subtle issues are best analyzed for specific target languages.   While this is a limitation, the paper also provides a starting framework for approaching other target languages.   \n\nI may have missed this in the discussion,   and I did not read the appendices,  but apart from a summary statement (line 366 - `the linguist supported the translators throughout the process and finally checked all the neutralizations')  there isn't much discussion about the quality of the data set that was produced.    It would be interesting to know whether there were challenges with inter-annotator agreement, i.e. whether the linguist needed to intervene often,  or how often the annotators 'failed to neutralise' (Table 2, iii-F,  line 395).   \n", "questions_to_authors": "Did you consider revisiting the survey done in section 3 but asking respondents to judge the created data itself?   \n    ", "paper_topic_and_main_contributions": "The paper describes the creation and analysis of GeNTE a `natural, bilingual test set for gender-neutral translation, whose creation was informed by a survey on the perception and use of neutral language.'     The authors provide an accurate summary of their contributions:\n(1) A study on the feasibility of neutral translation, (2) creation of GeNTE,1 the first natural, parallel corpus designed to test MT systems’ ability to generate neutral translations; (3) A comprehensive analysis of the (un)suitability of existing automatic metrics to evaluate neutral translation.  ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I'm not sure if this is intentional,  but I'd suggest settling on a single spelling of 'ambiguous'.\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a valuable resource to mitigate gender bias in machine translation in the form of GeNTE, an English to Italian test set for gender-neutral translation. It also investigates existing reference-based metrics for evaluating the quality of neutral translations and proposes a reference-free method that addresses the formers’ shortcomings. The strengths of the paper lie in its novelty and thoroughness, as indicated by its Soundness (three 4s, one 3) and Excitement (three 4s, one 3) scores. Multiple reviewers praised the clarity of the introduction, the motivation (particularly the human subjects survey on attitudes toward gender-neutral language), and discussion of limitations, though individual concerns were raised in regards to methodology, e.g. the use of regexes in data collection, the assumption that gender-neutral lexical items remain neutral in context, etc. The authors nevertheless addressed each of these concerns in their rebuttals to reviewer satisfaction.

In light of this, only minor revisions, addressing reviewers’ comments and questions, need to be made to ensure this paper is camera ready.
Decision: Accept-Main