Target-Agnostic Gender-Aware Contrastive Learning for Mitigating Bias in Multilingual Machine Translation
Authors: Minwoo Lee
Hyukhun Koh
Kang-il Lee
Dongdong Zhang
Minsung Kim
Kyomin Jung
Venue: EMNLP 2023 Main (2023)
Keywords: machine translation
bias mitigation
contrastive learning

Abstract: Gender bias is a significant issue in machine translation, leading to ongoing research efforts in developing bias mitigation techniques. However, most works focus on debiasing bilingual models without much consideration for multilingual systems. In this paper, we specifically target the gender bias issue of multilingual machine translation models for unambiguous cases where there is a single correct translation, and propose a bias mitigation method based on a novel approach. Specifically, we propose Gender-Aware Contrastive Learning, GACL, which encodes contextual gender information into the representations of non-explicit gender words. Our method is target language-agnostic and is applicable to pre-trained multilingual machine translation models via fine-tuning. Through multilingual evaluation, we show that our approach improves gender accuracy by a wide margin without hampering translation performance. We also observe that incorporated gender information transfers and benefits other target languages regarding gender accuracy. Finally, we demonstrate that our method is applicable and beneficial to models of various sizes.

Target-Agnostic Gender-Aware Contrastive Learning for Mitigating Bias in Multilingual Machine Translation
Minwoo Lee
Hyukhun Koh²
Kang-il Lee'
Dongdong Zhang'
Minsung Kim
Kyomin Jung
Dept. of ECE, Scoul National University, 2IAAI, Scoul National University,
Microsoft Research Asia
(minwoolee, hyukhunkoh-ai, ai, 4bkang, kms0805, kjung)@snu.cd ac. kr
bechang@microooof..cm
Abstract
[EN]
Input Text
The receptionist welcomed the lawyer because this is part of his job.
Gender bias is a significant issue in machine translation, leading to ongoing research efforts developing bias mitigation techniques. How- ever, most works focus debiasing bilingual models without much consideration for mul- tilingual systems. In this paper, we specifi- cally target the gender bias issue of multilin- gual machine translation models for unambigu- ous cases where there a single correct trans- lation, and propose a bias mitigation method based on a novel approach. Specifically, we propose Gender-Awan Contrastive Learning, GACL, which encodes contextual gender infor- mation into the representations non-explicit gender words. Our method is target language- agnostic and is applicable to pre-trained mul- tilingual machine translation models via fine- tuning. Through multilingual evaluation, show that our approach improves gender ac- curacy by wide margin without hampering translation performance. We also observe that incorporated gender information transfers and benefits other target languages regarding gen- der accuracy. Finally, we demonstrate that our method applicable beneficial models of various sizes.!
[FR]
réceptionniste salué l'avocat...
[DE]
Multilingual NMT
Die Empfangsdame begrüßte den Anwalt...
[FR]
réceptionniste salué l'avocat....
[DE]
Fine-tuning
Der Empfangschef begrüßte den Anwall...
Figure Example sentence from the WinoMT bench- mark and the corresponding translation outputs of mul- tilingual NMT systems. Existing systems often fail to translate with correct gender inflections.
Efforts have been made in recent studies to mit- igate the gender bias issue in machine translation (Choubey et al., 2021, Saunders and Byme, 2020, Costa-jussà and de Jorge, 2020). However, most of the works focus mitigating bilingual NMT mod- els and evaluate single language direction. Re- cently, Costa-jussà al. (2022) demonstrated that the shared encoder-decoder architecture in multilin- gual NMT systems leads to worse gender accuracy compared to Linguage-ppecifii modulcs. Nonethe- less, it remains unclear whether the existing debi- asing methods would yield similar effectiveness in multilingual NMT models.
1 Introduction
In machine translation research, gender bias has emerged significant problem, with recent neu- ral machine translation (NMT) systems exhibiting this bias (Prates al., 2020). This bias can mani- fest in various ways, such as the misgendering of individuals based on stereotypes or defaulting to masculine gender translations. As a result, there is a growing need to address and mitigate gender bias in machine translation systems to ensure fair and unbiased translations that accurately reflect the in- tended meaning without perpetuating gender-based assumptions.
In this work, investigate in detail the gender bias issue of multilingual NMT models. We focus on translating unambiguous cases where there is only one correct translation with respect to gender. We consider multiple target languages simultane- ously with various gender-based metrics and find that even the sate-oofthheaal multilingual NMT sys- tems still exhibit tendency to prefer gender stereo- types in translation.
Therefore, we propose new debiasing method for multilingual MT based on a new perspective of the problem. We hypothesize that the gen- der bias unambiguous settings is due to the
"Work done during internship at MSRA. Code available at httpp://gt /ag:::///ihhuie thub. com/mi hww.o///CCh

---

lack of gender information encoded into the non- explicit gender words and devise scheme to in- ject correct gender information into their latent embeddings. Specifically, we develop Gender- Aware Contrastive Learning, GACL, which as- signs gender pseudo-labels to text and encodes gender-specific information into encoder text rep- resentations. Our method is agnostic to the target translation language the learning is applied the encoder side of the model and can be applied debias pre-trained NMT models through fine- tuning. We also evaluate whether existing debias- ing techniques for bilingual NMT are equally ef- fective for multilingual systems and compare their effectiveness on different target languages.
Gender-related data filtering
Gender-aware
Contrastive Learning
Parallel corpus
Masculine
Feminine
CGc
LMT, LKD
Masculine
Feminine
NMT Model
Figure 2: Overview of proposed method. Train data is first filtered for sentences containing gender- related words. The processed dataset used fine- tune NMT model using machine translation and gender- aware contrastive learning objectives.
Experimental results show that our method is highly effective at improving gender bias metrics for all 12 evaluated languages, with negligible im- pact on the actual translation performance. We find our approach applicable to various model archi- tectures and very efficient that it demonstrates significant gender accuracy improvement with just a few thousand steps of fine-tuning. We also dis- cover that the debiasing effects extend to target language directions that are not trained on previous models. Through further analysis, demonstrate that our method effectively incorporates contextual gender information into the model encoder repre- sentations.
multilingual models that have encoder-decoder ar- chitectures and support English-to- many transla- tion directions. We filter gender-related data and fine-tune the pre-trained NMT model with the fil- tered data. The overview of the method is shown in Figure
2.1 Data Filtering and Preprocessing
We first filter the parallel train data for sentence pairs that contain gendered words in the English source sentence using the gender-related word list by Zhao et al. (2018). We exclude sentences that contain either no gendered words or gendered words of both male and female genders. After fil- tering the train data, we undersample the larger of the sentence pairs containing male and female gen- der words so that the number of samples for each gender is the same, in similar fashion to Choubey al. (2021).
In summary, the contributions of our work are as follows:
We find that recent multilingual NMT mod- els still suffer from gender bias and propose GACL, novel gender debiasing technique for multilingual NMT models based trastive learning.
2.2 Gender-aware Contrastive Loss
To the best of our knowledge, we are the first to show that the gender debiasing effect trans- fers across other languages multilingual NMT models that were not fine-tuned.
We devise contrastive loss that incorporates gen- der information into the encoder embeddings. Al- though the optimal approach would be to apply the contrastive scheme exclusively words that exhibit gender-based translation variations, this varies depending on the translated language and challenging to know in advance. Hence, we use mean-pon srean-pooled sentence-level embeddings for our contrastive learning scheme instead.
Through extensive evaluation and analysis, we show that our method is effective across mul- tiple architectures while having negligible impact on translation performance.
Given h, the encoder embedding of the souree sentence, we define positive samples be the set of sentence representations that have the same gender as h; and negative samples as the set of representa- tions that have the opposite gender. We correspond- ingly formulate contrastive loss as follows:
2 Method
In this paper, we propose GACL, gender-aware contrastive learning method for mitigating the un- ambiguous gender bias issue in multilingual ma- chine translation. Our approach applicable to

---

Four metrics are used to measure the gender bias with the WinoMT dataset.
Go =
Σ KtEH!
log EwentUng esim(hade)/
Accuracy measures whether the occupation is translated with the correct gender inflection based on the pronoun. The occupation word is deter- mined using source-target alignment algorithm. and the inflected gender is detected using target language-specifie morphological analysis.
where H is the set of positive samples, Hi is the set of negative samples, sim(.,) the cosine similarity function, and T the tempera- ture hyperparameter. Our formulation is equiva- lent to the supervised contrastive loss by Khosla et al. (2020), where we use the gender informa- tion as pseudo-labels to define positive and neg- ative pairs. In practice, we use positive samples H1* (M,) U [hyl9g = 9i) where NV is the repre- sentation based on different dropout seed, as in Gao al. (2021), and h are in-batch samples with the same gender marking gs. For negative samples, we use H, = {h& Mi # gi) where h4 are the in-batch samples with different gender markings.
AG AcCmate Accfemale measures the dif- ference in accuracy between male sentences and female sentences.
AS Accpro - Accanti measures the dif- ference in accuracy between sentences with pro-stereotypical and anti-stereotypical gender- occupation pairings defined by Zhao et al. (2018).
AR = Recallmate - Recall suggested by Choubey et al. (2021), measures the difference in the recall rate of male and female sentences. MT-GenEval (Currey al., 2022) is a recently released gender accuracy evaluation benchmark that provides realistic, gender-balanced sentences in cendder-mammiguous settings. We use the coun- terfactual subset, where for each sentence, there exists counterfactual version in the set with only the gender changed. MT-GenEval supports eight target languages: Arabic, German, Spanish, French, Hindi, Italian, Portuguese, and Russian.
In addition to the gender-aware contrastive loss, we train model with the original machine trans- lation loss to prevent forgetting. We also add knowl- edge distillation loss with the frozen machine trans- lation model as the teacher model to preserve the translation performance (Shao and Feng, 2022). In sum, our training objective is as follows:
Ltrain a1 LMY + a LKD + X Lac,
Four metrics are used to measure the gender bias with the MT-GenLival dataset.
where the machine tranlation loss LMT and the knowledge distillation loss LKD is added with weights based hyperparameter a, and our pro- posed loss Loc is added with multiplied hyperpa- rameter
Accuracy is measured based on whether the un- ambiguously gendered occupation has the correct gender inflection. Unlike WinoMT, however, ac- curacy is measured differently; using the counter- factual variants, words that are unique to single gender are extracted for each sentence, and trans- lation is marked correct if the words unique to a different gender are not included in the translation. However, this definition of accuracy has prob- lem in that even if the translation is incorrect, it could still be marked correct if the words unique to the different gender are not be contained. To avoid this problem, we define an alternative measure of accuracy, denoted Explicit Accuracy. In measur- ing explicit accuracy, a translation is marked cor- rect the words unique to different gender are not included in the translation, and the words unique to the same gender are explicitly included in the translation. This definition makes explicit accuracy stricter version of the original accuracy.
3 Experimental Framework
this section, we describe the details of the ex- periments, including the data, metrics, bascline methods, training architecture, and parameters.
3.1 Dataset and Metrics
order to measure the unambiguous gender bias in machine translation systems, we employ two eval- uation benchmarks: WinoMT and MT-GenEval. WinoMT (Stanovsky et al., 2019) is widely used gender bias evaluation benchmark consist- ing of 3,888 English sentences, where each sen- tence contains an occupation and gendered coref- erential pronoun. WinoMT supports ten target lan- guages: German, French, Italian, Ukrainian, Polish, Hebrew, Russian, Arabic, Spanish, and Czech.
AG AcCmate AcCfemale and E-AG =

---

Explicit feeeeae measures the difference of male and female sentences in terms of accuracy and explicit accuracy respec- tively.
85
p-0.
pa -99
71-
Neteety
85
We use the FLORES-200 (Costa-jussà et al., 2022), a standard multilingual NMT benchmark, to measure the translation performance of our mod- els. FLORES-20 200 consists of 3,001 sentences sam- pled from English Wikimedia projects and profes- sionally translated into 200+ languages. We use SentencePiece BLEU (spBLEU) and ChrF++ as evaluation metrics.
15-
-000)
68-
54
2
44-
15
10-
3.2 Baselines
36
32 40
#D0LEU
We compare three baseline methods that have been previously proposed to mitigate gender bias in ma- chine translation.
Figure 3: Relationships between translation perfor- mance and gender bias metrics of multilingual NMT models. Each point represents the average score of NMT system on the ten target languages of the WinoMT dataset.
Balanced: Costa-jussà and de Jorge (2020) pro- posed filter existing parallel corpora for sen- tences with gender mentions and subsample the data create a balanced version of the dataset. We fine-tune the machine translation system on the bal- anced dataset. We use the WMT18 en-de dataset as processed by Edunov et al. (2018).
schedule. The hyperparameters for the contrastive learning objective was set to À 1.0 and 0.4 based hyperparameter search (refer to Appendix B). We evaluate every 100 training steps and early stop based on the Explicit Accuracy score of the MT-GenEval development set for the fine-tuned language direction. For evaluation, we use beam search of beam size 5 during decoding.
GFST: Choubey et al. (2021) proposed method balanced parallel dataset with
to create gender-balanced source- and target-language filtering of pseudo- parallel data. In work, instead of re-training the model with GFST from scratch, fine-tune the initially trained model with the target-filtered data. We use the same news2018 corpus used in the original work for the monolingual corpus.
Results
Handcrafted: Saunders and Byrne (2020) pro-
In this section, we explore the gender bias issue of recent multilingual NMT models and report exper- imental results of the gender bias mitigation tech- niques multilingual machine translation models.
posed to use a small, high-quality handcrafted par- allel dataset containing balanced combination of gendered pronouns and occupations to fine-tune the machine translation system. We use the hand- crafted en2de dataset provided by the authors in our work, which consists of 388 samples.
4.1 Gender Bias Evaluation of Recent Multilingual NMT Models
We first evaluate existing multilingual NMT mod- els on gender bias and analyze their relationship with their translation performance. We test seven model variants of different sizes based on three model architectures: M2M-100, SMaLL-100, and NLLB-200. For gender bias, we evaluate on the WinoMT dataset for all 10 languages and average the results. Similarly for translation performance, evaluate the FLORES-200 devtest on the same 10 language directions and average the results. For the correlation measure between trans- lation performance and gender bias metrics, we use Pearson's 's correlation coefficient p.
3.3 Implementation Details
We use three pre-trained multilingual model ar- chitectures as our backbone for our experiments: M2M-100 (Fan et al., 2020), SMaLL-100 (Mo- hammadshahi et al., 2022a), which is a knowledge- distilled model from M2M-100, and NLLBB-200
(Costa-jussà et al., 2022). Due to resource limi- tations, we use 1.2 billion parameter variant for M2M-100 and 1.3 billion parameter distilled vari- ant for NLL.B-200 We train with batch size 8 and lcarning rate 4e-6 with 200 warmup steps and inverse square root learning rate decay

---

FLORES-200
WinoMT AGlu
80
Method
spBLEU,
Acc.t
AS
SMaLL-100
32.02
46.25
31.43
9.85
Accuracy
60
M2M-100
34.60
49.66
26.85
18.14
40
NI.LB-200
40.92
62.44
9.37
24.94
20
ChatGPT 27.02'
54.25
21.09
24.95
Table Average spBLEU score WinoMTT metrics of NMT systems and ChatGPT (gpt-3.5- t-tuubbo) the ten target languages of the WinoMT dataset. *ChatGPT spBLEU score obtained from al. (2023).
30
8
10-
As shown Figure 3, we find a strong positive correlation (p = 0.98) between the translation per- formance and the gender accuracy. As shown by negative correlation of AG (p = - 0.97) and A R 0.96),, the accuracy and recall gap between genders are also reduced translation performance improves. However, the correlation between trans- lation performance and AS positive (p = 0.91), implying that cetter-performing models rely more on occupation stereotypes rather than the original context for gender disambiguation.
he.
Uk
Tergell Language
Figure 4: Gender accuracy and AG of individual tar- get languages supported by WinoMT. Results of the SMaLL-I0M baseline model, model fine-tuned on en2de dataset, and model fine-tuned on en2tr dataset are re- ported.
the model single language direction and ob- serve its effect the corresponding language direc- tion (denoted in-domain, ID) as well as other lan- guage directions (denoted out-of-domain, OOD). We use the WMT18 en-de dataset (Edunov et al., 2018) for fine-tuning on English to German lan- guage direction. For evaluation, all target lan- guages supported by the dataset were evaluated. This involves 10 target languages for WinoMT and for MT-GenEvar Finally, languages covered by the datasets for evaluat- ing translation performance on the FLORES-200 dataset, which amounts to 12 target languages.
Overall, conclude that recent multilingual models continue to show similar tendencies MT systems previously reported by Koemi al. (2020), with positive correlation with gender aceuracy, neg- ative correlation with AG, and positive correlation with AS. This suggests that the development of NMT systems with a unidimensional focus on per- formance is insufficient to address the gender bias issue, and active consideration is required.
the union of the
Recently, ChatGPT has shown remarkable per- formance in various zero-shot NLP tasks, includ- ing machine translation. We evaluate the gen- der bias ChatGPT 1ott33.-turbbe in per- forming zero-shot machine translation. We use the prompt "Translate the following sentence into <lang>. <seni>", where <lang> is the name of target language and <sent> is the source sentence. As shown in Table ChatGPT falls short regard- ing the spBLEU score but achieves relatively high gender accuracy, surpassing M2M-100 with 54.25. However, we also find that the AS the highest for ChatGPT, indicating that its translation output often relies on gender stereotypes.
As shown in Table 2, the results on the SMaLL- 100 model show that our method, GACL, achieves the greatest improvement in in-domain gender ac- curacy for both WinoMT and MT-Gen with 27.3 and 13.0 absolute improvement from the base- line respectively. On the other hand, other baseline methods that were originally proposed for bilingual MT' systems proved to be less effective compared to our method. GFST was proved to be the least effective out of the baselines, with less than 3% improvement in gender accuracy, and fine-tuning on the Handcrafted set second was most effective, with 20.6% improvement. Based these results, suggest that for multilingual MT models, it is more effective use smaller, focused dataset on gender and occupation where the bias of the model exhibited.
4.2 Main Experimental Results
We report the results of fine-tuming multilingual NMT model with our GACL method along with other bascline methods. Specifically, we fine-tune
#hapss//chat.openaa.com
As shown by OOD gender accuracy and |AG]

---

WinoMT
MT-GenEval
FLORES-200
10
Method
Asc.
AG
Asc.
AAG|
E-Acc.t
E-AG|11
E-Acc.t
|E-AG]|
spBLEU,
spBLEU,
Baseline
57.4
24.2
45.0
32.2
54.7
10.0
42.3
27.1
36,0
32.7
Balanced
72.9
3.8
49.7
22,4
56.7
8.3
44.1
23.9
35.4
32.4
GFST
59.9
20.9
46.2
31.1
57.3
10.3
41.9
27.2
35.5
32.4
Handcrafted
78.0
-1.5
52.7
18.4
58.7
4.0
45.1
22.4
35.8
32.7
GACL (Ours)
84.7
-3.7
65.5
8.1
67.7
-2.0
56.2
12.9
36.0
32.7
Table Main experimental results the WinoMT, MT-GenEval, and FLORES-200 datasets for the SMaLL-100 model. The in-domain (ID) setting signifies the en-de language direction in which the model is fine-tuned, while the out-of-domain (OOD) setting encompasses the remaining language directions supported by the dataset.
WinoMT
MV-GenEval
FLORES-200
Method
Acc.t
|AG|,
[AS],
[&R
E-Acc.
E-AGI_
Acc.
[AG|,
spBLEU,
ChiF++y
SMaLL-100
Baseline
46.2
31.4
9.9
57.5
43.8
25.0
57.9
25.6
33.0
52.3
GACL (Ours)
67.4
7.7
6.4
18.4
57.6
11.5
73.2
11.1
33,0
52.2
with Larr Only
52.0
20.6
9.3
44.2
45.7
22.0
61.3
21.6
32.6
51.8
with Loe Only
63.5
9.0
7.5
21.7
55.3
13.0
71.6
12.7
32.5
51.7
M2M-100
Baseline
49.7
26.8
18.2
53.9
44.9
23.5
58.0
23.9 9
35.5
53,6
GACL. (Ours)
71.4
6,4
7.3
15.5
59.3
8.8
73.3
9.2
34.9
53.1
NLLB-200
Baseline
61.7
9.0
24.5
29.3
57.1
16.7
66.4
16.8
40.5
57.2
CACL(Our))
78.2
3,9
6.1
6.2
69.9
5.9
80.4
5.5
40.0
56.9
Table 3: Full experimental results averaged all covered languages the WinoMT, MT-GenEval, and FLORES- 200 datasets. Evvaluated model architectures are SMaLII-100, M2M-100-I.2B, and N...B-200-13B--iiiiii
metrics in Table 2, gender bias mitigation strategies also have positive effect on the unseen target lan- guages during fine-tuning, regardless of the method used. This implies that gender-related information is agnostic to the target language, and the debiasing effects are transferred to other languages. However, while other baseline methods have a much lower improvement in OOD than ID, our approach al- most effective in OOD. Fine-tuming on the Hand- crafted set improves WinoMT accuracy by 20.6% in ID and 7.7% in OOD, while GACL approach improves by 25.7% and 20.5% respectively.
significantly improve gender-related translations.
4.3 Results for Individual Target Languages
We report the individual results for each target lan- guage on the WinoMT dataset in Figure 4. To ob- serve the effect of the target language used during GACL fine-tuning, evaluate and report two language directions: English German (en2de) and English Turkish (en2tr). In contrast with the German language, which has rich gender mor- phology, the Turkish language is a gender-neutral language that lacks grammatical gender. We use the same TED2020 corpus (Reimers and Gurevych, 2020) for getting gender-balanced training data to rule out the effect of data domain from our experi- ments.
We report the full results all evaluated metries and model architectures in Table We observe that applying GACL improves upon all gender ac- curacy and bias metrics for all evaluated model architectures. Especially, we find that |AS] metric of NLLB-2000, which scored highest out of all meth- ods before fine-tuning, is reduced to 6. 1, the lowest out of all methods. On the other hand, the SpBLEU and ChrF++ metrics for M2M- 100 and NLLB-200 drop by an average of 0.5 points. We suggest that catastrophic forgetting did not occur during fine-tuning due to the model's fast conver- gence. Still, the fine-tuning was long enough to
Results show that the en2de-finetuned model has higher gender accuracy than the en2tr-finetuned model by an average of 3.6%. However, using en2tr is quite effective improving gender curacy and reducing AG on all evaluated target languages. Since the target language of Turkish does not contain gender-related words, results sug- gest that the gender-related knowledge is accessible from the source encoder representations, and our
find that

---

Positive
Negative In batch
MT-GenEval
MBX
Dropoct
In batch
Ratio
Acc.
[AG]
Baseline
61.01
8.95
1:B
66.82
3.11
B-1:B
66.
3.02
B:B
66.93
2.83
Table Ablation of different contrastive pair combina- tions on the MT GenEval development set. B denotes one-half of the train batch size.
10
biseline
GACL
Figure 5: t-SNE visualization of encoder output repre- sentation of occupation words. Circle and cross markers denote embeddings of the original SMaLL-100 model and GACLL-finttune variant, respectively. The colors of the marker denote the context the occupation word.
approach is able to mitigate the bias that lies within
it.
4.4 Ablation Study
Analysis
We compare the effects of using different con- trastive samples in Table 4. First, observe that using single dropout representation as the positive sample and in-batch sentence representations with different gender as negative samples achieves sub- stantial performance improvement on both gender accuracy and AG. We can also see that using just in-batch samples with the same gender as positive samples can similarly improve the gender accu- racy. However, we find that incorporating all avail- able samples within the batch for positive samples achieves the best results.
In this section, we investigate how our GACL fine- tuning method affects the model representations regarding gender. Specifically, we use 40 stereotyp- ical occupation words from the WinoMT dataset, where 20 are stereotypically assumed male while the remaining 20 are assumed female. We label the "stereotypical gender" of an occupation word as de- fined by this gender assumption. We then construct a sentence with the template "He is <occupation>." and "She is <occupation>." as encoder input. Here, the pronouns decide the "contextual gender" of the occupation word. Finally, single contextual repre- sentation vector is computed by taking the average of encoder output representations of the tokens that make up the occupation word. We use this same representation extraction process for both the base- line SMaLL-100 model and the SMaLL-100 model fine-tuned with GACL.
We also perform ablation study on training with just machine translation loss LIT,, which is equivalent the Baseline method, and just the gender-aware contrastive loss LCc- The results shown Table 3 show that training with MT on a gender-balanced balanced dataset improves over the baseline by a relatively small amount for all metrics. On the other hand, training with just LGe loss achieves surprisingly high performance on both gender eval- uation benchmarks. We point out that training on Lce only updates the encoder parameters of an encoder-decoderr model, and thus having gender- aware contextual embedding the encoder repre- sentations can be effective. We also observed dur- ing the fine-tuning process with only LGe that the performance converges quickly within 200 steps, and upon further fine-tuning, both translation per- formance and gender accuracy deteriorate very quickly. We thus conclude that losses like Carr and CKD are required to prevent catastrophic forgetting during fine-tuning and enable stable training to get optimal convergence on gender-aud gender-aware aware contrastive learning.
To examine the representations, we employ the t-SNE dimension reduction technique (van der Maaten and Hinton, 2008) to visualize the occu- pation representations in 2 dimensions, as shown in Figure 5. We observe that the representations for each occupation are clustered closely together regardless of the sentence context and model. This shows that the contextual gender has a relatively little contribution to the representation compared to the semantic meaning of the occupation. Also, our fine-tuning method induces relatively small change, preserving the semantic distinction be- tween the occupations. Finally, we note that the average distance between representations of differ- ent contexts is farther apart for the GACL repre- sentations (0.59) than the bascline representations (0.19), suggesting that the contrastive objective has

---

Cheng et al. (2021) and He al. (2022) use a con- trastive learning scheme reduce bias in sentence embeddings and language models respectively.
08-
0..
0.
2.50
0.0
Hacel Basaing CXCL
Multilingual Machine Translation Studies such as mBERT (Pires et al., 2019) and XLM-R (Goyal et al., 2021) have shown that it is possible to train language models on multiple languages si- multaneously, method referred to as multilingual training. Recent research has proven that multilin- gual training contributes NMT (Aharoni et al., 2019; Tran et al., 2021; Chi- ang et al., 2022). According to Carrión- and Casacuberta (2022), by training multilingual NMT models further with a few-shot regularization, a de- crease in the performance can be prevented. Knowl- edge distillation also helps NMT models preserve their original performance (Shao and Feng, 2022).
0.2
0.2
3.01 00
0.0
stereoppe
lacal
Figure 6: Clustering results on occupation word embed- dings based on stereotypical and contextual gender label assigmments. Lower scores are better for stereotypical gender labels, and higher scores are better for contextual gender labels.
positive impact on
guided the model to differentiate the occupation based on the gender context.
To investigate how much gender information encoded within the embeddings in-depth, we per- form k-means clustering with k = on the occu- pation embeddings, and evaluate the cluster quality based on stereotypical and contextual gender as label assignments. Based on the Purity and Nor- malized Mutual Information (NMI) metrics, see that clusters for both models have negligible alignment with the stereotypical gender assignment (Figure 6). On the other hand, we find that clus- tering based on GACL embeddings is very well aligned with the contextual gender, while the base- line model continues to be misaligned. This shows that GACL representations capture contextual gen- der information significantly better than the base- line representations.
Gender Bias in Machine Translation Bilin- gual NMT models have been shown to be eas- ily exposed gender biases (Prates al., 2020). Correspondingly. Zhao et al. (2018); Choubey et al. (2021); Currey et al. (2022) employ data augmentation-basee approach to reduce gender bias of MT models. In addition, Saunders and Byrne (2020) propose utilizing a transfer-leaming method, and Savoldi et al. (2021) develop unified framework to tackle the biases. However, these works mostly do not consider multilingual NMT models that support multiple language directions.
A Alternatively, Fleisig and Fellbaum (2022) pro- pose an adversarial learning framework to miti- gate gender bias in machine translation models by removing gender information when the input has masked gender context. Our approach, on the other hand, injects the correct contextual gender informa- tion from encoder output contrastively when given inputs have gender contexts.
6 Related Works
Gender Bias in NLP Chung et al. (2022) point out that the existing English-centric Large Language Models (LLMs)) suffer from gender-stereotypical words. However, fixing biases that are deeply in- grained in hidden representations a challenging task (Gonen and Goldberg. 2019; Orgad and Be- linkov, 2022). Previous researchers such as Rav- fogel et al. (2020); Kumar et al. (2020); Gaci et al. (2022b) debias the hidden representation using learning-based methods. However, Kumar et al. (2022) point out the limitations of such studies and recommend the use of data augmentation tech- niques (Webster et al., 2020; Sharma et al., 2021; Lauscher et al., 2021).
In the case of multilingual machine translation, Costa-jussà al. (2022) have shown that the shared encoder-decoder architecture of multilingual NMT models has a negative effect on the gender bias. Also, Mohammadshahi et al. (2022b) investigate how the multilingual model compression affects gender bias. Contemporary work by Cabrera and Nichues (2023) examines gender preservation in zero-shot multilingual machine translation. How- ever, no existing work that the authors are aware of specifically considers mitigating the unambigu- ous gender bias of multilingual NMT models for multiple language directions simultaneously.
addition to aforemen- tioned research, Attanasio et al. (2022) and Gaci et al. (2022a) focus on biased attention weights, and

---

language coverage may restrict the generalizability of the study's 's results a broader linguistic context. gender bias of multilingual NMT models, specif- ically focusing on unambiguous cases and eval- uating multiple target languages. Our findings indicated that even state-of-the-ar multilingual NMT systems tend to exhibit preference for gender stereotypes in translations. We then pro- posed a novel debiasing method, Gender-Aware Contrastive Learning (GACL), which injects con- textually consistent gender information into latent embeddings. Our experiments demonstrated that GACL effectively improves gender accuracy and reduces gender performance gaps in multilingual NMT models, with positive effects extending to target languages not included in fine-tuning. These findings highlight the importance of addressing gender bias in machine translation and provide a promising approach to mitigate in multilingual NMT systems.
7 Conclusion
this work, we conducted an investigation into the
The focus of this study primarily on evalu- ating gender bias in unambiguous settings where the intended gendered terms are clear. However, the investigation of gender bias in ambiguous set- tings, where the gendered term can have multiple interpretations, is not addressed in this study. Con- sequently, the study does not provide insights into potential biases in ambiguous gendered language use, which can also contribute to societal biases and stereotypes.
Our work focuses recent NMT models based on the encoder-decoder architecture, and hence the effect of our method on decoder-only NMT models remains unverified. Nevertheless, our approach is applicable to other model architectures as long as we can aggregate representation of the source input sentence. Specifically for decoder-only archi- tectures, one feasible strategy would be to pool the decoder model outputs of tokens up given souree input sentence. As decoder-only large language models such as ChatGPT are increasingly being considered for the MT task, we believe this an interesting direction for future work.
Limitations
The gender debiasing process in this study relies on a curated list of gender word pairs to identify and filter gendered terms in the dataset. However, this approach may not cover the whole of the gendered terms present in the world. The limited coverage of gendered terms could potentially introduce biases and inaccuracies in the evaluation results, certain gendered terms may be missed or appropriately accounted for.
Ethical Considerations
Our work attempts to reduce bias and misrepre- sentations in translation of masculine and femi- nine gendered referents. Our methodology and evaluation has been limited to considering binary genders, which overlooks non-binary genders and correspondingly doesn't consider bias in gender- inclusive and gender-neutral translations. Possible mitigations include extending the translation data to incorporate sentences with gender- inflec- tions and defining a separate gender pseudo-label for applying proposed contrastive loss. The lack of flexibility in gender could also be mitigated by extending our work to controlled generation where the preferred gender inflection is given as input.
Furthermore, our method only deals with binary gender, and do not consider the possible representa- tions mno--inaryy genders and their bias in transla- tion. As languages vary in their use of grammatical gender and they often lack clearly defined rules or established linguistic structures for non-binary genders, it is especially challenging to evaluate and mitigate bias in this context. This limitation highlights the need for further research and consid- eration of more diverse gender representations in bias mitigation.
Furthermore, in our work, the evaluated source language was limited to English, and evaluated target languages are mostly of high-resource. Correspondingly, our work may under-represent bias found in unevaluated low-resource languages. However, the findings in our work show poten- tial gender debiasing effects transferring to non fine-tuned languages, and extending gender bias evaluation resources to include low-resource lan- guages may help analyze and mitigate gender bias
While we extend gender bias evaluation to mul- tilingual settings, this study is still limited to the provided target languages, which predominantly in- clude medium-to-hipe high resource languages. Due to lack of evaluation data, the evaluated source lan- guage is also limited to English. Consequently, the findings and conclusions may not be representative of the gender biases present in languages that are not included in the evaluation. The limitations in

---

for those languages.
Pengyu Cheng, Weituo Hao, Siyang Yuan, Shijing Si, and Lawrence Carin. 2021. Fairfil: Contrastive neu- ral debiasing method for pretrained text encoders. In International Conference on Learning Representa- lions.
Acknowledgements
This research was supported by the MSIt(Ministry of Science, ICT), Korea, under the High-Potential Individuals Global Training Pr02ram)(2022- 00155958) supervised by the ITTP(Int for Information & Communications Technology Planning & Evaluation) (Contribution: 50%), and Institute of Information communications Technology Planning & (ITI grant funded by the Korea (OV))) [No. 2022-0-00184, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics], and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea goveramen(MSITT) INO.2021-0-4 Artificial Intelligence Graduate School Program (Seoul National University)], and Microsoft Research Asia. K. Jung is with Automation and Systems Research Institute (ASRI), Seoul National University.
Ting-Rui Rui Chiang, Yi-Pei Chen, Yi-Tin Ti-Ting Ych, and Gra- ham Neubig, 2022. Breaking down multilingual ma- chine translation. In Findings of the Associastion for Computational Linguistics: ACL 2022, pages 2766- 2780, Dublin, Ireland, Association for Computational Linguisties.
Prafulla Kumar Choubey, Anna Currey, Prashant Mathur, and Georgiana Dinu. 2021. GFST: Gender- filtered self-training for more accurate gender in trans- lation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages (640-1654, Online and Punta Cana, Domini- can Republic, Association for Computational Lin- guistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery. Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instraction-finetuned language models.
References
Roce Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 3874-381 3884, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint Ntti:2207.04672..
Giuseppe Attanasio, Debora Nozza, Dirk Hovy, and Elena Baralis. 2022. Entropy- based attention regu- larization frees unintended bias mitigation from lists. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1105-1119, Dublin,
Marta R. Costa-jussà and Adrià de Jorge. 2020. Fine- tuning neural machine translation on geader-balanced datasets. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 26 36-34, Barcelona, Spain (Online). Association for Computational Linguistics.
Ireland. Association for Computational Linguistics.
Lena Cabrera and Jan Niehues. 2023. Gender lost in translation: How bridging the gap between languages affects gender bias in zero-shot multilingual trans- lation. In Proceedings of the First Workshop on Gender-Inclusive Translation Technologies, pages 25-35, Tampere, Finland. European Association for Machine Translation.
Marta R. Costa-jusså, Carlos Escolano, Christine Basta, Javier Ferrando, Roser Batlle, and Ksenia Kharitonova. 2022. Interpreting gender bias in neu- machine translation: Multilingual architecture matters. Proceedings of the AAA! Conference on Artificial Intelligence, 36(1 1): 1855-11863.
Anna Currey, Maria Nadejde, Raghavendra Reddy Pap- pagari, Mia Mayer, Stanislas Lauly, Xing Niu, Ben- jamin Hsu, and Georgiana Dinu. 2022. MT-GenEval: A counterfactual and contextual dataset for evaluating gender accuracy machine translation. Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4287-4299, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Salvador Carrión-Ponz and Francisco Casacuberta. 2022. Few-shot regularization to tackle catastrophic forgetting in multilingual machine translation. In Proceedings of the 15th biennial conference of the Association for Machine Translation in the Amer- icas (Volume I: Research Track), pages 188-199, Orlando, USA. Association for Machine Translation in the Americas.

---

Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding tack-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489-500, Brussels, Belgium. Association for Computational Linguistics.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. 2020. Su- pervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, pages 18661-18673. Curran Associates, Inc.
Tom Kocmi, Tomasz Limisiewicz, and Gabriel Stanovsky. 2020. Gender coreference and bias eval- uation at WMT 2020, In Proceedings of the Fifth Conference on Machine Translation, pages 357-364, Online. Association for Computational Linguistics.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma. Ahmed Ei-Kishky, Siddharth Goyal, Man- deep Baines, Onur Celebi. Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vi- taliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020. Beyond english-centric multilingual machine translation.
Abhinav Kumar, Chenhao Tan, and Amit Sharma. 2022. Probing classifiers are unreliable for concept removal and detection. In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability.
Eve Fleisig and Christiane D. Fellbaum. 2022. Miti- gating gender bias in machine translation through adversarial learning. ArXiv, abs/2203.10675.
Vaibhav Kumar, Tenzin Singhay Bhotia, Vaibhav Ku- mar, and Tanmoy Chakraborty. 2020. Nurse is closer to woman than surgeon? mitigating cender-biased proximities in word embeddings. Transactions of the A Association for Computational Linguistics, 8:486- 503.
Yacine Gaci, Boualem Benatallah, Fabio Casati, and Khalid Benabdeslem. 2022a. Debiasing pretrained text encoders by paying attention to paying attention. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 9582 0202-960,,440 Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Anne Lauscher, Tobias Lueken, and Goran Glavaš. 2021. Sustainable modular debiasing of language models, In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4782-4797, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Yacine Gaci, Boualem Benatallah, Fabio Casati, and Khalid Benabdeslem. 2022b. Iterative adversarial re- moval of gender bias in pretrained word embeddings. In Proceedings of the 37th ACM/SIGAPP Symposium On Applied Computing, pages 829-836.
Hongyuan Lu, Haoyang Huang, Dongdong Zhang. Hao- ran Yang. Wai Lam, and Furu Wei. 2023. Chain- of of-dictionary prompting elicits translation in large language models.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894 6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Alireza Mohammadshahi, Vassilina Nikoulina, Alexan- dre Berard, Caroline Brun, James Henderson, and Laurent Besacier. 2022a. SMaL.L- Introduc- ing shallow multiliiguul machine translation model for low-resource languages. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 1348-8359, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Hila Gonen and Yoav Goldberg, 2019. Lipstick pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 609-6 614, Minncapolis. Minnesota. Association for Computa- tional Linguistics.
Alireza Mohammadshahi,, Vassilina Nikoulina, Alexan- dre Berard, Caroline Brun, James Henderson, and Laurent Besacier. 2022b. What do compressed mul- tilingual machine translation models forget? In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 4308-4329, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Naman Goyal, Jingfei Du Myle Ott, Giri Ananthara- man, and Alexis Conneau. 2021. Larger-scale trans- formers for multilingual masked language modeling. In Proceedings of the 6th Workshop on Represen- tation Learning for NLP (RePLANLP-2021), pages 33, Online. Association for Computational Lin- guistics.
Hadas Orgad and Yonatan Belinkov. 2022. Choose your lenses: Flaws in gender bias evaluation. In Proceed- ings of the 4th Workshop on Gender Bias in Natu- ral Language Processing (GeBNLP), pages 151-167, Seattle, Washington. Association for Computational Linguistics.
Jacqueline He, Mengzhou Xia, Christiane Fellbaum, and Danqi Chen. 2022. MABEL: Attenuating gen- der bias using textual entailment data. In Empirical Methods in Natural Language Processing (EMNLP).
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual multilingual BERT? Proceed- ings of the 57th Annual Meeting of the Association for

---

Computational Linguisties, pages 4996-5001, Flo- renee, Italy, Association for Computational Linguis-
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605.
tics.
Marcelo OR Prates, Pedro H Avelar, and Luís Lamb. 2020. Assessing gender bias in machine translation: a case study with google translate. Neural Computing and Applications, 32:6363-6381.
Kellie Webster, Xuezhi Wang. Ian Tenney, Alex Beutel, Emily Pitler. Ellie Pavlick, Jilin Chen, Ed H. Chi, and Slav Petrov. 2020. Measuring and reducing gendered correlations in pre-trained models. Technical report.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg, 2020. Null out: Guard- ing protected attributes by iterative nullspace projec- tion. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256, Online. Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang. Mark Yatskar, Vicente Or- donez. and Kai- Wei Chang. 2018. Gender bias coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Short Papers), pages 15-20, New Orleans, Louisiana, Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual us- ing knowledge distillation. Proceedings of 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512 4525, Online. Association for Computational Linguisties.
Danielle Saunders and Bill Byrne 2020. Reducing gen- der bias in neural machine translation domain adaptation problem. Proceedings of the 58th An- nual Meeting of the Association for Computational Online. Association for Computational Linguistics.
Linguistics. pages
Danielle Saunders, Rosie Sallis, and Bill Byrne. 2020. Neural machine translation doesn't translate gender coreference right unless you make it. In Proceedings of the Second Workshop on Gender Bias in Natural Langwage Processing, pages 35-43, Barcelona, Spain (Online). Association for Computational Linguistics.
Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Mat- teo Negri, and Marco Turchi. 2021. Gender bias in machine translation. Transactions of the Association for Computational Linguistics, 284.
Chenze Shao and Yang Feng. 2022. Overcoming catas- trophic forgetting beyond continual learning: Bal- anced training for neural machine translation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 2023-2036, Dublin, Ireland. As- sociation for Computational Linguistics.
Shanya Sharma, Manan Dey, and Koustuv Sinha. 2021. Evaluating gender bias natural language inference.
Gabriel Stanovsky, Noah A. Smith, and Luke Zettle- moyer. 2019. Evaluating gender bias in machine translation. ACL, Florence, Italy. Association for Computational Linguistics.
Chau Tran, Shruti Bhosale, James Cross, Philipp Kochn, Sergey Edunov, and Angela Fan. 2021. Facebook AI's WMT21 news translation task submission. In Proceedings of the Sixth Conference on Machine Translation, pages 205-215, Online. Association for Computational Linguistics.

---

A Experimental Details
Method
Dataset size
Steps
SMaLL-110
A.1 ChatGPT Generation
Balanced
172,862
3,700
For our experiments using ChatGPT, use the API provided by OpenAl for generating text, using the model gpt-3. turbo. For small number of cases, ChatGPT generated multiple lines of text de- limited by newline characters, leading to errors in source-target alignment during WinoMT evalution. For these cases, we split the sentence based on the newline character and take the first sentence as the translation.
GFST
1,802,832
1,900
Handcrafted
388
200
GACL
172,862
700
with Lac only
200
M2M-100
GACI.
172,862
400
NLLB-200
GACL
172,862
800
Table 5: Dataset size and number of training steps before early stopping for the fine-tuning experiments in our work.
A.2 Data Preprocessing Details
In our work, use the cleaned WMT18 en-de dataset, as pre-processed by Edunov et al. (2018). By filtering for sentences with gender-related words, we find 415,401 masculine and 86,431 fem- inine sentence sets. Next, masculine sentence sets are undersampled with random sampling while all samples from feminine set are used to create final balanced set of total 1186,431=172,866 samples. We do not perform additional processing handle other domain differences except gender.
68.0
875
67.0
S9BLE0
33.00
66.3
86.0
32,95
85.5
32,90
65.0
0.0
0.2
0.4 0.6
0.8
1.0
A.3 Fine-tuning Details
For our experiments, hyperparameter search was done manually with learning rate from (2e-6, 4e-6, 8e-6), batch size from {4, 8, 16, 32), and learning rate warmup steps from 1000, 200, )00} based on fine-tuning GACL the SMaLL-100 architecture and metric based on explicit accuracy the MT- GenEval development set. Our final selection of hyperparameters then used for all experiments in our paper. One exception for fine-tuning GACL with NLLB-200 architecture we set the learn- ing rate to 8c-6 due to its slower convergence dur- ing training in comparison to other architectures. For fine-tuning with our GACL method, we use balanced random sampling so that the number of sentences for each gender are equal within one mini-batch.
Figure 7: Experimental results tuning hyperparame- ter a. the spBLEU scores are the average score 10 language directions on FL.ORES200 development set and gender accuracy is the average explicit accuracy the MT-GenEval development set.
NVIDIA A100 80GB GPU. We use the pre-trained model checkpoints downloaded from the Hugging- face website.
B Additional Experiments
B.1 Many2many Translation Performance
The 12 target languages covered by the WinoMT and MT-GenEval are mostly categorized high-resource languages (Mohammadshahi et al., 2022a). Thus, we extend our FLORES-22 200 evalua- tion to languages of low and medium resources for both source and target languages to more accurately analyze the impact of our approach on multilingual translation performance.
The fine-tuned dataset size and number of finc- tuning steps before early stopping is shown in Table 5. We notice that Balanced and GFST data aug- mentation based methods trained for more than one thousands steps before carly stopping. On the other hand, our GACL method and Handcrafted method stopped fine-tuning within one thousand steps.
For this experiment, levels defined by Mohammadshahi al. (2022a): High (H), Medium (M), Low (L) and Very Low
the four resource
We fine-tuned the LMaLLL-100 model on NVIDIA A6000 GPU, and fine-tuned M2M-100 1.2B and NLLB-200 1.3B distilled models on
Phtpss//huggingffcc.oo

---

FLORES-200 QUBLEU
Method
YLAVL
VE2L
V1.2M
VI.H
L2VL
1.LL
LAH
MIVV
MIZ
M1M
512H
H1MM
B1311
X2X
SMALL-100
Bostline
26.6
14.5
21.8
15.3
12.7
9.4
13.7
11.3
12.2
19.7
11.7
21.2
12.6
15.3
12.2
16.1
GACT.
8.7
19.5
7.7
M2M--190
Baseline
8.3
10.7
30.2
99
1622
GACL
7.8
10.1
19.8
10.6
9.5
NLLB-200
23.6
GACT.
20.8
25.1
23.5
22.7
24.7
23.2
Table 6: Many2many translation performance evaluation by resource level FL.REES-200 using the spBLEU metric. X2X represents the total average score.
FLORES-20 Chel'+
Method
V1.2II.
VI.2I.
V1.2M
V1.2H
1.2VI.
1.21.
1.2M 1.2H
MIVE.
MI2.
M2M
MIH
H2VE.
H2I.
HEM
8221
XIX
SMALL-100
Buoecline:
48.0
14.4
35.3
15.9
38.7
27.7
27.4
30.2
44.7
31.9
33.7
31.5
41.1
32.4
28.7
33.0
34.8
GACL
$5.1
33.5
35.1
15.2
18.0
26.6
26.9
29.2
64.5
30,9
33.6
30.8
43.0
31.5
25.5
32.0
34,2
wib CXD
45,0
32.1
34.7
34.1
37.1
24.7
25.8
27.6
44.3
29.5
33.4
29.8
42.6
3000
27.8
30.6
33.3
M2M-109
Bascline
50.4
29.9
35.7
31.7
29.0
24.00
260
20.5
46.7
27.9
33,5
29,4
43.2
27.7
20.3
28.8
33.3
GACI.
90.3
29.1
35.2
31.5
39.1
23.2
25.7
29.3
46.7
27.1
33.1
29.1
43.0
26.8
28.8
28.2
32.9
NLLB-200
Bosallin:
S1.6
45.3
38.5
45.9
50.0
46.6
37.
45.
40.0
43.7
16.6
43.8
90.4
45.
37.9
44.6
44.4
GACL
36,0
45.0
Table 7: Many2many translation performance evaluation by resource level on FLORES-20 using the ChrF++ metric. X2X represents the total average score.
(VL), and evaluate many2many translation perfor- mance using spBLEU. Due to resource limitations, we sample 5 languages for each resource group and evaluate across all sampled language direc- tions. The languages used are as follows: High re- source languages include French, German, Italian, Russian, and Spanish (Latin America). Medium resource languages are Arabic, Bulgarian, Chi- nese (Simplified), Korean, and Turkish. Low re- source languages are Afrikaans, Amharic, Bosnian, Cebuano, and Kazakh. Very Low resource lan- guages are Belarusian, Croatian, Filipino (Taga- log), Nepali, and Occitan. The averaged results by resource group reported in Tables 6 and 7.
B.3 Relationship between translation
performance and gender bias metrics for each language
In Figure 8, we report results on the relationship between translation performance and gender bias metrics for each language. observe similar cor- relations between translation performance and gen- der bias metrics of multilingual MT systems across each independent target languages. However, the slope of the correlation differs by the target lan- guage.
B.4 Gender bias evaluation results for each target language
We report the evaluation results on WinoMT and MT-GenEval datasets for each of the supported target languages individually in Tables 8 and 9, For all evaluations, the source language is fixed to English, it the only provided source language for the datasets.
B.2 Ablation Results on Hyperparameter
We report the effects of changing the hyperparme- ter used to determine the relative weight between LM and LKD in the joint training loss em- ployed during fine-tuning.
As shown in Figure 7, we find that for translation performance, setting to 0,4 performs the best, while using single loss of either EMT (i.e. 0) and LKD (i.c. 1) performs slightly worse. For very
B.5 Statistical significance tests
We share the results statistical significance test- ing between our GACL model (Table 3; row 2) and ablation fine-tuned with Lae only (Table 3; row 4) which scored closely accuracy scores. We conduct a paired randomized permutation test with the number of resamples N set to 100,000. The v-values from the test are shown in Table 10.
gender accuracy, we find that trends clear, with a set 1.0 being the most effective and of 0.4 second most effective. Based on these findings, we choose to use a value of 0.4 for the rest of the experiments this paper.

---

Tuget lung.
de
ft
it
be
AVG
AG
LG
20
AG
AG
Aec.
AG
AG
AG
AG
Beseline
57.4
242
15.8
27.0
50.3
24.3
40.7
38.2
53.3
36.4
58.8
$5.5
47.7
21.0
45.6
59.5
44.2
42.3
46.9
35.5
46.2
31.4
Balanod
72.9
188
18.8
22.6
39.6
7.
434
29.1
64.7
8.0
12.5
69.2
17.0
50.5
28.1
46.8
34.0
54.3
22.8
52.0
20,6
$4.3
17.0
22.0
(UuS)
-78
52.5
Table 8: Accuracy and AG scores the 10 larget languages of the WinoMT dataset.
Teg lang.
de
it
bi
AVG
Acc.
AG
Aoc.
AG
Auc:
AG
Auc:
AG
Ans:
AG
Act:
AG
Acc:
AG
Acc.
AG
AG
Haseline
62.3
10.0
65.7
20.0
53.3
25.0
49.3
39.0
26.3
27.0
53.0
27.7
65.7
14.3
52.7
38.7
37.9
25.0
Balamoed
65.7
8.3
68.7
19.0
57.3
21.3
57.0
54.3
60.7
19.3
56.3
25.0
72.0
10.5
52.7
37.7
61.3
22.0
GPST
63.0
10.3
52.3
38.3
25.0
37.3
Handerafied
(Ourz)
76.0
-20
00.3
9.0
73
31.3
73.2
Table 9: Accuracy and AG scores the 8 target languages of the MT-GenEvva test set.
50
Darasct
Target Lang.
P-valoc
de
1e-8
1e-5
le-5 5
50
WinoNT
le-5
le-5
1e-5
le-5
25
le-5
70-
le-2
le-
20
Genlenl
40
2e-12 1
4a
46-5
30
45
50
20
10
40
50
<281
K2BLBU
Table 10: P-values from randomized pairwise permuta- tion test between accuracy scores of GACL and ablation. denotes evaluations that are not stastistically signifi- cant with respect to threshold of 0.05.
Relationships between translation perfor- mance and gender bias metrics of multilingual NMT model for various evaluated target languages.
Figure 8:
Method
Accprim.
Ace
Baseline
46.3
48.6
GFST
47.5
52.2
4.7
We found that the GACL method shows higher accuracy than the ablation on 10 out of 10 evalu- ated languages on WinoMT dataset and 4 out of 8 evaluated languages on MT-GenEval dataset di- rections with statistical significance of p-value less than 0.05.
Handerafted
55.2
59.0
3.8
GACL (Ours)
67.4
72.7
5.3
Table 11: Gender "Accuracy" of the primary and sec- ondary entities the WinoMT dataset.
(2020) and show the results in Table 11. We found that all previous methods well ours lead an consistent increase in gender accuracy of secondary entity by about 5% compared to gender accuracy of primary entity, which is consistent with findings by Saunders et al. (2020). Note that none of the evaluated methods, including ours, explicitly ac- count for entities with ambiguous gender and this issue is left for future research.
B.6
Gender evaluation on the secondary entity the WinoMT dataset
The WinoMT dataset is comprised of sentences that mention two entities; one entity has an unambigu- gender indicated by coreferential pronoun, while the gender of the other entity is ambiguous. this subsection, we evaluate the effect of gen- der debiasing methods on this secondary, gender- unspecified entity in the WinoMT dataset. We use the variation dataset proposed by Saunders et al.


Reviews:
[{"review_id": "dkkhRQvhsU", "reviewer": "Reviewer_fQoU", "summary": "", "questions": "While space is limited in this version, I would be interested if some qualitative examples could be given? Both where it does and does not work.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and constructive comments. We respond to them below.\n\n* On the data selection process:\n\nSimilarly to the GFST approach by Choubey et al., we undersample the larger of the M/F sentence sets to match the size of the smaller one. For the WMT18 en-de dataset we used, 415,401 masculine and 86,431 feminine sentence sets are extracted, and masculine sentence sets are undersampled with random sampling while all samples from feminine set are used to create a final balanced set of total 2*86,431=172,862 samples. We do not consider or perform additional processing to handle domain differences. We will update our paper to include these details.\n\n\n* On the finetuning hyperparameters:\n\nWe first like to clarify that the GFST baseline was trained for 1,900 steps, the Handcrafted baseline was trained for 200 steps, and GACL method was trained for 700 steps (Table 5 in Appendix), and the number of these fine-tuning steps were not fixed but rather based on early stopping on the best MT-Gender dev set performance (Line 288). Choubey et al may have required 30K steps for training their en-de model (Appendix A of their paper) because in their work they trained the transformer model from scratch. Our implementation, on the other hand, fine-tunes an already pre-trained multilingual NMT model and was able to converge in a fewer number of steps. \nWe will update the wording of “few thousand steps” (Line 376) to be more specific in our paper.\n\n* On the WinoMT dataset:\n\nFirstly, we agree that the WinoMT dataset is not ideal and some examples are not perfectly unambiguous in its interpretation. Consequently, we included evaluation on a more recent dataset, MT-GenEval, in our paper, which consists of professionally translated sentences containing unambiguous references to a single gender. With MT-GenEval, we show consistent findings that existing NMT models exhibit stereotypical gender bias and our approach helps mitigate the issue. \n\nSecondly, on the translation of the other secondary entity in the WinoMT which is (supposedly) ambiguous, we have run additional experiments using the variation dataset from (Saunders et al., 2020) and share the results. We found that all previous methods as well as ours lead to an consistent increase in gender accuracy of secondary entity by about 5% compared to gender accuracy of primary entity (Baseline: 46.3/48.6, GFST: 47.5/52.2, Handcrafted: 55.2/59.0, GACL 67.4/72.7), which is consistent with findings by (Saunders et al., 2020). We believe additional techniques (such as tagging, proposed by Saunders et al., 2020) to address this trend and possible ways handle the gender of an ambiguous entity is a great topic for future research, but out of scope of our work. We will update the paper to include the experimental results and this discussions.\n\n\n* On the discussion of extending beyond binary gender:\n\nOur work evaluated bias mitigation for the binary gender, which is a limitation that we will also clearly state in our ethical considerations section (please see response to ethics review). However, we believe our approach has a great flexibility to be extended to cover non-binary genders or gender-neutralizing translations. The supervised contrastive loss we employed for gender contrastive loss is not limited to binary classification tasks but applicable to multi-class labels, as the formulation relies only on defining positive pairs as samples from the same class, and negative pairs as samples from any different class. Correspondingly, augmenting the translation data with gender-neutral/inclusive versions of the translations and assigning pseudo-labels to apply multi-class supervised contrastive loss is viable within our framework. We will include these discussions in our paper.\n\n\n* On qualitative examples:\n\nThank you for the suggestion. We will include examples of translations before and after applying our fine-tuning method in the appendix of our paper.\n\nOverall, thank you for your comments. I'll be happy to engage in any further discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This is an interesting and possibly novel approach to bias in MT. It seems straightforward and sensible, and is clearly explained.\n\n- There are a good set of experiments, including a very wide range of languages, multiple different pre-trained models, as well as implementations of a sensible selection of similar approaches from the literature, with evaluation on two gender test sets with multiple metrics, and on Flores. I liked also the ablation experiments on using the different elements of GACL in isolation.\n\n- The results seem good overall, and their analysis is very interesting, particularly the results for the effect on language directions which were not directly trained.", "reasons_to_reject": "- The data selection process, which seems critical to the performance of the approach, is underspecified. It is briefly described in section 2.1, but important details are unclear: How does balanced sampling take place, are one or both of the M/F sentence sets undersampled? How is the domain difference between sentences with masculine vs feminine terms handled?\n\n- The finetuning hyperparameters are a little surprising - they don’t seem equally appropriate for a very tiny handcrafted set of 388 segments and a reinterpretation of the Choubey et al gender dataset which in that paper ran to hundreds of thousands of segments. Does it make equal sense to fine-tune on both for a “few thousand steps”?\n\n\n\n- The assumption is not unique to this paper, but I would strongly dispute that WinoMT contains examples of \"unambiguous cases where there is only one correct translation with respect to gender\". Firstly, it is possible to reinterpret many examples such that the pronoun is coreferent with either or neither of the professional entities in the input. Secondly, there are two professional entities in each sentences, and the \"correct\" translation of only one is implied by the pronoun - how is the other translated before or after the intervention? These are issues that have been raised in the literature for a while (Saunders et al 2020 NMT doesn't translate gender coreference right unless you make it, Gonzalez et al 2020 Type B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Bias) and the paper would be stronger if it addressed how they interact with the proposed approach.\n\n- The approach seems to hinge on a concept of binary gender - aiming for a 50/50 likelihood of the correct vs incorrect gender via the contrastive loss. It would strengthen the paper to discuss how such an approach might be extended for, for example, neutralization.\n\n\nEdit to add: The authors have responded clarifying many of these points. I am overall positive about this paper and would be happy to see it accepted with the proposed additional discussion and clarification.", "questions_to_authors": "While space is limited in this version, I would be interested if some qualitative examples could be given? Both where it does and does not work.", "paper_topic_and_main_contributions": "This paper attempts to mitigate gender bias in multilingual MT models by fine-tuning the encoder with a \"gender-aware contrastive loss\" in addition to the usual log likelihood. They also regulate with a teacher-loss relative to the original model.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HXnc9CglYI", "reviewer": "Reviewer_3Bkt", "summary": "", "questions": "In lines 127-131 it is stated that the training data is created so that it is balanced. Is this a requirement for the method? Has it been tested with unbalanced data and resampling?\nWas the method tested in the context of ambiguous referents?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review. We respond to your comments below.\n\n> 1. An impact statement is missing\n\nWe fully agree with the need for an impact statement/ethical considerations due to the topic of our work and we will update our paper to include it. Please see our response on the ethics review for more detail.\n\n> 2. It would be interesting to know why previous work has reported gains while going in the opposite direction\n\nThank you for pointing out a relevant and interesting work of (Fleisig and Fellbaum, 2022). We believe that while previous work’s approach seems to be in the opposite direction of ours, the motivation itself is shared in that both works try to “fix” model representations from having a fixed, stereotypical gender information. Their approach addresses this by removing gender information from encoder output adversarially when the given input has no gender context (via masking the pronouns in the input; see Figure 2 of their paper). Our approach, on the other hand, addresses this by injecting correct contextual gender information from encoder output contrastively when given inputs have gender contexts. We will update our paper to include the related work.\n\n> 3. Lack of statistical significance testing\n\nThank you for the suggestion. We share the results on paired randomized permutation tests (with number of resamples N=100,000) for comparing line 2 (GACL) and line 4 (Ablation with GC loss only) of table 3. We found that the GACL method shows higher accuracy than the ablation on 10 out of 10 evaluated languages on WinoMT dataset and 4 out of 8 evaluated languages on MT-GenEval dataset directions with statistical significance of p-value less than 0.05. We will include the specific p-values as well as individual results for each evaluated target languages in the paper. \n\n> 4. The effect of the solution on sentences where there is more than one referent is not evaluated.\n\nWe decided to focus on sentences with single referents in our work as 1) we found that existing NMT models still exhibit considerable stereotypical gender biases even with single referents, 2) there are statistically fewer cases compared to sentences with single referents (ratio of 7.2:1 on single to multiple referents during our filtering stage), and finally 3) lack of evaluation data/metrics considering gender bias of multiple referents. However, we believe this direction would be the natural next step for our work along with handling gender-ambiguous entities.\n\n> Q. In lines 127-131 it is stated that the training data is created so that it is balanced. Is this a requirement for the method? Has it been tested with unbalanced data and resampling? Was the method tested in the context of ambiguous referents?\n\nWe require sampling the same number of masculine and feminine samples in a single mini-batch during fine-tuning so that the number of positive and negative pairs for both genders are equal. While not tested, we believe our method can also be fine-tuned with unbalanced data as long as balanced resampling is applied. Our method was not tested in the context of ambiguous referents.\n\nOverall, thank you for your comments. We will also fix the typos and include the missing references.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " 1. The approach demonstrates effective, compares favourably to other strategies, and is efficient.\n 2. The evaluation is convincing and complete, with detailed analyses explaining the strengths of the proposed approach.", "reasons_to_reject": " 1. An impact statement is missing, while it would probably be required to delineate the effect of the techniques on NMT users, and how it relates to groups that are neglected in the study (e.g., non-binary individuals).\n 2. It would be interesting to know why previous work (https://arxiv.org/abs/2203.10675) has reported gains while going in the opposite direction (i.e., removing gender information). A comparison with such line of research would be very interesting.\n 3. Lack of statistical significance testing. Although the gaps between methods are large, confidence intervals of paired statistical significance would give more information when results are quite close (e.g., line 2 and 4 of table 3, ...).\n 4. The effect of the solution on sentences where there is more than one referent is not evaluated.", "questions_to_authors": "In lines 127-131 it is stated that the training data is created so that it is balanced. Is this a requirement for the method? Has it been tested with unbalanced data and resampling?\nWas the method tested in the context of ambiguous referents?", "paper_topic_and_main_contributions": "The paper introduces a novel strategy to mitigate gender bias in multilingual machine translation models by exploiting contrastive learning. The proposed approach consists in adding a new loss that takes the encoder outputs for all the samples in the batch and forces a higher similarity for those tensors that contain a referent of the same gender. In addition to this loss, the finetuning process accounts for other two losses: the traditional cross entropy and a knowledge distillation loss, which aim at avoiding the catastrophic forgetting problem. The results on different benchmarks demonstrate the ability of the approach in reducing gender bias, while retaining translation quality.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 388: \"less\" -> \"lower\"\n\nIn Table 1 up/down arrow for the metrics would help.", "Justification_for_Ethical_Concerns": "No impact statement is provided, although the paper deals with a dimension (gender) that directly impacts users."}, "extra_text": {"Missing_References": "Lines 222-233 define gender accuracy exactly as defined in:\nM. Gaido et al. 2020. \"Breeding Gender-aware Direct Speech Translation Systems\", Proc. of COLING 2020.\n\n\nContemporaneous work on gender bias in multilingual MT, which might be included in a camera ready:\nL Cabrera, J. Niehues. 2023. \"Gender Lost In Translation: How Bridging The Gap Between Languages Affects Gender Bias in Zero-Shot Multilingual Translation\", In Workshop on Gender-Inclusive Translation Technologies (GITT) at EAMT 2023.\n"}}, {"review_id": "fX5YwmxqCk", "reviewer": "Reviewer_KfGt", "summary": "", "questions": "A.  Line 135 states:  `Given h_i as the encoder embedding of the source sentence, we define positive samples to be the set of representations that have the same gender as h_i'.   Can this be clarified?   If h_i is a sentence embedding,  its not clear how it can have a gender.    Some explanation as to how gender is extended to the sentence level,  or perhaps a clarification that h_i is a word-level representation (?), would make this more clear.   Given that section 2.2 contains the main novelty in the paper, it should be made more explicit.\n\nB. The presentation seems to emphasize that the technique is suitable for encoder-decoder models, and modifying the encoder embedding makes for a nice presentation.  But couldn't the technique also be applied in a decoder architecture?   \n\nC. More a comment than a question, but readers might be as interested in BLEURT as spBLEU and ChrF++ .", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highlighting many strengths of our paper and pointing out parts that are unclear. We respond to your comments below.\n\n> As noted in the limitations section, the evaluated source language is limited to English.\n\nWe focused our work on extending evaluation to multiple target languages in a multilingual setting first and evaluated on a total of 12 target languages, which is a great increase in coverage in comparison to previous MT debiasing literature that usually considered bilingual models for three to five target languages (Choubey et al 2021, Saunders and Bryne 2020). However, our work is definitely extensible to source languages other than English, such as Chinese, if there are means to detect gendered terms in the sentence, as the proposed contrastive loss itself is not dependent on any specific language.\n\n>  Another limitation is the focus on encoder-decoder architectures\n\nOur work focuses on encoder-decoder architectures as most of the best-performing multilingual NMT models employed an encoder-decoder architecture in our review of literature. However, our approach is applicable to other model architectures as long as we can aggregate a representation of the source language input. Specifically for decoder-only architectures, an option would be to pool the decoder model outputs of tokens up to given source language input. As LLMs are increasingly being considered for the MT task, we believe this is an interesting direction for future work. \n\n> Q A. Line 135 states: `Given h_i as the encoder embedding of the source sentence, we define positive samples to be the set of representations that have the same gender as h_i'. Can this be clarified?\n\nh_i is a sentence embedding, and we use the gendered word mentioned within the source sentence as the gender pseudo-label for the sentence embedding during contrastive learning. While it might seem unintuitive to assign a gender to a sentence, we use sentence-level instead of word-level representations because it is difficult to know in advance which words in the given sentence will be translated to have a gender-specific marking and which will not, and this also varies by the target language. Therefore, we rely on indirectly incorporating gender information through the sentence embedding instead, which is computed by pooling all word representations. We will update section 2.2 to make this more explicit.\n\n> Q B. ... couldn't the technique also be applied in a decoder architecture?\n\nYes, we believe our method is applicable to decoder-only models by computing representations based on decoder output of the given source sentence input. However, we focused our work on studying encoder-decoder architecture as most state-of-the-art multilingual NMT models were based on them.\n\n> Q C. ... readers might be as interested in BLEURT as spBLEU and ChrF++ .\n\nThank you for the recommendation. We evaluated on spBLEU and ChrF++ primarily to allow comparison with published results from previous works on multilingual NMT. While extending evaluation with BLEURT would strengthen the comparison, due to the large number of language directions we were evaluating on and limited resources, we decided to focus more on the evaluation metrics related to gender bias.\n\n> Figure 3 shows average scores of a NMT system on ten target languages of WinoMT. Is the regression done against the individual scores of individual systems on specific LPs, or is it against the average scores? \n\nThe regression is done against the average scores for Figure 3. We will update our paper to make this clear. We also have regression plots for individual scores of individual systems for each target language in Figure 8 of Appendix. \n\nOverall, thank you for your comments and hope our responses have sufficiently addressed your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a fairly simple technique that gives good improvements to accuracy in translation for multilingual NMT models.   The technique has a practical flavour: the focus on the source embedding, rather than the target language,  yields improvements in gender accuracy into multiple target languages,  with good generalisation from the language pairs used in training with the contrastive loss.   A good range of language pairs and multilingual models are evaluated.   The paper also has many interesting observations, e.g. that ChatGPT often relies on gender stereotypes in its translation output (line 343).   The chosen baseline techniques are suitable for the proposed technique.", "reasons_to_reject": "As noted in the limitations section,  the evaluated source language is limited to English.   I can't recommend an easy fix for this, given what evaluation resources are available,  but it is a significant limitation.   Another limitation is the focus on encoder-decoder architectures (line 114).   Many recent LLMs will not be able to make use of this method.", "questions_to_authors": "A.  Line 135 states:  `Given h_i as the encoder embedding of the source sentence, we define positive samples to be the set of representations that have the same gender as h_i'.   Can this be clarified?   If h_i is a sentence embedding,  its not clear how it can have a gender.    Some explanation as to how gender is extended to the sentence level,  or perhaps a clarification that h_i is a word-level representation (?), would make this more clear.   Given that section 2.2 contains the main novelty in the paper, it should be made more explicit.\n\nB. The presentation seems to emphasize that the technique is suitable for encoder-decoder models, and modifying the encoder embedding makes for a nice presentation.  But couldn't the technique also be applied in a decoder architecture?   \n\nC. More a comment than a question, but readers might be as interested in BLEURT as spBLEU and ChrF++ .", "paper_topic_and_main_contributions": "from the abstract:  the paper specifically targets gender bias for multilingual systems and studies cases where there is a single correct (i.e. unambiguous) translation.   A straightforward modelling technique termed Gender-Aware Contrastive Learning is proposed that encodes contextual gender information.    This is a source-side technique,  in that it influences the encoder, and so offers improved gender translation even into target languages not seen in training (fine tuning).    The Gender-Aware Contrastive Learning technique is described in section 2.2 , and is motivated by recent in-batch contrasting training techniques.  Regularisation in fine tuning is also needed,  as is confirmed by ablation experiments in section 4.4.   Improvements in gender accuracy across a range of target languages and architectures are reported using the usual suite of metrics, including a newly proposed Explicit Accuracy.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "An example of the 'Explicit Accuracy' (line 226)  showing how it differs from normal accuracy (line 214) would make it easier to follow.   \n\nFigure 3 shows average scores of a NMT system on ten target languages of WinoMT.    Is the regression done against the individual scores of individual systems on specific LPs,  or is it against the average scores?    Its difficult to interpret this figure.\n\nNote that Table 2 column headings in the 'Method' row have misplaced | | 's for the \\Delta G 's"}}]

Author rebuttal:

Meta-review: This paper proposes reducing gender bias in multilingual MT models by fine-tuning the encoder with an additional "gender-aware contrastive loss". 

The results on different languages show that the approach reduces gender bias while retaining translation quality.

The analysis explaining the strengths of the proposed approach is also presented. 



Revisions discussed during the authors' response should be applied. 

Also, dealing only with binary gender should be mentioned in limitations (while including non-binary it is very easy in English, for most of the other languages there are no clearly defined rules (yet)).
Decision: Accept-Main