Generating Gender Alternatives in Machine Translation (2024)

Abstract: Machine translation (MT) systems often translate terms with ambiguous gender (e.g., English term “the nurse”) into the gendered form that is most prevalent in the systems’ training data (e.g., “enfermera”, the Spanish term for a female nurse). This often reflects and perpetuates harmful stereotypes present in society. With MT user interfaces in mind that allow for resolving gender ambiguity in a frictionless manner, we study the problem of generating all grammatically correct gendered translation alternatives. We open source train and test datasets for five language pairs and establish benchmarks for this task. Our key technical contribution is a novel semi-supervised solution for generating alternatives that integrates seamlessly with standard MT models and maintains high performance without requiring additional components or increasing inference overhead.

heading: 1 Introduction and Related Work
text: Gender1 biases present in train data are known to bleed into natural language processing (NLP) systems, resulting in dissemination and potential amplification of those biases (Sun et al., 2019). Such biases are often also the root cause of errors. A machine translation (MT) system might, for example, translate doctor to the Spanish term médico (masculine) instead of médica (feminine), given the input “The doctor asked the nurse to help her in the procedure” (Stanovsky et al., 2019). To avoid prescribing wrong gender assignment, MT systems need to disambiguate gender through context. When the correct gender cannot be determined through context, providing multiple translation alternatives that cover all valid gender choices is a reasonable approach.
∗Work done during an internship at Apple. †Equal senior contribution.
1“gender” in this work refers to binary grammatical gender, and not social gender (male, female, nonbinary). Please refer to §Limitations for a detailed discussion.
Numerous prior works have focused on producing correctly gendered translations given contextual gender “hints”, such as “to help her” in the example above (Stanovsky et al., 2019; Saunders and Byrne, 2020; Stafanovičs et al., 2020; Costa-jussà et al., 2022; Saunders et al., 2022; Renduchintala et al., 2021; Bentivogli et al., 2020; Currey et al., 2022). In contrast, the problem of generating all valid and grammatically correct gendered translations has seen far less attention (Kuczmarski and Johnson, 2018; Johnson, 2020; Sánchez et al., 2023).
Consider the example: “The secretary was angry with the boss.” The gender of both secretary and boss remain ambiguous in the absence of additional context: both entities can take either gender. However, and to the best of our knowledge, all existing approaches (Kuczmarski and Johnson, 2018; Johnson, 2020; Sánchez et al., 2023; Rarrick et al., 2023) for producing different gendered translations operate on “sentence-level”, instead of on “entity-level”: they only allow two sentence-level alternatives to surface, in which both secretary and boss are either masculine or feminine: • secretary, boss: El secretario estaba enojado con el jefe.2
• secretary, boss: La secretaria estaba enojada con la jefa.
In this work, we introduce a novel approach that operates on entity-level, i.e., it generates four alternatives corresponding to all grammatically valid combinations of gender choices for both entities:
• secretary, boss: El secretario estaba enojado con el jefe. • secretary, boss: El secretario estaba enojado con la jefa. • secretary, boss: La secretaria estaba enojada con el jefe. • secretary, boss: La secretaria estaba enojada con la jefa.
When integrated with a proper user interface, our approach provides users with the freedom to choose gender for each entity. We posit that any such system should meet the following practical quality criteria, making the problem challenging:
2Gendered translations in Spanish. Brown and teal represent masculine and feminine genders respectively.
237
• Alternatives should not be produced when the gender can be inferred from the sentence context, e.g., “She is a boss” should only produce the feminine translation “Ella es una jefa”.
• All alternatives should maintain grammatical gender agreement. Phrases like “El secretaria” or “secretaria estaba enojado” should not be produced as they break gender agreement by using different gendered forms for the same entity.
• Alternatives should differ only in gender inflections and not general wording, formality, etc., as any such differences can potentially encode bias.
This paper presents several key contributions towards studying the task of generating entity-level alternatives, meeting the above quality criteria:
• Producing entity-level alternatives for n genderambiguous entities requires generating 2n different translations. We propose an efficient approach that reduces the problem to generating a single structured translation where “gendersensitive phrases” are grouped together and aligned to corresponding ambiguous entities.
• We open source train datasets 3 for this task for 5 language pairs and establish supervised baselines. We extend an existing test set for this task: GATE (Rarrick et al., 2023) from 3 to 6 language pairs and open source the extended set.
• We develop a semi-supervised approach that leverages pre-trained MT models or large language models (LLMs) for data augmentation. Models trained on augmented data outperform the supervised baselines and can also generalize to language pairs not covered in the train sets.

heading: 2 Entity-Level Gender Alternatives
text: Our key insight for efficiently generating entitylevel gender alternatives is to reduce the problem to generating a single translation with embedded gender structures and their gender alignments.
Consider our previous example: “The secretary was angry with the boss.” We want to generate the following entity-level alternatives: • secretary, boss: El secretario estaba enojado con el jefe. • secretary, boss: El secretario estaba enojado con la jefa. • secretary, boss: La secretaria estaba enojada con el jefe. • secretary, boss: La secretaria estaba enojada con la jefa.
3https://github.com/apple/ ml-gendered-translation
Since we constraint the alternatives to only differ in gender inflections, we can instead produce a single translation with gender-sensitive phrases grouped together as gender structures, shown in () :
( El secretario La secretaria ) estaba ( enojado enojada ) con ( el jefe la jefa )
All alternatives can be derived from this single translation by choosing either the masculine or feminine form in each gender structure. However, doing this naively can give us invalid alternatives that break gender agreement, for example:
El secretario estaba enojada con el jefe
( El secretario La secretaria ) and ( enojado enojada ) correspond to the same entity secretary and cannot have different gender choices. By having gender alignments between each gender structure in the translation and its corresponding gender-ambiguous entity in the source, we can deduce which gender structures are linked together and need to be consistent with each other.
Let x = x1 . . . xn be the source sentence containing n tokens and let Ga ⊆ {1 . . . n} represent the set of indices of gender-ambiguous entities in x. We aim to produce a translation yS :
yS = y1 . . . ( M1 F1 ) . . . (Mk Fk ) . . . ym, (1)
containing a set of gender structures S = {S1 . . . Sk} where Si := ( Mi Fi ) is the ith gender structure. Translation yS is a sequence of two types of elements: {y1 . . . ym} = yS \ S are regular tokens that do not change based on the gender of any entity in Ga and M∗/F∗ are the masculine and feminine inflected forms of the phrases that do change based on the gender of an entity in Ga. Gender alignments can then be formally defined as a oneto-many mapping from Ga to S. An ambiguous entity is aligned to a gender structure ( M F ) iff the correct inflection form (M or F) in the translation depends on the gender of the entity. In our example, secretary is aligned to ( El secretario La secretaria ) , ( enojado enojada ) , and boss is aligned to ( el jefe
la jefa
) . Given the translation
with gender structures yS and gender alignments, alternatives corresponding to any combination of gender assignments of ambiguous entities can be easily derived as follows: for all ambiguous entities with male gender assignment, choose the male form for their aligned gender structures. Similarly, for all entities with female assignments, choose the female form for their aligned gender structures.

heading: 3 Datasets
text: To build and evaluate systems producing alternatives, we prepare train and test sets containing gender structures and gender alignment annotations.

heading: 3.1 Test data
text: We evaluate our models on a combination of two existing test sets that test complementary aspects:
• GATE (Rarrick et al., 2023) has source sentences with at least 1 and at most 3 gender-ambiguous entities with their entity-level alternatives satisfying our quality criteria. It evaluates the system on cases where alternatives should be produced. • MT-GenEval (Currey et al., 2022) contains sentences with entities whose gender can be inferred from the sentence context and are not ambiguous. This set is helpful for evaluating cases where alternatives should not be produced.
These two test sets have different annotation formats and guidelines. In order to unify them, we ask annotators to review and post-edit existing annotations using the following guidelines:
1. Marking gendered words: First, all words in the source referring to entities (people/animals) that can have masculine or feminine grammatical genders are marked. 2. Gender ambiguity annotation: Next, if multiple words refer to the same entity, a head word is selected among them. We guided the annotators to pick the one that acts the most like the subject as the head word. For each head word, if its gender can be inferred from the grammatical context, such as co-referring male/female pronouns, it is marked as such. If no gender can be inferred, the gender is marked as ambiguous. We only rely on grammatical sentence context and not on external knowledge/common gender associations of names/proper nouns. Appendix B discusses how our annotation guidelines handle the problem of masculine generics (Piergentili et al., 2023a), where masculine nouns/pronouns
can be used to refer to ambiguous or collective entities. 3. Gender aware translation: Finally, we ask the annotators to translate the source sentence. Entities without any ambiguity must be translated into the correct gender. If the translation depends on the gender of the ambiguous entities in the source, gender structures and gender alignments are annotated.
Table 1 explains the process with the help of an example annotation. We prepare this unified test set for 6 language pairs: English to German, French, Spanish, Portuguese, Russian, and Italian.4

heading: 3.2 Train data
text: We open source train data containing samples in the same format as the test set to ensure reproducibility and to encourage development of supervised/semisupervised systems for producing alternatives. In contrast to the test sets, which are created via human annotation, we rely on an automatic data augmentation approach (see Appendix C for details) to create train data at scale. The source sentences for the train sets are sampled from Europarl (Koehn, 2005), WikiTitles (Tiedemann, 2012), and WikiMatrix (Schwenk et al., 2021) corpora. The train data are partitioned into two different sets:
• G-Tag contains source sentences with head words for all entities with their gender ambiguity label: Masc., Fem. or Ambiguous. • G-Trans contains gender-ambiguous entities in the source sentences, gender structures in the translations and gender alignments.
To the best of our knowledge, this is the first large-scale corpus that contains gender ambiguities and how they effect gendered forms in the translation. We release these sets for 5 language pairs: English to German, French, Spanish, Portuguese, and Russian. G-Tag contains ∼ 12k sentences and
4We extend the original GATE corpus, which only includes English to Spanish, French, and Italian.
G-Trans contains ∼ 50k sentence pairs per language pair. Detailed statistics of the train and test sets can be found in Appendix A.

heading: 4 Training MT Models to Generate Gender Structures and Alignments
text: We first present how to train MT models that produce gender structures and alignments, assuming parallel data enriched with gender structures and alignments (for example, G-Trans) is available. We then describe a novel data augmentation pipeline that can enrich any regular parallel corpora with gender structures and alignments.
Given a source sentence x = {x1 . . . xn}, translation yS containing gender structures, and gender alignments A, we want to train the MT model to generate yS , A|x. Let’s assume that yS contains k gender structures and A = {a1 . . . ak} where ai represents the source token aligned to the ith gender structure. We serialize each gender structure in yS into a sequence of tokens as follows:
( M F ) → BEG M MID F END
where BEG, MID, and END are special tokens. The model is then trained to produce gender structures in the form of this sequence.
Garg et al. (2019) introduced a technique to train MT models to jointly generate translations and word-alignments. We use their approach to learn generation of gender alignments. Let m1 . . .mk denote the positions of the MID tokens of the gender structures. A specific cross-attention head is chosen and supervised to learn gender alignments. Let n and m denote the lengths of the source and the serialized target respectively and let Pm×n denote the attention probability distribution computed by the selected head. We train the model with regular cross entropy and an additional alignment loss:
L = Lcross-ent − λ
k
k∑
i=1
log(Pmi,ai)
where λ is a scaling factor. This added loss term encourages the attention head to place more probability mass on the aligned source token when generating the MID token belonging to that token’s gender structure. During inference, the gender alignment for the ith gender structure can be computed as:
ai = argmax s∈{x1...xn} Pmi,s
This model can generate gender structures and alignments without any additional inference overhead. Then, using the procedure described in section 2, all entity-level alternatives can be easily derived from the model outputs.

heading: 5 Data Augmentation Pipeline
text: G-Trans dataset provides supervised data to train MT models in the above manner. However, this dataset is small (50k examples per language pair) and has a restrictive domain, limiting the quality of the trained models. We propose a data augmentation pipeline that can take any regular parallel corpora (containing high quality but potentially biased translations) and augment the translations with gender structures and alignments whenever there are ambiguities in the source.
Algorithm 1 Data Augmentation Overview Input: x = {x1 . . . xn} (source sentence) and yB = {y1 . . . ym} (reference translation without gender structures, potentially biased)
▷ Step 1: Detect set of gender-ambiguous entities Ga in the source sentence: Ga ⊆ {1 . . . n} Ga ← GenderAmbiguousEntities(x) if Ga = ϕ then
Output: x, yB , ϕ end if
▷ Step 2: Transform yB into an all-masculine yM and all-feminine yF translations yM ← argmax p(y|x, yB , gender(xi) = male ∀i ∈ Ga) yF ← argmax p(y|x, yB , gender(xi) = female ∀i ∈ Ga)
▷ Step 3: Combine yM and yF into a single translation yS containing gender structures Let yM = y1 . . .M1 . . . yj . . .Mk . . . ym and Let yF = y1 . . . F1 . . . yj . . . Fk . . . ym where y∗ are the common tokens between yM and yF and {(Mi, Fi) | i ∈ 1 . . . k} be the k differing phrases. yS ← group(yM , yF ) = y1 . . . ( M1 F1 ) . . . (Mk Fk ) . . . ym ▷ Step 4: Align each gender structure Si := ( Mi Fi ) to its corresponding ambiguous entity in Ga A← ComputeGenderAlignments(x, yS)
Output: x, ys, A
Algorithm 1 gives an overview of the main components of the pipeline, which we describe in detail in the following subsections. It consists of first detecting gender-ambiguous entities in the source sentence (§5.1), followed by transforming the reference translation into all-masculine/all-feminine translations (§5.2, §5.3), condensing those into single translation with gender structures, and finally aligning the gender structures (§5.4).

heading: 5.1 Detecting gender-ambiguous entities
text: Traditionally, rule-based methods, which rely on dependency parsing and co-reference resolution, are used to detect gender-ambiguous entities in the source sentence (Rarrick et al., 2023; Habash et al., 2019). In contrast, we adopt a data-driven approach. G-Tag dataset contains English source sentences annotated with head-words, which refer to entities with their gender label derived from the grammatical sentence context: ambiguous, masculine, feminine. Following Alhafni et al. (2022), we fine-tune a (BERT-style) pre-trained language model (PLM) using this dataset to tag each source token with one of the four labels: ambiguous, masculine, feminine, or not a headword.

heading: 5.2 Generating all-masculine/feminine translations using fine-tuned MT models
text: If ambiguous entities are detected in the source sentence, then the next step is to transform the high-quality but potentially biased reference translation yB to all-masculine yM and all-feminine yF translations. yM and yF are equivalent to sentencelevel alternatives corresponding to masculine and feminine assignments for all ambiguous entities, respectively. We explore two methods for this task: fine-tuning pre-trained MT models (this subsection) and using LLMs (subsection 5.3).
We fine-tune a pre-trained MT model M on a bitext extracted from the G-Trans dataset. The source sentences of this bi-text contain ambiguous entities tagged as masculine or feminine using <M>/<F> tags, and the target translation has correct gender inflections given the gender tags. Table 2 explains this extraction process in detail using an example.
The fine-tuned model Mfine-tuned learns to generate translations with gender inflections in agreement with the gender assignments (<M>/<F>) in the source. We use Saunders and Byrne (2020)’s lattice rescoring approach to generate yM and yF . Let xM and xF denote source sentences in which
all ambiguous entities (Ga) have been tagged using <M> and <F> tags, respectively. Let I(yB) represent the search space consisting only of all possible gender inflection variants of yB . Mfine-tuned is used to decode yM and yF over the constrained search space I(yB):
yM = argmax y∈I(yB)
pMfine-tuned(y|xM )
yF = argmax y∈I(yB)
pMfine-tuned(y|xF )
This can be done efficiently using constrained beam search. This procedure guarantees that yM , yF , and yB differ only in terms of gender inflections, and therefore, yM and yF possess the same general translation quality as the reference translation yB .

heading: 5.3 Generating all-masculine/feminine translations using LLMs
text: LLMs’ ability to learn using in-context examples (Brown et al., 2020) provides us with an alternative approach for generating yM and yF . We
can provide selected instances from G-Trans as incontext examples in the prompt to the LLM and have it generate output for a test instance (Sánchez et al., 2023). Inspired by re-writing literature (Vanmassenhove et al., 2021; Sun et al., 2021), we design a prompt that treats the LLM as an editor: it edits/re-writes the given translation yB to match the provided gender assignments (all-masculine and all-feminine) in the prompt (See Figure 1 for an example).

heading: 5.4 Aligning gender structures
text: yM and yF are combined together in Step 3 as described in Algorithm 1 to produce a single translation yS containing gender structures. The final step is to align each gender structure in yS to an ambiguous entity in the source. We model this as a tagging task and fine-tune a PLM using alignment annotations in the G-Trans dataset.
Algorithm 2 Alignment Algorithm Input: x = {x1 . . . xn} (source sentence) and yS (transla-
tion with k gender structures) Let yS = y1 . . . ( M1 F1 ) . . . (Mk Fk ) . . . ym for ith gender structure Si := ( Mi Fi ) do
Let | be a special marker token yA ← y1 . . .M1 . . . |Mi| . . .Mk . . . ym ai ← PLM(x; yA) ▷ ; denotes concatenation
end for Output: A = {ai, ∀i ∈ 1 . . . k}
Each gender structure is aligned one-by-one as described in Algorithm 2. To align the ith gender structure Si, we take yM and enclose the phrase corresponding to Si by a special token | to get yA. Then x and yA are concatenated together and fed to the PLM, which is fine-tuned to tag all the tokens in x as aligned/not-aligned to Si (See Figure 5 in the Appendix for an example). The gold aligned/notaligned labels for fine-tuning are extracted from the G-Trans dataset.

heading: 6 Evaluation Metrics
text: We evaluate our systems’ performance using the following metrics:
• Alternatives metrics: These metrics compute the overlap between the set of sentences that have alternatives in the test set and the set of sentences for which the system produces alternatives. This overlap is measured using precision and recall and gives a sense of how often the
system produces alternatives and whether it produces them only when needed. • Structure metrics: These metrics are computed over the set of sentences for which both the test set and system output contain alternatives. They measure the quality of the generated alternatives by computing the overlap between the gender structures in the reference alternatives and the generated alternatives. The overlap is measured using precision and recall. • Alignment accuracy: This is measured as the % of gender structures that are aligned to the correct source entity and reflects the quality of gender agreement in the generated alternatives. • δ-BLEU: Lastly, following Currey et al. (2022), to measure the degree of bias towards a gender, we compute δ-BLEU as follows: We separate the masculine and feminine forms in gender structures (if any) for the reference and the system output, compute masculine and feminine BLEU scores (using sacrebleu (Post, 2018)), and measure the absolute difference between the two:
δ-BLEU = |BLEU(ŷm, ym)− BLEU(ŷf , yf )|
Higher δ-BLEU indicates more bias. Mathematical definitions of alternatives and structure metrics can be found in Appendix K.

heading: 7 Experiments and Results
text: We will first describe the experimental details and results of our data augmentation pipeline in 7.1 and 7.2. We then present the training details of the MT model generating alternatives end-to-end and how it benefits from data augmentation in 7.3 and 7.4.

heading: 7.1 Data augmentation pipeline details
text: The data augmentation pipeline consists of three components: detecting gender-ambiguous entities, generating all-masculine/feminine translations and aligning gender structures.
We build the ambiguous entity detector (§5.1) by fine-tuning xlm-roberta-large (Conneau et al., 2020) using transformers (Wolf et al., 2020). We use the combined G-Tag dataset across all 5 language pairs for fine-tuning.
To generate all-masculine/feminine translations, we explore two approaches: fine-tuning pre-trained MT models (§5.2), and using LLMs (§5.3). For the first approach, we fine-tune the M2M 1.2B (Fan et al., 2021) model using fairseq (Ott et al., 2019). The model is fine-tuned jointly on bi-text
extracted from the G-Trans dataset (as described in Table 2) for all 5 language pairs. The list of gender inflections used for lattice rescoring is collected from Wiktionary (Ylonen, 2022) and inflections present in the G-Trans train and test sets.
For the second approach, we use gpt-3.5-turbo as our LLM and follow the prompt design described in subsection 5.3 with 6 in-context examples. We provide additional ablation studies on the number of in-context examples, different prompt designs, and choice of LLM (gpt vs. OpenLlama-v2-7B (Geng and Liu, 2023)) in Appendix F. We find that using more in-context examples helps, but gains are minimal for > 6. Since LLM decoding does not use lattice rescoring, it is possible that the generated all-masculine/feminine translations differ in more than just gender inflections. To avoid this, we explicitly check the differences and don’t generate gender structures if the differences don’t match any entry in the list of gender inflections.
Lastly, to align gender structures we fine-tune xlm-roberta-large on source, targets, and gender alignments extracted from the G-Trans dataset jointly for all 5 language pairs. The hyperparameters for fine-tuning XLM and M2M models are decided based on validation performance on a held-out portion of the train sets and can be found in Appendices D, E and G.

heading: 7.2 Data augmentation pipeline results
text: The data augmentation pipeline takes source sentences and their reference translations (without gender structures, potentially biased) as inputs. For evaluating the data augmentation pipeline, we feed in the source sentences and their all-masculine reference translations from the test set as inputs. The pipeline returns these translations augmented with
gender structures and alignments. We can then compute the evaluation metrics described in section 6 on the generated gender structures and alignments. Table 3 summarizes the results.
Both M2M and GPT perform mostly on par with the exception of English-Russian, where GPT achieves much lower alternatives recall (58.7 compared to 89.3). The quality of generated gender structures is better for GPT on English-German and English-Portuguese and better for M2M on English-Spanish and English-Russian, as can be seen from the structure metrics. Note that we don’t have any G-Trans data for English-Italian, so the results of the M2M model and the alignment accuracy on English-Italian are purely due to zero-shot generalization of M2M and XLM models (Johnson et al., 2017). Overall, the zero-shot results are comparable to others in terms of alternatives metrics and alignment accuracy but fall behind on structure metrics. The alignment model performs well obtaining ≥ 91% accuracy on all language pairs. δ-BLEU depends on both alternatives and structure metrics and can be used as a single metric to compare systems’ performance. Overall, GPT wins in terms of not relying on any fine-tuning dataset and better performance on English to German, Spanish, and French. Fine-tuning M2M wins in terms of achieving better results on English to Portuguese and Russian and being much more efficient in terms of parameters and inference cost (M2M 1.2B can be fit on a single A100 GPU).
Finally, Table 5 compares the performance of our data augmentation pipeline using M2M against GATE’s sentence-level gender re-writer on their setup. We use our pipeline to re-write an allmasculine reference into an all-feminine form (M→F) and vice-versa (F→M). More details about
the setup and evaluation metrics used for this comparison can be found in Appendix I. We see significant improvements in recall at the cost of relatively small degradation in precision (except EnglishItalian). Our system is able to outperform GATE on their proposed F.5 metric on all 3 language pairs.

heading: 7.3 End-to-end MT model details
text: We train a vanilla multilingual MT model on all 6 language pairs using parallel corpora from Europarl, WikiMatrix, WikiTitles, Multi-UN (Chen and Eisele, 2012), NewsCommentary (Barrault et al., 2019) and Tilde MODEL (Rozis and Skadin, š, 2017). We refer to this as vanilla bi-text. We evaluate the models on gender-related metrics using our gender test set. The details of data pre-processing, training, and model architecture can be found in Appendix J.
A straightforward way to adapt this vanilla model to produce gender alternatives is to use domain adaptation methods towards the G-Trans dataset (which contains gender structures and alignments). To this end, we train another MT model with the vanilla bi-text plus the G-Trans dataset with a prefixed corpus tag <gender> using the loss and serialization described in section 4. Adding corpus tags when mixing corpora from different domains has proven to be quite effective (Kobus et al., 2017; Caswell et al., 2019; Costajussà et al., 2022). During inference, this tag is used to decode gender alternatives. We treat this model as the supervised baseline.
Finally, we train a third model, this time augmenting the entire vanilla bi-text with gender structures and alignments by passing it through our data augmentation pipeline (using M2M since running GPT at scale is cost-prohibitive).
To measure the impact of our approach on general domain translation performance, we evaluate the models on the FLoRes (Costa-jussà et al., 2022) test set. Since FLoRes references don’t contain gender structures, we also remove gender structures from the outputs of our models (if any are present) while evaluating against FLoRes. We do so by choosing the gender form which is more probable according to the model: concretely, for every gender structure BEG M MID F END, we choose either M or F depending on which phrase has a higher average token log probability.

heading: 7.4 End-to-end MT model results
text: Table 4 summarizes the results of these models. The vanilla model cannot generate alternatives and shows a huge bias towards generating masculine forms (δ-BLEU ranging from 5.3 to 12.5 points). This bias is greatly reduced by the supervised baseline. The model trained on augmented data further reduces the bias and obtains the best performance in terms of alternative metrics, alignment accuracy, and δ-BLEU. This shows the effectiveness of the data augmentation pipeline. Augmented data also allows us to train a competitive system for EnglishItalian which lacks supervised data.
Results on general domain translation quality (Column FLoRes BLEU from Table 4) show that compared to the vanilla baseline, the model trained on augmented data suffers no degradation on English to German and Spanish and some degradations (−0.3 to −0.7 BLEU) on Engish to French, Portuguese, Russian and Italian.

heading: 8 Conclusion and Future Work
text: In this work, we study the task of generating entitylevel alternatives when translating a sentence with gender ambiguities into a language with grammatical gender. We open source first train datasets, encouraging future research towards this task, and develop a data augmentation pipeline that leverages pre-trained MT models and LLMs to generate even larger train sets. Finally, we demonstrate that this data can be used effectively to train deploymentfriendly MT models that generate alternatives without any additional inference cost or model components.
Our models and pipeline can enable new translation UIs that support fine-grained gender control and can also find applications in aiding human translators to automatically point out ambiguities and recommend alternative translations.
Future work includes exploring other genderless source languages apart from English (e.g., Chinese, Korean, and Japanese) and associated challenges, as well as extending the approach to non-binary and gender-neutral forms (Lardelli, 2023; Piergentili et al., 2023b; Savoldi et al., 2024).
Bias Statement
This work focuses on the bias a machine translation system can manifest by solely generating one translation from multiple valid ones that exist
with respect to grammatical gender when translating from English to a more gendered language, e.g., French. Singling out one translation as such without offering users the ability to modify the output to match the grammatical gender the user intends for each entity causes two categories of harm: representation harm and quality-of-service harm (Madaio et al., 2020; Blodgett et al., 2020). It causes representational harm by reflecting the potential stereotypes that lead to the default translation (e.g., between occupations and gender) and quality-of-service harm by failing the users who need the output in the target language to be in a grammatical gender case other than what is generated by default. Our work advocates and proposes a solution for enabling users to choose from all equally correct translation alternatives.
Limitations
All mentions of “gender” in this work refer to the grammatical gender present in many languages of the world that are not genderless. Grammatical gender in linguistics is distinct from social gender: while grammatical gender is essentially a noun class system, the discussion surrounding social gender (male, female, nonbinary) encompasses a much more complex set of concepts, e.g., social constructs, norms, roles, and gender identities. Building effective solutions that facilitate inclusive conversations on these topics is not only an open problem in NLP, but many fields.
Moreover, the ambiguities in the linguistic grammatical gender are assumed to be, as in most of the gendered languages, binary: masculine and feminine. However, many languages have more grammatical genders (i.e., noun classes): e.g., Worrorra has masculine, feminine, terrestrial, celestial, and collective.
As such, our proposed resources, as presented so far, fall short of generating entity-level genderneutral translations or disambiguation beyond the binary system of masculine/feminine. However, it’s noteworthy that our pipeline, paired with suitable data resources, e.g., gender-neutral terms for lattice rescoring, forms a powerful instrument for addressing such more challenging settings.

heading: Acknowledgements
text: We would like to thank Yi-Hsiu Liao, Hendra Setiawan, and Telmo Pessoa Pires for their contributions and discussions through different stages of
the project, Matthias Sperber, António Luís Vilarinho dos Santos Lopes, and USC ISI’s CUTELABNAME members for their constructive feedback on the paper drafts, and the whole Machine Translation team at Apple for their support for the project.

heading: A Dataset details
text: Detailed train data statistics are listed in Table 6. Detailed test set statistics are shown in Table 7 and Figure 2.
We had to get the annotations in GATE and MTGenEval reviewed and post-edited from human annotators because their annotation guidelines differ from ours in the following respects:
• GATE defines a gender-ambiguous entity as an entity whose gender cannot be inferred from the grammatical sentence context and whose gender can influence changes in the translation. This second requirement makes this definition of ambiguous entity dependent on the target language/translation. E.g., in “I am going to the market”, despite the gender of I being ambiguous, it would not be marked as an ambiguous entity for EnglishSpanish, since the Spanish translation does not change based on the gender of I. The same entity would be marked as an ambiguous entity in case of English-Hindi where the translation changes based on the gender of I.
In our definition of an ambiguous entity, we drop the second requirement, making it independent of the translation and the target language. This enables us to train an ambiguity tagger solely on the Engish source sentences which can be used for any English-X language pair. This, however, forces us to re-annotate the GATE corpus.
• MT-GenEval corpus contains source sentences with annotated entities whose gender
can be inferred as masculine/feminine from the sentence context. This provides a valuable test-bed for catching false positive gender alternatives. However, we found that ∼ 50% of source sentences also contain one or more ambiguous entities which have not been annotated. Therefore we re-annotate the MTGenEval corpus as well to mark such entities.
Upon the deanonymized publication of this work, we plan to release the datasets under CC BY-SA license.

heading: B Problem of masculine generics during gender ambiguity annotation
text: It is fairly common to use masculine gendered words to refer to ambiguous entities. In administrative and legal text, masculine gendered words have been used to refer to collection of people (Piergentili et al., 2023a) for e.g. “A judge must certify that he has familiarized himself with...”. It is a complex problem to ascertain whether he refers to a masculine individual or a group of (ambiguous gendered) people at large.
In our annotation guidelines we informed the annotators that entities shouldn’t be marked as masculine solely because of masculine generic nouns like actor, sportsmen. However no special guidelines were provided around the trickier case of masculine generic pronouns (he, himself as shown in the example above)

heading: C Synthetically generated train data
text: We used human annotation to collect primary versions of G-Trans and G-Tag datasets (gender train sets) using the annotation process described in subsection 3.1. However, we are unable to release these “human-annotated” sets publicly due to legal and proprietary data restrictions. To make our approach and results reproducible to the community, we instead plan to release "synthetically generated sets" generated as follows: we trained our data augmentation pipeline (described in section 5) on the “human-annotated” training sets and then ran the data augmentation pipeline on corpora mentioned subsection 3.2. We then sampled the G-Trans and G-Tag datasets from the pipeline results and use them throughout our work.

heading: D Gender-ambiguous entity detector
text: The gender-ambiguous entity detector is fine-tuned using the following hyper-parameters:
• batch size: 64 • epochs: 2 • learning rate: 2e-5 • tokenizer: intl from sacrebleu library • subword model: default xlm-roberta-large tokenizer • output labels: <A> (ambiguous), <M> (masculine), <F> (feminine), <N> (not an entity) • linear tagging layer: 1024× 4 • Architecture hyper-parameters can be found
by loading xlm-roberta-large using AutoModelForTokenClassification in transformers. • The tagging loss is applied only on the first subword of each token. The prediction for each token is computed based on the label output for the first sub-word. • We fine-tune all the parameters of the pre-trained model along with the added linear layer. • All reported results are gathered from a single run.
Table 8 summarizes the results of the detector on tagging entities of different genders.

heading: E Generating all-masculine/feminine translations by finetuned-M2M model
text: We fine-tuned a pre-trained M2M-1.2B model with the following hyper-parameters: • batch size: 8192 • learning rate: 3e-5 • encoder layerdrop: disabled • decoder layerdrop: disabled • Rest of the hyper-parameters are the same as the
pre-trained model. • We fine-tune for a total of 40000 steps and se-
lect the best checkpoint based on loss on a held out validation set. • We use the sub-word model and dictionaries of the pre-trained M2M model. However, we add gender assignment tags (<M> and <F>) as new entries in the dictionary and train their embeddings from scratch. • We use a beam size of 5 while decoding allmasculine/feminine translations using latticerescoring. • All reported results are gathered from a single run.

heading: F Ablation studies on generating using LLMs
text: We study the effect of three factors on the effectiveness of LLMs for generating allmasculine/feminine translations as part of our data augmentation process: number of in-context examples, prompt design, and choice of LLM.
F.1 Number of in-context examples
In our preliminary experiments, we found using at least four in-context examples to be necessary for our task, with performance starting to plateau thereafter (see the chart below in Figure 3). We use six in-context examples in the rest of the experiments.
F.2 Choice of LLM and prompt design
In addition to GPT (gpt-3.5-turbo), we also experiment with OpenLLaMA (OpenLlama-v2-7B) (Geng and Liu, 2023), an open reproduction of LLaMA (Touvron et al., 2023). We find these two to vary in overall
performance and robustness to different kinds of prompts.
Specifically, besides the prompt design discussed in the main text, which has the LLM edit an existing translation to satisfy the provided grammatical gender requirements, we also experiment with an additional design: given the input and the grammatical gender requirements, we have the LLM generate the translation from scratch (Figure 4). We call the former the editor-view prompting, and the latter the generator-view prompting.
In editor-view prompting, the base translation can be sourced in any number of ways, including using the reference translation, as we did in subsection 7.2. However, to make the study between editor-view and generator-view fair and make sure reference translations do not give any advantage to the editor-view, we first prompt the LLM for base translations (first call) and then have it edit those (second call). This effectively breaks the task of generating gender alternatives down to two separate tasks for LLMs: translation, and then editing.
Table 9 reports and compares the results of prompting each of the two LLMs we experiment with, using each of the two prompt designs we use. All reported results are gathered from a single run. GPT, expectedly, outperforms OpenLLaMA. And while both generally benefit from breaking down the task under the editor-view (and perform better under editor-view than under generator-view), OpenLLaMA conspicuously profits more. Specifically, OpenLLaMA’s alternative recall under the generator-view suggests that it fails to generate alternatives following the in-context examples. However, under the editor-view, it is able to follow the in-context examples more. The wider gap between the performance of OpenLLaMA under the two prompting approaches compared to that of GPT, shows that for our task, it’s far less robust to different prompt designs.

heading: G Aligning gender-ambiguous entities
text: We fine-tune an xlm-roberta-large model for aligning gender structures to their corresponding ambiguous entities using the following hyperparameters:
• epochs: 1 • output labels: 1(aligned), 2 (not-
aligned) • linear tagging layer: 1024× 2 • Rest of the hyper-parameters are same as the
gender-ambiguous entity detector (Appendix D). • All reported results are gathered from a single
run.
Figure 5 shows an example of input and output when aligning a gender structure.

heading: H Running data augmentation pipeline on outputs of M2M and GPT
text: In this work we focus on running the data augmentation pipeline over parallel corpora to enrich them with gender structures and gender alignments. However, the pipeline can also be run over any translation system to generate entity-level gender alternatives. Table 10 shows the results when the data augmentation pipeline is run over translations from the pre-trained M2M and GPT models.
The pipeline uses fine-tuned M2M when run over translations from the M2M model and the editor-view prompting using GPT when run over translations from GPT. We can see that both M2M and GPT have large bias towards producing masculine translations (δ-BLEU values ranging from 6.5 to 12.7 points). The data augmentation pipeline has multiple components and much higher inference cost than the end-end student model, but can produce higher quality gender alternatives when compared to the end-end model (Table 4 vs. Table 10).

heading: I Comparison against GATE
text: For the comparison against GATE in Table 5, we use exactly the same setup and metrics (Precision/Recall/F0.5) from Rarrick et al. (2023). We evaluate our data augmentation pipeline on the gender re-writing task. Let’s consider the M → F rewriting case: Given a source sentence with ambiguous entities, the task is to re-write an all-masculine reference translation into an all-feminine reference translation. A system might not output a re-write (in case it fails to detect any ambiguous entities or if the re-written output is the same as the input) or it might actually do a re-write. If the system performs a re-write, it’s classified as correct if the re-write matches the all-feminine reference translation exactly. If there is any difference between the two, then the re-write is classified as incorrect. Given these definitions, the Precision and Recall is defined as:
Precision = number of correct re-writes
number of attempted re-writes
Recall = number of correct re-writes total number of examples

heading: J End-to-end MT model to generate alternatives
text: We extract the bi-text used for training end-to-end models using mtdata (Gowda et al., 2021). We
use sentencepiece (Kudo, 2018) to learn a vocabulary of size 36000 tokens. We remove sentence pairs with lengths ≥ 400 sentencepiece tokens or exceeding a token ratio of 1: 3. We train all end-toend models using the following hyper-parameters: • batch size: 458752 • decoder layers: 20 • decoder layers: 3 • lr: 7e-4 • We supervise an attention head in second from
the bottom decoder layer. The scaling factor λ for the alignment loss is set to 0.05. • embedding dim: 512 • shared encoder-decoder and input-output embed-
dings • learning rate: 3e-5 • All reported results are gathered from a single
run. The end-end models produce gender structures without any constraints. This can result in gender structures containing phrases that differ in more than just gender inflections. To avoid this, we explicitly check the gender structures against our collected list of gender inflections and retain only those structures which pass the check.

heading: K Evaluation Metrics
text: The alternatives metrics compute the sentence level precision and recall of generating alternatives. Let I(b) denote an indicator function:
I(b) = { 1 b = True 0 b = False
and given a sentence x, let ϕ(x) check whether x contains gender structures:
ϕ(x) = { True x contains gender structures False otherwise
Let y and ŷ denote the reference from the test set and the system hypothesis respectively, then alternatives precision and recall can be defined as follows:
Precision =
∑ y,ŷ
I(ϕ(y) ∧ ϕ(ŷ)) ∑ ŷ I(ϕ(ŷ))
Recall =
∑ y,ŷ
I(ϕ(y) ∧ ϕ(ŷ)) ∑ y I(ϕ(y))
We compute structure metrics over the subset S where both references and system outputs contain gender structures, i.e. S = {(y, ŷ) | ϕ(y)∧ϕ(ŷ) = True}. Over S, we compute the following statistics:
• Total structures: total number of gender structures present in y for (y, ŷ) ∈ S.
• Predicted structures: total number of gender structures present in ŷ for (y, ŷ) ∈ S
• Correct structures: total number of gender structures which are present in both y and ŷ for (y, ŷ) ∈ S
We can then compute structure precision and recall as follows:
Precision = Correct structures
Predicted structures
Recall = Correct structures Total structures

heading: Future Work
text: We open source first train datasets, encouraging future research towards this task, and develop a data augmentation pipeline that leverages pre-trained MT models and LLMs to generate even larger train sets. Finally, we demonstrate that this data can be used effectively to train deploymentfriendly MT models that generate alternatives without any additional inference cost or model components. Our models and pipeline can enable new translation UIs that support fine-grained gender control and can also find applications in aiding human translators to automatically point out ambiguities and recommend alternative translations. Future work includes exploring other genderless source languages apart from English (e. g. , Chinese, Korean, and Japanese) and associated challenges, as well as extending the approach to non-binary and gender-neutral forms (Lardelli, 2023; Piergentili et al. , 2023b; Savoldi et al. , 2024). Bias Statement
This work focuses on the bias a machine translation system can manifest by solely generating one translation from multiple valid ones that exist
with respect to grammatical gender when translating from English to a more gendered language, e. g. , French. Singling out one translation as such without offering users the ability to modify the output to match the grammatical gender the user intends for each entity causes two categories of harm: representation harm and quality-of-service harm (Madaio et al. , 2020; Blodgett et al. , 2020). It causes representational harm by reflecting the potential stereotypes that lead to the default translation (e. g. , between occupations and gender) and quality-of-service harm by failing the users who need the output in the target language to be in a grammatical gender case other than what is generated by default. Our work advocates and proposes a solution for enabling users to choose from all equally correct translation alternatives. Limitations
All mentions of “gender” in this work refer to the grammatical gender present in many languages of the world that are not genderless