SlowGuess/NAACL_2022_OCR huggingface dataset:
Bashar Alhafni, Nizar Habash, Houda Bouamor†
Carnegie Mellon University in Qatar
{alhafni, nizar.habash}@nyu.edu, hbouamor@qatar.cmu.edu
In this paper, we define the task of gender rewriting in contexts involving two users (I and/or You) – first and second grammatical persons with independent grammatical gender preferences. We focus on Arabic, a gender-marking morphologically rich language. We develop a multi-step system that combines the positive aspects of both rule-based and neural rewriting models. Our results successfully demonstrate the viability of this approach on a recently created corpus for Arabic gender rewriting, achieving  $88.42\mathrm{M}^2$ $\mathrm{F}_{0.5}$  on a blind test set. Our proposed system improves over previous work on the first-person-only version of this task, by 3.05 absolute increase in  $\mathbf{M}^2$ $\mathrm{F}_{0.5}$ . We demonstrate a use case of our gender rewriting system by using it to post-edit the output of a commercial MT system to provide personalized outputs based on the users' grammatical gender preferences. We make our code, data, and pretrained models publicly available. $^1$
Gender bias is a fundamental problem in natural language processing (NLP) and it has been receiving an increasing attention across a variety of core tasks such as machine translation (MT), co-reference resolution, and dialogue systems. Research has shown that NLP systems have the ability to embed and amplify gender bias (Sun et al., 2019), which not only degrades users' experiences but also creates representational harm (Blodgett et al., 2020). The embedded bias within NLP systems is usually attributed to training models on biased data that reflects the social inequalities of the world we live in. However, even the most balanced of models can still exhibit and amplify bias if they are designed to produce a single text output without taking their users' gender preferences into consideration. Therefore, to provide the correct user-aware
output, NLP systems should be designed to produce outputs that are as gender-specific as the users information they have access to. Users information could be either embedded as part of the input or provided externally by the users themselves. In cases where this information is unavailable to the system, generating all gender-specific forms or a gender-neutral form is more appropriate.
Producing user-aware outputs becomes more challenging for systems targeting multi-user contexts (first and second persons, with independent grammatical gender preferences), particularly when dealing with gender-marking morphologically rich languages. In this paper, we define the task of gender rewriting in contexts involving two users (I and/or You) – first and second grammatical persons with independent grammatical gender preferences and we focus on Arabic, a gender-marking morphologically rich language. The main contributions of our work are as follows:
1. We introduce a multi-step gender rewriting system that combines the positive aspects of rule-based and neural models.
2. We demonstrate our approach's effectiveness by establishing a strong benchmark on a publicly available multi-user Arabic gender rewriting corpus.
3. We show that our best system yields state-of-the-art results on the first-person-only version of this task, beating previous work.
4. We demonstrate a use case of our system by post-editing the output of an MT system to match users' grammatical gender preferences.
This paper is organized as follows. We first discuss related work (§2) as well as relevant Arabic linguistic facts (§3). We then define the gender rewriting task in §4 and describe the data we use and the gender rewriting model we build in §5 and §6. Lastly, we present our experimental setup (§7) and results (§8) and conclude in §9.
1https://github.com/CAMEL-Lab/gender-rewriting/
Human Language Technologies, pages 618 - 631
Substantial research has targeted the problem of gender bias in various NLP tasks such as MT (Rabinovich et al., 2017; Vanmassenhove et al., 2018; Stafanovičs et al., 2020; Savoldi et al., 2021), dialogue systems (Cercas Curry et al., 2020; Dinan et al., 2020; Liu et al., 2020a,b; Sheng et al., 2021), language modeling (Lu et al., 2018; Cordia and Bowman, 2019; Sheng et al., 2019; Vig et al., 2020; Nadeem et al., 2021), co-reference resolution (Rudinger et al., 2018; Zhao et al., 2018a), and named entity recognition (Mehrabi et al., 2019). While the majority of research has focused on tackling gender bias in English by debiasing word embeddings (Bolukbasi et al., 2016; Zhao et al., 2018b; Gonen and Goldberg, 2019; Manzini et al., 2019; Zhao et al., 2020) or by training systems on gender-balanced corpora built using counterfactual data augmentation techniques (Lu et al., 2018; Hall Maudslay et al., 2019; Zmigrod et al., 2019), our work falls under text rewriting through the controlled generation of gender alternatives for morphologically rich languages.
Within text rewriting, Vanmassenhove et al. (2021) and Sun et al. (2021) recently presented rule-based and neural rewriting models to generate gender-neutral sentences in English. For morphologically rich languages and specifically Arabic, Habash et al. (2019) introduced the Arabic Parallel Gender Corpus v1.0 (APGC v1.0) of first-person-singular constructions and designed a two-step gender identification and reinflection system to generate masculine and feminine grammatical gender alternatives. Alhafni et al. (2020) used APGC v1.0 to create a joint gender identification and reinflection model. They treated the problem as a user-aware grammatical error correction task and showed improvements over Habash et al. (2019)'s results. Both efforts modeled gender reinflection using character-level Seq2Seq models. More recently, Alhafni et al. (2022) extended APGC v1.0 to APGC v2.0 by including contexts involving first and second grammatical persons covering singular, dual, and plural constructions; and adding six times more sentences.
In our work, we use APGC v2.0 to build a multi-step gender rewriting system to generate gender alternatives in multi-user contexts. We also show improvements over both Habash et al. (2019)'s and Alhafni et al. (2020)'s results on APGC v1.0.
3 Arabic Linguistic Facts
We highlight two of the many challenges that face Modern Standard Arabic (MSA) NLP systems dealing with gender expressions.
Morphological Richness and Complexity Arabic has a rich and complex morphological system that inflects for many morphological features (gender, number, person, case, state, aspect, mood, voice), in addition to several attachable clitics (prepositions, particles, pronouns) (Habash, 2010). Arabic nouns, adjectives, and verbs inflect for gender: masculine  $(M)$  and feminine  $(F)$ , and for number: singular  $(S)$ , dual  $(D)$  and plural  $(P)$ .
Changing the grammatical gender of Arabic words involves either changing the form of the base word, changing the pronominal enclitics that are attached to the base word, or a combination of both. A base word in Arabic refers to the stem along with its attachable affixes (prefixes, suffixes, circumfixes). Changing the base word gender requires either a suffix change, a pattern change, or a lexical change as shown in Table 1(a-c). Arabic also has clitics that attach to the stem after affixes. A clitic is a morpheme that has the syntactic characteristics of a word but shows evidence of being phonologically bound to another word. In this respect, a clitic is distinctly different from an affix, which is phonologically and syntactically part of the word. Proclitics are clitics that precede the word (like a prefix), whereas enclitics are clitics that follow the word (like a suffix). Pronominal enclitics are pronouns that cliticize to previous words (Table 1(d)). It is worth noting that multiple affixes and clitics can appear in a single word in Arabic and changing the grammatical gender of such words requires changing the genders of both the base word and its clitics (Table 1(f-g)).
Orthographic Ambiguity Arabic uses diacritics to specify short vowels. However, these optional diacritics are usually omitted in Arabic orthography, leaving readers to infer the meaning of certain words based on the context (Habash, 2010). This increases the degree of word ambiguity as gender-specific words could only differ in terms of diacritics. For instance, the verb lsb  $l\varsigma b t^2$  can be diacritized as laibta 'you [masc.] played' or as laibti 'you [fem.] played'.
2Arabic transliteration is in the HSB scheme (Habash et al., 2007).
Paired Gender Alternatives	Rewrite Type
Âmyr (NOUN.MS)
prince	Âmyrh (NOUN.FS)
princess	Suffix Change
ÂHmr (ADJ.MS)
red	HmrA' (ADJ.FS)
red	Pattern Change
Âx (NOUN.MS)
brother	Âxt (NOUN.FS)
sister	Lexical Change
+ myr+km (NOUN.MS+PRON.2MP)
your (MP) prince	+ myr+kn (NOUN.MS+PRON.2FP)
your (FP) prince	Enclitic Change
Âr'A'(NOUN.MP)
princes	ÂrA'At (NOUN.FP)
princes	Pattern Change + Suffix Change
+ myr+km (NOUN.MS+PRON.2MP)
your (MP) prince	+ myr+kn (NOUN.FS+PRON.2FP)
your (FP) princess	Suffix Change + Enclitic Change
+ myr+hm (NOUN.MP+PRON.3MP)
their (MP) princes	+ myr+hn (NOUN.FP+PRON.3FP)
their (FP) princesses	Pattern Change + Suffix Change + Enclitic Change
Table 1: Examples of the changes needed to generate gender alternative forms of gender-specific words in Arabic.
4 The Gender Rewriting Task
We define the task of gender rewriting as generating alternatives of a given Arabic sentence to match different target user gender contexts (e.g., female speaker with a male listener, a male speaker with a male listener, etc.). This requires changing the grammatical gender (masculine or feminine) of certain words referring to the users (speaker/1 $^{\text{st}}$  person and listener/2 $^{\text{nd}}$  person). Previous work done by Habash et al. (2019) and Alhafni et al. (2020) refer to this task as gender reinflation, but we believe that gender rewriting is a more appropriate term given that it goes beyond reinflation. $^{3}$
Notation We will use four elementary symbols to facilitate the discussion of this task: 1M, 1F, 2M and 2F. The digit part of the symbol refers to the grammatical persons (1 $^{\text{st}}$  or  $2^{\text{nd}}$ ) and the letter part refers to the grammatical genders (masculine or feminine). Additionally, we will use B to refer to invariant/ambiguous gender.
We define the sentence-level gender using the following four labels: 1M/2F, 1F/2M, 1M/2F, and 1F/2F. These four labels indicate the grammatical persons and genders of the user contexts we are modeling.
We define the word-level gender based on the genders of the word's base form and its attachable pronominal enclitics (§3) using the notation: base form gender + enclitic gender. This results in
25 word-level gender labels (e.g., B+1F, 1F+2M). We use B to refer to gender invariant/ambiguous words. Examples of the word-level gender labels are shown in Table 2.
Task Definition Given an Arabic sentence and a sentence-level target gender, the goal is to rewrite the input sentence to match the target users' gender preferences.
Some of the models we explore only use sentence-level gender labels; while other models use word-level gender labels to identify which input words need to be rewritten to match the target users' gender preferences.
5 Data
For our experiments, we use the publicly available Arabic Parallel Gender Corpus (APGC) – a parallel corpus of Arabic sentences with gender annotations and gender rewritten alternatives of sentences selected from OpenSubtitles 2018 (Lison and Tiedemann, 2016). The corpus comes in two versions: APGC v1.0 and APGC v2.0. APGC v1.0 was introduced by Habash et al. (2019) and it contains 12,238 first-person-singular Arabic parallel gender-annotated sentences. Alhafni et al. (2022) expanded APGC v1.0 by including contexts involving first and second grammatical persons covering singular, dual, and plural constructions to create v2.0, which contains 80,326 gender-annotated parallel sentences (596,799 words). Both versions of APGC include the original English parallels of the Arabic sentences.
$^{3}$ Morphological reinflation usually refers to reinforcing either a lemma or an already inflected form to produce a desired form of a particular word (Cotterell et al., 2016, 2017).
English	Input	Target 1M/2M	Target 1F/2M	Target 1M/2F	Target 1F/2F
I want to talk to you	B B B B	B B B B	B B B B	B B B B	B B B B
Dad, I am here	B B 2M+B	B B 2M+B	B B 2M+B	B B 2F+B	B B 2F+B
I will tell you [plural] something	B B+2F	B B+2M	B B+2M	B B+2F	B B+2F
I am going to my office	B 1M+B	B 1M+B	B 1F+B	B 1M+B	B 1F+B
Because I am an idiot	1F+B	1M+B	1F+B	1M+B	1F+B
I am glad to know you [plural]	B+2M 1F+B	B+2M 1M+B	B+2M 1F+B	B+2F 1M+B	B+2F 1F+B
Table 2: Examples from the Arabic Parallel Gender Corpus v2.0 including the extended word-level annotations for each sentence and its rewrite to the opposite grammatical gender forms where appropriate. First person gendered words are in purple and second person gendered words are in red. M is Masculine; F is Feminine; and B is invariant.
In all of our experiments, we use an extended version of APGC v2.0 to train and test our systems. We also report results on the test set of APGC v1.0 to compare with previous work.
Annotations Each sentence in APGC v2.0 has word-level gender labels where each word is labeled as B, 1F, 2F, 1M, or 2M. All sentences containing gender-specific words referring to human participants have parallels representing their opposite gender forms. For the sentences without any gender-specific words, their parallels are trivial copies. Out of the 80,326 sentences in APGC v2.0,  $46\%$  (36,980) do not contain any gendered words, whereas sentences with gendered references constitute  $54\%$  (43,346). In terms of the word-level statistics,  $9.7\%$  (58,066) are gender-specific, whereas  $90.3\%$  (538,733) are marked as B.
Moreover, APGC v2.0 is organized into five parallel corpora that are fully aligned (1-to-1) at the word level: Input, Target 1M/2M, Target 1F/2M, Target 1M/2F, and Target 1F/2F. All five corpora are balanced in terms of gender, i.e., the number of words marked as 1F and 1M is the same; and the number of words marked as 2F and 2M is the same. The Input corpus contains sentences with all possible word types (B, 1F, 2F, 1M, 2M). The Target 1M/2M corpus contains sentences that consist of B, 1M, 2M words; the Target 1F/2M corpus contains sentences that consist of B, 1F, 2M words; the Target 1M/2F corpus contains sentences that consist of B, 1M, 2F words; and the Target 1F/2F corpus contains sentences that consist of B, 1F, 2F words. The four target corpora are intended to model the target
users' gender preferences for a particular input.
Splits We use Alhafni et al. (2022)'s splits: 57,603 sentences (427,523 words) for training (TRAIN), 6,647 sentences (49,257 words) for development (DEV), and 16,076 sentences (120,019 words) for testing (TEST).
Preprocessing the Word-Level Annotations Since gender information could be expressed at different parts of Arabic words (§3), we automatically extend the APGC v2.0 word-level annotations to mark the genders of both the base words and their pronominal enclitics.
Our preprocessing pipeline considers the labeled gendered words across the five parallel forms of each sentence in APGC v2.0. If the word ends with a gender marking pronominal enclitic, we label the gender of the enclitic based on predefined rules as 1F, 1M, 2F, or 2M. If the gendered word does not end with a gender-marking enclitic, then we label the enclitic as B. Once the enclitic is labeled, we compare the base form of the word across its parallel forms. If the base form is the same, we label it as B. Otherwise, we assign the base form the same label that is provided as part of APGC v2.0. All gender ambiguous words will be labeled as B. Table 2 presents some examples from APGC v2.0 with the extended word-level gender annotations.
The extended word-level statistics are presented in Appendix B. We make the extended word-level gender annotations publicly available as a new release of APGC (APGC v2.1).<sup>4</sup>
4http://resources.camel-lab.com/
6 The Multi-step Model Approach
Most of the recent work on gender rewriting relies on using Seq2Seq models (Habash et al., 2019; Alhafni et al., 2020; Sun et al., 2021; Jain et al., 2021; Vanmassenhove et al., 2021). However, the lack of large gender-annotated parallel datasets presents a challenge when training Seq2Seq models, and especially when dealing with morphologically rich languages. This issue is highlighted by Alhafni et al. (2020), who report that most of the errors  $(68\%)$  produced by their character-level Seq2Seq model are due to not making any changes to gender-specific words in the input sentences. Given the complexity of the gender rewriting task in Arabic and the relatively small training data size, we model the gender rewriting task using a multi-step system the combines the positive aspects of rule-based and neural models. Our system consists of three components: Gender identification, Out-of-context word gender rewriting, and In-context ranking and selection.
6.1 Gender Identification (GID)
We first identify the word-level gender label (base word + pronominal enclitic) for each word in the input sentence. We build a word-level classifier by leveraging a Transformer-based pretrained language model. There are many Arabic monolingual BERT models available such as AraBERT (Antoun et al., 2020), ARBERT (Abdul-Mageed et al., 2021), and QARIB (Abdelali et al., 2021). However, we chose to use CAMeLBERT MSA (Inoue et al., 2021) as it was pretrained on the largest MSA dataset to date. Following the work of Devlin et al. (2019), we fine-tune CAMeLBERT MSA using Hugging Face's transformers (Wolf et al., 2020) by adding a fully-connected linear layer with a softmax on top of its architecture. During fine-tuning, we use the representation of the first sub-token as an input to the linear layer.
6.2 Out-of-context Word Gender Rewriting
Given the desired sentence-level target gender as an input and the identified gender label for each word in the input sentence, we decide if a word-level gender rewrite is needed based on the compatibility between the provided sentence-level target gender and the predicted word-level gender labels. We implement three word-level gender alternative generation models: Corpus-based Rewriter, Morphological Rewriter, and Neural Rewriter.
Figure 1: The multi-step gender rewriting system. First person gendered words are in purple and second person gendered words are in red. The sentence-level target gender is  $1\mathrm{M} / 2\mathrm{M}$ . The input words glad (1F+B), know you (B+2F), and ladies (2F+B) are rewritten to their masculine forms.
Corpus-based Rewriter (CorpusR) We build a simple word-level lookup rewriting model by exploiting the fully aligned words in APGC v2.1. We implement this model as a bigram maximum likelihood estimator: given an input word with its bigram surrounding context  $(w_{i},w_{i - 1})$ , a gender alternative target word  $(y_{i})$ , and a desired word-level target gender  $(g)$ , the CorpusR model is built by computing  $P(y_{i}|w_{i},w_{i - 1},g)$  over the training examples. During inference, we generate all possible gender alternatives for the given input word  $(w_{i})$ . If the bigram context  $(w_{i},w_{i - 1})$  was not observed in the training data, we backoff to a unigram context. If the input word was not observed during training, we pass it to the output as it is.
Morphological Rewriter (MorphR) For the morphological rewriter, we use the morphological analyzer and generator provided by CAMeL Tools (Obeid et al., 2020). We extend the Standard Arabic Morphological Analyzer database (SAMA) (Graff et al., 2009) used by the morphological generator to produce controlled gender alternatives. We make our extensions to the database publicly available.[5]
<sup>5</sup>We provide code to reconstruct our extended database from the original SAMA 3.1 database (LDC2010L01) which can be obtained from the LDC.
Given an input word and a desired word-level target gender, the morphological generator has the ability to produce gender alternatives by either rewriting the base word, its pronominal enclitics, or both. If an input word does not get recognized by the morphological analyzer and generator, we pass it to the output as it is. It is worth noting that this rewriting model does not require any training data.
Neural Rewriter (NeuralR) Inspired by work done on out-of-context morphological reinfection (Kann and Schütze, 2016; Cotterell et al., 2018), we design a character-level encoder-decoder model with attention. Given an input word and word-level target gender, the encoder-decoder model would generate gender alternatives of the input word. For the encoder, we use a two-layer bidirectional GRU (Cho et al., 2014) and for the decoder we use a two-layer GRU with additive attention (Bahdanau et al., 2015). Furthermore, we employ side constraints (Sennrich et al., 2016) to control for the generation of gender alternatives. That is, we add the word-level target gender as a special token (e.g.,  $<1\mathrm{F}+\mathrm{B}>$ ) to the beginning of the input word and we feed that entire sequence to the model (i.e.,  $<1\mathrm{F}+\mathrm{B}>$ ). The intuition here is that the attentional encoder-decoder model would be able to learn to pay attention to the side constraints to generate the desired gender alternative of the input word. During inference, we use beam search to generate the top 3-best hypotheses.
6.3 In-Context Ranking and Selection
Since the three word-level gender alternative generation models we implement are out-of-context and given Arabic's morphological richness, we expect to get multiple output words when generating a single gender alternative for a particular input word. This leads to producing multiple candidate gender alternative output sentences. To select the best candidate output sentence, we rank all candidates in full sentential context based on their pseudo-log-likelihood (PLL) scores (Salazar et al., 2020). We first use Hugging Face's transformers to fine-tune the CAMeLBERT MSA model on the Input corpus of APGC v2.1 by using a masked language modeling (Devlin et al., 2019) objective. This helps in mitigating the domain shift (Gretton et al., 2006) issue between CAMeLBERT's pretraining data and APGC v2.1. We then compute the PLL score for each sentence using the fine-tuned CAMeLBERT MSA model by masking the sentence tokens one
by one. $^{6}$  We will refer to the in-context ranking and selection as simply selection throughout the paper.
Figure 1 presents an overview of our gender rewriting model. We describe the training settings and the model's hyperparameters in Appendix A.
7.1 Evaluation Metrics
We follow Alhafni et al. (2020) by treating the gender rewriting problem as a user-aware grammatical error correction task and use the MaxMatch  $(\mathbf{M}^2)$  scorer (Dahlmeier and Ng, 2012) as our evaluation metric. The  $\mathbf{M}^2$  scorer computes the precision (P), recall (R), and  $F_{0.5}$  by maximally matching phrase-level edits made by a system to gold-standard edits. The gold edits are computed by the  $\mathbf{M}^2$  scorer based on provided gold references. Moreover and to be consistent with previous work, we also report BLEU (Papineni et al., 2002) scores which are obtained using SacreBLEU (Post, 2018). We report the gender rewriting results in a normalized space for Alif, Ya, and Ta-Marbuta (Habash, 2010).
Do Nothing Our first baseline trivially passes the input sentences to the output as they are. This baseline highlights the level of similarity between the inputs and the outputs.