SlowGuess/NAACL_2022_OCR huggingface dataset:
Joint Baseline Model Our second baseline uses a variant of the sentence-level linguistically enhanced joint gender identification and rewriting model introduced by Alhafni et al. (2020). The main difference between this model and the one introduced by Alhafni et al. (2020) is that we model four multi-user target genders, whereas they only modeled two single-user target genders (1M, 1F). We implement this joint baseline model using a character-level GRU encoder-decoder with additive attention, where the learned input character-level representations are enriched with word-level morphological features obtained from the extended morphological analyzer that is part of CAMeL Tools. The representation of the input sentence-level target gender (1M/2M, 1F/2M, 1M/2F, 1F/2F) is learned as part of the model and used during decoding when generating gender alternatives. We refer to this baseline as Joint+Morph.
<sup>6</sup>We use Salazar et al. (2020)'s implementation to compute the PLL scores: https://github.com/awslabs/mlm-scoring
Extended Joint Baseline Models Our third and fourth baseline models reduce the complexity of the Joint+Morph model by not learning a representation for the input sentence-level target gender as part of the model. Instead, we provide the sentence-level target gender information as side constraints. We add the target gender as a special token to the beginning of the input sentence (e.g., <1M/2F>Input Sentence) when we feed it to the model. Moreover, we explore the effectiveness of enriching the input character representations with word-level morphological features. To do so, we omit the morphological features from the first joint variant, and we use them in the second. We refer to these models as Joint+Side Constraints and Joint+Morph+Side Constraints, respectively.
7.3 Our Multi-step Models
We explore five variants of the gender rewriting multi-step model described in §6. All five variants use the same gender identification (GID) and in-context selection models, but they differ in their out-of-context word-level gender rewriting generation setup. The first three variants use one word-level gender rewriting model each - CorpusR, MorphR, or NeuralR. The fourth multi-step model uses MorphR as a backoff if the input words that need to be rewritten are not observed by the CorpusR model during training (CorpusR»MorphR). The fifth system uses all three word-level gender alternative generation models in a backoff cascade: CorpusR»MorphR»NeuralR.
7.4 Data Augmentation
Given the relatively small size of APGC v2.1 and motivated by recent work on using data augmentation to improve grammatical error correction (Wan et al., 2020; Stahlberg and Kumar, 2021), we investigate adding additional training examples through data augmentation. We randomly selected 800K sentences from the English-Arabic portion of the OpenSubtitles 2018 dataset, which was used to build APGC. We ensured that all extracted pairs include either first or second (or both) person pronouns on the English side: I, me, my, mine, myself, and you, your, yours, yourself. To generate gender alternatives of the selected Arabic sentences, we pass each sentence four times through our best gender rewriting system to generate all four user gender contexts (1M/2M, 1F/2M, 1M/2F, 1F/2F). We add the 800K selected Arabic sentences and their 1M/2M, 1F/2M, 1M/2F, 1F/2F generated gen
der alternatives to the Input, Target 1M/2M, Target 1F/2M, Target 1M/2F, and Target 1F/2F corpora of the training split of APGC v2.1, respectively. At the end, we end up with 857,603 Arabic parallel sentences (6,209,958 words). We make the synthetically generated data publicly available.
Table 3 presents the DEV set results. Joint+Side Constraints and Joint+Morph+Side Constraints significantly improve over the Joint+Morph baseline with up to 13.87 increase in  $\mathrm{F}_{0.5}$ . The best performing system overall is our multi-step model using all rewrite components - Table 3(i), henceforth, Our Best Model. It improves over all the joint models in every compared metric, including a 22.84 increase in  $\mathrm{F}_{0.5}$  when compared to the Joint+Morph baseline. Our Best Model's biggest advantages seem to come from combining the three word-level out-of-context gender alternative generation models in a cascaded setup to deal with OOV words during the generation. Comparing (i) with (e,f,g) in Table 3, we observe improvements ranging from 3.91 to 6.02  $\mathrm{F}_{0.5}$ .
We used Our Best Model to conduct the data augmentation experiments as discussed in §7.4. The full set of augmentation experiment results are presented in Appendix C. The best augmented model's results, which benefits from augmentation in the GID and NeuralR components, are also presented in Table 3(j). However, its increase of 0.19 points in  $\mathrm{F}_{0.5}$  is not statistically significant with McNemar's (McNemar, 1947) test at  $p > 0.05$ .
The results of our best models on the TEST sets of APGC v2.1 and APGC v1.0 are presented in Table 4. The results on APGC v2.1 TEST show consistent conclusions with the DEV results. Our best augmented model improves over its nonaugmented variant in every compared metric, including a 0.25 absolute increase in  $\mathrm{F}_{0.5}$  that is statistically significant with McNemar's test at  $p < 0.05$ . For APGC v1.0, we do not report results with augmentation for fair comparison to previous results. In both TEST sets, we establish new SOTAs.
Error Analysis We conducted an error analysis over the output of our best augmented system on APGC v2.1 DEV. In total, there were 1,475 (5.5% out of 26,588) sentences with errors across the four target corpora. Table 5 presents a summary of the error types our best augmented gender rewriting model makes.
P	R	F0.5	BLEU
(a) Do Nothing	100.0	0.0	0.0	89.36
(b) Joint + Morph	64.76	67.40	65.27	93.31
(c) Joint + Side Constraints	77.10	77.71	77.22	95.60
(d) Joint + Morph + Side Constraints	78.97	79.84	79.14	96.17
(e) GID + CorpusR + Selection	88.22	71.22	84.20	96.54
(f) GID + MorphR + Selection	84.48	75.29	82.47	96.96
(g) GID + NeuralR + Selection	84.62	73.32	82.09	96.75
(h) GID + CorpusR > MorphR + Selection	88.59	85.84	88.02	97.96
(i) GID + CorpusR > MorphR > NeuralR + Selection	88.46	86.74	88.11	98.04
(j) GIDAug + CorpusR > MorphR > NeuralRAug + Selection	88.67	86.84	88.30	98.05
Table 3: Results of a number of systems on the DEV set of APGC v2.1. Aug indicates using augmented data.
P	R	F0.5	BLEU
APGC v2.1 Test	Joint + Morph + Side Constraints	79.27	80.44	79.50	96.19
GID + CorpusR > MorphR > NeuralR + Selection	88.70	86.13	88.17	97.98
GIDAug + CorpusR > MorphR > NeuralRAug + Selection	88.86	86.69	88.42	98.05
APGC v1.0 Test	Habash et al. (2019)	77.74	52.06	70.76	98.28
Alhafni et al. (2020)	78.98	60.32	74.38	98.49
GID + CorpusR > MorphR > NeuralR + Selection	78.57	73.17	77.43	98.92
Table 4: Gender rewriting results on the TEST sets of APGC v2.1 and APGC v1.0.
1M/2M	1F/2M	1M/2F	1F/2F
GID	150 56%	194 70%	325 68%	324 72%
Rewrite	69 26%	50 18%	82 17%	66 15%
Select	49 18%	35 13%	73 15%	58 13%
Total	268	279	480	448
Table 5: Error type statistics of our best augmented system's performance on APGC v2.1 DEV.
Target	1M/2M	1F/2M	1M/2F	1F/2F
Google Translate	13.59	13.15	11.38	10.96
Best SystemAug	13.71	13.64	13.30	13.23
Table 6: BLEU results on the post-edited Google Translate output of APGC v2.1 TEST using our best augmented system.
The majority of errors (67.3%) were caused by GID which achieves a word-level accuracy of 98.9% on DEV. The gender-rewriting errors constituted 18.1% and selection errors 14.6%. Considering different target corpora, we observe that every time an F target is added, the number of errors increases. The 1M/2M target outputs has the lowest number of errors (268 or 18%), while the 1M/2F targets outputs has the highest number of errors (480 or 33%).
Use Case: Post-Editing MT Output We demonstrate next how our proposed gender rewriting model could be used to personalize the output of user-unaware MT systems through post-editing. We use the English to Arabic Google Translate output sentences that are part of APGC v2.1. We evaluate Google Translate's output against all four target corpora (1M/2M, 1F/2M, 1M/2F, 1F/2F) separately. To re-target Google Translate's Arabic output for the four user gender contexts we model, we pass each Arabic sentence four times through our best augmented system (Table 3(j)). We present the evaluation in terms of BLEU in Table 6 over APGC v2.1 TEST. All the results are reported in an orthographically normalized space for Alif, Ya, and Ta-Marbuta.
Again, we observe that every time an M participant is switched to F, the BLEU scores drop for Google Translate's output. This is consistent to what have been reported by Alhafni et al. (2022) and highlights the bias the machine translation output has towards masculine grammatical gender preferences. The post-edited outputs generated by our best augmented system improves over Google Translate's across the four target user contexts, achieving the highest increase in 2.27 BLEU points for 1F/2F.
We defined the task of gender rewriting in contexts involving two users (I and/or You), and developed a multi-step system that combines the positive aspects of both rule-based and neural rewriting models. Our best models establish the benchmark for this newly defined task and the SOTA for a previously defined first person version of it. We further demonstrated a use case of our gender rewriting system by post-editing the output of a commercial MT system to provide personalized outputs based on the users' grammatical gender preferences.
In future work, we plan to explore the use of other pretrained models, and to work on the problem of gender rewriting in other languages and dialectal varieties.
Gender Rewriting Our underlying intention of developing a gender rewriting model for Arabic is to increase the inclusiveness of NLP applications that deal with gender-marking morphologically rich languages. Our work aims at empowering and allowing users to interact with NLP technology in a way that is consistent with their social identities. We acknowledge that by limiting the choice of gender expressions to the grammatical gender choices in Arabic, we exclude other alternatives such as non-binary gender or no-gender expressions. However, we are not aware of any sociolinguistics published research that discusses such alternatives for Arabic. We stress on the importance of adapting Arabic NLP models to new gender alternative forms as they emerge as part of the language usage. We further recognize the limitations of the gender identification component we use as part of our multi-step gender rewriting model as it relies on a language model that was pretrained on a large monolingual Arabic corpus, which could possibly contain biased text. We realize the potential risks of our proposed gender rewriting model if it is intentionally maliciously misused to produce gender alternatives that do not match the target users' gender preferences.
Data We use the publicly available Arabic Parallel Gender Corpus (APGC).<sup>7</sup> It is subject to its creators' own Copy Rights and User Agreement and we strictly adhere to its intended usage. It is worth
noting that APGC does not contain any heterocentric assumptions as part of its annotations (e.g., the word  $\mathcal{L}$  'my husband' is labeled as genderambiguous (B)). Moreover, all proper names are labeled as B, even when they have strong gender-specific association (Alhafni et al., 2022). The Arabic data we use for our data augmentation experiments was randomly sampled from OpenSubtitles 2018 (Lison and Tiedemann, 2016), which is publicly available. OpenSubtitles is distributed under a Creative Commons license.
We thank Alberto Chierici, Christian Khairallah, Go Inoue, and Ossama Obeid for the helpful discussions. We acknowledge the support of the High Performance Computing Center at New York University Abu Dhabi. Finally, we wish to thank the anonymous reviewers at NAACL 2022 for their feedback.
Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations.
Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088-7105, Online. Association for Computational Linguistics.
Bashar Alhafni, Nizar Habash, and Houda Bouamor. 2020. Gender-aware reinflection using linguistically enhanced neural models. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 139-150, Barcelona, Spain (Online). Association for Computational Linguistics.
Bashar Alhafni, Nizar Habash, and Houda Bouamor. 2022. The Arabic Parallel Gender Corpus 2.0: Extensions and analyses. In Proceedings of the 13th Language Resources and Evaluation Conference (LREC), Marseille, France.
Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association.
<sup>8</sup>https://opus.nlpl.eu/OpenSubtitles-v2018.php
${}^{9}$  Attribution-Non Commercial 4.0 International.
<sup>7</sup>http://resources.camel-lab.com/
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations (ICLR).
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. CoRR, abs/1506.03099.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454-5476, Online. Association for Computational Linguistics.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligram, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
Shikha Bordia and Samuel R. Bowman. 2019. Identifying and reducing gender bias in word-level language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 7-15, Minneapolis, Minnesota.
Amanda Cercas Curry, Judy Robertson, and Verena Rieser. 2020. Conversational assistants and gender stereotypes: Public perceptions and desiderata for voice personas. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 72-78, Barcelona, Spain (Online). Association for Computational Linguistics.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, Doha, Qatar.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Geraldine Walther, Ekaterina Vylomova, Arya D. McCarthy, Katharina Kann, Sabrina J. Mielke, Garrett Nicolai, Miikka Silfverberg, David Yarowsky, Jason Eisner, and Mans Hulden. 2018. The CoNLL-SIGMORPHON 2018 shared task: Universal morphological reinflection. In Proceedings of the CoNLL-SIGMORPHON 2018 Shared Task: Universal Morphological Reinfection, pages 1-27, Brussels. Association for Computational Linguistics.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Geraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017. CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinfection in 52 languages. CoRR, abs/1706.09031.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. The SIGMORPHON 2016 shared task—morphological reinflation. In Proceedings of the Workshop of the Special Interest Group on Computational Morphology and Phonology (SIGMORPHON), Berlin, Germany.
Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 568-572, Montreal, Canada.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. 2020. Queens are powerful too: Mitigating gender bias in dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8173-8188, Online. Association for Computational Linguistics.
Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them.
David Graff, Mohamed Maamouri, Basma Bouziri, Sondos Krouna, Seth Kulick, and Tim Buckwalter. 2009. Standard Arabic Morphological Analyzer (SAMA) Version 3.1. Linguistic Data Consortium LDC2009E73.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Scholkopf, and Alex Smola. 2006. A kernel method for the two-sample-problem. In Advances in Neural Information Processing Systems, volume 19. MIT Press.
Nizar Habash, Houda Bouamor, and Christine Chung. 2019. Automatic gender identification and reinfection in Arabic. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 155-165, Florence, Italy.
Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods, pages 15-22. Springer, Netherlands.
Nizar Y Habash. 2010. Introduction to Arabic natural language processing, volume 3. Morgan & Claypool Publishers.
Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell, and Simone Teufel. 2019. It's all in the name: Mitigating gender bias with name-based counterfactual data substitution. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing
and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5267-5275, Hong Kong, China.
Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92-104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
Nishtha Jain, Maja Popovic, Declan Groves, and Eva Vanmassenhove. 2021. Generating gender augmented data for NLP. In Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing, pages 93-102, Online. Association for Computational Linguistics.
Katharina Kann and Hinrich Schütze. 2016. Single-model encoder-decoder with explicit morphological representation for reinflation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 555-560, Berlin, Germany.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the Language Resources and Evaluation Conference (LREC), Portož, Slovenia.
Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao Liu, and Jiliang Tang. 2020a. Does gender matter? towards fairness in dialogue systems. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4403-4416, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Haochen Liu, Wentao Wang, Yiqi Wang, Hui Liu, Zitao Liu, and Jiliang Tang. 2020b. Mitigating gender bias for neural dialogue generation with adversarial learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 893-903, Online. Association for Computational Linguistics.
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2018. Gender bias in neural natural language processing.
Thomas Manzini, Lim Yao Chong, Alan W Black, and Yulia Tsvetkov. 2019. Black is to criminal as Caucasian is to police: Detecting and removing multiclass bias in word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 615-621, Minneapolis, Minnesota. Association for Computational Linguistics.
Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153-157.