SlowGuess/NAACL_2022_OCR huggingface dataset:
Ninareh Mehrabi, Thamme Gowda, Fred Morstatter, Nanyun Peng, and Aram Galstyan. 2019. Man is to person as woman is to location: Measuring gender bias in named entity recognition.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5356-5371, Online. Association for Computational Linguistics.
Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 7022-7032, Marseille, France. European Language Resources Association.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the Conference of the Association for Computational Linguistics (ACL), pages 311-318, Philadelphia, Pennsylvania, USA.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Brussels, Belgium.
Ella Rabinovich, Raj Nath Patel, Shachar Mirkin, Lucia Specia, and Shuly Wintner. 2017. Personalized machine translation: Preserving original author traits. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1074-1084, Valencia, Spain.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8-14, New Orleans, Louisiana.
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-2712, Online. Association for Computational Linguistics.
Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021. Gender Bias in Machine Translation. Transactions of the Association for Computational Linguistics, 9:845-874.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Controlling politeness in neural machine translation via side constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 35-40, San Diego,
California. Association for Computational Linguistics.
Emily Sheng, Josh Arnold, Zhou Yu, Kai-Wei Chang, and Nanyun Peng. 2021. Revealing persona biases in dialogue systems.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3407-3412, Hong Kong, China.
Arturs Stefanovics, Toms Bergmanis, and Marcis Pinnis. 2020. Mitigating gender bias in machine translation with target gender annotations.
Felix Stahlberg and Shankar Kumar. 2021. Synthetic data generation for grammatical error correction with tagged corruption models. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 37-47, Online. Association for Computational Linguistics.
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1630–1640, Florence, Italy. Association for Computational Linguistics.
Tony Sun, Kellie Webster, Apu Shah, William Yang Wang, and Melvin Johnson. 2021. They, them, theirs: Rewriting with gender-neutral english.
Eva Vanmassenhove, Chris Emmery, and Dimitar Shter-onov. 2021. NeuTral Rewritter: A rule-based and neural approach to automatic rewriting into gender neutral alternatives. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8940-8948, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting gender right in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3003-3008, Brussels, Belgium.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems, volume 33, pages 12388-12401. Curran Associates, Inc.
Zhaohong Wan, Xiaojun Wan, and Wenguang Wang. 2020. Improving grammatical error correction with data augmentation by editing latent representation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2202-2212, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini, Kai-Wei Chang, and Ahmed Hassan Awadallah. 2020. Gender bias in multilingual embeddings and crosslingual transfer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2896-2907, Online. Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018a. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15–20, New Orleans, Louisiana.
Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. 2018b. Learning gender-neutral word embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4847–4853, Brussels, Belgium.
Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell. 2019. Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1651-1661, Florence, Italy.
Gender Identification We fine-tune CAMeL-BERT MSA on a single GPU for 10 epochs with a learning rate of 5e-5, batch size of 32, a seed of 12345, and a maximum sequence length of 128. For the augmentation experiments, we use the same hyperparameters but we run the fine-tuning for 3 epochs. At the end of the fine-tuning, we pick the best checkpoint based on the performance on the DEV set. Our gender identification model has 108,506,901 parameters.
In-Context Ranking and Selection We fine-tune CAMeLBERT MSA on a single GPU for 3 epochs with a learning of 5e-5, batch size of 32, and a seed of 88. The fine-tuned CAMeLBERT MSA model has 109,112,880 parameters.
Neural Rewriter (NeuralR) For the character-level encoder-decoder neural rewritten model we use a character embedding size of 128, a hidden size of 256, a dropout probability of 0.2 on the outputs of each GRU layer, and gradient clipping with a maximum norm of 1. We train for 50 epochs on a single GPU with early stopping after 6 epochs if the loss does not decrease on the DEV set. We use the Adam (Kingma and Ba, 2014) optimizer with an initial learning rate of 5e-4, decaying by a factor of 0.5 if the loss on the DEV set does not decrease after 2 epochs. We train with greedy decoding and a batch size of 32. We also apply scheduled sampling (teacher forcing) (Bengio et al., 2015) with a constant sampling probability (0.3) during training. During inference, we use beam search with a beam width of 10 to produce the top 3-best hypotheses. Our NeuralR model has 3,287,110 parameters.
Joint Models The training settings and the hyperparameters of the joint models are identical to the ones we use in our NeuralR model.
The crucial difference between the Joint+Morph model and its extended variants (Joint+Side Constraints and Joint+Morph+Side Constraints) is that the rewriting in the Joint+Morph model is conditioned on the sentence-level target gender. The representation of the sentence-level target gender in the baseline model is learned as an embedding of size 10 during training and only used in the decoder.
Our Joint+Morph model has 3,481,178 parameters; the Joint+Side Constraints model has
3,293,258 parameters; and the Joint+Morph+Side Constraints model has 3,480,926 parameters.
Training Time The CorpusR model was trained on a single CPU and it took  $\approx 2$  minutes to be trained. All our neural models were trained on a single GPU. Fine-tuning CAMeLBERT MSA on the gender identification task took  $\approx 1$  hour; fine-tuning CAMeLBERT MSA on the MLM objective took  $\approx 1$  hour. Training the NeuralR model with different settings took  $\approx 12$  hours in total. All the baseline joint models took  $\approx 29$  hours to be trained.
It is worth noting that all the results presented in this work are reported over a single run and the hyperparameters of our neural models were manually tuned based on the performance on the DEV set.
B Arabic Parallel Gender Corpus v2.1: Extended Word-Level Annotations
Word Gender Label	Train	Dev	Test
B	385,693	44,629	108,411	538,733
B+1M	28	5	10	43
B+1F	28	5	10	43
B+2M	1,042	98	279	1,419
B+2F	1,042	98	279	1,419
1M+B	3,490	422	958	4,870
1F+B	3,490	422	958	4,870
2M+B	16,320	1,787	4,548	22,655
2F+B	16,320	1,787	4,548	22,655
1M+1F	0	0	0	0
1F+1M	0	0	0	0
1M+2F	1	0	0	1
1F+2M	1	0	0	1
1M+1M	9	0	1	10
1F+1F	9	0	1	10
1M+2M	1	0	0	1
1F+2F	1	0	0	1
2M+1M	1	0	0	1
2F+1M	1	0	0	1
2M+1F	1	0	0	1
2F+1F	1	0	0	1
2M+2M	22	2	8	32
2F+2F	22	2	8	32
2M+2F	0	0	0	0
2F+2M	0	0	0	0
427,523	49,257	120,019	596,799
Table 7: The statistics of the extended word-level gender annotations of APGC v2.1 across the TRAIN, DEV, and TEST splits.
P	R	F0.5	BLEU
(a)	GID + CorpusRAug	MorphR	NeuralR	+ Selection	88.19	86.66	87.88
(b)	GIDAug + CorpusR	MorphR	NeuralR	+ Selection	88.63	86.91	88.28
(c)	GID + CorpusR	MorphR	NeuralRAug	+ Selection	88.50	86.64	88.12
(d)	GIDAug + CorpusRAug	MorphR	NeuralR	+ Selection	88.41	86.89	88.10
(e)	GIDAug + CorpusR	MorphR	NeuralRAug	+ Selection	88.67	86.84	88.30
(f)	GIDAug + CorpusRAug	MorphR	NeuralRAug	+ Selection	88.39	86.87	88.08
Table 8: Results of the data augmentation experiments on the DEV set of APGC v2.1. Aug indicates that the component of the system is trained on the augmented data.
C Augmentation Experiments
When it comes to the data augmentation experiments, we took the best performing system (Table 3(i)) and explored training its different components on the augmented training data. Evaluation results on the DEV set of APGC v2.1 using data augmentation are presented in Table 8. Starting off with training the CorpusR model on the augmented data (Table 8(a)), we notice a decrease in performance by  $0.23\mathrm{F}_{0.5}$  compared to Our Best Model (Table 3(i)). This is attributed to the noisy coverage increase in the CorpusR model and can be observed by the decrease in precision (88.19) and recall (86.66). When we train the GID and the NeuralR models on the augmented data (Table 8(b-c)), we get an increase in  $\mathrm{F}_{0.5}$  reaching 88.28 and 88.12, respectively. However, using both the augmented GID and CorpusR models (Table 8(d)) decreases the performance slightly as it achieves 88.10 in  $\mathrm{F}_{0.5}$ . The best performing system was the one that uses both the augmented GID and NeuralR models (Table 8(e)) as it improves over its non-augmented variant reaching 88.30 (0.19 increase) in  $\mathrm{F}_{0.5}$ . Combining the three augmented GID, CorpusR, and NeuralR models (Table 8(f)) achieves 88.08 in  $\mathrm{F}_{0.5}$ .