SlowGuess/ACL_2020_OCR huggingface dataset:
Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Felix Stahlberg, Christopher Bryant, and Bill Byrne. 2019. Neural grammatical error correction with finite state transducers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4033-4039.
Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne. 2016. Syntactically guided neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 299-305, Berlin, Germany. Association for Computational Linguistics.
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679-1684, Florence, Italy. Association for Computational Linguistics.
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1630–1640, Florence, Italy. Association for Computational Linguistics.
Brian Thompson, Jeremy G Winnup, Huda Khayrallah, Kevin Duh, and Philipp Koehn. 2019. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2062-2068, Minneapolis, Minnesota. Association for Computational Linguistics.
Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting gender right in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3003-3008, Brussels, Belgium. Association for Computational Linguistics.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2Tensor for neural machine translation. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), pages 193-199, Boston, MA. Association for Machine Translation in the Americas.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000-6010.
Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. 2018. Denoising neural machine translation training with trusted data and online data selection. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 133-143, Belgium, Brussels. Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2979–2989, Copenhagen, Denmark. Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15-20, New Orleans, Louisiana. Association for Computational Linguistics.
Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 3530-3534, Porto Roz, Slovenia. European Language Resources Association (ELRA).
Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell. 2019. Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1651-1661, Florence, Italy. Association for Computational Linguistics.
A WinoMT male:female prediction ratio
We report  $\Delta G$  on WinoMT for easy comparison to previous work, but also find that M:F prediction ratio on WinoMT is an intuitive and interesting metric. Tables 6 and 7 expand on the results of Tables 3 and 4 respectively.
7735
en-de	en-es	en-he
BLEU	Acc	M:F	BLEU	Acc	M:F	BLEU	Acc	M:F
Baseline	42.7	60.1	3.4	27.8	49.6	6.3	23.8	51.3	2.2
Original	41.8	60.7	3.1	28.3	53.0	4.0	23.5	53.6	2.0
FTrans original	43.3	60.0	3.9	27.4	51.6	5.4	23.4	48.7	3.0
FTrans swapped	43.4	63.0	3.1	27.4	53.7	4.0	23.7	48.1	2.6
Balanced	42.5	64.0	2.7	27.7	52.8	4.3	23.8	48.3	2.7
Table 6: General test set BLEU and WinoMT scores after unregularised fine-tuning the baseline on four gender-based adaptation datasets.
en-de	en-es	en-he
BLEU	Acc	M:F	BLEU	Acc	M:F	BLEU	Acc	M:F
Baseline	42.7	60.1	3.4	27.8	49.6	6.3	23.8	51.3	2.2
Balanced	42.5	64.0	2.7	27.7	52.8	4.3	23.8	48.3	2.7
Handcrafted (no overlap)	40.6	71.2	1.7	26.5	64.1	2.4	23.1	56.5	0.8
Handcrafted	40.8	78.3	1.3	26.7	68.6	1.9	22.9	65.7	0.9
Handcrafted (converged)	36.5	85.3	0.9	25.3	72.4	1.5	22.5	72.6	1.0
Handcrafted EWC	42.2	74.2	1.6	27.2	67.8	2.0	23.3	65.2	1.2
Rescore 1 with 3	42.7	68.3	2.2	27.8	62.4	2.3	23.9	56.2	1.3
Rescore 1 with 4	42.7	74.5	1.6	27.8	64.2	2.1	23.9	58.4	1.3
Rescore 1 with 5	42.5	81.7	1.1	27.7	68.4	1.8	23.6	63.8	1.3
Table 7: General test set BLEU and WinoMT scores after fine-tuning on the handcrafted profession set, compared to fine-tuning on the most consistent counterfactual set. Lines 1-2 duplicated from Table 6. Lines 3-4 vary adaptation data. Lines 5-6 vary adaptation training procedure. Lines 7-9 apply lattice rescoring to baseline hypotheses.
7736