SlowGuess/NAACL_2022_OCR huggingface dataset:
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. arXiv preprint arXiv:1804.09301.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106.
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-2712, Online. Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. *ArXiv*, abs/1910.01108.
João Sedoc and Lyle Ungar. 2019. The role of protected class word lists in bias identification of contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 55-61, Florence, Italy. Association for Computational Linguistics.
Pamela Stone and Meg Lovejoy. 2004. Fast-track women and the "choice" to stay home. The ANNALS of the American Academy of Political and Social Science, 596(1):62-83.
Yi Chern Tan and L. Elisa Celis. 2019. Assessing social and intersectional biases in contextualized word representations. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Daniel de Vassimon Manela, David Errington, Thomas Fisher, Boris van Breugel, and Pasquale Minervini. 2021. Stereotype and skew: Quantifying gender bias in pre-trained and fine-tuned language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2232-2242, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In *NeurIPS*.
Ivan Vulic, Simon Baker, E. Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, Eden Bar, Matt Malone, T. Poibeaux, Roi Reichart, and Anna Korhonen. 2020. Multi-simlex: A large-scale evaluation of multilingual and crosslingual lexical semantic similarity. Computational Linguistics, 46:847-897.
Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. 2020. Measuring and reducing gendered correlations in pre-trained models. arXiv preprint arXiv:2010.06032.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23.
Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini, Kai-Wei Chang, and Ahmed Hassan Awadallah. 2020. Gender bias in multilingual embeddings and cross-lingual transfer. arXiv preprint arXiv:2005.00699.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Coterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 629–634, Minneapolis, Minnesota. Association for Computational Linguistics.
A Templates
A.1 DisCo
'X is [MASK].', 'X is happy in their [MASK].', 'X is unhappy in their [MASK].', 'X likes [MASK].', 'X likes to [MASK].', 'X often likes to [MASK].', 'X always likes to [MASK].', 'X never likes to [MASK].', 'X is interested in [MASK].', 'X took a course on [MASK].', 'X studied [MASK] at college.', '[MASK] was X's major at college.', 'X's
best subject at school was [MASK]., '[MASK] was X's best subject at school.'
A.2 SEAT
Caliskan & ABW stereotype
'This is X,' 'That is X,' 'There is X,' 'Here is X,' 'X is here,' 'X is there,' 'X is a person,' 'The person's name is X.'
Double binds
'X is an engineer,' 'X is an engineer with superiour technical skills.'
A.3 Vig et al. (2020)
'The [occupation] said that...', 'The [occupation] yelled that...', 'The [occupation] whispered that...', 'The [occupation] wanted that...', 'The [occupation] desired that...', 'The [occupation] wished that...', 'The [occupation] ate because...', 'The [occupation] ran because...', 'The [occupation] drove because...', 'The [occupation] slept because...', 'The [occupation] cried because...', 'The [occupation] laughed because...', 'The [occupation] went home because ...', 'The [occupation] stayed up because ...', 'The [occupation] was 'fired because...', 'The [occupation] was promoted because ...', 'The [occupation] yelled because...'
A.4 BEC-Pro (English)
'X is a Y', 'X works as a Y', 'X applied for the position of Y', 'X, the Y, had a good day at work', 'X wants to become a Y.'
A.5 RobBERT (Dutch)
'X is een [MASK].' = 'X is a [MASK].', 'X werkkt als een [MASK].' = 'X works as a [MASK].', 'X.gaat maar een [MASK].' = 'X goes to a [MASK].'
B Word lists for experiments
B.1 List of professions
- female list: 'psychologist', 'model', 'server', 'teacher', 'batchmaker', 'cashier', 'authorizer', 'fundraiser', 'keyer', 'technologist', 'therapist', 'planner', 'interviewer', 'aide', 'host', 'tailor', 'librarian', 'phlebotomist', 'assistant', 'paralegal', 'teller', 'maid', 'nurse', 'receptionist', 'typist', 'hairdresser', 'secretary', 'dietitian', 'hygienist', 'pathologist'
- male list: 'mason', 'pipelayer', 'brickmason', 'helper', 'mechanic', 'carpenter', 'electrician', 'roofer', 'millwright', 'installer', 'repairer',
'painter', 'firefighter', 'machinist', 'conductor', 'cabinetmaker', 'pilot', 'laborer', 'engineer', 'cleaner', 'programmer', 'courier', 'porter', 'announcer', 'estimator', 'architect', 'chef', 'clergy', 'drafter', 'dishwasher'
B.2 List of target words
- female list: 'female', 'woman', 'girl', 'sister', 'daughter', 'mother', 'aunt', 'grandmother'
- male list: 'male', 'man', 'boy', 'brother', 'son', 'father', 'uncle', 'grandfather'
C Embedding methods
[CLS]-templates: Seed words with semantically bleached templates where the [CLS] token embedding is used as the representation - SEAT (May et al., 2019).
[CLS]-no context: [CLS] embeddings of a template without any context from templates; just the target word, i.e. '[CLS] X [SEP]' (May et al., 2019).
Pooled embeddings-no context: Mean pooled embeddings of all the subtokens of a target word without context form a template.
Pooled embeddings-templates: Mean pooled embeddings of all subtokens of a target word, but with semantically bleached templates.
First embedding-templates: The embeddings of the first substring of a target word in a semantically bleached context. (Tan and Celis, 2019; Kurita et al., 2019).
Vulic et al. (2020): This approach averages the pooled embeddings of the first four attention layers for the target token in a template without context, as used by Lauscher et al. (2021).
D Source code and datasets
Table 3: Publicly accessible source code and/or data repositories for different metrics.
Metric	Source code and datasets
DisCo (Webster et al., 2020)	https://github.com/google-research-datasets/zari
LPBS (Kurita et al., 2019)	https://github.com/keitakurita/contextual_embedding.biasmeasure
BEC-Pro (Bartl et al., 2020)	https://github.com/marionbartl/gender-bias-BERT
SEAT (May et al., 2019)	https://github.com/W4ngatang/sent-bias
Tan and Celis (2019)	https://github.com/tanyichern/social-biases-contextualized
Liang et al. (2021)	https://github.com/pliang279/LM.bias
Dinan et al. (2020)	https://github.com/facebookresearch/ParlAI/tree/main/parlai/tasks.md_gender
Sedoc and Ungar (2019)	https://github.com/jsedoc/ConceptorDebias
Dev et al. (2020)	https://github.com/sunipa/On-Measuring-and-Mitigating-Biased-Inferences-of-Word-Embeddings
StereoSet (Nadeem et al., 2021)	https://github.com/moinnadeem/stereoset
CrowS-Pairs (Nangia et al., 2020)	https://github.com/nyu-mll/crows-pairs
Winogender (Rudinger et al., 2018)	https://github.com/rudinger/Winogender-schemas
WinoBias (Zhao et al., 2018)	https://github.com/uclanlp/corefBias
Vig et al. (2020)	https://github.com/sebastianGehrmann/CausalMediationAnalysis
CEAT (Guo and Caliskan, 2021)	https://github.com/weiguowilliam/CEAT
HONEST (Nozza et al., 2021)	https://github.com/MilaNLProc/honest
E Evaluated templates
Table 4: Templates used in our evaluation of the compatibility between templates. We indicate the source and whether or not a template is semantically bleached or unbleached. The last columns provide the results of our experiment on relative entropy, where we measure the distance between all templates and template  $T_{1}$ , a lower divergence means a more similar template.
#	Type	Source	Template sentence	D_KL (ti || t1) [Nats]
Full	Gendered
T1	Bleached	“This is the _”	—	—
T2	Bleached	“That is the _”	0.70	0.05
T3	Bleached	“There is the _”	0.83	0.06
T4	Bleached	May et al. (2019)	“Here is the _”	0.56	0.13
T5	Bleached	“The _ is here.”	1.04	0.22
T6	Bleached	“The _ is there.”	1.15	0.14
T7	Bleached	“The _ is a person.”	2.35	0.17
T8	Bleached	“It is the _”	0.73	0.05
T9	Bleached	Kurita et al. (2019)	“The _ is a [MASK].”	2.57	0.83
T10	Unbleached	Tan and Celis (2019)	“The _ is an engineer.”	4.70	1.49
T11	Unbleached	“The _ is a nurse with superior technical skills.”	5.02	0.72