SlowGuess/NAACL_2022_OCR huggingface dataset:
For the correlation analyses between templates and representation methods, we vary our seed words by creating subsets and we keep the language model (BERT-base-uncased) constant. We start by compiling the sets of attribute words (professionals) and target words (gendered words) following Caliskan et al. (2017) and Zhao et al. (2018), which are split in two sets of male and female "stereotyped" professions ( $\S$  B.1) and we create female and male sets of target words ( $\S$  B.2). We generate 20 subsets  $\{a_1, \dots, a_{20}\}$  by randomly sampling 10 professions for each set of attributes, thus for male and female professions (see  $\S$  B.1 for the full list). We expect that some subsets will show higher levels of bias than others and that given two "accurate" fairness metrics  $\mathcal{M}_1$  and  $\mathcal{M}_2$ , if  $\mathcal{M}_1$  indicates that  $a_1$  contains less bias than  $a_2$  which in turn contains less bias than  $a_3$ ,  $\mathcal{M}_2$  should likewise indicate bias for the three subsets. Caliskan et al. (2017); May et al. (2019); Lauscher et al. (2021); Tan and Celis (2019) used a similar approach to calculate distributional properties and quantify the variance. In our experiments, we use Pearson correlation coefficients.
For the third correlation experiment between fair
ness metrics (§ 4.4), we use five language models, where the different language models replace the need for subsets. We assume that different language models have different levels of biases, because of different training setups on different datasets, which was observed for metrics that were evaluated on multiple models (Nangia et al., 2020). We also use the templates and seed words for each metric as described in the original papers, since we compare the metrics as they are used.
4.2 Compatibility between templates
The choice of template for creating contexts for seed words plays a very important role in measuring bias in contextual word embeddings. Many papers propose the use of "semantically bleached" sentence templates for context which should contain no semantic meaning so that the embedding generated by inserting a seed word into such a template should only represent the seed word. May et al. (2019); Tan and Celis (2019) indicated that semantically bleached templates might still contain some semantics, at least related to the bias.
If these templates are semantically bleached with regard to a gender bias, all these templates should have a high correlation with other bleached templates. We test the bleached SEAT templates (May et al., 2019), listed in Table 2 ( $T_{1} - T_{8}$ ). We also compare with the masked template of used by Kurita et al. (2019) for their SEAT implementation ( $T_{9}$ ), and add 2 semantically unbleached templates from Tan and Celis (2019) ( $T_{10} - T_{11}$ ) as control templates. We test both the [CLS] embedding as
Table 2: Templates used in our evaluation of the compatibility between templates. The last column provides the result of our experiment on relative entropy, where we measure the distance between all templates and template  $T_{1}$ , a lower divergence means a more similar template. The source of the templates is indicated in Table 4 in Appendix E
#	Type	Template sentence	DKL
T1	Bl.	“This is the _”	—
T2	Bl.	“That is the _”	0.05
T3	Bl.	“There is the _”	0.06
T4	Bl.	“Here is the _”	0.13
T5	Bl.	“The _ is here.”	0.22
T6	Bl.	“The _ is there.”	0.14
T7	Bl.	“The _ is a person.”	0.17
T8	Bl.	“It is the _”	0.05
T9	Bl.	“The _ is a [MASK].”	0.83
T10	Unbl.	“The _ is an engineer.”	1.49
T11	Unbl.	“The _ is a nurse with superior technical skills.”	0.72
sentence representation May et al. (2019) and the target token embedding (Tan and Celis, 2019).
We test our hypothesis with a correlation analysis as described in § 4.1 and we additionally test how the distribution differs between templates. We operationalize semantically bleached templates as two templates  $T1, T2$  having the same contextualized probability for a set of tokens on position  $x_{m}$ , following  $P(x_{m} \mid T_{1}) = P(x_{m} \mid T_{2})$ .
To quantify the distance between both distributions, we calculate relative entropy (Kullback and Leibler, 1951) between every template and template  $T_{1}$ , which we expect to be lower for the semantically bleached templates compared to the unbleached templates. We perform this relative entropy experiment twice: (i) once with all tokens in the model's vocabulary and (ii) once with a set of gendered tokens (see § B.2). Both sets aim to evaluate how the contextualized distributions of the masked token  $t_{i} = P(x_{m} \mid T_{i})$  differ, but we expect a lower divergence in particular for the gendered subset. Figure 2a and Table 2 present our results for the correlation analysis and difference in distributions, where we make three observations.
Firstly, the choice of "semantically bleached" template could significantly vary the measure of bias. Although templates  $T_{1} - T_{9}$  are all bleached, there are weak and sometimes even negative correlations (e.g.  $T_{7}$ ). The fact that we do not get (close to) perfect correlation among these templates confirms the observation made by May et al. (2019) on the possible impact that "semantically bleached" templates could have on fairness evaluations.
Secondly, semantically and syntactically similar
Figure 3: Correlations between different representation methods. Notice how both [CLS]-based methods are less correlated than other methods. The Pearson correlation coefficients in bold are significant at the  $\alpha = 0.05$  level.
templates do not necessarily correlate strongly. E.g. "There is the  $\_$ " ( $T_{3}$ ) and "The  $\_$  is there." ( $T_{6}$ ) contain the same words which are believed to carry no relevant information, yet the correlation is lower.
Thirdly, the distributional distances between  $T_{1}$  and all other templates, as measured by the Kullback-Leibler divergence and shown in Table 2, highlight that the different templates are indeed not completely semantically bleached. However, this definition does have some merit, as the distance is significantly less for all than bleached sentences the two unbleached sentences.
Based on the above observations, we conclude that semantically bleached templates need to be used cautiously, and any results stemming from the use of such templates cannot be objectively maintained so long as there does not exist a standardized and validated scheme of selecting such templates.
4.3 Compatibility between representations
Word representations or embeddings could also be a source of inconsistency in evaluating contextualized language models. Since many techniques use templates, it is natural to use the entire sentence representation as the representation of the word in question, e.g. by mean-pooling over all target tokens or using the [CLS] embedding. We test these methods and some additional combinations that have been used in the literature, yet not necessarily for bias evaluations. A complete list with explanations can be found in Appendix C.
Firstly, we investigate whether there are inconsistencies between methods by conducting corre
Correlations between intrinsic and extrinsic measures
Figure 4: Correlations between different intrinsic and extrinsic fairness measures. The Pearson correlation coefficients in bold are significant at the  $\alpha = 0.05$  level.
lation analysis of bias scores produced by SEAT on scores from the subset of attribute words. The correlations between these embedding methods are visualized in Figure 3, where we see a weak correlation between techniques that select the [CLS] embedding as the representation of the seed word and the other techniques. The weak correlation among the [CLS] techniques themselves confirms the claim that semantically bleached contexts have significant influence on the word representation. Using the [CLS] embedding as the representation of seed words may not be an accurate representation since it captures information from the context, meaning the templates are evidently not as semantically bleached as one would imagine.
Secondly, we explore how other embedding selection methods withstand semantic influence from the context/template. Tan and Celis (2019) propose using the contextual word representation of the token of interest instead of [CLS]. We investigate the effectiveness of this approach by replicating the experiment in Figure 2a. The results on the correlations between template types show that using only the embeddings of the target word (Figure 2b) produces more consistent results than using the [CLS] embedding as the representation (Figure 2a). Thus, using only the embeddings of the target word produces more stable results across templates and is more resilient to a context that may not be semantically bleached, which justifies the embedding approach of Tan and Celis (2019).
4.4 Compatibility between metrics
In this section, our goal is to (i) see if there is a general relationship between intrinsic and extrinsic bias measures and (ii) how individual bias metrics correlate with extrinsic bias. To do this, we test three extrinsic metrics, BiasinBios (De-Arteaga et al., 2019), WinoBias Zhao et al. (2018), and skew (de Vassimon Manela et al., 2021). and we evaluate five popular language models<sup>2</sup>. For Wino-Bias, we adapt the models to the OntoNotes 5.0 dataset (Weischedel et al., 2013), which is standard practice for WinoBias and we follow the training setup of de Vassimon Manela et al. (2021).
We performed a correlation analysis between the results of the three extrinsic measures and a set of intrinsic fairness measures from Section 3; the results are presented in Figure 4. We observe that most correlations with the extrinsic BiasInBios measure are negative—which is expected since this measure gives a higher score if more bias is present—but still strongly correlated with some intrinsic measures, like a WEAT variant by Tan and Celis (2019). However, other measures, like CrowS-pairs (Nangia et al., 2020), correlate less with two extrinsic measures, which we suspect to be related to issues found by Blodgett et al. (2021), although more experiments are needed to confirm this. Part of these poor correlations are caused by the differences in templates ( $\S$  4.2) and representations ( $\S$  4.3) that we observed, but such differences remain worrisome.
5 Code
We make the source code available and also publish a package to bundle fairness metrics at https://github.com/iPieter/biased-rulers.
6 Discussion and ethical considerations
We mostly compare one of the most frequently studied settings, namely binary gender biases with a focus on professions. Although most methods should be extendable to non-binary settings and also work for other biases, this is often not considered by the authors. Furthermore, different works also consider different notions of gender and conflate multiple notions (Cao and Daumé III, 2020). Both issues should be addressed in future works.
$^{2}$ bert-base-uncased, bert-large-uncased, roberta-base, distilbert-base-uncased and bert-base-multilingual-uncased.
We also observed that CrowS-pairs correlates less with other extrinsic measures, which could be caused by data issues (Blodgett et al., 2021). Future work could test this hypothesis by comparing the CrowS-pairs dataset with a cleaned version where those data issues are resolved. However, such a version does currently not exist. Related to this, is the design of the templates. We observed excessive variation between templates, similar to the differences between few-shot prompts that are used with autoregressive models like GPT-2 (Lu et al., 2021). Future work could also focus on template designing and refine the concept of semantically bleached templates.
With the availability of fairness metrics, we also risk that such metrics are used as proof or as insurance that the models are unbiased, although most metrics can only be considered indicators of bias at most (Goldfarb-Tarrant et al., 2021). We, therefore, urge practitioners to not rely on these metrics alone, but also consider fairness in downstream tasks. We also did not draw much attention to many other negative impacts of language models that practitioners should consider, e.g. high energy usage or not including all stakeholders when training a language model (Bender et al., 2021).
In this paper, we presented an overview of fairness metrics for contextualized language models and we focused on which templates, embeddings and measures these metrics used. We evaluated how these metrics correlate with each other, as well as how parts of these metrics correlate. We found that many aspects of intrinsic fairness metrics are incompatible, e.g. choosing different templates, embeddings, or even metrics. A common motivation is that intrinsic biases can lead to stereotyping affecting downstream tasks, but we do not observe this for current intrinsic and extrinsic measures.
Our advice is to use a mix of some intrinsic measures of fairness that don't use embeddings directly and eliminate one source of variance, for example DisCo or LPBS, in addition to a measure like Tan and Celis (2019) that seems to correlate well with at least some notion of extrinsic bias. However, we also recommend to perform extrinsic fairness evaluations on downstream tasks, since this is where actual resource allocations happen and where intrinsic and extrinsic biases collude.
We thank Luc De Raedt for his continued support, Jessa Bekker for her practical advice on writing a survey, and Eva Vanmassenhove for sharing her knowledge on gender bias. Pieter Delobelle was supported by the Research Foundation - Flanders (FWO) under EOS No. 30992574 (VeriLearn). Both Pieter Delobelle and Ewoenam Kwaku Tokpo also received funding from the Flemish Government under the "Onderzoekprogramma Artificiele Intelligente (AI) Vlaanderen" programme. Bettina Berendt received funding from the German Federal Ministry of Education and Research (BMBF) - Nr. 16DII113.
Maria Antoniak and David Mimno. 2021. Bad seeds: Evaluating lexical methods for bias measurement. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 1889-1904. Association for Computational Linguistics.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2019. *Fairness and Machine Learning*. fairml-book.org. http://www.fairmlbook.org.
Marion Bartl, Malvina Nissim, and Albert Gatt. 2020. Unmasking contextual stereotypes: Measuring and mitigating BERT's gender bias. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 1-16, Barcelona, Spain (Online). Association for Computational Linguistics.
Elisa Bassignana, Valerio Basile, and Viviana Patti. 2018. Hurtlex: A multilingual lexicon of words to hurt. In 5th Italian Conference on Computational Linguistics, CLIc-it 2018, volume 2253, pages 1-6. CEUR-WS.
Christine Basta, Marta R. Costa-jussà, and Noe Casas. 2019. Evaluating the underlying gender bias in contextualized word embeddings. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 33-39, Florence, Italy. Association for Computational Linguistics.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models
be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610-623, New York, NY, USA. Association for Computing Machinery.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454-5476, Online. Association for Computational Linguistics.
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna M. Wallach. 2021. Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In ACL/IJCNLP.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, page 4356-4364, Red Hook, NY, USA. Curran Associates Inc.
Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186.
Yang Trista Cao and Hal Daumé III. 2020. Toward gender-inclusive coreference resolution. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4568-4595, Online. Association for Computational Linguistics.
Rodrigo Alejandro Chávez Mula and Gerasimos Spanakis. 2020. Evaluating bias in Dutch word embeddings. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 56-71, Barcelona, Spain (Online). Association for Computational Linguistics.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnamaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. New York, NY, USA. Association for Computing Machinery.
Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020. RobBERT: a Dutch RoBERTa-based Language Model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3255-3265, Online. Association for Computational Linguistics.
Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Srikumar. 2020. On measuring and mitigating biased inferences of word embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7659-7666.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. 2020. Multi-dimensional gender bias classification. arXiv preprint arXiv:2005.00614.
Ismael Garrido-Muñoz, Arturo Montejo-Ráez, Fernando Martínez-Santiago, and L. Ureña-López. 2021. A survey on bias in deep nlp. Applied Sciences, 11:3184.
Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic bias metrics do not correlate with application bias. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1926-1940, Online. Association for Computational Linguistics.
Hila Gonen and Y. Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In *NAACL*.
Anthony G Greenwald, Debbie E McGhee, and Jordan LK Schwartz. 1998. Measuring individual differences in implicit cognition: the implicit association test. Journal of personality and social psychology, 74(6):1464.
Wei Guo and Aylin Caliskan. 2021. Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 122-133.
Masahiro Kaneko and Danushka Bollegala. 2021. Unmasking the mask-evaluating social biases in masked language models. arXiv preprint arXiv:2104.07496.
S. Kullback and R. A. Leibler. 1951. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79 - 86.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166-172, Florence, Italy. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations.
Anne Lauscher, Tobias Lüken, and Goran Glavaš. 2021. Sustainable modular debiasing of language models. arXiv preprint arXiv:2109.03646.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning.
Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning, pages 6565-6576. PMLR.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786.
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 622-628, Minneapolis, Minnesota. Association for Computational Linguistics.
Tomas Mikolov, Kai Chen, G.s Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. Proceedings of Workshop at ICLR, 2013.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. Stereoset: Measuring stereotypical bias in pretrained language models. In ACL/IJCNLP.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953-1967, Online. Association for Computational Linguistics.
Debora Nozza, Federico Bianchi, and Dirk Hovy. 2021. HONEST: Measuring hurtful sentence completion in language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2398-2406, Online. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532-1543.