SlowGuess/NAACL_2022_OCR huggingface dataset:
Pieter Delobelle<sup>1</sup>, Ewoenam Kwaku Tokpo<sup>2</sup>, Toon Calders<sup>2</sup> and Bettina Berendt<sup>1,3</sup>
$^{1}$  Department of Computer Science, KU Leuven; Leuven.AI
$^{2}$  Department of Computer Science, University of Antwerp
<sup>3</sup> Faculty of Electrical Engineering and Computer Science, TU Berlin; Weizenbaum Institute
An increasing awareness of biased patterns in natural language processing resources such as BERT has motivated many metrics to quantify 'bias' and 'fairness' in these resources. However, comparing the results of different metrics and the works that evaluate with such metrics remains difficult, if not outright impossible. We survey the literature on fairness metrics for pre-trained language models and experimentally evaluate compatibility, including both biases in language models and in their downstream tasks. We do this by combining traditional literature survey, correlation analysis and empirical evaluations. We find that many metrics are not compatible with each other and highly depend on (i) templates, (ii) attribute and target seeds and (iii) the choice of embeddings. We also see no tangible evidence of intrinsic bias relating to extrinsic bias. These results indicate that fairness or bias evaluation remains challenging for contextualized language models, among other reasons because these choices remain subjective. To improve future comparisons and fairness evaluations, we recommend to avoid embedding-based metrics and focus on fairness evaluations in downstream tasks.
With the popularization of word embeddings by works such as Word2vec (Mikolov et al., 2013), GLoVe (Pennington et al., 2014) and, more recently, contextualized variants such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019), Natural Language Processing (NLP) has seen significant growth and advancement. Word embeddings and later language models have been adopted by many applications. Many of these embeddings have been probed by researchers for biases such as gender stereotypes.
Word embeddings are generally trained on real-world data such that they model statistical properties from the training data. Hence, they pick up
biases and stereotypes that are typically present in the data (Garrido-Muñoz et al., 2021). Although Kurita et al. (2019) and Webster et al. (2020) opine that this can pose significant challenges in downstream applications, this view has been questioned, especially for non-contextualized word embeddings (Goldfarb-Tarrant et al., 2021).
Early works such as Bolukbasi et al. (2016); Caliskan et al. (2017); Gonen and Goldberg (2019) widely explored fairness in non-contextualized embedding methods. In non-contextualized embeddings such as Word2vec and GLoVe embeddings, models are trained to generate vectors that map directly to dictionary words and hence are independent of the context in which the word is used. In contrast, contextualized word embeddings take polysemy (words could have multiple meanings, e.g. 'a stick' vs 'let's stick to') into consideration. Thus different embeddings are generated for a given word depending on the context in which it appears. Because of such differences between the two approaches, popular techniques for detecting and measuring bias in non-contextualized word embeddings, such as WEAT (Caliskan et al., 2017), do not apply naturally to contextualized variants.
Many techniques have been proposed to measure bias in contextualized word embeddings, either as a standalone method (May et al., 2019; Bartl et al., 2020) or as an additional contribution to evaluate fairness interventions (Webster et al., 2020; Lauscher et al., 2021; Kurita et al., 2019). This broad selection of methods makes it difficult for NLP practitioners to select an appropriate and reliable set of metrics to quantify bias and to compare results. This is further exacerbated as these quantifying techniques also involve different choices for attribute and target words, commonly jointly referred to as seed words, templates for context, and different methods for measuring similarity.
In this paper, we combine literature survey and experimental comparisons to compare fairness met
Human Language Technologies, pages 1693 - 1706
rics for contextualized language models. We are guided by the following research questions:
- Which fairness measures exist for contextualized language models such as BERT? (Section 3)
- What challenges do languages other than English pose? (§ 3.3)
- What are the relationships between fairness measures, the templates these measures use, embedding methods, and intrinsic vs extrinsic measures? (Section 4)
- Which set of measures do we recommended to evaluate language resources? (Section 7)
Static word embeddings have typically been used with recurrent neural networks (RNNs), optionally with an attention mechanism (Bahdanau et al., 2014). The transformer architecture (Vaswani et al., 2017) introduced a new paradigm relying only on attention, which proved faster and more accurate than RNNs and did not rely on static word embeddings. The transformer consists of two stacks of attention layers, the encoder and the decoder, with each layer consisting of multiple parallel attention heads. BERT (Devlin et al., 2019) is based on the encoder from this transformer and obtained state-of-the-art results for multiple NLP tasks using transfer learning with a pre-training step and a second finetuning step.
The pre-training task is to reconstruct missing words in a sentence, called masked language modeling (MLM), which helps capture interesting semantics. The training objective for a model with parameters  $\theta$  is to predict the original token on the position of a randomly masked token  $x_{m}$  based on the positional-dependent context  $\mathbf{x}_{/\mathbf{m}} = x_0,\ldots ,x_{m - 1},x_{m + 1},\ldots ,x_N$ , following  $\max_{\theta}\sum_{i = 1}^{N}\mathbf{1}_{x_i = \mathbf{x}_/\mathbf{m}}\log \left(P\left(x_i\mid \mathbf{x}_/\mathbf{m};\theta\right)\right)$  with  $\mathbf{1}_{x_i = \mathbf{x}_/\mathbf{m}}$  as indicator function. After training, the language model can infer the probability that a token occurs on the masked position. As an illustration with the original BERT model, the sentence '[MASK] is a doctor' is filled in with the token 'He' (62%), followed by 'She' (32%). Because the MLM task relies on co-occurrences, this example illustrates how this task captures stereotypes that are present in pre-training datasets, which is referred to as intrinsic bias.
Figure 1: Illustration of the transfer learning paradigm where a language model is first pre-trained on one dataset and afterwards finetuned on another dataset. Both stages can introduce biases.
As a second step, this pre-trained model can be finetuned on a new task, most commonly either sentence classification, which uses the contextualized embeddings of the first token  $x_0 = [\mathrm{CLS}]$ , or token classification, for which the embeddings of each respective token position are used. These embeddings are obtained from output states of the penultimate layer, after which a single linear layer is added and trained. This finetuning is typically done with different datasets that are labeled for the task at hand and here we can observe extrinsic bias with allocational harms (Goldfarb-Tarrant et al., 2021; Blodgett et al., 2020), e.g. gender imbalances in co-reference resolution (see § 3.2).
Many models improved on the original BERT architecture and training setup, e.g. RoBERTa (Liu et al., 2019) was trained on significantly more data for a longer period and without a second pretraining objective, next sentence prediction. ALBERT (Lan et al., 2019) used parameter sharing between attention layers to obtain a smaller model without significant performance degradation. Sanh et al. (2019) also created a smaller BERT variation, DistilBERT, by using knowledge distillation. All these models are MLMs, so this gives us the opportunity to compare bias metrics across models.
2.1 Fairness in word embeddings
Fairness in machine learning has a long standing history and a general introduction is out of scope for this paper, so we refer the reader to Barocas et al. (2019). Typical metrics, e.g. demographic parity, are not directly applicable to tasks dealing with natural language. Furthermore, many NLP applications finetune existing language models, which intertwines extrinsic and intrinsic biases as discussed earlier in Section 2.
Early methods for evaluating bias in noncontextualized embeddings like Word2vec, are WEAT (Caliskan et al., 2017) and a direct bias metric (Bolukbasi et al., 2016). The latter demonstrated that word embeddings contain a (linear) biased subspace, where for example 'man' and 'woman' can be projected on the same gender axis as 'computer programmer' and 'homemaker' (Bolukbasi et al., 2016). These analogies are calculated using cosine distance between vectors to define similarity and also to evaluate the authors' proposed debiasing strategies. In addition, pairs of gendered words were also evaluated using Principal Component Analysis (PCA). This showed that most of the variance stemming from gender could be attributed to a single principal component (Bolukbasi et al., 2016).
In parallel, the Word Embeddings Association Test (WEAT; Caliskan et al., 2017) was developed based on the Implicit Association Tests (IAT; Greenwald et al., 1998) from social sciences. WEAT measures associations between two sets of target words  $\mathcal{X},\mathcal{Y}$ , e.g. male and female names, and another two sets of attribute words  $\mathcal{A},\mathcal{B}$ , e.g. career and family-related words, following
$$
s (\mathcal {X}, \mathcal {Y}, \mathcal {A}, \mathcal {B}) = \sum_ {x \in \mathcal {X}} u (x, \mathcal {A}, \mathcal {B}) - \sum_ {y \in \mathcal {Y}} u (y, \mathcal {A}, \mathcal {B})
$$
with a similarity measure  $u(x, \mathcal{A}, \mathcal{B})^1$  that measures the association between one word embedding  $x$  and the word vectors of attributes  $a \in \mathcal{A}, b \in \mathcal{B}$ , defined as  $(x, \mathcal{A}, \mathcal{B}) = \mathrm{mean}_{a \in \mathcal{A}} \cos(x, a) - \mathrm{mean}_{b \in \mathcal{B}} \cos(x, b)$ . This method relies on a vector representation for each word and by providing a representation from a contextualized model, WEAT can also be adapted for contextualized language models, which we discuss in Section 3 and §4.3.
3 Measuring fairness in language models
3.1 Intrinsic measures
Discovery of Correlations (DisCo). Webster et al. (2020) presented an intrinsic measure Discovery of Correlations (DisCo) that uses templates with two slots such as 'likes to [MASK].', we provide a complete list in § A.1. The first slot  $(-)$  is filled with words based on a set of e.g. first names or nouns related to professions. The second masked slot is filled in by the language model
and the three top predictions are kept. If these predictions differ between sets, this is considered an indication of bias. Lauscher et al. (2021) slightly modified this method by filtering predictions with  $P(x_{m} \mid T) > 0.1$  instead of the top-three items.
Log Probability Bias Score (LPBS). This bias score presented by Kurita et al. (2019) is a template-based method that is similar to DisCo, but also corrects for the prior probability of the target attribute, as for example the token 'He' commonly has a higher prior than 'She'. The reasoning is that correction ensures that any measured difference between attributes can be attributed to the attribute and not to the prior of this token. Bartl et al. (2020) introduced an alternative dataset specifically for this evaluation method, called bias evaluation corpus with professions (BEC-Pro), with templates and seeds in both English and German. We will revisit the German results in § 3.3.
Sentence Embedding Association Test (SEAT). A limitation of WEAT (Caliskan et al., 2017) is that the method does not work directly on contextualized word embeddings, which SEAT solves by using context templates (May et al., 2019). These templates are semantically bleached, so there are no words in there that affect bias measurements, for instance '_ is a [MASK].' We will investigate this concept further in § 4.2.
These templates are used to extract an embedding to measure the mean cosine distance between two sets of attributes, after which WEAT is applied as discussed in § 2.1. This embedding is obtained from the [CLS] token in BERT. May et al. (2019) implemented three tests from WEAT. In addition, the authors also made new tests for double binds (Stone and Lovejoy, 2004) and angry Black woman stereotypes. An approach inspired by SEAT was taken by Lauscher et al. (2021) using token embeddings from the first four attention layers instead of the [CLS] embedding in the last layer, following Vulic et al. (2020). Tan and Celis (2019) also adapted SEAT by relying on the embedding of the token of interest in the last layer, instead of the [CLS] token. We will discuss these different embedding methods in § 4.3.
Contextualized Embedding Association Test (CEAT). Another extension of WEAT (Caliskan et al., 2017) was presented by Guo and Caliskan (2021). CEAT uses Reddit data (up to 9 tokens) as context templates, which provide more realistic
$^{1}$ Caliskan et al. (2017) originally used  $s(x, \mathcal{A}, \mathcal{B})$ .
Table 1: Overview of intrinsic measures of bias for language models. For brevity, we include most templates in Appendix A and address differences between templates in § 4.2. We also discuss the evaluation types (§ 3.1) and embedding types (§ 4.3). We also indicate if data and source code are both available (●), or if only a dataset is available (⊙), or if neither is publicly available (○). The repositories are linked in Appendix D.
Metric	Type	Templates	Models	Embedding type	Code
DisCo (Webster et al., 2020)	Association	§ A.1	BERT, ALBERT	—	○
Lauscher et al. (2021)	Association	BERT	○
LPBS (Kurita et al., 2019)	Association	‘X is a Y’, ‘X can do Y’	BERT	—	●
BEC-Pro (Bartl et al., 2020)	Association	§ A.4	BERT	—	●
Based on WEAT
SEAT (May et al., 2019)	Association	§ A.2	BERT, GPT, ELMo, ..	[CLS] (BERT)	●
Lauscher et al. (2021)	Association	‘[CLS] X [SEP]’	BERT	Vulic et al. (2020)	○
Tan and Celis (2019)	Association	§ A.2	BERT, GPT, GPT-2, ELMo	Target token	●
CEAT (Guo and Caliskan, 2021)	Association	Reddit	BERT, GPT-2, ELMo	Target token	●
CAT (Nadeem et al., 2021)	Association	Stereoset	●
CrowS-Pairs (Nangia et al., 2020)	Association	CrowS-Pairs	BERT, RoBERTa, ALBERT	—	●
Basta et al. (2019)	PCA	—	ELMo	—	○
Zhao et al. (2019)	PCA	—	ELMo	—	◎
Sedoc and Ungar (2019)	PCA	Not mentioned	BERT, ELMo	Mean	●
contexts compared to other WEAT extensions (May et al., 2019; Lauscher et al., 2021; Tan and Celis, 2019; May et al., 2019). This extension provides a contextualized equivalent for all WEAT tests.
Context Association Test (CAT). Nadeem et al. (2021) created StereoSet, a dataset with stereotypes with regard to professions, gender, race, and religion. Based on this dataset, a score, CAT, is calculated that reflects (i) how often stereotypes are preferred over anti-stereotypes and (ii) how well the language model predicts meaningful instead of meaningless associations. Blodgett et al. (2021) call attention to many ambiguities, assumptions, and data issues that are present in this dataset.
CrowS-Pairs. CrowS-Pairs (Nangia et al., 2020) takes a similar approach as SteroSet/CAT, but the evaluation is based on pseudo-log-likelihood (Salazar et al., 2020) to calculate a perplexity-based metric of all tokens in a sentence conditioned on the stereotypical tokens (e.g. 'He'). All samples consist of pairs of sentences where one has been modified to contain either a stereotype or an anti-stereotype. ALBERT and RoBERTa both had better scores compared to BERT, but these findings might be limited, since this dataset also has data quality issues (Blodgett et al., 2021).
All Unmaksed Likelihood (AUL). Kaneko and Bollegala (2021) modify the above CrowS-Pairs measure to consider multiple correct predictions, instead of only testing if the target tokens are predicted. In addition, the authors also argue against evaluations biases using [MASK] tokens, since
these tokens are not used in downstream tasks.
PCA-based methods. Both Basta et al. (2019); Zhao et al. (2019) analyzed gender subspaces in ELMo using a method that is very similar to Bolukbasi et al. (2016). This approach was then applied to BERT-based models (Sedoc and Ungar, 2019). We do not further compare to these methods, since they are less suited to obtain numerical bias scores as they rely on identifying a unique gender axis.
3.2 Extrinsic measures
Extrinsic measures are used to measure how bias propagates in downstream tasks such as occupation prediction and coreference resolution. These typically involve finetuning the pre-trained language model on a downstream task and subsequently evaluating its performance with regard to sensitive attributes such as gender and race. As elsewhere in the bias literature, most evaluations focus on gender bias due to the relative availability of gender-related datasets and the relatively widespread concern for gender-related biases.
BiasInBios. De-Arteaga et al. (2019) developed an English dataset as a classification benchmark for measuring bias in language models, which has been adopted as an extrinsic measure (Webster et al., 2020; Zhao et al., 2020). The task is to predict professions based on biographies of people. Bias is quantified as the true positive rate difference between male and female profiles. We will investigate BiasInBios as a fairness metric in (§ 4.4).
Winograd schemas. The Winograd schema (Levesque et al., 2012), originally designed to test
machine intelligence based on anaphora resolution, has been adapted in various works into benchmark datasets for bias evaluation. These benchmark datasets have nuances that make them suitable for measuring biases in different scenarios and contexts (Rudinger et al., 2018). Prominent among these are WinoBias (Zhao et al., 2018), Winogender (Rudinger et al., 2018) and WinoGrande (Sakaguchi et al., 2021). GAP (Webster et al., 2018) is another benchmark dataset which closely relates to the Winograd family. It has also been used to measure bias in pronoun resolution methods.
The WinoBias dataset covers 40 occupations and is used to measure the ability of a language model to resolve coreferencing of gender pronouns (female and male) in the context of pro-stereotype and anti-stereotype jobs. A pro-stereotype setting is when, for instance, a male pronoun is linked to a male-dominated job, whereas a female pronoun being linked to that same job will be an anti-stereotype. E.g. Pro-stereotype: [The janitor] reprimanded the accountant because [he] got less allowance. Anti-stereotype: [The janitor] reprimanded the accountant because [she] got less allowance. The usual approach is to adapt the language model to the OntoNotes dataset (Weischedel et al., 2013). A model is said to pass the WinoBias test if resolution is done with the same level of performance for pro-stereotype and anti-stereotype instances. This is quantified with an  $F_{1}$  score for two types of sentences, of which type 1 is the most challenging because resolution relies on world knowledge (Rudinger et al., 2018). Using this approach, de Vassimon Manela et al. (2021) extended WinoBias to include skew towards one gender, following  $\frac{1}{2}(|F_1^{f_{pro}} - F_1^{m_{pro}}| + |F_1^{f_{anti}} - F_1^{m_{anti}}|)$ . In (§ 4.4), we will also investigate WinoBias (type 1) and the skew variant as implemented by de Vassimon Manela et al. (2021).
3.3 Measuring biases in other languages
Many languages have some sort of grammatical gender, which can interfere with fairness evaluation metrics presented in § 3.1 that focus mostly on gender stereotyping by measuring associations. The assumption is that there should be no association between e.g. professions and gender. However, these associations can be expected in gendered languages. We provide a brief overview of some methods that address languages beyond English.
Delobelle et al. (2020) and Chávez Mulsa and Spanakis (2020) evaluated RobBERT, a Dutch language model. Delobelle et al. (2020) did this visually with three templates ( $\S$  A.5). Associations between gendered pronouns and professions were not considered an indicator of bias, since this is expected in Dutch. Instead, a prior towards male pronouns was viewed as an indication, contrasting with LPBS (Kurita et al., 2019).
For German, Bartl et al. (2020) evaluated BEC-Pro. The authors found that the scores for male and female professions were very similar, likely because of the gender system.
Finally, Nozza et al. (2021) presented a multilingual approach using HurtLex (Bassignana et al., 2018), focusing on six European languages (English, Italian, French, Portuguese, Romanian, and Spanish) with BERT and GPT-2. Both models replicated multiple stereotypes and reproduced derogatory words across languages, leading the authors to question the suitability for public deployment.
4 On the compatibility of measures
In this section, our goal is to objectively investigate the consistency in indicating bias between various techniques used by previous works. As mentioned earlier, besides the metric choice, three primary factors are important when measuring intrinsic bias in an embedding model: (i) choice of seed words, (ii) choice of templates and (iii) how representations for seed words are generated.
Recent works investigating bias in language models have found issues with inconsistencies between seed words (Antoniak and Mimno, 2021), unvoiced assumptions and data quality issues in StereoSet and CrowS-Pairs templates (Blodgett et al., 2021), and issues with semantically bleached templates (Tan and Celis, 2019). These issues raise some questions for the remaining two factors, for example whether or not the choice of template and technique for selecting embeddings to represent seed words matters in measuring bias? And are "semantically bleached" templates really semantically bleached? Meaning, do they not affect bias measurements? Or in the extreme, can bias in embedding model stay hidden by picking the "wrong" templates or representations? These are questions we seek to answer with a series of experimental analysis where we measure correlations between various approaches to test if these templates and representations measure the same bias.
(a) [CLS] embedding
(b) Target token embedding
Figure 2: Correlation of templates as listed in Table 2 when using two different embedding approaches, namely the [CLS] (Figure 2a) and the pooled target token embeddings (Figure 2b). Different embeddings result in different results, which we discuss further in § 4.3. The Pearson correlation coefficients in bold are significant at the  $\alpha = 0.05$  level.
We conduct correlation analyses between different templates (§ 4.2) and between representation methods (§ 4.3), as well as between measures themselves (§ 4.4). To create a context and to help draw concise conclusions, we focus all our experiments on binary gender bias with respect to professions.