SlowGuess/NAACL_2022_OCR huggingface dataset:
Hadas Orgad<sup>1</sup>
Seraphina Goldfarb-Tarrant $^2$
Yonatan Belinkov $^{1*}$
$^{1}$ Technion – Israel Institute of Technology
$^{2}$ University of Edinburgh
orgad.hadas@cs.technion.ac.il
s.tarrant@ed.ac.uk
belinkov@technion.ac.il
Common studies of gender bias in NLP focus either on extrinsic bias measured by model performance on a downstream task or on intrinsic bias found in models' internal representations. However, the relationship between extrinsic and intrinsic bias is relatively unknown. In this work, we illuminate this relationship by measuring both quantities together: we debias a model during downstream fine-tuning, which reduces extrinsic bias, and measure the effect on intrinsic bias, which is operationalized as bias extractability with information-theoretic probing. Through experiments on two tasks and multiple bias metrics, we show that our intrinsic bias metric is a better indicator of debiasing than (a contextual adaptation of) the standard WEAT metric, and can also expose cases of superficial debiasing. Our framework provides a comprehensive perspective on bias in NLP models, which can be applied to deploy NLP systems in a more informed manner.
Efforts to identify and mitigate gender bias in Natural Language Processing (NLP) systems typically target one of two notions of bias. Extrinsic evaluation methods and debiasing techniques focus on the bias reflected in a downstream task (De-Arteaga et al., 2019; Zhao et al., 2018), while intrinsic methods focus on a model's internal representations, such as word or sentence embedding geometry (Caliskan et al., 2017; Bolukbasi et al., 2016; Guo and Caliskan, 2021). Despite an abundance of evidence pointing towards gender bias in pretrained language models (LMs), the extent of harm caused by these biases is not clear when it is not reflected in a specific downstream task (Barocas
(a) Training a model on the original task's dataset
(b) Training another model on a debiased dataset
(c) Measuring our intrinsic metric on the debiased and original dataset using probing
Figure 1: Our proposed framework. Black arrows mark forward passes, red arrows mark things we measure. We first (a) train a model on a downstream task, then (b) train another model on the same task using a debiased dataset, and finally (c) measure intrinsic bias in both models and compare.
et al., 2017; Kate Crawford, 2017; Blodgett et al., 2020; Bommasani et al., 2021). For instance, while the word embedding proximity of "doctor" to "man" and "nurse" to "woman" is intuitively normatively wrong, it is not clear when such phenomena would lead to downstream predictions manifesting in social biases. Recently, Goldfarb-Tarrant et al. (2021) have shown that debiasing static embeddings intrinsically is not correlated with extrinsic gender bias measures, but the nature of the reverse relationship is unknown: how are extrinsic interventions reflected in intrinsic representations? Furthermore, Gonen and Goldberg (2019a) demonstrated that a number of intrinsic debiasing methods applied to static embeddings only partially remove the bias and that most of it is still hidden within the embed
*Supported by the Viterbi Fellowship in the Center for Computer Engineering at the Technion.
1Our code and model checkpoints are publicly available at https://github.com/technion-cs-nlp/gender_internal
Human Language Technologies, pages 2602 - 2628
ding. Complementing their view, we examine extrinsic debiasing methods, as well as demonstrate the possible harm this could cause. Contrary to their conclusion, we do not claim that these debiasing methods should not be trusted, as long as they are utilized with care.
Our goal is to gain a better understanding of the relationship between a model's internal representations and its extrinsic gender bias by examining the effects of various debiasing methods on the model's representations. Specifically, we fine-tune models with and without gender debiasing strategies, evaluate their external bias using various bias metrics, and measure intrinsic bias in the representations. We operationalize intrinsic bias via two metrics: First, we use CEAT (Guo and Caliskan, 2021), a contextual adaptation of the widely used intrinsic bias metric WEAT (Caliskan et al., 2017). Second, we propose to use an information-theoretic probe to quantify the degree to which gender can be extracted from the internal model representations. Then, we examine how these intrinsic metrics correlate with a variety of extrinsic bias metrics that we measure on the model's downstream performance. Our approach is visualised in Figure 1.
We perform extensive experiments on two downstream tasks (occupation prediction and coreference resolution); several debiasing strategies that involve alterations to the training dataset (such as removing names and gender indicators, or balancing the data by oversampling or downsampling); and a multitude of extrinsic bias metrics. Our analysis reveals new insights into the way language models encode and use information on gender:
- The effect of debiasing on internal representations is reflected in gender extractability, while not always in CEAT. Thus, gender extractability is a more reliable indicator of gender bias in NLP models.
- In cases of high gender extractability but low extrinsic bias metrics, the debiasing is superficial, and the internal representations are a good indicator for this: The bias is still present in internal representations and can be restored by retraining the classification layer. Therefore, our proposed measuring method can help in detecting such cases before deploying the model.
- The two tasks show different patterns of correlation between intrinsic and extrinsic bias.
The coreference task exhibits a high correlation. The occupation prediction task exhibits a lower correlation, but it increases after retraining (a case of superficial debiasing). Gender extractability shows higher correlations with extrinsic metrics than CEAT, increasing the confidence in this metric as a reliable measure for gender bias in NLP models.
In this study, we investigate the relationship between extrinsic bias metrics of a task and a model's internal representations, under various debiasing conditions, for two datasets in English. We perform extrinsic debiasing, evaluate various extrinsic and intrinsic bias metrics before and after debiasing, and examine correlations.
Dataset. Let  $D = \{X, Y, Z\}$  be a dataset consisting of input data  $X$ , labels  $Y$  and protected attributes  $Z$ . This work focuses on gender as the protected attribute  $z$ . In all definitions,  $F$  and  $M$  indicate female and male gender, respectively, as the value of the protected attribute  $z$ .
Trained Model. The model is optimized to solve the downstream task posed by the dataset. It can be formalized as  $f \circ g: X \to \mathbb{R}^{|\mathcal{Y}|}$ , where  $g(\cdot)$  is the feature extractor, implemented by a language model, e.g., RoBERTa (Liu et al., 2019),  $f(\cdot)$  is the classification function, and  $\mathcal{Y}$  is the set of the possible labels for the task.
2.1 Bias Metrics
Each bias evaluation method described in the literature can be categorized as extrinsic or intrinsic. In all definitions,  $\mathbf{r}$  indicates the model's output probabilities.
2.1.1 Extrinsic Metrics
Extrinsic methods involve measuring the bias of a model solving a downstream problem. The extrinsic metric is a function:
$$
E (\boldsymbol {X}, \boldsymbol {Y}, \boldsymbol {R}, \boldsymbol {Z}) \in \mathbb {R}
$$
The output represents the quantity of bias measured; the further from 0 the number is, the larger the bias is. Our analysis comprises a wide range
${}^{2}Z$  is by convention used for attributes for which we want to ensure fairness, such as gender, race, etc. It is purposefully broad, and depending on the task and data could refer to the gender of an entity in coreference, the subject of a text, the demographics of the author of a text, etc.
of extrinsic metrics, including some that have been measured in the past on the analyzed tasks (Zhao et al., 2018; De-Arteaga et al., 2019; Ravfogel et al., 2020; Goldfarb-Tarrant et al., 2021) and some that have never been measured before, and shows our results apply to many of them. For illustration, we will consider occupation prediction, a common task in research on gender bias (De-Arteaga et al., 2019; Ravfogel et al., 2020; Romanov et al., 2019). The input  $x$  is a biography and the prediction  $y$  is the profession of the person described in it. The protected attribute  $z$  is the gender of that person.
Performance gap. This is the difference in performance metric for two different groups, for instance two groups of binary genders, or a group of pro-stereotypical and a group of anti-stereotypical examples. We measure the following metrics: True Positive Rate (TPR), False Positive Rate (FPR), and Precision. In occupation prediction, for instance, the TPR gap for each profession  $y$  expresses the difference in the percentage of women and men whose profession is  $y$  and are correctly classified as such. We also measure F1 of three standard clustering metrics for coreference resolution. Each such performance gap captures a different facet of gender bias, and one might be more interested in one of the metrics depending on the application.
We compute two types of performance gap metrics: (1) the sum of absolute gap values over all classes; (2) the Pearson correlation between the performance gap for a class and the percentage of women in that class. For instance, if  $y$  is a profession, we measure the correlation between performance gaps and percentages of women in each profession. The two metrics are closely related but answer slightly different questions: the sum quantifies how a model behaves differently on different genders, and the correlation shows the relation of model behaviour to social biases (in the world or the data) without regard to actual gap size.
Statistical metrics. For breadth of analysis, we examine three additional statistical metrics (Barocas et al., 2019), which correspond to different notions of bias. All three are measured as differences  $(d)$  between two probability distributions, and we then obtain a single bias quantity per metric by summing all computed distances.
- Independence:  $d(P(\mathbf{r}|\mathbf{z} = z), P(\mathbf{r}))\forall z \in \{F, M\}$ . For instance, we measure the difference between the distribution of model's predictions on women and the distribution of all predictions. Independence is stronger as the prediction  $\mathbf{r}$  is less correlated with the protected attribute  $\mathbf{z}$ . It is measured with no relation to the gold labels.
- Separation:  $d\big(P(\mathbf{r}|\mathbf{y} = y, \mathbf{z} = z), P(\mathbf{r}|\mathbf{y} = y)\big) \forall y \in \mathcal{V}, z \in \{F, M\}$ . For instance, we measure the difference between the distribution of a model's predictions on women who are teachers and the distribution of predictions on all teachers. It encapsulates the TPR and FPR gaps discussed previously, and can be seen as a more general metric.
- Sufficiency:  $d\big(P(\mathbf{y}|\mathbf{r} = r, \mathbf{z} = z), P(\mathbf{y}|\mathbf{r} = r)\big)$ . For instance, we measure the difference between the distribution of gold labels on women classified as teachers by the model and the distribution of gold labels on all individuals classified as teachers by the model. Sufficiency relates to the concept of calibration in classification. A difference in the classifier's scores for men and for women indicates that it might be penalizing or over-promoting one of the genders.
2.1.2 Intrinsic Metrics
Intrinsic methods are applied to the representation obtained from the feature extractor. These methods are independent of any downstream task. The intrinsic metric is a function:
$$
I (g (\boldsymbol {X}), \boldsymbol {Z}) \in \mathbb {R}
$$
Compression. Our main intrinsic metric is the compression of gender information evaluated by a minimum description length (MDL) probing classifier (Voita and Titov, 2020), trained to predict gender from the model's representations. Probing classifiers are widely used for predicting various properties of interest from frozen model representations (Belinkov and Glass, 2019). MDL probes were proposed because a probe's accuracy may be misleading due to memorization and other issues (Hewitt and Liang, 2019; Belinkov, 2021). We use the MDL online code, where the probe is trained in timesteps, on increasing subsets of the training set, then evaluated against the rest of it. Higher compression indicates greater gender extractability.
CEAT. We also measure CEAT (Guo and Caliskan, 2021), which is a contextualized version
<sup>3</sup>Percentages for coreference resolution are taken from labour statistics, following Zhao et al. (2018). For occupation prediction we use training set statistics following De-Arteaga et al. (2019), before balancing.
of WEAT (Caliskan et al., 2017), a widely used bias metric for static word embeddings. WEAT defines sets  $\mathbb{X}$  and  $\mathbb{Y}$  of target words, and sets  $\mathbb{A}$  and  $\mathbb{B}$  of attribute words. For instance,  $\mathbb{A}$  and  $\mathbb{B}$  contain males and females names, while  $\mathbb{X}$  and  $\mathbb{Y}$  contain career and family related words, respectively. The bias is operationalized as the geometric proximity between the target and attribute word embeddings, and is quantified in CEAT by the Combined Effect Size (CES) and a p-value for the null hypothesis of having no biased associations. For more information on CEAT refer to Appendix A.4.3.
2.2 Debiasing Techniques
We debias models by modifying the downstream task's training data before fine-tuning. Scrubbing (De-Arteaga et al., 2019) removes first names and gender-specific terms ("he", "she", "husband", "wife", "Mr", "Mrs", etc.). Balancing subsamples or oversamples examples such that each gender is equally represented in the resulting dataset w.r.t each label. Anonymization (Zhao et al., 2018) removes named entities. Counterfactual Augmentation (Zhao et al., 2018) involves replacing male entities in an example with female entities, and adding the modified example to the training set. As some of these are dataset/task-specific, we give more details in the following section.
In each experiment, we fine-tune a model for a downstream task. For training, we use either the original dataset or a dataset debiased with one of the methods from Section 2.2. Figure 2 presents examples of debiasing methods for the two downstream tasks. We measure two intrinsic metrics by probing that model's inner representations for gender extractability (as measured by MDL) and by CEAT, and test various extrinsic metrics. The relation between one intrinsic and one extrinsic metric becomes one data point, and we repeat over many random seeds (for both the model and the probe). Further implementation details are in appendix A.
3.1 Occupation Prediction
The task of occupation prediction is to predict a person's occupations (from a closed set), based on their biography. We use the Bias in Bios dataset (De-Arteaga et al., 2019). Regardless of the training method, the test set is subsampled such that each profession has equal gender representation.
Figure 2: Examples of two debiasing methods performed on the data.
Model. Our main model is a RoBERTa model (Liu et al., 2019) topped with a linear classifier, which receives the [CLS] token embedding as input and generates a probability distribution over the professions. In addition, we experiment with training a baseline classifier layer on top of a frozen, non-finetuned RoBERTa. We also replicate our RoBERTa experiments with a DeBERTa model (He et al., 2020), to verify that our results are not model specific and hold more broadly.
Debiasing Techniques. Following De-Arteaga et al. (2019) we experiment with scrubbing the training dataset. Figure 2 shows an example biography snippet and its scrubbed version. We also conduct balancing (per profession, subsampling and oversampling to ensure an equal number of males and females per profession), which has not previously been used on this dataset and task.
Metrics. We measure all bias metrics from Section 2.1 except for F1.
Probing. The probing dataset for this task is the test set, and the gender label of a single biography is the gender of the person described in it. We probe the [CLS] token representation of the biography. In addition to the models described above, we measure baseline extractability of gender information from a randomly initialized RoBERTa model.
3.2 Coreference Resolution
The task of coreference resolution is to find all textual expressions referring to the same real-world entities. We train on Ontonotes 5.0 (Weischedel et al., 2013) and test on the Winobias challenge dataset (Zhao et al., 2018). Winobias consists of sentence pairs, pro- and anti-stereotypical variants, with individuals referred to by their profession. For example, "The physician hired the secretary be
Debiasing Strategy	Intrinsic	Extrinsic
Before	After
Compression	CEAT	TPR (P)	FPR (S)	Sep	Suff	TPR (P)	FPR (S)	Sep	Suff
Random	5.61*	0.12†	-	-	-	-	-	-	-	-
Pre-trained	10.12	0.49*	-	-	-	-	-	-	-	-
None	4.12	0.22	0.76	0.08	0.33	9.45	0.78	0.073	0.33	9.70
Oversampling	8.52*	0.29	0.73	0.09*	0.31	8.32*	0.81*	0.068*	0.33	10.91*
Subsampling	3.57	0.22	0.32*	0.03*	0.20*	1.22*	0.70*	0.08*	0.30*	1.32*
Scrubbing	1.70*	0.23	0.70*	0.06*	0.30	4.93*	0.71*	0.06*	2.56*	0.81*
(a) Occupation classification: Comparison of intrinsic and extrinsic metrics before and after retraining of classification layer, over 10 seeds per fine-tuned model and per retrained classification model.
Debiasing Strategy	Intrinsic	Extrinsic
Before	After
Compression	CEAT	F1 diff	FPR (S)	Sep	Suff	F1 diff	FPR (S)	Sep	Suff
Random	0.83*	0.12†	-	-	-	-	-	-	-	-
Pre-trained	0.96	0.49*	-	-	-	-	-	-	-	-
None	1.98	0.35	6.63	0.12	1.25	8.69	6.07	0.11	1.19	7.35
Anon	2.07*	0.31*	7.26	0.13	1.34	8.82	7.42*	0.13*	1.34*	8.66*
CA	1.50*	0.27*	2.30*	0.05*	0.54*	1.67*	3.67*	0.06*	0.67*	2.40*
Anon + CA	1.54*	0.25*	2.42*	0.049*	0.56*	1.56*	2.86*	0.05*	0.59*	1.65*
(b) Coreference resolution: Comparison of intrinsic and extrinsic metrics before and after retraining of classification layer, over 10 seeds per fine-tuned model and 5 seeds per retrained classification model.
Table 1: Results on both tasks. * marks significant reduction or increase in bias ( $p < 0.05$  on Pitman's permutation test), compared to the non-debiased model (debiasing strategy None). The lowest bias score in each column is marked with **bold. P = Pearson; S = Sum. † was computed only on 3 out of 10 tests for which CEAT's  $p < 0.05$ .
cause he/she was busy." is pro/anti-stereotypical, based on US labor statistics. A coreference system is measured by the performance gap between the pro- and anti-stereotypical subsets.
Model. We use the model presented in Lee et al. (2018a) with RoBERTa as a feature extractor.
Debiasing Techniques. Following Zhao et al. (2018), we apply anonymization (denoted as Anon) and counterfactual augmentation (CA) on the training set. These techniques were used jointly in previous work; we examine each individually as well.
Metrics. Following Zhao et al. (2018), we measure the F1 difference between anti- and stereotypical examples. We also interpret the task as a classification problem, and measure all metrics from Section 2.1. For more details refer to Appendix A.4.2.
Probing. We probe the representation of a profession word as extracted from Winobias sentences,
after masking out the pronouns. We define a profession's gender as the stereotypical gender for this profession. To prevent memorization by the probe—given the small number of professions—the dataset is sorted so that professions are gradually added to the training set, so a success on the validation set is on previously unseen professions.
Tables 1a and 1b present intrinsic and extrinsic metrics for RoBERTa models on the occupation prediction and coreference resolution tasks, respectively. We present a representative subset of the measured metrics that demonstrate the observed phenomena; full results are found in Appendix B. The DeBERTa model results are consistent with the RoBERTa model trends.
4.1 Compression Reflects Debiasing Effects
As shown in the tables, compression captures differences in models that were debiased differently. CEAT, however, cannot differentiate between occupation prediction models. For example, in occupation prediction (Table 1a) the compression rate
$^{4}$ Labor Force Statistics from the Current Population Survey, https://www.bls.gov/cps/cpsaat11.htm  $^{5}$ We combined the T1 and T2 datasets, as well as the dev and test datasets, to create a single held-out challenge dataset.
varies significantly between a non-debiased and a debiased model via scrubbing and oversampling, while CEAT detects no difference between the models. In coreference resolution (Table 1b), both compression and CEAT are able to identify differences between the non-debiased model and the others, such as CA, which has both a lower compression and CEAT effect. But the CEAT effect sizes are small (below 0.5), which implies no bias, in contrast to the extrinsic metrics.
4.2 High Gender Extractability Implies Superficial Debiasing
In occupation classification (Table 1a), somewhat surprisingly, subsampling the training data has the strongest effect on extrinsic metrics, but not on compression rate. Scrubbing reduces both intrinsic and extrinsic metrics, although its effect on extrinsic metrics is limited compared to subsampling. Training with oversampling caused less reduction in extrinsic bias metrics. A consequence of oversampling is that some metrics are less biased, but compression rates are increased, so gender information is more accessible. The effectiveness of subsampling over other metrics is further discussed in appendix C. In coreference resolution (Table 1b), while both CA and CA with anonymization reduced gender extractability as well as external bias metrics, anonymization alone increased intrinsic bias without affecting external bias metrics significantly.
Debiasing without fine-tuning. As the effect on extrinsic bias did not match the effect on intrinsic bias in several cases, we examined the role of the classification layer. We trained a model for occupation prediction without fine-tuning the underlying RoBERTa model. Training on a subsampled dataset also reduced the extrinsic metrics (0.15, 0.03, 0.20, and 0.31, respectively, on TPR gaps Pearson, FPR gaps sum, separation sum, and sufficiency sum). Detailed results of this experiment can be found in Appendix B. Since no updates were made to the LM, the internal representations could not be debiased, thus the debiasing observed in this model can only be superficial.
Retraining the classification layer. Fine-tuning of both tasks revealed that lower extrinsic metrics did not always lead to lower compression. Does this indicate cases where the debiasing process is only superficial, and the internal representations remain biased? To test this hypothesis, we froze the
previously fine-tuned LM's weights, and retrained the classification layer. We used the original (nondebiased) training set for retraining.
Tables 1a and 1b also compare extrinsic metrics before and after retraining. All models show bias restoration, due to the classification layer being trained on the biased dataset. $^{6}$  The amount of bias restored varies between models in a way that is predictable by the compression metric.
In the occupation prediction task, comparing Before and After numbers in Table 1a, the model fine-tuned using a scrubbed dataset—which has the lowest compression rate—displays the least bias restoration, confirming that the LM absorbed the process of debiasing. The model fine-tuned on subsampled data has higher extrinsic bias after retraining. Hence, the debiasing was primarily cosmetic, and the representations within the LM were not debiased. The model fine-tuned on oversampled data—which has the highest compression—has the highest extrinsic bias (except for FPR), even though this was not true before retraining.