SlowGuess/NAACL_2022_OCR huggingface dataset:
pages 1679-1684, Florence, Italy. Association for Computational Linguistics.
Jiao Sun and Nanyun Peng. 2021. Men are elected, women are married: Events gender bias on wikipedia.
Harini Suresh and John Guttag. 2021. A framework for understanding sources of harm throughout the machine learning life cycle. In Equity and Access in Algorithms, Mechanisms, and Optimization, EAAMO '21, New York, NY, USA. Association for Computing Machinery.
Yi Chern Tan and L. Elisa Celis. 2019. Assessing social and intersectional biases in contextualized word representations. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 183-196, Online. Association for Computational Linguistics.
Claudia Wagner, Eduardo Graells-Garrido, David Garcia, and Filippo Menczer. 2016. Women through the glass ceiling: gender asymmetries in wikipedia. EPJ Data Science, 5:1-24.
Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605-617.
R. Weischedel, E. Hovy, M. Marcus, and Martha Palmer. 2013. Ontonotes: A large training corpus for enhanced processing. LDC2013T19, Philadelphia, Penn.: Linguistic Data Consortium.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Liyan Xu and Jinho D. Choi. 2020. Revealing the myth of higher-order inference in coreference resolution. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8527-8533. Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 629-634, Minneapolis, Minnesota. Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15–20, New Orleans, Louisiana. Association for Computational Linguistics.
We used RoBERTa in all models (base size, 120M parameters). We use following random seeds in all repeated experiments: 0, 5, 11, 26, 42, 46, 50, 63, 83, 90. Our code was implemented mainly using the Python libraries Pytorch (Paszke et al., 2019), Transformers (Wolf et al., 2020), Sklearn (Pedregosa et al., 2011), and the experiments were logged using Wandb (Biewald, 2020).
A.1 Occupation Classification
We fine-tuned a RoBERTa-base model with a linear classification layer on top. Training was done for 10 epochs at a learning rate of 5e-5, batch size of 64. The input to RoBERTa was the biography tokens, which is limited to the first 128 tokens. The resulting [CLS] token embedding is fed to the classifier to predict the occupation. The probing task involves using the same [CLS] token and training the probing classifier to predict the gender of the person in the biography. The experiments without fine-tuning included either a pre-trained or a previously fine-tuned RoBERTa. We first extracted the pre-trained RoBERTa's embeddings of tokens from the [CLS] and then trained a linear classifier on them. The learning rate was 0.001 and the batch size was 64. We trained the classification layer with pre-trained RoBERTa on 300 epochs, but with fine-tuned RoBERTa, 10 epochs were sufficient. For all training processes, the epoch with the greatest validation accuracy was saved. Fine-tuning took 7 hours on a GeForce RTX 2080 Ti GPU. Bias in Bios contains almost 400k biographies, and we obtain validation (10%) and test set (25%) by splitting with Scikit-learn's (Pedregosa et al., 2011) test_train_split with our random seeds.
A.2 Coreference Resolution
We use the implementation of Xu and Choi (2020), a model that was introduced by Lee et al. (2018b) and has been adopted by many coreference resolution models. Coreference resolution is the process of clustering different mentions in a text that refers to the same real-world entities. The task is solved by detecting mentions through text spans and then predicting for each pair of spans if they represent the same entity. The span representations were extracted with a RoBERTa model, which is fine-tuned throughout the training process, except in the retraining experiment. Fine-tuning took 3 hours on an NVIDIA RTX A6000 GPU. Ontonotes 5.0 has
625k sentences and we use the standard validation and test splits.
A.3 Probing Classifier
We use the MDL probe (Voita and Titov, 2020) implementation by Mendelson and Belinkov (2021b). In all experiments, we use a linear probe and train it with a batch size of 16 and a learning rate of 1e-3. The timestamps used, meaning the accumulating fractions of data that the probe is trained on, are  $2.0\%$ ,  $3.0\%$ ,  $4.4\%$ ,  $6.5\%$ ,  $9.5\%$ ,  $14.0\%$ ,  $21.0\%$ ,  $31.0\%$ ,  $45.7\%$ ,  $67.6\%$ ,  $100\%$ .
A.4 Metrics
A.4.1 Fairness-Based Metrics Implementation
All three statistical fairness metrics measure the difference between two probability distributions, where this difference describes a notion of bias. We calculate Independence and Separation via Kullback-Leibler (KL) divergence, using the AllenNLP implementation (https://github.com/allenai/allennlp). We calculate Sufficiency via Wasserstein distance instead, which is motivated by Kwegyir-Aggrey et al. (2021). In this case, we cannot use KL divergence, since there are some classes that do not occur in model predictions for both male and female genders. This causes the probability distributions to not have the same support, and KL divergence is unbounded. Wasserstein distance lacks the requirement for equal support.
A.4.2 Classification Metrics Interpretation in Winobias
Winobias datasets contain pairs of stereotypical and anti-stereotypical sentences. The stereotypes are derived from the US labor statistics (for instance, a profession with a majority of males is stereotypically male). Since coreference resolution is viewed as a clustering problem, it is usually measured via clustering evaluation metrics. Coreference resolution is commonly measured as the average F1 score of these, and the same is true for Winobias. Nevertheless, coreference resolution is accomplished by making a prediction for each pair of mentions, so it can be seen as a classification task. Winobias can be viewed as a simpler task than general coreference resolution, as it contains exactly two mentions of professions and one pronoun, which refers to exactly one profession. Therefore, we reframe it as a classification problem. In a Winobias sentence with two professions  $x$  and  $y$ , as well as a pronoun  $p$ , where  $p$  is referring to  $x$ , a true positive
would be to cluster  $x$  and  $p$  together, while a false positive would be to cluster  $y$  and  $p$  together. Our classification metrics are derived based on these definitions. For instance, the TPR gap for profession "teacher", which is a stereotypical female occupation, is the TPR rate on pro-stereotypical sentences (with a female pronoun) minus the TPR rate on anti-stereotypical sentences (with a male pronoun).
A.4.3 CEAT
The Word Embedding Association Test (WEAT) developed by (Caliskan et al., 2017) is a method for evaluating bias in static word embeddings. The test is defined as follows: given two sets of target words  $\mathbb{X},\mathbb{Y}$  (e.g., 'executive', 'management', 'professional' and 'home', 'parents', 'children') and two sets of attribute words (e.g., male names and female names), and using  $\vec{w}$  to represent the word embedding for word  $w$ , the effect size is:
$$
\mathrm {E S} = \operatorname {m e a n} _ {x \in \mathbb {X}} s (x, \mathbb {A}, \mathbb {B}) - \operatorname {m e a n} _ {y \in \mathbb {Y}} s (y, \mathbb {A}, \mathbb {B})
$$
$$
\begin{array}{l} s (x, \mathbb {A}, \mathbb {B}) = \\ \frac {\operatorname {m e a n} _ {a \in \mathbb {A}} \cos (\vec {x} , \vec {a}) - \operatorname {m e a n} _ {a \in \mathbb {A}} \cos (\vec {x} , \vec {b})}{\operatorname {s t d - d e v} _ {w \in \mathbb {X} \bigcup \mathbb {Y}} s (w , \mathbb {A} , \mathbb {B})} \\ \end{array}
$$
In essence, the effect size measures how different are the distances between the embedding vectors of each target group and the attribute groups. Specifically, if  $s(x,\mathbb{A},\mathbb{B}) > 0$ ,  $\vec{x}$  is more similar to attribute words  $\mathbb{B}$  and vice versa. For instance, a larger effect size is observed if target words  $\mathbb{X}$  are more similar to attribute words  $\mathbb{A}$  and target words  $\mathbb{Y}$  are more similar to attribute words  $\mathbb{B}$ .  $|ES| > 0.5$  and  $|ES| > 0.8$  are considered medium and large effect sizes, respectively (Rice and Harris, 2005). The null hypothesis holds that there is no difference between the two sets of target words in terms of their relative similarity to the two sets of attribute words, indicating that there are no biased associations. Statistical significance is defined by the p-value of WEAT, which reflects the probability of observing the effect size under the null hypothesis.
Since a word can take on a great variety of vector representations in a contextual setting,  $ES$  varies according to the sentences used to extract word representation. Thus, to adopt WEAT to contextualized representations, the Combined Effect Size (CES) (Guo and Caliskan, 2021) is derived as the
distribution of WEAT effect sizes over many possible contextual word representations:
$$
\operatorname {C E S} (\mathbb {X}, \mathbb {Y}, \mathbb {A}, \mathbb {B}) = \frac {\sum_ {i = 1} ^ {N} v _ {i} E S _ {i}}{\sum_ {i = 1} ^ {N} v _ {i}}
$$
where  $ES_{i}$  denotes the WEAT effect size of the  $i$ 'th choice of word representations from a large corpus, and  $v_{i}$  is the inverse of the sum of in-sample variance  $V_{i}$  and between-sample variance in the distribution of random-effects. As in Guo and Caliskan (2021), the representation for each word is derived from 10,000 random sentences extracted from a corpus of Reddit comments.
The combined effect size of each of the models is examined on WEAT stimulus 6, which contains target words of career/family and attribute words of male/female names. This was the only one that detected bias on a pre-trained RoBERTa (CES close to 0.5 and  $p < 0.05$ ). The points that we kept in our analysis are those where  $p < 0.05$ , which make up  $90\%$  of the points in occupation prediction and  $95\%$  of the points in coreference resolution.
B Full Results
In this section we provide the full results of a RoBERTa model trained on the downstream task.
Table 3 presents results for the occupation prediction task after fine-tuning, Table 4 presents the retrained model results.
Figure 5 illustrates the correlations between extrinsic metrics and compression rate before and after retraining.
Table 5 presents the complete results for the occupation prediction task of the model trained without fine-tuning, meaning that the RoBERTa model is the pretrained version from Liu et al. (2019) and only the classification layer was updated. Subsampling the dataset has significant debiasing effects, which suggests that this debiasing method can achieve low extrinsic bias even when internal bias exists. The Pearson correlation on precision exhibits a different behavior. It makes sense nonetheless: precision is computed as  $TP \backslash (TP + FP)$ . A biased model will assign more examples of a specific profession to a specific gender (which aligns with the percentage of biographies of this profession with this gender on the training set), increasing both  $TP$  and  $FP$  and decreasing precision. The results on the coreference resolution task align with the results of occupation prediction.
Table 6 presents the results using a DeBERTa model (He et al., 2020) for the occupation classification task. The trends are similar to those of RoBERTa, with the same metrics showing an increase, no change, or decrease in correlation after re-training, suggesting a general trend in the behavior of these metrics in relation to internal model representations.
Table 7 displays the results on a finetuned model for the coreference resolution task and Table 8 displays the retraining results.
Figure 6 shows the correlations between compression rate and extrinsic metrics before and after the retraining.
Metric	Debiasing Strategy
None	Oversampling	Subsampling	Scrubbing
Compression	4.121 ± 1.238	8.522* ± 2.354	3.568 ± 1.516	1.699* ± 0.138
Accuracy	0.861 ± 0.005	0.852* ± 0.004	0.861 ± 0.003	0.851* ± 0.003
TPR gap (P)	0.763 ± 0.071	0.729 ± 0.067	0.319* ± 0.114	0.704* ± 0.068
TPR gap (S)	2.391 ± 0.257	2.145* ± 0.220	1.598* ± 0.273	2.019* ± 0.262
FPR gap (P)	0.591 ± 0.052	0.491* ± 0.059	0.087* ± 0.094	0.552 ± 0.063
FPR gap (S)	0.075 ± 0.010	0.085* ± 0.011	0.030* ± 0.006	0.057* ± 0.007
Precision gap (P)	-0.880 ± 0.031	-0.855 ± 0.115	-0.299* ± 0.215	-0.815* ± 0.040
Precision gap (S)	3.621 ± 0.337	3.401 ± 0.667	1.549* ± 0.229	2.590* ± 0.279
Independence gap (S)	0.009 ± 0.002	0.008 ± 0.002	0.001* ± 0.000	0.005* ± 0.001
Separation gap (S)	0.327 ± 0.051	0.305 ± 0.030	0.204* ± 0.032	0.296 ± 0.053
Sufficiency gap (S)	9.451 ± 1.945	8.324* ± 1.537	1.218* ± 0.330	4.930* ± 0.927
Table 3: Occupation Prediction: Results on a RoBERTa-based model trained over 10 seeds. Significant reduction or increase in a metric ( $p < 0.05$  on Pitman's permutation test), compared to the non-debiased model (debiasing strategy is None), is marked with *. The lowest bias score or highest performance metric in each column is marked with bold.  $\mathrm{P} =$  Pearson;  $\mathrm{S} =$  Sum.
Metric	Debiasing Strategy
None	Oversampling	Subsampling	Scrubbing
Compression	4.121 ± 1.238	8.522 ± 2.354	3.568 ± 1.516	1.699 ± 0.138
Accuracy	0.859 ± 0.004	0.856 ± 0.003	0.853 ± 0.003	0.854 ± 0.003
TPR gap (P)	0.777 ± 0.047	0.813* ± 0.040	0.704* ± 0.075	0.714* ± 0.068
TPR gap (S)	2.482 ± 0.238	2.593* ± 0.240	2.164* ± 0.284	1.989* ± 0.227
FPR gap (P)	0.596 ± 0.041	0.603 ± 0.047	0.602 ± 0.041	0.536* ± 0.038
FPR gap (S)	0.073 ± 0.008	0.068* ± 0.007	0.081* ± 0.012	0.059* ± 0.005
Precision gap (P)	-0.877 ± 0.027	-0.891* ± 0.023	-0.889* ± 0.035	-0.817* ± 0.058
Precision gap (S)	3.710 ± 0.251	3.996* ± 0.272	3.555* ± 0.598	2.703* ± 0.255
Independence gap (S)	0.009 ± 0.002	0.010* ± 0.002	0.009 ± 0.003	0.005* ± 0.001
Separation gap (S)	0.334 ± 0.050	0.328 ± 0.048	0.300* ± 0.049	0.274* ± 0.041
Sufficiency gap (S)	9.701 ± 1.305	10.908* ± 1.354	8.370* ± 2.558	5.239* ± 0.798
Table 4: Occupation Prediction after retraining: Results on a RoBERTa-based model after retraining of the classification layer with 10 seeds for each pre-trained model. Significant reduction or increase in a metric ( $p < 0.05$  on Pitman's permutation test), compared to the non-debiased model (debiasing strategy is None), is marked with * The lowest bias score or highest performance metric in each column is marked with **bold. P = Pearson; S = Sum.
Metric	Debiasing Strategy
None	Oversampling	Subsampling	Scrubbing
Accuracy	0.824 ± 0.003	0.815* ± 0.005	0.831* ± 0.001	0.807* ± 0.003
TPR gap (P)	0.839 ± 0.011	0.443* ± 0.053	0.158* ± 0.156	0.814 ± 0.029
TPR gap (S)	3.088 ± 0.192	1.545* ± 0.177	1.621* ± 0.088	3.154 ± 0.332
FPR gap (P)	0.598 ± 0.016	0.369* ± 0.029	0.067* ± 0.050	0.550* ± 0.012
FPR gap (S)	0.087 ± 0.004	0.041* ± 0.004	0.027* ± 0.003	0.112* ± 0.005
Precision gap (P)	-0.872 ± 0.028	-0.427* ± 0.074	-0.161* ± 0.162	-0.853 ± 0.019
Precision gap (S)	3.811 ± 0.253	1.736* ± 0.108	1.551* ± 0.195	3.907 ± 0.184
Independence gap (S)	0.014* ± 0.002	0.001* ± 0.000	0.000* ± 0.000	0.022* ± 0.001
Separation gap (S)	0.336* ± 0.044	0.214* ± 0.038	0.203* ± 0.024	0.432* ± 0.048
Sufficiency gap (S)	12.019* ± 1.721	2.105* ± 0.576	1.478* ± 0.394	13.798* ± 0.966
Table 5: Occupation Prediction: Results on a RoBERTa-based model trained without fine-tuning, over 5 seeds. The compression rate computed on a pre-trained RoBERTa model is 10.122. Significant reduction or increase in a metric ( $p < 0.05$  on Pitman's permutation test), compared to the non-debiased model (debiasing strategy is None), is marked with *. The lowest bias score or highest performance metric in each column is marked with **bold. P = Pearson; S = Sum.
Metric	R2Compression	R2CEAT
Before	After	Before	After
TPR gap (P)	0.023	0.120	0.051	0.006
TPR gap (S)	0.000	0.200	0.036	0.098
FPR gap (P)	0.083	0.153	0.121	0.149
FPR gap (S)	0.055	0.013	0.009	0.021
Precision gap (P)	0.002	0.135	0.046	0.031
Precision gap (S)	0.024	0.362	0.026	0.103
Independence gap (S)	0.034	0.084	0.0	0.054
Separation gap (S)	0.000	0.117	0.008	0.009
Sufficiency gap (S)	0.016	0.250	0.046	0.042
Table 6: Results for a DeBERTa model trained on occupation prediction task. Coefficient determination of the regression line taken on the compression rate or CEAT and each extrinsic metric, before and after retraining of the classification layer.  $\mathrm{P} =$  Pearson;  $\mathrm{S} =$  Sum. Coefficients are of lower magnitude for DeBERTa than RoBERTa models, but the same trends apply. They largely increase after retraining (save for FPR gap, and a few of the very low magnitude Pearson metrics). The increase after retraining does not apply to CEAT, and the correlations with CEAT are usually lower.
Figure 5: Occupation prediction: Before (left) and after (right) plots of compression rate versus and extrinsic metric. Cases of low correlation are discussed in D.1.
Metric	Debiasing Strategy
None	Anon	CA	Anon + CA
Compression	1.984 ± 0.101	2.073* ± 0.102	1.502* ± 0.075	1.540* ± 0.098
F1 (Onotonotes test)	76.406 ± 0.165	76.538 ± 0.176	77.187* ± 0.071	77.246* ± 0.230
F1 diff (pro - anti)	6.631 ± 1.013	7.256 ± 0.846	2.302* ± 0.466	2.422* ± 0.714
TPR gap (P)	0.654 ± 0.069	0.710* ± 0.047	0.607 ± 0.082	0.627 ± 0.100
TPR gap (S)	4.884 ± 0.698	4.870 ± 0.509	2.041* ± 0.228	2.014* ± 0.286
FPR gap (P)	0.602 ± 0.036	0.620 ± 0.056	0.572 ± 0.078	0.629 ± 0.107
FPR gap (S)	0.120 ± 0.015	0.128 ± 0.011	0.050* ± 0.006	0.049* ± 0.007
Precision gap (P)	-0.549 ± 0.051	-0.571 ± 0.052	-0.491* ± 0.081	-0.569 ± 0.122
Precision gap (S)	3.080 ± 0.275	3.266 ± 0.264	1.421* ± 0.181	1.390* ± 0.216
Independence gap (S)	0.027 ± 0.008	0.025 ± 0.004	0.004* ± 0.001	0.004* ± 0.001
Separation gap (S)	1.247 ± 0.150	1.344 ± 0.137	0.537* ± 0.061	0.557* ± 0.070
Sufficiency gap (S)	8.684 ± 1.883	8.816 ± 1.544	1.673* ± 0.354	1.557* ± 0.384
Table 7: Coreference resolution: results on Ontonotes test set and Winobias challenge set. Each model was trained over 10 seeds. * Marks significant reduction or increase in bias ( $p < 0.05$  on Pitman's permutation test), compared to the non-debiased model (debiasing strategy None). The lowest bias score or highest performance metric in each column is in **bold. P = Pearson; S = Sum.
Metric	Debiasing Strategy
None	Anon	CA	Anon + CA
Compression	1.984 ± 0.065	2.073* ± 0.104	1.502* ± 0.081	1.540* ± 0.079
F1 (Onotonotes test)	76.40* ± 0.16	76.48* ± 0.22	76.72* ± 0.15	76.91* ± 0.19
F1 diff (pro - anti)	6.072 ± 0.789	7.417* ± 1.280	3.674* ± 0.599	2.858* ± 0.382
TPR gap (P)	0.635 ± 0.053	0.688* ± 0.052	0.679* ± 0.062	0.654 ± 0.049
TPR gap (S)	4.561 ± 0.414	5.143* ± 0.713	2.590* ± 0.420	2.178* ± 0.201
FPR gap (P)	0.579 ± 0.046	0.637* ± 0.055	0.620* ± 0.070	0.692* ± 0.075
FPR gap (S)	0.113 ± 0.011	0.126* ± 0.016	0.063* ± 0.010	0.052* ± 0.004
Precision gap (P)	-0.512 ± 0.060	-0.581* ± 0.057	-0.550* ± 0.083	-0.632* ± 0.098
Precision gap (S)	2.943 ± 0.215	3.221* ± 0.384	1.690* ± 0.242	1.446* ± 0.146
Independence gap (S)	0.022 ± 0.003	0.026* ± 0.006	0.006* ± 0.002	0.004* ± 0.001
Separation gap (S)	1.188 ± 0.114	1.336* ± 0.175	0.670* ± 0.111	0.594* ± 0.057
Sufficiency gap (S)	7.350 ± 0.914	8.655* ± 1.726	0.2.401* ± 0.610	1.653* ± 0.294
Table 8: Coreference resolution after retraining: results on Ontonotes test set and extrinsic bias metrics on Winobias challenge set. Each model finetuned over 10 seeds and re-trained over 5 seeds. * Marks significant reduction or increase in bias ( $p < 0.05$  on Pitman's permutation test), compared to the non-debiased model (debiasing strategy None). The lowest bias score or highest performance metric in each column is in **bold. P = Pearson; S = Sum.
Figure 6: Coreference resolution: Before (left) and after (right) plots of compression rate versus and extrinsic metric. Cases of low and no correlation with the Pearson metrics are discussed in D.2.
Figure 6: Coreference resolution: Before (left) and after (right) plots of compression rate versus and extrinsic metric.
Female
Words	Male
Words
husband, women,	chief, companies
gender, listed,	computer,
practices, nurse,	applications,
specializes,	md, accepts,
children,	known, doctors,
ba, child,	npi, sports,
reading, families,	philosoph’,
location, place,	problems, rating,
affiliated, family,	no, systems,
experiences,	theory, practicing,
spanish,	software,
love, justice	security, major
Table 9: Top 20 significant words used to predict gender on all biographies, as obtained from a logistic regression model trained on predicting the gender of a person described in a biography. The words are sorted by importance.
Female
Words	Male
Words
husband , women, midwife , providing book , includes, joining, faculty	holds , emergency, vanderbilt, forces, registered, mental, assistant, president
Table 10: Top 8 words used to predict gender of female and male nurses, as obtained from a logistic regression model trained on predicting the gender of a person described in a biography. The words are sorted by importance.
C Why is scrubbing not as effective as subsampling?
The debiasing method of subsampling significantly reduced external biases in the occupation prediction task. Although compression rates show that scrubbing reduced more gender information, subsampling outperforms it as a debiasing method. We find that in spite of the scrubbing, a probe is able to correctly identify the gender from an internal representation with  $68.8\%$  accuracy compared to  $90.7\%$  on the original, non-scrubbed data. This means that although the scrubbing process reduces extrinsic bias significantly, gender information is still embedded in the [CLS] token embeddings.
To investigate the source of gender information after scrubbing, we use logistic regression (LR)
model to predict the gender from the Bag-of-Words of the scrubbed biographies. We perform an iterative process for automatic extra scrubbing: in each iteration we (1) train a LR model for gender prediction (2) scrub the n most significant words for each gender according to the LR weights. The most relevant words among 5 seeds of training with  $n = 10$  words scrubbed per iteration are displayed in Table 9. The model learns indirect correlations to gender in the absence of explicit gendered words. Because the significant words are related to male- or female-dominated professions, we conducted the process on a specific profession. Table 10 presents the most significant words for biographies of nurses. There are differences in wording even between females and males in the same profession. The results of this study are in line with the results of other studies that have been conducted on the way biographies are written for men and women (Wagner et al., 2016; Sun and Peng, 2021).