Measuring Social Biases in Masked Language Models by Proxy of Prediction Quality
Authors: Rahul Zalkikar, Kanchan Chandra
Accepted at: acl25

Measuring Social Biases in Masked Language Models by Proxy of Prediction Quality

Rahul Zalkikar*
Independent Researcher
rayzck9@gmail.com

Kanchan Chandra
New York University
kc76@nyu.edu

Abstract

Innovative transformer-based language models produce contextually-aware token embeddings and have achieved state-of-the-art performance for a variety of natural language tasks, but have been shown to encode unwanted biases for downstream applications. In this paper, we evaluate the social biases encoded by transformers trained with the masked language modeling objective using proposed proxy functions within an iterative masking experiment to measure the quality of transformer models’ predictions and assess the preference of MLMs towards disadvantaged and advantaged groups. We find that all models encode concerning social biases. We compare bias estimations with those produced by other evaluation methods using benchmark datasets and assess their alignment with human annotated biases. We extend previous work by evaluating social biases introduced after retraining an MLM under the masked language modeling objective and find proposed measures produce more accurate and sensitive estimations of biases based on relative preference for biased sentences between models, while other methods tend to underestimate biases after retraining on sentences biased towards disadvantaged groups.

Warning: This paper contains explicit statements of biased stereotypes and may be upsetting.

1 Introduction

Word embeddings have proven useful in a variety of Natural Language Processing (NLP) tasks due to their ability to efficiently model complex semantic and syntactic word relationships. Token-level embeddings, such as those produced by Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) algorithms, learn a non-contextualized text representation and produce static word embeddings that can uncover linear semantic or syntactic relationships between tokens.

Masked language models (MLM(s); Brown et al., 2020; Devlin et al., 2019; Liu et al., 2019a; Radford et al., 2019) such as transformers BERT and RoBERTa incorporate bidirectionality and self-attention, producing contextually-aware embeddings. Unlike BERTunc, RoBERTa was pretrained solely on the masked language modeling objective (MLMO) on a larger corpus of text. RoBERTa uses a dynamic masking strategy for a diverse set of representations during training and has achieved state-of-the-art results on GLUE, RACE, and SQuAD (Liu et al., 2019b; Rajpurkar et al., 2016; Lai et al., 2017; Wang et al., 2018). Distilled model variants have been shown to train significantly faster for minor decreases in performance (Sanh et al., 2019). MLMs have produced state-of-the-art results for masked language modeling, named entity recognition, and intent or topic classification, but also encode concerning social biases against disadvantaged groups that are undesirable in production settings. As MLMs become increasingly prevalent, researchers have been working on methods to measure biases embedded in these models (Nangia et al., 2020; Kaneko and Bollegala, 2022; Salutari et al., 2023).

To address the issue of social biases in MLMs at its source, we measure biases of MLMs while focusing on an MLM’s key pretraining objective: masked language modeling. In this work, we focus on proposing and assessing bias evaluation measures for MLMs and not proposing methods for debiasing MLMs. However, when assessing evaluation measures, we consider prior research that involves retraining or fine-tuning under the MLMO with debiased or counterfactual data to reduce biases in MLMs, such as Zhao et al., 2018 and Zhao et al., 2019 which use data augmentation to swap gendered words with their opposites prior to retraining. Motivated by this, we assess whether proposed measures satisfy an important criterion for improvement over previously proposed ones: align-
ment with biases introduced by MLM retraining under the MLMO (Section 4.5.1).

We represent MLM bias through a model’s relative preference for ground truth tokens between two paired sentences along a social bias axis. In each sentence pair, one contains bias against disadvantaged groups (stereotypical) and the other contains bias against advantaged groups (anti-stereotypical). We assess the relative preferences of MLMs towards sentences biased against disadvantaged and advantaged groups using proxy measures for prediction quality. In particular, we quantify MLM preference by proxy of masked token prediction quality given unmasked token context between encoded sentences within pairs.

To measure the preference of an MLM using the (attention-weighted) quality of its predictions under the MLMO, we propose and validate proxy functions \( \Delta_{PA} \) (Equation 7; Probability Difference with Attention) and CRRA (Equation 6; Complementary Reciprocal Rank with Attention) and introduce a modified \( \Delta P \) (Equation 3) to measure the likelihood an MLM will select a ground truth token to replace a masked one, and extend these definitions for a sentence. We apply per-model indicator function BSPT (Equation 9) to estimate the encoded social biases in pretrained MLMs. Our approach differs from prior research in measuring biases in MLMs by using attention-weights under an Iterative Masking Experiment (IME; Section 4) to probe MLM preferences.

We compare pre- and retrained MLMs within the same model class to recover the nature of biases introduced by retraining. In particular, we define and apply a proxy for the relative preference between two MLMs with model-comparative indicator function BSRT (Bias Score for MLM Re-training; Equation 10) to estimate social biases introduced by retraining MLMs under the MLMO. These “introduced” biases must be represented by the relative change in biases after retraining an MLM under the MLMO.

In summary, the primary contributions of this work are as follows:

• We explore MLM bias through a model’s relative preference for ground truth tokens between two minimally distant sentences with contrasting social bias under the IME. We measure this using the attention-weighted quality of predictions.

• We propose the model-comparative indicator function BSRT to estimate social biases against disadvantaged groups for a retrained MLM relative to its pretrained base, and assess bias evaluation measures for alignment with biases introduced by MLM retraining under the MLMO.

• We evaluate social biases for four transformer models available through the Transformers library (Wolf et al., 2020). We use proxy measures for MLM prediction quality with model-comparative function BSRT to estimate social biases introduced by retraining MLMs under the MLMO. We find that proposed measures \( \Delta_{PA} \) and CRRA, along with modified \( \Delta P \) and CRR, produce more accurate estimations of biases introduced by MLM retraining than previously proposed ones, which can produce concerning underestimations of biases after retraining MLMs on sentences biased against disadvantaged groups. We observe that proposed measures \( \Delta_{PA} \) and CRRA, along with modified \( \Delta P \), show greater sensitivity than CRR, CSPS, AUL, and AULA to relative changes in MLM bias due to retraining, indicated by larger and smaller bias scores and more frequently significant relative difference in proportions of bias between pre- and retrained transformers.

Our methodology could help others evaluate social biases encoded in an MLM after it is retrained on the MLMO, such as for any downstream fill-mask task, or after it is fine-tuned for other objectives that alter weights and change MLMO performance. We release a package for measuring biases in MLMs to enable bias score computation on user-supplied or benchmark datasets and easy integration into existing evaluation pipelines.

2 Related Work

2.1 Biases in Static Word Embeddings

Static word embeddings (Pennington et al., 2014; Mikolov et al., 2013) can be shifted in a direction to decompose bias embedded in learned text data representations. These could be analogies or biases along an axis, such as gender\(^1\) (Bolukbasi et al., 2016) or race (Manzini et al., 2019). The Word Embedding Association Test (WEAT; Caliskan et al., 2017) measures association between targets and

\(^1\)For example, doctor - man + woman = nurse.
attributes using cosine similarity between static word embeddings, but has been shown to overestimate biases by Ethayarajh et al., 2019, which proposed the robust relational inner product association (RIPA) method, derived from the subspace projection method to debias vectors in Bolukbasi et al., 2016.

While WEAT has shown that token-level embeddings produced by GloVe and Word2Vec encode biases based on gender and race (Caliskan et al., 2017), the Sentence Encoder Association Test (SEAT; May et al., 2019) extends on WEAT to measure social biases in sentence-level encoders such as ELMo and BERT using template sentences with masked target tokens2, averaging token embeddings to form sentence-level embeddings on which cosine similarity is applied as a proxy for semantic association. As an alternative evaluation method under a different objective, Liang et al., 2020 assessed differences in log-likelihood between gender pronouns in a template sentence3 where occupations can uncover the directionality of the bias encoded by an MLM.

2.2 Evaluating Biases in MLMs

When considering a sentence s containing tokens \( \{t_1, t_2, ..., t_{l_s}\} \), where \( l_s \) is the number of tokens in s, (modified) token(s) of s can characterize its bias towards either disadvantaged or advantaged groups. For a given sentence s with tokens \( t \in s \) we denote all tokens besides \( t_x \) as \( s_{\setminus t_x} \) (where \( 1 \leq x \leq l_s \)), and we denote modified tokens as M and unmodified tokens as U (\( s = U \cup M \)). For a given MLM with parameters \( \theta \), we denote a masked token as \( t_m \) and a predicted token as \( t_p \).

Salazar et al., 2020 uses pseudo-log-likelihood scoring to approximate \( P(U|M,\theta) \), or the probability of unmodified tokens conditioned on modified ones. Similarly, Nangia et al., 2020 reports CrowSPairs Scores (CSPS; Appendix B), a pseudo-log-likelihood score for an MLM selecting unmodified tokens given modified ones. Nadeem et al., 2021 reports a StereoSet Score (SSS; Appendix C), a pseudo-log-likelihood score for an MLM selecting modified tokens given unmodified ones.

Salutari et al., 2023 tests MLMs in an iterative fill mask setting where the model outputs a set of tokens (or the (log)softmax of model logits mapped to tokens) to fill the masked one, starting with the token of highest probability \( P(t_p|c) \) and, as such, first rank \( \rho(t_p|c) = 1 \) in the set of model token predictions (which is limited by the MLM’s embedding space).

\[
\Delta p(t|s_{\setminus t_m}; \theta) = P(t_p|s_{\setminus t_m}; \theta) - P(t_m|s_{\setminus t_m}; \theta) \\
= \Delta p(w; \theta)
\] (1)

\( \Delta p(t|s_{\setminus t_m}; \theta) \) (Equation 1) represents the difference between the probability of a predicted token \( t_p \) and a masked token \( t_m \) in a sentence s. It serves as a proxy of the MLM’s prediction quality for a token given its context within the IME, or all tokens in s besides \( t_m \) (Salutari et al., 2023).

\[
\text{CRR}(t|s_{\setminus t_m}; \theta) = (\rho(t_p|s_{\setminus t_m}; \theta)^{-1} - \rho(t_m|s_{\setminus t_m}; \theta)^{-1}) \\
= 1 - \rho(t_m|s_{\setminus t_m}; \theta)^{-1} \\
= \text{CRR}(w; \theta)
\] (2)

Salutari et al., 2023 proposes the Complementary Reciprocal Rank (CRR(\( t|s_{\setminus t_m}; \theta \)); Equation 2) for a masked token given its context, where \( \rho(t_p|s_{\setminus t_m})^{-1} \) is the reciprocal rank of the predicted token (and always equal to 1) and \( \rho(t_m|s_{\setminus t_m})^{-1} \) is the reciprocal rank of the masked token. Thus, \( \rho(t_m|s_{\setminus t_m})^{-1} \) provides a likelihood measure for \( t_m \) being chosen by the model as a candidate token to replace the ground truth (masked) one.

Salutari et al., 2023 defines \( \Delta p(s) \) (Appendix D) as the probability difference for a sentence s and CRR(s) (Appendix E; average of all single masked token CRRs with token ordering preserved) as the complementary reciprocal rank for a sentence s, and uses them as proxies for an MLM’s prediction quality or preference, claiming metrics based on CRR for a sentence s are necessary to fully capture the biases embedded in MLMs.

Kaneko and Bollegala, 2022 propose evaluation metrics All Unmasked Likelihood (AUL; Appendix F) and AUL with Attention weights (AULA; Appendix G), where both are generated by requiring the MLM to predict all tokens (unmasked input). By requiring the MLM to simultaneously predict all of the tokens in a given unmasked input sentence s, Kaneko and Bollegala, 2022 aim to eliminate biases associated with masked tokens under previously proposed pseudo-likelihood-based scoring methods (Nadeem et al., 2021, Nangia et al., 2020), which assumed that masked tokens are statistically independent, and selectional biases from masking a subset of input tokens, such as high frequency

2For example, "This is <mask>".
3For example, "<mask> is a/an [occupation]".
words (which are masked more often during training).

Kaneko and Bollegala, 2022 reference the use of sentence-level embeddings produced by MLMs for downstream tasks such as sentiment classification to argue that biases associated with masked tokens should not influence the intrinsic bias evaluation of an MLM, as opposed to the evaluation of biases introduced after an MLM is fine-tuned. In contrast, we focus on an MLM’s key pretraining objective, masked language modeling, to measure social biases of the MLM.4 In addition, we measure relative changes in biases w.r.t. the intrinsic biases of the same base MLM after retraining under the MLMO (as opposed to fine-tuning). Thus, we argue that biases associated with masked tokens are not undesirable in our case.

AUL and AULA were found to be sensitive to contextually meaningful inputs by randomly shuffling tokens in input sentences and comparing accuracies with and without shuffle. CRR is also conditional to the unmasked token context by definition. We argue measure sensitivity to unmasked token contexts is desirable when evaluating a given MLM’s preference under a fill-mask task, or when estimating biases using contextualized token-level embeddings produced by an MLM.

In contrast with previous methods such as SSS and cSPPS, we refrain from using strictly modified or unmodified subsets of input tokens as context and instead provide all tokens but the ground truth one as context for MLM prediction under each iteration of our masking experiment. In this sense, CRR and \( \Delta P \) consider all tokens equally, and like AUL, they might also benefit from considering the weight of MLM attention as a proxy for token importance when probing for MLM preferences for ground truth tokens between two paired sentences along a social bias axis.5

Existing benchmark datasets such as CPS are limited to one ground truth per masked token, so an important consideration is an MLM’s ability to predict multiple plausible tokens for a context that could qualify for concerning social bias but goes unrecognized during evaluation using previously proposed measures. CRR could perform better than pseudo-(log)likelihood-based measures for this sensitivity that yields larger relative differences in \( \Delta P(t|c) \) as opposed to \( \mathrm{CRR}(t|c) \)6, and is deemed critical for evaluation by Salutari et al., 2023 due to the possible uniformity of probabilities generated by a particular MLM w.r.t. others.

3 Methodology

3.1 Biases in Pretrained MLMs

We probe for MLM preferences using the IME, which masks one token at a time until all tokens have been masked, or until we have \( n \) logits or predictions for a sentence with \( n \) tokens. Appendix I shows an IME example for one model and text.

Table 1 shows masked token predictions (those with first rank and highest probability) produced by MLMs for example input contexts. The ground truth masked token rank, or \( \rho(t_m|c) \), represents the quality of the models’ predictions relative to a sentence biased against an advantaged or disadvantaged group. Special start and end character tokens for MLMs are not included in the span of tokens considered in our experiments to eliminate noise.7

\[
\Delta P(t|s_{\setminus t_m}; \theta) = (\log P(t_p|s_{\setminus t_m}; \theta) - \log P(t_m|s_{\setminus t_m}; \theta)) \\
= \Delta P(w; \theta)
\]

Motivated by the typically long-tail distribution of probabilities output by MLMs, we redefine Equation 1 and propose a modified version \( \Delta P(w) \) as shown in Equation 3. The log transformation reduces sensitivity to differences when probabilities are high but enhances relative differences when probabilities are low, improving sensitivity to subtle but potentially meaningful variations in low-probability token choices.

\[
\mathrm{CRR}(t|s_{\setminus t_m}; \theta) = a_m (1 - \log \rho(t_m|s_{\setminus t_m}; \theta)^{-1}) \\
= \mathrm{CRR}(w; \theta)
\]
(4)

\[
\Delta \mathrm{PA}(t|s_{\setminus t_m}; \theta) = a_m (\log P(t_p|s_{\setminus t_m}; \theta) \\
- \log P(t_m|s_{\setminus t_m}; \theta)) \\
= \Delta \mathrm{PA}(w; \theta)
\]
(5)

4Masked language modeling was a pretraining objective for all transformers considered in this work. Next sentence prediction was not used for RoBERTa and variants due to relatively lower performance with its inclusion (Liu et al., 2019b).

5By incorporating attention weights, \( \Delta \mathrm{PA} \) and CRRA consider token-level contributions to MLM predictions, leveraging the attention mechanism’s role in weighting contextually important tokens.

6The lowest rank of any possible ground truth token within a model’s vocabulary is equal to the size of the vocabulary (\( 1 \leq \rho(t_m|c) \leq V \) and \( 0 \leq \mathrm{CRR}(s) \leq 1 - (V)^{-1} \) for an MLM with embedding vocabulary of size \( V \)).

7Special start and end character tokens for MLMs are not considered by measures using attention weights and probabilities computed from the (log)softmax of model logits for the masked token index.
<table>
  <tr>
    <th>Model</th>
    <th>IME Input with masked token \( w_{11} = \) money</th>
    <th>\( w_{11} \) Rank</th>
  </tr>
  <tr>
    <td>distilRoBERTa<sub>P</sub></td>
    <td>The <b>poor</b> are really ignorant about how to handle the <b>problems</b> they do have</td>
    <td>5</td>
  </tr>
  <tr>
    <td>distilRoBERTa<sub>R</sub></td>
    <td>The <b>poor</b> are really ignorant about how to handle the <b>money</b> they do have</td>
    <td>1</td>
  </tr>
  <tr>
    <td>distilRoBERTa<sub>P</sub></td>
    <td>The <b>rich</b> are really ignorant about how to handle the <b>wealth</b> they do have</td>
    <td>4</td>
  </tr>
  <tr>
    <td>distilRoBERTa<sub>R</sub></td>
    <td>The <b>rich</b> are really ignorant about how to handle the <b>problems</b> they do have</td>
    <td>2</td>
  </tr>
</table>

Table 1: MLM predictions and ground truth masked token ranks for input sentences under the IME. The processed input sentences are “The [poor, rich] are really ignorant about how to handle the money they do have”, and are contained in \( S_{dis} \) or \( S_{adv} \) from CPS with characteristic words poor and rich respectively. Masked tokens are in bold.

For a given MLM with parameters \( \theta \), we propose attention-weighted measures CRRA(\( w \)) and \( \Delta PA(w) \) defined in Equations 4 and 5 respectively, where \( a_w \) is the average of all multi-head attentions associated with the ground truth token \( w \)\footnote{Averaging the multi-head attentions associated with a ground truth token provides a potentially balanced representation of token-level contributions by capturing diverse contextual relationships encoded by individual heads while reducing noise from any single head.}, and \( P(t_m|s_{\setminus t_m}) \) and \( \rho(t_m|s_{\setminus t_m}) \) are the probability score and rank of the masked token respectively. We extend these definitions for a sentence \( s \) as shown in Equations 6 and 7.

\[
\text{CRRA}(s) := \frac{1}{l_s} \sum_{w \in s} \text{CRRA}(w; \theta) \tag{6}
\]

\[
\Delta PA(s) := \frac{1}{l_s} \sum_{w \in s} \Delta PA(w; \theta) \tag{7}
\]

We compute measure \( f \), \( \forall f \in \{ \text{CRR}_T(s), \Delta P_T(s), \text{CRRA}_T(s), \Delta PA_T(s) \} \), where \( T \) is an MLM transformer and \( s \) is a sentence, \( \forall s \in S_{dis} \) and \( \forall s \in S_{adv} \).\footnote{We apply the Shapiro-Wilk test (Shapiro and Wilk, 1965) for normality to each measure and did not find evidence that the measures were not drawn from a normal distribution. The same was found for the difference of each of these measures between sentence sets relative to the same transformer \( T \).} Table 13 in the Appendix shows these measures (likelihood scores) for an example sentence \( s \) in SS and CPS.\footnote{Greater MLM preference based on prediction quality is reflected by CRR, CRRA, \( \Delta P \) and \( \Delta PA \) values closer to 0 (if a sentence with bias against advantaged groups has a greater value relative to its paired counterpart, the MLM is deemed to prefer bias against *disadvantaged* groups). The opposite is true for measures CSPS, SSS, AUL and AULA.} We define sets of measures \( M_1 \) and \( M_2 \) (\( M_1 \cap M_2 = \emptyset \)), where \( M_1 = \{ \text{CRR}, \text{CRRA}, \Delta PA, \Delta P \} \) and \( M_2 = \{ \text{AUL}, \text{AULA}, \text{CSPS}, \text{SSS} \} \).

\[
\Delta f_T(i) = \begin{cases}
f_T(S_{adv}(i)) - f_T(S_{dis}(i)), & \text{if } f \in M_1 \\
f_T(S_{dis}(i)) - f_T(S_{adv}(i)), & \text{if } f \in M_2
\end{cases} \tag{8}
\]

We apply Equation 8 to estimate the preference of a transformer \( T \) for \( s \in S_{dis} \) relative to \( s \in S_{adv} \) for paired sentence \( s \) with index \( i \) and measure \( f \),

\( \forall f \in \{ \text{CRR, CRRA, } \Delta P, \Delta PA, \text{CSPS, SSS, AUL, AULA} \} \). We define a bias score for a pretrained MLM as BSPT (Equation 9).

\[
\text{BSPT}(f) := \frac{100}{N} \sum_{i=1}^N \mathbb{I}(\Delta f_T(i) > 0) \tag{9}
\]

\( \mathbb{I} \) is a per-model indicator function which returns 1 if transformer \( T \) has a larger preference for \( S_{dis} \) relative to \( S_{adv} \) and 0 otherwise, as estimated for a measure \( f \) by Equation 8. BSPT represents the proportion of sentences with higher relative bias against disadvantaged groups for a given measure, where values above 50 indicate greater relative bias against disadvantaged groups for an MLM.

3.2 Biases Introduced by MLM Retraining

We define a bias score for a retrained MLM (relative to its pretrained base) as BSRT (Equation 10).

\[
\text{BSRT}(f) := \frac{100}{N} \sum_{i=1}^N \mathbb{I}(\Delta f_{T_1}(i) > \Delta f_{T_2}(i)) \tag{10}
\]

We define a proxy for the relative preference between two MLMs with the model-comparative indicator function \( \mathbb{I} \), which returns 1 if transformer \( T_1 \) has a larger preference for \( S_{dis} \) relative to \( S_{adv} \) than transformer \( T_2 \) and 0 otherwise, as estimated for a measure \( f \) by Equation 8. BSRT can be applied to compare pre- and retrained MLMs within the same model class and recover biases introduced by MLM retraining. Values above 50 indicate greater bias against disadvantaged groups for transformer \( T_1 \) relative to \( T_2 \), or a retrained transformer relative to its pretrained base.

4 Experiments and Findings

4.1 Benchmark Datasets for Social Bias

The Crowdsourced Stereotype Pairs Benchmark (CPS; Nangia et al., 2020) contains biased sentences towards historically advantaged and disadvantaged groups along nine forms of social biases. StereoSet (SS; Nadeem et al., 2021) contains intrasentence and intersentence (with context) pairs
for four forms of social biases, using the likelihood of modified tokens given unmodified token contexts as proxy for MLM preference. Similarly, CPS contains characteristic words that distinguish sentences within pairs and define the nature of a particular bias towards either advantaged or disadvantaged groups, but instead uses the relative likelihood of unmodified tokens being chosen by the MLM given a modified context (characteristic word) across a sentence pair.

CPS and SS contain biased sentences towards advantaged and disadvantaged groups, where CPS sentence pairs are categorized by bias types: race, age, socioeconomic, disability, religion, physical appearance, gender, sexual orientation, and nationality, and SS sentence pairs by: race, religion, gender, and profession (see Appendix A.1 for details on sentence counts for bias categories).

To probe for biases of interest that are encoded in MLMs, the scope of our experiments include all bias categories and sentences pairs in CPS and intrasentence pairs in SS, since intersentence pairs are not masked for bias evaluation (Kaneko and Bollegala, 2022).11 We estimate MLM preference towards a stereotypical sentence over a less stereotypical one for each bias category in CPS and SS.

4.2 Retraining Dataset

CPS provides a more diverse alternative to biases expressed by sentence pairs in SS. Biases widely acknowledged in the United States are well represented in CPS12 compared to SS, and there is greater diversity of sentence structures in CPS (Nangia et al., 2020). CPS has been found to be a more reliable benchmark for pretrained MLM bias measurement than SS, and the validation rate of CPS is 18% higher than SS. (Nangia et al., 2020). Based on these findings, paired with (1) the computational expense and time-consumption involved with retraining MLMs under the MLMO and (2) concerns regarding standard masked language modeling metric viability on SS, we proceed to use sentence sets in CPS to re-train MLMs and validate methods for estimating the biases that are introduced (Nangia et al., 2020) (see Appendix A.2 for details). We re-train MLMs \( \forall s \in S_{\text{dis}} \) or \( \forall s \in S_{\text{adv}} \), where \( s \) is a sentence in CPS biased towards either advantaged (\( S_{\text{adv}} \)) or disadvantaged groups (\( S_{\text{dis}} \)).

4.3 Transformer-based Language Models

We denote pre- and retrained transformers as \( T_P \) and \( T_R \) respectively. The subscript \( \text{unc} \) denotes an uncased model. We report results from the following transformer-based language models available through the HuggingFace library (Wolf et al., 2020): bert-base-uncased (BERT\(_{\text{unc}}\); Devlin et al., 2019), roberta-base (RoBERTA; Liu et al., 2019b), distilbert-base-uncased (distilBERT\(_{\text{unc}}\); Sanh et al., 2019), and distilroberta-base (distil-RoBERTa; Liu et al., 2019a).

4.4 Pretrained MLM Bias Scores

We use BSPT to compare MLM preferences, report overall bias scores in Table 2, and provide detailed results for all measures and MLMs in Appendix K.

<table>
  <tr>
    <th></th>
    <th>RoBERTa<sub>P</sub></th>
    <th>BERT<sub>P,unc</sub></th>
    <th>D-RoBERTa<sub>P</sub></th>
    <th>D-BERT<sub>P,unc</sub></th>
  </tr>
  <tr>
    <th>CPS Dataset</th>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>CSPS</td>
    <td>59.35</td>
    <td>60.48</td>
    <td>59.35</td>
    <td>56.83</td>
  </tr>
  <tr>
    <td>AUL</td>
    <td>58.75</td>
    <td>48.34</td>
    <td>53.32</td>
    <td>51.59</td>
  </tr>
  <tr>
    <td>AULA</td>
    <td>58.09</td>
    <td>48.21</td>
    <td>51.86</td>
    <td>52.65</td>
  </tr>
  <tr>
    <td>CRR</td>
    <td>58.89</td>
    <td><b>61.07</b></td>
    <td>57.76</td>
    <td>56.23</td>
  </tr>
  <tr>
    <td>CRRA</td>
    <td><b>60.68</b></td>
    <td>58.89</td>
    <td><b>61.94</b></td>
    <td><b>60.08</b></td>
  </tr>
  <tr>
    <td>\( \Delta P \)</td>
    <td>59.88</td>
    <td>60.08</td>
    <td>59.75</td>
    <td>57.49</td>
  </tr>
  <tr>
    <td>\( \Delta PA \)</td>
    <td>60.15</td>
    <td>60.81</td>
    <td>59.81</td>
    <td>58.02</td>
  </tr>
  <tr>
    <th>SS Dataset</th>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>SSS</td>
    <td>61.06</td>
    <td><b>59.16</b></td>
    <td><b>61.4</b></td>
    <td>60.59</td>
  </tr>
  <tr>
    <td>AUL</td>
    <td>59.45</td>
    <td>48.91</td>
    <td>60.21</td>
    <td>51.71</td>
  </tr>
  <tr>
    <td>AULA</td>
    <td>58.83</td>
    <td>50.28</td>
    <td>59.59</td>
    <td>51.66</td>
  </tr>
  <tr>
    <td>CRR</td>
    <td>57.83</td>
    <td>53.85</td>
    <td>54.37</td>
    <td>53.42</td>
  </tr>
  <tr>
    <td>CRRA</td>
    <td>62.06</td>
    <td>58.59</td>
    <td>60.54</td>
    <td><b>61.11</b></td>
  </tr>
  <tr>
    <td>\( \Delta P \)</td>
    <td>62.2</td>
    <td>58.64</td>
    <td><b>61.4</b></td>
    <td>59.31</td>
  </tr>
  <tr>
    <td>\( \Delta PA \)</td>
    <td><b>62.35</b></td>
    <td>58.21</td>
    <td>61.35</td>
    <td>59.31</td>
  </tr>
</table>

Table 2: Overall bias scores for pretrained MLMs using BSPT(f) with measures \( \forall f \in \{ \text{CRR}, \text{CRRA}, \Delta P, \Delta PA, \text{CSPS}, \text{SSS}, \text{AUL}, \text{AULA} \} \) on CPS and SS, where D- denotes distilled. Color-coded from lightest to darkest, with lower values represented by lighter shades and higher values by darker shades. Bold values indicate the highest bias score for an MLM across all measures.

Overall, all evaluation methods show concerning social biases against disadvantaged groups embedded in MLMs as observed in prior research (Kaneko and Bollegala, 2022, Nangia et al., 2020). Interestingly, BERT\(_{\text{unc}}\) has the lowest overall SSS, AUL, AULA, CRRA, \( \Delta P \), and \( \Delta PA \) (second lowest CRR) on SS, but conflicting results on CPS, where it has the highest CSPS, CRR, \( \Delta P \), and \( \Delta PA \) but the lowest AUL, AULA, and CRRA. RoBERTa and distilRoBERTa have higher overall bias than BERT\(_{\text{unc}}\) and distilBERT\(_{\text{unc}}\) according to (1) all but one measure on SS and (2) AUL and CRRA on CPS.

11Our experiments on the SS dataset include only intrasentence pairs as in experiment code used by Kaneko and Bollegala, 2022.
12CPS categories are a "narrowed" version of the US Equal Employment Opportunities Commission's list of protected categories (Nangia et al., 2020).
Kaneko and Bollegala, 2022 observe a higher bias score for religion in CPS across CSPS, AUL, and AULA with the roberta-large MLM. Nangia et al., 2020 also observe that roberta-large has relatively higher bias scores for the religion category in CPS, and relatively lower bias scores for the gender and race categories compared to SS.\footnote{BERT\textsubscript{unc} and RoBERTa in this paper are transformers roberta-base and bert-base-uncased as referenced in Section 4.3, whereas Kaneko and Bollegala, 2022 use roberta-large and bert-base-cased in their experiments.}
Similarly, we observe that gender has relatively lower scores in CPS compared to SS across MLMs, but that race bias remains low across all MLMs, measures, and datasets. We observe a relatively high religious bias across all MLMs in CPS, but find AUL and AULA tend to underestimate religious bias on SS and overestimate it on CPS compared to proposed measures for BERT\textsubscript{unc}. We find that other measures underestimate disability bias for BERT\textsubscript{unc} and distilBERT\textsubscript{unc} on CPS, but give similar estimates for RoBERTa and distilRoBERTa. We find that only AUL and AULA estimate overall bias scores below 50, where the corresponding MLM is BERT\textsubscript{unc} in each case.

4.4.1 Human Annotated Bias Alignment
We compare the alignment (agreement) between measures and bias ratings in CPS. Sentence pairs in CPS received five annotations in addition to the implicit annotation from the writer. We map sentences to a binary classification task, where a sentence is considered biased if it satisfies criteria from Nangia et al., 2020, where (1) at least three out of six annotators (including the implicit annotation) agree a given pair is socially biased and (2) the majority of annotators who agree a given pair is socially biased agree on the type of social bias being expressed.\footnote{This experiment setting gives 58 unbiased pairs and 1,450 biased pairs for binary classification.}

We compute evaluation measures derived from MLMs to predict whether a pair is biased or unbiased at varying thresholds. All measures are computed for each sentence in a pair. Thresholds for bias scores computed on sentences with bias against advantaged and disadvantaged groups respectively maximize area under the ROC Curve for that measure. We find that one or more of proposed measures \( \Delta P \) and CRRA, along with modified \( \Delta P \) and CRR, outperform AUL and AULA in their agreement with human annotators on CPS for RoBERTaP and BERT\textsubscript{P,unc} based on higher AUROC values if MLMs exhibit bias towards disadvantaged groups (ROC curves in Appendix L).

4.5 Retrained MLM Bias Scores
We re-train each transformer under consideration using the PyTorch Python library with P100 and T4 GPUs on cased (RoBERTa and distilRoBERTa) and uncased (BERT\textsubscript{unc} and distilBERT\textsubscript{unc}) versions of CPS sentences (see Appendix M.1 for more details).

![Difference between ECDFs for CRR distribution for sentences in Sdis and Sadv for pre- and retrained RoBERTa and the race bias category in CPS. The line at y = 0 separates what is biased against disadvantaged groups on the positive y-axis from what is biased against advantaged groups on the negative y-axis.](page_768_1092_484_312.png)

Figure 1: Difference between ECDFs for CRR distribution for sentences in \( S_{dis} \) and \( S_{adv} \) for pre- and retrained RoBERTa and the race bias category in CPS. The line at \( y = 0 \) separates what is biased against disadvantaged groups on the positive y-axis from what is biased against advantaged groups on the negative y-axis.

Figure 1 shows the difference in ECDFs for the \( \mathrm{CRR}_m(s) \) measure on the race bias category in CPS, where \( m \) is retrained MLM RoBERTa\(_R\). It illustrates how the difference in ECDF distributions for an MLM can visually represent a contextual shift in relative bias. When compared with pretrained RoBERTa (RoBERTa-P), we observe an *upwards* shift in the difference of ECDFs for the CRR difference between sentence sets after retraining RoBERTa on \( S_{dis} \) (RoBERTa-R-DIS), and a *downwards* shift after retraining on \( S_{adv} \) (RoBERTa-R-ADV). This is expected since retraining on \( S_{dis} \) or \( S_{adv} \) should shift the MLM preference towards the corresponding bias type.

We compute BSRT for retrained MLMs and all measures and bias categories on CPS and report detailed results in Appendix M.2.\footnote{We assess whether the relative differences in proportions of bias between pre- and retrained transformers are statistically significant according to McNemar’s test (McNemar, 1947) (p-value < 0.05), using binarized outcomes for bias as given by \( f_{R}(S_{adv}) > f_{R}(S_{dis}) \) and \( f_{P}(S_{adv}) > f_{P}(S_{dis}) \) to create contingency tables of outcome pairings between pre- and retrained transformers to test for marginal homogeneity.} In general, each measure produces a bias score in accordance with the particular retraining dataset used for almost
<table>
  <tr>
    <th>Model</th>
    <th>Retrain dataset</th>
    <th>\( f : \min / \max \) BSRT \( \forall c \in C \)</th>
    <th>\( f : p < a \forall c \in C \)</th>
  </tr>
  <tr>
    <td>RoBERTa<sub>R</sub></td>
    <td>\( \forall s \in S_{\text{dis}} \)<br>\( \forall s \in S_{\text{adv}} \)</td>
    <td>\( \max \) BSRT : \( \{\Delta P, \Delta PA\} \)<br>\( \min \) BSRT : \( \{CRRA, \Delta P\} \)</td>
    <td>CRR, CRRA, \( \Delta P, \Delta PA \)<br>CSPS, CRRA, CRRA, \( \Delta P, \Delta PA \)</td>
  </tr>
  <tr>
    <td>distilRoBERTa<sub>R</sub></td>
    <td>\( \forall s \in S_{\text{dis}} \)<br>\( \forall s \in S_{\text{adv}} \)</td>
    <td>\( \max \) BSRT : \( \{CRR, \Delta P, \Delta PA\} \)<br>\( \min \) BSRT : \( \{CRRA, \Delta P, \Delta PA\} \)</td>
    <td>\( \Delta P, \Delta PA \)<br>CRR, CRRA, \( \Delta P, \Delta PA \)</td>
  </tr>
  <tr>
    <td>BERT<sub>R</sub></td>
    <td>\( \forall s \in S_{\text{dis}} \)<br>\( \forall s \in S_{\text{adv}} \)</td>
    <td>\( \max \) BSRT : \( \{\Delta P, \Delta PA\} \)<br>\( \min \) BSRT : \( \{CRRA, \Delta P, \Delta PA\} \)</td>
    <td>CRR, CRRA, \( \Delta P, \Delta PA \)<br>CSPS, CRRA, CRRA, \( \Delta P, \Delta PA \)</td>
  </tr>
  <tr>
    <td>distilBERT<sub>R</sub></td>
    <td>\( \forall s \in S_{\text{dis}} \)<br>\( \forall s \in S_{\text{adv}} \)</td>
    <td>\( \max \) BSRT : \( \{AUL, CRR, \Delta P, \Delta PA\} \)<br>\( \min \) BSRT : \( \{CRRA, \Delta P, \Delta PA\} \)</td>
    <td>AUL, AULA, \( \Delta P, \Delta PA \)<br>CSPS, CRRA, \( \Delta P, \Delta PA \)</td>
  </tr>
</table>

Table 3: Measures \( \forall f \in \{CRR, CRRA, \Delta P, \Delta PA, CSPS, AUL, AULA\} \) with at least one min BSRT or max BSRT across bias categories in CPS for MLMs retrained on \( S_{\text{adv}} \) or \( S_{\text{dis}} \) respectively (BSRT; Equation 10). \( f : p < a \forall c \in C \) indicates that, for a measure \( f \) and every bias category in CPS, the relative difference in proportions of bias between pre- and retrained transformers is statistically significant according to McNemar’s test (p-value < 0.05).

all significant results across MLMs, demonstrating that proposed function BSRT can be applied to estimate the bias against disadvantaged groups for transformer \( T_1 \) relative to its pretrained base \( T_2 \).

In Table 3, we report (1) measures with at least one min BSRT or max BSRT across bias categories in CPS for MLMs retrained on \( S_{\text{adv}} \) or \( S_{\text{dis}} \) respectively, and (2) measures for which the relative difference in proportions of bias between pre- and retrained transformers is statistically significant according to McNemar’s test (p-value < 0.05) for every bias category in CPS.

For all MLMs retrained on \( S_{\text{dis}} \), CRR, \( \Delta P \), and \( \Delta PA \) give results above 50 for each bias category as expected, with higher scores than other measures in almost every case. CRRA also gives results above 50 for each bias category except for disability bias using distilBERT<sub>R</sub>. Similarly, for all MLMs retrained on \( S_{\text{adv}} \), CRR, CRRA, \( \Delta P \), and \( \Delta PA \) give results below 50 for each bias category as expected, with lower scores than other measures in almost every case. In contrast, AUL and CSPS give 3 and 4 bias scores below 50 respectively across all MLMs retrained on \( S_{\text{dis}} \). Notably, AUL and CRRA give 1 bias score below 50. AUL and AULA give 4 and 5 bias scores above 50 respectively across all MLMs retrained on \( S_{\text{adv}} \).

As shown in Table 3, in every case and across MLMs, one or more of proposed measures reports the highest retraining bias scores for MLMs retrained on \( S_{\text{dis}} \) and the lowest for MLMs retrained on \( S_{\text{adv}} \). In addition, \( \Delta P \) and \( \Delta PA \) give significant results for every bias type across all MLMs retrained on \( S_{\text{dis}} \) or \( S_{\text{adv}} \), and CRR and CRRA give significant results for every bias type using BERT<sub>R</sub> and RoBERTa<sub>R</sub>. In contrast, AUL, AULA, and CSPS have 5, 6, and 11 insignificant results respectively across all MLMs retrained on \( S_{\text{dis}} \), and 1, 12, and 13 insignificant results respectively across all MLMs retrained on \( S_{\text{adv}} \).

We find that \( \Delta PA, \Delta P, \) and CRRA show greater sensitivity than CRR, CSPS, AUL, and AULA for relative changes in MLM bias due to retraining, indicated by larger and smaller scores and more frequently significant relative difference in proportions of bias between pre- and retrained transformers. We find that CSPS, AUL, and AULA produce concerning underestimations of biases introduced by retraining MLMs only on sentences with biases against disadvantaged groups from CPS, and overestimations from retraining on sentences with biases against advantaged groups (see Appendix M.2 for more details).

4.5.1 Retraining Bias Alignment

<table>
  <tr>
    <th colspan="2">Retrain dataset: \( \forall s \in S^{\text{dis}} \)</th>
    <th colspan="6">Model</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>CSPS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>\( \Delta P \)</th>
    <th>\( \Delta PA \)</th>
  </tr>
  <tr>
    <td>BERT<sub>R,unc</sub></td>
    <td></td>
    <td>0.028</td>
    <td>0.028</td>
    <td>0.028</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
  </tr>
  <tr>
    <td>RoBERTa<sub>R</sub></td>
    <td></td>
    <td>0.028</td>
    <td>0.000</td>
    <td>0.083</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
  </tr>
  <tr>
    <td>D-BERT<sub>R,unc</sub></td>
    <td></td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.028</td>
    <td>0.000</td>
    <td>0.000</td>
  </tr>
  <tr>
    <td>D-RoBERTa<sub>R</sub></td>
    <td></td>
    <td>0.056</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
  </tr>
  <tr>
    <th colspan="2">Retrain dataset: \( \forall s \in S^{\text{adv}} \)</th>
    <th colspan="6">Model</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>CSPS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>\( \Delta P \)</th>
    <th>\( \Delta PA \)</th>
  </tr>
  <tr>
    <td>BERT<sub>R,unc</sub></td>
    <td></td>
    <td>0.000</td>
    <td>0.083</td>
    <td>0.083</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
  </tr>
  <tr>
    <td>RoBERTa<sub>R</sub></td>
    <td></td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
  </tr>
  <tr>
    <td>D-BERT<sub>R,unc</sub></td>
    <td></td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
  </tr>
  <tr>
    <td>D-RoBERTa<sub>R</sub></td>
    <td></td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.028</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.000</td>
  </tr>
</table>

Table 4: Error rates for MLMs using measures \( \forall f \in \{CRR, CRRA, \Delta P, \Delta PA, CSPS, AUL, AULA\} \), where D- denotes distilled. Bold values indicate the lowest error rate for an MLM across all measures.

We frame a binary classification task where BSRT above 50 indicates increased preference (after retraining) for sentences with bias against disadvantaged groups in CPS (1), and vice versa for scores below 50 (0).\footnote{There are 72 predictions per measure across MLMs and bias types. Half of binary truths are 1 and 0 respectively since MLMs retrained on \( S_{\text{dis}} \) should score above 50 for all bias types (vice versa for \( S_{\text{adv}} \)) and \( S_{\text{dis}} \) has the same length (number of sentences) as \( S_{\text{adv}} \).} We report error rates for MLMs in
Table 4, and find \( \Delta_{PA} \), \( \Delta_P \), CRRA, and CRR produce the lowest error rate for all MLMs. CRR, \( \Delta_P \), and \( \Delta_{PA} \) are 100% accurate and CRRA is 99% accurate, while AUL, AULA, and CSPS are 93%, 88%, and 94% accurate respectively. Based on this evaluation setting, \( \Delta_{PA} \), \( \Delta_P \), CRRA, and CRR are more accurate than CSPS, AUL, and AULA for estimating social biases introduced by retraining MLMs.

5 Conclusion

We represent MLM bias through a model’s relative preference for ground truth tokens between two paired sentences with contrasting social bias under the iME, measured using the (attention-weighted) quality of predictions. We evaluate social biases for four state-of-the-art transformers using benchmark datasets CPS and SS and approximate the distributions of proposed measures. We use BSRT to compute bias scores for pretrained MLMs using considered measures and find all encode concerning social biases. We find that gender has lower encoded biases on CPS compared to SS across MLMs, and that other measures can underestimate religious bias against disadvantaged groups on SS and disability bias on CPS. We propose BSRT to estimate social biases against disadvantaged groups for a retrained MLM relative to its pretrained base and assess measure alignment with biases introduced by retraining under the MLMO. We find that proposed measures \( \Delta_{PA} \) and CRRA, along with modified \( \Delta_P \) and CRR, produce more accurate estimations of introduced biases than previously proposed ones, which underestimate biases after retraining on sentences biased towards disadvantaged groups, and observe \( \Delta_{PA} \), CRRA, and \( \Delta_P \) show greater sensitivity than CRR, CSPS, AUL, and AULA to relative changes in MLM bias due to retraining.

6 Limitations

We anticipate that the limitations addressed in this section will be useful for future research evaluating social biases in MLMs.

As described in section 4, we leverage sentence pairs from two benchmark datasets, CPS and SS, to evaluate the social biases of pre- and retrained MLMs. Both datasets are limited to the English language and specific social bias types represented by binary sentence sets. Future research extending this work could consider and compare alternative benchmark datasets with different languages, social bias types, and sentence set structures. In addition, we acknowledge the dependency on human annotated biases in benchmark datasets when assessing discussed measures.

In this work, and as mentioned in section 2.2, we focus on an MLM’s key pretraining objective, masked language modeling, to measure social biases of the MLM. Different pretraining objectives such as next sentence prediction are beyond the scope of this paper. Furthermore, we measure relative changes in biases w.r.t. the intrinsic biases of a base MLM after retraining under the MLMO, and report results from the four transformers mentioned in section 4.3. A logical extension of this work would be considering MLMs with different architectures or training data. We propose the model-comparative function BSRT to measure the relative change in MLM biases after retraining. Future research could leverage this function to assess the sensitivity of discussed measures to a range of MLM retraining conditions.

We also encourage research assessing the agreement between relative changes in MLM biases introduced by retraining and biases embedded in the retraining corpus.

7 Ethical Considerations

The methods and measures employed and proposed as part of this work are intended to be used for measuring social biases in pre- and retrained MLMs. We do not condone the use of this research to further target disadvantaged groups in any capacity. Instead, we encourage the use of proposed measures in conjunction with model debiasing efforts to lessen encoded social biases against disadvantaged groups in MLMs used in production settings. No ethical issues have been reported concerning the datasets or measures used in this paper to the best of our knowledge.

References

Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu,
Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.

Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. 2019. Understanding undesirable word embedding associations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1696–1705, Florence, Italy. Association for Computational Linguistics.

Aaron Gokaslan and Vanya Cohen. 2019. Open-webtext corpus. http://Skylion007.github.io/OpenWebTextCorpus.

Masahiro Kaneko and Danushka Bollegala. 2022. Unmasking the mask – evaluating social biases in masked language models. Proceedings of the AAAI Conference on Artificial Intelligence, 36(11):11954–11962.

Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794, Copenhagen, Denmark. Association for Computational Linguistics.

Sheng Liang, Philipp Dufter, and Hinrich Schütze. 2020. Monolingual and multilingual reduction of gender bias in contextualized representations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5082–5093, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Improving multi-task deep neural networks via knowledge distillation for natural language understanding.

Yinhai Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.

Thomas Manzini, Lim Yao Chong, Alan W Black, and Yulia Tsvetkov. 2019. Black is to criminal as Caucasian is to police: Detecting and removing multi-class bias in word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 615–621, Minneapolis, Minnesota. Association for Computational Linguistics.

Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 622–628, Minneapolis, Minnesota. Association for Computational Linguistics.

Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space.

Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5356–5371, Online. Association for Computational Linguistics.

Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for Computational Linguistics.

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.

Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of
the Association for Computational Linguistics, pages 2699–2712, Online. Association for Computational Linguistics.

Flavia Salutari, Jerome Ramos, Hossein A. Rahmani, Leonardo LinguaGlossa, and Aldo Lipani. 2023. Quantifying the bias of transformer-based language models for african american english in masked language modeling. In Advances in Knowledge Discovery and Data Mining, pages 532–543, Cham. Springer Nature Switzerland.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108.

S. S. Shapiro and M. B. Wilk. 1965. An analysis of variance test for normality (complete samples). Biometrika, 52(3/4):591–611.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics.

Bernard L Welch. 1947. The generalization of student’s’ problem when several different population variances are involved. Biometrika, pages 1–25.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierre Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 629–634, Minneapolis, Minnesota. Association for Computational Linguistics.

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15–20, New Orleans, Louisiana. Association for Computational Linguistics.

Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 19–27.
A  Datasets

A.1 Benchmark Datasets for Social Biases

<table>
  <tr>
    <th>Bias (CPS)</th>
    <th>N (\( S_{dis} \) and \( S_{adv} \))</th>
  </tr>
  <tr>
    <td>Race</td>
    <td>516</td>
  </tr>
  <tr>
    <td>Religion</td>
    <td>105</td>
  </tr>
  <tr>
    <td>Nationality</td>
    <td>159</td>
  </tr>
  <tr>
    <td>Socioeconomic</td>
    <td>172</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>262</td>
  </tr>
  <tr>
    <td>Sexual orientation</td>
    <td>84</td>
  </tr>
  <tr>
    <td>Age</td>
    <td>87</td>
  </tr>
  <tr>
    <td>Disability</td>
    <td>60</td>
  </tr>
  <tr>
    <td>Physical appearance</td>
    <td>63</td>
  </tr>
  <tr>
    <th>Bias (SS)</th>
    <th>N (\( S_{dis} \) and \( S_{adv} \))</th>
  </tr>
  <tr>
    <td>Race</td>
    <td>962</td>
  </tr>
  <tr>
    <td>Religion</td>
    <td>79</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>255</td>
  </tr>
  <tr>
    <td>Profession</td>
    <td>810</td>
  </tr>
</table>

Table 5: Sentence counts for bias categories in \( S_{dis} \) (stereotypical) and \( S_{adv} \) (anti-stereotypical) on CPS and SS datasets.

A.2 Retraining Datasets

<table>
  <tr>
    <th>\( \forall s \in S_{dis} \) for CPS</th>
    <th>Uncased</th>
    <th>Cased</th>
  </tr>
  <tr>
    <td>Unique Tokens</td>
    <td>4631</td>
    <td>4800</td>
  </tr>
  <tr>
    <td>Lexical Diversity</td>
    <td>0.235</td>
    <td>0.244</td>
  </tr>
  <tr>
    <th>\( \forall s \in S_{adv} \) for CPS</th>
    <th>Uncased</th>
    <th>Cased</th>
  </tr>
  <tr>
    <td>Unique Tokens</td>
    <td>4607</td>
    <td>4768</td>
  </tr>
  <tr>
    <td>Lexical Diversity</td>
    <td>0.233</td>
    <td>0.241</td>
  </tr>
</table>

Table 6: Data profile for (cased and uncased) sentence sets used to re-train MLMs, where lines correspond to sentences in CPS. Includes 1508 lines and total tokens greater than \( 1.9 * 10^4 \) for both cased and uncased data.

B  Equation for CSPS(\( s \))

\[
\text{CSPS}(s) := \sum_{t \in s} \log P(t|U_{t}, M; \theta)
\] (11)

C  Equation for SSS(\( s \))

\[
\text{SSS}(s) := \frac{1}{l_s} \sum_{t \in s} \log P(t|U; \theta)
\] (12)

D  Equation for \( \Delta P(s) \)

\[
\Delta P(s) := \frac{1}{l_s} \sum_{w \in s} \Delta P(w; \theta).
\] (13)

E  Equation for CRR(\( s \))

\[
\text{CRR}(s) := \frac{1}{l_s} \sum_{w \in s} \text{CRR}(w; \theta).
\] (14)

F  Equation for AUL(\( s \))

\[
\text{AUL}(s) := \frac{1}{l_s} \sum_{w \in s} \log P(w_i|s; \theta)
\] (15)

G  Equation for AULA(\( s \))

\[
\text{AULA}(s) := \frac{1}{l_s} \sum_{w \in s} a_i \log P(w_i|s; \theta)
\] (16)

H  Example Plot of CRR(\( t|c \))

The input for the example plot is processed text from CPS with socioeconomic bias against a disadvantaged group: "The poor are really ignorant about how to handle the money they do have".

![Example plot of CRR(t|c) for distilBERT_Punc relative to the ground truth for a sentence s and token t ∈ s given context c, where c = s\setminus t_m and t_m is the ground truth (masked) token.](page_1012_1012_377_246.png)

Figure 2: Example plot of CRR(t|c) for distilBERT_Punc relative to the ground truth for a sentence s and token t ∈ s given context c, where c = s\setminus t_m and t_m is the ground truth (masked) token.

I  Example of the IME

The pretrained model input (\( t1 \)) for the IME example in Table 7 is "The businessman s body was found dead in an alley discarded and uncared for".

1. \( t1 \) with \( w_1 \) masked
The businessman s body was found dead in an alley discarded and uncared for

2. \( t1 \) with \( w_2 \) masked
The businessman s body was found dead in an alley discarded and uncared for

3. \( t1 \) with \( w_3 \) masked
The businessman s body was found dead in an alley discarded and uncared for

4. \( t1 \) with \( w_4 \) masked
The businessman s body was found dead in an alley discarded and uncared for ...

Table 7: Example of the IME for distilRoBERTa, where the language model encodes (tokenizes) text t and predicts for a masked token \( w_i \), where i is the original token index.
J  Difference Between ECDFs for CRR(s)

![Difference between ECDFs for CRR distribution for sentences in S_{dis} (S1) and S_{adv} (S2) and pre-trained transformers for the race bias category in CPS.](page_186_153_495_312.png)

Figure 3: Difference between ECDFs for crr distribution for sentences in \( S_{dis} \) (S1) and \( S_{adv} \) (S2) and pre-trained transformers for the race bias category in CPS.

K  Bias Scores by Category for Pretrained MLMs

CPS dataset  Disability has the highest CRR across all MLMs on CPS, along with the highest AULA, CRRA, \( \Delta P \) and \( \Delta PA \) for RoBERTa, and the second highest AUL and CSPS. Similarly, disability has the highest CRRA for distilRoBERTa, and the second highest CSPS, AUL and AULA. Disability has the highest \( \Delta P \) and \( \Delta PA \) and the second highest CRRA for BERT_{unc} and distilBERT_{unc}, but appears in lower ranks for CSPS, and even more so for AULA and AUL.\footnote{Measures SSS, AUL, AULA, \( \Delta P \) and \( \Delta PA \) could be impacted the result of relative uniformity in the distribution of RoBERTa probabilities relative to each other in practice, as discussed in 4.} In general, sexual orientation bias has higher scores compared to others for BERT_{unc} and distilBERT_{unc}. Physical appearance has higher bias scores for BERT_{unc} and distilBERT_{unc} compared to RoBERTa and distilRoBERTa. Religion, disability and socioeconomic bias have the highest and second highest scores across all measures for distilRoBERTa. Similarly, religion, disability and sexual orientation have the highest and second highest scores across all measures for RoBERTa. All measures reflect higher socioeconomic and lower sexual orientation biases in distilRoBERTa compared to RoBERTa, and while distilBERT_{unc} and BERT_{unc} are both lower in socioeconomic bias than the former two MLMs, they are higher in sexual orientation bias than distilRoBERTa. Religion bias remains high across all MLMs and measures, while gender and age biases remain low. Compared to proposed measures, others underestimate disability bias for BERT_{unc} and distilBERT_{unc}, but yield similar relative estimates for RoBERTa and distilRoBERTa.

SS dataset  Gender has the first and second highest scores across all measures for RoBERTa and distilRoBERTa on SS.\footnote{Gender has the highest score across every measure for RoBERTa. It has the highest SSS, AUL, AULA and CRRA for distilRoBERTa, and the second highest CRRA, \( \Delta P \) and \( \Delta PA \).} Gender also has the highest CRR and CRRA and the second highest \( \Delta P \) and \( \Delta PA \) for distilBERT_{unc}. In general, profession and religion biases also have high scores for RoBERTa and distilRoBERTa, while race bias ranks third or fourth according to every measure. This persists for BERT_{unc} and distilBERT_{unc}, where race bias ranks third or fourth according to all measures besides CRR and AULA, which both place it second for distilBERT_{unc} and BERT_{unc} respectively. Profession has the highest AUL and AULA for BERT_{unc} and distilBERT_{unc}, while gender has the second highest (with one exception). Interestingly, religion has the highest CRR, CRRA and \( \Delta P \) for BERT_{unc}, and the second highest \( \Delta PA \), but has the lowest AUL and AULA. Similar to BERT_{unc}, religion has the highest \( \Delta P \) but the lowest AUL and AULA for distilBERT_{unc}. However, unlike BERT_{unc}, religion also has the lowest CRR and second lowest \( \Delta PA \) for distilBERT_{unc}. This might be expected since proposed measures rank religious bias for distilBERT_{unc} relatively consistently with other measures in CPS, whereas CSPS, AUL and AULA tend to overestimate religious bias for BERT_{unc} in comparison. We observe the opposite in SS, where AUL and AULA tend to underestimate the religious bias compared to proposed measures, with the notable exceptions of SSS and CRR. Overall, we can observe a higher relative gender bias in RoBERTa and distilRoBERTa compared to BERT_{unc} and distilBERT_{unc} on SS.

K.0.1  Race Bias in Pretrained MLMs

BERT_{unc} and distilBERT_{unc} are trained on English Wikipedia (16GB) and BookCorpus (Zhu et al., 2015), while RoBERTa and distilRoBERTa are trained on OpenWebText (Gokaslan and Cohen, 2019). As referenced in Section 4, Salutari et al., 2023 found that RoBERTa’s and distilRoBERTa’s exposure to less standard English through training on the OpenWebCorpus likely exposed these MLMs to a less standard form of American English,
as both models have more relative bias against SAE than AAE. Overall, results from Nangia et al., 2020 confirm intuition that RoBERTa’s exposure to web content extracted from URLs shared on Reddit (as opposed to Wikipedia) would result in a relatively higher MLM preference for biased (stereotyping) text compared to others.

Indeed, we also observe that pretrained MLMs RoBERTa and distilRoBERTa have higher incidence of race bias against disadvantaged groups. We assess the difference between means for our proposed measures with a two-tailed Welch’s t-test (Welch, 1947) and report significance results in Table 19 in the Appendix for the race category, alongside the mean difference in measures between sentence sets \( S_{adv} \) and \( S_{dis} \), or \( \frac{1}{N} \sum_{i=1}^{N} f(S_{adv}(i)) - f(S_{dis}(i)), \forall f \) on SS and CPS across all MLMs. This mean difference between sentence sets \( S_{adv} \) and \( S_{dis} \) across every MLM and measure is greater than 0, indicating that MLMs do encode bias against disadvantaged groups in the race bias category (with a lower \( \frac{1}{N} \sum_{i=1}^{N} f(S_{dis}(i)) \) relative to \( \frac{1}{N} \sum_{i=1}^{N} f(S_{adv}(i)) \)), and in some cases significantly so.

As shown in Appendix A.1, the race bias category makes up about one third of data sentence pairs in CPS (516 examples). For the race category in CPS we observe that pretrained RoBERTa has significantly different means for all proposed measures and pretrained distilRoBERTa has significantly different means for three of four measures. Similarly, pretrained RoBERTa and distilRoBERTa have significantly different means in three of four measures for the race category on SS. Based on these results we can only infer that pretrained RoBERTa and distilRoBERTa have relatively higher bias against disadvantaged groups in the race category compared to pretrained BERT_unc and distilBERT_unc.
<table>
  <tr>
    <th colspan="8">MLM: RoBERTa<sub>P</sub></th>
  </tr>
  <tr>
    <th>Bias (CPS)</th>
    <th>CSPS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>ΔP</th>
    <th>ΔPA</th>
  </tr>
  <tr><td>Religion</td><td>74.29</td><td>57.14</td><td>53.33</td><td>66.67</td><td>63.81</td><td>67.62</td><td>64.76</td></tr>
  <tr><td>Nationality</td><td>64.15</td><td>60.38</td><td>56.6</td><td>55.35</td><td>57.86</td><td>54.09</td><td>55.35</td></tr>
  <tr><td>Race</td><td>54.07</td><td>54.26</td><td>56.78</td><td>59.11</td><td>62.02</td><td>60.47</td><td>62.21</td></tr>
  <tr><td>Socioeconomic</td><td>61.05</td><td>65.12</td><td>66.28</td><td>61.05</td><td>62.79</td><td>61.63</td><td>59.88</td></tr>
  <tr><td>Gender</td><td>54.96</td><td>56.49</td><td>53.44</td><td>55.73</td><td>55.73</td><td>56.49</td><td>56.49</td></tr>
  <tr><td>Sexual orientation</td><td>60.71</td><td>72.62</td><td>67.86</td><td>50.0</td><td>64.29</td><td>63.1</td><td>63.1</td></tr>
  <tr><td>Age</td><td>66.67</td><td>58.62</td><td>59.77</td><td>56.32</td><td>55.17</td><td>56.32</td><td>54.02</td></tr>
  <tr><td>Disability</td><td>66.67</td><td>68.33</td><td>68.33</td><td>71.67</td><td>70.0</td><td>68.33</td><td>70.0</td></tr>
  <tr><td>Physical appearance</td><td>60.32</td><td>58.73</td><td>52.38</td><td>63.49</td><td>60.32</td><td>58.73</td><td>58.73</td></tr>
  <tr>
    <th>Bias (SS)</th>
    <th>SSS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>ΔP</th>
    <th>ΔPA</th>
  </tr>
  <tr><td>Race</td><td>57.48</td><td>56.65</td><td>56.96</td><td>56.44</td><td>60.71</td><td>60.19</td><td>60.4</td></tr>
  <tr><td>Profession</td><td>62.59</td><td>61.98</td><td>60.37</td><td>58.89</td><td>62.47</td><td>62.72</td><td>63.09</td></tr>
  <tr><td>Gender</td><td>69.8</td><td>64.71</td><td>62.35</td><td>61.96</td><td>66.27</td><td>67.45</td><td>66.27</td></tr>
  <tr><td>Religion</td><td>60.76</td><td>50.63</td><td>54.43</td><td>50.63</td><td>60.76</td><td>64.56</td><td>65.82</td></tr>
</table>

Table 8: Bias scores for biases in CPS (top) and SS dataset (bottom) with RoBERTa<sub>P</sub> as given by BSPT (Equation 9) using measures CRR, CRRA, ΔP, ΔPA, CSPS, SSS, AUL and AULA.

<table>
  <tr>
    <th colspan="8">MLM: BERT<sub>P,unc</sub></th>
  </tr>
  <tr>
    <th>Bias (CPS)</th>
    <th>CSPS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>ΔP</th>
    <th>ΔPA</th>
  </tr>
  <tr><td>Religion</td><td>71.43</td><td>66.67</td><td>66.67</td><td>59.05</td><td>60.0</td><td>63.81</td><td>60.95</td></tr>
  <tr><td>Nationality</td><td>62.89</td><td>51.57</td><td>54.09</td><td>52.83</td><td>50.94</td><td>47.17</td><td>49.69</td></tr>
  <tr><td>Race</td><td>58.14</td><td>48.84</td><td>49.42</td><td>62.98</td><td>59.5</td><td>61.24</td><td>62.02</td></tr>
  <tr><td>Socioeconomic</td><td>59.88</td><td>43.02</td><td>40.7</td><td>59.3</td><td>58.72</td><td>58.72</td><td>62.21</td></tr>
  <tr><td>Gender</td><td>58.02</td><td>46.56</td><td>43.89</td><td>54.2</td><td>53.44</td><td>58.02</td><td>57.25</td></tr>
  <tr><td>Sexual orientation</td><td>67.86</td><td>50.0</td><td>50.0</td><td>72.62</td><td>72.62</td><td>71.43</td><td>71.43</td></tr>
  <tr><td>Age</td><td>55.17</td><td>51.72</td><td>49.43</td><td>59.77</td><td>57.47</td><td>50.57</td><td>52.87</td></tr>
  <tr><td>Disability</td><td>61.67</td><td>38.33</td><td>41.67</td><td>80.0</td><td>71.67</td><td>76.67</td><td>75.0</td></tr>
  <tr><td>Physical appearance</td><td>63.49</td><td>30.16</td><td>33.33</td><td>71.43</td><td>66.67</td><td>71.43</td><td>73.02</td></tr>
  <tr>
    <th>Bias (SS)</th>
    <th>SSS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>ΔP</th>
    <th>ΔPA</th>
  </tr>
  <tr><td>Race</td><td>56.03</td><td>46.88</td><td>49.48</td><td>52.7</td><td>56.55</td><td>57.48</td><td>56.34</td></tr>
  <tr><td>Profession</td><td>60.62</td><td>51.23</td><td>51.98</td><td>54.2</td><td>60.12</td><td>58.64</td><td>59.26</td></tr>
  <tr><td>Gender</td><td>66.67</td><td>49.8</td><td>48.63</td><td>53.73</td><td>60.39</td><td>61.57</td><td>61.18</td></tr>
  <tr><td>Religion</td><td>58.23</td><td>46.84</td><td>48.1</td><td>64.56</td><td>62.03</td><td>63.29</td><td>60.76</td></tr>
</table>

Table 9: Bias scores for biases in CPS (top) and SS dataset (bottom) with BERT<sub>P,unc</sub> as given by BSPT (Equation 9) using measures CRR, CRRA, ΔP, ΔPA, CSPS, SSS, AUL and AULA.
<table>
  <tr>
    <th colspan="8">MLM: distilBERT<sub>P,unc</sub></th>
  </tr>
  <tr>
    <th>Bias (CPS)</th>
    <th>CSPS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>ΔP</th>
    <th>ΔPA</th>
  </tr>
  <tr>
    <td>Religion</td>
    <td>70.48</td>
    <td>55.24</td>
    <td>52.38</td>
    <td>54.29</td>
    <td>65.71</td>
    <td>65.71</td>
    <td>65.71</td>
  </tr>
  <tr>
    <td>Nationality</td>
    <td>54.09</td>
    <td>47.8</td>
    <td>47.17</td>
    <td>53.46</td>
    <td>53.46</td>
    <td>50.31</td>
    <td>52.83</td>
  </tr>
  <tr>
    <td>Race</td>
    <td>53.29</td>
    <td>55.43</td>
    <td>56.2</td>
    <td>55.81</td>
    <td>60.08</td>
    <td>55.62</td>
    <td>56.78</td>
  </tr>
  <tr>
    <td>Socioeconomic</td>
    <td>55.81</td>
    <td>45.93</td>
    <td>47.67</td>
    <td>59.3</td>
    <td>58.14</td>
    <td>58.72</td>
    <td>58.14</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>54.58</td>
    <td>56.11</td>
    <td>55.73</td>
    <td>51.15</td>
    <td>55.73</td>
    <td>54.58</td>
    <td>54.58</td>
  </tr>
  <tr>
    <td>Sexual orientation</td>
    <td>70.24</td>
    <td>47.62</td>
    <td>52.38</td>
    <td>67.86</td>
    <td>79.76</td>
    <td>71.43</td>
    <td>70.24</td>
  </tr>
  <tr>
    <td>Age</td>
    <td>59.77</td>
    <td>39.08</td>
    <td>45.98</td>
    <td>51.72</td>
    <td>51.72</td>
    <td>47.13</td>
    <td>47.13</td>
  </tr>
  <tr>
    <td>Disability</td>
    <td>61.67</td>
    <td>43.33</td>
    <td>51.67</td>
    <td>75.0</td>
    <td>73.33</td>
    <td>75.0</td>
    <td>75.0</td>
  </tr>
  <tr>
    <td>Physical appearance</td>
    <td>55.56</td>
    <td>50.79</td>
    <td>49.21</td>
    <td>55.56</td>
    <td>63.49</td>
    <td>65.08</td>
    <td>65.08</td>
  </tr>
  <tr>
    <th>Bias (SS)</th>
    <th>SSS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>ΔP</th>
    <th>ΔPA</th>
  </tr>
  <tr>
    <td>Race</td>
    <td>58.42</td>
    <td>48.54</td>
    <td>48.86</td>
    <td>53.64</td>
    <td>59.36</td>
    <td>56.55</td>
    <td>57.07</td>
  </tr>
  <tr>
    <td>Profession</td>
    <td>62.47</td>
    <td>55.68</td>
    <td>55.06</td>
    <td>52.22</td>
    <td>62.1</td>
    <td>61.36</td>
    <td>61.36</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>61.57</td>
    <td>52.94</td>
    <td>52.94</td>
    <td>56.86</td>
    <td>63.92</td>
    <td>61.96</td>
    <td>61.18</td>
  </tr>
  <tr>
    <td>Religion</td>
    <td>64.56</td>
    <td>45.57</td>
    <td>46.84</td>
    <td>51.9</td>
    <td>63.29</td>
    <td>63.29</td>
    <td>59.49</td>
  </tr>
</table>

Table 10: Bias scores for biases in CPS (top) and SS dataset (bottom) with distilBERT<sub>P,unc</sub> as given by BSPT (Equation 9) using measures CRR, CRRA, ΔP, ΔPA, CSPS, SSS, AUL and AULA.

<table>
  <tr>
    <th colspan="8">MLM: distilRoBERTa<sub>P</sub></th>
  </tr>
  <tr>
    <th>Bias (CPS)</th>
    <th>CSPS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>ΔP</th>
    <th>ΔPA</th>
  </tr>
  <tr>
    <td>Religion</td>
    <td>71.43</td>
    <td>49.52</td>
    <td>44.76</td>
    <td>62.86</td>
    <td>64.76</td>
    <td>71.43</td>
    <td>72.38</td>
  </tr>
  <tr>
    <td>Nationality</td>
    <td>62.26</td>
    <td>54.72</td>
    <td>52.83</td>
    <td>54.09</td>
    <td>59.75</td>
    <td>59.12</td>
    <td>59.75</td>
  </tr>
  <tr>
    <td>Race</td>
    <td>56.59</td>
    <td>51.74</td>
    <td>50.78</td>
    <td>59.88</td>
    <td>64.73</td>
    <td>58.53</td>
    <td>59.5</td>
  </tr>
  <tr>
    <td>Socioeconomic</td>
    <td>61.63</td>
    <td>65.12</td>
    <td>70.93</td>
    <td>61.63</td>
    <td>66.86</td>
    <td>67.44</td>
    <td>67.44</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>53.05</td>
    <td>51.91</td>
    <td>49.24</td>
    <td>51.15</td>
    <td>54.58</td>
    <td>53.05</td>
    <td>53.05</td>
  </tr>
  <tr>
    <td>Sexual orientation</td>
    <td>65.48</td>
    <td>50.0</td>
    <td>41.67</td>
    <td>55.95</td>
    <td>64.29</td>
    <td>64.29</td>
    <td>63.1</td>
  </tr>
  <tr>
    <td>Age</td>
    <td>56.32</td>
    <td>49.43</td>
    <td>43.68</td>
    <td>52.87</td>
    <td>55.17</td>
    <td>51.72</td>
    <td>50.57</td>
  </tr>
  <tr>
    <td>Disability</td>
    <td>68.33</td>
    <td>63.33</td>
    <td>63.33</td>
    <td>66.67</td>
    <td>71.67</td>
    <td>63.33</td>
    <td>63.33</td>
  </tr>
  <tr>
    <td>Physical appearance</td>
    <td>61.9</td>
    <td>42.86</td>
    <td>42.86</td>
    <td>58.73</td>
    <td>53.97</td>
    <td>60.32</td>
    <td>53.97</td>
  </tr>
  <tr>
    <th>Bias (SS)</th>
    <th>SSS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>ΔP</th>
    <th>ΔPA</th>
  </tr>
  <tr>
    <td>Race</td>
    <td>58.11</td>
    <td>57.38</td>
    <td>56.86</td>
    <td>54.05</td>
    <td>60.5</td>
    <td>60.29</td>
    <td>60.29</td>
  </tr>
  <tr>
    <td>Profession</td>
    <td>61.36</td>
    <td>62.22</td>
    <td>61.85</td>
    <td>53.46</td>
    <td>59.63</td>
    <td>61.23</td>
    <td>61.6</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>71.76</td>
    <td>64.31</td>
    <td>63.53</td>
    <td>58.04</td>
    <td>61.96</td>
    <td>64.71</td>
    <td>62.35</td>
  </tr>
  <tr>
    <td>Religion</td>
    <td>68.35</td>
    <td>60.76</td>
    <td>56.96</td>
    <td>55.7</td>
    <td>65.82</td>
    <td>65.82</td>
    <td>68.35</td>
  </tr>
</table>

Table 11: Bias scores for biases in CPS (top) and SS dataset (bottom) with distilRoBERTa<sub>P</sub> as given by BSPT (Equation 9) using measures CRR, CRRA, ΔP, ΔPA, CSPS, SSS, AUL and AULA.
<table>
  <tr>
    <th>Measure \( f \)</th>
    <th>R1</th>
    <th>R2</th>
    <th>R3</th>
    <th>R4</th>
    <th>R5</th>
    <th>R6</th>
    <th>R7</th>
    <th>R8</th>
    <th>R1</th>
    <th>R2</th>
    <th>R3</th>
    <th>R4</th>
  </tr>
  <tr>
    <th colspan="13">RoBERTa<sub>P</sub></th>
  </tr>
  <tr>
    <th>CPS Dataset</th>
    <td colspan="12">SS Dataset</td>
  </tr>
  <tr>
    <td>CSPS</td>
    <td>Rel.</td><td>Dis.</td><td>Age</td><td>Nat.</td><td>Soc.</td><td>Ori.</td><td>Phy.</td><td>Gen.</td><td>Race</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>SSS</td>
    <td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>Gen.</td><td>Pro.</td><td>Rel.</td>
  </tr>
  <tr>
    <td>AUL</td>
    <td>Ori.</td><td>Dis.</td><td>Soc.</td><td>Nat.</td><td>Phy.</td><td>Age</td><td>Rel.</td><td>Gen.</td><td>Race</td><td>Gen.</td><td>Pro.</td><td>Race</td>
  </tr>
  <tr>
    <td>AULA</td>
    <td>Dis.</td><td>Ori.</td><td>Soc.</td><td>Age</td><td>Race</td><td>Nat.</td><td>Gen.</td><td>Rel.</td><td>Phy.</td><td>Gen.</td><td>Pro.</td><td>Race</td>
  </tr>
  <tr>
    <td>CRR</td>
    <td>Dis.</td><td>Rel.</td><td>Phy.</td><td>Soc.</td><td>Race</td><td>Age</td><td>Gen.</td><td>Nat.</td><td>Ori.</td><td>Gen.</td><td>Pro.</td><td>Race</td>
  </tr>
  <tr>
    <td>CRRA</td>
    <td>Dis.</td><td>Ori.</td><td>Rel.</td><td>Soc.</td><td>Race</td><td>Phy.</td><td>Nat.</td><td>Gen.</td><td>Age</td><td>Gen.</td><td>Pro.</td><td>Rel.</td>
  </tr>
  <tr>
    <td>\( \Delta P \)</td>
    <td>Dis.</td><td>Rel.</td><td>Ori.</td><td>Soc.</td><td>Race</td><td>Phy.</td><td>Gen.</td><td>Age</td><td>Nat.</td><td>Gen.</td><td>Rel.</td><td>Pro.</td>
  </tr>
  <tr>
    <td>\( \Delta_{PA} \)</td>
    <td>Dis.</td><td>Rel.</td><td>Ori.</td><td>Race</td><td>Soc.</td><td>Phy.</td><td>Gen.</td><td>Nat.</td><td>Age</td><td>Gen.</td><td>Rel.</td><td>Pro.</td>
  </tr>
  <tr>
    <th colspan="13">BERT<sup>P,unc</sup></th>
  </tr>
  <tr>
    <th>CPS Dataset</th>
    <td colspan="12">SS Dataset</td>
  </tr>
  <tr>
    <td>CSPS</td>
    <td>Rel.</td><td>Ori.</td><td>Phy.</td><td>Nat.</td><td>Dis.</td><td>Soc.</td><td>Race</td><td>Gen.</td><td>Age</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>SSS</td>
    <td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>Gen.</td><td>Pro.</td><td>Rel.</td>
  </tr>
  <tr>
    <td>AUL</td>
    <td>Rel.</td><td>Age</td><td>Nat.</td><td>Ori.</td><td>Race</td><td>Gen.</td><td>Soc.</td><td>Dis.</td><td>Phy.</td><td>Pro.</td><td>Gen.</td><td>Race</td>
  </tr>
  <tr>
    <td>AULA</td>
    <td>Rel.</td><td>Nat.</td><td>Ori.</td><td>Age</td><td>Race</td><td>Gen.</td><td>Dis.</td><td>Soc.</td><td>Phy.</td><td>Pro.</td><td>Race</td><td>Gen.</td>
  </tr>
  <tr>
    <td>CRR</td>
    <td>Dis.</td><td>Ori.</td><td>Phy.</td><td>Race</td><td>Age</td><td>Soc.</td><td>Rel.</td><td>Gen.</td><td>Nat.</td><td>Rel.</td><td>Pro.</td><td>Gen.</td>
  </tr>
  <tr>
    <td>CRRA</td>
    <td>Ori.</td><td>Dis.</td><td>Phy.</td><td>Rel.</td><td>Race</td><td>Soc.</td><td>Age</td><td>Gen.</td><td>Nat.</td><td>Rel.</td><td>Gen.</td><td>Pro.</td>
  </tr>
  <tr>
    <td>\( \Delta P \)</td>
    <td>Dis.</td><td>Ori.</td><td>Phy.</td><td>Rel.</td><td>Race</td><td>Soc.</td><td>Gen.</td><td>Age</td><td>Nat.</td><td>Rel.</td><td>Gen.</td><td>Pro.</td>
  </tr>
  <tr>
    <td>\( \Delta_{PA} \)</td>
    <td>Dis.</td><td>Phy.</td><td>Ori.</td><td>Soc.</td><td>Race</td><td>Rel.</td><td>Gen.</td><td>Age</td><td>Nat.</td><td>Gen.</td><td>Rel.</td><td>Pro.</td>
  </tr>
  <tr>
    <th colspan="13">distilRoBERTa<sub>P</sub></th>
  </tr>
  <tr>
    <th>CPS Dataset</th>
    <td colspan="12">SS Dataset</td>
  </tr>
  <tr>
    <td>CSPS</td>
    <td>Rel.</td><td>Ori.</td><td>Nat.</td><td>Phy.</td><td>Soc.</td><td>Race</td><td>Age</td><td>Gen.</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>SSS</td>
    <td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>Gen.</td><td>Rel.</td><td>Pro.</td>
  </tr>
  <tr>
    <td>AUL</td>
    <td>Soc.</td><td>Dis.</td><td>Nat.</td><td>Gen.</td><td>Race</td><td>Ori.</td><td>Rel.</td><td>Age</td><td>Phy.</td><td>Gen.</td><td>Pro.</td><td>Rel.</td>
  </tr>
  <tr>
    <td>AULA</td>
    <td>Soc.</td><td>Dis.</td><td>Nat.</td><td>Race</td><td>Gen.</td><td>Rel.</td><td>Age</td><td>Phy.</td><td>Ori.</td><td>Gen.</td><td>Pro.</td><td>Rel.</td>
  </tr>
  <tr>
    <td>CRR</td>
    <td>Dis.</td><td>Rel.</td><td>Soc.</td><td>Race</td><td>Phy.</td><td>Ori.</td><td>Nat.</td><td>Age</td><td>Gen.</td><td>Gen.</td><td>Rel.</td><td>Race</td>
  </tr>
  <tr>
    <td>CRRA</td>
    <td>Dis.</td><td>Soc.</td><td>Rel.</td><td>Race</td><td>Ori.</td><td>Nat.</td><td>Age</td><td>Gen.</td><td>Phy.</td><td>Rel.</td><td>Gen.</td><td>Race</td>
  </tr>
  <tr>
    <td>\( \Delta P \)</td>
    <td>Rel.</td><td>Soc.</td><td>Ori.</td><td>Dis.</td><td>Phy.</td><td>Nat.</td><td>Race</td><td>Gen.</td><td>Age</td><td>Rel.</td><td>Gen.</td><td>Pro.</td>
  </tr>
  <tr>
    <td>\( \Delta_{PA} \)</td>
    <td>Rel.</td><td>Soc.</td><td>Dis.</td><td>Ori.</td><td>Nat.</td><td>Race</td><td>Phy.</td><td>Gen.</td><td>Age</td><td>Rel.</td><td>Gen.</td><td>Pro.</td>
  </tr>
  <tr>
    <th colspan="13">distilBERT<sup>P,unc</sup></th>
  </tr>
  <tr>
    <th>CPS Dataset</th>
    <td colspan="12">SS Dataset</td>
  </tr>
  <tr>
    <td>CSPS</td>
    <td>Rel.</td><td>Ori.</td><td>Dis.</td><td>Age</td><td>Soc.</td><td>Phy.</td><td>Gen.</td><td>Nat.</td><td>Race</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>SSS</td>
    <td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>Rel.</td><td>Pro.</td><td>Gen.</td>
  </tr>
  <tr>
    <td>AUL</td>
    <td>Gen.</td><td>Race</td><td>Rel.</td><td>Phy.</td><td>Nat.</td><td>Ori.</td><td>Soc.</td><td>Dis.</td><td>Age</td><td>Pro.</td><td>Gen.</td><td>Race</td>
  </tr>
  <tr>
    <td>AULA</td>
    <td>Race</td><td>Gen.</td><td>Ori.</td><td>Rel.</td><td>Dis.</td><td>Phy.</td><td>Soc.</td><td>Nat.</td><td>Age</td><td>Pro.</td><td>Gen.</td><td>Race</td>
  </tr>
  <tr>
    <td>CRR</td>
    <td>Dis.</td><td>Ori.</td><td>Soc.</td><td>Race</td><td>Phy.</td><td>Rel.</td><td>Nat.</td><td>Age</td><td>Gen.</td><td>Gen.</td><td>Race</td><td>Pro.</td>
  </tr>
  <tr>
    <td>CRRA</td>
    <td>Ori.</td><td>Dis.</td><td>Rel.</td><td>Phy.</td><td>Race</td><td>Soc.</td><td>Gen.</td><td>Nat.</td><td>Age</td><td>Gen.</td><td>Rel.</td><td>Pro.</td>
  </tr>
  <tr>
    <td>\( \Delta P \)</td>
    <td>Dis.</td><td>Ori.</td><td>Rel.</td><td>Phy.</td><td>Soc.</td><td>Race</td><td>Gen.</td><td>Nat.</td><td>Age</td><td>Rel.</td><td>Gen.</td><td>Pro.</td>
  </tr>
  <tr>
    <td>\( \Delta_{PA} \)</td>
    <td>Dis.</td><td>Ori.</td><td>Rel.</td><td>Phy.</td><td>Soc.</td><td>Race</td><td>Gen.</td><td>Nat.</td><td>Age</td><td>Pro.</td><td>Gen.</td><td>Rel.</td>
  </tr>
</table>

Table 12: Relative bias category ranks for pretrained MLMs based on evaluation scores using measures CSPS, SSS, AUL, AULA, CRR, CRRA, \( \Delta P \) and \( \Delta_{PA} \) on CPS and SS datasets in Tables 8, 9, 10 and 11.
L  ROC Curves for Measures

ROC curves of measures AUL, AULA, CRR, CRRA, \( \Delta P \) and \( \Delta_{PA} \) for the classification task detailed in Section 4.4.1.

![ROC curve of AUL, AULA, CRR, CRRA, ΔP and ΔPA for MLM RoBERTa_P on CPS.](page_232_186_579_377.png)

Figure 4: ROC curve of AUL, AULA, CRR, CRRA, \( \Delta P \) and \( \Delta_{PA} \) for MLM RoBERTa_P on CPS.

![ROC curve of AUL, AULA, CRR, CRRA, ΔP and ΔPA for MLM BERT_P,une on CPS.](page_232_624_579_377.png)

Figure 5: ROC curve of AUL, AULA, CRR, CRRA, \( \Delta P \) and \( \Delta_{PA} \) for MLM BERT_P,une on CPS.
<table>
  <tr>
    <th>Model</th>
    <th>Set</th>
    <th>Example s</th>
    <th>CRR(s)</th>
    <th>CRRA(s)</th>
    <th>ΔP(s)</th>
    <th>ΔPA(s)</th>
  </tr>
  <tr>
    <td>distilBERT<sub>P,unc</sub></td>
    <td>S<sub>dis</sub></td>
    <td>ethiopian men are skinny</td>
    <td><b>0.862</b></td>
    <td><b>0.448</b></td>
    <td><b>4.842</b></td>
    <td><b>0.408</b></td>
  </tr>
  <tr>
    <td>distilBERT<sub>P,unc</sub></td>
    <td>S<sub>adv</sub></td>
    <td>ethiopian men are fat</td>
    <td>0.973</td>
    <td>0.536</td>
    <td>5.736</td>
    <td>0.482</td>
  </tr>
  <tr>
    <td>distilBERT<sub>P,unc</sub></td>
    <td>S<sub>dis</sub></td>
    <td>african-americans are needlessly confrontational</td>
    <td><b>0.167</b></td>
    <td><b>0.053</b></td>
    <td><b>0.131</b></td>
    <td><b>0.006</b></td>
  </tr>
  <tr>
    <td>distilBERT<sub>P,unc</sub></td>
    <td>S<sub>adv</sub></td>
    <td>white americans are needlessly confrontational</td>
    <td>0.389</td>
    <td>0.112</td>
    <td>1.237</td>
    <td>0.059</td>
  </tr>
</table>

Table 13: CRR(s), CRRA(s), ΔP(s) and ΔPA(s) likelihood scores for an example instance in SS (top) and CPS (bottom). Values closer to 0 for each of the measures is indicative of higher MLM preference towards \( s \in S_{adv} \) relative to \( s \in S_{dis} \) or vice versa. In this case the distilBERT<sub>P,unc</sub> model is (relatively) biased against disadvantaged groups for both inputs across all measures.

<table>
  <tr>
    <th>Model</th>
    <th>Set</th>
    <th>Example s</th>
    <th>CRR(s)</th>
    <th>CRRA(s)</th>
    <th>ΔP(s)</th>
    <th>ΔPA(s)</th>
  </tr>
  <tr>
    <td>distilBERT<sub>P,unc</sub></td>
    <td>S<sub>dis</sub></td>
    <td>he was a <i>black</i> man with a bad temper</td>
    <td>0.388</td>
    <td>0.096</td>
    <td>0.915</td>
    <td>0.044</td>
  </tr>
  <tr>
    <td>distilBERT<sub>P,unc</sub></td>
    <td>S<sub>adv</sub></td>
    <td>he was a <i>white</i> man with a bad temper</td>
    <td><b>0.314</b></td>
    <td><b>0.092</b></td>
    <td><b>0.881</b></td>
    <td><b>0.042</b></td>
  </tr>
  <tr>
    <td>distilBERT<sub>R,unc</sub></td>
    <td>S<sub>dis</sub></td>
    <td>he was a <i>black</i> man with a bad temper</td>
    <td>&lt;0.001</td>
    <td><b>0.044</b></td>
    <td>&lt;0.001</td>
    <td>&lt;0.001</td>
  </tr>
  <tr>
    <td>distilBERT<sub>R,unc</sub></td>
    <td>S<sub>adv</sub></td>
    <td>he was a <i>white</i> man with a bad temper</td>
    <td>0.093</td>
    <td>0.053</td>
    <td>0.295</td>
    <td>0.013</td>
  </tr>
</table>

Table 14: CRR, CRRA, ΔP and ΔPA with distilBERT<sub>P,unc</sub> and distilBERT<sub>R,unc</sub> for an example sentence pair in CPS, where the former has bias against advantaged groups and the latter against disadvantaged groups. The scores show a shift in MLM preference (measure values closer to 0) indicating greater relative bias against disadvantaged groups after retraining.
M  Bias Scores by Category for Retrained MLMs

M.1 Experiment Settings
We retrain MLMs with mlm_probability set to 0.15 (as in Devlin et al., 2019), using 80 percent of data for training and 20 percent for validation. MLMs were retrained for 30 epochs, reaching minimum validation loss at epoch 30.

M.2 Findings
Tables 15, 16, 17, and 18 report bias scores for measures CRR, CRRA, \( \Delta P \), \( \Delta PA \), CSPS, AUL, and AULA for retrained MLMs using BSRT (Equation 10). † indicates that the relative difference in proportions of bias between pre- and retrained transformers is statistically significant according to McNemar’s test (p-value < 0.05), using binarized outcomes for bias as given by \( f_R(S_{adv}) > f_R(S_{dis}) \) and \( f_P(S_{adv}) > f_P(S_{dis}) \) to create contingency tables of outcome pairings between pre- and retrained transformers to test for marginal homogeneity. Results are for MLMs retrained on \( S_{dis} \) (top; all sentences with bias against disadvantaged groups) and \( S_{adv} \) (bottom; all sentences with bias against advantaged groups) for biases in CPS.

For RoBERTaR retrained on \( S_{adv} \) from CPS, each measure gives scores below 50 as expected. However, AUL and AULA give insignificant results for physical appearance bias. CRR, CRRA, \( \Delta P \), and \( \Delta PA \) give significant results below 50 for each bias category. For BERTR retrained on \( S_{adv} \), AUL and AULA overestimate physical appearance, gender, disability, and socioeconomic biases and give insignificant results for each, while all proposed measures give significant results below 50 for each category as expected. Similarly, for distilRoBERTaR, AULA overestimates physical appearance bias and gives insignificant results for physical appearance, gender, and age, AUL gives insignificant results for physical appearance, and CSPS and AUL give insignificant results for age. AUL and AULA also give insignificant results for physical appearance, disability, and sexual orientation for distilBERTR retrained on \( S_{adv} \). Overall, we find that AUL, AULA, and CSPS overestimate physical appearance bias introduced by retraining MLMs compared to proposed measures and based on BSRT.

For all MLMs retrained on \( S_{dis} \), CRR, \( \Delta P \), and \( \Delta PA \) give results above 50 for each bias category as expected, with higher scores than other measures in almost every case. CRRA also gives results above 50 for each bias category except for disability bias using distilBERTR. For RoBERTaR, CSPS, and AULA give scores less than 50 for age bias. AULA also gives a score less than 50 for sexual orientation bias. In addition, proposed measures are significant for every bias type using BERTR and RoBERTaR. This is also true for distilBERTR (besides CRRA for sexual orientation) and distilRoBERTaR (besides CRRA for physical appearance and CRR for disability). In contrast, AUL, AULA, and CSPS have 5, 6, and 11 insignificant results respectively across all MLMs retrained on \( S_{dis} \), and AULA and CSPS give 3 and 4 bias scores below 50 respectively. Notably, AUL and CRRA give 1 bias score below 50. These are concerning underestimations of biases introduced by retraining MLMs only on sentences with biases against disadvantaged groups from CPS.
<table>
  <tr>
    <th colspan="8">MLM: RoBERTa<sub>R</sub></th>
  </tr>
  <tr>
    <th colspan="2">Retrain dataset: \( \forall s \in S_{\text{dis}} \) for CPS</th>
    <th>CSPS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>\( \Delta P \)</th>
    <th>\( \Delta_{PA} \)</th>
  </tr>
  <tr>
    <td>Religion</td>
    <td>56.19</td>
    <td>53.33 †</td>
    <td>55.24 †</td>
    <td>81.9 †</td>
    <td>77.14 †</td>
    <td><b>85.71</b> †</td>
    <td><b>85.71</b> †</td>
  </tr>
  <tr>
    <td>Nationality</td>
    <td>56.6 †</td>
    <td>62.26 †</td>
    <td>60.38 †</td>
    <td>78.62 †</td>
    <td>79.25 †</td>
    <td><b>89.94</b> †</td>
    <td>88.68 †</td>
  </tr>
  <tr>
    <td>Race</td>
    <td>60.47 †</td>
    <td>63.76 †</td>
    <td>56.4 †</td>
    <td>72.29 †</td>
    <td>70.54 †</td>
    <td><b>82.36</b> †</td>
    <td>80.23 †</td>
  </tr>
  <tr>
    <td>Socioeconomic</td>
    <td>57.56 †</td>
    <td>56.4 †</td>
    <td>49.42 †</td>
    <td>82.56 †</td>
    <td>76.16 †</td>
    <td><b>88.37</b> †</td>
    <td>87.21 †</td>
  </tr>
  <tr>
    <td>Disability</td>
    <td>58.33 †</td>
    <td>78.33 †</td>
    <td>70.0</td>
    <td>78.33 †</td>
    <td>76.67 †</td>
    <td>88.33 †</td>
    <td><b>90.0</b> †</td>
  </tr>
  <tr>
    <td>Physical Appearance</td>
    <td>50.79</td>
    <td>65.08 †</td>
    <td>69.84 †</td>
    <td>76.19 †</td>
    <td>73.02 †</td>
    <td><b>79.37</b> †</td>
    <td><b>79.37</b> †</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>61.07 †</td>
    <td>55.73 †</td>
    <td>56.11 †</td>
    <td>69.47 †</td>
    <td>68.32 †</td>
    <td><b>76.34</b> †</td>
    <td>74.81 †</td>
  </tr>
  <tr>
    <td>Sexual Orientation</td>
    <td>52.38</td>
    <td>51.19</td>
    <td>47.62</td>
    <td>71.43 †</td>
    <td>66.67 †</td>
    <td><b>83.33</b> †</td>
    <td><b>83.33</b> †</td>
  </tr>
  <tr>
    <td>Age</td>
    <td>45.98</td>
    <td>51.72</td>
    <td>47.13</td>
    <td>71.26 †</td>
    <td>74.71 †</td>
    <td>85.06 †</td>
    <td><b>87.36</b> †</td>
  </tr>
  <tr>
    <th colspan="2">Retrain dataset: \( \forall s \in S_{\text{adv}} \) for CPS</th>
    <th>CSPS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>\( \Delta P \)</th>
    <th>\( \Delta_{PA} \)</th>
  </tr>
  <tr>
    <td>Religion</td>
    <td>19.05 †</td>
    <td>21.9 †</td>
    <td>29.52 †</td>
    <td>22.86 †</td>
    <td>16.19 †</td>
    <td><b>12.38</b> †</td>
    <td>13.33 †</td>
  </tr>
  <tr>
    <td>Nationality</td>
    <td>25.16 †</td>
    <td>24.53 †</td>
    <td>32.08 †</td>
    <td>18.87 †</td>
    <td>21.38 †</td>
    <td><b>16.35</b> †</td>
    <td>17.61 †</td>
  </tr>
  <tr>
    <td>Race</td>
    <td>38.76 †</td>
    <td>26.16 †</td>
    <td>29.26 †</td>
    <td>14.92 †</td>
    <td>14.34 †</td>
    <td><b>11.05</b> †</td>
    <td>11.43 †</td>
  </tr>
  <tr>
    <td>Socioeconomic</td>
    <td>29.65 †</td>
    <td>22.09 †</td>
    <td>23.84 †</td>
    <td>19.77 †</td>
    <td>19.77 †</td>
    <td><b>11.63</b> †</td>
    <td>13.37 †</td>
  </tr>
  <tr>
    <td>Disability</td>
    <td>20.0 †</td>
    <td>18.33 †</td>
    <td>21.67 †</td>
    <td>16.67 †</td>
    <td>10.0 †</td>
    <td><b>1.67</b> †</td>
    <td>8.33 †</td>
  </tr>
  <tr>
    <td>Physical Appearance</td>
    <td>28.57 †</td>
    <td>33.33</td>
    <td>44.44</td>
    <td>25.4 †</td>
    <td>12.7 †</td>
    <td><b>7.94</b> †</td>
    <td>11.11 †</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>37.79 †</td>
    <td>33.59 †</td>
    <td>40.08 †</td>
    <td>28.63 †</td>
    <td><b>28.24</b> †</td>
    <td>28.63 †</td>
    <td>30.15 †</td>
  </tr>
  <tr>
    <td>Sexual Orientation</td>
    <td>23.81 †</td>
    <td>16.67 †</td>
    <td>23.81 †</td>
    <td>22.62 †</td>
    <td>25.0 †</td>
    <td><b>13.1</b> †</td>
    <td>14.29 †</td>
  </tr>
  <tr>
    <td>Age</td>
    <td>31.03 †</td>
    <td>25.29 †</td>
    <td>31.03 †</td>
    <td>16.09 †</td>
    <td>21.84 †</td>
    <td><b>12.64</b> †</td>
    <td>13.79 †</td>
  </tr>
</table>

Table 15: Bias scores for measures using BSRT (Equation 10) and RoBERTa<sub>R</sub>, where † indicates that the relative difference in proportions of bias between pre- and retrained transformers is statistically significant according to McNemar’s test. Color-coded from lightest to darkest, with lower values represented by lighter shades and higher values by darker shades, except for insignificant values, which are not color-coded. For MLMs retrained on \( S_{\text{dis}} \), bold values indicate the highest statistically significant bias score across all measures. For MLMs retrained on \( S_{\text{adv}} \), bold values indicate the lowest statistically significant bias score across all measures.
<table>
  <tr>
    <th colspan="8">MLM: distilRoBERTa<sub>R</sub></th>
  </tr>
  <tr>
    <th colspan="8">Retrain dataset: \( \forall s \in S_{dis} \) for CPS</th>
  </tr>
  <tr>
    <th>Bias (CPS)</th>
    <th>CSPS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>\( \Delta P \)</th>
    <th>\( \Delta PA \)</th>
  </tr>
  <tr>
    <td>Religion</td>
    <td>49.52</td>
    <td>76.19 †</td>
    <td>75.24 †</td>
    <td>75.24 †</td>
    <td>80.0 †</td>
    <td><b>85.71</b> †</td>
    <td><b>85.71</b> †</td>
  </tr>
  <tr>
    <td>Nationality</td>
    <td>57.23 †</td>
    <td>71.7 †</td>
    <td>70.44 †</td>
    <td>70.44 †</td>
    <td>71.07 †</td>
    <td><b>77.99</b> †</td>
    <td><b>77.99</b> †</td>
  </tr>
  <tr>
    <td>Race</td>
    <td>57.36 †</td>
    <td>73.84 †</td>
    <td>70.54 †</td>
    <td>72.48 †</td>
    <td>70.16 †</td>
    <td><b>79.65</b> †</td>
    <td>78.29 †</td>
  </tr>
  <tr>
    <td>Socioeconomic</td>
    <td>59.88 †</td>
    <td>74.42 †</td>
    <td>63.37 †</td>
    <td>81.4 †</td>
    <td>76.16 †</td>
    <td><b>85.47</b> †</td>
    <td><b>85.47</b> †</td>
  </tr>
  <tr>
    <td>Disability</td>
    <td>46.67</td>
    <td>80.0</td>
    <td>75.0</td>
    <td>56.67</td>
    <td>55.0 †</td>
    <td><b>80.0</b> †</td>
    <td>78.33 †</td>
  </tr>
  <tr>
    <td>Physical Appearance</td>
    <td>50.79</td>
    <td>76.19 †</td>
    <td>65.08 †</td>
    <td>71.43 †</td>
    <td>71.43</td>
    <td><b>80.95</b> †</td>
    <td><b>80.95</b> †</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>64.12 †</td>
    <td>65.27 †</td>
    <td>65.27 †</td>
    <td><b>72.52</b> †</td>
    <td>70.99 †</td>
    <td>71.37 †</td>
    <td>70.99 †</td>
  </tr>
  <tr>
    <td>Sexual Orientation</td>
    <td>57.14 †</td>
    <td>71.43 †</td>
    <td>72.62 †</td>
    <td>72.62 †</td>
    <td>72.62 †</td>
    <td>83.33 †</td>
    <td><b>86.9</b> †</td>
  </tr>
  <tr>
    <td>Age</td>
    <td>64.37 †</td>
    <td>71.26 †</td>
    <td>64.37 †</td>
    <td><b>73.56</b> †</td>
    <td>72.41 †</td>
    <td>72.41 †</td>
    <td>72.41 †</td>
  </tr>
  <tr>
    <th colspan="8">Retrain dataset: \( \forall s \in S_{adv} \) for CPS</th>
  </tr>
  <tr>
    <th>Bias (CPS)</th>
    <th>CSPS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>\( \Delta P \)</th>
    <th>\( \Delta PA \)</th>
  </tr>
  <tr>
    <td>Religion</td>
    <td>23.81 †</td>
    <td>25.71 †</td>
    <td>33.33 †</td>
    <td>20.0 †</td>
    <td><b>12.38</b> †</td>
    <td><b>12.38</b> †</td>
    <td><b>12.38</b> †</td>
  </tr>
  <tr>
    <td>Nationality</td>
    <td>27.04 †</td>
    <td>27.67 †</td>
    <td>33.33 †</td>
    <td>26.42 †</td>
    <td>22.64 †</td>
    <td><b>16.35</b> †</td>
    <td>19.5 †</td>
  </tr>
  <tr>
    <td>Race</td>
    <td>36.63 †</td>
    <td>35.47 †</td>
    <td>42.25 †</td>
    <td>21.12 †</td>
    <td>13.57 †</td>
    <td>12.4 †</td>
    <td><b>11.82</b> †</td>
  </tr>
  <tr>
    <td>Socioeconomic</td>
    <td>26.74 †</td>
    <td>28.49 †</td>
    <td>28.49 †</td>
    <td>21.51 †</td>
    <td>16.86 †</td>
    <td><b>11.63</b> †</td>
    <td><b>11.63</b> †</td>
  </tr>
  <tr>
    <td>Disability</td>
    <td>20.0 †</td>
    <td>41.67 †</td>
    <td>38.33 †</td>
    <td>18.33 †</td>
    <td>10.0 †</td>
    <td>5.0 †</td>
    <td>6.67 †</td>
  </tr>
  <tr>
    <td>Physical Appearance</td>
    <td>23.81 †</td>
    <td>47.62</td>
    <td>50.79</td>
    <td>26.98 †</td>
    <td><b>15.87</b> †</td>
    <td>19.05 †</td>
    <td>19.05 †</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>38.93 †</td>
    <td>37.4 †</td>
    <td>39.69</td>
    <td>40.46 †</td>
    <td>30.15 †</td>
    <td><b>25.95</b> †</td>
    <td>27.86 †</td>
  </tr>
  <tr>
    <td>Sexual Orientation</td>
    <td>22.62 †</td>
    <td>45.24 †</td>
    <td>47.62 †</td>
    <td>27.38 †</td>
    <td>22.62 †</td>
    <td><b>14.29</b> †</td>
    <td><b>14.29</b> †</td>
  </tr>
  <tr>
    <td>Age</td>
    <td>36.78</td>
    <td>28.74</td>
    <td>35.63</td>
    <td>26.44 †</td>
    <td>27.59 †</td>
    <td><b>14.94</b> †</td>
    <td>16.09 †</td>
  </tr>
</table>

Table 16: Bias scores for measures using BSRT (Equation 10) and distilRoBERTa<sub>R</sub>, where † indicates that the relative difference in proportions of bias between pre- and retrained transformers is statistically significant according to McNemar’s test. Color-coded from lightest to darkest, with lower values represented by lighter shades and higher values by darker shades, except for insignificant values, which are not color-coded. For MLMs retrained on \( S_{dis} \), bold values indicate the highest statistically significant bias score across all measures. For MLMs retrained on \( S_{adv} \), bold values indicate the lowest statistically significant bias score across all measures.
<table>
  <tr>
    <th colspan="8">MLM: BERT<sub>R</sub></th>
  </tr>
  <tr>
    <th colspan="2">Retrain dataset: \( \forall s \in S_{\text{dis}} \) for CPS</th>
    <th>CSPS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>\( \Delta P \)</th>
    <th>\( \Delta PA \)</th>
  </tr>
  <tr>
    <td>Religion</td>
    <td>51.43</td>
    <td>44.76</td>
    <td>48.57</td>
    <td>62.86 †</td>
    <td>62.86 †</td>
    <td><b>77.14</b> †</td>
    <td>74.29 †</td>
  </tr>
  <tr>
    <td>Nationality</td>
    <td>66.04 †</td>
    <td>55.97 †</td>
    <td>54.09 †</td>
    <td>76.1 †</td>
    <td>74.21 †</td>
    <td><b>77.99</b> †</td>
    <td>77.36 †</td>
  </tr>
  <tr>
    <td>Race</td>
    <td>59.11 †</td>
    <td>59.69 †</td>
    <td>58.53 †</td>
    <td>78.49 †</td>
    <td>71.32 †</td>
    <td><b>83.33</b> †</td>
    <td>81.01 †</td>
  </tr>
  <tr>
    <td>Socioeconomic</td>
    <td>65.7 †</td>
    <td>69.19 †</td>
    <td>69.19 †</td>
    <td>79.07 †</td>
    <td>67.44 †</td>
    <td><b>79.65</b> †</td>
    <td>77.91 †</td>
  </tr>
  <tr>
    <td>Disability</td>
    <td>56.67 †</td>
    <td>65.0 †</td>
    <td>66.67 †</td>
    <td>71.67 †</td>
    <td>58.33 †</td>
    <td><b>76.67</b> †</td>
    <td>73.33 †</td>
  </tr>
  <tr>
    <td>Physical Appearance</td>
    <td>46.03 †</td>
    <td>76.19 †</td>
    <td>71.43 †</td>
    <td>76.19 †</td>
    <td>71.43 †</td>
    <td><b>84.13</b> †</td>
    <td><b>84.13</b> †</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>65.27 †</td>
    <td>57.63 †</td>
    <td>60.69 †</td>
    <td>66.41 †</td>
    <td>62.98 †</td>
    <td><b>72.52</b> †</td>
    <td>71.76 †</td>
  </tr>
  <tr>
    <td>Sexual Orientation</td>
    <td>60.71 †</td>
    <td>54.76 †</td>
    <td>57.14 †</td>
    <td>82.14 †</td>
    <td>66.67 †</td>
    <td><b>89.29</b> †</td>
    <td><b>89.29</b> †</td>
  </tr>
  <tr>
    <td>Age</td>
    <td>57.47 †</td>
    <td>52.87</td>
    <td>52.87</td>
    <td>73.56 †</td>
    <td>70.11 †</td>
    <td><b>90.8</b> †</td>
    <td>88.51 †</td>
  </tr>
  <tr>
    <th colspan="2">Retrain dataset: \( \forall s \in S_{\text{adv}} \) for CPS</th>
    <th>CSPS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>\( \Delta P \)</th>
    <th>\( \Delta PA \)</th>
  </tr>
  <tr>
    <td>Religion</td>
    <td>23.81 †</td>
    <td>31.43 †</td>
    <td>35.24 †</td>
    <td>16.19 †</td>
    <td>12.38 †</td>
    <td><b>3.81</b> †</td>
    <td>6.67 †</td>
  </tr>
  <tr>
    <td>Nationality</td>
    <td>25.79 †</td>
    <td>39.62 †</td>
    <td>39.62 †</td>
    <td>22.01 †</td>
    <td>20.75 †</td>
    <td><b>13.21</b> †</td>
    <td>16.98 †</td>
  </tr>
  <tr>
    <td>Race</td>
    <td>31.78 †</td>
    <td>46.12 †</td>
    <td>47.48 †</td>
    <td>16.47 †</td>
    <td>17.83 †</td>
    <td><b>13.95</b> †</td>
    <td>15.31 †</td>
  </tr>
  <tr>
    <td>Socioeconomic</td>
    <td>30.23 †</td>
    <td>52.33</td>
    <td>55.23</td>
    <td>12.79 †</td>
    <td>16.28 †</td>
    <td><b>12.21</b> †</td>
    <td>14.53 †</td>
  </tr>
  <tr>
    <td>Disability</td>
    <td>20.0 †</td>
    <td>53.33</td>
    <td>51.67</td>
    <td>20.0 †</td>
    <td>10.0 †</td>
    <td><b>10.0</b> †</td>
    <td>13.33 †</td>
  </tr>
  <tr>
    <td>Physical Appearance</td>
    <td>22.22 †</td>
    <td>63.49</td>
    <td>61.9</td>
    <td>17.46 †</td>
    <td>7.94 †</td>
    <td><b>6.35</b> †</td>
    <td><b>6.35</b> †</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>33.21 †</td>
    <td>50.0</td>
    <td>53.44</td>
    <td>29.39 †</td>
    <td>24.05 †</td>
    <td><b>19.85</b> †</td>
    <td>23.66 †</td>
  </tr>
  <tr>
    <td>Sexual Orientation</td>
    <td>22.62 †</td>
    <td>48.81</td>
    <td>47.62</td>
    <td>14.29 †</td>
    <td>9.52 †</td>
    <td><b>7.14</b> †</td>
    <td>8.33 †</td>
  </tr>
  <tr>
    <td>Age</td>
    <td>32.18 †</td>
    <td>44.83 †</td>
    <td>49.43 †</td>
    <td>20.69 †</td>
    <td>22.99 †</td>
    <td><b>14.94</b> †</td>
    <td>18.39 †</td>
  </tr>
</table>

Table 17: Bias scores for measures using BSRT (Equation 10) and BERT<sub>R</sub>, where † indicates that the relative difference in proportions of bias between pre- and retrained transformers is statistically significant according to McNemar’s test. Color-coded from lightest to darkest, with lower values represented by lighter shades and higher values by darker shades, except for insignificant values, which are not color-coded. For MLMs retrained on \( S_{\text{dis}} \), bold values indicate the highest statistically significant bias score across all measures. For MLMs retrained on \( S_{\text{adv}} \), bold values indicate the lowest statistically significant bias score across all measures.
<table>
  <tr>
    <th colspan="8">MLM: distilBERT<sub>R</sub></th>
  </tr>
  <tr>
    <th colspan="2">Retrain dataset: \( \forall s \in S_{\text{dis}} \) for CPS</th>
    <th>CSPS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>\( \Delta P \)</th>
    <th>\( \Delta PA \)</th>
  </tr>
  <tr>
    <td>Bias (CPS)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Religion</td>
    <td>61.9 †</td>
    <td>68.57 †</td>
    <td>69.52 †</td>
    <td>75.24 †</td>
    <td>72.38 †</td>
    <td><b>90.48</b> †</td>
    <td>87.62 †</td>
  </tr>
  <tr>
    <td>Nationality</td>
    <td>69.81 †</td>
    <td>61.64 †</td>
    <td>62.89 †</td>
    <td>74.84 †</td>
    <td>84.28 †</td>
    <td><b>90.57</b> †</td>
    <td>89.31 †</td>
  </tr>
  <tr>
    <td>Race</td>
    <td>65.7 †</td>
    <td>62.6 †</td>
    <td>61.05 †</td>
    <td>77.71 †</td>
    <td>69.77 †</td>
    <td><b>87.21</b> †</td>
    <td>85.27 †</td>
  </tr>
  <tr>
    <td>Socioeconomic</td>
    <td>65.12 †</td>
    <td>66.86 †</td>
    <td>63.95 †</td>
    <td>75.0 †</td>
    <td>70.93 †</td>
    <td><b>85.47</b> †</td>
    <td>84.3 †</td>
  </tr>
  <tr>
    <td>Disability</td>
    <td>51.67</td>
    <td><b>80.0</b> †</td>
    <td>73.33 †</td>
    <td>58.33</td>
    <td>46.67 †</td>
    <td><b>80.0</b> †</td>
    <td>73.33 †</td>
  </tr>
  <tr>
    <td>Physical Appearance</td>
    <td>66.67 †</td>
    <td>68.25 †</td>
    <td>68.25 †</td>
    <td><b>85.71</b> †</td>
    <td>71.43 †</td>
    <td>82.54 †</td>
    <td>79.37 †</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>69.08 †</td>
    <td>53.82 †</td>
    <td>53.82 †</td>
    <td>68.7 †</td>
    <td>71.37 †</td>
    <td>80.92 †</td>
    <td><b>81.3</b> †</td>
  </tr>
  <tr>
    <td>Sexual Orientation</td>
    <td>58.33</td>
    <td>63.1 †</td>
    <td>59.52 †</td>
    <td>77.38 †</td>
    <td>52.38</td>
    <td><b>83.33</b> †</td>
    <td>79.76 †</td>
  </tr>
  <tr>
    <td>Age</td>
    <td>57.47</td>
    <td>66.67 †</td>
    <td>56.32 †</td>
    <td>75.86 †</td>
    <td>73.56 †</td>
    <td><b>85.06</b> †</td>
    <td><b>85.06</b> †</td>
  </tr>
  <tr>
    <th colspan="2">Retrain dataset: \( \forall s \in S_{\text{adv}} \) for CPS</th>
    <th>CSPS</th>
    <th>AUL</th>
    <th>AULA</th>
    <th>CRR</th>
    <th>CRRA</th>
    <th>\( \Delta P \)</th>
    <th>\( \Delta PA \)</th>
  </tr>
  <tr>
    <td>Bias (CPS)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Religion</td>
    <td>25.71 †</td>
    <td>38.1 †</td>
    <td>40.0 †</td>
    <td>22.86 †</td>
    <td><b>11.43</b> †</td>
    <td><b>11.43</b> †</td>
    <td>12.38 †</td>
  </tr>
  <tr>
    <td>Nationality</td>
    <td>27.04 †</td>
    <td>35.85 †</td>
    <td>35.22 †</td>
    <td>20.13 †</td>
    <td>17.61 †</td>
    <td><b>14.47</b> †</td>
    <td>15.09 †</td>
  </tr>
  <tr>
    <td>Race</td>
    <td>34.11 †</td>
    <td>35.66 †</td>
    <td>35.47 †</td>
    <td>17.44 †</td>
    <td>16.09 †</td>
    <td>13.76 †</td>
    <td><b>12.98</b> †</td>
  </tr>
  <tr>
    <td>Socioeconomic</td>
    <td>26.16 †</td>
    <td>49.42</td>
    <td>44.77</td>
    <td>19.19 †</td>
    <td>18.6 †</td>
    <td>13.95 †</td>
    <td><b>13.37</b> †</td>
  </tr>
  <tr>
    <td>Disability</td>
    <td>23.33 †</td>
    <td>46.67</td>
    <td>40.0</td>
    <td>11.67 †</td>
    <td><b>6.67</b> †</td>
    <td>8.33 †</td>
    <td>8.33 †</td>
  </tr>
  <tr>
    <td>Physical Appearance</td>
    <td>25.4 †</td>
    <td>47.62</td>
    <td>47.62</td>
    <td>30.16</td>
    <td>20.63 †</td>
    <td><b>12.7</b> †</td>
    <td>14.29 †</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>33.59 †</td>
    <td>41.22 †</td>
    <td>39.69 †</td>
    <td>32.82 †</td>
    <td>26.34 †</td>
    <td>19.47 †</td>
    <td><b>18.7</b> †</td>
  </tr>
  <tr>
    <td>Sexual Orientation</td>
    <td>17.86 †</td>
    <td>47.62</td>
    <td>42.86</td>
    <td>14.29 †</td>
    <td>13.1 †</td>
    <td><b>7.14</b> †</td>
    <td><b>7.14</b> †</td>
  </tr>
  <tr>
    <td>Age</td>
    <td>24.14 †</td>
    <td>49.43 †</td>
    <td>48.28 †</td>
    <td>24.14 †</td>
    <td>18.39 †</td>
    <td><b>17.24</b> †</td>
    <td>19.54 †</td>
  </tr>
</table>

Table 18: Bias scores for measures using BSRT (Equation 10) and distilBERT<sub>R</sub>, where † indicates that the relative difference in proportions of bias between pre- and retrained transformers is statistically significant according to McNemar’s test. Color-coded from lightest to darkest, with lower values represented by lighter shades and higher values by darker shades, except for insignificant values, which are not color-coded. For MLMs retrained on \( S_{\text{dis}} \), bold values indicate the highest statistically significant bias score across all measures. For MLMs retrained on \( S_{\text{adv}} \), bold values indicate the lowest statistically significant bias score across all measures.
<table>
  <tr>
    <th rowspan="2">Race Bias Model</th>
    <th colspan="2">\( f = \mathrm{CRR}(s) \)</th>
    <th colspan="2">\( f = \mathrm{CRRA}(s) \)</th>
    <th colspan="2">\( f = \Delta P(s) \)</th>
    <th colspan="2">\( f = \Delta PA(s) \)</th>
  </tr>
  <tr>
    <th>CPS</th>
    <th>SS</th>
    <th>CPS</th>
    <th>SS</th>
    <th>CPS</th>
    <th>SS</th>
    <th>CPS</th>
    <th>SS</th>
  </tr>
  <tr>
    <td>BERT<sub>P,unc</sub></td>
    <td>0.016</td>
    <td>0.011</td>
    <td>0.004</td>
    <td>0.009</td>
    <td>0.111</td>
    <td>0.168 †</td>
    <td>0.006</td>
    <td>0.011</td>
  </tr>
  <tr>
    <td>RoBERTa<sub>P</sub></td>
    <td>0.023 †</td>
    <td>0.017</td>
    <td>0.004 †</td>
    <td>0.007 †</td>
    <td>0.14 †</td>
    <td>0.217 †</td>
    <td>0.006 †</td>
    <td>0.01 †</td>
  </tr>
  <tr>
    <td>distilBERT<sub>P,unc</sub></td>
    <td>0.013</td>
    <td>0.008</td>
    <td>0.004</td>
    <td>0.01 †</td>
    <td>0.053</td>
    <td>0.166 †</td>
    <td>0.003</td>
    <td>0.011 †</td>
  </tr>
  <tr>
    <td>distilRoBERTa<sub>P</sub></td>
    <td>0.024 †</td>
    <td>0.014</td>
    <td>0.005</td>
    <td>0.007 †</td>
    <td>0.113 †</td>
    <td>0.166 †</td>
    <td>0.005 †</td>
    <td>0.008 †</td>
  </tr>
</table>

Table 19: Significance results for the difference between measure means with a two-tailed Welch’s t-test. We report the mean difference \( \frac{1}{N} \sum_{i=1}^{N} f(S_{adv}(i)) - f(S_{dis}(i)) \) in measure \( f \) between sentence sets \( S_{adv} \) and \( S_{dis} \) for pretrained transformers. † indicates that the difference between the means \( \frac{1}{N} \sum_{i=1}^{N} f(S_{adv}(i)) \) and \( \frac{1}{N} \sum_{i=1}^{N} f(S_{dis}(i)) \) for a transformer is statistically significant according to the two-tailed Welch’s t-test (p-value < 0.05), where \( N \) is the total number of sentences and equal across sentence sets \( S_{adv} \) and \( S_{dis} \) within bias categories on CPS and SS datasets.

Reviewer paper summaries: The authors propose several measures for social biases in masked language models (MLMs) based on the notion of measuring bias through a model's relative preference for masked tokens between sentence pairs with social biases. Moreover, they extend their measure to compute the relative change in model bias for MLMs re-trained using the masked language modeling objective. They find that their proposed measures are more accurate than previous ones, which either over- or under-estimate the biases present, which was measured by comparison with human-annotated bias datasets.

This paper measures social biases in masked language models. It proposes proxy functions to evaluate bias through iterative masking experiments. The research assesses these biases by examining the relative preferences of MLMs towards disadvantaged and advantaged groups.

The paper suggests a novel approach to measuring bias by utilizing attention information in Transformers, focusing particularly on Masked Language Models (MLMs). By purposefully retraining MLMs on a bias/anti-bias dataset, the authors evaluate how effectively the proposed measurement assesses the bias embedded within these models.
Summary of strengths: The paper is very clearly written, organized, and I really appreciate how all equations, notation, and metrics are explained in plain language.  The proposed metrics are novel and the empirical results in bias evaluation show that one or more outperform existing metrics (outperform = more accurate measurement). I also really appreciate the clear examples in appendices I and J to make the task more concrete.

1. The paper introduces innovative proxy functions such as CRRA and ΔPA for measuring the quality of MLM predictions as a proxy for bias evaluation.

2. The authors compare pre-trained and re-trained models, and show how re-training can amplify or reduce biases. The proposed BSRT metric (Bias Score for MLM ReTraining) quantifies biases introduced by re-training. This can be useful for assessing the effectiveness of de-biasing strategies or understanding the impact of training data modifications.

3. The new proxy metrics - CRRA and ΔPA consider attention-weighted prediction quality to assess model biases - it better reflects the contextual influence of words on MLM predictions.

The strengths of this paper are consistent with previous versions:

1. The authors introduce innovative metrics for measuring social bias by incorporating attention mechanisms into existing social bias measurement techniques, utilizing established bias datasets.
2. The experimental setup for retraining Masked Language Models (MLMs), which involves evaluating the change in each metric's score during the training of MLMs on biased datasets, is intuitive and the results demonstrate that the proposed metrics are well aligned with human compared to the baseline.
Summary of weaknesses: I am not too familiar with the use of MLMs in recent years as transformers are more popular now. So I am just curious whether the proposed metrics and the whole paradigm of re-training the MLMs under the MLMO is something realistically useful going forward in the field, either as-is or as a basis for future work to extend these metrics to more current models? I understand that having accurate and consistent bias measurements for models is important, but I can't find anywhere in the paper that fully motivates why they focus on MLMs and the MLMO as opposed to anything else, which makes me uncertain of the significance of the results.

1. Equations 6 and 7 aggregate token-level metrics over all tokens in a sentence to compute a sentence-level score by averaging them. This approach would assume all tokens contribute equally to the overall bias in the sentence. But different tokens carry different weights in bias formation.

2. The model’s prediction for a masked token depends on the surrounding unmasked tokens. When multiple tokens are iteratively masked, the context changes with each masking. What distortions in the results did the authors encounter due to this?

Compared to the previous version, there are still unresolved weaknesses:

1. The overall writing and organization of the paper still make it challenging to grasp the paper’s core contributions.
    - I believe that introducing novel bias measurements is one of the key contributions of the paper. However, the details of the proposed measurements are located in Section 3.4, under Experiments and Findings. The authors should dedicate a separate section to methodology to elaborate on the method in detail.
    - I see BSPT itself as not being a strong innovation because it merely uses existing measurements and aggregates results across the dataset.
2. The motivation and rationale for leveraging attention information have not been adequately explained. Moreover, the authors mention “non-attention-weighted measures” in line 306, while in all other parts of the paper, they state that their method uses attention. It’s difficult to understand how and why attention is introduced.
3. According to the retrained MLM scores (Tables 15~18), the proposed measurements (i.e., CRRA and $\Delta$PA) did not consistently outperform other baselines. Specifically, the differences compared to CRR and $\Delta$P seem marginal.
Comments, suggestions and typos: n/a

1. Consider explaining why averaging attention scores in CRRA is used. Is there a risk that attention scores may overfit to high-frequency tokens or certain token types?

2. Test the proposed metrics using alternative masking strategies like masking multiple tokens or using random masking.

1. For Tables 15~18, it's better to highlight the highest scores in bold for each row.
2. I believe the results of the bias scores for retrained MLMs are significant enough compared to other results. It would be better to summarize them and include them in the main paper.
Overall assessment scores: 3.5

3

3
Metareview score: 4
Metareview: The paper introduces a novel method for measuring social biases in Masked Language Models (MLMs) by leveraging attention mechanisms within Transformers. The authors propose using proxy functions to evaluate bias through iterative masking experiments, which assess the models' relative preferences for masked tokens in sentence pairs that reflect social biases. By retraining MLMs on bias/anti-bias datasets, they extend their approach to measure changes in model bias, finding that their proposed metrics offer more accurate assessments compared to previous methods, which often over- or under-estimate biases. The effectiveness of these measures is validated against human-annotated bias datasets, demonstrating their improved precision in capturing embedded biases.
Summary of reasons to publish: The paper presents an innovative approach to measuring bias in MLMs by incorporating attention mechanisms and context-aware proxy metrics such as CRRA and ΔPA. The authors demonstrate the impact of retraining on bias, with the BSRT metric effectively capturing these changes. The results show strong alignment with human-annotated bias data and surpass existing metrics in accuracy.