SlowGuess/NAACL_2022_OCR huggingface dataset:
John P. Lalor, $^{1,2}$  Yi Yang, $^{3}$ , Kendall Smith, $^{1,2}$  Nicole Forsgren, $^{4}$  Ahmed Abbasi $^{1,2}$
<sup>1</sup> Human-centered Analytics Lab, University of Notre Dame
$^{2}$  Department of IT, Analytics, and Operations, University of Notre Dame
$^{3}$  Department of Information Systems and Operations Management, HKUST
4 Microsoft Research
{john.lalor, ksmith77, aabbasi} @ nd.edu, imyiyang@ust.hk, nicole.forsgren@microsoft.com
There has been a recent wave of work assessing the fairness of machine learning models in general, and more specifically, on natural language processing (NLP) models built using machine learning techniques. While much work has highlighted biases embedded in state-of-the-art language models, and more recent efforts have focused on how to debias, research assessing the fairness and performance of biased/debiased models on downstream prediction tasks has been limited. Moreover, most prior work has emphasized bias along a single dimension such as gender or race. In this work, we benchmark multiple NLP models with regards to their fairness and predictive performance across a variety of NLP tasks. In particular, we assess intersectional bias - fairness across multiple demographic dimensions. The results show that while current debiasing strategies fare well in terms of the fairness-accuracy trade-off (generally preserving predictive power in debiased models), they are unable to effectively alleviate bias in downstream tasks. Furthermore, this bias is often amplified across demographic dimensions. We conclude with implications for future NLP debiasing research.
As state-of-the-art natural language processing (NLP) language models become increasingly powerful and pervasive, recent progress in NLP has underscored the need for deeper analyses of how such models perform with respect to underrepresented groups. Research on fairness in NLP has shown that distributed representations of words often encode stereotypes - particularly towards different demographic groups (Blodgett et al., 2020; Bender et al., 2021). There is a growing stream of research that looks at mitigating these biases, especially when it manifests in the learned embedding state (Bolukbasi et al., 2016; Zmigrod et al., 2019; Kaneko and Bollegala, 2021). While prior work
has undoubtedly moved the needle, recent surveys and research articles have identified several important gaps and issues (Blodgett et al., 2020; Tan and Celis, 2019). First, much of the current work on examining NLP bias (and proposing debiasing strategies) has focused on representational harm - how a model describes certain groups, including stereotyping and other misrepresentations (Blodgett et al., 2020; Suresh and Guttag, 2019). Conversely, there has been far less work exploring allocational harm in downstream NLP prediction tasks - when a system distributes resources or opportunities differently (Blodgett et al., 2020; Suresh and Guttag, 2019). Downstream tasks, such as sequence classification, also affect underrepresented groups, as these models show disparate impact on various demographic subsets, including women, African Americans, and the elderly (Blodgett et al., 2020; Bender et al., 2021; Shah et al., 2020).
Second, there has been limited work that examines intersectional bias across a wide array of relevant characteristics, including several demographic dimensions, for a variety of non-debiased and debiased embeddings, $^{1}$  on a multitude of downstream tasks. Some work has studied demographic intersections such as young men and old women from a theoretical perspective (e.g., Kearns et al., 2018). Other recent studies have empirically shown that the biases inherent in language models for gender and race intersections might exceed those observed for gender and race alone (Tan and Celis, 2019), and that only debiasing along a single dimension can be problematic (Subramanian et al., 2021). Based on these two gaps, there is a need for a more systematic analysis of how current state-of-the-art language models and mitigation strategies perform with regards to intersectional bias in down-
In this work, our scope is debiasing embeddings, not debiasing classifiers. While there is much work in the area of debiasing classifiers, here we restrict our focus to the debiasing of embeddings.
Human Language Technologies, pages 3598 - 3609
Figure 1: Overview of our fairness benchmarking analyses. We benchmark performance across datasets, models, and debiasing methods for tasks involving multiple demographic variables.
stream tasks.
Accordingly, in this study we perform a broad benchmark analysis of intersectional bias (Figure 1) encompassing the following key characteristics:
- Benchmark analysis on ten downstream sequence classification tasks related to five datasets that span common modes of user-generated content: Twitter, forums, Reddit, and survey responses. For these tasks, we also note the allocational harm implications of disparate impact, namely the harm associated with biased NLP-guided interventions.
- Inclusion of five demographic dimensions: gender, race, age, education, and income. Having three or more dimensions on many of the tasks affords opportunities to examine bias for various demographic intersection subgroups in a more in-depth manner. On four of the datasets, these demographics are self-reported as opposed to being algorithmically or heuristically inferred - an important consideration for debiasing research.
- Evaluation of three prominent word embeddings, BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and GloVe (Pennington et al., 2014), and four state-of-the-art model debiasing methods (Ravfogel et al., 2020; Kaneko and Bollegala, 2021; Zmigrod et al., 2019; Webster et al., 2020). This allows us to draw empirical insights regarding the effectiveness of mitigation strategies for downstream tasks.
Our results show that existing debiasing methods are generally very adept at preserving predictive power in downstream tasks. However, their ability to mitigate intersectional bias in such tasks is limited. In general, debiasing BERT/RoBERTa only incrementally alleviates disparate impact of model classifications. Further, while gender bias alone has disparate impact rates of  $5 - 10\%$  or less on most tasks, the range of bias is amplified for intersections - with unfairness rates often being 20 to  $50\%$  higher. On tasks such as inferring personality traits, literacy, or numeracy of users, these debiased models are still outside the fairness ranges recommended by governing bodies (Barocas and Selfbst, 2016). Interestingly, these biases are more pronounced in models using GloVe, suggesting that debiased transformer-based models generally have better predictive power, and are fairer.
Our main contributions are two-fold. First, we perform a large-scale examination of intersectional bias across an array of downstream tasks. Our benchmark evaluation offers empirical evidence that the concerns voiced in recent critical surveys about too much emphasis on representational debiasing devoid of explicit normative goals (Blodgett et al., 2020), relative to mitigation of downstream allocational harm, are well-founded. Second, we quantify the size and scope of the intersectional bias problem, and the risks it can introduce for select underprivileged sub-groups when deploying NLP models for sequence prediction tasks. We are hopeful our work will spur future research that further sheds light on intersectional biases in down-
stream tasks, as well as mitigation strategies for alleviating locational harm. Towards this goal, the code and data used in this work is publicly available via GitHub.[2]
2.1 Allocational and Representational Harms
In their survey on bias in NLP, Blodgett et al. (2020) drew a distinction between allocational and representational harms. They found that most papers in NLP describe methods for measuring and mitigating representational harms - when "a system (e.g., a search engine) represents some social groups in a less favorable light than others, demeans them, or fails to recognize their existence altogether" (Blodgett et al., 2020). One well-known example are stereotypes in word embeddings, such as certain ethnic groups being more closely associated with "housekeeper" (Garg et al., 2018).
In contrast, (Blodgett et al., 2020) only found four papers in their survey that were classified as having techniques for measuring/mitigating allocational harms - these "arise when an automated system allocates resources (e.g., credit) or opportunities (e.g., jobs) unfairly to different social groups." Allocational harm is often aligned with downstream tasks/interventions guided by the NLP model. For instance, all four of the aforementioned allocational harm papers measure and/or mitigate gender bias with respect to an NLP-based occupation classifier (De-Arteaga et al., 2019; Prost et al., 2019; Romanov et al., 2019; Zhao et al., 2020). More specifically, these studies examine the allocational harm of biased occupation classification predictions on decisions that affect humans, specifically whether an HR NLP system scraping web bios classifies individuals as relevant or not for a position. Our work builds on the nascent allocational harm literature by examining ten downstream tasks related to five data sets spanning Twitter, Reddit, forum, and survey response text.
2.2 Intersectional Biases
Intersectional biases arising as a result of interacting demographics have been studied in the broader machine learning literature, either from a theoretical perspective (Kearns et al., 2018; Yang et al., 2020), or in the context of facial recognition (Buolamwini and Gebru, 2018). In NLP, Tan and Celis (2019) evaluate and reveal important intersectional
biases in contextualized word embedding models such as BERT and GPT-2. However, in their study, intersectional biases are evaluated using the word association test with an emphasis on representational harm - it remains unclear how intersectional biases affect allocational harm in downstream NLP tasks. Subramanian et al. (2021) looked at intersectional biases of classification models specifically designed for unbiased prediction, but do not evaluate embedding debiasing techniques. We build on the emergent literature on intersectional biases by assessing datasets encompassing up to five demographic dimensions, in conjunction with state-of-the-art word embeddings and debiasing methods, on downstream tasks where biased predictions can lead to allocational harm (§3.1).
2.3 Debiasing
Pretrained word embeddings, including static word embeddings such as GloVe and contextualized word embeddings such as BERT, contain human-like biases and stereotypical associations (Caliskan et al., 2017; Garg et al., 2018; May et al., 2019). A burgeoning body of NLP work has explored debiasing techniques to mitigate biases in pretrained word embeddings. One body of work has focused on debiasing static word embeddings (Bolukbasi et al., 2016; Zhao et al., 2020, 2018; Kaneko and Bollegala, 2019; Ravfogel et al., 2020).
Given the wide adoption of transformer-based contextualized embedding models, recent research has investigated bias mitigation in models such as BERT and RoBERTa (Zmigrod et al., 2019; Webster et al., 2020; Garimella et al., 2021; Kaneko and Bollegala, 2021; Guo et al., 2022). Existing methods for debiasing static and contextualized embeddings have alleviated representational harm along demographic dimensions such as gender. However, Gonen and Goldberg (2019) raised the concern that some debiasing strategies geared towards static word embeddings simply cover up the biases - which can resurface. Moreover, the seemingly debiased static embeddings often do not alleviate biases in downstream NLP prediction tasks (Goldfarb-Tarrant et al., 2021). The extent to which state-of-the-art debiasing methods can mitigate downstream intersectional biases remains unclear. This is precisely one of the gaps our study attempts to shed light on.
$^{2}$ https://github.com/nd-hal/naacl-2022
Dataset	Task	Demographics	Data source	N
Psychometrics	Anxiety, Literacy, Numeracy, Trust	Gender, Race, Age, Income, Ed-ucation	Survey	8,395
Multilingual Twitter Corpus (MTC)	Hate Speech Identification	Gender, Race, Age	Twitter	83,078
Five Item Personality Inventory (FIPI)	Extraverted, Stable	Gender, Race, Age, Income, Ed-ucation	Survey	6,805
AskAPatient	Sentiment	Gender, Age	Forums	20,000
Myers-Briggs Type In-dicator (MBTI)	Perceiving, Thinking	Gender, Age	Reddit	7,406 (1,584)
Table 1: Details of the datasets used for benchmarking. For MBTI, users were able to provide multiple texts, we report unique users in parentheses.
3 Data, Models, Experiments
As previously depicted in Figure 1, our experimental setup is as follows. We assess predictive performance and fairness across five datasets spanning ten dependent variables/tasks and five demographic dimensions. We train three models (GloVe, BERT, and RoBERTa) as our prediction and fairness baselines. We then debias the input embeddings for these models (Ravfogel et al., 2020; Zmigrod et al., 2019; Kaneko and Bollegala, 2021) and re-train them to compare the performance. Details of the data, models, and evaluation metrics are below.
We examine five datasets (Table 1) across several NLP tasks: psychometric dimension prediction, hate speech identification, personality detection, and sentiment analysis. The psychometric data set (Abbasi et al., 2021) consists of free-text responses on four psychometric dimensions: subjective health literacy, numeracy, anxiety, and trust in doctors. These free-text responses were then linked to survey-based psychometric scores also provided by the participants (serving as gold-standard numeric response labels). The data also includes self-reported demographics for each individual: age, race, gender, income, and education level. This data set was collected using crowd workers from Amazon Mechanical Turk and Qualtrics.
Similarly, the Five Item Personality Inventory (FIPI) and Myers-Briggs Type Indicator (MBTI) datasets include free text responses to estimate one of the FIPI or MBTI personality traits (Gjurković et al., 2021). In particular, due to space constraints, we focus on the MBTI traits of perceiving and thinking, and the FIPI traits of extraverted and stable. For FIPI, available demographics are gender, race, age, income, and education. For MBTI, self-reported gender and age are available. The AskAP
patient dataset (Limsopatham and Collier, 2016) is taken from web forums and has labeled sentiment, along with gender and age information.
The Multilingual Twitter Corpus (MTC) hatespeech dataset contains labeled Twitter messages for the task of hate speech detection (Huang et al., 2020). The dataset also contains inferred author demographic factors. We use three demographics: gender, race, and age.
The Psychometrics, FIPI, AskAPatient, and MBTI tasks are all relevant from an allocational harms perspective. Biases in predictions for healthcare-related variables (Psychometrics), or personality type variables (MBTI, FIPI) can affect an individual's health care plan, personalized interventions, job prospects, etc. Biased predictions for drug rating sentiment can affect which drugs a future user chooses to take.
3.2 Models and Debiasing Methods
In the experiments, we considered several different text classification models. We used a word convolutional neural network (CNN) model, initialized with GloVe embeddings. We also considered two transformer-based contextualized embedding models: BERT and RoBERTa.
CNN We trained a word convolutional neural network (CNN) model, initialized with GloVE embeddings. The model consists of 3 concatenated CNN layers with kernel size of 1, 2 and 3 respectively. Each layer has a filter size of 256, rectified linear unit (ReLU) activation, L2 regularization (0.001), and global max pooling. The models were trained for 35 epochs with a batch size of 32 and learning rate of  $1e^{-4}$ .
Debiased-CNN We debiased the GloVe model using (Ravfogel et al., 2020). We kept all parameters the same as in the original paper based on
their publically available implementation. $^3$  The projection matrix was learned over 50 epochs.
BERT and RoBERTa We fine-tuned BERT and RoBERTa on downstream prediction tasks. We used BERT-base-uncased and RoBERTa-base model loaded from the transformers library. We fine-tuned BERT and RoBERTa model for five epochs using the following hyperparameters: a batch size to 32, learning rate of  $1e^{-5}$ , weight decay of 0.01. We saved the final model that achieves the lowest loss on validation set.
Debiased-BERT and Debiased-RoBERTa We debiased BERT and RoBERTa using (Kaneko and Bollegala, 2021). We obtained the gender word lists and stereotype word lists. We used Newscommentary-v15 corpus as the external corpus to locate sentences where the gender and stereotype words occur and then debias. All BERT or RoBERTa layers are debiased at the token level, and the debiasing loss weight is set to 0.8. The model is fine-tuned for three epochs used the following hyperparameters: a batch size of 32 and learning rate of  $5e^{-5}$
Training Details For each dataset we trained using five-fold cross validation, so that for each example in each dataset, we could generate predictions as unseen test data. Each test fold was then concatenated for a given model for fairness calculations. All models were trained on the same data with hyperparameter tuning. All prediction models, debiasing models are trained on a NVIDIA GeForce RTX 3090 GPU card, with 11.2 CUDA version.
Debiasing Strategy Static word embeddings (GloVe, Pennington et al., 2014) were debiased using WordED (Ravfogel et al., 2020) $^5$ . This method iteratively learns a projection of embeddings that removes the bias information with minimal impact on embedding distances.
Contextualized word embedding models BERT and RoBERTa were debiased using ContextED (Kaneko and Bollegala, 2021), which has been shown to work well at removing gender-bias encoded in embeddings. This method uses predefined word lists to identify sentences that contain the gendered or stereotype words, and then
fine-tunes the pretrained model parameters by encouraging gendered and stereotype words to have orthogonal representations.
We also assessed two alternative debiasing methods for the contextualized word embedding models: counterfactual data augmentation (CDA) (Zmigrod et al., 2019) and Dropout (Webster et al., 2020).7 CDA augments the training corpora with counterfactual data so that the language model is pretrained on gender-balanced text. Dropout mitigates gender biases by increasing the dropout rate in the pretrained models. Therefore, the debiasing methods in our experiments represent different ways of mitigating biases: dataset level (CDA), debiasing during pretraining (ContextED and Dropout), and post-tuning debiasing (WordED).
There are several definitions of fairness in the literature (Mehrabi et al., 2021), each with corresponding methods of assessment. In this work we rely on two prior metrics from the literature, and also present a new metric, adjusted disparate impact, to account for base rates in the dataset.
Disparate Impact One of the most common fairness assessments is disparate impact (DI, Friedler et al., 2019). DI measures the inequality of positive cases between privileged and non-privileged groups for a particular demographic. DI comes from the legal field, where certain regulations require DI be above a threshold of 0.8 (or below 1.2 in the inverse case). For true labels  $y$ , predicted labels  $\hat{y}$ , and relevant demographic group  $A$ :
$$
D I = \frac {p (\hat {y} = 1 \mid A = 0)}{p (\hat {y} = 1 \mid A = 1)} \tag {1}
$$
Where  $A = 0$  refers to the protected group and  $A = 1$  refers to the privileged group. A DI ratio of 1 indicates demographic parity, where the rates of positive predictions are consistent across demographic classes:  $P(\hat{y} = 1|A = 0) = P(\hat{y} = 1|A = 1)$  (Mehrabi et al., 2021).
Statistical Parity (SP) Subgroup Fairness Recent theoretical work on intersectional biases also assesses demographic parity, where the score compares group-specific rates to the global rate in the dataset instead of a comparison between privileged and protected classes (Kearns et al., 2018):
$$
p (A = g) \times | p (\hat {y} = 1) - p (\hat {y} = 1 | A = g) | \quad (2)
$$
<sup>7</sup>https://github.com/google-research-datasets/zari
<sup>3</sup>https://github.com/shauli-ravfogel/nullspace_projection
$^{4}$ https://github.com/kanekomasahiro/context-debias/
$^{5}$ https://github.com/shauli-ravfogel/nullspace_projection
$^{6}$ https://github.com/kanekomasahiro/context-debias
This value is compared to an acceptability parameter  $\lambda$  to assess fairness. As this method was proposed for the intersectional case, it gives a way to identify the upper-bound of the fairness violation in a dataset (Yang et al., 2020):
$$
F V = \max  _ {g \in G _ {f}} \left| T P R _ {g} - T P R _ {D} \right| \tag {3}
$$
Where  $G_{f}$  is the set of demographic groups under consideration for analysis,  $TPR_{g}$  is the true positive rate of the classifier on the instances in  $g$ , and  $TPR_{D}$  is the overall true positive rate for the classifier on the dataset. Prior work considered the average violation across groups (Subramanian et al., 2021), but for the purposes of this study we are interested in a worst case analysis.
Adjusted Disparate Impact We propose reweighting DI to account for differences in base rates. Adjusted DI (ADI) divides DI by the base rate ratio for the protected and privileged groups:  $DI^{*} = \frac{p(y = 1|A = 0)}{p(y = 1|A = 1)}, ADI = \frac{DI}{DI^{*}}$
Note that the disparate impact metrics are not defined for cases where there are no positive instances for either the protected or privilege classes in the data, or when there are no positive predictions for the privileged class (due to zero division). Therefore, we use additive smoothing when calculating DI and adjusted DI (Zhai and Lafferty, 2004).
Intersectional Fairness To assess intersectional fairness we enumerated all combinations for each  $n$ -demographic scenario (e.g., 2-demographic, 3-demographic, etc.). We set a reference demographic, specifically gender, because of the prior work on debiasing word embeddings for gender (Bolukbasi et al., 2016; Gonen and Goldberg, 2019; Kaneko and Bollegala, 2021). For intersectional cases, we calculated DI and FV for all possible combinations of demographics that included gender. For example, the 2-demographic case for the psychometrics dataset involves calculating DI and FV for the following protected groups: older women, lower education women, lower income women, and non-white women. Our privileged groups are the negations of the protected groups, e.g., for the above case they are younger men, higher education men, higher income men, and white men. By considering disjoint demographic groups, we avoid cumulative effects of merging fairness results from individual demographics during the intersectional phase. We follow the same procedure for enumerating protected groups for the
3- and 4-demographic cases. For 5-demographics we consider all demographics together. For all models and datasets, we calculated fairness and performance metrics. For performance, we report mean squared error (MSE), Pearson's  $r$ , F1, and area under the receiver operating curve (AUC). For fairness, we report adjusted DI and fairness violation (FV, §3.3).
Figure 2 shows the ADI results for BERT and GloVe using ContextED and WordED for debiasing, respectively. In most cases, particularly for BERT, disparate impact scores for gender alone are in a reasonable range (within  $10\%$ ). For GloVe, we do observe high gender ADI on Anxiety and Thinking. However, as the number of demographics under consideration grows, the range of ADI scores widens. While debiasing the word embeddings typically helps to reduce the unfairness for the target demographic (e.g., gender), in the intersectional cases the model still performs poorly. There are similar trends in FV scores as the number of demographics increases, with the extent of violations often increasing by a factor of  $3x$  to  $10x$  as intersections increase (Table 2).