SlowGuess/NAACL_2021_OCR huggingface dataset:
Xisen Jin $^{\S}$ , Francesco Barbieri $^{\dagger}$ , Brendan Kennedy $^{\S}$ , Aida Mostafazadeh Davani $^{\S}$ , Leonardo Neves $^{\dagger}$ , Xiang Ren $^{\S}$
$\S$  University of Southern California
†Snap Inc.
{xisenjin, mostafaz, btkenned, xiangren}@usc.edu
{fbarbieri, lneves}@snap.com
Fine-tuned language models have been shown to exhibit biases against protected groups in a host of modeling tasks such as text classification and coreference resolution. Previous works focus on detecting these biases, reducing bias in data representations, and using auxiliary training objectives to mitigate bias during fine-tuning. Although these techniques achieve bias reduction for the task and domain at hand, the effects of bias mitigation may not directly transfer to new tasks, requiring additional data collection and customized annotation of sensitive attributes, and re-evaluation of appropriate fairness metrics. We explore the feasibility and benefits of upstream bias mitigation (UBM) for reducing bias on downstream tasks, by first applying bias mitigation to an upstream model through fine-tuning and subsequently using it for downstream finetuning. We find, in extensive experiments across hate speech detection, toxicity detection, occupation prediction, and coreference resolution tasks over various bias factors, that the effects of UBM are indeed transferable to new downstream tasks or domains via finetuning, creating less biased downstream models than directly fine-tuning on the downstream task or transferring from a vanilla upstream model. Though challenges remain, we show that UBM promises more efficient and accessible bias mitigation in LM fine-tuning.[12]
The practice of fine-tuning pretrained language models (PTLMs or LMs), such as BERT (Devlin et al., 2019), has improved prediction performance in a wide range of NLP tasks. However, finetuned LMs may exhibit biases against certain protected groups (e.g., gender and ethnic minorities),
Figure 1: Comparison between the focus of our study (d) and previous works (a,b,c). We study the viability of obtaining an upstream model that could reduce bias in a number of downstream classifiers when fine-tuned.
as models may learn to associate certain features with positive or negative labels spuriously (Dixon et al., 2018), or propagate bias encoded in PTLMs to downstream classifiers (Caliskan et al., 2017; Bolukbasi et al., 2016). Among many examples, Kurita et al. (2019) demonstrates gender-bias in the pronoun resolution task when models are trained using BERT embeddings, and Kennedy et al. (2020) shows that hate speech classifiers finetuned from BERT result in more frequent false positive predictions for certain group identifier mentions (e.g., "muslim", "black").
Approaches for bias mitigation are mostly applied during fine-tuning to reduce bias in a specific downstream task or dataset (Park et al., 2018; Zhang et al., 2018; Beutel et al., 2017) (see Fig. 1 (a)). For example, data augmentation approaches reduce the influence of spurious features in the original dataset (Dixon et al., 2018; Zhao et al., 2018; Park et al., 2018), and adversarial learning approaches generate debiased data representations that are exclusive to the downstream model (Kumar et al., 2019; Zhang et al., 2018). These techniques act on biases particular to the given dataset, domain, or task, and require new bias mitigation when switching to a new downstream task or dataset. This can require auxiliary training objectives, the definition of task-specific fairness met
$^{1}$ Code and data: https://github.com/INK-UC/Upstream-Bias-Mitigation
2The work was partially done when Xisen Jin was an intern at Snap Inc.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3770-3783 June 6-11, 2021. ©2021 Association for Computational Linguistics
rics, the annotation of bias attributes (e.g., identifying African American Vernacular English), and the collection of users' demographic data. These drawbacks make bias mitigation inaccessible to the growing community, fine-tuning LMs to new datasets and tasks.
In contrast, we investigate initially mitigating bias while fine-tuning an "upstream" model in one or more upstream datasets, and subsequently achieving reduced bias when fine-tuning for downstream applications (Fig. 1 (d)), so that bias mitigation is no longer required in downstream training. Similar to transfer learning for enhancing predictive performance in common setups (Pan and Yang, 2010; Dai and Le, 2015), we suggest that LMs that undergo bias mitigation acquire inductive bias that is helpful for reducing harmful biases when fine-tuned on new domains and tasks. In four tasks with known bias factors — hate speech detection, toxicity detection, occupation prediction from short bios, and coreference resolution — we explore whether upstream bias mitigation of a LM followed by downstream fine-tuning reduces bias for the downstream model. Though previous work has addressed biases in frozen PTLM or word embeddings (Bolukbasi et al., 2016; Zhou et al., 2019; Bhardwaj et al., 2020; Liang et al., 2020; Ravfogel et al., 2020), for example by measuring associations between gender and occupations in an embedding space, they do not study their effect on downstream classifiers (Fig. 1 (b)), while some of them study the effects while keeping the embeddings frozen (Zhao et al., 2019; Kurita et al., 2019; Prost et al., 2019). Bias in these frozen representations can also be directly corrected by removing associations between feature and sensitive attributes (Elazar and Goldberg, 2018; Madras et al., 2018) (Fig. 1 (c)), but this does not allow predictions to be generated for new data.
Our experiments address the following research questions: (a) whether mitigating a single bias factor in the upstream stage is maintained when finetuning on new examples from the same domain and task, (b) whether transfer is viable when the downstream domains and tasks are different from the upstream model, and (c) whether we can address multiple kinds of bias with a single upstream model. We perform these experiments under a generic transfer learning framework, noted as Upstream Bias Mitigation (UBM) for Downstream Fine-Tuning for convenience, which consists of
two stages: first, in the upstream bias mitigation stage, a LM is fine-tuned with bias mitigation objectives on one or several "upstream" tasks, and subsequently the classification layer is re-initialized; then, in the downstream fine-tuning stage the encoder from the upstream model, jointly with the new classification layer, are again fine-tuned on a downstream task without additional bias mitigation steps. Using six datasets with previously recognized bias factors, our analysis show overall positive results for the questions above; still, there are challenges remaining to stabilize the results of bias mitigation in challenging setups, e.g., the multi-bias factor setting.
Our contributions are summarized as follows: (1) we propose a new research direction for mitigating bias in fine-tuned models; (2) we perform extensive experiments to study the viability of the upstream bias mitigation framework in various settings; (3) we demonstrate the effectiveness of this research direction, motivating further improvements, tests, and applications.
2 Exploring the Transferability of Bias Mitigation Effects
We consider biases against protected groups in classifiers fined-tuned from LMs. In our present analysis, bias is defined as disparate model performance on different subsets of data which are associated with different demographic groups (e.g., instances that mention or are generated by different social groups) (Blodgett et al., 2020). Our evaluation of bias aligns with the definition of equalized odds and equal opportunities (Hardt et al., 2016) in previous works of fairness in machine learning.
Here, we first outline our experimental setup for exploring the transferability of bias mitigation effects, in which we detail the process of applying UBM and pose three key research questions (section 2.1). We follow by introducing the bias factors studied and the corresponding classification tasks and datasets (section 2.2), and our evaluation protocols and metrics (section 2.3).
2.1 Experiment Setups of UBM
Our goal is to evaluate the transferability of bias mitigation effects for one or multiple bias factors in downstream fine-tuned models. We follow an Upstream Bias Mitigation (UBM) for Downstream Fine-Tuning procedure, pictured in Figure 2. First, in the Upstream Bias Mitigation phase, an upstream
Figure 2: Experiment setups to study Upstream Bias Mitigation (UBM) for Downstream Fine-Tuning. We consider the settings with the same or different upstream and downstream domains and tasks, while addressing one or more bias factors (e.g., both dialect bias and gender bias). The framework consists of two stages: (1) an upstream (source) model  $f_{s} = h_{s} \circ g_{s}$  is trained with bias mitigation algorithms and (2) the encoder  $g_{s}$  is transferred to the downstream (target) model  $f_{t}$  for fine-tuning.
source) model  $f_{s} = h_{s}\circ g_{s}$ , composed of a text encoder  $g_{s}$  and a classifier head  $h_s$ , is trained on one or more upstream datasets  $\mathcal{D}_s$  with bias mitigation algorithms. The encoder  $g_{s}$  is to be transferred to downstream (target) domains and tasks while the classifier head  $h_s$  is discarded. Then, in the Downstream Fine-Tuning phase, the downstream model  $f_{t} = h_{t}\circ g_{t}$  utilizes  $g_{s}$  to initialize the encoder weights and is fine-tuned for prediction performance without bias mitigation approaches on downstream datasets  $\mathcal{D}_t$ .
This UBM process is applied in three settings, summarized below, which each contribute to evaluating the transferability of bias mitigation effects.
1. Fine-Tuning on the Same Distribution. In the simplest setting, we fine-tune the downstream model over new examples from the same data distribution as the upstream model. In practice, each dataset is split into two halves, with one used for upstream bias mitigation and the other for downstream fine-tuning.
2. Cross-Domain and Cross-Task Fine-Tuning. Similar to how LMs are fine-tuned for various tasks and domains, in a more practical setup, we test whether transfer of bias mitigation effects is viable across domains and tasks. To achieve this, we apply bias mitigation while fine-tuning a LM on one dataset and perform fine-tuning on another.
3. Multiple Bias Factors. In the most challenging setup, we train a single upstream model to address multiple bias factors (e.g., both dialect bias and gender bias). Such upstream models can be trained with multi-task learning (i.e., jointly training over multiple datasets with shared encoder  $g$  but different classifier heads  $h$ ) while mitigating multiple kinds of bias. Subsequently, the resulting upstream model is transferred to downstream models as be
Dataset	Prediction Task	Bias
GHC (Kennedy et al., 2018)	Hate	Group Identifier
Stormfront (de Gibert et al., 2018)	Hate	Group Identifier
DWMW (Davidson et al., 2017)	Toxicity	AAVE Dialect
FDCL (Founta et al., 2018)	Toxicity	AAVE Dialect
BiasBios (De-Anteaga et al., 2019)	Occupation	Gender Stereotyping
OntoNotes 5.0 (Weischedel et al., 2013)	Coreference	Gender Stereotyping
Table 1: Summary of tasks and bias included for study.
fore. This is a key test of UBM's viability for widespread application.
2.2 Bias Factors and Datasets
To ensure our analysis holds true for a variety of domains, tasks, and bias factors, we experiment with three different bias factors studied in previous research along with six different datasets (also summarized in Table 1), described below.
Group Identifier Bias. This bias refers to higher false positive rates of hate speech predictions for sentences containing specific group identifiers, which is harmful to protected groups by misclassifying innocuous text (e.g., "I am a Muslim") as hate speech. We include two datasets for study, namely the Gab Hate Corpus (GHC; Kennedy et al., 2018) and the Stormfront corpus (de Gibert et al., 2018). Both datasets contain binary labels for hate and non-hate instances, though with differences in the labeling schemas and domains.
AAVE Dialect Bias. Sap et al. (2019) show that offensive and hate speech classifiers yield a higher false positive rate on text written in African American Vernacular English (AAVE). This bias brings significant harm to the communities that uses AAVE, for example, by leading to the disproportionate removal of the text written in AAVE in social media platforms (Blodgett et al., 2020). We include two datasets for study: FDCL (Founta et al., 2018) and DWMW (Davidson et al., 2017). In both datasets, we treat abusive, hateful and spam to
gather as harmful outcomes (i.e., false positives for each are harmful) to compute false positive rates. Following Sap et al. (2019), we use an off-the-shelf AAVE dialect predictor (Blodgett et al., 2016) to identify examples written in AAVE.
Gender Stereotypical Bias. Zhao et al. (2018) summarize a list of occupations that are prone to be stereotyped in practice, leading to coreference resolutions models and occupation prediction models having biases in performance in pro- and anti-stereotypical instances when trained on short bios. We train the coreference resolution model on the OntoNotes 5.0 dataset (Weischedel et al., 2013) and the occupation classifier on the BiasBios (De-Arteaga et al., 2019) dataset.
2.3 Evaluation Protocol and Metrics
We evaluate the overall performance of the models on downstream tasks along with appropriate bias metrics for each bias factor, analyzed for each dataset and task in previous works. We expect UBM to minimally affect classification performance while improving on bias metrics.
Classification Performance. We report in-domain F1 scores for GHC, Stormfront, OntoNotes 5.0, and accuracy scores for FDCL, DWMW and BiasBios. Following Zhang et al. (2018), for hate speech detection and toxicity detection datasets, we use the equal error rate (EER) threshold for prediction.
Group Identifier Bias Metrics. To evaluate group identifier bias, we evaluate false positive rate (FPR) differences, noted as FPRD, between examples mentioning one of 25 group identifiers provided by Kennedy et al. (2020) and the overall FPR. In addition, we followed Kennedy et al. (2020) in using a New York Times articles (NYT) corpus of  $25k$  non-hate sentences, each mentioning one of 25 group identifiers. This corpus specifically provides an opportunity to measure FPR—reported as (NYT Acc.), equivalent to 1-FPR. Additionally, following the evaluation protocol of Dixon et al. (2018) and Zhang et al. (2020), we incorporate the Identity Phrase Templates Test Sets (reported as IPTTS), which consists of  $77k$  hate and non-hate examples mentioning group identifiers, generated with templates. Following these works, for IPTTS we compute FPRD as  $\sum_{z}|\mathrm{FPR}_{z} - \mathrm{FPR}_{\text{overall}}|$ , where  $\mathrm{FPR}_{z}$  is false positive rate on sentences with the group identifier  $z$ , and  $\mathrm{FPR}_{\text{overall}}$  is the overall false positive rate.
AVE Dialect Bias Metrics. Given the sparsity
of AAVE examples in the datasets and the noisy outputs of AAVE classifier (Blodgett et al., 2016), we expect the in-domain FPRD metrics to be noisy. Therefore, following Xia et al. (2020), we incorporate the BROD (Blodgett et al., 2016) dataset, which is a large unlabeled collection of Twitter posts written in l. Since in practice only a small portion of texts are toxic or spam, we treat all examples from BROD as normal, and report the accuracy (which equals 1-FPR) on the dataset.
Gender Stereotype Metrics. We employ the WinoBias (Zhao et al., 2018) dataset which provides opportunities to evaluate models on stereotypical and anti-stereotypical coreference examples. We report the differences in F1 (F1-Diff) on two subsets of data. On occupation prediction, following Ravfogel et al. (2020), we report mean differences of true positive rate (TPR) differences in predicting each occupation for men and women.
Here, we detail the particular bias mitigation algorithms used for implementing UBM, as well as the other baselines used for verifying the transferability of bias mitigation effects.
3.1 Implementations of UBM
We implement UBM with two different bias mitigation algorithms in the upstream bias mitigation phase: explanation regularization (Kennedy et al., 2020), and adversarial de-biasing (Zhang et al., 2018; Madras et al., 2018; Xia et al., 2020), denoted here as  $\mathbf{UBM}_{reg}$  and  $\mathbf{UBM}_{adv}$ , respectively.
UBM with Explanation Regularization. Explanation regularization reduces importance placed on spurious surface patterns (i.e., words or phrases) during upstream model training. We apply  $\mathbf{UBM}_{reg}$  to group identifier and AAVE dialect bias, where the set of spurious patterns are group identifiers and the most frequent words, from statistics of the dataset, used by AAVE speakers; we find explanation regularization not effective for gender bias. The importance of a surface pattern  $w \in \mathcal{W}$  in the input  $\mathbf{x}$ , noted as  $\phi(w, \mathbf{x})$  is measured as the model prediction change when it is removed. The model is trained by optimizing the main learning objective  $\ell$  while penalizing importance attributed to patterns  $w \in \mathcal{W}$  that exist in the input  $\mathbf{x}$ .
$$
\min  _ {f} \quad \ell_ {c} + \alpha \sum_ {w \in \mathbf {x} \cap \mathcal {W}} | | \phi (w, \mathbf {x}) | | ^ {2}, \tag {1}
$$
UBM with Adversarial De-biasing. In  $\mathrm{UBM}_{adv}$ , the upstream model is trained with adversarial de-biasing techniques, so that sensitive attributes related to bias (e.g., the dialect of the sentence or the gender referenced in the sentence) cannot be predicted from the hidden representations  $\mathbf{z}$  given by the encoder  $g$ . During training, an adversarial classifier head  $h^{adv}$  is built upon the encoder and trained to predict sensitive attributes, while the encoder is optimized to prevent the adversarial classifier from success. Formally, the optimization objective is written as,
$$
\min  _ {g, h} \max  _ {h _ {a d v}} \quad \ell_ {c} + \ell_ {a d v} (h _ {a d v} \circ g (\mathbf {x}), a), \quad (2)
$$
where  $a$  notes the ground truth sensitive attribute, and  $\ell_{adv}$  is the cross entropy loss between the predicted sensitive attribute and the ground truth sensitive attribute.
As mentioned in Sec. 2.1, upstream models can be trained to mitigate multiple bias factors with multi-task learning on multiple datasets. We separately apply bias mitigation algorithms for each dataset (sharing the same encoder) and note the algorithms applied in the subscript (e.g.,  $\mathbf{UBM}_{reg + adv}$ ).
3.2 Other Baselines
We compare UBM with two families of methods.
Methods without Bias Mitigation. Two types of models were evaluated that did not address bias. First, the Vanilla model is a downstream classifier directly fine-tuned on downstream task from a LM (e.g., RoBERTa). Second, Van-Transfer is fine-tuned on upstream datasets without bias mitigation and fine-tuned on downstream datasets.
Downstream Bias Mitigation. For reference, we show the results of directly applying explanation regularization, noted as Expl. Reg., or adversarial de-biasing, noted as Adv. Learning, during downstream fine-tuning. In most cases, mitigating bias in downstream classifier should be the most effective way to reduce bias, though this is not always feasible in practice for reasons discussed above.
We also consider two simple baselines that could reduce bias in downstream models via heuristics. Emb. Zero zeros out the word embedding of spurious surface patterns (using the same word list as explanation regularization) in PTLMs before finetuning. We also include Emb. Zero. Trans, which
GHC B
Metrics	In-domain F1 (↑)	In-domain FPRD (↓)	IPTTS FPRD (↓)	NYT Acc (↑)
Non-Transfer (GHC . B)
Vanilla	37.91 ± 2.5	35.64 ± 2.2	21.50 ± 2.8	68.55 ± 20
Expl. Reg.	38.09 ± 2.7	18.68 ± 0.3	4.82 ± 1.1	84.05 ± 3.0
GHC . A → GHC . B
Van. Transfer	42.41 ± 1.0	37.44 ± 1.5	17.67 ± 2.1	75.35 ± 4.2
UBMReg	43.79 ± 1.9	34.34 ± 3.1	10.02 ± 1.1	81.40 ± 1.4
Table 2: Same-domain and task UBM with a single bias factor. The source datasets are noted before arrow  $(\rightarrow)$ . All metrics except In-domain F1 measure bias. See Table 6 in Appendix for complete results.
zeros out embeddings of spurious surface patterns before fine-tuning from an upstream model. The method does not apply to cases where surface patterns related to bias (e.g., gendered pronouns) are crucial for prediction, e.g., coreference resolution.
In this section, we present the results of UBM in three settings following the order in Sec. 2.1: transferring to the same data distribution, transferring to different data distributions, transferring from an upstream model with bias mitigation for multiple bias factors. We follow these main analyses with an investigation of the impact of freezing encoder weights before downstream fine-tuning, and lastly with a brief exploration of how UBM's positive results are achieved.
Implementation Details. In all experiments reported on below, models are initially fine-tuned from RoBERTa-base. The upstream model is trained for a fixed number of epochs and the checkpoint with the best prediction performance is transferred to the downstream model. See Appendix for more implementation details. We use  $\mathcal{D}_s\rightarrow \mathcal{D}_t$  as the transfer notation, in which upstream and downstream datasets are respectively represented in the left and right-hand side of the arrow.
4.1 UBM with the Same Data Distribution
We first briefly show the results when the downstream model sees new, unseen samples from the same data distribution as the upstream model. In this controlled setting, we isolate and test the basic viability of UBM, which requires that information from the upstream model is retained during downstream fine-tuning. GHC, Stormfront, FDCL and BiasBios were partitioned into two subsets with equal size, noted as subsets A and B of corresponding datasets, to train the upstream and downstream models respectively.
Downstream dataset	GHC	Stormfront	DWMW	OntoNotes 5.0
Metrics	In-domain F1 (↑)	In-domain FPRD (↓)	IPTTS FPRD (↓)	NYT Acc (↑)	In-domain F1 (↑)	In-domain FPRD (↓)	IPTTS FPRD (↓)	NYT Acc (↑)	In-domain Acc. (↑)	BROD Acc. (↑)	In-domain F1 (↑)	Winobias F1-Diff (↓)
Non-Transfer (GHC)	Non-Transfer (Stormfront)	Non-Transfer (DWMW)	Non-Transfer (OntoNotes 5.0)
Vanilla	49.60 ± 1.0	46.43 ± 2.5	20.01 ± 5.7	72.08 ± 7.3	53.74 ± 2.8	18.09 ± 2.7	11.51 ± 5.1	73.06 ± 10	91.46 ± 0.1	78.77 ± 0.3	76.53 ± 0.2	8.04 ± 0.5
Emb. Zero	43.76 ± 0.7	38.31 ± 2.0	11.95 ± 2.7	83.21 ± 5.2	49.97 ± 0.6	18.80 ± 2.0	8.20 ± 0.3	70.15 ± 4.4	90.59 ± 0.1	62.37 ± 0.4
Expl. Reg.	43.37 ± 1.8	29.29 ± 1.2	4.2 ± 1.6	81.22 ± 11	51.53 ± 1.1	13.43 ± 1.5	3.80 ± 0.4	83.73 ± 8.0	91.38 ± 0.1	76.61 ± 1.5
Adv. Learning	91.11 ± 0.3	77.53 ± 0.9
Stf. → GHC	GHC → Stf.	FDCL → DWMW	BiasBios → OntoNotes 5.0
Van-Transfer	47.83 ± 2.1	47.51 ± 4.6	14.00 ± 0.8	66.71 ± 10.6	55.79 ± 1.3	17.83 ± 2.2	8.26 ± 2.5	76.98 ± 1.1	91.27 ± 0.2	78.98 ± 1.1	76.65 ± 0.3	10.54 ± 0.7
Emb. Zero. Trans.	44.51 ± 0.5	40.92 ± 4.2	12.91 ± 0.2	80.11 ± 1.2	52.98 ± 0.6	16.35 ± 0.9	8.04 ± 2.1	81.11 ± 1.9	91.53 ± 0.0	81.01 ± 0.9
UBMReg	49.94 ± 1.0	42.71 ± 3.8	12.23 ± 3.3	75.34 ± 4.8	56.43 ± 0.6	18.03 ± 2.5	6.86 ± 1.1	81.18 ± 1.1	91.39 ± 0.0	80.27 ± 0.2
UBMAdv	91.20 ± 0.0	81.24 ± 0.2	76.34 ± 0.2	9.27 ± 1.4
Table 3: Cross-domain and task UBM with a single bias factor. The source datasets are noted before arrow  $(\rightarrow)$ . All metrics except In-domain F1 or In-domain Accuracy measures bias. The preferred outcomes for each metric are marked with arrows. The main comparators of UBM are Vanilla, Van-Transfer, Emb. Zero, and Emb. Zero. Trans that do not perform downstream bias mitigation. We see UBM maintains in-domain prediction performance while overall reduces bias. Results of Adv. Learning and  $\mathrm{UBM}_{Adv}$  on GHC, Stf. are not included because applying adversarial de-biasing to reduce group identifier bias yields degenerated classifiers.