SlowGuess/NAACL_2021_OCR huggingface dataset:
Ali Shafahi, Parsa Saadatpanah, Chen Zhu, Amin Ghiasi, Christoph Studer, David W. Jacobs, and Tom Goldstein. 2020. Adversarily robust transfer learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23.
Mengzhou Xia, Anjalie Field, and Yulia Tsvetkov. 2020. Demoting racial bias in hate speech detection. In Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media, pages 7-14, Online. Association for Computational Linguistics.
B. H. Zhang, B. Lemoine, and Margaret Mitchell. 2018. Mitigating unwanted biases with adversarial learning. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society.
Guanhua Zhang, Bing Bai, Junqi Zhang, Kun Bai, Conghui Zhu, and Tiejun Zhao. 2020. Demographics should not be the reason of toxicity: Mitigating discrimination in text classifications with instance weighting. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4134-4145, Online. Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Coterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 629-634, Minneapolis, Minnesota. Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15-20, New Orleans, Louisiana. Association for Computational Linguistics.
Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, Ryan Cotterell, and Kai-Wei Chang. 2019. Examining gender bias in languages with grammatical gender. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5276-5284, Hong Kong, China. Association for Computational Linguistics.
We use RoBERTa-base as our base model. In the bias mitigation phase, models for GHC, Stormfront, FDCL, DWMW, and BiasBios are trained with a learning rate  $1e^{-5}$ , and the checkpoint with the best validation F1 or accuracy score is provided to the fine-tuning phase. We train on GHC, FDCL, DWMW, BiasBios for maximum 5 epochs and Stormfront for maximum 10 epochs. The checkpoint with the best validation in-domain classification performance is kept. In the fine-tuning phase, we try the learning rate  $1e^{-5}$  and  $5e^{-6}$ , and report the results with a higher validation in-domain classification performance. For the coreference resolution model on OntoNotes 5.0, we adapt existing code implementation $^4$  (Joshi et al., 2019) to support loading RoBERTa-base as the base model. We use the same hyperparameter settings as BERT-base in the provided code implementation.
To report mean and standard deviation of performance are computed over 3 runs for most of the experiments, with the same set of random seeds; for GHC and Stf. experiments in Table 3, and  $\mathrm{UBM}_{reg,reg,adv}$  on OntoNotes 5.0, we run experiments for 6 runs. Models except coreference resolution models on OntoNotes, are trained on a single GTX 2080 Ti GPU. Coreference resolution models are trained on a single Quadro RTX 6000 GPU.
The training time per iteration is consistent over experiments in about 1.5 iteration per second, except the conference resolution. The training of coreference resolution model on OntoNotes 5.0 takes around 8 hours. The largest dataset among other datasets, BiasBios, takes 2 hours to train.
A.2 Details of Bias Mitigation Algorithms
For explanation regularization algorithm, we set the regularization strength  $\alpha$  as 0.03 for GHC and Stormfront experiments, and 0.1 for FDCL and DWMW experiments. We regularize importance score on 25 group identifiers in (Kennedy et al., 2018) for GHC and Stormfront. These group identifiers the ones that have the largest coefficient in a bag-of-words linear classifier. For FDCL, we extract 50 words with largest coefficient in the bag-of-words linear classifier with a AAE dialect probability higher than  $60\%$  (given by the off-the-shelf AAE dialect predictor (Blodgett et al., 2016)) on
its own. For adversarial de-biasing, the adversarial loss term has the same weight as the classification loss term.
Group Identifier Bias Experiments. We use a balanced split of the GHC dataset, where training, validation, and the test set consist of 22,767, 1,586, and 1,344 examples. We use the union of "human degradation" and "call for violence" has the hate label, which results in around  $9\%$  of hate examples for all the splits. Note that the split is different from Kennedy et al. (2020) where the test split has a much higher ratio of hate examples. We use the same split as Kennedy et al. (2020) for the Stormfront dataset, with 7,896, 978, and 1,998 examples in training, validation, and test sets. The NYT corpus contain  $12.5k$  non-hate sentences for testing.
AAVE Dialect Bias Experiments. We follow Sap et al. (2019) for the split ratio (73/12/15) of the DWMW dataset, which results in 17,994, 2,974, and 3,718 examples in each split. For the FDCL dataset, as only tweet ids are provided and some of the tweets are no longer available, the final dataset consists of 41191, 5149, 5149 examples for each split. We release the tweet ids used in each split. Following (Xia et al., 2020), we sample  $20k$  examples with an AAVE speaker probability (which is included in the dataset) greater than  $80\%$ . We manually verify a subset of examples in BROD following the protocols of (Sap et al., 2019) and found  $93\%$  of sentences clearly non-toxic.
Gender Stereotypical Bias Experiments. For the BiasBios dataset, we use the same split as Ravfogel et al. (2020) with 255,710 training examples  $(65\%)$ , 39,359 validation examples  $(10\%)$ , and 98,344  $(25\%)$  test examples. We use the official dataset split (Weischedel et al., 2013) for the OntoNotes 5.0 dataset.
We use the same train/test splits between "transfer" and "non-transfer" setup. Two partitions of datasets used for Same-distribution UBMexperiments have a random half of total examples for train/validation/test splits. For IPTTS/NYT/BROD, we use the same test set across tables.
A.4 Details of  $\ell^2$ -sp Regularizer
The  $\ell^2$ -sp regularizer (Li et al., 2018) we applied in Sec. 4.4 penalizes the distance between the weights and the initial point of fine-tuning. Formally, let  $\mathbf{w}_0$  be the initial weight of the encoder  $g_{t}$  before fine-tuning, and  $\mathbf{w}$  be the current weight of  $g_{t}$ . The
4 https://github.com/mandarjoshi90/cores
Method / Datasets	GHC B	Stormfront B	FDCL B	Biasbios B
Metrics	In-domain F1 (↑)	In-domain FPRD (↓)	IPTTS FPRD (↓)	NYT Acc (↑)	In-domain F1 (↑)	In-domain FPRD (↓)	IPTTS FPRD (↓)	NYT Acc (↑)	In-domain Acc. (↑)	BROD Acc. (↑)	In-domain F1 (↑)	Winbias F1-Diff (↓)
Non-Transfer (GHC B)	Non-Transfer (Stf. B)	Non-Transfer (FDCL. B)	Non-Transfer (Biasbios B)
Vanilla	37.91 ± 2.5	35.64 ± 2.2	21.50 ± 2.8	68.55 ± 20	55.56 ± 0.5	20.81 ± 4.9	10.99 ± 5.6	66.28 ± 8.1	75.72 ± 0.2	73.57 ± 1.2	85.52 ± 0.1	13.83 ± 0.2
Expl. Reg.	38.09 ± 2.7	18.68 ± 0.3	4.82 ± 1.1	84.05 ± 3.0	53.05 ± 1.0	15.97 ± 1.1	3.36 ± 3.3	65.23 ± 10	77.30 ± 0.2	76.72 ± 1.1	85.07 ± 0.0	9.61 ± 0.5
Adv. Learning	75.28 ± 0.2	77.12 ± 1.2
GHCA→GHC B	Stf. A→Stf. B	FDCLA→FDCL B	Biasbios A→Biasbios B
Van. Transfer	42.41 ± 1.0	37.44 ± 1.5	17.67 ± 2.1	75.35 ± 4.2	58.43 ± 1.2	17.96 ± 3.6	11.58 ± 4.8	74.12 ± 4.2	76.33 ± 0.6	70.35 ± 2.4	85.81 ± 0.2	12.59 ± 0.5
UBMReg	43.79 ± 1.9	34.34 ± 3.1	10.02 ± 1.1	81.40 ± 1.4	58.56 ± 1.0	16.42 ± 0.9	7.51 ± 2.4	69.45 ± 4.0	76.22 ± 0.5	69.29 ± 1.8	85.86 ± 0.1	11.99 ± 0.5
UBMAdv	75.88 ± 0.4	71.11 ± 1.6
Table 6: Same-domain and task UBM with a single bias factor. We partition a dataset to two subsets, noted as split A and split B. We train the upstream model with split A and fine-tune on split B. All metrics except In-domain F1 or In-domain Accuracy measures bias. The preferred outcomes for each metric are marked with arrows.
Metrics	In-domain F1 (↑)	In-domain FPRD (↓)	IPTTS FPRD (↓)	NYT Acc (↑)	In-domain F1 (↑)	BROD Acc. (↑)	In-domain F1 (↑)	In-domain TPRD (↓)
Upstream model	Stormfront A + FDCL A
Downstream model	Stormfront B	FDCL B
Van-Transfer	57.58 ± 2.7	13.97 ± 2.0	11.33 ± 2.2	75.72 ± 7.5	77.18 ± 0.5	71.12 ± 1.2
UBMReg,Reg	56.72 ± 1.7	17.91 ± 1.0	8.05 ± 0.6	77.40 ± 0.3	77.13 ± 0.3	72.17 ± 1.6
UBMReg,Adv	55.63 ± 2.5	17.14 ± 0.5	13.78 ± 4.3	70.37 ± 10	76.64 ± 0.6	76.55 ± 0.6
Upstream Model	GHCA + FDCL A
Downstream Model	GHC B	FDCL B
Van-Transfer	44.30 ± 0.7	41.06 ± 3.9	19.75 ± 6.9	74.60 ± 6.3	77.34 ± 0.4	72.96 ± 1.5
UBMReg,Reg	42.96 ± 2.0	33.98 ± 3.0	9.30 ± 2.1	86.05 ± 1.9	76.21 ± 0.4	73.10 ± 1.4
UBMReg,Adv	42.44 ± 3.5	33.96 ± 1.5	16.68 ± 1.7	81.79 ± 9.0	76.94 ± 0.4	76.55 ± 0.7
Upstream Model	GHCA + FDCL A+ BiasBios A
Downstream Model	GHC B	FDCL B	BiasBios B
Van-Transfer	42.80 ± 3.3	37.83 ± 10	17.38 ± 1.7	72.23 ± 13	77.30 ± 0.2	72.91 ± 1.1	85.81 ± 0.0	12.78 ± 0.0
UBMReg,Reg,Adv	42.81 ± 1.9	31.86 ± 1.1	9.61 ± 1.8	82.15 ± 9.5	76.95 ± 0.1	72.93 ± 0.6	85.81 ± 0.0	11.72 ± 0.8
UBMReg,Adv,Adv	41.66 ± 2.3	33.39 ± 0.2	10.00 ± 0.8	83.50 ± 3.0	77.03 ± 0.3	75.87 ± 0.8	85.79 ± 0.1	12.43 ± 0.5
$\ell^2$  -sp regularizer is written as  $\Omega (\mathbf{w}) = \beta ||\mathbf{w} - \mathbf{w}_0||_2^2$  , appended to the learning objective.  $\beta$  is a hyperparameter controlling the strength of the regularization. We reported results where  $\beta = 1$  We tried different values of  $\beta$  from  $1e^{-6}$  to 100, increasing  $\beta$  by 10 times each time, but we do not see changes in the conclusion.
B Complete Analysis of UBM over the Same Data Distribution
Table 6 show the results of same-domain transfer with a single bias factors. Table 7 further show the results of addressing multiple bias factors in this setup.
On GHC, Stormfront, and BiasBios, UBM overall reduces bias compared to Vanilla and Vanilla-Transfer. We notice the NYT accuracy on Stormfront in Stf. A  $\rightarrow$  Stf. B setup is an exception. However, we see the bias is not reduced on Stf. B even when we directly run explanation regularization in the target domain. We reason that the Half-Stormfront dataset is small and the average length of the sentences are quite different between Stormfront and NYT, so that a model trained on
Table 7: Dealing with multiple bias factors with a single upstream model with UBM, where the domains and tasks are the same in the upstream and the downstream model.  $\nearrow$  and  $\searrow$  show whether the metrics has increased or decreased (both imply improvement) compared to non-transfer Vanilla training in Table 3.
Metrics	In-domain F1 (↑)	In-domain FPRD (↓)	IPTTS FPRD (↓)	NYT Acc (↑)
Stormfront → GHC
Expl. Reg.	43.37 ± 1.8	29.29 ± 1.2	4.20 ± 1.6	81.22 ± 11
Van-Transfer + Reg.	45.25 ± 2.2	29.91 ± 1.9	5.01 ± 1.3	86.15 ± 2.9
UBMReg + Reg.	44.92 ± 2.0	28.85 ± 2.1	3.36 ± 1.2	89.33 ± 1.2
GHC → Stormfront
Expl. Reg.	51.53 ± 1.8	13.43 ± 1.5	3.80 ± 0.4	83.73 ± 8.0
Van-Transfer + Reg.	52.18 ± 1.3	13.12 ± 1.1	4.35 ± 0.3	80.54 ± 2.0
UBMReg + Reg.	53.58 ± 1.4	16.07 ± 1.3	4.53 ± 0.9	82.59 ± 1.4
Table 8: Applying both UBM and downstream bias mitigation  $(\mathrm{UBM}_{Reg} + \mathrm{Reg.})$ , compared to downstream bias mitigation only (Expl. Reg.) and downstream bias mitigation over Van-Transfer model (Van-Transfer + Reg.).
Stormfront hardly generalizes to NYT.
We find intriguing results on FDCL; From FDCL A  $\rightarrow$  FDCL B in Table 6, we find bias is not reduced with UBM. However, as shown in Table 7, when the upstream model is trained jointly with other datasets to reduce multiple bias factors (Stf A + FDCL A, GHC A + FDCL A, GHC A + FDCL A + BiasBios A), the bias is clearly reduced.
C Applying UBM with Downstream Bias Mitigation
In Table 8, we report the performance of performing both upstream and downstream bias mitigation, compared with downstream bias mitigation only, and downstream bias mitigation over a vanilla-transferred model. We see UBM further reduced bias in the Stormfront  $\rightarrow$  GHC setup, while fail to improve in  $\mathrm{GHC}\rightarrow$  Stormfront. Compared to our previous results in Tables 3 and 4, we see a clearer directionality of transfer of bias mitigation effects when downstream bias mitigation is also applied.