SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 2 presents the results for mitigating group identifier bias in the GHC. We see an overall bias reduction, via UBM, by comparing with Vanilla training and Van-Transfer. We include full results and discussions for this simple setting in Appendix.
4.2 Cross-domain and Task UBM
Following the result that UBM is effective in the same-domain setting, we now move to analyzing cross-domain settings in greater depth. For hate speech classification, we perform transfer learning from GHC to Stormfront and from Stormfront to GHC; and for toxicity classification, we perform transfer learning from FDCL to DWMW. We also perform transfer learning from BiasBios (occupation prediction) to OntoNotes 5.0 (coreference resolution). Table 3 shows the results of cross-domain and task transfer learning and non-transfer baselines. Our findings are summarized below.
UBM can reduce bias in different target domains and tasks compared to fine-tuning without bias mitigation. The results of cross-domain and task transfer learning (i.e., Stf.  $\rightarrow$  GHC, GHC  $\rightarrow$  Stf., FDCL  $\rightarrow$  DWMW), show that transferring from a less biased upstream model (UBMReg and UBMAdv) leads to better downstream bias mitigation compared to directly training without bias mitigation in the target domain (Vanilla). Meanwhile, the in-domain classification performance has improved (on GHC and Stormfront) or been preserved (on DWMW). It is notable that directly mitigating bias (Expl. Reg., Adv. Learning) on DWMW is not effective, which is previously observed by Xia et al. (2020), while transferring from FDCL is successful.
There are exceptions where UBM fails to reduce bias. We see the in-domain FPRD on Stormfront
does not improve; however, as discussed in our metrics section, the in-domain FPRD is computed over a much smaller set of examples compared to NYT and IPTTS datasets, and is thus less reliable. UBM does not reduce bias compared to Vanilla training on OntoNotes 5.0, but achieves less bias compared to Van-Transfer. This result confirms the effect of bias mitigation in upstream models, but the transfer learning itself has increased the bias.
Comparison with Emb. Zero and Emb. Zero. Trans. We find two alternative methods, Emb. Zero and Emb. Zero Trans, also reduce bias on some of the datasets. On GHC, Emb. Zero achieves an in-domain FPRD and IPTTS-FPRD lower than UBM. However, it comes with clear drop of indomain classification performance.
4.3 Mitigating Multiple Bias Factors
Having observed an overall positive effect of UBM across domains and tasks, next we present the results of experiments on mitigating multiple bias factors with a single upstream model. This involves training an upstream model with multiple bias mitigation objectives across multiple datasets, followed by fine-tuning on a single dataset without bias mitigation. We test three combinations of datasets. First, a multi-task model is trained to jointly mitigate group identifier bias and AAVE dialect bias using GHC and FDCL (GHC + FDCL), and transferred to Stormfront and DWMW. Next, a model is similarly trained jointly on group identifier and AAVE biases on and Stormfront and FDCL (Stf. + FDCL) and transferred to GHC and DWMW. Lastly, models were trained over source datasets
<sup>3</sup>We find UBM<sub>Reg, Reg, Adv</sub> yield degenerated classifiers for OntoNotes (Test F1 < 46.00) in 5 out of 6 runs. The result is from one successful run.
Bias Factor	Group Identifier Bias	AAVE Dialect Bias	Gender Stereotypical Bias
Metrics	In-domain F1 (↑)	In-domain FPRD (↓)	IPTTS FPRD (↓)	NYT Acc (↑)	In-domain F1 (↑)	BROD Acc. (↑)	In-domain F1 (↑) Winobias F1-Diff (↓)
Upstream model	Stormfront + FDCL
Downstream model	GHC	DWMW
Van-Transfer	49.71 ± 0.3	45.84 ± 3.8	12.43 ± 2.5	72.37 ± 7.4	91.64 ± 0.2	81.12 ± 0.1
UBMReg,Reg	50.21 ± 1.4	47.63 ± 0.7	12.29 ± 2.7	68.44 ± 8.6	91.66 ± 0.2	80.05 ± 0.1
UBMReg,Adv	49.89 ± 1.7	47.85 ± 1.2	21.25 ± 2.0	65.78 ± 5.7	91.55 ± 0.2	81.14 ± 1.5
Upstream Model	GHC + FDCL
Downstream Model	Stormfront	DWMW
Van-Transfer	56.78 ± 1.6	14.26 ± 0.8	11.04 ± 0.7	77.06 ± 5.1	91.65 ± 0.1	80.98 ± 0.4
UBMReg,Reg	53.87 ± 1.2	15.92 ± 1.2	8.40 ± 1.4	83.71 ± 3.2	91.79 ± 0.4	81.36 ± 0.8
UBMReg,Adv	53.63 ± 0.7	15.52 ± 2.2	8.90 ± 1.6	84.87 ± 1.1	91.33 ± 0.1	81.09 ± 0.4
Upstream Model	GHC + FDCL + BiasBios
Downstream Model	Stormfront	DWMW	OntoNotes 5.0
Van-Transfer	55.47 ± 0.7	16.74 ± 1.5	12.19 ± 0.7	64.15 ± 6.5	91.58 ± 0.1	80.74 ± 0.3	73.64 ± 0.3 9.91 ± 0.2
UBMReg,Reg,Adv	52.59 ± 0.5	21.17 ± 2.0	9.99 ± 2.8	74.58 ± 4.9	91.64 ± 0.3	81.07 ± 0.4	75.68 4.933
UBMReg,Adv,Adv	52.85 ± 0.9	18.55 ± 5.8	13.15 ± 3.7	70.00 ± 5.8	91.50 ± 0.1	81.08 ± 0.3	76.01 ± 0.4 8.67 ± 0.7
Table 4: Dealing with multiple bias factors with a single upstream model with UBM. We test three combination of upstream datasets, namely Stormfront + FDCL, GHC + FDCL, and GHC + FDCL + BiasBios, in reducing two or three bias factors.  $\nearrow$  and  $\searrow$  show whether the metrics has increased or decreased (both imply improvement) compared to non-transfer Vanilla training in Table 3.
Metrics	In-domain Acc. (↑)	In-domain FPRD (↓)	IPTTS FPRD (↓)	NYT Acc. (↑)	In-domain Acc. (↑)	BROD Acc. (↑)
Stf. → GHC, UBMReg	FDCL → DWMW, UBMAdv
Freeze	45.42	37.71	7.82	84.45	83.25	64.80
ℓ2-sp	49.31	47.03	14.24	71.88	91.38	79.95
Fine-tune	49.94	42.71	12.23	75.34	91.20	81.24
GHC → Stf. UBMReg
Freeze	47.32	25.02	8.24	64.60
ℓ2-sp	55.80	19.75	6.72	80.42
Fine-tune	56.43	18.03	6.86	81.18
Table 5: UBM while keeping the encoder frozen (Freeze), discouraging parameter changes  $(\ell^2$  -sp), or standard fine-tuning (Fine-tune). We see weight freezing and  $\ell^2$  -sp overall do not improve over simple fine-tuning on Stf.  $\rightarrow$  GHC and FDCL  $\rightarrow$  DWMW.
GHC, FDCL, BiasBios (GHC+FDCL+BiasBios) to mitigate all three bias factors, and transferred to Stormfront, DWMW, and OntoNotes. The results are shown in Table 4.
Comparison to Single-Dataset Vanilla Baselines. As a basic measure of bias mitigation success, we compare multi-dataset models' results with single-dataset Vanilla training and Van-Transfer. We see UBM with  $\mathrm{GHC} + \mathrm{FDCL}$  successfully reduces both group identifier bias and AAVE dialect bias in downstream models. UBM with  $\mathrm{GHC} + \mathrm{FDCL} + \mathrm{BiasBios}$  also successfully reduces group identifier bias in terms of IPTTS, FPRD (which is the most reliable metrics of bias given its large size), and AAVE bias. It also reduces gender stereotypical bias compared to Van-Transfer in some experimental runs, but in an unstable manner, demonstrated by the large variance of F1-Diff and degenerated runs of  $\mathrm{UBM}_{Reg,Reg,Adv}$ .
Results of UBM on Stf. + FDCL are less promising. We find  $\mathrm{UBM}_{Reg,Adv,Adv}$  is not successful in reducing group identifier bias.  $\mathrm{UBM}_{Reg,Reg,Adv}$
could reduce bias on IPTTS-FPRD, but does not improve other metrics. Notably, UBM on Stf. + FDCL clearly underperform UBM on Stf. only.
$\mathbf{UBM}_{Reg}$  versus  $\mathbf{UBM}_{Adv}$ . Empirically, we find using explanation regularization on FDCL  $(\mathbf{UBM}_{reg,reg},\mathbf{UBM}_{reg,reg,adv})$  instead of adversarial learning  $(\mathbf{UBM}_{reg,adv},\mathbf{UBM}_{reg,adv,adv})$  consistently improves bias mitigation performance on other bias factors.
Takeaways. Our results show it is possible to reduce multiple bias factors via UBM. However, we have shown that these effects are not automatic for each new dataset added to upstream models for multi-task bias mitigation.
4.4 Freezing or Regularizing Model Weights
In the experiments above, we have shown that the effect of mitigating bias is partially preserved with simple fine-tuning. Next, we study whether freezing the encoders or discouraging their weight changes improves bias mitigation in the target domain, as they intuitively try to retain effect of bias mitigation. However, we find a counterintuitive result: these approaches typically do not achieve reduced downstream bias, and in fact reduce in-domain classification performance. Table 5 shows the results when we keep the weights frozen (Freeze), discouraging weights from changing with  $\ell^2$ -sp regularizer (Li et al., 2018, details in appendix), or standard fine-tuning (fine-tune). In Stf.  $\rightarrow$  GHC, freezing the weights contributed to reducing the bias, while  $\ell^2$ -sp failed to help. In GHC  $\rightarrow$  Stf and FDCL  $\rightarrow$  DWMW, freezing the weights and  $\ell^2$ -sp both increased the bias. A possible reason is that by freezing the encoder, we reduce its expressive power. As a result, the encoder is prone
(a) Gab A  $\rightarrow$  Gab B
(b) Stf. A  $\rightarrow$  Stf. B
(c)  $\mathrm{Gab}\to \mathrm{Stf}$
(d) Stf.  $\rightarrow$  Gab
Figure 3: Gradient of importance attribution on group identifiers  $\phi (w,\mathbf{x})$  over time (in solid lines) and the corresponding values of  $\phi (w,\mathbf{x})$  (in dash lines) during downstream fine-tuning. The cross-marks show the gradient measured in the upstream model (before reinitialization of the classifier layer).  $\mathrm{UBM}_{reg}$  not only reduces importance attributed to group identifiers, but also the gradient norm of the importance.
to capture simple but spurious correlations.
4.5 Investigating Why UBM Reduces Bias
We attempt to interpret why fine-tuning from a debiased upstream model remains less biased during fine-tuning from the perspective of gradient of importance attributed to words  $w$  related to bias factors (e.g., group identifiers) by the input occlusion algorithm. A large importance attribution usually induces bias. Figure 3 plots the importance attribution of group identifiers  $\phi(w, \mathbf{x})$  and the norm of its gradient w.r.t. parameters  $\theta$  of the encoder  $g$ , noted as  $||\nabla_{\theta}\phi(w, \mathbf{x})||_2$ .
UBM reduces the gradient of  $\phi (w,\mathbf{x})$ , so that  $\phi (w,\mathbf{x})$  is less likely to change at the beginning of downstream fine-tuning. Fig. 3 shows UBM has not only reduced value of importance attributed to spurious patterns, but also reduced their gradients. The gradient norm is highly indicative about how the importance  $\phi (w,\mathbf{x})$  will change in the downstream model, because when the loss in Eq. 1 in the upstream model is minimized, the gradient  $\nabla_{\theta}\phi (w,\mathbf{x})$  has the same norm but the opposite direction as the main downstream classification objective  $\nabla_{\theta}\ell_{c}$ . It implies that whether the upstream model converges at an optimum where both objectives agree (i.e., gradients are small) can be an important indicator of the success of UBM.
The figure further shows that the gradient and the value of  $\phi (w,\mathbf{x})$  remain small for  $\mathrm{UBM}_{reg}$  over
the whole training process. We leave more study into the training dynamics of UBM as future works.
Here we review approaches that inform the present work (techniques for bias mitigation) and are related to the basic idea of UBM.
Mitigating bias in representations. Bias can be mitigated directly in representations of data. Zhang et al. (2018); Beutel et al. (2017) proposed training a classifier together with an adversarial predictor for sensitive attributes. Madras et al. (2018) further studied re-usable de-biased representations by training a new downstream classifier (potentially with a different classification task) using the learned representations. However, this practice relies on frozen representations (rather than models themselves), which precludes the possibility of generating predictions for new data.
Mitigating bias in pretrained models. Another line of work addresses bias in pretrained models (e.g., word vectors, BERT, Zhou et al., 2019; May et al., 2019; Bhardwaj et al., 2020; Liang et al., 2020). Many such studies again focus on bias in frozen data representations, and do not study their effects on downstream classifiers. Others alternatively assess the propagation of bias from pretrained models to downstream classifiers: Ravfogel et al. (2020) study algorithms for mitigating bias in pretrained models by de-biasing the learned representations, which can subsequently be used in classifiers as frozen representations.
Transferring learning of fairness and robustness. A few previous works have studied related research problems, with significant differences to our work. Though Schumann et al. (2019) theoretically analyzes the transferability of fairness across domains, it assumes simultaneous access of source and target domain data, which does not account for transferring upstream bias mitigation to arbitrary downstream fine-tuned models. Shafahi et al. (2020) study transfer learning of robustness to adversarial attacks under fine-tuning, but do not seek to mitigate bias.
We observe that the effects of bias mitigation are indeed transferable in fine-tuning LMs. Future works in fine-tuning LMs can use UBM in order to easily apply the positive effects of bias mitigation methods to new domains and tasks without customized
bias mitigation processes or access to sensitive user information. Though UBM does not rival directly mitigating bias on the downstream task, it is more efficient and accessible. Future works can develop the effectiveness of UBM beyond the default scenarios in this paper, and potentially apply it to tasks and settings beyond hate speech, toxicity classification, occupation prediction, and coreference resolution in English corpora.
Our analysis demonstrates the effectiveness of Upstream Bias Mitigation for Downstream Fine-Tuning. As we stated in the paper, the reduced efforts of downstream bias mitigation will facilitate broader application of bias mitigation in the growing deep learning community.
While we may expect to obtain an "off-the-shelf" language model that could reduce multiple kinds of bias with UBM, we emphasize that proper evaluation of bias may still be required in downstream side, especially for guaranteed bias mitigation. Currently, our initial analysis of UBM confirms that bias mitigation effects are transferable, but does not provide guarantees of bias mitigation or levels of bias mitigation in the direct setting. The findings in this analysis should identify the potential of UBM to the broader NLP and machine learning communities, which may be extended with new approaches within the UBM framework, or interpretation techniques (as in Sec. 4.5).
Alex Beutel, J. Chen, Zhe Zhao, and Ed Huai hsin Chi. 2017. Data decisions and theoretical implications when adversarially learning fair representations. ArXiv, abs/1707.00075.
Rishabh Bhardwaj, Navonil Majumder, and Soujanya Poria. 2020. Investigating gender bias in bert. ArXiv, abs/2009.05021.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454-5476, Online. Association for Computational Linguistics.
Su Lin Blodgett, Lisa Green, and Brendan O'Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages
1119-1130, Austin, Texas. Association for Computational Linguistics.
Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4349-4357.
A. Caliskan, J. Bryson, and A. Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356:183 - 186.
Andrew M. Dai and Quoc V. Le. 2015. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 3079-3087.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. ICWSM.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnamaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 120-128.
Ona de Gibert, Naiara Perez, Aitor Garcia-Pablos, and Montse Cuadros. 2018. Hate speech dataset from a white supremacy forum. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pages 11-20, Brussels, Belgium. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Lucas Dixon, John Li, Jeffrey Scott Sorensen, Nithum Thain, and L. Vasserman. 2018. Measuring and mitigating unintended bias in text classification. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society.
Yanai Elazar and Yoav Goldberg. 2018. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages
11-21, Brussels, Belgium. Association for Computational Linguistics.
Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, I. Leontiadis, Jeremy Blackburn, G. Stringhini, Athena Vakali, M. Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. ICWSM.
Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 3315-3323.
Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803-5808, Hong Kong, China. Association for Computational Linguistics.
Brendan Kennedy, Mohammad Atari, Aida M Davani, Leigh Yeh, Ali Omrani, Yehsong Kim, Kris Coombs Jr, Shreya Havaldar, Gwenyth Portillo-Wightman, Elaine Gonzalez, et al. 2018. The gab hate corpus: A collection of 27k posts annotated for hate speech. PsyArXiv. July, 18.
Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, and Xiang Ren. 2020. Contextualizing hate speech classifiers with post-hoc explanation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5435-5442, Online. Association for Computational Linguistics.
Sachin Kumar, Shuly Wintner, Noah A. Smith, and Yulia Tsvetkov. 2019. Topics to avoid: Demoting latent confounds in text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4153-4163, Hong Kong, China. Association for Computational Linguistics.
Keita Kurita, Nidhi Vyas, Ayush Parek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166-172, Florence, Italy. Association for Computational Linguistics.
Xuhong Li, Yves Grandvalet, and Franck Davoine. 2018. Explicit inductive bias for transfer learning with convolutional networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2830-2839. PMLR.
Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis Philippe Morency. 2020. Towards debiasing sentence representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5502-5515, Online. Association for Computational Linguistics.
David Madras, Elliot Creager, Toniann Pitassi, and Richard S. Zemel. 2018. Learning adversarially fair and transferable representations. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 3381-3390. PMLR.
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 622-628, Minneapolis, Minnesota. Association for Computational Linguistics.
Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22:1345-1359.
Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2799-2804, Brussels, Belgium. Association for Computational Linguistics.
Flavien Prost, Nithum Thain, and Tolga Bolukbasi. 2019. Debiasing embeddings for reduced gender bias in text classification. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 69-75, Florence, Italy. Association for Computational Linguistics.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256, Online. Association for Computational Linguistics.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668-1678, Florence, Italy. Association for Computational Linguistics.
Candice Schumann, Xuezhi Wang, Alex Beutel, J. Chen, Hai Qian, and Ed Huai hsin Chi. 2019. Transfer of machine learning fairness across domains. ArXiv, abs/1906.09688.