SlowGuess/ACL_2020_OCR huggingface dataset:
We present these debiasing results in Table 3, and see that for both binary gender bias and multiclass religion bias, our proposed method reduces the amount of bias as measured by the given tests and metrics. The reduction in bias is most pronounced when debiasing the pre-trained BERT encoder. We also observe that simply fine-tuning the BERT encoder for specific tasks also reduces the biases present as measured by the Caliskan tests, to some extent. However, fine-tuning does not lead to consistent decreases in bias and cannot be used as a standalone debiasing method. Furthermore, fine-tuning does not give us control over which type of bias to control for and may even amplify bias if the task data is skewed towards particular biases. For example, while the bias effect size as measured by Caliskan test C7 decreases from  $+0.542$  to  $-0.033$  and  $+0.288$  after fine-tuning on SST-2 and CoLA respectively, the effect size as measured by the multiclass Caliskan test increases from  $+0.035$  to  $+1.200$  and  $+0.243$  after fine-tuning on SST-2 and CoLA respectively.
3.4 Comparison with Baselines
We compare to three baseline methods for debiasing: 1) FastText derives debiased sentence embeddings using an average of debiased FastText word embeddings (Bojanowski et al., 2016) using word-level debiasing methods (Bolukbasi et al., 2016), 2)
Table 3: Debiasing results on BERT and ELMo sentence representations. First six rows measure binary SEAT effect sizes for sentence-level tests, adapted from Caliskan tests. SEAT scores closer to 0 represent lower bias. CN: test from Caliskan et al. (2017) row  $N$ . The last row measures bias in a multiclass religion setting using MAC (Manzini et al., 2019) before and after debiasing. MAC score ranges from 0 to 2 and closer to 1 represents lower bias. Results are reported as  $x_{1} \rightarrow x_{2}$  where  $x_{1}$  represents score before debiasing and  $x_{2}$  after, with lower bias score in bold. Our method reduces bias of BERT and ELMo for the majority of binary and multiclass tests.
Debiasing Method	Ave. Abs. Effect Size
BERT original (Devlin et al., 2019)	+0.354
FastText (Bojanowski et al., 2016)	+0.565
BERT word (Bolukbasi et al., 2016)	+0.861
BERT simple (May et al., 2019)	+0.298
SENT-DEBIAS BERT (ours)	+0.256
Table 4: Comparison of various debiasing methods on sentence embeddings. FastText (Bojanowski et al., 2016) (and BERT word) derives debiased sentence embeddings with an average of debiased FastText (and BERT) word embeddings using word-level debiasing methods (Bolukbasi et al., 2016). BERT simple adapts May et al. (2019) by using simple templates to debias BERT representations. SENT-DEBIAS BERT represents our method using diverse templates. We report the average absolute effect size across all Caliskan tests. Average scores closer to 0 represent lower bias.
BERT word obtains a debiased sentence representation from average debiased BERT word representations, again debiased using word-level debiasing methods (Bolukbasi et al., 2016), and 3) BERT simple adapts May et al. (2019) by using simple templates to debias BERT sentence representations. From Table 4, SENT-DEBIAS achieves a lower average absolute effect size and outperforms the baselines based on debiasing at the word-level and averaging across all words. This indicates that it is not sufficient to debias words only and that biases in a sentence could arise from their debiased word constituents. In comparison with BERT simple, we observe that using diverse sentence templates obtained from naturally occurring written and spoken text makes a difference on how well we can remove biases from sentence representations. This supports our hypothesis that using increasingly diverse templates estimates a bias subspace that generalizes to different words in their context.
Figure 1: Influence of the number of templates on the effectiveness of bias removal on BERT fine-tuned on SST-2 (left) and BERT fine-tuned on QNLI (right). All templates are from WikiText-2. The solid line represents the mean over different combinations of domains and the shaded area represents the standard deviation. As increasing subsets of data are used, we observe a decreasing trend and lower variance in average absolute effect size.
3.5 Effect of Templates
We further test the importance of sentence templates through two experiments.
1) Same Domain, More Quantity: Firstly, we ask: how does the number of sentence templates impact debiasing performance? To answer this, we begin with the largest domain WikiText-2 (13750 templates) and divide it into 5 partitions each of size 2750. We collect sentence templates using all possible combinations of the 5 partitions and apply these sentence templates in the contextualization step of SENT-DEBIAS. We then estimate the corresponding bias subspace, debias, and measure the average absolute values of all 6 SEAT effect sizes. Since different combinations of the 5 partitions result in a set of sentence templates of different sizes  $(20\%, 40\%, 60\%, 80\%, 100\%)$ , this allows us to see the relationship between size and debiasing performance. Combinations with the same percentage of data are grouped together and for each group we compute the mean and standard deviation of the average absolute effect sizes. We perform the above steps to debias BERT fine-tuned on SST-2 and QNLI and plot these results in Figure 1. Please refer to the appendix for experiments with BERT fine-tuned on CoLA, which show similar results.
For BERT fine-tuned on SST-2, we observe a decreasing trend in the effect size as increasing subsets of the data is used. For BERT fine-tuned on QNLI, there is a decreasing trend that quickly tapers off. However, using a larger number of templates reduces the variance in average absolute effect size and improves the stability of the SENTDEBIAS algorithm. These observations allow us to conclude the importance of using a large number of templates from naturally occurring text corpora.
2) Same Quantity, More Domains: How does
the number of domains that sentence templates are extracted from impact debiasing performance? We fix the total number of sentence templates to be 1080 and vary the number of domains these templates are drawn from. Given a target number  $k$ , we first choose  $k$  domains from our Reddit, SST, POM, WikiText-2 datasets and randomly sample  $1080 / k$  templates from each of the  $k$  selected domains. We construct 1080 templates using all possible subsets of  $k$  domains and apply them in the contextualization step of SENT-DEBIAS. We estimate the corresponding bias subspace, debias and measure the average absolute SEAT effect sizes. To see the relationship between the number of domains  $k$  and debiasing performance, we group combinations with the same number of domains  $(k)$  and for each group compute the mean and standard deviation of the average absolute effect sizes. This experiment is also performed for BERT fine-tuned on SST-2 and QNLI datasets. Results are plotted in Figure 2.
We draw similar observations: there is a decreasing trend in effect size as templates are drawn from more domains. For BERT fine-tuned on QNLI, using a larger number of domains reduces the variance in effect size and improves stability of the algorithm. Therefore, it is important to use a large variety of templates across different domains.
3.6 Visualization
As a qualitative analysis of the debiasing process, we visualize how the distances between sentence representations shift after the debiasing process is performed. We average the sentence representations of a concept (e.g. man, woman, science, art) across its contexts (sentence templates) and plot the t-SNE (van der Maaten and Hinton, 2008)
Figure 2: Influence of the number of template domains on the effectiveness of bias removal on BERT fine-tuned on SST-2 (left) and BERT fine-tuned on QNLI (right). The domains span the Reddit, SST, POM, WikiText-2 datasets. The solid line is the mean over different combinations of domains and the shaded area is the standard deviation. As more domains are used, we observe a decreasing trend and lower variance in average absolute effect size.
Pretrained BERT embeddings
Debiased BERT embeddings
Figure 3: t-SNE plots of average sentence representations of a word across its sentence templates before (left) and after (right) debiasing. After debiasing, non gender-specific concepts (in black) are more equidistant to genders.
embeddings of these points in 2D space. From Figure 3, we observe that BERT average representations of science and technology start off closer to man while literature and art are closer to woman. After debiasing, non gender-specific concepts (e.g. science, art) become more equidistant to both man and woman average concepts.
3.7 Performance on Downstream Tasks
To ensure that debiasing does not hurt the performance on downstream tasks, we report the performance of our debiased BERT and ELMo on SST-2 and CoLA by training a linear classifier on top of debiased BERT sentence representations. From Table 5, we observe that downstream task performance show a small decrease ranging from  $1 - 3\%$  after the debiasing process. However, the performance of ELMo on SST-2 increases slightly from 89.6 to 90.0. We hypothesize that these differences in performance are due to the fact that CoLA tests for linguistic acceptability so it is more concerned with low-level syntactic structure such as verb usage, grammar, and tenses. As a result, changes in sentence representations across bias directions may impact its performance more. For example,
sentence representations after the gender debiasing steps may display a mismatch between gendered pronouns and the sentence context. For SST, it has been shown that sentiment analysis datasets have labels that correlate with gender information and therefore contain gender bias (Kritchenko and Mohammad, 2018). As a result, we do expect possible decreases in accuracy after debiasing. Finally, we test the effect of SENT-DEBIAS on QNLI by training a classifier on top of debiased BERT sentence pair representations. We observe little impact on task performance: our debiased BERT fine-tuned on QNLI achieves  $90.6\%$  performance as compared to the  $91.3\%$  we obtained without debiasing.
4 Discussion and Future Work
Firstly, we would like to emphasize that both the WEAT, SEAT, and MAC metrics are not perfect since they only have positive predictive ability: they can be used to detect the presence of biases but not their absence (Gonen and Goldberg, 2019). This calls for new metrics that evaluate biases and can scale to the various types of sentences appearing across different individuals, topics, and in both
Test	BERT	debiased BERT	ELMo	debiased ELMo
SST-2	92.7	89.1	89.6	90.0
CoLA	57.6	55.4	39.1	37.1
QNLI	91.3	90.6	-	-
Table 5: We test the effect of SENT-DEBIAS on both single sentence (BERT and ELMo on SST-2, CoLA) and paired sentence (BERT on QNLI) downstream tasks. The performance (higher is better) of debiased BERT and ELMo sentence representations on downstream tasks is not hurt by the debiasing step.
spoken and written text. We believe that our positive results regarding contextualizing words into sentences implies that future work can build on our algorithms and tailor them for new metrics.
Secondly, a particular bias should only be removed from words and sentences that are neutral to that attribute. For example, gender bias should not be removed from the word "grandmother" or the sentence "she gave birth to me". Previous work on debiasing word representations tackled this issue by listing all attribute specific words based on dictionary definitions and only debiasing the remaining words. However, given the complexity of natural sentences, it is extremely hard to identify the set of neutral sentences and its complement. Thus, in downstream tasks, we removed bias from all sentences which could possibly harm downstream task performance if the dataset contains a significant number of non-neutral sentences.
Finally, a fundamental challenge lies in the fact that these representations are trained without explicit bias control mechanisms on large amounts of naturally occurring text. Given that it becomes infeasible (in standard settings) to completely retrain these large sentence encoders for debiasing (Zhao et al., 2018; Zhang et al., 2018), future work should focus on developing better post-hoc debiasing techniques. In our experiments, we need to re-estimate the bias subspace and perform debiasing whenever the BERT encoder was fine-tuned. It remains to be seen whether there are debiasing methods which are invariant to fine-tuning, or can be efficiently re-estimated as the encoders are fine-tuned.
This paper investigated the post-hoc removal of social biases from pretrained sentence representations. We proposed the SENT-DEBIAS method that accurately captures the bias subspace of sentence representations by using a diverse set of templates from naturally occurring text corpora. Our experiments show that we can remove biases that occur in
BERT and ELMo while preserving performance on downstream tasks. We also demonstrate the importance of using a large number of diverse sentence templates when estimating bias subspaces. Leveraging these developments will allow researchers to further characterize and remove social biases from sentence representations for fairer NLP.
PPL and LPM were supported in part by the National Science Foundation (Awards #1750439, #1722822) and National Institutes of Health. RS was supported in part by US Army, ONR, Apple, and NSF IIS1763562. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, National Institutes of Health, DARPA, and AFRL, and no official endorsement should be inferred. We also acknowledge NVIDIA's GPU support and the anonymous reviewers for their constructive comments.
Herve Abdi and Lynne J. Williams. 2010. Principal component analysis. WIRES Comput. Stat., 2(4):433-459.
Emily Alsentzer, John Murphy, William Boag, WeiHung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72-78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
David Bamman, A. Seza Dogruoz, Jacob Eisenstein, Dirk Hovy, David Jurgens, Brendan O'Connor, Alice Oh, Oren Tsur, and Svitlana Volkova. 2016. Proceedings of the first workshop on NLP and computational social science.
Solon Barocas and Andrew D Selbst. 2016. Big data's disparate impact. *Calif. L. Rev.*, 104:671.
Christine Basta, Marta R. Costa-jussà, and Noe Casas. 2019. Evaluating the underlying gender bias in contextualized word embeddings. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 33-39, Florence, Italy. Association for Computational Linguistics.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Proc. of NIPS, pages 4349-4357.
Shikha Bordia and Samuel R. Bowman. 2019. Identifying and reducing gender bias in word-level language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 7-15, Minneapolis, Minnesota. Association for Computational Linguistics.
Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science.
Robert Dale. 2019. Law and word order: Nlp in legal tech. Natural Language Engineering, 25(1):211-217.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H. Chi, and Alex Beutel. 2019. Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES '19, page 219-226, New York, NY, USA. Association for Computing Machinery.
Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 609-614, Minneapolis, Minnesota. Association for Computational Linguistics.
Anthony Greenwald, T Poehlman, Eric Luis Uhlmann, and Mahzarin Banaji. 2009. Understanding and using the implicit association test: Iii. meta-analysis of predictive validity.
Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. Clinicalbert: Modeling clinical notes and predicting hospital readmission. CoRR, abs/1904.05342.
Svetlana Kiritchenko and Saif Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 43-53, New Orleans,
Louisiana. Association for Computational Linguistics.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166-172, Florence, Italy. Association for Computational Linguistics.
Brian Larson. 2017. Gender as a variable in natural-language processing: Ethical considerations. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pages 1-11, Valencia, Spain. Association for Computational Linguistics.
Anne Lauscher and Goran Glavaš. 2019. Are we consistently biased? multidimensional analysis of biases in distributional word vectors. In *SEM 2019*.
Paul Pu Liang, Yao Chong Lim, Yao-Hung Hubert Tsai, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2019. Strong and simple baselines for multimodal utterance embeddings. In *NAACL-HLT*, pages 2599–2609, Minneapolis, Minnesota. Association for Computational Linguistics.
Paul Pu Liang, Ziyin Liu, AmirAli Bagher Zadeh, and Louis-Philippe Morency. 2018. Multimodal language analysis with recurrent multistage fusion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579-2605.
Thomas Manzini, Lim Yao Chong, Alan W Black, and Yulia Tsvetkov. 2019. Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 615-621, Minneapolis, Minnesota. Association for Computational Linguistics.
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017a. Pointer sentinel mixture
models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017b. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings.
Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2799-2804, Brussels, Belgium. Association for Computational Linguistics.
Sunghyun Park, Han Suk Shim, Moitreya Chatterjee, Kenji Sagee, and Louis-Philippe Morency. 2014. Computational analysis of persuasiveness in social multimedia: A novel dataset and multimodal prediction approach. In ICMI.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 527-536, Florence, Italy. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models