SlowGuess/ACL_2020_OCR huggingface dataset:
Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, Louis-Philippe Morency
Machine Learning Department and Language Technologies Institute
pliang@cs.cmu.edu
As natural language processing methods are increasingly deployed in real-world scenarios such as healthcare, legal systems, and social science, it becomes necessary to recognize the role they potentially play in shaping social biases and stereotypes. Previous work has revealed the presence of social biases in widely used word embeddings involving gender, race, religion, and other social constructs. While some methods were proposed to debias these word-level embeddings, there is a need to perform debiasing at the sentence-level given the recent shift towards new contextualized sentence representations such as ELMo and BERT. In this paper, we investigate the presence of social biases in sentence-level representations and propose a new method, SENTDEBIAS, to reduce these biases. We show that SENT-DEBIAS is effective in removing biases, and at the same time, preserves performance on sentence-level downstream tasks such as sentiment analysis, linguistic acceptability, and natural language understanding. We hope that our work will inspire future research on characterizing and removing social biases from widely adopted sentence representations for fairer NLP.
Machine learning tools for learning from language are increasingly deployed in real-world scenarios such as healthcare (Velupillai et al., 2018), legal systems (Dale, 2019), and computational social science (Bamman et al., 2016). Key to the success of these models are powerful embedding layers which learn continuous representations of input information such as words, sentences, and documents from large amounts of data (Devlin et al., 2019; Mikolov et al., 2013). Although word-level embeddings (Pennington et al., 2014; Mikolov et al., 2013) are highly informative features useful for
a variety of tasks in Natural Language Processing (NLP), recent work has shown that word-level embeddings reflect and propagate social biases present in training corpora (Lauscher and Glavaš, 2019; Caliskan et al., 2017; Swinger et al., 2019; Bolukbasi et al., 2016). Machine learning systems that incorporate these word embeddings can further amplify biases (Sun et al., 2019b; Zhao et al., 2017; Barocas and Selfbst, 2016) and unfairly discriminate against users, particularly those from disadvantaged social groups. Fortunately, researchers working on fairness and ethics in NLP have devised methods towards debiasing these word representations for both binary (Bolukbasi et al., 2016) and multiclass (Manzini et al., 2019) bias attributes such as gender, race, and religion.
More recently, sentence-level representations such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), and GPT (Radford et al., 2019) have become the preferred choice for text sequence encoding. When compared to word-level representations, these models have achieved better performance on multiple tasks in NLP (Wu and Dredze, 2019), multimodal learning (Zellers et al., 2019; Sun et al., 2019a), and grounded language learning (Urbanek et al., 2019). As their usage proliferates across various real-world applications (Huang et al., 2019; Alsentzer et al., 2019), it becomes necessary to recognize the role they play in shaping social biases and stereotypes.
Debiasing sentence representations is difficult for two reasons. Firstly, it is usually unfeasible to fully retrain many of the state-of-the-art sentence-based embedding models. In contrast with conventional word-level embeddings such as GloVe (Pennington et al., 2014) and word2vec (Mikolov et al., 2013) which can be retrained on a single machine within a few hours, the best sentence encoders such as BERT (Devlin et al., 2019), and GPT (Radford et al., 2019) are trained on massive amounts of text
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5502-5515 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
data over hundreds of machines for several weeks. As a result, it is difficult to retrain a new sentence encoder whenever a new source of bias is uncovered from data. We therefore focus on post-hoc debiasing techniques which add a post-training debiasing step to these sentence representations before they are used in downstream tasks (Bolukbasi et al., 2016; Manzini et al., 2019). Secondly, sentences display large variety in how they are composed from individual words. This variety is driven by many factors such as topics, individuals, settings, and even differences between spoken and written text. As a result, it is difficult to scale traditional word-level debiasing approaches (which involve bias-attribute words such as man, woman) (Bolukbasi et al., 2016) to sentences.
Related Work: Although there has been some recent work in measuring the presence of bias in sentence representations (May et al., 2019; Basta et al., 2019), none of them have been able to successfully remove bias from pretrained sentence representations. In particular, Zhao et al. (2019), Park et al. (2018), and Garg et al. (2019) are not able to perform post-hoc debiasing and require changing the data or underlying word embeddings and retraining which is costly. Cordia and Bowman (2019) only study word-level language models and also requires re-training. Finally, Kurita et al. (2019) only measure bias on BERT by extending the word-level Word Embedding Association Test (WEAT) (Caliskan et al., 2017) metric in a manner similar to May et al. (2019).
In this paper, as a compelling step towards generalizing debiasing methods to sentence representations, we capture the various ways in which bias-attribute words can be used in natural sentences. This is performed by contextualizing bias-attribute words using a diverse set of sentence templates from various text corpora into bias-attribute sentences. We propose SENT-DEBIAS, an extension of the HARD-DEBIAS method (Bolukbasi et al., 2016), to debias sentences for both binary $^{1}$  and multiclass bias attributes spanning gender and religion. Key to our approach is the contextualization step in which bias-attribute words are converted into bias-attribute sentences by using a diverse set
Binary Gender	Multiclass Religion
man, woman	jewish, christian, muslim
he, she	torah, bible, quran
father, mother	synagogue, church, mosque
son, daughter	rabbi, priest, imam
Table 1: Examples of word pairs to estimate the binary gender bias subspace and the 3-class religion bias subspace in our experiments.
of sentence templates from text corpora. Our experimental results demonstrate the importance of using a large number of diverse sentence templates when estimating bias subspaces of sentence representations. Our experiments are performed on two widely popular sentence encoders BERT (Devlin et al., 2019) and ELMo (Peters et al., 2018), showing that our approach reduces the bias while preserving performance on downstream sequence tasks. We end with a discussion about possible shortcomings and present some directions for future work towards accurately characterizing and removing social biases from sentence representations for fairer NLP.
2 Debiasing Sentence Representations
Our proposed method for debiasing sentence representations, SENT-DEBIAS, consists of four steps: 1) defining the words which exhibit bias attributes, 2) contextualizing these words into bias attribute sentences and subsequently their sentence representations, 3) estimating the sentence representation bias subspace, and finally 4) debiasing general sentences by removing the projection onto this bias subspace. We summarize these steps in Algorithm 1 and describe the algorithmic details in the following subsections.
1) Defining Bias Attributes: The first step involves identifying the bias attributes and defining a set of bias attribute words that are indicative of these attributes. For example, when characterizing bias across the male and female genders, we use the word pairs (man, woman), (boy, girl) that are indicative of gender. When estimating the 3-class religion subspace across the Jewish, Christian, and Muslim religions, we use the tuples (Judaism, Christianity, Islam), (Synagogue, Church, Mosque). Each tuple should consist of words that have an equivalent meaning except for the bias attribute. In general, for  $d$ -class bias attributes, the set of words forms a dataset  $\mathcal{D} = \{(w_1^{(i)}, \dots, w_d^{(i)})\}_{i=1}^m$  of  $m$  entries where each entry  $(w_1, \dots, w_d)$  is a  $d$ -tuple of words that are each representative of a particular
<sup>1</sup>Although we recognize that gender is non-binary and there are many important ethical principles in the design, ascription of categories/variables to study participants, and reporting of results in studying gender as a variable in NLP (Larson, 2017), for the purpose of this study, we follow existing research and focus on female and male gendered terms.
bias attribute (we drop the superscript  ${}^{(i)}$  when it is clear from the context). Table 1 shows some bias attribute words that we use to estimate the bias subspaces for binary gender and multiclass religious attributes (full pairs and triplets in appendix).
Existing methods that investigate biases tend to operate at the word-level which simplifies the problem since the set of tokens is bounded by the vocabulary size (Bolukbasi et al., 2016). This simple approach has the advantage of identifying the presence of biases using predefined sets of word associations, and estimate the bias subspace using the predefined bias word pairs. On the other hand, the potential number of sentences are unbounded which makes it harder to precisely characterize the sentences in which bias is present or absent. Therefore, it is not trivial to directly convert these words to sentences to obtain a representation from pretrained sentence encoders. In the subsection below, we describe our solution to this problem.
2) Contextualizing Words into Sentences: A core step in our SENT-DEBIAS approach involves contextualizing the predefined sets of bias attribute words to sentences so that sentence encoders can be applied to obtain sentence representations. One option is to use a simple template-based design to simplify the contextual associations a sentence encoder makes with a given term, similar to how May et al. (2019) proposed to measure (but not remove) bias in sentence representations. For example, each word can be slotted into templates such as "This is <word>.", "I am a <word>". We take an alternative perspective and hypothesize that for a given bias attribute (e.g. gender), a single bias subspace
exists across all possible sentence representations. For example, the bias subspace should be the same in the sentences "The boy is coding.", "The girl is coding.", "The boys at the playground.", and "The girls at the playground". In order to estimate this bias subspace accurately, it becomes important to use sentence templates that are as diverse as possible to account for all occurrences of that word in surrounding contexts. In our experiments, we empirically demonstrate that estimating the bias subspace using a large and diverse set of templates from text corpora leads to improved bias reduction as compared to using simple templates.
To capture the variety in syntax across sentences, we use large text corpora to find naturally occurring sentences. These naturally occurring sentences therefore become our sentence "templates". To use these templates to generate new sentences, we replace words representing a single class with another. For example, a sentence containing a male term "he" is used to generate a new sentence but replacing it with the corresponding female term "she". This contextualization process is repeated for all word tuples in the bias attribute word dataset  $\mathcal{D}$ , eventually contextualizing the given set of bias attribute words into bias attribute sentences. Since there are multiple templates which a bias attribute word can map to, the contextualization process results in a bias attribute sentence dataset  $\mathcal{S}$  which is substantially larger in size:
$$
\begin{array}{l} \mathcal {S} = \bigcup_ {i = 1} ^ {m} \text {C o n t e x t u a l i z e} \left(w _ {1} ^ {(i)}, \dots , w _ {d} ^ {(i)}\right) (1) \\ = \left\{\left(s _ {1} ^ {(i)}, \dots , s _ {d} ^ {(i)}\right) \right\} _ {i = 1} ^ {n}, | \mathcal {S} | > | \mathcal {D} | (2) \\ \end{array}
$$
Dataset	Type	Topics	Formality	Length	Samples
WikiText-2	written	everything	formal	24.0	“the mailing contained information about their history and advised people to read several books, which primarily focused on {jewish/christian/muslim} history”
SST	written	movie reviews	informal	19.2	“{his/her} fans walked out muttering words like horrible and terrible, but had so much fun dissing the film that they didn’t mind the ticket cost.”
Reddit	written	politics, electronics, relationships	informal	13.6	“roommate cut my hair without my consent, ended up cutting {himself/herself} and is threatening to call the police on me”
MELD	spoken	comedy TV-series	informal	8.1	“that’s the kind of strength that I want in the {man/woman} I love!”
POM	spoken	opinion videos	informal	16.0	“and {his/her} family is, like, incredibly confused”
Table 2: Comparison of the various datasets used to find natural sentence templates. Length represents the average length measured by the number of words in a sentence. Words in italics indicate the words used to estimating the binary gender or multiclass religion subspaces, e.g. (man, woman), (jewish, christian, muslim). This demonstrates the variety in our naturally occurring sentence templates in terms of topics, formality, and spoken/written text.
where CONTEXTUALIZE  $(w_{1},\dots,w_{d})$  is a function which returns a set of sentences obtained by matching words with naturally-occurring sentence templates from text corpora.
Our text corpora originate from the following five sources: 1) WikiText-2 (Merit et al., 2017a), a dataset of formally written Wikipedia articles (we only use the first  $10\%$  of WikiText-2 which we found to be sufficient to capture formally written text), 2) Stanford Sentiment Treebank (Socher et al., 2013), a collection of 10000 polarized written movie reviews, 3) Reddit data collected from discussion forums related to politics, electronics, and relationships, 4) MELD (Poria et al., 2019), a large-scale multimodal multi-party emotional dialog dataset collected from the TV-series Friends, and 5) POM (Park et al., 2014), a dataset of spoken review videos collected across 1,000 individuals spanning multiple topics. These datasets have been the subject of recent research in language understanding (Merit et al., 2017b; Liu et al., 2019; Wang et al., 2019) and multimodal human language (Liang et al., 2018, 2019). Table 2 summarizes these datasets. We also give some examples of the diverse templates that occur naturally across various individuals, settings, and in both written and spoken text.
3) Estimating the Bias Subspace: Now that we have contextualized all  $m$  word  $d$ -tuples in  $\mathcal{D}$  into  $n$  sentence  $d$ -tuples  $\mathcal{S}$ , we pass these sentences through a pre-trained sentence encoder (e.g. BERT, ELMo) to obtain sentence representations. Suppose we have a pre-trained encoder  $M_{\theta}$  with parameters  $\theta$ . Define  $\mathcal{R}_j, j \in [d]$  as sets that collect all sentence representations of the  $j$ -th entry in the  $d$ -tuple,  $\mathcal{R}_j = \{M_{\theta}(s_j^{(i)})\}_{i=1}^n$ . Each of these sets  $\mathcal{R}_j$  defines a vector space in which a specific bias
attribute is present across its contexts. For example, when dealing with binary gender bias,  $\mathcal{R}_1$  (likewise  $\mathcal{R}_2$ ) defines the space of sentence representations with a male (likewise female) context. The only difference between the representations in  $\mathcal{R}_1$  versus  $\mathcal{R}_2$  should be the specific bias attribute present. Define the mean of set  $j$  as  $\pmb{\mu}_{j} = \frac{1}{|\mathcal{R}_{j}|}\sum_{\mathbf{w}\in \mathcal{R}_{j}}\mathbf{w}$ . The bias subspace  $\mathbf{V} = \{\mathbf{v}_1,\dots,\mathbf{v}_k\}$  is given by the first  $k$  components of principal component analysis (PCA) (Abdi and Williams, 2010):
$$
\mathbf {V} = \mathbf {P C A} _ {k} \left(\bigcup_ {j = 1} ^ {d} \bigcup_ {\mathbf {w} \in \mathcal {R} _ {j}} \left(\mathbf {w} - \boldsymbol {\mu} _ {j}\right)\right). \qquad (3)
$$
$k$  is a hyperparameter in our experiments which determines the dimension of the bias subspace. Intuitively,  $\mathbf{V}$  represents the top-  $k$  orthogonal directions which most represent the bias subspace.
4) Debiasing: Given the estimated bias subspace  $\mathbf{V}$ , we apply a partial version of the HARD-DEBIAS algorithm (Bolukbasi et al., 2016) to remove bias from new sentence representations. Taking the example of binary gender bias, the HARD-DEBIAS algorithm consists of two steps:
4a) Neutralize: Bias components are removed from sentences that are not gendered and should not contain gender bias (e.g., I am a doctor., That nurse is taking care of the patient.) by removing the projection onto the bias subspace. More formally, given a representation  $\mathbf{h}$  of a sentence and the previously estimated gender subspace  $\mathbf{V} = \{\mathbf{v}_1,\dots,\mathbf{v}_k\}$ , the debiased representation  $\hat{\mathbf{h}}$  is given by first obtaining  $\mathbf{h}_{\mathbf{V}}$ , the projection of  $\mathbf{h}$  onto the bias subspace  $\mathbf{V}$  before subtracting  $\mathbf{h}_{\mathbf{V}}$  from  $\mathbf{h}$ . This results in a vector that is orthogonal to the bias subspace
5505
$\mathbf{V}$  and therefore contains no bias:
$$
\mathbf {h} _ {\mathbf {V}} = \sum_ {j = 1} ^ {k} \left\langle \mathbf {h}, \mathbf {v} _ {j} \right\rangle \mathbf {v} _ {j}, \tag {4}
$$
$$
\hat {\mathbf {h}} = \mathbf {h} - \mathbf {h} _ {\mathbf {V}}. \tag {5}
$$
4b) Equalize: Gendered representations are centered and their bias components are equalized (e.g. man and woman should have bias components in opposite directions, but of the same magnitude). This ensures that any neutral words are equidistant to biased words with respect to the bias subspace. In our implementation, we skip this Equalize step because it is hard to identify all or even the majority of sentence pairs to be equalized due to the complexity of natural sentences. For example, we can never find all the sentences that man and woman appear in to equalize them appropriately. Note that even if the magnitudes of sentence representations are not normalized, the debiased representations are still pointing in directions orthogonal to the bias subspace. Therefore, skipping the equalize step still results in debiased sentence representations as measured by our definition of bias.
We test the effectiveness of SENT-DEBIAS at removing biases and retaining performance on downstream tasks. All experiments are conducted on English terms and downstream tasks. We acknowledge that biases can manifest differently across different languages, in particular gendered languages (Zhou et al., 2019), and emphasize the need for future extensions in these directions. Experimental details are in the appendix and code is released at https://github.com/pliang279/ sent_debias.
3.1 Evaluating Biases
Biases are traditionally measured using the Word Embedding Association Test (WEAT) (Caliskan et al., 2017). WEAT measures bias in word embeddings by comparing two sets of target words to two sets of attribute words. For example, to measure social bias surrounding genders with respect to careers, one could use the target words programmer, engineer, scientist, and nurse, teacher, librarian, and the attribute words man, male, and woman, female. Unbiased word representations should display no difference between the two target words in terms of their relative similarity to the two sets of attribute words. The relative similarity as measured by WEAT is commonly known as the effect
size. An effect size with absolute value closer to 0 represents lower bias.
To measure the bias present in sentence representations, we use the method as described in May et al. (2019) which extended WEAT to the Sentence Encoder Association Test (SEAT). For a given set of words for a particular test, words are converted into sentences using a template-based method. The WEAT metric can then be applied for fixed-length, pre-trained sentence representations. To measure bias over multiple classes, we use the Mean Average Cosine similarity (MAC) metric which extends SEAT to a multiclass setting (Manzini et al., 2019). For the binary gender setting, we use words from the Caliskan Tests (Caliskan et al., 2017) which measure biases in common stereotypes surrounding gendered names with respect to careers, math, and science (Greenwald et al., 2009). To evaluate biases in the multiclass religion setting, we modify the Caliskan Tests used in May et al. (2019) with lexicons used by Manzini et al. (2019).
3.2 Debiasing Setup
We first describe the details of applying SENT-DEBIAS on two widely-used sentence encoders: BERT $^2$  (Devlin et al., 2019) and ELMo (Peters et al., 2018). Note that the pre-trained BERT encoder must be fine-tuned on task-specific data. This implies that the final BERT encoder used during debiasing changes from task to task. To account for these differences, we report two sets of metrics: 1) BERT: simply debiasing the pre-trained BERT encoder, and 2) BERT post task: first fine-tuning BERT and post-processing (i.e. normalization) on a specific task before the final BERT representations are debiased. We apply SENT-DEBIAS on BERT fine-tuned on two single sentence datasets, Stanford Sentiment Treebank (SST-2) sentiment classification (Socher et al., 2013) and Corpus of Linguistic Acceptability (CoLA) grammatical acceptability judgment (Warstadt et al., 2018). It is also possible to apply BERT (Devlin et al., 2019) on downstream tasks that involve two sentences. The output sentence pair representation can also be debiased (after fine-tuning and normalization). We test the effect of SENT-DEBIAS on Question Natural Language Inference (QNLI) (Wang et al., 2018) which converts the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016) into a binary classification task. These results are
2We used uncased BERT-Base throughout all experiments.
Test	BERT	BERT post SST-2	BERT post CoLA	BERT post QNLI	ELMo
C6: M/F Names, Career/Family	+0.477 → -0.096	+0.036 → -0.109	-0.009 → +0.149	-0.261 → -0.054	-0.380 → -0.298
C6b: M/F Terms, Career/Family	+0.108 → -0.437	+0.010 → -0.057	+0.199 → +0.186	-0.155 → -0.004	-0.345 → -0.327
C7: M/F Terms, Math/Arts	+0.253 → +0.194	-0.219 → -0.221	+0.268 → +0.311	-0.584 → -0.083	-0.479 → -0.487
C7b: M/F Names, Math/Arts	+0.254 → +0.194	+1.153 → -0.755	+0.150 → +0.308	-0.581 → -0.629	+0.016 → -0.013
C8: M/F Terms, Science/Arts	+0.399 → -0.075	+0.103 → +0.081	+0.425 → -0.163	-0.087 → +0.716	-0.296 → -0.327
C8b: M/F Names, Science/Arts	+0.636 → +0.540	-0.222 → -0.047	+0.032 → -0.192	-0.521 → -0.443	+0.554 → +0.548
Multiclass Caliskan	+0.035 → +0.379	+1.200 → +1.000	+0.243 → +0.757	-	-
reported as BERT post SST-2, BERT post CoLA, and BERT post QNLI respectively.
For ELMo, the encoder stays the same for downstream tasks (no fine-tuning on different tasks) so we just debias the ELMo sentence encoder. We report this result as ELMo.
3.3 Debiasing Results