SlowGuess/ACL_2020_OCR huggingface dataset:
Tianlu Wang $^{1*}$  Xi Victoria Lin $^{2}$  Nazneen Fatema Rajani $^{2}$
Bryan McCann² Vicente Ordonez¹ Caiming Xiong²
<sup>1</sup>University of Virginia {tw8cb, vicente} @ virginia.edu
$^{2}$ Salesforce Research  $\{\mathrm{xilin, nazneen.rajani, bmccann, cxiong}\} @$ salesforce.com
Word embeddings derived from human-generated corpora inherit strong gender bias which can be further amplified by downstream models. Some commonly adopted debiasing approaches, including the seminal Hard Debias algorithm (Bolukbasi et al., 2016), apply post-processing procedures that project pre-trained word embeddings into a subspace orthogonal to an inferred gender subspace. We discover that semantic-agnostic corpus regularities such as word frequency captured by the word embeddings negatively impact the performance of these algorithms. We propose a simple but effective technique, Double-Hard Debias, which purifies the word embeddings against such corpus regularities prior to inferring and removing the gender subspace. Experiments on three bias mitigation benchmarks show that our approach preserves the distributional semantics of the pre-trained word embeddings while reducing gender bias to a significantly larger degree than prior approaches.
Despite widespread use in natural language processing (NLP) tasks, word embeddings have been criticized for inheriting unintended gender bias from training corpora. Bolukbasi et al. (2016) highlights that in word2vec embeddings trained on the Google News dataset (Mikolov et al., 2013a), "programmer" is more closely associated with "man" and "homemaker" is more closely associated with "woman". Such gender bias also propagates to downstream tasks. Studies have shown that coreference resolution systems exhibit gender bias in predictions due to the use of biased word embeddings (Zhao et al., 2018a; Rudinger et al., 2018). Given the fact that pre-trained word embeddings
have been integrated into a vast number of NLP models, it is important to debias word embeddings to prevent discrimination in NLP systems.
To mitigate gender bias, prior work have proposed to remove the gender component from pre-trained word embeddings through post-processing (Bolukbasi et al., 2016), or to compress the gender information into a few dimensions of the embedding space using a modified training scheme (Zhao et al., 2018b; Kaneko and Bollegala, 2019). We focus on post-hoc gender bias mitigation for two reasons: 1) debiasing via a new training approach is more computationally expensive; and 2) pre-trained biased word embeddings have already been extensively adopted in downstream NLP products and post-hoc bias mitigation presumably leads to less changes in the model pipeline since it keeps the core components of the original embeddings.
Existing post-processing algorithms, including the seminal Hard Debias (Bolukbasi et al., 2016), debias embeddings by removing the component that corresponds to a gender direction as defined by a list of gendered words. While Bolukbasi et al. (2016) demonstrates that such methods alleviate gender bias in word analogy tasks, Gonen and Goldberg (2019) argue that the effectiveness of these efforts is limited, as the gender bias can still be recovered from the geometry of the debiased embeddings.
We hypothesize that it is difficult to isolate the gender component of word embeddings in the manner employed by existing post-processing methods. For example, Gong et al. (2018); Mu and Viswanath (2018) show that word frequency significantly impact the geometry of word embeddings. Consequently, popular words and rare words cluster in different subregions of the embedding space, despite the fact that words in these clusters are not semantically similar. This can degrade the ability of component-based methods for debiasing gender.
*This research was conducted during the author's internship at Salesforce Research.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5443-5453 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
(a) Change the frequency of "boy".
(b) Change the frequency of "daughter".
Figure 1:  $\Delta$  of cosine similarities between gender difference vectors before / after adjusting the frequency of word  $w$ . When the frequency of  $w$  changes, the cosine similarities between the gender difference vector  $(\overrightarrow{v})$  for  $w$  and other gender difference vectors exhibits a large change. This demonstrates that frequency statistics for  $w$  have a strong influence on the gender direction represented by  $\overrightarrow{v}$ .
Specifically, recall that Hard Debias seeks to remove the component of the embeddings corresponding to the gender direction. The important assumption made by Hard Debias is that we can effectively identify and isolate this gender direction. However, we posit that word frequency in the training corpora can twist the gender direction and limit the effectiveness of Hard Debias.
To this end, we propose a novel debiasing algorithm called Double-Hard Debias that builds upon the existing Hard Debias technique. It consists of two steps. First, we project word embeddings into an intermediate subspace by subtracting component(s) related to word frequency. This mitigates the impact of frequency on the gender direction. Then we apply Hard Debias to these purified embeddings to mitigate gender bias. Mu and Viswanath (2018) showed that typically more than one dominant directions in the embedding space encode frequency features. We test the effect of each dominant direction on the debiasing performance and only remove the one(s) that demonstrated the most impact.
We evaluate our proposed debiasing method using a wide range of evaluation techniques. According to both representation level evaluation (WEAT test (Caliskan et al., 2017), the neighborhood metric (Gonen and Goldberg, 2019)) and downstream task evaluation (coreference resolution (Zhao et al., 2018a)), Double-Hard Debias outperforms all pre
vious debiasing methods. We also evaluate the functionality of debiased embeddings on several benchmark datasets to demonstrate that Double-Hard Debias effectively mitigates gender bias without sacrificing the quality of word embeddings<sup>1</sup>.
2 Motivation
Current post-hoc debiasing methods attempt to reduce gender bias in word embeddings by subtracting the component associated with gender from them. Identifying the gender direction in the word embedding space requires a set of gender word pairs,  $\mathcal{P}$ , which consists of "she & he", "daughter & son", etc. For every pair, for example "boy & girl", the difference vector of the two embeddings is expected to approximately capture the gender direction:
$$
\overrightarrow {v} _ {\text {b o y , g i r l}} = \overrightarrow {w} _ {\text {b o y}} - \overrightarrow {w} _ {\text {g i r l}} \tag {1}
$$
Bolukbasi et al. (2016) computes the first principal component of ten such difference vectors and use that to define the gender direction. $^{2}$
Recent works (Mu and Viswanath, 2018; Gong et al., 2018) show that word frequency in a training
$^{1}$ Code and data are available at https://github.com/uvavision/Double-Hard-Debias.git
2The complete definition of  $\mathcal{P}$  is: "woman & man", "girl & boy", "she & he", "mother & father", "daughter & son", "gal & guy", "female & male", "her & his", "herself & himself", and "Mary & John" (Bolukbasi et al., 2016).
corpus can degrade the quality of word embeddings. By carefully removing such frequency features, existing word embeddings can achieve higher performance on several benchmarks after fine-tuning. We hypothesize that such word frequency statistics also interferes with the components of the word embeddings associated with gender. In other words, frequency-based features learned by word embedding algorithms act as harmful noise in the previously proposed debiasing techniques.
To verify this, we first retrain GloVe (Pennington et al., 2014) embeddings on the one billion English word benchmark (Chelba et al., 2013) following previous work (Zhao et al., 2018b; Kaneko and Bollegala, 2019). We obtain ten difference vectors for the gendered pairs in  $\mathcal{P}$  and compute pairwise cosine similarity. This gives a similarity matrix  $S$  in which  $S_{p_i,p_j}$  denotes the cosine similarity between difference vectors  $\vec{v}_{pair_i}$  and  $\vec{v}_{pair_j}$ .
We then select a specific word pair, e.g. "boy" & "girl", and augment the corpus by sampling sentences containing the word "boy" twice. In this way, we produce a new training corpus with altered word frequency statistics for "boy". The context around the token remains the same so that changes to the other components are negligible. We retrain GloVe with this augmented corpus and get a set of new offset vectors for the gendered pairs  $\mathcal{P}$ . We also compute a second similarity matrix  $S^{\prime}$  where  $S_{pi,p_j}^{\prime}$  denotes the cosine similarity between difference vectors  $\vec{v}_{pair_i}^{\prime}$  and  $\vec{v}_{pair_j}^{\prime}$ .
By comparing these two similarity matrices, we analyze the effect of changing word frequency statistics on gender direction. Note that the offset vectors are designed for approximating the gender direction, thus we focus on the changes in offset vectors. Because statistics were altered for "boy", we focus on the difference vector  $\vec{v}_{boy,girl}$  and make two observations. First, the norm of  $\vec{v}_{boy,girl}$  has a  $5.8\%$  relative change while the norms of other difference vectors show much smaller changes. For example, the norm of  $\vec{v}_{man,woman}$  only changes by  $1.8\%$ . Second, the cosine similarities between  $\vec{v}_{boy,girl}$  and other difference vectors also show more significant change, as highlighted by the red bounding box in Figure 1a. As we can see, the frequency change of "boy" leads to deviation of the gender direction captured by  $\vec{v}_{boy,girl}$ . We observe similar phenomenon when we change the frequency of the word "daughter" and present these results in Figure 1b.
Based on these observations, we conclude that word frequency plays an important role in gender debiasing despite being overlooked by previous works.
In this section, we first summarize the terminology that will be used throughout the rest of the paper, briefly review the Hard Debias method, and provide background on the neighborhood evaluation metric. Then we introduce our proposed method: Double-Hard Debias.
3.1 Preliminary Definitions
Let  $W$  be the vocabulary of the word embeddings we aim to debias. The set of word embeddings contains a vector  $\vec{w} \in \mathbb{R}^n$  for each word  $w \in W$ . A subspace  $B$  is defined by  $k$  orthogonal unit vectors  $B = \{b_1, \ldots, b_k\} \in \mathbb{R}^d$ . We denote the projection of vector  $v$  on  $B$  by
$$
v _ {B} = \sum_ {j = 1} ^ {k} (v \cdot b _ {j}) b _ {j}. \tag {2}
$$
Following (Bolukbasi et al., 2016), we assume there is a set of gender neutral words  $N \subset W$  such as "doctor" and "teacher", which by definition are not specific to any gender. We also assume a pre-defined set of  $n$  male-female word pairs  $D_{1}, D_{2}, \ldots, D_{n} \subset W$ , where the main difference between each pair of words captures gender.
Hard Debias. The Hard Debias algorithm first identifies a subspace that captures gender bias. Let
$$
\mu_ {i} := \sum_ {w \in D _ {i}} \overrightarrow {w} / | D _ {i} |. \tag {3}
$$
The bias subspace  $B$  is the first  $k$ $(\geq 1)$  rows of SVD(C), where
$$
\mathbf {C} := \sum_ {i = 1} ^ {m} \sum_ {w \in D _ {i}} \left(\vec {w} - \mu_ {i}\right) ^ {T} \left(\vec {w} - \mu_ {i}\right) / \left| D _ {i} \right| \tag {4}
$$
Following the original implementation of Bolukbasi et al. (2016), we set  $k = 1$ . As a result the subspace  $B$  is simply a gender direction. $^3$
Hard Debias then neutralizes the word embeddings by transforming each  $\overrightarrow{w}$  such that every word
3Bolukbasi et al. (2016) normalize all embeddings. However, we found it is unnecessary in our experiments. This is also mentioned in Ethayarajh et al. (2019)
Figure 2: Clustering accuracy after projecting out D-th dominating direction and applying Hard Debias. Lower accuracy indicates less bias.
$w\in N$  has zero projection in the gender subspace. For each word  $w\in N$  , we re-embed  $\overrightarrow{w}$  ..
$$
\overrightarrow {w} := \overrightarrow {w} - \overrightarrow {w} _ {B} \tag {5}
$$
Neighborhood Metric. The Neighborhood Metric proposed by (Gonen and Goldberg, 2019) is a bias measurement that does not rely on any specific gender direction. To do so it looks into similarities between words. The bias of a word is the proportion of words with the same gender bias polarity among its nearest neighboring words.
We selected  $k$  of the most biased male and females words according to the cosine similarity of their embedding and the gender direction computed using the word embeddings prior to bias mitigation. We use  $W_{m}$  and  $W_{f}$  to denote the male and female biased words, respectively. For  $w_{i} \in W_{m}$ , we assign a ground truth gender label  $g_{i} = 0$ . For  $w_{i} \in W_{f}$ ,  $g_{i} = 1$ . Then we run KMeans  $(k = 2)$  to cluster the embeddings of selected words  $\hat{g}_{i} = KMeans(\vec{w}_{i})$ , and compute the alignment score  $a$  with respect to the assigned ground truth gender labels:
$$
a = \frac {1}{2 k} \sum_ {i = 1} ^ {2 k} \mathbb {1} [ \hat {g} _ {i} = = g _ {i} ] \tag {6}
$$
We set  $a = \max(a, 1 - a)$ . Thus, a value of 0.5 in this metric indicates perfectly unbiased word embeddings (i.e. the words are randomly clustered), and a value closer to 1 indicates stronger gender bias.
3.2 Double-Hard Debiasing
According to Mu and Viswanath (2018), the most statistically dominant directions of word embeddings encode word frequency to a significant extent. Mu and Viswanath (2018) removes these frequency features by centralizing and subtracting components along the top  $D$  dominant directions
from the original word embeddings. These postprocessed embeddings achieve better performance on several benchmark tasks, including word similarity, concept categorization, and word analogy. It is also suggested that setting  $D$  near  $d / 100$  provides maximum benefit, where  $d$  is the dimension of a word embedding.
We speculate that most the dominant directions also affect the geometry of the gender space. To address this, we use the aforementioned clustering experiment to identify whether a direction contains frequency features that alter the gender direction.
More specifically, we first pick the top biased words (500 male and 500 female) identified using the original GloVe embeddings. We then apply PCA to all their word embeddings and take the top principal components as candidate directions to drop. For every candidate direction  $\mathbf{u}$ , we project the embeddings into a space that is orthogonal to  $\mathbf{u}$ . In this intermediate subspace, we apply Hard Debias and get debiased embeddings. Next, we cluster the debiased embeddings of these words
and compute the gender alignment accuracy (Eq. 6). This indicates whether projecting away direction  $\mathbf{u}$  improves the debiasing performance. Algorithm 1 shows the details of our method in full.
We found that for GloVe embeddings pre-trained on Wikipedia dataset, elimination of the projection along the second principal component significantly decreases the clustering accuracy. This translates to better debiasing results, as shown in Figure 2. We further demonstrate the effectiveness of our method for debaising using other evaluation metrics in Section 4.
In this section, we compare our proposed method with other debiasing algorithms and test the functionality of these debiased embeddings on word analogy and concept categorization task. Experimental results demonstrate that our method effectively reduces bias to a larger extent without degrading the quality of word embeddings.
We use 300-dimensional GloVe (Pennington et al., 2014)  $^4$  embeddings pre-trained on the 2017 January dump of English Wikipedia, containing 322,636 unique words. To identify the gender direction, we use 10 pairs of definitional gender words compiled by (Bolukbasi et al., 2016) $^6$ .
We compare our proposed method against the following baselines:
GloVe: the pre-trained GloVe embeddings on Wikipedia dataset described in 4.1. GloVe is widely used in various NLP applications. This is a nondebiased baseline for comparison.
GN-GloVe: We use debiased Gender-Neutral GN-GloVe embeddings released by the original authors (Zhao et al., 2018b). GN-GloVe restricts gender information in certain dimensions while neutralizing the rest dimensions.
GN-GloVe  $(w_{a})$ : We exclude the gender dimensions from GN-GloVe. This baseline tries to completely remove gender.
GP-GloVe: We use debiased embeddings released by the original authors (Kaneko and Bollegala,
2019). Gender-preserving Debiasing attempts to preserve non-discriminative gender information, while removing stereotypical gender bias.
GP-GN-GloVe:: This baseline applies Gender-preserving Debiasing on already debaised GN-GloVe embeddings. We also use debiased embeddings provided by authors.
Hard-GloVe: We apply Hard Debias introduced in (Bolukbasi et al., 2016) on GloVe embeddings. Following the implementation provided by original authors, we debias netural words and preserve the gender specific words.
Strong Hard-GloVe: A variant of Hard Debias where we debias all words instead of avoiding gender specific words. This seeks to entirely remove gender from GloVe embeddings.
Double-Hard GloVe: We debias the pre-trained GloVe embeddings by our proposed Double-Hard Debias method.
4.3 Evaluation of Debiasing Performance
We demonstrate the effectiveness of our debiasing method for downstream applications and according to general embedding level evaluations.
4.3.1 Debiasing in Downstream Applications
Coreference Resolution. Coreference resolution aims at identifying noun phrases referring to the same entity. Zhao et al. (2018a) identified gender bias in modern coreference systems, e.g. "doctor" is prone to be linked to "he". They also introduce a new benchmark dataset WinoBias, to study gender bias in coreference systems.
WinoBias provides sentences following two prototypical templates. Each type of sentences can be divided into a pro-stereotype (PRO) subset and a antistereotype (ANTI) subset. In the PRO subset, gender pronouns refer to professions dominated by the same gender. For example, in sentence "The physician hired the secretary because he was overwhelmed with clients.", "he" refers to "physician", which is consistent with societal stereotype. On the other hand, the ANTI subset consists of same sentences, but the opposite gender pronouns. As such, "he" is replaced by "she" in the aforementioned example. The hypothesis is that gender cues may distract a coreference model. We consider a system to be gender biased if it performs better in pro-stereotypical scenarios than in anti-stereotypical scenarios.
4Experiments on Word2Vec are included in the appendix.
<sup>5</sup>https://github.com/uclanlp/gn_glove
<https://github.com/tolga-b/debiaswe>
Embeddings	OntoNotes	PRO-1	ANTI-1	Avg-1	|Diff-1|	PRO-2	ANTI-2	Avg-2	|Diff-2|
GloVe	66.5	77.7	48.2	62.9	29.0	82.7	67.5	75.1	15.2
GN-GloVe	66.1	68.4	56.5	62.5	12.0	78.2	71.3	74.7	6.9
GN-GloVe(wa)	66.4	66.7	56.6	61.6	10.2	79.0	72.3	75.7	6.7
GP-GloVe	66.1	72.0	52.0	62.0	20.0	78.5	70.0	74.3	8.6
GP-GN-GloVe	66.3	70.0	54.5	62.0	15.0	79.9	70.7	75.3	9.2
Hard-GloVe	66.2	72.3	52.7	62.6	19.7	80.6	78.3	79.4	2.3
Strong Hard-GloVe	66.0	69.0	58.6	63.8	10.4	82.2	78.6	80.4	3.6
Double-Hard GloVe	66.4	66.0	58.3	62.2	7.7	85.4	84.5	85.0	0.9
Table 1: F1 score (\%) of coreference systems on OntoNotes test set and WinoBias dataset. |Diff | represents the performance gap between pro-stereotype (PRO) subset and anti-stereotype (ANTI) subset. Coreference system trained on our Double-Hard GloVe embeddings has the smallest |Diff | values, suggesting less gender bias.
We train an end-to-end coreference resolution model (Lee et al., 2017) with different word embeddings on OntoNotes 5.0 training set and report the performance on WinoBias dataset. Results are presented in Table1. Note that absolute performance difference (Diff) between the PRO set and ANTI set connects with gender bias. A smaller Diff value indicates a less biased coreference system. We can see that on both types of sentences in WinoBias, Double-Hard GloVe achieves the smallest Diff compared to other baselines. This demonstrates the efficacy of our method. Meanwhile, Double-Hard GloVe maintains comparable performance as GloVe on OntoNotes test set, showing that our method preserves the utility of word embeddings. It is also worth noting that by reducing gender bias, Double-Hard GloVe can significantly improve the average performance on type-2 sentences, from  $75.1\%$  (GloVe) to  $85.0\%$ .
4.3.2 Debiasing at Embedding Level
The Word Embeddings Association Test (WEAT). WEAT is a permutation test used to measure the bias in word embeddings. We consider male names and females names as attribute sets and compute the differential association of two sets of target words<sup>7</sup> and the gender attribute sets. We report effect sizes  $(d)$  and p-values  $(p)$  in Table2. The effect size is a normalized measure of how separated the two distributions are. A higher value of effect size indicates larger bias between target words with regard to gender. p-values denote if the bias is significant. A high p-value (larger than 0.05) indicates the bias is insignificant. We refer readers to Caliskan et al. (2017) for more details.
As shown in Table 2, across different target words sets, Double-Hard GloVe consistently outperforms other debiased embeddings. For Career & Family and Science & Arts, Double-Hard GloVe reaches the lowest effect size, for the latter one, Double-Hard GloVe successfully makes the bias insignificant (p-value  $>0.05$ ). Note that in WEAT test, some debiasing methods run the risk of amplifying gender bias, e.g. for Math & Arts words, the bias is significant in GN-GloVe while it is insignificant in original GloVe embeddings. Such concern does not occur in Double-Hard GloVe.
Neighborhood Metric. (Gonen and Goldberg, 2019) introduces a neighborhood metric based on clustering. As described in Sec 3.1, We take the top  $k$  most biased words according to their cosine similarity with gender direction in the original GloVe embedding space<sup>8</sup>. We then run k-Means to cluster them into two clusters and compute the alignment accuracy with respect to gender, results are presented in Table 3. We recall that in this metric, a accuracy value closer to 0.5 indicates less biased word embeddings.
Using the original GloVe embeddings, k-Means can accurately cluster selected words into a male group and a female group, suggesting the presence of a strong bias. Hard Debias is able to reduce bias in some degree while other baselines appear to be less effective. Double-Hard GloVe achieves the lowest accuracy across experiments clustering top 100/500/1000 biased words, demonstrating that the proposed technique effectively reduce gender bias. We also conduct tSNE (van der Maaten and Hinton, 2008) projection for all baseline embed
<sup>7</sup>All word lists are from Caliskan et al. (2017). Because GloVeembeddings are uncased, we use lower cased people names and replace "bill" with "tom" to avoid ambiguity.
To be fair, we exclude all gender specific words used in debiasing, so Hard-GloVe and Strong Hard-GloVe have same accuracy performance in Table 3
Embeddings	Career & Family	Math & Arts	Science & Arts
d	p	d	p	d	p
GloVe	1.81	0.0	0.55	0.14	0.88	0.04
GN-GloVe	1.82	0.0	1.21	6e-3	1.02	0.02
GN-GloVe(wa)	1.76	0.0	1.43	1e-3	1.02	0.02
GP-GloVe	1.81	0.0	0.87	0.04	0.91	0.03
GP-GN-GloVe	1.80	0.0	1.42	1e-3	1.04	0.01
Hard-GloVe	1.55	2e-4	0.07	0.44	0.16	0.62
Strong Hard-GloVe	1.55	2e-4	0.07	0.44	0.16	0.62
Double-Hard GloVe	1.53	2e-4	0.09	0.57	0.15	0.61
dings. As shown in Figure 3, original non-debiased GloVe embeddings are clearly projected to different regions. Double-Hard GloVe mixes up male and female embeddings to the maximum extent compared to other baselines, showing less gender information can be captured after debiasing.