Venue: NAACL 2019

# Black is to Criminal as Caucasian is to Police:

Detecting and Removing Multiclass Bias in Word Embeddings

Thomas Manzini\({}^{1*}\) , Yao Chong Lim\({}^{1*}\) , Yulia Tsvetkov\({}^{3}\), Alan W Black\({}^{4}\)

Microsoft AI Development Acceleration Program\({}^{1}\), Carnegie Mellon University\({}^{4}\)

Thomas.Manzini@microsoft.com, {yaochonl,ytsvetko,awb}@cs.cmu.edu

 Equal contributionsWork done while at CMU and The Microsoft AI Development Acceleration Program

###### Abstract

Online texts--across genres, registers, domains, and styles--are riddled with human stereotypes, expressed in overt or subtle ways. Word embeddings, trained on these texts, perpetuate and amplify these stereotypes, and propagate biases to machine learning models that use word embeddings as features. In this work, we propose a method to debias word embeddings in multiclass settings such as race and religion, extending the work of Bolubasi et al. (2016) from the binary setting, such as binary gender. Next, we propose a novel methodology for the evaluation of multiclass debiasing. We demonstrate that our multiclass debiasing is robust and maintains the efficacy in standard NLP tasks.

## 1 Introduction

In addition to possessing informative features useful for a variety of NLP tasks, word embeddings reflect and propagate social biases present in training corpora Caliskan et al. (2017); Garg et al. (2018). Machine learning systems that use embeddings can further amplify biases Barocas and Selbest (2016); Zhao et al. (2017), discriminating against users, particularly those from disadvantaged social groups.

Bolukbasi et al. (2016) introduced a method to _debias_ embeddings by removing components that lie in stereotype-related embedding subspaces. They demonstrate the effectiveness of the approach by removing gender bias from word2vec embeddings Mikolov et al. (2013), preserving the utility of embeddings and potentially alleviating biases in downstream tasks. However, this method was only for _binary_ labels (e.g., male/female), whereas most real-world demographic attributes, including gender, race, religion, are not binary but continuous or categorical, with more than two categories.

In this work, we show a generalization of Bolukbasi et al.'s (2016) which enables _multiclass_ debiasing, while preserving utility of embeddings (SS3). We train word2vec embeddings using the Reddit L2 corpus Rabinovich et al. (2018) and apply multiclass debiasing using lexicons from studies on bias in NLP and social science (SS4.2). We introduce a novel metric for evaluation of bias in collections of word embeddings (SS5). Finally, we validate that the utility of debiased embeddings in the tasks of part-of-speech (POS) tagging, named entity recognition (NER), and POS chunking is on par with off-the-shelf embeddings.

## 2 Background

As defined by Bolukbasi et al. (2016), debiasing word embeddings in a binary setting requires identifying the bias subspace of the embeddings. Components lying in that subspace are then removed

\begin{table}
\begin{tabular}{|p{99.6pt}|p{142.3pt}|} \hline \multicolumn{2}{|p{99.6pt}|}{**Gender Biased Analogies**} \\ \hline man → doctor woman → receptionist woman → secretary & woman → supervisor man → principal \\ \hline \multicolumn{2}{|p{99.6pt}|}{**Racially Biased Analogies**} \\ \hline black → criminalasia → doctor Caucasian → leader & caucasian → dd black → led \\ \hline \multicolumn{2}{|p{99.6pt}|}{**Religiously Biased Analogies**} \\ \hline muslim → terrorist jewish → philanthropist christian → unemployed & christian → stooge Jewish → pensioners \\ \hline \end{tabular}
\end{table}
Table 1: Examples of gender, racial, and religious biases in analogies generated from word embeddings trained on the Reddit data from users from the USA.



from each embedding.

### Identifying the bias subspace

Bolukbasi et al. (2016) define the gender subspace using _defining sets_ of words, where the words in each set represent different ends of the bias. For example, in the case of gender, one defining set might be the gendered pronouns {_he_, _she_} and another set might be the gendered nouns {_man_, _woman_}. The gender subspace is then computed from these defining sets by 1) computing the vector differences of the word embeddings of words in each set from the set's mean, and 2) taking the most significant components of these vectors.

#### Removing bias components

Following the identification of the gender sub-space, one can apply hard or soft debiasing Bolukbasi et al. (2016) to completely or partially remove the subspace components from the embeddings.

##### Hard debiasing

Hard debiasing (also called "Neutralize and Equalize") involves two steps. First, bias components are removed from words that are not gendered and should not contain gender bias (e.g., _doctor_, _nurse_), and second, gendered word embeddings are centered and their bias components are equalized. For example, in the binary case, _man_ and _woman_ should have bias components in opposite directions, but of the same magnitude. Intuitively, this then ensures that any neutral words are equidistant to any biased words with respect to the bias subspace.

More formally, to _neutralize_, given a bias subspace \(\mathcal{B}\) spanned by the vectors \(\{\mathbf{b_{1}},\mathbf{b_{2}},...,\mathbf{b_{k}}\}\), we compute the component of each embedding in this subspace:

\[\mathbf{w}_{\mathcal{B}}=\sum_{i=1}^{k}\langle\mathbf{w},\mathbf{b_{i}}\rangle\bm {b_{i}} \tag{1}\]

We then remove this component from words that should be bias-neutral and normalize to get the debiased embedding:

\[\mathbf{w}^{\prime}=\frac{\mathbf{w}-\mathbf{w}_{\mathcal{B}}}{\|\mathbf{w}- \mathbf{w}_{\mathcal{B}}\|} \tag{2}\]

To _equalize_ the embeddings of words in an equality set \(E\), let \(\mathbf{\mu}=\frac{1}{|E|}\sum_{\mathbf{w}\in E}\mathbf{w}\) be the mean embedding of the words in the set and \(\mathbf{\mu}_{\mathcal{B}}\) be its component in the bias subspace as calculated in Equation 1. Then, for \(\mathbf{w}\in E\),

\[\mathbf{w}^{\prime}=(\mathbf{\mu}-\mathbf{\mu}_{\mathcal{B}})+\sqrt{1-\|\mathbf{\mu}-\bm {\mu}_{\mathcal{B}}\|^{2}}\frac{\mathbf{w}_{\mathcal{B}}-\mathbf{\mu}_{\mathcal{ B}}}{\|\mathbf{w}_{\mathcal{B}}-\mathbf{\mu}_{\mathcal{B}}\|} \tag{3}\]

Note that in both Equations 2 and 3, the new embedding has unit length.

**Soft debiasing**

Soft debiasing involves learning a projection of the embedding matrix that preserves the inner product between biased and debiased embeddings while minimizing the projection onto the bias subspace of embeddings that should be neutral.

Given embeddings \(\mathbf{W}\) and \(\mathbf{N}\) which are embeddings for the whole vocabulary and the subset of bias-neutral words respectively, and the bias subspace \(\mathcal{B}\) obtained in Section 2.1, soft debiasing seeks for a linear transformation \(A\) that minimizes the following objective:

\[\begin{split}\|(\mathbf{AW})^{\intercal}&(\mathbf{ AW})-\mathbf{W}^{\intercal}\mathbf{W}\|_{F}^{2}\\ &+\lambda\|(\mathbf{AN})^{\intercal}(\mathbf{AQ})\|_{F}^{2} \end{split} \tag{4}\]

Minimizing the first term preserves the inner product after the linear transformation \(\mathbf{A}\), and minimizing the second term minimizes the projection onto the bias subspace \(\mathcal{B}\) of embeddings. \(\lambda\in\mathbb{R}\) is a tunable parameter that balances the two objectives.

## 3 Methodology

We now discuss our proposed extension of word embedding debiasing to the multiclass setting.

### Debiasing

As in the binary setting, debiasing consists of two steps: identifying the "bias subspace" and removing this component from the set of embeddings.

##### Identifying the bias subspace

The core contribution of our work is in _identifying_ the "bias subspace" in a multiclass setting; if we can identify the bias subspace then prior work can be used for multiclass debiasing.

Past work has shown that it is possible to linearly separate multiple social classes based on components of word embeddings Garg et al. (2018). Based on this we hypothesize that there exists some component of these embeddings which can capture multiclass bias. While a multiclass 

problem is inherently not a linearly separable problem, a one versus rest classifier is. Following from this, the computation of a multiclass bias subspace does not have any linear constraints, though it does come with a loss of resolution. As a result we can compute the principal components required to compute the "bias subspace" by simply adding an additional term for each additional bias class to each defining set.

Formally, given defining sets of word embeddings \(D_{1},D_{2},...,D_{n}\), let the mean of the defining set \(i\) be \(\mathbf{\mu}_{i}=\frac{1}{|D_{i}|}\sum_{\mathbf{w}\in D_{i}}\mathbf{w}\), where \(\mathbf{w}\) is the word embedding of \(w\). Then the bias subspace \(\mathcal{B}\) is given by the first \(k\) components of the following principal component analysis (PCA) evaluation:

\[\mathbf{PCA}\left(\bigcup^{n}_{i=1}\bigcup_{\mathbf{w}\in D_{i}}\mathbf{w}- \mathbf{\mu}_{i}\right) \tag{5}\]

The number of components \(k\) can be empirically determined by inspecting the eigenvalues of the PCA, or using a threshold. Also, note that the defining sets do not have to be the same size. We discuss the robustness of this method later.

#### Removing Bias Components

Following the identification of the bias subspace, we apply the hard Neutralize and Equalize debiasing and soft debiasing method presented in Bolukbasi et al. (2016) and discussed in Section 2.2 to completely or partially remove the subspace components from the embeddings.

For equalization, we take the defining sets to be the equality sets as well.

### Quantifying Bias Removal

We propose a new metric for the evaluation of bias in collections of words which is simply the mean average cosine similarity (MAC). This approach is motivated by the WEAT evaluation method proposed by Caliskan et al. (2017) but modified for a multiclass setting. To compute this metric the following data is required: a set of target word embeddings \(T\) containing terms that inherently contain some form of social bias (e.g. {_church_, _synagogue_, _mosque_}), and a set \(A\) which contains sets of attributes \(A_{1},A_{2},...,A_{N}\) containing word embeddings that should not be associated with any word embeddings contained in the set \(T\) (e.g. {_violent_, _liberal_, _conservative_}).

We define a function \(S\) that computes the mean cosine distance between a particular target \(T_{i}\) and all terms in a particular attribute set \(A_{j}\):

\[S(\mathbf{t},A_{j})=\frac{1}{N}\sum_{\mathbf{a}\in A_{j}}cos(\mathbf{t}, \mathbf{a}), \tag{6}\]

where the cosine distance is:

\[cos(\mathbf{u},\mathbf{v})=1-\frac{\mathbf{u}\cdot\mathbf{v}}{\lVert\mathbf{ u}\rVert_{2}\cdot\lVert\mathbf{v}\rVert_{2}}. \tag{7}\]

Finally, we define MAC as:

\[\text{MAC}(T,A)=\frac{1}{|T||A|}\sum_{T_{i}\in T}\sum_{A_{j}\in A}S(T_{i},A_{ j}) \tag{8}\]

We also perform a paired \(t\)-test on the distribution of average cosines used to calculate the MAC. Thus we can quantify the effect of debiasing on word embeddings in \(T\) and sets in \(A\).

### Measuring Downstream Utility

To measure the utility of the debiased word embeddings, we use the tasks of NER, POS tagging, and POS chunking. This is to ensure that the debiasing procedure has not destroyed the utility of the word embeddings. We evaluate test sentences that contain at least one word affected by debiasing. Additionally, we measure the change in performance after replacing the biased embedding matrix by a debiased one, and retraining the model on debiased embeddings.

## 4 Data

In this section we discuss the different data sources we used for our initial word embeddings, the social bias data used for evaluating bias, and the linguistic data used for evaluating the debiasing process.

\begin{table}
\begin{tabular}{|l|c|l|} \hline
**Gender Debiasing** & MAC & P-Value \\ \hline Biased & 0.623 & N/A \\ Hard Debiased & 1.000 & 1.582e-14 \\ Soft Debiased (\(\lambda\) = 0.2) & 0.747 & 1.711e-12 \\ \hline
**Race Debiasing** & MAC & P-Value \\ \hline Biased & 0.892 & N/A \\ Hard Debiased & 1.009 & 7.235e-04 \\ Soft Debiased (\(\lambda\) = 0.2) & 0.985 & 6.217e-05 \\ \hline
**Religion Debiasing** & MAC & P-Value \\ \hline Biased & 0.859 & N/A \\ Hard Debiased & 1.004 & 3.006e-07 \\ Soft Debiased (\(\lambda\) = 0.2) & 0.894 & 0.007 \\ \hline \end{tabular}
\end{table}
Table 2: The associated mean average cosine similarity (MAC) (defined in Section 3.2) and P-Values for debiasing methods for gender, race, and religious bias.

 



### Embedding Language Corpus

We used the L2-Reddit corpus [10], a collection of Reddit posts and comments by both native and non-native English speakers. The native countries of post authors are determined based on their posts in country- and region-specific subreddits (such as r/Europe and r/UnitedKingdom), and other metadata such as user flairs, which serve as self-identification of the user's country of origin.

In this work, we exclusively explore data collected from the United States. This was done to leverage extensive studies of social bias done in the United States. To obtain the initial biased word embeddings, we trained word2vec embeddings [35] using approximately 56 million sentences.

### Social Bias Data

We used the following vocabularies and studies to compile lexicons for bias detection and removal.1

Footnote 1: The lexicon used can be found here: [https://docs.google.com/spreadsheets/d/lBQBFIdv9BnuifxikjrcJNqLA@dxg2eAWk4kdu_OCUM](https://docs.google.com/spreadsheets/d/lBQBFIdv9BnuifxikjrcJNqLA@dxg2eAWk4kdu_OCUM).

For gender, we used vocabularies created by [10] and [7].

For race we consulted a number of different sources for each race: Caucasians [8, 10]; African Americans [11, 12, 13]; Brown Givens and Monahan, 2005; Chung-Herrera and Lankau, 2005; Hakanen, 1995; Welch, 2007; Kawai, 2005); and Asian Americans [10, 11, 12, 13, 14, 15, 16, 17, 18].

Finally, for religion we used the following sources and labels: Christians [14, 15, 16, 17, 18]; Jews [11, 12, 13]; and Muslims [15, 16, 17].

### Downstream Tasks

We evaluate biased and debiased word embeddings on several downstream tasks. Specifically, the CoNLL 2003 shared task [15, 16] which provides evaluation data for NER, POS tagging, and POS chunking.

## 5 Results and Discussion

In this section we review the results of our experiments and discuss what those results mean in the context of this work.

### Observations of Bias

We can observe bias in word embeddings in many different ways. However, for the purposes of demonstrating that bias exists in these word embeddings we use the analogy task that was used to demonstrate bias in [10]. We observe that bias is present in generated analogies by viewing them directly. A small subset of these analogies are in Table 1 to highlight our findings.

\begin{table}
\begin{tabular}{|c||c|c|c|c|c|c|c|c|c|} \hline  & \multicolumn{6}{c|}{Embedding Matrix Replacement} \\ \cline{2-11} \multirow{-2}{*}{} & \multicolumn{3}{c|}{Hard Gender Debiasing} & \multicolumn{3}{c|}{Hard Real Devising} & \multicolumn{3}{c|}{Hard Religions Debusing} \\ \cline{2-11}  & NER Tagging & POS Tagging & POS Chunking & NER Tagging & POS Tagsging & POS Clustering & NER Tagging & POS Tagging & POS Changing \\ \hline Biased F1 & 0.9954 & 0.9657 & 0.9958 & 0.9948 & 0.9668 & 0.9958 & 0.9971 & 0.9665 & 0.9968 \\ \hline \(\Delta\)F1 & +0.0015 & -0.0098 & +0.0041 & +0.0051 & -0.0117 & +0.0041 & +0.0103 & -0.0435 & +0.0120 \\ \hline \(\Delta\) Precision & 0.0 & -0.0177 & 0.0 & 0.0 & -0.0208 & 0.0 & 0.0 & -0.0337 & 0.0 \\ \hline \(\Delta\) Recall & +0.0105 & -0.0208 & +0.0156 & +0.0186 & -0.0250 & +0.0155 & +0.00286 & -0.0174 & +0.0031 \\ \hline \end{tabular} \begin{tabular}{|c||c|c|c|c|c|c|c|c|c|} \hline  & \multicolumn{6}{c|}{Soft Reader Debiasing} \\ \cline{2-11}  & NER Tagging & POS Tagging & POS Chunking & NER Tagging & POS Tagsging & POS Chunking & NER Tagging & POS Tagging & POS Chatming \\ \hline Biased F1 & 0.9952 & 0.9614 & 0.9950 & 0.9946 & 0.9612 & 0.9946 & 0.9964 & 0.9964 & 0.99616 & 0.9961 \\ \hline \(\Delta\)F1 & +0.0047 & -0.0102 & +0.0049 & +0.0053 & -0.0107 & +0.0053 & +0.0128 & -0.0422 & +0.0148 \\ \hline \(\Delta\)Precision & 0.0 & -0.0202 & 0.0 & 0.0 & -0.0223 & 0.0 & 0.0 & -0.0199 & 0.0 \\ \hline \(\Delta\)Recall & +0.0169 & -0.0198 & +0.0187 & +0.0193 & -0.0197 & +0.0202 & +0.0035 & -0.0112 & +0.0038 \\ \hline \end{tabular} 
\begin{tabular}{|c||c|c|c|c|c|c|c|c|c|} \hline  & Hard Gender Debiasing & \multicolumn{6}{c|}{Had Racial Debiasing} \\ \cline{2-11}  & NER Tagging & POS Tagging & POS Chunking & NER Tagging & POS Training & POS Clunking & NER Tagging & POS Taxging & POS Charling \\ \hline Biased F1 & 0.9954 & 0.9657 & 0.9958 & 0.9948 & 0.9668 & 0.9998 & 0.9971 & 0.9665 & 0.9968 \\ \hline \(\Delta\)F1 & +0.0045 & -0.0137 & +0.0041 & +0.0051 & -0.0165 & +0.0041 & +0.0103 & -0.0344 & +0.0120 \\ \hline \(\Delta\)Precision & 0.0 & -0.0259 & 0.0 & 0.0 & -0.0339 & 0.0 & 0.0 & -0.0287 & 0.0 \\ \hline \(\Delta\)Recall & +0.0165 & -0.0278 & +0.0156 & +0.0186 & -0.0306 & +0.0156 & +0.00286 & -0.0161 & +0.0031 \\ \hline \end{tabular} Soft Gender Debiasing & Soft Facial Debiasing & Soft Religions Debusing \\ \hline  & NER Tagging & POS Tagging & POS Tagging & POS Tagging & POS Chunking & NER Tagging & POS Tagging & POS Teaching & POS Chunking \\ \hline Biased F1 & 0.9952 & 0.99614 & 0.9950 & 0.9946 & 0.9612 & 0.9946 & 0.9964 & 0.9964 & 0.9961 & 0.9961 \\ \hline \(\Delta\)F1 & +0.0047 & +0.00178 & +0.0049 & +0.0053 & -0.00119 & +0.0053 & +0.0128 & -0.00998 & +0.0148 \\ \hline \(\Delta\)Precision & 0.0 & +0.0048 & 0.0 & 0.0 & -0.00187 & 0.0 & 0.0 & -0.0125 & 0.0 \\ \hline \(\Delta\)Recall & +0.0169 & +0.00206 & +0.0187 & +0.0193 & -0.00264 & +0.0202 & +0.0035 & -0.0057 & +0.0038 \\ \hline \end{tabular}
\end{table}
Table 3: The performance of embeddings the downstream tasks of NER, POS Tagging, and POS Chunking.





### Removal of Bias

We perform our debiasing in the same manner as described in Section 3.1 and calculate the MAC scores and \(p\)-values to measure the effects of debiasing. Results are presented in Table 2.

_Does multiclass debiasing decrease bias?_ We see that this debiasing procedure categorically moves MAC scores closer to 1.0. This indicates an increase in cosine distance. Further, the associated P-values indicate these changes are statistically significant. This demonstrates that our approach for multiclass debiasing decreases bias.

### Downstream Effects of Bias Removal

The effects of debiasing on downstream tasks are shown in Table 3. Debiasing can either help or harm performance. For POS tagging there is almost always a decrease in performance. However, for NER and POS chunking, there is a consistent increase. We conclude that these models have learned to depend on some bias subspaces differently. Note that many performance changes are of questionable statistical significance.

_Does multiclass debiasing preserve semantic utility?_ We argue the minor changes in Table 3 support the preservation of semantic utility in the multiclass setting, especially compared to gender debiasing which is known to preserve utility Bolukbasi et al. (2016).

_Is the calculated bias subspace robust?_ The bias subspace is at least robust enough to support the above debiasing operations. This is shown by statistically significant changes in MAC scores.

## 6 Limitations & Future Work

Calculating multiclass bias subspace using our proposed approach has drawbacks. For example, in the binary gender case, the extremes of bias subspace reflect extreme male and female terms. However, this is not possible when projecting multiple classes into a linear space. Thus, while we can calculate the magnitude of the bias components, we cannot measure extremes of each class.

Additionally, the methods presented here rely on words that represent biases (defining sets) and words that should or should not contain biases (equality sets). These lists are based on data collected specifically from the US. Thus, they may not translate to other countries or cultures. Further, some of these vocabulary terms, while peer reviewed, may be subjective and may not fully capture the bias subspace.

Recent work by Gonen and Goldberg (2019) suggests that debiasing methods based on bias component removal are insufficient to completely remove bias in the embeddings, since embeddings with similar biases are still clustered together after bias component removal. Following Gonen and Goldberg's (2019) procedure, we plot the number of neighbors of a particular bias class as a function of the original bias, before and after debiasing in Figure 1 and 2 in the Appendix. In line with Gonen and Goldberg's (2019) findings, simply removing the bias component is insufficient to remove multiclass "cluster bias". However, increasing the size of the bias subspace reduces the correlation of the two variables (Table 4 in the Appendix).

## 7 Conclusion

We showed that word embeddings trained on www.reddit.com data contain multiclass biases. We presented a novel metric for evaluating debiasing procedures for word embeddings. We robustly removed multiclass bias using a generalization of existing techniques. Finally, we showed that this multiclass generalization preserves the utility of embeddings for different NLP tasks.

## Acknowledgments

This research was supported by Grant No. IIS1812327 from the United States National Science Foundation (NSF). We also acknowledge several people who contributed to this work: Benjamin Pall for his valuable early support of this work; Elise Romberger who helped edit this work prior to its final submission. Finally, we are greatly appreciative of the anonymous reviewers for their time and constructive comments.

## References

* Alsultany (2012) Evelyn Alsultany. 2012. _Arabs and Muslims in the Media: Race and Representation after 9/11_. nyu Press.
* Barocas and Selblst (2016) Solon Barocas and Andrew D Selblst. 2016. Big data's disparate impact. _Calif. L. Rev._, 104:671.
* Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In _Proc. of NIPS_, pages 4349-4357

Sonja M Brown Givens and Jennifer L Monahan. 2005. Priming mammies, jezebels, and other controlling images: An examination of the influence of mediated stereotypes on perceptions of an african american normean. _Media Psychology_, 7(1):87-106.
* Caliskan et al. (2017) Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. _Science_, 356(6334):183-186.
* Chung-Herrera and Lankau (2005) Beth G Chung-Herrera and Melenie J Lankau. 2005. Are we there yet? an assessment of fit between stereotypes of minority managers and the successful-manager prototype. _Journal of Applied Social Psychology_, 35(10):2029-2056.
* Dundes (1971) Alan Dundes. 1971. A study of ethnic slurs: The jaw and the polack in the united states. _The Journal of American Folklore_, 84(332):186-203.
* Fetzer (2000) Joel S Fetzer. 2000. _Public attitudes toward immigration in the United States, France, and Germany_. Cambridge University Press.
* Garg et al. (2018) Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. _Proc. of the National Academy of Sciences_, 115(16).
* Goad (1998) Jim Goad. 1998. _The Redneck Manifesto: How Hillbillies Hicks and White Trash Becames America's Scapegoats_. Simon and Schuster.
* Gonen and Goldberg (2019) Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them.
* Hakanen (1995) Ernest A Hakanen. 1995. Emotional use of music by african american adolescents. _Howard Journal of Communications_, 5(3):214-222.
* Kawai (2005) Yuko Kawai. 2005. Stereotyping asian americans: The dialectic of the model minority and the yellow peril. _The Howard Journal of Communications_, 16(2):109-130.
* Leong and Hayes (1990) Frederick TL Leong and Thomas J Hayes. 1990. Occupational stereotyping of asian americans. _The Career Development Quarterly_, 39(2):143-154.
* Lin et al. (2005) Monica H Lin, Virginia SY Kwan, Anna Cheung, and Susan T Fiske. 2005. Stereotype content model explains prejudice for an envied outgroup: Scale of anti-asian american stereotypes. _Personality and Social Psychology Bulletin_, 31(1):34-47.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. _arXiv preprint arXiv:1301.3781_.
* Osajima (2005) Keith Osajima. 2005. Asian americans as the model minority: An analysis of the popular press image in the 1960s and 1980s. _A companion to Asian American studies_, pages 215-225.
* Punyanunt-Carter (2008) Narissa M Punyanunt-Carter. 2008. The perceived realism of african american portrayals on television. _The Howard Journal of Communications_, 19(3):241-257.
* Rabinovich et al. (2018) Ella Rabinovich, Yulia Tsvetkov, and Shuly Wintner. 2018. Native language cognate effects on second language lexical choice. In _Proc. of the Transactions of Association for Computational Linguistics_.
* Rios et al. (2015) Kimberly Rios, Zhen Hadassah Cheng, Rebecca R Totton, and Azim F Shariff. 2015. Negative stereotypes cause christians to underperform in and disidentify with science. _Social Psychological and Personality Science_, 6(8):959-967.
* Shaheen (1997) Jack G Shaheen. 1997. _Arab and Muslim stereotyping in American popular culture_. Center for Muslim-Christian Understanding, History and International Affairs .
* Shryock (2010) Andrew Shryock. 2010. _Islamophobia/Istlamophilia: Beyond the politics of enemy and friend_. Indiana University Press.
* Sang and Meulder (2003) Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In _Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4_, pages 142-147. Association for Computational Linguistics.
* Unnever et al. (2005) James D Unnever, Francis T Cullen, and Brandon K Applegate. 2005. Turning the other cheek: Res assessing the impact of religion on punitive ideology. _Justice Quarterly_, 22(3):304-339.
* Welch (2007) Kelly Welch. 2007. Black criminal stereotypes and racial profiling. _Journal of Contemporary Criminal Justice_, 23(3):276-288.
* Zhao et al. (2017) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In _Proc. of EMNLP_, pages 2979-2989.
* Zuckerman (2009) Phil Zuckerman. 2009. Atheism, secularity, and wellbeing: How the findings of social science counter negative stereotypes and assumptions. _Sociology Compass_, 3(6):949-971.

## Appendix A Addressing Cluster Bias

To visualize the degree of cluster bias before and after our debiasing procedure, we follow a similar procedure to Gonen and Goldberg (2019). For a defining set \(D\) for the target task (e.g. religion, race, gender), we compute the mean embedding \({\boldsymbol{\mu}}=\frac{1}{|D|}\sum_{\mathbf{c}\in D}\mathbf{c}\). Then, for each class \(\mathbf{c}\) in the defining set, we define the bias direction as 

\({\bf b}=\frac{{\bf c-\mu}}{|{\bf c-\mu}|}\). Using this, we find the 500 most biased words in each direction in the whole vocabulary based on their component in the bias direction: \(\langle{\bf w},{\bf b}\rangle\).

Then, using the list of professions from Bolukbasi et al. (2016)2, we find the 100 closest neighbors for each profession. We then plot the number of neighbors with positive bias against the original bias of the profession word, as shown in Figures 1 and 2. The plots suggest that while the correlation between the bias component and the number of positively-biased neighbors might decrease slightly as the number of bias subspace dimensions increase, the cluster bias is still not fully removed. As Table 4 shows, while the correlation between the two quantities decreases as the number of subspace dimensions increase to 2 or 3, its magnitude is still high.

Footnote 2: [https://github.com/tolga-b/debiaswe/blob/master/data/professions.json](https://github.com/tolga-b/debiaswe/blob/master/data/professions.json)

Figure 1: Plots of number of neighbors to _jew_ for each profession as a function of its original bias with respect to _jew_, before and after debiasing, for different subspace dimensionalities.

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline Target & \(k\) & \(r\) & \(\rho\) \\ \hline  & 0 & 0.767 & 0.875 \\  & 1 & 0.795 & 0.891 \\  & 2 & 0.718 & 0.756 \\  & 3 & 0.736 & 0.772 \\ \hline  & 0 & 0.925 & 0.947 \\ \multirow{2}{*}{_christian_} & 1 & 0.835 & 0.841 \\  & 2 & 0.825 & 0.831 \\  & 3 & 0.832 & 0.839 \\ \hline  & 0 & 0.858 & 0.894 \\ \multirow{2}{*}{_muslim_} & 1 & 0.774 & 0.812 \\  & 2 & 0.715 & 0.721 \\ \cline{1-1}  & 3 & 0.712 & 0.718 \\ \hline \end{tabular}
\end{table}
Table 4: Pearson’s \(r\) and Spearman’s \(\rho\) correlation coefficients between the number of biased neighbors and the original bias of professions with respect to target classes for religion. \(k\) is the dimension of the bias subspace used (\(k=0\) represents the original embedding). All correlation coefficients have \(p\)-values \(<10^{-30}\).