SlowGuess/ACL_2020_OCR huggingface dataset:
Human-selected vs. Learned Directions. Our method differs from the common projection-based approach by two main factors: the numbers of directions we remove, and the fact that those directions are learned iteratively from data. Perhaps the benefit is purely due to removing more directions? We compare the ability to linearly classify words by gender bias after removing 10 directions by our method (running Algorithm 1 for 10 iterations) with the ability to do so after removing 10 manually-chosen directions defined by the difference vectors between gendered pairs  ${}^{4}$  . INLP-based debiasing results in a very substantial drop in classification accuracy (54.4%), while the removal of the predefined directions only moderately decreases accuracy (80.7%). This shows that data-driven identification of gender-directions outperforms manually selected directions: there are many subtle ways in which gender is encoded, which are hard for people to imagine.
Discussion. Both the previous method and our method start with the main gender-direction of  $\overrightarrow{he} -\overrightarrow{she}$ . However, while previous attempts take this direction as the information that needs to be neutralized, our method instead considers the labeling induced by this gender direction, and then iteratively finds and neutralizes directions that correlate with this labeling. It is likely that the  $\overrightarrow{he} -\overrightarrow{she}$  direction is one of the first to be removed, but we then go on and learn a set of other directions that correlate with the same labeling and which are predictive of it to some degree, neutralizing each of
them in turn. Compared to the 10 manually identified gender-directions from Bolukbasi et al. (2016), it is likely that our learned directions capture a much more diverse and subtle set of gender clues in the embedding space.
Effect of debiasing on the embedding space. In appendix §A.2 we provide a list of 40 random words and their closest neighbors, before and after INLP, showing that INLP doesn't significantly damage the representation space that encodes lexical semantics. We also include a short analysis of the influence on a specific subset of inherently gendered words: gendered surnames (Appendix §A.4).
Additionally, we perform a semantic evaluation of the debiased embeddings using multiple word similarity datasets (e.g. SimLex-999 (Hill et al., 2015)). We find large improvements in the quality of the embeddings after the projection (e.g. on SimLex-999 the correlation with human judgements improves from 0.373 to 0.489) and we elaborate more on these findings in Appendix §A.3.
Clustering. Figure 1 shows t-SNE (Maaten and Hinton, 2008) projections of the 2,000 most female-biased and 2,000 most male-biased words, originally and after  $t = 3$ ,  $t = 18$  and  $t = 35$  projection steps. The results clearly demonstrate that the classes are no longer linearly separable: this behavior is qualitatively different from previous word vector debiasing methods, which were shown to maintain much of the proximity between female and male-biased vectors (Gonen and Goldberg, 2019). To quantify the difference, we perform K-means clustering to  $K = 2$  clusters on the vectors, and calculate the V-measure (Rosenberg and Hirschberg, 2007) which assesses the degree of overlap between the two clusters and the gender groups. For the t-SNE projected vectors, the measure drops from  $83.88\%$  overlap originally, to  $0.44\%$  following the projection; and for the original space, the measure drops from  $100\%$  to  $0.31\%$ .
WEAT. While our method does not guarantee attenuating the bias-by-neighbors phenomena that is discussed in Gonen and Goldberg (2019), it is still valuable to quantify to what extent it does mitigate this phenomenon. We repeat the Word Embedding Association Test (WEAT) from Caliskan et al. (2017) which aims to measure the association in vector space between male and female concepts and stereotypically male
3Interestingly, RBF-kernel SVM (used by Gonen and Goldberg (2019)) achieves random accuracy. 4We use the following pairs, taken from Bolukbasi et al. (2016): ("woman", "man"), ("girl", "boy"), ("she", "he"), ("mother", "father"), ("daughter", "son"), ("gal", "guy"), ("female", "male"), ("her", "his"), ("herself", "himself"), ("mary", "john").
7242
or female professions. Following Gonen and Goldberg (2019), we represent the male and female groups with common names of males and females, rather than with explicitly gendered words (e.g. pronouns). Three tests evaluate the association between a group of male names and a groups of female names to (1) career and family-related words; (2) art and mathematics words; and (3) artistic and scientific fields. In all three tests, we find that the strong association between the groups no longer exists after the projection (non-significant p-values of 0.855, 0.302 and 0.761, respectively).
Bias-by-Neighbors. To measure bias-by-neighbors as discussed in (Gonen and Goldberg, 2019), we consider the list of professions provided in (Bolukbasi et al., 2016) and measure the correlation between bias-by projection and bias by neighbors, quantified as the percentage of the top 100 neighbors of each profession which were originally biased-by-projection towards either of the genders. We find strong correlation of 0.734 (compared with 0.852 before), indicating that much of the bias-by-neighbors remains.[5]
6.2 Fair Classification: Controlled Setup
We now evaluate using INLP with a deeper classifier, with the goal of achieving fair classification.
Classifier bias measure: TPR-GAP. To measure the bias in a classifier, we follow De-Arteaga et al. (2019) and use the TPR-GAP measure. This measure quantifies the bias in a classifier by considering the difference (GAP) in the True Positive Rate (TPR) between individuals with different protected attributes (e.g. gender/race). The TPR-GAP is tightly related to the notion of fairness by equal opportunity (Hardt et al., 2016): a fair classifier is expected to show similar success in predicting the task label  $Y$  for the two populations, when conditioned on the true class. Formally, for a binary protected attribute  $z$  and a true class  $y$ , define:
$$
T P R _ {z, y} = P [ \hat {Y} = y | Z = z, Y = y ] \tag {2}
$$
$$
G A P _ {z, y} ^ {T P R} = T P R _ {z, y} - T P R _ {z ^ {\prime}, y} \tag {3}
$$
where  $Z$  is a random variable denoting binary protected attribute,  $z$  and  $z'$  denote its two values, and
Ratio	Sentiment	TPR-Gap
Original	INLP	Original	INLP
0.5	0.76	0.75	0.19	0.16
0.6	0.78	0.74	0.29	0.22
0.7	0.81	0.66	0.38	0.24
0.8	0.84	0.67	0.45	0.15
Table 1: The Sentiment scores (in accuracy, higher is better) and TPR differences (lower is better) as a function of the ratio of tweets written by black individuals in the positive-sentiment class.
$Y, \hat{Y}$  are random variables denoting the correct class and the predicted class, respectively.
Experiment setup. We begin by experimenting with a controlled setup, where we control for the proportion of the protected attributes within each main-task class. We follow the setup of Elazar and Goldberg (2018) which used a twitter dataset, collected by Blodgett et al. (2016), where each tweet is associated with "race" information and a sentiment which was determined by their belonging to some emoji group.
Naturally, the correlation between the protected class labels and the main-class labels may influence the fairness of the model, as high correlation can encourage the model to condition on the protected attributes. We measure the TPR-GAP on predicting sentiment for the different race groups (African American English (AAE) speakers and Standard American English (SAE) speakers), with different imbalanced conditions, with and without application of our "classifier debiasing" procedure.
In all experiments, the dataset is overly balanced with respect to both sentiment and race (50k instances for each). We change only the proportion of each race within each sentiment class (e.g., in the 0.7 condition, the "happy" sentiment class is composed of  $70\%$  AAE /  $30\%$  SAE, while the "sad" class is composed of  $30\%$  AAE /  $70\%$  SAE).
Our classifier is based on the DeepMoji encoder (Felbo et al., 2017), followed by a 1-hideen-layer MLP. The DeepMoji model was trained on millions of tweets in order to predict their emojis; a model which was proven to perform well on different classification tasks (Felbo et al., 2017), but also encodes demographic information (Elazar and Goldberg, 2018). We train this classifier to predict sentiment. We then follow the procedure in §5: training a guarding function on the hidden layer of the MLP, and re-training the final linear layer on the guarded vectors. Table 1 presents the results.
<sup>5</sup>Note that if, for example, STEM-related words are originally biased towards men, the word "chemist" after the projection may still be regarded as male-biased by neighbors, not because an inherent bias but due to its proximity to other originally biased words (e.g. other STEM professions).
7243
BoW	FastText	BERT
Accuracy (profession)	Original	78.2	78.1	80.9
+INLP	80.1	73.0	75.2
\( GAP_{male}^{TPR,RMS} \)	Original	0.203	0.184	0.184
+INLP	0.124	0.089	0.095
Table 2: Fair classification on the Biographies corpus.
Figure 3: t-SNE projection of BERT representations for the profession "professor" (left) and for a random sample of all professions (right), before and after the projection.
Figure 4: The correlation between  $GAP_{\text{female},y}^{TPR}$  and the relative proportion of women in profession  $y$ , for BERT representation, before (left; R=0.883) and after (right; R=0.470) the projection.
As expected the TPR-GAP grows as we increase the correlation between class labels and protected attributes. The accuracy grows as well. Applying our debiasing technique significantly reduced the TPR gap in all settings, although hurting more the main task accuracy in the highly-imbalanced setting. In Appendix A.5, we give some more analysis on the balance between performance and TPR-Gap and show that one can control for this ratio, by using more iterations of INLP.
6.3 Fair Classification: In the Wild
We now evaluate the fair classification approach in a less artificial setting, measuring gender bias in biography classification, following the setup of De-Arteaga et al. (2019).
They scraped the web and collected a dataset of short biographies, annotated by gender and profession. They trained logistic regression classifiers
to predict the profession of the biography's subject based on three different input representation: bag-of-words (BOW), bag of word-vectors (BWV), and RNN based representation. We repeat their experiments, using INLP for rendering the classifier oblivious of gender.
Setup. Our data contains 393,423 biographies. We follow the train:dev:test split of De-Arteaga et al. (2019), resulting in 255,710 training examples  $(65\%)$ , 39,369 development examples  $(10\%)$  and 98,344  $(25\%)$  test examples. The dataset has 28 classes (professionals), which we predict using a multiclass logistic classifier (in a one-vs-all setting). We consider three input representations: BOW, BWV and BERT (Devlin et al., 2019) based classification. In BOW, we represent each biography as the sum of one-hot vectors, each representing one word in the vocabulary. In the BWV representation, we sum the FastText token representations (Joulin et al., 2017) of the words in the biography. In BERT representation, we represent each biography as the last hidden state of BERT over the  $CLS$  token. Each of these representations is then fed into the logistic classifier to get final prediction. We do not finetune FastText or BERT.
We run INLP with scikit-learn Pedregosa et al. (2011) linear classifiers. We use 100 logistic classifiers for BOW, 150 linear SVM classifiers for BWV, and 300 linear SVM classifiers for BERT.
Bias measure. We use the TPR-GAP measure for each profession. Following Romanov et al. (2019), we also calculate the root-mean square of  $GAP_{g,y}^{TPR}$  over all professions  $y$ , to get a single per-gender bias score:
$$
G A P _ {g} ^ {T P R, R M S} = \sqrt {\frac {1}{| C |} \sum_ {y \in C} \left(G A P _ {g , y} ^ {T P R}\right) ^ {2}} \tag {4}
$$
where  $C$  is the set of all labels (professions).
De-Arteaga et al. (2019) have shown that  $GAP_{g,y}^{TPR}$  strongly correlates with the percentage of women in profession  $y$ , indicating that the true positive rate of the model is influenced by gender.
6.3.1 Results
Main results The results are summarized in Table 2. INLP moderately changes main-task accuracy, with a  $1.9\%$  increase in BOW, a  $5.1\%$  decrease in performance in BWV and a  $5.51\%$  decrease in BERT.  $GAP_{g}^{TPR,RMS}$  is significantly
<sup>6</sup>The original dataset had 399,000 examples, but 5,557 biographies were no longer available on the web.
7244
decreased, indicating that on average, the true positive rate of the classifiers for male and female become closer: in BOW representation, from 0.203 to 0.124 (a  $38.91\%$  decrease); in BWV, from 0.184 to 0.089 (a  $51.6\%$  decrease); and in BERT, from 0.184 to 0.095 (a  $48.36\%$  decrease). We measure the correlation between  $GAP_{y,female}^{TPR}$  for each profession  $y$ , and the percentage of biographies of women in that profession. In BOW representation, the correlation decreases from 0.894 prior to INLP to 0.670 after it (a  $33.4\%$  decrease). In BWV representation, the correlation decreases from 0.896 prior to INLP to 0.425 after it (a  $52.5\%$  decrease). In BERT representation, the correlation decreases from 0.883 prior to INLP to 0.470 following it (a  $46.7\%$  decreases; Figure 4b). De-Arteaga et al. (2019) report a correlation of 0.71 for BWV representations when using a "scrubbed" version of the biographies, with all pronouns and names removed. INLP significantly outperforms this baseline, while maintaining all explicit gender markers in the input.
Analysis. How does imposing fairness influence the importance the logistic classifier attribute to different words in the biography? We take advantage of the BOW representation and visualize which features (words) influence each prediction (profession), before and after the projection. According to Algorithm 1, to debias an input  $x$ , we multiply  $W(Px)$ . Equivalently, we can first multiply  $W$  by  $P$  to get a "debiased" weight matrix  $W'$ . We begin by testing how much the debiased weights of words that are considered to be biased were changed during the debiasing, compared to random vocabulary words. We compare the relative change before and after the projection of these words, for every occupation. Biased words undergo an average relative change of x1.23 compared to the average change of the entire vocabulary, demonstrating that biased words indeed change more. The per-profession breakout is available in Figure 2 in Appendix §A.6.1.
Next, we test the words that were changed the most during the INLP process. We compare the weight difference before and after the projection. We sort each profession words by weight, and average their location index for each professions. Many words indeed seem gender specific (e.g. ms., mr., his, her, which appears in locations 1, 2, 3 and 4 respectively), but some seem unrelated, perhaps due to spurious correlations in the data. The complete
list is available in Table 4 in the Appendix §A.6.1; an analogous analysis for the FastText representation is available at Appendix §A.6.2.
A limitation of our method when used in the context of fairness is that, like other learning approaches, it depends on the data  $X, Z$  that is fed to it, and works under the assumption that the training data is sufficiently large and is sampled i.i.d from the same distribution as the test data. This condition is hard to achieve in practice, and failure to provide sufficiently representative training data may lead to biased classifications even after its application. Like other methods, there are no magic guarantees, and the burden of verification remains on the user. It is also important to remember that the method is designed to achieve a very specific sense of protection: removal of linear information regarding a protected attribute. While it may correlate with fairness measures such as demographic parity, it is not designed to ensure them. Finally, it is designed to be fed to a linear decoder, and the attributes are not protected under non-linear classifiers.
We present a novel method for removing linearly-represented information from neural representations. We focus on bias and fairness as case studies, and demonstrate that across increasingly complex settings, our method is capable of attenuating societal biases that are expressed in representations learned from data.
While we focused on bias, Iterative Nullspace Projection has broader possible use-cases, and can be utilized to remove specific components from a representation, in a controlled manner. This method can be applicable for other end goals, such as style-transfer, disentanglement of neural representations and increasing their interpretability. We aim to explore those directions in a future work.
We thank Jacob Goldberger and Jonathan Berant for fruitful discussions. This project received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme, grant agreement No. 802774 (iEXTRACT).
7245
Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and wordnet-based approaches. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 19-27.
Matthew Barker and William Rayens. 2003. Partial least squares for discrimination. Journal of Chemometrics: A Journal of the Chemometrics Society, 17(3):166-173.
Maria Barrett, Yova Kementchedjhieva, Yanai Elazar, Desmond Elliott, and Anders Søgaard. 2019. Adversarial removal of demographic attributes revisited. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6331-6336.
Adi Ben-Israel. 2015. Projectors on intersections of subspaces. Contemporary Mathematics, pages 41-50.
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1798-1828.
Su Lin Blodgett, Lisa Green, and Brendan O'Connor. 2016. Demographic dialectal variation in social media: A case study of african-american english. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119-1130.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in neural information processing systems, pages 4349-4357.
Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186.
Maximin Coavoux, Shashi Narayan, and Shay B Cohen. 2018. Privacy-preserving neural representations of text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1-10.
Maria De-Arteaga, Alexey Romanov, Hanna M. Wallach, Jennifer T. Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Cem Geyik, Krishnamaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency,
FAT* 2019, Atlanta, GA, USA, January 29-31, 2019, pages 120-128. ACM.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Yanai Elazar and Yoav Goldberg. 2018. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 11-21. Association for Computational Linguistics.
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. 2019. Understanding undesirable word embedding associations. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 1696-1705. Association for Computational Linguistics.
Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
Yaroslav Ganin and Victor S. Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pages 1180-1189.
Paul Geladi and Bruce R Kowalski. 1986. Partial least-squares regression: a tutorial. Analytica chimica acta, 185:1-17.
Yoav Goldberg. 2019. Assessing bert's syntactic abilities. arXiv preprint arXiv:1901.05287.
Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 609-614. Association for Computational Linguistics.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. *Generative adversarial nets.* In *Advances in Neural Information Processing Systems* 27: Annual Conference on Neural Information Processing Systems
7246