SlowGuess/ACL_2020_OCR huggingface dataset:
2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2672-2680.
Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. 2012. Large-scale learning of word relatedness with constraints. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1406-1414.
Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 3315-3323.
Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. 1998. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18-28.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4129-4138. Association for Computational Linguistics.
Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665-695.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 2: Short Papers, pages 427-431. Association for Computational Linguistics.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. TACL, 4:521-535.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579-2605.
David Madras, Elliot Creager, Toniann Pitassi, and Richard S. Zemel. 2019. Fairness through causal awareness: Learning causal latent-variable models for biased data. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* 2019, Atlanta, GA, USA, January 29-31, 2019, pages 349-358. ACM.
Michaël Mathieu, Junbo Jake Zhao, Pablo Sprechmann, Aditya Ramesh, and Yann LeCun. 2016. Disentangling factors of variation in deep representation using adversarial training. In Advances in
Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 5041-5049.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 2463-2473. Association for Computational Linguistics.
Yehezkel Resheff, Yanai Elazar, Moni Shahar, and Oren Shalom. 2019. Privacy and fairness in recommender systems via adversarial training of user representations. In Proceedings of the 8th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM., pages 476-482. INSTICC, SciTePress.
Alexey Romanov, Maria De-Arteaga, Hanna M. Wallach, Jennifer T. Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Cem Geyik, Krishnamaram Kenthapadi, Anna Rumshisky, and Adam Kalai. 2019. What's in a name? reducing bias in bios without access to protected attributes. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4187-4195. Association for Computational Linguistics.
Andrew Rosenberg and Julia Hirschberg. 2007. V-measure: A conditional entropy-based external cluster evaluation measure. In EMNLP-CoNLL 2007, Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, June 28-30, 2007, Prague, Czech Republic, pages 410-420. ACL.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4593-4601. Association for Computational Linguistics.
Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. 2017. Controllable invariance through adversarial feature learning. In Advances in
7247
Neural Information Processing Systems, pages 585-596.
Ke Xu, Tongyi Cao, Swair Shah, Crystal Maung, and Haim Schweitzer. 2017. Cleaning the null space: A privacy mechanism for predictors. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 2789-2795. AAAI Press.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 335-340. ACM.
Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. 2018. Learning gender-neutral word embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 4847-4853. Association for Computational Linguistics.
7248
A.1 INLP Guarantees
In this section, we prove, for the binary case, an orthogonality property for INLP classifiers: each two classifiers  $w_{i}$  and  $w_{j}$  from two iterations steps  $i$  and  $j$  are orthogonal (Lemma A.1). Several useful properties of the matrix  $P$  that is returned from INLP emerge as a direct result of orthogonality: the product of the projection matrices calculated in the different INLP steps is commutative (Corollary A.1.1);  $\mathrm{P}$  is a valid projection (Corollary A.1.2); and  $\mathrm{P}$  projects to a subspace which is the intersection of the nullspaces of all INLP classifiers  $N(w_{1}) \cap N(w_{2}) \cap \dots \cap N(w_{n})$  (Corollary A.1.3). Furthermore, we bound the influence of  $P$  on the structure of the representation space, demonstrating that its impact is limited only to those parts of the vectors that encode the protected attribute (Lemma A.2).
We prove those properties for two consecutive projection matrices  $P_{1}$  and  $P_{2}$  from two consecutive iterations of Algorithm 1, presented below in 5. The general property follows by induction.
1.  $w_{1} = \operatorname{argmin}_{w}\mathcal{L}(w;X;Z)$
2.  $P_{1} \coloneqq P_{N(w_{1})} = \text{GetProjectionMatrix}(N(w_{1}))$
3.  $X^{\prime} = P_{1}x$
4.  $w_{2} = \operatorname{argmin}_{w}\mathcal{L}(w;X^{\prime} = P_{1}X;Z)$
5.  $P_{2}:= P_{N(w_{2})}=$  GetProjectionMatrix  $(N(w_{2}))$
INLP Projects to the Intersection of Nullspaces.
Lemma A.1. if  $w_{2}$  is initialized as the zero vector and trained with SGD, and the loss  $\mathcal{L}$  is convex, then  $w_{2}$  is orthogonal to  $w_{1}$ , that is,  $w_{1} \cdot w_{2} = 0$ .
Proof. In line 4 of the algorithm, we calculate  $w_{2} = \operatorname{argmin}_{w}\mathcal{L}(w;X^{\prime} = P_{1}X;Z)$ . For a convex  $\mathcal{L}$  and a linear model  $w$ , it holds that the gradient with respect to  $w$  is a linear function of  $x$ :  $\nabla_w\mathcal{L}(x_i) = \alpha_ix_i$  for some scalar  $\alpha_{i}$ . It follows that after  $t$  stochastic SGD steps,  $w_{2}^{t}$  is a linear combination of input vectors  $x_{1},\ldots ,x_{i},\ldots ,x_{t}$ . Since we constrain the optimization to  $x_{i}\in N(w_{1})$ , and considering that fact the nullspace is closed under addition, at each step  $t$  in the optimization it holds
that  $w_2^t \in N(w_1)$ . In particular, this also holds for the optimal  $w_2^{*7}$ .
We proceed to prove commutativity based on this property.
Corollary A.1.1.  $P_{1}P_{2} = P_{2}P_{1}$
Proof. By Lemma A.1,  $w_{1} \cdot w_{2} = 0$ , so  $P_{R(w_{1})}P_{R(w_{2})} = P_{R(w_{2})}P_{R(w_{1})} = 0$ , where  $P_{R(w_{i})}$  is the projection matrix on the row-space of  $w_{i}$ . We rely on the relation  $P_{N(w_{i})} = I - P_{R(w_{i})}$  and write:
$$
\begin{array}{c} P _ {1} P _ {2} = (I - P _ {R (w _ {1})}) (I - P _ {R (w _ {2})}) = I - \\ P _ {R (w _ {1})} - P _ {R (w _ {2})} - P _ {R (w _ {1})} P _ {R (w _ {2})} = I - P _ {R (w _ {1})} - \\ P _ {R (w _ {2})}. \end{array}
$$
$$
\begin{array}{c} P _ {2} P _ {1} = (I - P _ {R (w _ {2})}) (I - P _ {R (w _ {1})}) = I - \\ P _ {R (w _ {2})} - P _ {R (w _ {1})} - P _ {R (w _ {2})} P _ {R (w _ {1})} = I - P _ {R (w _ {1})} - \\ P _ {R (w _ {2})}, \text {w h i c h c o m p l e t e s t h e p r o o f}. \end{array}
$$
Corollary A.1.2.  $P = P_{2}P_{1}$  is a projection, that is,  $P^2 = P$ .
Proof.  $P^2 = (P_2P_1)^2 = P_2P_1P_2P_1 \stackrel{*}{=} P_2P_2P_1P_1 = P_2^2P_1^2 \stackrel{**}{=} P_2P_1 = P$ , where * follows from Corollary A.1.1 and ** follows from  $P_1$  and  $P_2$  being projections.
Corollary A.1.3.  $P_{2}P_{1}$  is a projection onto  $N(w_{1})\cap N(w_{2})$
Proof. Let  $x \in \mathbb{R}^n$ .  $P_2(P_1x) \in N(w_2)$ , as  $P_2$  is the projection matrix to  $N(w_2)$ . Similarly,  $P_2(P_1x) = P_1(P_2x) \in N(w_1)$ , so  $Px \in N(w_1) \cap N(w_2)$ . Conversely, let  $x \in N(w_1) \cap N(w_2)$ . Then  $P_1x = x = P_2x$ , so  $Px = P_2P_1x = P_2x = x$ , so  $x$  is mapped by  $P$  to  $N(w_1) \cap N(w_2)$ .
Note that in practice, we enforce Corollary A.1.3 by using the projection Equation 1 (section 4). As such, the matrix  $P$  that is returned from Algorithm 1 is a valid projection matrix to the intersection of the nullspaces even if the conditions in Lemma A.1 do not hold, e.g. when  $\mathcal{L}$  is nonconvex or  $w_{2}$  is not initialized as the zero vector.
<sup>7</sup>If we performed proper dimensionality reduction at stage 3 - i.e., not only zeroing some directions, but completely removing them - the optimization in 4 would have a unique solution, as the input would not be rank-deficient. Then, we could use an alternative construction that relies on the Referender theorem, which allows expressing  $w_{2}$  as a weighted sum of the inputs:  $w_{2} = \sum_{x_{i}\in X^{\prime} = P_{1}X}\alpha_{i}x_{i}$ , for some scalars  $\alpha_{i}$ . As each  $x_{i}$  is inside the nullspace, so is any linear combinations of them, and in particular  $w_{2}$ .
7249
INLP Approximately Preserves Distances.
While the projection operations remove the protected information from the representations, ostensibly it could have had a detrimental impact on the structure of the representations space: as a trivial example, the zero matrix  $\pmb{O}$  is another operator that removes the protected information, but at a price of collapsing the entire space into the zero vector. The following lemma demonstrate this is not the case. The projection minimally damages the structure of the representation space, as measured by distances between arbitrary vectors: the change in (squared) distance between  $x, x' \in \mathbb{R}^n$  is bounded by the difference between the "gender components" of  $x$  and  $x'$ .
Lemma A.2. Let  $\vec{w} \in \mathbb{R}^n$  be a unit gender direction found in one INLP iteration, and let  $x, x' \in \mathbb{R}^n$  be arbitrary input vectors. Let  $P \coloneqq P_{N(\vec{w})}$  be the nullspace projection matrix corresponding to  $\vec{w}$ . Let  $d(x, x') \coloneqq ||x - x'||$  and  $d^P(x, x') \coloneqq ||Px - Px'||$  be the distances between  $x, x'$  before and after the projection, respectively. Then the following holds:
$$
\left(d \left(x, x ^ {\prime}\right) - d ^ {P} \left(x, x ^ {\prime}\right)\right) ^ {2} \leq \left(x \vec {w} - x ^ {\prime} \vec {w}\right) ^ {2}
$$
Proof. notation: we denote the  $i$ th entry of a vector  $x$  by  $x_{i}$ .
Since  $\vec{w}$  is the parameter vector of a gender classifier, a point  $x\in \mathbb{R}^n$  can be classified to a gender  $g\in \{0,1\}$  according to the sign of the dot product  $x\vec{w}$ . Note that in the binary case, the nullspace projection matrix  $P$  is given by
$$
P = I - \vec {w} \vec {w} ^ {T} \tag {5}
$$
Where  $\vec{w}\vec{w}^T$  is the outer product. By definition, if  $\vec{w}$  is in the direction of one of the axes, say without loss of generality the first axis, such that  $\vec{w} = [1,0,\dots,0]$ , then the following holds:
$$
\vec {w} \vec {w} ^ {T} = \left[ \begin{array}{c c c c} 1 & 0 & \dots & 0 \\ \vdots & 0 & & \\ & & \ddots & \\ 0 & \dots & & 0 \end{array} \right] \tag {6}
$$
Such that  $\vec{w}\vec{w}^T$  is the zero matrix except its (1,1) entry, and then  $P$  is simplified to
$$
P = \left[ \begin{array}{c c c c} 0 & 0 & \dots & 0 \\ \vdots & 1 & 0 \dots & 0 \\ 0 & & \ddots & \\ 0 & \dots & & 1 \end{array} \right] \tag {7}
$$
I.e, the unit matrix, except of a zero in the  $(1,1)$  position. Hence, the projection operator  $Px$  keeps  $x$  intact, apart from zeroing the first coordinate  $x_{1}$ . We will take advantage of this property, and rotate the axes such that  $\vec{w}$  is the direction of the first axis. We will show that the results we derive this way still apply to the original axes system.
Let  $R$  be a rotation matrix, such that after the rotation, the first coordinate of  $Rx$  is aligned with  $\vec{w}$ :
$$
(R x) _ {1} = x \vec {w} \tag {8}
$$
One can always find such rotation of the axes. Let  $x' \in \mathbb{R}^n$  be another point in the same space. Given the original squared distance between  $x$  and  $x'$ :
$$
d \left(x, x ^ {\prime}\right) = \left\| x - x ^ {\prime} \right\| ^ {2} \tag {9}
$$
Our goal is to bound the squared distance between the projected points in the new coordinate system:
$$
d ^ {P, R} (x, x ^ {\prime}) := \left\| [ P ] _ {R} R x - [ P ] _ {R} R x ^ {\prime} \right\| ^ {2} \tag {10}
$$
Where  $[P]_R$  denotes the projection matrix  $P$  in the rotated coordinate system, which takes the form 7.
Note that  $R$ , being a rotation matrix, is orthogonal. By a known result in linear algebra, multiplication by orthogonal matrices preserves dot product and distances. That means that the distance is the same before and after the rotation:  $d^{P,R}(x,x') = d^{P}(x,x')$ , so we can safely bound  $d^{P,R}(x,x')$  and the same bound would hold in the original coordinate system.
7250
By 7,
$$
\begin{array}{l} d ^ {P, R} (x, x ^ {\prime}) \\ = \sqrt {(0 - 0) ^ {2} + \sum_ {i = 2} ^ {n} ([ R x ] _ {i} - [ R x ^ {\prime} ] _ {i}) ^ {2}} \tag {11} \\ = \sqrt {d (x , x ^ {\prime}) ^ {2} - ([ R x ] _ {1} - [ R x ^ {\prime} ] _ {1}) ^ {2}} \\ \end{array}
$$
Note that in general it holds that for any  $a \geq b \geq 0$
$$
\begin{array}{l} \sqrt {a - b} = \sqrt {a + b - 2 b} = \sqrt {a + b - 2 \sqrt {b} \sqrt {b}} \tag {12} \\ \geq \sqrt {a + b - 2 \sqrt {a} \sqrt {b}} = \sqrt {(\sqrt {a} - \sqrt {b}) ^ {2}} \\ = \sqrt {a} - \sqrt {b} \\ \end{array}
$$
Combining 12 with 11 when taking  $a = d(x, x')^2$ ,  $b = ([Rx]_1 - [Rx']_1)^2$  we get:
$$
d ^ {P, R} \left(x, x ^ {\prime}\right) \geq d \left(x, x ^ {\prime}\right) - \left| \left(\left[ R x \right] _ {1} - \left[ R x ^ {\prime} \right] _ {1}\right) \right| \tag {13}
$$
From 11 one can also trivially get
$$
\begin{array}{l} d ^ {P, R} (x, x ^ {\prime}) = \sqrt {d (x , x ^ {\prime}) ^ {2} - ([ R x ] _ {1} - [ R x ^ {\prime} ] _ {1}) ^ {2})} \tag {14} \\ \leq \sqrt {d (x , x ^ {\prime}) ^ {2}} = d (x, x ^ {\prime}) \\ \end{array}
$$
Combining 14 and 13 we finally get:
$$
\begin{array}{l} d \left(x, x ^ {\prime}\right) - \left| \left(\left[ R x \right] _ {1} - \left[ R x ^ {\prime} \right] _ {1}\right) \right| \leq d ^ {P, R} \left(x, x ^ {\prime}\right) \tag {15} \\ <   d (x, x ^ {\prime}) \\ \end{array}
$$
Or, equivalently, after subtracting  $d(x,x^{\prime})$  from all elements and multiplying by -1:
$$
\begin{array}{l} \left| \left(\left[ R x \right] _ {1} - \left[ R x ^ {\prime} \right] _ {1}\right) \right| = \left| x \vec {w} - x ^ {\prime} \vec {w} \right| \\ \geq d (x, x ^ {\prime}) - d ^ {P, R} (x, x ^ {\prime}) \geq 0 \\ \end{array}
$$
$$
\begin{array}{l} (d (x, x ^ {\prime}) - d ^ {P, R} (x, x ^ {\prime})) ^ {2} \\ \leq \left(\left[ R x \right] _ {1} - \left[ R x ^ {\prime} \right] _ {1}\right) ^ {2} = \left(x \vec {w} - x ^ {\prime} \vec {w}\right) ^ {2} \\ \end{array}
$$
Note that this result has a clear interpretation: the difference between the distance of the projected  $x$ ,  $x'$  and the distance of the original  $x$ ,  $x'$  is bounded by the difference of  $x$  and  $x'$  in the gender direction  $\vec{w}$ . In particular, if  $x$  and  $x'$  are equally male-biased, their distance would not change at all; if  $x$  is very male-biased and  $x'$  is very female-biased, the projection would significantly alter the distance between them.
A.2 Influence on Local Neighbors in Glove Space
Word	Neighbors before	Neighbors after
order	orders, ordering, purchase	orders, ordering, ordered
crack	keygen, cracks, torrent	keygen, cracks, warez
Craiglist	ebay, craigslist, ads	ebay, craigslist, freecycle
populations	population, species, communities	population, species, habitats
epub	ebook, mobi, pdf	mobi, ebook, kindle
finals	semifinals, playoffs, championship	semifinals, semifinal, quarterfinals
installed	install, installing, installation	install, installing, installs
identifiable	disclose, identify, identifying	disclose, pii, distinguishable
photographs	photograph, photos, images	photograph, images, photos
ta	si, tu, ti	que, bien, ele
couch	sofa, sitting, bed	sofa, couches, loveseat
cooler	coolers, cooling, warmer	coolers, cooling, warmer
becky	debbie, kathy, julie	debbie, steph, Jen
appreciated	appreciate, greatly, thanks	appreciate, muchly, thanks
negotiation	negotiating, negotiations, mediation	negotiating, negotiations, mediation
initial	subsequent, prior, following	initial, inital, subsequent
chloe	chanel, emma, lauren	chloe, chanel, handbags
filipino	pinoy, filipinos, philippine	filipinos, pinoy, tagalog
relying	rely, relied, relies	rely, relied, relies
perpetual	eternal, continual, irrevocable	irrevocable, datejust, perpetuity
himself	him, herself, his	herself, oneself, he
seaside	beach, beachside, picturesque	beachside, idyllic, seafront
measure	measures, measuring, measured	measures, measuring, measured
yorkshire	staffordshire, leeds, lancashire	staffordshire, dales, lancashire
merchandise	goods, items, apparel	goods, items, merchandize
sub	subs, k, def	subs, subbed, svS
tones	tone, hues, muted	tone, polyphonic, muted
therapist	therapists, psychologist, therapy	therapists, physiotherapist, psychologist
leaned	sighed, smiled, glanced	leant, leaning, sighed
tho	nnd, cuz, tlie	nnd, tlio, tlie
lawyers	attorneys, lawyer, attorney	attorneys, lawyer, attorney
compile	compiling, compiler, compiles	compiling, compiler, compiles
chord	chords, progressions, guitar	chords, progressions, voicings
aims	aim, aimed, aiming	aim, aimed, aiming
ensure	ensuring, assure, ensures	ensuring, ensures, assure
aerospace	aviation, engineering, automotive	aeronautics, aviation, aeronautical
clubhouse	pool, playground, amenities	clubhouses, pool, playground
locking	lock, locks, latch	lock, locks, latch
reign	reigns, emperor, throne	reigns, reigned, emperor
vulnerable	susceptible, fragile, affected	susceptible, vunerable, fragile
Table 1: 3-nearest words before and after the INLP projection
7252
Table 1 above presents the results of word-embeddings similarity test mentioned in 6.1. This table lists the top 3-nearest neighbors of sampled words from GloVe, before and after the INLP process. It is evident that INLP does not alter the neighbors of the random sample in a detrimental way.
A.3 Quantitative Influence of Gender Debiasing on Glove Embeddings
In Appendix A.2 we provide a sample of words to qualitatively evaluate the influence of INLP on semantic similarity in Glove word embeddings (Section 6.1). We observe minimal change to the nearest neighbors. To complement this measure, we use a quantitative measure: measuring performance on established word-similarity tests, for the original Glove embeddings, and for the debiased ones. Those tests measure correlation between cosine similarity in embedding space and human judgements of similarity. Concretely, we test the embeddings similarities using three dataset, which contain four similarity tests that measure similarity or relatedness between words. We use the following datasets: SimLex999 (Hill et al., 2015), Word-Sim353 (Agirre et al., 2009) which contain two evaluations, on words similarity and relatedness and finally on Mturk-771 (Halawi et al., 2012).
The test sets are composed of word pairs, where each pair was annotated by humans to give a similarity or relatedness score. To evaluate a model against such data, each pair is given a score (in the case of word embedding, cosine similarity) and then we calculate Spearman correlation between all the score pairs. The results on the regular Glove embeddings before and after the gender debiasing are presented in Table 3. We observe a major improvements across all evaluation sets after the projection: between 0.044 to 0.116 points.
This major difference in performance is rather surprising. It is not clear how to interpret the positive influence on correlation with human judgements. This puzzle is further compounded by the fact the projection reduces the rank of the embedding spaces, and by definition induces loss of information. We hypothesize that many of the words in the embedding space contain a significant gender component, which is not correlated with humans judgements of similarity. While intriguing, testing this hypothesis is beyond the scope of this work, and we leave the more rigorous answer to a future
A.4 Influence on Local Neighbors of Surnames Representations in Glove Space
Word	Neighbors before	Neighbors after
ruth	helen, esther, margaret	etting, esther, gehrig
charlotte	raleigh, nc, atlanta	raleigh, greensboro, nc
abigail	hannah, lydia, eliza	hannah, phebe, josiah
sophie	julia, marie, lucy	moone, baxter, marceau
nicole	nicole, kimberly, kayla	nicole, mya, heiress
emma	emily, lucy, sarah	grint, frain, watson
olivia	emma, rachel, kate	munn, thirlby, wilde
ava	devine, zoe, isabella	viticultural, devine, appellation
isabella	sophia, josephine, isabel	rossellini, beeton, ferdinand
sophia	anna, lydia, julia	hagia, antipolis, topkapi
mia	bella, mamma, mama	bangg, mamma, culpa
amelia	earhart, louisa, caroline	earhart, fernandina, bedelia
james	john, william, thomas	jassie, nightfire, perse
john	james, william, paul	deere, scatman, betjeman
robert	richard, william, james	pattinson, mccammon, blacksportonline
michael	david, mike, brian	micheal, franti, moorcock
william	henry, edward, james	edward, henry, sir
david	stephen, richard, michael	bisbal, magen, sylvian
richard	robert, william, david	clayderman, brautigan, roty
joseph	francis, charles, thomas	joesph, dreamcoat, abboud
thomas	james, william, john	szasz, deshaun, tomy
ariel	sharon, alexis, hanna	peterpan, mermaid, cinderella
mike	brian, chris, dave	mignola, birbiglia, dave
Table 2: 3-nearest words before and after the INLP projection, for surenames