SlowGuess/ACL_2020_OCR huggingface dataset:
The results in Table 1 suggest that, as expected, the projection has little influence on the lexical semantics of unbiased words, as measured by their closest neighbors in embedding space. But how does the projection influence inherently gendered words? Table 2 contains the closest-neighbors to the Glove representations of gendered surnames, before and after the projection. We observe an interesting tendency to move from neighbors which are other gendered surnames, towards family names, which are by definition gender-neutral (for instance, the closest neighbor of "Robert" changes from "Richard" to "Pattinson"). Another interesting tendency is to move towards place names bearing a connection to that surnames (For instance, the closest neighbor of "Sophia" changes to "Hagia"). At the same time, some gendered surnames remain close neighbors even after the projection.
A.5 Performance and "Fair Classification" as a Function of INLP Iterations
In Section 6.2 where we compare the accuracy and TPR-Gap before and after using INLP for a certain amount of iterations. The number of iterations chosen is somehow arbitrary, but we emphasize that this can be controlled for as the number of iterations used with INLP. By sacrificing the main task performance, one can improve the TPR-Gap
7253
Eval	Before	After
SimLex999	0.373	0.489
WordSim353 - Sim	0.695	0.799
WordSim353 - Rel	0.599	0.698
Mturk-771	0.684	0.728
of their model. In Figure 1 we detail these trade-offs for the 0.8 ratio, where the original TPR-Gap originally is the highest.
We note that the performance is minimally damaged for the first 180 iterations, while the TPR-Gap improves greatly, after-which, both metric account for larger drops. Using this trade-off, one can decide how much performance they are willing to sacrifice in order to get a less biased model.
A.6 Biographies dataset: Words Most-Associated with Gender
A.6.1 Bag-of-Words Model
In this section, we present the raw results of the experiment aimed to assess the influence of INLP on specific words, under the bag-of-words model, for the biographies experiments (Section 6.3.1).
Table 4 lists the words most influenced by INLP projection (on average over all professions) after the debiasing procedure explained in Section 6.3.
Figure 2 presents the relative change of biased word for each profession, compared to a random sample.
Table 3: Word similarity scores on Glove embeddings, before and after INLP. The scores are the Spearman correlation coefficient between the similarity scores.
Most Changed Words
ms., mr., his, her, he, she, mrs., specializes, english, practices, ', ' , him, spanish, speaks, with, affiliated, and, medicine, ms, state, #, the, medical, michael, in, residency, at, of, psychology, dr., 's, law, research, practice, about, where, business, education, 5, -, is, first, women, america, insurance, more, john, university, location, ph.d., surgery, ( , mental, ), that, engineering, graduated, language, bs, litigation, collection, united, 1, graduate, humana, cpas, cancer, npi, completed, 10, book, hospital, c, out, family, or, when, oklahoma, certified, ohio, number, training, for, like, a, than, be, nursing, ], -, can, writing, patients, no, orthopaedic, attorney, over, ny, mr, „,
Table 4: Top 100 words influenced by INLP projection (BOW representation, biographies dataset).
Figure 2: The relative change of biased vs. random words, per profession.
A.6.2 Bag-of-Word-Vectors Model
In this section, we present an analysis for the influence of INLP projection on the FastText representation of individual words, under the bag-of-word-vectors model, for the biographies experiments (Section 6.3.1). We begin by ordering the vocabulary items by their cosine similarity to each of the top 15 gender directions found in INLP (i.e., their similarity to the weight vector of each classifier). For each gender direction  $w_{i}$ , we focus on the
7254
Gender direction	Male-biased	Female-biased
0	his, he, His, himself	herself, she, She, her
1	himself, him, Gavin, His	Hatha, midwifery, Midwifery, feminist
2	Mark, Jon, Darren, Luke	Actress, Zumba, Diana, woman
3	Gordon, he, wind, charge	hers, recipe, Challenge, cookbooks
4	1935, 1955, namely, 1958	Roots, Issue, FHM, yoga
5	M, Mickey, KS, Bethesda	Vietnam, Subject, Elle, Ecuador
6	Keys, correct, address, fuel	leap, Embedded, textile, femininity
7	Papers, Categories, wherein, Newark	Botox, LASIK, periodontal, UnityPoint
8	binding, closely, MT, command	Aventura, brunette, HTML, Disclosure
9	82, 92, 91, 86	ASP.NET, committer, Twilight, Seth
10	t, Cisco, Philips, Sharp	preschool, caregivers, homeowners, Preschool
11	Toulouse, Aviv, scored, commended	intersectional, Equality, equality, ASME
12	addressing, segment, inequalities, segments	Wire, loose, anything, Vincents
13	comparison, Hart, 480, refereed	Matthew, independence, couples, LGBTQ
14	manufacturer, organizers, scope, specifications	homeschooling, ligament, loyalty, graduating
Table 5: words closest to top 15 INLP gender directions (FastText representation, biographies dataset).
20,000 most common vocabulary items, and calculate the closest words to  $w_{i}$  (to get male-biased words) as well as the closest words to  $-w_{i}$  (to get female-biased words). The result are presented in Table 5.
The first gender direction seems to capture pronouns. Other gender directions capture socially biased terms, such as "preschool" (direction 10), "cookbooks" (direction 3) or other gender-related terms, such as "LGBTQ" (direction 15) or "femininity" (direction 6). Interestingly, those are mostly female-biased terms. As for the male-biased words, some directions capture surnames, such as "Gordon" and "Aviv". Other words which were found to be male-biased are less interpretable, such as words specifying years (direction 4), organizational terms such as "Organizers", "specifications" (direction 14), or the words "Papers", "Categories" (direction 7). It is not clear if those are the result of spurious correlations/noise, or whether they reflect actual subtle differences in the way the biographies of men and women are written.
Gender rowspace The above analysis focuses on what information do individual gender directions convey. Next, we aim to demonstrate the influence of the final INLP projection on the representation of words. To this end, we rely on the rowspace of the INLP matrix  $P$ . Recall that the rowspace is the orthogonal complement of the nullspace. As the INLP matrix  $P$  projects to the intersection of nullspaces of the gender directions, the complement  $P_R := I - P$  projects to the union of rowspaces of individual gender directions. This is a subspace which is spanned by all gender direc
tions, and thus can be thought of as an empirical gender subspace within the representation space.
For a given word vector  $w$ , the "gender norm" – the norm of its projection on the rowspace,  $||P_Rw||$  – is a scalar quantity which can serve as a measure for the gender-bias of the word. We sort the vocabulary by the ratio between the gender norm and the original norm,  $\frac{||P_Rw||}{||w||}$  and present the 200 most gendered words (Table 6).
As before, we see a combination of inherently-gendered words ("motherhood", "women", "gender", "masculinities"), socially-biased terms ("teacher", "raising", "semiconductors", "B.Tech", "IEEE", "STEM", "fashion") and other words whose connection to gender is less interpretable, and potentially represent spurious correlations ("trauma", "Vitae", "smile", "920", "forgiveness").
Top words by component on the gender subspace
motherhood, SSHRC, microfinance, preschool, genocide, IFP, CSE, intersectional, student, homeschooling, photoshoot, intersectionality, 920, breastfeeding, STEM, photojournalistic, haiku, kindergarten, FreeOnes, UNESCO, menstrual, turbulence, NTR, ASME, HFN, ECE, IEEE, feminism, noir, Jadavpur, Motherhood, reportage, Contra, TU, WebSphere, counsellor, photovoltaic, J2EE, contraception, university, PEN, masculinities, parenting, EAP, Politecnico, Feminism, trauma, Universiti, counselling, curriculum, Kanpur, women, edits, Pune, Nanjing, ethnographic, Pinterest, surrealist, taught, Hindustan, students, CNRS, Bangalore, Mumbai, consortium, tooth, Vitae, Kindergarten, nanoscale, school, ACL, scholarships, cloud, Goa, NIJC, Montessori, JSPS, scholarship, Neha, DAAD, endometriosis, carrier, UCI, activism, Ambedkar, EECS, semiconductor, scholar, microfluidic, bikini, Raising, teacher, Feminist, vinyasa, NBER, ethnography, Twilight, Sunil, Shankar, viral, earthquake, semiconductors, historiography, vampire, HMO, PSU, bioenergy, historian, Ravi, Breastfeeding, Raman, resettlement, Shweta, ICTs, UNDP, NVIDIA, HIV, Counselling, HEC, KDD, Hyderabad, contraceptive, macro, Ghaziabad, sexuality, CAS, documentary, mic, biography, postdoc, transnationalism, AMD, CFD, B.Tech, physicist, LGBT, parenthood, HKU, HIP, internationalization, M.Tech, BDS, acne, theorist, HPV, Meerut, ageing, smile, Rajesh, psychoeducational, PUNE, grief, AHA, Essays, discourses, secrets, Swati, EPFL, coaching, IIE, Manoj, BIDMC, infertility, fashion, Chicana, Vaishali, Graduation, sociologist, Gender, EA, MIT, teach, gift, IETF, NPPA, counselor, JPL, gender, menopause, LGBTQ, Waseda, perceptions, praxis, birthday, Jawaharlal, fertility, gendered, coverage, stills, PIH, Balaji, Tagged, baking, USM, postpartum, Goenka, Pooja, forgiveness
Table 6: Words by gender norm.
7256