Fair is Better than Sensational:Man is to Doctor as Woman is to Doctor

Abstract: Analogies such as man is to king as woman is to X are often used to illustrate the amazing power of word embeddings. Concurrently, they have also exposed how strongly human biases are encoded in vector spaces built on natural language. While finding that queen is the answer to man is to king as woman is to X leaves us in awe, papers have also reported finding analogies deeply infused with human biases, like man is to computer programmer as woman is to homemaker, which instead leave us with worry and rage. In this work we show that,often unknowingly, embedding spaces have not been treated fairly. Through a series of simple experiments, we highlight practical and theoretical problems in previous works, and demonstrate that some of the most widely used biased analogies are in fact not supported by the data. We claim that rather than striving to find sensational biases, we should aim at observing the data"as is", which is biased enough. This should serve as a fair starting point to properly address the evident, serious, and compelling problem of human bias in word embeddings.

Word embeddings are distributed representations of texts which capture similarities between words. Beside improving a wide variety of NLP tasks, the power of word embeddings is often also tested intrinsically. Together with the idea of training word embeddings, BIBREF0 introduced the idea of testing the soundness of embedding spaces via the analogy task. Proportional analogies are equations of the form INLINEFORM0 , or simply A is to B as C is to D. Given the terms INLINEFORM1 , the model must return the word that correctly stands for INLINEFORM2 in the given analogy. A most classic example is man is to king as woman is to X, where the model is expected to return queen, by subtracting “manness" from the concept of king to obtain some general royalty, and then re-adding some “womanness" to obtain the concept of queen ( INLINEFORM3 ).
Beside this kind of magical power, however, embeddings have been shown to carry worrying biases present in our society and thus encoded in language. Recent studies BIBREF1 , BIBREF2 found that embeddings yield biased analogies such as the classic man is to doctor as woman is to nurse, or man is to computer programmer as woman is to homemaker.
Attempts at reducing bias, either via postprocessing BIBREF1 or directly in training BIBREF3 have nevertheless left two outstanding issues: bias is still encoded implicitly BIBREF4 , and it is debatable whether we should aim at removal or rather at transparency and awareness BIBREF5 , BIBREF4 .
With an eye to transparency, we took a closer look at the analogy structure. In the original proportional analogy implementation, all terms of the equation INLINEFORM0 are distinct BIBREF0 , BIBREF6 . In other words, the model is forced to return a different concept than the original ones. Given an analogy of the form INLINEFORM1 , the model is not allowed to yield any term INLINEFORM2 such that INLINEFORM3 , or INLINEFORM4 , or INLINEFORM5 , since the code explicitly prevents this. While this constraint is helpful when all terms of the analogy are expected to be different, it becomes a problem, and even a dangerous artifact, when the terms could or even should be the same.
We investigate this issue using the original analogy test set BIBREF0 , and examples from the literature. We test all examples on different embedding spaces built for English, using two settings for the analogy code: when all terms must be different (as in the original, widely used, implementation), and without this constraint, meaning that any word, including the input terms, can be returned. As far as we know, this is the first work that evaluates and reports analogies in an unrestricted fashion, since the analogy code is always used as is. Our experiments and results suggest that the mainstream examples as well as the use of the analogy task itself as a tool to detect bias should be revised and reconsidered.
Warning This work does not mean at all to downplay the presence and danger of human biases in word embeddings. On the contrary: embeddings do encode human biases, and we believe that this issue deserves the full attention of the field. However, we also believe that overemphasising and specifically seeking biases to achieve sensational results is not beneficial. It is also not necessary: what we observe naturally is worrying and sensational enough. Rather, we should aim at transparency and experimental clarity so as to ensure the fairest and most efficient treatment of the problem.
For both word2vec BIBREF0 and gensim BIBREF7 we adapted the code so that the input terms of the analogy query are allowed to be returned. Throughout this article, we use two different embedding spaces. The first is the widely used representation built on GoogleNews BIBREF8 . The second is taken from BIBREF2 , and was trained on a Reddit dataset BIBREF9 .
We test analogies using the code with and without modification, with the aim of showing the drawbacks and dangers of constraining (and selecting) the output of analogy queries to word embeddings. The analogies we use in this article come from three sources: the original analogy dataset proposed by BIBREF0 (Section SECREF3 ), a small selection of additional analogies to highlight the need to be able to return input vectors (Section SECREF3 ), and a collection of examples found in papers that address the problem of (human) biases in word embeddings (Section SECREF4 ). We follow BIBREF0 , BIBREF1 and BIBREF2 by using 3cosadd to calculate the analogies, as shown in Equation EQREF2 :
 DISPLAYFORM0 
All the examples used in this article, plus any new query, can be tested on any of the embeddings in the original and modified analogy code, and through our online demo.
The original, widely used, analogy test set introduced by BIBREF0 consists of two main categories: semantic analogies (Paris is to France as Tokyo is to Japan) and morpho-syntactic analogies (car is to cars as table is to tables). Within these, examples are classified in more specific sub-categories, as shown in the left column of Table TABREF5 . In the same table we report two scores based on the Google News embeddings as well as for the reddit embeddings from BIBREF2 . Under “orig" we report the score obtained using the original analogy code, and under “fair" we report the score yielded by our altered version, where the query terms ( INLINEFORM0 ) can be returned.
The results show a drastic drop in performance for the fair setting. In most cases, this is because the second term is returned as answer (man is to king as woman is to king, thus INLINEFORM0 ), but in some cases it is the third term that gets returned (big is to bigger as cold is to cold, thus INLINEFORM1 ). Results are over 50% lower in the semantic set, and the drop is even more serious in the syntactic examples, with the exception of “nationality-adj".
In the default evaluation set, and in the extended set proposed by BIBREF10 to cover additional linguistic relations, there are no word-pairs for which the gold target word is one of the three query words, in other words: A, B or C is the correct answer. Thus one might deem it a reasonable decision that the original analogy code does not let any of the original vectors to be returned. However, these conditions do exist, and this choice has consequences. The major consequence we observe is discussed in Section 4, and has to do with analogies affected by human bias. But even for the analogy types of Table 1, there are cases where this constraint is undesirable, due to homography. Additionally, there are other analogy types for which such constraint is utterly counterproductive, such as is-a or part-of relations.
One of the most well known analogies brought as example of human bias in word embeddings is man is to doctor as woman is to nurse BIBREF1 , BIBREF2 . This heavily biased analogy reflecting gendered stereotypes in our society, is however truly meaningful only if the system were allowed to yield “doctor” (arguably the expected answer in absence of bias) instead of “nurse”, and it doesn't. But we know that the system isn't allowed to return this candidate, since the original analogy code rules out the possibility of returning as D any of the query terms INLINEFORM0 , making it impossible to obtain man is to doctor as woman is to doctor (where INLINEFORM1 ).
This means that the bias isn't necessarily (or at least not only) in the representations themselves, rather in the way we query them. So, what do the embedding spaces actually tell if you let them return any word in the vocabulary?
We took a selection of mainstream, striking examples from the literature on embedding bias, and tested them fairly, without posing any constraint on the returned term, exactly as we did for all analogies in Section SECREF3 . In Table TABREF9 we report these examples, organised by the papers which discussed them, together with the returned term as reported in the paper itself, and the top two terms returned when using our modified code (1st and 2nd, respectively). Each example is tested over the same embedding space used in the corresponding paper.
What immediately stands out is that, bar a few exceptions, we do not obtain the term reported in the respective paper. One reason for this is that the model is now allowed to return the input vectors, and in most cases it does just that (especially INLINEFORM0 ).
In Section SECREF3 , we saw how this heavily affects the results on the original analogy test, and we also discussed why it would nevertheless be beneficial to impose no restrictions on the returned answer. When analogies are used to study human bias, though, the problem is more serious: How can we claim the model is biased because it does not return doctor if the model is simply not allowed to return doctor?
As a further constraint to the allowed output, BIBREF1 add an empirical threshold to Equation EQREF2 to ensure that terms that are too similar to INLINEFORM0 are excluded. Consequences are non-trivial. By not allowing the returned vector to be too close to the input vectors, this method basically skips potentially valid, unbiased answers until a potentially more biased answer is found. It isn't necessarily the case that more distance corresponds to more bias, but it is usually the case that less distance is akin to less bias (for example, gynecologist is a less biased answer than nurse to the query man is to doctor as woman is to X).
A closer look at the results makes things even more worrying. If the top answer yielded by our unrestricted code is one of the input vectors (e.g. doctor), the original code would not have shown it. It would have instead yielded what we obtain as our second answer. This is what we should see in the reported analogies. However, Table TABREF9 (column Index) shows that this is not always the case.
The threshold method of BIBREF1 described in Section SECREF10 is the cause for this outcome in their examples, as vectors higher in the rank have been excluded as too close to the input vector. Unfortunately, while surely successful over standard, factual analogy cases, this strategy turns out to be essentially a way of selecting the output.
For example, their strategy not only excludes lovely (input term), but also magnificent as a possible answer for she is to lovely as he is to X, since the vector for magnificent is not distant enough from the vector of input term lovely. As can be seen in Table TABREF9 , lovely and magnificent would be the first and second words returned otherwise. The term brilliant is only returned in 10th position by an unrestricted search. While aiming at returning a vector distant enough from the input term might be desirable for some of the analogies, this threshold-based strategy is not fair when researching bias, as it potentially forces the exclusion of unbiased terms (in this case, after magnificent, one would find the following terms before encountering brilliant: marvelous, splendid, nice, fantastic, delightful, terrific, wonderful). In the example she is to sewing as he is to X., the threshold was strong enough to even exclude a potentially biased answer (woodworking).
 BIBREF2 also use the analogy test to demonstrate bias, starting from a pre-selection of terms to construct their queries from a variety of sources. In addition to using the original analogy code, thus missing out on what the actual returned term would be, they state that rather than reporting the top term, they hand-pick an example from the returned top-N words. While qualitatively observing and weighing the bias of a large set of returned answers makes sense, it can be misleading to cherry-pick and report very biased terms in sensitive analogies. At the very least, when reporting term-N, one should report the top-N terms to provide a more accurate picture. In Table TABREF12 , we report the top-10 candidates for asian is to engineer as black is to X in both the Reddit embeddings of BIBREF2 as well as GoogleNews, for completeness. Similarly, we now know that an unrestricted analogy search for man is to doctor as woman is to X returns doctor, but this does not provide a complete picture. Reporting the top-10 for this query as well as the top-10 for the inverted query (Table TABREF12 ) surely allows for a much better informed analysis rather than simply reporting doctor, or picking nurse.
If the analogy really is a symptom of a biased vector space, we should find similar biases for synonyms or closely related words to the input word INLINEFORM0 . However, with computer_programmer for example, this does not seem to be the case. If we use the term programmer instead of computer_programmer, homemaker is not very close (165), while for coder (13,374), developer (26,117) and hacker (56,646) it does not even appear in the top 10,000. Also, when using white instead of the less frequent and more specialised (and in a way less parallel to black) caucasian in the analogy of black is to criminal as caucasian is to X, lawful is found at position 40 instead of 13.
In a way, examples are always cherry-picked, but when making substantial claims on observed biases, the fact that the results we obtain are due to a carefully chosen word (rather than a similar one, possibly even more frequent), should not be overlooked.
If we do not operate any manipulations on the returned vectors, neither by setting constraints nor by cherry-picking the output, we observe that in many cases, independently of the analogy type and the query terms, the model simply returns one of the input terms, and in particular INLINEFORM0 . Perhaps, this is a weakness of embeddings in modelling certain relations, or the analogy task as such is not apt at capturing them.
Such observations relate to two points raised in previous work. First, the suggestive power of analogies should not be overestimated. It has been argued that what is observed through the analogy task might be mainly due to irrelevant neighborhood structure rather than to the vector offset that supposedly captures the analogy itself BIBREF11 , BIBREF12 . Indeed, BIBREF13 have also shown that the 3cosadd method is not able to capture all linguistic regularities present in the embeddings. Interestingly, the analogy task has not been recently used anymore to evaluate the soundness of contextualised embeddings BIBREF14 , BIBREF15 . Second, bias isn't fully captured anyway via the analogy task. In fact, BIBREF4 suggest that analogies are not quite reliable diagnostics for uncovering bias in word embeddings, since bias is anyway often encoded implicitly. As a side note, we would like to mention that in an earlier version of their paper, BIBREF18 accidentally searched for the inverse of the intended query, and still managed to find biased examples. This seems to be a further, strong, indication that strategies like this are not fully suitable to demonstrate the presence of bias in embeddings.
If analogies might not be the most appropriate tool to capture certain relations, surely matters have been made worse by selecting results in order to prove (and emphasise) the presence of human bias. Using such sensational “party tricks" BIBREF4 is harmful, as they get easily propagated both in science itself BIBREF19 , BIBREF20 , BIBREF21 , even outside NLP and AI BIBREF22 , as well as in popularised articles of the calibre of Nature BIBREF23 . This is even more dangerous, because of the widened pool of readers, and because such readers are usually in no position to verify the reliability of such examples.
In any case, anyone who constructs and uses analogies to uncover human biases must do this fairly and transparently, and be aware of their limitations. In this sense, it is admirable that BIBREF5 try to better understand their results by checking them against actual job distributions between the two genders. Aiming primarily at scientific discovery rather than sensational findings is a strict pre-requisite to truly understand how and to what extent embeddings encode and reflect the biases of our society, and how to cope with this, both socially and computationally.

Figure and table captions:
Table 1: Performance on the standard analogy test set (Mikolov et al. 2013a) using the original and the fair versions of the analogy code. The fair version allows for any term in the vocabulary to be returned, including the input terms, while the original one does not allow any of the input terms to be returned.
Table 2: Results of analogy task on examples where one of the original words is the correct answer. Bold indicates the correct answer.
Table 3: Fair results of embedding analogies. We use the same embedding set for our analysis as is used in the respective papers. “Index” denotes the position where the reported biased answer was actually found in our experiments. 1st answer and 2nd answer report what we actually find in first and second position. In brackets: the index of the reported answer as obtained using the swapped query, C is to B as A is to X.
Table 4

Questions about this paper:
Which embeddings do they detect biases in?

Answers:
Word embeddings trained on GoogleNews and Word embeddings trained on Reddit dataset