Abstract: Analogies such as man is to king as woman is to X are often used to illustrate the amazing power of word embeddings. Concurrently, they have also exposed how strongly human biases are encoded in vector spaces built on natural language. While finding that queen is the answer to man is to king as woman is to X leaves us in awe, papers have also reported finding analogies deeply infused with human biases, like man is to computer programmer as woman is to homemaker, which instead leave us with worry and rage. In this work we show that,often unknowingly, embedding spaces have not been treated fairly. Through a series of simple experiments, we highlight practical and theoretical problems in previous works, and demonstrate that some of the most widely used biased analogies are in fact not supported by the data. We claim that rather than striving to find sensational biases, we should aim at observing the data"as is", which is biased enough. This should serve as a fair starting point to properly address the evident, serious, and compelling problem of human bias in word embeddings.

Introduction: Word embeddings are distributed representations of texts which capture similarities between words. Beside improving a wide variety of NLP tasks, the power of word embeddings is often also tested intrinsically. Together with the idea of training word embeddings, BIBREF0 introduced the idea of testing the soundness of embedding spaces via the analogy task. Proportional analogies are equations of the form INLINEFORM0 , or simply A is to B as C is to D. Given the terms INLINEFORM1 , the model must return the word that correctly stands for INLINEFORM2 in the given analogy. A most classic example is man is to king as woman is to X, where the model is expected to return queen, by subtracting “manness" from the concept of king to obtain some general royalty, and then re-adding some “womanness" to obtain the concept of queen ( INLINEFORM3 ). Beside this kind of magical power, however, embeddings have been shown to carry worrying biases present in our society and thus encoded in language. Recent studies BIBREF1 , BIBREF2 found that embeddings yield biased analogies such as the classic man is to doctor as woman is to nurse, or man is to computer programmer as woman is to homemaker. Attempts at reducing bias, either via postprocessing BIBREF1 or directly in training BIBREF3 have nevertheless left two outstanding issues: bias is still encoded implicitly BIBREF4 , and it is debatable whether we should aim at removal or rather at transparency and awareness BIBREF5 , BIBREF4 . With an eye to transparency, we took a closer look at the analogy structure. In the original proportional analogy implementation, all terms of the equation INLINEFORM0 are distinct BIBREF0 , BIBREF6 . In other words, the model is forced to return a different concept than the original ones. Given an analogy of the form INLINEFORM1 , the model is not allowed to yield any term INLINEFORM2 such that INLINEFORM3 , or INLINEFORM4 , or INLINEFORM5 , since the code explicitly prevents this. While this constraint is helpful when all terms of the analogy are expected to be different, it becomes a problem, and even a dangerous artifact, when the terms could or even should be the same. We investigate this issue using the original analogy test set BIBREF0 , and examples from the literature. We test all examples on different embedding spaces built for English, using two settings for the analogy code: when all terms must be different (as in the original, widely used, implementation), and without this constraint, meaning that any word, including the input terms, can be returned. As far as we know, this is the first work that evaluates and reports analogies in an unrestricted fashion, since the analogy code is always used as is. Our experiments and results suggest that the mainstream examples as well as the use of the analogy task itself as a tool to detect bias should be revised and reconsidered. Warning This work does not mean at all to downplay the presence and danger of human biases in word embeddings. On the contrary: embeddings do encode human biases, and we believe that this issue deserves the full attention of the field. However, we also believe that overemphasising and specifically seeking biases to achieve sensational results is not beneficial. It is also not necessary: what we observe naturally is worrying and sensational enough. Rather, we should aim at transparency and experimental clarity so as to ensure the fairest and most efficient treatment of the problem.

Question: What are the two PharmaCoNER subtasks?
Answer: Entity identification with offset mapping and concept indexing