Detecting and Mitigating Indirect Stereotypes in Word Embeddings
Authors: Erin George
Deanna Needell
Venue: Submitted to ICLR 2023 (2023)
Keywords: word embeddings
stereotypes
bias
bias mitigation
indirect bias

Abstract: Societal biases in the usage of words, including harmful stereotypes, are frequently learned by common word embedding methods.  These biases manifest not only between a word and an explicit marker of its stereotype, but also between words that share related stereotypes.  This latter phenomenon, sometimes called ``indirect bias,'' has resisted prior attempts at debiasing.  In this paper, we propose a novel method to mitigate indirect bias in distributional word embeddings by modifying biased relationships between words before embeddings are learned.  This is done by considering how the co-occurrence probability of a given pair of words changes in the presence of words marking an attribute of bias, and using this to average out the effect of a bias attribute.  To evaluate this method, we perform a series of common tests and demonstrate that the semantic quality of the word embeddings is retained while measures of bias in the embeddings are reduced.  In addition, we conduct novel tests for measuring indirect stereotypes by extending the Word Embedding Association Test (WEAT) with new test sets for indirect binary gender stereotypes.  With these tests, we demonstrate that this method can reduce the presence of more subtle stereotypes not properly addressed by previous work.

Under review as a conference paper at ICLR 2023
DETECTING AND MITIGATING INDIRECT STEREOTYPES IN WORD EMBEDDINGS
Anonymous authors Paper under double-bling review
ABSTRACT
Societal biases in the usage of words, including harmful stereotypes, are fre- quently learned by common word embedding methods. These biases manifest not only between word and an explicit marker of its stereotype, but also between words that share related stereotypes. This latter phenomenon, sometimes called "indirect bias," has resisted prior attempts at debiasing. In this paper, we propose a novel method mitigate indirect bias in distributional word embeddings by mod- ifying biased relationships between words before embeddings are learned. This is done by considering how the co-occurrence probability of given pair of words changes in the presence of words marking an attribute of bias, and using this to average out the effect of bias attribute. To evaluate this method, we perform a series of common tests and demonstrate that measures of bias in the word embed- dings are reduced in exchange for some reduction in the semantic quality of the embeddings. In addition, we conduct novel tests for measuring indirect stereo- types by extending the Word Embedding Association Test (WEAT) with new test sets for indirect binary gender stereotypes. With these tests, we demonstrate the presence of more subtle stereotypes not addressed by previous work. The pro- posed method is able to reduce the presence of some of these stereotypes, serving as a crucial next step towards non-stereotyped word embeddings.
FIX
FIX
INTRODUCTION
Distributional word embeddings, such Word2Vec (Mikolov et al. 2013a) and GloVe (Pennington al. 2014), are computer representations of words as vectors in semantic space. These embed- dings are popular because the geometry of the vectors corresponds to semantic and syntactic struc- ture (Mikolov et al. 2013b). Unfortunately, societal stereotypes, such as those pertaining to race, gender, national origin, sexuality, are typically reflected in word embeddings (Bolukbasi et al. 2016 Caliskan et al.) 2017] Garg et al.] 2018 PPapakyrikkopollos et al.) 2020). These stereotypes 50 pervasive that they have proved resistant to many existing debiasing techniques (Gonen & Goldberg 2019).
Techniques attempting remove mitigate bias in word vectors common in the literature. The typical case study for bias mitigation methods in the literature is binary ods, such as hard debiasing from Bolukbasi al. (2016) and GN-GloVe from Zhao et al. (2018b),
binary[ggender gender. Subspace meth-
attempt to identify or create a vector subspace of gender-related information (typically "gender direction") and drop this subspace. Counterfactual Data Substitution from Maudslay
(0019)
based on Counterfactual Data Augmentation from Lu al. (2020). swaps explicitly gendered words counter stereotyped associations. James & Alvarez- (2019) and Qian al. (2019) both pro- pose methods to reduce bias towards binary gender by encouraging learned conditional probabilities of words appearing with "he" and with "she" be equal.
Gonen & Goldberg (2019) showed that common "debiasing" methods failed to meaningfully reduce bias in word embeddings. They describe how bias can manifest not only as undesirable association
We use the phrase "binary gender" to refer to the common yet unrealistic simplification of gender as just "male" or "female", which we take be the main source of hias of study in this work. This is limitation of this work.

---

Under review as a conference paper at ICLR 2023
between stereotyped words and words marking a bias attribute but also between stereotyped words themselves. These manifestations are sometimes called direct bias and indirect bias using the terminology introduced by Boluubasi et al. (201)). An example of this second manifestation of bias is that the word "doctor" might be associated more strongly with stereotypically masculine words than with stereotypically feminine words. At the time, bias mitigation algorithms commonly attempted to address direct bias while leaving indirect bias mostly present.
A common trend in the study of the indirect bias is the departure from stereotypes as the object of study in favor of clustering. While the measures introduced by Bolukbasi et al. (2016): [Caliskan ct al. (2017) attempt to quantify the existence of commonly understood stereotypes, work on indirect bias typically uses the measures introduced by |Gonen & Goldberg (2019) which merely attempt to measure how well proposed bias mitigation methods disperse words with similar relationships to the bias attribute in the embedding space. These clustering measures, while useful at capturing some forms of indirect bias, are limited. In particular, it is unclear how dispersed stereotyped words should be in the embedding space, given that the stereotype of word is not entirely arbitrary and can potentially be estimated based on its semantic, non-stereotypical, meaning.
These new bias measures have inspired countless new bias mitigation methods. Nearest neighbor bias mitigation from Dames & Alvarez-Melis (2019) attempts to equalize each word's association with its masculine (defined by the original undebiased embeddings) neighbors and its feminine neighbors. Double hard debias from Wang et al.] (2020) projects off the direction defined by the "most gender-biased words" (again, based alignment in the original embedding's 's "gender direc- tion"") in addition to the standard gender-related subspace. [Bordia & Bowman ( (2019) modify the loss function when learning word embeddings to penalize neutral words having large components in the gender-related subspace which can then be dropped off. Kumar et al. (2020) propose RAN-Debias which attempts to disperse words in the embedding space that share similar binary gender biases (defined again by the original word embeddings) while preserving the original geometry as much as possible. [Lauscher et al. (2020) describe multiple bias mitigation methods: the standard projection method, averaging original word vectors with an orthogonal transformation that attempts to swap the bias attribute, and neural method that uses a loss function to group together words exhibiting a bias attribute away from neutral words. These methods, similarly to the bias measures of Gonen & Goldberg 2019), focus on the clustering and dispersion of words in relation to the bias attribute.
In current state-of of- the-art models, word embeddings have largely been replaced by contextualized embeddings from transformer models such as BERT (Devlin et al.) (0188) and GPT (Radford et al. 2018). However, word embeddings remain a popular object of study when quantifying bias in NIP, in part due to their simplicity and theoretical results that make them easier to reason about. As advances in the understanding of bias and stereotypes in word embeddings have been adapted for these newer models (Liang et al. [2020] Maa et al. 2019), novel techniques to measure and mitigate bias in word embeddings remain relevant.
2 BACKGROUND
2.1 WORD EMBEDDINGS
Many word embedding algorithms use the empirical probability that two given words appear near each other in the corpus (Levy & Goldberg, 2014; |Pennington et al., 2014). This empirical prob- ability is computed by counting how many times one word appears in the context of another word-context pair. A word-context pair is defined as a pair of words from the corpus that appear within certain fixed distance from each other, the window size, and within the same sentence. A word-conted pair designates one word as appearing in the context of another; in this paper, we will refer to
pair (a, b) where is the word appearing in the context of the
We use the phrase "bias attribute" to refer an attribute associated with stereotypes that are to be re- moved. The reader can assume "bias attribute" in this work will always refer to binary gender without loss of comprehension. We prefer "bias attribute" for generality.
*Or "explicit" and "implicit" hias. In this work we will refer to these "direct hias" and "indirect bias" for clarity, even though they are actually just two different manifestations of the same phenomenon. "Following the recommendations of Devinney et al. (022)) we prefer the terms "masculine" and "feminine" over "male" and "female" in this work.
2

---

Under review as a conference paper at ICLR 2023
word b. Contexts can be unidirectional or bidirectional. In the unidirectional case, in word-conter context pair (a, b), always occurs before in the corpus (or alternatively, always after). In the bidirectional case, for any pair of nearby words and b, there are two word-contens pairs: (a, b) and a).
From these corpus statistics, word embedding algorithms learn vectors for each word in a way that the word co-oceurrence statisties be derived from the geometry of the vectors. The exact details for how this is done is dependent on the exact word embedding algorithm used and is not important for this work.
2.2 WORD EMBEDDING ASSOCIATION TEST
The Word Embedding Association Test (WEAT), introduced by Caliskan et al. (2017). 7) is common test used to quantify the presence of specific stereotypes in word embeddings. Given two sets of target words X and Y of equal size and two sets of attribute words A and B, WEAT measures association between the targets and the attributes. The sets are chosen so that X and A are stereotypically linked with each other, and similarly for Y and B.
The association of a word w E X U Y with the attributes is
1
swe A B
cos( wi, a)
cos( (7, b) BEB
A
aE
where iž denotes the word vector corresponding to the word u and
0 0
cos(1,, 17, 0)
is the cosine similarity between ūł and ह.
The outputs of WEAT are the test statistic, effect size, and p-value (of permutation test), defined by the following equations respectively:
s (X B)
Σ
st A. B)
Σ (c,,A,B)
ZEX
DEY
Exex (z, A I EpEy AB
EffectSize (X, Y.A. B) A
stddov mEXUY 8 (w, A, B) p P(s A,) s YX A.A
where (X,, Y.) is a random partition of X Y into two sets of equal size.
Of these three measures, the effect size is most commonly used in the literature. In this work, port the effect size along with the p-value. The effect size retains its original meaning from from CClissan al. (2017) of measuring the strength and direction of the tested stercotype. We interpret the p- value not as measure of statistical significance, as originally conceived. Instead, we interpret it as another measure of the tested stereotype, indicating how easily words in X can separated from words in Y according to their association with A and B: p-values of O, 0.5, and correspond to perfect separation according to the stereotype, no separation, and perfect separation according to the opposite stereotype, respectively. We will interpret these as relative measures: WEAT effect size closer to zero and a WEAT p-value closer 0.5 both suggest reduced presence of the stereotype in the embeddings.
3 INDIRECT STEREOTYPES
Previous work measuring stereotypes has typically focused measuring associations between stereotyped words and explicit markers of the stereotype, such as name and semantically gendered words. In turn, previous work on mitigating bias have used these same markers input for bias mitigation methods. By looking at the same sets of words for quantifying and mitigating bias,
'hile names do not inherently mark any bias attribute, aggregate groups of names typically isolate bias attribute more directly than other groups of words can and, more importantly for this work, are commonly used reference points in bias mitigation algorithms.
3

---

Under review as a conference paper at ICLR 2023
easy to overestimate the effect of bias mitigation. This is what led Gonen & Goldberg (2019) to propose their new bias measures. These are based around measuring how clustered previously biased words are, but it unclear what result should be desired from these measures. To address this issue, we demonstrate that we can capture some forms of the remaining bias as stereotypes, which refer to indirect stereotypes.
man
intensified relationship
suppressed
handsome
engineer
relationship
sentimental
woman
Figure 1: An example showing indirect stereotypes.
An example of an indirect stereotype is shown in Figure [ There are three words here exhibiting a stereotype in regards to binary gender: "handsome" and "engineer," which are both masculine and not feminine, and "sentimental," which feminine and masculine. gender stereotypes, we would expect quantifiably stronger association between "handsome" and "engineer" than between "sensitive" and "engineer" to exist purely because they are both stereotyp- ically used to refer to men even if there is not stereotype that engineers are more handsome than they are sensitive.
corpus with binary
These stereotypes can come about as a result of sentences that exhibit multiple stereotypes at the same time, For example, the sentence "he an a handsome engineer" exhibits the masculine stereo- types for "handsome" and "engineer". In corpus that exhibits both of these stereotypes, this sen- tence would be more likely to occur than the following related sentences:
She is handsome engineer.
He is sensitive engineer.
She is sensitive engineer.
This will result in "handsome" occurring with "engineer" more frequently than "sensitive" does. However, when word embeddings are "debiased" with respect to binary gender, the goal in previous work is typically to obtain word vectors that would predict each pair of the previous examples that differ only in choice of pronoun equally likely to occur. This is not enough to correct the as- sociations between "handsome," "engineer,"" and "sensitive." Furthermore, the association between "engineer" and "handsome" should exist even in sentences without explicit reference binary gen- der, as those cases "engineer" is more likely to refer to a man than a woman. More details on this analysis can be found in Appendix A
To test indirect stereotypes, we use WEAT with the categories that are stereotyped according to binary gender (these are science/art, math/art, and career/family), well test set of professions and adjectives that are stereotypically masculine and feminine. These new test sets and the process used to generate them for stereotypes in word embeddings, subsequent work has called attention to shortcomings of the measure. We give a more thorough discussion with our analysis, the end of Section 5
detailed in Appendix B While we will use WEAT
primary test
NEW
4 BIAS MITIGATION METHOD
Instead of only looking at the co-occurrence probability of two words, we consider the probability word co-occurs with another and one of the two also occurs near word marking the bias attribute.
ssuming there other relationship hetween "engineer" and "handsome" "sensitive".
4

---

Under review as a conference paper at ICLR 2023
By using the word marking bias attribute a proxy for the bias attribute itself, determine how the relationship between the two words varies as the bias attribute varies. Then, we can approx- imate what the relationship between two words would be if one of them had no association with the bias attribute.
FIX
Example 1 Suppose we would like to mitigate one of the stereotypes in Figure ! perhaps that "engi- neer" and "handsome" are more closely associated than they should be based non-stereotypical semantics. In the corpus, it might be that an engineer is described as handsome three times as often when "engineer" refers to a man as opposed to when it refers that "engineer" refers to a man twice as often as it refers to woman. If "engineer" is adjusted so that it refers to men just as often as women, then the probability that an engineer is referved to as handsome should also be adjusted to be
woman. It might also be the case
P( (""handoome" mman) + 1/2 P("handsome' woman) 6
(1)
P("handsomee" " man) + 3 P( """handsomee woman) 7
of what it was before, at least when just considering occurrences where "engineer" refers to man or woman. Whether "engineer" refers to man or woman not known to us a priori, but can be estimated based on whether "engineer" is near word marking binary gender. Occurrences where "engineer" is not near such a word could be left as is or also be adjusted with this factor based the reasoning in Section 3 In this work, take the latter approach. We note that word embeddings do not attempt to quantify whether or not "handsome" " describes an engineer, but rather if "handsome" occurs near "engineer". However, the same principle still applies.
The general method is follows. Suppose word-conter pair in the corpus is selected uniformly at random. Consider two words a and where at least one of a or is a word to be neutralized with respect this bias. Additionally, let X and Y be two sets of words that are similar in usage but differ in terms of the bias attribute that is being mitigated. For example, in the case of binary gender, X could be the set ("he", "man", :...} and Y could be the set ("she", "woman", ..
Define the following events for a random word-context pair (c, d):
A c= a, B d=b.
X : the pair appears near word in X, Y: : the pair appears near a word in W.
A word-context pair is defined being near word in or in Y if this word is within a certain distance from e or from d. This distance can be taken to be the window size used for determining word-context pairs, for simplicity. In this work, we weight occurrences where a word in X or Y appears near just one of or d half as much as if the word appeared near both of e and d, but we could just as easily weight these occurrences the same.
The probability the word-context pair (c, d) is (a, b) can be decomposed as
P(AnB) - P(BOX))(XBBX X) + P(B
P(ANBOXXY)
P(XUY|AnB)
using the definition of conditional probability and inclusion-exclusien This representation isolates the contribution of the bias attribute on P(A n B). The terms P(B X) and P(B n Y) indicate the bias b has, if any, while the terms (AA n X) and P(A|B n Y) describe how the relationship between a and is influenced by the bias attribute.
Now if b is replaced with a hypothetical neutral word b' that has the same meaning except with no association to the bias attribute, the probability of // occurring with a can be approximated. Denote by B' the event that a randomly selected word-context pair (c, d) satisfies d = b.
The following exact equalities hold in this setting:
P((XUY)nB) = P((XUY)
P(XUY|A = P(XUY|AnB)
P(XNB') = P(YnB')
The first two identities hold because and b' have the same meaning except for the bias. The last identity holds because // has bias.
5

---

Under review as a conference paper at ICLR 2023
In addition, the following relations are approximately true:
P(A/BX) ~ P(A[B' nx) P(ABnY) R P(A|B' nY)
P(BOXNY) & P(B'DXXY) P(ANBOXOY) & PIANB'
In the first two lines, the explicit presence of the bias attribute of interest likely to overpower the bias b has. In the last two lines, the approximate relationship is likely to hold as all the probabilities are small: the events are the intersection of three or more events and are subsets of Y.
These relations imply
P(XNB'> P(YnB') v P(XПB) + P(YNB)
and
P(ANB) ~ P(XUYIANB B) -P(ANBOXOY))
(2)
Reconsider Example The factor in front is the original co-occurrence probability divided by the denominator of equation with semantics replaced by word co-occurrence. In the example, P(A n B n X n Y) 0 and (P(B X) P(B n Y)) With these substitutions, the remaining terms are the numerator of equation With the switch from semantics to co-occurrence, this is now an approximation.
NEW
A further approximation can be obtained by looking at a. Since b' is neutral, if a is replaced with neutral word a' and A replaced with a corresponding event A', holds that P(A' s P(AnB'). A similar argument shows equation 2 holds when the role of and 6 are switched. These two approximations can be averaged to yield a better approximation for P(A' nB'):
1
P(A'nB)) ~ P(XUY|ANB) [( + P(B P(BN + P(ABBYY)
(3)
+ + + (LIPANX-PPANY))PBAXXX-PPPPP + - -P(ANBUXNY))
This approximation is the basis of our proposed bias mitigation method. We propose the right hand side of equation 3 can replace the probability of a word-context pair being (a. b) whenever this quantity would be used, where easily with word embedding methods that directly use the probability or counts of a specific word- context pair occurring, such Glo Ve PPMI factoring, but in principle could be adapted even to methods that only indirectly use these probabilities.
is word that is wanted to be neutralized. This can be done
For all the terms in equation be defined, least one occurrence each of AnX, AnY, BnX, and
B n Y must be present in the corpus. Therefore, modified entries in the co-occurrence matrix must
NEW
be limited and in a pair satisfying Y. In practice this may not be a strong restriction. In test case with binary gender, although only 34% of the words can have co-occurrences modified in this way, this includes most frequently occurring words. The vast majority of words that cannot be modified with this method have less than 100 occurrences in the corpus and would therefore have noisier embeddings. This is shown in Figure / Thus, words that cannot be applied equation can be dropped, yielding more pronounced bias mitigation. This is possible even with only eight words each in and Y.
only those where both words appear the corpus in a word-context pair satisfying X
Of course, binary gender in English is particularly well-suited for this method, as words that mark binary gender are rather common (e.g.. "he", "she", "man", "woman", etc.). For other types of biases, would likely have to use less common words, such as names, resulting in fewer words can apply equation 3 to. The entries of the original co-occurrence matrix with these words can be kept, although there will likely be less bias mitigation result.

---

Under review as a conference paper at ICLR 2023
150000
Embedded words
Number of wwods
Bias mitigation possible
100000
50000
0
102
104
106
10a
Occurrences of word
Figure Number of embedded words versus number of occurrences of words in the corpus.
5 RESULTS
We conduct experiments by training GloVe embeddings on the UMBC webbase corpus (Han et al., 2013) and then attempting to mitigate the presence of bias from binary gender. We compare the pro- posed method from Section[[wit Glo with and without Counterfactual Data Subsitution (Maud- slay et al. 2019). These are labeled as "Proposed", "Original", and "CDS" respectively in the fol- lowing tables. Due to computational constraints, we only train CDS on 30 million sentences from the corpus. The results with training on the reduced corpus are shown on the left side cach table, while the results for the full corpus are shown the right side of each table. In all tables, the best results for each half row are shown bold.
NEW
FIX
For all methods, we train Glo Ve with embedding dimension of 300. In determining the cooc- currence matrix, we use a window size of 15 and skip over words with less than 5 occurrences in the corpus. We also preprocess than five words, and using Stanford CoreNLP (Manning et al. 2014) to separate out certain lexemes from words (e.g., n't and ss).
corpus by removing capitalization, skipping sentences with less
For the proposed bias mitigation method, we use the following sets of words:
X ("he", "him", "his", "himself", "mman", "men", "boy", "boys"] V ("she", "her", "hers", "herself", "woman", "women", "girl", "girls")..
All results are obtained by training word vectors multiple times and reporting the median of the respective measures. Five sets each of word vectors trained for the reduced corpus, while three sets are trained for the full corpus.
As first test, evaluate all three word embeddings trzebski et al.) 2017). The relative change in the median results these tests after the two tested bias mitigation methods tation of the tested semantics in the word embeddings. The tests are grouped into three categories; from top bottom, these are word similarity tasks, categorization tasks, and analogy tasks. Overall, our proposed method typically results in worse degradation of semantics than CDS does. This is most pronounced in the analogy tasks, where there is an approximately 15% decrease in accuracy in the smaller corpus. With the full corpus, there is less degradation in semantics. The analogy tasks still see an approximately 10% decrease in accuracy, but there surprisingly substantial increase in performance in the word similarity tasks and mixed results in classification.
series of common semantic tests (Jas-
NEW
shown in Table T For all tests, higher score indicates better represen-
FIX
NEW
NEW
The tasks that suffer with largest degradation in the full corpus (AP, BLESS, Battig, Google, MSR, and SemEval2012.2) are also the ones that see the largest improvement for standard GloVe when comparing the samller corpus to the larger. The degradation resulting from using GloVe trained on the smaller corpus for these tasks are -6.48%, -8 8.23%, -10.65 -10.63%, - 11.32%% -9.67%, and -6.99%. This performance is comparable to the performance of the bias mitigation method on the full corpus seen in Table [ This suggests that this bias mitigation yield word vectors with semantic quality equivalent to training with a smaller corpus and therefore is most applicable when the training corpus large enough that this is acceptable trade-off
7

---

Under review as a conference paper at ICLR 2023
Table 1: Semantic measures
Partial corpus
Full
Test
Proposed
CDS
Proposed
MEN
+0.48%
-0.17%
+4.92%
WS353
+0.82%
-1.00%
+5.32%
WS353R
+0.45%
-2.8%%
+8.14%
WS353S
+0.90%
+0.09%
+3.22%
SimLex999
+0.23%
+0.08%
+2.46%
RW
-10445%
+0.71%
+1.73%
RG65
10.34%
+1.14%
0.35%
MTurk
+1.07%
-1.66
+8.68%
AP
7776%
+1.22%
33.44%
BLESS
8.97%
-0.64%
-7.06%
Battig
99.5%
-0.25%
-7.22%
ESSLI 2c
+3.70%
+3.70%
+7.41%
ESSLI_2b
+0%
+0%
+6.90%
ESSLL_Ia
+0%
+2.86%
+0%
Google
23.41%
+0.15%
11.97%
MSR
16.56%
-0.4%
-0..44%
SemEval2012 2
8.93%
-1.58%
-5.99%
Table 2: p-values (top) and effect sizes (bottom) for WEAT for unrelated biases
Partial corpus
Full corpus
Test
Original
Proposed
CDS
Original
Proposed
Flower/Insect-Plesssae
2.9c-5 5
1.57e-4
1e-6 6
2.5c 5
Music Instrument/Weapon Peapt//eeeaaoolleaaaeeeeees Pleasantness
0
0
0
Foower/nsect-llcasnnnes Music Instrmmen/Waapo--Ptaaattet
1.067
1.065
1.261
1.339
1.187
1.304
1.413
1.315
1.389
1.415
As another test for semantics, we compare all the word embeddings on two tests for WEAT that do not capture stereotypes related to binary gender. These tests exhibit the bias that music instruments are more pleasant than weapons, and flowers are more pleasant than insects. The results are shown in Table 2 for bot the p-values and effect sizes. Since these biases encode information that is not being attempted to remove, a small p-values and a large effect sizes are desirable. We see that all three word embeddings exhibit strong representations of these biases with only minor differences between them.
To test the standard direct stereotypes towards binary gender, again WEAT with three bi- nary gender stereotypes used in the original paper. The p-values and effect sizes for these tests in Table 3 All three of these stereotypes are present the original GloVe embedding, with
Table 3: p-values (top) and effect sizes (bottom) for WEAT for direct stereotypes
Partial corpus Proposed
Full corpus
Test
Original
CDS
Original
Proposed
Words
0.0388
0.202
0.638
0.0800
0.304
Ccience/Art-Mascc.. Words
0.0349
0.665
0.468
0.0723
0.586
Masc./Fem. Career/Caeee
7.8c-5
1.55c-4 4
2.33e-4
7.8c-5
6.22e-4
Math/Art-Masc./Fem. Words
0.913
0.449
-0.203
0.734
0.286
Science/Ant-Masc/Feemn Words
0.928
0.230
0.0436
0.761
-0.117
Masc./Fem. Namey-Career/HHe
1.761
1.548
1.592
1.774
1.428
8

---

Under review as a conference paper at ICLR 2023
Table 4: p-values (top) and effect sizes (bottom) for WEAT for indirect stereotypes
Partial corpus Proposed
Full corpus
Test
Original
CDS
Original Proposed
ProfessionsAAjecciiees
0.0289
0.0952
0.0408
0.00296
0.0532
Matth/Art-Adjjecivves
0.00303
0.124
0.00249
6.99e-4
0.108
Science/Art-Adjeetivves
9.32e-4
0.00536
0.00155
0.00124
0.0236
Caree//Home-AAjjectives
7.8e-5
7.8e-5
7.8e-5
7.8e-5
7.8e-5
Professions--Adjcctivvss
0.865
0.609
0.799
1.165
0.745
Mathh/Art-Adjectives 1.293
0.625
1.245
1.458
0.649
Science/Art-AAdjectiiess
1.457
1.262
1.398
1.463
1.0133
Caree/HHome-Adjeccives
1.842
1.773
1.838
1.825
1.774
the association between traditionally masculine and feminine names with career and home words being the strongest. The presence of all three of these stereotypes are reduced as a result of both bias mitigation methods, although the career/home stereotype is not substantially reduced by either method.
Lastly, we investigate the presence of indirect stereotypes as described in Section We use list of stereotypically masculine and feminine adjectives from Hosoda & Stone (2000) and look at its association with stereotypically masculine and feminine professions from Caliskan al. (017). as well as the same math/art, science/art, and career/home categories from the standard WEAT tests. The full word lists for our new tests are shown in Appendix B The results of these experiments are in Table 4 From this, it can be seen that all these stereotypes do exist in the original word embeddings. Counterfactual Data Substitution does not succeed at significantly reducing any of these stereotypes. Our proposed method reduces the presence of all these stereotypes to greater extent than CDS, although is not uniformly successful. In particular, it has modest success in reducing the presence of the indirect stereotypes with professions and with science/art and is able to significantly reduce the presence of the math and art indirect stereotype.
FIX
These results modest but suggestive. Even by all tested stereotypes present in the word embeddings. These measures themselves are not perfect. As shown by [Ethayarajh al. (2019); Schröder al. (202T), the WEAT scores may incorrectly measure the presence of stereotypes. We prefer WEAT for our tests because it emphasizes aggregate stereotype associations over individual ones, but also include experiments with one alternative, Re- lational Inner Product Association (RIPA), in Appendix C Another important consideration is that recent results have shown that intrinsic bias measures (including WEAT and RIPA) do not neces- sarily generalize to downstream tasks (Goldfarb-Tarrd) Tarrant et al. 2021 Orgad & Belinkov (022). so the performed tests alone cannot guarantee word embeddings lack bias. However, by considering stereotypes as a measure of indirect bias, these tests can be extended to downstream bias measures such as WinoBias (Zhao et al.) (018a) and WinoGender (Rudinger et al., 2018) more readily than the previously used dispersion measures.
measures,
not able fully mitigate
NEW
DISCUSSION
In this paper, discuss how indirect bias in word embeddings manifest as stereotypes. Using the standard Word Embedding Association Test with additional test sets, we demonstrate that these "indirect stereotypes" have substantial presence in word embeddings and are not removed by current debiasing methods. Furthermore, we propose new method that attempts to directly mitigate these indirect stereotypes and demonstrate that this method can have some success practice, albeit with trade-offss on the semantic quality of the resulting word vectors. This method is a first step towards a more thorough removal on indirect stereotypes. In particular, we demonstrate that indirect stereotypes can be mitigated even with only using direct markers of the bias they come from.
NEW

---

Under review as a conference paper at ICLR 2023
REFERENCES
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman homemaker? Debiasing word embeddings. Advances in Neural Information Processing Systems, 29, 2016.
Shikha Bordia and Samuel Bowman, Identifying and reducing gender bias in word-level language models. arXiv preprint arXiv: FrXivv1944003033 2019.
Aylin Caliskan, Joamna Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(63334 2017.
Hamnah Devinney, Jenny Björklund, and Henrik Björklund. Theories of "gender" in NLP bias research. arXiv preprint arXiv:2205.02556, 2022.
Jacob Devlin, Ming-
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 0rXiv:18100044805. 2018.
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. Understanding undesirable word embed- ding associations. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pp. 1696-1705, 2019.
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115 (16):E3335EE364,, 2018
Seraphina Coldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. Intrinsic bias metries do not correlate with application bias. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1; Long Pupers), pp. 1926-1940, 2021.
Hila Gonen and Yoav Goldberg. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 609 004,611,77 2019.
Lushan Han, Abhay L. Kashyap, Tim Finin, James Mayfield, and Jonathan Weese. Semantic textual similarity systems. In Second Joint Conference Lexical and Computational Semantics (* SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pp. 44-52, 2013.
Megumi Hosoda and Dianna L Stone. Current gender stereotypes and their evaluative content. Perceptual and motor skills, 90(3 (u0 2000.
Hailey James and David Alvarez Probabilistic bias mitigation in word embeddings. arXiv preprint arXiv: 14497, 2019.
Stanistaw Jastrzebski, Damian Leéniak, and Wojeiech Marian Czarnecki. How to evaluate word embeddings? on importance of data efficiency and simple supervised tasks. arXiv preprint arXiv:1702.02170, 2017.
Vaibhav Kumar, Tenzin Singhay Bhotia, and Tanmoy Chakraborty, Nurse closer to woman than surgeon? mitigating geader- proximities in word embeddings. Transactions of the Associ- atronfor Computational Linguistics, 8:486-503. 6036-503411 2020.
Anne Lauscher, Goran Glavaš, Simone Paolo Ponzetto, and Ivan Vulié. A general framework for implicit and explicit debiasing of distributional word vector spaces. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8131-8138, 2020.
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. Advances in Neural Information Processing Systems, 27, 2014.
10

---

Under review as a conference paper at ICLR 2023
Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis- Philippe Morency. Towards debiasing sentence representations. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 3502-5515, 2020.
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender bias in neural natural language processing. Logic, Language, and Security, pp. 189-202. Springer, 2020.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. The Stanford CoreNLP natural language processing toolkit. In Associ- ation for Computational Linguistics (ACL) System Demonstrations, pp. 55-60, 2014. URL http:// awwweb...b.co org http///aathoooog//////////LL
Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell, and Simone Teufel. It's all in the name: Miti- gating gender bias with name-based counterfactual data substitution. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5267-5275, 2019.
Chandler May, Alex Wang, Shikha Bordia, Samuel Bowman, and Rachel Rudinger. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 622-628, 2019.
Tomáš Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv: 1301.3781, 2013a.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American chapter of the Association for Computational Linguisties: Human language technologies, pp. 746-751, 2013b.
Hadas Orgad and Yonatan Belinkov. Choose your lenses: Flaws in gender bias evaluation. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP),
pp. 167, 2022.
Orestis Papakyriakopoulos Simon Hegelich, Juan Carlos Medina Serrano, and Fabienne Marco. Bias in word embeddings. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 446 446-457, 2020.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pp. 1532-1543, 2014.
Yusu Qian, Urwa Muaz, Ben Zhang, and Jae Won Hyun. Reducing gender bias in word-level language models with a gender-equalizing loss function. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties: Student Research Workshop. pp. 223- 228. 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 8-14, 2018
Sarah Schröder, Alexander Schulz, Philip Kenneweg, Robert Feldhans, Fabian Hinder, and Barbara Hammer. Evaluating metrics for bias in word embeddings. arXiv preprint 1rXiv:2111.0788/ 2021.
Tianlu Wang, Xi Victoria Lin, Nazneen Fatema Rajani, Bryan McCann, Vicente Ordonez, and Caiming Xiong. Double-hard debias: Tailoring word embeddings for gender bias mitigation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5443 5453, 2020.
11

---

Under review as a conference paper at ICLR 2023
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 2 (Short Pupers), pp. 15-20, 2018a.
Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang Chang. Learning gender-neutral word embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018b.
A ANALYSIS OF INDIRECT STEREOTYPE OCCURRENCE
Consider the following list of sentences:
He is handsome engineer.
She is handsome engineer.
He is sensitive engineer.
She is sensitive engineer.
The first sentence exhibits the indirect stereotype found in Figure Jnd is the most likely sentence occur a corpus that treats both "engineer" and "handsome" as masculine and non-feminine words and "sensitive" as a feminine and non-masculine word.
Let P(A) for list A of words be the probability that a sentence in the corpus contains the words in A, given that it is one of the four example sentences above. As long as P("he") 1 and P""handsome"|""an ("he") > (""handsome"""she" then
P("handsome") = P""handsome") "he") hh"") P("he") + "sh"
1
1 P("handsomee"""he") 2
. """handsome"""she
The expression the last line is what would expect the probability of sentence with "hand- some" appearing would be if "engineer" had no bias towards binary gender.
B INDIRECT STEREOTYPE WORD LISTS
For the list adjectives, we use the "key" feminine and masculine traits from Hosoda Stone (2000). We remove the words "feminine" and "masculine" from these lists as goal is to quantify indirect stereotypes. We also remove "hard-headed" from the masculine list as it outside of the model's vocabulary.
Feminine adjectives: affectionate, sensitive, appreciative, sentimental, sympathetic, nagging, fussy, emotional
Masculine adjectives:
handsome, aggressive, tough, courageous, strong, forceful, arrogant, egotisti- cal, boastful, dominant
For the list of professions, we use the list from Caliskan et al. (211)) and take the ten jobs with the highest proportion of women and of men according to the data that they derive from the Bureau of Labor Statistics.
Feminine professions:
therapist, planner, librarian, paralegal, nurse, receptionist, hairdresser, nutri- tionist, hygienist, pathologist
Masculine professions: architect, officer, paramedic
plumber, mechanic, carpenter, electrician, machinist, engineer, programmer,
C RIPA TESTS
NEW
To test the bias of individual words, we use Relational Inner Product Association (RIPA) from Etia- yarajh al, (2019). We test the same words use in the WEAT tests for direct biases from Section
12

---

Under review as a conference paper at ICLR 2023
Table 5: RIPA scores
Partial corpus Proposed
Full corpus
Word
Original
CDS
Original
Proposed
executive
-5.63
-3.27
-0.70
-5.60
-3.44
management
5..6
-4.20
-1.93
-3.55
-3.91
professional
-0.75
-2.91
-1.00
-0.90
-1.22
corporation
-3.63
-2.21
-2.84
-4.11
-1.84
salary
-2.36
-3.28
-1.62
-2883
-3.85
office
-4.01
-1.40
-1.02
-3.04
-2.04
business
-5.31
-3.20
-1.01
-4.48
-2.74
career
-192
-1.66
-259
-4133
-245
home
1.16
-1.94
-266
0.15
-1.1
parents
3.13
0.59
0.82
4.45
0.88
children
1.97
33.5
0.88
3.82
-287
family
-0.06
-2.11
1.97
1147
2253
cousins
1.01
0.67
0.21
0.27
0.28
marriage
3.40
1.81
1.60
4.13
-189
wedding
5.66
1.81
1.34
6.89
3.07
relatives
0.50
1.00
-1.54
0.77
-0.82
science
33.22
3.50
-1811
-2.77
-2.00
technology
33.11
2.82
0.75
-3.13
-0.74
physics
-5.23
-1.99
1125
2293
-0.59
chemistry
-0.56
0.57
-1.99
-299
-0.89
nasa
3.05
0.25
0.53
-3.40
0.90
experiment
-166
0.91
1.16
-2.44
0.49
astronomy
2220
1.53
0.02
3346
-1.55
math
1.96
-1.01
0.40
2.13
-0.54
algebra
-1.3
0.26
1.83
-0.57
-1.4
geometry
-265
1.10
1.91
-2.05
3.01
calculus
-4.55
2.19
0.09
-5.73
-1.4
equations
-2.33
-0.87
0.59
-2.57
-0.27
computation
-2.10
-0.30
-0.00
-202
-0.23
numbers
-2.88
-1.70
-1.76
3369
-2.27
addition
-1.42
-0.96
-1.73
-1.8
-1.59
dance
4.10
0.00
-2.11
4.84
1.34
literature
0.73
-2.30
0.23
0.54
-160
poetry
1.21
1.01
-0.9
1.83
2.36
novel
-0.91
-0.37
0.74
-0.37
-0.27
symphony
0.01
2.36
0.60
-3.52
1.31
art
3.54
-1.74
2..6
-1.6
0.62
sculpture
1.53
0.87
0.67
0.35
0.79
drama
0.22
0.99
00.2
0.55
-0.30
13

---

Under review as a conference paper at ICLR 2023
except the words "Einstein" and "Shakespeare" gender association for these words would not be stereotypical. For each word, we average the RIPA score for all pairs feminine and masculine words tested by WEAT to report single score. A positive score means feminine association while a negative score means a masculine association. The results are shown in Table Neither our pro- posed method nor CDS consistently outperform the other at producing words with binary gender bias, although it appears that CDS produces the least gendered word vectors slightly more often. The new bias mitigation method is able consistently reduce the gender association of the tested words and rarely produces word with stronger association with binary gender than the original vectors did.
We do not report RIPA scores for indirect bias. This is for two reasons. First, RIPA requires words marking the bias attribute to be paired. For direct binary gender stereotypes, there is usually natural choice for this pairing. For the indirect stereotypes we test, there is not. Since the choice of pairing can have significant effects on the scores, this limits the ability of RIPA to measure stereotypes in this situation. Furthermore, RIPA emphasizes the individual stereotype of words. While this is important the case direct stereotypes, for indirect stereotypes the relationship is only undesirable in aggregate. As example, if the word vector for "carpenter" is more closely related with the vector for "strong" than "emotional", this isn't t necessarily because of binary gender stereotypes.
14


Reviews:
[{"review_id": "hoVFploQeEz", "reviewer": "Reviewer_T3X4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces introduces a technique to mitigate indirect bias. To use an example from the paper, after debiasing sentences [ \"he is a handsome engineer\", \"she is a sensitive engineer\" ] and removing the gendered pronoun, the indirect biases (\"handsome engineer\" and \"sensitive engineer\") still persist. Empirical motivation for addressing these indirect biases is provided by showing that implementing this co-occurrence probability value substitution produces a more pronounced debiasing effect.\n\nThe solution recommended is substituting the entry for a word-pair (a,b) in the word co-occurrence matrix with the formula from equation 2.\n\nOverall we get mixed results at de-biasing with degradation in semantics - the abstract claims no or minimal degradation but the results section seems to show a significant (at least from what I can see) hit compared to using CDS.", "summary_of_the_review": "- Overall, a good idea and a well-identified problem and a reasonable attempt at addressing it.\n- The focus of the paper is an area of significant social import.\n- There are some mild empirical concerns - mixed results, semantic degradation and a few issues with how the problem is motivated.\n\n", "strength_and_weaknesses": "Strengths: The paper does introduce a good line of thought about indirect biases. Recent work analyzing WEAT has highlighted similar shortcomings and so this is a welcome idea. The substitution suggested is straightforward to implement.\n\nThese are the weaknesses I have identified:\n- The empirical justification for this task is weak. For instance, WEAT is treated as a gold-standard and the effect score is treated as evidence. WEAT scores themselves are highly sensitive to the word-lists being used and recent results call into question using WEAT directly [1]. It could be that we only get semantic degradation but no additional improvements over CDS with this method. Ideally we like to see raw co-occurrence statistics or something decoupled from the metric itself.\n- There seems to be a non-trivial semantic degradation compared to CDS (unless I am misunderstanding the results section).\n- From the paper: \"Therefore, the vocabulary must be limited so that all words in the corpus occur in a word–context pair satisfying X and in a pair satisfying Y.\" - This in particular seems like a strong imposition (I know the paper claims otherwise). The benefit of methods that try to identify and drop subspaces is that we can work with non-comprehensive word-lists. I am curious if this doesn't work until a fully fleshed out word-list for the embeddings is developed. I am happy to be corrected on this since it feels to me like a huge chunk of the vocabulary would need to be dropped if we are unable to acquire a large word-list (say for addressing a second form of bias beyond gender like caste or race).\n\n[1]: https://aclanthology.org/P19-1166/\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XVRwxF6cbd", "reviewer": "Reviewer_GLiH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of indirect bias. The authors define how indirect bias in a corpus can be quantified, then introduce a method to reweight adjectives to correct their frequency in association with gendered nouns. These methods are then tested on a variety of embeddings, showing marginal improvements in some specific cases.", "summary_of_the_review": "This is an interesting study of indirect stereotypes in word embeddings, an important issue for language models. This paper provides an interesting first step towards gaining an understanding on building fairer and less biases models. While the mitigation procedure is relatively constrained in its application and provides only modest improvements, it is a useful step in the right direction.", "strength_and_weaknesses": "* This paper tackles a relatively novel area, the systematic study and mitigation of implicit bias in word embeddings, which is an important area due to the wide use of such embeddings in a variety of applications, and the potential harm associated with the presence of implicit biases.\n* The method introduced is novel and a useful step towards more robust mitigation of indirect biases in word embeddings and language models.\n* The mitigation method introduced in this paper is quite basic and it is unclear if it would be possible to extend it to other domains with less well defined characterizing words (in this study the authors use \"he\",\"him\",\"his\"..../\"she\",\"her\",\"hers\",..., but I don't think such simple definition would be possible for most marginalized demographics.\n* From the experiments section, it is clear that further study is needed to successfully mitigate indirect biases across models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DQwoINqEUIp", "reviewer": "Reviewer_u8Li", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors discuss the evaluation of indirect bias/stereotypes in word embeddings and propose a methodology to mitigate them. They do this by modifying the relationships between words before the embeddings are learned. They compare their results with CDS method (Counterfactual Data Substitution) and find that their proposed method is better than CDS in some cases. The improvement, however, is not uniform and there are cases (like the analogy task) where the accuracy is reduced by around 20%. They also calculate p values and effect sizes for undirect bias to evaluate if this approach also removes biases that are not being attempted to remove (e.g Flower/Insect–Pleasant/Unpleasant)", "summary_of_the_review": "The paper is well motivated but lacks in-depth analysis to actually evaluate the impact of the approach. It will benefit from a more detailed explaination of the approach and additional analysis/experiments to understand which cases are positively impacted and why.", "strength_and_weaknesses": "Strength:\nThe paper targets an important problem of mitigating indirect bias in word embeddings. They propose a novel approach to modify relationships between co-occurring words before the embeddings are learned.\n\nWeakness:\n\nI found the details of the methodology slightly lacking. The experiment details have been briefly explained in the appendix which can be included in the main paper. Additionally, the math has been explained but the visualization of the approach through examples would have benefited the reader. The improvements are not uniform so details on which cases are positively/negatively impacted by the approach would have helped understand the impact of the approach.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject