SlowGuess/ACL_2020_OCR huggingface dataset:
Towards Understanding Gender Bias in Neural Relation Extraction
Andrew Gaut\*, Tony Sun\*, Shirlyn Tang, Yuxin Huang
Jing Qian†, Mai ElSherief††, Jieyu Zhao‡,
Diba Mirza†, Elizabeth Belding†, Kai-Wei Chang‡, and William Yang Wang†
$^{\dagger}$ Department of Computer Science, UC Santa Barbara
$^{\ddagger}$ Department of Computer Science, UC Los Angeles
†† School of Interactive Computing, Georgia Institute of Technology
{ajg, tonysun, shirlyntang, yuxinhuang}@ucsb.edu
{jing_qian, dimirza, ebelding, william}@cs.ucsb.edu
melsherief@gatech.edu
{jyzhao, kwchang}@cs.ucla.edu
Recent developments in Neural Relation Extraction (NRE) have made significant strides towards automated knowledge base construction. While much attention has been dedicated towards improvements in accuracy, there have been no attempts in the literature to evaluate social biases exhibited in NRE systems. In this paper, we create WikiGenderBias, a distantly supervised dataset composed of over 45,000 sentences including a  $10\%$  human annotated test set for the purpose of analyzing gender bias in relation extraction systems. We find that when extracting spouse and hypernym (i.e., occupation) relations, an NRE system performs differently when the gender of the target entity is different. However, such disparity does not appear when extracting relations such as birth date or birth place. We also analyze two existing bias mitigation techniques, word embedding debiasing and data augmentation. Unfortunately, due to NRE models relying heavily on surface level cues, we find that existing bias mitigation approaches have a negative effect on NRE. Our analysis lays groundwork for future quantifying and mitigating bias in relation extraction.
With the wealth of information being posted online daily, relation extraction has become increasingly important. Relation extraction aims specifically to extract relations from raw sentences and represent them as succinct relation tuples of the form (head, relation, tail) e.g., (Barack Obama, spouse, Michelle Obama).
The concise representations provided by relation extraction models have been used to extend Knowledge Bases (KBs) (Riedel et al., 2013; Subasic et al., 2019; Trisedya et al., 2019). These KBs are then used heavily in NLP systems, such as question answering systems (Bordes et al., 2014; Yin et al., 2016; Cui et al., 2019). In recent years, much focus in the Neural Relation Extraction (NRE) community has been centered on improvements in model precision and the reduction of noise (Lin et al., 2016; Liu et al., 2017; Wu et al., 2017; Feng et al., 2018; Vashisth et al., 2018; Qin et al., 2018). Yet, little attention has been devoted towards the fairness of such systems.
We take the first step at understanding and evaluating gender bias in NRE systems by measuring the differences in model performance when extracting relations from sentences written about females versus sentences written about males. If a NRE model predicts a relation such occupation with higher recall on male entities, this could lead to the resulted knowledge bases having more occupation information for males than for females (see the illustration in Figure 1). Eventually, the gender bias in knowledge bases may affect downstream predictions, causing undesired allocative harms (Crawford, 2017) and reinforcing gender-stereotypical beliefs in society.
In this paper, we present an evaluation framework to analyze social bias in NRE models. Specifically, we evaluate gender bias in English language predictions of a collection of popularly used and open source NRE models $^{1}$  (Lin et al., 2016; Wu et al., 2017; Liu et al., 2017; Feng et al., 2018). We evaluate on two fronts: (1) examining gender bias
<https://github.com/thunlp/OpenNRE/
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2943-2953
Figure 1: An illustration of gender bias in relation extraction and how it affects a downstream application. In their Wikipedia articles, both Beatrice (female) and Ben (male) are described as engineers. These sentences contain the (entity; occupation; engineer) relation. However, the model only predicts that the sentence from the male article expresses the occupation relation. If on a large scale, models extract the (entity; occupation; engineer) relation more often for males, knowledge bases will contain information for male engineers more often than female. Question answering models that query these knowledge bases may give biased answers and propagate gender bias downstream.
exhibited in a model that is trained on a relation extraction dataset; and (2) examining if the existing bias mitigation techniques (Bolukbasi et al., 2016; Zhao et al., 2018; Lu et al., 2018) can be applied to reduce the bias in an NRE system while maintaining its performance.
Carrying out such an evaluation is difficult with existing NRE datasets, such as the NYT dataset (Sandhaus, 2018), because there is no reliable way to obtain gender information about the entities mentioned in input sentences. Therefore, we create a new dataset, WikiGenderBias, specifically aimed at evaluating gender bias for NRE. WikiGenderBias is a distantly supervised dataset extracted using Wikipedia and DBPedia. It contains 45,000 sentences, each of which describe either a male or female entity with one of four relations: spouse, hypernym (i.e., occupation), birthDate, and birthPlace. We posit that a biased NRE system leverages gender information as a proxy when extracting knowledge tuples with spouse and hypernym relations. However, gender of the entity does not affect the extraction of relations such as birthDate and birthPlace, as they are not intuitively related to gender. Experiment results confirm our conjecture.
Our contributions are as such:
- We create WikiGenderBias, a new dataset for evaluating gender bias in NRE systems.
- We present an evaluation framework to demonstrate that gender bias is exhibited in NRE model outputs.
- We test several existing bias mitigation ap
proaches to reducing gender bias in NRE system. Our analysis sheds light for designing future mitigating techniques.
Gender Bias Measurement. Existing studies have revealed gender bias in various NLP tasks (Zhao et al., 2017; Rudinger et al., 2018; Zhao et al., 2018; Dixon et al., 2018; Lu et al., 2018; Kiritchenko and Mohammad, 2018; Romanov et al., 2019; Sheng et al., 2019; Sun et al., 2019). People have proposed different metrics to evaluate gender bias, for example, by using the performance difference of the model on male and female datapoints for bias evaluation (Lu et al., 2018; Kiritchenko and Mohammad, 2018). Other metrics have been proposed to evaluate fairness of predictors and allocative bias (Dwork et al., 2012; Hardt et al., 2016), such as Equality of Opportunity. In this work, we use both of these metrics to evaluate NRE models.
Mitigation Methods. After discovering gender bias existing, prior work has developed various methods to mitigate that bias (Escudé Font and Costa-jussà, 2019; Bordia and Bowman, 2019). Those mitigation methods can be applied in different levels of a model, including in the training phase, in the embedding layer, or in the inference procedure. In this paper, we test three existing debiasing approaches, namely data augmentation (Zhao et al., 2018; Lu et al., 2018), and word embedding debiasing technique (Hard Debiasing (Bolukbasi et al., 2016)) for mitigating bias in NRE models.
Original Dataset	Equalized Dataset
Entity Pairs	Instances	Entity Pairs	Instances
M	F	M	F	M	F	M	F
Train	12,139	4,571	27,048	9,391	2,479	4,571	9,465	9,415
Development	1,587	553	3,416	1,144	336	553	1,144	1,144
Test	1,030	1,101	2,320	2,284	1,030	1,101	2,320	2,284
Total	14,756	6,225	32,784	12,819	3,845	6,225	12,929	12,843
Table 1: WikiGenderBias's Dataset splits. Entity Pairs means distinct pairs  $(e1,e2)$  such that  $(e_1, relation, e_2)$  is a relation in WikiGenderBias. Instances are the total number of  $(e1, relation, e2, sentence)$  tuples in WikiGender-Bias, where sentence is distantly supervised. We categorize an entity pair as male (female) if  $e1$  is male (female), since the sentence in the instance is taken from  $e1$  's article and we define datapoints as male (female) if that is the gender of the subject of the article. The left two entries are for the dataset taken from the true distribution; the right two are the gender-equalized dataset created by down-sampling male instances.
Neural Relation Extraction. Relation extraction is a task in NLP with a long history that typically seeks to extract structured tuples  $(e_1, r, e_2)$  from texts (Bach and Badaskar, 2007). Early on, learning algorithms for relation extraction models were typically categorized as supervised, including feature-based methods (Kambhatla, 2004; Zhou et al., 2005; Zhao and Grishman, 2005) and kernel-based methods (Lodhi et al., 2002; Zelenko et al., 2003), or semi-supervised (Brin, 1998; Agichtein and Gravano, 2000; Etzioni et al., 2005; Pantel and Pennacchiotti, 2006), or purely unsupervised (Etzioni et al., 2008). Supervised approaches suffer from the need for large amounts of labelled data, which is sometimes not feasible, and generalizes poorly to open domain relation extraction, since labeled data is required for every entity-relation type (Bach and Badaskar, 2007; Mintz et al., 2009). Many semi-supervised approaches rely on pattern-matching, which is not robust, and many are unable to extract intra-sentence relations (Bach and Badaskar, 2007). When data annotation is insufficient or hard to obtain and semi-supervised approaches are insufficient, the distant supervision assumption is used to collect data to train supervised models (Mintz et al., 2009). Given a relation  $(e_1, r, e_2)$  in a knowledge base (KB), distant supervision assumes any sentence that contains both  $e_1$  and  $e_2$  expresses  $r$  (Mintz et al., 2009). Great efforts have been made to improve NRE models by mitigating the effects of noise in the training data introduced by Distant Supervision (Hoffmann et al., 2011; Surdeanu et al., 2012; Lin et al., 2016; Liu et al., 2017; Feng et al., 2018; Qin et al., 2018). However, to our knowledge, there are no studies on bias or ethics in NRE, which is filled by this work.
3 WikiGenderBias
We define gender bias in NRE as a difference in model performance when predicting on sentences from male versus female articles. Thus, we need articles written about entities for which we can identify the gender information. However, to obtain gender information for existing annotated datasets could be costly or impossible. Thus, we elected to create WikiGenderBias with this gender information to be able to detect scenarios like that in Figure 1. The data statistics of WikiGenderBias are given in Table 1.
3.1 Dataset Creation
Wikipedia is associated with a knowledge base, DBPedia, that contains relation information for entities with articles on Wikipedia (Mendes et al., 2012). Many of these entities have gender information and their corresponding articles are readily available. Therefore, we create our dataset based on sentences extracted from Wikipedia.
To generate WikiGenderBias, we use a variant of the distant supervision assumption: for a given relation between two entities, if one sentence from an article written about one entity also mentions the other entity, then we assume that such sentence expresses the relation. For instance, if we know (Barack, spouse, Michelle) is a relation tuple and we find the sentence He and Michelle were married in Barack's Wikipedia article, then we assume that sentence expresses the (Barack, spouse, Michelle) relation. This assumption is similar to that made by Mintz et al. (2009) and allows us to scalably create the dataset.
WikiGenderBias considers four relations that stored in DBPedia: spouse, hypernym, birthDate,
2945
Relation	Head Entity	Tail Entity	Sentence
Birthdate	Robert M. Kimmitt	December 19, 1947	Robert M. Kimmitt ( born December 19 , 1947 ) was United States Deputy Secretary of the Treasury under President George W. Bush .
Birthplace	Charles Edward Stuart	Rome	Charles was born in the Palazzo Muti , Rome , Italy , on 31 December 1720 , where his father had been given a residence by Pope Clement XI
Spouse	John W. Caldwell	Sallie J. Barclay	Caldwell married Sallie J. Barclay , and the couple had one son and two daughters .
hybernym	Handry Satriago	CEO	Handry Satriago ( born in Riau , Pekanbaru on June 13 , 1969 ) is the CEO of General Electric Indonesia .
Table 2: Examples of relations of each type in WikiGenderBias.
and birthPlace. Note that the hypernym relation on DBPedia is similar to occupation, with entities having hypernym labels such as Politician. We also generate negative examples by obtaining datapoints for three unrelated relations: parents, deathDate, and almaMater. We label them as NA (not a relation). As each sentence only labelled with one relation based on our distant supervision assumption, WikiGenderBias is a 5-class classification relation extraction task. Figure 2 lists the label distribution.
We hypothesize that a biased relation extraction model might use gender as a proxy to influence predictions for spouse and hypernym relations, since words pertaining to marriage are more often mentioned in Wikipedia articles about female entities and words pertaining to hypernym (which is similar to occupation) are more often mentioned in Wikipedia articles about male entities (Wagner et al., 2015; Graells-Garrido et al., 2015). On the other hand, we posit that birthDate and birthPlace would operate like control groups and believe gender would correlate with neither relation.
To simplify the analysis, we only consider the head entities that associated with at least one of the four targeted relations. We set up our experiment such that head entities are not repeated across the train, dev, and test sets so that the model will see only new head entities at the test time. Since we obtain the distantly supervised sentences for a relation from the head entity's article, this guarantees the model will not reuse sentences from an article. However, it is possible that the head entity will appear as a tail entity in other relations because an entity could appear in multiple articles. The data splits are given in Table 1.
Besides, Wikipedia includes more articles written about males than about females. Therefore,
there are more male instances than female instances in WikiGenderBias as well. To remove the effect of dataset bias in our analysis, we also create a gender-equalized version of the training and development sets by down-sampling male instances. We discuss the creation of gender-equalized test set below.
3.2 Test Sets
We equalize the male and female instances in the test set. In this way, a model cannot achieve high performance by performing well only on the dominant class. Furthermore, since some data instances that are collected using distant supervision are noisy, we annotated the correctness of the test instances using Amazon Mechanical Turk annotations to perform a fair comparison.
Specifically, we asked workers to determine whether or not a given sentence expressed a given relation. If the majority answer was "no", then we labeled that sentence as expressing "no relation" (we denote them as NA). Each sentence was annotated by three workers. Each worker was paid 15 cents per annotation. We only accepted workers from England, the US or Australia and with HIIT Approval Rate greater than  $95\%$  and Number of HIITs greater than 100. We found the pairwise inter-annotator agreement as measured by Fleiss' Kappa (Fleiss, 1971)  $\kappa$  is 0.44, which is consistent across both genders and signals moderate agreement. We note that our  $\kappa$  value is affected by asking workers to make binary classifications, which limits the degree of agreement that is attainable above chance. We also found the pairwise inter-annotator agreement to be  $84\%$ .
Figure 2: Proportion of sentences corresponding to a given relation over total sentences in WikiGenderBias for each entity. This demonstrates that, of the entities we sampled to create WikiGenderBias, the spouse relation is expressed more often relative to the birthdate, birthplace, and hypernym relations in articles about female entities than in articles about male entities. Additionally, hypernym is mentioned more often relative to the other relations in articles about male entities than in articles about female entities.
We build on the work of Graells-Garrido et al. (2015), who discovered that female entities are more likely to have spouse information in the Infoboxes on their Wikipedia page than male entities. Figure 2 demonstrates a further discrepancy: amongst articles we sampled, proportionally, the spouse relation is mentioned more often relative to hypernym, birthPlace, and birthDate in female articles than in male articles. Additionally, we show that amongst female and male articles we sampled, hypernyms are mentioned more often in male than female articles relative to spouse, birthPlace, and birthDate (see Section 2). This observation aligns with the literature, arguing that authors do not write about the two genders equally (Wagner et al., 2015; Graells-Garrido et al., 2015).
4 Gender Bias in NRE
We evaluate OpenNRE (Han et al., 2019), a popular open-source NRE system. OpenNRE implements the approach from (Lin et al., 2016). To convert sentences into vectors, researchers propose convolutional neural networks as well as the piecewise convolutional neural networks (PCNN) which retain more structural information between entities (Zeng et al., 2015). In this work, we use a PCNN with Selective Attention for the experiments.
We train every encoder selector combination on the training set of WikiGenderBias and its genderequalized version. We input Word2Vec (Mikolov et al., 2013) word embeddings trained on WikiGen
derBias to the models $^2$ . We use commit 709b2f from the OpenNRE repository tensorflow branch to obtain the models.
4.1 Performance Parity Score
The goal of a successful relation extraction model is to maximize F1 score while minimizing the model performance gender gap (or disparity score). However, when comparing different systems, it is hard to decide what is the right balance between these two objectives. On one end, a model which has zero gender gap but has only  $10\%$  accuracy for both male and female test instances has almost no practical value. Other methods that have high accuracy or F1 score may do so at the cost of a wide gender gap. Although our test set for WikiGenderBias is gender-equalized, one can imagine that improving performance on a test set that is heavily skewed towards males can be done by focusing on male test instances while largely ignoring female ones. Therefore, it is important to strike a balance between model performance and inter-group parity.
To measure model performance, we use Macro-average F1 score. To measure inter-group parity, we use the pairwise difference in F1 scores averaged over all the groups for predictions on a given relation  $i$ . We describe the average difference over all relations as Disparity Score (DS):
$$
D S = \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{x} \sum_ {j = 1} ^ {x} \sum_ {k = j + 1} ^ {x} \left| F _ {1 _ {i k}} - F _ {1 _ {i j}} \right|,
$$
where  $n$  denotes the number of relations (e.g. {birthDate, birthPlace, spouse, hypernym}).  $x$  denotes the number of groups (e.g. {male, female}).  $F_{1_{rk}}$  is the  $F_1$  score for the model when predicting datapoints with true label relation  $r$  that belong to group  $k$ . (So, for instance,  $F_{1_{spouse,male}}$  is the  $F_1$  score on sentences that express the spouse relation from male articles.) The Disparity Score measures the F1 score gap between predictions on male and female data points.
Bringing these two metrics together, we propose the Performance Parity Score (PPS). PPS is the Macro-average difference (equally weighted) of the
2We performed Grid Search to determine the optimal hyperparameters. We set epochs  $= 60$  , learning rate  $\eta = 0.5$  early stopping with patience of 10, batch size  $= 160$  , and sliding window size  $= 3$  (for CNN and PCNN). These hyperparameters are similar to the default settings found in the OpenNRE repository tensorflow branch, which uses epochs  $= 60$  learning rate  $\eta = 0.5$  , and early stopping with patience of 20.
Figure 3: Aggregate performance of the NRE model for each relation (left) and male-female F1 score gender gap for each relation (right). An ideal model maximizes performance and minimizes the gender gap. The experiment is run five times. We give the mean values and standard error bars.
F1 score subtracted by the model performance gender gap, which we defined as the Disparity Score, per relation. We place equal importance on the F1 score and Disparity Score by giving each score an implicit weight of 1. In our formula for PPS above, we also divide the final result by the number of relations  $n$ . This keeps the range of PPS within  $[-1, 1]$ , although PPS will generally fall between  $[0, 1]$  because it is highly unlikely that the Disparity Score will be greater than the overall F1 score. PPS seeks to incentivize a combination of both model performance and inter-group parity for the task of relation extraction:
$$
\begin{array}{l} P P S = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(F _ {1 _ {i}} - \frac {1}{x} \sum_ {j = 1} ^ {x} \sum_ {k = j + 1} ^ {x} \left| F _ {1 _ {i k}} - F _ {1 _ {i j}} \right|\right) \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} F _ {1 _ {i}} - \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{x} \sum_ {j = 1} ^ {x} \sum_ {k = j + 1} ^ {x} \left| F _ {1 _ {i k}} - F _ {1 _ {i j}} \right| \\ = \text {M a c r o} F _ {1} \text {s c o r e} - \text {D i s p a r i t y S c o r e}. \\ \end{array}
$$
4.2 Measuring Performance Differences
Similar to the parity term in PPS, gender bias can be measured as the difference in a performance metric for a model when evaluated on male and female datapoints (De-Arteaga et al., 2019). We define male (female) datapoints to be relations for which the head entity is male (female), which means the distantly supervised sentence is taken from a male (female) article. Prior work has used area under the precision-recall curve and F1 score to measure NRE model performance (Gupta et al., 2019; Han et al., 2019; Kuang et al., 2019). We use MacroF1 score as our performance metric. We denote the F1 gender difference as  $F1_{Gap}$ , which is used to calculate the disparity score. A larger disparity score indicates higher bias in predictions.
4.3 Equality of Opportunity Evaluation
Equality of Opportunity (EoO) was originally proposed to measure and address allocative biases (Hardt et al., 2016). Consequently, we examine this metric in the context of relation extraction to better understand how allocative biases can begin to emerge at this stage.
Equality of Opportunity (EoO) is defined in terms of the joint distribution of  $(X, A, Y)$ , where  $X$  is the input,  $A$  is a protected attribute that should not influence the prediction, and  $Y$  is the true label (Hardt et al., 2016). A predictor satisfies Equality of Opportunity if and only if:  $P(\hat{Y} = 1|A = male, Y = 1)$  and  $P(\hat{Y} = 1|A = female, Y = 1)$ . In our case  $A = \{male, female\}$ , because gender is our protected attribute and we assume it to be binary. We evaluate EoO on a per-relation, one-versus-rest basis. Thus, when calculating EoO for spouse,  $Y = 1$  indicates the true label is spouse and  $\hat{Y} = 1$  indicates a prediction of spouse. We do this for each relation. Note that this is equivalent to measuring per-relation recall for each gender.
4.4 Result