Question: What do they evaluate in their paper?
Evidence: 
With the wealth of information being posted online daily, Relation Extraction (RE) has become increasingly important. RE aims specifically to extract relations from raw sentences and represent them as succinct relation tuples of the form (head, relation, tail). An example is (Barack Obama, spouse, Michelle Obama).
The concise representations provided by RE models have been used to extend Knowledge Bases (KBs) BIBREF0, BIBREF1. These KBs are then used heavily in NLP systems, such as Task-Based Dialogue Systems. In recent years, much focus in the NRE community has been centered on improvements in model precision and the reduction of noise BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6. Yet, little attention has been devoted towards the fairness of such systems.
In this paper, we take the first step at understanding and evaluating gender bias in NRE systems. We analyze gender bias by measuring the differences in model performance when extracting relations from sentences written about females versus sentences written about males. Significant discrepancies in performance between genders could diminish the fairness of systems and distort outcomes in applications that use them. For example, if a model predicts the occupation relation for with higher recall for male entities, this could lead to KBs having more occupation information for males. Downstream search tasks using that KB could produce biased predictions, such as ranking articles about female computer scientists below articles about their male peers.
We provide the first evaluation of social bias in NRE models; specifically, we evaluate gender bias in English language predictions of a collection of popularly used and open source NRE models BIBREF2, BIBREF4, BIBREF3, BIBREF5. We evaluate OpenNRE on two fronts: (1) examining Equality of Opportunity BIBREF7 when OpenNRE is trained on an unmodified dataset and (2) examining the effect that various debiasing options BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12 have on both absolute F1 score and the difference in F1-scores on male and female datapoints.
However, carrying out such an evaluation is difficult with existing NRE datasets, such as the NYT dataset from BIBREF13, because there is no reliable way to obtain gender information about the entities. Thus, we create a new dataset specifically aimed at evaluating gender bias for NRE, just as prior work has done for other tasks like Coreference Resolution BIBREF14, BIBREF9. We call our dataset WikiGenderBias and make it publicly available. Our contributions are as such:
WikiGenderBias is the first dataset aimed at training and evaluating NRE systems for gender bias. It contains ground truth labels for the test set and about 45,000 sentences in total.
We provide the first evaluation of NRE systems for gender bias and find that it exhibits gender bias.
We demonstrate that using both gender-swapping and debiased embeddings effectively mitigates bias in the model's predictions and that using genderswapping improves the model's performance when the training data contains contextual biases.
In this paper, we take the first step at understanding and evaluating gender bias in NRE systems.
Answer: Gender bias in Relation Extraction (RE) models.