SlowGuess/ACL_2020_OCR huggingface dataset:
As shown in Figure 3, the NRE system performs better when predicting the spouse relation on sentences from articles about male entities than from articles on female entities (see Figure 3, right). Further, there is a large recall gap (see EoO column, row 1, in Table 3). Notably, the gender difference in performance is much smaller on birthDate, birthPlace, and hypernym relations, although the gender difference is non-zero for birthPlace and hyperpym. This is interesting given that a higher percentage of female instances in WikiGenderBias are spouse relations than male (see Figure 2). We encourage future work to explore whether the writing style differences between male and female spouse in
Spouse	Birth Date	Birth Place	Hybernym	Total
F1Gap	EoO	F1Gap	EoO	F1Gap	EoO	F1Gap	EoO	F1 Score	Disparity Score	PPS
PCNN,ATT	.041	.058	.004	.000	-.003	-.017	.015	.009	.886	.016	.870
CNN,ATT	.034	.043	-.003	.001	.014	.004	.028	.014	.882	.020	.862
RNN,ATT	.032	.043	.015	.019	.005	-.011	-.006	-.006	.889	.014	.875
BIRNN,ATT	.039	.061	.013	.021	-.016	-.033	-.013	-.026	.884	.020	.864
PCNN,AVE	.034	.044	.005	.010	-.001	-.011	.005	-.005	.903	.011	.892
CNN,AVE	.027	.028	.013	.029	.007	.009	.002	-.028	.895	.012	.883
RNN,AVE	.039	.036	.004	.021	.016	.020	.006	-.012	.912	.016	.895
BIRNN,AVE	.024	.018	.001	.015	.009	.018	-.005	-.022	.913	.010	.903
Table 3: Results from running combinations of encoders and selectors of the OpenNRE model for the male and female genders of each relation. A positive  $F1_{Gap}$  indicates a higher F1 on male instances. A higher Equality of Opportunity (EoO) indicates higher recall on male instances. A higher PPS score indicates a better balance of performance and parity (see Section 4.1). We ran the experiment five times and report the mean values. Varying the encoder and selector appears to have no conclusive effect on bias, although models using the average selector doe achieve better aggregate performance. These results were obtained using the gender unequalized training data.
stances causes those male instances to be easier to classify.
In addition, we explore different types of sentence encoder and sentence-level attention used in the creation of the bag representation for each entity pair and examined how these models performed on our dataset. Notably, the bias in spouse relation persists across OpenNRE architectures (see Table 3). It seems models using average attention, which merely averages all the sentence vectors in the bag to create a representation of the entire bag, allows for better aggregate performance on WikiGender-Bias. However, the effect on the Disparity Score (and therefore the bias exhibited in the predictions) seems negligible.
We note that these results do not necessarily indicate that the model itself contains biases given that males and females are written about differently on Wikipedia. These results do, however, demonstrate that we must be cautious when deploying NRE models, especially those trained on Wikipedia data, since they can propagate biases latent in their training data to the knowledge bases they help create.
5 Bias Mitigation
We examine data augmentation and Hard-Debiasing as bias mitigation techniques for reducing gender bias in NRE system.
5.1 Bias Mitigation Techniques
Equalizing the Gender Distribution Sometimes, the true distribution contains an imbalance in gendered data. For instance, perhaps the training set contains more instances from male articles than female. To mitigate this, one can simply downsample the male instances until the male and female
instances are approximately equal, then train on this modified, equalized distribution.
Data Augmentation. The contexts in which males and females are written about can differ; for instance, on Wikipedia women are more often written about with words related to sexuality than men (Graells-Garrido et al., 2015). Data augmentation mitigates these contextual biases by replacing masculine words in a sentence with their corresponding feminine words and vice versa for all sentences in a corpus, and then training on the union of the original and augmented corpora<sup>3</sup> (Zhao et al., 2018; Lu et al., 2018; Dixon et al., 2018; Maudslay et al., 2019; Zhao et al., 2019).
Word Embedding Debiasing Word embeddings can encode gender biases (Bolukbasi et al., 2016; Caliskan et al., 2017; Garg et al., 2018) and this can affect bias in downstream predictions for models using the embeddings (Zhao et al., 2018; Font and Costa-Jussa, 2019). In this work, we apply the Hard-Debiasing technique (Bolukbasi et al., 2016). We applied Hard-Debiasing to Word2Vec embeddings (Mikolov et al., 2013), which we trained on the sentences in WikiGenderBias. When used in conjunction with data augmentation, the embeddings are re-trained on the union of the two corpora. Below, we give metrics used for measuring model performance and bias in our experiments.
3We use the following list to perform data augmentation: https://github.com/uclanlp/coreBias/blob/master/WinoBias/wino/generalized(swaps.txt
Figure 4: Bias in relation extraction model on each relation as measured by male - female F1 score gender gap (used to calculate disparity score) for the default training set without modifications (left) and equalized training set (right). This is evaluated on the model with No Debiasing and two bias mitigation methods: debiased embeddings and data augmentation. The experiment is run five times. We give the mean values and standard error bars.
#	Equalization	Debiased Embeddings	Data Aug.	EoO ↓	PPS Score ↑	Macro F1 Score ↑	Disparity Score ↓
1	.012	.870	.886	.016
2	✓	-.011	.851	.860	0.010
3	✓	.015	.886	.902	.016
4	✓	.014	.841	.866	.026
5	✓	✓	.001	.863	.872	.009
6	✓	✓	-.024	.805	.835	.030
7	✓	✓	.018	.868	.891	.023
8	✓	✓	✓	.006	.867	.877	.010
Table 4: PPS Scores when using debiased embeddings and data augmentation with the unequalized, original dataset. We find that using debiased embeddings alone leads to the best PPS score. Other combinations of debiasing parameters lowers either F1 score, disparity score, or both. We bold the best values, which represent the maximum for PPS score and F1 score and minimum for Disparity Score.
5.2 Effectiveness of Bias Mitigation
We note that by downsampling the training instances to equalize the number of male and female datapoints, the difference in performance on male versus female sentences decreases to almost 0 for every relation aside from hypernym (see Figure 4, right). Additionally, the drop in aggregate is performance is relatively small (see Macro F1, Table 4). Given that we down-sampled male instances to create this equalized dataset, training on the equalized data was also more efficient.
We also examined the effect of various debiasing techniques. Table 4 shows the results. Unfortunately, most of these techniques cause a significant performance drop and none of them is effective in reducing the performance gap between genders. Interestingly, debiasing embeddings increased aggregate performance by achieving slightly better F1 performance. As none of these mitigation approaches is effective, their combinations are not effective as well. They either lowering Macro F1
or raising Disparity Score or both.
We further examine the performance of various bias mitigation techniques evaluated in each relation in Figure 4. NRE relies heavily on surface-level cues such as context, the entities, and their positions. Data augmentation might potentially introduce artifacts and biases, causing the NRE system captures unwanted patterns and spurious statistics between contexts.
In our study, we create and publicly release WikiGenderBias: the first dataset aimed at evaluating bias in NRE models. We train NRE models on the WikiGenderBias dataset and test them on gender-separated test sets. We find a difference in F1 scores for the spouse relation between predictions on male sentences and female for the model's predictions. We also examine existing bias mitigation techniques and find that naive data augmentation causes a significant performance drop.
It is an open and difficult research question to build unbiased neural relation extractors. One possibility is that some bias mitigation methods that add noise to the dataset encourage neural relation extraction models to learn spurious correlations and unwanted biases. We encourage future work to dive deeper into this problem.
While these findings will help future work avoid gender biases, this study is preliminary. We only consider binary gender, but future work should consider non-binary genders. Additionally, future work should further probe the source of gender bias in the model's predictions, perhaps by visualizing attention or looking more closely at the model's outputs.
We thank anonymous reviewers for their helpful feedback. This material is based upon work supported in part by the National Science Foundation under IIS Grant 1927554 and Grant 1821415: Scaling the Early Research Scholars Program.
Eugene Agichtein and Luis Gravano. 2000. Snowball: Extracting relations from large plain-text collections. In Proceedings of the Fifth ACM International Conference on Digital Libraries (ACM '00), pages 85-94.
Nguyen Bach and Sameer Badaskar. 2007. A review of relation extraction. Literature review for Language and Statistics II, 2:1-15.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man Is to Computer Programmer As Woman Is to Homemaker? Debiasing Word Embeddings. In *Neural Information Processing Systems* (NIPS '16).
Antoine Bordes, Sumit Chopra, and Jason Weston. 2014. Question answering with subgraph embeddings. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 615-620, Doha, Qatar. Association for Computational Linguistics.
Shikha Bordia and Samuel R. Bowman. 2019. Identifying and reducing gender bias in word-level language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 7-15, Minneapolis, Minnesota. Association for Computational Linguistics.
Sergey Brin. 1998. Extracting Patterns and Relations from the World Wide Web. In International Work-
shop on The World Wide Web and Databases at EDBT '98, pages 172-183. Springer.
Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics Derived Automatically from Language Corpora Contain Human-Like Biases. Science, 356(6334):183-186.
Kate Crawford. 2017. The Trouble With Bias. Keynote at Neural Information Processing Systems (NIPS'17).
Wanyun Cui, Yanghua Xiao, Haixun Wang, Yangqiu Song, Seung-won Hwang, and Wei Wang. 2019. Kbqa: Learning question answering over qa corpora and knowledge bases.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnamaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in Bios: A Case Study of Semantic Representation Bias in a High-stakes Setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 120-128. ACM.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and Mitigating Unintended Bias in Text Classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (AAAI'18), pages 67-73. ACM.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness Through Awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pages 214-226. ACM.
Joel Escudé Font and Marta R. Costa-jussà. 2019. Equalizing gender bias in neural machine translation with word embeddings techniques. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 147-154, Florence, Italy. Association for Computational Linguistics.
Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S Weld. 2008. Open Information Extraction from the Web. 51(12):68-74.
Oren Etzioni, Michael Cafarella, Doug Downey, Ana Maria Popescu, Tal Shaked, Stephen Soderland, Daniel S Weld, and Alexander Yates. 2005. Unsupervised Named-Entity Extraction from the Web: An Experimental Study. Artificial Intelligence, 165(1):91-134.
Jun Feng, Minlie Huang, Li Zhao, Yang Yang, and Xiaoyan Zhu. 2018. Reinforcement Learning for Relation Classification from Noisy Data. In Thirty-Second Conference on Advancement of Artificial Intelligence (AAAI '18).
Joseph L Fleiss. 1971. Measuring Nominal Scale Agreement Among Many Raters. Psychological bulletin, 76(5):378.
Joel Escudé Font and Marta R Costa-Jussa. 2019. Equalizing gender biases in neural machine translation with word embeddings techniques. arXiv preprint arXiv:1901.03116.
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635-E3644.
Eduardo Graells-Garrido, Mounia Lalmas, and Filippo Menczer. 2015. First Women, Second sex: Gender Bias in Wikipedia. In Proceedings of the 26th ACM Conference (ACM '15), pages 165-174. ACM.
Pankaj Gupta, Subburam Rajaram, Hinrich Schütze, and Thomas Runkler. 2019. Neural Relation Extraction Within and Across Sentence Boundaries. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI '19), volume 33, pages 6513-6520.
Xu Han, Tianyu Gao, Yuan Yao, Demin Ye, Zhiyuan Liu, and Maosong Sun. 2019. OpenNRE: An Open and Extensible Toolkit for Neural Relation Extraction. arXiv preprint arXiv:1909.13078.
Moritz Hardt, Eric Price, and Srebro. 2016. Equality of Opportunity in Supervised Learning. In Advances in Neural Information Processing Systems (NIPS '16), pages 3315-3323.
Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011. Knowledge-based Weak Supervision for Information Extraction of Overlapping Relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL '11), pages 541-550. Association for Computational Linguistics.
Nanda Kambhatla. 2004. Combining Lexical, Syntactic, and Semantic Features with Maximum Entropy Models for Extracting Relations. In Proceedings of the ACL 2004 on Interactive Poster and Demonstration Sessions, pages 22-es.
Svetlana Kiritchenko and Saif M Mohammad. 2018. Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. arXiv preprint arXiv:1805.04508.
Jun Kuang, Yixin Cao, Jianbing Zheng, Xiangnan He, Ming Gao, and Aoying Zhou. 2019. Improving Neural Relation Extraction with Implicit Mutual Relations. arXiv preprint arXiv:1907.05333.
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural Relation Extraction with Selective Attention Over Instances. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL '16), volume 1, pages 2124-2133.
Tianyu Liu, Kexiang Wang, Baobao Chang, and Zhifang Sui. 2017. A Soft-Label Method for Noise-Tolerant Distantly Supervised Relation Extraction.
In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP '17), pages 1790-1795.
Huma Lodhi, Craig Saunders, John Shawe-Taylor, Nello Cristianini, and Chris Watkins. 2002. Text Classification using String Kernels. Journal of Machine Learning Research, 2(Feb):419-444.
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2018. Gender Bias in Neural Natural Language Processing.
Rowan Hall Maudslay, Hila Gonen, Ryan Coterell, and Simone Teufel. 2019. It's All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution. arXiv preprint arXiv:1909.00871.
Pablo N. Mendes, Max Jakob, and Christian Bizer. 2012. Dbpedia for nlp: A multilingual cross-domain knowledge base. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12), Istanbul, Turkey.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and Their Compositionality. In Advances in Neural Information Processing Systems (NIPS '13), pages 3111-3119.
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant Supervision for Relation Extraction Without Labeled Data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the Association of Computational Linguistics (ACL '09), pages 1003-1011. Association for Computational Linguistics.
Patrick Pantel and Marco Pennacchiotti. 2006. *Espresso: Leveraging generic patterns for automatically harvesting semantic relations*. In *Proceedings of the 21st International Conference on Computational Linguistics* and 44th Annual Meeting of the Association for Computational Linguistics, pages 113-120, Sydney, Australia. Association for Computational Linguistics.
Pengda Qin, Weiran Xu, and William Yang Wang. 2018. Robust distant supervision relation extraction via deep reinforcement learning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2137-2147, Melbourne, Australia. Association for Computational Linguistics.
Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M Marlin. 2013. Relation Extraction with Matrix Factorization and Universal Schemas. In booktitle=North American Chapter of the Association for Computational Linguistics (NAACL '13), pages 74-84.
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra
Chouldechova, Sahin Geyik, Krishnamaram Kenthapadi, Anna Rumshisky, and Adam Tauman Kalai. 2019. What's in a Name? Reducing Bias in Bios without Access to Protected Attributes. arXiv preprint arXiv:1904.05233.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender Bias in Coreference Resolution. In North American Chapter of the Association for Computational Linguistics (NAACL '18).
Evan Sandhaus. 2018. The New York Times Annotated Corpus. Linguistic Data Consortium (LDC '08).
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3407-3412, Hong Kong, China. Association for Computational Linguistics.
Pero Subasic, Hongfeng Yin, and Xiao Lin. 2019. Building Knowledge Base through Deep Learning Relation Extraction and Wikidata. In AAAI Spring Symposium: Combining Machine Learning with Knowledge Engineering.
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating Gender Bias in Natural Language Processing: Literature Review.
Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher D Manning. 2012. Multi-instance Multi-label Learning for Relation Extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing (EMNLP '12), pages 455-465. Association for Computational Linguistics.
Bayu Distiawan Trisedya, Gerhard Weikum, Jianzhong Qi, and Rui Zhang. 2019. Neural Relation Extraction for Knowledge Base Enrichment. In Association for Computational Linguistics (ACL '19). ACL.
Shikhar Vashishth, Rishabh Joshi, Sai Suman Prayaga, Chiranjib Bhattacharyya, and Partha Talukdar. 2018. Reside: Improving Distantly-Supervised Neural Relation Extraction Using Side Information.
Claudia Wagner, David Garcia, Mohsen Jadidi, and Markus Strohmaier. 2015. It's a Man's Wikipedia? Assessing Gender Inequality in an Online Encyclopedia. In Ninth International AAAI Conference on Web and Social Media (AAAI'15).
Yi Wu, David Bamman, and Stuart Russell. 2017. Adversarial Training for Relation Extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP '17), pages 1778-1783.
Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, and Xiaoming Li. 2016. Neural generative question answering. In Proceedings of the Workshop on Human-Computer Question Answering, pages 36-42, San Diego, California. Association for Computational Linguistics.
Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2003. Kernel Methods for Relation Extraction. Journal of Machine Learning Research, 3(Feb):1083-1106.
Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015. Distant Supervision for Relation Extraction Via Piecewise Convolutional Neural Networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP '15), pages 1753-1762.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender Bias in Contextualized Word Embeddings. arXiv preprint arXiv:1904.03310.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men Also Like Shopping: Reducing Gender Bias Amplification Using Corpus-Level Constraints. In Empirical Methods of Natural Language Processing (EMNLP '17).
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. In North American Chapter of the Association for Computational Linguistics (NAACL '18).
Shubin Zhao and Ralph Grishman. 2005. Extracting relations with integrated information using kernel methods. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 419-426, Ann Arbor, Michigan. Association for Computational Linguistics.
GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005. Exploring various knowledge in relation extraction. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 427-434, Ann Arbor, Michigan. Association for Computational Linguistics.