SlowGuess/ACL_2020_OCR huggingface dataset:
Ninareh Mehrabi, Thamme Gowda, Fred Morstatter, Nanyun Peng, and Aram Galstyan. 2019. Man is to Person as Woman is to Location: Measuring Gender Bias in Named Entity Recognition. arXiv preprint arXiv:1910.10872.
Michela Menegatti and Monica Rubini. 2017. Gender bias and sexism in language. In Oxford Research Encyclopedia of Communication. Oxford University Press.
Inom Mirzaev, Anthony Schulte, Michael Conover, and Sam Shah. 2019. Considerations for the interpretation of bias measures of word embeddings. arXiv preprint arXiv:1906.08379.
Salikoko S. Mufwene, Guy Bailey, and John R. Rickford, editors. 1998. African-American English: Structure, History, and Use. Routledge.
Michael J. Muller. 2007. Participatory Design: The Third Space in HCI. In *The Human-Computer Interaction Handbook*, pages 1087–1108. CRC Press.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. StereoSet: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456.
Dong Nguyen, Rilana Gravel, Dolf Trieschnigg, and Theo Meder. 2013. "How Old Do You Think I Am?: A Study of Language and Age in Twitter. In Proceedings of the Conference on Web and Social Media (ICWSM), pages 439-448, Boston, MA.
Malvina Nissim, Rik van Noord, and Rob van der Goot. 2020. Fair is better than sensational: Man is to doctor as woman is to doctor. Computational Linguistics.
Debora Nozza, Claudia Volpetti, and Elisabetta Fersini. 2019. Unintended Bias in Misogyny Detection. In Proceedings of the Conference on Web Intelligence, pages 149-155.
Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Emre Kiciman. 2019. Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries. Frontiers in Big Data, 2.
Alexandra Olteanu, Kartik Talamadupula, and Kush R. Varshney. 2017. The Limits of Abstract Evaluation Metrics: The Case of Hate Speech Detection. In Proceedings of the ACM Web Science Conference, Troy, NY.
Orestis Papakyriakopoulos, Simon Hegelich, Juan Carlos Medina Serrano, and Fabienne Marco. 2020. Bias in word embeddings. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 446-457, Barcelona, Spain.
Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing Gender Bias in Abusive Language Detection. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 2799-2804, Brussels, Belgium.
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics, 7:677-694.
Xiangyu Peng, Siyan Li, Spencer Frazier, and Mark Riedl. 2020. Fine-Tuning a Transformer-Based Language Model to Avoid Generating Non-Normative Text. arXiv preprint arXiv:2001.08764.
Radomir Popovic, Florian Lemmerich, and Markus Strohmaier. 2020. Joint Multiclass Debiasing of Word Embeddings. In Proceedings of the International Symposium on Intelligent Systems, Graz, Austria.
Vinodkumar Prabhakaran, Ben Hutchinson, and Margaret Mitchell. 2019. Perturbation Sensitivity Analysis to Detect Unintended Model Biases. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 5744-5749, Hong Kong, China.
Shrimai Prabhumoye, Elijah Mayfield, and Alan W. Black. 2019. Principled Frameworks for Evaluating Ethics in NLP Systems. In Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications, Florence, Italy.
Marcelo Prates, Pedro Avelar, and Luis C. Lamb. 2019. Assessing gender bias in machine translation: A case study with google translate. *Neural Computing and Applications*.
Rasmus Precenth. 2019. Word embeddings and gender stereotypes in Swedish and English. Master's thesis, Uppsala University.
Dennis R. Preston. 2009. Are you really smart (or stupid, or cute, or ugly, or cool)? Or do you just talk that way? Language attitudes, standardization and language change. Oslo: Novus forlag, pages 105-129.
Flavien Prost, Nithum Thain, and Tolga Bolukbasi. 2019. Debiasing Embeddings for Reduced Gender Bias in Text Classification. In Proceedings of the Workshop on Gender Bias in Natural Language Processing, pages 69-75, Florence, Italy.
Reid Pryzant, Richard Diehl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang. 2020. Automatically Neutralizing Subjective Bias in Text. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), New York, NY.
Arun K. Pujari, Ansh Mittal, Anshuman Padhi, Anshul Jain, Mukesh Jadon, and Vikas Kumar. 2019. Debiasing Gender biased Hindi Words with Word-embedding. In Proceedings of the International Conference on Algorithms, Computing and Artificial Intelligence, pages 450-456.
Yusu Qian, Urwa Muaz, Ben Zhang, and Jae Won Hyun. 2019. Reducing gender bias in word-level
language models with a gender-equalizing loss function. In Proceedings of the ACL Student Research Workshop, pages 223-228, Florence, Italy.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection. In Proceedings of the Association for Computational Linguistics (ACL).
John R. Rickford and Sharese King. 2016. Language and linguistics on trial: Hearing Rachel Jeantel (and other vernacular speakers) in the courtroom and beyond. *Language*, 92(4):948-988.
Anthony Rios. 2020. FuzzE: Fuzzy Fairness Evaluation of Offensive Language Classifiers on African-American English. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), New York, NY.
Gerald Roche. 2019. Articulating language oppression: colonialism, coloniality and the erasure of TibetáÁZs minority languages. *Patterns of Prejudice*.
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnamaram Kenthapadi, Anna Rumshisky, and Adam Tauman Kalai. 2019. What's in a Name? Reducing Bias in Bios without Access to Protected Attributes. In Proceedings of the North American Association for Computational Linguistics (NAACL), pages 4187-4195, Minneapolis, MN.
Jonathan Rosa. 2019. Contesting Representations of Migrant "Illegality" through the Drop the I-Word Campaign: Rethinking Language Change and Social Change. In Netta Avineri, Laura R. Graham, Eric J. Johnson, Robin Conley Riner, and Jonathan Rosa, editors, Language and Social Justice in Practice. Routledge.
Jonathan Rosa and Christa Burdick. 2017. Language Ideologies. In Ofelia García, Nelson Flores, and Massimiliano Spotti, editors, The Oxford Handbook of Language and Society. Oxford University Press.
Jonathan Rosa and Nelson Flores. 2017. Unsettling race and language: Toward a raciolinguistic perspective. Language in Society, 46:621-647.
Sara Rosenthal and Kathleen McKeown. 2011. Age Prediction in Blogs: A Study of Style, Content, and Online Behavior in Pre- and Post-Social Media Generations. In Proceedings of the North American Association for Computational Linguistics (NAACL), pages 763-772, Portland, OR.
Candace Ross, Boris Katz, and Andrei Barbu. 2020. Measuring Social Biases in Grounded Vision and Language Embeddings. arXiv preprint arXiv:2002.08911.
Richard Rothstein. 2017. The Color of Law: A Forgotten History of How Our Government Segregated America. Liveright Publishing.
David Rozado. 2020. Wide range screening of algorithmic bias in word embedding models using large sentiment lexicons reveals underreported bias types. PLOS One.
Elayne Ruane, Abeba Birhane, and Anthony Ventresque. 2019. Conversational AI: Social and Ethical Considerations. In Proceedings of the Irish Conference on Artificial Intelligence and Cognitive Science, Galway, Ireland.
Rachel Rudinger, Chandler May, and Benjamin Van Durme. 2017. Social bias in elicited natural language inferences. In Proceedings of the Workshop on Ethics in Natural Language Processing, pages 74-79, Valencia, Spain.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender Bias in Coreference Resolution. In Proceedings of the North American Association for Computational Linguistics (NAACL), pages 8-14, New Orleans, LA.
Elizabeth B.N. Sanders. 2002. From user-centered to participatory design approaches. In Jorge Frascara, editor, Design and the Social Sciences: Making Connections, pages 18-25. CRC Press.
Brenda Salenave Santana, Vinicius Woloszyn, and Leandro Krug Wives. 2018. Is there gender bias and stereotype in Portuguese word embeddings? In Proceedings of the International Conference on the Computational Processing of Portuguese Student Research Workshop, Canela, Brazil.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the Association for Computational Linguistics (ACL), pages 1668-1678, Florence, Italy.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social Bias Frames: Reasoning about Social and Power Implications of Language. In Proceedings of the Association for Computational Linguistics (ACL).
Hanna Sassaman, Jennifer Lee, Jenessa Irvine, and Shankar Narayan. 2020. Creating Community-Based Tech Policy: Case Studies, Lessons Learned, and What Technologists and Communities Can Do Together. In Proceedings of the Conference on Fairness, Accountability, and Transparency, Barcelona, Spain.
Danielle Saunders and Bill Byrne. 2020. Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem. In Proceedings of the Association for Computational Linguistics (ACL).
Tyler Schnoebelen. 2017. Goal-Oriented Design for Ethical Machine Learning and NLP. In Proceedings of the Workshop on Ethics in Natural Language Processing, pages 88-93, Valencia, Spain.
Sabine Sczesny, Magda Formanowicz, and Franziska Moser. 2016. Can gender-fair language reduce gender stereotyping and discrimination? Frontiers in Psychology, 7.
João Sedoc and Lyle Ungar. 2019. The Role of Protected Class Word Lists in Bias Identification of Contextualized Word Representations. In Proceedings of the Workshop on Gender Bias in Natural Language Processing, pages 55-61, Florence, Italy.
Procheta Sen and Debasis Ganguly. 2020. Towards Socially Responsible AI: Cognitive Bias-Aware Multi-Objective Learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), New York, NY.
Deven Shah, H. Andrew Schwartz, and Dirk Hovy. 2020. Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview. In Proceedings of the Association for Computational Linguistics (ACL).
Judy Hanwen Shen, Lauren Fratamico, Iyad Rahwan, and Alexander M. Rush. 2018. Darling or Babygirl? Investigating Stylistic Bias in Sentiment Analysis. In Proceedings of the Workshop on Fairness, Accountability, and Transparency (FAT/ML), Stockholm, Sweden.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The Woman Worked as a Babbysitter: On Biases in Language Generation. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 3398-3403, Hong Kong, China.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2020. Towards Controllable Biases in Language Generation. arXiv preprint arXiv:2005.00268.
Seungjae Shin, Kyungwoo Song, JoonHo Jang, Hyemi Kim, Weonyoung Joo, and Il-Chul Moon. 2020. Neutralizing Gender Bias in Word Embedding with Latent Disentanglement and Counterfactual Generation. arXiv preprint arXiv:2004.03133.
Jesper Simonsen and Toni Robertson, editors. 2013. Routledge International Handbook of Participatory Design. Routledge.
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the Association for Computational Linguistics (ACL), pages 1679-1684, Florence, Italy.
Yolande Strengers, Lizhe Qu, Qiongkai Xu, and Jarrod Knibbe. 2020. Adhering, Steering, and Queering: Treatment of Gender in Natural Language Generation. In Proceedings of the Conference on Human Factors in Computing Systems (CHI), Honolulu, HI.
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating Gender Bias in Natural Language Processing: Literature Review. In Proceedings of the Association for Computational Linguistics (ACL), pages 1630-1640, Florence, Italy.
Adam Sutton, Thomas Lansdall-Welfare, and Nello Cristianini. 2018. Biased embeddings from wild data: Measuring, understanding and removing. In Proceedings of the International Symposium on Intelligent Data Analysis, pages 328-339, 's-Hertogenbosch, Netherlands.
Chris Sweeney and Maryam Najafian. 2019. A Transparent Framework for Evaluating Unintended Demographic Bias in Word Embeddings. In Proceedings of the Association for Computational Linguistics (ACL), pages 1662-1667, Florence, Italy.
Chris Sweeney and Maryam Najafian. 2020. Reducing sentiment polarity for demographic attributes in word embeddings using adversarial learning. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 359-368, Barcelona, Spain.
Nathaniel Swinger, Maria De-Arteaga, Neil Thomas Heffernan, Mark D.M. Leiserson, and Adam Tauman Kalai. 2019. What are the biases in my word embedding? In Proceedings of the Conference on Artificial Intelligence, Ethics, and Society (AIES), Honolulu, HI.
Samson Tan, Shafiq Joty, Min-Yen Kan, and Richard Socher. 2020. It's Morphin' Time! Combating Linguistic Discrimination with Inflectional Perturbations. In Proceedings of the Association for Computational Linguistics (ACL).
Yi Chern Tan and L. Elisa Celis. 2019. Assessing Social and Intersectional Biases in Contextualized Word Representations. In Proceedings of the Conference on Neural Information Processing Systems, Vancouver, Canada.
J. Michael Terry, Randall Hendrick, Evangelos Evangelou, and Richard L. Smith. 2010. Variable dialect switching among African American children: Inferences about working memory. *Lingua*, 120(10):2463-2475.
Joel Tetrault, Daniel Blanchard, and Aoife Cahill. 2013. A Report on the First Native Language Identification Shared Task. In Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications, pages 48-57, Atlanta, GA.
Mike Thelwall. 2018. Gender Bias in Sentiment Analysis. Online Information Review, 42(1):45-57.
Kristen Vaccaro, Karrie Karahalios, Deirdre K. Mulligan, Daniel Kluttz, and Tad Hirsch. 2019. Contestability in Algorithmic Systems. In *Conference Companion Publication* of the 2019 on Computer
Supported Cooperative Work and Social Computing, pages 523-527, Austin, TX.
Ameya Vaidya, Feng Mai, and Yue Ning. 2019. Empirical Analysis of Multi-Task Learning for Reducing Model Bias in Toxic Comment Detection. arXiv preprint arXiv:1909.09758v2.
Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting Gender Right in Neural Machine Translation. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 3003-3008, Brussels, Belgium.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias. arXiv preprint arXiv:2004.12265.
Tianlu Wang, Xi Victoria Lin, Nazneen Fatema Rajani, Bryan McCann, Vicente Ordonez, and Caiming Xiong. 2020. Double-Hard Debias: Tailoring Word Embeddings for Gender Bias Mitigation. In Proceedings of the Association for Computational Linguistics (ACL).
Zili Wang. 2019. MSnet: A BERT-based Network for Gendered Pronoun Resolution. In Proceedings of the Workshop on Gender Bias in Natural Language Processing, pages 89-95, Florence, Italy.
Kellie Webster, Marta R. Costa-jussà, Christian Hardmeier, and Will Radford. 2019. Gendered Ambiguous Pronoun (GAP) Shared Task at the Gender Bias in NLP Workshop 2019. In Proceedings of the Workshop on Gender Bias in Natural Language Processing, pages 1-7, Florence, Italy.
Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605-618.
Walt Wolfram and Natalie Schilling. 2015. American English: Dialects and Variation, 3 edition. Wiley Blackwell.
Austin P. Wright, Omar Shaikh, Haekyu Park, Will Epperson, Muhammed Ahmed, Stephane Pinel, Diyi Yang, and Duen Horng (Polo) Chau. 2020. RECAST: Interactive Auditing of Automatic Toxicity Detection Models. In Proceedings of the Conference on Human Factors in Computing Systems (CHI), Honolulu, HI.
Yinchuan Xu and Junlin Yang. 2019. Look again at the syntax: Relational graph convolutional network for gendered ambiguous pronoun resolution. In Proceedings of the Workshop on Gender Bias in Natural Language Processing, pages 96-101, Florence, Italy.
Kai-Chou Yang, Timothy Niven, Tzu-Hsuan Chou, and Hung-Yu Kao. 2019. Fill the GAP: Exploiting BERT for Pronoun Resolution. In Proceedings of the Workshop on Gender Bias in Natural Language Processing, pages 102-106, Florence, Italy.
Zekun Yang and Juan Feng. 2020. A Causal Inference Method for Reducing Gender Bias in Word Embedding Relations. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), New York, NY.
Daisy Yoo, Anya Ernest, Sofia Serholt, Eva Eriksson, and Peter Dalsgaard. 2019. Service Design in HCI Research: The Extended Value Co-creation Model. In Proceedings of the Halfway to the Future Symposium, Nottingham, United Kingdom.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018. Mitigating unwanted biases with adversarial learning. In Proceedings of the Conference on Artificial Intelligence, Ethics, and Society (AIES), New Orleans, LA.
Guanhua Zhang, Bing Bai, Junqi Zhang, Kun Bai, Conghui Zhu, and Tiejun Zhao. 2020a. Demographics Should Not Be the Reason of Toxicity: Mitigating Discrimination in Text Classifications with Instance Weighting. In Proceedings of the Association for Computational Linguistics (ACL).
Haoran Zhang, Amy X. Lu, Mohamed Abdalla, Matthew McDermott, and Marzyeh Ghassemi. 2020b. Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings. In Proceedings of the ACM Conference on Health, Inference, and Learning.
Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini, Kai-Wei Chang, and Ahmed Hassan Awadallah. 2020. Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer. In Proceedings of the Association for Computational Linguistics (ACL).
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterrall, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender Bias in Contextualized Word Embeddings. In Proceedings of the North American Association for Computational Linguistics (NAACL), pages 629-634, Minneapolis, MN.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 2979-2989, Copenhagen, Denmark.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018a. Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. In Proceedings of the North American Association for Computational Linguistics (NAACL), pages 15–20, New Orleans, LA.
Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. 2018b. Learning Gender-Neutral Word Embeddings. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 4847-4853, Brussels, Belgium.
Alina Zhiltsova, Simon Caton, and Catherine Mulwa. 2019. Mitigation of Unintended Biases against Non-Native English Texts in Sentiment Analysis. In Proceedings of the Irish Conference on Artificial Intelligence and Cognitive Science, Galway, Ireland.
Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, and Kai-Wei Chang. 2019. Examining gender bias in languages with grammatical genders. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 5279-5287, Hong Kong, China.
Ran Zmigrod, S. J. Mielke, Hanna Wallach, and Ryan Cotterell. 2019. Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. In Proceedings of the Association for Computational Linguistics (ACL), pages 1651-1661, Florence, Italy.
In Table 3, we provide examples of the papers' motivations and techniques across several NLP tasks.
A.1 Categorization details
In this section, we provide some additional details about our method—specifically, our categorization.
What counts as being covered by an NLP task?
We considered a paper to cover a given NLP task if it analyzed "bias" with respect to that task, but not if it only evaluated overall performance on that task. For example, a paper examining the impact of mitigating "bias" in word embeddings on "bias" in sentiment analysis would be counted as covering both NLP tasks. In contrast, a paper assessing whether performance on sentiment analysis degraded after mitigating "bias" in word embeddings would be counted only as focusing on embeddings.
What counts as a motivation? We considered a motivation to include any description of the problem that motivated the paper or proposed quantitative technique, including any normative reasoning.
We excluded from the "Vague/unstated" category of motivations the papers that participated in the Gendered Ambiguous Pronoun (GAP) Shared Task at the First ACL Workshop on Gender Bias in NLP. In an ideal world, shared task papers would engage with "bias" more critically, but given the nature of shared tasks it is understandable that they
do not. As a result, we excluded them from our counts for techniques as well. We cite the papers here; most propose techniques we would have categorized as "Questionable correlations," with a few as "Other representational harms" (Abzaliev, 2019; Attree, 2019; Bao and Qiao, 2019; Chada, 2019; Ionita et al., 2019; Liu, 2019; Lois et al., 2019; Wang, 2019; Xu and Yang, 2019; Yang et al., 2019).
We excluded Dabas et al. (2020) from our survey because we could not determine what this paper's user study on fairness was actually measuring.
Finally, we actually categorized the motivation for Liu et al. (2019) (i.e., the last row in Table 3) as "Questionable correlations" due to a sentence elsewhere in the paper; had the paragraph we quoted been presented without more detail, we would have categorized the motivation as "Vague/unstated."
A.2 Full categorization: Motivations
Allocational harms Hovy and Spruit (2016); Caliskan et al. (2017); Madnani et al. (2017); Dixon et al. (2018); Kiritchenko and Mohammad (2018); Shen et al. (2018); Zhao et al. (2018b); Bhaskaran and Bhallamudi (2019); Bordia and Bowman (2019); Brunet et al. (2019); Chaloner and Maldonado (2019); De-Arteaga et al. (2019); Dev and Phillips (2019); Font and Costa-jussa (2019); James-Sorenson and Alvarez-Melis (2019); Kurita et al. (2019); Mayfield et al. (2019); Pujari et al. (2019); Romanov et al. (2019); Ruane et al. (2019); Sedoc and Ungar (2019); Sun et al. (2019); Zmigrod et al. (2019); Hutchinson et al. (2020); Papakyriakopoulos et al. (2020); Ravfogel et al. (2020); Strengers et al. (2020); Sweeney and Najafian (2020); Tan et al. (2020); Zhang et al. (2020b).
Stereotyping Bolukbasi et al. (2016a,b); Caliskan et al. (2017); McCurdy and Serbetci (2017); Rudinger et al. (2017); Zhao et al. (2017); Curry and Rieser (2018); Diaz et al. (2018); Santana et al. (2018); Sutton et al. (2018); Zhao et al. (2018a,b); Agarwal et al. (2019); Basta et al. (2019); Bhaskaran and Bhallamudi (2019); Cordia and Bowman (2019); Brunet et al. (2019); Cao and Daumé (2019); Chaloner and Maldonado (2019); Cho et al. (2019); Dev and Phillips (2019); Font and Costa-jussà (2019); Gonen and Goldberg (2019); James-Sorenson and Alvarez-Melis (2019); Kaneko and Bollegala (2019); Karve et al. (2019); Kurita et al. (2019); Lauscher and Glavaš (2019); Lee et al. (2019); Manzini et al. (2019); Mayfield
Table 3: Examples of the categories into which the papers' motivations and proposed quantitative techniques for measuring or mitigating "bias" fall. Bold text in the quotes denotes the content that yields our categorizations.