SlowGuess/ACL_2020_OCR huggingface dataset:
More specifically, we recommend that work analyzing "bias" in NLP systems be reoriented around the following question: How are social hierarchies, language ideologies, and NLP systems coproduced? This question mirrors Benjamin's (2020) call to examine how "race and technology are coproduced"—i.e., how racial hierarchies, and the ideologies and discourses that maintain them, create and are re-created by technology. We recommend that researchers and practitioners similarly ask how existing social hierarchies and language ideologies drive the development and deployment of NLP systems, and how these systems therefore reproduce these hierarchies and ideologies. As a starting point for reorienting work analyzing "bias" in NLP systems around this question, we
provide the following concrete research questions:
How do social hierarchies and language ideologies influence the decisions made during the development and deployment lifecycle? What kinds of NLP systems do these decisions result in, and what kinds do they foreclose?
$\diamond$  General assumptions: To which linguistic norms do NLP systems adhere (Bender, 2019; Ruane et al., 2019)? Which language practices are implicitly assumed to be standard, ordinary, correct, or appropriate?
$\diamond$  Task definition: For which speakers are NLP systems (and NLP resources) developed? (See Joshi et al. (2020) for a discussion.) How do task definitions discretize the world? For example, how are social groups delineated when defining demographic attribute prediction tasks (e.g., Koppel et al., 2002; Rosenthal and McKeown, 2011; Nguyen et al., 2013)? What about languages in native language prediction tasks (Tetreault et al., 2013)?
Data: How are datasets collected, preprocessed, and labeled or annotated? What are the impacts of annotation guidelines, annotator assumptions and perceptions (Olteanu et al., 2019; Sap et al., 2019; Geiger et al., 2020), and annotation aggregation processes (Pavlick and Kwiatkowski, 2019)?
$\diamond$  Evaluation: How are NLP systems evaluated? What are the impacts of evaluation metrics (Olteanu et al., 2017)? Are any non-quantitative evaluations performed?
How do NLP systems reproduce or transform language ideologies? Which language varieties or practices come to be deemed good or bad? Might "good" language simply mean language that is easily handled by existing NLP systems? For example, linguistic phenomena arising from many language practices (Eisenstein, 2013) are described as "noisy text" and often viewed as a target for "normalization." How do the language ideologies that are reproduced by NLP systems maintain social hierarchies?
$\triangleright$  Which representational harms are being measured or mitigated? Are these the most normatively concerning harms, or merely those that are well handled by existing algorithmic fairness techniques? Are there other representational harms that might be analyzed?
3Language ideologies encompass much more than this; see, e.g., Lippi-Green (2012), Alim et al. (2016), Rosa and Flores (2017), Rosa and Burdick (2017), and Charity Hudley (2017).
4.2 Conceptualizations of "bias"
Turning now to (R2), we argue that work analyzing "bias" in NLP systems should provide explicit statements of why the system behaviors that are described as "bias" are harmful, in what ways, and to whom, as well as the normative reasoning underlying these statements. In other words, researchers and practitioners should articulate their conceptualizations of "bias." As we described above, papers often contain descriptions of system behaviors that are understood to be self-evident statements of "bias." This use of imprecise terminology has led to papers all claiming to analyze "bias" in NLP systems, sometimes even in systems developed for the same task, but with different or even inconsistent conceptualizations of "bias," and no explanations for these differences.
Yet analyzing "bias" is an inherently normative process—in which some system behaviors are deemed good and others harmful—even if assumptions about what kinds of system behaviors are harmful, in what ways, for whom, and why are not stated. We therefore echo calls by Bardzell and Bardzell (2011), Keyes et al. (2019), and Green (2019) for researchers and practitioners to make their normative reasoning explicit by articulating the social values that underpin their decisions to deem some system behaviors as harmful, no matter how obvious such values appear to be. We further argue that this reasoning should take into account the relationships between language and social hierarchies that we described above. First, these relationships provide a foundation from which to approach the normative reasoning that we recommend making explicit. For example, some system behaviors might be harmful precisely because they maintain social hierarchies. Second, if work analyzing "bias" in NLP systems is reoriented to understand how social hierarchies, language ideologies, and NLP systems are coproduced, then this work will be incomplete if we fail to account for the ways that social hierarchies and language ideologies determine what we mean by "bias" in the first place. As a starting point, we therefore provide the following concrete research questions:
What kinds of system behaviors are described as "bias"? What are their potential sources (e.g., general assumptions, task definition, data)?
In what ways are these system behaviors harmful, to whom are they harmful, and why?
$\triangleright$  What are the social values (obvious or not) that
underpin this conceptualization of "bias?"
4.3 Language use in practice
Finally, we turn to (R3). Our perspective, which rests on a greater recognition of the relationships between language and social hierarchies, suggests several directions for examining language use in practice. Here, we focus on two. First, because language is necessarily situated, and because different social groups have different lived experiences due to their different social positions (Hanna et al., 2020)—particularly groups at the intersections of multiple axes of oppression—we recommend that researchers and practitioners center work analyzing “bias” in NLP systems around the lived experiences of members of communities affected by these systems. Second, we recommend that the power relations between technologists and such communities be interrogated and reimagined. Researchers have pointed out that algorithmic fairness techniques, by proposing incremental technical mitigations—e.g., collecting new datasets or training better models—maintain these power relations by (a) assuming that automated systems should continue to exist, rather than asking whether they should be built at all, and (b) keeping development and deployment decisions in the hands of technologists (Bennett and Keyes, 2019; Cifor et al., 2019; Green, 2019; Katell et al., 2020).
There are many disciplines for researchers and practitioners to draw on when pursuing these directions. For example, in human-computer interaction, Hamidi et al. (2018) study transgender people's experiences with automated gender recognition systems in order to uncover how these systems reproduce structures of transgender exclusion by redefining what it means to perform gender "normally." Value-sensitive design provides a framework for accounting for the values of different stakeholders in the design of technology (e.g., Friedman et al., 2006; Friedman and Hendry, 2019; Le Dantec et al., 2009; Yoo et al., 2019), while participatory design seeks to involve stakeholders in the design process itself (Sanders, 2002; Muller, 2007; Simonsen and Robertson, 2013; DiSalvo et al., 2013). Participatory action research in education (Kemmis, 2006) and in language documentation and reclamation (Junker, 2018) is also relevant. In particular, work on language reclamation to support decolonization and tribal sovereignty (Leonard, 2012) and work in sociolinguistics focus
ing on developing co-equal research relationships with community members and supporting linguistic justice efforts (e.g., Bucholtz et al., 2014, 2016, 2019) provide examples of more emancipatory relationships with communities. Finally, several workshops and events have begun to explore how to empower stakeholders in the development and deployment of technology (Vaccaro et al., 2019; Givens and Morris, 2020; Sassaman et al., 2020) $^{4}$  and how to help researchers and practitioners consider when not to build systems at all (Barocas et al., 2020).
As a starting point for engaging with communities affected by NLP systems, we therefore provide the following concrete research questions:
How do communities become aware of NLP systems? Do they resist them, and if so, how?
$\triangleright$  What additional costs are borne by communities for whom NLP systems do not work well?
Do NLP systems shift power toward oppressive institutions (e.g., by enabling predictions that communities do not want made, linguistically based unfair allocation of resources or opportunities (Rosa and Flores, 2017), surveillance, or censorship), or away from such institutions?
Who is involved in the development and deployment of NLP systems? How do decision-making processes maintain power relations between technologists and communities affected by NLP systems? Can these processes be changed to reimagine these relations?
5 Case study
To illustrate our recommendations, we present a case study covering work on African-American English (AAE). Work analyzing "bias" in the context of AAE has shown that part-of-speech taggers, language identification systems, and dependency parsers all work less well on text containing features associated with AAE than on text without these features (Jorgensen et al., 2015, 2016; Blodgett et al., 2016, 2018), and that toxicity detection systems score tweets containing features associated with AAE as more offensive than tweets without them (Davidson et al., 2019; Sap et al., 2019).
These papers have been critical for highlighting AAE as a language variety for which existing NLP
systems may not work, illustrating their limitations. However, they do not conceptualize "racial bias" in the same way. The first four of these papers simply focus on system performance differences between text containing features associated with AAE and text without these features. In contrast, the last two papers also focus on such system performance differences, but motivate this focus with the following additional reasoning: If tweets containing features associated with AAE are scored as more offensive than tweets without these features, then this might (a) yield negative perceptions of AAE; (b) result in disproportionate removal of tweets containing these features, impeding participation in online platforms and reducing the space available online in which speakers can use AAE freely; and (c) cause AAE speakers to incur additional costs if they have to change their language practices to avoid negative perceptions or tweet removal.
More importantly, none of these papers engage with the literature on AAE, racial hierarchies in the U.S., and raciolinguistic ideologies. By failing to engage with this literature—thereby treating AAE simply as one of many non-Penn Treebank varieties of English or perhaps as another challenging domain—work analyzing “bias” in NLP systems in the context of AAE fails to situate these systems in the world. Who are the speakers of AAE? How are they viewed? We argue that AAE as a language variety cannot be separated from its speakers—primarily Black people in the U.S., who experience systemic anti-Black racism—and the language ideologies that reinforce and justify racial hierarchies.
Even after decades of sociolinguistic efforts to legitimize AAE, it continues to be viewed as "bad" English and its speakers continue to be viewed as linguistically inadequate—a view called the deficit perspective (Alim et al., 2016; Rosa and Flores, 2017). This perspective persists despite demonstrations that AAE is rule-bound and grammatical (Mufwene et al., 1998; Green, 2002), in addition to ample evidence of its speakers' linguistic adroitness (e.g., Alim, 2004; Rickford and King, 2016). This perspective belongs to a broader set of raciolinguistic ideologies (Rosa and Flores, 2017), which also produce allocational harms; speakers of AAE are frequently penalized for not adhering to dominant language practices, including in the education system (Alim, 2004; Terry et al., 2010), when seeking housing (Baugh, 2018), and in the judicial system, where their testimony is misunderstood or,
4Also https://participatoryml.github.io/
5This language variety has had many different names over the years, but is now generally called African-American English (AAE), African-American Vernacular English (AAVE), or African-American Language (AAL) (Green, 2002; Wolfram and Schilling, 2015; Rickford and King, 2016).
worse yet, disbelieved (Rickford and King, 2016; Jones et al., 2019). These raciolinguistic ideologies position racialized communities as needing linguistic intervention, such as language education programs, in which these and other harms can be reduced if communities accommodate to dominant language practices (Rosa and Flores, 2017).
In the technology industry, speakers of AAE are often not considered consumers who matter. For example, Benjamin (2019) recounts an Apple employee who worked on speech recognition for Siri:
"As they worked on different English dialects —
Australian, Singaporean, and Indian English — [the
employee] asked his boss: 'What about African
American English?' To this his boss responded:
Well, Apple products are for the premium market."
The reality, of course, is that speakers of AAE tend not to represent the "premium market" precisely because of institutions and policies that help to maintain racial hierarchies by systematically denying them the opportunities to develop wealth that are available to white Americans (Rothstein, 2017)—an exclusion that is reproduced in technology by countless decisions like the one described above.
Engaging with the literature outlined above situates the system behaviors that are described as "bias," providing a foundation for normative reasoning. Researchers and practitioners should be concerned about "racial bias" in toxicity detection systems not only because performance differences impair system performance, but because they reproduce longstanding injustices of stigmatization and disenfranchisement for speakers of AAE. In re-stigmatizing AAE, they reproduce language ideologies in which AAE is viewed as ungrammatical, uneducated, and offensive. These ideologies, in turn, enable linguistic discrimination and justify enduring racial hierarchies (Rosa and Flores, 2017). Our perspective, which understands racial hierarchies and raciolinguistic ideologies as structural conditions that govern the development and deployment of technology, implies that techniques for measuring or mitigating "bias" in NLP systems will necessarily be incomplete unless they interrogate and dismantle these structural conditions, including the power relations between technologists and racialized communities.
We emphasize that engaging with the literature on AAE, racial hierarchies in the U.S., and raciolinguistic ideologies can generate new lines of engagement. These lines include work on the ways that the decisions made during the development
and deployment of NLP systems produce stigmatization and disenfranchisement, and work on AAE use in practice, such as the ways that speakers of AAE interact with NLP systems that were not designed for them. This literature can also help researchers and practitioners address the allocational harms that may be produced by NLP systems, and ensure that even well-intentioned NLP systems do not position racialized communities as needing linguistic intervention or accommodation to dominant language practices. Finally, researchers and practitioners wishing to design better systems can also draw on a growing body of work on anti-racist language pedagogy that challenges the deficit perspective of AAE and other racialized language practices (e.g. Flores and Chaparro, 2018; Baker-Bell, 2019; Martínez and Mejía, 2019), as well as the work that we described in section 4.3 on reimagining the power relations between technologists and communities affected by technology.
By surveying 146 papers analyzing "bias" in NLP systems, we found that (a) their motivations are often vague, inconsistent, and lacking in normative reasoning; and (b) their proposed quantitative techniques for measuring or mitigating "bias" are poorly matched to their motivations and do not engage with the relevant literature outside of NLP. To help researchers and practitioners avoid these pitfalls, we proposed three recommendations that should guide work analyzing "bias" in NLP systems, and, for each, provided several concrete research questions. These recommendations rest on a greater recognition of the relationships between language and social hierarchies—a step that we see as paramount to establishing a path forward.
This paper is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. 1451512. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. We thank the reviewers for their useful feedback, especially the suggestion to include additional details about our method.
Artem Abzaliev. 2019. On GAP coreference resolution shared task: insights from the 3rd place solution. In Proceedings of the Workshop on Gender Bias in Natural Language Processing, pages 107-112, Florence, Italy.
ADA. 2018. Guidelines for Writing About People With Disabilities. ADA National Network. https://bit.ly/2KREbkB.
Oshin Agarwal, Funda Durupinar, Norman I. Badler, and Ani Nenkova. 2019. Word embeddings (also) encode human personality stereotypes. In Proceedings of the Joint Conference on Lexical and Computational Semantics, pages 205-211, Minneapolis, MN.
H. Samy Alim. 2004. You Know My Steez: An Ethnographic and Sociolinguistic Study of Styleshifting in a Black American Speech Community. American Dialect Society.
H. Samy Alim, John R. Rickford, and Arnetha F. Ball, editors. 2016. Racial linguistics: How Language Shapes Our Ideas About Race. Oxford University Press.
Sandeep Attree. 2019. Gendered ambiguous pronouns shared task: Boosting model confidence by evidence pooling. In Proceedings of the Workshop on Gender Bias in Natural Language Processing, Florence, Italy.
Pinkesh Badjatiya, Manish Gupta, and Vasudeva Varma. 2019. Stereotypical bias removal for hate speech detection task using knowledge-based generalizations. In Proceedings of the International World Wide Web Conference, pages 49-59, San Francisco, CA.
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. 2019. Differential Privacy Has Disparate Impact on Model Accuracy. In Proceedings of the Conference on Neural Information Processing Systems, Vancouver, Canada.
April Baker-Bell. 2019. Dismantling anti-black linguistic racism in English language arts classrooms: Toward an anti-racist black language pedagogy. Theory Into Practice.
David Bamman, Sejal Popat, and Sheng Shen. 2019. An annotated dataset of literary entities. In Proceedings of the North American Association for Computational Linguistics (NAACL), pages 2138-2144, Minneapolis, MN.
Xingce Bao and Qianqian Qiao. 2019. Transfer Learning from Pre-trained BERT for Pronoun Resolution. In Proceedings of the Workshop on Gender Bias in Natural Language Processing, pages 82-88, Florence, Italy.
Shaowen Bardzell and Jeffrey Bardzell. 2011. Towards a Feminist HCI Methodology: Social Science, Feminism, and HCI. In Proceedings of the Conference on Human Factors in Computing Systems (CHI), pages 675-684, Vancouver, Canada.
Solon Barocas, Asia J. Biega, Benjamin Fish, Jedrzej Niklas, and Luke Stark. 2020. When Not to Design, Build, or Deploy. In Proceedings of the Conference on Fairness, Accountability, and Transparency, Barcelona, Spain.
Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The Problem With Bias: Allocative Versus Representational Harms in Machine Learning. In Proceedings of SIGCIS, Philadelphia, PA.
Christine Basta, Marta R. Costa-jussà, and Noe Casas. 2019. Evaluating the underlying gender bias in contextualized word embeddings. In Proceedings of the Workshop on Gender Bias for Natural Language Processing, pages 33-39, Florence, Italy.
John Baugh. 2018. Linguistics in Pursuit of Justice. Cambridge University Press.
Emily M. Bender. 2019. A typology of ethical risks in language technology with an eye towards where transparent documentation can help. Presented at The Future of Artificial Intelligence: Language, Ethics, Technology Workshop. https://bit.ly/2P9t9M6.
Ruha Benjamin. 2019. *Race After Technology: Abolitionist Tools for the New Jim Code*. John Wiley & Sons.
Ruha Benjamin. 2020. 2020 Vision: Reimagining the Default Settings of Technology & Society. Keynote at ICLR.
Cynthia L. Bennett and Os Keyes. 2019. What is the Point of Fairness? Disability, AI, and The Complexity of Justice. In Proceedings of the ASSETS Workshop on AI Fairness for People with Disabilities, Pittsburgh, PA.
Camiel J. Beukeboom and Christian Burgers. 2019. How Stereotypes Are Shared Through Language: A Review and Introduction of the Social Categories and Stereotypes Communication (SCSC) Framework. Review of Communication Research, 7:1-37.
Shruti Bhargava and David Forsyth. 2019. Exposing and Correcting the Gender Bias in Image Captioning Datasets and Models. arXiv preprint arXiv:1912.00578.
Jayadev Bhaskaran and Isha Bhallamudi. 2019. Good Secretaries, Bad Truck Drivers? Occupational Gender Stereotypes in Sentiment Analysis. In Proceedings of the Workshop on Gender Bias in Natural Language Processing, pages 62-68, Florence, Italy.