SlowGuess/ACL_2020_OCR huggingface dataset:
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett
blodgett@cs.umass.edu
Solon Barocas
solon@microsoft.com
Hanna Wallach
wallach@microsoft.com
We survey 146 papers analyzing "bias" in NLP systems, finding that their motivations are often vague, inconsistent, and lacking in normative reasoning, despite the fact that analyzing "bias" is an inherently normative process. We further find that these papers' proposed quantitative techniques for measuring or mitigating "bias" are poorly matched to their motivations and do not engage with the relevant literature outside of NLP. Based on these findings, we describe the beginnings of a path forward by proposing three recommendations that should guide work analyzing "bias" in NLP systems. These recommendations rest on a greater recognition of the relationships between language and social hierarchies, encouraging researchers and practitioners to articulate their conceptualizations of "bias"—i.e., what kinds of system behaviors are harmful, in what ways, to whom, and why, as well as the normative reasoning underlying these statements—and to center work around the lived experiences of members of communities affected by NLP systems, while interrogating and reimagining the power relations between technologists and such communities.
A large body of work analyzing "bias" in natural language processing (NLP) systems has emerged in recent years, including work on "bias" in embedding spaces (e.g., Bolukbasi et al., 2016a; Caliskan et al., 2017; Gonen and Goldberg, 2019; May et al., 2019) as well as work on "bias" in systems developed for a breadth of tasks including language modeling (Lu et al., 2018; Cordia and Bowman,
2019), coreference resolution (Rudinger et al., 2018; Zhao et al., 2018a), machine translation (Vanmassenhove et al., 2018; Stanovsky et al., 2019), sentiment analysis (Kiritchenko and Mohammad, 2018), and hate speech/toxicity detection (e.g., Park et al., 2018; Dixon et al., 2018), among others.
Although these papers have laid vital groundwork by illustrating some of the ways that NLP systems can be harmful, the majority of them fail to engage critically with what constitutes "bias" in the first place. Despite the fact that analyzing "bias" is an inherently normative process—in which some system behaviors are deemed good and others harmful—papers on "bias" in NLP systems are rife with unstated assumptions about what kinds of system behaviors are harmful, in what ways, to whom, and why. Indeed, the term "bias" (or "gender bias" or "racial bias") is used to describe a wide range of system behaviors, even though they may be harmful in different ways, to different groups, or for different reasons. Even papers analyzing "bias" in NLP systems developed for the same task often conceptualize it differently.
For example, the following system behaviors are all understood to be self-evident statements of "racial bias": (a) embedding spaces in which embeddings for names associated with African Americans are closer (compared to names associated with European Americans) to unpleasant words than pleasant words (Caliskan et al., 2017); (b) sentiment analysis systems yielding different intensity scores for sentences containing names associated with African Americans and sentences containing names associated with European Americans (Kir-itchenko and Mohammad, 2018); and (c) toxicity
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454-5476 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
detection systems scoring tweets containing features associated with African-American English as more offensive than tweets without these features (Davidson et al., 2019; Sap et al., 2019). Moreover, some of these papers focus on "racial bias" expressed in written text, while others focus on "racial bias" against authors. This use of imprecise terminology obscures these important differences.
We survey 146 papers analyzing "bias" in NLP systems, finding that their motivations are often vague and inconsistent. Many lack any normative reasoning for why the system behaviors that are described as "bias" are harmful, in what ways, and to whom. Moreover, the vast majority of these papers do not engage with the relevant literature outside of NLP to ground normative concerns when proposing quantitative techniques for measuring or mitigating "bias." As a result, we find that many of these techniques are poorly matched to their motivations, and are not comparable to one another.
We then describe the beginnings of a path forward by proposing three recommendations that should guide work analyzing "bias" in NLP systems. We argue that such work should examine the relationships between language and social hierarchies; we call on researchers and practitioners conducting such work to articulate their conceptualizations of "bias" in order to enable conversations about what kinds of system behaviors are harmful, in what ways, to whom, and why; and we recommend deeper engagements between technologists and communities affected by NLP systems. We also provide several concrete research questions that are implied by each of our recommendations.
Our survey includes all papers known to us analyzing “bias” in NLP systems—146 papers in total. We omitted papers about speech, restricting our survey to papers about written text only. To identify the 146 papers, we first searched the ACL Anthology<sup>1</sup> for all papers with the keywords “bias” or “fairness” that were made available prior to May 2020. We retained all papers about social “bias,” and discarded all papers about other definitions of the keywords (e.g., hypothesis-only bias, inductive bias, media bias). We also discarded all papers using “bias” in NLP systems to measure social “bias” in text or the real world (e.g., Garg et al., 2018).
To ensure that we did not exclude any relevant
NLP task	Papers
Embeddings (type-level or contextualized)	54
Coreference resolution	20
Language modeling or dialogue generation	17
Hate-speech detection	17
Sentiment analysis	15
Machine translation	8
Tagging or parsing	5
Surveys, frameworks, and meta-analyses	20
Other	22
Table 1: The NLP tasks covered by the 146 papers.
papers without the keywords “bias” or “fairness,” we also traversed the citation graph of our initial set of papers, retaining any papers analyzing “bias” in NLP systems that are cited by or cite the papers in our initial set. Finally, we manually inspected any papers analyzing “bias” in NLP systems from leading machine learning, human-computer interaction, and web conferences and workshops, such as ICML, NeurIPS, AIES, FAccT, CHI, and WWW, along with any relevant papers that were made available in the “Computation and Language” and “Computers and Society” categories on arXiv prior to May 2020, but found that they had already been identified via our traversal of the citation graph. We provide a list of all 146 papers in the appendix. In Table 1, we provide a breakdown of the NLP tasks covered by the papers. We note that counts do not sum to 146, because some papers cover multiple tasks. For example, a paper might test the efficacy of a technique for mitigating “bias” in embedding spaces in the context of sentiment analysis.
Once identified, we then read each of the 146 papers with the goal of categorizing their motivations and their proposed quantitative techniques for measuring or mitigating "bias." We used a previously developed taxonomy of harms for this categorization, which differentiates between so-called *allocational* and *representational* harms (Barocas et al., 2017; Crawford, 2017). Allocational harms arise when an automated system allocates resources (e.g., credit) or opportunities (e.g., jobs) unfairly to different social groups; representational harms arise when a system (e.g., a search engine) represents some social groups in a less favorable light than others, demeans them, or fails to recognize their existence altogether. Adapting and extending this taxonomy, we categorized the 146 papers' motivations and techniques into the following categories:
<https://www.aclweb.org/anthology/>
$\triangleright$  Allocational harms.
Category	Papers
Motivation	Technique
Allocational harms	30	4
Stereotyping	50	58
Other representational harms	52	43
Questionable correlations	47	42
Vague/unstated	23	0
Surveys, frameworks, and meta-analyses	20	20
Table 2: The categories into which the 146 papers fall.
$\triangleright$  Representational harms:2
$\triangleright$  Stereotyping that propagates negative generalizations about particular social groups.
Differences in system performance for different social groups, language that misrepresents the distribution of different social groups in the population, or language that is denigrating to particular social groups.
Questionable correlations between system behavior and features of language that are typically associated with particular social groups.
$\triangleright$  Vague descriptions of "bias" (or "gender bias" or "racial bias") or no description at all.
$\triangleright$  Surveys, frameworks, and meta-analyses.
In Table 2 we provide counts for each of the six categories listed above. (We also provide a list of the papers that fall into each category in the appendix.) Again, we note that the counts do not sum to 146, because some papers state multiple motivations, propose multiple techniques, or propose a single technique for measuring or mitigating multiple harms. Table 3, which is in the appendix, contains examples of the papers' motivations and techniques across a range of different NLP tasks.
3 Findings
Categorizing the 146 papers' motivations and proposed quantitative techniques for measuring or mitigating "bias" into the six categories listed above enabled us to identify several commonalities, which we present below, along with illustrative quotes.
Papers state a wide range of motivations, multiple motivations, vague motivations, and sometimes no motivations at all. We found that the papers' motivations span all six categories, with several papers falling into each one. Appropriately, papers that provide surveys or frameworks for analyzing "bias" in NLP systems often state multiple motivations (e.g., Hovy and Spruit, 2016; Bender, 2019; Sun et al., 2019; Rozado, 2020; Shah et al., 2020). However, as the examples in Table 3 (in the appendix) illustrate, many other papers  $(33\%)$  do so as well. Some papers  $(16\%)$  state only vague motivations or no motivations at all. For example,
"[N]o human should be discriminated on the basis of demographic attributes by an NLP system."
—Kaneko and Bollegala (2019)
"[P]rominent word embeddings [...] encode systematic biases against women and black people [...] implicating many NLP systems in scaling up social injustice." —May et al. (2019)
These examples leave unstated what it might mean for an NLP system to "discriminate," what constitutes "systematic biases," or how NLP systems contribute to "social injustice" (itself undefined).
Papers' motivations sometimes include no normative reasoning. We found that some papers  $(32\%)$  are not motivated by any apparent normative concerns, often focusing instead on concerns about system performance. For example, the first quote below includes normative reasoning—namely that models should not use demographic information to make predictions—while the other focuses on learned correlations impairing system performance.
"In [text classification], models are expected to make predictions with the semantic information rather than with the demographic group identity information (e.g., 'gay', 'black') contained in the sentences." —Zhang et al. (2020a)
"An over-prevalence of some gendered forms in the training data leads to translations with identifiable errors. Translations are better for sentences involving men and for sentences containing stereotypical gender roles."
—Saunders and Byrne (2020)
Even when papers do state clear motivations, they are often unclear about why the system behaviors that are described as "bias" are harmful, in what ways, and to whom. We found that even papers with clear motivations often fail to explain what kinds of system behaviors are harmful, in what ways, to whom, and why. For example,
2We grouped several types of representational harms into two categories to reflect that the main point of differentiation between the 146 papers' motivations and proposed quantitative techniques for measuring or mitigating "bias" is whether or not they focus on stereotyping. Among the papers that do not focus on stereotyping, we found that most lack sufficiently clear motivations and techniques to reliably categorize them further.
"Deploying these word embedding algorithms in practice, for example in automated translation systems or as hiring aids, runs the serious risk of perpetuating problematic biases in important societal contexts." Brunet et al. (2019)
"[I]f the systems show discriminatory behaviors in the interactions, the user experience will be adversely affected." —Liu et al. (2019)
These examples leave unstated what "problematic biases" or non-ideal user experiences might look like, how the system behaviors might result in these things, and who the relevant stakeholders or users might be. In contrast, we find that papers that provide surveys or frameworks for analyzing "bias" in NLP systems often name who is harmed, acknowledging that different social groups may experience these systems differently due to their different relationships with NLP systems or different social positions. For example, Ruane et al. (2019) argue for a "deep understanding of the user groups [sic] characteristics, contexts, and interests" when designing conversational agents.
Papers about NLP systems developed for the same task often conceptualize "bias" differently. Even papers that cover the same NLP task often conceptualize "bias" in ways that differ substantially and are sometimes inconsistent. Rows 3 and 4 of Table 3 (in the appendix) contain machine translation papers with different conceptualizations of "bias," leading to different proposed techniques, while rows 5 and 6 contain papers on "bias" in embedding spaces that state different motivations, but propose techniques for quantifying stereotyping.
Papers' motivations conflate allocational and representational harms. We found that the papers' motivations sometimes (16%) name immediate representational harms, such as stereotyping, alongside more distant allocational harms, which, in the case of stereotyping, are usually imagined as downstream effects of stereotypes on résumé filtering. Many of these papers use the imagined downstream effects to justify focusing on particular system behaviors, even when the downstream effects are not measured. Papers on "bias" in embedding spaces are especially likely to do this because embeddings are often used as input to other systems:
However, none of these papers [on embeddings] have recognized how blatantly sexist the embeddings are and hence risk introducing biases of various types into real-world systems.
Bolukbasi et al. (2016a)
"It is essential to quantify and mitigate gender bias in these embeddings to avoid them from affecting downstream applications." —Zhou et al. (2019)
In contrast, papers that provide surveys or frameworks for analyzing "bias" in NLP systems treat representational harms as harmful in their own right. For example, Mayfield et al. (2019) and Ruane et al. (2019) cite the harmful reproduction of dominant linguistic norms by NLP systems (a point to which we return in section 4), while Bender (2019) outlines a range of harms, including seeing stereotypes in search results and being made invisible to search engines due to language practices.
3.2 Techniques
Papers' techniques are not well grounded in the relevant literature outside of NLP. Perhaps unsurprisingly given that the papers' motivations are often vague, inconsistent, and lacking in normative reasoning, we also found that the papers' proposed quantitative techniques for measuring or mitigating "bias" do not effectively engage with the relevant literature outside of NLP. Papers on stereotyping are a notable exception: the Word Embedding Association Test (Caliskan et al., 2017) draws on the Implicit Association Test (Greenwald et al., 1998) from the social psychology literature, while several techniques operationalize the well-studied "Angry Black Woman" stereotype (Kiritchenko and Mohammad, 2018; May et al., 2019; Tan and Celis, 2019) and the "double bind" faced by women (May et al., 2019; Tan and Celis, 2019), in which women who succeed at stereotypically male tasks are perceived to be less likable than similarly successful men (Heilman et al., 2004). Tan and Celis (2019) also examine the compounding effects of race and gender, drawing on Black feminist scholarship on intersectionality (Crenshaw, 1989).
Papers' techniques are poorly matched to their motivations. We found that although  $21\%$  of the papers include allocational harms in their motivations, only four papers actually propose techniques for measuring or mitigating allocational harms.
Papers focus on a narrow range of potential sources of "bias." We found that nearly all of the papers focus on system predictions as the potential sources of "bias," with many additionally focusing on "bias" in datasets (e.g., differences in the number of gendered pronouns in the training data (Zhao et al., 2019)). Most papers do not interrogate
the normative implications of other decisions made during the development and deployment lifecycle—perhaps unsurprising given that their motivations sometimes include no normative reasoning. A few papers are exceptions, illustrating the impacts of task definitions, annotation guidelines, and evaluation metrics: Cao and Daumé (2019) study how folk conceptions of gender (Keyes, 2018) are reproduced in coreference resolution systems that assume a strict gender dichotomy, thereby maintaining cisnormativity; Sap et al. (2019) focus on the effect of priming annotators with information about possible dialectal differences when asking them to apply toxicity labels to sample tweets, finding that annotators who are primed are significantly less likely to label tweets containing features associated with African-American English as offensive.
4 A path forward
We now describe how researchers and practitioners conducting work analyzing "bias" in NLP systems might avoid the pitfalls presented in the previous section—the beginnings of a path forward. We propose three recommendations that should guide such work, and, for each, provide several concrete research questions. We emphasize that these questions are not comprehensive, and are intended to generate further questions and lines of engagement.
Our three recommendations are as follows:
(R1) Ground work analyzing "bias" in NLP systems in the relevant literature outside of NLP that explores the relationships between language and social hierarchies. Treat representational harms as harmful in their own right.
(R2) Provide explicit statements of why the system behaviors that are described as “bias” are harmful, in what ways, and to whom. Be forthright about the normative reasoning (Green, 2019) underlying these statements.
(R3) Examine language use in practice by engaging with the lived experiences of members of communities affected by NLP systems. Interrogate and reimagine the power relations between technologists and such communities.
4.1 Language and social hierarchies
Turning first to (R1), we argue that work analyzing "bias" in NLP systems will paint a much fuller picture if it engages with the relevant literature outside of NLP that explores the relationships between
language and social hierarchies. Many disciplines, including sociolinguistics, linguistic anthropology, sociology, and social psychology, study how language takes on social meaning and the role that language plays in maintaining social hierarchies. For example, language is the means through which social groups are labeled and one way that beliefs about social groups are transmitted (e.g., Maass, 1999; Beukeboom and Burgers, 2019). Group labels can serve as the basis of stereotypes and thus reinforce social inequalities: “[T]he label content functions to identify a given category of people, and thereby conveys category boundaries and a position in a hierarchical taxonomy” (Beukeboom and Burgers, 2019). Similarly, “controlling images,” such as stereotypes of Black women, which are linguistically and visually transmitted through literature, news media, television, and so forth, provide “ideological justification” for their continued oppression (Collins, 2000, Chapter 4).
As a result, many groups have sought to bring about social changes through changes in language, disrupting patterns of oppression and marginalization via so-called "gender-fair" language (Sczesny et al., 2016; Menegatti and Rubini, 2017), language that is more inclusive to people with disabilities (ADA, 2018), and language that is less dehumanizing (e.g., abandoning the use of the term "illegal" in everyday discourse on immigration in the U.S. (Rosa, 2019)). The fact that group labels are so contested is evidence of how deeply intertwined language and social hierarchies are. Taking "gender-fair" language as an example, the hope is that reducing asymmetries in language about women and men will reduce asymmetries in their social standing. Meanwhile, struggles over language use often arise from dominant social groups' desire to "control both material and symbolic resources"—i.e., "the right to decide what words will mean and to control those meanings"—as was the case in some white speakers' insistence on using offensive place names against the objections of Indigenous speakers (Hill, 2008, Chapter 3).
Sociolinguists and linguistic anthropologists have also examined language attitudes and language ideologies, or people's metalinguistic beliefs about language: Which language varieties or practices are taken as standard, ordinary, or unmarked? Which are considered correct, prestigious, or appropriate for public use, and which are considered incorrect, uneducated, or offensive (e.g., Campbell-
Kibler, 2009; Preston, 2009; Loudermilk, 2015; Lanehart and Malik, 2018)? Which are rendered invisible (Roche, 2019)?<sup>3</sup> Language ideologies play a vital role in reinforcing and justifying social hierarchies because beliefs about language varieties or practices often translate into beliefs about their speakers (e.g. Alim et al., 2016; Rosa and Flores, 2017; Craft et al., 2020). For example, in the U.S., the portrayal of non-white speakers' language varieties and practices as linguistically deficient helped to justify violent European colonialism, and today continues to justify enduring racial hierarchies by maintaining views of non-white speakers as lacking the language "required for complex thinking processes and successful engagement in the global economy" (Rosa and Flores, 2017).
Recognizing the role that language plays in maintaining social hierarchies is critical to the future of work analyzing "bias" in NLP systems. First, it helps to explain why representational harms are harmful in their own right. Second, the complexity of the relationships between language and social hierarchies illustrates why studying "bias" in NLP systems is so challenging, suggesting that researchers and practitioners will need to move beyond existing algorithmic fairness techniques. We argue that work must be grounded in the relevant literature outside of NLP that examines the relationships between language and social hierarchies; without this grounding, researchers and practitioners risk measuring or mitigating only what is convenient to measure or mitigate, rather than what is most normatively concerning.