SlowGuess/NAACL_2022_OCR huggingface dataset:
Maarten Sap $^{\diamondsuit}$  Swabha Swayamdipta $^{\spadesuit}$  Laura Vianna $^{\diamond}$
Xuhui Zhou\* Yejin Choi\*\* Noah A. Smith\*\*
$^{\diamond}$ Paul G. Allen School of Computer Science, University of Washington, Seattle, WA, USA
$\spadesuit$ Allen Institute for AI, Seattle, WA, USA
$\diamond$ Department of Psychology, University of Washington, Seattle, WA, USA
\*Georgia Institute of Technology, Atlanta, GA, USA
{maartens, swabhas} @allenai.org
Warning: this paper discusses and contains content that is offensive or upsetting.
The perceived toxicity of language can vary based on someone's identity and beliefs, but this variation is often ignored when collecting toxic language datasets, resulting in dataset and model biases. We seek to understand the who, why, and what behind biases in toxicity annotations. In two online studies with demographically and politically diverse participants, we investigate the effect of annotator identities (who) and beliefs (why), drawing from social psychology research about hate speech, free speech, racist beliefs, political leaning, and more. We disentangle what is annotated as toxic by considering posts with three characteristics: anti-Black language, African American English (AAE) dialect, and vulgarity. Our results show strong associations between annotator identity and beliefs and their ratings of toxicity. Notably, more conservative annotators and those who scored highly on our scale for racist beliefs were less likely to rate anti-Black language as toxic, but more likely to rate AAE as toxic. We additionally present a case study illustrating how a popular toxicity detection system's ratings inherently reflect only specific beliefs and perspectives. Our findings call for contextualizing toxicity labels in social variables, which raises immense implications for toxic language annotation and detection.
Determining whether a text is toxic (i.e., contains hate speech, abuse, or is offensive) is inherently a subjective task that requires a nuanced understanding of the pragmatic implications of language (Fiske, 1993; Croom, 2011; Waseem et al., 2021). Without this nuance, both humans and machines are prone to biased judgments, such as over-relying on seemingly toxic keywords (e.g., expletives, swearwords; Dinan et al., 2019; Han
and Tsvetkov, 2020) or backfiring against minorities (Yasin, 2018; Are, 2020, i.a.). For example, racial biases have been uncovered in toxic language detection where text written in African American English (AAE) is falsely flagged as toxic (Sap et al., 2019; Davidson et al., 2019).
The crux of the issue is that not all text is equally toxic for everyone (Waseem, 2016; Al Kuwatly et al., 2020). Yet, most previous research has treated this detection as a simple classification with one correct label, obtained by averaging judgments by a small set of human raters per post (Waseem and Hovy, 2016; Wulczyn et al., 2017; Davidson et al., 2017; Founta et al., 2018; Zampieri et al., 2019). Such approaches ignore the variance in annotations (Pavlick and Kwiatkowski, 2019; Geva et al., 2019; Arhin et al., 2021; Akhtar et al., 2021) based on who the annotators are, and what their beliefs are.
In this work, we investigate the who, why, and what behind biases<sup>1</sup> in toxicity annotations, through online studies with demographically and politically diverse participants. We measure the effects of annotator identities (who annotates as toxic) and attitudes or beliefs (why they annotate as toxic) on toxicity perceptions, through the lens of social psychology research on hate speech, free speech, racist beliefs, altruism, political leaning, and more. We also analyze the effect of what is being rated, by considering three text characteristics: anti-Black or racially prejudiced meaning, African American English (AAE), and vulgar words.
We seek to answer these questions via two online studies. In our breadth-of-workers controlled study, we collect ratings of toxicity for a set of 15 hand curated posts from 641 annotators of different races, attitudes, and political leanings.
<sup>1</sup>We use the term "bias" to denote both simple skews or variation in annotations (e.g., for variation in detecting vulgar content as toxic) or representational harms (e.g., AAE being over-detected as toxic or anti-Black content being underdetected as toxic; Barocas et al., 2017; Blodgett et al., 2020).
Human Language Technologies, pages 5884 - 5906
Going to Africa. Hope I don't get AIDS. Just kidding. I'm white!
Less offensive/racist for annotators who... score higher in FREEOFFSpeech, RACISTBELIEFS, LINGPURISM, TRADITIONALISM; are more conservative, men, White
Less offensive/racist for annotators who... score higher in RACIST BELIEFS
More offensive/racist for annotators who... score higher in EMPATHY, ALTRUISM, HARMOFHATESpeech; are more liberal, women, Black
More offensive/racist for annotators who... score higher in HARMOfHATESpeech
If u grown & still get thirsty for Jordans known erbody else gon havem & u still feel like u accomplished something that say alot about u
More racist for annotators who... score higher in RACISTBELIEFS; are more conservative
More racist for annotators who... score higher in FREEOffSpeech,
TRADITIONALISM; are more conservative
More offensive for annotators who... score higher in RACISTBELIEFS, TRADITIONALISM; are more conservative
I got mosquito bites on my foot and they fucking hurt
More offensive for annotators who... score higher in LINGPURISM, TRADITIONALISM; are more conservative
Anti-Black
AAE
Vulgar
Figure 1: Annotator identities and attitudes can influence how they rate toxicity in text. We summarize the key findings from our analyses of biases in toxicity (offensiveness or racism) ratings for three types of language: anti-Black content, African American English (AAE), and vulgar language.
Then, in our breadth-of-posts study, we simulate a typical toxic language annotation setting by collecting toxicity ratings for  $\sim 600$  posts, from a smaller but diverse pool of 173 annotators.
Distilled in Figure 1, our most salient results across both studies show that annotators scoring higher on our racist beliefs scale were less likely to rate anti-Black content as toxic (§4). Additionally, annotators' conservatism scores were associated with higher ratings of toxicity for AAE (§5), and conservative and traditionalist attitude scores with rating vulgar language as more toxic (§6).
We further provide a case study which shows that PERSPECTIVEAPI, a popular toxicity detection system, mirrors ratings by annotators of certain attitudes and identities over others (§7). For instance, for anti-Black language, the system's scores better reflect ratings by annotators who score high on our scale for racist beliefs. Our findings have immense implications for the design of toxic language annotation and automatic detection—we recommend contextualizing ratings in social variables and looking beyond aggregated discrete decisions (§8).
2 The Who, Why, and What of Toxicity Annotations
We aim to investigate how annotators' ratings of the toxicity of text is influenced by their own identities (who they are; §2.1), and their beliefs
(why they consider something toxic; §2.2) on specific categories of text (what they consider toxic; §2.3)—namely, text with anti-Black language, presence of African American English (AAE), and presence of vulgar or profane words. To this end, we design two online studies (§3) and discuss who find each of these text characteristics offensive and why as separate research questions in Sections §4, §5, and §6, respectively.
2.1 Demographic Identities: Who considers something as toxic?
Prior work has extensively shown links between someone's gender, political leaning, and race affects how likely they are to perceive or notice harmful speech or racism (Cowan et al., 2002; Norton and Sommers, 2011; Carter and Murphy, 2015; Prabhakaran et al., 2021). Grounded in this prior literature, our study considers annotators' race, gender, and political leaning. Since perceptions of race and political attitudes vary vastly across the globe, we restrict our study to participants exclusively from the United States.
2.2 Attitudes: Why does someone consider something toxic?
While some annotator toxicity ratings may highly correlate with demographic factors at face value (Prabhakaran et al., 2021; Jiang et al., 2021), we aim to go beyond demographics to investigate annotator beliefs that explain these correlations. Based on prior work in social psychology, polit
Text Categories
Key Findings: Identity and Attitude Biases in Toxicity Detection
Breadth-of-Workers
Breadth-of-Posts
2Please contact the authors for the anonymized study data.
ical science, and sociolinguistics, we select seven attitude dimensions, which we operationalize via scales (in SMALL CAPS), as described below. $^3$
Valuing the freedom of offensive speech (FREEOFFSPEECH): the belief that any speech, including offensive or hateful speech, should be unrestricted and free from censorship. Recently, this belief has become associated with majority and conservative identities (Cole, 1996; Gillborn, 2009; White and Crandall, 2017; Elers and Jayan, 2020). We use the scale by Cowan and Khatchadourian (2003); see Appendix A.1.
Perceiving the HARMOFHATESPEECH: the belief that hate speech or offensive language can be harmful for the targets of that speech (Soral et al., 2018; Nadal, 2018). This belief is correlated with socially-progressive philosophies (Downs and Cowan, 2012, see also Nelson et al., 2013). We use the scale by Cowan and Khatchadourian (2003); see Appendix A.2.
Endorsement of RACISTBELIEFS: the beliefs which deny the existence of racial inequality, or capture resentment towards racial minorities (Poteat and Spanierman, 2012). We measure RACISTBELIEFS using items from the validated Modern Racism Scale (McConahay, 1986); see Appendix A.3.
TRADITIONALISM: the belief that one should follow established norms and traditions, and be respectful of elders, obedient, etc. In the US, these beliefs are associated with generally conservative ideologies (Johnson and Tamney, 2001; Knuckey, 2005). We use an abridged version<sup>4</sup> of the TRADITIONALISM scale (Bouchard Jr. and McGue, 2003) that measures annotators' adherence to traditional values; see Appendix A.4.
Language Purism (LINGPURISM): the belief that there is a "correct" way of using English (Jernudd and Shapiro, 1989). Typically, this belief also involves negative reactions to non-canonical ways of using language (Sapolsky et al., 2010; DeFrank and Kahlbaugh, 2019). We created and validated a four-item LINGPURISM scale to measure this concept; see Appendix A.5.
EMPATHY: one's tendency to see others' perspectives and feel others' feelings. Research in social psychology has linked higher levels of empathy to
the ability and willingness of recognizing and labeling hate speech (Cowan and Khatchadourian, 2003). We measure EMPATHY using an abbreviated Interpersonal Reactivity Index (Pulos et al., 2004); see Appendix A.6.
ALTRUISM: one's attitude of selfless concern about others' well-being, which can move people to act when harm or injustice happens (Wagstaff, 1998; Gavrilets and Fortunato, 2014; Riar et al., 2020), including harms through hate speech (Cowan et al., 2002). We gathered the items to measure ALTRUISM with an adapted scale taken from Steg et al. (2014); see Appendix A.7.
It is worth noting that some of the above attitudes, though not all, correlate with demographics very strongly. Table 8 in Appendix A.8 details these correlations from our study.
2.3 Text Characteristics: What is considered offensive?
Not all toxic text is toxic for the same reasons. We aim to understand how characteristics of text can affect ratings of toxicity, in addition to annotator attitudes and identities. Specifically, we consider three dimensions or categories of text, based on recent work on text characteristics that tend to be over- or under-detected as toxic (Dinan et al., 2019; Sap et al., 2019; Han and Tsvetkov, 2020; Zhou et al., 2021): anti-Black language, presence of African American English (AAE) dialect markers, and vulgar language (e.g., swearwords, slurs). We distinguish between two types of vulgarity, following Zhou et al. (2021): swearwords or explicit words that do not reference identities (offensive, non-identity referring; ONI), and (reclaimed) slurs or other identity-referring vulgarity (offensive identity-referring; OI). In our analyses, we focus on ONI vulgarity unless explicitly noted.
3 Data & Study Design
We design two online studies to study the effect of annotator identities and attitudes on their toxicity ratings on posts with different characteristics. In either study, annotators are asked to rate how offensive and how racist they consider a post to be (see Appendix B for the exact questions). We specifically focus on readers' perceptions or opinions, instead of imposing prescriptive definitions
For both studies, we sought explicit consent from participants, paid participants above minimum wage, and obtained approval by our institution's ethics board (IRB).
3We abstain from conclusions beyond our abstractions.
4This was done to reduce cognitive load on annotators.
Vulgar (ONI), not Anti-Black, not AAE
I sort of fuckin love how close my gf is with my family like that she knows about my family jokes before I fuckin do
I'm hoping that Cersei gets her ass handed to her but I love Daenerys and Jon;3: I don't want either of em losing xD
I got mosquito bites on my foot and they fucking hurt  
Personally I think it would be fucking hilarious to put Franziska or Godot in but idk if they're important enough characters to make the cut
I just keep telling myself that shitty things happen to me but something really great is gonna happen bc romantic comedies have ruined me
AAE, not Vulgar, not Anti-Black
If u grown & still get thirsty for Jordans knowin erbody else gon havem & u still feel like u accomplished something that say alot about u
Just spent a whole 20k(imaginary money) on a home studio.. And used the last 3$2 get some flowers n a gift card for YA BIHH.
When everything is going accordingly the enemy always try to come creep up on you but I ain't havin it!! buhh byeee
You can't slang no tea cause he been putting me on bout everything. I could only wish dawg the best
ain't no problem in culling ppl off. save that space for somebody who'll benefit from u and vice versa
Anti-Black, not Vulgar, not AAE
I don't get government benefits just for being white Where is white history month? Why can't I have white pride day?
I don't like black doctors, it's just my preference. Anti-Black racism isn't that big of a deal because people treated Italians like this in the 60's and 70's
of toxicity or hate speech which previous work has shown still suffers from large annotator disagreement (Ross et al., 2017). In the sections §4-6, we report only (near-)significant associations; see Appendix E and F for all results.
3.1 Breadth-of-Workers Study
Our first study focuses on collecting toxicity ratings from a wide and diverse set of participants for a controlled set of posts. Shown in Table 1, we hand curated a set of 15 posts that belong exclusively to one text category (e.g., vulgar but non-AAE and non-anti-Black; see Appendix C.1 for more data selection and validation details). To exclude confounding effects of offensive identity mentions (OI; e.g., slurs) which could be both vulgar and anti-Black (or sexist, homophobic, etc.), we only considered posts with vulgar terms that are non-identity referring (ONI; e.g., swearwords).
We ran our study on a 641 participants that were recruited using a pre-qualifier survey on Amazon Mechanical Turk (MTurk) to ensure racial and political diversity. Our final participant pool spanned various racial (13% Black, 85% White), political (29% conservative, 59% liberal), and gender identities (54% women, 45% men, 1% non-binary). Each participant gave each of the 15 posts toxicity ratings, after which they answered a series of questions about their attitudes and their identity. We
Table 1: All 15 posts for our breadth-of-workers study.
cat.	Anti-Black	AAE	Vulgar (ONI)	Vulgar (OI)
count	113	270	196	217
Table 2: Counts for each text category for the 571 posts in our breadth-of-posts study. OI: identity-referring vulgarity, ONI: non-identity referring vulgarity; categories are explained in §2.3. Posts could belong to multiple categories (Figure 5 in Appendix F).
used three attention checks to ensure data quality. For further details, please see Appendix C.
In our subsequent analyses, we compute associations between the toxicity ratings and identities or attitudes by computing the effect sizes (Pearson  $r$  correlation or Cohen's  $d$ ) between the average toxicity rating of the posts in each category and annotator identities or attitude scores.
3.2 Breadth-of-Posts Study
Our second study focuses on collecting ratings for a larger set of posts, but with fewer annotators per post to simulate a crowdsourced dataset on toxic language. In contrast to the previous study, we consider anti-Black or AAE posts that could also be vulgar, and allow this vulgarity to cover both potentially offensive identity references (OI) as well as non-identity vulgar words (ONI; see §2.3). We do not consider posts that are anti-Black and AAE, since the pragmatic toxicity implications of anti-Black meaning expressed in AAE are very complex (e.g., in-group language with self-deprecation, sarcasm, reclaimed slurs; Greengross and Miller, 2008; Croom, 2011), and are thus beyond the scope of this study.
We draw from two existing toxic language detection corpora to select 571 posts (Table 2). For AAE and possibly vulgar posts, we draw from Founta et al. (2018), using an automatic AAE detector by Blodgett et al.  $(2016)^{6}$  and the vulgarity word list from Zhou et al. (2021) for detecting OI and ONI terms. For anti-Black and possibly vulgar posts, we select posts annotated as anti-Black in Vidgen et al. (2021), using the same method by Zhou et al. (2021) for detecting vulgar terms. See Appendix D.1 for more data selection details.
As with the previous study, we ran our annotation study on 173 participants recruited through a pre-qualifier survey. Our annotators varied racially
Breadth-of-Workers study
Breadth-of-Posts study
6The text-only AAE detector by Blodgett et al. (2016) strongly correlates  $(r = .60)$  with more race-aware AAE detectors (Sap et al., 2019).
Anti-Black posts	Rated as Offensive	Rated as Racist
EMPATHY	r=	0.285	**	r=	0.286
ALTRUISM	r=	0.380	**	r=	0.441
HARMOfHATESpeechECH	r=	0.451	**	r=	0.528
FREEOFFSpeechECH	r=	-0.394	**	r=	-0.467
RACISTBELIEFS	r=	-0.513	**	r=	-0.574
LINGPURISM	r=	-0.154	**	r=	-0.167
TRADITIONALISM	r=	-0.206	**	r=	-0.237
Politics (lib.: 0, cons.: 1)	r=	-0.374	**	r=	-0.441
Gender (men: 0, women: 1)	d=	0.321	**	d=	0.341
Race (White: 0, Black: 1)	d=	0.301	*	n.s.
(20% Black, 76% White), politically (30% conservative, 54% liberal), and in gender (45% women, 53% men,  $< 2\%$  non-binary). Each post was annotated by 6 participants from various racial and political identities. Additionally, we asked participants one-item versions of our attitude scales, using the question from each scale that correlated best with toxicity in our breadth-of-workers study as explained in Appendix D.3. See Appendix D for more study design details.
In our analyses, we examine toxicity of anti-Black and potentially vulgar posts (§4.2) and of AAE and potentially vulgar posts (§5.2), but not of vulgar posts separately, due to confounding effects of the AAE or anti-Black characteristics that those posts could have. Additionally, unlike the breadth-of-workers study, here each annotator could rate a varying number of posts. Thus, we compute associations between toxicity ratings and identities or attitudes using a linear mixed effects model with random effects for each participant.
4 Who finds anti-Black posts toxic, and why?
Anti-Black language denotes racially prejudiced or racist content—subtle (Breitfeller et al., 2019) or overt—which is often a desired target for toxic language detection research (Waseem, 2016; Vid
Table 3: Associations between annotator variables and ratings of offensiveness and racism for the anti-Black posts in the breadth-of-workers study. We use the Holm correction for multiple comparisons for non-hypothesized associations and only present significant Pearson  $r$  or Cohen's  $d$  effect sizes (*:  $p < {0.05}$  ,**:  $p$  < 0.001; n.s.: not significant).
Anti-Black posts	Rated as Offensive	Rated as Racist
HARMOFHATESPEECH	0.117†	0.154*
FREEOFFSPEECH	n.s.	-0.138†
RACISTBELIEFS	-0.131*	-0.185*
Table 4: Associations for anti-Black (and potentially also vulgar) posts from the breadth-of-posts study, shown as the  $\beta$  coefficients from a mixed effects model with a random effect for each annotator ( $\dagger$ :  $p < 0.075$ ,  $*$ :  $p < 0.05$ ,  $**$ :  $p < 0.001$ ; Holm-corrected for multiple comparisons; n.s.: not significant).
gen et al., 2021). Based on prior work on linking conservative ideologies, endorsement of unrestricted speech, and racial prejudice with reduced likelihood to accept the term "hate speech" (Duckitt and Fisher, 2003; White and Crandall, 2017; Roussos and Dovidio, 2018; Elers and Jayan, 2020), we hypothesize that conservative annotators and those who score highly on the RACISTBELIEFS or FREEOFFSPEECH scales will rate anti-Black tweets as less toxic, and vice-versa. Conversely, based on findings by Cowan and Khatchadourian (2003), we hypothesize that annotators with high HARMOFHATESPEECH scores will rate anti-Black tweets are more toxic.
4.1 Breadth-of-Workers Results
As shown in Table 3, we found several associations between annotator beliefs and toxicity ratings for anti-Black posts, confirming our hypotheses. The three most salient associations with lower racism ratings were annotators who scored higher in RACISTBELIEFS, FREEOFFSPEECH, and those who leaned conservative. We find similar trends for offensiveness ratings.
Conversely, we found that participants who scored higher in HARMOFHATESPEECH were much more likely to rate anti-Black posts as more offensive, and more racist. Finally, though both white and Black annotators rated these posts very high in offensiveness (with means  $\mu_{\mathrm{Black}} = 3.85$  and  $\mu_{\mathrm{White}} = 3.59$  out of 5), our results show that Black participants were slightly more likely than white participants to rate them as offensive.
Our exploratory analyses unearthed other significant associations: negative correlations with LINGPURISM, TRADITIONALISM, and gender (male), and positive correlations with high EMPATHY, ALTRUIISM, and gender (female).
For each post, we collected toxicity ratings from two white conservative workers, two from white liberal workers, and two from Black workers.
$^{8}$ Using the statsmodels implementation: https://www.statsmodels.org/stable/generated/statsmodels.formula api.mixedlm.html
4.2 Breadth-of-Posts Results
Table 4 shows similar results as in the breadth-of-workers analyses, despite the posts now potentially containing vulgarity. Specifically, we find that annotators who scored higher in RACIST-BELIEFS rated anti-Black posts as less offensive, whereas those who scored higher in HARMOFHATESPEECH rated them as more offensive. Ratings of racism showed similar effects, along with a near-significant association between higher FREEOFFSPEECH scores and lower ratings of racism for anti-Black posts.
4.3 Perceived Toxicity of Anti-Black Language
Overall, our results from both studies corroborate previous findings that studied associations between attitudes toward hate speech and gender and racial identities, specifically that conservatives, white people, and men tend to value free speech more, and that liberals, women, and nonwhite people perceive the harm of hate speech more (Cowan and Khatchadourian, 2003; Downs and Cowan, 2012). Our results also support the finding that those who hold generally conservative ideologies tend to be more accepting towards anti-Black or racially prejudiced content (Goldstein and Hall, 2017; Lucks, 2020; Schaffner, 2020).