Venue: NAACL 2022

# Annotators with Attitudes:

How Annotator Beliefs And Identities Bias Toxic Language Detection

 Maarten Sap\({}^{\diamondsuit}\) Swabha Swayamdipta\({}^{\spadesuit}\) Laura Vianna\({}^{\diamondsuit}\)

Xuhui Zhou\({}^{\clubsuit}\) Yejin Choi\({}^{\diamondsuit}\) Noah A. Smith\({}^{\diamondsuit}\)

\({}^{\diamondsuit}\)Paul G. Allen School of Computer Science, University of Washington, Seattle, WA, USA

\({}^{\spadesuit}\)Allen Institute for AI, Seattle, WA, USA

\({}^{\diamondsuit}\)Department of Psychology, University of Washington, Seattle, WA, USA

\({}^{\clubsuit}\)Georignainstitute of Technology, Atlanta, GA, USA {maartens,swabhas}@allenai.org

###### Abstract

_Warning_: this paper discusses and contains content that is offensive or upsetting. The perceived toxicity of language can vary based on someone's identity and beliefs, but this variation is often ignored when collecting toxic language datasets, resulting in dataset and model biases. We seek to understand the _who_, _why_, and _what_ behind biases in toxicity annotations. In two online studies with demographically and politically diverse participants, we investigate the effect of annotator identities (_who_) and beliefs (_why_), drawing from social psychology research about hate speech, free speech, racist beliefs, political leaning, and more. We disentangle _what_ is annotated as toxic by considering posts with three characteristics: anti-Black language, African American English (AAE) dialect, and vulgarity. Our results show strong associations between annotator identity and beliefs and their ratings of toxicity. Notably, more conservative annotators and those who scored highly on our scale for racist beliefs were less likely to rate anti-Black language as toxic, but more likely to rate AAE as toxic. We additionally present a case study illustrating how a popular toxicity detection system's ratings inherently reflect only specific beliefs and perspectives. Our findings call for contextualizing toxicity labels in social variables, which raises immense implications for toxic language annotation and detection.

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5884 - 5906

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Determining whether a text is toxic (i.e., contains hate speech, abuse, or is offensive) is inherently a subjective task that requires a nuanced understanding of the pragmatic implications of language Fiske (1993); Croom (2011); Waseem et al. (2021). Without this nuance, both humans and machines are prone to biased judgments, such as over-relying on seemingly toxic keywords (e.g., expletives, swearwords; Dinan et al., 2019; Han and Tsvetkov, 2020) or backfiring against minorities Yasin (2018); Are (2020), i.a.). For example, racial biases have been uncovered in toxic language detection where text written in African American English (AAE) is falsely flagged as toxic Sap et al. (2019); Davidson et al. (2019).

The crux of the issue is that not all text is equally toxic for everyone Waseem (2016); Al Kuwatly et al. (2020). Yet, most previous research has treated this detection as a simple classification with one correct label, obtained by averaging judgments by a small set of human raters per post Waseem and Hovy (2016); Wulczyn et al. (2017); Davidson et al. (2017); Founta et al. (2018); Zampieri et al. (2019). Such approaches ignore the variance in annotations Pavlick and Kwiatkowski (2019); Geva et al. (2019); Arhin et al. (2021); Akhtar et al. (2021) based on who the annotators are, and what their beliefs are.

In this work, we investigate the _who_, _why_, and _what_ behind biases1 in toxicity annotations, through online studies with demographically and politically diverse participants. We measure the effects of annotator identities (_who_ annotates as toxic) and attitudes or beliefs (_why_ they annotate as toxic) on toxicity perceptions, through the lens of social psychology research on hate speech, ricest beliefs, altruism, political leaning, and more. We also analyze the effect of _what_ is being rated, by considering three text characteristics: anti-Black or racially prejudiced meaning, African American English (AAE), and vulgar words.

Footnote 1: We use the term “bias” to denote both simple skews or variation in annotations (e.g., for variation in detecting vulgar content as toxic) or representational harms (e.g., AAE benefiting over-detected as toxic or anti-Black content being undetected as toxic; Barocas et al., 2017; Blodgett et al., 2020).

We seek to answer these questions via two online studies. In our **breadth-of-workers** controlled study, we collect ratings of toxicity for a set of 15 hand curated posts from 641 annotators of different races, attitudes,

Then, in our **breadth-of-posts** study, we simulate a typical toxic language annotation setting by collecting toxicity ratings for \(\sim\)600 posts, from a smaller but diverse pool of 173 annotators.2

Footnote 2: Please contact the authors for the anonymized study data.

Distilled in Figure 1, our most salient results across both studies show that annotators scoring higher on our racist beliefs scale were less likely to rate anti-Black content as toxic (SS4). Additionally, annotators' conservatism scores were associated with higher ratings of toxicity for AAE (SS5), and conservative and traditionalist attitude scores with rating vulgar language as more toxic (SS6).

We further provide a case study which shows that PerspectiveAPI, a popular toxicity detection system, mirrors ratings by annotators of certain attitudes and identities over others (SS7). For instance, for anti-Black language, the system's scores better reflect ratings by annotators who score high on our scale for racist beliefs. Our findings have immense implications for the design of toxic language annotation and automatic detection--we recommend contextualizing ratings in social variables and looking beyond aggregated discrete decisions (SS8).

## 2 The _Who_, _Why_, and _What_ of Toxicity Annotations

We aim to investigate how annotators' ratings of the toxicity of text is influenced by their own identities (_who they are_; SS2.1), and their beliefs (_why they consider something toxic_; SS2.2) on specific categories of text (_what they consider toxic_; SS2.3)--namely, text with anti-Black language, presence of African American English (AAE), and presence of vulgar or profane words. To this end, we design two online studies (SS3) and discuss _who_ find each of these text characteristics offensive and _why_ as separate research questions in Sections SS4, SS5, and SS6, respectively.

### Demographic Identities: _Who_ considers something as toxic?

Prior work has extensively shown links between someone's gender, political leaning, and race affects how likely they are to perceive or notice harmful speech or racism Cowan et al. (2002); Norton and Sommers (2011); Carter and Murphy (2015); Prabhakaran et al. (2021). Grounded in this prior literature, our study considers annotators' **race**, **gender**, and **political leaning**. Since perceptions of race and political attitudes vary vastly across the globe, we restrict our study to participants exclusively from the United States.

### Attitudes: _Why_ does someone consider something toxic?

While some annotator toxicity ratings may highly correlate with demographic factors at face value Prabhakaran et al. (2021); Jiang et al. (2021), we aim to go beyond demographics to investigate annotator _beliefs_ that explain these correlations. Based on prior work in social psychology, polit

Figure 1: Annotator identities and attitudes can influence how they rate toxicity in text. We summarize the key findings from our analyses of biases in toxicity (offensiveness or racism) ratings for three types of language: anti-Black content, African American English (AAE), and vulgar language.

 

ical science, and sociolinguistics, we select seven attitude dimensions, which we operationalize via scales (in small caps), as described below.3

Footnote 2: We use the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer to the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “_HarmOfHateSpeech_” to refer the term “HarmOfHateSpeech_” to refer the term “HarmOfHateSpeech_” to refer the term “HarmOfHateSpeech_” to refer the term “HarmOfHateSpeech_” to refer the term “HarmOfHateSpeech_” to refer the term “HarmOfHateSpeech_” to refer the term “HarmOfHateSpeech_” to refer the term “HarmOfHateSpeech_” to refer the term “HarmOfHateSpeech_” to refer the term “HarmOfHateSpeech_” to refer the term “HarmOfHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_” to the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_" to the term “HarmHateSpeech_” to refer the term “HarmHateSpeech_ 

of toxicity or hate speech which previous work has shown still suffers from large annotator disagreement (Ross et al., 2017). In the sections SS4-6, we report only (near-)significant associations; see Appendix E and F for all results.

### Breadth-of-Workers Study

Our first study focuses on collecting toxicity ratings from a wide and diverse set of participants for a controlled set of posts. Shown in Table 1, we hand curated a set of 15 posts that belong exclusively to one text category (e.g., vulgar but non-AAE and non-anti-Black; see Appendix C.1 for more data selection and validation details). To exclude confounding effects of offensive identity mentions (OI; e.g., slurs) which could be both vulgar and anti-Black (or sexist, homophobic, etc.), we only considered posts with vulgar terms that are non-identity referring (ONI; e.g., swearwords).

We ran our study on a 641 participants that were recruited using a pre-qualifier survey on Amazon Mechanical Turk (MTurk) to ensure racial and political diversity. Our final participant pool spanned various racial (13% Black, 85% White), political (29% conservative, 59% liberal), and gender identities (54% women, 45% men, 1% non-binary). Each participant gave each of the 15 posts toxicity ratings, after which they answered a series of questions about their attitudes and their identity. We used three attention checks to ensure data quality. For further details, please see Appendix C.

In our subsequent analyses, we compute associations between the toxicity ratings and identities or attitudes by computing the effect sizes (Pearson \(r\) correlation or Cohen's \(d\)) between the average toxicity rating of the posts in each category and annotator identities or attitude scores.

### Breadth-of-Posts Study

Our second study focuses on collecting ratings for a larger set of posts, but with fewer annotators per post to simulate a crowdsourced dataset on toxic language. In contrast to the previous study, we consider anti-Black or AAE posts that could also be vulgar, and allow this vulgaris to cover both potentially offensive identity references (OI) as well as non-identity vulgar words (ONI; see SS2.3). We do not consider posts that are anti-Black and AAE, since the pragmatic toxicity implications of anti-Black meaning expressed in AAE are very complex (e.g., in-group language with self-deprecation, sarcasm, reclaimed slurs; Greengross and Miller, 2008; Croom, 2011), and are thus beyond the scope of this study.

We draw from two existing toxic language detection corpora to select 571 posts (Table 2). For AAE and possibly vulgar posts, we draw from Founta et al. (2018), using an automatic AAE detector by Blodgett et al. (2016)6 and the vulgarity word list from Zhou et al. (2021) for detecting OI and ONI terms. For anti-Black and possibly vulgar posts, we select posts annotated as anti-Black in Vidgen et al. (2021), using the same method by Zhou et al. (2021) for detecting vulgar terms. See Appendix D.1 for more data selection details.

Footnote 6: The text-only AAE detector by Blodgett et al. (2016) strongly correlates (\(r\)=.60) with more race-aware AAE detectors (Sap et al., 2019).

As with the previous study, we ran our annotation study on 173 participants recruited through a pre-qualifier survey. Our annotators varied racially

\begin{table}
\begin{tabular}{c c c c}  & **Vapar (ONI), net Anti-Black, nat AAE** \\ \hline _sort_ of n�tachio new class _my off k with my family like that the known colour may imply below twenty touch locatio_ ok & \\ _from_ high grade Greek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pek _pepek _pek _pek _pepek _pek _pepek _pek _pepek _pepek _pepek _pepek _pepek _pepek _pepek _pepepek _pepepek _ 

(20% Black, 76% White), politically (30% conservative, 54% liberal), and in gender (45% women, 53% men, \(<\)2% non-binary). Each post was annotated by 6 participants from various racial and political identities.7 Additionally, we asked participants one-item versions of our attitude scales, using the question from each scale that correlated best with toxicity in our breadth-of-workers study as explained in Appendix D.3. See Appendix D for more study design details.

Footnote 7: For each post, we collected toxicity ratings from two white conservative workers, two from white liberal workers, and two from Black workers.

In our analyses, we examine toxicity of anti-Black and potentially vulgar posts (SS4.2) and of AAE and potentially vulgar posts (SS5.2), but not of vulgar posts separately, due to confounding effects of the AAE or anti-Black characteristics that those posts could have. Additionally, unlike the breadth-of-workers study, here each annotator could rate a varying number of posts. Thus, we compute associations between toxicity ratings and identities or attitudes using a linear mixed effects model8 with random effects for each participant.

## 4 Who finds anti-Black posts toxic, and why?

Anti-Black language denotes racially prejudiced or racist content--subtle Breitfeller et al. (2019) or overt--which is often a desired target for toxic language detection research Waseem (2016); Vidgen et al. (2021). Based on prior work on linking conservative ideologies, endorsement of unrestricted speech, and racial prejudice with reduced likelihood to accept the term "hate speech" Duckitt and Fisher (2003); White and Crandall (2017); Roussos and Dovidio (2018); Elers and Jayan (2020), we hypothesize that conservative annotators and those who score highly on the RacistBeliefs or FreeOffSpeech scales will rate anti-Black tweets as less toxic, and vice-versa. Conversely, based on findings by Cowan and Khatchadourian (2003), we hypothesize that annotators with high HarmOfHateSpeech scores will rate anti-Black tweets are more toxic.

### Breadth-of-Workers Results

As shown in Table 3, we found several associations between annotator beliefs and toxicity ratings for anti-Black posts, confirming our hypotheses. The three most salient associations with _lower racism_ ratings were annotators who scored higher in RacistBeliefs, FreeOffSpeech, and those who leaned conservative. We find similar trends for offensiveness ratings.

Conversely, we found that participants who scored higher in HarmOfHateSpeech were much more likely to rate anti-Black posts as _more offensive_, and _more racist_. Finally, though both white and Black annotators rated these posts very high in offensiveness (with means \(\mu_{\text{Black}}\) = 3.85 and \(\mu_{\text{white}}\) = 3.59 out of 5), our results show that Black participants were slightly more likely than white participants to rate them as offensive.

Our exploratory analyses unearthed other significant associations: negative correlations with LingPurism, Traditionalism, and gender (male), and positive correlations with high Empathy, Altruism, and gender (female).

\begin{table}
\begin{tabular}{l|r||c} \hline _Anti-Black posts_ & _Rated at Offensive_ & _Rated at Racist_ \\ \hline HarmOfAttAttSpeech & 0.117  \\ FireOrSweet & \(n.s.\) & -0.138  \\ RectifBlatters & -0.131  \\ \hline \end{tabular}
\end{table}
Table 4: Associations for _anti-Black_ (and potentially also vulgar) posts from the _breadth-of-posts_ study, shown as the \(\beta\) coefficients from a mixed effects model with a random effect for each annotator (‘\(p\) < 0.075, *’: \(p\) < 0.05, **’: \(p\) < 0.001; Holm-corrected for multiple comparisons; \(n.s.\): not significant).

\begin{table}
\begin{tabular}{l|c c|c c} _Anti-Black posts_ & _Rated at Offensive_ & _Rated at Racist_ \\ \hline Empty & \(r\) = & 0.285  & \(r\) = & 0.286  \\ Altugum & \(r\) = & 0.380  & \(r\) = & 0.441  \\ HammOfAttSpeech & \(r\) = & 0.451  & \(r\) = & 0.528  \\ FresOptSpeech & \(r\) = & -0.394  & \(r\) = & -0.467  \\ RacistBeliefs & \(r\) = & -0.543  & \(r\) = & -0.548  \\ LispPursam & \(r\) = & -0.154  & \(r\) = & -0.167  \\ Tradstromlassium & \(r\) = & -0.206  & \(r\) = & -0.237  \\ Politics & (\(\hbar b.a. cons. i) & \(r\) = & -0.374  & \(r\) = & -0.441  \\ Gender & (\(\hbar m.o. twitter. l) & \(d\) = & 0.321  & \(d\) = & 0.341  \\ Race & (\hbar m.o. black. l) & \(d\) = & 0.301  & \(\pi.s.\) \\ \hline \end{tabular}
\end{table}
Table 3: Associations between annotator variables and ratings of offensiveness and racism for the _anti-Black_ posts in the _breadth-of-workers_ study. We use the Holm correction for multiple comparisons for non-hypothesized associations and only present significant Pearson \(r\) or Cohen’s \(d\) effect sizes (*: \(p\) < 0.05, **: \(p\) < 0.001; \(n.s.\): not significant).

 



### Breadth-of-Posts Results

Table 4 shows similar results as in the breadth-of-workers analyses, despite the posts now potentially containing vulgarity. Specifically, we find that annotators who scored higher in RacistBelieffs rated anti-Black posts as _less_ offensive, whereas those who scored higher in HamOfHateSpeech rated them as _more_ offensive. Ratings of racism showed similar effects, along with a near-significant association between higher FreeOffSpeech scores and lower ratings of racism for anti-Black posts.

### Perceived Toxicity of Anti-Black Language

Overall, our results from both studies corroborate previous findings that studied associations between attitudes toward hate speech and gender and racial identities, specifically that conservatives, white people, and men tend to value free speech more, and that liberals, women, and non-white people perceive the harm of hate speech more Cowan and Khatchadourian (2003); Downs and Cowan (2012). Our results also support the finding that those who hold generally conservative ideologies tend to be more accepting towards anti-Black or racially prejudiced content Goldstein and Hall (2017); Lucks (2020); Schaffner (2020).

In the context of toxicity annotation and detection, our findings highlight the need to consider the attitudes of annotators towards free speech, racism, and their beliefs on the harms of hate speech, for an accurate estimation of anti-Black language as toxic, offensive, or racist (e.g., by actively taking into consideration annotator ideologies; Waseem (2016); Vidgen et al. (2021). This can be especially important given that hateful content very often targets marginalized groups and racial minorities Silva et al. (2016); Sap et al. (2020), and can catalyze violence against them O'Keeffe et al. (2011); Cleland (2014).

## 5 Who finds AAE posts toxic, and why?

African American English (AAE) is a set of well-studied varieties or dialects of U.S. English, common among, but not limited to, African-American or Black speakers Green (2002); Edwards (2004). This category has been shown to be considered "worse" English by non-AAE speakers Hilliard (1997); Blake and Cutler (2003); Champion et al. (2012); Beneke and Cheatham (2015); Rosa and Flores (2017), and is often mistaken as obscene or toxic by humans and AI models Spears et al. (1998); Sap et al. (2019), particularly due to dialect-specific lexical markers (e.g., words, suffixes).

Based on prior work that correlates racial prejudice with negative attitudes towards AAE Gaidher et al. (2015); Rosa (2019), we hypothesize that annotators who are white and who score high in RacistBelieffs will rate AAE posts as more toxic. Additionally, since AAE can be considered non-canonical English Sapolsky et al. (2010); DeFrank and Kahlbaugh (2019), we hypothesize that annotators who are more conservative and who score higher in Traditionalism and LingPurisim will rate AAE posts with higher toxicity.

### Breadth-of-Workers Results

Table 5 shows significant associations between annotator identities and beliefs and their ratings of toxicity of AAE posts. Partially confirming our hypothesis, we found that ratings of racism had somewhat significant correlations with annotators' conservative political leaning, and their scores on our RacistBelieffs scale. However, contrary to our expectations, we found that white and Black annotators did not differ in how offensive they rated AAE tweets (\(d=0.14\), \(p>0.1\)). We found no additional hypothesized or exploratory associations for racism ratings, and no significant associations for offensiveness ratings.

### Breadth-of-Posts Results

Shown in Table 6, our results for AAE and potentially vulgar breadth-of-posts study show higher offensiveness ratings from conservative raters, and those who scored higher in Traditionalism and, almost significantly, RacistBelieffs. We also find that conservative annotators and those who scored higher in FreeOffSpeech (and near-significantly, Traditionalism) rated AAE posts as more racist.

\begin{table}
\begin{tabular}{l|c c} _AAE posts_ & _Rated as Racist_ \\ \hline RacusBelieffs & \(r=0.089\) * \\ Politics _(ib: 0, conv: 1)_ & \(r=0.076\) † \\ \end{tabular}
\end{table}
Table 5: Associations between ratings of racism and annotor variables, for the _AAE_ posts from the _breadth-of-workers_ study. As with the previous results, we correct for multiple comparisons for non-hypothesized associations and only show significant results (†: \(p<0.075\), *: \(p<0.05\)).



As an additional investigation, we measure whether attitudes or identities affects toxicity ratings of AAE posts that contain the word "_n*gga_," a (reclaimed) slur that has very different pragmatic interpretations depending on speaker and listener identity Croom (2011). Here, we find that raters who are more conservative tended to score those posts as significantly more racist (\(\beta=0.465,p=0.003\); corrected for multiple comparisons).

### Perceived Toxicity of AAE

Our findings suggest that annotators perceive that AAE posts are associated with the Black racial identity Rosa (2019), which could cause those who score highly on the RacistBeliefs scale to annotate them as racist, potentially as a form of colorblind racism (e.g., where simply mentioning race is considered racist; Bonilla-Silva (2006). Moreover, specific markers of AAE could have been perceived as obscene by non-AAE speakers Spears et al. (1998), even though some of these might be reclaimed slurs (e.g., "n*gga"; Croom (2011); Galinsky et al. (2013)). Contrary to expectations, annotators' own racial identity did not affect their ratings of AAE posts in our studies. Future work should investigate this phenomenon further, in light of the variation in perceptions of AAE within the Black community Rahman (2008); Johnson et al. (2022), and the increased acceptance and usage of AAE by non-Black people in social media Ilbury (2020); Ueland (2020).

These findings shed some light on the racial biases found in hate speech detection Davidson et al. (2019); Sap et al. (2019), partially explaining why AAE is perceived as toxic. Based on our results, future work in toxic language detection should account for this over-estimation of AAE as racist. For example, annotators could explicitly include speakers of AAE, or those who understand that AAE or its lexical markers are not inherently toxic, or are primed to do so Sap et al. (2019). Avoiding an incorrect estimation of AAE as toxic is crucial to avoid upholding racing-linguistic hierarchies and thus representational harms against AAE speakers Rosa (2019); Blodgett et al. (2020).

## 6 Who finds vulgar posts toxic, and why?

Vulgarity can correspond to non-identity referring swearwords (e.g., _f*ck_, _sh*t_; denoted as OnI) or identity-referring slurs (e.g., _b*th_, _n*gga_; denoted as OI). Both types of vulgarity can be mistaken for toxic despite also having non-hateful usages (e.g., to indicate emotion or social belonging: Croom (2011); Dynel (2012); Galinsky et al. (2013).

Given that vulgarity can be considered non-canonical or impolite language Jay and Janschewitz (2008); Saopolsky et al. (2010); DeFrank and Kahlbaugh (2019), we hypothesize that annotators who score high on LingPurism, Traditionalism, and who are more conservative will rate vulgar posts as more offensive. Importantly, here, we focus on the posts that are exclusively vulgar (ONI) from only our breadth-of-workers study, to avoid confounding effects of vulgar posts with anti-Black meaning or in AAE (both of those cases were analyzed in SS4.2 and SS5.2). We refer the reader to Appendix F for the results on vulgar posts in the breadth-of-posts study.

### Breadth-of-Workers Results

Confirming our hypotheses, we found that offensiveness ratings of vulgar (OnI) posts indeed correlated with annotators' Traditionalism and LingPurism scores, and conservative political leaning (Table 7). We found no associations between attitudes and racism ratings for vulgar posts.

\begin{table}
\begin{tabular}{l|r||c} _AAE posts_ & _Rated as Offensive_ & _Rated as Racist_ \\ \hline FreeOtrSpeech & _n.s._ & **0.217  \\ RacistBeliefs & 0.133  & _n.s._ \\ Traditionalsim & 0.137  & 0.110  \\ \end{tabular}
\end{table}
Table 6: Associations between _AAE_ (and potentially also vulgar) post ratings from the _breadth-of-posts_ study and annotator variables, shown as the \(\beta\) coefficients from a mixed effects model with a random effect for each annotator. We only show significant results (†: \(p\) ¡ 0.075, *: \(p\) ¡ 0.05, **: \(p\) ¡ 0.001; Holm-corrected for multiple comparisons; _n.s._: not significant).

\begin{table}
\begin{tabular}{l|l} _Vulgar (OnI) posts_ & _Rated as Offensive_ \\ \hline LisePurnism & \(r=0.106\)  \\ Traditionalsim & \(r=0.252\)  \\ Politics _(hh: 0, cons. 1)_ & \(r=0.171\)  \\ \end{tabular}
\end{table}
Table 7: Associations between toxicity ratings and annotator variables for the _vulgar_ posts from the _breadth-of-workers_ study. We correct for multiple comparisons for non-hypothesized associations and only show significant results (�: \(p\) ¡ 0.05, **: \(p\) ¡ 0.001).

 



### Perceived Toxicity of Vulgar Language

Our findings corroborate prior work showing how adherence to societal traditional values is often opposed to the acceptability of vulgar language (Sapolsky et al., 2010). Traditional values and conservative beliefs have been connected to finding vulgar language as a direct challenging the moral order Jay (2018); Sterling et al. (2020); Muddiman (2021). Our results suggest that vulgarity is a very specific form of offensiveness that deserves special attention. Specifically, future work might consider studying the specific toxicity of individual identity-referring vulgar (OI) words, which can carry prejudiced meaning as well (e.g., slurs such as "_n*gg*r_"). Moreover, annotators across different levels of traditionalism could be considered when collecting ratings of vulgarity, especially since perceptions might vary with generational and cultural norms Dynel (2012).

## 7 Toxicity Detection System Case Study: PerspectiveAPI

Our previous findings indicated that there is strong potential for annotator identities and beliefs to affect their toxicity ratings. We are additionally interested in how this influences the behavior of toxicity detection models trained on annotated data. We present a brief case study to answer this question with the PerspectiveAPI,9 a widely used, commercial system for toxicity detection. Appendix G provides a more in-depth description.

Footnote 9: www.perspectiveapi.com

We investigate whether PerspectiveAPI scores align with toxicity ratings from workers with specific identities or attitudes, using the 571 posts from our breadth-of-posts study. Specifically, we compare the correlations between PerspectiveAPI scores and ratings from annotators, broken down by annotators with different identities (e.g., men and women) or with higher or lower scores on attitude scales (split at the mean). See Appendix G.1 for details about this methodology.

Our investigation shows that Perspective scores can be significantly more aligned with ratings from certain identities or groups scoring higher or lower on attitude dimensions (see Table 12 in Appendix G.2). Our most salient results show that for anti-Black posts, Perspective scores are somewhat significantly more aligned with racism ratings by annotators who score high in RacistBelieffs (\(r_{\text{high}}\) = 0.29, \(r_{\text{low}}\) = 0.17, \(\Delta r\) = 0.12, \(p\) = 0.056; Figure 2). Additionally, for AAE posts, Perspective scores are slightly more correlated with racism ratings by annotators who were women (\(\Delta r\) = 0.22, \(p\) < 0.001) or white (\(\Delta r\) = 0.08, \(p\) = 0.07), and who scored higher in LingPurism (\(\Delta r\) = 0.14, \(p\) = 0.003) or Traditionalism (\(\Delta r\) = 0.10, \(p\) = 0.030).

Overall, our findings indicate that PerspectiveAPI toxicity score predictions align with specific viewpoints or ideologies, depending on the text category. Particularly, it seems that the API underestimates the toxicity of anti-Black posts in a similar way to annotators who scored higher on the RacistBelieffs scale, and aligns more with white annotator's perception of AAE toxicity (vs. Black annotators). This corroborate prior findings that show that toxicity detection models inherently encode a specific positionality Cambo (2021) and replicate human biases Davani et al. (2021).

## 8 Discussion & Conclusion

Overall, our analyses showed that perceptions of toxicity are indeed affected by annotators' demographic identities and beliefs (SS2). We found--via a breadth-of-workers study and a breadth-of-posts study (SS3)--several associations when isolating specific text characteristics: anti-Black (SS4), AAE (SS5), and vulgarity (SS6). Finally, we showed that a popular toxicity detection system yields toxicity scores that are more aligned with raters with certain attitudes and identities than others (SS7). We discuss implications of our findings below.

**Variation in toxicity ratings in hate speech datasets.** In our study we deliberately sought rat

Figure 2: Correlation between PerspectiveAPI toxicity scores and racism ratings for anti-Black posts, broken down by by participants scoring high and low in RacistBelieffs.



ing of _perceptions_ of toxicity of posts by racially and politically diverse participants. However, many existing hate speech datasets instructed annotators to adhere to detailed definitions of toxicity Davidson et al. (2017); Founta et al. (2018), and some even selected crowdworkers for their liberal ideology Waseem (2016); Sap et al. (2020); Vidgen et al. (2021). While those annotation setups and annotator homogeneity could cause less variation in toxicity annotations of anti-Black, AAE, and vulgar posts, there is still empirical evidence of anti-AAE racial biases in some of these datasets Sap et al. (2019); Davidson et al. (2019).

Given the large variation in perceptions of toxicity that we showed and the implicit encoding of perspectives by toxicity models, we recommend researchers and dataset creators investigate and report annotator attitudes and demographics; researchers could collect attitude scores based on relevant social science research, perhaps in lightweight format as done in our breadth-of-posts study, and report those scores along with the dataset (e.g., in datasheets; Gebru et al. (2018).

Contextualize toxicity predictions in social variablesAs shown in our results and previous studies (e.g., Waseem, 2016; Ross et al., 2017; Waseem et al., 2021), determining what is toxic is subjective. However, given this subjectivity, the open question remains: _whose perspective should be considered when using toxicity detection models?_ To try answering this question, we urge researchers and practitioners to consider all stakeholders and end users on which toxicity detection systems might be deployed (e.g., through human-centered design methods; Sanders (2002); Friedman et al. (2008); Hovy and Yang (2021). While currently, the decision of content moderation often solely lies in the hands of the platforms, we encourage the exploration of alternative solutions (e.g., community fact checkers, digital juries; Maronikolakis et al., 2022; Gordon et al., 2022).

In general, we urge people to embrace that each design decision has socio-political implications Green (2020); Cambo (2021), and encourage them to develop technologies to shift power to the targets of oppression Blodgett et al. (2020); Kalluri (2020); Birhane (2021). Finally, given the increasingly essential role of online platforms in people's daily lives Rahman (2017), we echo calls for policy regulating online spaces and toxicity detection Jiang (2020); Benesch (2020); McGuffie and Newhouse (2020); Gillespie et al. (2020).

Beyond toxicity classification: modeling distributions and generating explanationsOur findings on the subjectivity of the toxicity detection tasks suggests that standard approaches of obtaining binary (or even \(n\)-ary) labels of toxicity and averaging them into a majority vote are inadequate. Instead, researchers could consider modeling the distribution or variation in toxicity labels with respect to individual annotators Geva et al. (2019); Fornaciari et al. (2021); Davani et al. (2021) or to specific identities or beliefs.

But, perhaps more importantly, we encourage re-thinking the toxicity detection paradigm altogether. With the goal to assist human content moderators,10 creating systems that explain biased implications of posts could be more helpful than opaque toxicity scores Thus, we advocate for moving away from classification frameworks, and towards more nuanced, holistic, and explainable frameworks for inferring the desired concepts of toxicity and social biases (e.g., Social Bias Frames; Sap et al., 2020).

Limitations and open questionsOur work had several limitations and raised several open research questions, some of which we outline below. First, our particular choices of attitudes and scales could affect our results; other scales (e.g., Gerdes et al., 2011, for measuring empathy) as well as other psychological variables (e.g., propensity to volunteer or to value dignity) could be studied in the context of toxicity perceptions. Additionally, the automatic AAE detector in the breadth-of-posts study could have induced data selection biases, despite being strongly correlated with race-aware dialect detection (as noted in footnote 6). Furthermore, our analysis of the attitudes encoded in the PerspectiveAPI in SS7 was merely a pilot study; we hope future work will explore more in-depth methods for assess model positionality.

Footnote 10: Note that while content moderation can induce significant psychological harms in moderators Roberts (2017); Steiger et al. (2021), full automation also has significant risks.

While our study focused on racial discrimination by studying AAE and anti-Black posts, future work should explore other axes of discrimination (e.g., sexism, homophobia, ableism, etc.). Additionally, our study focused only on U.S.-centric perspectives; we hope researchers will explore variations in toxicity perceptions in other cultural contexts (e.g., variati 



## References

* Akhtar et al. (2021) Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021. Whose opinions matter? perspective-aware models to identify opinions of hate speech victims in abusive language detection. ArXiv preprint arXiv:2106.15896.
* Kuwatly et al. (2020) Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020. Identifying and measuring annotator bias based on annotators' demographic characteristics. In _Proceedings of the Fourth Workshop on Online Abuse and Harms_.
* Are (2020) Carolina Are. 2020. How instagram's algorithm is censoring women and vulnerable users but helping online abusers. _Feminist media studies_, 20(5).
* examining the challenges with annotating toxic text datasets.
* Barocas et al. (2017) Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Allocative versus representational harms in machine learning. In _SIGCIS_.
* Beneke and Cheatham (2015) Margaret Beneke and Gregory A Cheatham. 2015. Speaking up for african american english: Equity and inclusion in early childhood settings. _Early Childhood Education Journal_, 43(2).
* Benesch (2020) Susan Benesch. 2020. Proposals for improved regulation of harmful online content. Technical report.
* Birhane (2021) Abeba Birhane. 2021. Algorithmic injustice: a relational ethics approach. _Patterns (New York, N.Y.)_, 2(2).
* Blake and Cutler (2003) Renee Blake and Cecilia Cutler. 2003. AAE and variation in teachers' attitudes: A question of school philosophy? _Linguistics and education_, 14(2).
* Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daume III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
* Blodgett et al. (2016) Su Lin Blodgett, Lisa Green, and Brendan O'Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_.
* Bonilla-Silva (2006) Eduardo Bonilla-Silva. 2006. _Racism without racists: Color-blind racism and the persistence of racial inequality in the United States_.
* Bouchard Jr. and McGue (2003) Thomas J. Bouchard Jr. and Matt McGue. 2003. Genetic and environmental influences on human psychological differences. _Journal of Neurobiology_, 54(1).
* Breitfeller et al. (2019) Luke Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019. Finding microaggressions in wild: A case for locating elusive phenomena in social media posts. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
* Burnham et al. (2018) Martin J Burnham, Yen K Le, and Ralph L Piedmont. 2018. Who is mturk? personal characteristics and sample consistency of these online workers. _Mental Health, Religion & Culture_, 21(9-10).
* Cambo (2021) Scott Allen Cambo. 2021. _Model Positionality: A Novel Framework for Data Science with Subjective Target Concepts_. Ph.D. thesis, Northwestern University.
* Carter and Murphy (2015) Evelyn R Carter and Mary C Murphy. 2015. Group-based differences in perceptions of racism: What counts, to whom, and why? _Social and personality psychology compass_, 9(6).
* Champion et al. (2012) Tempii B Champion, Deirdre Cobb-Roberts, and Linda Bland-Stewart. 2012. Future educators' perceptions of african american verncacular english (AAVE). _Online Journal of Education Research_, 1(5).
* Cleland (2014) Jamie Cleland. 2014. Racism, football fans, and online message boards: How social media has added a new dimension to racist discourse in English football. _J. Sport Soc. Issues_, 38(5).
* Cole (1996) D Cole. 1996. Racist speech should be protected by the constitution. _Hate crimes_.
* Cowan and Khatchadourian (2003) Gloria Cowan and Desiree Khatchadourian. 2003. Empathy, ways of knowing, and interdependence as mediators of gender differences in attitudes toward hate speech and freedom of speech. _Psychology of Women Quarterly_, 27(4).
* Cowan et al. (2002) Gloria Cowan, Miriam Resendez, Elizabeth Marshall, and Ryan Quist. 2002. Hate speech and constitutional protection: Priming values of equality and freedom. _The Journal of social issues_, 58(2).
* Croom (2011) Adam M Croom. 2011. Slurs. _Language Sciences_, 33(3).
* Davani et al. (2021) Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy, and Morteza Dehghani. 2021. Hate speech classifiers learn human-like social stereotypes.
* Davidson et al. (2019) Thomas Davidson, Debasmitta Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In _Proceedings of the Third Workshop on Abusive Language Online_.
* Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In _ICWSM_

Melanie DeFrank and Patricia Kahlbaugh. 2019. Language choice matters: When profainty affects how people are judged. _Journal of Language and Social Psychology_, 38(1).
* Dinan et al. (2019) Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
* Downs and Cowan (2012) Daniel M Downs and Gloria Cowan. 2012. Predicting the importance of freedom of speech and the perceived harm of hate speech. _Journal of applied social psychology_, 42(6).
* Duckitt and Fisher (2003) John Duckitt and Kirstin Fisher. 2003. The impact of social threat on worldview and ideological attitudes. _Political Psychology_, 24(1).
* Dynel (2012) Marta Dynel. 2012. Swearing methodologically : the (im)politicess of expletives in anonymous commentaries on youtube. _Journal of English Studies_, 10(0).
* Edwards (2004) Walter F Edwards. 2004. African american vernacular english: phonology. In _A Handbook of Varieties of English: Morphology and syntax_.
* Elers and Jayan (2020) Christine Helen Elers and Pooja Jayan. 2020. "this is us": Free speech embedded in whiteness, racism and coloniality in aotearoa, new zealand. _First Amendment Studies_, 54(2).
* Fiske (1993) S T Fiske. 1993. Controlling other people. the impact of power on stereotyping. _The American psychologist_, 48(6).
* Formaciari et al. (2021) Tommaso Formaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poseio. 2021. Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
* Founta et al. (2018) Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. In _ICWSM_.
* Friedman et al. (2008) Batya Friedman, Peter H Kahn, and Alan Borning. 2008. Value sensitive design and information systems. _The handbook of information and computer ethics_.
* Gaither et al. (2015) Sarah E Gaither, Ariel M Cohen-Goldberg, Calvin L Gidney, and Keith B Maddox. 2015. Sounding black or white: Priming identity and biracial speech. _Frontiers in Psychology_, 6.
* Galinsky et al. (2013) Adam D Galinsky, Cynthia S Wang, Jennifer A Whitsson, Eric M Anicich, Kurt Hugenberg, and Galen V Bodenhausen. 2013. The reappropriation of stigmatizing labels: the reciprocal relationship between power and self-labeling. _Psychol. Sci._, 24(10).
* Gavrilets and Fortunato (2014) Sergey Gavrilets and Laura Fortunato. 2014. A solution to the collective action problem in beta-group conflict with within-group inequality. _Nature communications_, 5(1).
* Gebru et al. (2018) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daume III, and Kate Crawford. 2018. Datashheets for datasets. In _FAccT_*.
* Gerdes et al. (2011) Karen E Gerdes, Cynthia A Lietz, and Elizabeth A Segal. 2011. Measuring empathy in the 21st century: Development of an empathy index rooted in social cognitive neuroscience and social justice. _Social Work Research_, 35(2).
* Geva et al. (2019) Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
* Gillborn (2009) David Gillborn. 2009. Risk-free racism: Whiteness and so-called free speech. _Make Forest law review_, 44.
* Gillespie et al. (2020) Tarleton Gillespie, Patricia Aufderheide, Elinor Carmi, Ysabel Gerard, Robert Gorwa, Ariadna Matamoros-Fernandez, Sarah T Roberts, Aram Sinmreich, and Sarah Myers West. 2020. Expanding the debate about content moderation: Scholarly research agendas for the coming policy debates. _Internet Policy Review_, 9(4).
* Goldstein and Hall (2017) Donna M Goldstein and Kira Hall. 2017. Postelection surrealism and nostalgic racism in the hands of donald trump. _HAU: Journal of Ethnographic Theory_, 7(1).
* Gordon et al. (2022) Mitchell L Gordon, Michelle S Lam, Jeon Sung Park, Kayur Patel, Jeffrey T Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022. Jury learning: Integrating dissenting voices into machine learning models. In _CHI_.
* Green (2020) Ben Green. 2020. Data science as political action: Grounding data science in a politics of justice. _SSRN Electronic Journal_.
* Green (2002) Lisa Green. 2002. _African American English: A Linguistic Introduction_, 8.3.2002 edition edition.
* Greengross and Miller (2008) Gil Greengross and Geoffrey F Miller. 2008. Dissing oneself versus missing rivals: effects of status, personality, and sex on the Short-Term and Long-Term attractiveness of Self-Deprecating and Other-Depreacting humor. _Evolutionary Psychology_, 6(3).

 

Xiaochuang Han and Yulia Tsvetkov. 2020. Fortifying toxic speech detectors against veiled toxicity. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
* Hilliard [1997] A G Hilliard. 1997. Language, culture and the assessment of african american children. Assessment for equity and inclusion: Embracing all our children.
* Hovy and Yang [2021] Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
* Huff and Tingley [2015] Connor Huff and Dustin Tingley. 2015. "who are these people?" evaluating the demographic characteristics and political preferences of mturk survey respondents. Research & Politics, 2(3).
* Ilbury [2020] Christian Ilbury. 2020. "sassy queens": Stylistic orthographic variation in twitter and the enregisterment of AAVE. Journal of sociolinguistics, 24(2).
* Jay [2018] Timothy Jay. 2018. Swearing, moral order, and online communication. Journal of Language Aggression and Conflict, 6(1).
* Jay and Janschewitz [2008] Timothy Jay and Kristin Janschewitz. 2008. The pragmatics of swearing. Journal of Politeness Research, 4.
* Jernudd and Shapiro [1989] Bjorn H Jernudd and Michael J Shapiro. 1989. The politics of language purism.
* Jiang [2020] J A Jiang. 2020. Identifying and addressing design and policy challenges in online content moderation. Extended Abstracts of the 2020 CHI Conference on.
* Jiang et al. [2021]Jialun Aaron Jiang, Morgan Klaus Scheuerman, Casey Fiesler, and Jed R Brubaker. 2021. Understanding international perceptions of the severity of harmful content online. PloS one, 16(8).
* Johnson et al. [2022] Darin G Johnson, Bradley D Mattan, Nelson Flores, Nina Luuharatanahirun, and Emily B Falk. 2022. Social-Cognitive and affective antecedents of code switching and the consequences of linguistic racism for black people and people of color. Affective Science, 3(1).
* Johnson and Tamney [2001] Stephen D Johnson and Joseph B Tamney. 2001. Social traditionalism and economic conservatism: Two conservative political ideologies in the united states. The Journal of social psychology, 141(2).
* Kalluri [2020] Pratyusha Kalluri. 2020. Don't ask if artificial intelligence is good or fair, ask how it shifts power. Nature, 583(7815).
* Knuckey [2005] Jonathan Knuckey. 2005. A new front in the culture war? moral traditionalism and voting behavior in us house elections. American Politics Research, 33(5).
* Loepp and Kelly [2020] Eric Loepp and Jarrod T Kelly. 2020. Distinction without a difference? an assessment of mturk worker types. Research & Politics, 7(1).
* Lucks [2020] Daniel S Lucks. 2020. _Reconsidering Reagan: Racism, Republicans, and the Road to Trump_.
* Maronikolakis et al. [2022] Antonis Maronikolakis, Axel Wisiorek, Leah Nann, Haris Jabbar, Sahana Udupa, and Hinrich Schutze. 2022. Listening to affected communities to define extreme speech: Dataset and experiments. In ACL 2022 Findings.
* McConahay [1986] John B McConahay. 1986. Modern racism, ambivalence, and the modern racism scale. In Prejudice, discrimination, and racism, volume 337.
* McGuffie and Newhouse [2020] Kris McGuffie and Alex Newhouse. 2020. The radicalization risks of gpt-3 and advanced neural language models.
* Muddiman [2021] Ashley Muddiman. 2021. Conservatives and incivility. In Conservative Political Communication.
* Nadal [2018] Kevin L Nadal. 2018. _Microaggressions and traumatic stress: Theory, research, and clinical treatment_.
* Nelson et al. [2013] Jessica C Nelson, Glenn Adams, and Phia S Salter. 2013. The marley hypothesis: Denial of racism reflects ignorance of history. Psychological science, 24(2).
* Norton and Sommers [2011] Michael I Norton and Samuel R Sommers. 2011. Whites see racism as a Zero-Sum game that they are now losing. Perspectives on psychological science: a journal of the Association for Psychological Science, 6(3).
* O'Keeffe et al. [2011] Gwenn Schurgin O'Keeffe, Kathleen Clarke-Pearson, and Council on Communications and Media. 2011. The impact of social media on children, adolescents, and families. Pediatrics, 127(4).
* Pavlick and Kwiatkowski [2019] Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7.
* Poteat and Spanierman [2012] V Paul Poteat and Lisa B Spanierman. 2012. Modern racism attitudes among white students: The role of dominance and authoritarianism and the mediating effects of racial color-blindness. The Journal of Social Psychology, 152(6).
* Prabhakaran et al. [2021] Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing Annotator-Level labels and information in datasets. In Proc. of LAW-DMR workshop at EMNLP.
* Pulos et al. [2004] Steven Pulos, Jeff Elison, and Randy Lennon. 2004. The hierarchical structure of the interpersonal reactivity index. Social behavior and personality, 32(4).
* Rahman [2008] Jacquelyn Rahman. 2008. Middle-class african americans: Reactions and attitudes toward african american english. American speech, 83(2).

 

K Sabeel Rahman. 2017. The new utilities: Private power, social infrastructure, and the revival of the public utility concept. _Cardozo law review_, 39.
* Rier et al. (2020) Marc Riar, Benedikt Morschheuser, Juho Hamari, and Rudiger Zarnekow. 2020. How game features give rise to altruism and collective action? implications for cultivating cooperation by gamification. In _Proceedings of the 53rd Hawaii International Conference on System Sciences_.
* Roberts (2017) Sarah T Roberts. 2017. Social media's silent filter. _The Atlantic_.
* Rosa (2019) Jonathan Rosa. 2019. _Looking Like a Language, Sounding Like a Race_.
* Rosa and Flores (2017) Jonathan Rosa and Nelson Flores. 2017. Unsettling race and language: Toward a racialophinguistic perspective. _Language In Society_, 46(5).
* Ross et al. (2017) Bjorn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2017. Measuring the reliability of hate speech annotations: the case of the european refugee crisis. In _NLP 4 CMC Workshop_.
* Rousso and Dovidio (2018) Gina Roussos and John F Davidio. 2018. Hate speech is in the eye of the beholder: The influence of racial attitudes and freedom of speech beliefs on perceptions of racially motivated threats of violence. _Social psychological and personality science_, 9(2).
* Sanders (2002) Elizabeth Sanders. 2002. _From user-centered to participatory design approaches_.
* Sap et al. (2019) Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
* Sap et al. (2020) Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
* Sapolsky et al. (2010) Barry S. Sapolsky, Daniel M. Shafer, and Barbara K. Kaye. 2010. Rating offensive words in three television program contexts. _Mass Communication and Society_, 14(1).
* Schaffner (2020) Brian F Schaffner. 2020. The heightened importance of racism and sexism in the 2018 US midterm elections. _British journal of political science_.
* Silva et al. (2016) Leandro Silva, Mainack Mondal, Denzil Correa, Fabricio Benevenuto, and Ingmar Weber. 2016. Analyzing the targets of hate in online social media. In _Tenth international AAAI conference on web and social media_.
* Silver and Dunlap (1987) N Clayton Silver and William P Dunlap. 1987. Averaging correlation coefficients: should fisher's z transformation be used? _Journal of applied psychology_, 72(1).
* Soral et al. (2018) Wiktor Soral, Michal Bilewicz, and Mikolaj Winiewski. 2018. Exposure to hate speech increases prejudice through desensitization. _Aggressive behavior_, 44(2).
* Spears et al. (1998) Arthur K Spears et al. 1998. African-american language use: Ideology and so-called obscenity. _African-American English: Structure, history, anduse_.
* Steg et al. (2014) Linda Steg, Ooda Perlaviciute, Ellen Van der Werff, and Judith Lavrink. 2014. The significance of hedonic values for environmentally relevant attitudes, preferences, and actions. _Environment and behavior_, 46(2).
* Steiger et al. (2021) Miriah Steiger, Timir J Bharucha, Sukrit Venkatagiri, Martin J Riedl, and Matthew Lease. 2021. The psychological Well-Being of content moderators: The emotional labor of commercial moderation and avenues for improving support. In _CHI_, number Article 341 in CHI '21.
* Sterling et al. (2020) Joanna Sterling, John T Jost, and Richard Bonneau. 2020. Political psycholinguistics: A comprehensive analysis of the language habits of liberal and conservative social media users. _Journal of personality and social psychology_, 118(4).
* Ueland (2020) Ane Ueland. 2020. _Language and identity: a study of African American Vernacular English and its status in American society_. Ph.D. thesis, University of Stavanger, Norway.
* Vidgen et al. (2021) Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021. Learning from the worst: Dynamically generated datasets to improve online hate detection. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_.
* Wagstaff (1998) Graham F Wagstaff. 1998. Equity, justice, and altruism. _Current Psychology_, 17(2).
* Waseem (2016) Zeerak Waseem. 2016. Are you a racist or am I seeing things? annotator influence on hate speech detection on Twitter. In _Proceedings of the First Workshop on NLP and Computational Social Science_.
* Waseem and Hovy (2016) Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on Twitter. In _Proceedings of the NAACL Student Research Workshop_.
* Waseem et al. (2021) Zeerak Waseem, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein. 2021. Disembodied machine learning: On the illusion of objectivity in nlp. Anonymous preprint under review 



## References

* [1] A. B. Kibble, Phys. Rev. D **49**, 2378 (1994); M. C. Bowers and R. J. Glauber, Phys. Lett. B **50**, 63 (1994); S. G. Fadin, Nucl. Phys. B **403**, 1 (1994); E. Brezin and L. I. McLerran, Phys. Rev. D **49**, 2378 (1994); M. C. Bowers and R. J. Glauber, Phys. Rev. D **49**, 2378 (1994); M. C. Bowers, R. J. Glauber, Phys. Rev. D **49**, 2378 (1994); M. C. Bowers, R. J. Glauber, Phys. Rev. D **49**, 2378 (1994); M. C. Bowers, R. J. Glauber, Phys. Rev. D **49**, 2381 (1994); M. C. Bowers, R. J. Glauber, Phys. Rev. D **49**, 2390 (1994); M. C. Bowers, R. J. Glauber, Phys. Rev. D **49**, 2491 (1994); M. C. Bowers, R. J. Glauber, Phys. Rev. D **49**, 2492 (1994); M. C. Bowers, R. J. Glauber, Phys. Rev. D **49**, 2493 (1994); M. C. Bowers, R. J. Glauber, Phys. Rev. D **49**, 2493 (1994); M. C. Bowers, R. J. Glauber, Phys. Rev. D **49**, 2494 (1994); M. C. Bowers, R. J. Glauber, Phys. Rev. D **49**, 2493 (1994); M. C. Bowers, R. J. Glauber, Phys. Rev. D **49**, 2494 (1994); M. C. Bowers, R. J. Glauber, Phys.





## References

* [1] A. Datta, Phys. Rev. Lett. **68**, 2390 (1997); M. Beneke and G. F. Giudice, Nucl. Phys. **B450**, 1 (1997); J. C. Collins and R. S. Thorne, Phys. Rev. **D48**, 1 (1993); E. Eichten, Phys. Rev. **D49**, 1 (1994); L. Frankfurt, Phys. Rev. **D49**, 1 (1994); W. H. Freeman, Z. Phys. **C32**, 1 (1994); T. Hahn, Phys. Rev. **D49**, 1 (1994); K. Goulianos, Phys. Rev. **D50**, 1 (1994); V. Balitsky, Phys. Rev. **D51**, 1 (1995); P. Hoods, Phys. Rev. **D52**, 1 (1995); M. Beneke, G. F. Giudice, Phys. Rev. **D53**, 1 (1996); M. Beneke, G. F. Giudice, Phys. Rev. **D54**, 1 (1996); M. Beneke, G. F. Giudice, Phys. Rev. **D55**, 1 (1997); M. Beneke, G. F. Giudice, Phys. Rev. **D56**, 1 (1997); M. Beneke, G. F. Giudice, Phys. Rev. **D57**, 1 (1998); M. Beneke, G. F. Giudice, Phys. Rev. **D58**, 1 (1998); M. Beneke, G. F. Giudice, Phys. Rev. **D59**, 1 (1999); M. Beneke, G. F. Giudice, Phys. Rev. **D59**, 1 (1999); M.





## References

* [1] A. Datta, Phys. Rev. Lett. **68**, 2390 (1997); M. Beneke and G. F. Giudice, Nucl. Phys. **B450**, 1 (1997); J. C. Collins and R. S. Thorne, Phys. Rev. **D48**, 1 (1993); E. Eichten, Phys. Rev. **D49**, 1 (1994); L. Frankfurt, Phys. Rev. **D49**, 1 (1994); W. H. Freeman, Z. Phys. **C32**, 1 (1994); T. Hahn, Phys. Rev. **D49**, 1 (1994); K. Goulianos, Phys. Rev. **D50**, 1 (1994); V. Balitsky, Phys. Rev. **D51**, 1 (1995); P. Hoods, Phys. Rev. **D52**, 1 (1995); M. Beneke, G. F. Giudice, Phys. Rev. **D53**, 1 (1996); M. Beneke, G. F. Giudice, Phys. Rev. **D54**, 1 (1996); M. Beneke, G. F. Giudice, Phys. Rev. **D55**, 1 (1997); M. Beneke, G. F. Giudice, Phys. Rev. **D56**, 1 (1997); M. Beneke, G. F. Giudice, Phys. Rev. **D57**, 1 (1998); M. Beneke, G. F. Giudice, Phys. Rev. **D58**, 1 (1998); M. Beneke, G. F. Giudice, Phys. Rev. **D59**, 1 (1999); M. Beneke, G. F. Giudice, Phys. Rev. **D59**, 1 (1999); M.



sociated with higher scores on the Empathy, HarmOfHateSpeech, and Altruism scales, whereas conservative political leaning was associated with higher scores on the Traditionalism, LingPurism, FreeOffSpeech and RacistBeliefs scales.

## Appendix B Toxicity questions

Following crowdsourcing setups in prior work (Waseem, 2016; Davidson et al., 2017; Wulczyn et al., 2017; Founta et al., 2018; Sap et al., 2019), we asked three fine-grained questions to annotators for each post in both our studies:

* "How toxic/hateful/disrespectful or offensive does this post seem _to you?_"
* "How much could this post be seen as toxic/hateful/disrespectful or offensive _to anyone?_"
* "In your opinion, how _racist_ is this post?" Given the high correlations between the two offensiveness variables (Pearson \(r\geq.7\); Table 9), we use create an _offensiveness_ ("_off_:") score by taking the average rating given to the "to you" and "to anyone" questions. In all our analyses, we use that _offensiveness_ score, along with the raw _racism_ score.

## Appendix C Small-scale controlled study details

### Data Selection & Validation

We aimed to select online posts that were very indicative of each of the above characteristics (vullgar, AAE, anti-Black) but not indicative of the others, in order to tease out the effect of that category. We selected vulgar and AAE posts from a publicly available large corpus of tweets annotated for hate speech by Founta et al. (2018).11 For each tweet in that corpus, we detected the presence of nonidentity related profanity or swearwords using the list from Zhou et al. (2021), and extracted the likelihood that the tweet is in AAE using a lexical detector by Blodgett et al. (2016). As candidates, we selected 10 vulgar tweets that have low likelihood of being AAE, and 26 tweets that have high likelihood of being AAE but contain no vulgarity. For anti-Black posts, we selected 11 candidate online posts curated by Zevallos (2017).

Footnote 11: We only used the training subset of the corpus.

We ran a human validation study to verify that the candidate posts are truly indicative of their respective categories. We created an annotation scheme to collect binary ratings for two questions per post: "does it contain vulgar language", and "is it offensive to minorities"; a post could belong to either category or neither. Each post was manually annotated by three undergraduate research assistants trained for the task. Post validation, we manually selected 5 posts per category with perfect inter-annotator agreement. Table 1 lists the final 15 posts used for our study.

### Participant Recruitment

We ran our study on Amazon Mechanical Turk (MTurk), a crowdsourcing platform that is often used to collect offensiveness annotations.12 With the task at hand, we sought a racially and politically diverse pool of participants, which can be challenging given that MTurk workers are usually tend to be predominantly white and skew liberal Huff and Tingley (2015); Burnham et al. (2018); Loepp and Kelly (2020). Therefore, we ran a pre-selection survey to collect race and political ideology of workers, noting that this presurvey could grant them access to a longer survey on free speech, hate speech, and offensiveness in language.13\({}^{,}\)14 We stopped recruiting once we reached at least 200 Black and 200 conservative participants.

Footnote 12: Note, this study was approved by the author’s institutional review board (IRB).

Footnote 13: To better recruit for our pre-survey, we noted in the title that “BIPOC people and conservatives were encouraged to participate,” and also varied the title’s working to emphasize free speech or hate speech in different recruiting rounds.

Footnote 14: We compensated workers $0.02-$0.03 for this pre-survey.

### Study Setup

We ran our study on the widely used survey platform Qualtrics, using an MTurk HIT to recruit and compensate participants.15 Participants were first

\begin{table}
\begin{tabular}{c c|c c|c c|c c|c c|c c|c c}  & \multicolumn{3}{c|}{**Overall**} & \multicolumn{3}{c|}{**Overall**} \\ \hline \multirow{2}{*}{**Category**} & \multirow{2}{ 

asked to consent to the task, then were shown instructions for annotating the 15 posts (with occasional reminders of the instructions). Then we asked participants their views on several topics using the scales described in SS2.2 and SSA and finally their demographics. Throughout the study, we added three attention checks to ensure the quality of responses.

Allowing only Black, white liberal, and white conservative workers to participate, we ran our survey for 4 weeks from March 10 to April 5 2021, occasionally reminding participants from our preserve that they could take the survey.

## Appendix D Breadth-of-Posts annotation study details

### Data Selection

In this study, we draw from two existing corpora of posts labeled for toxicity, hate, or offensiveness. First, we select posts that are automatically detected as AAE and/or vulgar from Founta et al. (2018), using the lexical detector of AAE by Blodgett et al. (2016) and the vulgar wordlist by Zhou et al. (2021). Second, we select posts that are automatically detected as vulgar and/or annotated as anti-Black from Vidgen et al. (2021). Importantly, in this large-scale study, we consider posts that potentially have multiple characteristics (e.g., AAE and vulgar), and thus consider both posts with potentially offensive identity references (vulgar-OI) as well as non-identity vulgar words (vulgar-ONI). However, to circumvent potential racial biases in what is labelled as "racist" in the Vidgen et al. (2021) corpus Sap et al. (2019); Davidson et al. (2019), we do not consider posts that are annotated as anti-Black but detected as AAE.

Given an initial set of posts from our categories, we then randomly sample up to 600 posts, stratifying by toxicity label, vulgarity, AAE, and anti-Black meaning. Our final sample contains 571 posts, as outlined in Table 2 and Figure 5.

### Breadth-of-Posts Survey details

As in the breadth-of-workers study, we recruit participants using a pre-qualifying survey on MTurk. Then, we set up a second MTurk task to collect toxicity ratings, and annotator attitudes and identities. For each post, we collected two ratings from white conservative workers, two from white liberal workers, and two from Black workers. To better mirror the crowdsourcing setting and to reduce the annotator burden, we shorten the task to only ask one question per attitude (listed in SSA). We also asked one attention check question to ensure data quality.

For this study, our final dataset contains 3,171 ratings from \(N=173\) participants.16 Our participants were 53% were men, 45% women, and \(<\)2% non-binary, identified as 76% white, 20% Black, and \(<\)4% some other race, and spanned the political spectrum from 54% liberal to 30% conservative, with 16% centrists or moderates.

Footnote 16: As before, we discard 255 ratings where workers failed an attention check.

### Selecting Attitude Questions

In order to simplify the annotation task for annotators, we abridged the attitude scales to only one item. Using the data from the breadth-of-workers study, we select the question that best correlated with all toxicity ratings. Specifically, for each scale, we first take the tweet category with the highest correlation with toxicity (e.g., anti-Black posts for RacistBelieffs), and then take the item whose response scores correlated most with the toxicity rating for those posts. Those items are bolded in SSA.

## Appendix E Further Breadth-of-Workers Results

We show all associations between attitudes and toxicity ratings in Table 10.

Additionally, we investigate the differences in the overall toxicity ratings of anti-Black vs. AAE

Figure 4: Average ratings of offensiveness and racism for each tweet category in the breadth-of-workers controlled study. All differences are significant (\(p<0.001\)) after correcting for multiple comparisons.

 

vs. vulgar posts? (Figure 4). Overall, anti-Black tweets were rated as substantially more offensive and racist than AAE or vulgar tweets (with effect sizes ranging from \(d\) = 2.4 to \(d\) = 3.6). Additionally, vulgar tweets were rated as more offensive than AAE tweets (\(d\) = -0.29, \(p\) < 0.001).

Surprisingly, we also found that AAE tweets were considered slightly more racist than vulgar tweets (\(d\) = 0.19, \(p\) < 0.001). To further inspect this phenomenon, we performed exploratory analyses by computing the differences in ratings of racism for AAE and vulgar broken down by annotator gender, race, and political leaning. We found that AAE tweets were rated as significantly more racist than vulgar tweets only by annotators who were white or liberal (\(d\) = 0.20 and \(d\) = 0.22, respectively, with \(p\) < 0.001 corrected for multiple comparisons), compared to Black or conservative. There were no significant differences when looking at men and women separately.

## Appendix F Further Breadth-of-Posts Results

To account for the varying number of posts that each annotators could rate, we use a linear mixed effects model 17 to compute associations between each post's toxicity ratings and identities or attitudes. Specifically, we our linear model regresses the attitude score onto the toxicity score, with a random effect for each worker.18

Footnote 17: Using the Python statsmodels implementation.

Footnote 18: In R-like notation, toxicity\(\sim\)attitude+(l|WorkerId)!/the API was accessed in October 2021

See Figure 5 and Table 11.

See Figure 5 and Table 11.

## Appendix G PerspectiveAPI Case Study: Details & Results

### Details

We first obtain Perspective toxicity scores for all the posts in our breadth-of-posts study (SS3.2).19 Then, we split workers into two different groups for each of our attitudes and identity dimensions. For attitudes and political leaning, we assign each annotator to a "high" or "low" group based on whether they scored higher or lower than the mean score on that attitude scale. For gender and race, we use binary bins for man/woman and white/black.

Footnote 19: The API was accessed in October 2021

Then, for each attitude or identity dimension, we compute the Pearson \(r\) correlation between the Perspective score and the toxicity ratings from the high and low groups, considering posts from potentially overlapping categories (e.g., AAE and potentially vulgar posts).Finally, we compare the high and low correlations using Fisher's \(r\)-to-\(z\) transformation Silver and Dunlap [19].

Figure 5: Venn diagram of number of tweets in each of the categories.

 



### Results

See Table 12 and Figures 6-13.

\begin{table}
\begin{tabular}{l|c c|c||c|c|c|c|c|c}  & \multicolumn{4}{c|}{_Rated as Offensive_} & \multicolumn{5}{c}{_Rated as Racist_} \\ Anti-Black & AAE & Vulgar OI & Vulgar OnI & Anti-Black & AAE & Vulgar OI & Vulgar OnI \\ \hline Emparity & _n.s._ & _n.s._ & 0.168 \({}^{\dagger}\) & _n.s._ & _n.s._ & _n.s._ & _n.s._ & _n.s._ \\ Altvism & _n.s._ & _n.s._ & _n.s._ & _n.s._ & _n.s._ & _n.s._ & _n.s._ & _n.s._ \\ HammerOfListSpeech & 0.117 \({}^{\dagger}\) & _n.s._ & 0.169 \({}^{\dagger}\) & _n.s._ & 0.154 \({}^{*}\) & _n.s._ & _n.s._ & _n.s._ \\ FreeOtSpreech & _n.s._ & _n.s._ & _n.s._ & _n.s._ & 0.138 \({}^{+}\) & ** & ** & ** & ** \\ RacisBellities & 0.131 \({}^{*}\) & 0.133 \({}^{\dagger}\) & _n.x._ & _n.s._ & 0.185 \({}^{*}\) & ** & ** & ** & ** \\ LispPuxsm & _n.s._ & _n.s._ & _n.s._ & _n.s._ & _n.s._ & _n.s._ & _n.s._ & _n.k._ \\ Traditionalism & _n.s._ & 0.137 \({}^{*}\) & 0.152 \({}^{*}\) & _n.s._ & _n.s._ & 0.110 \({}^{\dagger}\) & _n.s._ & _n.s._ \\ Politics (_iik.: (a, cons.: 1)_ & _n.s._ & 0.143 \({}^{*}\) & _n.s._ & 0.134 \({}^{+}\) & _n.s._ & 0.206 \({}^{**}\) & 0.172 \({}^{*}\) & 0.196 \({}^{*}\) \\ Gender & _(non: q, women: l)_ & _n.s._ & _n.s._ & _n.s._ & _n.s._ & _n.s._ & _n.s._ & _n.k._ \\ Race & _(Whiuc. @ Black: l)_ & _n.s._ & _n.s._ & _n.s._ & _n.s._ & _n.s._ & _n.s._ & _n.k._ \\ \end{tabular}
\end{table}
Table 11: Associations between the annotator demographic and attitude variables and their ratings of offensiveness and racism on the posts from the breadth-of-posts study. We break down the results by category, but categories are overlapping. Only significant associations (\(\beta\) coefficients from a mixed effects model) are shown (\({}^{\dagger}\colon p<0.075\), \({}^{*}\): \(p<0.05\), \({}^{**}\colon p<0.001\); Holm-corrected for multiple comparisons).





\begin{table}
\begin{tabular}{l|c c||c|c|c|c|c|c|c|c}  & \multicolumn{4}{c|}{_Rated as Offensive_} & \multicolumn{3}{c|}{Rated as Racist} \\  & Anti-Black & AAE & Vulgar OI & Velgar OOI & Anti-Black & AAE & Vulgar OI & Valgar OII \\ \hline Empathy & \(n.x.\) & \(n.x.\) & 0.10 * low & \(n.s.\) & \(n.s.\) & \(n.s.\) & \(n.s.\) & \(n.s.\) & \(n.s.\) \\ Altevin & \(n.x.\) & 0.08 * low & 0.11 * low & \(n.s.\) & \(n.s.\) & 0.10 * low & \(n.s.\) & \(n.s.\) \\ HashOffAntiSpiction & \(n.x.\) & \(n.x.\) & 0.10 * low & \(n.s.\) & \(n.s.\) & \(n.s.\) & \(n.s.\) & \(n.s.\) \\ FriendSpace & \(n.x.\) & \(n.x.\) & 0.18 * high & \(n.s.\) & \(n.s.\) & \(n.s.\) & \(n.s.\) & \(n.s.\) \\ FlactShellerts & \(n.x.\) & 0.15 * high & \(n.s.\) & 0.12 * high & \(n.s.\) & \(n.s.\) & \(n.s.\) & \(n.s.\) \\ LapPursim & \(n.x.\) & 0.09 * high & 0.11 * high & \(n.s.\) & \(n.s.\) & \(n.s.\) & 0.06 * high & \(n.s.\) & 0.00 * high \\ Traditionalism & \(n.x.\) & \(n.x.\) & \(n.x.\) & \(n.s.\) & \(n.s.\) & 0.10 * high & \(n.s.\) & 0.00 * high \\ \hline Politics (_ibk. 0, cons. : 1)_ & \(n.x.\) & \(n.x.\) & 0.09 * lb. & \(n.s.\) & \(n.s.\) & \(n.s.\) & \(n.s.\) & \(n.s.\) \\ Gender (nonw. 0, nonsw. 1) & \(n.x.\) & 0.10 * wom. & 0.10 * men & \(n.s.\) & \(n.s.\) & 0.28 * wom. & \(n.s.\) & \(n.s.\) \\ Race (Whae; 0, Black-1) & \(n.x.\) & -0.07 * white & \(n.s.\) & \(n.s.\) & \(n.s.\) & -0.08 * white & \(n.s.\) & \(n.s.\) \\ \end{tabular}
\end{table}
Table 12: We correlated the PerspectiveAPI toxicity scores with offensiveness/racism ratings by our annotators, breaking them into two bins based on their attitude scores. Then, we used Fisher’s z-to-r test to measure whether the differences in correlations between the annotators who are high/low were significant (‘\(p<0.1\), ‘\(p<0.05\)).

Figure 6: PerspectiveAPI and ratings of _offensiveness_ of anti-Black tweets.

Figure 7: PerspectiveAPI and ratings of _racist_ of anti-Black tweets.



Figure 8: PerspectiveAPI and ratings of _offensiveness_ of AAE tweets.

Figure 10: PerspectiveAPI and ratings of _offensiveness_ of vulgar-OI tweets.

Figure 9: PerspectiveAPI and ratings of _racist_ of AAE tweets.

 

Figure 12: PerspectiveAPI and ratings of _offensiveness_ of vulgar-OnI tweets.

Figure 13: PerspectiveAPI and ratings of _racist_ of vulgar-OnI tweets.