SlowGuess/NAACL_2022_OCR huggingface dataset:
In the context of toxicity annotation and detection, our findings highlight the need to consider the attitudes of annotators towards free speech, racism, and their beliefs on the harms of hate speech, for an accurate estimation of anti-Black language as toxic, offensive, or racist (e.g., by actively taking into consideration annotator ideologies; Waseem, 2016; Vidgen et al., 2021). This can be especially important given that hateful content very often targets marginalized groups and racial minorities (Silva et al., 2016; Sap et al., 2020), and can catalyze violence against them (O’Keeffe et al., 2011; Cleland, 2014).
5 Who finds AAE posts toxic, and why?
African American English (AAE) is a set of well-studied varieties or dialects of U.S. English, common among, but not limited to, African-American or Black speakers (Green, 2002; Edwards, 2004). This category has been shown to be considered "worse" English by non-AAE speakers (Hilliard, 1997; Blake and Cutler, 2003; Champion et al., 2012; Beneke and Cheatham, 2015; Rosa and Flo
AAE posts	Rated as Racist
RACIST BELIEFS	r = 0.089	*
Politics (lib: 0, cons: 1)	r = 0.076	†
Table 5: Associations between ratings of racism and annotator variables, for the AAE posts from the breadth-of-workers study. As with the previous results, we correct for multiple comparisons for non-hypothesized associations and only show significant results ( $\dagger$ :  $p < 0.075$ ,  $*$ :  $p < 0.05$ ).
res, 2017), and is often mistaken as obscene or toxic by humans and AI models (Spearst et al., 1998; Sap et al., 2019), particularly due to dialect-specific lexical markers (e.g., words, suffixes).
Based on prior work that correlates racial prejudice with negative attitudes towards AAE (Gaither et al., 2015; Rosa, 2019), we hypothesize that annotators who are white and who score high in RACISTBELIEFS will rate AAE posts as more toxic. Additionally, since AAE can be considered non-canonical English (Sapolsky et al., 2010; DeFrank and Kahlbaugh, 2019), we hypothesize that annotators who are more conservative and who score higher in TRADITIONALISM and LINGPURISM will rate AAE posts with higher toxicity.
5.1 Breadth-of-Workers Results
Table 5 shows significant associations between annotator identities and beliefs and their ratings of toxicity of AAE posts. Partially confirming our hypothesis, we found that ratings of racism had somewhat significant correlations with annotators' conservative political leaning, and their scores on our RACISTBELIEFS scale. However, contrary to our expectations, we found that white and Black annotators did not differ in how offensive they rated AAE tweets ( $d = 0.14$ ,  $p > 0.1$ ). We found no additional hypothesized or exploratory associations for racism ratings, and no significant associations for offensiveness ratings.
5.2 Breadth-of-Posts Results
Shown in Table 6, our results for AAE and potentially vulgar breadth-of-posts study show higher offensiveness ratings from conservative raters, and those who scored higher in TRADITIONALISM and, almost significantly, RACISTBELIEFS. We also find that conservative annotators and those who scored higher in FREEOFFSPEECH (and nearly significantly, TRADITIONALISM) rated AAE posts as more racist.
AAE posts	Rated as Offensive	Rated as Racist
FREEOFFSpeechCH	n.s.	0.217*
RACISTBELIEFS	0.133†	n.s.
TRADITIONALISM	0.137*	0.110†
Politics (lib.: 0, cons.: 1)	0.143*	0.206**
As an additional investigation, we measure whether attitudes or identities affects toxicity ratings of AAE posts that contain the word "n*gga," a (reclaimed) slur that has very different pragmatic interpretations depending on speaker and listener identity (Croom, 2011). Here, we find that raters who are more conservative tended to score those posts as significantly more racist ( $\beta = 0.465$ ,  $p = 0.003$ ; corrected for multiple comparisons).
5.3 Perceived Toxicity of AAE
Our findings suggest that annotators perceive that AAE posts are associated with the Black racial identity (Rosa, 2019), which could cause those who score highly on the RACISTBELIEFS scale to annotate them as racist, potentially as a form of colorblind racism (e.g., where simply mentioning race is considered racist; Bonilla-Silva, 2006). Moreover, specific markers of AAE could have been perceived as obscene by non-AAE speakers (Spears et al., 1998), even though some of these might be reclaimed slurs (e.g., "n*gga"; Croom, 2011; Galinsky et al., 2013). Contrary to expectations, annotators' own racial identity did not affect their ratings of AAE posts in our studies. Future work should investigate this phenomenon further, in light of the variation in perceptions of AAE within the Black community (Rahman, 2008; Johnson et al., 2022), and the increased acceptance and usage of AAE by non-Black people in social media (Ilbury, 2020; Ueland, 2020).
These findings shed some light on the racial biases found in hate speech detection (Davidson et al., 2019; Sap et al., 2019), partially explaining why AAE is perceived as toxic. Based on our results, future work in toxic language detection should account for this over-estimation of AAE as racist. For example, annotators could explicitly in
Table 6: Associations between AAE (and potentially also vulgar) post ratings from the breadth-of-posts study and annotator variables, shown as the  $\beta$  coefficients from a mixed effects model with a random effect for each annotator. We only show significant results ( $\dagger$ :  $p < 0.075$ ,  $*$ :  $p < 0.05$ ,  $\*\*:$ $p < 0.001$ ; Holm-corrected for multiple comparisons; n.s.: not significant).
Vulgar (OnI) posts	Rated as Offensive
LINGPURISM	r = 0.106*
TRADITIONALISM	r = 0.252**
Politics (lib: 0, cons: 1)	r = 0.171**
Table 7: Associations between toxicity ratings and annotator variables for the vulgar posts from the breadth-of-workers study. We correct for multiple comparisons for non-hypothesized associations and only show significant results (*:  $p < {0.05}$  ,**:  $p < {0.001}$  ).
clude speakers of AAE, or those who understand that AAE or its lexical markers are not inherently toxic, or are primed to do so (Sap et al., 2019). Avoiding an incorrect estimation of AAE as toxic is crucial to avoid upholding racio-linguistic hierarchies and thus representational harms against AAE speakers (Rosa, 2019; Blodgett et al., 2020).
6 Who finds vulgar posts toxic, and why?
Vulgarity can correspond to non-identity referring swearwords (e.g.,  $f^{*}ck$ ,  $sh^{*}t$ ; denoted as ONI) or identity-referring slurs (e.g.,  $b^{*}tch$ ,  $n^{*}gga$ ; denoted as OI). Both types of vulgarity can be mistaken for toxic despite also having non-hateful usages (e.g., to indicate emotion or social belonging; Croom, 2011; Dynel, 2012; Galinsky et al., 2013).
Given that vulgarity can be considered noncanonical or impolite language (Jay and Janschewitz, 2008; Sapolsky et al., 2010; DeFrank and Kahlbaugh, 2019), we hypothesize that annotators who score high on LINGPURISM, TRADITIONALISM, and who are more conservative will rate vulgar posts as more offensive. Importantly, here, we focus on the posts that are exclusively vulgar (ONI) from only our breadth-of-workers study, to avoid confounding effects of vulgar posts with anti-Black meaning or in AAE (both of those cases were analyzed in §4.2 and §5.2). We refer the reader to Appendix F for the results on vulgar posts in the breadth-of-posts study.
6.1 Breadth-of-Workers Results
Confirming our hypotheses, we found that offensiveness ratings of vulgar (ONI) posts indeed correlated with annotators' TRADITIONALISM and LINGPURISM scores, and conservative political leaning (Table 7). We found no associations between attitudes and racism ratings for vulgar posts.
6.2 Perceived Toxicity of Vulgar Language
Our findings corroborate prior work showing how adherence to societal traditional values is often opposed to the acceptability of vulgar language (Sapolsky et al., 2010). Traditional values and conservative beliefs have been connected to finding vulgar language as a direct challenging the moral order (Jay, 2018; Sterling et al., 2020; Muddiman, 2021). Our results suggest that vulgarity is a very specific form of offensiveness that deserves special attention. Specifically, future work might consider studying the specific toxicity of individual identity-referring vulgar (OI) words, which can carry prejudiced meaning as well (e.g., slurs such as “ $n^{*}gg^{*}r$ ) . Moreover, annotators across different levels of traditionalism could be considered when collecting ratings of vulgarity, especially since perceptions might vary with generational and cultural norms (Dynel, 2012).
7 Toxicity Detection System Case Study: PERSPECTIVEAPI
Our previous findings indicated that there is strong potential for annotator identities and beliefs to affect their toxicity ratings. We are additionally interested in how this influences the behavior of toxicity detection models trained on annotated data. We present a brief case study to answer this question with the PERSPECTIVEAPI, a widely used, commercial system for toxicity detection. Appendix G provides a more in-depth description.
We investigate whether PERSPECTIVE API scores align with toxicity ratings from workers with specific identities or attitudes, using the 571 posts from our breadth-of-posts study. Specifically, we compare the correlations between PERSPECTIVE API scores and ratings from annotators, broken down by annotators with different identities (e.g., men and women) or with higher or lower scores on attitude scales (split at the mean). See Appendix G.1 for details about this methodology.
Our investigation shows that PERSPECTIVE scores can be significantly more aligned with ratings from certain identities or groups scoring higher or lower on attitude dimensions (see Table 12 in Appendix G.2). Our most salient results show that for anti-Black posts, PERSPECTIVE scores are somewhat significantly more aligned with racism ratings by annotators who score high in RACISTBELIEFS ( $r_{\mathrm{high}} = 0.29$ ,  $r_{\mathrm{low}} = 0.17$ ,
Figure 2: Correlation between PERSPECTIVE API toxicity scores and racism ratings for anti-Black posts, broken down by by participants scoring high and low in RACISTBELIEFS.
$\Delta r = 0.12, p = 0.056$ ; Figure 2). Additionally, for AAE posts, PERSPECTIVE scores are slightly more correlated with racism ratings by annotators who were women ( $\Delta r = 0.22, p < 0.001$ ) or white ( $\Delta r = 0.08, p = 0.07$ ), and who scored higher in LINGPURISM ( $\Delta r = 0.14, p = 0.003$ ) or TRADITIONALISM ( $\Delta r = 0.10, p = 0.030$ ).
Overall, our findings indicate that PERSPEC-TIVEAPI toxicity score predictions align with specific viewpoints or ideologies, depending on the text category. Particularly, it seems that the API underestimates the toxicity of anti-Black posts in a similar way to annotators who scored higher on the RACISTBELIEFS scale, and aligns more with white annotator's perception of AAE toxicity (vs. Black annotators). This corroborate prior findings that show that toxicity detection models inherently encode a specific positionality (Cambo, 2021) and replicate human biases (Davani et al., 2021).
Overall, our analyses showed that perceptions of toxicity are indeed affected by annotators' demographic identities and beliefs (§2). We found—via a breadth-of-workers study and a breadth-of-posts study (§3)—several associations when isolating specific text characteristics: anti-Black (§4), AAE (§5), and vulgarity (§6). Finally, we showed that a popular toxicity detection system yields toxicity scores that are more aligned with raters with certain attitudes and identities than others (§7). We discuss implications of our findings below.
Variation in toxicity ratings in hate speech datasets. In our study we deliberately sought rat
9www.perspectiveapi.com
ing of perceptions of toxicity of posts by racially and politically diverse participants. However, many existing hate speech datasets instructed annotators to adhere to detailed definitions of toxicity (Davidson et al., 2017; Founta et al., 2018), and some even selected crowdworkers for their liberal ideology (Waseem, 2016; Sap et al., 2020; Vidgen et al., 2021). While those annotation setups and annotator homogeneity could cause less variation in toxicity annotations of anti-Black, AAE, and vulgar posts, there is still empirical evidence of anti-AAE racial biases in some of these datasets (Sap et al., 2019; Davidson et al., 2019).
Given the large variation in perceptions of toxicity that we showed and the implicit encoding of perspectives by toxicity models, we recommend researchers and dataset creators investigate and report annotator attitudes and demographics; researchers could collect attitude scores based on relevant social science research, perhaps in lightweight format as done in our breadth-of-posts study, and report those scores along with the dataset (e.g., in datasheets; Gebru et al., 2018).
Contextualize toxicity predictions in social variables. As shown in our results and previous studies (e.g., Waseem, 2016; Ross et al., 2017; Waseem et al., 2021), determining what is toxic is subjective. However, given this subjectivity, the open question remains: whose perspective should be considered when using toxicity detection models? To try answering this question, we urge researchers and practitioners to consider all stakeholders and end users on which toxicity detection systems might be deployed (e.g., through human-centered design methods; Sanders, 2002; Friedman et al., 2008; Hovy and Yang, 2021). While currently, the decision of content moderation often solely lies in the hands of the platforms, we encourage the exploration of alternative solutions (e.g., community fact checkers, digital juries; Maronikolakis et al., 2022; Gordon et al., 2022).
In general, we urge people to embrace that each design decision has socio-political implications (Green, 2020; Cambo, 2021), and encourage them to develop technologies to shift power to the targets of oppression (Blodgett et al., 2020; Kalluri, 2020; Birhane, 2021). Finally, given the increasingly essential role of online platforms in people's daily lives (Rahman, 2017), we echo calls for policy regulating online spaces and toxicity detection algorithms (Jiang, 2020; Benesch, 2020; McGuffie
and Newhouse, 2020; Gillespie et al., 2020).
Beyond toxicity classification: modeling distributions and generating explanations. Our findings on the subjectivity of the toxicity detection tasks suggests that standard approaches of obtaining binary (or even  $n$ -ary) labels of toxicity and averaging them into a majority vote are inadequate. Instead, researchers could consider modeling the distribution or variation in toxicity labels with respect to individual annotators (Geva et al., 2019; Fornaciari et al., 2021; Davani et al., 2021) or to specific identities or beliefs.
But, perhaps more importantly, we encourage re-thinking the toxicity detection paradigm altogether. With the goal to assist human content moderators,[10] creating systems that explain biased implications of posts could be more helpful than opaque toxicity scores. Thus, we advocate for moving away from classification frameworks, and towards more nuanced, holistic, and explainable frameworks for inferring the desired concepts of toxicity and social biases (e.g., Social Bias Frames; Sap et al., 2020).
Limitations and open questions. Our work had several limitations and raised several open research questions, some of which we outline below. First, our particular choices of attitudes and scales could affect our results; other scales (e.g., Gerdes et al., 2011, for measuring empathy) as well as other psychological variables (e.g., propensity to volunteer or to value dignity) could be studied in the context of toxicity perceptions. Additionally, the automatic AAE detector in the breadth-of-posts study could have induced data selection biases, despite being strongly correlated with race-aware dialect detection (as noted in footnote 6). Furthermore, our analysis of the attitudes encoded in the PERSPECTIVEAPI in §7 was merely a pilot study; we hope future work will explore more in-depth methods for assess model positionality.
While our study focused on racial discrimination by studying AAE and anti-Black posts, future work should explore other axes of discrimination (e.g., sexism, homophobia, ableism, etc.). Additionally, our study focused only on U.S.-centric perspectives; we hope researchers will explore variations in toxicity perceptions in other cultural contexts (e.g., variations based on caste in India).
<sup>10</sup>Note that while content moderation can induce significant psychological harms in moderators (Roberts, 2017; Steiger et al., 2021), full automation also has significant risks.
Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021. Whose opinions matter? perspective-aware models to identify opinions of hate speech victims in abusive language detection. ArXiv preprint arXiv:2106.15896.
Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020. Identifying and measuring annotator bias based on annotators' demographic characteristics. In Proceedings of the Fourth Workshop on Online Abuse and Harms.
Carolina Are. 2020. How instagram's algorithm is censoring women and vulnerable users but helping online abusers. *Feminist media studies*, 20(5).
Kofi Arhin, Ioana Baldini, Dennis Wei, Karthikeyan Natesan Ramamurthy, and Moninder Singh. 2021. Ground-Truth, whose truth? - examining the challenges with annotating toxic text datasets.
Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Allocative versus representational harms in machine learning. In SIGCIS.
Margaret Beneke and Gregory A Cheatham. 2015. Speaking up for african american english: Equity and inclusion in early childhood settings. Early Childhood Education Journal, 43(2).
Susan Benesch. 2020. Proposals for improved regulation of harmful online content. Technical report.
Abeba Birhane. 2021. Algorithmic injustice: a relational ethics approach. *Patterns (New York, N.Y.)*, 2(2).
Renée Blake and Cecilia Cutler. 2003. AAE and variation in teachers' attitudes: A question of school philosophy? Linguistics and education, 14(2).
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Su Lin Blodgett, Lisa Green, and Brendan O'Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
Eduardo Bonilla-Silva. 2006. *Racism without racists: Color-blind racism and the persistence of racial inequality in the United States*.
Thomas J. Bouchard Jr. and Matt McGue. 2003. Genetic and environmental influences on human psychological differences. Journal of Neurobiology, 54(1).
Luke Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019. Finding microaggressions in the wild: A case for locating elusive phenomena in social media posts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Martin J Burnham, Yen K Le, and Ralph L Piedmont. 2018. Who is mturk? personal characteristics and sample consistency of these online workers. *Mental Health, Religion & Culture*, 21(9-10).
Scott Allen Cambo. 2021. Model Positionality: A Novel Framework for Data Science with Subjective Target Concepts. Ph.D. thesis, Northwestern University.
Evelyn R Carter and Mary C Murphy. 2015. Group-based differences in perceptions of racism: What counts, to whom, and why? Social and personality psychology compass, 9(6).
Tempii B Champion, Deirdre Cobb-Roberts, and Linda Bland-Stewart. 2012. Future educators' perceptions of african american vernacular english (AAVE). Online Journal of Education Research, 1(5).
Jamie Cleland. 2014. Racism, football fans, and online message boards: How social media has added a new dimension to racist discourse in English football. *J. Sport Soc. Issues*, 38(5).
D Cole. 1996. Racist speech should be protected by the constitution. *Hate crimes*.
Gloria Cowan and Désirée Khatchadourian. 2003. Empathy, ways of knowing, and interdependence as mediators of gender differences in attitudes toward hate speech and freedom of speech. Psychology of Women Quarterly, 27(4).
Gloria Cowan, Miriam Resendez, Elizabeth Marshall, and Ryan Quist. 2002. Hate speech and constitutional protection: Priming values of equality and freedom. The Journal of social issues, 58(2).
Adam M Croom. 2011. Slurs. Language Sciences, 33(3).
Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy, and Morteza Dehghani. 2021. Hate speech classifiers learn human-like social stereotypes.
Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In ICWSM.
Melanie DeFrank and Patricia Kahlbaugh. 2019. Language choice matters: When profanity affects how people are judged. Journal of Language and Social Psychology, 38(1).
Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Daniel M Downs and Gloria Cowan. 2012. Predicting the importance of freedom of speech and the perceived harm of hate speech. Journal of applied social psychology, 42(6).
John Duckitt and Kirstin Fisher. 2003. The impact of social threat on worldview and ideological attitudes. Political Psychology, 24(1).
Marta Dynel. 2012. Swearing methodologically: the (im)politeness of expletives in anonymous commentaries on youtube. Journal of English Studies, 10(0).
Walter F Edwards. 2004. African american vernacular english: phonology. In A Handbook of Varieties of English: Morphology and syntax.
Christine Helen Elers and Pooja Jayan. 2020. "this is us": Free speech embedded in whiteness, racism and coloniality in aotearoa, new zealand. First Amendment Studies, 54(2).
S T Fiske. 1993. Controlling other people: the impact of power on stereotyping. The American psychologist, 48(6).
Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. In ICWSM.
Batya Friedman, Peter H Kahn, and Alan Borning. 2008. Value sensitive design and information systems. The handbook of information and computer ethics.
Sarah E Gaither, Ariel M Cohen-Goldberg, Calvin L Gidney, and Keith B Maddox. 2015. Sounding black or white: Priming identity and biracial speech. Frontiers in Psychology, 6.