SlowGuess/NAACL_2022_OCR huggingface dataset:
Adam D Galinsky, Cynthia S Wang, Jennifer A Whitson, Eric M Anicich, Kurt Hugenberg, and Galen V Bodenhausen. 2013. The reappropriation of stigmatizing labels: the reciprocal relationship between power and self-labeling. Psychol. Sci., 24(10).
Sergey Gavrilets and Laura Fortunato. 2014. A solution to the collective action problem in between-group conflict with within-group inequality. Nature communications, 5(1).
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daume III, and Kate Crawford. 2018. Datasheets for datasets. In  $FAccT^{*}$ .
Karen E Gerdes, Cynthia A Lietz, and Elizabeth A Segal. 2011. Measuring empathy in the 21st century: Development of an empathy index rooted in social cognitive neuroscience and social justice. Social Work Research, 35(2).
Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
David Gillborn. 2009. Risk-free racism: Whiteness and so-called free speech. Wake Forest law review, 44.
Tarleton Gillespie, Patricia Aufderheide, Elinor Carmi, Ysabel Gerrard, Robert Gorwa, Ariadna Matamoros-Fernandez, Sarah T Roberts, Aram Sinnreich, and Sarah Myers West. 2020. Expanding the debate about content moderation: Scholarly research agendas for the coming policy debates. *Internet Policy Review*, 9(4).
Donna M Goldstein and Kira Hall. 2017. Postelection surrealism and nostalgic racism in the hands of donald trump. HAU: Journal of Ethnographic Theory, 7(1).
Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeffrey T Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022. Jury learning: Integrating dissenting voices into machine learning models. In CHI.
Ben Green. 2020. Data science as political action: Grounding data science in a politics of justice. SSRN Electronic Journal.
Lisa Green. 2002. African American English: A Linguistic Introduction, 8.3.2002 edition edition.
Gil Greengross and Geoffrey F Miller. 2008. Dissing oneself versus dissing rivals: effects of status, personality, and sex on the Short-Term and Long-Term attractiveness of Self-Deprecating and Other-Deprecating humor. Evolutionary Psychology, 6(3).
Xiaochuang Han and Yulia Tsvetkov. 2020. Fortifying toxic speech detectors against veiled toxicity. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
A G Hilliard. 1997. Language, culture and the assessment of african american children. Assessment for equity and inclusion: Embracing all our children.
Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Connor Huff and Dustin Tingley. 2015. "who are these people?" evaluating the demographic characteristics and political preferences of mturk survey respondents. Research & Politics, 2(3).
Christian Ilbury. 2020. "sassy queens": Stylistic orthographic variation in twitter and the enregisterment of AAVE. Journal of sociolinguistics, 24(2).
Timothy Jay. 2018. Swearing, moral order, and online communication. Journal of Language Aggression and Conflict, 6(1).
Timothy Jay and Kristin Janschewitz. 2008. The pragmatics of swearing. Journal of Politeness Research, 4.
Björn H Jernudd and Michael J Shapiro. 1989. The politics of language purism.
J A Jiang. 2020. Identifying and addressing design and policy challenges in online content moderation. *Extended Abstracts of the 2020 CHI Conference on*.
Jialun Aaron Jiang, Morgan Klaus Scheuerman, Casey Fiesler, and Jed R Brubaker. 2021. Understanding international perceptions of the severity of harmful content online. *PloS one*, 16(8).
Darin G Johnson, Bradley D Mattan, Nelson Flores, Nina Lauharatanahirun, and Emily B Falk. 2022. Social-Cognitive and affective antecedents of code switching and the consequences of linguistic racism for black people and people of color. *Affective Science*, 3(1).
Stephen D Johnson and Joseph B Tamney. 2001. Social traditionalism and economic conservatism: Two conservative political ideologies in the united states. The Journal of social psychology, 141(2).
Pratyusha Kalluri. 2020. Don't ask if artificial intelligence is good or fair, ask how it shifts power. Nature, 583(7815).
Jonathan Knuckey. 2005. A new front in the culture war? moral traditionalism and voting behavior in us house elections. American Politics Research, 33(5).
Eric Loepp and Jarrod T Kelly. 2020. Distinction without a difference? an assessment of mturk worker types. Research & Politics, 7(1).
Daniel S Lucks. 2020. Reconsidering Reagan: Racism, Republicans, and the Road to Trump.
Antonis Maronikolakis, Axel Wisiorek, Leah Nann, Haris Jabbar, Sahana Udupa, and Hinrich Schütze. 2022. Listening to affected communities to define extreme speech: Dataset and experiments. In ACL 2022 Findings.
John B McConahay. 1986. Modern racism, ambivalence, and the modern racism scale. In *Prejudice, discrimination, and racism*, volume 337.
Kris McGuffie and Alex Newhouse. 2020. The radicalization risks of gpt-3 and advanced neural language models.
Ashley Muddiman. 2021. Conservatives and incivility. In Conservative Political Communication.
Kevin L Nadal. 2018. Microaggressions and traumatic stress: Theory, research, and clinical treatment.
Jessica C Nelson, Glenn Adams, and Phia S Salter. 2013. The marley hypothesis: Denial of racism reflects ignorance of history. Psychological science, 24(2).
Michael I Norton and Samuel R Sommers. 2011. Whites see racism as a Zero-Sum game that they are now losing. *Perspectives on psychological science: a journal of the Association for Psychological Science*, 6(3).
Gwenn Schurgin O'Keeffe, Kathleen Clarke-Pearson, and Council on Communications and Media. 2011. The impact of social media on children, adolescents, and families. Pediatrics, 127(4).
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7.
V Paul Poteat and Lisa B Spanierman. 2012. Modern racism attitudes among white students: The role of dominance and authoritarianism and the mediating effects of racial color-blindness. *The Journal of Social Psychology*, 152(6).
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing Annotator-Level labels and information in datasets. In Proc. of LAWDMR workshop at EMNLP.
Steven Pulos, Jeff Elison, and Randy Lennon. 2004. The hierarchical structure of the interpersonal reactivity index. Social behavior and personality, 32(4).
Jacquelyn Rahman. 2008. Middle-class african americans: Reactions and attitudes toward african american english. American speech, 83(2).
K Sabeel Rahman. 2017. The new utilities: Private power, social infrastructure, and the revival of the public utility concept. *Cardozo law review*, 39.
Marc Riar, Benedikt Morschheuser, Juho Hamari, and Rüdiger Zarnekow. 2020. How game features give rise to altruism and collective action? implications for cultivating cooperation by gamification. In Proceedings of the 53rd Hawaii International Conference on System Sciences.
Sarah T Roberts. 2017. Social media's silent filter. The Atlantic.
Jonathan Rosa. 2019. Looking Like a Language, Sounding Like a Race.
Jonathan Rosa and Nelson Flores. 2017. Unsettling race and language: Toward a raciolinguistic perspective. Language In Society, 46(5).
Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2017. Measuring the reliability of hate speech annotations: the case of the european refugee crisis. In NLP 4 CMC Workshop.
Gina Roussos and John F Dovidio. 2018. Hate speech is in the eye of the beholder: The influence of racial attitudes and freedom of speech beliefs on perceptions of racially motivated threats of violence. Social psychological and personality science, 9(2).
Elizabeth Sanders. 2002. From user-centered to participatory design approaches.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Barry S. Sapolsky, Daniel M. Shafer, and Barbara K. Kaye. 2010. Rating offensive words in three television program contexts. *Mass Communication and Society*, 14(1).
Brian F Schaffner. 2020. The heightened importance of racism and sexism in the 2018 US midterm elections. British journal of political science.
Leandro Silva, Mainack Mondal, Denzil Correa, Fabricio Benevenuto, and Ingmar Weber. 2016. Analyzing the targets of hate in online social media. In Tenth international AAAI conference on web and social media.
N Clayton Silver and William P Dunlap. 1987. Averaging correlation coefficients: should fisher's z transformation be used? Journal of applied psychology, 72(1).
Wiktor Soral, Michal Bilewicz, and Mikołaj Winiewski. 2018. Exposure to hate speech increases prejudice through desensitization. Aggressive behavior, 44(2).
Arthur K Spears et al. 1998. African-american language use: Ideology and so-called obscenity. African-American English: Structure, history, and use.
Linda Steg, Goda Perlaviciute, Ellen Van der Werff, and Judith Lurvink. 2014. The significance of hedonic values for environmentally relevant attitudes, preferences, and actions. Environment and behavior, 46(2).
Miriah Steiger, Timir J Bharucha, Sukrit Venkatagiri, Martin J Riedl, and Matthew Lease. 2021. The psychological Well-Being of content moderators: The emotional labor of commercial moderation and avenues for improving support. In CHI, number Article 341 in CHI '21.
Joanna Sterling, John T Jost, and Richard Bonneau. 2020. Political psycholinguistics: A comprehensive analysis of the language habits of liberal and conservative social media users. Journal of personality and social psychology, 118(4).
Ane Ueland. 2020. Language and identity: a study of African American Vernacular English and its status in American society. Ph.D. thesis, University of Stavanger, Norway.
Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021. Learning from the worst: Dynamically generated datasets to improve online hate detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Graham F Wagstaff. 1998. Equity, justice, and altruism. Current Psychology, 17(2).
Zeerak Waseem. 2016. Are you a racist or am I seeing things? annotator influence on hate speech detection on Twitter. In Proceedings of the First Workshop on NLP and Computational Social Science.
Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on Twitter. In Proceedings of the NAACL Student Research Workshop.
Zeerak Waseem, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein. 2021. Disembodied machine learning: On the illusion of objectivity in nlp. Anonymous preprint under review.
Mark H White and Christian S Crandall. 2017. Freedom of racist speech: Ego and expressive threats. Journal of personality and social psychology, 113(3).
Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017.
Danyaal Yasin. 2018. Black and banned: Who is free speech for? . Accessed: 2018-12-6.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Zuleyka Zevallos. 2017. Sociology of race. . Accessed: 2021-10-10.
Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah Smith. 2021. Challenges in automated debiasing for toxic language detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume.
A Attitude scales
Below, we list the individual items from each of our seven attitude dimensions. In our breadth-of-workers study, we asked participants all of the items in each scale, but in the larger-scale study, we only asked the bolded items (one per scale). (R) denotes a reverse coded item.
A.1 Valuing the freedom of offensive speech (FREEOFFSPEECH).
Participants were instructed to rate their level of agreement with the item statements, using a 5-point Likert scale ranging from "not at all" (1) to "very much so" (5).
- People should face consequences for saying something offensive online. (R)
- Offensive posts and comments online should be removed from the platforms.
- There is no such a thing as hate speech, only "speech".
- People should be allowed to say things that others may consider offensive.
- It's more important to protect free speech rights than to protect people's feelings when it comes to offensive language.
This scale was taken from Cowan et al. (2002).
A.2 Perceiving the HARMOFHATESPEECH.
- When someone posts something offensive towards minorities online it intimates and casts fear in the heart of individuals who are part of the minority group.
- Offensive language encourages discrimination against minorities.
- Offensive language is harmful to minorities.
A.3 Endorsement of RACISTBELIEFS.
- Discrimination against racial minorities is no longer a problem in the United States.
- It is easy to understand the anger of racial minorities people in America. (R)
- Racial minorities are getting too demanding in their push for equal rights.
- Over the past few years, racial minorities have gotten more economically than they deserve.
- Over the past few years, the government and news media have shown more respect to racial minorities than they deserve.
These items form the validated Modern Racism Scale, created by McDonahay (1986).
A.4 TRADITIONALISM.
Participants were asked: "Please tell us how important each of these is as a guiding principle in your life." They answered each item on a 5-point Likert scale, ranging from "not at all important to me" (1) to "extremely very important to me" (5).
- Being obedient, dutiful, meeting obligations.
- Self-discipline, self-restraint, resistance to temptations.
- Honoring parents and elders, showing respect.
Traditions and customs.
This is an abridged version of the traditionalism scale by Bouchard Jr. and McGue (2003).
A.5 Language Purism (LINGPURISM).
- I dislike when people make simple grammar or spelling errors.
- It is important to master the English language properly, and not make basic spelling mistakes or misuse a common word.
- I am not afraid to correct people when they make simple grammar or spelling errors.
There exists such a thing as good proper English.
This scale was created by the authors.
A.6 EMPATHY.
- Before criticizing somebody, I try to imagine how I would feel if I were in his/her place.
- I don't usually become sad when I see other people crying. (R)
Figure 3: Distributions of the attitude scores of the workers in the breadth-of-workers study.
EMPATHY	TRADITIONALISM	LINGPURISM	RACISTBELIEFS	HARMOFHATESPEECH	ALTRUISM	FREEOFFSPEECH
attitudes	EMPATHY	n.s.	-0.137	*	-0.309	**	0.359	**	0.456	**	-0.312	**
TRADITIONALISM	n.s.	0.260	**	0.409	**	-0.275	**	-0.136	*	0.161	**
LINGPURISM	-0.137	*	0.260	**	0.299	**	-0.162	**	-0.165	**	0.190	**
RACISTBELIEFS	-0.309	**	0.409	**	0.299	**	-0.711	**	-0.641	**	0.572	**
HARMOFHATESPEECH	0.359	**	-0.275	**	-0.162	**	-0.711	**	0.595	**	-0.677	**
ALTRUISM	0.456	**	-0.136	*	-0.165	**	-0.641	**	0.595	**	-0.496	**
FREEOFFSPEECH	-0.312	**	0.161	**	0.190	**	0.572	**	-0.677	**	-0.496	**
demog	Politics (lib.: 0, cons.: 1)	-0.123	*	0.531	**	0.263	**	0.703	**	-0.570	**	-0.515	**	0.491	**
Gender (men: 0, women: 1)	0.260	**	n.s.	n.s.	-0.133	*	0.128	*	0.172	**	-0.198	**
Race (White: 0, Black: 1)	n.s.	n.s.	n.s.	-0.191	**	n.s.	0.159	**	-0.178	**
Table 8: Pearson  $r$  correlations between the attitude and demographic variables from participants in our breadth-of-workers study. We only show significant correlations (*:  $p < {0.05}$  ,**:  $p < {0.001}$  ),and denote non-significant correlations with "n.s." . Our demographic variables are not correlated with each other.
- When someone is feeling 'down' I can usually understand how they feel.
- I have tender, concerned feelings for people or groups of people less fortunate than me.
This scale is an abbreviated version of the widely used Interpersonal Reactivity Index by Pulos et al. (2004).
A.7 ALTRUISM.
- Social justice, correcting injustice, caring for the weak.
- Equality, equal opportunity for all.
These items are taken from the altruism part of the scale by (Steg et al., 2014).
A.8 Attitude distributions & inter-variable correlations
We plot the distributions of our breadth-of-workers participants on the seven attitude scales in Figure 3. While most attitudes have wider distributions, RACISTBELIEFS notably stands out as having a skewed distributions towards people scoring low on the scale.
While some attitudes may highly correlate with demographic factors at face value (e.g., TRADITIONALISM and politically conservatism); other forms of biases may not be easily explained by demographics alone. We examine the relationship between our attitude measurements and annotator demographic identity variables. Shown in Table 8, we find strong significant correlations between several of our annotator variables.
Notably, we find that an annotator's political orientation correlated strongly with several variables, with liberal leaning identities being as
sociated with higher scores on the EMPATHY, HARMOFHATESPEECH, and ALTRUISM scales, whereas conservative political leaning was associated with higher scores on the TRADITIONALISM, LINGPURISM, FREEOFFSPEECH and RACIST-BELIEFS scales.
B Toxicity questions
Following crowdsourcing setups in prior work (Waseem, 2016; Davidson et al., 2017; Wulczyn et al., 2017; Founta et al., 2018; Sap et al., 2019), we asked three fine-grained questions to annotators for each post in both our studies:
- "How toxic/hateful/disrespectful or offensive does this post seem to you?"
- "How much could this post be seen as toxic/hateful/disrespectful or offensive to anyone?"
- "In your opinion, how racist is this post?"
Given the high correlations between the two offensiveness variables (Pearson  $r \geq .7$ ; Table 9), we use create an offensiveness ("off.") score by taking the average rating given to the "to you" and "to anyone" questions. In all our analyses, we use that offensiveness score, along with the raw racism score.
overall	anti-Black	AAE	Vulgar
to you	to any	ratist	to you	to any	ratist	to you	to any	ratist	to you	to any
off.	0.962	0.965	0.838	0.946	0.925	0.801	0.931	0.951	0.412	0.911	0.931
to you	0.857	0.823	0.751	0.794	0.773	0.405	0.696
to any	0.857	0.793	0.751	0.698	0.773	0.374	0.696
Table 9: Correlations between different offensiveness questions for each tweet (all  $p < 0.001$ ). Since offensiveness "to you" and "to any" are very strongly correlated, we average them into a single offensiveness score (off.).
C Small-scale controlled study details
C.1 Data Selection & Validation
We aimed to select online posts that were very indicative of each of the above characteristics (vulgar, AAE, anti-Black) but not indicative of the others, in order to tease out the effect of that category. We selected vulgar and AAE posts from a publicly available large corpus of tweets annotated for hate speech by Founta et al. (2018). For each tweet in that corpus, we detected the presence of nonidentity related profanity or swearwords using the
list from Zhou et al. (2021), and extracted the likelihood that the tweet is in AAE using a lexical detector by Blodgett et al. (2016). As candidates, we selected 10 vulgar tweets that have low likelihood of being AAE, and 26 tweets that have high likelihood of being AAE but contain no vulgarity. For anti-Black posts, we selected 11 candidate online posts curated by Zevallos (2017).
We ran a human validation study to verify that the candidate posts are truly indicative of their respective categories. We created an annotation scheme to collect binary ratings for two questions per post: "does it contain vulgar language", and "is it offensive to minorities"; a post could belong to either category or neither. Each post was manually annotated by three undergraduate research assistants trained for the task. Post validation, we manually selected 5 posts per category with perfect inter-annotator agreement. Table 1 lists the final 15 posts used for our study.
C.2 Participant Recruitment
We ran our study on Amazon Mechanical Turk (MTurk), a crowdsourcing platform that is often used to collect offensiveness annotations.12 With the task at hand, we sought a racially and politically diverse pool of participants, which can be challenging given that MTurk workers are usually tend to be predominantly white and skew liberal (Huff and Tingley, 2015; Burnham et al., 2018; Loepp and Kelly, 2020). Therefore, we ran a pre-selection survey to collect race and political ideology of workers, noting that this presurvey could grant them access to a longer survey on free speech, hate speech, and offensiveness in language.13,14 We stopped recruiting once we reached at least 200 Black and 200 conservative participants.
C.3 Study Setup
We ran our study on the widely used survey platform Qualtrics, using an MTurk HIT to recruit and compensate participants.[15] Participants were first
<sup>12</sup>Note, this study was approved by the author's institutional review board (IRB).
<sup>13</sup>To better recruit for our pre-survey, we noted in the title that "BIPOC people and conservatives were encouraged to participate," and also varied the title's wording to emphasize free speech or hate speech in different recruiting rounds.
14We compensated workers  $0.02-$ 0.03 for this presurvey.
<sup>15</sup> Participants were compensated $4.33 for the entire survey, equivalent to an average hourly compensation of $22/h.
11We only used the training subset of the corpus.
asked to consent to the task, then were shown instructions for annotating the 15 posts (with occasional reminders of the instructions). Then we asked participants their views on several topics using the scales described in §2.2 and §A and finally their demographics. Throughout the study, we added three attention checks to ensure the quality of responses.
Allowing only Black, white liberal, and white conservative workers to participate, we ran our survey for 4 weeks from March 10 to April 5 2021, occasionally reminding participants from our presurvey that they could take the survey.
D Breadth-of-Posts annotation study details
D.1 Data Selection
In this study, we draw from two existing corpora of posts labeled for toxicity, hate, or offensiveness. First, we select posts that are automatically detected as AAE and/or vulgar from Founta et al. (2018), using the lexical detector of AAE by (Blodgett et al., 2016) and the vulgar wordlist by Zhou et al. (2021). Second, we select posts that are automatically detected as vulgar and/or annotated as anti-Black from Vidgen et al. (2021). Importantly, in this large-scale study, we consider posts that potentially have multiple characteristics (e.g., AAE and vulgar), and thus consider both posts with potentially offensive identity references (vulgar-OI) as well as non-identity vulgar words (vulgar-ONI). However, to circumvent potential racial biases in what is labelled as "racist" in the Vidgen et al. (2021) corpus (Sap et al., 2019; Davidson et al., 2019), we do not consider posts that are annotated as anti-Black but detected as AAE.