SlowGuess/NAACL_2021_OCR huggingface dataset:
Bertie Vidgen†, Dong Nguyen†*, Helen Margetts†‡, Patricia Rossini¶, Rebekah Tromble†§
† The Alan Turing Institute; * Utrecht University; ‡ University of Oxford;
University of Liverpool; George Washington University
bvidgen@turing.ac.uk
Online abuse can inflict harm on users and communities, making online spaces unsafe and toxic. Progress in automatically detecting and classifying abusive content is often held back by the lack of high quality and detailed datasets. We introduce a new dataset of primarily English Reddit entries which addresses several limitations of prior work. It (1) contains six conceptually distinct primary categories as well as secondary categories, (2) has labels annotated in the context of the conversation thread, (3) contains rationales and (4) uses an expert-driven group- adjudication process for high quality annotations. We report several baseline models to benchmark the work of future researchers. The annotated dataset, annotation guidelines, models and code are freely available.
Social media platforms have enabled unprecedented connectivity, communication and interaction for their users. However, they often harbour harmful content such as abuse and hate, inflicting myriad harms on online users (Waseem and Hovy, 2016; Schmidt and Wiegand, 2017a; Fortuna and Nunes, 2018; Vidgen et al., 2019c). Automated techniques for detecting and classifying such content increasingly play an important role in moderating online spaces.
Detecting and classifying online abuse is a complex and nuanced task which, despite many advances in the power and availability of computational tools, has proven remarkably difficult (Vidgen et al., 2019a; Wiegand et al., 2019; Schmidt and Wiegand, 2017b; Waseem et al., 2017). As Jurgens et al. (2019) argued in a recent review, research has 'struggled to move beyond the most obvious tasks in abuse detection.' One of the biggest barriers to creating higher performing, more robust, nuanced and generalisable classification systems
is the lack of clearly annotated, large and detailed training datasets. However, creating such datasets is time-consuming, complicated and expensive, and requires a mix of both social and computational expertise.
We present a new annotated dataset of  $\sim 25,000$  Reddit entries. It contains four innovations that address limitations of previous labelled abuse datasets. First, we present a taxonomy with six conceptually distinct primary categories (Identity-directed, Person-directed, Affiliation-directed, Counter Speech, Non-hateful Slurs and Neutral). We also provide salient subcategories, such as whether personal abuse is directed at a person in the conversation thread or to someone outside it. This taxonomy offers greater coverage and granularity of abuse than previous work. Each entry can be assigned to multiple primary and/or secondary categories (Section 3).
Second, we annotate content in context, by which we mean that each entry is annotated in the context of the conversational thread it is part of. Every annotation has a label for whether contextual information was needed to make the annotation. To our knowledge, this is the first work on online abuse to incorporate a deep level of context. Third, annotators provided rationales. For each entry they highlighted the part of the text which contains the abuse (and the relevant parts for Counter Speech and Non-hateful Slurs). Fourth, we provide high quality annotations by using a team of trained annotators and a time-intensive discussion-based process, facilitated by experts, for adjudicating disagreements (Section 4).
This work addresses the need for granular and nuanced abusive content datasets, advancing efforts to create accurate, robust, and generalisable classification systems. We report several baseline models to benchmark the work of future researchers (Section 5). The annotated dataset, annotation
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2289-2303 June 6-11, 2021. ©2021 Association for Computational Linguistics
codebook and code have been made available. $^{1}$  A full description of the dataset is given in our data statement in the Appendix (Bender and Friedman, 2018).
Taxonomies of abuse Taxonomies vary in terms of the scope of abusive behaviours they cover. Some offer categories for abuse against both individuals and groups (Zampieri et al., 2020), others cover only abuse against identities (Davidson et al., 2017; Fortuna and Nunes, 2018; Kiela et al., 2020), against only a single identity, such as misogyny (Anzovino et al., 2018) or Islamophobia (Vidgen and Yasseri, 2019), or only abuse against individuals (Wulczyn et al., 2017). Some research distinguishes between content in different languages or taken from different platforms (Kumar et al., 2018).
Waseem et al. (2017) outline two dimensions for characterising online abuse. First, whether it is directed against individuals or groups. Second, whether it is implicit or explicit (also referred to as 'covert' or 'overt' (Kumar et al., 2018) and 'weak' or 'strong' (Vidgen and Yasseri, 2019)). These two dimensions (strength and target) have been further developed in other studies. Zampieri et al. (2019) use a hierarchical three-level approach to annotation, separating (a) offensive from not-offensive tweets, (b) offensive into targeted and untargeted statements and (c) for targeted statements, identification of what is attacked (group, individual or other). Vidgen et al. (2019a) propose a tripartite distinction, also separating 'concept-directed' abuse from group-directed and person-directed abuse. However, this is problematic as concept-directed content may be better understood as legitimate critique.
Many taxonomies include fine-grained labels for complex subcategories of abuse. Palmer et al. (2020) label implicit varieties of hate, including 'adjectival nominalization', 'distancing' and 'Othering' language. Anzovino et al. (2018) label content for six subtypes of misogyny: discrediting, using stereotypes, objectifying, sexually harassing, threatening violence, dominating or derailing. Sanguinetti et al. (2018) provide annotations for which group is targeted and the linguistic action (i.e., dehumanizing, delegitimizing or aiming to inflict harm). They provide flags for aggressive-
ness, offensiveness, irony and stereotypes. Sap et al. (2020) provide annotations for 'social frames' (i.e., biases and stereotypes) about groups. They provide labels for (a) offence (yes/no), (b) whether a group is targeted, and (c) whether the abuse is intentional. Wulczyn et al. (2017) identify different interpersonal abuse, including toxicity, aggression and attacks.
Some taxonomies explicitly separate abuse from closely-related but non-abusive forms of online expression. This reflects social scientific insights which emphasize the importance, but also difficulty, of making such distinctions (Rossini, 2019, 2020). Vidgen et al. (2020) distinguish hostility against East Asia from criticism of East Asia, as well as counter speech and discussion of prejudice. Procter et al. (2019) distinguish cyber hate from counter speech, as do Qian et al. (2019) and Mathew et al. (2019), amongst others.
Annotation and Data The quality of annotations for abusive datasets has been widely critiqued, and inter-rater agreement scores are often remarkably low. Wulczyn et al. (2017) report an Alpha of 0.45, Sanguinetti et al. (2018) Kappas from  $k = 0.37$  for offence to  $k = 0.54$  for hate, Gomez et al. (2020) report Kappa of 0.15 in the "MMH150" dataset of hateful memes, and Fortuna and Nunes (2018) report a Kappa of 0.17 for a text-only task. In a classification study of prejudice against East Asia, Vidgen et al. (2020) find that  $27\%$  of classification errors are due to annotation mistakes. Low agreement is partly because abuse is inherently ambiguous and subjective, and individuals can perceive the same content very differently (Salminen et al., 2019, 2018).
Many abusive content datasets use crowdsourced annotations (Zampieri et al., 2019; Fortuna and Nunes, 2018; Davidson et al., 2017). They are cheap and scalable but can be low quality and are often ill-suited to complicated tasks (Sabou et al., 2014). Trained experts with clear guidelines are often preferable for ensuring consistency (Vidgen and Derczynski, 2020). Whether experts- or crowdsourced annotators are used, a diverse pool is needed as annotators encode their biases, backgrounds and assumptions into their annotations (Sap et al., 2019; Waseem et al., 2017). Most datasets use a simple majority vote over annotations to determine the final labels. However, majority agreement does not guarantee that content is correctly labelled, especially for complex edge
https://github.com/dongpng/cad_ naacl2021
cases. One option is to use a method that adjusts annotators' impact based on their quality, such as MACE (Hovy et al., 2013). However, this may not work well on the most ambiguous content. Group-decision making processes present a promising way of improving annotation quality. Breitbartler et al. (2019) use a collaborative multi-stage process to label micro-aggression and Card et al. (2015) use a similar process for labelling news articles. This ensures more oversight from experts and reflection by annotators on the difficult content. It also provides a feedback loop for annotators to learn from mistakes and improve.
A well-established problem with abusive content datasets is that each bit of content is marked up individually, without taking into account any content that came before (Gao and Huang, 2017; Mubarak et al., 2017). This can lead to poor quality annotations when content is ambiguous or unclear without knowing the context. Detection systems which do not account for context are likely to be less applicable in the real-world, where nearly all content appears in a certain context (Seaver, 2015). Pavlopoulos et al. (2020) systematically investigate the role of context in a dataset of Wikipedia comments by providing annotators the 'parent' before showing them the 'child' entry. In one experiment at least  $5\%$  of the data was affected. In a study of Twitter conversations Procter et al. (2019) label replies to tweets based on whether they 'agree' or 'disagree' with the original message. Notwithstanding these studies, further work is needed to better understand the role of context and how abuse emerges within threads, as well as the challenges of detecting deeply contextual content.
3 Taxonomy
We present a hierarchical taxonomy of abusive content, which comprises six primary categories and additional secondary categories. It builds on critical social scientific research (Marwick and Miller, 2014; Citron and Norton, 2011; Lenchart et al., 2016), and addresses issues in previous taxonomies, including those provided by Zampieri et al. (2020), Waseem et al. (2017), Founta et al. (2018) and Vidgen et al. (2019a). It offers greater coverage by including three conceptually distinct types of abusive content (Identity-directed abuse, Affiliation-directed abuse and Person-directed abuse) as well as three types of non-abusive content (Neutral, Counter Speech and Non-hateful Slurs). The tax-
Figure 1: Primary and Secondary categories.
onomic structure is shown in Figure 1. Indicative examples are given in Table 1.
3.1 Identity-directed abuse
Content which contains a negative statement made against an identity. An 'identity' is a social category that relates to a fundamental aspect of individuals' community, socio-demographics, position or self-representation (Jetten et al., 2004). It includes but is not limited to Religion, Race, Ethnicity, Gender, Sexuality, Nationality, Disability/Ableness and Class. The secondary category comprises five subtypes of identity-directed abuse: Derogation, Animosity, Threatening language, Glorification and Dehumanization.
Derogation Language which explicitly attacks, demonizes, demeans or insults a group. Derogation includes representing or describing a group in extremely negative terms and expressing negative emotions about them. Derogation is the basis of most 'explicit' forms of abuse in existing hateful content taxonomies, although it is often referred to
Primary	Secondary	Example
Identity-directed	Derogation	Muslims cant speak English, they're savages
Identity-directed	Animosity	I dont think black people face any discrimination
Identity-directed	Threatening	Gotta kick those immigrants out... now!
Identity-directed	Glorification	Adolf had it right, yknow?
Identity-directed	Dehumanization	Travellers are nothing but trash
Affiliation-directed	Derogation	So sick of these undesirable leftist fools
Person-directed	To them	Fuck off @user
Person-directed	About them	Trump is a massive bellend
Non-hateful Slurs	/	Y'all see me and my n*ggas come in here?
Counter Speech	Identity-directed	Sorry but that's just factually incorrect
Counter Speech	Affiliation-directed	You should be nicer about theMODs, they do alot for us
Counter Speech	Person-directed	That's no way to talk to someone!
Neutral	/	I've had a right bloody day of it
Table 1: Indicative examples of the categories.
with different terms. For instance, Davidson et al. (2017) define hate as content that is 'derogatory', Waseem and Hovy (2016) include 'attacks' in their account of hate and Zampieri et al. (2019) 'insults'.
Animosity Language which expresses abuse against a group in an implicit or subtle manner. The lynchpin of this category is that negativity is directed at the group (i.e., there must be some aspect which is discernibly abusive or demeaning about the group in question) but this is not expressed explicitly. Animosity includes undermining the experiences and treatment of groups, ridiculing them, and accusing them of receiving 'special treatment'. Animosity is similar to the 'implicit' category used in other taxonomies (Waseem et al., 2017; Vidgen and Yasseri, 2019; Kumar et al., 2018).
Threatening language Language which either expresses an intent/desire to inflict harm on a group, or expresses support for, encourages or incites such harm. Harm includes physical violence, emotional abuse, social exclusion and harassment. This is one of the most harmful forms of hateful language (Marwick and Miller, 2014; Citron and Norton, 2011) yet usually it is part of an 'explicit' hate category (Zampieri et al., 2019; Wulczyn et al., 2017; Waseem and Hovy, 2016) and few datasets have treated it as a separate category, see Golbeck et al. (2017), Anzovino et al. (2018), and Hammer (2014) for exceptions.
Dehumanization Language which maliciously describes groups as insects, animals and nonhumans (e.g., leeches, cockroaches, insects, germs, rats) or makes explicit comparisons. Dehuman-
ization has been linked with real-world violence and is a particularly important focus for computational work (Leader Maynard and Benesch, 2016; Matsuda et al., 1993), yet is often combined into a broader 'explicit' category (Palmer et al., 2020; Vidgen et al., 2020; Kiela et al., 2020) and has been insufficiently studied on its own, apart from Mendelsohn et al. (2020).
Glorification of hateful entities Language which explicitly glorifies, justifies or supports hateful actions, events, organizations, tropes and individuals (which, collectively, we call 'entities'). It includes denying that identity-based atrocities took place (e.g., Genocide). Glorification is one of the least studied forms of hate computationally, likely because it is more ambiguous, particularly when individuals only express interest in the entities (de Gibert et al., 2018).
3.2 Affiliation-directed abuse
Content which express negativity against an affiliation. We define 'affiliation' as a (more or less) voluntary association with a collective. Affiliations include but are not limited to: memberships (e.g. Trade unions), party memberships (e.g. Republicans), political affiliations (e.g. Right-wing people) and occupations (e.g. Doctors). The same secondary categories for Identity-directed abuse apply to Affiliation-directed. In some previous taxonomies, affiliations have been mixed in with identities (Founta et al., 2018; Zampieri et al., 2019), although in general they have been excluded as out of scope (e.g. Waseem and Hovy (2016)).
3.3 Person-directed abuse
Content which directs negativity against an identifiable person, who is either part of the conversation thread or is named. Person-directed abuse includes serious character based attacks, such as accusing the person of lying, as well as aggression, insults and menacing language. Person- and Identity- directed forms of abuse are often addressed in separate taxonomies, although in some studies they have been merged into a more general 'toxic' category (Wulczyn et al., 2017; Golbeck et al., 2017). Recent work have addressed both types of content, recognising that they are conceptually different but often co-occur in the real-world and share syntactical and lexical similarities (Zampieri et al., 2019; Mandl et al., 2019). We provide two secondary categories for person-directed abuse: Abuse at a person who is part of the conversation thread and Abuse about a person who is not part of the conversation thread. The person must be clearly identified, either by their actual name, username or status (e.g. 'the president of America'). To our knowledge, this distinction has not been used previously.
3.4 Counter Speech
Content which challenges, condemns or calls out the abusive language of others. Counter Speech can take several forms, including directly attacking/condemning abusive language in unambiguous terms, challenging the original content and 'calling out' the speaker for being abusive. We use a similar approach to Qian et al. (2019) and Mathew et al. (2019) who also treat counter speech as a relational act that responds to, and challenges, actual abuse.
3.5 Non-hateful Slurs
A slur is a collective noun, or term closely derived from a collective noun, which is pejorative. Slurs include terms which are explicitly insulting (e.g. 'n*gga' or 'kebabi') as well as terms which implicitly express animosity against a group (e.g. 'Rainy' or 'Chad'). A slur by itself does not indicate identity-directed abuse because in many cases slurs are not used in a derogatory way but, rather, to comment on, counter or undermine genuine prejudice (Jeshion, 2013) — or they have been reclaimed by the targeted group, such as use of 'n*gga' by black communities (Davidson et al., 2017; Davidson and Weber, 2019). In this category we mark up only the non-hateful use of slurs. Hateful uses of slurs would fall under Identity-directed abuse.
3.6 Neutral
Content which does not contain any abuse, Nonhateful Slurs or Counter Speech and as such would not fall into any of the other categories.
4.1 Data collection
The low prevalence of online abuse in 'the wild' (likely as little as  $0.1\%$  in English language social media (Vidgen et al., 2019b)) means that most training datasets have used some form of purposive (or 'directed') sampling to ensure enough entries are in the positive class (Fortuna et al., 2020). However, this can lead to biases in the dataset (Ousidhoum et al., 2020) which, in turn may impact the performance, robustness and fairness of detection systems trained on them (Sap et al., 2019). Notably, the widely-used practice of keyword sampling can introduce topic and author biases, particularly for datasets with a high proportion of implicit abuse (Wiegand et al., 2019).
Accordingly, like Qian et al. (2019), we use community-based sampling, selecting subreddits which are likely to contain higher-than-average levels of abuse and a diverse range of abuse. This should lead to a more realistic dataset where the abusive and non-abusive content share similarities in terms of topic, grammar and style. We identified 117 subreddits likely to contain abusive content, which we we filtered to just 16, removing subreddits which (1) had a clear political ideology, (2) directed abuse against just one group and (3) did not have recent activity. 187,806 conversation threads were collected over 6 months from 1st February 2019 to 31st July 2019, using the PushShift API (Gaffney and Matias, 2018). We then used stratified sampling to reduce this to 1,394 posts and 23,762 comments (25,156 in total) for annotation. See Data Statement in the Appendix for more information on how the initial 117 subreddits were identified.
All posts and comments were annotated. The titles main body of posts were treated separately, resulting in 1,394 post titles, 1,394 post bodies and 23,762 comments being annotated (26,550 entries in total). All entries were assigned to at least one of the six primary categories. Entries could be assigned to several primary categories and/or several
secondary categories. The dataset contains 27,494 distinct labels.
All entries were first independently annotated by two annotators. Annotators underwent 4 weeks training and were either native English speakers or fluent. See Data Statement in the Appendix for more information. Annotators worked through entire Reddit conversations, making annotations for each entry with full knowledge of the previous content in the thread. All disagreements were surfaced for adjudication. We used a consensus-based approach in which every disagreement was discussed by the annotators, facilitated by an expert with reference to the annotation codebook. This is a time-consuming process which helps to improve annotators' understanding, and identify areas that guidelines need to be clarified and improved. Once all entries were annotated through group consensus they were then reviewed in one-go by the expert to ensure consistency in how labels were applied. This helped to address any issues that emerged as annotators' experience and the codebook evolved throughout the annotation process. In some cases the labels may appear counter-intuitive. For instance, one entry starts "ITT: Bernie Sanders is imperfect and therefore is a garbage human being." This might appear like an insult, however the remainder of the statement shows that it is intended ironically. Similarly, use of "orange man bad" may appear to be an attack against Donald Trump. However, in reality it is supporting Trump by mocking left-wing people who are opposed to him. Nuances such as these only become apparent after multiple reviews of the dataset and through group-based discussions.
Targets of abuse For Identity-directed, Affiliation-directed and Non-hateful Slurs, annotators inductively identified targets. Initially, 1,500 targets were identified (including spelling variations), which was reduced to 185 through review and cleaning. All important distinctions, including intersectional identities and specific subgroups and outlooks (e.g., 'non-gender dysphoric transgender people') were retained. The identities were then grouped into 8 top level categories. The top level categories for Identity-directed abuse include Gender, Ableness/disability and Race.
Context For every annotation a flag for 'context' was given to capture how the annotation was made. If the primary/secondary label was based on just
the entry by itself then 'Current' was selected. If knowledge of the previous content in the conversation thread was required then 'Previous' was selected. Context was primarily relevant in two ways. First, for understanding who a generic pronoun referred to (e.g., 'they'). Second, to express support for another users' abuse (e.g., Person 1 writes 'I want to shoot some X' and person 2 responds 'Go do it!'). If this context is not taken into account then the abuse would be missed. In some cases, only the context of a single previous statement was needed to understand an entry (as with the example just given), whereas in other cases several previous statements were required. For Neutral, no label is given for context. For Non-hateful Slurs, only 'Current' could be selected. Our definition of Counter Speech is relational, and so all Counter Speech require 'Previous' context. For Affiliation-, Identity-, and Person- directed approximately  $25 - 32\%$  of content were labelled with 'Previous' context.