Introduction
Abusive online content, such as hate speech and harassment, has received substantial attention over the past few years for its malign social effects. Left unchallenged, abusive content risks harminng those who are targeted, toxifying public discourse, exacerbating social tensions and could lead to the exclusion of some groups from public spaces. As such, systems which can accurately detect and classify online abuse at scale, in real-time and without bias are of central interest to tech companies, policymakers and academics.
Most detection systems rely on having the right training dataset, reflecting one of the most widely accepted mantras in computer science: Garbage In, Garbage Out. Put simply: to have systems which can detect and classify abusive online content effectively, one needs appropriate datasets with which to train them. However, creating training datasets is often a laborious and non-trivial task – and creating datasets which are non-biased, large and theoretically-informed is even more difficult (BIBREF0 p. 189). We address this issue by examining and reviewing publicly available datasets for abusive content detection, which we provide access to on a new dedicated website, hatespeechdata.com.
In the first section, we examine previous reviews and present the four research aims which guide this paper. In the second section, we conduct a critical and in-depth analysis of the available datasets, discussing first what their aim is, how tasks have been described and what taxonomies have been constructed and then, second, what they contain and how they were annotated. In the third section, we discuss the challenges of open science in this research area and elaborates different ways of sharing training datasets, including the website hatespeechdata.com In the final section, we draw on our findings to establish best practices when creating datasets for abusive content detection.
Background
The volume of research examining the social and computational aspects of abusive content detection has expanded prodigiously in the past five years. This has been driven by growing awareness of the importance of the Internet more broadly BIBREF1, greater recognition of the harms caused by online abuse BIBREF2, and policy and regulatory developments, such as the EU's Code of Conduct on Hate, the UK Government's `Online Harms' white paper BIBREF3, Germany's NetzDG laws, the Public Pledge on Self-Discipline for the Chinese Internet Industry, and France's anti-hate regulation BIBREF2. In 2020 alone, three computer science venues will host workshops on online hate (TRAC and STOC at LREC, and WOAH at EMNLP), and a shared task at 2019's SemEval on online abuse detection reports that 800 teams downloaded the training data and 115 submitted detection systems BIBREF4. At the same time, social scientific interventions have also appeared, deepening our understanding of how online abuse spreads BIBREF5 and how its harmful impact can be mitigated and challenged BIBREF6.
All analyses of online abuse ultimately rely on a way of measuring it, which increasingly means having a method which can handle the sheer volume of content produced, shared and engaged with online. Traditional qualitative methods cannot scale to handle the hundreds of millions of posts which appear on each major social media platform every day, and can also introduce inconsistencies and biase BIBREF7. Computational tools have emerged as the most promising way of classifying and detecting online abuse, drawing on work in machine learning, Natural Language Processing (NLP) and statistical modelling. Increasingly sophisticated architectures, features and processes have been used to detect and classify online abuse, leveraging technically sophisticated methods, such as contextual word embeddings, graph embeddings and dependency parsing. Despite their many differences BIBREF8, nearly all methods of online abuse detection rely on a training dataset, which is used to teach the system what is and is not abuse. However, there is a lacuna of research on this crucial aspect of the machine learning process. Indeed, although several general reviews of the field have been conducted, no previous research has reviewed training datasets for abusive content detection in sufficient breadth or depth. This is surprising given (i) their fundamental importance in the detection of online abuse and (ii) growing awareness that several existing datasets suffer from many flaws BIBREF9, BIBREF10. Close relevant work includes:
Schmidt and Wiegand conduct a comprehensive review of research into the detection and classification of abusive online content. They discuss training datasets, stating that `to perform experiments on hate speech detection, access to labelled corpora is essential' (BIBREF8, p. 7), and briefly discuss the sources and size of the most prominent existing training datasets, as well as how datasets are sampled and annotated. Schmidt and Wiegand identify two key challenges with existing datasets. First, `data sparsity': many training datasets are small and lack linguistic variety. Second, metadata (such as how data was sampled) is crucial as it lets future researchers understand unintended biases, but is often not adequately reported (BIBREF8, p. 6).
Waseem et al.BIBREF11 outline a typology of detection tasks, based on a two-by-two matrix of (i) identity- versus person- directed abuse and (ii) explicit versus implicit abuse. They emphasise the importance of high-quality datasets, particularly for more nuanced expressions of abuse: `Without high quality labelled data to learn these representations, it may be difficult for researchers to come up with models of syntactic structure that can help to identify implicit abuse.' (BIBREF11, p. 81)
Jurgens et al.BIBREF12 discuss also conduct a critical review of hate speech detection, and note that `labelled ground truth data for building and evaluating classifiers is hard to obtain because platforms typically do not share moderated content due to privacy, ethical and public relations concerns.' (BIBREF12, p. 3661) They argue that the field needs to `address the data scarcity faced by abuse detection research' in order to better address more complex rsearch issues and pressing social challenges, such as `develop[ing] proactive technologies that counter or inhibit abuse before it harms' (BIBREF12, pp. 3658, 3661).
Vidgen et al. describe several limitations with existing training datasets for abusive content, most noticeably how `they contain systematic biases towards certain types and targets of abuse.' BIBREF13[p.2]. They describe three issues in the quality of datasets: degradation (whereby datasets decline in quality over time), annotation (whereby annotators often have low agreement, indicating considerable uncertainty in class assignments) and variety (whereby `The quality, size and class balance of datasets varies considerably.' [p. 6]).
Chetty and AlathurBIBREF14 review the use of Internet-based technologies and online social networks to study the spread of hateful, offensive and extremist content BIBREF14. Their review covers both computational and legal/social scientific aspects of hate speech detection, and outlines the importance of distinguishing between different types of group-directed prejudice. However, they do not consider training datasets in any depth.
Fortuna and NunesBIBREF15 provide an end-to-end review of hate speech research, including the motivations for studying online hate, definitional challenges, dataset creation/sharing, and technical advances, both in terms of feature selection and algorithmic architecture (BIBREF15, 2018). They delineate between different types of online abuse, including hate, cyberbullying, discrimination and flaming, and add much needed clarity to the field. They show that (1) dataset size varies considerably but they are generally small (mostly containing fewer than 10,000 entries), (2) Twitter is the most widely-studied platform, and (3) most papers research hate speech per se (i.e. without specifying a target). Of those which do specify a target, racism and sexism are the most researched. However, their review focuses on publications rather than datasets: the same dataset might be used in multiple studies, limiting the relevance of their review for understanding the intrinsic role of training datasets. They also only engage with datasets fairly briefly, as part of a much broader review.
Several classification papers also discuss the most widely used datasets, including Davidson et al. BIBREF16 who describe five datasets, and Salminen et al. who review 17 datasets and describe four in detail BIBREF17.
This paper addresses this lacuna in existing research, providing a systematic review of available training datasets for online abuse. To provide structure to this review, we adopt the `data statements' framework put forward by Bender and Friedman BIBREF18, as well as other work providing frameworks, schema and processes for analysing NLP artefacts BIBREF19, BIBREF20, BIBREF21. Data statements are a way of documenting the decisions which underpin the creation of datasets used for Natural Language Processing (NLP). They formalise how decisions should be documented, not only ensuring scientific integrity but also addressing `the open and urgent question of how we integrate ethical considerations in the everyday practice of our field' (BIBREF18, p. 587). In many cases, we find that it is not possible to fully recreate the level of detail recorded in an original data statement from how datasets are described in publications. This reinforces the importance of proper documentation at the point of dataset creation.
As the field of online abusive content detection matures, it has started to tackle more complex research challenges, such as multi-platform, multi-lingual and multi-target abuse detection, and systems are increasingly being deployed in `the wild' for social scientific analyses and for content moderation BIBREF5. Such research heightens the focus on training datasets as exactly what is being detected comes under greater scrutiny. To enhance our understanding of this domain, our review paper has four research aims.
Research Aim One: to provide an in-depth and critical analysis of the available training datasets for abusive online content detection.
Research Aim Two: to map and discuss ways of addressing the lack of dataset sharing, and as such the lack of `open science', in the field of online abuse research.
Research Aim Three: to introduce the website hatespeechdata.com, as a way of enabling more dataset sharing.
Research Aim Four: to identify best practices for creating an abusive content training dataset.
Analysis of training datasets
Relevant publications have been identified from four sources to identify training datasets for abusive content detection:
The Scopus database of academic publications, identified using keyword searches.
The ACL Anthology database of NLP research papers, identified using keyword searches.
The ArXiv database of preprints, identified using keyword searches.
Proceedings of the 1st, 2nd and 3rd workshops on abusive language online (ACL).
Most publications report on the creation of one abusive content training dataset. However, some describe several new datasets simultaneously or provide one dataset with several distinct subsets of data BIBREF22, BIBREF23, BIBREF24, BIBREF25. For consistency, we separate out each subset of data where they are in different languages or the data is collected from different platforms. As such, the number of datasets is greater than the number publications. All of the datasets were released between 2016 and 2019, as shown in Figure FIGREF17.
Analysis of training datasets ::: The purpose of training datasets ::: Problems addressed by datasets
Creating a training dataset for online abuse detection is typically motivated by the desire to address a particular social problem. These motivations can inform how a taxonomy of abusive language is designed, how data is collected and what instructions are given to annotators. We identify the following motivating reasons, which were explicitly referenced by dataset creators.
Reducing harm: Aggressive, derogatory and demeaning online interactions can inflict harm on individuals who are targeted by such content and those who are not targeted but still observe it. This has been shown to have profound long-term consequences on individuals' well-being, with some vulnerable individuals expressing concerns about leaving their homes following experiences of abuse BIBREF26. Accordingly, many dataset creators state that aggressive language and online harassment is a social problem which they want to help address
Removing illegal content: Many countries legislate against certain forms of speech, e.g. direct threats of violence. For instance, the EU's Code of Conduct requires that all content that is flagged for being illegal online hate speech is reviewed within 24 hours, and removed if necessary BIBREF27. Many large social media platforms and tech companies adhere to this code of conduct (including Facebook, Google and Twitter) and, as of September 2019, 89% of such content is reviewed in 24 hours BIBREF28. However, we note that in most cases the abuse that is marked up in training datasets falls short of the requirements of illegal online hate – indeed, as most datasets are taken from public API access points, the data has usually already been moderated by the platforms and most illegal content removed.
Improving health of online conversations: The health of online communities can be severely affected by abusive language. It can fracture communities, exacerbate tensions and even repel users. This is not only bad for the community and for civic discourse in general, it also negatively impacts engagement and thus the revenue of the host platforms. Therefore, there is a growing impetus to improve user experience and ensure online dialogues are healthy, inclusive and respectful where possible. There is ample scope for improvement: a study showed that 82% of personal attacks on Wikipedia against other editors are not addressed BIBREF29. Taking steps to improve the health of exchanges in online communities will also benefit commercial and voluntary content moderators. They are routinely exposed to such content, often with insufficient safeugards, and sometimes display symptoms similar to those of PTSD BIBREF30. Automatic tools could help to lessen this exposure, reducing the burden on moderators.
Analysis of training datasets ::: The purpose of training datasets ::: Uses of datasets: How detection tasks are defined
Myriad tasks have been addressed in the field of abusive online content detection, reflecting the different disciplines, motivations and assumptions behind research. This has led to considerable variation in what is actually detected under the rubric of `abusive content', and establishing a degree of order over the diverse categorisations and subcategorisations is both difficult and somewhat arbitrary. Key dimensions which dataset creators have used to categorise detection tasks include who/what is targeted (e.g. groups vs. individuals), the strength of content (e.g. covert vs. overt), the nature of the abuse (e.g. benevolent vs. hostile sexism BIBREF31), how the abuse manifests (e.g. threats vs. derogatory statements), the tone (e.g. aggressive vs. non-aggressive), the specific target (e.g. ethnic minorities vs. women),and the subjective perception of the reader (e.g. disrespectful vs. respectful). Other important dimensions include the theme used to express abuse (e.g. Islamophobia which relies on tropes about terrorism vs. tropes about sexism) and the use of particular linguistic devices, such as appeals to authority, sincerity and irony. All of these dimensions can be combined in different ways, producing a large number of intersecting tasks.
Consistency in how tasks are described will not necessarily ensure that datasets can be used interchangeably. From the description of a task, an annotation framework must be developed which converts the conceptualisation of abuse into a set of standards. This formalised representation of the `abuse' inevitably involves shortcuts, imperfect rules and simplifications. If annotation frameworks are developed and applied differently, then even datasets aimed at the same task can still vary considerably. Nonetheless, how detection tasks for online abuse are described is crucial for how the datasets – and in turn the systems trained on them – can subsequently be used. For example, a dataset annotated for hate speech can be used to examine bigoted biases, but the reverse is not true. How datasets are framed also impacts whether, and how, datasets can be combined to form large `mega-datasets' – a potentially promising avenue for overcoming data sparsity BIBREF17.
In the remainder of this section, we provide a framework for splitting out detection tasks along the two most salient dimensions: (1) the nature of abuse and (2) the granularity of the taxonomy.
Analysis of training datasets ::: The purpose of training datasets ::: Uses of datasets: How detection tasks are defined ::: Detection tasks: the nature of abuse
This refers to what is targeted/attacked by the content and, subsequently, how the taxonomy has been designed/framed by the dataset creators. The most well-established taxonomic distinction in this regard is the difference between (i) the detection of interpersonal abuse, and (ii) the detection of group-directed abuse BIBREF11). Other authors have sought to deductively theorise additional categories, such as `concept-directed' abuse, although these have not been widely adopted BIBREF13. Through an inductive investigation of existing training datasets, we extend this binary distinction to four primary categories of abuse which have been studied in previous work, as well as a fifth `Mixed' category.
Person-directed abuse. Content which directs negativity against individuals, typically through aggression, insults, intimidation, hostility and trolling, amongst other tactics. Most research falls under the auspices of `cyber bullying', `harassment' and `trolling' BIBREF23, BIBREF32, BIBREF33. One major dataset of English Wikipedia editor comments BIBREF29 focuses on the `personal attack' element of harassment, drawing on prior investigations that mapped out harassment in that community. Another widely used dataset focuses on trolls' intent to intimidate, distinguishing between direct harassment and other behaviours BIBREF34. An important consideration in studies of person-directed abuse is (a) interpersonal relations, such as whether individuals engage in patterns of abuse or one-off acts and whether they are known to each other in the `real' world (both of which are a key concern in studies of cyberbullying) and (b) standpoint, such as whether individuals directly engage in abuse themselves or encourage others to do so. For example, the theoretically sophisticated synthetic dataset provided by BIBREF33 identifies not only harassment but also encouragement to harassment. BIBREF22 mark up posts from computer game forums (World of Warcraft and League of Legends) for cyberbullying and annotate these as $\langle $offender, victim, message$\rangle $ tuples.
Group-directed abuse. Content which directs negativity against a social identity, which is defined in relation to a particular attribute (e.g. ethnic, racial, religious groups)BIBREF35. Such abuse is often directed against marginalised or under-represented groups in society. Group-directed abuse is typically described as `hate speech' and includes use of dehumanising language, making derogatory, demonising or hostile statements, making threats, and inciting others to engage in violence, amongst other dangerous communications. Common examples of group-directed abuse include sexism, which is included in datasets provided by BIBREF36, BIBREF37, BIBREF38, BIBREF39, BIBREF33 and racism, which is directly targeted in BIBREF36, BIBREF40. In some cases, specific types of group-directed abuse are subsumed within a broader category of identity-directed abuse, as in BIBREF41, BIBREF42, BIBREF4. Determining the limits of any group-directed abuse category requires careful theoretical reflection, as with the decision to include ethnic, caste-based and certain religious prejudices under `racism'. There is no `right' answer to such questions as they engage with ontological concerns about identification and `being' and the politics of categorization.
Flagged content. Content which is reported by community members or assessed by community and professional content moderators. This covers a broad range of focuses as moderators may also remove spam, sexually inappropriate content and other undesirable contributions. In this regard, `flagged' content is akin to the concept of `trolling', which covers a wide range of behaviours, from jokes and playful interventions through to sinister personal attacks such as doxxing BIBREF43. Some forms of trolling can be measured with tools such as the Global Assessment of Internet Trolling (GAIT) BIBREF43.
Incivility. Content which is considered to be incivil, rude, inappropriate, offensive or disrespectful BIBREF24, BIBREF25, BIBREF44. Such categories are usually defined with reference to the tone that the author adopts rather than the substantive content of what they express, which is the basis of person- and group- directed categories. Such content usually contains obscene, profane or otherwise `dirty' words. This can be easier to detect as closed-class lists are effective at identifying single objectionable words (e.g. BIBREF45). However, one concern with this type of research is that the presence of `dirty' words does not necessarily signal malicious intent or abuse; they may equally be used as intensifiers or colloquialisms BIBREF46. At the same time, detecting incivility can be more difficult as it requires annotators to infer the subjective intent of the speaker or to understand (or guess) the social norms of a setting and thus whether disrespect has been expressed BIBREF42. Content can be incivil without directing hate against a group or person, and can be inappropriate in one setting but not another: as such it tends to be more subjective and contextual than other types of abusive language.
Mixed. Content which contains multiple types of abuse, usually a combination of the four categories discussed above. The intersecting nature of online language means that this is common but can also manifest in unexpected ways. For instance, female politicians may receive more interpersonal abuse than other politicians. This might not appear as misogyny because their identity as women is not referenced – but it might have motivated the abuse they were subjected to. Mixed forms of abuse require further research, and have thus far been most fully explored in the OLID dataset provided by BIBREF4, who explore several facets of abuse under one taxonomy.
Analysis of training datasets ::: The purpose of training datasets ::: Uses of datasets: How detection tasks are defined ::: Detection tasks: Granularity of taxonomies
This refers to how much detail a taxonomy contains, reflected in the number of unique classes. The most important and widespread distinction is whether a binary class is used (e.g. Hate / Not) or a multi-level class, such as a tripartite split (typically, Overt, Covert and Non-abusive). In some cases, a large number of complex classes are created, such as by combining whether the abuse is targeted or not along with its theme and strength.
In general, Social scientific analyses encourage creating a detailed taxonomy with a large number of fine-grained categories. However, this is only useful for machine learning if there are enough data points in each category and if annotators are capable of consistently distinguishing between them. Complex annotation schemas may not result in better training datasets if they are not implemented in a robust way. As such, it is unsurprising that binary classification schemas are the most prevalent, even though they are arguably the least useful given the variety of ways in which abuse can be articulated. This can range from the explicit and overt (e.g. directing threats against a group) to more subtle behaviours, such as micro-aggressions and dismissing marginalised groups' experiences of prejudice. Subsuming both types of behaviour within one category not only risks making detection difficult (due to considerable in-class variation) but also leads to a detection system which cannot make important distinctions between qualitatively different types of content. This has severe implications for whether detection systems trained on such datasets can actually be used for downstream tasks, such as content moderation and social scientific analysis.
Drawing together the nature and granularity of abuse, our analyses identify a hierarchy of taxonomic granularity from least to most granular:
Binary classification of a single `meta' category, such as hate/not or abuse/not. This can lead to very general and vague research, which is difficult to apply in practice.
Binary classification of a single type of abuse, such as person-directed or group-directed. This can be problematic given that abuse is nearly always directed against a group rather than `groups' per se.
Binary classification of abuse against a single well-defined group, such as racism/not or Islamophobia/not, or interpersonal abuse against a well-defined cohort, such as MPs and young people.
Multi-class (or multi-label) classification of different types of abuse, such as:
Multiple targets (e.g. racist, sexist and non-hateful content) or
Multiple strengths (e.g. none, implicit and explicit content).
Multiple types (e.g. threats versus derogatory statements or benevolent versus hostile statements).
Multi-class classification of different types of abuse which is integrated with other dimensions of abuse.
Analysis of training datasets ::: The content of training datasets ::: The `Level' of content
49 of the training datasets are annotated at the level of the post, one dataset is annotated at the level of the user BIBREF47, and none of them are annotated at the level of the comment thread. Only two publications indicate that the entire conversational thread was presented to annotators when marking up individual entries, meaning that in most cases this important contextual information is not used. 49 of the training datasets contain only text. This is a considerable limitation of existing research BIBREF13, especially given the multimodal nature of online communication and the increasing ubiquity of digital-specific image-based forms of communication such as Memes, Gifs, Filters and Snaps BIBREF48. Although some work has addressed the task of detecting hateful images BIBREF49, BIBREF50, this lead to the creation of a publically available labelled training dataset in only one case BIBREF51. To our knowledge, no research has tackled the problem of detecting hateful audio content. This is a distinct challenge; alongside the semantic content audio also contains important vocal cues which provide more opportunities to investigate (but also potentially misinterpret) tone and intention.
Analysis of training datasets ::: The content of training datasets ::: Language
The most common language in the training datasets is English, which appears in 20 datasets, followed by Arabic and Italian (5 datasets each), Hindi-English (4 datasets) and then German, Indonesian and Spanish (3 datasets). Noticeably, several major languages, both globally and in Europe, do not appear, which suggests considerable unevenness in the linguistic and cultural focuses of abusive language detection. For instance, there are major gaps in the coverage of European languages, including Danish and Dutch. Surprisingly, French only appears once. The dominance of English may be due to how we sampled publications (for which we used English terms), but may also reflect different publishing practices in different countries and how well-developed abusive content research is.
Analysis of training datasets ::: The content of training datasets ::: Source of data
Training datasets use data collected from a range of online spaces, including from mainstream platforms, such as Twitter, Wikipedia and Facebook, to more niche forums, such as World of Warcraft and Stormfront. In most cases, data is collected from public sources and then manually annotated but in others data is sourced through proprietary data sharing agreements with host platforms. Unsurprisingly, Twitter is the most widely used source of data, accounting for 27 of the datasets. This reflects wider concerns in computational social research that Twitter is over-used, primarily because it has a very accessible API for data collection BIBREF52, BIBREF53. Facebook and Wikipedia are the second most used sources of data, accounting for three datasets each – although we note that all three Wikipedia datasets are reported in the same publication. Many of the most widely used online platforms are not represented at all, or only in one dataset, such as Reddit, Weibo, VK and YouTube.
The lack of diversity in where data is collected from limits the development of detection systems. Three main issues emerge:
Linguistic practices vary across platforms. Twitter only allows 280 characters (previously only 140), provoking stylistic changes BIBREF54, and abusive content detection systems trained on this data are unlikely to work as well with longer pieces of text. Dealing with longer pieces of text could necessitate different classification systems, potentially affecting the choice of algorithmic architecture. Additionally, the technical affordances of platforms may affect the style, tone and topic of the content they host.
The demographics of users on different platforms vary considerably. Social science research indicates that `digital divides' exist, whereby online users are not representative of wider populations and differ across different online spaces BIBREF53, BIBREF55, BIBREF56. Blank draws attention to how Twitter users are usually younger and wealthier than offline populations; over reliance on data from Twitter means, in effect, that we are over-sampling data from this privileged section of society. Blank also shows that there are also important cross-national differences: British Twitters are better-educated than the offline British population but the same is not true for American Twitter users compared with the offline American population BIBREF56. These demographic differences are likely to affect the types of content that users produce.
Platforms have different norms and so host different types and amounts of abuse. Mainstream platforms have made efforts in recent times to `clean up' content and so the most overt and aggressive forms of abuse, such as direct threats, are likely to be taken down BIBREF57. However, more niche platforms, such as Gab or 4chan, tolerate more offensive forms of speech and are more likely to contain explicit abuse, such as racism and very intrusive forms of harassment, such as `doxxing' BIBREF58, BIBREF59, BIBREF60. Over-reliance on a few sources of data could mean that datasets are biased towards only a subset of types of abuse.
Analysis of training datasets ::: The content of training datasets ::: Size
The size of the training datasets varies considerably from 469 posts to 17 million; a difference of four orders of magnitude. Differences in size partly reflect different annotation approaches. The largest datasets are from proprietary data sharing agreements with platforms. Smaller datasets tend to be carefully collected and then manually annotated. There are no established guidelines for how large an abusive language training dataset needs to be. However, smaller datasets are problematic because they contain too little linguistic variation and increase the likelihood of overfitting. Rizoiu et al.BIBREF61 train detection models on only a proportion of the Davidson et al. and Waseem training datasets and show that this leads to worse performance, with a lower F1-Score, particularly for `data hungry' deep learning approaches BIBREF61. At the same time, `big' datasets alone are not a panacea for the challenges of abusive content classification. Large training datasets which have been poorly sampled, annotated with theoretically problematic categories or inexpertly and unthoughtfully annotated, could still lead to the development of poor classification systems.
The challenges posed by small datasets could potentially be overcome through machine learning techniques such as `semi-supervised' and `active' learning BIBREF62, although these have only been limitedly applied to abusive content detection so far BIBREF63. Sharifirad et al. propose using text augmentation and new text generation as a way of overcoming small datasets, which is a promising avenue for future research BIBREF64.
Analysis of training datasets ::: The content of training datasets ::: Class distribution and sampling
Class distribution is an important, although often under-considered, aspect of the design of training datasets. Datasets with little abusive content will lack linguistic variation in terms of what is abusive, thereby increasing the risk of overfitting. More concerningly, the class distribution directly affects the nature of the engineering task and how performance should be evaluated. For instance, if a dataset is 70% hate speech then a zero-rule classification system (i.e. where everything is categorised as hate speech) will achieve 70% precision and 100% recall. This should be used as a baseline for evaluating performance: 80% precision is less impressive compared with this baseline. However, 80% precision on an evenly balanced dataset would be impressive. This is particularly important when evaluating the performance of ternary classifiers, when classes can be considerably imbalanced.
On average, 35% of the content in the training datasets is abusive. However, class distributions vary considerably, from those with just 1% abusive content up to 100%. These differences are largely a product of how data is sampled and which platform it is taken from. Bretschneider BIBREF22 created two datasets without using purposive sampling, and as such they contain very low levels of abuse ( 1%). Other studies filter data collection based on platforms, time periods, keywords/hashtags and individuals to increase the prevalence of abuse. Four datasets comprise only abusive content; three cases are synthetic datasets, reported on in one publication BIBREF65, and in the other case the dataset is an amendment to an existing dataset and only contains misogynistic content BIBREF37.
Purposive sampling has been criticised for introducing various forms of bias into datasets BIBREF66, such as missing out on mis-spelled content BIBREF67 and only focusing on the linguistic patterns of an atypical subset of users. One pressing risk is that a lot of data is sampled from far right communities – which means that most hate speech classifiers implicitly pick up on right wing styles of discourse rather than hate speech per se. This could have profound consequences for our understanding of online political dialogue if the classifiers are applied uncritically to other groups. Nevertheless, purposive sampling is arguably a necessary step when creating a training dataset given the low prevalence of abuse on social media in general BIBREF68.
Analysis of training datasets ::: The content of training datasets ::: Identity of the content creators
The identity of the users who originally created the content in training datasets is described in only two cases. In both cases the data is synthetic BIBREF65, BIBREF33. Chung et al. use `nichesourcing' to synthetically generate abuse, with experts in tackling hate speech creating hateful posts. Sprugnoli et al. ask children to adopt pre-defined roles in an experimental classroom setup, and ask them to engage in a cyberbullying scenario. In most of the non-synthetic training datasets, some information is given about the sampling criteria used to collect data, such as hashtags. However, this does not provide direct insight into who the content creators are, such as their identity, demographics, online behavioural patterns and affiliations.
Providing more information about content creators may help address biases in existing datasets. For instance, Wiegand et al. show that 70% of the sexist tweets in the highly cited Waseem and Hovy dataset BIBREF36 come from two content creators and that 99% of the racist tweets come from just one BIBREF66. This is a serious constraint as it means that user-level metadata is artificially highly predictive of abuse. And, even when user-level metadata is not explicitly modelled, detection systems only need to pick up on the linguistic patterns of a few authors to nominally detect abuse. Overall, the complete lack of information about which users have created the content in most training datasets is a substantial limitation which may be driving as-yet-unrecognised biases. This can be remedied through the methodological rigour implicit in including a data statement with a corpus.
Analysis of training datasets ::: Annotation of training datasets ::: Annotation process
How training datasets are annotated is one of the most important aspects of their creation. A range of annotation processes are used in training datasets, which we split into five high-level categories:
Crowdsourcing (15 datasets). Crowdsourcing is widely used in NLP research because it is relatively cheap and easy to implement. The value of crowdsourcing lies in having annotations undertaken by `a large number of non-experts' (BIBREF69, p. 278) – any bit of content can be annotated by multiple annotators, effectively trading quality for quantity. Studies which use crowdsourcing with only a few annotators for each bit of content risk minimising quality without counterbalancing it with greater quantity. Furthermore, testing the work of many different annotators can be challenging BIBREF70, BIBREF71 and ensuring they are paid an ethical amount may make the cost comparable to using trained experts. Crowdsourcing has also been associated with `citizen science' initiatives to make academic research more accessible but this may not be fully realised in cases where annotation tasks are laborious and low-skilled BIBREF72, BIBREF20.
Academic experts (22 datasets). Expert annotation is time-intensive but is considered to produce higher quality annotations. Waseem reports that `systems trained on expert annotations outperform systems trained on amateur annotations.' BIBREF73 and, similarly, D'Orazio et al. claim, `although expert coding is costly, it produces quality data.' BIBREF74. However, the notion of an `expert' remains somewhat fuzzy within abusive content detection research. In many cases, publications only report that `an expert' is used, without specifying the nature of their expertise – even though this can vary substantially. For example, an expert may refer to an NLP practitioner, an undergraduate student with only modest levels of training, a member of an attacked social group relevant to the dataset or a researcher with a doctorate in the study of prejudice. In general, we anticipate that experts in the social scientific study of prejudice/abuse would perform better at annotation tasks then NLP experts who may not have any direct expertise in the conceptual and theoretical issues of abusive content annotation. In particular, one risk of using NLP practitioners, whether students or professionals, is that they might `game' training datasets based on what they anticipate is technically feasible for existing detection systems. For instance, if existing systems perform poorly when presented with long range dependencies, humour or subtle forms of hate (which are nonetheless usually discernible to human readers) then NLP experts could unintentionally use this expectation to inform their annotations and not label such content as hateful.
Professional moderators (3 datasets). Professional moderators offer a standardized approach to content annotated, implemented by experienced workers. This should, in principle, result in high quality annotations. However, one concern is that moderators are output-focused as their work involves determining whether content should be allowed or removed from platforms; they may not provide detailed labels about the nature of abuse and may also set the bar for content labelled `abusive' fairly high, missing out on more nuance and subtle varieties. In most cases, moderators will annotate for a range of unacceptable content, such as spam and sexual content, and this must be marked in datasets.
A mix of crowdsourcing and experts (6 datasets).
Synthetic data creation (4 datasets). Synthetic datasets are an interesting option as they are inherently non-authentic and therefore not necessarily representative of how abuse manifests in real-world situations. However, if they are created in realistic conditions by experts or relevant content creators then they can mimic real behaviour and have the added advantage that they may have broader coverage of different types of abuse. They are also usually easier to share.
Analysis of training datasets ::: Annotation of training datasets ::: Identity of the annotators
The data statements framework given by Bender and Friedman emphasises the importance of understanding who has completed annotations. Knowing who the annotators are is important because `their own “social address" influences their experience with language and thus their perception of what they are annotating.' BIBREF18 In the context of online abuse, Binns et al. show that the gender of annotators systematically influences what annotations they provide BIBREF75. No annotator will be well-versed in all of the slang or coded meanings used to construct abusive language. Indeed, many of these coded meanings are deliberately covert and obfuscated BIBREF76. To help mitigate these challenges, annotators should be (a) well-qualified and (b) diverse. A homogeneous group of annotators will be poorly equipped to catch all instances of abuse in a corpus. Recruiting an intentionally mixed groups of annotators is likely to yield better recall of abuse and thus a more precise dataset BIBREF77.
Information about annotators is unfortunately scarce. In 23 of the training datasets no information is given about the identity of annotators; in 17 datasets very limited information is given, such as whether the annotator is a native speaker of the language; and in just 10 cases is detailed information given. Interestingly, only 4 out of these 10 datasets are in the English language. Relevant information about annotators can be split into (i) Demographic information and (ii) annotators' expertise and experience. In none of the training sets is the full range of annotator information made available, which includes:
Demographic information. The nature of the task affects what information should be provided, as well as the geographic and cultural context. For instance, research on Islamophobia should include, at the very least, information about annotators' religious affiliation. Relevant variables include:
Age
Ethnicity and race
Religion
Gender
Sexual Orientation
Expertise and experience. Relevant variables include:
Field of research
Years of experience
Research status (e.g. research assistant or post-doc)
Personal experiences of abuse. In our review, none of the datasets contained systematic information about whether annotators had been personally targeted by abuse or had viewed such abuse online, even though this can impact annotators' perceptions. Relevant variables include:
Experiences of being targeted by online abuse.
Experiences of viewing online abuse.
Analysis of training datasets ::: Annotation of training datasets ::: Guidelines for annotation
A key source of variation across datasets is whether annotators were given detailed guidelines, very minimal guidelines or no guidelines at all. Analysing this issue is made difficult by the fact that many dataset creators do not share their annotation guidelines. 21 of the datasets we study do not provide the guidelines and 14 only provide them in a highly summarised form. In just 15 datasets is detailed information given (and these are reported on in just 9 publications). Requiring researchers to publish annotation guidelines not only helps future researchers to better understand what datasets contain but also to improve and extend them. This could be crucial for improving the quality of annotations; as Ross et al. recommend, `raters need more detailed instructions for annotation.' BIBREF78
The degree of detail given in guidelines is linked to how the notion of `abuse' is understood. Some dataset creators construct clear and explicit guidelines in an attempt to ensure that annotations are uniform and align closely with social scientific concepts. In other cases, dataset creators allow annotators to apply their own perception. For instance, in their Portuguese language dataset, Fortuna et al. ask annotators to `evaluate if according to your opinion, these tweets contain hate speech' BIBREF38. The risk here is that authors' perceptions may differ considerably; Salminen et al. show that online hate interpretation varies considerably across individuals BIBREF79. This is also reflected in inter-annotator agreement scores for abusive content, which is often very low, particularly for tasks which deploy more than just a binary taxonomy. However, it is unlikely that annotators could ever truly divorce themselves from their own social experience and background to decide on a single `objective' annotation. Abusive content annotation is better understood, epistemologically, as an intersubjective process in which agreement is constructed, rather than an objective process in which a `true' annotation is `found'. For this reason, some researchers have shifted the question of `how can we achieve the correct annotation?' to `who should decide what the correct annotation is?' BIBREF73. Ultimately, whether annotators should be allowed greater freedom in making annotations, and whether this results in higher quality datasets, needs further research and conceptual examination.
Some aspects of abusive language present fundamental issues that are prone to unreliable annotation, such as Irony, Calumniation and Intent. They are intrinsically difficult to annotate given a third-person perspective on a piece of text as they involve making a judgement about indeterminate issues. However, they cannot be ignored given their prevalence in abusive content and their importance to how abuse is expressed. Thus, although they are fundamentally conceptual problems, these issues also present practical problems for annotators, and should be addressed explicitly in coding guidelines. Otherwise, as BIBREF80 note, these issues are likely to drive type II errors in classification, i.e. labelling non-hate-speech utterances as hate speech.
Analysis of training datasets ::: Annotation of training datasets ::: Guidelines for annotation ::: Irony
This covers statements that have a meaning contrary to that one might glean at first reading. Lachenicht BIBREF81 notes that Irony goes against Grice's quality maxim, and as such Ironic content requires closer attention from the reader as it is prone to being misinterpreted. Irony is a particularly difficult issue as in some cases it is primarily intended to provide humour (and thus might legitimately be considered non-abusive) but in other cases is used as a way of veiling genuine abuse. Previous research suggests that the problem is widespread. Sanguinetti et al. BIBREF82 find irony in 11% of hateful tweets in Italian. BIBREF25 find that irony is one of the most common phenomena in self-deleted comments; and that the prevalence of irony is 33.9% amongst deleted comments in a Croatian comment dataset and 18.1% amongst deleted comments in a Slovene comment dataset. Furthermore, annotating irony (as well as related constructs, such as sarcasm and humour) is inherently difficult. BIBREF83 report that agreement on sarcasm amongst annotators working in English is low, something echoed by annotations of Danish content BIBREF84. Irony is also one of the most common reasons for content to be re-moderated on appeal, according to Pavlopoulos et al. BIBREF24.
Analysis of training datasets ::: Annotation of training datasets ::: Guidelines for annotation ::: Calumniation
This covers false statements, slander, and libel. From the surveyed set, this is annotated in datasets for Greek BIBREF24 and for Croatian and Slovene BIBREF25. Its prevalence varies considerably across these two datasets and reliable estimations of the prevalence of false statements are not available. Calumniation is not only an empirical issue, it also raises conceptual problems: should false information be considered abusive if it slanders or demeans a person? However, if the information is then found out to be true does it make the content any less abusive? Given the contentiousness of `objectivity', and the lack of consensus about most issues in a `post-truth' age BIBREF85, who should decide what is considered true? And, finally, how do we determine whether the content creator knows whether something is true? These ontological, epistemological and social questions are fundamental to the issue of truth and falsity in abusive language. Understandably, most datasets do not taken any perspective on the truth and falsity of content. This is a practical solution: given error rates in abusive language detection as well as error rates in fact-checking, a system which combined both could be inapplicable in practice.
Analysis of training datasets ::: Annotation of training datasets ::: Guidelines for annotation ::: Intent
This information about the utterer's state of mind is a core part of how many types of abusive language are defined. Intent is usually used to emphasize the wrongness of abusive behaviour, such as spreading, inciting, promoting or justifying hatred or violence towards a given target, or sending a message that aims at dehumanising, delegitimising, hurting or intimidating them BIBREF82. BIBREF81 postulate that “aggravation, invective and rudeness ... may be performed with varying degrees of intention to hurt", and cite five legal degrees of intent BIBREF86. However, it is difficult to discern the intent of another speaker in a verbal conversation between humans, and even more difficult to do so through written and computer-mediated communications BIBREF87. Nevertheless, intent is particularly important for some categories of abuse such as bullying, maliciousness and hostility BIBREF34, BIBREF32. Most of the guidelines for the datasets we have studied do not contain an explicit discussion of intent, although there are exceptions. BIBREF88 include intent as a core part of their annotation standard, noting that understanding context (such as by seeing a speakers' other online messages) is crucial to achieving quality annotations. However, this proposition poses conceptual challenges given that people's intent can shift over time. Deleted comments have been used to study potential expressions of regret by users and, as such, a change in their intent BIBREF89, BIBREF25; this has also been reported as a common motivator even in self-deletion of non-abusive language BIBREF90. Equally, engaging in a sequence of targeted abusive language is an indicator of aggressive intent, and appears in several definitions. BIBREF23 require an “intent to physically assert power over women" as a requirement for multiple categories of misogynistic behaviour. BIBREF34 find that messages that are “unapologetically or intentionally offensive" fit in the highest grade of trolling under their schema.
Kenny et al. BIBREF86 note how sarcasm, irony, and humour complicate the picture of intent by introducing considerable difficulties in discerning the true intent of speakers (as discussed above). Part of the challenge is that many abusive terms, such as slurs and insults, are polysemic and may be co-opted by an ingroup into terms of entertainment and endearment BIBREF34.
Dataset sharing ::: The challenges and opportunities of achieving Open Science
All of the training datasets we analyse are publicly accessible and as such can be used by researchers other than the authors of the original publication. Sharing data is an important aspect of open science but also poses ethical and legal risks, especially in light of recent regulatory changes, such as the introduction of GPDR in the UK BIBREF91, BIBREF92. This problem is particularly acute with abusive content, which can be deeply shocking, and some training datasets from highly cited publications have not been made publicly available BIBREF93, BIBREF94, BIBREF95. Open science initiatives can also raise concerns amongst the public, who may not be comfortable with researchers sharing their personal data BIBREF96, BIBREF97.
The difficulty of sharing data in sensitive areas of research is reflected by the Islamist extremism research website, `Jihadology'. It chose to restrict public access in 2019, following efforts by Home Office counter-terrorism officials to shut it down completely. They were concerned that, whilst it aimed to support academic research into Islamist extremism, it may have inadvertently enabled individuals to radicalise by making otherwise banned extremist material available. By working with partners such as the not-for-profit Tech Against Terrorism, Jihadology created a secure area in the website, which can only be accessed by approved researchers. Some of the training datasets in our list have similar requirements, and can only be accessed following a registration process.
Open sharing of datasets is not only a question of scientific integrity and a powerful way of advancing scientific knowledge. It is also, fundamentally, a question of fairness and power. Opening access to datasets will enable less-well funded researchers and organisations, which includes researchers in the Global South and those working for not-for-profit organisations, to steer and contribute to research. This is a particularly pressing issue in a field which is directly concerned with the experiences of often-marginalised communities and actors BIBREF36. For instance, one growing concern is the biases encoded in detection systems and the impact this could have when they are applied in real-world settings BIBREF9, BIBREF10. This research could be further advanced by making more datasets and detection systems more easily available. For instance, Binns et al. use the detailed metadata in the datasets provided by Wulczyn et al. to investigate how the demographics of annotators impacts the annotations they make BIBREF75, BIBREF29. The value of such insights is only clear after the dataset has been shared – and, equally, is only possible because of data sharing.
More effective ways of sharing datasets would address the fact that datasets often deteriorate after they have been published BIBREF13. Several of the most widely used datasets provide only the annotations and IDs and must be `rehydrated' to collect the content. Both of the datasets provided by Waseem and Hovy and Founta et al. must be collected in this way BIBREF98, BIBREF36, and both have degraded considerably since they were first released as the tweets are no longer available on Twitter. Chung et al. also estimate that within 12 months the recently released dataset for counterspeech by Mathew et al. had lost more than 60% of its content BIBREF65, BIBREF58. Dataset degradation poses three main risks: First, if less data is available then there is a greater likelihood of overfitting. Second, the class distributions usually change as proportionally more of the abusive content is taken down than the non-abusive. Third, it is also likely that the more overt forms of abuse are taken down, rather than the covert instances, thereby changing the qualitative nature of the dataset.
Dataset sharing ::: Research infrastructure: Solutions for sharing training datasets
The problem of data access and sharing remains unresolved in the field of abusive content detection, much like other areas of computational research BIBREF99. At present, an ethical, secure and easy way of sharing sensitive tools and resources has not been developed and adopted in the field. More effective dataset sharing would (1) greater collaboration amongst researchers, (2) enhance the reproducibility of research by encouraging greater scrutiny BIBREF100, BIBREF101, BIBREF102 and (3) substantively advance the field by enabling future researchers to better understand the biases and limitations of existing research and to identify new research directions.
There are two main challenges which must be overcome to ensure that training datasets can be shared and used by future researchers. First, dataset quality: the size, class distribution and quality of their content must be maintained. Second, dataset access: access to datasets must be controlled so that researchers can use them, whilst respecting platforms' Terms of Service and not allowing potential extremists from having access. These problems are closely entwined and the solutions available, which follow, have implications for both of them.
Synthetic datasets. Four of the datasets we have reviewed were developed synthetically. This resolves the dataset quality problem but introduces additional biases and limitations because the data is not real. Synthetic datasets still need to be shared in such a way as to limit access for potential extremists but face no challenges from Platforms' Terms of Services.
Data `philanthropy' or `donations'. These are defined as `the act of an individual actively consenting to donate their personal data for research' BIBREF97. Donated data from many individuals could then be combined and shared – but it would still need to be annotated. A further challenge is that many individuals who share abusive content may be unwilling to `donate' their data as this is commonly associated with prosocial motivations, creating severe class imbalances BIBREF97. Data donations could also open new moral and ethical issues; individuals' privacy could be impacted if data is re-analysed to derive new unexpected insights BIBREF103. Informed consent is difficult given that the exact nature of analyses may not be known in advance. Finally, data donations alone do not solve how access can be responsibly protected and how platforms' Terms of Service can be met. For these reasons, data donations are unlikely to be a key part of future research infrastructure for abusive content detection.
Platform-backed sharing. Platforms could share datasets and support researchers' access. There are no working examples of this in abusive content detection research, but it has been successfully used in other research areas. For instance, Twitter has made a large dataset of accounts linked to potential information operations, known as the “IRA" dataset (Internet Research Agency). This would require considerably more interfaces between academia and industry, which may be difficult given the challenges associated with existing initiatives, such as Social Science One. However, in the long term, we propose that this is the most effective solution for the problem of sharing training datasets. Not only because it removes Terms of Service limitations but also because platforms have large volumes of original content which has been annotated in a detailed way. This could take one of two forms: platforms either make content which has violated their Community Guidelines available directly or they provide special access post-hoc to datasets which researchers have collected publicly through their API - thereby making sure that datasets do not degrade over time.
Data trusts. Data trusts have been described as a way of sharing data `in a fair, safe and equitable way' ( BIBREF104 p. 46). However, there is considerable disagreement as to what they entail and how they would operate in practice BIBREF105. The Open Data Institute identifies that data trusts aim to make data open and accessible by providing a framework for storing and accessing data, terms and mechanisms for resolving disputes and, in some cases, contracts to enforce them. For abusive content training datasets, this would provide a way of enabling datasets to be shared, although it would require considerable institutional, legal and financial commitments.
Arguably, the easiest way of ensuring data can be shared is to maintain a very simple data trust, such as a database, which would contain all available abusive content training datasets. This repository would need to be permissioned and access controlled to address concerns relating to privacy and ethics. Such a repository could substantially reduce the burden on researchers; once they have been approved to the repository, they could access all datasets publicly available – different levels of permission could be implemented for different datasets, depending on commercial or research sensitivity. Furthermore, this repository could contain all of the metadata reported with datasets and such information could be included at the point of deposit, based on the `data statements' work of Bender and Friedman BIBREF18. A simple API could be developed for depositing and reading data, similar to that of the HateBase. The permissioning system could be maintained either through a single institution or, to avoid power concentrating amongst a small group of researchers, through a decentralised blockchain.
Dataset sharing ::: A new repository of training datasets: Hatespeechdata.com
The resources and infrastructure to create a dedicated data trust and API for sharing abusive content training datasets is substantial and requires considerable further engagement with research teams in this field. In the interim, to encourage greater sharing of datasets, we have launched a dedicated website which contains all of the datasets analysed here: https://hatespeechdata.com. Based on the analysis in the previous sections, we have also provided partial data statements BIBREF18. The website also contains previously published abusive keyword dictionaries, which are not analysed here but some researchers may find useful. Note that the website only contains information/data which the original authors have already made publicly available elsewhere. It will be updated with new datasets in the future.
Best Practices for training dataset creation
Much can be learned from existing efforts to create abusive language datasets. We identify best practices which emerge at four distinct points in the process of creating a training dataset: (1) task formation, (2) data selection, (3) annotation, and (4) documentation.
Best Practices for training dataset creation ::: Task formation: Defining the task addressed by the dataset
Dataset creation should be `problem driven' BIBREF106 and should address a well-defined and specific task, with a clear motivation. This will directly inform the taxonomy design, which should be well-specified and engage with social scientific theory as needed. Defining a clear task which the dataset addresses is especially important given the maturation of the field, ongoing terminological disagreement and the complexity of online abuse. The diversity of phenomena that fits under the umbrella of abusive language means that `general purpose' datasets are unlikely to advance the field. New datasets are most valuable when they address a new target, generator, phenomenon, or domain. Creating datasets which repeat existing work is not nearly as valuable.
Best Practices for training dataset creation ::: Selecting data for abusive language annotation
Once the task is established, dataset creators should select what language will be annotated, where data will be sampled from and how sampling will be completed. Any data selection exercise is bound to give bias, and so it is important to record what decisions are made (and why) in this step. Dataset builders should have a specific target size in mind and also have an idea of the minimum amount of data this is likely to be needed for the task. This is also where steps 1 and 2 intersect: the data selection should be driven by the problem that is addressed rather than what is easy to collect. Ensuring there are enough positive examples of abuse will always be challenging as the prevalence of abuse is so low. However, given that purposive sampling inevitably introduces biases, creators should explore a range of options before determining the best one – and consider using multiple sampling methods at once, such as including data from different times, different locations, different types of users and different platforms. Other options include using measures of linguistic diversity to maximize the variety of text included in datasets, or including words that cluster close to known abusive terms.
Best Practices for training dataset creation ::: Annotating abusive language
Annotators must be hired, trained and given appropriate guidelines. Annotators work best with solid guidelines, that are easy to grasp and have clear examples BIBREF107. The best examples are both illustrative, in order to capture the concepts (such as `threatening language') and provide insight into `edge cases', which is content that only just crosses the line into abuse. Decisions should be made about how to handle intrinsically difficult aspects of abuse, such as irony, calumniation and intent (see above). Annotation guidelines should be developed iteratively by dataset creators; by working through the data, rules can be established for difficult or counter-intuitive coding decisions, and a set of shared practices developed. Annotators should be included in this iterative process. Discussions with annotators the language that they have seen “in the field" offers an opportunity to enhance and refine guidelines - and even taxonomies. Such discussions will lead to more consistent data and provide a knowledge base to draw on for future work. To achieve this, it is important to adopt an open culture where annotators are comfortable providing open feedback and also describing their uncertainties. Annotators should also be given emotional and practical support (as well as appropriate financial compensation), and the harmful and potentially triggering effects of annotating online abuse should be recognised at all times. For a set of guidelines to help protect the well-being of annotators, see BIBREF13.
Best Practices for training dataset creation ::: Documenting methods, data, and annotators
The best training datasets provide as much information as possible and are well-documented. When the method behind them is unclear, they are hard to evaluate, use and build on. Providing as much information as possible can open new and unanticipated analyses and gives more agency to future researchers who use the dataset to create classifiers. For instance, if all annotators' codings are provided (rather than just the `final' decision) then a more nuanced and aware classifier could be developed as, in some cases, it can be better to maximise recall of annotations rather than maximise agreement BIBREF77.
Our review found that most datasets have poor methodological descriptions and few (if any) provide enough information to construct an adequate data statement. It is crucial that dataset creators are up front about their biases and limitations: every dataset is biased, and this is only problematic when the biases are unknown. One strategy for doing this is to maintain a document of decisions made when designing and creating the dataset and to then use it to describe to readers the rationale behind decisions. Details about the end-to-end dataset creation process are welcomed. For instance, if the task is crowdsourced then a screenshot of the micro-task presented to workers should be included, and the top-level parameters should be described (e.g. number of workers, maximum number of tasks per worker, number of annotations per piece of text) BIBREF20. If a dedicated interface is used for the annotation, this should also be described and screenshotted as the interface design can influence the annotations.
Best Practices for training dataset creation ::: Best practice summary
Unfortunately, as with any burgeoning field, there is confusion and overlap around many of the phenomena discussed in this paper; coupled with the high degree of variation in the quality of method descriptions, it has lead to many pieces of research that are hard to combine, compare, or re-use. Our reflections on best practices are driven by this review and the difficulties of creating high quality training datasets. For future researchers, we summarise our recommendations in the following seven points:
Bear in mind the purpose of the dataset; design the dataset to help address questions and problems from previous research.
Avoid using `easy to access' data, and instead explore new sources which may have greater diversity. Consider what biases may be created by your sampling method.
Determine size based on data sparsity and having enough positive classes rather than `what is possible'.
Establish a clear taxonomy to be used for the task, with meaningful and theoretically sound categories.
Provide annotators with guidelines; develop them iteratively and publish them with your dataset. Consider using trained annotators given the complexities of abusive content.
Involve people who have direct experience of the abuse which you are studying whenever possible (and provided that you can protect their well-being).
Report on every step of the research through a Data Statement.
Conclusion
This paper examined a large set of datasets for the creation of abusive content detection systems, providing insight into what they contain, how they are annotated, and how tasks have been framed. Based on an evidence-driven review, we provided an extended discussion of how to make training datasets more readily available and useful, including the challenges and opportunities of open science as well as the need for more research infrastructure. We reported on the development of hatespeechdata.com – a new repository for online abusive content training datasets. Finally, we outlined best practices for creation of training datasets for detection of online abuse. We have effectively met the four research aims elaborated at the start of the paper.
Training detection systems for online abuse is a substantial challenge with real social consequences. If we want the systems we develop to be useable, scalable and with few biases then we need to train them on the right data: garbage in will only lead to garbage out.

Question: Have any baseline model been trained on this abusive language dataset?
Answer: No

Question: How big are this dataset and catalogue?
Answer:  from 469 posts to 17 million

Question: What is open website for cataloguing abusive language data?
Answer: hatespeechdata.com