SlowGuess/NAACL_2021_OCR huggingface dataset:
Rationales For all categories other than Neutral, annotators highlighted the part of the entry related to the category. This is important for Reddit data where some comments are very long; the longest entry in our dataset has over 10k characters. As part of the adjudication process, just one rationale was selected for each entry, giving a single 'gold standard'.
Inter annotator agreement Inter annotator agreement for the primary categories was measured using Fleiss' Kappa. It was 'moderate' overall (0.583) (Mchugh, 2012). This compares favourably with other abusive content datasets (Gomez et al., 2020; Fortuna and Nunes, 2018; Wulczyn et al., 2017), especially given that our taxonomy contains six primary categories. Agreement was highest for Non-hateful slurs (0.754). It was consistently 'moderate' for Neutral (0.579), Person (0.513), Affiliation (0.453) and Identity (0.419) but was lower for Counter Speech (0.267). This reflects Counter Speech's low prevalence (meaning annotators were less experienced at identifying it) and the subjective nature of judging whether content counters abuse or is implicitly supportive. One challenge is that if annotators missed a category early on in a thread then they would also miss all subsequent context-dependent entries.
4.3 Prevalence of categories
The prevalence of the primary and secondary categories in the dataset is shown in Table 3. Non
Category	Fleiss' Kappa
Affiliation directed	0.453
Identity directed	0.498
Person directed	0.513
Counter Speech	0.267
Non-hateful Slurs	0.754
Neutral	0.579
AVERAGE	0.583
hateful Slurs and Neutral entries do not have secondary categories and so only the total is shown. Neutral entries dominate, accounting for  $79.8\%$  of the data, followed by Identity-directed abuse which accounts for  $9.9\%$ , Affiliation-directed abuse  $(5.0\%)$ , Person-directed abuse  $(4.0\%)$ , Counter Speech  $(0.8\%)$  and Non-hateful use of slurs  $(0.5\%)$ . Animosity and Derogation are the most frequent secondary categories in Identity-directed and Affiliation-directed abuse, with Threatening language, Dehumanization and Glorification accounting for less than  $5\%$  combined. This is unsurprising given the severity of such language. Other training datasets for online abuse generally report similar or slightly higher levels of non-neutral content, e.g., in Gomez et al. (2020)  $82\%$  is neutral, in Waseem and Hovy (2016)  $68\%$  is not hateful, in both Zampieri et al. (2019) and Vidgen et al. (2020)  $67\%$ , and in Founta et al. (2018)  $58\%$  is neutral.
Data splits For our classification experiments, we exclude entries that are “[removed]”, “[deleted]” or empty because they were either a blank entry associated with a post title or a entry that only contained an image. We also exclude entries written by two prolific bots (SnapshillBot and AutoModerator) and non-English entries, which were identified by langid.py (Lui and Baldwin, 2012) and then manually verified. Entries with an image were included but the image was not used for classification. The dataset used for experiments contains 23,417 entries and is split into a train (13,584;  $58\%$ ), development (4,526;  $19.3\%$ ) and test set (5,307;  $22.7\%$ ). All entries belonging to the same thread are assigned to the same split. A small set of subreddits only occurs in either the development or the test set; this allows us to test performance on entries in subreddits that were not included in training.
Table 2: Average Kappa scores for primary categories
Primary	Secondary	Number	Percentage
Affiliation-directed	Derogation	629	46.0
Animosity	676	49.5
Threatening	30	2.2
Dehumanization	31	2.3
Glorification	0	0.0
Total	1,366	100
Identity-directed	Derogation	1,026	37.8
Animosity	1,577	58.1
Threatening	31	1.1
Dehumanization	29	1.1
Glorification	49	1.8
Total	2,712	100
Person-directed	About a person	552	49.6
To a person	560	50.4
Total	1,112	100
Counter Speech	Affiliation	53	24.1
Identity	115	52.3
Person	52	23.6
Total	220	100
Non-hateful Slurs	Total	149	100
Neutral	Total	21,935	100
TOTAL	27,494	100
Table 3: Prevalence of the categories
Hyperparameters are tuned on the development set.
Classification task We automatically classify the primary categories. Due to the low prevalence of Non-hateful Slurs, these are not used as a separate category in the classification experiments. Instead, for the experiments, we re-assign entries with only a Non-hateful Slur label to Neutral. For entries that have a Non-hateful Slur label and at least one other label, we simply ignore the Nonhateful Slur label<sup>2</sup>.  $1.94\%$  of entries in the training set have more than one primary category. When we exclude Neutral entries (because these entries cannot have another category), this increases to  $10.5\%$ . The training data has a label cardinality of 1.02 (Tsoumakas and Katakis, 2007). We thus formulate the task as a multilabel classification problem. It is challenging given the highly skewed label distributions, the influence of context, and the multilabel setup.
We compare several popular baseline models. We only use the texts of entries as input. The context of entries (e.g., previous entries in a thread) are
2This is in-line with our taxonomy, whereby entries assigned to Neutral cannot be assigned to any of the other categories.
not taken into account; integrating context could be explored in future work.
Logistic Regression (LR) We use Logistic Regression with L2 regularization, implemented using scikit-learn (Pedregosa et al., 2011). There are different approaches to multilabel classification (Boutell et al., 2004; Tsoumakas and Katakis, 2007). One common approach is the Label Power-set method, where a new label is created for each unique label combination. However, this approach is not suitable for our data; many label combinations only have a few instances. Furthermore, classifiers would not be able to recognise unseen label combinations. We therefore use a binary relevance setup, where binary classifiers are trained for each label separately. Because the class distribution is heavily skewed, classes are weighted inversely proportional to their frequencies in the training data.
BERT and DistilBERT We finetune the BERT base uncased model (Devlin et al., 2019) with commonly used hyperparameters (see the Appendix). Given BERT's sensitivity to random seeds (Dodge et al., 2020), each setting was run with five different random seeds. Our implementation uses the Hugging Face's Transformers library (Wolf et al., 2019). We use a binary cross entropy loss and encode the labels as multi-hot vectors. Classes are weighted by their ratio of negative over positive examples in the training data. We also finetune DistilBERT (Sanh et al., 2019), a lighter version of BERT trained with knowledge distillation.
Evaluation metrics The precision, recall and F1 score for each primary category are reported in Table 4. In Table 5, we report micro and macro average F1 scores. Because of the highly skewed class distribution, we favor macro F1 scores. We also report the exact match accuracy (the fraction of entries for which the full set of labels matches).
Classifier comparison BERT performs best and achieves a substantial performance improvement over Logistic Regression (Macro F1 of 0.455 vs. 0.343). The performance of DistilBERT is slightly lower, but very close to BERT's performance. With both BERT and DistilBERT there is still much room for improvement on most categories. Note that a majority class classifier which labels everything as Neutral would achieve a high accuracy (0.818) but a low F1 macro score (0.180). There
were no clear performance differences between entries from subreddits that were or were not included in the training data.
Primary categories Performance differs substantially between the different categories (Table 4). All classifiers attain high F1 scores on Neutral entries (LR: 0.859, BERT: 0.902); this is expected as the class distribution is highly skewed towards Neutral. Performance is lowest on Counter Speech (LR: 0.042, BERT: 0.091), possibly due to a combination of factors. First, this category has the lowest number of training instances. Second, inter-annotator agreement was lowest on Counter Speech. And third, all Counter Speech annotations are based on previous content in the thread.
Error analysis Qualitative analysis shows that the BERT model often misclassifies neutral content which mention identities (e.g., non-misogynistic discussions of women) or contains profanities and aggressive language. It tends to classify Affiliation- and Identity-directed abuse which uses less aggressive language and contains fewer abusive keywords as Neutral. Surprisingly, many of the Person-directed entries which are misclassified as Neutral contain clear signals of abuse, such as profanities and overt aggression. No discernible pattern was observed with Counter Speech which was misclassified as a different category. For this category, the low performance may be attributed mostly to its low frequency in the training data.
Context Our benchmark models do not explicitly take into account context for prediction. As expected, all our models are worse at predicting the primary categories of entries where context was required for the annotation. For example, with logistic regression, the recall for Identity-directed abuse is  $21.1\%$  for entries where the annotation was based on previous content compared with  $46.3\%$  for entries where the annotation is based only on the current content. Similarly, with BERT the recall for Identity-directed abuse increases from  $25.3\%$  ('Previous') to  $60.1\%$  ('Current').
Secondary categories We compare recall between the secondary categories. For Person-directed abuse, the recall with LR for abuse targeting a person who is not in the thread is substantially lower than for entries that are directed to a person in the thread with (25.2% vs. 35.6%). For BERT and DistilBERT, the performance difference
LR	DistilBERT	BERT
P	R	F1	P	R	F1	P	R	F1
Neutral	0.872	0.845	0.859	0.880	0.917	0.898	0.883	0.922	0.902
Identity-directed	0.281	0.398	0.330	0.414	0.473	0.441	0.411	0.510	0.455
Affiliation-directed	0.229	0.395	0.290	0.368	0.450	0.405	0.368	0.481	0.416
Person-directed	0.145	0.304	0.196	0.359	0.404	0.380	0.356	0.488	0.411
Counter Speech	0.032	0.061	0.042	0.083	0.073	0.076	0.107	0.088	0.091
Table 4: Scores per category on the test set. For DistilBERT and BERT these are the means over 5 runs.
Accuracy	F1 (macro)	F1 (micro)
LR	0.634	0.343	0.711
DistilBERT	0.769 (0.005)	0.440 (0.007)	0.797 (0.005)
BERT	0.762 (0.005)	0.455 (0.006)	0.799 (0.005)
Table 5: Results on the test set. For BERT and DistilBERT the standard deviations are also reported.
between these two secondary categories is small (e.g., BERT:  $48.6\%$  vs.  $49.0\%$ ). Furthermore, for Identity-directed abuse the recall for animosity (LR:  $36.2\%$ , BERT:  $45.3\%$ ) tends to be lower than the recall for derogation (LR:  $49.0\%$ , BERT:  $65.9\%$ ), which is expected as animosity expresses abuse in an implicit manner and is often more nuanced. The larger difference for BERT vs. logistic regression shows the promise of more advanced models in distinguishing subcategories. For Affiliation-directed abuse, the differences are smaller. Here, the recall for animosity is ( unexpectedly) slightly higher (LR:  $43.3\%$ , BERT:  $49.5\%$ ) than for derogation (LR:  $36.1\%$ , BERT:  $48.0\%$ ).
Label dependence The multilabel setup of this classification task makes this a challenging problem. All models tend to assign too many labels. For example, DistilBERT predicts only too few labels in  $1.17\%$  of the cases, the remainder predicting the right number  $(91.88\%)$  or too many  $(6.96\%)$ . For BERT, the difference is even higher  $(1.06\%)$  too few;  $9.21\%$  too many labels).
Dependencies between labels are sometimes violated. In our taxonomy, entries which are Neutral cannot have another label, but our models violate this constraint in many cases. With DistilBERT  $3.8\%$  of the entries are classified as Neutral and at least one other class, this is even more so for BERT  $(5.4\%)$  and (LR:  $10.7\%$ ). Future work could therefore explore modeling relationships between labels.
We have presented a detailed dataset for training abusive content classification systems. It incorporates relevant social scientific concepts, providing a more nuanced and robust way of characterising—and therefore detecting—abuse. We have also presented benchmark experiments, which show much room for improvement.
Our analyses indicate numerous areas to explore further, including creating systems which explicitly model the conversation threads to account for context. Predictive methods could be applied to understand and forecast when a conversation is turning toxic, potentially enabling real-time moderation interventions. More powerful models could also be applied to better distinguish the primary categories and to begin classification of the secondary categories. This could be achieved by also using the images to classify the content, which we did not do. Finally, we would also expect the rationales to be of considerable use in future experiments, both for classification and to understand the annotation process.
The current work has several limitations. First, the class distribution is heavily skewed towards the Neutral class and some abusive categories have low frequencies. This better reflects real-world prevalence of abuse but can limit the signals available for classification. Second, inter-annotator agreement was in-line with other research in this domain but could still be improved further, especially with 'edge case' content.
We follow the ACM's Code of Ethics and Professional conduct<sup>3</sup>, as well as academic guidelines for ethically researching activity on social media (Townsend and Wallace, 2017; Williams, 2019). Online abuse poses substantial risk of harm to online users and their communities, and there is a
<sup>3</sup>https://www.acm.org/code-of-ethics
strong social justification for conducting this work.
Dataset collection We used the Pushshift API to collect data from Reddit4, which we accessed through the data dumps on Google's BigQuery using  $\mathbb{R}^5$ . The Pushshift API is a wrapper which allows large quantities of Reddit data to be accessed reliably and easily (Baumgartner et al., 2020; Gaffney and Matias, 2018). Our collection is consistent with Reddit's Terms of Service.
Ethical approval This project was given ethical approval on 18th March 2019, before any research had started, by The Alan Turing Institute (submission C1903-053). Reddit can be considered a public space in that discussion are open and posts are aimed at a large audience. In this way, it differs from a one-to-one or 'private' messaging service. When users sign up to Reddit, they consent to have their data made available to third parties, such as academics. Many users are aware of this and choose to use non-identifiable pseudonyms. Existing ethical guidance indicates that in this situation explicit consent is not required from each user (which is often infeasible), provided that harm to users is minimized at all times (Williams, 2019) and no 'real' quotes are attributed to them in the paper. We follow this guidance and do not provide any direct quotes. The examples given in Table 1 are synthetic. We also minimized how many entries we collected from each user so that each one comprises only a small part of the total dataset. At no point did any of the research team contact any Reddit users, minimizing the risk that any harm could be caused to them. Further, we decided not to review any profile information about the users, substantially minimizing the risk that any personally identifiable information is included in the dataset.
Treatment of annotators We used trained annotators that were carefully recruited through the host institution (in line with their HR procedures). Crowdsourced workers were not used. Annotators were carefully supervised with weekly meetings and regular one-to-one discussions. We followed the guidelines provided by Vidgen et al. (2019a) for ensuring annotator welfare during the work. We provided annotators with access to support services throughout the project, including counselling support, although they were not used. Annotators were
paid substantially above the living wage. They were paid holiday and all meetings and training time was paid.
Research team wellbeing To protect the wellbeing of the research team, we had regular catchup discussions, and made sure that the lead researchers were not exposed excessively to harmful content. We did not post anything about the project whilst it was conducted (to minimize the risk of attracting the attention of malicious online actors) and did not engage with any of the Reddit users or communities being studied.
Dataset information and quality We provide a Data Statement in the Appendix, following Bender and Friedman (2018), with full information about the dataset.
Baseline models We present baseline classification models in the paper. We have carefully considered how these models could be deployed and believe that this is highly unlikely given their performance. There is a risk of bias in any dataset, and associated models, and we have sought to provide as much information as possible in our dataset, documentation and other artefacts to enable future researchers to investigate these issues. We do not use demographic or identity characteristics in the formation of the dataset. We also do not provide information about individual annotators, only giving the overall profile of the annotation team. The computational time/power involved in creating the baselines was minimal.
This work was supported by Wave 1 of The UKRI Strategic Priorities Fund under the EPSRC Grant EP/T001569/1, particularly the "Criminal Justice System" theme within that grant, and The Alan Turing Institute.
The authors would like to thank Zeerak Waseem and Paul Röttger for their helpful advice, as well as feedback from attendees at the Hertie School's 2020 workshop on online hate regulation.
Maria Anzovino, Elisabetta Fersini, and Paolo Rosso. 2018. Automatic identification and classification of misogynistic language on Twitter. In NLDB, pages 57-64.
4https://pushshift.io/api-parameters/ 5https://pushshift.io/ using-bigquery-with-redis-it-data/
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The Pushshift Reddit Dataset. In Proceedings of the 14th International Conference on Web and Social Media, pages 830-839.
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604.
Matthew R. Boutell, Jiebo Luo, Xipeng Shen, and Christopher M. Brown. 2004. Learning multi-label scene classification. Pattern Recognition, 37(9):1757-1771.
Luke M. Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019. Finding microaggressions in the wild: A case for locating elusive phenomena in social media posts. In Proceedings of the 9th EMNLP-IJCNLP, pages 1664-1674.
Dallas Card, Amber E. Boydstun, Justin H. Gross, Philip Resnik, and Noah A. Smith. 2015. The media frames corpus: Annotations of frames across issues. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 438-444.
Danielle Citron and Helen Norton. 2011. Intermediaries and Hate Speech: Fostering Digital Citizenship for Our Information Age. Boston University Law Review, 91(16):1435-2131.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated Hate Speech Detection and the Problem of Offensive Language. In Proceedings of the 11th ICWSM, pages 1-4.
Thomas Davidson and Ingmar Weber. 2019. Racial Bias in Hate Speech and Abusive Language Detection Datasets. In 3rd Workshop on Abusive Language Online (ACL), pages 1-11.
Ona de Gibert, Naiara Perez, Aitor Garcia-Pablos, and Montse Cuadros. 2018. Hate Speech Dataset from a White Supremacy Forum. In Proceedings of the Second Workshop on Abusive Language Online (ACL), pages 11-20.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.
Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. ACM Comput. Surv., 51(4).
Paula Fortuna, Juan Soler-company, and Leo Wanner. 2020. Toxic, Hateful, Offensive or Abusive? What Are We Really Classifying? An Empirical Analysis of Hate Speech Datasets. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC), pages 6786-6794.
Antigoni-maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior. In ICWSM, pages 1-11.
Devin Gaffney and J Nathan Matias. 2018. Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus. Plos ONE, 13(7):1-13.
Lei Gao and Ruihong Huang. 2017. Detecting Online Hate Speech Using Context Aware Models. In Proceedings of Recent Advances in Natural Language Processing, pages 260-266.
Jennifer Golbeck, Alicia A. Geller, Jayanee Thanki, Shalmali Naik, Kelly M. Hoffman, Derek Michael Wu, Alexandra Berlinger, Priyanka Vengataraman, Shivika Khare, Zahra Ashktorab, Marianna J. Martindale, Gaurav Shahane, Paul Cheakalos, Jenny Hottle, Siddharth Bhagwan, Raja Rajan Gunasekaran, Rajesh Kumar Gnanasekaran, Rashad O. Banjo, Piyush Ramachandran, Lisa Rogers, Kristine M. Rogers, Quint Gergory, Heather L. Nixon, Meghna Sardana Sarin, Zijian Wan, Cody Buntain, Ryan Lau, and Vichita Jienjitlert. 2017. A Large Labeled Corpus for Online Harassment Research. In Proceedings of the ACM Conference on Web Science, pages 229-233.
Raul Gomez, Jaume Gibert, Lluis Gomez, and Dimosthenis Karatzas. 2020. Exploring Hate Speech Detection in Multimodal Publications. In Proceedings of the Winter Conference on Applications of Computer Vision, pages 1-8.
Hugo Lewi Hammer. 2014. Detecting threats of violence in online discussions using bigrams of important words. In Proceedings - 2014 IEEE Joint Intelligence and Security Informatics Conference, JISIC 2014, page 319.
Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning Whom to Trust with MACE. In Proceedings of NAACL-HLT, pages 1120-1130.
Robin Jeshion. 2013. Slurs and Stereotypes. Analytic Philosophy, 54(3):314-329.
Jolanda Jetten, Russell Spears, and Tom Postmes. 2004. Intergroup Distinctiveness and Differentiation: A Meta-Analytic Integration. Journal of personality and social psychology, 86(6):862-879.