SlowGuess/NAACL_2021_OCR huggingface dataset:
David Jurgens, Eshwar Chandrasekharan, and Libby Hemphill. 2019. A Just and Comprehensive Strategy for Using NLP to Address Online Abuse. In Proceedings of the 57th Annual Meeting of the ACL, pages 3658-3666.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes. arXiv:2005.04790v2, pages 1-17.
Ritesh Kumar, Atul Kr Ojha, Shervin Malmasi, and Marcos Zampieri. 2018. Benchmarking Aggression Identification in Social Media. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying., 1, pages 1-11.
Jonathan Leader Maynard and Susan Benesch. 2016. Dangerous Speech and Dangerous Ideology: An Integrated Model for Monitoring and Prevention. Genocide Studies and Prevention, 9(3):70-95.
Amanda Lenhart, Michelle Ybarra, Kathryn Zickuhr, and Myeshia Price-Feeney. 2016. Online harrassment, digital abuse, and cyberstalk in America. Data & Society, New York.
Marco Lui and Timothy Baldwin. 2012. langid.py: An off-the-shelf language identification tool. In Proceedings of the ACL 2012 System Demonstrations, pages 25-30, Jeju Island, Korea. Association for Computational Linguistics.
Thomas Mandl, Sandip Modha, Prasenjit Majumder, Daksh Patel, Mohana Dave, Chintak Mandlia, and Aditya Patel. 2019. Overview of the HASOC track at FIRE 2019: Hate speech and offensive content identification in Indo-European languages. In FIRE '19: Proceedings of the 11th Forum for Information Retrieval Evaluation, pages 14-17.
Alice E Marwick and Ross Miller. 2014. Online Harassment, Defamation, and Hateful Speech: A Primer of the Legal Landscape Recommended Citation. Center on Law and Information Policy at Fordham Law School, New York.
Binny Mathew, Punyajoy Saha, Hardik Tharad, Subham Rajgaria, Prajwal Singhania, Suman Kalyan Maity, Pawan Goyal, and Animesh Mukherjee. 2019. Thou shalt not hate: Countering online hate speech. In Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019, pages 369-380.
Mary Matsuda, Charles Lawrence, Richard Delgado, and Kimberlé Crenshaw. 1993. Words that Wound: Critical race theory, assaultive speech and the First Amendment. Routledge, New York.
Mary L Mchugh. 2012. Interrater reliability: the Kappa statistic. Biochemia Medica, 22(3):276-282.
Julia Mendelsohn, Yulia Tsvetkov, and Dan Jurafsky. 2020. A Framework for the Computational Linguistic Analysis of Dehumanization. Frontiers in Artificial Intelligence, 3(August):1-24.
Hamdy Mubarak, Kareem Darwish, and Walid Magdy. 2017. Abusive Language Detection on Arabic Social Media. In Proceedings of the First Workshop on Abusive Language Online, pages 52-56.
Nedjma Ousidhoum, Yangqiu Song, and Dit-yan Yeung. 2020. Comparative Evaluation of Label-Agnostic Selection Bias in Multilingual Hate Speech Datasets. In Proceedings of EMNLP. Association for Computational Linguistics.
Alexis Palmer, Christine Carr, Melissa Robinson, and Jordan Sanders. 2020. COLD: Annotation scheme and evaluation data set for complex offensive language in English. The Journal for Language Technology and Computational Linguistics, 34(1):1-28.
John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. 2020. Toxicity Detection: Does Context Really Matter? arXiv:2006.00998, pages 1-10.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Rob Procter, Helena Webb, Pete Burnap, William Housley, Adam Edwards, Matthew Williams, and Marina Jirotka. 2019. A Study of Cyber Hate on Twitter with Implications for Social Media Governance Strategies. In Proceedings of Truth and Trust 2020.
Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth Belding, and William Yang Wang. 2019. A Benchmark Dataset for Learning to Intervene in Online Hate Speech. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4757-4766.
Patricia Rossini. 2019. Toxic for whom? Examining the targets of uncivil and intolerant discourse in online political talk. In *Voices: Exploring the shifting contours of communication*, pages 221-242. Peter Lang, New York.
Patricia Rossini. 2020. Beyond Incivility: Understanding Patterns of Uncivil and Intolerant Discourse in Online Political Talk. Communication Research, 47(8).
Marta Sabou, Kalina Bontcheva, Leon Derczynski, and Arno Scharl. 2014. Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines. In LREC 2018, Eleventh International Conference on Language Resources and Evaluation, pages 859-866.
Joni Salminen, Hind Almerekhi, Ahmed Mohamed Kamel, Soon Gyo Jung, and Bernard J. Jansen. 2019. Online hate ratings vary by extremes: A statistical analysis. In CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, pages 213-217.
Joni Salminen, Fabio Veronesi, Hind Almerekhi, Soon-gyo Jung, and Bernard J. Jansen. 2018. Online Hate Interpretation Varies by Country, But More by Individual. In Proceedings of SNAMS, pages 1-7.
Manuela Sanguinetti, Fabio Poletto, Cristina Bosco, Viviana Patti, and Marco Stranisci. 2018. An Italian Twitter Corpus of Hate Speech against Immigrants. In LREC, pages 2798-2805.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, Noah A Smith, and Paul G Allen. 2019. The Risk of Racial Bias in Hate Speech Detection. In Proceedings of the 57th Annual Meeting of the ACL, pages 1668-1678.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477-5490, Online. Association for Computational Linguistics.
Anna Schmidt and Michael Wiegand. 2017a. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media. Association for Computational Linguistics.
Anna Schmidt and Michael Wiegand. 2017b. A Survey on Hate Speech Detection using Natural Language Processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1-10. Association for Computational Linguistics.
Nick Seaver. 2015. The nice thing about context is that everyone has it. Media, Culture and Society, 37(7):1101-1109.
Leanne Townsend and Prof Claire Wallace. 2017. Social Media Research: A Guide to Ethics. The University of Aberdeen, Aberdeen.
Grigorios Tsoumakas and Ioannis Katakis. 2007. Multi-label classification: An overview. International Journal of Data Warehousing and Mining (IJDWM), 3(3):1-13.
Bertie Vidgen and Leon Derczynski. 2020. Directions in Abusive Language Training Data: Garbage In, Garbage Out. arXiv:2004.01670, pages 1-26.
Bertie Vidgen, Scott Hale, Ella Guest, Helen Margetts, David Broniatowski, Zeerak Waseem, Austin Botelho, Matthew Hall, and Rebekah Tromble. 2020. Detecting East Asian prejudice on social media. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 162-172, Online. Association for Computational Linguistics.
Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019a. Challenges and frontiers in abusive content detection. In Proceedings of the Third Workshop on Abusive Language Online (ACL), pages 80-93.
Bertie Vidgen, Helen Margetts, and Alex Harris. 2019b. How much online abuse is there? A systematic review of evidence for the UK. The Alan Turing Institute, London.
Bertie Vidgen, Rebekah Tromble, Alex Harris, Scott Hale, Dong Nguyen, and Helen Margetts. 2019c. Challenges and frontiers in abusive content detection. In 3rd Workshop on Abusive Language Online.
Bertie Vidgen and Taha Yasseri. 2019. Detecting weak and strong Islamophobic hate speech on social media. Journal of Information Technology & Politics, 17(1):66-78.
Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding Abuse: A Typology of Abusive Language Detection Subtasks. In Proceedings of the First Workshop on Abusive Language Online, pages 78-84.
Zeerak Waseem and Dirk Hovy. 2016. Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter. In *NAACL-HLT*, pages 88-93.
Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of Abusive Language: the Problem of Biased Datasets. In *NAACL-HLT*, pages 602-608, Minneapolis. ACL.
Matthew Williams. 2019. Hatred behind the scenes: a report on the rise of online hate speech. Mishcon de Reya, London.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex Machina: Personal Attacks Seen at Scale. In Proceedings of the International World Wide Web Conference, pages 1391-1399.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. In Proceedings of NAACL HLT 2019, volume 1, pages 1415-1420.
Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Căgni Öltekin. 2020. SemEval-2020 task 12: Multilingual offensive language identification in social media (OffenseEval 2020). In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1425–1447, Barcelona (online). International Committee for Computational Linguistics.
A Data Statement
A. CURRATION RATIONALE In order to study the classification of abusive content online, and the role of context, we collected data from Reddit, using the PushShift API. We sampled data by collecting content from specific cubreddits, rather than by using keywords.
To identify subreddits which contain substantial levels of abuse (and as such are suitable for inclusion in this dataset), we reviewed four sources, which returned 117 unique subreddits. Note that many prolific abusive subreddits had been banned prior to our study, such as r/AgainstGayMarriage, r/BlackPeopleHate, r/coontown and r/physical_removal.
1. Curated list of 21 hateful subreddits, hosted on r/AgainstHateSubreddits. $^6$
2. Curated list of 30 'reactionary/fascist' subreddits, hosted on r/GenderCynicial.7
3. List of 13 hateful subreddits from a news article in Vice.<sup>8</sup>
4. Every subreddit mentioned on r/AgainstHateSubreddits in the 12 months from January to December 2018 (93 in total).
Initially, the number of conversation threads that we sampled from each subreddit was proportionate to the total number of conversation threads they hosted during the period (such that more active subreddits featured more heavily in the dataset). Due to large differences in how active users were across subreddits, we then boosted the number of conversation threads from subreddits with fewer posts. We also stratified by date and limited the
Subreddit	# threads
r/Drama	148
r/conspiracy	145
r/bakchodi	123
r/TrueOffMyChest	123
r/subredditcancer	106
r/ImGoingToHellForThis	104
r/ShitPoliticsSays	100
r/TumblrInAction	98
r/SubredditDrama	96
r/4chan	77
r/WatchRedditDie	77
r/CCJ2	44
r/Negareddit	43
r/HateCrimeHoaxes	41
r/smuggies	38
r/imgoingtohellforthis2	31
TOTAL	1,394
Table 6: Number of conversation threads from each subreddit.
number of entries from each author to maximize diversity. The number of conversation threads from each subreddit is shown in Table 6.
B. LANGUAGE VARIETY Most of the content was in English. Annotators were instructed to mark up non-English where possible, in order to retain the conversation structure in the final dataset. We checked language by first applying langid.py and then manually checking all of the entries which were flagged as non-English. 1,407 entries were flagged as non-English, of which 353 were identified as genuinely non-English by human review. We excluded non-English entries from the dataset for experiments.
C. SPEAKER DEMOGRAPHICS The creators of the Reddit entries ('speakers') were not directly approached and thus we could not ask them for demographic information. Further, Reddit users provide relatively little information about themselves in their profiles, and we opted not to collect the little information that was available due to ethical concerns.
Outside of two moderator bot accounts ('SnapshillBot' and 'Automoderator'), the most common users appeared over 60 times in the dataset. 11,122
$^{9}$ https://github.com/saffsd/langid.py
$^{6}$ https://www.reddit.com/r/  
Against HateSubreddits  
 $^{7}$ https://www.reddit.com/r/  
GenderCynical/comments/bdrtvq/a_  
comprehensive_list_of_transphobic_  
subreddits/  
 $^{8}$ https://www.vice.com/en_uk/article/  
8xxymb/here-are-reddits-whiniest-most/  
low-key-toxic-subreddits
users appear in total.
D. ANNOTATOR DEMOGRAPHICS The dataset includes annotations from 12 trained analysts. They were recruited through a competitive process. They underwent 4 weeks of training, including numerous one-to-one sessions. Work was conducted over 12 weeks, with each annotator working between 10 and 20 hours each week. Of the 12 annotators who contributed to the final dataset, 11 consented to provide information about their demographics. Age: 7 annotators were 18-29, 3 were 30-39 and 1 was 40-49. Gender: 4 were female and 7 were male. Ethnicity: 8 were white, 1 Latino, 1 of Middle Eastern ethnic origin and 1 was mixed. National identity: 7 were British, 1 American, 1 Ecuadorian, 1 Jordanian and 1 Polish. Social media use: 9 used social media more than once per day, and 2 use it once per day. Exposure to online abuse: All annotators had witnessed online abuse in the previous year, with 10 stating they had witnessed it more than 3 times and 1 stating they had witness it 2-3 times. Disagreements were adjudicated through group discussion with an expert in abusive online content. They are a post-doctoral researcher with extensive experience.
E. SPEECH SITUATION All Reddit comments and posts were made between 1st February 2019 and 31st July 2019. The intended audience is unknown but was most likely the other members of the subreddit.
F. TEXT CHARACTERISTICS The composition of the dataset, including the distribution of the Primary and Secondary categories, is described in the paper.
B Data and Model fitting
The application of the Context flag for the primary categories is shown in Table 7.
B.2 Model fitting
Training details We fine-tune for 3 epochs, a training batch size of 16, and 100 warm up steps. We experiment with learning rates of  $\{2\mathrm{e} - 5,3\mathrm{e} - 5,4\mathrm{e} - 5,5\mathrm{e} - 5\}$  and a weight decay of  $\{0,0.01,0.03\}$ . Experiments were run with a single NVIDIA Quadro RTX 6000 GPU. Finetuning one BERT model took around 24 minutes. Finetuning one DistilBERT model took around 12 minutes.
Primary category	Context	Number	Percent
Affiliation	Current	1,033	75.6%
Previous	333	24.4%
Identity	Current	2,046	75.4%
Previous	666	24.6%
Person	Current	765	68.8%
Previous	347	31.2%
Counter Speech	Previous	220	100.0%
Non-hateful Slurs	Current	149	100.0%
Neutral	None	21,935	100.0%
Table 7: Use of context flag, split by primary category.