SlowGuess/NAACL_2021_OCR huggingface dataset:
Datasets that contain a higher proportion of implicit abuse mostly suffer from biases caused by the sampling of the underlying raw data. (Typically, one samples microposts containing certain keywords or topics that may coincide with abusive language.) As Wiegand et al. (2019) showed, classifiers trained on these datasets may correctly detect implicitly abusive instances on unseen test instances of the same datasets. However, these correct classifications are not produced by grasping the concept of implicit abuse but by exploiting some artifacts contained in the dataset. Such artifacts can be frequently occurring words, such as women and football, that, due to the sampling process, coincidentally only occur in abusive microposts.
Although additional datasets containing larger amounts of implicit abuse have been released since Wiegand et al. (2019) published their findings, we found that these new datasets also suffer from biases. We outline these biases on the most recent dataset that displays a high degree of implicit abuse and that is also fairly large (Table 1): the dataset by Sap et al. (2020) (SBFrames). Of the recent datasets, it is also the only dataset to cover a significant amount of abusive instances targeting common
identity groups (e.g. Jews, Muslims).
In order to get a larger amount of microposts, existing datasets (e.g. Founta et al. (2018)) were merged into SBFrames. In addition, further raw data was added, such as posts from the white-supremacist platform stormfront.org or subReddit on abusive jokes from reddit.com. While these additional data undoubtedly yield more abusive content, it is problematic to merge data from different domains into one corpus. The resulting dataset is bound to be fairly heterogeneous in terms of style.
For example, most jokes from `redisit.com` follow a specific syntactic pattern: a question is asked to which some (short) abusive answer is given. This is illustrated by (47) and (48).
(47) What's worse than an angry black woman? Nothing.
(48) How do you pick up a Jewish girl? With a shovel.
Since the dataset does not explicitly state the origin of each micropost, we approximated the set of jokes by mining for the above syntactic pattern. More than  $80\%$  of the jokes are abusive. Due to the recurring syntactic pattern of jokes, classifiers trained on the corpus from Sap et al. (2020) will find it easy to detect abusive utterances. They basically have to look for a joke, i.e. a question followed by an answer. They do not really have to understand the joke or the concept of abuse. This observation is particularly significant to the detection of implicit abuse since more than  $40\%$  of the implicitly abusive microposts that we randomly sampled from the dataset were jokes (Table 2).
The above `redisit-joke-bias` is just one example of that corpus. We also noticed that identity groups (i.e. Jews, Muslims, blacks etc.), which comprise the typical targets of the dataset, also highly correlate with abuse (Table 3). For instance, almost all mentions of  $\text{Jew}(s)$  are abusive. This property makes the detection of such abusive instances considerably easier since a classifier can predict all cases including mentions of these words as abuse and reaches a high classification performance.
Simply removing the mentions of identity groups is insufficient. Microposts addressing those particular identity groups would still be restricted to the abusive microposts. Supervised classifiers are likely to infer that a micropost refers to some identity group although it has been removed. For instance, one can easily infer that (49) is about Jews and (50) is about Muslims due to further contextual clues (Hitler & gas (49); ISIS & Al-Qaeda (50)).
(49) I'm pretty sure Hitler just said "I wanna glass of juice" not I wanna gas the <IDENTITY_GROUP>.
(50) Being a <IDENTITY_GROUP> I have a confusion choosing my career. Either to go with ISIS or Al-Qaeda?
Moreover, we have to assume further biases in the dataset from Sap et al. (2020): The proportion of abuse across the different sources from which this dataset is created seems to vary considerably: Abusive utterances in Founta et al. (2018) (this is one source of the dataset) are rare (14%) while the majority of posts from the white-supremacy site stormfront.org (another source of the dataset) should be abusive. This is so since the major topic of this platform (i.e. white supremacy) is racist. Since these texts also vary much in style across the different sources (the former are tweets, while the latter are longer posts with fully grammatical sentences), a classifier that learns to detect the style of the different sources will already have a good prior as to whether a particular post is abusive.
5.2 Divide and Conquer
We argue that by creating one dataset to cover all phenomena of abusive language, the creators of those datasets lose sight of appropriate negative data. By negative data, we mean those instances that are not abusive and contrast the abusive instances so that a classifier can learn a good distinction between abusive and non-abusive instances. By using inappropriate negative data, biases as those described in §5.1 will notably distort classification performance. If datasets are created for individual subtypes of implicit abuse (§4.1-§4.8) we obtain a less heterogeneous set of abusive instances for which it is easier to produce suitable negative instances. In order to classify unrestricted text, it would simply take a final meta-classifier that collects the predictions of all the specialized classifiers for specific subtypes of abuse.
5.3 More training data does NOT necessarily mean better training data
As we outlined in §5.1, increasing the size of data by merging different corpora is highly problematic. Supervised classifiers may simply produce higher classification scores as a result of further biases introduced by the merging process.
Thinking about negative data is important. If there are certain artifacts that coincide with the abusive instances due to the sampling process (i.e. they are not representative of abusive language), then one can neutralize that bias by enforcing it
identity group	woman	lesbian	gay	black	muslim	jew
% abusive	67.3	71.7	75.2	87.2	87.8	93.8
Table 3:Abusive posts with identity group.
to also occur in the negative data. For supervised classifiers, this artifact will then be ignored as it will occur in all classes equally.
For example, the mentions of identity groups (Jews, Muslims, women, gay people etc.) are mostly limited to abusive instances (Table 3). A less biased dataset would enforce mentions of identity groups in the negative data. Although the resulting overall dataset may be smaller as a result of selecting specific negative data, the overall quality of the training data should rise. In general, the NLP community is increasingly aware of such biased constructions in datasets and measures, as we propose, are an approved means to produce datasets to evaluate classifiers under more realistic conditions (McCoy et al., 2019).
Another problem of randomly sampling data is that due to the fact that the frequency distribution of a language vocabulary is generally a power law distribution (Zipf, 1965), instances will always be dominated by a few frequently occurring words. Supervised classifiers may achieve high classification performance by just focusing on these particular words. However, a dataset would be much harder if we tried to represent words more equally.
For example, if we were to produce a dataset for learning to detect identity groups depicted as perpetrators (§4.2), the best way would be to sample microposts with mentions of co-occurrences of an identity group and some negative polar expression (e.g. Muslims rape, Muslims criticize). In order to build a dataset that captures the long tail of rare constructions, we would need to ensure that we do not only include the frequently occurring negative polar expressions (e.g. kill, murder, rape) but also the infrequent ones (e.g. calumniate, concoct, racketeer). As a consequence, a dataset with 10k microposts that focuses on the frequent polar expressions may be less suitable for training a classifier on than a dataset that comprises 1k microposts but includes a wide set of polar expressions with each expression only occurring a few times.
Our call for smaller datasets that do not contain similar non-informative instances but a sample of the task that allows for sharper decision boundaries echoes ideas from the field from active learning (Settles, 2012) and the recent proposal for NLP
evaluation in terms of contrast sets (Gardner et al., 2020).
5.4 Classification Below the Micropost-Level
Previous research considered entire microposts as instances from which to learn abusive language. However, there may be good reason to focus on smaller meaningful units, such as sentences or even clauses. This view is also shared by parts of the community. SemEval 2021 includes a shared task that addresses the detection of abusive text spans within a micropost.<sup>6</sup> In the following, we describe how such classification schemes would facilitate learning implicit abuse.
Given that social-media platforms commonly used for obtaining natural language data, such as Twitter, increasingly ban abusive language on their sites<sup>7</sup>, the amount of data available in which abusive language is actually used is decreasing.<sup>8</sup> However, there are still many mentions of abuse available, such as reported cases (Chiril et al., 2020), including implicit abuse (51)-(52).
(51) @USER exposes the hypocrisy of claims that [Muslims want to suppress free speech]abusive clause.
(52) The Texas GOP thinks that [gay people need a cure]abusive clause.
For example, we randomly sampled 50 tweets from Twitter containing the abusive clause homosexuality is unnatural. After manual inspection we found that  $76\%$  of the tweets just reported this claim and the author clearly opposed that view.
Sometimes, the presence of emojis (53) or interjections (54) also suggests that the author of the tweet does not share the stated proposition.
(53) [Black people are aliens] abusive clause now
(54)Wow, [Jews control everything]abusive clause, cool lol
Given the above observations, we suspect that there are many abusive clauses that are only available as embedded abuse (51)-(54). In order to use them as training data for genuine abuse (such clauses may occur as genuine abuse, i.e. abuse that is not embedded, in unseen test data), we think it would suffice to isolate the actual abusive clauses and train on them instead of the entire microposts.
<sup>6</sup>https://sites.google.com/view/toxicspans
<sup>7</sup>https://techcrunch.com/2020/03/05/twitter-bans-hatespeech-around-age-disability-and-in-the-wake-of-the-coronavirus-outbreak-disease/
Alternative social-media platforms which are known to contain a higher proportion of abusive language, such as gab.com, are considerably more difficult to process, as technical support equivalent to Twitter API is typically not available.
Recent research on the helpfulness of context may also support our view to restrict the context for training data. In an in-depth study, Pavlopoulos et al. (2020) found that increasing the context for abusive language detection by considering microposts neighbouring the post to be classified actually harms classification performance. Microposts, such as tweets from Twitter, themselves can already be fairly long (up to 280 characters) representing a paragraph of sentences. Future research should investigate whether the non-abusive sentences of a longer abusive micropost already negatively affect learning abusive language.
Apart from that, an abusive micropost often contains more than one predictive clue. For such microposts, a supervised classifier may not need to detect all of these clues. Typically, the classifier is more effective in spotting the easier clues, which, in the case of abusive language detection, are (explicitly) abusive words. (55) is a micropost that includes both explicit abuse (i.e. the word sneaky) and implicit abuse (i.e. an abusive clause expressing some anti-Semitic stereotype). If we want to effectively learn the more difficult implicit clues, it may be useful to focus only on the implicitly abusive clauses by removing the explicit clues from microposts that also include implicit abuse.
(55) Sneaky [Jews are controlling the world through their banking]abusive clause.
6 The Role of Machine Learning
Despite the continuing success of machine learning in many areas of NLP, particularly fairly generic methods, we should be careful in considering this the magic bullet for every problem including the detection of implicitly abusive language.
Already in some subtasks of (explicitly) abusive language detection, machine learning has not produced the anticipated results. For example, supervised learning still produces fairly poor classification performance on the cross-domain detection of abusive language, with lexicon-based approaches performing much stronger (Wiegand et al., 2018). Further, statistical debiasing methods for abusive language detection have also been reported to yield very limited success (Zhou et al., 2021). The authors of that research argue that spending more efforts in ensuring a high quality of the datasets during their creation is more worthwhile than applying sophisticated machine learning.
We anticipate that there are also some subtasks in the realm of implicit abuse that may not be solved
with the help of supervised learning approaches. One such example may be the task of detecting novel or unknown stereotypes. If we compare the two stereotypes (56) and (57), we find that these sentences differ in meaning, sentiment and also in terms of syntactic structure.
(56) Asian children are intelligent.
(57) All Asian people lie.
If we train a classifier on (56) it is unlikely to identify (57) as an instance of the same category due to the lack of similar features. As a consequence, learning-based approaches are unlikely to succeed in this task.
Although generic supervised methods may always represent a good baseline, the community should also be open that other more linguistically informed approaches can be more effective for particular subtasks in the detection of implicitly abusive language. Riloff et al. (2013) demonstrated that mining for a particular linguistic construction is an effective means to recognize a specific type of sarcasm. We envisage that similar approaches may be effective for the detection of implicit abuse.
Due to the susceptibility of supervised learning to overfitting, we also recommend an experimental set-up in which a cross-domain evaluation is included in order to check whether the resulting classifiers generalize beyond the training data.
There are different subtypes of implicit abuse. Some of them are frequent in available datasets (e.g. jokes or stereotypes) while others are sparse (e.g. dehumanization or euphemisms). As far as frequent subtypes of implicit abuse (e.g. stereotypes and perpetrators) are concerned, unsuitable sampling causes biases that prevent classifiers from really learning these phenomena. Simply adding instances by merging datasets does not solve the problem. It may introduce further detrimental biases. Overall, our analysis supports the claim that the currently available datasets are not really suitable for effectively learning implicit abuse. We strongly argue for new datasets that focus on particular subtypes of implicit abuse. This will also facilitate thinking about appropriate negative data. Larger datasets are not necessarily the best datasets to train a classifier on, especially if they are dominated by frequently observed words. Finally, it may also make sense to learn on smaller units, such as clauses, rather than on entire microposts.
This paper contains real-life examples of abusive language taken from actual web data. We are aware of the fact that some readers may feel offended by these examples, particularly since many of them address entire identity groups (e.g. Muslims, Jews etc.). We chose those examples deliberately in order to demonstrate that despite not being instances of explicit abuse, implicit abuse can still be extremely severe. Consequently, the automatic detection of implicit abuse should be considered equally pressing as the detection of explicit abuse.
The examples used in this paper in no way reflect the opinion of the authors. All mentions of specific user names were anonymized in order to comply with privacy principles.
Our work is critical of the design of existing datasets for abusive language detection. We would like to clarify that we do not generally challenge the usefulness of these datasets per se. Our criticism only relates to using these datasets for learning implicit abuse.
Shohini Bhattasali, Jeremy Cytryn, Elana Feldman, and Joonsuk Park. 2015. Automatic Identification of Rhetorical Questions. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 743-749, Beijing, China.
Pete Burnap, Omer F. Rana, Nick Avis, Matthew Williams, William Housley, Adam Edwards, Jeffrey Morgan, and Luke Sloan. 2015. Detecting tension in online communities with computational Twitter analysis. Technological Forecasting, 95:96-108.
Pete Burnap and Matthew L. Williams. 2015. Cyber Hate Speech on Twitter: An Application of Machine Classification and Statistical Modeling for Policy and Decision Making. *Policy & Internet*, 7(2):223-242.
Mike Cardwell. 1999. Dictionary of Psychology. Fitzroy Dearborn.
Patricia Chiril, Véronique Moriceau, Farah Benamara, Alda Mari, Gloria Origgi, and Marlene Coulomb-Gully. 2020. He said "who's gonna take care of your children when you are at ACL?": Reported sexist acts are not sexist. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 4055-4066, Online.
Jenna Cryan, Shiliang Tang, Xinyi Zhang, Miriam Metzger, Haitao Zheng, and Ben Y. Zhao. 2020. Detecting Gender Stereotypes: Lexicon vs. Supervised
Learning Methods. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI), pages 1-11, Honolulu, HI, USA.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated Hate Speech Detection and the Problem of Offensive Language. In Proceedings of the International AAAI Conference on Weblogs and Social Media (ICWSM), Montreal, Canada.
Christian Felt and Ellen Riloff. 2020. Recognizing Euphemisms and Dysphemisms Using Sentiment Analysis. In Proceedings of the Workshop on Figurative Language Processing, pages 136-145, Online.
Paula Fortuna and Sérgio Nunes. 2018. A Survey on Automatic Detection of Hate Speech in Text. ACM Computing Surveys, 51(4):85:1-85:30.
Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large Scale Crowdsourcing and Characterization of Twitter Abusive Behaviour. In Proceedings of the International AAAI Conference on Weblogs and Social Media (ICWSM), Stanford, CA, USA.
Annemarie Friedrich and Manfred Pinkal. 2015. Automatic recognition of habituals: a three-way classification of clausal aspect. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2471-2481, Lisbon, Portugal.
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating Models' Local Decision Boundaries via Contrast Sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1307-1320, Online.
Raul Gomez, Jaume Gibert, Lluis Gomez, and Dimosthenis Karatzas. 2020. Exploring Hate Speech Detection in Multimodal Publications. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1470-1478.
Nick Haslam. 2006. Dehumanization: An integrative review. *Personality and social psychology review*, 10:252-264.
Nick Haslam and Steve Loughnan. 2016. Recent research on dehumanization. *Current Opinion in Psychology*, 11:25-29.
Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri. 2018. Benchmarking Aggression Identification in Social Media. In Proceedings of
the Workshop on Trolling, Aggression and Cyberbullying (TRAC), pages 1-11, Santa Fe, NM, USA.
Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri. 2020. Evaluating Aggression Identification in Social Media. In Proceedings of the Workshop on Trolling, Aggression and Cyberbullying, pages 1-5.
E. D. Macmillan. 2007. Macmillan English Dictionary. Michael Rundell.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 3428-3448, Florence, Italy.
Julia Mendelsohn, Yulia Tsvetkov, and Dan Jurafsky. 2020. A Framework for the Computational Linguistic Analysis of Dehumanization. Frontiers in Artificial Intelligence.
Rada Mihalcea and Carlo Strapparava. 2006. Learning to laugh (automatically): Computational models for humor recognition. Computational Intelligence, 22(2):126-142.
Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang. 2016. Abusive Language Detection in Online User Content. In Proceedings of the International Conference on World Wide Web (WWW), pages 145-153, Republic and Canton of Geneva, Switzerland.
Shereen Oraby, Vrindavan Harrison, Amita Misra, Ellen Riloff, and Marily Walker. 2017. Are you serious?: Rhetorical Questions and Sarcasm in Social Media Dialog. In Proceedings of the Annual SIGdial Meeting on Discourse and Dialogue, pages 310-319, Saarbrücken, Germany.
Alexis Palmer, Melissa Robinson, and Kristy K. Phillips. 2017. Illegal is not a Noun: Linguistic Form for Detection of Pejorative Nominalizations. In Proceedings of the Workshop on Abusive Language Online (ALW), pages 91-100, Vancouver, BC, Canada.
John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. 2020. Toxity Detection: Does Context Realler Matter? In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 4296-4305, Online.
Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. 2020. Resources and benchmark corpora for hate speech detection: a systematic review. *Language Resources and Evaluation*.
Ashequl Qadir, Ellen Riloff, and Marilyn A. Walker. 2015. Learning to Recognize Affective Polarity in