SlowGuess/NAACL_2021_OCR huggingface dataset:
Implicitly Abusive Language – What does it actually look like and why are we not getting there?
Leibniz Institute for German Language
Abusive language detection is an emerging field in natural language processing which has received a large amount of attention recently. Still the success of automatic detection is limited. Particularly, the detection of implicitly abusive language, i.e. abusive language that is not conveyed by abusive words (e.g. dumbass or scum), is not working well. In this position paper, we explain why existing datasets make learning implicit abuse difficult and what needs to be changed in the design of such datasets. Arguing for a divide-and-conquer strategy, we present a list of subtypes of implicitly abusive language and formulate research tasks and questions for future research.
Abusive or offensive language is commonly defined as hurtful, derogatory or obscene utterances made by one person to another person or group of persons. Examples are (1)-(3). In the literature, closely related terms include hate speech (Waseem and Hovy, 2016) or cyberbullying (Zhong et al., 2016). While there may be nuanced differences in meaning, they are all compatible with the general definition above.
(1) stop editing this, you dumbass.
(2) Just want to slap the stupid out of these bimbos!!
(3) Go lick a pig you arab muslim piece of scum.
Due to the rise of user-generated web content, the amount of abusive language is growing. NLP methods are required to focus human review efforts towards the most relevant microposts.
Though there has been much work on abusive language detection in general, comparatively little work has been focusing on implicit forms of abusive language (4)-(5) (Waseem et al., 2017). By implicit abuse we understand abusive language that is not conveyed by (unambiguously) abusive words (e.g. dumbass, bimbo, scum).
(4) I haven't had an intelligent conversation with a woman in my whole life.
(5) Why aren't there any Mexicans on Star Trek? Because they don't work in the future either.
Detailed analyses of the output of existing classifiers have also revealed that currently only explicit abuse can be reliably detected (van Aken et al., 2018; Wiegand et al., 2019).
In this position paper, we want to shed more light on the nature of implicitly abusive language. We identify subtypes of implicit abuse that can be found in existing datasets and the literature. We also outline shortcomings that prevent implicitly abusive language from really being learned on its own terms. With this study, we hope to guide future research on implicitly abusive language.
Our contributions in this paper are:
- We present a list of subtypes of implicit abuse. This is accompanied by quantitative information from publicly available datasets.
- We derive research tasks and questions regarding those subtypes for future research.
- We detail properties of existing datasets that make them less suitable for training classifiers to detect implicit abuse.
- We propose key issues that need to be considered when building new datasets for implicitly abusive language.
<sup>1</sup>http://thelawdictionary.org/abusive-language
2The examples in this work are included to illustrate the severity of abusive language. They are taken from actual web data and in no way reflect the opinion of the authors.
Association for Computational Linguistics: Human Language Technologies, pages 576-587
2 The Story So Far
By far the most prominent classification approaches applied to abusive language detection are supervised learning methods. Whereas initially, traditional learning algorithms, such as SVMs or logistic regression, were among the most popular methods for this task (Warner and Hirschberg, 2012; Burnap et al., 2015; Nobata et al., 2016), at present, best results are obtained by deep-learning methods, particularly transformers (Struß et al., 2019; Kumar et al., 2020; Zampieri et al., 2020). A more detailed summary of the methods explored can be found in Schmidt and Wiegand (2017) and Fortuna and Nunes (2018).
Unfortunately, so far there has been little error analysis of system output for abusive language detection. As a consequence, the community is fairly unaware of what types of errors are made and why.
The most notable exception is van Aken et al. (2018) who carry out experiments on the dataset of Google's Toxic Comment Classification Challenge and the dataset by Davidson et al. (2017).
As prominent errors that a supervised classifier makes, van Aken et al. (2018) list toxicity without swearwords, rhetorical questions and comparisons/metaphorical language. All these phenomena can be subsumed by implicit abuse. Unfortunately, the study by van Aken et al. (2018) is only of limited help since one of two datasets considered, namely the dataset from the Toxic Comment Classification Challenge, contains a high degree of explicitly abusive language (Table 1). The other dataset, i.e. the dataset by Davidson et al. (2017), is not considered in our work, since it is not a dataset for the detection of abusive language but the disambiguation of potentially abusive words. $^4$
Wiegand et al. (2019) find that supervised classifiers with a reasonable cross-domain performance are those that are trained on datasets with a high degree of explicit abuse. Classifiers trained on datasets with a high degree of implicit abuse perform poorly on other datasets, no matter whether one deals with implicit or explicit abuse. From that the authors conclude that classifiers are not effectively learning implicit abuse.
dataset	publication	size	%expl.
Kumar	(Kumar et al., 2018)	15,000	32.7
SBFrames	(Sap et al., 2020)	45,318	37.6
Waseem	(Waseem and Hovy, 2016)	16,165	44.4
Warner	(Warner and Hirschberg, 2012)	3,438	51.3
OffensEval	(Zampieri et al., 2019)	13,240	54.0
Razavi	(Razavi et al., 2010)	1,525	64.7
Founta	(Founta et al., 2018)	59,357	75.9
Kaggle	(Wulczyn et al., 2017)	312,737	76.7
Table 1: Statistics of datasets (size: number of microposts; expl.: explicit abuse).
3 Datasets Considered in this Study
Recent years have seen a notable increase of datasets for abusive language detection. Since a survey would be beyond the scope of this section, we refer the reader to Poletto et al. (2020) and Vidgen and Derczynski (2020). However, implicit abuse is not covered in these publications.
Due to the limited space, we only focus on English datasets in this paper. We also just consider the common binary classification task of whether a micropost is abusive or not. Table 1 shows the proportion of explicit abuse on the different datasets. We compute these scores by checking each abusive micropost from a dataset for the presence of an abusive word according to the lexicon of abusive words from Wiegand et al. (2018). The complementary proportion to each score can be considered a proxy for the degree of implicit abuse (e.g.  $67.3\%$  for Kumar). However, such scores should just be considered an upper bound for implicit abuse since we will have missed explicitly abusive microposts. (Even the lexicon from Wiegand et al. (2018) is not exhaustive.) From the scores in Table 1, we can conclude that the datasets Kumar, SBFrames, Waseem, Warner and OffensEval have a fairly high proportion of implicit abuse, which is why we focus on these datasets in the remainder of this paper.
4 Different Subtypes of Implicit Abuse
For each dataset, we manually annotated a random sample of 500 implicitly abusive instances (according to our proxy described in §3) for their subtypes, i.e. 2,500 instances in total. The subtypes we used were either mentioned in previous work (van Aken et al., 2018) or frequently observed in the examined datasets. In the following, we describe these subtypes:
4.1 Stereotypes
By stereotypes, we understand a fixed, overgeneralized belief about a particular group or class of
$^{3}$ www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/overview
4 In other words, it deals with the question in which contexts a mention of a potentially abusive word (e.g. fuck) is really used in an abusive manner and what type of abuse is conveyed, i.e. hate speech or mere profanity.
people (Cardwell, 1999):
(6) Jews have undue influence.
Stereotypes are difficult to detect as there are quite a few stereotypes per identity group. Moreover, stereotypes need not be negative in tone:
(7) Jews are good at making money.
As a consequence, using sentiment analysis as a pre-filtering step by isolating only negative statements may miss a substantial fraction of stereotypical remarks. However, as a research task it may be a reasonable starting point since not every negative sentence focusing on some identity group conveys some (abusive) stereotype (e.g. (8)-(10)). A first research question could be how to detect stereotypical statements among negative statements.
(8) Gay people fight for the right to be accepted.
(9) Muslims groan under the recession.
(10) Jews mourn the loss of a member of their community.
We believe that specific linguistic properties may be indicative for automatic classification. For example, stereotypes are more likely to co-occur with habitual aspect (11) rather than non-habitual aspect (12) (Friedrich and Pinkal, 2015).
(11) Jews always support terror instability.
(12) Jews currently fear displaying their faith in public.
One should also examine whether generic phrases regarding identity groups (13) correlate with stereotypes (Reiter and Frank, 2010). Previous work already established that the definite article, which represents a subset of such generic phrases, is predictive for abusive language (Burnap and Williams, 2015; Palmer et al., 2017).
(13) The jew does not care about the humankind.
Further, the same stereotype can be expressed in different ways. For example, (14)-(17) convey the sexist stereotype that women belong in the kitchen.
(14) Men should drive and women should cook.
(15) This is how America should be. 5 women slaving over a hot stove.
(16) Get back in the kitchen.
(17) Women should all stay at home in an apron, chained to the oven!
We could also envisage the following research task for stereotype classification. Rather than classifying arbitrary statements on identity groups as stereotype or not, one could start with a set of existing stereotypes (e.g. black people are criminal)
and learn to align all statements on identity groups crawled from a social-media site to these existing stereotypes. Such a setting may be simpler than the plain classification since the stereotypes are known in advance. This setting would even be in line with reality since the set of stereotypes regarding identity groups is not infinite.
To date, the only significant work analyzing the detection of stereotypes is Cryan et al. (2020) who compare a lexicon-based classifier with a supervised classifier for detecting gender stereotypes. The dataset used for supervised classification and the lexicon are created via crowdsourcing. No deeper linguistic analysis of the data is performed.
4.2 Perpetrators
(18)-(21) depict some identity group as perpetrators.
(18) Jews are raping kids.
(19) Muslims are looting old people's homes.
(20) Black people steal everything.
(21) Jews scheme on world domination daily.
By perpetrator, we understand a person who commits, an illegal, criminal, or evil act. An individual being depicted as a perpetrator (22) may be just considered an accusation. However, if identity groups (i.e. Jews, Muslims etc.) are presented in this way, (18)-(21) this is perceived as abusive language, particularly if the victims are protected classes (e.g. kids, old people) as in (18) and (19).
(22) Our neighbour is beating his wife.
This form of implicit abuse is a proper subset of stereotypes (§4.1). However, we think that abuse conveyed by depicting someone as a perpetrator has some notably different properties than the other stereotypes. These properties justify a separate category. The actions that characterize perpetrators are often criminal offenses (e.g. raping, murdering, stealing) or are at least morally contemptible (e.g. adultery, lying, scheming). Thus, we consider them to be universal actions that can apply to different targets (i.e. identity groups). In contrast, the other stereotypes are target-specific and less universal. Switching identity groups does not necessarily preserve the abusiveness as shown in (23) and (24).
(23) Jews belong in the kitchen.
(24) Women are good at making money.
5www.dictionary.com/browse/perpetrator
We assume that the depiction as a perpetrator is also largely tied to (fairly unambiguous) lexical units, i.e. a subset of action predicates (primarily verbs) being negative polar expressions. From a computational perspective, it should, therefore, be feasible to detect such cases reliably. The depiction of other stereotypes may be less tied to specific lexical items. Therefore, we believe the detection of those stereotypes to be more challenging.
Abusive comparisons are comparisons in which the vehicle (you in (25)) is compared to some offensive entity, action or state (idiot in (25)). Abusive comparisons need not be explicitly abusive (25) but can also be implicitly abusive (26)-(27).
(25) You talk like an idiot.
(26) You look like someone only a mother could love.
(27) You sing like a dying bird.
A research question that would need to be answered is whether detecting abusive comparisons is not (almost) identical to the detection of comparisons conveying a negative sentiment. Such classification of comparisons into positive (28), neutral (29) and negative comparisons (30) has already been addressed by Qadir et al. (2015).
(28) You look like a princess.
(29) You look like your brother.
(30) You look like a crackhead.
Another research question would be to examine whether abusive comparisons are not identical to (negative) comparisons using figurative language (i.e. similes as (31)). Intuitively, comparisons employing literal language should be less abusive (32).
(31) You look like the back end of a bus.
(32) You look like you have slept badly.
4.4 Dehumanization
By dehumanization, we commonly understand the act of perceiving or treating people as less than human (Haslam and Loughnan, 2016). While Haslam and Loughnan (2016) propose a fairly comprehensive set of different properties that characterize dehumanization, we focus on the most commonly accepted property of likening members of the target group to non-human entities (Haslam, 2006), such as machines, animals or diseases.
We observed two different realizations of dehumanization. On the one hand, the target is explicitly equated with non-human entities (33).
(33) Black people are monkeys.
On the other hand, a more difficult form of dehumanization involves metaphorical language in which the target is not explicitly equated to a non-human entity but their actions or properties are reminiscent of such entities as in (34)-(37).
(34) A wild flock of Jews is grazing outside a bagel store.
(35) Headscarfed muslims waddle around our streets all over.
(36) I own my wife and her money.
(37) How come bunches of gay people mushroom out of the ground these years?
Different classification approaches may be suitable for the detection of this second type of dehumanization. One may compile a corpus with mentions of animals, diseases etc. and learn the language (i.e. how non-human entities are depicted) by supervised learning. Alternatively, one might compile a lexicon that captures predicates describing actions of animals (e.g. waddle) or properties of objects/diseases (e.g. mushroom out) and then use this resource as a look-up.
Dehumanization in natural language processing has not yet been properly addressed. The only exception is the in-depth descriptive study by Mendelsohn et al. (2020) examining the dehumanizing connotation of the two words homosexual and gay in different temporally-indexed corpora.
4.5 Euphemistic Constructions
We observed several abusive remarks that were disguised as an euphemistic construction (38)-(40), typically some form of negation (39) & (40).
(38) You inspire my inner serial killer.
(39) Liberals are not very smart.
(40) I'm not excited about your existence.
If we translate these euphemisms into their unequivocal counterparts (41)-(43), the abusive nature of these statements becomes more obvious.
(41) I want to kill you.
(42) Liberals are retarded.
(43) I hate you.
With the exception of Felt and Riloff (2020), euphemisms have not been addressed in natural language processing so far.
As a research question, one would need to answer how abusive euphemisms can be detected and translated to their unequivocal counterpart.
4.6 Call for Action
Calls for action represent another type of implicitly abusive language. By that we understand that the author of a micropost asks that something, typically some form of punishment, needs to be done to the abused target (44)-(46). In particular violent actions may be shrouded in allusion. For example, (46) is an obscure way to demand that someone should be killed by electrocution.
(44) Thank you for your fortitude and perseverance. Please give McConnell a kick in the butt from some of us.
(45) @USER Liberals are so easy to figure out! Make America great again. Get rid of all liberal women.
(46) He should be given 5000 volts!
Given an appropriate dataset with sufficient occurrences, automatic methods should be able to detect this type of abuse, even in microposts, such as (46), although it is not an explicit call for killing someone. The presence of the modal verb should and the exclamation mark indicate the presence of an obligation or even command. In addition, the keyword volt in combination with a command may be a clear indicator that the author wants some violent action to take place. State-of-the-art classifiers should be able to learn such correlations.
The problem for studying this type of abusive language lies in its sparsity in the publicly available data. In many countries calling for violent actions is considered a crime. This deters many users from expressing such content on the web.
4.7 Multimodal Abuse
Most social-media platforms allow users to embed images or videos in their posts. In many cases, the abusive content of a micropost is hidden in the nontextual components or results as an interplay of text and image/video. One could also regard many of these abusive posts as instances of implicit abuse since many of them do not contain mentions of abusive words. Therefore, a comprehensive classifier to detect implicitly abusive microposts should also consist of a multimodal component that analyses image or video content and fuses this information with text analysis.
Indeed the community is aware of this form of abuse and there have been several attempts for multimodal analysis (Singh et al., 2017; Yang et al., 2019; Gomez et al., 2020). In our work, however, we do not address the aspect of multimodal abuse simply because many datasets only include the textual component of a micropost and the reconstruction of non-textual components of posts can only
be reconstructed with greater effort or even not be obtained at all.
4.8 Phenomena Requiring World Knowledge and Inferences
Of the subtypes we present as implicit abuse, the final subtypes present the most difficult kind of abusive language. We subsume all those phenomena which can effectively only be detected with the help of inferencing and additional world knowledge. Given some appropriate training data and (linguistic) feature design, automatic methods should be able to detect any of the previous subtypes to a certain degree. All of the following types of implicit abuse, however, are unlikely to be established on the basis of such approaches.
- Jokes. Jokes as (47) can be severely abusive.
(47) What's better than winning gold in the paral-lympics? Walking.
The computational modeling of humor remains a challenging task (Mihalcea and Strapparava, 2006). We are not aware of any research on the detection of abusive humor.
- Sarcasm. Sarcasm is largely defined as the activity of saying [...] the opposite of what you mean (Macmillan, 2007). The way in which is spoken is intended to make someone else feel stupid or show them that you are angry. This explains the strong connection towards abusive language as in (48):
(48) It's always fun watching sports with a woman in the room.
Although the automatic detection of sarcasm has been investigated (Tsur et al., 2010; Riloff et al., 2013), the classification performance is still fairly limited.
- Rhetorical questions. Rhetorical questions are asked not to elicit information but to make a statement (Bhattasali et al., 2015). They have been examined on social-media texts (Ranganath et al., 2016; Oraby et al., 2017). Future work needs to address what makes a rhetorical question abusive:
(49) Did Stevie Wonder choose these "models"?
- Other implicit abuse. Our final category comprises all further forms of implicit abuse that require world knowledge and inferencing:
(50) She still thinks she matters.
(51) I live in Ethiopia. Happy new year 1219!
(52) These girls know skinny sausages are no fun.
(53) Welcome to the Hotel Islamfornia. You may check out any time but you can never leave.
datasets
subtype	Kumar	SBFrames	Waseem	Warner	OffensEval	average
other implicit abuse	9.8	28.4	12.8	30.4	2.4	16.8
perpetrator	18.2	2.4	22.0	17.1	15.2	15.0
stereotype	13.4	2.0	12.2	20.0	14.2	12.4
joke	0.0	40.8	0.2	2.5	0.0	8.7
call for action	3.8	1.6	1.0	4.6	2.8	2.8
dehumanization	2.2	0.6	1.0	2.5	3.0	1.9
euphemistic construction	1.4	0.6	2.0	1.3	3.8	1.8
rhetorical question	1.2	1.6	1.6	2.1	0.6	1.4
comparison	0.6	0.0	1.4	0.0	0.0	0.4
sarcasm	1.0	0.0	0.2	0.0	0.6	0.4
unknown	37.0	11.0	37.8	10.8	23.0	23.9
explicit abuse (abus. word missing in Wiegand et al. (2018))	11.4	10.0	7.8	8.8	34.4	14.5
Table 2: Percentage of different subtypes of implicit abuse (including overlooked explicit abuse) within a dataset. The numbers are obtained by manually inspecting 500 implicit texts from each of the datasets.
4.9 Distribution of Subtypes
Table 2 shows the distribution of the subtypes of implicit abuse in the examined samples of the datasets. It also includes cases of explicit abuse missed by the lexicon from Wiegand et al. (2018) and unknown cases of implicit abuse which we could not assign to any of the previous subtypes. We were surprised by the high number of unknown cases, most notably in Kumar, Waseem and OffensEval. Some of posts are pretty short, such as RIP, Why so or Ouch! A large part of those unknown microposts requires the inclusion of further context information (e.g. multi-media attachments or links) in order to comprehend their abusive nature.
Most subtypes of implicit abuse are rare in all datasets, so none of them is an appropriate source for learning to detect these subtypes. Stereotypes, perpetrators and other implicit abuse are frequent in most datasets, however. SBFrames has a large amount of jokes. We assume that the sampling process to produce this dataset notably distorted the distribution of subtypes. We discuss this in §5.1.
Though we only found very few comparisons in the samples of abusive microposts (Table 2), comparisons seem a fairly natural form of abuse. Indeed, by manually inspecting the general dataset for comparisons by Qadir et al. (2015), we found that  $2/3$  of the person-targeted negative comparisons are abusive comparisons. About  $75\%$  of those abusive comparisons are implicitly abusive.
5 What should(n't) the datasets for implicit abuse look like?
Driven by the requirements of data-hungry deep-learning methods, the most common strategy for abusive language detection is to create a single dataset and train a classifier on it. That dataset
should be as large as possible. Unfortunately, most of the datasets that are created in this way are of little use to really learn implicit abuse.
For one thing, large datasets for abusive language detection that are produced by random sampling usually have an overwhelming proportion of explicit abuse among abusive instances (Wiegand et al., 2019). Currently, we do not know whether this is due to the predominance of explicit abuse on most social-media platforms or the fact that human annotators more readily detect explicit abuse.
5.1 Biases