SlowGuess/NAACL_2022_OCR huggingface dataset:
Michael Wiegand
Digital Age Research Center (D!ARC)
Alpen-Adria-Universität Klagenfurt
AT-9020 Klagenfurt, Austria
michael.wiegand@aau.at
Elisabeth Eder
Institut für Germanistik
elisabeth.eder@aau.at
Josef Ruppenhofer
Leibniz Institute for the German Language
D-68161 Mannheim, Germany
ruppenhofer@ids-mannheim.de
We address the task of distinguishing implicitly abusive sentences on identity groups (Muslims terrorize the world daily) from other group-related negative polar sentences (Muslims despise terrorism). Implicitly abusive language are utterances not conveyed by abusive words (e.g. bimbo or scum). So far, the detection of such utterances could not be properly addressed since existing datasets displaying a high degree of implicit abuse are fairly biased. Following the recently proposed strategy to solve implicit abuse by separately addressing its different subtypes, we present a new focused and less biased dataset that consists of the subtype of atomic negative sentences about identity groups. For that task, we model components that each address one facet of such implicit abuse, i.e. depiction as perpetrators, aspectual classification and non-conformist views. The approach generalizes across different identity groups and languages.
Abusive language is commonly defined as hurtful, derogatory or obscene utterances made by one person to another person. Examples are (1)-(2).
In the literature, closely related terms include hate speech (Waseem and Hovy, 2016) or cyber bullying (Zhong et al., 2016). While there may be nuanced differences in meaning, they are all compatible with the general definition above.
(1) stop editing this, you dumbass.
(2) Go lick a pig you arab muslim piece of scum.
Due to the rise of user-generated web content, the amount of abusive language is growing. NLP
methods are required to focus human review efforts towards the most relevant microposts. Though there has been much work on abusive language detection in general, there has been little work focusing on implicit forms of abusive language (3)-(4) (Waseem et al., 2017). By implicit we understand abusive language that is not conveyed by (unambiguously) abusive words (e.g. bimbo, scum).
(3) Did Stevie Wonder choose these models?
(4) You inspire my inner serial killer.
Detailed analyses of the output of existing classifiers have also revealed that currently only explicit abuse can be reliably detected (van Aken et al., 2018; Wiegand et al., 2019, 2021b).
In this paper, we define implicit abuse as those abusive utterances that lack any abusive word according to the largest lexicon of abusive words available, i.e. the lexicon by Wiegand et al. (2018).
In particular, datasets focusing on abuse towards identity groups (Jews, gay people etc.) contain a high degree of implicit abuse. For example, according to Wiegand et al. (2021b), on the dataset from Waseem and Hovy (2016),  $56\%$  of the abusive instances are implicit, while on the dataset from Sap et al. (2020), as many as  $62\%$  are.
So far, existing research on implicitly abusive language detection on identity groups has been limited by various biases on existing datasets (Arango et al., 2019; Wiegand et al., 2019), most prominently the identity-group bias (Dixon et al., 2018): mentions of identity groups almost exclusively occur in microposts that are considered abusive. As a consequence, most classifiers erroneously learn identity groups as clues for abusive language.
Given that implicit abuse is a challenging problem, Wiegand et al. (2021b) argue that the only
http://thelawdictionary.org
Human Language Technologies, pages 5600 - 5612
reasonable approach to solve this problem is to address specific subtypes individually rather than consider all types of implicit abuse at once.
In this paper, we address the task of distinguishing implicitly abusive remarks on identity groups (5)-(7) from other negative polar sentences (8)-(10). The task is a binary classification problem. Each instance is an individual atomic sentence.
(5) Jews succumb to cultural degeneracy. (ABUSIVE)
(6) Gay people are contaminating our planet. (ABUSIVE)
(7) Women fabricate menopausal symptoms. (ABUSIVE)
(8) Jews grieve for Orlando. (OTHER)
(9) Gay people are defying stereotypes! (OTHER)
(10) Women dread return of Taliban overseas. (OTHER)
We create a novel less biased dataset for this task. In NLP, there is an increasing awareness of the importance of producing such data (Gardner et al., 2020). Moreover, Zhou et al. (2021) find that ensuring the quality of datasets during their creation is considerably more effective than even the most sophisticated statistical debiasing techniques.
Unlike previous work, we focus on a linguistically informed classification approach and show that this approach is equally effective for different identity groups and can be used to outperform supervised classifiers trained on existing datasets.
We consider only negative polar utterances, since implicitly abusive microposts have a predominantly negative sentiment. For instance, on a random sample of 200 implicitly abusive instances from the dataset by Sap et al. (2020), we could not find a single remark with a positive or neutral sentiment.
Our contributions are the following:
- We present the first extensive study on how to detect implicitly abusive remarks among negative atomic remarks on identity groups.
- We establish the predictiveness of 3 linguistic features, namely, aspectual classification, the detection of perpetrators and non-conformist views. The latter two features are addressed for the first time, in general.
- We present a new dataset for this task.
- We introduce new lexical resources for detecting perpetrators and non-conformist views.
This paper only addresses one subset of implicit abuse. However, we consider this focus appropriate, since it is not trivial to detect these instances. As a comprehensive classifier that can detect all these types, we envisage a meta-classifier that collects predictions of individual classifiers designed for different subtypes of abusive language.
All resources created as part of this research are made publicly available. They are contained in the supplementary material<sup>2</sup> to this paper, which also includes implementation details.
Much of the previous work in abusive language detection follows a one-size-fits-all approach (Fortuna and Nunes, 2018). Surveys on existing datasets do not address implicit abuse (Vidgen and Derczynski, 2020; Poletto et al., 2021).
Wiegand et al. (2021b) present a roadmap on implicit abuse arguing that this type of abusive language has not adequately been addressed in previous work. No classification experiments are presented. Next to implicit abuse towards identity groups, they identify as subtypes dehumanization, euphemisms, call for action, multimodal abuse and comparisons. Comparisons are also addressed by Wiegand et al. (2021a) who present the first dataset for this subtype along with classification experiments. The comparisons do not target identity groups. Therefore, our novel dataset and the comparison dataset comprise different sentence types.
Breitfeller et al. (2019) present a study on microaggressions which are comments or actions expressing a prejudiced attitude towards marginalized groups unconsciously. Such instances are cases of implicit abuse. Since this is a descriptive study no data for classification are introduced.
Han and Tsvetkov (2020) propose a classification approach for what they call veiled toxicity, an umbrella term for many different subtypes of implicit abuse. The approach is evaluated on the dataset by Sap et al. (2020) which Wiegand et al. (2021b) report to have considerable biases.
ElSherief et al. (2021) introduce a general dataset for implicit abuse which is sampled from tweets by hate groups. The authors report biases in the dataset, such as the identity-group bias.
As a source for our data, we chose Twitter since it is a platform that contains a high degree of abusive language. We focused on 4 identity groups that cover a range of different characteristics (religion, sexual orientation and gender) and that can also be frequently found in existing datasets. Moreover, they need to occur with sufficient frequency in both
$^{2}$ https://github.com/miwieg/naacl2022_identity_groups
languages we are going to examine. The groups are gay people<sup>3</sup>, Jews, Muslims and women.<sup>4</sup>
The abusive utterances we are looking for are essentially stereotypical sentences on identity groups. Such remarks typically realize the abused target, i.e. the identity group, as the agent (i.e. logical subject) of the verb (5)-(7). Our new dataset focuses only on this argument position since stereotypical remarks usually depict identity groups as the entities performing some action (agent) rather than being affected by it (patient, i.e. logical object). We obtain such utterances by extracting tweets containing mentions of our identity groups followed by a negative polar verb. (This strategy has been proposed by Wiegand et al. (2021b) in order to ensure lexical variability.) The focus on verbs rather than on nouns and adjectives was motivated by the fact that the latter two are more likely to be explicitly abusive words. For example, these parts of speech compose  $91\%$  of the lexicon by Wiegand et al. (2018). In this work, we are interested in implicit abuse, however. To test the recall of our sampling approach, we inspect two random samples of 200 abusive (atomic) instances from two popular datasets that focus on identity groups (Sap et al., 2020; Waseem and Hovy, 2016). We find that  $80/84\%$  of the instances realize the identity group as an agent.  $70/70\%$  of the predicates are verbs, the remainder being adjectives and nouns. Of the verbal predicates,  $79/92\%$  were negative polar verbs.
Vidgen et al. (2021b) recently introduced a dataset similar to ours: It focuses on identity groups and also aims at having annotators create suitable non-abusive data. Their goal is to reduce the identity-group bias on their data by a large degree. We refer to this dataset as DynaB. We examined the non-abusive instances in DynaB for our 4 identity groups (Table 1) and found that more than  $80\%$  of the instances are cases of reported abuse (Chiril et al., 2020), as in (11), negations (12), or simply positive or neutral utterances (13).
(11) It's rude to keep saying Jews own the media.
(12) Jews do not drive climate change.
(13) Jews are industrious.
Our dataset, however, consists of atomic sentences,
3For this group, we used the terms gay people and lesbians. Other expressions, such as gays or queer, were too infrequent.
4Ideally, we would also have included black people as an additional identity group. However, it was not possible to obtain a sufficient amount of implicitly abusive data for this identity group in both languages that we consider in this paper.
i.e. there is no negation or reported abuse (5)-(10). Further, all sentences convey a negative sentiment. We believe this to be more challenging since a classifier needs a proper understanding of the atomic utterances themselves rather than looking for positive/neutral sentiment (13) or context clues indicating a non-abusive nesting, such as negation words (e.g. not (11)) or reporting verbs (e.g. say (12)).
We implemented the following measures proposed by Wiegand et al. (2021b) for producing less biased data for the detection of implicit abuse.
- Our data is sampled from one textual source, i.e. Twitter. Both abusive and non-abusive sentences are sampled by the same pattern (i.e. mention of identity group preceding a negative verb). Thus no biases are caused by merging instances from different text sources.
- In order to avoid any user biases, tweets were sampled from a wide set of different users. The average number of tweets per user is 1.1.
- In order to avoid a focus on frequently occurring verbs, we sampled our dataset from a wide set of negative polar verbs. On average, each verb occurs twice in the final dataset. Unlike previous datasets, this sampling strategy thus puts due emphasis on the "long tail" of the verb distribution.
- We only included sentences that do not contain explicitly abusive words. Otherwise, classifiers could easily detect the respective abusive utterances since they would just have to focus on these explicit clues.
- We remove any text co-occurring with our sentences that might give rise to spurious correlations, e.g. hashtags or user names. We observed that particularly hashtags, such as #banIslam or #feminismIsCancer, often strongly correlate with abusive tweets. Such hashtags display a behaviour similar to explicitly abusive words.
We created a gold standard for English and another, less-resourced language, German. Exactly the same sampling procedure was applied to both datasets. However, due to the sparsity of German language content on Twitter (Hong et al., 2011), the German dataset is smaller.
Both datasets were annotated via the crowdsourcing platform Prolific. The label of each
We used the list of negative polar verbs contained in the resources by Wiegand et al. (2018).
<https://www.prolific.co/
property	English	German
sentences	2221	970
abusive sentences	56.24%	52.16%
non-abusive sentences	43.76%	47.84%
sentences on gay people	403	154
sentences on Jews	545	184
sentences on Muslims	782	367
sentences on women	491	265
no. of unique verbs	965	534
avg. frequency of verbs	2.30	1.82
avg. sentence length (in tokens)	7.75	7.00
avg. no. of sentences per user	1.05	1.10
Table 1: Statistics of the datasets
instance represents the majority vote of 5 different crowdworkers, who were native speakers. We opted for a very high approval rate (i.e.  $95\%$  or higher) in order to guarantee a sufficiently high annotation quality. (The supplementary material contains annotation guidelines.) Table 1 offers some descriptive statistics.
On a random sample of 200 sentences, we computed the agreement between the majority vote of our crowdsourced judgments and one co-author of this paper. We measured substantial agreement of  $\kappa = 0.87$  on the English and  $\kappa = 0.82$  on the German dataset (Landis and Koch, 1977).
4 Supervised Classifiers and Evaluation
We consider RoBERTa (Liu et al., 2019) as a baseline for generic supervised classification for English data. For our German data, we use the best transformer according to Chan et al. (2020). We fine-tune the pretrained models on the given task using the FLAIR framework (Akbik et al., 2019). (The supplementary notes contain more details on all classifiers including hyperparameter settings.)
As evaluation measures, we use macro-average precision, recall, F1-score. For all classifiers built with transformers, we report the average over 5 training runs (including standard deviation). All other classifiers produce deterministic output.
5 Linguistically Informed Classifier
We propose a linguistically informed classifier which models 3 component tasks. We describe how this classifier is built for English. The component tasks represent concepts which have been suggested to be predictive for this task (Wiegand et al., 2021b) but, so far, could not be tested due to the lack of data. In order to avoid overfitting, each component comes with a separate classifier being built on training data different to the test data of our main task. Since we manually labeled our
dataset also for each of the component tasks<sup>7</sup> we can conduct an intrinsic evaluation of each component, too. In order to have an unbiased annotation, each crowdworker was only allowed to participate in exactly one of our annotation tasks.
5.1 Component 1: Aspectual Classifier
The Task. In our first task we address aspectual classification. Abusive utterances regarding identity groups are usually stereotypes (Sap et al., 2020). Per definition, stereotypes coincide with habitual (or non-episodic) aspect (14)-(15). On the other hand, episodic aspect (16)-(17), i.e. utterances that express information about a single event (Friedrich and Pinkal, 2015), despite the fact that they may be tendentious (Mendelsohn et al., 2021) or even be cases of fake news (Zhou and Zafarani, 2020), is more likely to be non-abusive. We distinguish between episodic and non-episodic sentences.
(14) Muslims are vandalising Hindu temples every day. (non-episodic)
(15) The Jews damage our souls. (non-episodic)
(16) Muslims vandalise newspaper offices in Odisha over publication of Mohammed's images. (episodic)
(17) Jews damage olive trees in West Bank. (episodic)
The Method. Aspectual classification was investigated by Friedrich and Pinkal (2015) and an implementation of their classifier is available as part of *silent* (Friedrich et al., 2016). However, we observed substantial issues with *silent* when applied to our data. The tool was trained on Wikipedia and MASC (Ide et al., 2008). On these datasets, episodic aspect is biased towards past tense. However, our data originates from Twitter and both episodic and non-episodic sentences co-occur in present tense.
As a consequence, we decided to build a classifier from scratch. As no suitable labeled training data for our domain (i.e. social media) is available, we decided to apply a form of distant supervision (Mintz et al., 2009). As a proxy for episodic sentences, we sampled tweets from news feeds (e.g. LGBT_news or GazaTVNews) from Twitter. Such tweets typically report on specific events (18)-(19).
(18) Israel strikes Iranian targets inside of Syria.
(19) North Texas Student Expelled for Being Gay
For the non-episodic sentences, we considered the implied statements (21) from the social bias frames
For all component tasks, we obtained a substantial agreement with the lowest being at  $\kappa = 0.65$  (detection of perpetrators) using the same random sample as for the main task.
feature	example	episodic?
is the sentence in progressive tense?	Women are unbalancing the world.	no
is there a mention denoting a specific point in time?	Lesbians are wrestling right now on Jerry springer.	yes
is there a generalizing adverbial phrase?	Muslims slander Christians all the time.	no
is there some quantification?	Muslims assassinate 2 Christian aid workers.	yes
does the verb describe a state?	Women hate short men.	no
is there a concrete noun?	Muslims Steal Ambulance.	yes
is there a mention of a person name?	Jews Censor David Duke's Youtube Channel.	yes
is there a mention of a (specific) location?	Muslims Brawl At NY Amusement Park.	yes
Table 2: Feature set of the feature-based aspectual (baseline) classifier (more details in the supplementary notes).
majority-class	silent	feature-based	RoBERTa
F1	38.4	53.0	76.6	76.9 (±1.0)
Table 3: Evaluation of aspectual classification.
corpus (Sap et al., 2020). In that dataset, the annotators added for each abusive instance (20) the stereotype that the remark alludes to (21).
(20) What do you call a movie with an all-Muslim cast? A box office bomb.
(21) implied statement: Muslims are all terrorists
For our training set, we randomly sampled 1000 news tweets (= episodic) and 1000 implied statements (= non-episodic). As classifiers, we trained RoBERTa and a feature-based baseline. The latter was included since generic supervised classifiers (such as RoBERTa) are susceptible of learning spurious correlations contained in training data. Such correlations cannot be ruled out as our training data for the two classes was sampled from different sources. Our feature-based baseline, which is a logistic regression trained on high-level features that are fairly domain independent, makes such overfitting less likely. The features for detecting episodic sentences check for mentions of concrete entities or a specific point in time, while features for non-episodic sentences try to detect states and generalizations. Table 2 lists the full feature set.
Table 3 shows the result of the different classifiers on our English dataset (§3). silent performs poorly. We attribute it to the tense bias reported above. The feature-based baseline is strong but it does not outperform RoBERTa. Therefore, RoBERTa does not seem to be seriously affected by spurious correlations. We use the output of RoBERTa in all subsequent experiments. In order to facilitate the combination with other components of our classifier, we use the majority vote of the 5 runs of this classifier.
5.2 Component 2: Perpetrator Classifier
The Task. A common stereotype that can be observed with every identity group is the depiction
as perpetrators (22)-(24). By perpetrators, we understand persons who commit an illegal, criminal, or evil act. Although different identity groups are typically depicted as different perpetrators (e.g. Muslims are depicted as terrorists (22), women are considered to be dishonest (23), while gay men are accused of being pedophiles (24)), all these stereotypes describe actions that involve criminal offenses (e.g. raping, stealing) or morally contemptible behaviour (e.g. adultery, lying). We think it is most economical to frame the detection of perpetrators as a single task.
(22) [Muslims]agent terrorize the world daily.
(23) [Women]agent betray their partners.
(24) [Gay people]agent are raping our children.
We consider the task a form of semantic role labeling (Gildea and Jurafsky, 2002), i.e. perpetrators are specific entities evoked by particular verbs. Therefore, we need to find perpetrator-evoking verbs (e.g. terrorize, betray, rape) and the respective argument position of the perpetrator.
The Method. In order to obtain a labeled dataset of perpetrator-evoking verbs, we randomly sampled 500 negative polar verbs from the Subjectivity Lexicon (Wilson et al., 2005) and asked crowdworkers to form simple sentences (only a main clause) in which the given verb evokes an event that includes some perpetrator. The 500 verbs are in no way tuned for our test data ( $\S 3$ ).<sup>9</sup> Since we do not want crowdworkers to invent any anti-Semitic, homophobic, Islamophobic or misogynist sentences, we invented a fictitious people whose name has no phonetic resemblance to existing identity groups. The crowdworkers were asked to depict these people as perpetrator, if possible. Obviously, plausible sentences can only be formed with the subset of perpetrator-evoking verbs we are looking for. For other verbs, such as grieve or dread, forming such
8www.dictionary.com/browse/perpetrator
<sup>9</sup>Our English dataset contains 965 verbs of which only 373 can be found among the 500 from the Subjectivity Lexicon.
sentences is not possible. Therefore, crowdworkers were asked not to provide a sentence in case they felt that they were unable to meet the criterion of constructing a context with a perpetrator being a participating entity of the event evoked by the given verb. Only if the majority of 5 crowdworkers managed to produce such sentences for the same verb, did we consider it as a perpetrator-evoking verb. This setting also allowed us to identify the semantic role of the perpetrator. Overall, 165 out of 500 verbs were identified as perpetrator-evoking verbs. In  $96\%$  of the respective sentences, the semantic role of the perpetrator was the agent of the verb (as in (22)-(24)).
In a second step, we extended the list of perpetrator-evoking verbs. Our aim is to obtain a (nearly) exhaustive list of perpetrator-evoking verbs. Therefore, we train a classifier on our 500 verbs (each verb labeled as either perpetrator-evoking or other) and classify each verb from the largest list of publicly available negative polar verbs. We took the verbs from the set of negative polar words from Wiegand et al. (2018) (totaling 1,700 negative verbs). We trained a logistic regression classifier where each verb was represented by its (publicly available) word embedding induced on Common Crawl (Mikolov et al., 2018).<sup>10</sup> We ended up with 491 perpetrator-evoking verbs. Our lexicon-based classifier identifies a perpetrator if it is observed as an agent of one of these 491 verbs. This classifier is run on our dataset ( $\S 3$ ). The output is evaluated against the gold annotation for this component task.