Venue: NAACL 2022

Identifying Implicitly Abusive Remarks about Identity Groups using a Linguistically Informed Approach

 Michael Wiegand

Digital Age Research Center (D'ARC)

Alpen-Adria-Universitat Klagenfurt

AT-9020 Klagenfurt, Austria

michael.wiegand@aau.at Elisabeth Eder

Institut fur Germanistik

Alpen-Adria-Universitat Klagenfurt

AT-9020 Klagenfurt, Austria

elisabeth.eder@aau.at Josef Ruppenhofer

Leibniz Institute for the German Language

D-68161 Mannheim, Germany

ruppenhofer@ids-mannheim.de

###### Abstract

We address the task of distinguishing implicitly abusive sentences on identity groups (_Muslims terrorize the world daily_) from other group-related negative polar sentences (_Muslims decspise terrorism_). Implicitly abusive language are utterances not conveyed by abusive words (e.g. _bimbo_ or _scum_). So far, the detection of such utterances could not be properly addressed since existing datasets displaying a high degree of implicit abuse are fairly biased. Following the recently proposed strategy to solve implicit abuse by separately addressing its different subtypes, we present a new focused and less biased dataset that consists of the subtype of atomic _negative_ sentences about identity groups. For that task, we model components that each address one facet of such implicit abuse, i.e. depiction as perpetrators, aspectual classification and non-conformist views. The approach generalizes across different identity groups and languages.

+
Footnote †: FOOTNOTE:3]Footnote †: footnotetext: 

## 1 Introduction

Abusive language is commonly defined as hurtful, derogatory or obscene utterances made by one person to another person.1 Examples are (1)-(2).

Footnote 1: [http://thelawdictionary.org](http://thelawdictionary.org)

In the literature, closely related terms include _hate speech_Waseem and Hovy (2016) or _cyber bullying_Zhong et al. (2016). While there may be nuanced differences in meaning, they are all compatible with the general definition above.

* stop editing this, you dumbass.
* Go lick a pig you arab muslim piece of scum.

Due to the rise of user-generated web content, the amount of abusive language is growing. NLP methods are required to focus human review efforts towards the most relevant microposts. Though there has been much work on abusive language detection in general, there has been little work focusing on **implicit** forms of abusive language (3)-(4) Waseem et al. (2017). By _implicit_ we understand abusive language that is _not_ conveyed by (unambiguously) abusive words (e.g. _bimbo_, _scum_).

* Did Stevie Wonder choose these models?
* You inspire my inner serial killer.

Detailed analyses of the output of existing classifiers have also revealed that currently only explicit abuse can be reliably detected van Aken et al. (2018); Wiegand et al. (2019, 2021).

In this paper, we define implicit abuse as those abusive utterances that lack any abusive word according to the largest lexicon of abusive words available, i.e. the lexicon by Wiegand et al. (2018).

In particular, datasets focusing on abuse towards **identity groups** (_Jews_, _gay people_ etc.) contain a high degree of implicit abuse. For example, according to Wiegand et al. (2021), on the dataset from Waseem and Hovy (2016), 56% of the abusive instances are implicit, while on the dataset from Sap et al. (2020), as many as 62% are.

So far, existing research on implicitly abusive language detection on identity groups has been limited by various biases on existing datasets Arango et al. (2019); Wiegand et al. (2019), most prominently the **identity-group bias**Dixon et al. (2018): mentions of identity groups almost exclusively occur in microposts that are considered abusive. As a consequence, most classifiers erroneously learn identity groups as clues for abusive language.

Given that implicit abuse is a challenging problem, Wiegand et al. (2021) argue that the only 

reasonable approach to solve this problem is to address specific subtypes individually rather than consider all types of implicit abuse at once.

In this paper, we address the **task** of distinguishing implicitly abusive remarks on identity groups (5)-(7) from other negative polar sentences (8)-(10). The task is a **binary classification problem**. Each instance is an individual atomic sentence.

* Jews succumb to cultural degeneracy. _(ABUSIVE)_
* Gay people are contaminating our planet. _(ABUSIVE)_
* Women fabricate menopausal symptoms. _(ABUSIVE)_
* Jews grieve for Orlando. _(OTHER)_
* Gay people are defying stereotypes! _(OTHER)_
* Women dread return of Taliban overseas. _(OTHER)_

We create a novel less biased dataset for this task. In NLP, there is an increasing awareness of the importance of producing such data Gardner et al. (2020). Moreover, Zhou et al. (2021) find that ensuring the quality of datasets during their creation is considerably more effective than even the most sophisticated statistical debiasing techniques.

Unlike previous work, we focus on a **linguistically informed classification approach** and show that this approach is equally effective for different identity groups and can be used to outperform supervised classifiers trained on existing datasets.

We consider only negative polar utterances, since implicitly abusive microposts have a predominantly negative sentiment. For instance, on a random sample of 200 implicitly abusive instances from the dataset by Sap et al. (2020), we could not find a single remark with a positive or neutral sentiment.

Our **contributions** are the following:

* We present the first extensive study on how to detect implicitly abusive remarks among negative atomic remarks on identity groups.
* We establish the predictiveness of 3 linguistic features, namely, aspectual classification, the detection of perpetrators and non-conformist views. The latter two features are addressed for the first time, in general.
* We present a new dataset for this task.
* We introduce new lexical resources for detecting perpetrators and non-conformist views.

This paper only addresses one subset of implicit abuse. However, we consider this focus appropriate, since it is not trivial to detect these instances. As a comprehensive classifier that can detect all these types, we envisage a meta-classifier that collects predictions of individual classifiers designed for different subtypes of abusive language.

All resources created as part of this research are made **publicly available**. They are contained in the supplementary material2 to this paper, which also includes implementation details.

Footnote 2: [https://github.com/miwieg/naacl2022_identity_groups](https://github.com/miwieg/naacl2022_identity_groups)

## 2 Related Work

Much of the previous work in abusive language detection follows a one-size-fits-all approach Fortuna and Nunes (2018). Surveys on existing datasets do not address implicit abuse Vidgen and Derczynski (2020); Poletto et al. (2021).

Wiegand et al. (2021b) present a roadmap on implicit abuse arguing that this type of abusive language has not adequately been addressed in previous work. No classification experiments are presented. Next to implicit abuse towards identity groups, they identify as subtypes dehumanization, euphemisms, call for action, multimodal abuse and comparisons. Comparisons are also addressed by Wiegand et al. (2021a) who present the first dataset for this subtype along with classification experiments. The comparisons do not target identity groups. Therefore, our novel dataset and the comparison dataset comprise different sentence types.

Breitfeller et al. (2019) present a study on _microaggressions_ which are comments or actions expressing a prejudiced attitude towards marginalized groups unconsciously. Such instances are cases of implicit abuse. Since this is a descriptive study no data for classification are introduced.

Han and Tsvetkov (2020) propose a classification approach for what they call _veiled toxicity_, an umbrella term for many different subtypes of implicit abuse. The approach is evaluated on the dataset by Sap et al. (2020) which Wiegand et al. (2021b) report to have considerable biases.

Elsherief et al. (2021) introduce a general dataset for implicit abuse which is sampled from tweets by hate groups. The authors report biases in the dataset, such as the identity-group bias.

## 3 Data

As a source for our data, we chose Twitter since it is a platform that contains a high degree of abusive language. We focused on 4 identity groups that cover a range of different characteristics (religion, sexual orientation and gender) and that can also be frequently found in existing datasets. Moreover, they need to occur with sufficient frequency in both 

languages we are going to examine. The groups are _gay people3_, Jews, Muslims and women.4

Footnote 3: For this group, we used the terms _gap people_ and _lesbians_. Other expressions, such as _gays_ or _queer_, were too infrequent.

Footnote 4: Ideally, we would also have included _black people_ as an additional identity group. However, it was not possible to obtain a sufficient amount of implicitly abusive data for this identity group in both languages that we consider in this paper.

The abusive utterances we are looking for are essentially stereotypical sentences on identity groups. Such remarks typically realize the abused target, i.e. the identity group, as the agent (i.e. logical subject) of the verb (5)-(7). Our new dataset focuses only on this argument position since stereotypical remarks usually depict identity groups as the entities performing some action (agent) rather than being affected by it (patient, i.e. logical object). We obtain such utterances by extracting tweets containing mentions of our identity groups followed by a negative polar verb. (This strategy has been proposed by Wiegand et al. [2018] in order to ensure lexical variability.) The focus on verbs rather than on nouns and adjectives was motivated by the fact that the latter two are more likely to be explicitly abusive words. For example, these parts of speech compose 91% of the lexicon by Wiegand et al. [2018]. In this work, we are interested in implicit abuse, however. To test the recall of our sampling approach, we inspect two random samples of 200 abusive (atomic) instances from two popular datasets that focus on identity groups [Sap et al., 2020; Waseem and Hovy, 2016]. We find that 80/84% of the instances realize the identity group as an agent. 70/70% of the predicates are verbs, the remainder being adjectives and nouns. Of the verbal predicates, 79/92% were negative polar verbs.

Vidgen et al. [2021b] recently introduced a dataset similar to ours: It focuses on identity groups and also aims at having annotators create suitable non-abusive data. Their goal is to reduce the identity-group bias on their data by a large degree. We refer to this dataset as \(\textit{DynaB}\). We examined the non-abusive instances in \(\textit{DynaB}\) for our 4 identity groups (Table 1) and found that more than 80% of the instances are cases of _reported_ abuse [Chiril et al., 2020], as in (11), negations (12), or simply positive or neutral utterances (13).

* **It's rude to keep saying** Jews own the media.
* Jews do **not** drive climate change.
* Jews are **industrious**.

Our dataset, however, consists of **atomic** sentences, i.e. there is no negation or reported abuse (5)-(10). Further, all sentences convey a **negative** sentiment. We believe this to be more challenging since a classifier needs a proper understanding of the atomic utterances themselves rather than looking for positive/neutral sentiment (13) or context clues indicating a non-abusive nesting, such as negation words (e.g. _not_ (11)) or reporting verbs (e.g. _say_ (12)).

We implemented the following measures proposed by Wiegand et al. [2021b] for producing less biased data for the detection of implicit abuse.

* Our data is sampled from one textual source, i.e. Twitter. Both abusive and non-abusive sentences are sampled by the same pattern (i.e. mention of identity group preceding a negative verb). Thus no biases are caused by merging instances from different text sources.
* In order to avoid any user biases, tweets were sampled from a wide set of different users. The average number of tweets per user is 1.1.
* In order to avoid a focus on frequently occurring verbs, we sampled our dataset from a wide set of negative polar verbs.5 On average, each verb occurs twice in the final dataset. Unlike previous datasets, this sampling strategy thus puts due emphasis on the "long tail" of the verb distribution. Footnote 5: We used the list of negative polar verbs contained in the resources by Wiegand et al. [2018].
* We only included sentences that do **not** contain explicitly abusive words. Otherwise, classifiers could easily detect the respective abusive utterances since they would just have to focus on these explicit clues.
* We remove any text co-occurring with our sentences that might give rise to spurious correlations, e.g. hashtags or user names. We observed that particularly hashtags, such as _#banIslam_ or _#feminismIsCancer_, often strongly correlate with abusive tweets. Such hashtags display a behaviour similar to explicitly abusive words.

We created a gold standard for English and another, less-resourced language, German. Exactly the same sampling procedure was applied to both datasets. However, due to the sparsity of German language content on Twitter [Hong et al., 2011], the German dataset is smaller.

Both datasets were **annotated via the crowdsourcing platform Prolific.6** The label of each 

instance represents the majority vote of 5 different crowdworkers, who were native speakers. We opted for a very high approval rate (i.e. 95% or higher) in order to guarantee a sufficiently high annotation quality. (_The supplementary material contains annotation guidelines._) Table 1 offers some descriptive statistics.

On a random sample of \(200\) sentences, we computed the agreement between the majority vote of our crowdsourced judgments and one co-author of this paper. We measured substantial agreement of \(\kappa=0.87\) on the English and \(\kappa=0.82\) on the German dataset Landis and Koch (1977).

## 4 Supervised Classifiers and Evaluation

We consider RoBERTa Liu et al. (2019) as a baseline for generic supervised classification for English data. For our German data, we use the best transformer according to Chan et al. (2020). We fine-tune the pretrained models on the given task using the FLAIR framework Akbik et al. (2019). (_The supplementary notes contain more details on all classifiers including hyperparameter settings._)

As evaluation measures, we use macro-average precision, recall, F1-score. For all classifiers built with transformers, we report the average over 5 training runs (including standard deviation). All other classifiers produce deterministic output.

## 5 Linguistically Informed Classifier

We propose a linguistically informed classifier which models 3 component tasks. We describe how this classifier is built for English. The component tasks represent concepts which have been suggested to be predictive for this task Wiegand et al. (2021) but, so far, could not be tested due to the lack of data. In order to avoid overfitting, each component comes with a separate classifier being built on training data different to the test data of our main task. Since **we manually labeled our dataset also for each of the component tasks7 we can conduct an intrinsic evaluation of each component, too**. In order to have an unbiased annotation, each crowdworker was only allowed to participate in exactly one of our annotation tasks.

Footnote 7: For all component tasks, we obtained a substantial agreement with the lowest being at \(\kappa=0.65\) (detection of perpetrators) using the same random sample as for the main task.

### Component 1: Aspectual Classifier

#### The Task.

In our first task we address aspectual classification. Abusive utterances regarding identity groups are usually stereotypes Sap et al. (2020). Per definition, stereotypes coincide with habitual (or non-episodic) aspect (14)-(15). On the other hand, episodic aspect (16)-(17), i.e. utterances that express information about a single event Friedrich and Pinkal (2015), despite the fact that they may be tendentious Mendelsohn et al. (2021) or even be cases of fake news Zhou and Zafarani (2020), is more likely to be non-abusive. We **distinguish between episodic and non-episodic sentences**.

* Muslims are vandalising Hindu temples every day. (_non-episodic_)
* The Jews damage our souls. (_non-episode_)
* Muslims vandalise newspaper offices in Odisha over publication of Mohammed's images. (_episodic_)
* Jews damage olive trees in West Bank. (_episodic_)

#### The Method.

Aspectual classification was investigated by Friedrich and Pinkal (2015) and an implementation of their classifier is available as part of _sitent_Friedrich et al. (2016). However, we observed substantial issues with _sitent_ when applied to our data. The tool was trained on Wikipedia and MASC Ide et al. (2008). On these datasets, episodic aspect is biased towards past tense. However, our data originates from Twitter and both episodic and non-episodic sentences co-occur in present tense.

As a consequence, we decided to build a classifier from scratch. As no suitable labeled training data for our domain (i.e. social media) is available, we decided to apply a form of _distant supervision_Mintz et al. (2009). As a proxy for episodic sentences, we sampled tweets from news feeds (e.g. _LGBT__news_ or _GazaTVNews_) from Twitter. Such tweets typically report on specific events (18)-(19).

For the non-episodic sentences, we considered the _implied statements_ (21) from the _social bias frames_

\begin{table}
\begin{tabular}{|l|c|c|} \hline
**property** & **English** & **German** \\ \hline sentences & 2221 & 970 \\ abusive sentences & 56.24\% & 52.16\% \\ non-abusive sentences & 43.76\% & 47.84\% \\ \hline sentences on gay people & 403 & 154 \\ sentences on Jews & 545 & 184 \\ sentences on Muslims & 782 & 367 \\ sentences on women & 491 & 265 \\ \hline no. of unique verbs & 965 & 534 \\ avg. frequency of verbs & 2.30 & 1.82 \\ avg. sentence length (in tokens) & 7.75 & 7.00 \\ avg. no. of sentences per user & 1.05 & 1.10 \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of the datasets 

corpus_(Sap et al., 2020). In that dataset, the annotators added for each abusive instance (20) the stereotype that the remark alludes to (21).
3. What do you call a movie with an all-Muslim cast? A box office bomb.
4. _implied statement:_ Muslims are all terrorists

For our training set, we randomly sampled 1000 news tweets (=episodic) and 1000 implied statements (=non-episode). As classifiers, we trained RoBERTa and a feature-based baseline. The latter was included since generic supervised classifiers (such as RoBERTa) are susceptible of learning spurious correlations contained in training data. Such correlations cannot be ruled out as our training data for the two classes was sampled from different sources. Our feature-based baseline, which is a logistic regression trained on high-level features that are fairly domain independent, makes such overfitting less likely. The features for detecting episodic sentences check for mentions of concrete entities or a specific point in time, while features for non-episodic sentences try to detect states and generalizations. Table 2 lists the full feature set.

Table 3 shows the result of the different classifiers on our English dataset (SS3). _sitent_ performs poorly. We attribute it to the tense bias reported above. The feature-based baseline is strong but it does not outperform RoBERTa. Therefore, RoBERTa does not seem to be seriously affected by spurious correlations. We use the output of RoBERTa in all subsequent experiments. In order to facilitate the combination with other components of our classifier, we use the majority vote of the 5 runs of this classifier.

### Component 2: Perpetrator Classifier

**The Task.** A common stereotype that can be observed with every identity group is the depiction as perpetrators (22)-(24). By perpetrators, we understand persons who commit an illegal, criminal, or evil act.8 Although different identity groups are typically depicted as different perpetrators (e.g. Muslims are depicted as terrorists (22), women are considered to be dishonest (23), while gay men are accused of being pedophiles (24)), all these stereotypes describe actions that involve criminal offenses (e.g. raping, stealing) or morally contemptible behaviour (e.g. adultery, lying). We think it is most economical to frame the detection of perpetrators as a single task.

Footnote 8: www.dictionary.com/browse/perpetrator

* \([{\text{Muslims}}]_{agent}\) terrorize the world daily.
* \([{\text{Women}}]_{agent}\) betray their partners.
* \([{\text{Gay people}}]_{agent}\) are raping our children.

We consider the task a form of semantic role labeling Gildea and Jurafsky (2002), i.e. perpetrarors are specific entities evoked by particular verbs. Therefore, we need to **find perpetrator-evoking verbs** (e.g. _terrorize_, _betray_, _rape_) **and the respective argument position of the perpetrator**.

**The Method.** In order to obtain a labeled dataset of perpetrator-evoking verbs, we randomly sampled 500 negative polar verbs from the _Subjectivity Lexicon_(Wilson et al., 2005) and asked crowdworkers to form simple sentences (only a main clause) in which the given verb evokes an event that includes some perpetrator. The 500 verbs are in no way tuned for our test data (SS3).9 Since we do not want crowdworkers to invent any anti-Semitic, homophobic, Islamophobic or misogynist sentences, we invented a fictitious people whose name has no phonetic resemblance to existing identity groups. The crowdworkers were asked to depict these people as perpetrator, if possible. Obviously, plausible sentences can only be formed with the subset of perpetrator-evoking verbs we are looking for. For other verbs, such as _grieve_ or _dread_, forming such

\begin{table}
\begin{tabular}{|l|p{176.8pt}|c|} \hline
**feature** & **example** & **episodic?** \\ \hline is the sentence in progressive tense? & _Women are unbalancing the world._ & no \\ is there a mention denoting a specific point in time? & _Lesh’s are wrestling right now on Jerry springer._ & yes \\ is there a generalizing adverbial phrase? & _Muslims slander Christians all the time._ & no \\ is there some quantification? & _Mustims assassinate 2 Christian aid workers._ & yes \\ does the verb describe a state? & _Women hate short men._ & no \\ is there a concrete noun? & _Muslims Steel Ambivalence._ & yes \\ is there a mention of a person name? & _Jews Censor David Duke’s Joutube Channel._ & yes \\ is there a mention of a (specific) location? & _Muslims Brasyl At NY Amsunent Park._ & yes \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 2: Feature set of the feature-based aspectual (baseline) classifier (_more details in the supplementary notes_).

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline
**majorify-class** & **sitent** & **feature-based** & **RoBERTa** \\ \hline
**F1** & 38.4 & 53.0 & 76.6 & **76.9**\(\pm\)10) \\ \hline \end{tabular}
\end{table}
Table 3: Evaluation of aspectual classification.

 

sentences is not possible. Therefore, crowdworkers were asked not to provide a sentence in case they felt that they were unable to meet the criterion of constructing a context with a perpetrator being a participating entity of the event evoked by the given verb. Only if the majority of 5 crowdworkers managed to produce such sentences for the same verb, did we consider it as a perpetrator-evoking verb. This setting also allowed us to identify the semantic role of the perpetrator. Overall, 165 out of 500 verbs were identified as perpetrator-evoking verbs. In 96% of the respective sentences, the semantic role of the perpetrator was the agent of the verb (as in (22)-(24)).

In a second step, we extended the list of perpetrator-evoking verbs. Our aim is to obtain a (nearly) exhaustive list of perpetrator-evoking verbs. Therefore, we train a classifier on our 500 verbs (each verb labeled as either _perpetator-evoking_ or _other_) and classify each verb from the largest list of publicly available negative polar verbs. We took the verbs from the set of negative polar words from Wiegand et al. (2018) (totaling 1,700 negative verbs). We trained a logistic regression classifier where each verb was represented by its (publicly available) word embedding induced on Common Crawl Mikolov et al. (2018).10 We ended up with 491 perpetrator-evoking verbs. Our **lexicon-based classifier** identifies a perpetrator if it is observed as an agent of one of these 491 verbs. This classifier is run on our dataset (SS3). The output is evaluated against the gold annotation for this component task.

Footnote 10: [https://dl.fbaipublicfiles.com/fasttext/vectors-english/crawl-300d-2M.vec.zip](https://dl.fbaipublicfiles.com/fasttext/vectors-english/crawl-300d-2M.vec.zip)

As a baseline, we run a very fine-grained semantic-role labeling system based on **FrameNet**[Baker et al., 1998] on our data. We chose _open sesame_Swayamdipta et al. (2017) which is the most recent publicly available tool for semantic-role labeling based on FrameNet. Due to its fine-grained inventory, there are frame elements (this is the term for semantic roles in FrameNet) which semantically correspond to our concept of perpetrators. More precisely, we considered text spans as perpetrators if they are predicted to be one of the following frame elements: _Abuser_, _Assailant_, _Counter_actor_, _Destroyer_, _Invader_, _Killer_, _Manipulator_, _Offender_, _Perpetrator_ and _Wrongdoer_.

Table 4 shows the performance of the different classifiers to detect mentions of perpetrators in our English dataset (SS3). Our lexicon-based classifier outperforms FrameNet, which is known to have a limited lexical coverage Das and Smith (2011).

### Component 3: Non-Conformist Views

The TaskFor our third component task, we consider the sentiment of the agent towards the patient (as conveyed by the main verb in the sentence) in combination with the sentiment expected a priori towards the patient. (The agent is always the mention of the identity group.) This is illustrated in Table 5. We observe a systematic relationship between abusive language and fine-grained sentiment: **If the sentiment of the identity group (i.e. the agent) towards the patient is opposite to the prior sentiment of the patient, then this utterance depicts the identity group as having a non-conformist view.**11 Such views are perceived as abusive utterances: If someone attributes non-conformist views to some identity group, then, one often intends to stigmatize this group as not belonging to their own community. This phenomenon is referred to as _othering_Burnap and Williams (2016).

Footnote 11: In Table 5, we only distinguish between positive and negative sentiment. There is no neutral sentiment. In the context of these sentiment patterns, we found that neutral sentiment follows the same pattern as positive sentiment. Conflating positive and neutral sentiment facilitated automatic processing.

The MethodIn order to detect the above pattern indicating non-conformist views, we need the output of two modules: the first determines the prior sentiment of the patient (i.e. the phrase representing the logical object); the second determines the sentiment of the agent towards the patient. The prior sentiment of the patient can be easily detected by running a sentiment text classifier on that phrase. For this, we use _TweetEval_Barbieri et al. (2020).

The difficult part is to detect the sentiment of the agent towards the patient. Sentiment text classifiers are unable to determine such fine-grained sentiment information. They capture the general sentiment of a given text which may be different. For instance, (25) conveys a positive sentiment of _Muslims_ towards _violence_, while the sentiment of the sentence is generally considered negative.

\begin{table}
\begin{tabular}{|p{42.7pt}|p{42.7pt}|p{42.7pt}|p{42.7pt}|} \hline  & **majority-class** & **FrameNet** & **lexicon-based classifier** \\ \hline
**F1** & 33.9 & 60.1 & **70.5** \\ \hline \end{tabular}
\end{table}
Table 4: Evaluation of perpetrator classification.

 

Instead of a text classifier, we seek a lexicon that specifies for any negative polar verb (out of context) whether it conveys a positive sentiment of the agent towards the patient (e.g. _glorify_, _long (for)_, _pray (to)_) or a negative sentiment towards it (e.g. _abhor_, _dislike_, _suffer_). The only lexicons with such information are _EffectWordNet_ (**effectWN**)_Choi and Wiebe [2014] and the _connotation-frames lexicon_ (**frames**) Rashkin et al. [2016]. Unfortunately, both resources only cover about 40% of the verbs in our dataset. We also determined a significant level of noise in these resources (_as detailed in the supplementary notes_). Therefore, we decided to create a novel lexicon with that information. It should cover all possible negative verbs. We first had crowdworkers annotate for some (seed) negative verbs the sentiment of the agent towards the patient. We chose the 500 verbs we already used in SS5.2. The majority of the crowdworkers' judgements represent our gold standard annotation. On these annotated 500 verbs we trained a logistic regression classifier. As features, we represented each verb by its word embedding from Common Crawl. (Using such a representation is common practice for this task rashkin2016language.) The resulting classifier was run on the same large set of 1,700 negative verbs we used in SS5.2. For each verb the classifier predicts the sentiment of the agent towards the patient. The result is our novel lexicon. Since we have also manually annotated the sentiment of the agent towards the patient for each verb in the sentences of our labeled dataset (SS3), we can evaluate this lexicon against our labeled dataset.

Table 6 evaluates the different lexicons to determine the sentiment of the agent towards the patient on our novel dataset. The table shows that our novel bootstrapped lexicon produces a notable improvement over the existing resources.

### How the Final Classifier is Built

Figure 1 shows how the component tasks introduced in SS5.1-5.3 are combined to produce our linguistically informed classifier: We consider those sentences as abusive that are non-episodic and which either depict the identity group as perpetrator or attribute non-conformist views to it. We use the best-performing component classifiers as determined by our previous evaluation (SS5.1-SS5.3).

We also experimented with a supervised classifier that uses the predictions from our component classifiers as features. However, since the classification performance was on a par with our proposed (rule-based) classifier (Figure 1), we decided in favor of the latter classifier. It has a clear advantage over supervised classification in that it does not require any labeled training data to combine the predictions of the component classifiers.

## 6 Evaluation on English Dataset

We evaluate the linguistically informed classifier (Figure 1) on our new English dataset (for implicit abuse) against other classifiers trained on existing datasets. We carry out a **cross-dataset evaluation**: None of the classifiers, including our linguistically informed classifier, has been trained on our English dataset. Given the recent criticism against within-dataset evaluation Arango et al. [2019]; Wiegand et al. [2019] in which high performance is often the result of overfitting, this is a fairly unbiased set up.

As datasets for training supervised baselines, we chose those that focus on implicit abuse ElSherief et al. [2021] or abuse towards identity groups

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline  & **majority-class** & **frames** & **effectWN** & **novel lexicon** \\ \cline{1-2}
**F1** & 41.9 & 64.1 & 64.8 & **71.6** \\ \hline \end{tabular}
\end{table}
Table 6: Evaluation of fine-grained sentiment analysis.

Figure 1: Linguistically informed classifier.

 

(Waseem and Hovy, 2016; Sap et al., 2020; Vidgen et al., 2021a,b). We also included Founta et al. (2018) as a more general dataset sampled from Twitter. For each dataset, we fine-tune the pre-trained RoBERTa model (SS4) on the training partition of the respective dataset. As a further baseline, we run the state-of-the-art classifier for abusive language detection _PerspectiveAPI12_ on our dataset.

Footnote 12: www.perspectiveapi.com

We also include an **oracle** version of our linguistically informed classifier, that combines the gold standard annotation for the component tasks (SS5.1-SS5.3) rather than the outputs of the respective classifiers. This can be considered the upper bound for the linguistically informed classifier.

Finally, we also consider a **human classifier** as a general upper bound. We randomly sampled the judgment of one individual annotator from the crowdsourced gold-standard annotation for the detection of abusive language. This individual judgement may notably differ from the gold standard label which is the majority label of 5 annotators.

Table 7 displays the results. The classifiers trained on existing datasets do not perform well on our new dataset. The best classifier among them is the one trained on the _DynaB_-dataset. For _DynaB_ (unlike the other datasets), special attention was paid to the inclusion of non-abusive instances (SS3). Still, our linguistically informed classifier is more effective. _DynaB_ suffers from the _identity-group bias_ (SS1): its recall for non-abusive instances is only at 20%. As detailed in SS3, _DynaB_ focuses on non-abusive _nesting_ of abusive statements (such as (11) or (12)). However, it only contains very few non-abusive _atomic_ utterances (8)-(10). With about 68%, our linguistically informed classifier has no perfect recall for non-abusive instances either. Since both this classifier and _DynaB_ have in general a high precision (with _DynaB_ having the highest of all classifiers), it makes sense to combine them in order to raise the overall recall. We combine the two classifiers by predicting a non-abusive sentence if one of the two classifiers predicts one. This combination further increased performance. Thus we could outperform _DynaB_ by 8%-points in macro-average F1.

The strong performance of the oracle version of our linguistically informed classifier (77.7% F1) is proof that our 3 linguistic concepts are predictive of abuse on identity groups. The fact that it outperforms our best automatic solution (73.8% F1) suggests that there is still room for improvement.

Table 8 examines the performance of the individual components of our linguistically informed classifier. Since the combined classifier outperforms every individual classifier, we can conclude that the information contained in the components is complementary to a certain degree.

Table 8 also shows that the individual components are effective across the 4 targets which suggests that they are target independent.

## 7 Evaluation on German Dataset

Our final experiments focus on our German dataset. As baselines, we consider supervised classifiers (SS4) trained on the German datasets for abusive lan

\begin{table}
\begin{tabular}{|l|c c c c c||c c c|} \hline  & \multicolumn{2}{c|}{**ABUSIVE**} & \multicolumn{2}{c|}{**OTHER**} & \multicolumn{2}{c|}{**AVERAGE**} \\ \cline{2-9}
**training data** & **Prec** & **Rec** & **F1** & **Prec** & **Rec** & **F1** & **Prec** & **Rec** & **F1 (std)** \\ \hline majority-class classifier & 56.2 & 100.0 & 72.0 & 0.1\({}^{\dagger}\) & 0.0 & 0.0 & 28.1 & 50.0 & 36.0 \\ \hline (Vidgen et al., 2021a)* & 50.0 & 58.1 & 53.7 & 53.3 & 54.4 & 53.8 & 51.6 & 56.2 & 52.8 (\(\pm\)1.1) \\ (Waseen and Hovy, 2016)* & 63.0 & 22.1 & 32.7 & 45.4 & **82.9** & 58.7 & 54.2 & 52.5 & 53.3 (\(\pm\)1.3) \\ (Founta et al., 2018)* & 65.5 & 61.4 & 63.4 & 54.1 & 56.5 & 55.3 & 59.8 & 59.0 & 59.4 (\(\pm\)1.2) \\ (Sap et al., 2020)* & 61.5 & 90.4 & 73.2 & 71.4 & 26.1 & 38.2 & 66.4 & 58.3 & 62.0 (\(\pm\)4.3) \\ PerspectiveAPI & 67.2 & 65.3 & 66.2 & 57.0 & 59.1 & 58.0 & 62.1 & 62.2 & 62.2 \\ ElSherief et al., 2021)* & 70.5 & 57.8 & 63.5 & 55.9 & 67.3 & 61.1 & 63.2 & 62.6 & 62.9 (\(\pm\)3.4) \\ DynaB (Vidgen et al., 2021b)* & 61.1 & **98.0** & 75.3 & **88.4** & 19.6 & 32.1 & **74.8** & 58.8 & 65.8 (\(\pm\)2.2) \\ linguistically informed classifier & 75.2 & 76.0 & 75.6 & 68.7 & 67.8 & 68.2 & 72.0 & 71.9 & 71.9 \\ linguistically informed classifier + DynaS* & **78.1** & 74.9 & **76.5** & 69.3 & 73.0 & **71.1** & 73.7 & **73.9** & **73.8 (\(\pm\)0.5)** \\ \hline lingutically informed classifier (oracle) & 81.3 & 79.1 & 80.2 & 74.0 & 76.5 & 75.2 & 77.6 & 77.8 & 77.7 \\ \hline human classifier (upper bound) & 81.7 & 85.4 & 83.5 & 82.1 & 77.8 & 79.9 & 81.6 & 81.9 & 81.8 \\ \hline \end{tabular}
\end{table}
Table 7: Cross-dataset evaluation on English dataset (\({}^{\dagger}\): _strictly speaking the value for this score is not defined, however, following common practice we considered it \(0\) which enables the computation of the average score_;

\({}^{*}\): _RoBERTa has been used as classifier_).

\begin{table}
\begin{tabular}{|l||c|c|c|c|} \hline
**targets** & **perpetr.** & **non-conf. views** & **aspect** & **combined** \\ \hline gay people & 67.6 & 62.0 & 68.4 & **70.6** \\ Jews & 61.2 & 62.4 & 67.0 & **71.8** \\ Muslims & 58.5 & 62.6 & 66.9 & **72.5** \\ women & 63.0 & 61.2 & 70.0 & **71.4** \\ \hline all & 62.0 & 62.5 & 67.7 & **71.9** \\ \hline \end{tabular}
\end{table}
Table 8: Evaluation of different linguistic components on the different targets (_evaluation measure: F1-score_).

 

guage detection, i.e. _GermEval 2019_Struss et al. (2019) and _GermEval 2021_Risch et al. (2021). Next to a classifier that replicates our linguistically informed classifier on German data, we also test a cross-lingual classifier. Following previous work Zampieri et al. (2020), we fine-tune the multilingual transformer XLM-RoBERTa Conneau et al. (2020) on our English dataset. Since this language model also covers German, the resulting classifier can also be applied on our German dataset.

Table 9 shows the results. The human baseline is notably higher than on the English dataset. German tweets are predominantly posted by native speakers resulting in more fluent language. This makes the manual annotation for the human baseline easier.

With the linguistically informed classifier we outperform both _GermEval_ classifiers. The oracle version is even notably better. These results suggest that the linguistic properties of our 3 components are language independent. The fact that the multilingual transformer performs best indicates that, in general, the type of implicit abuse we address in this work, is valid across different languages.

## 8 Discussion

As the performance of our oracle classifier shows, even a perfect linguistically informed classifier is still below human performance. We could identify two types of ambiguous utterances in our misclassifications that may be responsible: A few sentences are underspecified as to whether they report facts or reflect the author's opinion being biased by their stereotypical views (26)-(27). Only the interpretation as an opinion is perceived abusive.

\begin{tabular}{l|c c|c} \hline
**training data \& classifier** & Prec & FCI [\(std\)] \\ \hline majority-cases classified & 26.1 & 50.0 & 43.3 \\ \hline GermEval-2021* [Facebook] & 65.8 & 55.7 & 60.2 (+3.7) \\ GermEval-2019* [Twitter] & 69.5 & 59.3 & 63.9 (+4.9) \\ Inquistically informed classifier & 70.7 & 70.6 & 70.6 \\ ing. inf. class. 4-GermEval-2019* & 73.4 & 72.6 & 73.0 (+1.6) \\ English-dataset (XLM-RoBERTA) & **81.1** & **80.7** & **80.9** (+0.8) \\ \hline Ing. informed classifier (oracle) & 82.9 & 83.0 & 82.9 \\ \hline human classifier (upper bound) & 87.9 & 87.8 & 87.8 \\ \hline \end{tabular}

Moreover, the prior sentiment of the patient may occasionally depend on the ideology of the reader. For instance, atheists may consider (28) abusive while religious persons would not. Similarly, feminists and non-feminists may have a different perception of (29). It may be debatable that unique class labels as we have assigned to (26)-(29) are adequate. One may argue that without further context these ambiguities cannot be properly resolved.

\begin{table}
\begin{tabular}{l|c c|c} \hline
**Training data \& classifier** & Prec & Rec & FI [\(std\)] \\ \hline majority-cases classified & 26.1 & 50.0 & 43.3 \\ \hline Germany-2021* [TaveBook] & 65.8 & 55.7 & 60.2 (+3.7) \\ GermanyEval-2019* [Twitter] & 69.5 & 59.3 & 63.9 (+4.9) \\ Inquistically informed classifier & 70.7 & 70.6 & 70.6 \\ ing. inf. class. 4-GermEval-2019* & 73.4 & 72.6 & 73.0 (+1.6) \\ English-dataset (XLM-RoBERTa) & **81.1** & **80.7** & **80.9** (+0.8) \\ \hline Ing. informed classifier (oracle) & 82.9 & 83.0 & 82.9 \\ \hline human classifier (upper bound) & 87.9 & 87.8 & 87.8 \\ \hline \end{tabular}
\end{table}
Table 9: Evaluation on German dataset 

Ethical Considerations

Most of our new gold standard data were created with the help of crowdsourcing. All crowdworkers were compensated following the wage recommended by the crowdsourcing platform Prolific (i.e. SS9.60 per hour). Since we were aware of the offensive nature of the data that the crowdworkers had to annotate, we inserted a respective warning in the task advertisement. In order to keep the psychological strain of the crowdworkers at an acceptable level, the data to be annotated was split into bins of 100-200 instances. Furthermore, we allowed each crowdworker to take part in one single task only. We also made it very clear in the task description that we follow a linguistic purpose with our crowdsourcing tasks and the opinion expressed in the sentences to be annotated in no way reflects the opinion of (us) researchers designing the tasks.

One of our crowdsourcing tasks included inventing sentences in which a group of people is framed as a perpetrator (SS5.2). Since we did not want crowdworkers to invent any anti-Semitic, homophobic, Islamophobic or misogynist content, we introduced the name of a fictitious people which the crowdworkers were to use in their sentences. We also made sure that the particular name did not have any obvious phonetic resemblance to existing identity groups. Although the resulting sentences being invented are not directed against any existing identity groups they may still be considered abusive. However, we think that this is justifiable in this particular context since we are not aware of any existing dataset that contains a similar content (i.e. a focused dataset for learning perpetrator-evoking verbs) that we could have used for our experiments. In principle, creating morally disputable content as part of research is not unusual. Both in plagiarism detection Potthast et al. (2010), deception Ott et al. (2011) and, quite recently, abusive language detection itself Vidgen et al. (2021b; Wiegand et al., 2021a) a procedure similar to ours was pursued.

One substantial part of the data we are going to make publicly available as part of this research will include sentences extracted from Twitter. In order to protect the privacy rights of the authors of the tweets and individuals mentioned in them, we anonymized our data by discarding mentions of usernames. The public release of a limited number of tweets as in the range of our dataset is also in accordance with the regulations of Twitter.

A datasheet describing our novel dataset of labeled sentences for the task of detecting implicitly abusive remarks about identity groups (both English and German version) following the specification of Gebru et al. (2018) was added to the supplementary material.

Our current data focuses on the four identity groups _Jews_, _Muslims_ and _gay people_ and _women_. This choice was mainly motivated by the fact that these groups are among the most abused identity groups on social media. As a consequence, it was also possible to obtain a reasonable amount of data (even with our restrictive measures to ensure less biased datasets). Moreover, these identity groups are well represented in existing datasets. This allows us to compare our proposed classifier against baseline classifiers trained on these existing datasets. We acknowledge that abusive language on the web is also directed against other identity groups. We leave their automatic detection to future work. However, our study suggests that abusive language that targets these other identity groups will follow the same language patterns as the instances of abusive language examined in this paper.

## 11 Acknowlegdgements

The authors would like to thank Sybille Sornig for manually annotating parts of the data on which our descriptive statistics in Section 3 are based. We are also grateful to Ines Rehbein for feedback on earlier drafts of this paper.

## References

* A. Akbik, T. Bergmann, D. Blythe, K. Rasul, M. Schweter, and R. Vollgraf (2019)FLAIR: An easy-to-use framework for state-of-the-art NLP. In Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL (HLT/NAACL), Minneapolis, MN, USA, pp. 54-59. External Links: Link Cited by: SS1.
* A. Arango, J. Perez, and B. Poblete (2019)Hate Speech Detection is Not as Easy as You May Think: A Closer Look at Model Validation. In Proceedings of the ACM Special Interest Group on Information Retrieval (SIGIR), Paris, France, pp. 45-53. External Links: ISBN 978-1-467-3861-3 Cited by: SS1.
* C. F. Baker, C. J. Fillmore, and J. B. Lowe (1998)The Berkeley FrameNet Project. In Proceedings of the International Conference on Computational Linguistics and Annual Meeting of the Association for Computational Linguistics (COLING/ACL), Montreal, Quebec, Canada, pp. 86-90. External Links: Document Cited by: SS1.

 

Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa-Anke, and Leonardo Neves. 2020. TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification. In _Findings of Association for Computational Linguistics: EMNLP 2020_, 1644-1650, Online.
* Breitfeller et al. (2019) Luke M. Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019. Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1664-1674, Hong Kong, China.
* Burnap and Williams (2016) Pete Burnap and Matthew L. Williams. 2016. Us and them: identifying cyber hate on Twitter across multiple protected characteristics. _EPJ Data Science_, 5(1):11.
* Chan et al. (2020) Branden Chan, Stefan Schweter, and Timo Moller. 2020. German's Next Language Model. In _Proceedings of the International Conference on Computational Linguistics (COLING)_, pages 6788-6796, Barcelona, Spain (Online).
* Chiril et al. (2020) Patricia Chiril, Veronique Moriceau, Farah Benamara, Alda Mari, Gloria Origgi, and Marlene Coulomb-Gully. 2020. He said "who's gonna take care of your children when you are at ACL?": Reported sexist acts are not sexist. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 4055-4066, Online.
* Choi and Wiebe (2014) Yoonjung Choi and Janyce Wiebe. 2014. +/-EffectWordNet: Sense-level Lexicon Acquisition for Opinion Inference. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1181-1191, Doha, Qatar.
* Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning of Scale. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 8440-8451, Online.
* Das and Smith (2011) Dipanjan Das and Noah A. Smith. 2011. Semi-Supervised Frame-Semantic Parsing for Unknown Predicates. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 1435-1444, Portland, OR, USA.
* Dixon et al. (2018) Lucas Dixon, John Li, Jeffrey S. Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and Mitigating Unintended Bias in Text Classification. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES)_, pages 67-73, New Orleans, LA, USA.
* ElSherief et al. (2021) Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021. Latent Hatter: A Benchmark for Understanding Implicit Hate Speech. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 345-363, Online and Punta Cana, Dominican Republic.
* Fortuna and Nunes (2018) Paula Fortuna and Sergio Nunes. 2018. A Survey on Automatic Detection of Hate Speech in Text. _ACM Computing Surveys_, 51(4):85:1-85:30.
* Founta et al. (2018) Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior. In _Proceedings of the International AAAI Conference on Weblogs and Social Media (ICWSM)_, Stanford, CA, USA.
* Friedrich et al. (2016) Annemarie Friedrich, Alexis Palmer, and Manfred Pinkal. 2016. Situation entity types: automatic classification of clause-level aspect. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 1757-1768, Berlin, Germany.
* Friedrich and Pinkal (2015) Annemarie Friedrich and Manfred Pinkal. 2015. Automatic recognition of habituals: a three-way classification of clausal aspect. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2471-2481, Lisbon, Portugal.
* Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sinao Chen, Prodeep Dasiqi, Theberu Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating Models' Local Decision Boundaries via Contrast Sets. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1307-1320, Online.
* Gebru et al. (2018) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daume III, and Kate Crawford. 2018. Datasheets for Datasets. In _Proceedings of the Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT ML)_, Stockholm, Sweden.
* Gildea and Jurafsky (2002) Daniel Gildea and Daniel Jurafsky. 2002. Automatic Labeling of Semantic Roles. _Computational Linguistics_, 28(3):245-288.
* Han and Tsvetkov (2020) Xiauchuang Han and Yulia Tsvetkov. 2020. Fortifying Toxic Speech Detection Against Veiled Toxicity. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7732-7739, Online.
* Hong et al. (2011) Lichan Hong, Gregorio Convertino, and Ed Chi. 2011. Language matters in Twitter: A large scale study. In _Proceedings of the International AAAI Conference 

on Weblogs and Social Media (ICWSM)_, Barcelona, Catalonia, Spain.
* Ide et al. [2008] Nancy Ide, Collin Baker, Christiane Fellbaum, Charles Fillmore, and Rebecca Passonneau. 2008. MASC: the Manually Annotated Sub-Corpus of American English. In _Proceedings of the Conference on Language Resources and Evaluation (LREC)_, pages 2455-2461, Marrakech, Morocco.
* Landis and Koch [1977] J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. _Biometrics_, 33(1):159-174.
* Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. _arXiv preprint arXiv:1907.11692_.
* Mendelsohn et al. [2021] Julia Mendelsohn, David Jurgens, and Ceren Budak. 2021. Modeling Framing in Immigration Discourse on Social Media. In _Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL (HLT/NAACL)_, 2219-2263, Online.
* Mikolov et al. [2018] Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advanced in Pre-Training Distributed Word Representations. In _Proceedings of the Conference on Language Resources and Evaluation (LREC)_, pages 52-55, Miyazaki, Japan.
* Mintz et al. [2009] Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant Supervision for Relation Extraction without Labeled Data. In _Proceedings of the Joint Conference of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing (ACLI/IJCNLP)_, pages 1003-1011, Singapore.
* Ott et al. [2011] Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T. Hancock. 2011. Finding Deceptive Opinion Spam by Any Stretch of the Imagination. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 309-319, Portland, OR, USA.
* Poletto et al. [2021] Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. 2021. Resources and benchmark corpora for hate speech detection: a systematic review. _Language Resources and Evaluation_, 55:477-523.
* Potthast et al. [2010] Martin Potthast, Benno Stein, Alberto Barron-Cedeno, and Paolo Rosso. 2010. An Evaluation Framework for Plagiriarm Detection. In _Proceedings of the International Conference on Computational Linguistics (COLING)_, pages 997-1005, Beijing, China.
* Rashkin et al. [2016] Hannah Rashkin, Sameer Singh, and Yejin Choi. 2016. Connotation Frames: A Data-Driven Investigation. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 311-321, Berlin, Germany.
* Risch et al. [2021] Julian Risch, Anke Stoll, Lena Wilms, and Michael Wiegand. 2021. Overview of the GermEval 2021 Shared Task on the Identification Toxic, Engaging, and Fact-Claiming Comments. In _Proceedings of the GermanEval 2021 Shared Task on the Identification of Toxic, Engaging, and Fact-Claiming Comments_, pages 1-12, Online.
* Sap et al. [2020] Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. SOCIAL BIAS FRAMES: Reasoning about Social and Power Implications of Language. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 5477-5490, Online.
* Struss et al. [2019] Julia Maria Struss, Melanie Siegel, Josef Ruppenhofer, Michael Wiegand, and Manfred Klenner. 2019. Overview of GermEval Task 2, 2019 Shared Task on the Identification of Offensive Language. In _Proceedings of the GermEval Workshop_, pages 352-363, Erlangen, Germany.
* Suwayadipita et al. [2017] Swabha Suwayadipita, Sam Thomson, Chris Dyer, and Noah A. Smith. 2017. Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic Scaffold. _arXiv preprint arXiv:1706.09528_.
* van Achen et al. [2018] Betty van Achen, Julian Risch, Ralf Krestel, and Alexander Loser. 2018. Challenges for Toxic Comment Classification: An In-Depth Error Analysis. In _Proceedings of the Workshop on Abusive Language Online (ALB)_, pages 33-42, Brussels, Belgium.
* Vidgen and Derczynski [2020] Bertie Vidgen and Leon Derczynski. 2020. Directions in Abusive Language Training Data. _PLoS One_, 15(12).
* Vidgen et al. [2021a] Bertie Vidgen, Dong Nguyen, Helen Margetts, Patricia Rossini, and Rebekah Tromble. 2021a. Introducing CAD: the Contextual Abuse Dataset. In _Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL (HLT/NAACL)_, pages 2289-2303, Online.
* Vidgen et al. [2021b] Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021b. Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 1667-1682, Online.
* Waseem et al. [2017] Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding Abuse: A Typology of Abusive Language Detection Subtasks. In _Proceedings of the ACL-Workshop on Abusive Language Online_, pages 78-84, Vancouver, BC, Canada.
* Student Research Workshop_, pages 88-93, San Diego, CA, USA
 

Michael Wiegand, Maja Geulis, and Josef Ruppenhofer. 2021a. Implicitly Abusive Comparisons - A New Dataset and Linguistic Analysis. In _Proceedings of the Conference on European Chapter of the Association for Computational Linguistics (EACL)_, pages 358-368, Online.
* What does it actually look like and why are we not getting there? In _Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL (HLT/NAACL)_, pages 576-587, Online.
* Wiegand et al. [2019] Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of Abusive Language: the Problem of Biased Datasets. In _Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL (HLT/NAACL)_, pages 602-608, Minneapolis, MN, USA.
* A Feature-Based Approach. In _Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL (HLT/NAACL)_, pages 1046-1056, New Orleans, LA, USA.
* Wilson et al. [2005] Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing Contextual Polarity in Phraselevel Sentiment Analysis. In _Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing (HLT/EMNLP)_, pages 347-354, Vancouver, BC, Canada.
* Zampieri et al. [2020] Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Cagn Coltekin. 2020. SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020). In _Proceedings of SemEval_, page 1425-1447, Online.
* Zhong et al. [2016] Haoti Zhong, Hao Li, Anna Cinzia Squicciarini, Sarah Michele Rajtmajer, Christopher Griffin, David J. Miller, and Cornelia Caragea. 2016. Content-Driven Detection of Cyberbullying on the Instagram Social Network. In _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_, pages 3952-3958, New York City, NY, USA.
* Zhou and Zafarani [2020] Xinyi Zhou and Reza Zafarani. 2020. A Survey of Fake News: Fundamental Theories, Detection Methods, and Opportunities. _ACM Computing Surveys_, 53(5):109:1-109:40.
* Zhou et al. [2021] Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2021. Challenges in Automated Debiasing for Toxic Language Detection. In _Proceedings of the Conference on European Chapter of the Association for Computational Linguistics (EACL)_, pages 3143-3155, Online.