SlowGuess/NAACL_2022_OCR huggingface dataset:
As a baseline, we run a very fine-grained semantic-role labeling system based on FrameNet (Baker et al., 1998) on our data. We chose open sesame (Swayamdipta et al., 2017) which is the most recent publicly available tool for semantic-role labeling based on FrameNet. Due to its fine-grained inventory, there are frame elements (this is the term for semantic roles in FrameNet) which semantically correspond to our concept of perpetrators. More precisely, we considered text spans as perpetrators if they are predicted to be one of the following frame elements: Abuser, Assailant, Counter Actor, Destroyer, Invader, Killer, Manipulator, Offender, Perpetrator and Wrongdoer.
Table 4 shows the performance of the different classifiers to detect mentions of perpetrators in our
majority-class	FrameNet	lexicon-based classifier
F1	33.9	60.1	70.5
Table 4: Evaluation of perpetrator classification.
English dataset (§3). Our lexicon-based classifier outperforms FrameNet, which is known to have a limited lexical coverage (Das and Smith, 2011).
5.3 Component 3: Non-Conformist Views
The Task. For our third component task, we consider the sentiment of the agent towards the patient (as conveyed by the main verb in the sentence) in combination with the sentiment expected a priori towards the patient. (The agent is always the mention of the identity group.) This is illustrated in Table 5. We observe a systematic relationship between abusive language and fine-grained sentiment: If the sentiment of the identity group (i.e. the agent) towards the patient is opposite to the prior sentiment of the patient, then this utterance depicts the identity group as having a non-conformist view.[11] Such views are perceived as abusive utterances: If someone attributes non-conformist views to some identity group, then, one often intends to stigmatize this group as not belonging to their own community. This phenomenon is referred to as othering (Burnap and Williams, 2016).
The Method. In order to detect the above pattern indicating non-conformist views, we need the output of two modules: the first determines the prior sentiment of the patient (i.e. the phrase representing the logical object); the second determines the sentiment of the agent towards the patient. The prior sentiment of the patient can be easily detected by running a sentiment text classifier on that phrase. For this, we use TweetEval (Barbieri et al., 2020).
The difficult part is to detect the sentiment of the agent towards the patient. Sentiment text classifiers are unable to determine such fine-grained sentiment information. They capture the general sentiment of a given text which may be different. For instance, (25) conveys a positive sentiment of Muslims towards violence, while the sentiment of the sentence is generally considered negative.
(25) [Muslims]agent glorify [violence]patient.
In Table 5, we only distinguish between positive and negative sentiment. There is no neutral sentiment. In the context of these sentiment patterns, we found that neutral sentiment follows the same pattern as positive sentiment. Conflating positive and neutral sentiment facilitated automatic processing.
$^{10}$ https://dl.fbaipublicfiles.com/  
fasttext/vectors-english/crawl-300d-2M.  
vec.zip
fine-grained sentiment
example sentences (2 sentences for each type; all sentences are non-episodic)	agent to patient	patient	abuse
[ Jews]agent long for [a safe Israel]patient.
[Women]agent abhor [violence]patient.	[ Muslims]agent grieve for [their brothers]patient.
[Jews]agent suffer from [ethnic cleansing]patient.	positive
negative	positive
negative
[Lesbians]agent pray to [Satan]patient.
[Muslims]agent dislike [peace]patient.	[ Muslims]agent revert to [stoning victims]patient.
[Lesbians]agent disrespect [God's plan]patient.	positive
negative	negative
positive	✓
✓
Table 5: Implicitly abusive language and fine-grained sentiment; non-conformist views are sentences in which sentiment of agent to patient and sentiment of patient disagree; non-conformist views coincide with abuse.
majority-class	frames	effectWN	novel lexicon
F1	41.9	64.1	64.8	71.6
Table 6: Evaluation of fine-grained sentiment analysis.
Instead of a text classifier, we seek a lexicon that specifies for any negative polar verb (out of context) whether it conveys a positive sentiment of the agent towards the patient (e.g. glorify, long (for), pray (to)) or a negative sentiment towards it (e.g. abhor, dislike, suffer). The only lexicons with such information are EffectWordNet (effectWN) (Choi and Wiebe, 2014) and the connotation-frames lexicon (frames) (Rashkin et al., 2016). Unfortunately, both resources only cover about  $40\%$  of the verbs in our dataset. We also determined a significant level of noise in these resources (as detailed in the supplementary notes). Therefore, we decided to create a novel lexicon with that information. It should cover all possible negative verbs. We first had crowdworkers annotate for some (seed) negative verbs the sentiment of the agent towards the patient. We chose the 500 verbs we already used in §5.2. The majority of the crowdworkers' judgements represent our gold standard annotation. On these annotated 500 verbs we trained a logistic regression classifier. As features, we represented each verb by its word embedding from Common Crawl. (Using such a representation is common practice for this task (Rashkin et al., 2016).) The resulting classifier was run on the same large set of 1,700 negative verbs we used in §5.2. For each verb the classifier predicts the sentiment of the agent towards the patient. The result is our novel lexicon. Since we have also manually annotated the sentiment of the agent towards the patient for each verb in the sentences of our labeled dataset (\$3), we can evaluate this lexicon against our labeled dataset.
Table 6 evaluates the different lexicons to determine the sentiment of the agent towards the patient on our novel dataset. The table shows that our novel bootstrapped lexicon produces a notable improvement over the existing resources.
Figure 1: Linguistically informed classifier.
5.4 How the Final Classifier is Built
Figure 1 shows how the component tasks introduced in §5.1-5.3 are combined to produce our linguistically informed classifier: We consider those sentences as abusive that are non-episodic and which either depict the identity group as perpetrator or attribute non-conformist views to it. We use the best-performing component classifiers as determined by our previous evaluation (§5.1-§5.3).
We also experimented with a supervised classifier that uses the predictions from our component classifiers as features. However, since the classification performance was on a par with our proposed (rule-based) classifier (Figure 1), we decided in favor of the latter classifier. It has a clear advantage over supervised classification in that it does not require any labeled training data to combine the predictions of the component classifiers.
6 Evaluation on English Dataset
We evaluate the linguistically informed classifier (Figure 1) on our new English dataset (for implicit abuse) against other classifiers trained on existing datasets. We carry out a cross-dataset evaluation: None of the classifiers, including our linguistically informed classifier, has been trained on our English dataset. Given the recent criticism against within-dataset evaluation (Arango et al., 2019; Wiegand et al., 2019) in which high performance is often the result of overfitting, this is a fairly unbiased set up.
As datasets for training supervised baselines, we chose those that focus on implicit abuse (ElSherief et al., 2021) or abuse towards identity groups
ABUSIVE	OTHER	AVERAGE
training data	Prec	Rec	F1	Prec	Rec	F1	Prec	Rec	F1 (std)
majority-class classifier	56.2	100.0	72.0	0.0†	0.0	0.0†	28.1	50.0	36.0
(Vidgen et al., 2021a)*	50.0	58.1	53.7	53.3	54.4	53.8	51.6	56.2	52.8 (±1.1)
(Waseem and Hovy, 2016)*	63.0	22.1	32.7	45.4	82.9	58.7	54.2	52.5	53.3 (±1.3)
(Founta et al., 2018)*	65.5	61.4	63.4	54.1	56.5	55.3	59.8	59.0	59.4 (±2.1)
(Sap et al., 2020)*	61.5	90.4	73.2	71.4	26.1	38.2	66.4	58.3	62.0 (±4.3)
PerspectiveAPI	67.2	65.3	66.2	57.0	59.1	58.0	62.1	62.2	62.2
(ElSherief et al., 2021)*	70.5	57.8	63.5	55.9	67.3	61.1	63.2	62.6	62.9 (±3.4)
DynaB (Vidgen et al., 2021b)*	61.1	98.0	75.3	88.4	19.6	32.1	74.8	58.8	65.8 (±2.2)
linguistically informed classifier	75.2	76.0	75.6	68.7	67.8	68.2	72.0	71.9	71.9
linguistically informed classifier + DynaB*	78.1	74.9	76.5	69.3	73.0	71.1	73.7	73.9	73.8 (±0.5)
linguistically informed classifier (oracle)	81.3	79.1	80.2	74.0	76.5	75.2	77.6	77.8	77.7
human classifier (upper bound)	81.7	85.4	83.5	82.1	77.8	79.9	81.6	81.9	81.8
(Waseem and Hovy, 2016; Sap et al., 2020; Vidgen et al., 2021a,b). We also included Founta et al. (2018) as a more general dataset sampled from Twitter. For each dataset, we fine-tune the pretrained RoBERTa model (§4) on the training partition of the respective dataset. As a further baseline, we run the state-of-the-art classifier for abusive language detection PerspectiveAPI<sup>12</sup> on our dataset.
We also include an oracle version of our linguistically informed classifier, that combines the gold standard annotation for the component tasks ( $\S 5.1-\S 5.3$ ) rather than the outputs of the respective classifiers. This can be considered the upper bound for the linguistically informed classifier.
Finally, we also consider a human classifier as a general upper bound. We randomly sampled the judgment of one individual annotator from the crowdsourced gold-standard annotation for the detection of abusive language. This individual judgement may notably differ from the gold standard label which is the majority label of 5 annotators.
Table 7 displays the results. The classifiers trained on existing datasets do not perform well on our new dataset. The best classifier among them is the one trained on the DynaB-dataset. For DynaB (unlike the other datasets), special attention was paid to the inclusion of non-abusive instances (\$3). Still, our linguistically informed classifier is more effective. DynaB suffers from the identity-group bias (\$1): its recall for non-abusive instances is only at  $20\%$ . As detailed in  $\S 3$ , DynaB focuses on non-abusive nesting of abusive statements (such as (11) or (12)). However, it only contains very few non-abusive atomic utterances (8)-(10). With about  $68\%$ , our linguistically informed classifier has no perfect recall for non-abusive instances ei
Table 7: Cross-dataset evaluation on English dataset ( $\dagger$ : strictly speaking the value for this score is not defined, however, following common practice we considered it 0 which enables the computation of the average score;  $*$ : RoBERTa has been used as classifier).
targets	perpetr.	non-conf. views	aspect	combined
gay people	67.6	62.0	68.4	70.6
Jews	61.2	62.4	67.0	71.8
Muslims	58.5	62.6	66.9	72.5
women	63.0	61.2	70.0	71.4
all	62.0	62.5	67.7	71.9
Table 8: Evaluation of different linguistic components on the different targets (evaluation measure: F1-score).
ther. Since both this classifier and DynaB have in general a high precision (with DynaB having the highest of all classifiers), it makes sense to combine them in order to raise the overall recall. We combine the two classifiers by predicting a non-abusive sentence if one of the two classifiers predicts one. This combination further increased performance. Thus we could outperform DynaB by  $8\%$  -points in macro-average F1.
The strong performance of the oracle version of our linguistically informed classifier (77.7% F1) is proof that our 3 linguistic concepts are predictive of abuse on identity groups. The fact that it outperforms our best automatic solution (73.8% F1) suggests that there is still room for improvement.
Table 8 examines the performance of the individual components of our linguistically informed classifier. Since the combined classifier outperforms every individual classifier, we can conclude that the information contained in the components is complementary to a certain degree.
Table 8 also shows that the individual components are effective across the 4 targets which suggests that they are target independent.
7 Evaluation on German Dataset
Our final experiments focus on our German dataset. As baselines, we consider supervised classifiers (§4) trained on the German datasets for abusive lan
12www.perspectiveapi.com
training data & classifier	Prec	Rec	F1 (std)
majority-class classifier	26.1	50.0	34.3
GermEval-2021* [Facebook]	65.8	55.7	60.2 (±3.7)
GermEval-2019* [Twitter]	69.5	59.3	63.9 (±4.9)
linguistically informed classifier	70.7	70.6	70.6
ling. inf. class.+GermEval-2019*	73.4	72.6	73.0 (±1.6)
English-dataset (XLM-RoBERTa)	81.1	80.7	80.9 (±0.8)
ling. informed classifier (oracle)	82.9	83.0	82.9
human classifier (upper bound)	87.9	87.8	87.8
Table 9: Evaluation on German dataset (*: using best transformer from Chan et al. (2020)).
guage detection, i.e. GermEval 2019 (Struß et al., 2019) and GermEval 2021 (Risch et al., 2021). Next to a classifier that replicates our linguistically informed classifier on German data, we also test a cross-lingual classifier. Following previous work (Zampieri et al., 2020), we fine-tune the multilingual transformer XLM-RoBERTa (Conneau et al., 2020) on our English dataset. Since this language model also covers German, the resulting classifier can also be applied on our German dataset.
Table 9 shows the results. The human baseline is notably higher than on the English dataset. German tweets are predominantly posted by native speakers resulting in more fluent language. This makes the manual annotation for the human baseline easier.
With the linguistically informed classifier we outperform both GermEval classifiers. The oracle version is even notably better. These results suggest that the linguistic properties of our 3 components are language independent. The fact that the multilingual transformer performs best indicates that, in general, the type of implicit abuse we address in this work, is valid across different languages.
As the performance of our oracle classifier shows, even a perfect linguistically informed classifier is still below human performance. We could identify two types of ambiguous utterances in our misclassifications that may be responsible: A few sentences are underspecified as to whether they report facts or reflect the author's opinion being biased by their stereotypical views (26)-(27). Only the interpretation as an opinion is perceived abusive.
(26) Women overuse makeup.  
(27) Muslims suppress Christian life in Iraq.
Moreover, the prior sentiment of the patient may occasionally depend on the ideology of the reader. For instance, atheists may consider (28) abusive while religious persons would not. Similarly, fem
inists and non-feminists may have a different perception of (29). It may be debatable that unique class labels as we have assigned to (26)-(29) are adequate. One may argue that without further context these ambiguities cannot be properly resolved.
(28) Muslims surrender to God's will.  
(29) Women unmake patriarchy.
A general limitation of our approach is that our data exclusively originate from Twitter. Therefore, we cannot rule out that certain results reported in this paper only hold for data from this platform. Given, however, that we made sure that the data from that platform that we use are not affected by any obvious user or topic biases (§3) and given that our proposed method works across 4 different identity groups and 2 different languages, we estimate the likelihood that this limitation has significantly affected our results to be very low.
Another limitation of our work is the focus on atomic sentences in which the identity group is the agent of some negative verb. As we have motivated in §3, our exploratory data analysis suggests that this is the most frequent surface realization of such abusive remarks. However, implicitly abusive remarks targeting identity groups may also be expressed in other ways, such as (30) where the identity group is not an agent of some negative polar verb.
(30) Once again, we find Jews and money money money.
While constructions such as (30) are possible, we are unaware of any sampling method that would enable us to capture such constructions. We expect these constructions also to be more infrequent than the more prototypical atomic sentences. Therefore, we leave it to future work to address them.
We presented a new focused dataset for implicitly abusive remarks among negative polar utterances on identity groups. We identified 3 linguistic properties which allow us to effectively detect such abusive remarks across different identity groups and across different languages. The utterances have to be non-episodic and the identity group is either depicted as a perpetrator or attributed to a non-conformist view. We are also able to notably outperform classifiers trained on previous datasets.
Most of our new gold standard data were created with the help of crowdsourcing. All crowdworkers were compensated following the wage recommended by the crowdsourcing platform Prolific (i.e. \(9.60 per hour). Since we were aware of the offensive nature of the data that the crowdworkers had to annotate, we inserted a respective warning in the task advertisement. In order to keep the psychological strain of the crowdworkers at an acceptable level, the data to be annotated was split into bins of 100-200 instances. Furthermore, we allowed each crowdworker to take part in one single task only. We also made it very clear in the task description that we follow a linguistic purpose with our crowdsourcing tasks and the opinion expressed in the sentences to be annotated in no way reflects the opinion of (us) researchers designing the tasks.
One of our crowdsourcing tasks included inventing sentences in which a group of people is framed as a perpetrator (§5.2). Since we did not want crowdworkers to invent any anti-Semitic, homophobic, Islamophobic or misogynist content, we introduced the name of a fictitious people which the crowdworkers were to use in their sentences. We also made sure that the particular name did not have any obvious phonetic resemblance to existing identity groups. Although the resulting sentences being invented are not directed against any existing identity groups they may still be considered abusive. However, we think that this is justifiable in this particular context since we are not aware of any existing dataset that contains a similar content (i.e. a focused dataset for learning perpetrator-evoking verbs) that we could have used for our experiments. In principle, creating morally disputable content as part of research is not unusual. Both in plagiarism detection (Potthast et al., 2010), deception detection (Ott et al., 2011) and, quite recently, abusive language detection itself (Vidgen et al., 2021b; Wiegand et al., 2021a) a procedure similar to ours was pursued.
One substantial part of the data we are going to make publicly available as part of this research will include sentences extracted from Twitter. In order to protect the privacy rights of the authors of the tweets and individuals mentioned in them, we anonymized our data by discarding mentions of usernames. The public release of a limited number of tweets as in the range of our dataset is also in accordance with the regulations of Twitter.
A datasheet describing our novel dataset of labeled sentences for the task of detecting implicitly abusive remarks about identity groups (both English and German version) following the specification of Gebru et al. (2018) was added to the supplementary material.
Our current data focuses on the four identity groups Jews, Muslims and gay people and women. This choice was mainly motivated by the fact that these groups are among the most abused identity groups on social media. As a consequence, it was also possible to obtain a reasonable amount of data (even with our restrictive measures to ensure less biased datasets). Moreover, these identity groups are well represented in existing datasets. This allows us to compare our proposed classifier against baseline classifiers trained on these existing datasets. We acknowledge that abusive language on the web is also directed against other identity groups. We leave their automatic detection to future work. However, our study suggests that abusive language that targets these other identity groups will follow the same language patterns as the instances of abusive language examined in this paper.
The authors would like to thank Sybille Sornig for manually annotating parts of the data on which our descriptive statistics in Section 3 are based. We are also grateful to Ines Rehbein for feedback on earlier drafts of this paper.
Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An easy-to-use framework for state-of-the-art NLP. In Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL (HLT/NAACL), pages 54-59, Minneapolis, MN, USA.
Aymé Arango, Jorge Pérez, and Barbara Poblete. 2019. Hate Speech Detection is Not as Easy as You May Think: A Closer Look at Model Validation. In Proceedings of the ACM Special Interest Group on Information Retrieval (SIGIR), pages 45-53, Paris, France.
Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet Project. In Proceedings of the International Conference on Computational Linguistics and Annual Meeting of the Association for Computational Linguistics (COLING/ACL), pages 86-90, Montréal, Quebec, Canada.
Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa-Anke, and Leonardo Neves. 2020. TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification. In Findings of Association for Computational Linguistics: EMNLP 2020, 1644-1650, Online.
Luke M. Breitbart, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019. Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1664-1674, Hong Kong, China.
Pete Burnap and Matthew L. Williams. 2016. Us and them: identifying cyber hate on Twitter across multiple protected characteristics. *EPJ Data Science*, 5(1):11.
Branden Chan, Stefan Schweter, and Timo Möller. 2020. German's Next Language Model. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 6788-6796, Barcelona, Spain (Online).
Patricia Chiril, Véronique Moriceau, Farah Benamara, Alda Mari, Gloria Origgi, and Marlene Coulomb-Gully. 2020. He said "who's gonna take care of your children when you are at ACL?": Reported sexist acts are not sexist. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 4055-4066, Online.
Yoonjung Choi and Janyce Wiebe. 2014. EffectWordNet: Sense-level Lexicon Acquisition for Opinion Inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1181-1191, Doha, Qatar.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 8440-8451, Online.
Dipanjan Das and Noah A. Smith. 2011. Semi-Supervised Frame-Semantic Parsing for Unknown Predicates. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 1435-1444, Portland, OR, USA.
Lucas Dixon, John Li, Jeffrey S. Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and Mitigating Unintended Bias in Text Classification. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES), pages 67-73, New Orleans, LA, USA.
Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021. Latent Hatred: A Benchmark for Understanding Implicit Hate Speech. In