Venue: NAACL 2019

# Detection of Abusive Language: the Problem of Biased Datasets

 Michael Wiegand\({}^{*0}\), Josef Ruppenhofer\({}^{\dagger \ddagger}\), Thomas Kleinbauer\({}^{*}\)

\({}^{*}\)Spoken Language Systems, Saarland University, Saarbrucken, Germany

\({}^{\dagger}\)Leibniz ScienceCampus, Heidelberg/Mannheim, Germany

\({}^{\ddagger}\)Institute for German Language, Mannheim, Germany

michael.wiegand@lsv.uni-saarland.de

ruppenhofer@ids-mannheim.de

thomas.kleinbauer@lsv.uni-saarland.de

###### Abstract

We discuss the impact of data bias on abusive language detection. We show that classification scores on popular datasets reported in previous work are much lower under realistic settings in which this bias is reduced. Such biases are most notably observed on datasets that are created by focused sampling instead of random sampling. Datasets with a higher proportion of implicit abuse are more affected than datasets with a lower proportion.

1
Footnote 1: [http://helawdictionary.org/](http://helawdictionary.org/)

2
Footnote 2: [https://www.stat.ucsdavis.edu/~klinbauer/](https://www.stat.ucsdavis.edu/~klinbauer/)

3
Footnote 4: [https://github.com/journals/pierai/](https://github.com/journals/pierai/)

## 1 Introduction

Abusive or offensive language is commonly defined as hurtful, derogatory or obscene utterances made by one person to another person.1 Examples are (1)-(3). In the literature, closely related terms include _hate speech_Waseem and Hovy (2016) or _cyber bullying_Zhong et al. (2016). While there may be nuanced differences in meaning, they are all compatible with the general definition above.

Footnote 1: [http://helawdictionary.org/](http://helawdictionary.org/)

1. stop editing this, you **dumbass**.
2. Just want to slap the **stupid** out of these **binbos!!!**
3. Go lick a pig you arab muslim piece of **scum**.

Due to the rise of user-generated web content, in particular on social media networks, the amount of abusive language is also steadily growing. NLP methods are required to focus human review efforts towards the most relevant microposts.

In this paper, we examine the issue of data bias. For the creation of manually annotated datasets, randomly sampling microposts from large social media platforms typically results in a too small proportion of abusive comments Wulczyn et al. (2017); Founta et al. (2018). Therefore, more focused sampling strategies have to be applied which cause biases in the resulting datasets. We show what implications this has on classifiers trained on these datasets: Previous evaluations reported high classification performance on datasets with difficult cases of abusive language, e.g. implicit abuse (SS2). Contrarily, we find that the high classification scores are likely to be the result of modeling the bias in those datasets.

Although we will explicitly name shortcomings of existing individual datasets, our paper is not intended as a reproach of those who created them. On the contrary, we acknowledge the great efforts the researchers have taken to provide these resources. Without them, much existing research would not have been possible. However, we also noticed a lack of awareness of the special properties of those datasets among researchers using them. As we will illustrate with specific examples, this may result in unforeseen results of particular classification approaches.

## 2 Explicit and Implicit Abuse

One major distinction that has been proposed in the literature is the division into **explicitly** and **implicitly** abusive language Waseem et al. (2017). The former are microposts that employ some abusive words (1)-(3) (e.g. _dumbass_ or _scum_), while the latter represents the more difficult case in which the abusive nature is conveyed by other means, such as sarcasm, jokes, and particularly the usage of negative stereotypes etc. (4)-(5).

1. i haven had an intelligent conversation with a woman.
2. Jews don't marry children. Muslims do. All the time.

To determine which of the datasets that we consider in this work contain which type of abusive language, we proceeded as follows. On the set of abusive microposts of each dataset, we computed the proportion of microposts that include at least

one abusive word according to the lexicon of abusive words from Wiegand et al. (2018). Datasets with a high proportion of abusive words typically contain a high amount of explicitly abusive microposts, whereas datasets with a low proportion contain a higher amount of implicitly abusive language. The resulting figures, of course, are only a lower bound estimate for explicit language abuse. There will also be microposts containing abusive words that are missing from the lexicon. However, after manual inspection of a sample of microposts, we are fairly confident that this does not significantly change the _relative_ order of datasets when ranked according to their degree of explicit language abuse.

## 3 Datasets and Their Properties

Due to the limited space of this paper, we restrict our discussion to frequently cited (publicly available) datasets and datasets from shared tasks. Substantial interannotation agreement has also been reported with these datasets.

As we focus on the detection of abusive language in general, for those datasets containing more fine-grained class inventories describing subtypes of abusive language2, we conflate the categories to one general category. As a result, there are always only two categories: _abuse_ and _nobuse_. This merging removes differences between the individual annotation schemes that would otherwise impede a meaningful comparison.

Footnote 2: For example, Waseem and Hovy (2016) distinguish between _sexism_ and _racism_.

Table 1 shows a brief summary of the different datasets. Among the properties, we list the performance of a text classifier in the right-most column. Since in previous work performance on the different datasets was reported on the basis of different types of classifiers and also varying evaluation metrics, we ran the same classifier on all datasets in order to ensure a meaningful comparison. We chose FastText, which is an efficient supervised classifier known to produce state-of-the-art performance on many text classification tasks3Joulin et al. (2017) and whose results are easy to reproduce. Performance is evaluated in a 10-fold crossvalidation setting using the macro-average F1-score.

Footnote 3: More involved classifiers achieve better performance, however, the relative differences between the datasets remain.

Table 1 also describes the way the datasets were sampled. Not a single dataset has been produced by _pure_ random sampling. This would always result in tiny proportions of abusive language. For example, Founta et al. (2018) estimate that on Twitter, there are only between \(0.1\%\) up to at most \(3\%\) abusive tweets. What comes closest to random sampling is the procedure followed by Founta et al. (2018), Razavi et al. (2010) and the Kagglechallenge.4 They took a random sample and applied some heuristics in order to boost the proportion of abusive microposts. For instance, in the Kaggle-challenge, further microposts from users were added who had been blocked due to being reported to post personal attacks.

Footnote 4: www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge

The procedures applied by other researchers are more drastic because, as we show in SS4 and SS5, they affect more heavily the topic distribution of the dataset. These approaches do not even start with a random sample. The topic distribution is mostly determined by the creators of the dataset themselves. For example, Waseem and Hovy (2016) extract tweets matching query words likely to co-occur with abusive content. Kumar et al. (2018) choose Facebook-pages covering topics that similarly coincide with abusive language. The resulting datasets are far from representing a natural sample of the underlying social-media sites.

Table 1 shows that datasets that apply biased sampling (_Warner_, _Waseem_, _Kumar_) contain a high degree of implicit abuse. Boosted random sampling, which provides a more realistic cross section of microposts, on the other hand, captures a larger amount of explicit abuse. Future work should explore whether this is due to the predominance of explicit abuse on social media or some other reason, for example, the fact that human annotators more readily detect explicit abuse.

Intuitively, one would expect that the lower the proportion of explicit abuse is on the set of abusive microposts of a dataset, the lower the F1-score becomes because implicit abuse is not conveyed by lexical cues that are easy to learn. Table 1 confirms this notion, yet _Waseem_ is the notable exception. We need to find an explanation for this deviation since _Waseem_ is by far the most frequently used dataset for detecting abusive language Badjatiya et al. (2017); Bourgonje et al. (2017); Pitsilis et al. (2018); Agrawal and Awekar (2018); Karan and Snajder (20 

2018; Mishra et al., 2018a,b; Park et al., 2018; Qian et al., 2018; Sahlgren et al., 2018; Sharifirad et al., 2018; Unsvag and Gamback, 2018; Wiegand et al., 2018). This investigation is only possible since, fortunately, _Waseem_ is one of the datasets whose creation process has been meticulously documented.

## 4 Topic Bias

The _Waseem_-dataset has been sampled in such a way that it contains a high proportion of micropost's discussing the role of women in sports, particularly their suitability as football commentators. Such micropost's also very often coincide with sexist remarks. However, the authors did not make any attempt to _debias_ their dataset. As a consequence, domain-specific expressions such as _announcement_, _commentator_, _football_ or _sport_ occur very frequently and almost exclusively in abusive tweets. Yet intuitively these words should not be representative of abusive language. There are many texts on the web including Twitter that contain mentions of these expressions but that are not abusive. The current dataset, however, does not reflect that.

Table 2 illustrates this bias by listing the words with the highest Pointwise Mutual Information (PMI) towards abusive micropost's. It compares the _Founta_-dataset (a dataset representing almost random sampling) with the _Waseem_-dataset (a dataset produced by biased sampling). We deliberately chose two datasets sampled from the _same_ social-media site, namely Twitter, as otherwise the difference we report could be ascribed to differences in the underlying text sources. Table 2 shows that on the _Founta_-dataset, abusive words occupy the high ranks. Most of the highly ranked words of the _Waseem_-dataset, however, are not abusive. Similar observations can be made on the other datasets produced by biased sampling (i.e. _Warner_ and _Kumar_). In the _Warner_-dataset, the words _CBS_ and _Hollywood_ are two of the most predictive words. They refer to the anti-semitic prejudice that Jews are supposed to control most of the US media. On that dataset, the bias of identity terms is also extreme: Almost \(80\%\) of the \(256\) mentions of the identity term _Jew_ occur in abusive micropost's. On the _Kumar_-dataset, even common Arabic person names, such as _Azan_ or _Nahid_, strongly correlate with abusive language.

In order to demonstrate the detrimental effects such biases have, we now report the performance of further classifiers trained on the _Wassem_-dataset. Similar results could be obtained on the _Warner_- and _Kumar_-dataset. Yet they are most pronounced on the _Waseem_-dataset, which is also the dataset on which unexpectedly high classification performance has been observed in Table 1. Presumably, it is also the most biased dataset.

In our first experiment, we tested a FastText classifier (SS3) trained on the _Waseem_-dataset on a random sample of \(500\) additional tweets that include mentions of t

\begin{table}
\begin{tabular}{|l|c|r|r r|p{79.7pt}||c|} \hline
**name** & **publication** & **source** & **micropost’s** & **\#abusive** & **sampling** & **\%explicit** & **F1** \\ \hline Kaggle\({}^{\dagger}\) & (Wulczyn et al., 2017) & Wikipedia & 312,737 & 9.6 & boosted random sampling & 76.9 & 88.2 \\ Founta & (Foutna et al., 2018) & Twitter & 59,357 & 14.1 & boosted random sampling & 75.9 & 87.3 \\ Razavi & (Razavi et al., 2010) & diverse & 1,525 & 31.9 & boosted random sampling & 64.7 & 83.3 \\ Warner & (Warner and Hirschberg, 2012) & diverse & 3,438 & 14.3 & biased sampling & 51.3 & 71.8 \\ Waseem & (Waseem and Howy, 2016) & Twitter & 16,165 & 35.3 & biased sampling & 44.4 & **80.5** \\ Kumar & (Kumar et al., 2018) & Facebook & 15,000 & 58.1 & biased sampling & 32.7 & 70.4 \\ \hline  & \multicolumn{1}{p{79.7pt}}{} & \multicolumn{1}{p{79.7pt}}{} & \multicolumn{1}{p{79.7pt}}{} & \multicolumn{1}{p{79.7pt}}{} & \multicolumn{1}{p{79.7pt}}{} & \multicolumn{1}{p{79.7pt}}{} & \multicolumn{1}{p{79.7pt}}{} \\ \end{tabular}
\end{table}
Table 1: Properties of the different datasets. (\({}^{*}\): proportion of explicitly abuse micropost’s among abusive micropost’s. \({}^{\dagger}\): This is an extension of the dataset presented in Walczyn et al. (2017). Details on the corpus creation about Kaggle can therefore be found in that publication.)

\begin{table}
\begin{tabular}{|l|c c|} \hline
**rank** & **Founta** & **Wasemen** \\ \hline
1 & bitch & **commentator** \\
2 & ninggas & conedian \\
3 & motherfucker & **football** \\
4 & fucking & **announcer** \\
5 & nigga & pedophile \\
6 & idiot & mankind \\
7 & asshole & sexist \\
8 & trick & **sprt** \\
9 & fuckin & outlaw \\
10 & pussy & driver \\ \hline \end{tabular}
\end{table}
Table 2: Top 10 words having strongest correlation with abusive micropost’s according to PMI on _Founta_ (dataset representing almost random sample) and _Waseem_ (dataset produced by biased sampling).

\begin{table}
\begin{tabular}{|l|c c c|} \hline
**Feature Set** & **Prec** & **Rec** & **F1** \\ \hline all words & 80.91 & 80.08 & 80.49 \\ (ii) query words removed & 76.65 & 76.02 & 76.33 \\ (i) topic words removed & 75.07 & 74.41 & 74.72 \\ \hline \end{tabular}
\end{table}
Table 3: Impact of removing specific words from classifier trained and tested on _Waseem_.

 

_sport_. One would expect a low proportion of these particular tweets to be predicted as abusive. However, due to the fact that the abusive training data have such a large topic bias towards sports, the proportion of tweets predicted to be abusive is unreasonably high (i.e. 70%). Manual inspection confirmed that only a small proportion (up to \(5\%\)) was actually abusive. This result shows us that classifiers trained on the _Waseem_-dataset hardly generalize to the concept of abusive language. Difficult tweets on that dataset, e.g. instances of implicit abuse, may be classified correctly just because biased words such as _football_ or _sport_ occur in them.

In our second experiment, we train and test a classifier on the original _Waseem_-dataset in 10-fold crossvalidation. However, we remove either of the two types of biased words from the dataset:

* We remove 25 topic words from the 100 most correlating words that we thought bear no relation towards abusive language (e.g. _announcement_, _commentator_, _football_ or _sport_).
* We remove the 17 words that were used as a query by Waseem and Hovy (2016) to produce the dataset.

With (i) we want to show how good classifiers are that do not have access to biased words. This would be a realistic setting since words, such as _football_ or _sport_, only have this bias towards abusive language on the _Waseem_-dataset. Such removal is also necessary since otherwise these biased words cause a huge amount of false positives when testing on other datasets (as shown above).

With (ii) we want to show that query words themselves are biased, too. For example, we observed that the query word _WomenAgainstFeminism_ correlates with abusive tweets while _gamegate_ correlates with non-abusive tweets. The purpose of query words is to retrieve tweets that address specific topics. The fact that they correlate with the classes of the dataset further proves that the focused sampling process introduces data bias.

The results of these two configurations are displayed in Table 3. It shows that the removal of a very few words (i.e. \(0.2\%\) of the overall vocabulary) already causes the classification performance to drop notably. Please note that these experiments do not capture the full impact of the bias in this dataset. That is, there will be more biased words beyond the 25 words we identified on the list of top 100 words ranked according to PMI since the cut-off value of 100 was arbitrarily chosen.

## 5 Author Bias

Datasets may also be affected by author bias. By that, we mean that information relating to the author of a micropost may artificially boost classification performance. Author information can be explicitly derived from _meta-information_ of a micropost, for example, a feature that encodes the user name of a particular tweet that is to be classified. However, even if we do not explicitly encode such information, a (lexical) supervised classifier, such as the FastText-classifier from Table 1, may indirectly be affected by author biases. If the set of tweets belonging to a certain class predominantly comes from the same author, then a supervised classifier may largely pick up the writing style or the topics addressed by that author. Whenever the writing style or those topics are recognized, abusive language is predicted. This may work on a biased dataset but not beyond it.

We found that the distribution of abusive tweets on the _Waseem_-dataset is highly skewed towards 3 different authors as shown in Table 4. More than 70% of the sexist tweets originate from the two authors _Male Tears #4648_ and _Yes, They're Sexist_. 99% of the racist tweets originate from a single author (i.e. _Vile_Islam_). If virtually all racist tweets originate from the same author, a classifier just needs to consider tweets from that author and can predict tweets from every other author as nonracist. On this particular dataset, such a strategy leads to good results: Both Qian et al. (2018) and Mishra et al. (2018a) proposed classification approaches that add author information to common text-level features. These approaches were solely evaluated on the _Waseem_-dataset. However, the author distribution on the _Waseem_-dataset does not reflect reality where abusive tweets originate from far more than a very few authors. In reality, we therefore should expect author information to be less predictive.

\begin{table}
\begin{tabular}{|l c|c c|} \hline
**arcist** & \multicolumn{2}{c|}{**sexist**} \\ \hline
**author name** & **freq** & **author name** & **freq** \\ \hline Vile\_Islam & 1915 & Yes You ve Seexist & 1320 \\ Yes You’ reSexist & 8 & Male Tears \#4648 & 948 \\ Standing Up + Trump & 5 & Vile\_Islam & 50 \\ YESMarriageEquality & 1 & LiLlgasv91 & 1 \\ LiBeasy91 & 1 & N!ck & 9 \\ \hline \end{tabular}
\end{table}
Table 4: The 5 most sexist and racist authors on the _Waseem_-dataset and the number of their microposts.

 

How to Avoid a Biased Evaluation

A possible way to prevent classification scores from looking unreasonably well is by applying cross-domain classification, i.e. testing a classifier on a dataset different from the one it was trained on. The specific biases we pointed out should be primarily restricted to individual datasets and not carry over to other ones. This is illustrated by Table 5. Compared to in-domain classification (Table 1), all classifiers perform worse. So all classifiers seem to be affected by data bias to some degree. Datasets with explicit abuse and less biased sampling (_Kaggle_, _Founta_, _Razavi_) still perform reasonably when trained among each other, i.e. they are not heavily affected, whereas datasets with implicit abuse and biased sampling (_Warner_, _Waseem_, _Kumar_) perform poorly. This time this also includes _Waseem_ which implies that the good performance in in-domain classification (Table 1) was indeed caused by data bias.

Of course, cross-domain classification may not always be practical, particularly if a specific subtype of language abuse is studied for which there is only one dataset available. However, even then, simple methods such as computing the words that highly correlate with the different classes on that dataset, similar to what we did in Table 2, may already indicate that there are biases hidden in the dataset. If only a very small amount of biased words is identified, then usually it suffices to manually _debias_ the dataset. By that, one understands sampling additional microposts containing the words manually detected to be biased [30; 47]. For example, in the case of the _Waseem_-dataset, randomly sampling additional tweets matching the words _announcement_, _commentator_, _football_ or _sort_, would reduce the sexism bias we reported in this paper (simply because random tweets are unlikely to contain sexist remarks unlike the existing tweets from the _Waseem_-dataset).5 In order to avoid author bias to interfere with classification, one could restrict the number of microposts per author. This would result in a more balanced distribution of microposts per author.

Footnote 5: Please note, however, that in the case of the _Waseem_-dataset, this form of debiasing would not completely solve the data bias since this dataset contains biased words beyond the four words mentioned above.

## 7 Related Work

Previous work already established that identity terms (e.g. _gay_, _Jew_ or _woman_) have a bias to co-occur with abusive language [30; 68]. In this work, we showed that this problem is not restricted to the small set of identity terms. Most biases are introduced by the sampling method used on a dataset and they have a huge impact on classification performance.

## 8 Conclusion

We examined the impact of data bias on abusive language detection and showed that this problem is closely related to how data have been sampled. On the popular _Waseem_-dataset, we illustrated that under more realistic settings, where such biases would be less prominent, classification performance is much lower than reported in research publications. Currently, datasets with a higher degree of implicit abuse are more affected by data bias. Such bias often goes unnoticed in in-domain classification which is why we recommend cross-domain classification. Our finding that under a realistic evaluation classification performance is actually quite poor particularly on implicit abuse, is also in line with assessments from industry on the quality of the state of the art6 which suggests that there is still a long way to go.

Footnote 6: [https://www.businessinsider.de/facebook-ceo-zuckerberg-sys-hate-speech-stumps-ai-2018-4](https://www.businessinsider.de/facebook-ceo-zuckerberg-sys-hate-speech-stumps-ai-2018-4)

## Acknowledgements

The authors were partially supported by the German Research Foundation (DFG) under grants RU 1873/2-1 and WI 4204/2-1.

## References

* Agrawal and Awekar (2018) Sweeta Agrawal and Amit Awekar. 2018. Deep Learning for Detecting Cyberbullying Across Multiple Social Media Platforms. In _Proceedings of the European Conference in Information Retrieval (ECIR)_. Grenoble, France, pages 141-153.

\begin{table}
\begin{tabular}{|l c|c|c|c|c|} \hline  & \multicolumn{4}{c|}{**test**} \\ \hline
**train.** & **Kaggle** & **Founta** & **Razavi** & **Warner** & **Waseman Kumar** \\ \hline
**Kag.** & N/A & **85.83** & **76.15** & 63.91 & 60.32 & 62.48 \\
**Fou.** & **84.70** & N/A & **70.11** & 66.12 & 64.80 & 61.25 \\
**Raz.** & **72.15** & **73.34** & N/A & 60.22 & 61.76 & 60.61 \\
**Waz.** & 54.79 & 55.66 & 49.32 & N/A & 61.78 & 52.94 \\
**Was.** & 61.23 & 60.88 & 53.00 & 61.66 & N/A & 55.20 \\
**Kum.** & 69.31 & 65.93 & 62.98 & 55.74 & 59.20 & N/A \\ \hline \end{tabular}
\end{table}
Table 5: Cross-domain classification (_eval.: F1_).

 

Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017. Deep Learning for Hate Speech Detection in Tweets. In _Proceedings of the International Conference on World Wide Web (WWW)_. Perth, Australia, pages 759-760.
* Bourgonje et al. [2017] Peter Bourgonje, Julian Moreno-Schneider, Ankit Srivastava, and Georg Rehm. 2017. Automatic Classification of Abusive Language and Personal Attacks in Various Forms of Online Communication. In _Proceedings of the German Society for Computational Linguistics and Language Technology (GSCL)_. Springer, Berlin, Germany, LNAI, pages 180-191.
* Dixon et al. [2017] Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2017. Measuring and Mitigating Unintended Bias in Text Classification. In _Proceedings of the National Conference on Artificial Intelligence (AAAI)_. San Francisco, CA, USA.
* Founta et al. [2018] Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large Scale Crowdsourcing and Characterization of Twitter Abusive Behaviour. In _Proceedings of the International AAAI Conference on Weblogs and Social Media (ICWSM)_. Stanford, CA, USA.
* Joulin et al. [2017] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of Tricks for Efficient Text Classification. In _Proceedings of the Conference on European Chapter of the Association for Computational Linguistics (EACL)_. Valencia, Spain, pages 427-431.
* Karan and Snajder [2018] Miladen Karan and Jan Snajder. 2018. Cross-Domain Detection of Abusive Language Online. In _Proceedings of the Workshop on Abusive Language Online (ALW)_. Brussels, Belgium, pages 132-137.
* Kahirsagar et al. [2018] Rohan Kahirsagar, Tyrus Cukuvac, Kathleeen McKeown, and Susan McGregor. 2018. Predictive Embeddings for Hate Speech Detection on Twitter. In _Proceedings of the Workshop on Abusive Language Online (ALW)_. Brussels, Belgium, pages 26-32.
* Kumar et al. [2018a] Ritesh Kumar, Atil Kr. Ojaha, Shervin Malmasi, and Marcos Zampieri. 2018. Benchmarking Aggression Identification in Social Media. In _Proceedings of the Workshop on Troiling, Aggression and Cyberbulysing (TRAC)_. Santa Fe, NM, USA, pages 1-11.
* Mishra et al. [2018a] Pushkar Mishra, Marco Del Tredici, Helen Yannakoudakis, and Ekaterina Shutova. 2018a. Author Profiling for Abuse Detection. In _Proceedings of the International Conference on Computational Linguistics (COLING)_. Santa Fe, NM, USA, pages 1088-1098.
* Mishra et al. [2018b] Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. 2018b. Neural Character-based Composition Models for Abuse Detection. In _Proceedings of the Workshop on Abusive Language Online (ALW)_. Brussels, Belgium, pages 1-10.
* Park et al. [2018c] Ji Ho Park, Jamin Shin, and Pascale Fung. 2018c. Reducing Gender Bias in Abusive Language Detection. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Brussels, Belgium, pages 2799-2804.
* Pitsilis et al. [2018d] Georgios K. Pitsilis, Heri Ramampiaro, and Helge Langseth. 2018d. Detecting offensive language in tweets using deep learning. In _arXiv preprint arXiv: 1801.04433_.
* Qian et al. [2018e] Jing Qian, Mai ElSherief, Elizabeth M. Belding, and William Yang Wang. 2018. Leveraging intra-user and inter-user representation learning for automated hate speech detection. In _Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL (HLT/NAACL)_. New Orleans, LA, USA, pages 118-123.
* Razavi et al. [2010] Amir Hossein Razavi, Diana Inkpen, Sasha Uritsky, and Stan Matwin. 2010. Offensive Language Detection Using Multi-level Classification. In _Proceedings of the Canadian Conference on Artificial Intelligence_. Ottawa, Canada, pages 16-27.
* Sahlgren et al. [2018] Magnus Sahlgren, Tim Isbister, and Fredrik Olsson. 2018. Learning Representations for Detecting Abusive Language. In _Proceedings of the Workshop on Abusive Language Online (ALW)_. Brussels, Belgium, pages 115-123.
* Sharifiraad et al. [2018] Sima Sharifiroad, Borna Jafarpour, and Stan Matwin. 2018. Boosting Text Classification Performance on Sexist Tweets by Text Augmentation and Text Generation Using a Combination of Knowledge Graphs. In _Proceedings of the Workshop on Abusive Language Online (ALW)_. Brussels, Belgium, pages 107-101.
* Unvsug and Gamback [2018] Elise Fehn Unvsug and Bjorn Gamback. 2018. The Effects of User Features on Twitter Hate Speech Detection. In _Proceedings of the Workshop on Abusive Language Online (ALW)_. Brussels, Belgium, pages 75-85.
* Warner and Hirschberg [2012] William Warner and Julia Hirschberg. 2012. Detecting Hate Speech on the World Wide Web. In _Proceedings of the Workshop on Language in Social Media (LSM)_. Montreal, Canada, pages 19-26.
* Waseem et al. [2017] Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding Abuse: A Typology of Abusive Language Detection Subtasks. In _Proceedings of the ACL-Workshop on Abusive Language Online_. Vancouver, BC, Canada, pages 78-84.
* Student Research Workshop_. San Diego, CA, USA, pages 88
 

Michael Wiegand, Josef Ruppenhofer, Anna Schmidt, and Clayton Greenberg. 2018a Inducing a Lexicon of Abusive Words - A Feature-Based Approach. In _Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL (HLT/NAACL)_. New Orleans, LA, USA, pages 1046-1056.
* Wiegand et al. [2018b] Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. 2018b Overview of the GermEval 2018 Shared Task on the Identification of Offensive Language. In _Proceedings of the GermEval Workshop_. Vienna, Austria, pages 1-10.
* Wulczyn et al. [2017] Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex Machina: Personal Attacks Seen at Scale. In _Proceedings of the International Conference on World Wide Web (WWW)_. Perth, Australia, pages 1391-1399.
* Zhong et al. [2016] Haoti Zhong, Hao Li, Anna Cinzia Squicciarini, Sarah Michele Rajtmajer, Christopher Griffin, David J. Miller, and Cornelia Caragea. 2016. Content-Driven Detection of Cyberbullying on the Instagram Social Network. In _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_. New York City, NY, USA, pages 3952-3958.