Tackling Online Abuse: A Survey of Automated Abuse Detection Methods

Abstract: Abuse on the Internet represents an important societal problem of our time. Millions of Internet users face harassment, racism, personal attacks, and other types of abuse on online platforms. The psychological effects of such abuse on individuals can be profound and lasting. Consequently, over the past few years, there has been a substantial research effort towards automated abuse detection in the field of natural language processing (NLP). In this paper, we present a comprehensive survey of the methods that have been proposed to date, thus providing a platform for further development of this area. We describe the existing datasets and review the computational approaches to abuse detection, analyzing their strengths and limitations. We discuss the main trends that emerge, highlight the challenges that remain, outline possible solutions, and propose guidelines for ethics and explainability

With the advent of social media, anti-social and abusive behavior has become a prominent occurrence online. Undesirable psychological effects of abuse on individuals make it an important societal problem of our time. Munro munro2011 studied the ill-effects of online abuse on children, concluding that children may develop depression, anxiety, and other mental health problems as a result of their encounters online. Pew Research Center, in its latest report on online harassment BIBREF0 , revealed that INLINEFORM0 of adults in the United States have experienced abusive behavior online, of which INLINEFORM1 have faced severe forms of harassment, e.g., that of sexual nature. The report goes on to say that harassment need not be experienced first-hand to have an impact: INLINEFORM2 of American Internet users admitted that they stopped using an online service after witnessing abusive and unruly behavior of their fellow users. These statistics stress the need for automated abuse detection and moderation systems. Therefore, in the recent years, a new research effort on abuse detection has sprung up in the field of NLP.
That said, the notion of abuse has proven elusive and difficult to formalize. Different norms across (online) communities can affect what is considered abusive BIBREF1 . In the context of natural language, abuse is a term that encompasses many different types of fine-grained negative expressions. For example, Nobata et al. nobata use it to collectively refer to hate speech, derogatory language and profanity, while Mishra et al. mishra use it to discuss racism and sexism. The definitions for different types of abuse tend to be overlapping and ambiguous. However, regardless of the specific type, we define abuse as any expression that is meant to denigrate or offend a particular person or group. Taking a course-grained view, Waseem et al. W17-3012 classify abuse into broad categories based on explicitness and directness. Explicit abuse comes in the form of expletives, derogatory words or threats, while implicit abuse has a more subtle appearance characterized by the presence of ambiguous terms and figures of speech such as metaphor or sarcasm. Directed abuse targets a particular individual as opposed to generalized abuse, which is aimed at a larger group such as a particular gender or ethnicity. This categorization exposes some of the intricacies that lie within the task of automated abuse detection. While directed and explicit abuse is relatively straightforward to detect for humans and machines alike, the same is not true for implicit or generalized abuse. This is illustrated in the works of Dadvar et al. davdar and Waseem and Hovy waseemhovy: Dadvar et al. observed an inter-annotator agreement of INLINEFORM0 on their cyber-bullying dataset. Cyber-bullying is a classic example of directed and explicit abuse since there is typically a single target who is harassed with personal attacks. On the other hand, Waseem and Hovy noted that INLINEFORM1 of all the disagreements in annotation of their dataset occurred on the sexism class. Sexism is typically both generalized and implicit.
In this paper, we survey the methods that have been developed for automated detection of online abuse, analyzing their strengths and weaknesses. We first describe the datasets that exist for abuse. Then we review the various detection methods that have been investigated by the NLP community. Finally, we conclude with the main trends that emerge, highlight the challenges that remain, outline possible solutions, and propose guidelines for ethics and explainability. To the best of our knowledge, this is the first comprehensive survey in this area. We differ from previous surveys BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 in the following respects: 1) we discuss the categorizations of abuse based on coarse-grained vs. fine-grained taxonomies; 2) we present a detailed overview of datasets annotated for abuse; 3) we provide an extensive review of the existing abuse detection methods, including ones based on neural networks (omitted by previous surveys); 4) we discuss the key outstanding challenges in this area; and 5) we cover aspects of ethics and explainability.
Supervised learning approaches to abuse detection require annotated datasets for training and evaluation purposes. To date, several datasets manually annotated for abuse have been made available by researchers. These datasets differ in two respects:
In what follows, we review several commonly-used datasets manually annotated for abuse.
Dataset descriptions. The earliest dataset published in this domain was compiled by Spertus smokey. It consisted of INLINEFORM0 private messages written in English from the web-masters of controversial web resources such as NewtWatch. These messages were marked as flame (containing insults or abuse; INLINEFORM1 ), maybe flame ( INLINEFORM2 ), or okay ( INLINEFORM3 ). We refer to this dataset as data-smokey. Yin et al. Yin09detectionof constructed three English datasets and annotated them for harassment, which they defined as “systematic efforts by a user to belittle the contributions of other users". The samples were taken from three social media platforms: Kongregate ( INLINEFORM4 posts; INLINEFORM5 harassment), Slashdot ( INLINEFORM6 posts; INLINEFORM7 harassment), and MySpace ( INLINEFORM8 posts; INLINEFORM9 harassment). We refer to the three datasets as data-harass. Several datasets have been compiled using samples taken from portals of Yahoo!, specifically the News and Finance portals. Djuric et al. djuric created a dataset of INLINEFORM10 user comments in English from the Yahoo! Finance website that were editorially labeled as either hate speech ( INLINEFORM11 ) or clean (data-yahoo-fin-dj). Nobata et al. nobata produced four more datasets with comments from Yahoo! News and Yahoo! Finance, each labeled abusive or clean: 1) data-yahoo-fin-a: INLINEFORM12 comments, 7.0% abusive; 2) data-yahoo-news-a: INLINEFORM13 comments, 16.4% abusive; 3) data-yahoo-fin-b: INLINEFORM14 comments, 3.4% abusive; and 4) data-yahoo-news-b: INLINEFORM15 comments, 9.7% abusive.
Several groups have investigated abusive language in Twitter. Waseem and Hovy waseemhovy created a corpus of INLINEFORM0 tweets, each annotated as one of racism ( INLINEFORM1 ), sexism, ( INLINEFORM2 ) or neither (data-twitter-wh). We note that although certain tweets in the dataset lack surface-level abusive traits (e.g., @Mich_McConnell Just “her body” right?), they have nevertheless been marked as racist or sexist as the annotators took the wider discourse into account; however, such discourse information or annotation is not preserved in the dataset. Inter-annotator agreement was reported at INLINEFORM3 , with a further insight that INLINEFORM4 of all the disagreements occurred on the sexism class alone. Waseem waseem later released a dataset of INLINEFORM5 tweets annotated as racism ( INLINEFORM6 ), sexism ( INLINEFORM7 ), both ( INLINEFORM8 ), or neither (data-twitter-w). data-twitter-w and data-twitter-wh have INLINEFORM9 tweets in common. It should, however, be noted that the inter-annotator agreement between the two datasets is low (mean pairwise INLINEFORM10 ) BIBREF6 .
Davidson et al. davidson created a dataset of approximately INLINEFORM0 tweets, manually annotated as one of racist ( INLINEFORM1 ), offensive but not racist ( INLINEFORM2 ), or clean ( INLINEFORM3 ). We note, however, that their data sampling procedure relied on the presence of certain abusive words and, as a result, the distribution of classes does not follow a real-life distribution. Recently, Founta et al. founta crowd-sourced a dataset (data-twitter-f) of INLINEFORM4 tweets, of which INLINEFORM5 were annotated as normal, INLINEFORM6 as spam, INLINEFORM7 as hateful and INLINEFORM8 as abusive. The OffensEval 2019 shared task used a recently released dataset of INLINEFORM9 tweets BIBREF7 , each hierarchically labeled as: offensive ( INLINEFORM10 ) or not, whether the offence is targeted ( INLINEFORM11 ) or not, and whether it targets an individual ( INLINEFORM12 ), a group ( INLINEFORM13 ) or otherwise ( INLINEFORM14 ).
Wulczyn et al. wulczyn annotated English Talk page comments from a dump of the full history of Wikipedia and released three datasets: one focusing on personal attacks ( INLINEFORM0 comments; INLINEFORM1 abusive), one on aggression ( INLINEFORM2 comments), and one on toxicity ( INLINEFORM3 comments; INLINEFORM4 abusive) (data-wiki-att, data-wiki-agg, and data-wiki-tox respectively). data-wiki-agg contains the exact same comments as data-wiki-att but annotated for aggression – the two datasets show a high correlation in the nature of abuse (Pearson's INLINEFORM5 ). Gao and Huang gao2017detecting released a dataset of INLINEFORM6 Fox News user comments (data-fox-news) annotated as hateful ( INLINEFORM7 ) or non-hateful. The dataset preserves context information for each comment, including user's screen-name, all comments in the same thread, and the news article for which the comment is written.
Some researchers investigated abuse in languages other than English. Van Hee et al. vanhee gathered INLINEFORM0 Dutch posts from ask.fm to form a dataset on cyber-bullying (data-bully; INLINEFORM1 cyber-bullying cases). Pavlopoulos et al. pavlopoulos-emnlp released a dataset of ca. INLINEFORM2 comments in Greek provided by the news portal Gazzetta (data-gazzetta). The comments were marked as accept or reject, and are divided into 6 splits with similar distributions (the training split is the largest one: INLINEFORM3 accepted and INLINEFORM4 rejected comments). As part of the GermEval shared task on identification of offensive language in German tweets BIBREF8 , a dataset of INLINEFORM5 tweets was released, of which INLINEFORM6 were labeled as abuse, INLINEFORM7 as insult, INLINEFORM8 as profanity, and INLINEFORM9 as other. Around the same time, INLINEFORM10 Facebook posts and comments, each in Hindi (in both Roman and Devanagari script) and English, were released (data-facebook) as part of the COLING 2018 shared task on aggression identification BIBREF9 . INLINEFORM11 of the comments were covertly aggressive, INLINEFORM12 overtly aggressive and INLINEFORM13 non-aggressive. We note, however, that some issues were raised by the participants regarding the quality of the annotations. The HatEval 2019 shared task (forthcoming) focuses on detecting hate speech against immigrants and women using a dataset of INLINEFORM14 tweets in Spanish and INLINEFORM15 in English annotated hierarchically as hateful or not; and, in turn, as aggressive or not, and whether the target is an individual or a group.
Remarks. In their study, Ross et al. ross stressed the difficulty in reliably annotating abuse, which stems from multiple factors, such as the lack of “standard” definitions for the myriad types of abuse, differences in annotators' cultural background and experiences, and ambiguity in the annotation guidelines. That said, Waseem et al. W17-3012 and Nobata et al. nobata observed that annotators with prior expertise provide good-quality annotations with high levels of agreement. We note that most datasets contain discrete labels only; abuse detection systems trained on them would be deprived of the notion of severity, which is vital in real-world settings. Also, most datasets cover few types of abuse only. Salminen et al. salminen2018anatomy suggest fine-grained annotation schemes for deeper understanding of abuse; they propose 29 categories that include both types of abuse and their targets (e.g., humiliation, religion).
In this section, we describe abuse detection methods that rely on hand-crafted rules and manual feature engineering. The first documented abuse detection method was designed by Spertus smokey who used a heuristic rule-based approach to produce feature vectors for the messages in the data-smokey dataset, followed by a decision tree generator to train a classification model. The model achieved a recall of INLINEFORM0 on the flame messages, and INLINEFORM1 on the non-flame ones in the test set. Spertus noted some limitations of adopting a heuristic rule-based approach, e.g., the inability to deal with sarcasm, and vulnerability to errors in spelling, punctuation and grammar. Yin et al. Yin09detectionof developed a method for detecting online harassment. Working with the three data-harass datasets, they extracted local features (tf–idf weights of words), sentiment-based features (tf–idf weights of foul words and pronouns) and contextual features (e.g., similarity of a post to its neighboring posts) to train a linear support vector machine (svm) classifier. The authors concluded that important contextual indicators (such as harassment posts generally being off-topic) cannot be captured by local features alone. Their approach achieved INLINEFORM2 F INLINEFORM3 on the MySpace dataset, INLINEFORM4 F INLINEFORM5 on the Slashdot dataset, and INLINEFORM6 F INLINEFORM7 on the Kongregate dataset.
Razavi et al. razavi were the first to adopt lexicon-based abuse detection. They constructed an insulting and abusing language dictionary of words and phrases, where each entry had an associated weight indicating its abusive impact. They utilized semantic rules and features derived from the lexicon to build a three-level Naive Bayes classification system and apply it to a dataset of INLINEFORM0 messages ( INLINEFORM1 flame and the rest okay) extracted from the Usenet newsgroup and the Natural Semantic Module company's employee conversation thread ( INLINEFORM2 accuracy). Njagi et al. gitari also employed such a lexicon-based approach and, more recently, Wiegand et al. wiegand proposed an automated framework for generating such lexicons. While methods based on lexicons performed well on explicit abuse, the researchers noted their limitations on implicit abuse.
Bag-of-words (bow) features have been integral to several works on abuse detection. Sood et al. sood2012 showed that an svm trained on word bi-gram features outperformed a word-list baseline utilizing a Levenshtein distance-based heuristic for detecting profanity. Their best classifier (combination of SVMs and word-lists) yielded an F INLINEFORM0 of INLINEFORM1 . Warner and Hirschberg warner employed a template-based strategy alongside Brown clustering to extract surface-level bow features from a dataset of paragraphs annotated for antisemitism, and achieved an F INLINEFORM2 of INLINEFORM3 using svms. Their approach is unique in that they framed the task as a word-sense disambiguation problem, i.e., whether a term carried an anti-semitic sense or not. Other examples of bow-based methods are those of Dinakar et al. dinakar2011modeling, Burnap and Williams burnap and Van Hee et al. vanhee who use word n-grams in conjunction with other features, such as typed-dependency relations or scores based on sentiment lexicons, to train svms ( INLINEFORM4 F INLINEFORM5 on the data-bully dataset). Recenlty, Salminen et al. salminen2018anatomy showed that a linear SVM using tf–idf weighted n-grams achieves the best performance (average F INLINEFORM6 of INLINEFORM7 ) on classification of hateful comments (from a YouTube channel and Facebook page of an online news organization) as one of 29 different hate categories (e.g., accusation, promoting violence, humiliation, etc.).
Several researchers have directly incorporated features and identity traits of users in order to model the likeliness of abusive behavior from users with certain traits, a process known as user profiling. Dadvar et al. davdar included the age of users alongside other traditional lexicon-based features to detect cyber-bullying, while Galán-García et al. galan2016supervised utilized the time of publication, geo-position and language in the profile of Twitter users. Waseem and Hovy waseemhovy exploited gender of Twitter users alongside character n-gram counts to improve detection of sexism and racism in tweets from data-twitter-wh (F INLINEFORM0 increased from INLINEFORM1 to INLINEFORM2 ). Using the same setup, Unsvåg and Gambäck unsvaag2018effects showed that the inclusion of social network-based (i.e., number of followers and friends) and activity-based (i.e., number of status updates and favorites) information of users alongside their gender further enhances performance ( INLINEFORM3 gain in F INLINEFORM4 ).
In this section, we review the approaches to abuse detection that utilize or rely solely on neural networks. We also include methods that use embeddings generated from a neural architecture within an otherwise non-neural framework.
Distributed representations. Djuric et al. djuric were the first to adopt a neural approach to abuse detection. They utilized paragraph2vec BIBREF10 to obtain low-dimensional representations for comments in data-yahoo-fin-dj, and train a logistic regression (lr) classifier. Their model outperformed other classifiers trained on bow-based representations (auc INLINEFORM0 vs. INLINEFORM1 ). In their analysis, the authors noted that words and phrases in hate speech tend to be obfuscated, leading to high dimensionality and large sparsity of bow representations; classifiers trained on such representations often over-fit in training.
Building on the work of Djuric et al., Nobata et al. nobata evaluated the performance of a large range of features on the Yahoo! datasets (data-yahoo-*) using a regression model: (1) word and character n-grams; (2) linguistic features, e.g., number of polite/hate words and punctuation count; (3) syntactic features, e.g., parent and grandparent of node in a dependency tree; (4) distributional-semantic features, e.g., paragraph2vec comment representations. Although the best results were achieved with all features combined (F INLINEFORM0 INLINEFORM1 on data-yahoo-fin-a, INLINEFORM2 on data-yahoo-news-a), character n-grams on their own contributed significantly more than other features due to their robustness to noise (i.e., obfuscations, misspellings, unseen words). Experimenting with the data-yahoo-fin-dj dataset, Mehdad and Tetreault mehdad investigated whether character-level features are more indicative of abuse than word-level ones. Their results demonstrated the superiority of character-level features, showing that svm classifiers trained on Bayesian log-ratio vectors of average counts of character n-grams outperform the more intricate approach of Nobata et al. nobata in terms of AUC ( INLINEFORM3 vs. INLINEFORM4 ) as well as other rnn-based character and word-level models.
Samghabadi et al. W17-3010 utilized a similar set of features as Nobata et al. and augmented it with hand-engineered ones such as polarity scores derived from SentiWordNet, measures based on the LIWC program, and features based on emoticons. They then applied their method to three different datasets: data-wiki-att, a Kaggle dataset annotated for insult, and a dataset of questions and answers (each labeled as invective or neutral) that they created by crawling ask.fm. Distributional-semantic features combined with the aforementioned features constituted an effective feature space for the task ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 F INLINEFORM3 on data-wiki-att, Kaggle, ask.fm respectively). In line with the findings of Nobata et al. and Mehdad and Tetreault, character n-grams performed well on these datasets too.
Deep learning in abuse detection. With the advent of deep learning, many researchers have explored its efficacy in abuse detection. Badjatiya et al. badjatiya evaluated several neural architectures on the data-twitter-wh dataset. Their best setup involved a two-step approach wherein they use a word-level long-short term memory (lstm) model, to tune glove or randomly-initialized word embeddings, and then train a gradient-boosted decision tree (gbdt) classifier on the average of the tuned embeddings in each tweet. They achieved the best results using randomly-initialized embeddings (weighted F INLINEFORM0 of INLINEFORM1 ). However, working with a similar setup, Mishra et al. mishra recently reported that glove initialization provided superior performance; a mismatch is attributed to the fact that Badjatiya et al. tuned the embeddings on the entire dataset (including the test set), hence allowing for the randomly-initialized ones to overfit.
Park and Fung parkfung utilized character and word-level cnns to classify comments in the dataset that they formed by combining data-twitter-w and data-twitter-wh. Their experiments demonstrated that combining the two levels of granularity using two input channels achieves the best results, outperforming a character n-gram lr baseline (weighted F INLINEFORM0 from INLINEFORM1 to INLINEFORM2 ). Several other works have also demonstrated the efficacy of cnns in detecting abusive social media posts BIBREF11 . Some researchers BIBREF12 , BIBREF13 have shown that sequentially combining cnns with gated recurrent unit (gru) rnns can enhance performance by taking advantage of properties of both architectures (e.g., 1-2% increase in F INLINEFORM3 compared to only using cnns).
Pavlopoulos et al. pavlopoulos,pavlopoulos-emnlp applied deep learning to the data-wiki-att, data-wiki-tox, and data-gazzetta datasets. Their most effective setups were: (1) a word-level gru followed by an lr layer; (2) setup 1 extended with an attention mechanism on words. Both setups outperformed a simple word-list baseline and the character n-gram lr classifier (detox) of Wulczyn et al. wulczyn. Setup 1 achieved the best performance on data-wiki-att and data-wiki-tox (auc INLINEFORM0 and INLINEFORM1 respectively), while setup 2 performed the best on data-gazzetta (auc INLINEFORM2 ). The attention mechanism was additionally able to highlight abusive words and phrases within the comments, exhibiting a high level of agreement with annotators on the task. Lee et al. W18-5113 worked with a subset of the data-twitter-f dataset and showed that a word-level bi-gru along with latent topic clustering (whereby topic information is extracted from the hidden states of the gru BIBREF14 ) yielded the best weighted F INLINEFORM3 ( INLINEFORM4 ).
The GermEval shared task on identification of offensive language in German tweets BIBREF8 saw submission of both deep learning and feature engineering approaches. The winning system BIBREF15 (macro F INLINEFORM0 of INLINEFORM1 ) employed multiple character and token n-gram classifiers, as well as distributional semantic features obtained by averaging word embeddings. The second best approach BIBREF16 (macro F INLINEFORM2 INLINEFORM3 ), on the other hand, employed an ensemble of cnns, the outputs of which were fed to a meta classifier for final prediction. Most of the remaining submissions BIBREF17 , BIBREF18 used deep learning with cnns and rnns alongside techniques such as transfer learning (e.g., via machine translation or joint representation learning for words across languages) from abuse-annotated datasets in other languages (mainly English). Wiegand et al. wiegand2018overview noted that simple deep learning approaches themselves were quite effective, and the addition of other techniques did not necessarily provide substantial improvements.
Kumar et al. kumar2018benchmarking noted similar trends in the shared task on aggression identification on data-facebook. The top approach on the task's English dataset BIBREF19 comprised rnns and cnns along with transfer learning via machine translation (macro F INLINEFORM0 of INLINEFORM1 ). The top approach for Hindi BIBREF20 utilized lexical features based on word and character n-grams (F INLINEFORM2 62.92%).
Recently, Aken et al. van2018challenges performed a systematic comparison of neural and non-neural approaches to toxic comment classification, finding that ensembles of the two were most effective.
User profiling with neural networks. More recently, researchers have employed neural networks to extract features for users instead of manually leveraging ones like gender, location, etc. as discussed before. Working with the data-gazzetta dataset, Pavlopoulos et al. W17-4209 incorporated user embeddings into Pavlopoulos' setup 1 pavlopoulos,pavlopoulos-emnlp described above. They divided all the users whose comments are included in data-gazzetta into 4 types based on proportion of abusive comments (e.g., red users if INLINEFORM0 comments and INLINEFORM1 abusive comments), yellow (users with INLINEFORM2 comments and INLINEFORM3 abusive comments), green (users with INLINEFORM4 comments and INLINEFORM5 abusive comments), and unknown (users with INLINEFORM6 comments). They then assigned unique randomly-initialized embeddings to users and added them as additional input to the lr layer, alongside representations of comments obtained from the gru, increasing auc from INLINEFORM7 to INLINEFORM8 . Qian et al. N18-2019 used lstms for modeling inter and intra-user relationships on data-twitter-wh, with sexist and racist tweets combined into one category. The authors applied a bi-lstm to users' recent tweets in order to generate intra-user representations that capture their historic behavior. To improve robustness against noise present in tweets, they also used locality sensitive hashing to form sets semantically similar to user tweets. They then trained a policy network to select tweets from such sets that a bi-lstm could use to generate inter-user representations. When these inter and intra-user representations were utilized alongside representations of tweets from an lstm baseline, performance increased significantly (from INLINEFORM9 to INLINEFORM10 F INLINEFORM11 ).
Mishra et al. mishra constructed a community graph of all users whose tweets are included in the data-twitter-wh dataset. Nodes in the graph were users while edges the follower-following relationship between them on Twitter. They then applied node2vec BIBREF21 to this graph to generate user embeddings. Inclusion of these embeddings into character n-gram based baselines yielded state of the art results on data-twitter-wh (F INLINEFORM0 increased from INLINEFORM1 and INLINEFORM2 to INLINEFORM3 and INLINEFORM4 on the racism and sexism classes respectively). The gains were attributed to the fact that user embeddings captured not only information about online communities, but also some elements of the wider conversation amongst connected users in the graph. Ribeiro et al. ribeiro and Mishra et al. mishragcn applied graph neural networks BIBREF22 , BIBREF23 to social graphs in order to generate user embeddings (i.e., profiles) that capture not only their surrounding community but also their linguistic behavior.
Current trends. English has been the dominant language so far in terms of focus, followed by German, Hindi and Dutch. However, recent efforts have focused on compilation of datasets in other languages such as Slovene and Croatian BIBREF24 , Chinese BIBREF25 , Arabic BIBREF26 , and even some unconventional ones such as Hinglish BIBREF27 . Most of the research to date has been on racism, sexism, personal attacks, toxicity, and harassment. Other types of abuse such as obscenity, threats, insults, and grooming remain relatively unexplored. That said, we note that the majority of methods investigated to date and described herein are (in principle) applicable to a range of abuse types.
While the recent state of the art approaches rely on word-level cnns and rnns, they remain vulnerable to obfuscation of words BIBREF28 . Character n-gram, on the other hand, remain one of the most effective features for addressing obfuscation due to their robustness to spelling variations. Many researchers to date have exclusively relied on text based features for abuse detection. But recent works have shown that personal and community-based profiling features of users significantly enhance the state of the art.
Ethical challenges. Whilst the research community has started incorporating features from user profiling, there has not yet been a discussion of ethical guidelines for doing so. To encourage such a discussion, we lay out four ethical considerations in the design of such approaches. First, the profiling approach should not compromise the privacy of the user. So a researcher might ask themselves such questions as: is the profiling based on identity traits of users (e.g., gender, race etc.) or solely on their online behavior? And is an appropriate generalization from (identifiable) user traits to population-level behavioural trends performed? Second, one needs to reflect on the possible bias in the training procedure: is it likely to induce a bias against users with certain traits? Third, the visibility aspect needs to be accounted for: is the profiling visible to the users, i.e., can users directly or indirectly observe how they (or others) have been profiled? And finally, one needs to carefully consider the purpose of such profiling: is it intended to take actions against users, or is it more benign (e.g. to better understand the content produced by them and make task-specific generalizations)? While we do not intend to provide answers to these questions within this survey, we hope that the above considerations can help to start a debate on these important issues.
Labeling abuse. Labeling experiences as abusive provides powerful validation for victims of abuse and enables observers to grasp the scope of the problem. It also creates new descriptive norms (suggesting what types of behavior constitute abuse) and exposes existing norms and expectations around appropriate behavior. On the other hand, automated systems can invalidate abusive experiences, particularly for victims whose experiences do not lie within the realm of `typical' experiences BIBREF29 . This points to a critical issue: automated systems embody the morals and values of their creators and annotators BIBREF30 , BIBREF29 . It is therefore imperative that we design systems that overcome such issues. For e.g., some recent works have investigated ways to mitigate gender bias in models BIBREF31 , BIBREF32 .
Abuse over time and across domains. New abusive words and phrases continue to enter the language BIBREF33 . This suggests that abuse is a constantly changing phenomenon. Working with the data-yahoo-*-b datasets, Nobata et al. nobata found that a classifier trained on more recent data outperforms one trained on older data. They noted that a prominent factor in this is the continuous evolution of the Internet jargon. We would like to add that, given the situational and topical nature of abuse BIBREF1 , contextual features learned by detection methods may become irrelevant over time.
A similar trend also holds for abuse detection across domains. Wiegand et al. wiegand showed that the performance of state of the art classifiers BIBREF34 , BIBREF35 decreases substantially when tested on data drawn from domains different to those in the training set. Wiegand et al. attributed the trend to lack of domain-specific learning. Chandrasekharan et al. chandrasekharan2017bag propose an approach that utilizes similarity scores between posts to improve in-domain performance based on out-of-domain data. Possible solutions for improving cross-domain abuse detection can be found in the literature of (adversarial) multi-task learning and domain adaptation BIBREF36 , BIBREF37 , BIBREF38 , and also in works such as that of Sharifirad et al. jafarpour2018boosting who utilize knowledge graphs to augment the training of a sexist tweet classifier. Recently, Waseem et al. waseem2018bridging and Karan and Šnajder karan2018cross exploited multi-task learning frameworks to train models that are robust across data from different distributions and data annotated under different guidelines.
Modeling wider conversation. Abuse is inherently contextual; it can only be interpreted as part of a wider conversation between users on the Internet. This means that individual comments can be difficult to classify without modeling their respective contexts. However, the vast majority of existing approaches have focused on modeling the lexical, semantic and syntactic properties of comments in isolation from other comments. Mishra et al. mishra have pointed out that some tweets in data-twitter-wh do not contain sufficient lexical or semantic information to detect abuse even in principle, e.g., @user: Logic in the world of Islam http://t.co/xxxxxxx, and techniques for modeling discourse and elements of pragmatics are needed. To address this issue, Gao and Huang gao2017detecting, working with data-fox-news, incorporate features from two sources of context: the title of the news article for which the comment was posted, and the screen name of the user who posted it. Yet this is only a first step towards modeling the wider context in abuse detection; more sophisticated techniques are needed to capture the history of the conversation and the behavior of the users as it develops over time. NLP techniques for modeling discourse and dialogue can be a good starting point in this line of research. However, since posts on social media often includes data of multiple modalities (e.g., a combination of images and text), abuse detection systems would also need to incorporate a multi-modal component.
Figurative language. Figurative devices such as metaphor and sarcasm are common in natural language. They tend to be used to express emotions and sentiments that go beyond the literal meaning of words and phrases BIBREF39 . Nobata et al. nobata (among others, e.g., Aken et al. van2018challenges) noted that sarcastic comments are hard for abuse detection methods to deal with since surface features are not sufficient; typically the knowledge of the context or background of the user is also required. Mishra mishrathesis found that metaphors are more frequent in abusive samples as opposed to non-abusive ones. However, to fully understand the impact of figurative devices on abuse detection, datasets with more pronounced presence of these are required.
Explainable abuse detection. Explainability has become an important aspect within NLP, and within AI generally. Yet there has been no discussion of this issue in the context of abuse detection systems. We hereby propose three properties that an explainable abuse detection system should aim to exhibit. First, it needs to establish intent of abuse (or the lack of it) and provide evidence for it, hence convincingly segregating abuse from other phenomena such as sarcasm and humour. Second, it needs to capture abusive language, i.e., highlight instances of abuse if present, be they explicit (i.e., use of expletives) or implicit (e.g., dehumanizing comparisons). Third, it needs to identify the target(s) of abuse (or the absence thereof), be it an individual or a group. These properties align well with the categorizations of abuse we discussed in the introduction. They also aptly motivate the advances needed in the field: (1) developments in areas such as sarcasm detection and user profiling for precise segregation of abusive intent from humor, satire, etc.; (2) better identification of implicit abuse, which requires improvements in modeling of figurative language; (3) effective detection of generalized abuse and inference of target(s), which require advances in areas such as domain adaptation and conversation modeling.
Online abuse stands as a significant challenge before society. Its nature and characteristics constantly evolve, making it a complex phenomenon to study and model. Automated abuse detection methods have seen a lot of development in recent years: from simple rule-based methods aimed at identifying directed, explicit abuse to sophisticated methods that can capture rich semantic information and even aspects of user behavior. By comprehensively reviewing the investigated methods to date, our survey aims to provide a platform for future research, facilitating progress in this important area. While we see an array of challenges that lie ahead, e.g., modeling extra-propositional aspects of language, user behavior and wider conversation, we believe that recent progress in the areas of semantics, dialogue modeling and social media analysis put the research community in a strong position to address them. Summaries of public datasets In table TABREF4 , we summarize the datasets described in this paper that are publicly available and provide links to them. A discussion of metrics The performance results we have reported highlight that, throughout work on abuse detection, different researchers have utilized different evaluation metrics for their experiments – from area under the receiver operating characteristic curve (auroc) BIBREF79 , BIBREF48 to micro and macro F INLINEFORM0 BIBREF28 – regardless of the properties of their datasets. This makes the presented techniques more difficult to compare. In addition, as abuse is a relatively infrequent phenomenon, the datasets are typically skewed towards non-abusive samples BIBREF6 . Metrics such as auroc may, therefore, be unsuitable since they may mask poor performance on the abusive samples as a side-effect of the large number of non-abusive samples BIBREF52 . Macro-averaged precision, recall, and F INLINEFORM1 , as well as precision, recall, and F INLINEFORM2 on specifically the abusive classes, may provide a more informative evaluation strategy; the primary advantage being that macro-averaged metrics provide a sense of effectiveness on the minority classes BIBREF73 . Additionally, area under the precision-recall curve (auprc) might be a better alternative to auroc in imbalanced scenarios BIBREF46 . 

Figure and table captions:
Table 1: Links and summaries of datasets mentioned in the paper that are publicly available.

Questions about this paper:
What datasets were used in this work?

Answers:
DATA-TWITTER-WH, DATA-TWITTER-W, DATA-TWITTER-DAVID, DATA-TWITTER-F, DATA-WIKI-ATT, DATA-WIKI-AGG, DATA-WIKI-TOX, DATA-FOX-NEWS, DATA-GAZZETTA, DATA-FACEBOOK, Arabic News, GermEval, Ask.fun