Introduction
Social media is a very popular way for people to express their opinions publicly and to interact with others online. In aggregation, social media can provide a reflection of public sentiment on various events. Unfortunately, many users engaging online, either on social media, forums or blogs, will often have the risk of being targeted or harassed via abusive language, which may severely impact their online experience and the community in general. The existence of social networking services creates the need for detecting user-generated hateful messages prior to publication. All published text that is used to express hatred towards some particular group with the intention to humiliate its members is considered a hateful message.
Although hate speech is protected under the free speech provisions in the United States, there are other countries, such as Canada, France, United Kingdom, and Germany, where there are laws prohibiting it as being promoting violence or social disorder. Social media services such as Facebook and Twitter have been criticized for not having done enough to prohibit the use of their services for attacking people belonging to some specific race, minority etc. BIBREF0 . They have announced though that they would seek to battle against racism and xenophobia BIBREF1 . Nevertheless, the current solutions deployed by them have attempted to address the problem with manual effort, relying on users to report offensive comments BIBREF2 . This not only requires a huge effort by human annotators, but it also has the risk of applying discrimination under subjective judgment. Moreover, a non-automated task by human annotators would have strong impact on system response times, since a computer-based solution can accomplish this task much faster than humans. The massive rise in the user-generated content in the above social media services, with manual filtering not being scalable, highlights the need for automating the process of on-line hate-speech detection.
Despite the fact that the majority of the solutions for automated detection of offensive text rely on Natural Language Processing (NLP) approaches, there is lately a tendency towards employing pure machine learning techniques like neural networks for that task. NLP approaches have the drawback of being complex, and to a large extent dependent on the language used in the text. This provides a strong motivation for employing alternative machine learning models for the classification task. Moreover, the majority of the existing automated approaches depend on using pre-trained vectors (e.g. Glove, Word2Vec) as word embeddings to achieve good performance from the classification model. That makes the detection of hatred content unfeasible in cases where users have deliberately obfuscated their offensive terms with short slang words.
There is a plethora of unsupervised learning models in the existing literature to deal with hate-speech BIBREF3 , as well as in detecting the sentiment polarity in tweets BIBREF4 . At the same time, the supervised learning approaches have not been explored adequately so far. While the task of sentence classification seems similar to that of sentiment analysis; nevertheless, in hate-speech even negative sentiment could still provide useful insight. Our intuition is that the task of hate-speech detection can be further benefited by the incorporation of other sources of information to be used as features into a supervised learning model. A simple statistical analysis on an existing annotated dataset of tweets by BIBREF5 , can easily reveal the existence of significant correlation between the user tendency in expressing opinions that belong to some offensive class (Racism or Sexism), and the annotation labels associated with that class. More precisely, the correlation coefficient value that describes such user tendency was found to be 0.71 for racism in the above dataset, while that value reached as high as 0.76 for sexism. In our opinion, utilizing such user-oriented behavioural data for reinforcing an existing solution is feasible, because such information is retrieva2ble in real-world use-case scenarios like Twitter. This highlights the need to explore the user features more systematically to further improve the classification accuracy of a supervised learning system.
Our approach employs a neural network solution composed of multiple Long-Short-Term-Memory (LSTM) based classifiers, and utilizes user behavioral characteristics such as the tendency towards racism or sexism to boost performance. Although our technique is not necessarily revolutionary in terms of the deep learning models used, we show in this paper that it is quite effective.
Our main contributions are: INLINEFORM0 ) a deep learning architecture for text classification in terms of hateful content, which incorporates features derived form the users' behavioural data, INLINEFORM1 ) a language agnostic solution, due to no-use of pre-trained word embeddings, for detecting hate-speech, INLINEFORM2 ) an experimental evaluation of the model on a Twitter dataset, demonstrating the top performance achieved on the classification task. Special focus is given to investigating how the additional features concerning the users' tendency to utter hate-speech, as expressed by their previous history, could leverage the performance. To the best of our knowledge, there has not been done any previous study on exploring features related to the users tendency in hatred content that used a deep learning model.
The rest of the paper is organized as follows. In Section SECREF2 we describe the problem of hate speech in more detail, and we refer to the existing work in the field in Section SECREF3 . In Section SECREF4 we present our proposed model, while in Section SECREF5 we refer to the dataset used, the evaluation tests we performed and we discuss the results received. Finally, in Section SECREF6 we summarize our contributions and discuss the future work.
Problem Statement
The problem we address in this work can be formally described as follows: Let INLINEFORM0 be an unlabeled short sentence composed of a number of words, posted by a user INLINEFORM1 . Let INLINEFORM2 , INLINEFORM3 , INLINEFORM4 be three classes denoting Neutrality, Sexism and Racism respectively in a textual content. Members of these classes are those postings with content classified as belonging to the corresponding class, for which the following holds: INLINEFORM5 . Further, given that user INLINEFORM6 has a previous history of message postings INLINEFORM7 , we assume that any previous posting INLINEFORM8 by that user is already labeled as belonging to any of the classes N,S,R. Similarly, other postings by other users have also been labeled accordingly, forming up their previous history. Based on these facts, the problem is to identify the class, which the unlabeled sentence INLINEFORM9 by user INLINEFORM10 belongs to.
The research question we address in this work is:
How to effectively identify the class of a new posting, given the identity of the posting user and the history of postings related to that user?
To answer this question, our main goals can be summarized as follows:
Note that existing solutions for automatic detection are still falling short to effectively detect abusive messages. Therefore there is a need for new algorithms which would do the job of classification of such content more effectively and efficiently. Our work is towards that direction.
Related Work
Simple word-based approaches, if used for blocking the posting of text or blacklisting users, not only fail to identify subtle offensive content, but they also affect the freedom of speech and expression. The word ambiguity problem – that is, a word can have different meanings in different contexts – is mainly responsible for the high false positive rate in such approaches. Ordinary NLP approaches on the other hand, are ineffective to detect unusual spelling, experienced in user-generated comment text. This is best known as the spelling variation problem, and it is caused either by unintentional or intentional replacement of single characters in a token, aiming to obfuscate the detectors.
In general, the complexity of the natural language constructs renders the task quite challenging. The employment of supervised learning classification methods for hate speech detection is not new. BIBREF6 reported performance for a simple LSTM classifier not better than an ordinary SVM, when evaluated on a small sample of Facebook data for only 2 classes (Hate, No-Hate), and 3 different levels of strength of hatred. BIBREF7 described another way of detecting offensive language in tweets, based on some supervised model. They differentiate hate speech from offensive language, using a classifier that involves naive Bayes, decision trees and SVM. Also, BIBREF8 attempted to discern abusive content with a supervised model combining various linguistic and syntactic features in the text, considered at character uni-gram and bi-gram level, and tested on Amazon data. In general, we can point out the main weaknesses of NLP-based models in their non-language agnostic nature and the low scores in detection.
Unsupervised learning approaches are quite common for detecting offensive messages in text by applying concepts from NLP to exploit the lexical syntactic features of sentences BIBREF9 , or using AI-solutions and bag-of-words based text-representations BIBREF10 . The latter is known to be less effective for automatic detection, since hatred users apply various obfuscation tricks, such as replacing a single character in offensive words. For instance, applying a binary classifier onto a paragraph2vec representation of words has already been attempted on Amazon data in the past BIBREF11 , but it only performed well on a binary classification problem. Another unsupervised learning based solution is the work by BIBREF12 , in which the authors proposed a set of criteria that a tweet should exhibit in order to be classified as offensive. They also showed that differences in geographic distribution of users have only marginal effect on the detection performance. Despite the above observation, we explore other features that might be possible to improve the detection accuracy in the solution outlined below.
The work by BIBREF5 applied a crowd-sourced solution to tackle hate-speech, with the creation of an additional dataset of annotations to extend the existing corpus. The impact of the experience of annotators in the classification performance was investigated. The work by BIBREF13 dealt with the classification problem of tweets, but their interest was on sexism alone, which they distinguished into `Hostile', `Benevolent' or `Other'. While the authors used the dataset of tweets from BIBREF12 , they treated the existing `Sexism' tweets as being of class `Hostile', while they collected their own tweets for the `Benevolent' class, on which they finally applied the FastText by BIBREF14 , and SVM classification.
BIBREF15 approached the issue with a supervised learning model that is based on a neural network. Their method achieved higher score over the same dataset of tweets than any unsupervised learning solution known so far. That solution uses an LSTM model, with features extracted by character n-grams, and assisted by Gradient Boosted Decision Trees. Convolution Neural Networks (CNN) has also been explored as a potential solution in the hate-speech problem in tweets, with character n-grams and word2vec pre-trained vectors being the main tools. For example, BIBREF16 transformed the classification into a 2-step problem, where abusive text first is distinguished from the non-abusive, and then the class of abuse (Sexism or Racism) is determined. BIBREF17 employed pre-trained CNN vectors in an effort to predict four classes. They achieved slightly higher F-score than character n-grams.
In spite of the high popularity of NLP approaches in hate-speech classification BIBREF3 , we believe there is still a high potential for deep learning models to further contribute to the issue. At this point it is also relevant to note the inherent difficulty of the challenge itself, which can be clearly noted by the fact that no solution thus far has been able to obtain an F-score above 0.93.
Description of our Recurrent Neural Network-based Approach
The power of neural networks comes from their ability to find data representations that are useful for classification. Recurrent Neural Networks (RNN) are a special type of neural network, which can be thought of as the addition of loops to the architecture. RNNs use back propagation in the training process to update the network weights in every layer. In our experimentation we used a powerful type of RNN known as Long Short-Term Memory Network (LSTM). Inspired by the work by BIBREF15 , we experiment with combining various LSTM models enhanced with a number of novel features in an ensemble. More specifically we introduce:
Features
We first elaborate into the details of the features derived to describe each user's tendency towards each class (Neutral, Racism or Sexism), as captured in their tweeting history. In total, we define the three features INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , representing a user's tendency towards posting Neutral, Racist and Sexist content, respectively. We let INLINEFORM3 denote the set of tweets by user INLINEFORM4 , and use INLINEFORM5 , INLINEFORM6 and INLINEFORM7 to denote the subsets of those tweets that have been labeled as Neutral, Racist and Sexist respectively. Now, the features are calculated as INLINEFORM8 , INLINEFORM9 ,and INLINEFORM10 .
Furthermore, we choose to model the input tweets in the form of vectors using word-based frequency vectorization. That is, the words in the corpus are indexed based on their frequency of appearance in the corpus, and the index value of each word in a tweet is used as one of the vector elements to describe that tweet. We note that this modelling choice provides us with a big advantage, because the model is independent of the language used for posting the message.
Classification
To improve classification ability we employ an ensemble of LSTM-based classifiers.
In total the scheme comprises a number of classifiers (3 or 5), each receiving the vectorized tweets together with behavioural features (see Section SECREF5 ) as input.
The choice of various characteristics was done with the purpose to train the neural network with any data associations existing between the attributes for each tweet and the class label given to that tweet. In each case, the characteristic feature is attached to the already computed vectorized content for a tweet, thereby providing an input vector for one LSTM classifier. A high level view of the architecture is shown in Figure FIGREF7 , with the multiple classifiers. The ensemble has two mechanisms for aggregating the classifications from the base classifiers; namely Voting and Confidence. The preferred method is majority voting, which is employed whenever at least two of the base classifiers agrees wrt. classification of a given tweet. When all classifiers disagree, the classifier with strongest confidence in its prediction is given preference. The conflict resolution logic is implemented in the Combined Decision component.
Ensemble classifier [1] INLINEFORM0 INLINEFORM1 classifiers INLINEFORM2 INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6 INLINEFORM7 decision INLINEFORM8 decision INLINEFORM9 decision for INLINEFORM10
We present the above process in Algorithm SECREF6 . Here mode denotes a function that provides the dominant value within the inputs classes INLINEFORM0 and returns NIL if there is a tie, while classifier is a function that returns the classification output in the form of a tuple (Neutral, Racism, Sexism).
Data Preprocessing
Before training the neural network with the labeled tweets, it is necessary to apply the proper tokenization to every tweet. In this way, the text corpus is split into word elements, taking white spaces and the various punctuation symbols used in the language into account. This was done using the Moses package for machine translation.
We choose to limit the maximum size of each tweet to be considered during training to 30 words, and padded tweets of shorter size with zeros. Next, tweets are converted into vectors using word-based frequency, as described in Section SECREF5 . To feed the various classifiers in our evaluation, we attach the feature values onto every tweet vector.
In this work we experimented with various combinations of attached features INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 that express the user tendency. The details of each experiment, including the resulting size of each embedding can be found in Table TABREF10 , with the latter denoted `input dimension' in the table.
Deep learning model
In our evaluation of the proposed scheme, each classifier is implemented as a deep learning model having four layers, as illustrated in Figure FIGREF16 , and is described as follows:
The Input (a.k.a Embedding) Layer. The input layer's size is defined by the number of inputs for that classifier. This number equals the size to the word vector plus the number of additional features. The word vector dimension was set to 30 so that to be able to encode every word in the vocabulary used.
The hidden layer. The sigmoid activation was selected for the the hidden LSTM layer. Based on preliminary experiments the dimensionality of the output space for this layer was set to 200. This layer is fully connected to both the Input and the subsequent layer.
The dense layer. The output of the LSTM was run through an additional layer to improve the learning and obtain more stable output. The ReLU activation function was used. Its size was selected equal to the size of the input layer.
The output layer. This layer has 3 neurons to provide output in the form of probabilities for each of the three classes Neutral, Racism, and Sexism. The softmax activation function was used for this layer.
In total we experimented with 11 different setups of the proposed scheme, each with a different ensemble of classifiers, see Table TABREF17 .
Dataset
We experimented with a dataset of approximately 16k short messages from Twitter, that was made available by BIBREF12 . The dataset contains 1943 tweets labeled as Racism, 3166 tweets labeled as Sexism and 10889 tweets labeled as Neutral (i.e., tweets that neither contain sexism nor racism). There is also a number of dual labeled tweets in the dataset. More particularly, we found 42 tweets labeled both as both `Neutral' and `Sexism', while six tweets were labelled as both `Racism' and `Neutral'. According to the dataset providers, the labeling was performed manually.
The relatively small number of tweets in the dataset makes the task more challenging. As reported by several authors already, the dataset is imbalanced, with a majority of neutral tweets. Additionally, we used the public Twitter API to retrieve additional data associated with the user identity for each tweet in the original dataset.
Experimental Setting
To produce results in a setup comparable with the current state of the art BIBREF15 , we performed 10-fold cross validation and calculated the Precision,Recall and F-Score for every evaluated scheme. We randomly split each training fold into 15% validation and 85% training, while performance is evaluated over the remaining fold of unseen data. The model was implemented using Keras. We used categorical cross-entropy as learning objective, and selected the ADAM optimization algorithm BIBREF18 . Furthermore, the vocabulary size was set to 25000, and the batch-size during training was set to 500.
To avoid over-fitting, the model training was allowed to run for a maximum number of 100 epochs, out of which the optimally trained state was chosen for the model evaluation. An optimal epoch was identified so, such that the validation accuracy was maximized, while at the same time the error remained within INLINEFORM0 of the lowest ever figure within the current fold. Throughout the experiment we observed that the optimal epochs typically occurred after between the 30 and 40 epochs.
To achieve stability in the results produced, we ran every single classifier for 15 times and the output values were aggregated. In addition, the output from each single classifier run was combined with the output from another two single classifiers to build the input of an ensemble, producing INLINEFORM0 combinations. For the case of the ensemble that incorporates all five classifiers we restricted to using the input by only the first five runs of the single classifiers ( INLINEFORM1 combinations). That was due to the prohibitively very large number of combinations that were required.
Results
We now present the most interesting results from our experiments. For the evaluation we used standard metrics for classification accuracy, suitable for studying problems such as sentiment analysis. In particular we used Precision and Recall, with the former calculated as the ratio of the number of tweets correctly classified to a given class over the total number of tweets classified to that class, while the latter measures the ratio of messages correctly classified to a given class over the number of messages from that class. Additionally, the F-score is the harmonic mean of precision and recall, expressed as INLINEFORM0 . For our particular case with three classes, P, R and F are computed for each class separately, with the final F value derived as the weighted mean of the separate INLINEFORM1 -scores: INLINEFORM2 ; recall that INLINEFORM3 , INLINEFORM4 and INLINEFORM5 . The results are shown in Table TABREF24 , along with the reported results from state of the art approaches proposed by other researchers in the field. Note that the performance numbers P,R and F of the other state of the art approaches are based on the authors' reported data in the cited works. Additionally, we report the performance of each individual LSTM classifier as if used alone over the same data (that is, without the ensemble logic). The F-score for our proposed approaches shown in the last column, is the weighted average value over the 3 classes (Neutral,Sexism,Racism). Moreover, all the reported values are average values produced for a number of runs of the same tested scheme over the same data. Figure FIGREF23 shows the F-Score as a function of the number of training samples for each ensemble of classifiers. We clearly see that the models converge. For the final run the F-score has standard deviation value not larger than 0.001, for all classifiers.
As can be seen in Table TABREF24 , the work by BIBREF12 , in which character n-grams and gender information were used as features, obtained the quite low F-score of INLINEFORM0 . Later work by the same author BIBREF5 investigated the impact of the experience of the annotator in the performance, but still obtaining a lower F-score than ours. Furthermore, while the first part of the two step classification BIBREF16 performs quite well (reported an F-score of 0.9520), it falls short in detecting the particular class the abusive text belongs to. Finally, we observe that applying a simple LSTM classification with no use of additional features (denoted `single classifier (i)' in Table TABREF24 ), achieves an F-score that is below 0.93, something that is in line with other researchers in the field, see BIBREF15 .
Very interestingly, the incorporation of features related to user's behaviour into the classification has provided a significant increase in the performance vs. using the textual content alone, INLINEFORM0 vs. INLINEFORM1 .
Another interesting finding is the observed performance improvement by using an ensemble instead of a single classifier; some ensembles outperform the best single classifier. Furthermore, the NRS classifier, which produces the best score in relation to other single classifiers, is the one included in the best performing ensemble.
In comparison to the approach by BIBREF13 , which focuses on various classes of Sexism, the results show that our deep learning model is doing better as far as detecting Sexism in general, outperforming the FastText algorithm they include in their experiments (F=0.87). The inferiority of FastText over LSTM is also reported in the work by BIBREF15 , as well as being inferior over CNN in, BIBREF16 . In general, through our ensemble schemes is confirmed that deep learning can outperform any NLP-based approaches known so far in the task of abusive language detection.
We also present the performance of each of the tested models per class label in Table TABREF25 . Results by other researchers have not been included, as these figures are not reported in the existing literature. As can be seen, sexism is quite easy to classify in hate-speech, while racism seems to be harder; similar results were reported by BIBREF7 . This result is consistent across all ensembles.
For completion, the confusion matrices of the best performing approach that employs 3 classifiers (ensemble viii) as well as of the ensemble of the 5 classifiers (xi), are provided in Table TABREF26 . The presented values is the sum over multiple runs.
The code and results associated with this paper will be available on-line soon at: https://github.com/gpitsilis/hate-speech/
Conclusions and Future Work
In this work we present an ensemble classifier that is detecting hate-speech in short text, such as tweets. The input to the base-classifiers consists of not only the standard word uni-grams, but also a set of features describing each user's historical tendency to post abusive messages. Our main innovations are: i) a deep learning architecture that uses word frequency vectorisation for implementing the above features, ii) an experimental evaluation of the above model on a public dataset of labeled tweets, iii) an open-sourced implementation built on top of Keras.
The results show that our approach outperforms the current state of the art, and to the best of our knowledge, no other model has achieved better performance in classifying short messages. The approach does not rely on pre-trained vectors, which provides a serious advantage when dealing with short messages of this kind. More specifically, users will often prefer to obfuscate their offensive terms using shorter slang words or create new words by `inventive' spelling and word concatenation. For instance, the word `Islamolunatic' is not available in the popular pre-trained word embeddings (Word2Vec or GloVe), even though it appears with a rather high frequency in racist postings. Hence, word frequency vectorization is preferable to the pre-trained word embeddings used in prior works if one aims to build a language-agnostic solution.
We believe that deep learning models have a high potential wrt. classifying text or analyzing the sentiment in general. In our opinion there is still space for further improving the classification algorithms.
For future work we plan to investigate other sources of information that can be utilized to detect hateful messages. In addition, we intend to generalize the output received in the current experiment, with evaluation over other datasets, including analyzing texts written in different languages.

Question: what rnn classifiers were used?
Answer: LSTM classifier with no additional features, Neutral & Sexism, Neutral & Racism, Racism & Sexism and Neutral, Racism & Sexism.

experiment with combining various LSTM models enhanced with a number of novel features (O No additional features,  NS Neutral & Sexism,  NR Neutral & Racism, RS Racism & Sexism,  NRS Neutral, Racism & Sexism) in an ensemble.

Long Short-Term Memory Network (LSTM)

Question: what results did their system obtain?
Answer: Best authors' system achieved 0.9320 F1 score.

The best model achieved a 0.9320 F-score

The best performing single classifier produces F1 0.9265. The best ensemble classifier  (O+NS+RS+NR+NRS) produce  F1 0.9320.

Question: what are the existing approaches?
Answer: BIBREF12 , in which character n-grams and gender information were used as features, BIBREF5 investigated the impact of the experience of the annotator in the performance, two step classification BIBREF16, BIBREF13 , which focuses on various classes of Sexism, CNN in, BIBREF16