Automatic detection of hate speech in text: an overview of the topic and dataset annotation with hierarchical classes

The use of the internet and social networks, in particular for communication, has significantly increased in recent years. This growth has also resulted in the adoption of more aggressive communication. Therefore it is important that governments and social network platforms have tools to detect this type of communication, because it can be harmful to its targets. In this thesis we investigate the problem of detecting hate speech posted online. The first goal of our work was to make a complete overview on the topic, focusing on the perspective of computer science and engineering. We adopted an exhaustive and methodical approach that we called Systematic Literature Review. As a result, we critically summarized different perspectives on the hate speech concept and complemented our definition with rules, examples, and a comparison with other related concepts, such as cyberbullying and abusive language. Regarding the past work in the topic, we observed that the majority of the studies tackles this problem as a machine learning classification task and the studies use either general text mining features (e.g. n-grams, word2vec), or hate speech specific features (e.g. othering discourse). In the majority of these studies new datasets are collected, but those remain private, which makes more difficult to compare results across different works. We concluded also that this field is still in an early stage, with several open research opportunities. As we found no research on the topic in Portuguese, the second goal of this work was to annotate a dataset for this language and to make it available as well. Regarding the dataset annotation, we built a classification system using a hierarchical structure. The main advantage of this strategy is that it allows to better consider nuances in the hate speech concept, such as the existence and intersectionality of the subtypes of hate speech. Our data was collected from Twitter, and manually annotated by following a set of rules, that are also a valuable product of our work. We annotated a dataset with 5,668 messages from 1,156 distinct users, where 85 distinct classes of hate speech were considered. From the total 5,668 messages, around 22% contain some type of hate speech. Regarding the annotators agreement, using the hierarchical approach allowed us to improve results, however this was still an issue in identifying hate speech. Further analysis pointed out that the several types of hate speech present different characteristics (e.g. distinct number of messages, time occurrences, vocabulary size, distinct n-grams and POS). A final goal of our thesis was to investigate the potential advantages of using hierarchical classes to annotate a dataset. For this, we used the dataset annotated for Portuguese and we conducted an experiment with training, validation and test phases. In this experiment we compare two different approaches: we called unimodel to the model using only the hate speech class; and multimodel to the model using the several hierarchical classes. The main conclusion of our experiment was that the performance of the multimodel seemed to be slightly better than the unimodel in the F1 metric, and additionally, our method helped to identify a larger number of hate speech messages. This is the case because it has a better recall, in detriment of the precision. Finally, we think that in the future this experiment can be extended in order to better identify hate speech and the respective subtypes.