Source-driven Representations for Hate Speech Detection
Authors: Flavio Merenda, Claudia Zaghi, Tommaso Caselli, Malvina Nissim (University of Groningen, Università di Salerno, 2018)
Topic: Sentiment, Emotion, Irony, Hate

Abstract
Sources, in the form of selected English. Sources, in the form of selected Facebook pages, can be used as indicators of hate-rich content. Polarized distributed representations created over such content prove superior to generic embeddings in the task of hate speech detection. The same content seems to carry a too weak signal to proxy silver labels in a distant supervised setting. However, this signal is stronger than gold labels which come from a different distribution, leading to re-think the process of annotation in the context of highly subjective judgments.

Introduction
Hate speech is “the use of aggressive, hatred or offensive language, targeting a speciﬁc group of people sharing a common trait: their gender, ethnic group, race, religion, sexual orientation, or disability” (Merriam-Webster’s collegiate dictionary, 1999). The phenomenon is widely spread on-line, and Italian Social Media is deﬁnitely not an exception (Gagliardone et al., 2015). To monitor the problem, social networks and websites have introduced a stricter code of conduct and regularly remove hateful content ﬂagged by users (Bleich, 2014). However, the volume of data requires that ways are found to classify on-line content automatically (Nobata et al., 2016; Kennedy et al., 2017). The Italian NLP community is active on this front (Poletto et al., 2017; Del Vigna et al., 2017), with the development of labeled data, including the organization of a dedicated shared task at the EVALITA 2018 campaign1. Relying on manually labeled data has limitations, though: i.) annotation is time and resource consuming; ii.) portability to new domains is scarce2; iii.) biases are unavoidable in annotated data, especially in the form of annotation decisions. This is both due to the intrinsic subjectivity of the task itself, and to the fact that there is not, as yet, a shared set of deﬁnitions and guidelines across the different projects that yield annotated datasets. Introduced as a new take on data annotation (Mintz et al., 2009; Go et al., 2009), distant supervision is used to automatically assign (silver) labels based on the presence or absence of speciﬁc hints, such as happy/sad emoticons (Go et al., 2009) to proxy positive/negative labels for sentiment analysis, Facebook reactions (Pool and Nissim, 2016; Basile et al., 2017) for emotion detection, or speciﬁc strings to assign gender (Emmery et al., 2017). Such an approach has the advantage of being more scalable (portability to different languages or domains) and versatile (time and resources needed to train), than pure supervised learning algorithms, while preserving competitive performance. Apart from the ease of generating labeled data, distant supervision has a valuable ecological aspect in not relying on third-party annotators to interpret the data (Purver and Battersby, 2012). This reduces the risk of adding extra bias (see also point (iii) about limitation in the previous paragraph), modulo the choices related to which proxies should be considered. Novelty and Contribution We promote a special take on distant supervision where we use as proxies the sources where the content is published on-line rather than any hint in the content itself. Through a battery of experiments on hate speech detection in Italian we show that this approach yields meaningful representations and an increase in performance over the use of generic representations. Contextually, we show the limitations of silver labels, but also of gold labels that come from a different dataset with respect to the evaluation set.

Conclusion
We exploited distant supervision to automatically obtain representations from Facebook-scraped content in two forms. First, we generated polarized, hate-rich distributed representations which proved superior to larger, generic embeddings when used both in a CNN and an SVM model for hate speech detection. Second, we used the scraped data as training material directly, proxing 8We also experimented with adding progressively larger batches of silver data to gold (2K, 3K, 5K, etc.), but this yielded a steady decrease in performance. labels (hate vs non-hate) with the sources where the data was coming from (Facebook pages). This did not prove as a successful alternative nor complementary strategy to using gold data, though performance above baseline indicates some signal is present. Importantly, though, our experiments also suggest that gold data is not better than silver data if it comes from a different dataset. This highlights a crucial aspect related to the creation of manually labeled datasets, especially in the highly subjective area of hate speech and affective computing in general, where different guidelines and different annotators clearly introduce large biases and discrepancies across datasets. All considered, we believe that obtaining data in a distant, more ecological way should be further pursued and reﬁned. How to better exploit the information that comes from polarized embeddings in combination with other features is also left to future work.