DisaggregHate It Corpus: A Disaggregated Italian Dataset of Hate Speech
Authors: Marco Madeddu, Simona Frenda, Mirko Lai, Viviana Patti, Valerio Basile (Università di Torino, Aequa-tech, 2023)
Topic: Sentiment, Emotion, Irony, Hate

Abstract
Recent studies in Machine Learning advocate for the exploitation of disagreement between annotators to train models in line with the different opinions of humans about a specific phenomenon. This means that datasets where the annotations are aggregated by majority voting are not enough. In this paper, we present an Italian disaggregated dataset concerning hate speech and encoding some information about the annotators: the DisaggregHate It Corpus. The corpus contains Italian tweets that focus on the topic of racism and has been annotated by native Italian university students. We explain how the dataset was gathered by following the recommendation of the perspectivist approach, encouraging the annotators to give some socio-demographic information about them. To exploit the disagreement in the learning process, we proposed two types of soft labels: softmax and standard normalization. We investigated the benefit of using disagreement by creating a baseline binary model and two regression models that were respectively trained on the ‘hard’ (aggregated label by majority voting) and the two types of ‘soft’ labels. We tested the models in an in-domain and out-of-domain setting, evaluating their performance using the cross-entropy as a metric, and showing that the models trained on the soft labels performed better.

Introduction
The rise of the Internet and social media platforms has given many users the opportunity to express their opinion online. Unfortunately, this leads to the diffusion of a new online phenomenon: hate speech. To prevent the viral spread of these expressions on social media, hate speech detection has become a popular task in Natural Language Processing (NLP). A lot of tools have been created to detect and counter hate speech. Recently, there have been studies suggesting shifting away from the golden standard approach in Machine Learning, especially in tasks that are partly subjective and influenced by the social and cultural context, such as hate speech. These works advocate that different opinions given in the annotation process are not a noise factor but can be used to create better systems. This shift has inspired scholars to explore different techniques to train models using datasets where the target label is not simply determined by majority voting on the annotations. In this line, two theoretical paradigms have been established, both focusing on the inclusion of different perspectives: learning from disagreement and perspectivism. The former can be considered a ‘soft perspectivist approach’ because it takes into account the presence of disagreement in the annotated data, while the latter overcomes the idea of “ground truth” in dataset construction and in the creation and evaluation of NLP models, focusing more on who the annotators are. Our work can be seen as an attempt to approach hate speech detection by exploiting the possible disagreement among annotators. Usually, models are trained on data associated with a ‘hard’ label. In binary classification, each item is assigned a label, either 0 or 1. The hard label value is commonly obtained through majority voting, meaning that controversial instances have the same label as those where all annotators agree. This may lead to a loss of valuable information that could be used during training. On the other hand, ‘soft’ labels aim to avoid this waste of data by assigning a real number to the label. Different functions, such as standard normalization or the softmax function, can be used to determine the value of the soft label. In this context, we created the DisaggregHate It Corpus, a new disaggregated dataset about hate speech in the Italian language that includes some socio-demographic information about annotators. A corpus like this could be beneficial in exploring how different segments of the population are sensitive to social issues like hate speech and how this information can be leveraged to create better systems. After explaining the characteristics of the dataset in Section 3, we validate the corpus by using it as the training set for different models in Section 4. The experiments show that training models on a soft label rather than a hard label leads to better results. As suggested by previous research, we used cross-entropy as a metric for evaluating the models.

Conclusion
In this work, we presented the DisaggregHate It Corpus, a new disaggregated dataset in the Italian language for hate speech detection. To our knowledge, it is the first dataset released with disaggregated labels, incorporating socio-demographic information about the annotators. Upon computing the agreement among annotators with similar profiles, we found that the Krippendorff’s α was very low. Additionally, by pairing this information with the number of ties obtained through majority voting, we observed that disagreement is a significant factor in the corpus. This insight motivates the need for hate speech detection models that can capture the different opinions of human annotators. To address this, we experimented with the use of soft labels, exploring two methods of computing these labels: standard and softmax normalization. To further investigate the role of disagreement in model learning, we carried out experiments testing the performance of our models in two specific settings: in-domain and out-of-domain. We created a binary model based on hard labels and two regression models trained on the soft labels computed using the two normalization methods. Inspired by previous works, we evaluated the models by computing the cross-entropy between the soft labels of the annotations and the model predictions. The results showed that the regression models performed better in both in-domain and out-of-domain test sets. This indicates that using soft labels helps to integrate the disagreement among annotators into the models, aligning them more closely with the distribution of opinions among human annotators. Based on these findings, we plan to extend the study by using the same DisaggregHate It Corpus to explore a stronger perspectivist approach. This would involve modeling the perspectives of different groups of annotators based on their socio-demographic traits or other commonalities, further enhancing the ability of models to reflect diverse viewpoints.