# Time of Your Hate: The Challenge of Time in Hate Speech Detection on Social Media
## 4 Data
The tweets were then annotated by three independent contributors on Figure Eight, now Appen ([[https://appen.com/](https://appen.com/)]([https://appen.com/](https://appen.com/))), chosen to be Italian speakers, geolocated in Italy and using the same guidelines as in [12]. More in detail, each tweet had a so-called confidence score that captures the level of agreement between multiple contributors and indicates the \"confidence\" in the validity of the labeling. This index, based on [PERSON]'s \(\alpha\) metric, also incorporates a weighted average over the annotators' trust scores that tracks the dependability and consistency of each annotator's labeling history over time on the platform. In the following sections, we will refer collectively to all the data used for training the models as Haspeede+. Our aim was to investigate the temporal robustness of BERT in predicting hate speech \begin{table}
\begin{tabular}{l l} \hline \hline
**Keywords in Italian** & **English Translation** \\ \hline clandestin\({}^{\star}\) & _illegal immigrant(s)_ \\ corano & _Quran_ \\ emigrant\({}^{\star}\) & _emigrant(s)_ \\ emigrat\({}^{\star}\) & _emigrant(s)_ \\ esul\({}^{\star}\) & _exile(s)_ \\ fondamentalist\({}^{\star}\) & _fundamentalist(s)_ \\ imam & _imam_ \\ imamigrant\({}^{\star}\) & _immigrant(s)_ \\ immigrat\({}^{\star}\) & _immigrant(s)_ \\ Islam & _Islam_ \\ islamismo & _islamism_ \\ islamit\({}^{\star}\) & _Islamist(s)_ \\ maomettan\({}^{\star}\) & _Mohammad(s)_ \\ migrant\({}^{\star}\) & _migrant(s)_ \\ migrazione & _migration_ \\ musulman\({}^{\star}\) & _muslim(s)_ \\ musulmanesimo & _Islam_ \\ musulman\({}^{\star}\) & _muslim(s)_ \\ nond\({}^{\star}\) & _nomad(s)_ \\ profug\({}^{\star}\) & _refuge(s)_ \\ sfollat\({}^{\star}\) & _displaced_ \\ stranier\({}^{\star}\) & _foreigner(s)_ \\ \hline \hline \end{tabular}
\end{table} 
Table 1: List of keywords in Italian (and their English translation) used to compile the dataset. on Twitter messages related to immigration phenomena in Italy. In this context, we needed a set of data on this topic with a temporal structure that allowed us to capture variations in language and topics over time and then measure the hate speech detection systems performance using standard metrics, such as precision, recall, and F1-score. For this purpose, the data filtered as part of the \"Contro I'odio\" project, described in Section 1, were the perfect solution, both in terms of topic and temporal distribution. We used as test data random monthly samples of roughly 2000 tweets per set, from September 2018 to February 2019. This dataset was also entirely annotated on Appen with the same strategy illustrated before. In Table 2, we list the detailed size and class balance of all our datasets. We notice that the percentage of HS tweets decreases with time, while tweets are being annotated by the same set of annotators. A possible explanation of this is that the data from 2019 may be significantly different from the examples given to the annotators as guidelines (which belonged to previous years), and this yields to an inconsistently in the quality of the annotation results and ultimately to this class imbalance. The average length of the tweets is 24 words in the training sets and 38 words in the test sets. However, the training sets were collected using the standard Twitter API truncating messages longer than 140 characters, while the test sets were collected with the updated API returning the full messages. In terms of language variability, the type-token ratios are expectedly low, ranging from 10% to 18% across data sets.