# Time of Your Hate: The Challenge of Time in Hate Speech Detection on Social Media
## 5 Experimental Evaluation
### Experimental Design  
For what regards the prediction systems, we decided to compare AlBERTo against a traditional SVM, as it is the one in use in the \"Contro I'odio\" project. We then trained each system in two different scenarios: a _sliding window_ model and an _incremental_ model. In the first case we trained the system on month \(t_{i}\) and then tested it on the following month \(t_{i+1}\).  
In the second case instead we progressively incremented the size of the training set: we tested the models on month \(t_{i}\) but trained them on data from all the previous months, from \(t_{0}\) to \(t_{i-1}\). The rationale behind this choice was to evaluate how the system performance vary while injecting information on language and discourse about the recent past. To explore the interplay between the size of the training set and the temporal gap with the test data, we performed a second set of experiments with a fixed test set but adding Haspeede+ to each of the two training schemes. The reason for this was to evaluate how the systems performed when trained on a larger but older dataset, injected with information on language and topics far away in the past. For comparison, we also tested both systems after having trained them only on Haspeede+, as a baseline for comparison with the other settings.  
To smooth out any possible random effect, we repeated every single experiment for all the possible setups five times, each of which had a different random seed. We then computed the arithmetical average of the standard metrics.  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Dataset** & **Size** & **\% non-HS** & **\% HS** \\ \hline Haspeede Set & 4000 & 67.6 & 32.4 \\ Figure Eight Train Set 1 (data from 2015) & 3809 & 85.5 & 14.5 \\ Figure Eight Train Set 2 (data from 2017) & 3200 & 82.7 & 17.3 \\ \hline test 2018\_09 & 1991 & 67.5 & 32.5 \\ test 2018\_10 & 2000 & 82.9 & 17.1 \\ test 2018\_11 & 2000 & 84.2 & 15.8 \\ test 2018\_12 & 2000 & 84.1 & 15.9 \\ test 2019\_01 & 2000 & 90.2 & 9.8 \\ test 2019\_02 & 2000 & 91.4 & 8.6 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Dataset size and class balance.  
Moving from the consideration that we are performing a hate speech detection task and not a more general classification task, we decided to focus on precision, recall, and F1 for the positive class (the presence of hate speech) and macro F1-score on both classes. The reason behind this choice is that we believe that the key point in our experiment was to measure the effectiveness of the algorithm when correctly detecting hate speech messages, rather than correctly labeling non-hate speech messages. As an example, we believed that for us, the model needed to be able to correctly classify the hate speech sentence \"You are ugly, kill yourself\" more than classifying the sentence \"Today is a good day\" as not hate speech. We also computed the macro F1-score that averages on both classes because, as seen in Table 2, the distribution of the label in our training datasets is unbalanced, and this last metrics provides a better insight on the system performance in this specific case.