# Time of Your Hate: The Challenge of Time in Hate Speech Detection on Social Media
## 4 Data  
We present in the following the training and test data used in our experiments. In particular, both the test set and part of the training are drawn from the same source: TWITA [22], a large-scale collection of Italian tweets. Our training data originates from two different sets of annotated data. The first set consists of both the training and reference dataset of the Haspeede (Hate Speech Detection) shared task, organized within Evalita 2018 [12]: a total of 4000 tweets (3000 tweets in the training test and 1000 in the reference set) collected from 1 October 2016 to 25 April 2017. These messages were annotated with a mixed procedure: a subset was manually annotated by five independent experts while the rest of the data was crowdsourced on Figure Eight (The annotation guidelines are available here: [[https://github.com/msang/hate-speech-corpus/blob/master/GUIDELINES.pdf](https://github.com/msang/hate-speech-corpus/blob/master/GUIDELINES.pdf)]([https://github.com/msang/hate-speech-corpus/blob/master/GUIDELINES.pdf](https://github.com/msang/hate-speech-corpus/blob/master/GUIDELINES.pdf))). The full annotation process (including information on the inter-annotator agreement) is presented in [46; 47]. The second part of our training set is formed by data filtered from TWITA [22], a large-scale collection of tweets started in 2012 and currently ongoing. It relies on the Twitter Streaming API to download a sample of messages in Italian posted each day. Although the collection counts over half a billion tweets between 2012-2017, the subset targeted for our purposes is much smaller, resulting from a topic-based selection. We draw a selection of 3809 tweets from 2015 and 3200 from 2017 using a list of topic-based keywords and imposing the constrain of the tweet having a geotag in Italy, as we are interested in linguistics phenomena in Italian. The handcrafted list of  
Figure 1: On the left it is showed the SVM model, on the right the one based on AlBERTo. keywords extends the work described in [46; 47], where a set of terms was compiled by assessing which minority groups are most likely to be targeted for HS in the online discourse about immigration. This choice was based on the results of the 2015 Eurobarometer Survey on discrimination in the EU ([[http://ec.europa.eu/justice/fundamental-rights/files/factsheet_eurobarometer_fundamental_rights_2015.pdf](http://ec.europa.eu/justice/fundamental-rights/files/factsheet_eurobarometer_fundamental_rights_2015.pdf)]([http://ec.europa.eu/justice/fundamental-rights/files/factsheet_eurobarometer_fundamental_rights_2015.pdf](http://ec.europa.eu/justice/fundamental-rights/files/factsheet_eurobarometer_fundamental_rights_2015.pdf))). We further expanded this collection of terms using the Open Multilingual Wordnet ([[http://compiling.hss.ntu.edu.sg/omw/](http://compiling.hss.ntu.edu.sg/omw/)]([http://compiling.hss.ntu.edu.sg/omw/](http://compiling.hss.ntu.edu.sg/omw/))) (OMW), not only to collect a larger number of messages, but also to capture a wider variety of expressions on this topic. For each of these keywords, we collected from OMW all the related hypernyms and co-hypernyms. We then manually cleaned this new set of keywords to retain only the most relevant ones. The final list of our keywords is in Table 1.  
When filtering out text data, it is crucial to take into account the possibility of substring matching, e.g., \"Rom\" would match \"Roma\" (the capital city of Italy). We addressed this issue by implementing regular expressions in our filtering algorithm to match only entire keywords preceded and followed by white-spaces, punctuation, or beginning/end of a sentence. Our approach was purely string-based; therefore, we could collect tweets containing keywords occurring with a different meaning from expected, such as named entities (e.g., \"Nomadi\" is the Italian form for \"nomads\", but also the name of a popular music band). However, upon manual inspection, we noticed that these occurrences are extremely rare in our collection.