## 5 Experimental Evaluation
### Data Preparation and Evaluation Metrics  
To perform the experimental evaluation through the proposed investigation framework, we performed data collection and labeling. Data collection is the process of collecting messages (from Twitter) that are relevant to the problem domain. It is a crucial step, because it strongly determines the results obtained by classifiers. Messages were collected from Twitter by using Tweepy API [48]; 133,306 messages were collected from different accounts. The next step was to manually label the messages with a pool of predefined topics. In this process, we involved five volunteer students of the Masters degree at University of Sialkot (Pakistan), to label messages. Each message was labeled by two different students, that worked separately: in the case two different labels were assigned to the same message, the message was discarded from the data set. This way, only messages labeled with the same class by two different students were considered: messages that did not clearly talk about one of the selected topics were not considered. Hereafter, we list the topics considered as classes and the criteria adopted for labeling messages with each single topic. * _Business_: Messages talk about stocks, business activities, oil prices, Wall Street and companies' shares. * _Health_: Messages that talk about disease, medicine, surgeries, viruses, hospitals and related arguments are included in this topic. * _Politics_: Messages regarding elections, democracy, government and its policies are included in this topic. * _Entertainment_: Messages regarding show business, movies, music, TV shows and similar arguments belong to this topic. * _Sports_: Messages regarding all kinds of sports, athletes, matches and sports tournaments belong to this topic. * _Technology_: Messages talk about new tech, gadgets, software and related information. * _Weather_: Messages talk about weather, rains, storms and weather forecasting. The list of topics was inspired by [49], that proposed a list of categories for classifying sensitive tweets; we did not consider all the list proposed in [49], because some of the proposed categories did not denote topics that users would use to label messages (e.g., racism); we selected and integrated those that, presumably, could be often used by users. Table 3 provides a sample message for each one of the topical classes. In order to have a homogeneous distribution among classes, the training set \(TR\) contained 3500 messages for each class, while the test set \(TE\) contained 1500 messages for each class. Consequently, the training set \(TR\) contained 24,500 messages, while the test set \(TE\) contained 10,500 messages; the total number of messages was 35,000, that constitute the input for the investigation framework. All messages were written in English. Remember that messages in the test set \(TE\) were labeled by hand as well, in order to allow module M3 to automatically compute accuracy. To evaluate the results, the investigation framework computed accuracy, precision, recall and F1-measure. These measures are typical metrics adopted in information retrieval. Since we operated in a context of multi-label classification, we adopted the definitions reported in [50; 51]. Given a set \(C=\{c_{1},c_{2},\ldots,c_{n}\}\) of class labels, for each class \(c_{j}\) we define the following counts:  
* \(TP_{j}\) (true positives) is the number of items correctly assigned to class \(c_{j}\);
* \(FP_{j}\) (false positives) is the number of items incorrectly assigned to class \(c_{j}\);
* \(FN_{j}\) (false negatives) is the number of items incorrectly not assigned to class \(c_{j}\);
* \(TN_{j}\) (true negatives) is the number of items correctly not assigned to class \(c_{j}\). For each class \(c_{j}\), we can define the four above-mentioned metrics.