# Time Aware F-Score for Cybersecurity Early Detection Evaluation
## 3 Materials and Methods
### Datasets  
For the sake of variety in the experiments developed in this paper, we selected two different datasets. In order to study the outcome under particular circumstances, we choose two where both their nature and particular characteristics differ. The first one, obtained from Kitsune dataset [2], is the OS Scan Attack, which presents some particularities that will be shown later in this section. The second one was specially collected for the eRisk 2017 Workshop [1]. In this case, posts from the website Reddit were included after a selection process and being identified as written by users with depression or users without depression. As for Kitsune OS Scan Attack, an analysis is included next. Kitsune dataset is composed by data traffic from a video monitoring network and includes different attacks performed over the network through several days. It was designed and utilized to evaluate the Intrusion Detection System--Network based (NIDS) described in [2]. In particular we use the OS Scan Attack, which examines the devices connected trying to detect hosts and which operating systems (OS) they are using, to find potential vulnerabilities, and it will be referred as \"Network Attack\" from now on. This dataset is composed by a set of network packets from which a group of engineered features are extracted and tagged as \"Attack\" or \"Normal\". In order to further analyse the traffic, we divided it into bidirectional flows as described in [20; 21], defined as the aggregation of packets with same pair source IP address--destination IP address, source port--destination port and protocol over a defined period of time. To account for the time division, we used timestamp of the packets as a split point using 0.1 s as threshold for the inter-packet time and 1 s as threshold for the flow span as described in [22]. For the experiments performed, we used the features described in [2]. As presented in the original paper only characteristics of the packet itself and its relation in time with the rest of the traffic are used in the creation of the dataset without including specific information of the flow they belong to. Table 1 summarizes the main statistics for the datasets, among which OS Scan Attack from Kitsune dataset can be found. The amount of individual packets reaches nearly 1.7 million, distributed in 75,700 bidirectional flows. As the dataset represents an OS scan attack, most of anomalous flows are going to be around 2 packets in size, like it can be seen in the average of packets per flow for attack class, which accounts for the request and the reply from the device under attack. This, results on getting the majority of \(Entities\) of Anomalous type even if the greater part of \(items\) or packets belong to the Normal class. Figure 1: Items distribution for streaming evaluation. The eRisk depression dataset, referred as \"Depression Dataset\" from now on was expressly collected for the 2017 edition of eRisk workshop on Early Detection [1]. It is composed by a set of publicly available Reddit posts published by users in about a year of use period. Those posts were later marked as \"Depressed\" or \"Non-depressed\" guided by self-reported diagnoses of depression. In this case, subjects correspond with the _Entities_ (\(E\)) and each of the _items_ (\(I_{i}^{e}\)) with the subject's posts. As for the features used in the experiments we will use the ones defined in [23; 24]. Although, we will not include features created by the aggregation of a sequence of posts by only including individual post characteristics. Also, we include in Table 1, along with the ones of the previous described dataset, the main statistics computed for the subjects and posts considered. This dataset is significantly smaller both in terms of number of _Entities_ and _Items_ but it has a higher average ratio of _items_ per _entity_. With only 887 _Entities_ and slightly over 500,000 items, it achieves nearly 600 items for each entity.