# Insider Threat Detection Using Machine Learning Approach
## 3 Materials and Methods
### Dataset  
The lack of actual data is a major barrier for researchers studying the insider threat problem. These data involve log files that contain private user information [3]. In order to protect their users and assets, organisations frequently deny researchers access to real data. However, under specific regulations, an organisation may agree to give the researchers restricted access after anonymizing the private and confidential attributes of the data.  
This problem makes it difficult for the researchers to continue their work. In order to solve the insider threat detection problem, it is, therefore, preferable to use synthetic data in the system's design and evaluation. The DARPA ADAMS [48] and Schonlau datasets [49], among others, have been used in research papers in the past, even though they were not specifically created to address the insider threat issue. However, these datasets are less useful due to the complexity of the insider threat problem.  
The CMU-CERT dataset(s) has seen a significant increase in usage for insider threat detection systems over the past decade. The CERT insider threat dataset [50] is a collection of artificial datasets produced by the Community Emergency Response Team (CERT) at Carnegie Mellon University (CMU) [51]. It is widely used in insider threat detection studies [43]. There are several releases of the CERT insider threat dataset, and the most used versions are r4.2 and r6.2. Table 2 shows the two versions' statistics, demonstrating that compared to prior CERT dataset versions, CERT r4.2 has a higher rate of malicious activities.  
Figure 1: Research methodology.  
Thus, we used the CERT r4.2 dataset, which consists of five different events--logon/logoff events, email transmission, devices, files, and HTTP events--as illustrated in Table 3. These events document the activities of 1000 employees in an organisation over a period of 17 months. The dataset has 32,770,222 events from 1000 normal and abnormal users. Experts intentionally injected 7323 malicious insider instances. The CERT dataset contains the psychometric score for each employee, also known as the \"Big Five personality characteristics\". These characteristics are stored in the psychometric.csv file.  
In terms of insider threats, Version r4.2 of the dataset consists of three primary scenarios described as follows:  
1. A user who never worked after hours or used removable drives starts logging in after hours, using a removable drive, uploading information to wikileaks.org, and then leaves the company shortly after;
2. A user begins searching for career opportunities on job search websites and contacting potential employers. Before leaving the office, they use a thumb drive to take data (at a rate noticeably higher than their prior actions);
3. A dissatisfied system administrator downloads a key logger and transfers it to his supervisor's computer using a thumb drive. The following day, he logs into his company's network as his boss and sends out an alarming mass email, causing widespread concern, and immediately leaves the organisation.  
As already indicated, we have focused mainly on the CERT r4.2 dataset because the CERT r4.2 dataset contains a high number of insider threats compared with previous and later versions. Our attention is drawn to the first scenario, which corresponds to a data theft attack, as compared with privilege abuse, privilege escalation, or sabotage attack. We extrapolate data from the files device.csv, http.csv, psychometric.csv, and logon.csv. The description of the scenario includes the following: hours worked (logon.csv), using a removable device (device.csv), and uploading information (http.csv). We exclude file.csv and email.csv due to runtime considerations when using DFS, as outlined in Section 3.3.1.