# APT-Attack Detection Based on Multi-Stage Autoencoders
## 4 Proposed Methodology
### Data Pre-Processing--Statistical Analysis  
In ML, large amounts of data are analysed to identify common patterns between data records. For this purpose, statistical analysis is conducted to derive valuable insights from the raw data records. In this case, the analysis is applied to check the plausibility of features and to assess their suitability for anomaly detection. Subsequently, suitable features are selected and used in conjunction with individual detection methods. The statistical analysis includes the following steps:  
* Descriptive analysis: Determination of statistical metrics for the plausibility check of feature values.
* Histograms and boxplots: Visualisation and comparison of statistical distributions of normal flows and attacks in datasets.
* Correlation analysis: The examination of features to identify groups of highly correlated or identical features.
* Principal component analysis (PCA): Estimation of the extent of possible dimensionality reduction for individual features.  
In this paper, a statistical analysis was conducted on the combined--i.e., the merged Contagio and CICIDS2017--datasets. This analysis resulted both in the reduction in feature vectors and the reduction in the number of features. In particular, 620 of the feature vectors containing negative values were removed. Furthermore, the feature pool was reduced from 78 to a subset of 54 numerical features as a result of the analysis.  
As a matter of fact, the initial analysis of 78 original features revealed 8 features containing only zero values, 3 features containing higher amounts of implausible negative data, and 2 features containing infinity values. Next, a correlation analysis revealed 7 redundant features. Thus, 20 features were excluded from further evaluation, leaving 58 features for further analysis. Subsequently, including boxplots identified suspicious data in 4 features, which were then excluded from further anomaly detection. In concrete terms, the affected features exhibit a connection between the feature values and the day of the week on which the data was recorded. However, this observation stands in strong contrast to the network behaviour described by the remaining features. This inexplicable relationship is most apparent in a boxplot for the \"Idle Max\" feature, which depicts a linear increase in values from Monday to Friday, as shown in Figure 1.  
Figure 1: Boxplot of the feature “Idle Max”.  
Furthermore, the feature distributions of both test data sets--independent (\"independent\") and completely unknown (\"zero-day attack\")--were compared with one another. In this case, a distinction was made between benign data (\"benign\"), background attacks from CICIDS2017 (\"attack\") and the inserted APT attacks (\"APT\"). For some features, especially for the data of the background attacks, clear differences in statistical distributions can be recognized (for example, the boxplots framed in yellow in Figure 2). In fact, the results illustrate the difficulty of operating reliable anomaly detection for completely unknown data or application scenarios: changed feature distributions, as they exist for the \"zero-day attack\" test data set, pose a great challenge for detection algorithms, which usually results in a weaker detection performance for such data.