# Unsupervised Feature Selection for Outlier Detection on Streaming Data to Enhance Network Security
## 5 Evaluation
### Test Environment  
Experiments were conducted on a virtualized Ubuntu 20.04.1 LTS equipped with 12 x Intel(R) Xeon(R) CPU E5-2430 at 2.20 GHz and 32 GB memory running on a Proxmox server environment. All programs are coded in Python 3.7. Unless otherwise stated, commonly used Python libraries, e.g., numpy, are used for instance with the argsort function applying Merge Sort to sort the feature scores. We implemented ufssod_clustering in Python according to CKmeans.1d.dp [58] with only the relevant functions for BIC and the one-dimensional cluster-routine and adapted it with respect to Section 4.3. As of now, however, we did not implement the return of additional clusters based on the cluster center distance verification. For FS, we integrated the code ([[https://github.com/takuti/stream-feature-selection](https://github.com/takuti/stream-feature-selection)]([https://github.com/takuti/stream-feature-selection](https://github.com/takuti/stream-feature-selection)), accessed on 11 February 2021) for FSDS [28] and implemented IBFS according to the original paper [46] in the code ([[https://github.com/mgckind/iso_forest](https://github.com/mgckind/iso_forest)]([https://github.com/mgckind/iso_forest](https://github.com/mgckind/iso_forest)), accessed on 11 February 2021) of iForest. UFSSOD has been implemented according to Section 4.3 in Python as well. For online OD on SD, we have chosen six off-the-shelf ensemble algorithms. Most of them have been shown to outperform numerous standard detectors [59]. The Python framework called PySAD ([[https://pypi.org/project/pysad/](https://pypi.org/project/pysad/)]([https://pypi.org/project/pysad/](https://pypi.org/project/pysad/)), accessed on 19 February 2021) (Python Streaming Anomaly Detection) proposed in [60] is used to provide multiple implementations for online/sequential anomaly detection. It contains among others RS-Hash [61], HS-Trees [62], iForest_ASD [53], Loda [52], Kitsune [63] and xStream [14]. However, some of them are not yet fully implemented, e.g., iForest_ASD, or their offline (batch) implementation is included, taken from PyOD [64], rather than their online counterparts as for Loda and xStream. Thus, we integrated RS-Hash, HS-Trees, and Kitsune from PySAD and used our own Python implementation of Loda\({}_{\text{Two\,Hist.}}\). We  
Figure 4: Visualization aid of ufssod_clustering with four exemplary clusters and semicircles around the cluster centroids. have avoided the use of the Rozenholc formula, stated by [52], to obtain the optimum number of bins per histogram and used default numbers of histograms and bins for all runs. Furthermore, iForest_ASD from scikit-multiflow [11] and the iMForest implementation ([[https://github.com/bghojogh/iMondrian](https://github.com/bghojogh/iMondrian)]([https://github.com/bghojogh/iMondrian](https://github.com/bghojogh/iMondrian)), accessed on 19 February 2021) provided along with [48] was added for the online case. However, iForest_ASD was not included in the measurements due to (i) the lack of properly setting the drift threshold depending on the used datasets, which led to intense time-consuming measurements, and (ii) our desire to rely on the more advanced and recent iForest streaming competitor iMForest. xStream was taken from the StreamAD library ([[https://pypi.org/project/streamad/](https://pypi.org/project/streamad/)]([https://pypi.org/project/streamad/](https://pypi.org/project/streamad/)), accessed on 19 February 2021) providing the streaming version rather than the static one, as with PySAD. Unless otherwise stated, the default hyperparameters of the algorithms have been used and outlier thresholds have been fixed for all measurements. In order for the classifiers not to start their classification with empty models, all measurements are first initialized by processing 40% of the same input data later used for classification. This approach seems legitimate to us since we do not focus on the actual performance of each individual online OD classifier but rather want to evaluate the impacts of FS to them.