# Unsupervised Feature Selection for Outlier Detection on Streaming Data to Enhance Network Security
## 5 Evaluation
### Evaluation Methodology  
For the comparison of FSDS and IBFS, as representatives of FS for SD and OD, with UFSSOD, our methodology is shown in a flow chart in Figure 5. For each dataset of Table 3 we have performed extensive measurements by first setting an FS algorithm. It must be noted that FSDS is relying on the number of singular values \(k\). The authors of FSDS state that this parameter should be equal to the number of clusters in the dataset. However, since our focus is on classification rather than on clustering, we have performed measurements with different values for \(k\) to test the applicability of FSDS for OD. After the FS algorithms yield their scores for each feature, the subset is obtained by different \(\gamma\), e.g., the top 25% ranked features, randomly chosen \(\gamma\) features, or automatically proposing \(\gamma\) features by using ufssod_clustering to avoid the top-\(\gamma\)-problem. Each obtained subset has been used by six online capable OD algorithms per measurement yielding \(avg\_t\) and \(F1\) per classifier. Finally, the results have been averaged across 10 independent runs, since most of the methods are non-deterministic, e.g., negatively affected by random projection.  
Since no other unsupervised FS algorithms for OD with regard to a streaming fashion exist, we have tested our conceptualization in the two proposed settings of Figures 2 and 3. Thus, we let UFSSOD compute an outlier score and propose the top-\(\gamma\)-features for each data instance that will immediately be used and processed sequentially by xStream. For  
Figure 5: Flowchart of the evaluation measurements.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**ID** & **Dataset** & **\# Instances** & **\# Dimensions** & **Outliers (\%)** \\ \hline
1 & arrhythmia & 452 & 274 & 14.6 \\
2 & cardio & 1831 & 21 & 9.6 \\
3 & glass & 214 & 9 & 4.2 \\
4 & ionosphere & 351 & 33 & 35.9 \\
5 & letter & 1600 & 32 & 6.25 \\
6 & mnist & 2603 & 100 & 26.9 \\
7 & musk & 1000 & 166 & 9.7 \\
8 & optdigits & 2216 & 64 & 6.8 \\
9 & pendigits & 2000 & 16 & 2.3 \\
10 & pima & 768 & 8 & 34.9 \\
11 & satellite & 3000 & 36 & 31.1 \\
12 & satimage-2 & 1750 & 36 & 1.0 \\
13 & shuttle & 3000 & 9 & 7.9 \\
14 & vowels & 1456 & 12 & 3.4 \\
15 & wbc & 378 & 30 & 5.6 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Characteristics of the utilized and partially adapted datasets from ODDS [65].  
\(\text{Lodat}_{\text{Two Hist.}}\) we have used the parallel approach and let Loda build histograms using the current feature subset while classifying with the old one. The window size was set to 200 samples rather than 256 used by [52] because of the limited number of data instances in the _glass_ dataset. After the window size is reached, the current classifying histograms are replaced with the ones built and the new histograms are built using the currently proposed subset by UFSSOD. The streaming setting was only performed on the datasets with IDs 2, 5, 6, 8, 9, 11, 12, 13, and 14 due to their more meaningful number of data instances for a streaming setting. Since we stick to the default threshold values for most of the classifiers and do not extract the score values but their binary prediction, we have not utilized the scaled combination approach so far. Despite that, we have combined the results of UFSSOD, xStream, and \(\text{Lodat}_{\text{Two Hist.}}\) by a simple logical _or_-conjunction in three settings: UFSSOD with xStream, UFSSOD with \(\text{Lodat}_{\text{Two Hist.}}\) and UFSSOD with xStream with \(\text{Lodat}_{\text{Two Hist.}}\) as a combination of UFSSOD with two downstream applied OD algorithms. Although achieving a higher \(TP\) while also producing more \(FP\), our approach is legitimate under the assumption that it is more important to detect attacks while coping with \(FP\) in a consecutive alert analysis. Albeit not within the scope of this work, it must be noted that most of the classifiers are not able to perform root cause analysis, e.g., Kitsune as stated in [68], which might be relevant for a subsequent alert analysis.