# Efficacy Improvement of Anomaly Detection by Using Intelligence Sharing Scheme
## 5 Experimental Evaluation and Discussion
### Global Reputation Table Evaluation  
In this section, the classification performance of four global reputation tables produced by the four algorithms proposed in Section 4.2 are evaluated and compared. All four of the algorithms use average as their combination function; their normalization functions and ways of dealing with missing values differ. The evaluated algorithms are listed below. Figure 5: Number of flows counts per network according to the number of company’s identification ID. \begin{table}
\begin{tabular}{c c} \hline \hline
**Datasets Description Collected Over** & **Experimental Evaluation Inclusive** \\ \hline Number of networks & 310 \\ Total number of flows & 12,932,245,944 \\ Total number of malicious flows & 394,951 \\ Proportion of malicious flows & 1:32,744 \\ \% of HTTP(S) connects in background flows & 43.2\% \\ \% of HTTP(S) connects in malicious flows & 7.3\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Characteristics of the datasets used for experimental evaluation. 1. average: no normalization is performed;
2. norm-isect: \(\frac{x-u}{\sigma}\) normalization on the intersection of hostnames (list-wise deletion);
3. norm-all: \(\frac{x-u}{\sigma}\) normalization on all observed hostnames (pairwise deletion);
4. pairwise: pairwise optimization of normalization error. From now on, these algorithms will be referred by their identifications as specified above. For the norm-isect algorithm, there is need of a meaningful intersection of all networks. This is not the case when all of the networks in the dataset can be used. There are even networks that do not have any network flows in the training portion of the dataset, so the intersection is empty. Therefore, we need to use only a subset of networks for norm-isect to be applicable. A simple way to choose a reasonable set of networks is to choose the \(n\) networks with the largest numbers of unique hostnames. The sizes of intersections of the biggest \(n\) networks are visualized in Figure 6. In the given figure, the horizontal axis indicates the number of networks in an intersection (the networks with the largest number of hostnames are taken), and the vertical axis indicates the number of hostnames in the intersection on a logarithmic scale. A total of 100 networks were used, which corresponds to the intersection size of 1054 hostnames. Based on this figure, there is no immediately obvious number of companies that should be chosen. To strike a balance between calibration precision and utility, we arbitrarily choose to use 100 networks; their intersection contains 1054 unique hostnames. The same set of 100 networks is used for all four algorithms in order for their ROC curves to be comparable. The resulting global reputation score ROC curves are shown in Figure 7. The global reputation table of ROC curve comparison on the biggest 100 networks and the corresponding AUC values are listed in Table 3. The global reputation table of AUC score comparison on the biggest 100 networks shows that the best-performing algorithms are average and pairwise, which are nearly tied in their AUC scores and ROC curve shapes. The average algorithm delivered the best performance and pairwise occupies a close second place. Of the other two algorithms, norm-all is in third place and norm-isect clearly performed worst. Figure 6: Network intersection sizes. As expected, norm-isect did not perform particularly well. It was able to exploit only a tiny fraction of the available information to perform normalization: only 1054 hostnames were used compared to roughly 6.1 million hostnames utilized by norm-all. This probably caused substantial noise in the estimation of normalization parameters, degrading the ensemble performance. While norm-all exploits all of the available data, it suffers from a different problem: it standardizes every network output in isolation. The sets of observed hostnames are different for different networks, which then translates to different statistical characteristics of the individual network anomaly detection outputs.