# Evaluating Ensemble Learning Mechanisms for Predicting Advanced Cyber Attacks
## 5 Evaluation and Discussion
### Discussion  
The findings of this experimental study indicate that ensembles can improve performance and eliminate misclassifications that are present in individual classifiers. The essence of the machine learning algorithm is to search for the most accurate hypothesis in the possible hypothesis space (\(\mathbf{\mathcal{H}}\)), that is, the hypothesis closest to the unknown function (\(\mathbf{\mathcal{F}}\)). When the hypothesis space is large, large amounts of training data are required to limit the search for good approximations. Each training example eliminates any hypothesis in (\(\mathbf{\mathcal{H}}\)), that misclassifies it, or makes it less credible. In binary classification problems, each training example can eliminate half of the hypotheses in (\(\mathbf{\mathcal{H}}\)). Another reason why an ensemble is useful is that the training data may not provide enough information to learn from (\(\mathbf{\mathcal{H}}\)). Most learning algorithms consider a very large hypothesis space, so even after eliminating the hypothesis of misclassifying training samples, there are still many observations. From an existing set of hypotheses, an ensemble of classifiers can be easily constructed and combined using the methods described throughout this study. A third reason for integration is that the hypothesis space (\(\mathbf{\mathcal{H}}\)) may not contain the real function (\(\mathbf{\mathcal{F}}\)). Instead, it can contain several equally good approximations of (\(\mathbf{\mathcal{F}}\)). By using weighted combinations of these approximations, we may be able to represent classes other than (\(\mathbf{\mathcal{H}}\)) [37].  
In the cybersecurity context, network traffic analysis is critical for protection against cyberattacks. Therefore, it is vital to improve the performance of the prediction models  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Ensemble Mechanism** & & **Precision** & **Recall** & **F-Score** & **Support** \\ \hline \multirow{8}{*}{**Accuracy**} & **Normal** & 1.00 & 0.99 & 0.99 & 665,902 \\  & **Attack** & 0.95 & 0.97 & 0.96 & 96,113 \\  & **Accuracy** & & & 0.99 & 762,015 \\  & **Macro avg** & 0.97 & 0.91 & 0.92 & 762,015 \\  & **Weighted avg** & 0.99 & 0.92 & 0.92 & 762,015 \\ \hline \multirow{8}{*}{**Accuracy**} & **Normal** & 1.00 & 1.00 & 1.00 & 665,902 \\  & **Attack** & 0.98 & 0.98 & 0.98 & 96,113 \\ \cline{1-1}  & **Accuracy** & & & 0.99 & 762,015 \\ \cline{1-1}  & **Macro avg** & 0.99 & 0.99 & 0.99 & 762,015 \\ \cline{1-1}  & **Weighted avg** & 0.99 & 0.99 & 0.99 & 762,015 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Detailed Performance Results of the Three Ensemble mechanisms.  
to obtain more accurate detection results and at the same time, false positives should be lower to the minimum. Throughout this research, three heterogeneous ensemble-learning approaches--stacking, boosting, and bagging--are used to combine four ML algorithms. According to our findings, ensemble-learning algorithms are able to achieve greater prediction accuracies than base classifiers. The employment of meta-classifiers by the stacking bagging, and boosting ensembles to reduce errors made during the learning phase led to greater prediction performance. The stacking mechanism can improve the accuracy of base classifiers by increasing the internal diversity of the stacking model and reducing bias and variance. The structure of heterogeneous ensemble-learning models brings more reliable and robust results to support IDS functionality to detect unknown attacks. An overall performance metric across all potential classification criteria is measured by AUC as illustrated in Figure 9.  
The confusion matrices of the three applied mechanisms are displayed in Figure 10, where \"0\" indicates Normal and \"1\" is used for Attack.  
Figure 9: Performance Metrics. (**a**) False Positive Rates-Individual Algorithms, (**b**) Precision vs. Recall-Individual Algorithms, (**c**) False Positive Rates-Ensemble Model, (**d**) Precision vs. Recall-Ensemble Model.