# A Heterogeneous Machine Learning Ensemble Framework for Malicious Webpage Detection
## 4 Experiment and Result
### Prediction Accuracy of the Ensemble Technique  
We estimated the prediction accuracy by applying ensemble techniques to the classification results from the machine learning selections. This includes three methods: hard voting, soft voting, and weighted soft voting, and their performances are summarized in Table 4. In hard voting, a model with a majority vote wins. In soft voting, the probabilities given by each model for each class are summed, and the greatest sum of probabilities wins. In weighted soft voting, predictions are weighted by the model's importance and summed.  
When we applied the ensemble techniques, the highest precision was observed in weighted soft voting. Because both precision and recall had good records; the F1-score was also good. In terms of accuracy, we obtained the best results for hard voting. However, the difference in F1-score and accuracy between weighted soft voting and hard voting was only 0.0115 and 0.0139, respectively.  
The poorest result of accuracy was found in soft voting, with a difference of 0.1403 compared with weighted soft voting. In terms of precision, the difference was 0.14. Poor performance in soft voting was mainly due to a substantial gap between the prediction classification assessments in different machine learning models. As shown in Table 3, the prediction classification of an SVM was 60%, which led to a lower probability of soft voting.  
Figure 6 shows the ROC curve of the weighted soft voting model. The AUC value was 0.924, indicating that the TP and FP ratios were not biased and high detection accuracy was confirmed. The red-dot line in Figure 6 represents the worst case.  
An experiment was performed with AdaBoost, a different ensemble technique, to compare the performance of the proposed ensemble model. Table 5 presents the results of the AdaBoost experiment. In conclusion, the accuracy performance of the hard voting method was the highest with 0.9548, and the weighted soft voting method showed high precision with 0.9587, recall 0.9587, and F1-score 0.9587 for the remaining performance indicators. The proposed method dynamically adjusts weights by reflecting the characteristics of each model and thus exhibits a high accuracy performance compared to other methods. However, as aforementioned, there is a disadvantage that depends on the performance of the sub-model. Therefore, the proposed soft voting ensemble technique based on dynamic weighting is suitable for malicious web detection.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Ensemble Method** & **Precision** & **Recall** & **F1-Score** & **G-Mean** & **Accuracy** \\ \hline Hard Voting & 0.955 & 0.9549 & 0.9573 & 0.9549 & 0.9548 \\ \hline Soft Voting & 0.8286 & 0.8284 & 0.8284 & 0.8285 & 0.8284 \\ \hline Weighted Soft Voting & **0.9686** & **0.9689** & **0.9688** & **0.9687** & **0.9687** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance of the Voting Ensemble Techniques based on Dynamic Model Weight.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Ensemble Method** & **Precision** & **Recall** & **F1-Score** & **G-Mean** & **Accuracy** \\ \hline AdaBoost & 0.9526 & 0.9504 & 0.9516 & **0.9516** & **0.9516** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Performance of the AdaBoost Ensemble Techniques.