# A Heterogeneous Machine Learning Ensemble Framework for Malicious Webpage Detection
## 4 Experiment and Result
### Classification Results by Machine Learning Models  
We performed a classification of the dataset with six machine learning models: SVM, decision tree, random forest, XGBoost, logistic regression, and convolutional neural networks. The learning data accounted for 80% of the total dataset, and the remaining dataset was used for the validation. For learning and validation purposes, malicious and benign webpages were prepared in the same ratio. The data features consist of 26 vectors: 11 lexical-based features, 8 malicious features, 5 HTML/JS-base features, and 2 domain-based features. These datasets resulted in classification performance for each machine learning model, as summarized in Table 3.  
Table 3 above shows the precision (expressed in a percentage) of each machine learning model in descending order: light gradient boosting machine (0.9546), random forest (0.9511) is in the first place, followed by XGBoost (0.9499), logistic regression (0.9337), decision tree (0.9162), 1D-CNN (0.804), and SVM (0.6096).  
The highest difference between precision and accuracy was for logistic regression, which recorded a value of 0.03. This was mainly attributable to the parameter properties due to alternating model selections. However, an SVM recorded a very low level of performance (approximately 60%). This poor performance was due to sparse features, which made it difficult to find a hyperplane pattern against the Kernel-SVM. The malicious web detection dataset structure is not compatible with the SVM kernel. In addition, when the scale or format of feature values is the same and the number of data samples is small (10,000 or less), an SVM can show good performance regardless of the data dimension [28]. The dataset used in this study consists of heterogeneous features and the number of samples is approximately 100, 000. Therefore, it is considered that the SVM result is low.  
In addition, we performed experiments with the high-performance extreme learning machines (HP-ELM) model [29] and light gradient boosting machine (LGBM) [30] to compare the latest models. There are studies applied to this model in the field of malware detection [31]. As a result of the experiment, the F1-score was measured as 0.6946. The type and scale of the features of the dataset are diverse, so it is not suitable for HP-ELM, which is thought to indicate low performance. The LGBM is a tree-based ensemble model.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Type** & **Dataset** & **Learning (80\%)** & **Validation (20\%)** \\ \hline Malicious URL & 57,504 & & \\ \hline Benign URL & 57,492 & & \\ \hline Total & 114,996 & & 114,996 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Dataset Composition.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Precision** & **Recall** & **F1-Score** & **G-Mean** & **Accuracy** \\ \hline Support Vector Machine & 0.6096 & 0.6095 & 0.6093 & 0.60 & 0.61 \\ \hline Decision Tree & 0.9162 & 0.9162 & 0.9161 & 0.91 & 0.91 \\ \hline
**Random Forest** & **0.9511** & **0.951** & **0.951** & **0.95** & **0.95** \\ \hline XGBoost & 0.9499 & 0.9498 & 0.9498 & 0.94 & 0.94 \\ \hline Logistic Regression & 0.9337 & 0.9329 & 0.933 & 0.93 & 0.90 \\ \hline
1D-Convolutional Neural Networks & 0.804 & 0.802 & 0.8018 & 0.80 & 0.80 \\ \hline
**LGBM** & **0.9541** & **0.9546** & **0.9543** & **0.95** & **0.95** \\ \hline HP-ELM & 0.6535 & 0.6774 & 0.6653 & 0.65 & 0.65 \\ \hline
**Average** & **0.8465** & **0.8491** & **0.8475** & **0.84** & **0.83** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Classification Performance by Machine Learning Models.  
Looking at the performance of this model with an F1-score as high as 0.9543, it can be considered that our dataset has a structure suitable for a tree-based model. The RF is thought to exhibit relatively high performance because the influence of outliers or heterogeneous feature values on classification performance is offset through subtree construction and pruning.