# MM-ConvBERT-LMS: Detecting Malicious Web Pages via Multi-Modal Learning and Pre-Trained Model
## 4 Experiment and Analysis
### Comparative Experiment
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Methods** & **Models** & **Accuracy** & **Precision** & **Recall** & **F1 Score** \\ \hline \multirow{4}{*}{Machine Learning} & NB & 89.91 & 97.21 & 82.34 & 89.16 \\  & SVM & 91.15 & 95.55 & 86.47 & 90.78 \\  & KNN & 83.99 & 91.03 & 75.69 & 82.66 \\  & DT & 83.25 & 84.40 & 81.90 & 83.13 \\  & RF & 91.03 & 92.23 & 89.76 & 90.98 \\ \hline \multirow{4}{*}{Deep Learning} & Bi-LSTM & 93.60 & 96.73 & 90.36 & 93.43 \\  & Bi-GRU & 93.75 & 96.37 & 91.03 & 93.62 \\  & TextCNN & 95.02 & 96.38 & 93.63 & 94.99 \\  & DPCNN & 93.89 & 94.73 & 93.06 & 93.88 \\  & ALBERT \({}^{\star}\) & 96.93 & 97.06 & 96.85 & 96.95 \\  & BERT & 97.47 & 97.65 & 97.32 & 97.49 \\  & ConvBERT & 98.06 & 98.29 & 97.86 & 98.07 \\  & MM-ConvBERT-LMS(ours) & **98.72** & **98.90** & **98.55** & **98.73** \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Experimental results based on multi-modal data. samples tends to contain meaningful, regular words, while negative samples tend to contain numbers more frequently, and strings in the body part rarely represent complete semantic information. To precisely observe the improvement in the effectiveness of the multi-modal model compared to the single-modal model, we plotted the performance gain of each model on different types of features, as shown in Figures 8 and 9. The results demonstrate a clear improvement in accuracy when using multi-modal features, with an average increase of 0.44% and 11.29% compared to models based on URL and HTML data, respectively. With the exception of random forest and decision tree models, which exhibit a performance decrease after incorporating multi-modal features, all other models can aggregate and analyze information from different perspectives, leading to meaningful representations and reliable decision-making. To observe the reliability of classification for each model, we select the SVM, CNN, original ConvBERT, and MM-ConvBERT-LMS models, which exhibit better performance and greater variability in principle, as representatives and visualize their ROC and PR curves, as shown in Figure 10. The AUC scores of the MM-ConvBERT-LMS model outperform all three baselines, which proves that the model has better classification ability and robustness. Moreover, we plotted the ROC and PR curves of the models trained based on different data, as shown in Figure 11. The multi-modal model displayed superior performance compared to the single-modal model in both ROC and PR curves, reaffirming the effectiveness of our proposed approach. Figure 8: Accuracy improvement of multi-modal models compared to pure URL-based trained models. Figure 9: Accuracy improvement of multi-modal models compared to pure HTML-based trained models. While the above experiments were conducted with a sufficient number of training samples, labeled data in real-world scenarios is often extremely limited. As a result, we also explored the impact of the size of the training set on the model's effectiveness. The results, presented in Figure 12, show a decreasing trend in accuracy as the size of the training set is reduced. However, even when the training set is only 5% of the test set size, the MM-ConvBERT-LMS model outperforms the compared models and the single-modal model, with an accuracy exceeding 93%. In contrast, the TextCNN model tends to overfit more easily than other models and its accuracy is even lower than that of the SVM model. These results suggest that the use of multi-modal models can be beneficial for scenarios with limited labeled data. Figure 11: ROC and PR curves to compare the performance of single-modal and multi-modal ConvBERT models. Figure 12: Effect of dataset size on detection effectiveness. **Left**: Comparison between multimodal models. **Right**: Comparison between single-modal and multimodal models. Figure 10: ROC and PR curves to compare the performance of different multi-modal models. [PERSON] [18] et al. have shown that unbalanced data distribution is another challenge for real-world malicious web page detection tasks.