# MM-ConvBERT-LMS: Detecting Malicious Web Pages via Multi-Modal Learning and Pre-Trained Model
## 4 Experiment and Analysis
### Comparative Experiment  
Methods from both statistical machine learning and deep neural network categories were selected as the baselines for the experiments. For statistical machine learning methods, Naive Bayesian (NB), Decision Tree (DT), Random Forest (RF), Support Vector Machine (SVM), and K-Nearest Neighbors (KNN) were selected as comparison models. For deep learning methods, in addition to comparing with commonly used Text Convolutional Neural Network (TextCNN), Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), and Deep Pyramid Convolutional Neural Network (DPCNN), we also selected several pre-trained models as a baseline, including BERT, ALBERT [37], and the original ConvBERT. In our experiments, we evaluated the performance of each model on three datasets: pure HTML data, pure URL data, and multi-modal dataset (HTML + URL). The experimental results are presented in Tables 3-5, respectively. The optimal results are marked in black bold. According to the results, deep learning methods clearly outperform statistical machine learning model approaches, with several deep neural networks achieving accuracy rates of more than 90%. Meanwhile, among all the neural network models, the pre-trained methods outperform classical neural network models trained from scratch, demonstrating the benefits of using dynamic word vectors as prior knowledge. The MM-ConvBERT-LMS model even achieved the best results across all three experiments, achieving 98.72% accuracy on the task of detecting malicious web pages based on multi-modal features, and also outperforming the baselines in terms of precision, recall, and F1 score. Figure 7: In our experiment, the meanings of TP, TN, FP, and FN. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Methods** & **Models** & **Accuracy** & **Precision** & **Recall** & **F1 Score** \\ \hline \multirow{4}{*}{Machine Learning} & SVM & 76.39 & 77.56 & 74.80 & 76.15 \\  & KNN & 74.35 & 77.49 & 69.21 & 73.12 \\  & DT & 77.83 & 79.10 & 76.13 & 77.59 \\  & RF & 84.24 & 82.61 & 87.06 & 84.78 \\ \hline \multirow{4}{*}{Deep Learning} & Bi-LSTM & 75.67 & 75.94 & 75.71 & 75.83 \\  & Bi-GRU & 76.68 & 79.67 & 72.14 & 75.72 \\  & TextCNN & 85.68 & 83.78 & 88.77 & 86.20 \\  & DPCNN & 83.89 & 85.64 & 81.75 & 83.65 \\  & ALBERT & 85.65 & 86.25 & 85.10 & 85.67 \\  & BERT & 86.45 & 86.81 & 86.21 & 86.51 \\  & ConvBERT & 87.13 & **87.02** & 87.52 & 87.27 \\  & ConvBERT-LMS(ours) & **87.45** & 86.71 & **88.69** & **87.69** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Experimental results based on HTML data. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Methods** & **Models** & **Accuracy** & **Precision** & **Recall** & **F1 Score** \\ \hline \multirow{4}{*}{Machine Learning} & NB & 88.72 & 97.52 & 79.64 & 87.68 \\  & SVM & 89.63 & 95.27 & 83.57 & 89.04 \\  & KNN & 84.11 & 92.28 & 74.72 & 82.58 \\  & DT & 89.41 & 93.51 & 84.88 & 88.99 \\  & RF & 91.37 & 94.62 & 87.88 & 91.12 \\ \hline \multirow{4}{*}{Deep Learning} & Bi-LSTM & 92.49 & 94.97 & 89.86 & 92.34 \\  & Bi-GRU & 92.18 & 93.50 & 90.79 & 92.13 \\  & TextCNN & 92.70 & 94.93 & 90.34 & 92.58 \\  & DPCNN & 92.38 & 94.10 & 90.56 & 92.30 \\  & ALBERT & 95.70 & 95.84 & 95.62 & 95.73 \\  & BERT & 96.95 & 97.08 & 96.87 & 96.97 \\  & ConvBERT & 97.63 & 97.66 & **97.64** & 97.65 \\  & ConvBERT-LMS(ours) & **97.76** & **97.93** & 97.62 & **97.77** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Experimental results based on URL data.