# MM-ConvBERT-LMS: Detecting Malicious Web Pages via Multi-Modal Learning and Pre-Trained Model
## 4 Experiment and Analysis
### Ablation Experiment
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Modules** & **Model** & **Accuracy** & **Precision** & **Recall** & **F1 Score** \\ \hline  & MM-ConvBERT-LMS & 98.72 & 98.90 & 98.55 & 98.73 \\ \hline \multirow{4}{*}{The input modules} & MM-ConvBERT-LMS w/o HTML & 96.42 (2.30 \(\downarrow\)) & 94.19 (4.71 \(\downarrow\)) & 99.01 (0.46 \(\uparrow\)) & 96.54 (2.19 \(\downarrow\)) \\  & MM-ConvBERT-LMS w/o URL & 86.20 (12.52 \(\downarrow\)) & 86.81 (12.09 \(\downarrow\)) & 85.63 (12.92 \(\downarrow\)) & 86.22 (12.51 \(\downarrow\)) \\  & MM-ConvBERT-LMS w/o MTE & 98.41 (0.31 \(\downarrow\)) & 98.68 (0.22 \(\downarrow\)) & 98.15 (0.40 \(\downarrow\)) & 98.42 (0.31 \(\downarrow\)) \\  & MM-ConvBERT-LMS w/o & 97.07 (1.65 \(\downarrow\)) & 96.06 (2.84 \(\downarrow\)) & 98.21 (0.34 \(\downarrow\)) & 97.13 (1.60 \(\downarrow\)) \\  & MTE \& HTMLs w/o & 82.50 (16.22 \(\downarrow\)) & 82.46 (16.44 \(\downarrow\)) & 82.92 (15.63 \(\downarrow\)) & 82.69 (16.04 \(\downarrow\)) \\ \hline \multirow{3}{*}{The output modules} & MM-ConvBERT-LMS w/o modal-matching & 98.26 (0.46 \(\downarrow\)) & 98.43 (0.47 \(\downarrow\)) & 98.12 (0.43 \(\downarrow\)) & 98.27 (0.46 \(\downarrow\)) \\  & MM-ConvBERT-LMS w/o LMS & 98.61 (0.11 \(\downarrow\)) & 98.42 (0.48 \(\downarrow\)) & 98.83 (0.28 \(\uparrow\)) & 98.62 (0.11 \(\downarrow\)) \\ \hline Main structure & Two-stream MM-ConvBERT-LMS & 98.63 (0.09 \(\downarrow\)) & 98.90 (0) & 98.37 (0.18 \(\downarrow\)) & 98.64 (0.09 \(\downarrow\)) \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Results of the ablation experiments. MTE is the abbreviation for modal-type encoding. After the change, the result of performance reduction is marked in red, the improvement is marked in blue. \(\uparrow\) means the metric is improved, while \(\downarrow\) means the metric is decreased. After removing the modal-type encoding, it is noteworthy that the model's performance on pure HTML input deteriorates significantly, and it tends to rely more on easily learnable URL features. This suggests that the model treats URLs and HTML as the same type of information flow during training, and under the influence of the attention mechanism, the model considers URLs with more salient features as meaningful information while regarding the HTML part with difficult-to-explore features as noisy data, ultimately leading to the neglect of HTML features. However, when using modal-type encoding, ConvBERT can effectively distinguish between HTML and URLs as two different types of data, and the modal matching task can further ensure that the model does not overlook any features. Therefore, the MM-ConvBERT-LMS model not only yields satisfactory results on multimodal data but also proves to be effective in handling modal-missing samples. The detection accuracy of the two-stream network is slightly lower than that of the single-stream structure. One main reason is the insufficient attention-based modal interaction between the two-stream network's two inputs, where feature fusion occurs only near the output. This limitation results in the model's inability to fully explore the correlation between the two modalities. Meanwhile, the single-stream network has a significant advantage in terms of reducing both the training and inference cost, with 50% fewer model parameters and a training time that is approximately 39% shorter. Therefore, a single-stream network is more cost-effective. The above experiments indicate that pre-trained models may outperform those trained from scratch. However, the findings are inconclusive, possibly due to the inconsistent parameter sizes between the two types of models, and the possibility that better results achieved with pre-training may also be due to its network architecture or the number of parameters. Therefore, we establish baselines with a model without pre-trained weights (No PTW) and models with frozen layers and observe the impact of freezing different layers on the model's detection performance.