# MM-ConvBERT-LMS: Detecting Malicious Web Pages via Multi-Modal Learning and Pre-Trained Model
## 5 Conclusions  
In this paper, we propose a novel approach to address the limitations of current methods in detecting malicious web pages, such as lack of flexibility, insufficient feature extraction, and high  
Figure 16: Effect of pre-trained weights on the detection effect. The area of the circle represents the training time.  
rates of false positives and false negatives. Our proposed approach is an efficient model based on multi-modal learning and a pre-trained model. In terms of model structure, the ConvBERT-based backbone network can represent the local and global information of the input features through the mixed attention module. By adding the modal-type encoding, this model can avoid feature confusion and other problems, and enhance its capacity to analyze multi-modal features. On the other hand, the coarse-grained modal matching task is a simple yet efficient auxiliary training method that helps the model learn the relationship between two modalities. The optimization objective based on LMS also enables the model to learn larger classification boundary distances and improve robustness. Experimental results show that our proposed model achieves a detection accuracy of 98.72%, outperforming the baseline methods. Thus, our approach shows promise in the field of malicious web page detection.  
However, there are still certain aspects that require improvement:  
(1) **More adapted vocabulary and tokenizer**. In this paper, we use the vocabulary and tokenizer used in the ConvBERT model pre-training process to pre-process the input data. However, as these pre-training tasks are built on a regular corpus of text, the vocabulary may lack feature elements in HTML or URLs. Therefore, it is critical to improving the vocabulary and tokenizer to further enhance the model's effectiveness.  
(2) **A more complex yet realistic dataset**. In this paper, the multi-modal samples detected are those where both URL and HTML are malicious. Nevertheless, in a real-world scenario, only one of the HTML or URLs may be malicious, while the other features could be normal. Therefore, collecting such data and designing fine-grained modal matching tasks for these scenarios can broaden the scope of the proposed approach's application. It is essential to consider this complexity to evaluate the model's effectiveness in a more realistic setting.  
In the next step of our research, we will not only focus on enhancing the detection performance of the model but also improve its robustness against adversarial sample attacks [38] without affecting the detection accuracy. Furthermore, techniques such as model distillation [39] and pruning will be explored to improve the operational efficiency and practical performance of the model in real-world scenarios. Additionally, by integrating the model into browser plug-ins, we hope to provide real-time protection to users.  
Methodology, X.T., B.J. and J.W.; Validation, Y.Y.; Writing--review & editing, Q.S. and Y.W. All authors have read and agreed to the published version of the manuscript.  
This research was funded by National Social Science Foundation Key Project (20 AZD114) and the National Key Research and Development Program of China (2022 YFB3103200).  
Not applicable.  
All datasets are publicly available.  
The authors wish to express their appreciation to the reviewers for their helpful suggestions which greatly improved the presentation of this paper.  
The authors declare no conflict of interest.