# MM-ConvBERT-LMS: Detecting Malicious Web Pages via Multi-Modal Learning and Pre-Trained Model
## 3 Method
### ConvBERT-Based Backbone Network  
To incorporate prior knowledge into the model, we use a pre-trained ConvBERT as the backbone. ConvBERT is an improved BERT-based [32] pre-trained model, which has been optimized in both pre-training methods and model structure to achieve a balance of effectiveness and efficiency. Regarding the pre-training approach, ConvBERT uses a similar training process as Electra [33], both of which refer to the generative adversarial network and use Replaced Token Detection as a pre-training task, the main process of which is depicted in Figure 3.  
During the pre-training process, both the generator and discriminator are trained simultaneously. The generator is a small masked language model (MLM) whose main goal is to reconstruct the masked tokens in the input, whereas the discriminator is a classifier that operates at the token level to identify whether each word in a sentence has been replaced or not. The pre-training objective of the ConvBERT model is shown in Equation (2), where \(G\) represents the generator, \(D\) represents the discriminator, \(\theta\) represents the model parameters, and \(t\) is the position in the input vector marked by the token [MASK]. \(x^{masked}\) represents the sequence after masking and \(x^{corrpt}\) is the input reconstructed by the generator. By conducting large-scale corpus-based pre-training, the model can become more sensitive to  
Figure 3: The pre-training process of ConvBERT. some elements in URL or HTML data that have a specific meaning. This prior knowledge can help improve the accuracy of detection. \[ \begin{split}&\mathcal{L}_{pt}=\underset{\theta_{c}\phi_{D}}{\min} \mathcal{L}_{MLM}(x,\theta_{G})+\lambda\times\mathcal{L}_{Disc}(x,\theta_{D}) \\ &=\mathbb{E}\bigg{(}\underset{i}{\Sigma}-\log pG(x_{i}\big{|}x^{ msnsbd})\bigg{)}\\ &+\lambda\times\mathbb{E}\bigg{(}\underset{t}{\Sigma}-\mathbb{I} (x_{t}^{corrupt}=x_{t})\log D(x^{corrupt},t)-\mathbb{I}(x_{t}^{corrupt}\
eq x _{t})\log(1-D(x^{corrupt},t))\bigg{)}\end{split}  \tag{2}\]  
From a structural perspective, ConvBERT makes significant improvements to the multi-head self-attention mechanism included in the Transformer architecture. This mechanism, which is a core part of the Transformer, is shown in Equation (3), where \(Q\) is the query matrix, and \(K\) and \(V\) correspond to the key matrix and the value matrix, respectively. Through the use of multi-head attention, ConvBERT is able to learn features across multiple representation subspaces, providing a more comprehensive understanding of the input data. As shown in Equation (4), where _head_ represents the input vector projected into the subspace, \(W^{0}\) is the parameter matrix used for projection, and \(d_{k}\) is the dimension of the vector \(K\). The multi-headed self-attention has an expansive perceptual field of view that covers the entire output. This allows it to effectively extract global features, enabling analysis of not only long-term relationships in HTML sequences but also for modeling cross-modal relationships. \[\text{Attention}(Q,K,V)=\text{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V \tag{3}\]  
\[ \begin{split}\text{MultiHead}(Q,K,V)=\text{Concat}(head_{1}, head_{2},\dots,head_{n})W^{0}\\ \text{where}\;head_{i}=\text{Attention}(QW_{i}^{0},KW_{i}^{K}, VW_{i}^{V})\end{split}  \tag{4}\]  
However, for URL data, its local features are more useful for the detection task. For example, adjacent elements can form words with independent semantics, which in turn reflect the characteristics of the corresponding web page. Lightweight convolution is shown to compensate for the lack of local information modeling by multi-headed self-attentiveness, which works as shown in Equation (5), where \(X\) is the input vector, W represents the weights of the convolution kernel, \(i\) represents the position of the feature map currently being processed, and \(K\) represents the size of the convolution kernel. The improved dynamic convolution uses variable convolution kernel parameters and thus enhances the diversity of the representation.