# Research on the Construction of Malware Variant Datasets and Their Detection Method
## 4 Pretrained Model of Malware Detection Based on BERT  
### Construction of the BERT Pretrained Model  
A pretrained model is an application of transfer learning. It can use as much training data as possible to extract as much common information as possible so that the commonness can be transferred to the model of specific tasks and then fine-tuned with a relatively small amount of annotation data in relevant specific fields so that the downstream tasks can start from commonness. Furthermore, the model only needs to learn the characteristics of the task itself and the knowledge of a specific task. In deep learning models, the dependence on the amount of data is usually high, and a large number of parameters need to be learned. If the amount of data is small, it will inevitably lead to poor model overfitting and generalization. This pretrained model can make full use of a wide range of network resources and does not need to manually mark the data to obtain general knowledge after a large amount of data pretraining. Then, a small amount of annotation data on the target task is used for fine tuning so that the fine-tuned model can deal with the target task well. BERT is a pretrained language expression model that is trained on a large-scale unlabeled corpus through a two-way transformer encoder to integrate the deep two-way expression of left and right context information to realize the deep two-way representation of sentences to obtain the text vector containing a large amount of semantic information. When building the BERT pretrained model, it is necessary to establish a word table through word pieces. One of the main implementation methods of word piece is called BPE (byte pair encoding). Generally, the process of BPE can be understood as the re-splitting of a word, which makes the expression of the word more concise and the meaning clearer to reduce the size of the word list without losing semantic features. The word splitting process is shown in Figure 7.  
The essence of the BERT pretrained model is unsupervised training, so an index is needed to quantify the training effect. BERT will randomly mask out part of the data when initializing the data, then predict the masked language model, and select sentence A and sentence B to predict whether the two sentences are connected, that is, next sentence prediction, to quantify the effect. The specific training process is shown in Figure 8. First, the thesaurus is built through word pieces and API sequences. After building the input vector, it is input into the transformer encoder layer of the BERT model for training, and then the pretrained model is obtained. Figure 7: Word splitting process. ### Construction of the BERT Pretrained Model  
With the recent development of natural language processing (NLP) techniques, printable strings became more effective to detect malware [32]. Considering the outstanding performance of the BERT model in the NLP field, in this paper, we used BERT to detect malware. The classification task based on BERT needs a pretrained model based on a malware API sequence, but there is no public BERT model in the malware field, so the pretraining of the model is needed. In this paper, the open malware API call sequence dataset provided by reference [25] was used for unsupervised training of the pretrained model. The main work was divided into two parts. The first is to build a thesaurus. When processing pretraining data, many API sequences end with A, W, ExA and ExW. A means using ANSI coding as the text coding of standard input and output streams, W means using Unicode as the coding, Ex means the expansion of parameters and functions, and ExA and ExW are the combination of A, W and Ex. This part of the data can be regarded as the suffix of API call words. The functional semantics represented by relevant APIs are similar, so word pieces can be used to split them. For API words similar to DnsQuery_W, BERT will think_is the link symbol of a word to represent a phrase, so the relevant underlying information should also be removed.