# A Method for Processing Static Analysis Alarms Based on Deep Learning
## 3 Methodology
### CodeBERT Model and Data Representations  
In this study, we selected a representative pre-trained CodeBERT model, which is a bimodal model pre-trained on source code and natural-language descriptions. CodeBERT follows the BERT [29] model and adopts a multilayer bidirectional Transformer as the model architecture, which is composed of 12 layers of Transformer with 12 attention heads, and the size of the representation in each Transformer layer was set to 768. It was pre-trained on CodeSearchNet [30], a large-scale of code corpora collected from GitHub across six programming languages. To generate each training input sequence, we took the concatenation of warning details and code snippets from the corpus, defined as \(warning_{1}=\{w_{1},w_{2},\cdots,w_{n}\}\) and \(code_{1}=\{c_{1},c_{2},\cdots,c_{m}\}\), where \(n\) and \(m\) denote the lengths of two segments, respectively. The two segments were always connected by a special separator token [SEP]. The initial and final tokens of each sequence were consistently padded with a special classification token [CLS] and a concluding token [EOS], respectively. There is an example in Figure 2 to explain how the warning details and code snippets are normalized and tokenized in vector form.  
In conclusion, the input sequence of each pre-training sample is represented as follows:  
\[token=[\text{CLS}],\underbrace{w_{1},w_{2},\ldots,w_{n}}_{warning_{1}}[ \text{SEP}],\underbrace{c_{1},c_{2},\ldots,c_{m}}_{code_{1}}[\text{EOS}].\]  
As shown in Figure 3, we propose an innovative method that takes warning details and code snippets as input sequences and then utilize the CodeBERT model to convert  
Figure 2: The warning details and code snippets are normalized and tokenized in vector form.  
them into a suitable format for further analysis. The tokenizer parses the input sequences into individual tokens, which allows the model to process the input in a more efficient and structured way. Once the input has been tokenized, each token is subsequently transformed into a vector through word embedding. Word embedding is a way of representing words as high-dimensional vectors, where each dimension represents a different semantic feature of the word. Then, we perform positional encoding on the input embeddings, which enables the model to more effectively capture the relative position information of each token within the original input sequence. After position encoding, the input sequence can be passed through multiple Transformer layers. Each Transformer layer typically contains a multi-head attention mechanism and a feedforward neural network, both of which help the model capture the semantics of the input sequence. In addition, the Transformer introduces layer normalization operations on the basis of residual connections to improve the performance and stability of the model. After passing through multiple Transformer layers, we obtain context embeddings, which are high-dimensional vector representations of the input sequence. The context embeddings are obtained by passing the output of the final Transformer layer through a pooling operation. The pooling operation aggregates the output vectors of the final layer into a single fixed-length vector, which captures the most relevant information about the input sequence. Once the context embeddings are obtained from the CodeBERT model, we can use them as input features for neural networks to predict whether alarms are actionable warnings.  
Figure 3: The process of generating context embedding for code snippets and warning details based on pre-trained CodeBERT model.