# A Method for Processing Static Analysis Alarms Based on Deep Learning
## 5 Experiments and Results
### Research Questions and Findings
#### 5.3.2 RQ2: Effectiveness  
To demonstrate the effectiveness of our approach, we conducted a comprehensive evaluation of four neural network algorithms for classifying context embeddings generated from warning details and code snippets. Table 2 accurately reflects the performance of our approach; it provides a comparison of the accuracy, precision, recall, and F1-score metrics between the four different algorithms. As observed in the table, the CNN model exhibited the highest accuracy and precision compared to the other models, achieving an accuracy of 95.10% and a precision of 96.49%, which indicated its superior performance in correctly identifying actionable alarms. However, the BiRNN model demonstrated superior performance in terms of the recall and F1-score. It achieved a remarkable recall of 98.07% and an impressive F1-score of 96.90%. These metrics highlight the model's ability to effectively recognize actionable alarms while maintaining a favorable balance between precision and recall. Both the BiRNN and CNN models demonstrated promising potential for practical deployment in accurately identifying actionable warnings. The training and testing processes of these two classifiers can be observed in Figure 8.  
Furthermore, we implemented a function called _predict_ to determine whether the warning detail and code snippet pair inputted by the user is an actionable alert or a false alarm. The _predict_ function accepts the warning details and corresponding code snippets as the input from the user. It leverages the pre-trained CodeBERT model to generate context embeddings for the input and then utilizes a neural network classifier to predict the labels based on these context embeddings. By visualizing the ROC curves and calculating the AUC score, we assessed the performance of our proposed approach in accurately identifying actionable warnings. The results, as depicted in Figure 9, demonstrate that the BiRNN model attained the highest AUC score of 0.983, closely followed by the CNN model with an AUC score of 0.976. The attention model exhibited an AUC score of 0.94, while the MLP model yielded a comparatively lower AUC score of 0.847, indicating its inferior performance compared to the other models. \begin{table}
\begin{tabular}{c c c c c} \hline
**Model** & **Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline MLP & 76.38\% & 88.15\% & 72.21\% & 79.39\% \\ Attention & 86.81\% & 93.75\% & 84.60\% & 88.94\% \\ BiRNNs & 93.72\% & 95.77\% & 98.07\% & 96.90\% \\ CNNs & 95.10\% & 96.49\% & 93.50\% & 94.97\% \\ \hline \end{tabular}
\end{table} 
Table 2: Comparison of performance between different models in our experiment. Figure 8: The training and testing processes for BiRNN and CNN classifiers. Table 3 shows the benchmark evaluation results. The comparative experiments in the table were all replicated and validated by the author using the same dataset on different models in the same environment. Our designed BiRNN model, with its ability to capture contextual information from both the forward and backward directions, achieved the highest AUC score in our evaluation, closely followed by our CNN model. [PERSON] at al. [34] presented a classifier based on CNNs for identifying false positive alarms, and it achieved an average precision of 81.72% and an average recall of 73.09%. The convolutional neural network model designed in this study achieved an accuracy of 96.49% and a recall rate of 93.5% in the static analysis alarm classification task after preprocessing. The AUC score was 97.6%, which shows it can effectively identify false positive alarms. Using the support vector machine (SVM) proposed by [PERSON] et al. [35] for the experiments, the recall rate exceeded 87% with AUC scores of over 97% when predicting actionable warnings. While [PERSON] et al. would not recommend DL models, [PERSON] et al. [36] demonstrated that neural networks that learn via source code outperform traditional models. This sparked our interest in attempting the application of DL models for processing static analysis alarms.