# False Alarm Reduction Method for Weakness Static Analysis Using BERT Model
## 4 Experimental Setup and Evaluating the Proposed Model  
In this study, two deep learning models are used; therefore, it is necessary to separate the Juliet test suite C/C++ 1.3 version dataset used for the first time. A total of 60% and 40% of datasets were split for the BWA model and ACM. Then, for each model, the resulting data were divided into training, validation, and testing data. To train the model, we used PyTorch 1.12.1 with CUDA 11.4 on top of Python 3.10, a machine with 1 Terabyte RAM and 8 NVIDIA Tesla V100 PCIe 32 GB.  
To evaluate the performance of the proposed model, scores were calculated for Precision, Accuracy, F1-Score, and Receiver Operating Characteristic area under the curve (AUC). Table 5 shows the evaluation of the BERT-based Line-level weakness analysis and analysis alarm classification model. For the final model, we report several evaluation metrics, including those used in the initial work for each dataset. The metrics are true alarm (true positive), false alarm (false positive), and false alarm rate (false positive rate). This way, we can perform a comparison; the comparison results are shown in Table 6.  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline
**File Name** & **Line** & **CPC** & **CLG** & **FLF** & **IFR** & **FRC** & **PVS** & **BWA** & **Target** \\ \hline /NULL\_Pointer \_\_ \_01.c & 7 & 1 & 1 & 0 & 1 & 1 & 1 & 0.85 & 1 \\ \hline /NULL\_Pointer \_\_ \_01.c & 18 & 0 & 0 & 1 & 0 & 0 & 1 & 0.25 & 0 \\ \hline /NULL\_Pointer \_\_ \_02.c & 10 & 0 & 1 & 0 & 1 & 0 & 0 & 0.35 & 0 \\ \hline /NULL\_Pointer \_\_ \_03.c & 15 & 1 & 0 & 0 & 0 & 0 & 0 & 0.1 & 0 \\ \hline \end{tabular}
\end{table}
Table 4: Example data of integrating analysis results from each tool for Alarm Classification model.  
\begin{table}
\begin{tabular}{c c c c} \hline  & **Precision** & **Accuracy** & **F1-Score** \\ \hline BERT-based Line-level Weakness Analysis model & 0.96 & 0.92 & 0.94 \\ \hline  & **F1-Score** & **Accuracy** & **AUC** \\ \hline Alarm Classification model & 0.89 & 0.96 & 0.89 \\ \hline \end{tabular}
\end{table}
Table 5: Evaluation of BERT-based Line-level Weakness Analysis and Alarm Classification model.  
Table 6 shows the experimental results for each CWE using the proposed method. The contents of Table 6 are CWE-ID, CWE name, total weakness of each CWE, the analysis result with static analysis tools, selected alarm (number of alarms that ACM classified true alarms), rejected alarm (number of alarms that ACM classified false alarms), and reduce false alarm rate (rate of false positive has reduced). As shown in Table 6, the false positive rate (*) is initially high based on tool analysis results, but the false positive rate (**) is reduced significantly by detecting and excluding false alarms through the proposed Alarm Classification model. For instance, based on all the test case files of Buffer Over-read (CWE-125) weakness, there are 195 weaknesses. Through analysis by static analysis tools, a total of 1224 alarms were generated but only 263 of these were true alarms and the remaining 961 were false alarms. The calculated false alarm rate was 79%, and a total of 144 weaknesses were analyzed. The false alarm rate can be considered high. The proposed model was implemented to classify 1224 alarms, of which 300 were true alarms and 924 were false alarms. However, among the classification results, there are several results that are incorrectly classified (it is a true alarm but classified false alarm: 28, and the other way: 65), and the false alarm rate of the alarm classification model can be calculated as 7%. The false alarm rate of 79% is significantly reduced by 71% when we look at the final results. The overall false detection rate has significantly dropped, despite there being differences in the false alarm rate, which is reduced in accordance with the learning outcomes of each CWE.