# Automated Vulnerability Detection in Source Code Using Minimum Intermediate Representation Learning
## 5 Experiments and Results
### Comparative Analysis  
In order to examine the effectiveness of the proposed method in vulnerability detection, we performed a comparative experiment with state-of-the-art methods. We chose open-source static analysis tool Flawfinder [4] and commercial static analysis tool Checkmarx [5]. They represent traditional vulnerability detection methods based on static analysis. VUDDY [19] was chosen to represent the similarity-based method. VulDeePecker and our methods detect vulnerabilities based on learning vulnerable programming patterns. The difference is that VulDeePecker [25] has no representation learning stage, and it uses one dataset through the entire method. All the results in Table 8 are based on the same dataset. The results of Checkmarx and VulDeePecker are from the paper [25]. This because we are not able to access the commercial tool Checkmarx and VulDeePecker is not an open-source tool. It can be seen from Table 6 that our method outperforms the state-of-the-art methods. Specifically, traditional static analysis methods incur high FNR and FPR. The result of Checkmarx was relatively superior to Flawfinder. This is because the former applies efficient data flow analysis algorithms. VUDDY has a low FPR (3.5%), but has a poor performance on FNR (91.3%), P (47.0%), R (8.7%), and F1 (14.7%). This means that most vulnerabilities are not detected. This is because the similarity-based method is suitable for detecting vulnerabilities caused by code cloning, and is not adept at detecting general vulnerabilities. VulDeePecker and our method outperform other methods in most indicators. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Vulnerability** & **FPR (\%)** & **FNR (\%)** & **P (\%)** & **R (\%)** & **F1 (\%)** \\ \hline CWE-119 & 1.5 & 9.6 & 95.7 & 90.4 & 93.0 \\ CWE-399 & 0.1 & 3.0 & 98.7 & 97.0 & 97.8 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Performance of our method on different types of vulnerability. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Vulnerability** & \multicolumn{2}{c}{**Pre-Training**} & \multicolumn{2}{c}{**Representation Learning**} & \multicolumn{2}{c}{**Classifier Training**} \\ \cline{2-5}  & **Time (s)** & **Memory (MB)** & **Time (s)** & **Memory (MB)** & **Time (s)** & **Memory (MB)** \\ \hline CWE-119 & \multirow{2}{*}{216.5} & \multirow{2}{*}{278.0} & 301.1 & 2356.6 & 7.0 & 39.0 \\ CWE-399 & & & 137.4 & 2341.9 & 4.2 & 20.6 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: The complexity of our method on different types of vulnerability. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**System** & **FPR (\%)** & **FNR (\%)** & **P (\%)** & **R (\%)** & **F1 (\%)** \\ \hline Flawfinder [4] & 46.3 & 69.0 & 23.7 & 40.5 & 29.9 \\ Checkmarx [5] & 43.1 & 41.1 & 39.6 & 58.9 & 47.3 \\ VUDDY [19] & 3.5 & 91.3 & 47.0 & 8.7 & 14.7 \\ VulDeePecker [25] & 2.9 & 18.0 & 91.7 & 82.0 & 86.6 \\ Our method & 1.5 & 9.6 & 95.7 & 90.4 & 93.0 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Comparative experimental results. This indicates that the vulnerable pattern-based method excels at detecting general vulnerabilities. Compared with VulDeePecker, our method improved by 1.4% in FPR, 8.4% in FNR, 4.0% in P, 7.6% in R, and 6.4% in F1. This indicates that compared with training classifier directly, the application of transfer learning and representation learning can effectively improve the effect of vulnerability detection. The above experimental results show that our method has a fine granularity and representation ability by transforming the sample into the minimum intermediate representation. This facilitates audits and alleviates long-term dependency issues. By applying transfer learning, unlabeled data can be utilized to improve the effectiveness of vulnerability detection. By using the representation learning approach, our approach is able to learn the features of vulnerabilities from context. The comparative experiment results show that our method outperforms state-of-the-art methods.