# Automated Vulnerability Detection in Source Code Using Minimum Intermediate Representation Learning
## 5 Experiments and Results
### Effectiveness of Pre-Training  
In order to validate the effectiveness of pre-training, we projected the 200-dimensional embeddings to 2-dimensional vectors. Three types of tokens with different semantics were picked out. They are memory manipulation functions, logical comparisons and custom identifiers. The projected embeddings are shown in Figure 7.  
As can be seen from Figure 7, different types of tokens were grouped into separable clusters. This proves that through the pre-training on the extended corpus, our method can learn the common semantic information. In addition, custom identifiers with similar semantics have similar positions. This proves the ability of our method to alleviate the OoV problem.  
In addition, we compared the effectiveness of our method with four different embedding methods. A random forest is used as a classifier in all five methods. The first three methods that use Vocabulary, N-Gram, IT-IDF language models respectively were performed on the same dataset (CWE-119). Their classifiers are trained using the real value vectors encoded respectively. The last two methods both use the CBOW language model as the embedding method, and a representation learning is performed to obtained high-level features. The difference is that the former uses the same dataset (CWE-119) for pre-training, representation learning, and classifier training. The latter is our approach that utilizes an  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Algorithm** & **FPR (\%)** & **FNR (\%)** & **P (\%)** & **R (\%)** & **F1 (\%)** \\ \hline LR & 2.4 & 12.0 & 93.0 & 88.0 & 90.5 \\ NB & 18.4 & 13.1 & 62.6 & 86.9 & 72.8 \\ SVM & 1.2 & 10.9 & 96.4 & 89.1 & 92.6 \\ MLP & 2.8 & 12.6 & 91.8 & 87.4 & 89.5 \\ GBDT & 1.3 & 11.3 & 96.0 & 88.7 & 92.2 \\ RF & 1.5 & 9.6 & 95.7 & 90.4 & 93.0 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison of different classification algorithms.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **FPR (\%)** & **FNR (\%)** & **P (\%)** & **R (\%)** & **F1 (\%)** \\ \hline Sequential CNN & 6.3 & 22.0 & 85.9 & 68.0 & 75.9 \\ Sequential LSTM & 17.6 & 30.8 & 69.0 & 69.2 & 69.1 \\ BiLSTM & 29.4 & 7.3 & 64.1 & 92.3 & 75.7 \\ CNN+BiLSTM & 13.1 & 10.4 & 80.6 & 89.6 & 84.9 \\ CNN +BiLSTM+Attention & 5.5 & 33.9 & 87.1 & 66.1 & 75.1 \\ Concatenated CNN & 1.8 & 15.0 & 94.4 & 85.0 & 89.5 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison of different representation learning models.  
extended corpus for pre-training and vulnerability dataset (CWE-119) for representation learning and classifier training.  
Table 5 summarizes the experimental results. The first three methods have approximate performances. A significant improvement over the first three methods is achieved by the fourth method. This shows that the application of pre-training can improve the effectiveness of vulnerability detection. The last result is achieved by our method that has the best performance. This is because, through pre-training from a large database, the common syntax of a programming language is learned, and every token is represented by its context. Even if tokens are unique in a different program, it will eventually be represented as a similar vector as long as the identifier has a similar context. By using this transfer learning approach, we mitigate the impact of the OoV issue. The results validate Hypothesis 1.