# Automated Vulnerability Detection in Source Code Using Minimum Intermediate Representation Learning
## 5 Experiments and Results
### Comparison of Different Neural Networks  
In order to select the best model for representation learning, we adopted a 10-fold cross validation on train data. We compared six neural networks. Sequential CNN has three sequentially connected convolutional neural networks (CNN). Each convolutional neural network has 128 filters, and the size of filters is 3, 5, 7, respectively. Sequential LSTM has two long short-term memory neural networks (LSTM). Each LSTM has a 128-dimensional output. BiLSTM is composed of a forward LSTM and a backward LSTM, and it has a 128-dimensional output. In CNN + BiLSTM, the output of CNN with 128  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **Samples** & **Vulnerable** & **Not Vulnerable** & **Vocabulary** \\ \hline CWE-119 & 39,753 & 10,440 & 29,313 & 16,983 \\ CWE-399 & 21,885 & 7285 & 14,600 & 7811 \\ Extended dataset & 482,265 & - & - & 61,638 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics on training data and pre-training data.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Description** \\ \hline Input\_dim & The size of vocabulary (16983). \\ \hline Output\_dim & The dimensionality of vectors that the tokens are converted to (200). \\ \hline Sequence\_length & The length of each sample (400). \\ \hline CNN units & There are 3 concatenated convolutional neural networks (CNN). The number of filters \\  & all is 128. The size of filers is 3, 5, 7 respectively. \\ \hline Batch\_size & The number of samples that are propagated through the network (128). \\ \hline Loss function & A function to calculate the loss between the predicted value and real value \\  & (binary\_crossentropy) \\ \hline Optimizer & The algorithm to optimize the neural network (Adam) \\ \hline Monitor & The matric to be monitored for early stop (F1) and patience (10) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Tuned Parameters for representation learning.  
5-dimensional filters is the input of LSTM with 128 output. CNN + BiLSTM + Attention has an extra self-attention layer in addition to CNN + BiLSTM. Concatenated CNN has an output that concatenates the output of three convolutional neural networks. Each neural network is connected to a dense layer for training. The comparison results are showed in Table 3.  
We observe that although BiLSTM achieves the best results on FNR and Recall, it has the worst results on P. Compared with other models, Concatenated CNN has balanced performance and achieves the best results in FPR, P, and F1. Therefore, in the representation learning stage, we chose Concatenated CNN as the final model to learn high-level features.  
In order to compare the effects of different classifiers at the stage of vulnerability detection, we used the learned high-level features to train six classifiers: Logistic Regression (LR), Naive Bayesian (NB), Support Vector Machine (SVM), Multi-Layer Perceptron (MLP), Gradient Boosting Decision Tree (GBDT), Random Forest (RF). Table 4 shows the comparison results. Among the six classifiers, SVM had the best performance on FPR and P metrics, while RF achieved the best performance on FNR, R, and F1 metrics. Therefore, RF has a more balanced performance.