# PreNNsem: A Heterogeneous Ensemble Learning Framework for Vulnerability Detection in Software
## 4 Experiments and Results
### Comparison of Different Classification Algorithms  
Through Section 4.4, we observed that concatenated CNNs are the appropriate deep learning model to extract features. In Table 5, we directly use traditional machine learning after the word-embedding layer. In order to improve the classification results, we chose a different ensemble learning model [49] to substitute the simple activation sigmoid after CNNs. We chose boosting and bagging as our homogeneous ensemble model, including gradient boosting decision tree (GBDT) and random forest (RF). We used stacking for generating ensembles of heterogeneous classifiers, logistic regression (LR) and MultinomialNB (NB) as the base classifiers, and RF as the final classifier. For comparison with ensemble classifiers, we also chose traditional classifiers, including KNeighbors (KN), NB, and LR. Finally, Table 5 shows the comparison results. \begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Methods** & **Corpus** & **FPR (\%)** & **FNR (\%)** & **P (\%)** & **R (\%)** & **F1 (\%)** & **Memory (MB)** & **Time (s/epoch)** \\ \hline CNN-random & - & 3.4 & 12.3 & 90.1 & 87.7 & 88.9 & 1675.1 & 15.0 \\ CNN-static & CWE-119 & 3.9 & 16.6 & 88.4 & 83.4 & 85.8 & 1671.0 & 9.9 \\ CNN-non-static & CWE-119 & 2.5 & 10.7 & 92.6 & 89.3 & 90.9 & 1694.1 & 14.9 \\ CNN-static & SySeVR & 2.5 & 13.6 & 92.4 & 86.4 & 89.3 & 1670.9 & 10.1 \\ CNN-non-static & SySeVR & 2.6 & 9.7 & 92.3 & 90.2 & 91.2 & 1673.5 & 15.2 \\ LSTM-random & - & 2.1 & 12.9 & 93.5 & 87.1 & 90.2 & 1377.2 & 200.5 \\ LSTM-static & CWE-119 & 2.3 & 13.4 & 92.8 & 86.6 & 89.6 & 1365.6 & 183.4 \\ LSTM-non-static & CWE-119 & 1.9 & 9.8 & 94.1 & 90.2 & 92.1 & 1366.8 & 201.9 \\ LSTM-static & SySeVR & 1.6 & 10.5 & 95.1 & 89.5 & 92.2 & 1370.4 & 183.6 \\ LSTM-non-static & SySeVR & 2.3 & 10.7 & 93.0 & 89.3 & 91.1 & 1373.5 & 197.1 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison of different embedding methods on different corpora. Memory, memory consumption; Time, time consumption. \begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Methods** & **Base Classifier** & **FPR (\%)** & **FNR (\%)** & **P (\%)** & **R (\%)** & **F1 (\%)** & **Time (s)** \\ \hline Traditional ML & KN & 6.5 & 22.3 & 80.6 & 77.7 & 79.1 & 0.7 \\ Traditional ML & LR & 5.6 & 27.1 & 81.9 & 72.9 & 77.1 & 0.7 \\ CNNs + Traditional ML & KN & 3.3 & 6.6 & 90.7 & 93.4 & 93.0 & 0.6 \\ CNNs + Traditional ML & NB & 5.6 & 6.1 & 85.4 & 93.9 & 89.5 & 0.1 \\ CNNs + Traditional ML & LR & 2.1 & 7.9 & 93.9 & 92.1 & 92.9 & 0.6 \\ CNNs + Boosting & GBDT & 1.6 & 9.1 & 95.5 & 90.7 & 93.0 & 48.8 \\ CNNs + Bagging & RF & 1.6 & 8.8 & 95.8 & 91.2 & 93.1 & 10.7 \\ CNNs + Stacking & LR, NB, RF & 1.5 & 8.0 & 95.4 & 91.9 & 93.6 & 13.7 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Comparison of different classification algorithms. ML, machine learning; KN, KNeighbors; LR, logistic regression; NB, MultinomialNB; GBDT, gradient boosting decision tree; RF, random forest. Table 5 shows that the first two lines did not use representation learning to extract features, and the classification effect was poor. Machine learning with CNNs performed better than traditional machine learning. We concluded that word embedding can only extract the granular features of words. CNNs can obtain the features of code structure, not only word semantics. Therefore, multiple granularity features help to improve the performance of the classifier. The results of the last three lines (CNN + Ensemble) were generally better than those of lines three to five. Although CNN + NB produced the best recall results (93.9%), its precision was worse, at 85.4%, resulting in an F1 score of only 89.5%, which represents comprehensive performance. Low precision leads to spending more effort and time on the wrong detection results. Therefore, ensemble learning can further improve vulnerability detection.