# A Context-Aware Neural Embedding for Function-Level Vulnerability Detection
## 3 Experiments and Evaluation
### Result Analysis
#### 3.4.2 The Structure of Bi-LSTM  
To examine how different numbers of Bi-LSTM layers affect the learning of long-term dependency of the code sequences, we use different networks with different number of Bi-LSTM layers ranging from 0 to 3, and compare the results generated by these networks. As Figure 5 shows, when using only \(1\) Bi-LSTM layer in our framework, the best top-200 recall are achieved on all three projects. With the number of Bi-LSTM layer increases from 1 to 3, there is a continuous performance decrease as measured by the top-200 recall for all projects, except for LibTIFF. However, compared with using only \(1\) Bi-LSTM layer, a performance drop is still observed for LibTIFF. With more Bi-LSTM layers added to the network, the time required for training increases dramatically. Hence, adding more bi-LSTM layers does not contribute to performance increase for vulnerability detection on our data sets. For the following experiments, our framework contains only one Bi-LSTM layer.  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Software**} & \multirow{2}{*}{**Embedding**} & \multicolumn{6}{c}{**\# of Vulnerable Functions Found in Top-\(k\) (Top-\(k\) Precision)**} \\ \cline{3-10}  & & **Top 10** & **Top 20** & **Top 30** & **Top 40** & **Top 50** & **Top 100** & **Top 150** & **Top 200** \\ \hline \multirow{6}{*}{FFmpeg} & ELMo + RF & 9 (90\%) & 18 (90\%) & 22 (73\%) & 26 (65\%) & 31 (62\%) & 48 (48\%) & 56 (37\%) & 63 (32\%) \\ \cline{2-10}  & Word2 Vec + RF & 8 (80\%) & 14 (70\%) & 16 (53\%) & 20 (50\%) & 21 (42\%) & 32 (32\%) & 39 (26\%) & 49 (25\%) \\ \cline{2-10}  & FastText + RF & 8 (80\%) & 13 (65\%) & 17 (57\%) & 19 (48\%) & 21 (42\%) & 32 (32\%) & 43 (29\%) & 51 (26\%) \\ \cline{2-10}  & GloVe + RF & 7 (70\%) & 9 (45\%) & 12 (40\%) & 15 (38\%) & 17 (34\%) & 27 (27\%) & 37 (25\%) & 45 (23\%) \\ \cline{2-10}  & BERT + RF & 7 (70\%) & 9 (45\%) & 11 (37\%) & 12 (30\%) & 18 (36\%) & 27 (27\%) & 38 (25\%) & 44 (22\%) \\ \hline \multirow{6}{*}{LibTIFF} & ELMo + RF & 8 (80\%) & 13 (65\%) & 16 (53\%) & 20 (50\%) & 26 (52\%) & 42 (42\%) & 48 (32\%) & 52 (26\%) \\ \cline{2-10}  & Word2 Vec + RF & 5 (50\%) & 9 (45\%) & 11 (37\%) & 13 (33\%) & 16 (32\%) & 26 (26\%) & 38 (25\%) & 48 (24\%) \\ \cline{2-10}  & FastText + RF & 8 (80\%) & 13 (65\%) & 15 (50\%) & 18 (45\%) & 20 (40\%) & 32 (32\%) & 41 (27\%) & 48 (24\%) \\ \cline{2-10}  & GloVe + RF & 7 (70\%) & 12 (60\%) & 13 (43\%) & 17 (43\%) & 17 (34\%) & 30 (30\%) & 42 (28\%) & 52 (26\%) \\ \cline{2-10}  & BERT + RF & 4 (40\%) & 9 (45\%) & 12 (40\%) & 13 (33\%) & 16 (32\%) & 30 (30\%) & 40 (27\%) & 51 (26\%) \\ \hline \multirow{6}{*}{OpenSSL} & ELMo + RF & 9 (90\%) & 19 (95\%) & 27 (90\%) & 33 (83\%) & 38 (76\%) & 53 (53\%) & 60 (40\%) & 66 (33\%) \\ \cline{2-10}  & Word2 Vec + RF & 5 (50\%) & 9 (45\%) & 11 (37\%) & 13 (33\%) & 16 (32\%) & 26 (26\%) & 38 (25\%) & 48 (24\%) \\ \cline{1-1} \cline{2-10}  & FastText + RF & 8 (80\%) & 15 (75\%) & 24 (80\%) & 27 (68\%) & 32 (64\%) & 38 (38\%) & 50 (33\%) & 53 (27\%) \\ \cline{1-1} \cline{2-10}  & GloVe + RF & 8 (80\%) & 14 (70\%) & 19 (63\%) & 25 (63\%) & 26 (52\%) & 35 (35\%) & 39 (26\%) & 43 (22\%) \\ \cline{1-1} \cline{2-10}  & BERT + RF & 8 (80\%) & 13 (65\%) & 17 (57\%) & 18 (45\%) & 22 (44\%) & 35 (35\%) & 40 (27\%) & 45 (23\%) \\ \hline \hline \end{tabular}
\end{table}
Table 3: The comparison between the ELMo module, Word2 Vec, GloVe, FastText and BERT model in terms of their effectiveness of generating representations for source code functions on open source projects: FFmpeg, LibTIFF and OpenSSL. The generated representations are used as features to train a random forest (RF) classifier.