# VulEye: A Novel Graph Neural Network Vulnerability Detection Approach for PHP Application
## 5 Evaluation
### Result  
To be more evidently and intuitive, we evaluated the advantages of VulEye by designing four questions and answering them. **Q1: How is VulEye's detecting ability? **  
We first evaluated VulEye on the binary classification task on this question and chose the Receiver Operating Characteristic (ROC) curve to show the performance. Figure 6 offers VulEye's ROC performance. We can see that the area under the curve (AUC) was up to 0.9993, meaning that VulEye has adequate detection capability. Figures 7 and 8 show the tendencies of accuracy and loss in the training and valid stage. We can see that accuracy continues and stabilizes when the training steps increases and reaches 97% when the training step is 8000. Meanwhile, the loss gradually reduced from 0.5 to about 0.02. Tables 7 and 8 for the classification report of VulEye shows that the model's best macro-average accuracy reached 99%. So, VulEye performs very nicely on the binary classification job. To additionally confirm the model's capability in detecting multi-classes of vulnerabilities, we show the results by a multi-class confusion matrix, as shown in Figure 9. The results show that only CWE-78 and CWE-95's detection abilities are lower than 90%. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Precision** & **Recall** & **F1 Score** & **Support** \\ \hline Safe & 1.00 & 1.00 & 1.00 & 2111 \\ CWE-78 & 1.00 & 1.00 & 1.00 & 1 \\ CWE-79 & 1.00 & 1.00 & 1.00 & 193 \\ CWE-89 & 0.95 & 1.00 & 0.97 & 35 \\ CWE-90 & 0.98 & 0.97 & 0.97 & 100 \\ CWE-91 & 1.00 & 1.00 & 1.00 & 15 \\ CWE-95 & 0.67 & 0.50 & 0.57 & 4 \\ CWE-98 & 1.00 & 1.00 & 1.00 & 42 \\ CWE-601 & 1.00 & 1.00 & 1.00 & 172 \\ CWE-862 & 1.00 & 1.00 & 1.00 & 3 \\ \hline accuracy & & & 1.00 & 2676 \\ macro avg & 0.96 & 0.95 & 0.95 & 2676 \\ weighted avg & 1.00 & 1.00 & 1.00 & 2676 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Multi-classes Classification report of VulEye. Figure 7: VulEye training performance. Figure 8: VulEye valid performance. Figure 9: VulEye confusion matrix for multi-classes classification results. **Q2: Is the SDG slicing method beneficial to vulnerability detection? **  
We compare VulEye's SDG representation method with the traditional CFG (Control Flow Graph) and PDG (Program Dependence Graph) representation methods to verify the effectiveness of the SDG slicing method proposed in this paper. We trained binary and multi-classification models for each program representation method during the experiment to observe the vulnerability detection performance in binary and multi-classification scenarios. Table 9 shows that VulEye applies to different program representation methods, and the accuracy and precision of detection are more than 87%. Among them, the program representation method using PDG has a better detection effect than CFG, with an accuracy rate exceeding 96%, a precise detection rate for vulnerable programs exceeding 95%, and achieved an F1 score of 93.47%. The program representation method based on the PDG sub-graph slice SDG proposed in this paper can further improve the detection effect of the model. Its indicators are optimal, with a detection accuracy rate of 98.43%, a precise rate of vulnerable programs of 95.99%, and an F1 score of 96.29%. **Q3: How about the VulEye's detection ability compared with other models and tools? **  
Since many outstanding detection models and mature tools for PHP vulnerability detection worldwide exist, we selected six advanced models and tools for comparison: the machine-learning-based model Wirecaml [46] and TAP [36], the static-analysis-based tools Progpilt [47], RIPS [29], and WAP, and the Graph Neural Network based model DeepTective. We use these tools and models to conduct PHP vulnerability detection experiments compared with VulEye and measure their detection performance. Table 10 shows the performance result. We can see that the detection ability of VulEye significantly outperformed the existing open-source detection tools Progpilt, RIPS, and WAP.