# VulEye: A Novel Graph Neural Network Vulnerability Detection Approach for PHP Application
## 5 Evaluation
### Result
The main goal of these tools is generally to use simple rules to parse and check vulnerabilities. It is difficult to generate all kinds of vulnerability-triggering rules. Moreover, the rules' generation depends on expert knowledge, and some amiss rules may enlarge False Negatives and False Positives in vulnerability detection. the machine learning tools TAP and Wirecaml-SQLI performed better than open-source tools based on static analysis. The TAP model achieved an accuracy of 0.9385 and an F1 score of 0.8788. As far as we know, DeepTective is the latest PHP vulnerability detection method based on a Graph Neural Network which combines gated recurrent units and graph convolutional networks to detect SQLI (SQL injection), XSS (cross-site scripting), and OSCI (OS command injection) vulnerabilities. Since it is not an open-source project, we implemented the model and tried our best to train it as the authors have. The results show that VulEye is also better than the DeepTective model. \begin{table}
\begin{tabular}{l l l l l l} \hline \hline  & **Class** & **Precision** & **Recall** & **F1** & **Accuracy** \\ \hline \multirow{2}{*}{VulEye-CFG} & Safe & 0.8741 & 0.959 & 0.9146 & \multirow{2}{*}{0.8756} \\ \cline{2-2} \cline{5-6}  & Unsafe & 0.8805 & 0.6865 & 0.7715 & \\ \hline \multirow{2}{*}{VulEye-PDG} & Safe & 0.9628 & 0.9824 & 0.9725 & \\ \cline{2-2} \cline{5-6}  & Unsafe & 0.9576 & 0.913 & 0.9347 & \\ \hline \multirow{2}{*}{VulEye-SDG} & Safe & 0.9909 & 0.9892 & 0.99 & \\ \cline{2-2} \cline{5-6}  & Unsafe & 0.9599 & 0.9659 & 0.9629 & \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Different representation methods’ results comparison. **Q4: How is the generalization capability of VulEye? **  
We evaluate the generalization ability of VulEye through the actual PHP vulnerability programs contained in the two open-source vulnerability experimental environments, SQL-Labs [26] and XSS-Labs [27]. In the experiment, we selected our best model to detect the vulnerable file in the SQL-Labs and XSS-Labs and find how many files VulEye can successfully discover. There are 69 SQLI vulnerable files in SQL-Labs and 20 XSS vulnerable files in XSS-Labs. We deleted two vulnerable files in XSS-Labs, as shown in the code Listings 5 and 6, because the vulnerability is not caused by PHP code but javascript. Table 11 shows the result of VulEye's vulnerability detection capability in an actual PHP project. VulEye detected 61 SQLI vulnerable files in a total of 69 and 13 XSS vulnerable files out of 18. The accuracy of SQLI vulnerability detection is 0.8841, and the XSS vulnerability detection accuracy is 0.7222. **Listing 5. ** XSS-Labs sample 6. ... <script> if (localStorage.getItem(\"labs-tegal1337\") === null) { localStorage.setItem(\"labs-tegal1337\", \"labs-tegal1337\"); } </script>...  
<script> if (localStorage.getItem(\"labs-tegal1337\")!== null) {  document.getElementById(\"content\").innerHTML = \"\  <span>Current value of <b>labs-tegal1337</b> \   LocalStorage is: </span>\"+localStorage.getItem(\"labs-tegal1337\")+\"\"; } </script>... \begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Tool** & **Accuracy** & **Precision** & **Recall** & **F1** \\ \hline VulEye & 0.9843 & 0.9599 & 0.9659 & 0.9629 \\ DeepTective & 0.9726 & 0.9331 & 0.9373 & 0.9352 \\ \hline TAP & 0.9385 & 0.925 & 0.837 & 0.8788 \\ Wirecaml-XSS & 0.3864 & 0.2274 & 0.8851 & 0.3618 \\ Wirecaml-SQLI & 0.7171 & 0.1269 & 1 & 0.2252 \\ \hline Progpilot & 0.413 & 0.2227 & 0.4839 & 0.305 \\ WAP & 0.657 & 0.2843 & 0.1902 & 0.2279 \\ RIPS & 0.6923 & 0.3333 & 0.1562 & 0.212 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Tools’ Results Comparison. \begin{table}
\begin{tabular}{l l l l} \hline \hline  & **Contained** & **Detected** & **Accuracy** \\ \hline SQL-Labs & 69 & 61 & 0.8841 \\ XSS-Labs & 18 & 13 & 0.7222 \\ \hline \hline \end{tabular}
\end{table} 
Table 11: generalization capability test results. **Listing 6. ** XSS-Labs sample 8.