## 4 Exploratory Study
### Capability Evaluation  
To examine the proficiency of LLMs in SQLI black-box detection, we assessed the performance of LLM4 Sqli-Manual and LLM4 Sqli-Sqlmap using three advanced LLMs (GPT-3.5, GPT-4, and Claude-3) as base models on the SqliMicroBenchmark. The results are presented in Table 1. The findings reveal that LLM4 Sqli-Sqlmap detected an average of 30 out of 45 targets, whereas LLM4 Sqli-Manual identified only 2 on average. This indicates that LLMs guiding scanners for SQLI black-box detection are significantly more effective than those manually constructing payloads for SQLI black-box detection.  
In the context of LLM4 Sqli-Manual's results, GPT-4 demonstrated the best performance by detecting 4 out of 28 targets of basic difficulty. Claude-3 followed, identifying two targets of basic difficulty, while GPT-3.5 failed to detect any targets of basic difficulty. None of the models succeeded in detecting any targets of advanced difficulty.  
The results suggest that LLM4 Sqli-Sqlmap has a pronounced advantage in detecting basic difficulty targets. Specifically, GPT-4 excelled in this mode, detecting 27 basic difficulty targets and 4 advanced difficulty targets. Both Claude-3 and GPT-3.5 showed comparable performance, each detecting 26 basic difficulty targets and 4 advanced difficulty targets. Both LLM4 Sqli-Manual and LLM4 Sqli-Sqlmap struggled with advanced difficulty targets. These targets incorporate defense mechanisms such as encoding, keyword filtering, or Web Application Firewalls (WAFs), necessitating deep analysis and validation to identify and  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{1}{c}{**Scanner**} & \multicolumn{1}{c}{**Basic Targets (28)**} & \multicolumn{1}{c}{**Advanced Targets (17)**} & \multicolumn{1}{c}{**Total (45)**} \\ \hline \multirow{4}{*}{LLM4 Sqli-Manual} & GPT-3.5 & 0 (0\%) & 0 (0\%) & 0 (0\%) \\  & GPT-4 & 4 (14\%) & 0 (0\%) & 4 (9\%) \\  & Claude-3 & 2 (7\%) & 0 (0\%) & 2 (4\%) \\  & GPT-3.5 & 26 (93\%) & **4 (24\%)** & 30 (67\%) \\ LLM4 Sqli-Sqlmap & GPT-4 & **27 (96\%)** & **4 (24\%)** & **31 (69\%)** \\  & Claude-3 & 26 (93\%) & **4 (24\%)** & 30 (67\%) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Performance of LLM4 Sqli-Manual and LLM4 Sqli-Sqlmap on SqliMicroBenchmark.  
Figure 1: LLM4 Sqli-Manual (**left**) and LLM4 Sqli-Sqlmap (**right**) workflow.  
bypass these defenses. Thus, the poor performance of LLMs on these complex problems is predictable. **In summary, LLM4 Sqli-Sqlmap exhibits significant potential in SQLI black-box detection, whereas LLM4 Sqli-Manual demonstrates weak performance.**  
Compared to LLM4 Sqli-Manual, LLM4 Sqli-Sqlmap performs better in SQLI black-box detection. This superior performance is attributed to the fact that directly generating payloads to detect SQLI imposes higher demands on LLMs' depth of knowledge and reasoning ability within the SQLI domain. In contrast, LLM4 Sqli-Sqlmap employs automated SQLI scanners to execute the task, which significantly reduces these requirements. Experimental results indicate that current LLMs, such as GPT-3.5, GPT-4, and Claude-3, have not yet achieved the capability to efficiently resolve SQLI issues independently. Consequently, our research will utilize LLM4 Sqli-Sqlmap as the foundational framework for further exploration.