# SSQLi: A Black-Box Adversarial Attack Method for SQL Injection Based on Reinforcement Learning
## 4 Experiment
### Performance Comparison
#### 4.2.2 Experiment of Attack Effect  
The purpose of this experiment was to verify the effect of SSQLi on the detector attack. Since WAFs did not require training, in contrast to other other learning-based detection models, we first tested the effect of the two WAFs alone, as shown in Table 7. We observed that after being attacked by the adversarial sample, the detection rate of both WAFs decreased dramatically.  
Next, we tested the detectors on deep-learning, machine-learning, and integrated-learning models. First, however, we had to train each detection model to improve their performance, as shown in Table 8.  
Then, we attacked the detector with the SAC-algorithm-based SSQLi, proposed in this paper, an SSQLi using another reinforcement-learning algorithm DDQN, and the WAF-A-Mole method. The experimental results are shown in Table 9. Moreover, we have documented the overall time consumption of the SSQLi attack detector in Table 10. When not being attacked, all detectors, except for SafeDog, had detection rates over 99%, and the vast majority of the attack samples had been detected. After attacking with the SAC-algorithm-based SSQLi, the detection rate of each detector decreased significantly, and the detection rates of the SVM and LSTM models were below 5%. During the training process, we recorded the changes in the average detection rates of the three attack methods, which are shown in Figure 8. Among all the detector models, the ensemble-learning-based detector had the best robustness with a consistent detection rate of approximately 60% even after an attack. As compared to the SAC-based SSQLi, the DDQN-based SSQLi only had a significant effect on the SafeDog, LSTM, and SVM models, but the degree of effect was relatively low. WAF-A-Mole had only a subtle effect on the detection model.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **30** & **40** & **50** & **60** & **70** \\ \hline ER & 59.61\% & 60.50\% & 62.94\% & 63.60\% & 62.68\% \\ Training time & 3627 s & 4818 s & 5714 s & 6489 s & 7808 s \\ \hline \hline \end{tabular}
\end{table}
Table 6: Experimental results of parameter selection of epochs.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Detection Model** & **Accuracy** & **Precision** & **Recall** & **F1 Score** \\ \hline LSTM & 99.97\% & 99.99\% & 99.96\% & 99.97\% \\ CNN & 99.96\% & 99.33\% & 99.96\% & 99.64\% \\ MLP & 99.93\% & 99.91\% & 99.96\% & 99.93\% \\ SVM & 99.94\% & 100\% & 99.88\% & 99.94\% \\ AdaBoost & 99.95\% & 99.96\% & 99.94\% & 99.95\% \\ RandomForest & 99.97\% & 100\% & 99.93\% & 99.97\% \\ XGBoost & 99.96\% & 99.98\% & 99.93\% & 99.96\% \\ GradientBoost & 99.95\% & 99.97\% & 99.93\% & 99.96\% \\ \hline \hline \end{tabular}
\end{table}
Table 8: The performance of original detection model.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Original Payload** & **Adversarial Example** \\ \hline  & **ER** & **ER** \\ \hline SafeDog & 15.27\% & 94.96\% \\ Modescecurity & 0.10\% & 62.94\% \\ \hline \hline \end{tabular}
\end{table}
Table 7: The performances of WAF applications.  
Moreover, as a black-box-attack method, SSQLi automatically adopted different attack strategies for different detection methods without knowing the specific parameters of the detector. As shown in Figure 9, we were able to demonstrate the variation in the adversarial attack models by using SafeDog, ModSecurity, and LSTM as examples.  
\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & **SafeDog** & **ModSecurity** & **LSTM** & **CNN** & **MLP** & **SVM** & **AdaBoost** & **RandomForest** & **XGBoost** & **GradientBoost** \\ \hline Training Time & 2149.74 s & 3536.26 s & 29.970.53 s & 34,843.29 s & 12,522.81 s & 2834.67 s & 47,894.34 s & 43,474.43 s & 43,463.45 s & 39,957.42 s \\ \hline \hline \end{tabular}
\end{table}
Table 10: Time consumption of SSQLi.  
Figure 8: Mean detection rate variation curve of each detector against adversarial attacks.