# SSQLi: A Black-Box Adversarial Attack Method for SQL Injection Based on Reinforcement Learning
## 3 Proposed Method
### SAC Agent  
When launching an attack on a detection model in a real-world scenario, the exact parameters of the model's settings are typically unknown. Therefore, in order to simulate a real environment, we designed a black-box attack model that could adaptively adjust the choice of attack strategy based only on the outcomes of the detector. We used reinforcement learning in our design, wherein an agent selected the attack strategy. We considered the attack strategy matrix as the action space A and the results returned by the detector as the environment feedback, with the goal of allowing the agent to select the best attack strategies. The process by which the agent identified the optimal strategy in order to maximize the reward provided by the environment was expressed as Equation (5), where \(\mathbb{E}_{(s_{t},a_{t})\sim\rho_{\pi}}[r(s_{t},a_{t})]\) denotes the reward expectation based on strategy \(\pi\). To avoid the effect of the localized optima on the final strategy selection [25], we used the stochastic-strategy-gradient algorithm.  
\[\pi^{*}=\arg\max_{\pi}\sum_{t}\mathbb{E}_{(s_{t},a_{t})\sim\rho_{\pi}}[r(s_{t},a_{t})] \tag{5}\]  
[PERSON] et al. [26] introduced the concept of entropy to optimize strategic exploration and prevent a model from becoming enmeshed in localized optimal solutions, and this approach also encouraged the agent to explore more diverse strategies during the training phase. The attack strategy matrix proposed in this paper enabled the overlapping of multiple strategies in order to promote strategic diversification. We added entropy via \(\alpha\mathcal{H}(\pi(\cdot\mid s_{t}))\) to Equation (5) to create Equation (6), where b indicates the weight of the entropy value, where a larger value of \(\alpha\) indicates more focus on a multi-modal strategic selection, and a smaller value indicates more focus on the maximum cumulative feedback expectation.  
\[\pi^{*}=\arg\max_{\pi}\sum_{t}\mathbb{E}_{(s_{t},a_{t})\sim\rho_{\pi}}[r(s_{t},a_{t})+\alpha\mathcal{H}(\pi(\cdot\mid s_{t}))] \tag{6}\]  
The state-value function was represented by Equation (7), and the soft Q-function was represented by Equation (8), as follows:  
\[v_{\pi}(s)=\mathbb{E}_{\pi}[R_{t+1}+\gamma v_{\pi}(S_{t+1})\mid S_{t}=s] \tag{7}\]  
\[q_{\pi}(s,a)=\mathbb{E}_{\pi}[R_{t+1}+\gamma q_{\pi}(S_{t+1},A_{t+1})\mid S_{ t}=s,A_{t}=a] \tag{8}\]  
After the introduction of the entropy value, the reward function could be expressed as follows:  
\[r_{\text{soft}}\left(s_{t},a_{t}\right)=r(s_{t},a_{t})+\gamma\alpha\mathbb{E} _{s_{t+1}\sim\rho}H(\pi(\cdot\mid s_{t+1})) \tag{9}\]  
In order to improve the efficiency of the sample utilization and ensure the rapid convergence of the model, this study used the SAC algorithm to train the actor network to implement strategy selection and the critic network to evaluate the value of actions [27]. Since the original SAC algorithm had been applied to continuous data, in the subsequent training, we referred to [28] for the implementation of the SAC algorithm. The training process of SSQLi is shown in Algorithm 1.  
```
1:Detector: SQL injection detector; Payloads: the set of original payloads; SAC: SAC agent; state_function: state-vector-representation algorithm attack_matrix; MaxStep; epoch; back_size;
2:SAC model; adversarial_examples;
3:for each epoch do
4:for each payload in payloads do
5:whileend_control == 0 do
6: state = state_function(payload)
7: action = SAC.pick_action(state)
8: attack_function = attack_matrix(action)
9: new_payload = attack_function(payload)
10: reward = Detector.test(new_payload)
11: n_state = state_function(new_payload)
12: experiences.push([state,action,n_state,reward])
13:if reward\(>\)0 then
14: adversarial_examples.push(new_payload)
15:end_control = 1
16:endif
17:if step_number\(>\)MaxStep then
18: end_control = 1
19:endif
20:endwhile
21: history = experience.sample(batch_size)
22: SAC_learn(history)
23:endfor
24:endfor
25:return SAC model; adversarial_examples;
```  
**Algorithm 1** SSQLi based on SAC.