## 3 Proposed Approach
### DDQN  
In reinforcement learning, the decision maker or learner is called the Agent. The sum of all other interactions with them is called the Environment, and the environment generated such as game scores is called the Reward. DeepMind's VMnih first proposed the deep Q-networks (DQN) network in 2013. DQN uses end-to-end reinforcement learning to learn strategies from high-dimensional input and achieves comparable results to professional players in the many challenges of the Atari2600 game [14]. DQN is a deep Q network based on the Q-learning algorithm. The main improvements are as follows:  
1. Limit the reward value and the error term to a limited range and define the Q value and the gradient to be within a valid limited range to improve the stability of the model. 2. Adopt the experience replay training mechanism. The training of deep neural network requires that the samples be independent and identically distributed. However, the data acquired by reinforcement learning have a strong correlation. Direct training will lead to the problem of training instability. DQN stores the transferred samples of each step into memory D based on the experience replay mechanism, and each time, a mini-batch transfer sample is randomly extracted from D. The parameters are updated according to the gradient descent and randomized by introducing the experience replay sampling, which weakens the correlation between the data, thereby improving the stability of the model. 3. Use a deep neural network to approximate the current value function. In addition, use another network to generate a target value function. Every Ground, the target value function is updated with the current value function, and the other rounds keep the target value function unchanged. There is an overestimation problem in the DQN algorithm, which may lead to overestimation of a non-optimal action, such that the Q value exceeds the Q value of the optimal action. Finally, the model cannot obtain the optimal action. In order to solve the problem of overestimation of the DQN algorithm, [PERSON] et al. [15] proposed the double DQN algorithm (DDQN), which decomposes the target value function into the action selection value function and action evaluation value function. It can solve the problem of overestimation by decoupling action selection and action evaluation. The neural network part of DQN can be seen as a new neural network with an old neural network. They have the same structure, but their internal parameter updates are delayed. The idea of DDQN is to use another neural network to eliminate some effects of maximum error. Therefore, DDQN uses the Q-estimated neural network to estimate the maximum action value of Qmax(s', a') in Q reality. Then, we use this action estimated by Q to select Q(s') in Q reality. The basic operating principle of DDQN is shown in Figure 2.  
The time and space complexity of RLXSS depends mainly on the DDQN algorithm:  
\[Y_{t}^{DoubleDQN}\equiv R_{t+1}+\gamma Q(S_{t+1},\underset{a}{argmax}\,Q(S_{t+1},a;\theta_{t})\theta_{t}^{-}) \tag{2}\]  
In the algorithm, the parameter t represents the current step of the algorithm; the parameter a represents the action selected by the algorithm; \(S_{t+1}\) represents the new state of DDQN; \(\theta\) represents the weight in the new neural network. We can see from the algorithm that DDQN is the time complexity of recursion, and its spatial complexity is twice that of the neural network. B. XSS escape technology and action space  
Figure 3 shows a schematic diagram of the structure of common XSS attack vectors, including tags, attribute expressions, event expressions, content, and other components. According to the characteristics of XSS attack, we present four types of XSS attack escape technologies proposed in this paper, including encoding obfuscation, sensitive word substitution, position morphology transformation, and adding special characters.