# Autonomous Penetration Testing Based on Improved Deep Q-Network
## 3 Methodology
### Integration and Decoupling  
We combine those extensions together and propose the NDSPI-DQN algorithm to train the pentesting agent. As shown in Figure 1, we apply the dueling network architecture with added noise for both the \(Q\) network and the target network. The input of the neural network is the environment state vector, the hidden layers are fully connected to the input and output layers, and the output is the predicted values of all actions. \(\epsilon\)-greedy policy is no longer used, but the action is directly selected by the _softmax_ policy as Equation (7). The goal is set to maximize both the external reward and internal reward that is generated by ICM. Prioritized experience replay to store and sample important transitions more frequently. After combining those extensions, the action-value function can be written as \(Q(s,a,\epsilon;\psi)\), where \(\epsilon\) and \(\psi\), respectively, denote the noise variable and the concatenation network parameters. Thus, the target \(Q\) value is calculated as  
\[y=r+\gamma[\sigma log\sum_{a^{\prime}}exp(\frac{1}{\sigma}Q(s^{\prime},a^{ \prime},\epsilon^{\prime};\psi^{-}))], \tag{12}\]  
where \(r=r^{i}+r^{e}\) and the overall loss function is defined as  
\[L(\psi,\theta_{I},\theta_{F})=\lambda\mathbb{E}_{(s,a,F,s^{\prime})\sim D}[y-Q (s,a,\epsilon;\psi)]^{2}+(1-\delta)L_{I}+\delta L_{F}, \tag{13}\]  
where \(\lambda\) and \(\delta\) are the scalar that weights the importance of the loss of the policy network and ICM model. However, it is not enough to merely improve the agent's exploration. In order to reduce the cost of trial and error in the exploration process of the agent, we reduce the action space by decoupling the action that can be seen as an attack vector \(a=<h,o>\). As illustrated in Figure 2, the crucial insight of our method is that we split the the NDSPI-DQN network into two separate streams: one estimates the value of hosts, and the other estimates the value of the operations, which means the agent can independently select the victim host and the operation taken against the target at one time according to the current environment state vector. For the agent in an environment with \(M\) target hosts and \(N\) executable operations, the output of the neural network is divided into two parts: the \(Q\) values of \(M\) hosts and the \(Q\) values of \(N\) operations; thus, the action space is reduced from the original \(O(M\times N)\) to \(O(M+N)\). Two streams independently output the elements of actions given the input state using \(\pi(h|s)=softmax_{h}(\frac{1}{\sigma_{h}}Q(s,h,\varepsilon;\psi_{h}))\) and \(\pi(o|s)=softmax_{o}(\frac{1}{\sigma_{o}}Q(s,o,\varepsilon;\psi_{o}))\). The selected host \(h\) and operation \(o\) are then combined into the attack vector to act on the  
Figure 1: The learning cycle (**Top**) and schematic illustration of the NDSPI-DQN (bottom). (**Top**) The agent observes current environment state \(s_{t}\) and outputs an action \(a_{t}\) according to the learned policy; then, the external reward \(r_{t}^{e}\) is obtained from the environment. Meanwhile, ICM takes \((s_{t},a_{t},s_{t+1})\) as input and outputs the internal reward \(r_{t}^{i}\). Both \(r_{t}^{i}\) and \(r_{t}^{e}\) are used to optimize the policy. (**Bottom**) The details of the algorithm architecture are explained in Section 3.2. The orange layers denote fully connected layers and the red layers denote dueling network architectures with noisy linear layers. The input to the neural network is the vector of environment states, and the output is the action value. Figure 2: Schematic illustration of the decoupling NDSPI-DQN. Two streams share the hidden layers and each of them outputs its \(Q\) value using the dueling network architecture with the noisy linear layers. environment.