# Autonomous Penetration Testing Based on Improved Deep Q-Network
## 3 Methodology
### Extensions to DQN  
Five extensions to DQN are selected for the reason that they are proved to strengthen the exploration of the agent from different perspectives and each of them improves the overall performance. Soft \(Q\)-learning [25] is a method proposed for learning the maximum entropy policies. It allows for learning diverse strategies and better exploration, and it is proved to perform better in continuous action tasks. When combining it with DQN to handle discrete action tasks, the target \(Q\) value is calculated as Equation (6), where \(\sigma\) is a parameter used to measure the relative importance of entropy and reward. \[y^{soft}=r+\gamma[\sigma log\sum_{a^{\prime}}exp(\frac{1}{\sigma}Q(s^{\prime}, a^{\prime};\theta^{-}))]. \tag{6}\]  
As for the action selection strategy, instead of using the greedy policy in traditional DQN, soft Q-learning uses a _softmax_ policy as Equation (7), which improves the exploration by assigning a probability of being selected to each action. \[\pi(a|s)=softmax_{a}(\frac{1}{\sigma}Q(s,a;\theta)). \tag{7}\]  
Dueling network [26] improves the performance of policy evaluation by changing the neural network architecture of DQN. It decomposes the \(Q\) value function as the sum of value function and advantage function:  
\[Q(s,a;\psi)=V(s;\theta,\beta)+[A(s,a;\theta,\alpha)-\frac{1}{|\mathcal{A}|} \sum_{a^{\prime}}A(s,a^{\prime};\theta,\alpha)], \tag{8}\]  
where \(\theta\), \(\alpha\), and \(\beta\) denote respectively the shared network parameters, the value stream parameters, and the advantage stream parameters, and \(\psi\) is their concatenation. DQN uses experience replay to ensure that samples are independent and identically distributed, however, the traditional random sampling method ignores the significance of different samples. In order to make experience replay more efficient, prioritized experience replay [27] uses temporal difference error (TD error, \(\delta\)) to measure the significance of transitions. It samples transition with probability  
\[P(i)=\frac{p_{i}^{\alpha}}{\sum\limits_{k}p_{k}^{\alpha}}, \tag{9}\]  
where \(\alpha\) denotes how much prioritization is used and \(p(i)=|\delta_{i}|+\mu\) is the priority of transitions with a small positive constant \(\mu\). Noisy net [28] improves efficient exploration by adding parametric Gaussian noise to the linear layer of DQN. DQN applies \(\epsilon\)-greedy policy to select actions greedily based on the \(Q\) function or randomly pick actions with probability \(\epsilon\), compared to which noisy net has more abundant exploration on problems with large action space. The noisy linear layer of noisy net is defined as  
\[y=b+b_{noisy}\odot\epsilon^{b}+(W+W_{noisy}\odot\epsilon^{w})x, \tag{10}\]  
where \(\epsilon^{b},\epsilon^{w}\) denote the noise random variables, and \(\odot\) represents the elementwise product. This transformation is used to replace the standard linear \(y=b+Wx\). [PERSON] et al. [29] use curiosity as the internal reward signal to drive the agent to explore in an environment where external rewards are sparse. The external reward is given by the environment while the internal is generated from the self-supervised intrinsic curiosity module (ICM); thus, the goal of the curiosity-driven agent is maximizing the sum of internal rewards \(r^{i}\) and external rewards \(r^{e}\). ICM consists of two submodels: the inverse model is trained to predict the action \(\hat{a}_{t}\), given the feature vectors \(\boldsymbol{\phi}(s_{t}),\boldsymbol{\phi}(s_{t+1})\) that are encoded by the state \(s_{t},s_{t+1}\); the forward model takes as input \(\boldsymbol{\phi}(s_{t}),a_{t}\) and predicts the feature vector of the next state \(\boldsymbol{\hat{\phi}}(s_{t+1})\). Curiosity is defined as the prediction error of the environment state; thus, the internal reward is calculated as  
\[r^{i}_{t}=\frac{\eta}{2}\big{\|}\boldsymbol{\hat{\phi}}(s_{t+1})-\boldsymbol{ \phi}(s_{t+1})\big{\|}_{2}^{2}, \tag{11}\]  
where \(\eta>0\) is a scaling factor.