# Autonomous Penetration Testing Based on Improved Deep Q-Network
## 4 Experiments
### Results and Discussion  
The first part of our experiments is comparing the convergence performance of NDSPI-DQN, decoupling NDSPI-DQN, and the DQN algorithm under the same hyperparameter values (shown in Table 4). DQN is the baseline, as it has been widely used in many prior research studies [14; 15; 16]. Three algorithms are tested using scenario 1, and we compare the performance of them by looking at two metrics: the mean rewards over training steps during training episodes and the number of steps that the agent used in each episode. As shown in Figures 4 and 5, at the start of training, the mean reward obtained by the agent is low and the mean episode steps reach the set maximum, indicating that the agent tends to use random policy to choose actions. As the training progresses, the agent gradually learns to use as few steps as possible to obtain the maximum rewards. Finally, both NDSPI-DQN and its decoupling version can converge on the approximate optimal value within limited episodes (\(\sim\)600 episodes for the decoupling version and \(\sim\)1000 for the NDSPI-DQN) while DQN fails to converge during the training process. Figure 4: Mean episode rewards versus training episodes for different algorithms in scenario 1. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Scenario** & **Hosts** & **Sensitive** & **Subnets** & **Services** & **Processes** & **Operations** & **Honeypots** \\ \hline Scenario 1 & 17 & 2 & 7 & 5 & 3 & 12 & yes \\ Scenario 2 & 80 & 2 & 17 & 10 & 4 & 18 & no \\ Scenario 3 & 100 & 2 & 21 & 10 & 4 & 18 & no \\ Scenario 4 & 150 & 2 & 31 & 10 & 4 & 18 & no \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Experimental network scenarios list. The second part of experiments is investigating the scaling performance of our improved algorithms in different generated scenarios. The general procedure is to use scenario 2, 3 and 4, which are different in network sizes, test the NDSPI-DQN algorithm and its decoupling version with the same hyperparameter values (Table 4), and measure the convergence performance by looking at the mean rewards across the limited training episodes (Figure 6). DQN is ignored for its poor performance in the first part of experiments. Figure 6 shows that the performance of the NDSPI-DQN algorithm is significantly affected by the scale of the network: when the network size increases to 120 hosts (Figure 6b) or more, the algorithm becomes difficult to converge. By contrast, the decoupling NDSPI-DQN algorithm can stably converge within limited episodes for all test network scenarios (\(\sim\)200 episodes for scenario 2, \(\sim\)300 episodes for scenario 3, and \(\sim\)400 episodes for scenario 4). The size of the network hardly affects the convergence performance of the algorithm, which means that the algorithm has better robustness in large-scale network scenarios. The experimental results indicate that the decoupling NDSPI-DQN converges faster and has the best performance in large-scale scenarios. The major reasons are:  
1. Integration of improvements promotes efficient exploration so as to alleviate the sparse reward problem. Rewards drive the agent's learning and act as supervisory signals. When the network scale increases, the reward becomes sparse, which means that the agent cannot obtain positive rewards in most of the training steps. By integrating a series of improved methods, we have greatly improved the exploration ability of the DQN algorithm, so that the agent can learn the optimal policy efficiently within a limited number of training steps per episode. 2. The reduction of action space reduces the cost of trial and error. As we mentioned previously, an action is an attack vector that is defined as \(<h,o>\), and the action space of the agent is up to \(O(M\times N)\). For example, assuming that the agent in scenario 1 has compromised the host (4,1), then it can take action \(<(2,0),scan>\) to scan the host (2,0).