# Autonomous Penetration Testing Based on Improved Deep Q-Network
## 1 Introduction  
Penetration testing (short PT or pentesting) is active and authorized simulated cyberattack, aiming at assessing cybersecurity and discovering the hidden vulnerabilities. Currently, pentesting plays a crucial role in strengthening the defense of computer systems against cyberattacks, as digital assets are more frequently exposed to hackers' persistent, varied, and complex threats than ever before. However, the traditional pentesting methods mainly rely on highly skilled cybersecurity experts with domain-specific knowledge and experience, which requires prohibitive labor and time costs. Consequently, pentesting is conducted infrequently in many organizations. Autonomous pentesting can be a solution to this problem. Compared to the human-based method, performing pentesting autonomously is a more efficient and intelligent way. It can realize regular security testing without expensive specialists and make the pentesting process accessible to those nonexperts. Researches on autonomous pentesting originate from the attack tree [1] and attack graph [2]. Both methods offer interpretable and formal models for evaluating system security and discovering potential attack paths. The limitation of them is that they need complete information of the target topology and all hosts' configuration, which is unrealistic from the perspective of real-world attackers. They cannot be applied to large network scenarios due to the state space explosion and their complex modeling process. Another approach to autonomous pentesting is the partially observable Markov decision process (POMDP) model [3; 4; 5; 6]. Using the POMDP method allows modeling the incompleteknowledge characteristic of the real-world hackers and accounting for the uncertainty in the pentesting process. However, the POMDP-based method is computationally expensive in large-scale network scenarios. Currently, remarkable success has been achieved by reinforcement learning (RL), and in some games, the RL-based agents have outperformed human players, such as AlphaGo [7], OpenAI Five [8], Alpha Star [9], and MOBA AI [10]. Similar to many game rules, pentesting is also a process of dynamic decision making based on the state of the environment. Expressing the pentesting process in terms of a game is a promising way to realize autonomous pentesting. The RL-based pentesting agent can be trained to observe the network environment and learn the optimal policy using the trial-and-error method [11]. One of the main advantages of using the reinforcement learning approach is that the agent can learn without assumed prior knowledge for any given scenario. Recently reinforcement learning algorithms, such as tabular Q-learning and deep Q-network (DQN), have been applied to solve the pentesting problem in some researches. Tabular Q-learning has been applied to solving CTF (capture the flag) problems [12] and exploiting SQL injection vulnerabilities [13], proving the feasibility of applying reinforcement learning to simple and particular autonomous pentesting problems. Compared to the tabular Q-learning, DQN based methods [14; 15; 16] have a natural advantage in dealing with large state pace problems using the deep neural network for function approximation. However, the previous DQN-based methods are still troubled in the sparse reward problem [17] and the large action space caused by the increase in network size. Current research studies on autonomous pentesting based on reinforcement learning have only been applied in simple network scenarios, and there is a large room for improvement in convergence and scaling performance. On the basis of previous research studies, this paper models pentesting as a Markov decision process (MDP) problem and analyzes the main challenges of autonomous pentesting in large-scale networks. Then, an improved DQN algorithm is proposed to train the pentesting agent that can learn the optimal attack policy without prior knowledge. The proposed algorithm has better convergence performance in large-scale networks.