## 2 Background  
### Manual Penetration Test and AI-Driven Penetration Test  
A penetration test is generally regarded as a vulnerability assessment method that simulates malicious hackers' attacks on target networks. There are currently various frameworks and processes for the specific steps of penetration testing. [PERSON] et al. [12] describe the whole process of vulnerability assessment and penetration testing (VAPT) and all the methods, models and criteria. At the same time, they divided the penetration test into four major stages: planning and preparation stage, detection and penetration exploitation stage, post-penetration and data filtering stage and reporting and trace clearing stage. The proportion of time and resources consumed at each stage in the whole penetration test process is shown in Figure 1.  
It can be seen from Figure 1 that the preliminary collecting of information and the exploitation of the target network occupy most of the time and resources in the penetration test process. These tasks are often repetitive and complicated and require certain expert knowledge and experience [13]. The entire penetration test process consumes a lot of human costs. Therefore, the realization of an automated penetration test is currently a prospective research direction. With the development of artificial intelligence, AI-driven penetration tests provide new solutions to the problems that existed in manual penetration tests. The comparison of AI-driven penetration tests with manual penetration tests is shown in Figure 2.  
Figure 1: Penetration test phases—effort and time distribution. Figure 2: Comparison of manual penetration test with AI-driven penetration test. To solve the problems of high human cost and expert knowledge, the automated penetration test was proposed. At present, mature automated penetration test tools include APT2, Autosploit and Awesome-Hacking-Tools. These penetration test tools improve the efficiency of penetration tests. However, there are still some problems [14]. From the host level, these penetration test tools simply integrate existing network attack tools and lack the reasoning ability of network attack and defense knowledge. They cannot intelligently select attack payloads and configure the paframeters of the payload for a penetration test based on the state of the target host. From the network level, most of these penetration test tools perform a vulnerability assessment for a single host without expanding to the entire target network. In addition, these tools lack the ability to discover potential attack paths of the target network. The key to solving the above problems is how to intelligently realize the selection of vulnerability exploitation for targets and the selection of attack payloads. Artificial intelligence is widely used in automated penetration tests and could improve the efficiency of the potential attack path discovery in the target network. According to the current target network status, the artificial intelligent penetration test (AIPT) selects the exploit target and related method intelligently. In essence, the AIPT solves the sequential decision-making problem in a specific environment [15]. In the interactive process of continuous learning and training, AIPT could make a decision autonomously for the penetration test of the target network. RL is to learn how to map the current states into actions. It makes the agent dynamically explore and exploit the environment to find appropriate strategies for actions [16]. One of the main advantages of using RL is that it allows us to deal with problems without assuming any presupposition knowledge or model of the probability of a given state and action's outcome. Instead, it allows agents to learn this knowledge, which provides a solution to automated penetration test problems [17] by interacting with the environment. [PERSON] et al. modeled a penetration test as a POMDP (partially observable Markov decision process) [18].