## 4 Survey from the Attacker's Perspective
### Related Work
[PERSON] built an automated penetration testing system called \"IAPTS\" to solve the problem of automated Pentest in large network environments [46]. The system preprocesses the original data, and obtains the evaluations by constructing POMDP model to use PERSEUS [15], GIP [47] and PEGASUS [60] algorithms in combination, and then made it available to human experts. The experiments are carried out in small and medium networks. Experimental results showed that the improved GIP algorithm performed better than manual experts on small networks, but the computational cost was higher on medium LANs, requiring more in-depth research on the algorithm. **Pentest Based on MDP:** Through existing research, it can be seen that when using POMDP to model the attack decision-making process, due to the computational complexity of algorithms, the scale of applicable scenarios is limited. Therefore, more and more researchers choose to model the \"Pentest\" task with MDP, where the outcomes of actions are deterministic [40]. This change allows researchers to build the task with more consideration for the similarity of actions to the real world, the changes in the environment, and the presence of the opponent and so on. [PERSON] designed and built a new open-source network attack simulator (NAS) for studying the attack decision-making process [48], and compared different RL algorithms\" convergence times. This system promoted the research of reinforcement learning algorithms in the direction of \"Pentest\", but the abstraction of state space is too simple and lacks consideration of the dynamic environment, so it needs to be further deepened. [PERSON] et al., focused on the problem of large-scale action space reduction [49]. The author combined the DQN algorithm with \"Wolpertinger Architecture(WA)\" [50] to embed multilevel action. The sensitive machine attacked proportion of the proposed algorithm was obviously better than other algorithms, and was also applicable in the medium and large-scale network. Besides, the graph embedding method is first used to represent the state space, which is worth exploring. By focusing on how to reduce the number of features represented in the state space and how to approximate the state-action space with a small set of features, [PERSON] et al. performed the experiment to compare the advantages and disadvantages of three algorithms: Q-learning based on table, RNN with Q-learning and RNN with Q-learning after network configuration changes [51]. The results showed that, compared with Q-learning based on table, RNN with Q-learning can converge faster and respond to the change of environment, showing the potential of model-free learning for automated pentest. [PERSON] et al., used the real payloads of attack tools as actions for attack decision-making, and applied A2C, Q-learning and SARSA algorithms to solve the task [52]. The experimental results showed that the A2C algorithm has the best effect in cumulative average reward. However, the defined state representation in the article lacks a certain degree of interpretability. In order to test the stability and generalization of various DRL algorithms [53], [PERSON] et al. built five experimental scenarios with different topologies, services, and application configurations based on the Network Attack Simulator (NASim) platform [59]. In the test of stability, the returns of DQN algorithm and PPO were relatively stable. In the test of generalization, when the evaluation network deviates from the training environment, the performance of DQN and PPO algorithms failed to adapt to all experimental scenarios, even worse than the performance of random algorithm in the same environment, which pointed out the possible research directions. **Pentest Based on Markov Game:** Markov Game in \"Pentest\" treats the attacker and the defender as the competing agents and designs a Game model to compare the effectiveness of the two sides's policies. [PERSON] et al.