# A Survey on Penetration Path Planning in Automated Penetration Testing
## 5 Planning Methods in Penetration Path Planning
### Reinforcement Learning
#### 5.2.2 Deep Reinforcement Learning
Besides refining DRL algorithms, researchers are enhancing the realism and complexity of simulated network environments to bolster the robustness of intelligent agents' strategies. [PERSON] et al. [103] contended that conventional attack graphs, predicated on CVSS, inadequately capture the intricacies of target networks. To address this, they introduced the concept of 'cyber terrain', which integrates firewalls, network protocols, and software into the MDP model, thereby enhancing the decision-making capabilities of intelligent agents. [PERSON] et al. [104] explored using RL for penetration path planning in environments with dynamic defenses, like port hopping and network reorganization. As network complexity grows, it becomes harder for agents to find effective strategies, making the learning process more challenging. Policy-based RL constructs a policy network through function approximation, optimizing its parameters to maximize reward [105]. In contrast to the value function approach, which is limited to discrete actions, the policy gradient approach can accommodate continuous actions. The actor-critic framework combines the strengths of both policy and value functions, optimizing them simultaneously for efficient and stable training [106]. In this framework, the policy network selects actions, while the value function network evaluates state value and provides feedback, leading to more effective learning. Currently, most of the cutting-edge policy-based algorithms belong to the actor-critic algorithmic architecture, such as Proximal Policy Optimization (PPO) [107], Advantage Actor-Critic (A2C), Asynchronous Advantage Actor-Critic [108], Soft Actor-Critic [109], etc. [PERSON] et al. [110] first applied the A2C algorithm to penetration path planning in post-penetration and combined it with the PowerShell Empire post-penetration framework to realize automated post-penetration. Under the same benchmark conditions, A2C shows better experimental results than traditional value functions, proving the superiority of the actor-critic framework. Addressing the challenge of multi-target penetration path planning, [PERSON] et al. [111] have integrated a curiosity-driven mechanism with multi-objective RL, leveraging the PPO algorithm to achieve strategic planning for multiple objectives. In the context of partially observable network environments, [PERSON] et al. [112] have introduced an efficient partially observable RL agent, which is an enhancement of the PPO algorithm. This algorithm employs an implicit belief module to estimate the probability distribution of the current environmental state and utilizes a distributed RL framework known as \"Sample Factory\" for efficient data collection and parameter optimization across multiple parallel environments. This approach significantly enhances the training efficiency of the agent and bolsters its capacity to address penetration path planning challenges within partially observable settings. The actor-critic framework faces challenges related to high computational costs when dealing with large action spaces. [PERSON] et al. [113] introduced the wolpertinger architecture (WA), which is an extension of the actor-critic framework. WA leverages the actor-critic framework for policy optimization while utilizing KNN to efficiently explore and exploit the discrete action space, ensuring effective decision-making even in environments with a vast number of potential actions. [PERSON] et al. [114] further refined this by dividing penetration actions into three levels--action type, network structure, and service vulnerability--using WA. This hierarchical approach improves the representation of complex actions and boosts training efficiency. Building on this, [PERSON] et al. [115] proposed a dual-intelligent-body architecture for penetration path planning using the A2C algorithm. One body selects the target host, while the other performs scanning or exploiting actions, enhancing the decision-making process. [PERSON] et al.