# A Survey on Penetration Path Planning in Automated Penetration Testing
## 5 Planning Methods in Penetration Path Planning
### Reinforcement Learning
#### 5.2.2 Deep Reinforcement Learning  
Deep learning has surpassed human perception in some scenarios. Deep reinforcement learning (DRL) combines deep learning's perceptual capabilities with RL to handle complex, high-dimensional environments and make decisions [89]. The primary methods in RL fall into three categories: value-based, policy-based, and model-based. Value-based DRL approximates the value or action-value function using deep neural networks, updated via temporal difference learning. Deep Q-Network (DQN) [90; 91] is the first algorithm to demonstrate that deep learning combined with RL can solve complex control tasks. The success of the DQN inaugurated a new epoch in DRL, giving rise to the development of numerous subsequent algorithms. The DDQN [92] later refined this with a double Q-learning approach to stabilize learning, while the DRQN [93] improved upon the DQN by incorporating LSTM for better handling of partial observability. Additionally, enhancements such as dueling networks, prioritized replay, and noise mechanisms have further propelled the advancement of these algorithms, making them more robust and efficient in applications. In recent years, DQN and its derivatives have been widely used in automated PT. [PERSON] et al. [94] combined an MSF and DQN agent to create a framework that uses Nessus scan results to construct an attack graph and simulation environment and then plans optimal penetration paths with a DQN. [PERSON] et al. [95] developed an automated PT framework based on the DQN to plan optimal penetration paths for real-world network topologies. In further development, [PERSON] and [PERSON] [96] enhanced the DRL algorithm with the DDQN, thereby improving the performance of penetration path planning. Modifying the neural network architecture can enhance the agent's ability to handle specific problem characteristics, such as leveraging historical information to address partial observability in POMDPs. [PERSON] et al. [97] proposed a framework for host-based PT that uses Recurrent Neural Networks to refine strategies over time. However, it faces challenges with training stability and convergence. To overcome these, [PERSON] et al. [98] applied POMDPs to simulate the testing process and proposed the ND3 RQN algorithm, which improves on the DRQN by using historical data to enhance memory and exploration. [PERSON] al. [99] further advanced the DRQN with the DEU DRQNper algorithm, incorporating prioritized experience replay, a hybrid exploration and selection strategy, and Double DQN to calculate Q-values for effective path planning in partially observable environments. Traditional value-based RL methods require enumerating all possible actions, necessitating the computation and comparison of expected returns for each action in every state to identify the optimal one. Consequently, these methods become computationally prohibitive as the action space grows large. To address this, researchers have introduced various strategies. [PERSON] et al. [100] proposed an action decomposition strategy for PT that divides the action space into L subspaces and is tackled with L DQN agents. This strategy enhances DQN's exploration efficiency within large action spaces. [PERSON] et al. [101] presented an enhanced DQN algorithm that dynamically constrains the agent's action space by the expanding effective action space as the agent navigates the network environment. This method enhances the agent's decision-making accuracy and eliminates ineffective exploration paths. The curiosity mechanism encourages intelligent agents to actively explore uncharted territories, thereby efficiently discovering valuable actions. [PERSON] et al. [102] improved DQN's exploration with an intrinsic curiosity module, proposing the NDSPI-DQN algorithm. It integrates noisy networks and other techniques to boost training and penetration path planning in complex networks. Additionally, they used action vector decomposition to cut exploration costs and complexity.