# A Survey on Penetration Path Planning in Automated Penetration Testing
## 6 Simulation Environments  
Unlike traditional machine learning tasks, the training and testing data for RL are derived exclusively from the interactions between the agent and its environment. Constructing a realistic environment for training automated PT is impractical due to the high costs and security risks involved. Instead, simulation environments are often used to emulate real-world conditions and provide a safe, controlled space for training and testing.  
Several researchers have developed network scenario-based security simulation environments using Gym [140] for training and testing intelligent agents in the context of penetration path planning, such as NASAim [141], FARLAND [142; 143], CyberBattleSim [144; 145], CybORG [146], CyGIL [147], and NASimEmu [148]. Among these, CyberBattleSim stands out as it not only facilitates the training of offensive agents but also supports the simultaneous training of defensive agents within the same environment, offering a comprehensive platform for cyber security attack and defense research. Gym-based simulation environments are widely adopted due to their high degree of modularity and support for standard RL libraries and algorithms. This compatibility makes the training process for intelligent agents efficient and straightforward. The framework of these simulation environments is illustrated in Figure 6.  
Building on the foundations outside of the Gym framework, some researchers have developed more customized and efficient simulation environments. [PERSON] et al. [86] transformed a network's attack graph into a state transition graph for training RL agents, assigning reward values based on the CVSS. Expanding on this approach, [PERSON] et al. [95] optimized the generation process of state transition graphs. [PERSON] [104] proposed a training environment that more closely mimics real-world attacks and supports penetration path planning using LLM agents. [PERSON] et al. [149] developed CYST, offering a highly flexible simulation environment that supports complex behavior control and seamless integration with real-world systems. CYST is designed with a focus on high-fidelity vulnerability representation and multi-agent support, significantly enhancing the realism and practical applicability of cybersecurity training. In a further advancement, [PERSON] et al. [150] introduced a novel method that simultaneously generates complex attack scenarios and their corresponding terrain, ensuring that the generated environments are not only realistic but also logically coherent and executable. A summary of these simulation environments is provided in Table 4.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Type** & **References** & **Year** & **Source** & **Nature of Agent** & **Open Source** \\ \hline \multirow{4}{*}{Gym} & [141] & 2019 & Individual Researcher & Red * & Yes \\  & [142; 143] & 2021 & US MITRE and National Security Agency & Red * & No \\  & [144; 145] & 2021 & US Microsoft Research & Red * and Blue * & Yes \\  & [146] & 2021 & Australia Defense Science and Technology & Red * & Yes \\  & [147] & 2022 & Canada Defense Research and Development & Red * & No \\  & [148] & 2023 & Individual Researcher & Red * & Yes \\ \hline \multirow{4}{*}{Self-Defined} & [86] & 2018 & Individual Researcher & Red * & No \\  & [95] & 2020 & Individual Researcher & Red * & Yes \\ \cline{1-1}  & [104] & 2023 & Individual Researcher & Red * and Blue * & Yes \\ \cline{1-1}  & [150] & 2024 & Individual Researcher & Red * & Yes \\ \hline \hline \end{tabular}  
* Red represents offensive agent; blue represents defensive agent.  
\end{table}
Table 4: Simulation environment for penetration path planning.  
Figure 6: Simulation environments framework.