# STPA-RL: Integrating Reinforcement Learning into STPA for Loss Scenario Exploration
###### Abstract  
Experience-based methods like reinforcement learning (RL) are often deemed less suitable for the safety field due to concerns about potential safety issues. To bridge this gap, we introduce STPA-RL, a methodology that integrates RL with System-Theoretic Process Analysis (STPA). STPA is a safety analysis technique that identifies causative factors leading to unsafe control actions and system hazards through loss scenarios. In the context of STPA-RL, we formalize the Markov Decision Process based on STPA analysis results to incorporate control algorithms into the system environment. The agent learns safe actions through reward-based learning, tracking potential hazard paths to validate system safety. Specifically, by analyzing various loss scenarios related to the Platform Screen Door, we assess the applicability of the proposed approach by evaluating hazard trajectory graphs and hazard frequencies in the system. This paper streamlines the RL process for loss scenario identification through STPA, contributing to self-guided loss scenarios and diverse system modeling. Additionally, it offers effective simulations for proactive development to enhance system safety and provide practical assistance in the safety field. S 2024 14 2916 202  
environment. It models the changing appearance of a system based on feedback from its components like the real world. The key activity of STPA is identifying loss scenarios, which explain unsafe control measures and possible causes of risk [9]. STPA has the advantage of improving the overall safety and reliability of the system by proactively identifying system vulnerabilities and preventing safety accidents [10]. A human safety analyst's trial-and-error approach to examining combinations of risk variables by hand, especially for highly complex systems, becomes impractical [11]. In addressing this challenge, the utilization of RL agents in lieu of humans can efficiently contribute to the automated derivation of loss scenarios. Furthermore, similar to potential bugs in software, systems involve faults and hazards. There are techniques that utilize RL for automating the detection of sequences leading to system crashes or halts, resembling bug scenarios, as reported by [PERSON] et al. [12]. In this context, our objective was to leverage RL to derive loss scenarios related to potential hazards in systems analyzed through STPA. If the agent's learning is carried out at a tolerable level, it can acquire the ability to track the previous system state and the interaction that caused the situation where risks occurred in the system. As loss scenarios represent possible events or sequences of events that may result in undesired outcomes or compromised safety in a system, analysts must assess the likelihood and consequences of each potential hazard. In this regard, RL provides a more advantageous approach by enabling the evaluation of all these factors. In this paper, we present an approach named STPA-RL. By deriving loss scenarios and tracking changes in the system state, we identify the flow in which control commands are performed incorrectly and help inform design decisions and manage risk to ensure the safety of complex systems. By implementing an RL environment with MDP and rewarding safe actions, our approach allows for the identification of self-guided loss scenarios that lead to hazardous actions, enabling the discovery of potential loss scenarios at the system level. It provides the ability to simulate system behavior under different environmental conditions using RL before applying it to the actual system. In summary, our contribution is threefold: First, we present STPA-RL, which combines the safety analysis technique STPA with reinforcement learning. Second, we implement the RL environment using MDPs based on the system analysis results obtained by STPA. Third, we derive loss scenarios for the Platform Screen Door system from the state hazard trajectories and compare the probability of hazard occurrence.