# STPA-RL: Integrating Reinforcement Learning into STPA for Loss Scenario Exploration
## 8 Conclusions  
Infusing trial-and-error-based techniques such as RL into the realm of safety analysis remains a relatively uncommon practice, underscoring the urgent need for research in this domain. In light of this gap, our study endeavors to bridge this divide by delving into the integration of RL with STPA. Recognizing the paramount importance of deriving loss scenarios within STPA for bolstering safety, our research focuses on identifying potential hazards and their consequential outcomes within a system. Traditionally, STPA analysts cataloged loss scenarios using context tables with the combinations of causal factors. However, this approach has drawbacks as it is difficult to cover all the risks that may arise in practice and may not adequately prioritize risks based on their impact and likelihood of occurrence. Therefore, this paper proposes the applicability of RL to STPA and presents the experimentation result with the case study of the PSD system. By implementing the current control environment and rewarding safe actions, RL enables the identification of self-guided loss scenarios that lead to hazards. RL allows us to freely model various systems and interactions when implementing a control environment. The choice of RL algorithms to suit the requirements of the system can provide the ability to track loss scenarios that efficiently reach hazards and reflect real-world systems' uncertainty.  
In this paper, we derived 4 hazards, 11 UCAs, and 38 loss scenarios by using STPA-RL for the PSD system. These loss scenarios identify the flow in which control actions are performed incorrectly and determine the frequency of hazards that may occur. Tracking causes of hazards is efficient while searching for potentially unsafe situations in the system. Seeing what happens rather than guessing the likelihood of the presence of causal factors can point to the specific state from the flows. Based on the results of the testing, this derived information supports changing the system control algorithm where high-frequency hazards appear and set safety constraints to reduce hazards before developing the system. This helps prioritize which hazards to pay attention to in providing safety.  
Through system simulation with STPA-RL, we provide an easy test to discover problems, tweak control algorithms, and devise a design-change method. It is particularly suitable for application in the early stages of system development, allowing for the identification of potential risks and loss scenarios before the actual system is constructed and deployed. In conclusion, our approach not only enhances system safety through simulation but also allows the agent to learn about conditions leading to unsafe situations and appropriate responses. This enables the swift development of proactive measures, improving safety and reducing costs associated with system failures. Future studies should include verifying the suitability of the RL model for the STPA-analyzed system and comparing loss scenario results for larger systems.  
Conceptualization, J.C., R.K. and G.K.; methodology, J.C. and R.K.; validation, R.K. and G.K.; writing--original draft preparation, J.C. and R.K.; writing--review and editing, R.K. and G.K.; visualization, J.C.; supervision, G.K. All authors have read and agreed to the published version of the manuscript.  
This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (No. 2021-0-00122, Safety Analysis and Verification Tool Technology Development for High Safety Software Development).  
Not applicable.  
The raw data supporting the conclusions of this article will be made available by the authors on request.  
The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.