# STPA-RL: Integrating Reinforcement Learning into STPA for Loss Scenario Exploration
## 7 Discussion  
### Safety Analysis with Reinforcement Learning  
STPA has been recommended as an international standard for evaluations of automotive-related safety-critical systems, as indicated by SAE J3187 (SAE, Sydney, Australia) [31]. Therefore, there is a need for various studies exploring the diverse applications and utilization of STPA in the context of safety assessments. STPA, as a structured and systematic safety analysis method, can serve as a solid foundation for guiding the integration of Machine Learning (ML). Its emphasis on identifying unsafe control actions and analyzing causal factors aligns well with the requirements of ML algorithms in understanding system behaviors [32].  
The system modeling method using MDP for RL is not subject to major formal restrictions and helps improve the implementation of control algorithms in current systems. [PERSON] et al. [7] introduced a method that capitalizes on partially observable Markov Decision Processes to amalgamate nominal actions of the system along with unsafe control actions posed by the STPA. [PERSON] et al. [33] use the STPA method to formulate a parallel control model for the systems of maritime autonomous surface ships and simulates this with Markov Chains.  
Modern systems feature complex interactions and numerous subsystems, making control challenging and verification costly. Nevertheless, our proposed method can handle these systems that require many states and continuously identify state changes that lead to losses in control algorithms by deriving loss scenarios for potential risks while testing the learning results. RL is highly adaptable to various environmental and system changes and provides continuous learning from experience through trial and error [8], enabling the performance improvement of the system. It is available in situations where the transition probability is unknown and provides a measure of learning accuracy to derive loss scenarios.  
However, during the process of implementing the system into an environment, accuracy is dependent on the accuracy of the control algorithm and the information provided by STPA.  
Despite this limitation, this method can be implemented close to the actual system, enabling the identification of causes of state changes for possible real-life hazards. Formal methods such as Model Checking on STPA [15; 34], are idealistic and systematic but require a high level of knowledge and experience to be performed by engineers of the system in the real world. Their ability to explore and learn from various scenarios contributes to accurately identifying scenarios that could be missed in conventional analysis and design. The control algorithm for the STPA-based system shall be detailed to facilitate the modeling of the system environment for the RL algorithm.  
In [35], investigating tes- input generation using RL, the focus is mainly on exploring and improving specific behaviors of software systems or detecting bugs by maximizing rewards. When applying STPA-RL, the primary objective is to emphasize safety and identify unsafe states or actions within the system to derive and enhance loss scenarios.