# STPA-RL: Integrating Reinforcement Learning into STPA for Loss Scenario Exploration
## 5 Results and Analysis  
Model simulations were conducted over 1000 episodes. The model can now identify loss scenarios that lead to the hazard state, tracking previous paths--the trajectories of states and actions until UCA. Initially, we analyze the testing results to derive loss scenarios for the PSD system, focusing on hazard frequency. When there is a lack of statistically accumulated data for the system, engineers sometimes evaluate the system based on intuition or experience. To address this, our goal is to provide quantitative metrics for safety design decisions by deriving the frequency of loss scenarios through simulations. Hazard frequency refers to the occurrence of a hazard corresponding to the door position, train motion, obstacle state, and action for hazards in H01 to H04. In Figure 5(a), all possible states of the PSD system are depicted, half of which are safety states, and the other half are the hazard states. Among the hazard states, H02 comprises the majority. However, in Figure 5(b), during the system testing, it was discovered that H04 hazards occurred most frequently, accounting for 38% of the total hazard frequency. The hazard frequency analysis method identifies critical PSD system states by measuring each state's occurrence frequency and corresponding hazards in the RL model simulation, estimating hazard probabilities, prioritizing them, and repeatedly running the model to ensure result consistency. Eight examples of hazard trajectories (\(\sigma_{H}\)) and their corresponding loss scenarios (\(\sigma_{LS}\)) are presented in Tables 2 and 3. As shown in the last row of the Table 2, when the door position is fully opened, train motion is ready, and there is no obstacle detected, the action of open results in a hazard with a frequency of 52%, making up 59% of all H01 hazards. The third row indicates that when learning is highly effective, the safe action will be selected with a frequency of 83%. Identifying critical system states with a high risk of hazards is possible through this analysis. Table 3 presents a list of loss scenario paths for the PSD system, which show the sequence of normal actions and states that lead to the hazard state. These paths help identify the specific state changes that caused the hazard to occur. The final step in the paths corresponds to the hazard state reached as a result of the agent's action. To determine the cause of the hazard, analyze the preceding steps in the path. By these paths, we find the design mistake of the control algorithm or the origin of the loss for the safety of the system. The last row shows an example loss scenario of (1,0,1,1) scenario, which is 11.31% of the total, which resulted in a hazard state with (2,1,0,1) of H04. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Hazard** & **Door Position** & **Train Motion** & **Obstacle State** & **Action = Close** & **Action = Open** & **Hazard** \\  & & & & & & **Frequency** \\ \hline H01 & fully closed & moving & not exist & 72\% (safe) & 28\% (hazard) & 61\% of H01 \\ H01 & opening & moving & not exist & 43\% (safe) & 57\% (hazard) & 31\% of H01 \\ H02 & closing & moving & exists & 17\% (hazard) & 83\% (safe) & 1\% of H02 \\ H02 & fully closed & moving & exists & 52\% (hazard) & 48\% (safe) & 39\% of H02 \\ H03 & opening & stop & exists & 57\% (hazard) & 43\% (safe) & 36\% of H03 \\ H03 & fully closed & stop & not exist & 72\% (hazard) & 28\% (safe) & 54\% of H03 \\ H04 & closing & ready & not exist & 35\% (safe) & 65\% (hazard) & 40\% of H04 \\ H04 & fully opened & ready & not exist & 48\% (safe) & 52\% (hazard) & 59\% of H04 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Sample cases of hazard frequency and action ratio. Figure 6: Result of hazard proportion. (**a**) All possible states of PSD system, (**b**) hazard steps of loss scenarios.