# STPA-RL: Integrating Reinforcement Learning into STPA for Loss Scenario Exploration
## 4 Case Study: Platform Screen Door
### Applying STPA-RL  
To derive a loss scenario, STPA-RL is utilized following the steps below as in Figure 3.  
1. Implement the PSD environment with the results of STPA 1-3 stages. Define states, actions, hazards, control algorithms, and the reward structure of the environment. 2. Train the model by using the A2C algorithm and rewarding the safe action. The algorithm can be chosen differently depending on the characteristics of the system. 3. Derive loss scenarios from the state trajectories that reach the hazard state with unsafe action. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Control Action** & **Type A: Not Providing Causes Hazard** & **Type B: Providing Causes Hazard** & **Type C: Too Late/To Soon/Out of Order** & **Type D: Stoped Too Soon/Applied Too Long** \\ \hline \multirow{3}{*}{Close Door} & UCA01: not closing door & UCA03: closing door after & UCA05: closed the door too \\  & when train is ready and detected that obstacle does & detecting obstacle exists (H02). & late that obstacle appeared \\  & not exist (H04). & UCA04: closing door when & while closing (H02). \\  & UCA02: not closing door & UCA04: closing door when & UCA06: closed the door too \\  & the door needs to be opened & the door needs to be opened & late while train is ready, so \\  & while train is moving (H01). & because the train is stopped (H03). & the train started to move \\ \hline \multirow{3}{*}{Open Door} & UCA07: not opening door & UCA09: opening door when & UCA11: detected obstacle \\  & after detecting obstacle & UCA09: opening door when & and try to open door, but it \\  & exists and while door is & train is not stopped (H01). & was too late that the door \\  & closing (H02). & stop closed (H02). & \\  & UCA08: not opening door & UCA10: opening door when & got closed (H02). \\  & after the train is stopped & obstacle is not-detected & \\  & (H03). & while train is ready (H01). & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Unsafe control actions. Figure 5: Control structure of the platform screen door. Initially, state variables and inputs are defined to implement the PSD environment with the system's control algorithm and feedback. These variables are structured in a 4-tuple format called \(Step\), comprising three state variables of Process Model and agent action. Formally, we define a \(Step\) as follows:  
* \(Step=S\times A\)
* \(S=door\_position\times train\_motion\times obstacle\_state\)
* \(door\_position=\{fully\_closed,opening,fully\_opened,closing\}\)
* \(train\_motion=\{stop,ready,moving\}\)
* \(obstacle\_state=\{not\_exist,exists\}\)
* \(A=\{close,open\}\)  
Informally, \(Step\) means what action the agent has selected based on the value of the process model at a specific time. Values of \(Step\) are all non-negative integers listed starting from zero and increasing sequentially. In the system environment, the agent can perform two actions: close or open. Performing these actions will cause a change in the \(door\_position\) state. It changes from \(fully\_closed\) to \(opening\) when the action open was made, and from \(opening\) to \(fully\_opened\) when it is opened again. When it is \(fully\_opened\) and the action close was made, it changes to \(closing\), and when it is closed again, it becomes \(fully\_closed\). \(train\_motion\) varies depending on the current state of \(door\_position\) and the state of the train. \(obstacle\_state\) allows objects to exist or not exist at random rates. In particular, it is implemented following the hazards and safety constraints defined in identifying risks. Therefore, if the agent performs the action close, the priority of the hazard is H02, and the next hazard that is not applicable is H03. If the action open is performed, H04 is the priority, and the next is set to H01. The agent receives a reward when the action open is performed while \(obstacle\_state\) is \(1\) or \(train\_motion\) is \(0\). In addition, when \(obstacle\_state\) is \(0\) and \(train\_motion\) is \(1\) or \(2\), the action close is rewarded.