# STPA-RL: Integrating Reinforcement Learning into STPA for Loss Scenario Exploration
## 2 Background
### Reinforcement Learning  
RL is a sub-field of Machine Learning that involves an agent learning to make decisions by interacting with an environment [18]. RL is based on the concept of trial and error, where the agent receives feedback in the form of rewards or penalties based on its actions. MDP is used to mathematically define and model RL problems. And it is defined as a tuple consisting of a set of states \(S\), a set of actions \(A\), a reward function \(R(s,a,s^{\prime})\), and a transition probability function \(P(s^{\prime}|s,a)\)[19]. MDP supports decision-making by modeling the probabilistic relationship between the state and behavior of the system, and STPA explores the causal relationship between cause and effect to analyze system safety [7]. We conduct a safety analysis of the system by considering it as an MDP problem through information obtained with STPA. In STPA, events and analyzed conditions, including system states, are mapped to states of MDPs that are in specific situations. The action that an agent can take in the system of MDP is mapped to a control action or UCA in STPA. Reward in MDP is related to the agent's performance in each state, and it maps to account for the effectiveness and risk reduction of safety measures in STPA. The transition probability represents the probability of performing a particular action in a specific state and then moving to the next state, and STPA defines the system state change as the transition probability by considering the interaction. Figure 1: Procedure of STPA. Source: [[http://psas.scripts.mit.edu/home/materials/](http://psas.scripts.mit.edu/home/materials/)]([http://psas.scripts.mit.edu/home/materials/](http://psas.scripts.mit.edu/home/materials/)) (accessed on 15 February 2024). The goal of RL is to maximize the expected cumulative reward. \(R\) is the cumulative reward, \(T\) is the time horizon or the number of steps in the episode, \(r_{t}\) is the reward obtained at time \(t\), and \(\gamma\) is a discount factor that determines the importance of future rewards relative to immediate rewards [20]. \[R=\sum_{t=0}^{T}\gamma^{t}r_{t}\]  
RL agents have explicit goals, can sense aspects of their environments, and can choose actions to influence their environments [21]. The agent operates in a dynamic environment, where the state of the environment can change as a result of the agent's actions. This requires the agent to continuously adapt its behavior in response to new information. The Advantage Actor-Critic (A2C) in Figure 2 is an RL technique that combines the benefits of both policy-based and value-based methods [22]. In the A2C algorithm, the MLPPolicy refers to the policy based on Multi-Layer Perception (MLP). This policy function outputs the probabilities for all possible actions given a state, providing a stochastic way of selecting actions based on the given state. The MLPPolicy is part of the Actor (Policy Network), which represents the policy [23]. The actor is responsible for choosing actions and provides the advantage of computing continuous actions without the need for optimization procedures on a value function [24]. The critic (value network) estimates the expected return and provides low-variance knowledge of the performance [25]. The actor is updated using the policy gradient method. It collects episodes using the current policy and then computes gradients of the returns and probabilities of actions selected at each state. \(\theta_{actor}\) and \(\theta_{critic}\) are the parameters of each network. \(\alpha\) is the learning rate. \(J(\theta_{actor})\) is the objective function for the average reward of the policy. \[\theta_{actor}\leftarrow\theta_{actor}+\alpha_{actor}\
abla_{\theta_{actor}}J (\theta_{actor})\]  
The critic is updated using TD (temporal difference) learning. TD learning updates the value of the current state using the value of the next state. It is trained to predict the state-value function, and its parameters are updated to minimize the TD error [26]. \(L(\theta_{critic})\) is the objective function for the TD error.