SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We again divide the full optimization problem in stochastic settings into infeasible and feasible ones similar to Equations 1 and 2. However, we consider the infeasible formulation with likelihood the current state is in a safest policy's infeasible state, or  $\phi^{*}(s)$ . Similarly, we account for the feasible optimization formulation with likelihood the current state is in a safest policy's feasible set,  $1 - \phi^{*}(s)$ . The complete Reachability Estimation for Safe Policy Optimization (RESPO) can be rewritten as:
$$
\max  _ {\pi} \underset {s \sim d _ {0}} {\mathbb {E}} \left[ V ^ {\pi} (s) \cdot \left(1 - \phi^ {*} (s)\right) - V _ {c} ^ {\pi} (s) \cdot \phi^ {*} (s) \right], \text {s . t .}, V _ {c} ^ {\pi} (s) = 0, \text {w . p .} 1 - \phi^ {*} (s), \forall s \in S _ {I}.
$$
In sum, the RESPO framework provides several benefits when compared with other constrained Reinforcement Learning and reachability-based approaches. Notably, 1) it maintains persistent safety when in the feasible set unlike CMDP-based approaches, 2) compared with other reachability-based approaches, RESPO considers performance optimization in addition to maintaining safety, 3) it maintains the behavior of a safest policy in the infeasible set and even reenters the feasible set when possible, 4) RESPO employs rigorously defined reachability definitions even in stochastic settings.
5.3 Overall Algorithm
We describe our algorithms by breaking down the novel components. Our algorithm predicts reachability membership to guide the training toward optimizing the right portion of the optimization equation (i.e., feasibility case or infeasibility case). Furthermore, it exclusively uses the discounted sum of costs as the safety value function – we can avoid having to learn the reachability value function while having the benefit of exploiting the improved signal in the cost value function.
Optimization in infeasible set versus feasible set. If the agent is in the infeasible set, this is the simplest case. We want to find the optimal policy that maximizes  $-V_{c}^{\pi}(s)$ . This would be the only term that needs to be considered in optimization.
On the other hand, if the agent is in the feasible set, we must solve the constraint optimization  $\max_{\pi} V^{\pi}(s)$  subject to  $V_{c}^{\pi}(s) = 0$ . This could be solved via a Lagrangian-based method:
$$
\min_{\pi}\max_{\lambda}L(\pi ,\lambda) = \min_{\pi}\max_{\lambda}\Bigg(\mathbb{E}_{s\sim d_{0}}\left[ - V^{\pi}(s) + \lambda V_{c}^{\pi}(s)\right]\Bigg).
$$
Now what remains is obtaining the reachability estimation function  $\phi^{*}$ . First, we address the problem of acquiring optimal likelihood of being feasible. It is nearly impossible to accurately know before training if a state is in a safest policy's infeasible set. We propose learning a function guaranteed to converge to this REF (with some discount factor for  $\gamma$ -contraction mapping) by using the recursive Bellman formulation proved in Theorem 1.
We learn a function  $p(s)$  to capture the probability  $\phi^{*}(s)$ . It is trained like a reachability function:
$$
p (s) = \max  \left\{\mathbb {1} _ {s \in S _ {v}}, \gamma \cdot p \left(s ^ {\prime}\right) \right\},
$$
where  $S_{v}$  is the violation set,  $s'$  is the next sampled state, and  $\gamma$  is a discount parameter  $0 \ll \gamma < 1$  to ensure convergence of  $p(s)$ . Furthermore, and crucially, we ensure the learning rate of this REF is on a slower time scale than the policy and its critics but faster than the lagrange multiplier.
Bringing the concepts covered above, we present our full optimization equation:
$$
\min  _ {\pi} \max  _ {\lambda} L (\pi , \lambda) = \min  _ {\pi} \max  _ {\lambda} \left(\mathbb {E} _ {s \sim d _ {0}} \left[ \left[ - V ^ {\pi} (s) + \lambda \cdot V _ {c} ^ {\pi} (s) \right] \cdot (1 - p (s)) + V _ {c} ^ {\pi} (s) \cdot p (s) \right]\right). \tag {4}
$$
We show the design of our algorithm RESPO in an actor-critic framework in Algorithm 1. Note that the  $V$  and  $V_{c}$  have corresponding  $Q$  functions:  $V^{\pi}(s) = \mathbb{E}_{a\sim \pi (\cdot |s)}Q(s,a)$  and  $V_{c}^{\pi}(s) = \mathbb{E}_{a\sim \pi (\cdot |s)}Q_{c}(s,a)$ . The gradients' definitions are found in the appendix. We use operator  $\Gamma_{\Theta}$  to indicate the projection of vector  $\theta \in \mathbb{R}^n$  to the closest point in compact and convex set  $\Theta \subseteq \mathbb{R}^n$ . Specifically,  $\Gamma_{\Theta} = \arg \min_{\hat{\theta}\in \Theta}||\hat{\theta} -\theta ||^2$ .  $\Gamma_{\Omega}$  is similarly defined.
Algorithm 1 RESP0 Actor Critic
Require: Randomly initialized policy  $\pi_{\theta}$ 's parameters  $\theta_0$ , reward critic  $Q$ ’s parameters  $\eta_0$ , cost critic  $Q_{c}$ 's parameters  $\kappa_0$ , REF  $p$ ’s parameters  $\xi_0$ , Lagrange multiplier  $\lambda$ ’s parameters  $\omega_0$ , horizon  $T$
Require: Convex projection operators  $\Gamma_{\Theta}$  and  $\Gamma_{\Omega}$ , and reward and cost critic learning rate  $\zeta_1(k)$ , policy learning rate  $\zeta_2(k)$ , REF learning rate  $\zeta_3(k)$ , lagrange multiplier learning rate  $\zeta_4(k)$
1: for  $k = 0, 1, 2, \ldots$  do
2: for  $i = 0,1,2,\ldots$  do
3: Sample trajectories  $\tau_{i}:\{(s_{j},a_{j},s_{j}^{\prime},r_{j},h_{j})\} \sim \pi_{\theta}$
4: Rew. Update  $\eta_{k + 1} = \eta_k - \zeta_1(k)\nabla_\eta Q(s_t,a_t)\cdot [Q(s_t,a_t) - (r(s_t,a_t) + \gamma Q(s_{t + 1},a_{t + 1}))]$
5: Cost Update  $\kappa_{k + 1} = \kappa_k - \zeta_1(k)\nabla_\kappa Q_c(s_t,a_t)\cdot [Q_c(s_t,a_t) - (h(s_t) + \gamma Q_c(s_{t + 1},a_{t + 1}))]$
6: Policy Update  $\theta_{k+1} =$
7:  $\Gamma_{\Theta}\Bigg(\theta_k - \zeta_2(k)\gamma^t\left[-Q(s_t,a_t)[1 - p(s_t)] + Q_c(s_t,a_t)[\lambda (1 - p(s_t)) + p(s_t)]\right]\nabla_\theta \log \pi_\theta (a_t|s_t)\Bigg)$
8: REF Update  $\xi_{k + 1} = \xi_k - \zeta_3(k)\nabla_\xi p(s_t)\cdot [p(s_t) - \max \{\mathbb{1}_{h(s_t) > 0},\gamma p(s_{t + 1})\} ]$
9: Lagrange multiplier Update  $\omega_{k + 1} = \Gamma_{\Omega}\bigl (\omega_{k} - \zeta_{4}(k)Q_{c}(s_{t},a_{t})(1 - p(s_{t}))\nabla_{\omega}\lambda \bigr)$
Figure 1: We compare the performance of our algorithm with other SOTA baselines in Safety Gym (left two figures), Safety PyBullet (middle two figures), and Safety MuJoCo (right two figures).
5.4 Convergence Analysis
We provide convergence analysis of our algorithm for Finite MDPs (finite bounded state and action space sizes, maximum horizon  $T$ , reward bounded by  $R_{\mathrm{max}}$ , and cost bounded by  $H_{\mathrm{max}}$ ) under reasonable assumptions. We demonstrate our algorithm almost surely finds a locally optimal policy for our RESPO formulation, based on the following assumptions:
- A1 (Step size): Step sizes follow schedules  $\{\zeta_1(k)\}$ ,  $\{\zeta_2(k)\}$ ,  $\{\zeta_3(k)\}$ ,  $\{\zeta_4(k)\}$  where:
$\sum_{k}\zeta_{i}(k) = \infty$  and  $\sum_{k}\zeta_{i}(k)^{2} < \infty, \forall i \in \{1, 2, 3, 4\},$  and  $\zeta_{j}(k) = o(\zeta_{j-1}(k)), \forall j \in \{2, 3, 4\}$ .
The reward returns and cost returns critic value functions must follow the fastest schedule  $\zeta_1(k)$ , the policy must follow the second fastest schedule  $\zeta_2(k)$ , the REF must follow the second slowest schedule  $\zeta_3(k)$ , and finally, the lagrange multiplier should follow the slowest schedule  $\zeta_4(k)$ .
A2 (Strict Feasibility):  $\exists \pi (\cdot |\cdot ;\theta)$  such that  $\forall s\in S_I$  where  $\phi^{*}(s) = 0,V_{c}^{\pi_{\theta}}(s)\leq 0$
- A3 (Differentiability and Lipschitz Continuity): For all state-action pairs  $(s, a)$ , we assume value and cost Q functions  $Q(s, a; \eta)$ ,  $Q_{c}(s, a; \kappa)$ , policy  $\pi(a|s; \theta)$ , and REF  $p(s, a; \xi)$  are continuously differentiable in  $\eta$ ,  $\kappa$ ,  $\theta$ ,  $\xi$  respectively. Furthermore,  $\nabla_{\omega} \lambda_{\omega}$  and, for all state-action pairs  $(s, a)$ ,  $\nabla_{\theta} \pi(a|s; \theta)$  are Lipschitz continuous functions in  $\omega$  and  $\theta$  respectively.
The detailed proof of the following result is provided in the appendix.
Theorem 2. Given Assumptions A1-A3, the policy updates in Algorithm 1 will almost surely converge to a locally optimal policy for our proposed optimization in Equation RESPO.
Baselines. The baselines we compare are CMDP-based or solve for hard constraints. The CMDP baselines are Lagrangian-based Proximal Policy Optimization (PPOLag) based on [7], Constraint-Rectified Policy Optimization (CRPO) [35], Penalized Proximal Policy Optimization (P3O) [32], and Projection-Based Constrained Policy Optimization (PCPO) [3]. The hard constraints baselines are RCRL [27], CBF with constraint  $\dot{h}(s) + \nu \cdot h(s) \leq 0$ , and Feasible Actor-Critic (FAC) [9]. We classify FAC among the hard constraint approaches because we make its cost threshold  $\chi = 0$  in order to better compare using NN lagrange multiplier with our REF approach in RESPO. We include the unconstrained Vanilla PPO [33] baseline for reference.
Benchmarks. We compare RESPO with the baselines in a diverse suite of safety environments. We consider high-dimensional environments in Safety Gym [30] (namely PointButton and CarGoal), Safety PyBullet [50] (namely DroneCircle and BallRun), and Safety MuJoCo [51], (namely Safety HalfCheetah and Reacher). We also show our algorithm in a multi-drone environment with multiple hard and soft constraints. More detailed experiment explanations and evaluations are in the appendix.
6.1 Main Experiments in Safety Gym, Safety PyBullet, and MuJoCo
We compare our algorithm with SOTA benchmarks on various high-dimensional (up to 76D observation space), complex environments in the stochastic setting, i.e., where the environment and/or policy are stochastic. Particularly, we examine environments in Safety Gym, Safety PyBullet, and Safety MuJoCo. The environments provide reward for achieving a goal behavior or location, while the cost is based on tangible (e.g., avoiding quickly moving objects) and non-tangible (e.g., satisfying speed limit) constraints. Environments like PointButton require intricate behavior where specific buttons must be reached while avoiding multiple moving obstacles, stationary hazards, and wrong buttons.
Overall, RESPO achieves the best balance between optimizing reward and minimizing cost violations across all the environments. Specifically, our approach generally has the highest reward performance (see the red lines from the top row of Figure 2) among the safety-constrained algorithms while maintaining reasonably low to 0 cost violations (like in HalfCheetah). When RESPO performs the second highest, the highest-performing safety algorithm always incurs several times more violations than RESPO – for instance, RCRL in PointButton or PPOLag in Drone Circle. Non-primal-dual CMDP approaches, namely CRPO, P3O, and PCPO generally satisfy their cost threshold constraints, but their reward performances rarely exceed that of PPOLag. RCRL generally has extremes of high reward and high cost, like in BallRun, or low reward and low cost, like in CarGoal. FAC and CBF generally have conservative behavior that sacrifices reward performance to minimize cost.
Figure 2: Comparison of RESPO with baselines in Safety Gym and PyBullet environments. The plots in the first row show performance measured in rewards (higher is better); those in second row show cost (lower is better). RESPO (red curves) achieves the best balance of maximizing reward and minimizing cost. When other methods achieve higher rewards than RESPO, they achieve much higher costs as well. E.g., in PointButton, RCRL has slightly higher rewards, but accumulates over  $3 \times$  violations than RESPO. Note, Vanilla PPO is unconstrained.
6.2 Hard and Soft Constraints
We also demonstrate RESPO's performance in an environment with multiple hard and soft constraints. The environment requires controlling two drones to pass through a tunnel one at a time while respecting certain distance requirements. The reward is given for quickly reaching the goal positions. The two hard constraints involve (H1) ensuring neither drone collides into the wall and (H2) the distance between the two drones is more than 0.5 to ensure they do not collide. The soft constraint is that the two drones are within 0.8 of each other to ensure real-world communication. It is preferable to prioritize hard constraint H1 over hard constraint H2, since colliding with the wall may have more serious consequences to the drones rather than violations of an overly precautious distance constraint.
Our approach, in the leftmost of Figure 3, successfully reaches the goal while avoiding the wall obstacles in all time steps. We are able to prioritize this wall avoidance constraint over the second hard constraint. This can be seen particularly in between the blue to cyan time period where the higher
Figure 3: Comparison of RESPO with baselines trajectories in Hard & Soft Constraints multi-Drone control. Starting at gold circles, drones must enter the tunnel one at a time and reach green stars. Hard constraints are wall avoidance and ensuring drones are farther than 0.5 meters from each other. Soft constraint is drones are within 0.8 meters of each other. Trajectory colors correspond to time. RESPO (on left) successfully controls drones to reach goals while always avoiding walls and usually respecting distance constraints. Other baselines cannot manage multiple constraints: they collide with the wall and have many distance constraint violations.
Figure 4: Comparison of RESPO with baselines in MuJoCo. Higher rewards (first row plots) and lower costs (second row plots) are better. In HalfCheetah, RESPO has highest reward among safety baselines, with 0 violations. In Reacher, RESPO has good rewards, low costs.
Figure 5: Ablation study on the learning rate of REF. Higher rewards (first row plots) are better; lower costs (second row plots) are better. When changing REF's learning rate to violate timescale assumptions, REF produces suboptimal feasible sets.
Drone makes way for the lower Drone to pass through but needs to make a drop to make a concave parabolic trajectory to the goal. Nonetheless, the hard constraints are almost always satisfied, thereby producing the behavior of allowing one drone through the tunnel at a time. The soft constraints are satisfied at the beginning and end but are violated, reasonably, in the middle of the episode since only one drone can pass through the tunnel at a time, thereby forcing the other drone into a standby mode.
6.3 Ablation Studies
We also perform ablation studies to experimentally confirm the design choices we made based on the theoretically established convergence and optimization framework. We particularly investigate the effects of changing the learning rate of our reachability function as well as changing the optimization framework. We present the results of changing the learning rate for REF in Figure 5 while our results for the ablation studies on our optimization framework can be seen in Figure 6.
In Figure 5, we show the effects of making the learning rate of REF slower and faster than the one we use in accordance with Assumption 1. From these experiments, changing the learning rate in either direction produces poor reward performance. A fast learning rate makes the REF converge to the likelihood of infeasibility for the current policy, which can be suboptimal. But a very slow learning rate means the function takes too long to converge – the lagrange multiplier may meanwhile become very large, thus making it too difficult to optimize for reward returns. In both scenarios, the algorithm with modified learning rates produces conservative behavior that sacrifices reward performance.
In Figure 6, we compare RESPO with RCRL implemented with our REF and PPOLag in the CMDP framework with cost threshold  $\chi = 0$  to ensure hard constraint satisfaction. The difference between RESPO and the RCRL-based ablation approach is that the ablation still uses  $V_{h}^{\pi}$  instead of  $V_{c}^{\pi}$ . The ablation approach's high cumulative cost can be attributed to the limitations of using  $V_{h}^{\pi}$  - particularly, the lower sensitivity of  $V_{h}^{\pi}$  to safety improvement and its lack of guarantees on feasible set (re)entrance. PPOLag with  $\chi = 0$  produces low violations but also very low reward performance that's close to zero. Naively using  $V_{c}^{\pi}$  in a hard constraints framework leads to very conservative
Figure 6: Ablation study on optimization framework. Top row plots show performance measured in reward (higher is better). Bottom row plots show cost (lower is better). RESPO (red curve) achieves best balance of maximizing reward and minimizing cost. RCRL framework implemented with our REF without  $V_{c}^{\pi}$  incurs very high costs. PPOLag in CMDP framework with  $\chi = 0$  has very low reward performance. So, learning REF and learning  $V_{c}^{\pi}$  are both crucial components in our design and work in tandem to contribute to RESPO's efficacy.
behavior that sacrifices reward performance. Ultimately, this ablation study experimentally highlights the importance of learning our REF and using value function  $V_{c}^{\pi}$  in our algorithm's design.
In summary, we proposed a new optimization formulation and a class of algorithms for safety-constrained reinforcement learning. Our framework optimizes reward performance for states in least-violation policy's feasible state space while maintaining persistent safety as well as providing the safest behavior in other states by ensuring entrance into the feasible set with minimal cumulative discounted costs. Using our proposed reachability estimation function, we prove our algorithm's class of actor-critic methods converge a locally optimal policy for our proposed optimization. We provide extensive experimental results on a diverse suite of environments in Safety Gym, PyBullet, and MuJoCo, and an environment with multiple hard and soft constraints, to demonstrate the effectiveness of our algorithm when compared with several SOTA baselines. We leave open various extensions to our work to enable real-world deployment of our algorithm. These include constraining violations during training, guaranteeing safety in single-l lifetime reinforcement learning, and ensuring policies don't forget feasible sets as environment tasks change. Our approach of learning the optimal REF to reduce the state space into a low-dimensional likelihood representation to guide training for high-dimensional policies can have applications in other learning problems in answering binary classification or likelihood-based questions about dynamics in high-dimension feature spaces.
This material is based on work supported by NSF Career CCF 2047034, NSF CCF DASS 2217723, ONR YIP N00014-22-1-2292, and Amazon Research Award.
[56] Charles Dawson, Sicun Gao, and Chuchu Fan. Safe control with learned certificates: A survey of neural lyapunov, barrier, and contraction methods for robotics and control. IEEE Transactions on Robotics, 2023.
[57] Jason J Choi, Donggun Lee, Koushil Sreenath, Claire J Tomlin, and Sylvia L Herbert. Robust control barrier-value functions for safety-critical control. In 2021 60th IEEE Conference on Decision and Control (CDC), pages 6814-6821. IEEE, 2021.
[58] Eric Yang Yu, Zhizhen Qin, Min Kyung Lee, and Sicun Gao. Policy optimization with advantage regularization for long-term fairness in decision systems, 2022.
[59] Ppo lagrangian pytorch. https://github.com/akjayant/PPO_Lagrangian_PyTorch, 2022.
[60] Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. arXiv preprint arXiv:2305.09304, 2023.
r	reward function
h	safety loss function
Vπ	cumulative discounted reward value function
Vcπ	cumulative discounted cost value function
Vhπ	reachability value function
1s∈Sv	instantaneous violation indicator function
φπ	reachability estimation function (REF) of a given policy
φ*	reachability estimation function (REF) of safest policy
p	predicted reachability estimation function (REF) of safest policy
Hmax	upper bound of function h
Hmin	minimum non-zero value of of function h
λmax	maximum value to clip lagrange multiplier
Rmax	upper bound of function r
Ss	safe set
Sv	unsafe set
Sf	feasible set (persistent safe set)
Es′~π,P	expectation taken over possible next states
Eτ~π,P	expectation taken over possible trajectories
Es~d0	expectation taken over initial distribution
Table 1: Notation used in the paper.
B Gradient estimates
The Q value losses based on the MSE between the Q networks and the respective sampled returns result in the gradients:
$$
\hat {\nabla} _ {\eta} J _ {Q} (\eta) = \nabla_ {\eta} Q \left(s _ {t}, a _ {t}; \eta\right) \cdot \left[ Q _ {\eta} \left(s _ {t}, a _ {t}\right) - \left(r \left(s _ {t}, a _ {t}\right) + \gamma Q \left(s _ {t + 1}, a _ {t + 1}; \eta\right)\right) \right]
$$
$$
\hat {\nabla} _ {\kappa} J _ {Q _ {c}} (\kappa) = \nabla_ {\kappa} Q _ {c} (s _ {t}, a _ {t}; \kappa) \cdot [ Q _ {\kappa} (s _ {t}, a _ {t}) - (h (s _ {t}) + \gamma Q _ {c} (s _ {t + 1}, a _ {t + 1}; \kappa)) ]
$$
Similarly the REF gradient update is:
$$
\hat {\nabla} _ {\eta} J _ {p} (\xi) = \nabla_ {\xi} p (s _ {t}; \xi) \cdot [ p (s _ {t}; \xi) - \max  \{\mathbb {1} _ {s _ {t} \in S _ {v}}, \gamma p (s _ {t + 1}; \xi) \} ]
$$
From the policy gradient theorem in [52], we get the policy gradient loss as:
$$
\begin{array}{l} \hat {\nabla} _ {\theta} J _ {\pi} (\theta) = \gamma^ {t} \left[ - Q _ {\eta} \left(s _ {t}, a _ {t}\right) \left[ 1 - p _ {\xi} \left(s _ {t}\right) \right] \right. \\ \left. + Q _ {c} \left(s _ {t}, a _ {t}\right) \left[ \lambda_ {\omega} \left(1 - p _ {\xi} \left(s _ {t}\right)\right) + p _ {\xi} \left(s _ {t}\right) \right] \right] \nabla_ {\theta} \log \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) \\ \end{array}
$$
and the stochastic gradient of the multiplier is
$$
\hat {\nabla} _ {\omega} J _ {\lambda} (\omega) = Q _ {c} (s _ {t}, a _ {t}; \kappa) (1 - p _ {\xi} (s _ {t})) \nabla_ {\omega} \lambda_ {\omega}
$$
and  $\lambda_{\omega}$  is clipped to be in range  $[0, \lambda_{\max}]$  (in particular, projection operator  $\Gamma_{\Omega}(\lambda_{\omega}) = \arg \min_{\hat{\lambda}_{\omega} \in [0, \lambda_{\max}]} ||\lambda_{\omega} - \hat{\lambda}_{\omega}||^2$ ).
C.1 Theorem 1 with Proof
Theorem 3. The REF can be reduced to the following recursive Bellman formulation:
where  $s' \sim \pi$ ,  $P(s)$  is a sample of the immediate successive state (i.e.,  $s' \sim P(\cdot | s, a \sim \pi(\cdot | s))$ ) and the expectation is taken over all possible successive states.
$$
\begin{array}{l} \phi^ {\pi} (s) := \underset {\tau \sim \pi , P (s)} {\mathbb {E}} \max  _ {s _ {t} \in \tau} \mathbb {1} _ {s _ {t} ^ {\pi} \in S _ {v}} \\ = \underset {\tau \sim \pi , P (s)} {\mathbb {E}} \max  \left\{\mathbb {1} _ {s \in S _ {v}}, \max  _ {s _ {t} \in \tau \backslash \{s \}} \mathbb {1} _ {s _ {t} ^ {\pi} \in S _ {v}} \right\} \\ = \max  \left\{\mathbb {1} _ {s \in S _ {v}}, \underset {\tau \sim \pi , P (s)} {\mathbb {E}} \max  _ {s _ {t} \in \tau \backslash \{s \}} \mathbb {1} _ {s _ {t} ^ {\pi} \in S _ {v}} \right\} \\ = \max  \left\{\mathbb {1} _ {s \in S _ {v}}, \underset {s ^ {\prime} \sim \pi , P (s)} {\mathbb {E}} \underset {\tau^ {\prime} \sim \pi , P \left(s ^ {\prime}\right)} {\mathbb {E}} \max  _ {s _ {t} \in \tau^ {\prime}} \mathbb {1} _ {s _ {t} ^ {\pi} \in S _ {v}} \right\} \\ = \max  \bigl \{\mathbb {1} _ {s \in S _ {v}}, \underset {s ^ {\prime} \sim \pi , P (s)} {\mathbb {E}} \phi^ {\pi} \bigl (s ^ {\prime} \bigr) \bigr \} \\ \end{array}
$$
Note that we use the notation  $\tau \sim \pi, P(s)$  to indicate a trajectory sampled from the MDP with transition probability  $P$  under policy  $\pi$  starting from state  $s$ , and use the notation  $s' \sim \pi, P(s)$  to indicate the next immediate state from the MDP with transition probability  $P$  under policy  $\pi$  starting from state  $s$ . The third line holds because the indicator function is either 0 or 1, so if it's 1 then  $\phi^{\pi}(s) = \mathbb{E}_{\tau \sim \pi, P(s)} 1 = 1$  else  $\phi^{\pi}(s) = \mathbb{E}_{\tau \sim \pi, P(s)} \max_{s_t \in \tau \setminus \{s\}} \mathbb{1}_{s_t^{\pi} \in S_v}$ .
C.2 Proposition 1 with Proof