SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition 3. The cost value function  $V_{c}^{\pi}(s)$  is zero for state  $s$  if and only if the persistent safety is guaranteed for that state under the policy  $\pi$ .
Proof. (IF) Assume for a given policy  $\pi$ , the persistent safety is guaranteed, i.e.  $h(s_{t}|s_{0} = 0,\pi) = 0$  holds for all  $s_t\in \tau$  for all possible trajectories  $\tau$  sampled from the environment with control policy  $\pi$ . We then have:
$$
V _ {c} ^ {\pi} (s) := \underset {\tau \sim \pi , P (s)} {\mathbb {E}} [ \sum_ {s _ {t} \in \tau} \gamma^ {t} h (s _ {t}) ] = 0.
$$
(ONLY IF) Assume for a given policy  $\pi$ ,  $V_{c}^{\pi}(s) = 0$ . Since the image of the safety loss function  $h(s)$  is non-negative real, and  $V_{c}^{\pi}(s)$  is the expectation of the sum of non-negative real values, the only way  $V_{c}^{\pi}(s) = 0$  is if  $h(s_{t}|s_{0} = 0,\pi) = 0$ ,  $\forall s_t\in \tau$  for all possible trajectories  $\tau$  sampled from the environment with control policy  $\pi$ .
C.3 Proposition 2 with Proof
Proposition 4. If  $\exists \pi$  that produces trajectory  $\tau = \{(s_i), i \in \mathbb{N}, s_1 = s\}$  in deterministic MDP  $\mathcal{M}$  starting from state  $s$ , and  $\exists m \in \mathbb{N}, m < \infty$  such that  $s_m \in S_f^\pi$ , then  $\exists \epsilon > 0$  where if discount factor  $\gamma \in (1 - \epsilon, 1)$ , then the optimal policy  $\pi^*$  of Main paper Equation 3 will produce a trajectory  $\tau' = \{(s_j'), j \in \mathbb{N}, s_1' = s\}$ , such that  $\exists n \in \mathbb{N}, n < \infty$ ,  $s_n' \in S_f^{\pi^*}$  and  $V_c^{\pi^*}(s) = \min_{\pi'} V_c^{\pi'}(s)$ .
In other words the proposition is stating for some state  $s$ , if there is a policy that enters its feasible set in a finite number  $(m - 1)$  of steps, then by ensuring discount factor  $\gamma$  is close to 1 we can guarantee that the optimal policy  $\pi^*$  of Main paper Equation 3 will also enter the feasible set in a finite number of steps with the minimum cumulative discounted sum of the costs. Note that  $\pi^*$  will always produce trajectories with the minimum discounted sum of costs whether the state is in the feasible or infeasible set of the policy by virtue of its optimization which constrains  $V_c^\pi$ .
Proof. We consider two cases: (Case 1)  $m = 1$  and (Case 2)  $m > 1$ .
Case 1  $m = 1$ : In this case, there exists a policy  $\pi$  in which the current state  $s$  is in the feasible set of that policy. By definition, that means that in a trajectory  $\tau$  sampled in the MDP using that
policy, starting from state  $s$ , there are no future violations incurred in  $\tau$ . Thus  $V_{c}^{\pi}(s) = 0$ . Since  $\pi^{*}$  incurs the minimum cumulative violation,  $V_{c}^{\pi^{*}}(s) = 0$  trivially. Therefore,  $s$ , the first state of the trajectory, is in the feasible set of  $\pi^{*}$ .
Case 2  $m > 1$ : Since policy  $\pi^*$  produces the minimum cumulative discounted cost for a given state  $s$ , the core of this proof will be demonstrating that the minimum cumulative discounted cost of entering the feasible set (call this value  $H_E$ ) is less than the minimum cumulative discounted cost of not entering the feasible set (call this value  $H_N$ ), and therefore  $\pi^*$  will choose the route of entering the feasible set.
The proof will proceed by deriving a sufficient condition for  $H_{E} < H_{N}$  by establishing bounds on them.
We place an upper bound on the minimum cumulative discounted cost of entering the feasible set  $H_{E}$ . Since  $\exists \pi$  that enters the feasible set in  $m - 1$  steps, entering the feasible set can be at most the highest possible cost that  $\pi$  incurs. Since the maximum cost at any state is  $H_{\mathrm{max}}$ , the upper bound is the discounted sum of  $m - 1$  steps of violations  $H_{\mathrm{max}}$ , or
$$
H _ {E} <   \frac {H _ {\mathrm {m a x}} (1 - \gamma^ {m - 1})}{(1 - \gamma)}
$$
We place a lower bound on the minimum cumulative discounted cost of not entering the feasible set  $H_{N}$ . In this case, say in the sampled trajectory, the maximum gap between any two non-zero violations is  $w$ . By definition, the trajectory cannot have an infinite sequence of violation-free states since the trajectory never enters the feasible set. Therefore  $w$  is finite. Now recall  $H_{\mathrm{min}}$  is the lower bound on the non-zero values of  $h$ . So the minimum cumulative discounted cost of not entering the feasible set must be at least the cost of the trajectory with a violation of  $H_{\mathrm{min}}$  at intervals of  $w$  steps. That is:
$$
\frac {H _ {\operatorname* {m i n}} (\gamma^ {w})}{(1 - \gamma^ {w})} <   H _ {N}
$$
Now  $H_{E} < H_{N}$  will be true if the upper bound of  $H_{E}$  is less than the lower bound of  $H_{N}$ . In other words  $H_{E} < H_{N}$  is true if:
$$
\frac {H _ {\max } \left(1 - \gamma^ {m - 1}\right)}{(1 - \gamma)} <   \frac {H _ {\min } \left(\gamma^ {w}\right)}{\left(1 - \gamma^ {w}\right)} \tag {5}
$$
Rearranging, we get:
$$
\frac {H _ {\operatorname* {m a x}}}{H _ {\operatorname* {m i n}}} <   \frac {(1 - \gamma) \cdot (\gamma^ {w})}{(1 - \gamma^ {m - 1}) \cdot (1 - \gamma^ {w})} \tag {6}
$$
Let's define the RHS of the Inequality 6 as the function  $v(\gamma)$ . Consider  $\gamma \in (0,1)$ . It is not difficult to demonstrate that  $v(\gamma)$  in this domain range is a continuous function and that left directional limit  $\lim_{\gamma \to 1^{-}} v(\gamma) = \infty$ . This suggests that there is an open interval of values for  $\gamma$  (whose supremum is 1) for which  $H_{\max} / H_{\min} < v(\gamma)$  and so  $H_E < H_N$ . So we establish that  $\exists \epsilon > 0$  such that for  $\gamma \in (1 - \epsilon, 1)$ , we satisfy the sufficient condition  $H_E < H_N$  so that the optimal policy will enter its feasible set.
Thus, we prove that if there is a policy entering its feasible set from state  $s$ , then there is a range of values for  $\gamma$  that are close enough to 1 ensuring that the optimal policy of Main paper Equation 3 will enter its feasible set in a finite number of steps with minimum discounted sum of costs.
C.4 Theorem 2 with Proof
Theorem 4. Given Assumptions A1-A3 in Main paper, the policy updates in Algorithm 1 will almost surely converge to a locally optimal policy for our proposed optimization in Equation RESPO.
We first provide an intuitive explanation behind why our REF learns to converge to the safest policy's REF, then a proof overview, and then the full proof.
Figure 7: The predicted feasible set converges to a safest policy's feasible set since the misclassified regions  $\mathcal{X}$  and  $\mathcal{Y}$  are corrected over time.
C.4.1 Intuition behind REF convergence
The approach can be explained by considering what happens in the individual regions of space. Consider a deterministic environment for simplicity. As seen in Figure 7, there are two subsets of the initial state space: a safest policy's "true" feasible set  $S_{hI}$  and REF predicted feasible set  $S_{pI}$ , and they create 4 regions in the initial state space  $S_I$ :  $\mathcal{W} = \overline{S_{hI}} \cap \overline{S_{pI}}$ ,  $\mathcal{X} = \overline{S_{hI}} \cap S_{pI}$ ,  $\mathcal{Y} = S_{hI} \cap \overline{S_{pI}}$ ,  $\mathcal{Z} = S_{hI} \cap S_{pI}$ . Consider a point during training when the lagrange multiplier  $\lambda$  is sufficiently large. For states in  $\mathcal{W}$ , the set of correctly classified infeasible states, the algorithm will simply minimize cumulative violations  $V_c^{\pi_\theta}(s)$ , and thereby remain as safe as possible since the policy and critics learning rates are faster than that of REF.  $\mathcal{X}$ , which is the set of infeasible states that are misclassified, is very small if we ensure the policy and REF are trained at much faster time scales than the multiplier and so when the agent starts in true infeasible states, it will by definition reach violations and therefore be labeled as infeasible. In  $\mathcal{Y}$ , the set of truly feasible states that are misclassified, the algorithm also minimizes cumulative violations, which by the definition of feasibility should be 0. It will then have no violations and enter the correctly predicted feasible set  $\mathcal{Z}$ . And when starting in states in  $\mathcal{Z}$ , the algorithm will optimize the lagrangian, and since the multiplier  $\lambda$  is sufficiently large, it will converge to a policy that optimizes for reward while ensuring safety, i.e. no future violations, and therefore the state will stay predictably feasible in  $\mathcal{Z}$ . In this manner, REF's predicted feasible set will converge to the optimal feasible set, and the agent will be safe and have optimal performance in the feasible set and be the safest behavior outside the feasible set. Thereby, the algorithm finds a locally optimal solution to the proposed optimization formulation.
C.4.2 Proof Overview
We show our algorithm convergence to the optimal policy by utilizing the proof framework of multi-time scale presented in [8, 10, 53, 27]. Specifically, we have 4 time scales for (1) the critics, (2) policy, (3) REF function, and (4) lagrange multiplier, listed in order from fastest to slowest. The overview of each timescale proof step is as follows:
1 We demonstrate the almost sure convergence of the critics to the corresponding fixed point optimal critic functions of the policy.
2 Using multi-timescale theory, we demonstrate the policy almost surely converges to a stationary point of a continuous time system, which we show has a Lyapunov function certifying its locally asymptotic stability at the stationary point.
3 We demonstrate the almost sure convergence of the REF function to the REF of the policy that is safe insofar as the lagrange multiplier is sufficiently large.
4 We demonstrate the almost sure convergence of the lagrange multiplier to a stationary point similar to the proof in the policy timecale.
Finally, we demonstrate that the stationary points for the policy and lagrange multiplier form a saddle point, and so by local saddle point theorem we almost surely achieve the locally optimal policy of our proposed optimization.
C.4.3 Proof Details
Proof. Step 1 (convergence of the critics  $V_{\eta}$  and  $V_{\kappa}$  updates): From the multi-time scale assumption, we know that  $\eta$  and  $\kappa$  will converge on a faster time scale than the other parameters  $\theta$ ,  $\xi$ , and  $\omega$ . Therefore, we can leverage Lemma 1 of Chapter 6 of [10] to analyze the convergence properties while updating  $\eta_{k}$  and  $\kappa_{k}$  by treating  $\theta$ ,  $\xi$ , and  $\omega$  as fixed parameters  $\theta_{k}$ ,  $\xi_{k}$ , and  $\omega_{k}$ . In other words, the policy, REF, and lagrange multiplier are fixed while computing  $Q^{\pi_{\theta_k}}(s,a)$  and  $Q_c^{\pi_{\theta_k}}(s,a)$ . With the Finite MDP assumption and policy evaluation convergence results of [52], and assuming sufficiently expressive function approximator (i.e. wide enough neural networks) to ensure convergence to global minimum, we can use the fact that the bellman operators  $\mathcal{B}$  and  $\mathcal{B}_c$  which are defined as
$$
\mathcal{B}[Q](s,a) = r(s,a) + \gamma_{s^{\prime},a^{\prime}\sim \pi ,P(s)}\mathbb{E}[Q(s^{\prime},a^{\prime})]
$$
$$
\mathcal{B}[Q_{c}](s,a) = h(s) + \gamma_{s^{\prime},a^{\prime}\sim \pi ,P(s)}\mathbb{E}[Q_{c}(s^{\prime},a^{\prime})]
$$
are  $\gamma$ -contraction mappings, and therefore as  $k$  approaches  $\infty$ , we can be sure that  $Q(s,a;\eta_k)\to Q(s,a;\eta^*) = Q^{\pi_{\theta_k}}(s,a)$  and  $Q_{c}(s,a;\kappa_{k})\rightarrow Q_{c}(s,a;\kappa^{*}) = Q_{c}^{\pi_{\theta_{k}}} (s,a)$ . So since  $\eta_{k}$  and  $\kappa_{k}$  converge to  $\eta^{*}$  and  $\kappa^{*}$ , we prove convergence of the critics in Time scale 1.
Step 2 (convergence of the policy  $\pi_{\theta}$  update): Because  $\xi$  and  $\omega$  updated on slower time scales than  $\theta$ , we can again use Lemma 1 of Chapter 6 of [10] and treat these parameters are fixed at  $\xi_{k}$  and  $\omega_{k}$  respectively when updating  $\theta_{k}$ . Additionally in Time scale 2, we have  $||Q(s,a;\eta_k) - Q(s,a;\eta^*)||\to 0$  and  $||Q_c(s,a;\kappa_k) - Q_c(s,a;\kappa^*)||\to 0$  almost surely. Now the update of the policy  $\theta$  using the gradient from Equation 4 is:
$$
\begin{array}{l} \theta_ {k + 1} = \Gamma_ {\Theta} \left[ \theta_ {k} - \zeta_ {2} (k) \left(\nabla_ {\theta} L \left(\theta , \xi_ {k}, \omega_ {k}\right) \mid_ {\theta = \theta_ {k}}\right) \right] \\ = \Gamma_ {\Theta} \left[ \theta_ {k} - \zeta_ {2} (k) \left[ \gamma^ {t} \left[ - Q _ {\eta} \left(s _ {t}, a _ {t}\right) \right] \left[ 1 - p _ {\xi_ {k}} \left(s _ {t}\right) \right] \right. \right. \\ \left. + Q _ {c} \left(s _ {t}, a _ {t}\right) \left[ \lambda_ {\omega} \left(1 - p _ {\xi_ {k}} \left(s _ {t}\right)\right) + p _ {\xi_ {k}} \left(s _ {t}\right) \right] \right] \nabla_ {\theta} \log \pi \left(a _ {t} \mid s _ {t}; \theta\right) \mid_ {\theta = \theta_ {k}} ] ] \\ = \Gamma_ {\Theta} \left[ \theta_ {k} - \zeta_ {2} (k) \left(\nabla_ {\theta} L (\theta , \xi_ {k}, \omega_ {k}) \right| _ {\theta = \theta_ {k}, \eta = \eta^ {*}, \kappa = \kappa^ {*}} + \delta \theta_ {k + 1} + \delta \theta_ {\epsilon}) \right] \\ \end{array}
$$
$$
\begin{array}{l} \delta \theta_ {k + 1} = \sum_ {s _ {i}, a _ {i}} \left[ d _ {0} (s _ {0}) P ^ {\pi_ {\theta_ {k}}} (s _ {i}, a _ {i} | s _ {0}) \gamma^ {i} \left[ - Q _ {\eta} (s _ {i}, a _ {i}) \left[ 1 - p _ {\xi_ {k}} (s _ {i}) \right. \right. \right. \\ \left. + Q _ {c} \left(s _ {i}, a _ {i}\right)\left[ \lambda_ {\omega} \left(1 - p _ {\xi_ {k}} \left(s _ {i}\right)\right) + p _ {\xi_ {k}} \left(s _ {i}\right)\right]\right] \nabla_ {\theta} \log \pi \left(a _ {i} \mid s _ {i}; \theta\right) | _ {\theta = \theta_ {k}} \left. \right] \\ - \gamma^ {t} \left[ - Q _ {\eta} \left(s _ {t}, a _ {t}\right) \left[ 1 - p _ {\xi_ {k}} \left(s _ {t}\right) \right] + Q _ {c} \left(s _ {t}, a _ {t}\right) \left[ \lambda_ {\omega} \left(1 - p _ {\xi_ {k}} \left(s _ {t}\right)\right) + p _ {\xi_ {k}} \left(s _ {t}\right) \right] \right. \\ \cdot \nabla_ {\theta} \log \pi (a _ {t} | s _ {t}; \theta) | _ {\theta = \theta_ {k}} \\ \end{array}
$$
$$
\begin{array}{l} \delta \theta_ {\epsilon} = \sum_ {s _ {i}, a _ {i}} d _ {0} (s _ {0}) P ^ {\pi_ {\theta_ {k}}} (s _ {i}, a _ {i} | s _ {0}) \Bigg [ \\ - \gamma^ {i} \left[ - Q \left(s _ {i}, a _ {i}; \eta_ {k}\right) \left[ 1 - p _ {\xi_ {k}} \left(s _ {i}\right) \right] + Q _ {c} \left(s _ {i}, a _ {i}; \kappa_ {k}\right) \left[ \lambda_ {\omega} \left(1 - p _ {\xi_ {k}} \left(s _ {i}\right)\right) + p _ {\xi_ {k}} \left(s _ {i}\right) \right] \right] \\ \cdot \nabla_ {\theta} \log \pi (a _ {i} | s _ {i}; \theta) | _ {\theta = \theta_ {k}} \\ + \gamma^ {i} \left[ - Q ^ {\pi_ {\theta_ {k}}} \left(s _ {i}, a _ {i}\right) \left[ 1 - p _ {\xi_ {k}} \left(s _ {i}\right) \right] + Q _ {c} ^ {\pi_ {\theta_ {k}}} \left(s _ {i}, a _ {i}\right) \left[ \lambda_ {\omega} \left(1 - p _ {\xi_ {k}} \left(s _ {i}\right)\right) + p _ {\xi_ {k}} \left(s _ {i}\right) \right] \right] \\ \left. \cdot \nabla_ {\theta} \log \pi \left(a _ {i} \mid s _ {i}; \theta\right) \mid_ {\theta = \theta_ {k}} \right] \\ \end{array}
$$
Lemma 1: We can first demonstrate that  $\delta \theta_{k + 1}$  is square integrable. In particular,
$$
\begin{array}{l} \mathbb {E} \big [ | | \delta \theta_ {k + 1} | | ^ {2} | \mathcal {F} _ {\theta , k} \big ] \\ \leq 2 | | \nabla_ {\theta} \log \pi (a | s; \theta) | _ {\theta = \theta_ {k}} \mathbb {1} _ {\pi (a | s; \theta_ {k}) > 0} | | _ {\infty} ^ {2} \cdot \left(| | Q (s, a; \eta_ {k}) | | _ {\infty} ^ {2} \cdot | | 1 - p _ {\xi_ {k}} (s) | | _ {\infty} ^ {2} \right. \\ \left. + | | Q _ {c} (s, a; \kappa_ {k}) | | _ {\infty} ^ {2} \cdot \left[ | | \lambda_ {\omega} | | _ {\infty} ^ {2} \cdot | | 1 - p _ {\xi_ {k}} (s) | | _ {\infty} ^ {2} + | | p _ {\xi_ {k}} (s) | | _ {\infty} ^ {2} \right]\right) \\ \leq 2 \frac {\left| \left| \nabla_ {\theta} \pi (a | s ; \theta) \right| _ {\theta = \theta_ {k}} \right| _ {\infty} ^ {2}}{\min  \left\{\pi (a | s ; \theta_ {k}) \mid \pi (a | s ; \theta_ {k}) > 0 \right\}} \cdot \left(\left| \left| Q (s, a; \eta_ {k}) \right| \right| _ {\infty} ^ {2} \cdot \left| \left| 1 - p _ {\xi_ {k}} (s) \right| \right| _ {\infty} ^ {2} \right. \\ \left. + \left| \left| Q _ {c} (s, a; \kappa_ {k}) \right| \right| _ {\infty} ^ {2} \cdot \left[ \left| \left| \lambda_ {\omega} \right| \right| _ {\infty} ^ {2} \cdot \left| \left| 1 - p _ {\xi_ {k}} (s) \right| \right| _ {\infty} ^ {2} + \left| \left| p _ {\xi_ {k}} (s) \right| \right| _ {\infty} ^ {2} \right]\right) \\ \end{array}
$$
Note that  $\mathcal{F}_{\theta,k} = \sigma(\theta_m, \delta \theta_m, m \leq k)$  is the filtration for  $\theta_k$  generated by different independent trajectories [8]. Also note that the indicator function is used because the expectation of  $||\delta \theta_{k+1}||^2$  is taken with respect to  $P^{\pi_{\theta_k}}$  and  $P^{\pi_{\theta_k}}(s, a|s_0) = 0$  if  $\pi(a|s; \theta_k) = 0$ . From the Assumptions on Lipschitz continuity and Finite MDPs reward and costs, we can bound the values of the functions and the gradients of functions. Specifically
$$
\begin{array}{l} \left| \left| \right| \nabla_ {\theta} \pi (a | s; \theta) \right| _ {\theta = \theta_ {k}} \left| \right| _ {\infty} ^ {2} \leq K _ {1} (1 + \left| \right| \theta_ {k} \left| \right| _ {\infty} ^ {2}), \\ \left| \left| Q (s, a; \eta_ {k}) \right| \right| _ {\infty} ^ {2} \leq \frac {R _ {\max}}{1 - \gamma}, \\ \left| \left| Q _ {h} (s, a; \kappa_ {k}) \right| \right| _ {\infty} ^ {2} \leq \frac {H _ {\max}}{1 - \gamma}, \\ \left\| \lambda_ {\omega} \right\| _ {\infty} ^ {2} \leq \lambda_ {\max }, \\ \left| \left| 1 - p _ {\xi_ {k}} (s) \right| \right| _ {\infty} ^ {2} \leq 1, \\ | | p _ {\xi_ {k}} (s) | | _ {\infty} ^ {2} \leq 1 \\ \end{array}
$$
where  $K_{1}$  is a Lipschitz constant. Furthermore, note that because we are sampling,  $\pi (a|s;\theta_k)$  will take on only a finite number of values, so its nonzero values will be bounded away from zero. Thus we can say
$$
\frac {1}{\min  \left\{\pi \left(a | s ; \theta_ {k}\right) \mid \pi \left(a | s ; \theta_ {k}\right) > 0 \right\}} \leq K _ {2}
$$
for some large enough  $K_{2}$ . Thus using the bounds from these conditions, we can demonstrate
$$
\mathbb {E} \left[ \| \delta \theta_ {k + 1} \| ^ {2} | \mathcal {F} _ {\theta , k} \right] \leq 2 \cdot K _ {1} \left(1 + \| \theta_ {k} \| _ {\infty} ^ {2}\right) \cdot K _ {2} \left(\frac {R _ {\max }}{1 - \gamma} \cdot 1 + \frac {H _ {\max }}{1 - \gamma} \cdot \left(\lambda_ {\max } \cdot 1 + 1\right)\right) <   \infty
$$
Therefore  $\delta \theta_{k + 1}$  is square integrable.
Lemma 2: Secondly, we can demonstrate  $\delta \theta_{\epsilon} \to 0$
$$
\begin{array}{l} \delta \theta_ {\epsilon} = \sum_ {s _ {i}, a _ {i}} d _ {0} (s _ {0}) P ^ {\pi_ {\theta_ {k}}} (s _ {i}, a _ {i} | s _ {0}) \left[ \gamma^ {i} \left[ \left(Q (s _ {i}, a _ {i}; \eta_ {k}) - Q ^ {\pi_ {\theta_ {k}}} (s _ {i})\right) [ 1 - p _ {\xi_ {k}} (s _ {i}) ] \right. \right. \\ \left.\left. + \left(- Q _ {c} \left(s _ {i}, a _ {i}; \kappa_ {k}\right) + Q _ {c} ^ {\pi_ {\theta_ {k}}} \left(s _ {i}, a _ {i}\right)\right)\left[ \lambda_ {\omega} \left(1 - p _ {\xi_ {k}} \left(s _ {i}\right)\right) + p _ {\xi_ {k}} \left(s _ {i}\right)\right]\right] \nabla_ {\theta} \log \pi \left(a _ {i} \mid s _ {i}; \theta\right)\right| _ {\theta = \theta_ {k}} \left. \right] \\ \leq \sum_ {s _ {i}, a _ {i}} d _ {0} (s _ {0}) P ^ {\pi_ {\theta_ {k}}} (s _ {i}, a _ {i} | s _ {0}) \left[ \gamma^ {i} \left[ (Q (s _ {i}, a _ {i}; \eta_ {k}) - Q (s _ {i}, a _ {i}; \eta^ {*})) [ 1 - p _ {\xi_ {k}} (s _ {i}) ] \right. \right. \\ + \left(- Q _ {c} (s _ {i}, a _ {i}; \kappa_ {k}) + Q _ {c} (s _ {i}, a _ {i}; \kappa^ {*})\right) \left[ \lambda_ {\omega} (1 - p _ {\xi_ {k}} (s _ {i})) + p _ {\xi_ {k}} (s _ {i}) \right] \bigg ] \nabla_ {\theta} \log \pi (a _ {i} | s _ {i}; \theta) | _ {\theta = \theta_ {k}} \bigg ] \\ \leq \sum_ {s _ {i}, a _ {i}} d _ {0} (s _ {0}) P ^ {\pi_ {\theta_ {k}}} (s _ {i}, a _ {i} | s _ {0}) \left[ \gamma^ {i} \left[ | | Q (s _ {i}, a _ {i}; \eta_ {k}) - Q (s _ {i}, a _ {i}; \eta^ {*}) | | [ 1 - p _ {\xi_ {k}} (s _ {i}) ] \right. \right. \\ + \left. | | - Q _ {c} (s _ {i}, a _ {i}; \kappa_ {k}) + Q _ {c} (s _ {i}, a _ {i}; \kappa^ {*}) | | [ \lambda_ {\omega} (1 - p _ {\xi_ {k}} (s _ {i})) + p _ {\xi_ {k}} (s _ {i}) ] \right] \nabla_ {\theta} \log \pi (a _ {i} | s _ {i}; \theta) | _ {\theta = \theta_ {k}} \Biggr ] \\ \end{array}
$$
And because we have  $||Q(s,a;\eta_k) - Q(s,a;\eta^*)||\to 0$  and  $||Q_{c}(s,a;\kappa_{k}) - Q_{c}(s,a;\kappa^{*})||\to 0$  almost surely, we can therefore say  $\delta \theta_{\epsilon}\rightarrow 0$
Lemma 3: Finally, since  $\hat{\nabla}_{\theta}J_{\pi}(\theta)|_{\theta = \theta_k}$  is a sample of  $\nabla_{\theta}L(\theta ,\xi_k,\omega_k)|_{\theta = \theta_k}$  based on the history of sampled trajectories, we conclude that  $\mathbb{E}[\delta \theta_{k + 1}|\mathcal{F}_{\theta ,k}] = 0$
From the 3 above lemmas, the policy  $\theta$  update is a stochastic approximation of a continuous system  $\theta(t)$  defined by [10]
$$
\dot {\theta} = \Upsilon_ {\Theta} \left[ - \nabla_ {\theta} L (\theta , \xi , \omega) \right] \tag {7}
$$
$$
\Upsilon_ {\Theta} [ M (\theta) \triangleq \lim _ {0 <   \psi \to 0} \frac {\Gamma_ {\Theta} (\theta + \psi M (\theta)) - \Gamma_ {\Theta} (\theta)}{\psi}
$$
or in other words the left directional derivative of  $\Gamma_{\Theta}(\theta)$  in the direction of  $M(\theta)$ . Using the left directional derivative  $\Upsilon_{\Theta}[-\nabla_{\theta}L(\theta, \xi, \omega)]$  in the gradient descent algorithm for learning the policy  $\pi_{\theta}$  ensures the gradient will point in the descent direction along the boundary of  $\Theta$  when the  $\theta$  update hits its boundary. Using Step 2 in Appendix A.2 from [8], we have that  $dL(\theta, \xi, \omega)/dt = -\nabla_{\theta}L(\theta, \xi, \omega)^T \cdot \Upsilon_{\Theta}[-\nabla_{\theta}L(\theta, \xi, \omega)] \leq 0$  and the value is non-zero if  $||\Upsilon_{\Theta}[-\nabla_{\theta}L(\theta, \xi, \omega)||| \neq 0$ . Now consider the continuous system  $\theta(t)$ . For some fixed  $\xi$  and  $\omega$ , define a Lyapunov function
$$
\mathcal {L} _ {\xi , \omega} (\theta) = L (\theta , \xi , \omega) - L \left(\theta^ {*}, \xi , \omega\right)
$$
where  $\theta^{*}$  is a local minimum point. Then there exists a ball centered at  $\theta^{*}$  with a radius  $\rho$  such that  $\forall \theta \in \mathfrak{B}_{\theta^{*}}(\rho) = \{\theta ||\theta -\theta^{*}||\leq \rho \}$ ,  $\mathcal{L}_{\xi ,\omega}(\theta)$  is a locally positive definite function, that is  $\mathcal{L}_{\xi ,\omega}(\theta)\geq 0$ . Using Proposition 1.1.1 from [54], we can show that  $\Upsilon_{\Theta}[-\nabla_{\theta}L(\theta ,\xi ,\omega)]|_{\theta = \theta^{*}} = 0$  meaning  $\theta^{*}$  is a stationary point. Since  $dL(\theta ,\xi ,\omega) / dt\leq 0$ , through Lyapunov theory for asymptotically stable systems presented in Chapter 4 of [55], we can use the above arguments to demonstrate that with any initial conditions of  $\theta (0)\in \mathfrak{B}_{\theta^{*}}(\rho)$ , the continuous state trajectory of  $\theta (t)$  converges to  $\theta^{*}$ . Particularly,  $L(\theta^{*},\xi ,\omega)\leq L(\theta (t),\xi ,\omega)\leq L(\theta (0),\xi ,\omega)$  for all  $t > 0$ .
Using these aforementioned properties, as well as the facts that 1)  $\nabla_{\theta}L(\theta ,\xi ,\omega)$  is a Lipschitz function (using Proposition 17 from [8]), 2) the step-sizes of Assumption on steps sizes, 3)  $\delta \theta_{k + 1}$  is a square integrable Martingale difference sequence and  $\delta \theta_{\epsilon}$  is a vanishing error almost surely, and 4)  $\theta_{k}\in \Theta ,\forall k$  implying that  $\sup_k||\theta_k|| < \infty$  almost surely, we can invoke Theorem 2 of chapter 6 in [10] to demonstrate the sequence  $\{\theta_k\}$ ,  $\theta_{k}\in \Theta$  converges almost surely to the solution of the ODE defined by Equation 7, which additionally converges almost surely to the local minimum  $\theta^{*}\in \Theta$ .