SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {x} _ {0}} \left[ Y _ {t + 1} \mid \mathcal {F} _ {\mathbf {x} _ {0}, t} \right] (\rho) \\ = \mathbb {E} _ {\mathbf {x} _ {0}} \left[ Y _ {t + 1} \cdot \mathbb {I} \left(V \left(\mathbf {x} _ {t + 1}\right) <   \lambda\right) \mid \mathcal {F} _ {\mathbf {x} _ {0}, t} \right] (\rho) + \mathbb {E} _ {\mathbf {x} _ {0}} \left[ Y _ {t + 1} \cdot \mathbb {I} \left(V \left(\mathbf {x} _ {t + 1}\right) \geq \lambda\right) \mid \mathcal {F} _ {\mathbf {x} _ {0}, t} \right] (\rho) \\ = \frac {1}{\gamma^ {t + 1}} \cdot \mathbb {E} _ {\mathbf {x} _ {0}} \left[ X _ {t + 1} \cdot \mathbb {I} (V (\mathbf {x} _ {t + 1}) <   \lambda) \mid \mathcal {F} _ {\mathbf {x} _ {0}, t} \right] (\rho) + \mathbb {E} _ {\mathbf {x} _ {0}} \left[ Y _ {t} \cdot \mathbb {I} (V (\mathbf {x} _ {t + 1}) \geq \lambda) \mid \mathcal {F} _ {\mathbf {x} _ {0}, t} \right] (\rho) \\ = \frac {1}{\gamma^ {t + 1}} \cdot \mathbb {E} _ {\mathbf {x} _ {0}} [ X _ {t + 1} \mid \mathcal {F} _ {\mathbf {x} _ {0}, t} ] (\rho) - \frac {1}{\gamma^ {t + 1}} \cdot \mathbb {E} _ {\mathbf {x} _ {0}} [ X _ {t + 1} \cdot \mathbb {I} (V (\mathbf {x} _ {t + 1}) \geq \lambda) \mid \mathcal {F} _ {\mathbf {x} _ {0}, t} ] (\rho) \\ + \mathbb {E} _ {\mathbf {x} _ {0}} \left[ Y _ {t} \cdot \mathbb {I} \left(V \left(\mathbf {x} _ {t + 1}\right) \geq \lambda\right) \mid \mathcal {F} _ {\mathbf {x} _ {0}, t} \right] (\rho) \\ \leq \frac {1}{\gamma^ {t + 1}} \cdot \gamma \cdot X _ {t} (\rho) - \mathbb {E} _ {\mathbf {x} _ {0}} \left[ \left(\frac {1}{\gamma^ {t + 1}} \cdot X _ {t + 1} - Y _ {t}\right) \cdot \mathbb {I} (V (\mathbf {x} _ {t + 1}) \geq \lambda) \mid \mathcal {F} _ {\mathbf {x} _ {0}, t} \right] (\rho) \\ = Y _ {t} (\rho) - \mathbb {E} _ {\mathbf {x} _ {0}} \left[ \left(\frac {1}{\gamma^ {t + 1}} \cdot X _ {t + 1} - Y _ {t}\right) \cdot \mathbb {I} \left(V \left(\mathbf {x} _ {t + 1}\right) \geq \lambda\right) \mid \mathcal {F} _ {\mathbf {x} _ {0}, t} \right] (\rho) \\ \leq Y _ {t} (\rho). \\ \end{array}
$$
The first equality holds by the law of total probability. The second equality holds by the definition of  $Y_{t + 1}$ . The third equality holds by the law of total probability. The fourth inequality holds since above we proved that  $\mathbb{E}_{\mathbf{x}_0}[X_{t + 1} \mid \mathcal{F}_{\mathbf{x}_0,t}](\rho) \leq \gamma \cdot X_t$  whenever  $V(\mathbf{x}_i) < \lambda$  for all  $0 \leq i \leq t$  (cases 1 and 2 above). The fifth equality holds by definition of  $Y_t$  and the assumption that  $V(\mathbf{x}_i) < \lambda$  for all  $0 \leq i \leq t$ . Finally, the sixth inequality follows by observing that in the case when  $V(\mathbf{x}_i) < \lambda$  for all  $0 \leq i \leq t$  but  $V(\mathbf{x}_{t + 1}) \geq \lambda$ , we have  $X_{t + 1} / \gamma^{t + 1} \geq \lambda / \gamma^{t + 1} \geq \lambda / \gamma^t \geq Y_t$ .
2. If  $V(\mathbf{x}_i) = \lambda$  for some  $0 \leq i \leq t$ , then  $\mathbb{E}_{\mathbf{x}_0}[Y_{t + 1} \mid \mathcal{F}_{\mathbf{x}_0, t}](\rho) = Y_t(\rho) = Y_i(\rho)$ .
Hence, we have proved that  $(Y_{t})_{t = 0}^{\infty}$  is a nonnegative supermartingale.
We conclude the theorem claim by observing that
$$
\begin{array}{l} \mathbb {P} _ {\mathbf {x} _ {0}} \left[ \sup _ {t \geq 0} X _ {t} <   \lambda \right] = \mathbb {P} _ {\mathbf {x} _ {0}} \left[ \sup _ {t \geq 0} \gamma^ {t} \cdot Y _ {t} <   \lambda \right] = \mathbb {P} _ {\mathbf {x} _ {0}} \left[ \sup _ {t \geq N} \gamma^ {t} \cdot Y _ {t} <   \lambda \right] \\ = \mathbb {P} _ {\mathbf {x} _ {0}} \left[ \gamma^ {N} \cdot \sup  _ {t \geq N} \gamma^ {t - N} \cdot Y _ {t} <   \lambda \right] = \mathbb {P} _ {\mathbf {x} _ {0}} \left[ \sup  _ {t \geq N} \gamma^ {t - N} \cdot Y _ {t} <   \frac {\lambda}{\gamma^ {N}} \right] \\ \geq \mathbb {P} _ {\mathbf {x} _ {0}} \left[ \sup  _ {t \geq N} Y _ {t} <   \frac {\lambda}{\gamma^ {N}} \right] \geq \mathbb {P} _ {\mathbf {x} _ {0}} \left[ \sup  _ {t \geq 0} Y _ {t} <   \frac {\lambda}{\gamma^ {N}} \right] \geq 1 - \frac {1}{\lambda} \cdot \gamma^ {N}. \\ \end{array}
$$
Three non-trivial steps are the first and the second equality and the last inequality. The first equality holds since, if  $\sup_{t\geq 0}X_t < \lambda$ , then we also have  $Y_{t} = X_{t} / \gamma^{t}$  for each  $t$  by the definition of  $Y_{t}$ . The second equality holds since the system cannot reach a state in which  $V\geq \lambda$  and so  $X_{t}\geq \lambda$  in less than  $N$  time steps. On the other hand, for the last inequality we the inequality in Theorem 7. Applying the inequality to  $(Y_{t})_{t = 0}^{\infty}$  and  $\frac{\lambda}{\gamma^N}$  and observing that  $\mathbb{E}[Y_0] = \mathbb{E}[X_0]\leq 1$  by the Initial condition in Definition 2 yields  $\mathbb{P}_{\mathbf{x}_0}[\sup_{t\geq 0}Y_t\geq \lambda /\gamma^N ]\leq \frac{\gamma^N}{\lambda}$  and thus the last inequality.
E Learning Policies with Reach-avoid Supermartingales
We now present the POLICY+RASM subprocedure that we use for simultaneously learning a policy  $\pi_{\mu}$  and an RASM  $V_{\theta}$ , both of which are parametrized as neural networks with parameters  $\mu$  and  $\theta$ . The subprocedure POLICY+RASM is identical to the algorithm of [68], thus we keep this exposition brief and refer the reader to [68] for details. The reason why we can reuse this algorithm even though it learns additive RASMs is that additive and multiplicative RASMs are equivalent by Theorem 1. As we show below, the algorithm does not need to explicitly set an additive term  $\epsilon$  or a multiplicative factor  $\gamma$ , thus it is applicable to learning both additive and multiplicative RASMs. We then show how to use  $V_{\theta}$  to extract the bound in Theorem 2.
Analogously as in [68], the value  $\lambda > 1$  in Definition 3 is an algorithm parameter and we initialize it to  $\lambda = \frac{1}{1 - p'}$  so that the Theorem 2 bound  $1 - \frac{1}{\lambda} \cdot \gamma^{N} \geq 1 - \frac{1}{\lambda} = p'$  implies satisfaction of the desired probabilistic reach-avoid specification. If the algorithm succeeds in learning  $\pi_{\mu}$  and  $V_{\theta}$  with this value of  $\lambda$ , then the reach-avoid problem is solved. Otherwise, the algorithm gradually decreases the value of  $\lambda$  and tries to relearn  $\pi_{\mu}$  and  $V_{\theta}$  so that the resulting bound in Theorem 2 still exceeds  $p'$ . Thus, our new bound also yields an improvement in the algorithm.
The algorithm consists of two modules called learner and verifier, which are composed into a loop. In each loop iteration, the learner first learns a policy  $\pi_{\mu}$  and an RASM candidate  $V_{\theta}$ . These are then passed to the verifier which formally checks whether  $V_{\theta}$  satisfies all conditions in Definition 2. If the verification is successful, the algorithm returns the policy. Otherwise, the verifier identifies counterexample states at which the additive RASM conditions are violated. These are then passed to the learner and are used to fine-tune the previously learned policy and RASM by refining the loss function using the computed counterexamples.
Learner A policy  $\pi_{\mu}$  and an additive RASM candidate  $V_{\theta}$  are learned by minimizing the loss function
$$
\mathcal {L} (\theta , \nu) = \mathcal {L} _ {\text {I n i t}} (\nu) + \mathcal {L} _ {\text {U n s a f e}} (\nu) + \mathcal {L} _ {\text {D e c}} (\theta , \nu) + \mathcal {L} _ {\text {L i p s c h i t z}} (\theta , \nu).
$$
The first three loss terms are constructed from the sets  $C_{\mathrm{init}}$ ,  $C_{\mathrm{unsafe}}$  and  $C_{\mathrm{dec}}$  which are initialized by computing finite discretizations of  $\mathcal{X}_0$ ,  $\mathcal{X}_u$  and  $\mathcal{X} \backslash \mathcal{X}_t$  and are later extended by counterexamples computed by the verifier. The loss terms are used to guide the learner towards learning a true additive RASM which satisfies the Initial, Safety and Expected decrease conditions. Each loss term is designed to incur a loss at a counterexample whenever that counterexample violates the corresponding condition. In order for the Nonnegativity condition to be satisfied by default, the algorithm applies the softplus activation function to the output of  $V_{\theta}$ . The loss term  $\mathcal{L}_{\mathrm{Lipschitz}}(\theta, \nu)$  is a regularization term that does not enforce any of the defining conditions of additive RASMs, however it helps in decreasing the Lipschitz constants of neural networks. Each loss term is defined as follows:
$$
\mathcal{L}_{\text{Init}}(\nu) = \max_{\mathbf{x}\in C_{\text{init}}}\left\{V_{\nu}(\mathbf{x}) - 1,0\right\}
$$
$$
\mathcal {L} _ {\text {U n s a f e}} (\nu) = \max  _ {\mathbf {x} \in C _ {\text {u n s a f e}}} \left\{\frac {1}{1 - p} - V _ {\nu} (\mathbf {x}), 0 \right\}
$$
$$
\mathcal {L} _ {\text {D e c r e a s e}} (\theta , \nu) = \frac {1}{| C _ {\text {d e c}} |}.
$$
$$
\sum_ {\mathbf {x} \in C _ {\text {d e c r e a s e}}} \left(\max  \left\{\sum_ {\omega_ {1}, \dots , \omega_ {N} \sim \mathcal {N}} \frac {V _ {\nu} \left(f (\mathbf {x} , \pi_ {\theta} (\mathbf {x}) , \omega_ {i})\right)}{N} - V _ {\theta} (\mathbf {x}) + \tau \cdot K, 0 \right\}\right)
$$
The last loss term  $\mathcal{L}_{\mathrm{Lipschitz}}(\theta, \nu) = t \cdot (\mathcal{L}_{\mathrm{Lipschitz}}(\theta) + \mathcal{L}_{\mathrm{Lipschitz}}(\nu))$  is the regularization term used to guide the learner towards learning neural networks whose Lipschitz constants are below a tolerable threshold  $\rho$ , where  $t > 0$  is a regularization constant. By preferring networks with small Lipschitz constants, we allow the verifier to use a wider mesh and thus make verification condition easier to satisfy. We have The regularization term for  $\pi_{\theta}$  (and analogously for  $V_{\nu}$ ) is defined via
$$
\mathcal {L} _ {\text {L i p s c h i t z}} (\theta) = \max  \left\{\prod_ {W, b \in \theta} \max  _ {j} \sum_ {i} | W _ {i, j} | - \rho , 0 \right\},
$$
where  $W$  and  $b$  weight matrices and bias vectors for each layer in  $\pi_{\theta}$ .
**Verifier** The verifier checks whether  $V_{\theta}$  satisfies the defining properties of additive RASMs in Definition 2. Recall, the Nonnegativity condition is satisfied by default due to the softplus activation function applied to the output layer of  $V_{\theta}$ . Hence, the verifier only needs to check the Initial, Safety and Expected decrease conditions.
Since  $f$ ,  $\pi_{\mu}$  and  $V_{\theta}$  are continuous functions defined over a compact domain  $\mathcal{X}$  thus also Lipschitz continuous, the verifier may check the (both Multiplicative and Additive) expected decrease condition by checking a slightly stricter condition at finitely many discretization points. A discretization of the state space  $\mathcal{X}$  with mesh  $\tau > 0$  is a finite set  $\tilde{\mathcal{X}} \subseteq \mathcal{X}$  such that, for every  $\mathbf{x} \in \mathcal{X}$ , there exists  $\tilde{\mathbf{x}} \in \tilde{\mathcal{X}}$  such that  $||\mathbf{x} - \tilde{\mathbf{x}}||_1 < \tau$ . The discretization is computed by taking a grid of mesh  $\tau$ . Then, to check the expected decrease condition, it was showed in [68] that it suffices to check for each  $\tilde{\mathbf{x}} \in \tilde{\mathcal{X}}$  whose
adjacent discretization grid cells contain a non-target state and over which  $V$  attains a value that is less than or equal to  $\lambda$  that
$$
\mathbb {E} _ {\omega \sim d} \left[ V _ {\theta} \left(f (\tilde {\mathbf {x}}, \pi (\tilde {\mathbf {x}}), \omega)\right) \right] <   V _ {\theta} (\tilde {\mathbf {x}}) - \tau \cdot K,
$$
where  $K = L_V \cdot (L_f \cdot (L_\pi + 1) + 1)$  and  $L_f, L_\pi$  and  $L_V$  are Lipschitz constants of  $f$ ,  $\pi_\mu$  and  $V_\theta$ . It is assumed that  $L_f$  is provided and  $L_\pi$  and  $L_V$  are computed by the method of [57]. To verify the Initial condition, the verifier collects the set  $\mathsf{Cells}_{\mathcal{X}_0}$  of all cells of the discretization grid that intersect the initial set  $\mathcal{X}_0$ . Then, for each cell  $\in \mathsf{Cells}_{\mathcal{X}_0}$ , it checks if  $\sup_{\mathbf{x} \in \mathsf{cell}} V_\theta(\mathbf{x}) \leq 1$ , where the supremum is bounded from above by using interval arithmetic abstract interpretation (IA-AI) [22, 27] to propagate across neural network layers the extreme values that  $V_\theta$  can attain over a cell. Similarly, to verify the Unsafe condition, the verifier collects the set  $\mathsf{Cells}_{\mathcal{X}_u}$  of all cells of the discretization grid that intersect the initial set  $\mathcal{X}_u$ . Then, for each cell  $\in \mathsf{Cells}_{\mathcal{X}_u}$ , it uses IA-AI to check if  $\inf_{\mathbf{x} \in \mathsf{cell}} V_\theta(\mathbf{x}) \geq \lambda$ .
If the verifier shows that  $V_{\theta}$  satisfies the above checks, it concludes that  $V_{\theta}$  is an additive (and therefore multiplicative) RASM for the system under the policy  $\pi_{\mu}$  and returns the policy together with the lower bound on the probability of satisfying the reach-avoid specification as in Theorem 2. The fact that the verifier is correct was proved in [68, Theorem 2]. Otherwise, if a counterexample  $\tilde{x}$  to any of the checks is found, it is added to one of the three counterexample sets  $C_{\mathrm{init}}$ ,  $C_{\mathrm{unsafe}}$  and  $C_{\mathrm{dec}}$  that are then used by the learner to fine-tune  $V_{\theta}$  and  $\pi_{\mu}$ .
Soundness and computation of  $\gamma$ . The following theorem establishes that the above is a sound verification procedure and provides a closed-form expression for the values of  $\delta > 0$  and  $\gamma \in (0,1)$  for which  $V_{\theta}$  is a  $(\gamma, \delta, \lambda)$ -multiplicative RASM. Hence, to compute the lower bound on the probability of satisfying reach-avoidance in Theorem 2, one may use the value of  $\gamma$  implied by the theorem together with  $\lambda$  which is fixed by the algorithm,  $L_{V}$  which is computed by the algorithm and the maximal step size  $\Delta$  which we assume is provided by the user.
Theorem 8. If the verifier returns neural networks  $\pi_{\mu}$  and  $V_{\theta}$ , then  $V_{\theta}$  is a  $(\gamma, \delta, \lambda)$ -multiplicative RASM with
$$
\delta = \min \left\{\min _ {\tilde {\mathbf {x}} \in \tilde {\mathcal {X}}} \Big (V _ {\theta} (\tilde {\mathbf {x}}) - \tau \cdot K - \mathbb {E} _ {\omega \sim d} [ V _ {\theta} (f (\tilde {\mathbf {x}}, \pi (\tilde {\mathbf {x}}), \omega)) ] \Big), \lambda \right\}, \gamma = 1 - \frac {\delta}{\lambda}.
$$
Proof. Since it was shown in [68] that the verifier provides a sound verification procedure for checking that  $V_{\theta}$  is an additive RASM, by the equivalence in Theorem 1 it follows that it is also sound for checking that  $V_{\theta}$  is a  $(\gamma, \delta, \lambda)$ -multiplicative RASM for some values of  $\gamma$  and  $\delta$ . Hence, it remains to show that the values of  $\gamma$  and  $\delta$  in the theorem statement are correct.
First, we show that the Strict positivity outside  $\mathcal{X}_t$  condition is satisfied with the above value of  $\delta$ . To see this, let  $\mathbf{x} \in \mathcal{X} \backslash \mathcal{X}_t$ . Since  $\delta \leq \lambda$ , suppose without loss of generality that  $V_{\theta}(\mathbf{x}) \leq \lambda$ . Then,  $\mathbf{x}$  is contained in a discretization grid cell which contains a non-target state and over which  $V$  attains a value that is  $\leq \lambda$ . Hence, the verifier has shown that  $\mathbb{E}_{\omega \sim d}[V_{\theta}(f(\tilde{\mathbf{x}}, \pi(\tilde{\mathbf{x}}), \omega))] < V_{\theta}(\tilde{\mathbf{x}}) - \tau \cdot K$  holds for each  $\tilde{\mathbf{x}} \in \tilde{\mathcal{X}}$  which is a vertex of this cell. Taking a vertex  $\tilde{\mathbf{x}} \in \tilde{\mathcal{X}}$  of this cell for which  $||\mathbf{x} - \tilde{\mathbf{x}}||_1 \leq \tau$ , by the definition of Lipschitz constants we have
$$
\begin{array}{l} \mathbb {E} _ {\omega \sim d} \left[ V \left(f (\mathbf {x}, \pi (\mathbf {x}), \omega)\right) \right] \\ \leq \mathbb {E} _ {\omega \sim d} \left[ V \left(f (\tilde {\mathbf {x}}, \pi (\tilde {\mathbf {x}}), \omega)\right) \right] + \| f (\tilde {\mathbf {x}}, \pi (\tilde {\mathbf {x}}), \omega) - f (\mathbf {x}, \pi (\mathbf {x}), \omega) \| _ {1} \cdot L _ {V} \\ \leq \mathbb {E} _ {\omega \sim d} \left[ V \left(f (\tilde {\mathbf {x}}, \pi (\tilde {\mathbf {x}}), \omega)\right) \right] + | | (\tilde {\mathbf {x}}, \pi (\tilde {\mathbf {x}}), \omega) - (\mathbf {x}, \pi (\mathbf {x}), \omega) | | _ {1} \cdot L _ {V} \cdot L _ {f} \\ \leq \mathbb {E} _ {\omega \sim d} \left[ V \left(f (\tilde {\mathbf {x}}, \pi (\tilde {\mathbf {x}}), \omega)\right) \right] + | | \tilde {\mathbf {x}} - \mathbf {x} | | _ {1} \cdot L _ {V} \cdot L _ {f} \cdot (1 + L _ {\pi}) \\ \leq \mathbb {E} _ {\omega \sim d} \left[ V \left(f (\tilde {\mathbf {x}}, \pi (\tilde {\mathbf {x}}), \omega)\right) \right] + \tau \cdot L _ {V} \cdot L _ {f} \cdot (1 + L _ {\pi}). \\ \end{array}
$$
Hence, by the Nonnegativity condition we also have
$$
\begin{array}{l} V (\mathbf {x}) \geq V (\mathbf {x}) - \mathbb {E} _ {\omega \sim d} \left[ V \left(f (\mathbf {x}, \pi (\mathbf {x}), \omega)\right) \right] \\ \geq V (\tilde {\mathbf {x}}) - \tau \cdot L _ {V} - \mathbb {E} _ {\omega \sim d} \left[ V \left(f (\tilde {\mathbf {x}}, \pi (\tilde {\mathbf {x}}), \omega)\right) \right] - \tau \cdot L _ {V} \cdot L _ {f} \cdot (1 + L _ {\pi}) \tag {2} \\ = V (\tilde {\mathbf {x}}) - \tau \cdot K - \mathbb {E} _ {\omega \sim d} \Big [ V \Big (f (\tilde {\mathbf {x}}, \pi (\tilde {\mathbf {x}}), \omega) \Big) \Big ] \\ \geq \delta , \\ \end{array}
$$
since  $K = L_V \cdot (L_f \cdot (L_\pi + 1) + 1)$ , which concludes the proof.
Second, we show that Multiplicative expected decrease condition with the above value of  $\gamma$  holds. Let  $\mathbf{x} \in \mathcal{X} \backslash \mathcal{X}_t$  be such that  $V_{\theta}(\mathbf{x}) \leq \lambda$ . We need to show that  $\gamma \cdot V(\mathbf{x}) \geq \mathbb{E}_{\omega \sim d}[V(f(\mathbf{x}, \pi(\mathbf{x}), \omega))]$ . Since we showed in eq. (2) that  $V(\mathbf{x}) \geq V(\mathbf{x}) - \mathbb{E}_{\omega \sim d}[V(f(\mathbf{x}, \pi(\mathbf{x}), \omega))] \geq \delta > 0$  and since  $V_{\theta}(\mathbf{x}) \leq \lambda$ , we have
$$
\begin{array}{l} \frac {\mathbb {E} _ {\omega \sim d} [ V (f (\mathbf {x} , \pi (\mathbf {x}) , \omega)) ]}{V (\mathbf {x})} = 1 - \frac {V (\mathbf {x}) - \mathbb {E} _ {\omega \sim d} [ V (f (\mathbf {x} , \pi (\mathbf {x}) , \omega)) ]}{V (\mathbf {x})} \\ \leq 1 - \frac {\delta}{V (\mathbf {x})} \leq 1 - \frac {\delta}{\lambda}. \\ \end{array}
$$
Proof. In order to prove that  $\pi$  guarantees satisfaction of the probabilistic specification  $(\phi, p)$ , by Theorem 5 it suffices to show that  $\pi$  satisfies abstract reachability for the abstract graph  $G$  with probability at least  $p$ .
To prove abstract reachability for  $G$  with probability at least  $p$ , we show that a random trajectory of the system under policy  $\pi$  satisfies reach-avoid specifications of the edges along the finite path  $s = v_{i_0}, v_{i_1}, \ldots, v_{i_k} = t$  exhibited above with probability at least  $p$ . To prove this, we proceed by induction on  $0 \leq j \leq k$  to show that a random trajectory of the system under policy  $\pi$  satisfies reach-avoid specifications of each edge along a prefix  $s = v_{i_0}, v_{i_1}, \ldots, v_{i_j}$  of this path with probability at least  $\operatorname{Prob}[v_{i_j}]$ . Recall, Prob is the dictionary computed by Algorithm 1. Abstract reachability for  $G$  with probability at least  $p$  then follows if we set  $j = k$ , since  $v_{j_k} = t$  and we must have  $\operatorname{Prob}[t] \geq p$  for Algorithm 1 to output a policy (lines 15-17).
The base case  $j = 0$  follows trivially since the system starts in the initial region  $\beta(s) = \mathcal{X}_0$  and since  $\operatorname{Prob}[v_{i_0}] = \operatorname{Prob}[s] = 1$  by line 6. For the inductive step, suppose that  $0 \leq j \leq k - 1$  and that  $\pi$  satisfies reach-avoid specifications of each edge along  $s = v_{i_0}, v_{i_1}, \ldots, v_{i_j}$  with probability at least  $\operatorname{Prob}[v_{i_j}]$ . The claim for the prefix of length  $j + 1$  then follows by our construction of the finite path  $s = v_{i_0}, v_{i_1}, \ldots, v_{i_k} = t$ , as it implies that Algorithm 1 has successfully learned an edge policy for the edge  $(v_{i_j}, v_{i_{j+1}})$  that ensures satisfaction of the associated reach-avoid specification with probability at least  $p_{(v_{i_j}, v_{i_{j+1}})}$  and for which  $\operatorname{Prob}[v_{i_{j+1}}] = p_{(v_{i_j}, v_{i_{j+1}})} \cdot \operatorname{Prob}[v_{i_j}]$ . Since the right-hand-side of this equality is a lower bound on the probability of  $\pi$  satisfying reach-avoid specifications of each edge along  $s = v_{i_0}, v_{i_1}, \ldots, v_{i_j}$  multiplied by a lower bound on the probability of it satisfying the reach-avoid specification of the edge  $(v_{i_j}, v_{i_{j+1}})$ , the claim follows. This concludes the proof by induction.