SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Syntax A specification in SPECTRL is defined in terms of predicates and specification formulas. An atomic predicate is a boolean function  $a: \mathcal{X} \to \{\text{true, false}\}$  which for each system state specifies whether it satisfies the predicate, and a predicate is defined as a boolean combination of atomic predicates. Specification formulas in SPECTRL are then defined by the grammar
$$
\phi := \text {a c h i e v e} b \mid \phi_ {1} \text {e n s u r i n g} b \mid \phi_ {1}; \phi_ {2} \mid \phi_ {1} \text {o r} \phi_ {2} \tag {1}
$$
where  $b$  is a predicate and  $\phi_1$  and  $\phi_2$  are specification formulas. Intuitively, achieve  $b$  requires the agent to reach a state in which the predicate  $b$  is satisfied and  $\phi_1$  ensuring  $b$  requires the agent to satisfy the specification  $\phi$  while only visiting states in which the predicate  $b$  is satisfied. The clause  $\phi_1; \phi_2$  requires the agent to first satisfy specification  $\phi_1$  and then satisfy specification  $\phi_2$ . Finally,  $\phi_1$  or  $\phi_2$  requires satisfaction of at least one of the specifications  $\phi_1$  or  $\phi_2$ .
Semantics Given a trajectory  $\rho = (\mathbf{x}_t, \mathbf{u}_t, \omega_t)_{t=0}^{\infty}$  and writing  $\rho^K = (\mathbf{x}_t, \mathbf{u}_t, \omega_t)_{t=0}^K$  for its finite prefix of length  $K$ , the semantics of each SPECTRAL clause are formally defined as follows:
$$
\begin{array}{l} \rho \quad \vDash \phi \quad \exists K \in \mathbb {N} _ {0} \mathrm {s . t .} \rho^ {K} \models \phi \\ \rho^ {K} \quad \vDash \text {a c h i e v e} p \quad \exists t \leq K \text {s . t .} p \left(\rho_ {t} ^ {K}\right) = \text {t r u e} \\ \rho^ {K} \models \phi_ {1} \text {e n s u r i n g} p \quad \rho^ {K} \models \phi_ {1} \wedge \forall t \leq K. p \left(\rho_ {t} ^ {K}\right) = \text {t r u e} \\ \rho^ {K} \models \phi_ {1}; \phi_ {2} \quad \exists t \leq K \text {s . t .} \rho_ {[ 0: t ]} ^ {K} \models \phi_ {1} \text {a n d} \rho_ {[ t: K ]} ^ {K} \models \phi_ {2} \\ \rho^ {K} \models \phi_ {1} \text {o r} \phi_ {2} \quad \rho^ {K} \models \phi_ {1} \text {o r} \rho^ {K} \models \phi_ {2} \\ \end{array}
$$
Here,  $\rho_t^K$  denotes the  $t$ -th state along  $\rho^K$ ,  $\rho_{[0:t]}^K$  denotes the prefix of  $\rho^K$  consisting of the first  $t + 1$  states along  $\rho^K$  and  $\rho_{[t:K]}^K$  denotes the suffix of  $\rho^K$  that starts in the  $(t + 1)$ -st state along  $\rho^K$ .
Theorem 4. For each  $\phi \in \mathrm{Finitary}$  there exists  $\phi' \in \mathrm{SPECTRAL}$  such that, for any word  $w$ , the word  $w$  is accepted by  $\phi$  iff the word  $w$  is accepted by  $\phi'$ .
Proof. Let  $\phi$  be a finitary specification defined over the set of atomic predicates  $AP$ . Since  $\phi$  is finitary, there exist a finite time horizon  $H$  and a set  $L$  of words over  $AP$  of length  $H$  such that an infinite word over the alphabet  $AP$  is accepted by  $\phi$  iff its prefix of length  $H$  is contained in  $L$ . Define a SPECTRAL formula  $\phi'$  via:
$$
\phi^{\prime} = \bigvee_{(w_{1},\ldots ,w_{H})\in L}p(w_{1});  p(w_{2});\ldots ;  p(w_{H})
$$
where each  $p(w_{i})$  is an atomic predicate associated to the  $i$ -th letter in the word  $(w_{1},\ldots ,w_{H})$  and; denotes sequential composition of SPECTRL specifications. Then, an infinite word  $w$  is accepted by  $\phi$  if and only if the prefix of  $w$  of length  $H$  is contained in  $L$ , which holds if and only if  $w$  is accepted by the SpectRL formula  $\phi^{\prime}$ . This completes our reduction.
B Abstract Reachability Definition and Proof of Theorem 5
Given a trajectory  $\rho = (\mathbf{x}_t, \mathbf{u}_t, \omega_t)_{t=0}^{\infty}$  of the system and an abstract graph  $G = (V, E, \beta, s, t)$ , we say that  $\rho$  satisfies abstract reachability for  $G$  (written  $\rho \models G$ ) if it gives rise to a path in  $G$  that traverses  $G$  from  $s$  to  $t$  and satisfies reach-avoid specifications of all traversed edges. Formally, we require that there exists a sequence of time steps  $0 = i_0 < i_1 < \dots < i_k$  and a finite path  $s = v_0, v_1, \ldots, v_k = t$  in  $G$  such that
1.  $\mathbf{x}_{i_j} \in \beta(v_j)$  holds for each  $0 \leq j \leq k$ , and
2.  $\mathbf{x}_t\in \beta (v_j,v_{j + 1})$  holds for each  $0\leq j < k$  and  $i_j\leq t\leq i_{j + 1}$
Intuitively, the first condition encodes that the trajectory satisfies reachability specifications of traversed vertices in  $G$  while the second condition encodes that it satisfies avoidance specifications of traversed edges in  $G$ . We then say that a policy  $\pi$  for the system satisfies abstract reachability for  $G$  with probability  $p \in [0,1]$  at an initial state  $\mathbf{x}_0 \in \mathcal{X}_0$ , if we have that  $\mathbb{P}_{\mathbf{x}_0}[\rho \in \Omega_{\mathbf{x}_0} \mid \rho \models G] \geq p$ .
We now provide the proof of Theorem 5.
Theorem 5. Consider a stochastic feedback loop system with an initial set of states  $\mathcal{X}_0\subseteq \mathcal{X}$  and let  $\phi$  be a SPECTRL specification. Then there exists an abstract graph  $G = (V,E,\beta ,s,t)$  with  $|V|$  in  $\mathcal{O}(|\phi |)$  such that, for each trajectory  $\rho$  of the system, we have  $\rho \models \phi$  if and only if  $\rho \models G$ . Hence, for each policy  $\pi$  and initial state  $\mathbf{x}_0\in \mathcal{X}_0$ , we have  $\mathbb{P}_{\mathbf{x}_0}[\rho \in \Omega_{\mathbf{x}_0}\mid \rho \models \phi ] = \mathbb{P}_{\mathbf{x}_0}[\rho \in \Omega_{\mathbf{x}_0}\mid \rho \models G]$ .
Proof. Given a SPECTRL specification  $\phi$ , one can construct an abstract graph  $G$  such that for each trajectory  $\rho$  of the system we have  $\rho \models \phi$  iff  $\rho \models G$  as follows. First, the specification  $\phi$  is parsed according to the grammar of SPECTRL in eq. 1 in order to construct the parse tree of  $\phi$ . We then start by constructing an abstract graph for each leaf formula in the parse tree, and traverse the parse tree bottom-up in order to construct abstract graphs of parent formulas. The abstract graph of the specification  $\phi$  is then obtained by taking the abstract graph constructed for the root in the parse tree. The leaves of the parse tree are formulas of the form achieve  $p$ , for which we construct an abstract graph with two vertices  $s$  and  $t$ , a single edge  $e = (s,t)$  and set  $\beta(s) = \mathcal{X}_0$ ,  $\beta(t) = \{\mathbf{x} \in \mathcal{X} \mid p(\mathbf{x}) = \text{true}\}$  and  $\beta(e) = \mathcal{X}$ . For a formula  $\phi_1$  ensuring  $p$ , we take an abstract graph  $(V,E,\beta,s,t)$  for the specification  $\phi_1$  which was already constructed for the child node and define the abstract graph  $G = (V,E,\beta',s,t)$  by simply modifying the map  $\beta$  via  $\beta'(e) = \beta(e) \cap \{\mathbf{x} \in \mathcal{X} \mid p(\mathbf{x}) = \text{true}\}$  for each  $e \in E$ . For a formula  $\phi_1;\phi_2$ , we take the abstract graph of the specifications  $\phi_1$  and  $\phi_2$  which were already constructed for the child nodes and merge them by identifying the target node of  $\phi_1$  with the source node of  $\phi_2$  and using the region associated to it by the abstract graph of  $\phi_2$ . Finally, for a formula  $\phi_1$  or  $\phi_2$ , we introduce a novel source node  $s$  with  $\beta(s) = \mathcal{X}_0$ , take the abstract graph of  $\phi_1$  and  $\phi_2$  and connect the novel source node  $s$  to them by an edge. Note that this construction yields a graph with  $V$  in  $\mathcal{O}(|\phi|)$ .
Since the above construction soundly encodes the semantics of each SPECTRL grammar element as a reach-avoid specification, it follows by induction on the depth of the parse tree that for each trajectory  $\rho$  of the system we have  $\rho \models \phi$  iff  $\rho \models G$ . The claim of Theorem 5 follows.
1. If a continuous function  $V: \mathcal{X} \to \mathbb{R}$  is an  $(\epsilon, \lambda)$ -additive RASM, then it is also a  $(\frac{\lambda - \epsilon}{\lambda}, \min \{\epsilon, \lambda\}, \lambda)$ -multiplicative RASM.
2. If a continuous function  $\bar{V} : \mathcal{X} \to \mathbb{R}$  is a  $(\gamma, \delta, \lambda)$ -multiplicative RASM, then it is also an  $((1 - \gamma) \cdot \delta, \lambda)$ -additive RASM.
1. Let  $\delta = \min\{\epsilon, \lambda\}$  and  $\gamma = \frac{\lambda - \epsilon}{\lambda}$ . To show that  $V$  is a  $(\gamma, \delta, \lambda)$ -multiplicative RASM, we need to show that the Strict positivity outside  $\mathcal{X}_t$  and the Multiplicative expected decrease conditions hold. By the Additive expected decrease condition, for each  $\mathbf{x} \in \mathcal{X} \setminus \mathcal{X}_t$  at which  $V(\mathbf{x}) \leq \lambda$  we have  $V(\mathbf{x}) \geq \epsilon$ . So as  $\delta = \min\{\epsilon, \lambda\}$ , the Strict positivity outside  $\mathcal{X}_t$  follows. On the other hand, observe that for every  $\mathbf{x} \in \mathcal{X} \setminus \mathcal{X}_t$  at which  $V(\mathbf{x}) \leq \lambda$ , we have
$$
\frac {\mathbb {E} _ {\omega \sim d} [ V (f (\mathbf {x} , \pi (\mathbf {x}) , \omega)) ]}{V (\mathbf {x})} \leq \frac {V (\mathbf {x}) - \epsilon}{V (\mathbf {x})} \leq \frac {\lambda - \epsilon}{\lambda} = \gamma ,
$$
where the first inequality follows by the Additive expected decrease condition and the second inequality follows since  $\frac{z - \epsilon}{z}$  is monotonically increasing on the domain  $z > \epsilon$ . Hence, the Multiplicative expected decrease condition holds.
2. Let  $\epsilon = (1 - \gamma) \cdot \delta$ . To show that  $V$  is an  $(\epsilon, \lambda)$ -additive RASM, we need to show that the Additive expected decrease condition holds. We show this by observing that, for each  $\mathbf{x} \in \mathcal{X} \backslash \mathcal{X}_t$  such that  $V(\mathbf{x}) \leq \lambda$ , we have
$$
V (\mathbf {x}) - \mathbb {E} _ {\omega \sim d} [ V (f (\mathbf {x}, \pi (\mathbf {x}), \omega)) ] \geq V (\mathbf {x}) - \gamma \cdot V (\mathbf {x}) = (1 - \gamma) \cdot V (\mathbf {x}) \geq (1 - \gamma) \cdot \delta ,
$$
where the first inequality holds by the Multiplicative expected decrease condition and the last inequality holds by the Strict positivity outside  $\mathcal{X}_t$  condition. Hence, the Additive expected decrease condition is satisfied.
We first provide an overview of definitions and results from martingale theory that we use in the proof. We then present the proof.
Probability theory A probability space is a triple  $(\Omega, \mathcal{F}, \mathbb{P})$  of a state space  $\Omega$ , a sigma-algebra  $\mathcal{F}$  and a probability measure  $\mathbb{P}$  that satisfies Kolmogorov axioms [64]. A random variable in  $(\Omega, \mathcal{F}, \mathbb{P})$  is a function  $X: \Omega \to \mathbb{R}$  that is  $\mathcal{F}$ -measurable, i.e. for each  $a \in \mathbb{R}$  we have  $\{\omega \in \Omega \mid X(\omega) \leq a\} \in \mathcal{F}$ . A (discrete-time) stochastic process is a sequence  $(X_i)_{i=0}^{\infty}$  of random variables in  $(\Omega, \mathcal{F}, \mathbb{P})$ .
Conditional expectation Let  $X$  be a random variable in  $(\Omega, \mathcal{F}, \mathbb{P})$ . Given a sub-sigma-algebra  $\mathcal{F}' \subseteq \mathcal{F}$ , a conditional expectation of  $X$  given  $\mathcal{F}'$  is an  $\mathcal{F}'$ -measurable random variable  $Y$  such that, for each  $A \in \mathcal{F}'$ , we have
$$
\mathbb {E} [ X \cdot \mathbb {I} (A) ] = \mathbb {E} [ Y \cdot \mathbb {I} (A) ].
$$
Here,  $\mathbb{I}(A):\Omega \to \{0,1\}$  is an indicator function of  $A$ , given by  $\mathbb{I}(A)(\omega) = 1$  if  $\omega \in A$ , and  $\mathbb{I}(A)(\omega) = 0$  if  $\omega \notin A$ . Intuitively, conditional expectation of  $X$  given  $\mathcal{F}'$  is an  $\mathcal{F}'$ -measurable random variable that behaves like  $X$  upon evaluating its expected value on events in  $\mathcal{F}'$ . It is known that every nonnegative random variable admits a conditional expectation [64]. Moreover, the conditional expectation is almost-surely unique, meaning that for any two  $\mathcal{F}'$ -measurable random variables  $Y$  and  $Y'$  which are conditional expectations of  $X$  given  $\mathcal{F}'$  we have  $\mathbb{P}[Y = Y'] = 1$ . Therefore, we pick any such random variable as a canonical conditional expectation and denote it by  $\mathbb{E}[X|\mathcal{F}]^\prime$ .
Supermartingales Let  $(\Omega, \mathcal{F}, \mathbb{P})$  be a probability space and  $(\mathcal{F}_i)_{i=0}^{\infty}$  be an increasing sequence of sub-sigma-algebras in  $\mathcal{F}$ , i.e.  $\mathcal{F}_0 \subseteq \mathcal{F}_1 \subseteq \dots \subseteq \mathcal{F}$ . A nonnegative supermartingale with respect to  $(\mathcal{F}_i)_{i=0}^{\infty}$  is a stochastic process  $(X_i)_{i=0}^{\infty}$  such that each  $X_i$  is  $\mathcal{F}_i$ -measurable, and  $X_i(\omega) \geq 0$  and  $\mathbb{E}[X_{i+1} \mid \mathcal{F}_i](\omega) \leq X_i(\omega)$  hold for each  $\omega \in \Omega$  and  $i \geq 0$ . Intuitively, the second condition is the expected decrease condition, and it is formally captured via conditional expectation.
We now present two results from martingale theory that will be used in the proof. Let  $(\Omega, \mathcal{F}, \mathbb{P})$  be a probability space and  $(\mathcal{F}_i)_{i=0}^{\infty}$  be an increasing sequence of sub- $\sigma$ -algebras in  $\mathcal{F}$ .
Theorem 6 (Supermartingale convergence theorem [64]). Let  $(X_{i})_{i = 0}^{\infty}$  be a nonnegative supermartingale with respect to  $(\mathcal{F}_i)_{i = 0}^{\infty}$ . Then, there exists a random variable  $X_{\infty}$  in  $(\Omega ,\mathcal{F},\mathbb{P})$  to which the supermartingale converges to with probability 1, i.e.  $\mathbb{P}[\lim_{i\to \infty}X_i = X_\infty ] = 1$ .
Theorem 7 ([35]). Let  $(X_{i})_{i = 0}^{\infty}$  be a nonnegative supermartingale with respect to  $(\mathcal{F}_i)_{i = 0}^{\infty}$ . Then, for every  $\lambda >0$ , we have
$$
\mathbb {P} \left[ \sup  _ {i \geq 0} X _ {i} \geq \lambda \right] \leq \frac {\mathbb {E} \left[ X _ {0} \right]}{\lambda}.
$$
Proof. Fix an initial state  $\mathbf{x}_0\in \mathcal{X}_0$ . We need to show that  $\mathbb{P}_{\mathbf{x}_0}\big[\mathrm{ReachAvoid}(\mathcal{X}_t,\mathcal{X}_u)\big]\geq 1 - \frac{1}{\lambda}\cdot \gamma^N$  with  $N = \lfloor (\lambda -1) / (L_V\cdot \Delta)\rfloor$
Before proceeding with the proof, we define some notions. Consider the probability space  $(\Omega_{\mathbf{x}_0},\mathcal{F}_{\mathbf{x}_0},\mathbb{P}_{\mathbf{x}_0})$  over the set of all system trajectories that start in  $\mathbf{x}_0\in \mathcal{X}_0$ . For each time step  $t\in \mathbb{N}_0$ , define  $\mathcal{F}_{\mathbf{x}_0,t}\subseteq \mathcal{F}_{\mathbf{x}_0}$  to be a sub- $\sigma$ -algebra which contains events that are defined in terms of the first  $t$  states along a trajectory. Formally, for each  $j\in \mathbb{N}_0$ , let  $C_j:\Omega_{\mathbf{x}_0}\to \mathcal{X}$  assign to each trajectory  $\rho = (\mathbf{x}_t,\mathbf{u}_t,\omega_t)_{t\in \mathbb{N}_0}\in \Omega_{\mathbf{x}_0}$  the  $j$ -th state  $\mathbf{x}_j$  along the trajectory.  $\mathcal{F}_i$  is then defined as the smallest  $\sigma$ -algebra over  $\Omega_{\mathbf{x}_0}$  with respect to which  $C_0,C_1,\ldots ,C_i$  are all measurable. The sequence  $(\mathcal{F}_{\mathbf{x}_0,t})_{t = 0}^{\infty}$  defines a filtration in  $(\Omega_{\mathbf{x}_0},\mathcal{F}_{\mathbf{x}_0},\mathbb{P}_{\mathbf{x}_0})$ .
Proceeding with the proof, we show that  $V$  induces a supermartingale in the probability space  $(\Omega_{\mathbf{x}_0},\mathcal{F}_{\mathbf{x}_0},\mathbb{P}_{\mathbf{x}_0})$  over the set of all system trajectories that start in  $\mathbf{x}_0\in \mathcal{X}_0$ . For each  $t\in \mathbb{N}_0$ , define a random variable  $X_{t}$  in  $(\Omega_{\mathbf{x}_0},\mathcal{F}_{\mathbf{x}_0},\mathbb{P}_{\mathbf{x}_0})$  via
$$
X _ {t} (\rho) = \left\{ \begin{array}{l l} V (\mathbf {x} _ {t}), & \text {i f} \mathbf {x} _ {i} \notin \mathcal {X} _ {t} \text {a n d} V (\mathbf {x} _ {i}) <   \lambda \text {f o r a l l} 0 \leq i \leq t \\ 0, & \text {i f} \mathbf {x} _ {i} \in \mathcal {X} _ {t} \text {f o r s o m e} 0 \leq i \leq t \text {a n d} V (\mathbf {x} _ {j}) <   \lambda \text {f o r a l l} 0 \leq j \leq i \\ \lambda , & \text {o t h e r w i s e} \end{array} \right.
$$
for each trajectory  $\rho = (\mathbf{x}_t, \mathbf{u}_t, \omega_t)_{t \in \mathbb{N}_0} \in \Omega_{\mathbf{x}_0}$ . Intuitively,  $X_t$  is equal to  $V$  at  $\mathbf{x}_t$  until either the target set  $\mathcal{X}_t$  is reached upon which  $X_t$  is set to 0, or some  $V(\mathbf{x}_t) \geq \lambda$  is reached upon which  $X_t$  is set to  $\lambda$ . We claim that  $(X_t)_{t=0}^{\infty}$  is a nonnegative supermartingale with respect to the filtration  $(\mathcal{F}_{\mathbf{x}_0,t})_{t=0}^{\infty}$  in the probability space  $(\Omega_{\mathbf{x}_0}, \mathcal{F}_{\mathbf{x}_0}, \mathbb{P}_{\mathbf{x}_0})$  and that it with probability 1 converges to either 0 or to a value that is  $\geq \lambda$ .
Clearly, each  $X_{t}$  is nonnegative. To prove that  $(X_{t})_{t = 0}^{\infty}$  is a supermartingale, note first that each  $X_{t}$  is  $\mathcal{F}_{\mathbf{x}_0,t}$  measurable as it is defined in terms of the first  $t$  states along a trajectory. To show that the expected decrease condition is satisfied, we show that  $\mathbb{E}_{\mathbf{x}_0}[X_{t + 1}|\mathcal{F}_{\mathbf{x}_0,t}](\rho)\leq X_t(\rho)$  holds for each  $t\in \mathbb{N}_0$  and  $\rho = (\mathbf{x}_t,\mathbf{u}_t,\omega_t)_{t\in \mathbb{N}_0}$ . We consider 3 cases based on the definition of  $V$ :
1. If  $\mathbf{x}_0, \mathbf{x}_1, \ldots, \mathbf{x}_t \notin \mathcal{X}_t$  and  $V(\mathbf{x}_i) < \lambda$  for each  $0 \leq i \leq t$ , then
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {x} _ {0}} \left[ X _ {t + 1} \mid \mathcal {F} _ {\mathbf {x} _ {0}, t} \right] (\rho) \\ = \mathbb {E} _ {\mathbf {x} _ {0}} \left[ X _ {t + 1} \cdot \left(\mathbb {I} \left(\mathbf {x} _ {t + 1} \notin \mathcal {X} _ {t} \wedge V (\mathbf {x} _ {t + 1}) <   \lambda\right) + \mathbb {I} \left(\mathbf {x} _ {t + 1} \in \mathcal {X} _ {t}\right) + \mathbb {I} (V (\mathbf {x} _ {t + 1}) \geq \lambda)\right) \mid \mathcal {F} _ {\mathbf {x} _ {0}, t} \right] (\rho) \\ = \mathbb {E} _ {\mathbf {x} _ {0}} \left[ X _ {t + 1} \cdot \mathbb {I} \left(\mathbf {x} _ {t + 1} \notin \mathcal {X} _ {t}\right) \mid \mathcal {F} _ {\mathbf {x} _ {0}, t} \right] (\rho) + 0 + \lambda \cdot \mathbb {E} \left[ \mathbb {I} \left(V \left(\mathbf {x} _ {t + 1}\right) \geq \lambda\right) \mid \mathcal {F} _ {\mathbf {x} _ {0}, t} \right] (\rho) \\ \leq \mathbb {E} _ {\omega \sim d} [ V (f (\mathbf {x} _ {t}, \mathbf {u} _ {t}, \omega_ {t})) \cdot \mathbb {I} (\mathbf {x} _ {t + 1} \notin \mathcal {X} _ {t} \wedge V (\mathbf {x} _ {t + 1}) <   \lambda) ] \\ + \mathbb {E} _ {\omega \sim d} [ V (f (\mathbf {x} _ {t}, \mathbf {u} _ {t}, \omega_ {t})) \cdot \mathbb {I} (\mathbf {x} _ {t + 1} \in \mathcal {X} _ {t}) ] \\ + \mathbb {E} _ {\omega \sim d} [ V (f (\mathbf {x} _ {t}, \mathbf {u} _ {t}, \omega_ {t})) \cdot \mathbb {I} (V (\mathbf {x} _ {t + 1}) \geq \lambda) ] \\ = \mathbb {E} _ {\omega \sim d} [ V (f (\mathbf {x} _ {t}, \mathbf {u} _ {t}, \omega_ {t})) ] \\ \leq \gamma \cdot V (\mathbf {x} _ {t}) \leq V (\mathbf {x} _ {t}). \\ \end{array}
$$
The first equality follows by the law of total probability, the second equality follows by definition of  $X_{t}$ , the third inequality follows by observing that  $V(\mathbf{x}_{t + 1})\geq X_{t + 1}(\rho)$  if  $\mathbf{x}_{t + 1}\in \mathcal{X}_t$ , the fourth equality is just the sum of expectations over disjoint sets, and finally the fifth inequality follows by the Multiplicative expected decrease condition of  $V$  and the assumption that  $\mathbf{x}_t\notin \mathcal{X}_t$  and  $V(\mathbf{x}_t) < \lambda$ .
2. If  $\mathbf{x}_i\in \mathcal{X}_t$  for some  $0\leq i\leq t$  and  $V(\mathbf{x}_j) < \lambda$  for all  $0\le j\le i$ , then we have  $\mathbb{E}_{\mathbf{x}_0}[X_{t + 1}|\mathcal{F}_{\mathbf{x}_0,t}](\rho) = \gamma \cdot X_{t + 1} = X_{t + 1}(\rho) = 0.$
3. Otherwise, we must have  $V(\mathbf{x}_i) \geq \lambda$  and  $\mathbf{x}_0, \ldots, \mathbf{x}_i \notin \mathcal{X}_t$  for some  $0 \leq i \leq t$ , thus  $\mathbb{E}_{\mathbf{x}_0}[X_{t+1} \mid \mathcal{F}_{\mathbf{x}_0,t}](\rho) = X_{t+1}(\rho) = \lambda$ .
Hence, we have proved that  $(X_{t})_{t = 0}^{\infty}$  is a nonnegative supermartingale.
By Supermartingale Convergence Theorem, we then know that  $(X_{t})_{t = 0}^{\infty}$  with probability 1 converges to some value. We claim furthermore that this value is either  $0$  or  $\geq \lambda$  and that the value is attained. To see this, recall that in Theorem 1 we showed that  $V$  is also an  $(\epsilon ,\lambda)$ -additive RASM with  $\epsilon = (1 - \gamma)\cdot \delta$ . Then, the same sequence of inequalities as in the case one above shows that  $\mathbb{E}_{\mathbf{x}_0}[X_{t + 1}|\mathcal{F}_{\mathbf{x}_0,t}](\rho)\leq X_t(\rho) - \epsilon$  if  $\mathbf{x}_0,\mathbf{x}_1,\ldots ,\mathbf{x}_t\notin \mathcal{X}_t$  and  $V(\mathbf{x}_i) < \lambda$  for each  $0\leq i\leq t$ . Thus, the limit to which  $(X_{t})_{t = 0}^{\infty}$  converges cannot be in the open interval  $(0,\lambda)$  and the claim follows.
To prove the theorem claim, we show that  $(X_{t})_{t = 0}^{\infty}$  converges to a value with  $\geq \lambda$  with probability at most  $\frac{1}{\lambda}\cdot \gamma^{N}$ . Then, since  $(X_{t})_{t = 0}^{\infty}$  converging to 0 implies that system reaches a state in which  $V < \delta$  while never reaching a state in which  $V\geq \lambda$ , which by the Safety and the Strict positivity outside  $\mathcal{X}_t$  conditions implies that reach-avoidance is satisfied, this will imply the theorem claim. The proof until this point is analogous to the proof of [68, Theorem 1].
The technical novelty of our proof begins in the following step. We define another stochastic process  $(Y_{t})_{t = 0}^{\infty}$  from  $(X_{t})_{t = 0}^{\infty}$  by letting
$$
Y _ {t} = \left\{ \begin{array}{l l} X _ {t} / \gamma^ {t}, & \text {i f V (\mathbf {x} _ {i}) <   \lambda f o r a l l 0 \leq i \leq t} \\ Y _ {t - 1}, & \text {o t h e r w i s e} \end{array} \right.
$$
We claim that  $(Y_{t})_{t = 0}^{\infty}$  is also a nonnegative supermartingale with respect to the filtration  $(\mathcal{F}_{\mathbf{x}_0,t})_{t = 0}^{\infty}$ . The nonnegativity part of the claim clearly holds since each  $X_{t}$  is nonnegative. To check the expected decrease condition of supermartingales, for each  $t\in \mathbb{N}_0$  and for each  $\rho \in \Omega_{\mathbf{x_0}}$  we have distinguish two cases:
1. If  $V(\mathbf{x}_i) < \lambda$  for all  $0 \leq i \leq t$ , then