SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \Pr \left[ \text {e s c a p e} f r o m s _ {0} \right] \geq \Pr \left[ t = 1: \left(s _ {0}, s ^ {\prime}\right), s ^ {\prime} \notin \hat {F} \right] \\ + \operatorname * {P r} [ t = 1: (s _ {0}, s), s \neq s _ {a} ] \cdot \operatorname * {P r} [ t = 2: (s, s ^ {\prime}), s ^ {\prime} \notin \hat {F} | t = 1: (s _ {0}, s), s \neq s _ {a} ] \\ + \Pr [ t = 1: (s _ {0}, s _ {a}) ] \cdot \Pr [ t = 2: (s _ {a}, s ^ {\prime}), s ^ {\prime} \notin \hat {F} | t = 1: (s _ {0}, s _ {a}) ] \\ = \frac {d - (k - 1)}{d} + \frac {k - 2}{d} \cdot \frac {d - (k - 1)}{d} + \frac {1}{d} \cdot \frac {d - (k - 2)}{d} \\ = 1 - \frac {k - 1}{d} + \frac {k - 2}{d} - \frac {(k - 2) (k - 1)}{d ^ {2}} + \frac {1}{d} - \frac {k - 2}{d ^ {2}} = \\ 1 - \frac {k - 2}{d ^ {2}} (k - 1 + 1) = 1 - \frac {k (k - 2)}{d ^ {2}} \\ \end{array}
$$
$$
\Pr \left[ s t a y i n g \right] \leq \frac {k (k - 2)}{d ^ {2}} <   \frac {(k - 1) ^ {2}}{d ^ {2}}.
$$
2. If  $s_0 = s_a$ , then
$$
\begin{array}{l} \Pr \left[ \text {e s c a p e f r o m} s _ {0} \right] \geq \Pr \left[ t = 1: \left(s _ {0}, s ^ {\prime}\right), s ^ {\prime} \notin \hat {F} \right] \\ + \Pr [ t = 1: (s _ {0}, s), s \in \hat {F} ] \cdot \Pr [ t = 2: (s, s ^ {\prime}), s ^ {\prime} \notin \hat {F} | t = 1: (s _ {0}, s), s \in \hat {F} ] \\ = \frac {d - (k - 2)}{d} + \frac {k - 2}{d} \cdot \frac {d - (k - 1)}{d} \\ = 1 - \frac {k - 2}{d} + \frac {k - 2}{d} - \frac {(k - 2) (k - 1)}{d ^ {2}} \\ = 1 - \frac {(k - 2) (k - 1)}{d ^ {2}} \\ \end{array}
$$
$$
\operatorname * {P r} [ s t a y i n g ] \leq \frac {(k - 2) (k - 1)}{d ^ {2}} <   \frac {(k - 1) ^ {2}}{d ^ {2}}.
$$
Given an environment, a policy, and a SAFEZONE, one could compute exactly how much safe it is (see Appendix E for details), from which we deduce our following corollary.
Corollary A.3. SAFEZONE is NP-complete.
We note that for  $H = 1$ , the GreEDy AT EACH STEP Algorithm is optimal.
B Proofs of Section 3
B.1 Greedy by Threshold Algorithm
A naive approach to the SAFEZONE problem is to return all states  $s \in S$  with probability  $p(s) \geq \beta$ , for some parameter  $\beta > 0$ , see Algorithm 3.
Algorithm 3 Greedy by Threshold
Parameter:  $\beta >0,\{p(s)\}_{s\in S}$  return  $\{s\in \mathcal{S}:p(s)\geq \beta \}$
Proof. There are at most  $\frac{H}{\beta}$  states with probability  $p(s) \geq \beta$ . Thus  $|F| \leq \frac{H}{\beta}$ .
Denote by  $F^{*}$  an optimal  $(\rho, k^{*}) - \mathrm{SAFEZONE}$  set. By the law of total probability,
$$
\operatorname * {P r} _ {\tau} [ \tau \not \subseteq F ] \leq \operatorname * {P r} _ {\tau} [ \tau \not \subseteq F ^ {*} ] + \operatorname * {P r} _ {\tau} [ \tau \subseteq F ^ {*} \setminus F ].
$$
Looking at the R.H.S of the inequality, the left term is smaller than  $\rho$  by the definition of SAFEZONE. The right term is equal to the probability of reaching a state in  $F^{*}$  that its probability is smaller than  $\beta$ , i.e., a state in  $F^{*} \setminus F$ .
Using union bound, this can be bounded by  $k^{*}\beta$
Lemma B.1. For every  $\rho \in (0, 1/2)$ ,  $H \in \mathbb{N}$ , there exists an MDP and a minimal integer  $k$  such that the MDP has a  $(\rho, k)$ -SAFEZONE, but for  $\beta = \rho / k$  GreEDy BY THRESHOLD Algorithm returns  $F$  with escape probability  $\leq 2\rho$  and of size  $|F| = \Omega(H/\beta)$ .
Proof. Fix  $\rho \in (0,1)$ . For ease of the presentation, we will assume that  $\frac{1 - \rho}{\beta}$  is an integer (if not, it should be rounded to the nearest integer). Define  $A$  to contain  $\frac{1 - \rho}{\beta} \cdot H$  states,  $B$  to contain  $k - 1$  states, and  $S = \{s_0\} \cup A \cup B$ . Consider the following MDP with states  $S$  and starting state  $s_0$ . The transition function is defined as follows:
- For every  $i \in A$ ,  $\operatorname{Pr}[s_{1,i}^A | s_0] = \beta$  and for every  $j \in [H - 1]$ ,  $\operatorname{Pr}[s_{j + 1,i}^A | s_{j,i}^A] = 1$ .
For  $s\in B$ $\operatorname *{Pr}[s|s_0] = \frac{1 - \rho}{k - 1}$
- For  $s \in B$ ,  $\operatorname{Pr}[s|s] = 1$
The MDP is illustrated in Figure 2. Clearly,  $\{s_0\} \cup B$  is a  $(\rho, k)$ -SAFEZONE. In addition, GreEDY BY THRESHOLD ALGORITHM returns the set of all states, as for every state  $s \in A$  we have that  $p(s) = \beta$ ,  $p(s_0) = 1 > \rho \geq \beta$ , and for every  $s \in B$  we have that  $p(s) = \frac{1 - \rho}{k - 1} > \frac{\rho}{k} = \beta$ . Thus the size of the returned set is  $\mathcal{S}$  is  $\Omega(H / \beta)$ , which completes the proof.
Figure 2: Lower bound for GreEDy BY THRESHOLD Algorithm.
B.2 Simulation Algorithm
Algorithm 4 Simulation Algorithm
Input:  $m = \frac{1}{\beta}\ln \frac{k^*}{0.005}$ $F\gets \{s_0\}$   
for  $i = 1\dots m$  do  
     $\tau \gets$  choose a random trajectory  
     $F\gets F\cup \tau$   
end for  
return  $F$
Proof. Denote by  $F^{*}$  the optimal  $(\rho, k^{*}) - \mathrm{SAFEZONE}$  set. By the law of total expectation, we can split  $\mathbb{E}[|F|]$  into two parts, depending on whether trajectories are entirely in  $F^{*}$  or not:
- Trajectories that are entirely in  $F^{*}$  contribute at most  $k^{*}$  states to  $F$ .
- A trajectory that is not contained in  $F^{*}$  contributes at most  $H$  states to  $F$ .
$$
\mathbb {E} [ | F | ] \leq k ^ {*} + \rho \cdot \left(\frac {1}{\beta} \ln \frac {k ^ {*}}{0 . 0 0 5}\right) \cdot H = O \left(k ^ {*} + \frac {\rho H \ln k ^ {*}}{\beta}\right).
$$
We use Markov's inequality to get the desired bound on  $|F|$ .
For the safety, we first denote the set of all states in  $F^{*}$  with probability at least  $\beta$  as  $\Gamma = \{s \in F^{*} \mid p(s) \geq \beta\}$ . We will show that with probability at least 0.9995, it holds that  $\Gamma \subseteq F$ , which will prove our claim, similarly to Lemma 3.1.
For a fixed state  $s \in \Gamma$ , the probability that  $s \notin F$  is bounded by  $(1 - p(s))^{\frac{1}{\beta} \ln \frac{k^*}{0.005}} \leq e^{-\frac{\beta}{\beta} \cdot \ln \frac{k^*}{0.005}} = \frac{0.005}{k^*}$ . Using union bound, the probability that there is a state  $s \in \Gamma$  which is not in  $F$  is bounded by  $k^* \cdot \frac{0.005}{k^*} = 0.005$ .
In other words, with probability at least 0.995,  $\Gamma \subseteq F$ , thus implementing the greedy approach in Algorithm 3 and proving that the probability that a random trajectory escapes  $F$  is bounded by  $\rho + k^{*}\beta$ .
Lemma B.2. For every  $\rho, \gamma \in (0,1)$ ,  $H, k \in \mathbb{N}$ , and  $\beta = \frac{\rho}{k}$ , there is an integer  $r \in \mathbb{N}$  and MDP with  $(\rho, k) - \text{SAFEZONE}$ , but with probability  $\geq 1 - \gamma$ . SIMULATION algorithm returns  $F$  of size  $\mathbb{E}[|F|] \geq kH\ln k$  with escape probability  $\Delta(F) = O(\rho)$ .
Proof. Fix  $\rho, \gamma \in (0,1)$ . Recall that  $m = \frac{1}{\beta} \ln \frac{k^*}{0.005}$  and take  $r = \lceil \frac{m^2}{\gamma} \rceil$ . Define  $A$  to contain  $rH$  states,  $B$  to contain  $k - 1$  states, and  $S = \{s_0\} \cup A \cup B$ .
Consider the following MDP with states  $S$  and starting state  $s_0$ . The transition function is defined as follows:
- For every  $i \in A$ ,  $\operatorname{Pr}[s_{1,i}^{A}|s_{0}] = \frac{\rho}{r}$  and for every  $j \in [H - 1]$ ,  $\operatorname{Pr}[s_{j + 1,i}^{A}|s_{j,i}^{A}] = 1$ .
For  $s\in B$ $\operatorname *{Pr}[s|s_0] = \frac{1 - \rho}{k - 1}$
- For  $s \in B$ ,  $\operatorname{Pr}[s|s] = 1$
The MDP is illustrated in Figure 3.
Figure 3: Lower bound for SIMULATION Algorithm.
The set  $B\cup \{s_0\}$  is  $\rho$  -safe with  $k$  states.
We will show that:
- After adding  $\geq \frac{1}{\beta} \ln k = \frac{k}{\rho} \ln k$  random trajectories, with probability  $\geq 1 - \gamma$  we have that  $|F| \geq kH \ln k$ .
- After adding  $m$  random trajectories, we have that with high probability  $F^{*} \subseteq F$ , thus  $\Delta(F) \leq \Omega(\rho)$ .
To prove the first property, we claim that with probability  $\geq 1 - \gamma$ , every time we add a trajectory  $\tau$  such that  $\tau \cap A \neq \emptyset$ , we add  $H$  new states.
Notice that if we ignore  $s_0$ , trajectories in  $A$  are entirely unconnected, and each trajectory is chosen randomly with probability  $\operatorname*{Pr}[s_{1,i}^A |s_0] = \frac{\rho}{r}$ . This yields that if  $s_{1,i}^A \notin F$ , then  $s_{j,i}^A \notin F$  for every  $j \in [H]$ . As a result, every time we add a new  $s_{1,i}^A$  to  $F$ , we add  $H - 1$  more states to  $F$ . Let  $N$  denote the number of trajectories sampled with states from  $A$ . The probability that their intersection contains only  $s_0$  is
$$
\frac {r \cdot (r - 1) \cdot \ldots \cdot (r - N)}{r ^ {N}} \geq \left(\frac {r - N}{r}\right) ^ {N} = \left(1 - \frac {N}{r}\right) ^ {N} \geq 1 - \frac {N ^ {2}}{r} = 1 - \gamma .
$$
From the structure of the MDP, we have that  $\mathbb{E}[N] = \rho m$ . Therefore, with probability  $\geq 1 - \gamma$
$$
\mathbb {E} [ | F | ] \geq \mathbb {E} [ N ] \cdot H = \rho \cdot m \cdot H \geq \rho \cdot \frac {1}{\beta} \ln k \cdot H = k H \ln k.
$$
The second property follows from Lemma 3.2.
B.3 Greedy at Each Step
Algorithm 5 Greedy at Each Step
Input:  $\rho >0,\{p(s)\}_{s\in S}$ $F\gets \{s_0\}$    
for  $i = 1\dots H$  do Sort states in  $S_{i},p(s_{i}^{1})\geq \ldots \geq p(s_{i}^{|S_{i}|})$ $j^{*}\gets \arg \min_{j\in [|S_i|]}\sum_{r = 1}^{j}p(s_i^r)\geq 1 - \rho$ $F\leftarrow F\cup \left\{s_i^1,\dots s_i^{j^*}\right\}$    
end for   
return  $F$
Proof. Take a random trajectory  $\tau = (s_1, s_2, \ldots)$ . For every  $s_i \in \tau$ , the probability that  $s_i \notin F$  is bounded by  $\rho$ , thus using union bound, the probability that  $\tau$  has state  $s_i$  such that  $s_i \notin F$  is at most  $\rho H$ .
The construction of  $F$  guarantees that  $F$  is the minimal subset of states such that for every  $i$ , the probability that  $s_i$  is in the subset is at least  $1 - \rho$ . Assume by contradiction that  $|F| > k^*$ . Then there is a time step  $i$  such that  $\operatorname{Pr}[s_i \in F^*] < 1 - \rho$ , which is a contradiction, since  $\operatorname{Pr}[\tau \in F^*] \leq \min_i \operatorname{Pr}[s_i \in F^*]$ .
Lemma B.3. For any  $\rho \in (0,1)$ , there is an MDP and an integer  $k$  such that there is a  $(\rho ,k)$ -SAFEZONE, but GreEDy AT EACH STEP Algorithm returns  $F$  with escape probability  $\Delta (F)\geq \Omega (H\rho)$ .
Proof. Fix  $\rho \in (0,1)$  and take  $k = 3H + 1$ .
Consider the MDP illustrated in Figure 4. The set  $\{s_0\} \cup \{s_1^i\}_i \cup \{s_2^i\}_i \cup \{s_3^i\}_i$  form a  $(\rho, 3H + 1)$ -SAFEZONE.
We will prove by induction that the for every time  $i$ ,
$p(s_1^i) = 1 - 2\rho$
$p(s_2^i) = p(s_3^i) = p(s_4^i) = \frac{\rho}{2}$ , and
- For every  $j \in  \{ 5,\ldots ,k + 4\} ,p\left( {s}_{j}^{i}\right)  = \frac{\rho }{2k}$  .
Figure 4: Lower bound for GreEDy AT EACH STEP Algorithm.
It is easy to see that the two properties hold for  $i = 1$ .
For  $i > 1$
$$
p \left(s _ {1} ^ {i}\right) = p \left(s _ {1} ^ {i - 1}\right) (1 - \rho) + p \left(s _ {2} ^ {i - 1}\right) \frac {\rho}{2} + p \left(s _ {3} ^ {i - 1}\right) \frac {\rho}{2} = (1 - 2 \rho) (1 - \rho) + 2 (1 - 2 \rho) \frac {\rho}{2} = 1 - 2 \rho
$$
$$
p (s _ {2} ^ {i}) = p _ {i - 1} (s _ {1} ^ {i - 1}) \frac {\rho}{2} + p (s _ {2} ^ {i - 1}) \rho + p (s _ {3} ^ {i - 1}) \rho = (1 - 2 \rho) \frac {\rho}{2} + \frac {\rho^ {2}}{2} + \frac {\rho^ {2}}{2} = \frac {\rho}{2}
$$
Similarly,  $p(s_3^i) = \frac{\rho}{2}$ .
$$
p (s _ {4} ^ {i}) = \frac {1}{2} p (s _ {4} ^ {i - 1}) + \sum_ {j = 5} ^ {k + 4} \frac {p (s _ {j} ^ {i - 1})}{2} = \frac {\rho}{4} + k \frac {\rho}{4 k} = \frac {\rho}{2}
$$
For every  $j\in \{5,\ldots ,k + 4\}$
$$
p (s _ {j} ^ {i}) = \frac {1}{2 k} p (s _ {4} ^ {i - 1}) + \sum_ {m = 5} ^ {k + 4} \frac {p (s _ {m} ^ {i - 1})}{2 k} = \frac {\rho}{4 k} + k \frac {\rho}{4 k ^ {2}} = \frac {\rho}{2 k}.
$$
The algorithm might return  $\{s_0\} \cup \{s_1^i\}_i \cup \{s_2^i\}_i \cup \{s_4^i\}_i$ , i.e., instead of taking  $\cup_i\{s_3^i\}_i$  it takes  $\cup_i\{s_4^i\}_i$ . Finally, the observation  $\Delta(\{s_0\} \cup \{s_1^i\}_i \cup \{s_2^i\}_i \cup \{s_4^i\}_i) \geq \frac{\rho H}{4}$  completes the proof.
C Proofs of Section 4
For convenience, we state here Hoeffding's inequality.
Lemma C.1. [Hoeffding's Inequality] Let  $y_{1}, \ldots, y_{N}$  be independent random variables such that  $y_{i} \in [a, b]$  for every  $y_{i}$  with probability 1. Then, for any  $\epsilon > 0$ ,
$$
\Pr \left[ \left| \frac {1}{N} \sum_ {i = 1} ^ {N} y _ {i} - \mathbb {E} [ y _ {i} ] \right| \geq \epsilon \right] \leq 2 e ^ {- 2 N \epsilon^ {2} / (b - a) ^ {2}}.
$$
In this section, we provide a complete proof for Theorem 4.1. Throughout the section, we define a few terms and notions. We will start with proving guarantees regarding a single iteration of the while-loop.
Recall that  $F^{*}$  denotes a minimal  $\rho$ -safe set (of size  $k^{*}$ ). If there are multiple optimal solutions, choose one arbitrarily. For the convince of analysis, we denote the values of the algorithm variables at the end of each iteration  $i$  of the while-loop by  $\tau_{i}, F_{i}, \text{accept}_{i}$ . Let  $j(i)$  denote the value of variable  $j$  during the  $i$ -th call to EstimateSafety Subroutine. In addition, let  $N_{i}$  denote the number of trajectories sampled for the  $j$ -th time of calling Subroutine EstimateSafety, i.e.,  $N_{i} = \frac{1}{2\epsilon^{2}} \ln \frac{2}{\lambda_{j(i)}}$  for  $j(i) \leq i$ .
For ease of presentation, we recall some of the definitions from the proof technique description. We say that a trajectory  $\tau$  is good if all the states in  $\tau$  are in  $F^{*}$  and bad if it escapes it. I.e., a trajectory is good if  $\tau \subseteq F^{*}$  and bad if  $\tau \not\subseteq F^{*}$ . Additionally, we say that a state  $s \in S$  is good if it is in  $F^{*}$  and bad otherwise. Namely, a state  $s$  is good if  $s \in F^{*}$  and bad if  $s \notin F^{*}$ . Let  $G_{i}(F_{i-1})$  and  $B_{i}(F_{i-1})$  be the number of good and bad states added to  $F_{i-1}$  in iteration  $i$ , respectively (notice that  $G_{i}(F_{i-1})$  and  $B_{i}(F_{i-1})$  are random variables that depends on  $F_{i-1}$ ). For short, whenever it is clear from the context, we write  $G_{i}$  and  $B_{i}$  respectively.
The following lemma bounds the error in approximating the escape probability.
Lemma C.2. Let  $F_{i-1} \subseteq S$  be a subset of states and  $\epsilon, \lambda_j > 0$  be some parameters. Let  $S_i$  be a sample of  $N_i \geq \frac{1}{2\epsilon^2} \ln \frac{2}{\lambda_{j(i)}}$  i.i.d. random trajectories. Then,
$$
\Pr_ {S _ {i}} \left[ \left| \widehat {\Delta} (F _ {i - 1}) - \Delta (F _ {i - 1}) \right| \geq \epsilon \right] \leq \lambda_ {j}.
$$
Also, as  $\lambda_{j} = \frac{3\lambda}{2(\pi j)^{2}}$
$$
\Pr \left[ \exists i \left| \widehat {\Delta} (F _ {i - 1}) - \Delta (F _ {i - 1}) \right| \geq \epsilon \right] \leq \lambda / 4,
$$
Where the last probability is over all the samples  $S_{i}$  made by EstimateSafety Subroutine.
Proof. The first part follows directly from Hoeffding's inequality by taking  $y_{i} = \mathbb{I}[\tau \nsubseteq F]$ .
Assigning  $\lambda_{j} = \frac{3\lambda}{2(\pi j)^{2}}$  and applying union bound, we get
$$
\begin{array}{l} \Pr \left[ \exists i \left| \widehat {\Delta} (F _ {i - 1}) - \Delta (F _ {i - 1}) \right| \geq \epsilon \right] \leq \sum_ {i} \Pr_ {S _ {i}} \left[ \left| \widehat {\Delta} (F _ {i - 1}) - \Delta (F _ {i - 1}) \right| \geq \epsilon \right] \\ \le_ {(*)} \sum_ {j (i)} \lambda_ {j (i)} \le \sum_ {j = 1} ^ {\infty} \lambda_ {j} = \sum_ {j = 1} ^ {\infty} \frac {3 \lambda}{2 (\pi j) ^ {2}} = \frac {\lambda}{4}. \\ \end{array}
$$
The inequality marked by  $(\ast)$  follows from the fact that  $\Delta (F)$  is estimated once for every time  $j$  increases.
We define the event that EstimateSafety always provides good estimations by
$$
\mathcal {E} = \{\forall i \left| \widehat {\Delta} (F _ {i - 1}) - \Delta (F _ {i - 1}) \right| \leq \epsilon \}.
$$
By the above, we have that  $\operatorname{Pr}[\mathcal{E}] \geq 1 - \lambda / 4$ .
In the following lemma we assume that if the current escape probability is at least  $2\rho$ , then the fraction of bad trajectories that escape  $F_{i-1}$  is bounded from above by the fraction of good trajectories that escape  $F_{i-1}$ .
Lemma C.3. Let  $\rho > 0$  and assume that  $\Delta(F_{i-1}) \geq 2\rho$ . Then,
$$
\operatorname * {P r} _ {\tau} \left[ n e w _ {F _ {i - 1}} (\tau) \neq 0 \wedge \tau \nsubseteq F ^ {*} \right] \leq \operatorname * {P r} _ {\tau} \left[ n e w _ {F _ {i - 1}} (\tau) \neq 0 \wedge \tau \subseteq F ^ {*} \right],
$$
where the probabilities are over random trajectories.
Proof. To prove the lemma, we will bound the probability  $\operatorname{Pr}_{\tau}[new_{F_{i-1}}(\tau) \neq 0 \land \tau \not\subseteq F^*]$  from above and the probability  $\operatorname{Pr}_{\tau}[new_{F_{i-1}}(\tau) \neq 0 \land \tau \subseteq F^*]$  from below. Since  $\Delta(F^*) \leq \rho$ ,
$$
\Pr_ {\tau} \left[ n e w _ {F _ {i - 1}} (\tau) \neq 0 \wedge \tau \nsubseteq F ^ {*} \right] \leq \Pr_ {\tau} \left[ \tau \nsubseteq F ^ {*} \right] \leq \rho . \tag {4}
$$
The assumption  $\Delta (F_{i - 1})\geq 2\rho$  implies that
$$
\begin{array}{l} 2 \rho \leq \Delta (F _ {i - 1}) = \Pr_ {\tau} [ n e w _ {F _ {i - 1}} (\tau) \neq 0 ] = \Pr_ {\tau} [ n e w _ {F _ {i - 1}} (\tau) \neq 0 \wedge \tau \subseteq F ^ {*} ] + \Pr_ {\tau} [ n e w _ {F _ {i - 1}} (\tau) \neq 0 \wedge \tau \nsubseteq F ^ {*} ] \\ \leq \operatorname * {P r} _ {\tau} [ n e w _ {F _ {i - 1}} (\tau) \neq 0 \land \tau \subseteq F ^ {*} ] + \operatorname * {P r} _ {\tau} [ \tau \nsubseteq F ^ {*} ] \leq \operatorname * {P r} _ {\tau} [ n e w _ {F _ {i - 1}} (\tau) \neq 0 \land \tau \subseteq F ^ {*} ] + \rho , \\ \end{array}
$$
$$
\rho \leq \Pr_ {\tau} [ \text {n e w} _ {F _ {i - 1}} (\tau) \neq 0 \wedge \tau \subseteq F ^ {*} ]. \tag {5}
$$
Putting (4) and (5) together yields the statement.
Now, as long as the algorithm is inside the while-loop (i.e., the escape probability holds  $\widehat{\Delta}(F) > 2\rho + \epsilon$ ), it follows that  $\Delta(F) \geq 2\rho$  with high probability from Lemma C.2. Combining it with Lemma C.3 would yield that with high probability over a random trajectory, if the trajectory escapes  $F$  then in expectation, it is at least as likely to be good as it is to be bad.
We move on to show the main ingredient of the proof, namely that for any iteration, with high probability, the expected number of good states added to the current set  $F$  is larger or equal to the expected number of bad states.
For every iteration  $i$  in which we sample  $\tau_{i}$  both  $G_{i}$  and  $B_{i}$  depends on the following:
1. The realizations of the sampled trajectory,  $\tau_{i}$ , and in particular on  $new_{F_{i-1}}(\tau_i)$ .
2. The probability of adding it to  $F$ , i.e.,  $1 / new_{F_i - 1}(\tau_i)$ .
Next, we prove Equation (3).
Lemma C.4. Assume event  $\mathcal{E}$  holds. Thus, for all iterations  $i$  inside the while-loop we have
$$
\mathbb {E} [ B _ {i} | F _ {i - 1} ] \leq \mathbb {E} [ G _ {i} | F _ {i - 1} ],
$$
where the expectation is over the trajectory  $\tau$  that is sampled from the MC dynamics and added to  $F_{i - 1}$  according to  $Q_{F_{i - 1}}$ .
Proof. Since event  $\mathcal{E}$  holds, we have that  $\Delta(F_{i-1}) \geq 2\rho$  as long as we do not terminate in iteration  $i$ .
We can use it to bound  $\mathbb{E}_{\tau}[B_i|F_{i - 1}]$  by
$$
\begin{array}{l} \mathbb {E} _ {\tau} \left[ B _ {i} \mid F _ {i - 1} \right] \leq \sum_ {h = 1} ^ {H} \frac {\Pr_ {\tau} \left[ n e w _ {F _ {i - 1}} (\tau) = h \wedge \tau \not \subseteq F ^ {*} \right]}{h} \cdot h \\ = \Pr_ {\tau} [ n e w _ {F _ {i - 1}} (\tau) \neq 0 \wedge \tau \not \subseteq F ^ {*} ] \underset {L e m m a C. 3} {\leq} \Pr_ {\tau} [ n e w _ {F _ {i - 1}} (\tau) \neq 0 \wedge \tau \subseteq F ^ {*} ] \\ = \sum_ {h = 1} ^ {H} \frac {\Pr_ {\tau} [ n e w _ {F _ {i - 1}} (\tau) = h \wedge \tau \subseteq F ^ {*} ]}{h} \cdot h \leq \mathbb {E} _ {\tau} [ G _ {i} | F _ {i - 1} ]. \\ \end{array}
$$
Proof. Assume that the event  $\mathcal{E}$  holds, and recall that
$$
\Pr [ \mathcal {E} ] \geq 1 - \lambda / 4. \tag {6}
$$
We start with the first clause. Since the event  $\mathcal{E}$  holds, Lemma C.2 in particular implies that  $\Delta(F) \leq 2\rho + 2\epsilon$ , hence the first clause holds. For second clause, we will bound  $\mathbb{E}[|F| \mid \mathcal{E}]$  from above by  $2k^*$ . Since  $\mathcal{E}$  holds, we have that  $\Delta(F_{i-1}) \geq 2\rho$ , for every  $i$  inside the while-loop, thus Lemma C.4 yields
$$
\mathbb {E} \left[ B _ {i} \mid F _ {i - 1} \right] \leq \mathbb {E} \left[ G _ {i} \mid F _ {i - 1} \right].
$$
$$
\mathbb {E} [ | F | \mid \mathcal {E} ] \leq 2 \sum_ {i} \mathbb {E} _ {F _ {i - 1}} [ \mathbb {E} [ G _ {i} | F _ {i - 1} ] ] | \mathcal {E} ] \leq 2 k ^ {*}, \tag {7}
$$
where the last inequality follows from the definition of  $G_{i}$ , as  $\sum_{i}G_{i}\leq |F^{*}| = k^{*}$
We continue with the third clause of the theorem. Let  $M$  denote the sample complexity of the algorithm, namely  $M = M_{F} + M_{E}$  where  $M_{F}$  is the expected total number of trajectories sampled within the FINDING SAFEZONE Algorithm (without the samples made by EstimateSafety Subroutine) and  $M_{E}$  is the total number of trajectories sampled using EstimateSafety. We will bound each term separately.
Since  $\mathcal{E}$  holds, whenever we are inside the while-loop,  $\Delta (F_i)\geq 2\rho$ , which implies that it takes at most  $1 / 2\rho$  trajectories in expectation to sample a trajectory that escapes  $F_{i}$ , and such trajectory is accepted with probability at least  $1 / H$ .
Thus, from Wald's identity, it follows that
$$
\mathbb {E} \left[ M _ {F} \mid \mathcal {E} \right] = \frac {H}{2 \rho} \cdot \mathbb {E} [ | F | \mid \mathcal {E} ] \leq \frac {H k ^ {*}}{\rho}.
$$
From Markov's inequality on the above inequality, with probability at least  $1 - \frac{\lambda}{4}$ ,
$$
\Pr \left[ M _ {F} \geq \frac {4 H k ^ {*}}{\rho \lambda} \mid \mathcal {E} \right] \leq \frac {\lambda}{4}. \tag {8}
$$
Moving on to bound  $M_E$ . Since  $\mathcal{E}$  holds, it follows from Equation (7) and Markov's inequality that
$$
\Pr \left[ | F | \geq \frac {8 k ^ {*}}{\lambda} \mid \mathcal {E} \right] = \Pr \left[ | F | \geq 2 k ^ {*} \cdot \frac {4}{\lambda} \mid \mathcal {E} \right] = \Pr \left[ | F | \geq \mathbb {E} [ | F | \mid \mathcal {E} ] \cdot \frac {4}{\lambda} \mid \mathcal {E} \right] \leq \frac {\lambda}{4}. \tag {9}
$$
If  $|F| \leq \frac{8k^*}{\lambda}$ , the number of calls for Subroutine EstimateSafety is also bounded by  $8\pi k^* / \lambda$  (we only call EstimateSafety after we added states to  $F$ ). It also implies that  $\frac{3\lambda^3}{2(8\pi k^*)^2} \leq \lambda_j$  for every  $j \geq 1$ . Thus, if  $|F| \leq \frac{8k^*}{\lambda}$ ,
$$
\begin{array}{l} M _ {E} = \sum_ {j = 1} ^ {| F |} N _ {i} \leq \sum_ {j} ^ {\frac {8 k ^ {*}}{\lambda}} \frac {1}{2 \epsilon^ {2}} \ln \frac {2}{\lambda_ {j}} \leq \sum_ {j} ^ {\frac {8 k ^ {*}}{\lambda}} \frac {1}{2 \epsilon^ {2}} \ln \frac {2}{\frac {3 \lambda^ {3}}{2 (8 \pi k ^ {*}) ^ {2}}} \leq \sum_ {j} ^ {\frac {8 k ^ {*}}{\lambda}} \frac {1}{2 \epsilon^ {2}} \ln \frac {8 6 (\pi k ^ {*}) ^ {2}}{\lambda^ {3}} \\ = \frac {8 k ^ {*}}{2 \lambda \epsilon^ {2}} \ln \frac {8 6 (\pi k ^ {*}) ^ {2}}{\lambda^ {3}} = \frac {4 k ^ {*}}{\lambda \epsilon^ {2}} \ln \frac {8 6 (\pi k ^ {*}) ^ {2}}{\lambda^ {3}} \\ \end{array}
$$
Combining the above with Equation (9), we get
$$
\Pr \left[ M _ {E} > \frac {4 k ^ {*}}{\lambda \epsilon^ {2}} \ln \frac {8 6 (\pi k ^ {*}) ^ {2}}{\lambda^ {3}} \mid \mathcal {E} \right] \leq \frac {\lambda}{4} \tag {10}
$$
As  $M = M_F + M_E$ , union bound over Equation (6), Equation (8) and Equation (10) implies that with probability  $\geq 1 - 3\lambda /4 > 1 - \lambda$
$$
M = O \left(\frac {k ^ {*}}{\lambda \epsilon^ {2}} \ln \frac {k ^ {*}}{\lambda} + \frac {H k ^ {*}}{\rho \lambda}\right) \tag {11}
$$
For each trajectory we sample we run in time  $O(H)$ , e.g., by using a lookup table for maintaining the current set  $F$ . Consequently, if the event in Equation (11) holds then the running time of the algorithm is bounded by