SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
O \left(\frac {H k ^ {*}}{\lambda \epsilon^ {2}} \ln \frac {k ^ {*}}{\lambda} + \frac {H ^ {2} k ^ {*}}{\rho \lambda}\right).
$$
Overall, all the clauses in the lemma hold with probability  $\geq 1 - \lambda$ .
Proof. Assume we run FINDING SAFEZONE Algorithm for  $m = \frac{2\ln 300}{\delta}$  times and denote each algorithm output by  $F^i$ . Return the smallest set  $F = \operatorname*{argmin}_{F^i}|F^i|$ .
It follows from Theorem 4.1 that for every  $\lambda \in (0,1)$ , each  $F^i$  is of expected size  $\mathbb{E}[|F^i|] \leq 2k^*$ , and is  $(2\rho + 2\epsilon)$ -safe with probability  $\geq 1 - \lambda$ . Choosing  $\lambda = \frac{0.01}{3m}$  implies
$$
\Pr [ \Delta (F) > 2 \rho + 2 \epsilon ] \leq \frac {0 . 0 1}{3}. \tag {12}
$$
In addition, from Markov's inequality it follows that for every  $\delta > 0$ ,
$$
\begin{array}{l} \Pr \left[ | F ^ {i} | > (2 + \delta) k ^ {*} \right] \leq \Pr \left[ | F ^ {i} | > (2 + \delta) k ^ {*} | \mathcal {E} \right] + \Pr [ \mathcal {E} ] \\ \leq \frac {2 k ^ {*}}{(2 + \delta) k ^ {*}} + \lambda \\ = 1 - \frac {\delta / 2}{1 + \delta / 2} + \lambda \\ = 1 - \frac {\delta / 2 - \lambda - \lambda \delta / 2}{1 + \delta / 2} \\ \end{array}
$$
From the independence of the algorithm runs, for  $m = \frac{2\ln 300}{\delta}$ ,
$$
\begin{array}{l} \Pr \left[ | F | > (2 + \delta) k ^ {*} \right] \leq \Pr \left[ \forall i: \left(\left| F ^ {i} \right| > (2 + \delta) k ^ {*}\right) \right] \\ \leq \prod_ {i \in [ m ]} \Pr \left[ \left| F ^ {i} \right| > (2 + \delta) k ^ {*} \right] \\ \leq \left(1 - \frac {\delta / 2 - \lambda - \lambda \delta / 2}{1 + \delta / 2}\right) ^ {m} \\ \leq e ^ {- m \left(\frac {\delta / 2 - \lambda - \lambda \delta / 2}{1 + \delta / 2}\right)} \leq \frac {0 . 0 1}{3}. \\ \end{array}
$$
$$
\Pr [ | F | > (2 + \delta) k ^ {*} ] \leq \frac {0 . 0 1}{3}. \tag {13}
$$
As for the sample complexity, let  $M_{i}$  denote the (random) sample complexity of the  $i$ -th run, and let us denote
$$
\bar {M} = \frac {4 k ^ {*}}{\lambda \epsilon^ {2}} \ln \frac {8 6 (\pi k ^ {*}) ^ {2}}{\lambda^ {3}} + \frac {4 H k ^ {*}}{\rho \lambda}.
$$
From Theorem 4.1,  $M_{i} > \bar{M}$  with probability  $< \lambda$ .
By taking the union bound on the sample complexity bound per one run, we get
$$
\Pr \left[ \exists i: M _ {i} > \bar {M} \right] \leq \sum_ {i \in [ m ]} \Pr \left[ M _ {i} > \bar {M} \right] \leq m \cdot \lambda = \frac {0 . 0 1}{3}.
$$
Where the last inequality follows from Theorem 4.1, and  $\lambda = \frac{0.01}{3m}$ .
Assigning  $m = \frac{2\ln 300}{\delta}$  and  $\lambda = \frac{0.01}{3m} = \frac{0.01\delta}{6\ln 300}$ , we get that with probability  $\geq 1 - \frac{0.01}{3}$ ,
$$
\sum_ {i = 1} ^ {m} M _ {i} = O \left(\frac {m k ^ {*}}{\lambda \epsilon^ {2}} \ln \frac {k ^ {*}}{\lambda} + \frac {m H k ^ {*}}{\rho \lambda}\right) = O \left(\frac {k ^ {*}}{\delta^ {2} \epsilon^ {2}} \ln \frac {k ^ {*}}{\delta} + \frac {H k ^ {*}}{\rho \delta^ {2}}\right) \tag {14}
$$
Since the algorithm runs in time  $O(H)$  for every trajectory sampled, if the sample complexity is bounded by the above term, then the total running time is bounded by  $O\left(\frac{Hk^*}{\delta^2\epsilon^2}\ln \frac{Hk^*}{\delta} +\frac{Hk^*}{\rho\delta^2}\right)$ .
Finally, from union bound over Equation (12), Equation (13) and Equation (14) all the theorem properties hold with probability  $\geq 0.99$ .
D Additional Figures for Section 5
D.1 Comparing SAFEZONE of two policies
In this section, we empirically explore the SAFEZONE of two different policies within the same MDP. The first policy, described in the previous section, first goes right and then to the middle, and the second policy first goes to the middle and then goes right. See Figure 6 in the appendix. These seemingly similar policies induce very different SAFEZONES as can be seen in Figure 8 which depicts the number of visits in each state. It shows that the second policy requires fewer states to achieve the same level of safety, even though in terms of minimizing the number of steps to get to the goal state it is outperformed by the first policy (intuitively, the second policy has more fail attempts to go up in expectation since the lowest row of the grid cannot get worst). In Figure 7 we see that already with  $14\%$  of the states, all three algorithms achieve trajectory coverage of more than  $85\%$ .
(a) Set chosen by GreEDy AT EACH STEP Algorithm.
(b) Set chosen by SAFEZONE Algorithm.
Figure 5: Empirical results regarding Coverage of the different algorithms, FINDING SAFEZONES and state visit frequency.
Figure 6 depicts the two policies discussed in the paper when  $N = 7$ .
Figure 7 depicts coverage percentage for the different algorithms discussed in the paper when applied to the second policy.
(a) %Coverage: difference from GreEDy Algorithm.
(b)  $\%$  Coverage: absolute values.
(c) Total number of visits at each state from 2000 episodes. Zero visits in grey.
Figure 6(c) depicts the number of total visits at each state using the described policy.
Figure 8 shows the visits of the policies described in the main paper for  $N = 30$ . It is immediately clear that the SAFEZONE of the two policies are fundamentally different. As mentioned, this affects their SAFEZONE sizes. Namely, when trying to go right from a current state in the lowest row it is impossible to get to a square that is lower than that, and the first policy takes advantage of this. In contrast, the second policy keeps trying to go up from the lowest row, which implies that in expectation it goes down more times compared to the first.
E Exact Computation
In this section, we assume that the transition function is known to the algorithm and show how to compute  $\Delta(F)$ .
(d) Go right and then to the goal state.
(e) Go to the middle and then right.
Figure 6: Two policies for the same MDP with  $N = 7$ . Starting state,  $s_0$ , in blue, and the goal state in red.
Given a Markov Chain  $\langle S, P, s_0 \rangle$  and a set  $F \subseteq S$  we create a new Markov Chain  $\langle S', P', s_0 \rangle$  as follows. We add a new state  $s_{sink} \notin S$ , and set  $S' = F \cup \{s_{sink}\}$ . For each transition from a state  $s \in F$  to a state  $s' \notin F$  we modify and make the transition in  $P'$  to the sink  $s_{sink}$ . In  $P'$ , when we are in  $s_{sink}$  we always stay in  $s_{sink}$ . More formally: (1) if  $s, s' \in F$  then  $P'(s'|s) = P(s'|s)$ , (2) we set  $P'(s_{sink}|s) = \sum_{s' \notin F} P(s'|s)$  and (3)  $P'(s_{sink}|s_{sink}) = 1$  and  $P'(s|s_{sink}) = 0$  for  $s \neq s_{sink}$ .
Figure 7: SAFEZONE coverage for the second policy.
Now we claim that  $\Delta (F) = \operatorname *{Pr}_{P^{\prime}}[s_H = s_{sink}]$ , since any trajectory that reaches a state not in  $F$  will reach the sink in  $P^{\prime}$  and stay there. We can compute  $\operatorname *{Pr}_{P^{\prime}}[s_H = s_{sink}]$  using standard dynamics programming.
The running time of constructing  $\langle S', P', s_0 \rangle$  is  $O(|S|^2)$ . Computing the probability of  $\operatorname{Pr}_{P'}[s_H = s_{sink}]$  takes  $O(H|S|^2)$ . Therefore we have established the following.
Lemma E.1. Given a Markov chain  $\langle S, P, s_0 \rangle$  and a set  $F \subseteq S$  we can compute  $\Delta(F)$  in time  $O(|S|^2 H)$ .
Note that the above lemma implements an exact version of the EstimateSafety Subroutine.
(a) Number of visits at each state for policy "Go right and then to the middle"
(b) Number of visits at each state for policy "Go to the middle and then right"
Figure 8: Total number of visits for the two policies.