SlowGuess/NeurIPS_2023_OCR huggingface dataset:
While favoring trajectories with higher probabilities, which we already get by the sampling process, another key idea would guide this new distribution: To prefer trajectories that gradually increase the size of  $F$ . To implement this idea, we will ensure that the probability of adding a trajectory  $\tau$  to  $F$  should be inversely proportional to  $new_{F}(\tau)$ .
Formally, the support of  $Q_{F}$  is the trajectories with new states, i.e.,  $X = \{\tau | \text{new}_F(\tau) \neq 0\}$ . For every  $\tau \in X$
$$
Q _ {F} (\tau) \propto \frac {\operatorname * {P r} [ \tau ]}{\operatorname * {n e w} _ {F} (\tau)},
$$
where  $\operatorname{Pr}[\tau]$  is the probability of trajectory  $\tau$  under the Markov Chain with dynamics  $P$ . Note that the new distribution depends on the current set  $F$ , and changes as we modify it. Intuitively, adding trajectories to  $F$  according to  $Q_{F}$  instead of adding trajectories sampled directly from the dynamics (as we do in SIMULATION) would increase the expected ratio between the added safety and the number of new states we add to  $F$ , thus improving the set size guarantee of the output set. We elaborate on this in Section 4.2.
Our main algorithm is FINDING SAFEZONE, Algorithm 1. The algorithm receives, in addition to the safety parameter  $\rho$ , parameters  $\epsilon$ ,  $\lambda \in (0,1)$ , and maintains a set  $F$  that is initiated to  $\{s_0\}$ . On a high level, to implement the idea of adding trajectories to  $F$  according to  $Q_F$ , we use rejection sampling. Namely, in each iteration of the while-loop we first sample a trajectory  $\tau$ , and if  $new_{F}(\tau) \neq 0$ , we accept it with probability  $1 / new_{F}(\tau)$ . If the trajectory is accepted, it is added to  $F$ . More precisely, if  $new_{F}(\tau) \neq 0$ , we sample a Bernoulli random variable, accept  $\sim Br(1 / new_{F}(\tau))$ . If accept = 1, we add  $\tau$  to  $F$ . This process of adding trajectories to  $F$  generates the desired distribution,  $Q_F$ . Whenever a trajectory is added to  $F$ , we estimate the escape probability  $\Delta(F)$  (w.r.t. the updated set,  $F$ ).
The algorithm stops adding states to  $F$  and returns it as output when it becomes "safe enough". To be precise, let  $\widehat{\Delta}(F)$  denote the result of the escape probability estimation (by sampling trajectories as suggested in Proposition 2.2). If  $\widehat{\Delta}(F) \leq 2\rho + \epsilon$ , it means that  $F$  is  $(2\rho + 2\epsilon)$ -safe with probability  $\geq 1 - \lambda_j > 1 - \lambda$ , in which case the algorithm terminates and returns  $F$  as output.
To implement the estimation  $\widehat{\Delta}(F)$ , the algorithm calls EstimateSafety Subroutine. The subroutine samples  $N_{j} = \Theta\left(\frac{1}{\epsilon^{2}} \ln \frac{2}{\lambda_{j}}\right)$  trajectories, and returns the fraction of trajectories that escaped  $F$ .
For cases in which the transition function  $P$  is known to the learner, we provide an alternative implementation for EstimateSafety which computes the exact probability  $\Delta(F)$  (see Lemma E.1 in Appendix E).
Algorithm 1 FINDING SAFEZONE  
Input:  $\rho \in (0,1)$   
Parameters:  $\epsilon, \lambda \in (0,1)$ $F \gets \{s_0\}, j \gets 1, \widehat{\Delta}(F) \gets 1$   
while  $\widehat{\Delta}(F) > 2\rho + \epsilon$  do  
 $\tau \gets$  sample a random trajectory  
Compute  $new_F(\tau)$   
if  $new_F(\tau) \neq 0$  then  
sample accept  $\sim Br(1/new_F(\tau))$   
if  $accept = 1$  then  
 $F \gets F \cup \tau$ $\lambda_j \gets \frac{3\lambda}{2(j\pi)^2}, j \gets j + 1$ $\widehat{\Delta}(F) \gets EstSafety(\epsilon, \lambda_j, F)$   
end if  
end if  
end while  
return  $F$
Algorithm 2 EstSafety Subroutine  
Input: subset  $F$   
Parameters:  $\epsilon, \lambda_j \in (0,1)$ $\widehat{\Delta}(F) \gets 0$ $\mathcal{T} \gets \text{sample } N_j = \frac{1}{2\epsilon^2} \ln \frac{2}{\lambda_j}$  trajectories  
for  $\tau \in \mathcal{T}$  do  
if  $\tau \not\subseteq F$  then  
 $\widehat{\Delta}(F) \gets \widehat{\Delta}(F) + \frac{1}{N_j}$   
end if  
end for  
return  $\widehat{\Delta}(F)$
4.1 Algorithm Analysis
We define the event  $\mathcal{E} = \{\forall i |\widehat{\Delta}(F_{i-1}) - \Delta(F_{i-1})| \leq \epsilon\}$ , which states that all our EstimateSafety Subroutine estimations are accurate. We show that  $\mathcal{E}$  holds with high probability using Hoeffding's inequality. In most of the analysis, we condition on  $\mathcal{E}$  to hold.
The following theorem is the central component in the proof of the main theorem that follows it.
Theorem 4.1. Given  $\rho, \epsilon, \lambda \in (0,1)$ , FINDING SAFEZONE Algorithm returns a subset  $F \subseteq S$  such that:
1. The escape probability is bounded from above by  $\Delta (F)\leq 2\rho +2\epsilon$  , with probability  $1 - \lambda$
2. The expected size of  $F$  given  $\mathcal{E}$  is bounded by  $\mathbb{E}[|F|\mid \mathcal{E}]\leq 2k^{*}$
3. The sample complexity of the algorithm is bounded by  $O\left(\frac{k^*}{\lambda \epsilon^2} \ln \frac{k^*}{\lambda} + \frac{Hk^*}{\rho \lambda}\right)$ , and the running time is bounded by  $O\left(\frac{Hk^*}{\lambda \epsilon^2} \ln \frac{k^*}{\lambda} + \frac{H^2k^*}{\rho \lambda}\right)$ , with probability  $1 - \lambda$ .
To obtain the main theorem, we run FINDING SAFEZONE Algorithm several times and return the smallest output set,  $F$ , see the next section for more details.
Theorem 4.2. (main theorem) Given  $\epsilon, \rho, \delta > 0$ , if we run FINDING SAFEZONE for  $\Theta\left(\frac{1}{\delta}\right)$  times and return the smallest output set,  $F \subseteq S$ , then with probability  $\geq 0.99$
1. The escape probability is bounded by  $\Delta (F)\leq 2\rho +2\epsilon$
2. The size of  $F$  is bounded from above by  $|F|\leq (2 + \delta)k^{*}$
3. The total sample complexity and running time are bounded by  $O\left(\frac{k^{*}}{\delta^{2}\epsilon^{2}} \ln \frac{k^{*}}{\delta} + \frac{Hk^{*}}{\rho\delta^{2}}\right)$ , and  $O\left(\frac{Hk^{*}}{\delta^{2}\epsilon^{2}} \ln \frac{k^{*}}{\delta} + \frac{H^{2}k^{*}}{\rho\delta^{2}}\right)$ , respectively.
Finding an almost  $2\rho$ -safe SAFEZONE, nearly  $2k^{*}$  in size can be valuable. For example, it enhances the safety of popular commuting routes, promotes trust in autonomous vehicles, and aligns with potential regulatory restrictions, making most commutes driverless within this secure zone. Regarding sample complexity, we expect the dependency on  $k^{*}$  to be nearly optimal. The reason why is the following. Consider an induced MC with  $k^{*} - 1$  trajectories of size 2, each starting from an initial
state  $s^0$ , ends with a unique corresponding state  $1, \ldots, k^* - 1$ , and has a probability of  $\frac{1 - \rho}{k^* - 1}$ . If the MC has significantly more than  $k^*$  states with non-zero probability, it would take at least  $K^* - 1$  samples to find a  $(\rho, k^*)$ -SAFEZONE. As for the parameter  $\delta$ , this could be treated as a constant. For example, selecting  $\delta = 1/3$  yields that we need to run the algorithm  $6 \cdot \ln 300$  times and a solution of size  $7/3k^*$  w.h.p.
4.2 Proof Technique
Escape Probability Set Size Bounds. To ease the presentation of the proof, we assume that  $\widehat{\Delta}(F) = \Delta(F)$ . For full proofs, we refer to Appendix C. This case is interesting on its own since if the policy and transition function are known, we can compute  $\Delta(F)$  efficiently using dynamic programming (see Appendix E). As a result, event  $\mathcal{E}$  always holds. In addition, it is clear that the termination of the algorithm implies that  $\widehat{\Delta}(F) = \Delta(F) \leq 2\rho$ , thus  $F$  is  $(2\rho + 2\epsilon)$ -safe. The main challenge is bounding  $|F|$ .
A few notations before we start. Let  $F^{*}$  denote a minimal  $\rho$ -safe set (of size  $k^{*}$ ). Consider iteration  $i$  inside the while-loop. The random variable  $G_{i}(F)$  is the number of states in  $F^{*}$  that are added to  $F$  in iteration  $i$  and  $B_{i}(F)$  is the number of states added to  $F$  in iteration  $i$  that are not in  $F^{*}$  ( $G$  stands for good and  $B$  for bad). For ease of presentation, from here on we write  $G_{i}$  and  $B_{i}$  instead of  $G_{i}(F)$  and  $B_{i}(F)$ , respectively. Notice that the size of the output set is exactly  $\sum_{i} B_{i} + G_{i}$  and that  $\sum_{i} G_{i} \leq k^{*}$ .
The main idea of the proof technique is to show that by adding trajectories according to the new distribution  $Q_{F}$ , we ensure that, in expectation, there are at least as many good states that are added to  $F$  as bad states. Suppose the trajectory  $\tau$  was chosen to be added to  $F$  by the algorithm. If  $\tau \subseteq F^{*}$ , then  $G_{i}$  is equal to  $new_{F}(\tau)$  and  $B_{i} = 0$ . If  $\tau \not\subseteq F^{*}$ , then  $B_{i} \leq new_{F}(\tau)$ . Summarizing these observations, we have the following bounds
$$
G _ {i} \geq n e w _ {F} (\tau) \cdot \mathbb {I} [ \tau \subseteq F ^ {*} ] \text {a n d} B _ {i} \leq n e w _ {F} (\tau) \cdot \mathbb {I} [ \tau \nsubseteq F ^ {*} ],
$$
where  $\mathbb{I}[\cdot ]$  is the indicator function.
Moreover, a direct consequence of the probability in which  $\tau$  is added to  $F$  is that for any set of trajectories  $T$ ,
$$
\begin{array}{l} \mathbb {E} _ {\tau \sim Q _ {F}} [ n e w _ {F} (\tau) \cdot \mathbb {I} [ \tau \in T ] ] = \sum_ {\tau \in T} Q _ {F} (\tau) n e w _ {F} (\tau) \\ = \frac {1}{Z} \sum_ {\tau \in T, n e w _ {F} (\tau) \neq 0} \left(\frac {\Pr [ \tau ]}{n e w _ {F} (\tau)}\right) n e w _ {F} (\tau) \tag {1} \\ = \frac {1}{Z} \Pr_ {\tau} [ \tau \in T \wedge n e w _ {F} (\tau) \neq 0 ], \\ \end{array}
$$
where  $Z$  is the normalization factor of  $Q_{F}$ .
To bound the size of  $F$ , we want to show that the algorithm does not add too many states outside of  $F^{*}$ . We therefore bound  $\mathbb{E}[B_i] / \mathbb{E}[G_i]$ , where the expectations are over the trajectory  $\tau$  that is added to  $F$  according to  $Q_{F}$ . Applying Equation (1) twice, once with  $T = \{\tau \mid \tau \subseteq F^{*}\}$  and once with  $T = \{\tau \mid \tau \not\subsetneq F^{*}\}$ , we bound the ratio between  $B_{i}$  and  $G_{i}$  by
$$
\frac {\mathbb {E} \left[ B _ {i} \right]}{\mathbb {E} \left[ G _ {i} \right]} \leq \frac {\Pr_ {\tau} \left[ \tau \not \subseteq F ^ {*} \wedge n e w _ {F} (\tau) \neq 0 \right]}{\Pr_ {\tau} \left[ \tau \subseteq F ^ {*} \wedge n e w _ {F} (\tau) \neq 0 \right]}. \tag {2}
$$
We know that  $\operatorname{Pr}_{\tau}[\tau \nsubseteq F^{*}]$  is always smaller than  $\rho$ , so the numerator is  $\leq \rho$ . A lower bound for the denominator is  $\operatorname{Pr}_{\tau}[new_{F}(\tau) \neq 0] - \operatorname{Pr}_{\tau}[\tau \nsubseteq F^{*}]$ . In addition, whenever the algorithm is inside the main loop, the safety is at least  $\operatorname{Pr}_{\tau}[new_{F}(\tau) \neq 0] = \Delta(F) > 2\rho$ . Thus, the denominator is at least  $\rho$ . Hence, the RHS of (2) is less or equal to 1, thus
$$
\mathbb {E} \left[ B _ {i} \right] \leq \mathbb {E} \left[ G _ {i} \right]. \tag {3}
$$
This completes the proof because we know that the algorithm does not add too many states outside of  $F^{*}$ . More precisely,
$$
\mathbb {E} [ | F | ] = \mathbb {E} \left[ \sum_ {i} B _ {i} + G _ {i} \right] \leq \mathbb {E} \left[ 2 \sum_ {i} G _ {i} \right] \leq 2 k ^ {*}.
$$
Figure 1: Coverage percentage: difference from GreEDy Algorithm.
Sample Complexity. To discuss the sample complexity, we drop the assumption that the MC is known to the learner and use EstimateSafety Subroutine to approximate  $\Delta(F)$ . The number of calls to EstimateSafety is bounded by the size of the output set,  $F$ . Hence, this part of the sample complexity is bounded by  $|F| \cdot N_{|F|}$  and we show that is  $O\left(\frac{k^*}{\epsilon^2} \log k^*\right)$ . Another source of sampling is trajectories sampled for purposes of potentially adding them to  $F$ . Observe that at any iteration the set  $F$  has an escape probability of at least  $2\rho$ , and each trajectory that escapes  $F$  is accepted with a probability of at least  $1 / H$ . This implies a lower bound for the probability that a random trajectory is accepted is  $2\rho / H$ . This gives an upper bound of  $\frac{2|F| \rho}{H}$  for the expected sample complexity.
Amplification. Theorem 4.1 shows that if  $\mathcal{E}$  holds, then the set size,  $|F|$ , is bounded in expectation by  $2k^{*}$ . As  $\operatorname{Pr}[\mathcal{E}] \geq 1 - \lambda$  implies, from Markov's inequality, that the size  $(2 + \delta)k^{*}$  with small probability of about  $\delta + \lambda = O(\delta)$ . If we want to make sure that the actual size is at most  $(2 + \delta)k^{*}$  with high probability, we can repeat the process about  $\Theta\left(\frac{1}{\delta}\right)$  times and take the smallest size set.
5 Empirical Demonstration
This section demonstrates the qualitative and quantitative performance of the described algorithms in the paper. For additional figures, we refer the reader to Appendix D.
The MDP. We focus on a grid of size  $N \times N$ , for some parameter  $N$ . The agent starts off at mid-left state,  $(0, \lfloor \frac{N}{2} \rfloor)$  and wishes to reach the (absorbing) goal state at  $(N - 1, \lfloor \frac{N}{2} \rfloor)$  with a minimal number of steps. At each step, it can take one of four actions: up, down, right, and left by 1 grid square. With probability 0.9, the intended action is performed and with probability 0.1 there is a drift down. The agent stops either way after  $H = 300$  steps.
5.1 FINDING SAFEZONE vs. naive approaches
To compare the FINDING SAFEZONE Algorithm to the naive approaches presented in Section 3 we focus on the policy that first goes to the right and when it reaches the rightmost column, it goes up. The policy is described in the appendix, Figure 6(d). We take  $N = 30$  and 2000 episodes.
We run the FINDING SAFEZONE, GREEDY, and SIMULATION algorithms, and estimate their coverage based on a test set containing 2000 random trajectories. Figure 1 depicts the trajectories coverage of each algorithm minus the coverage of the GREEDY algorithm. For a figure with absolute values, we refer the reader to Figure 6(b) in the appendix. We see that the new algorithm exhibits better performance compared to its competitors. We also see that taking less than  $30\%$  of the states  $(k = 250$  out of 900 states) is enough to get coverage of more than  $80\%$  of the trajectories.
Figures 5(a), 5(b) show the sets found for  $k = 60$  both by the Finding SAFEZONE Algorithm and GreEDY. We see that GreEDY chooses an unconnected set for this small  $k$ , leading to a coverage of 0. While the new algorithm chooses a few states which consist of several trajectories, thus leading to a coverage larger than 0.
6 Discussion and Open Problems
In this paper, we have introduced the SAFEZONE problem. We have shown that it is NP-hard, even when the model is known, and designed a nearly  $(2\rho, 2k^{*})$  approximation algorithm for the case where the model and policy are unknown to the algorithm. Beyond improving the approximation factors (or showing that it cannot be done unless  $P = NP$ ), a natural direction for future work is the following. Given a small  $\rho > 0$  and a (known or unknown to the learner) MDP, find a policy with a small  $\rho$ -safe subset. If the value of the policy, when restricted to the SAFEZONE states, is close to the optimal value of the original MDP, restricting the policy to the SAFEZONE states generates a compact policy representation with a value close to optimal, and most trajectories are completed in the SAFEZONE.
This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement No. 882396), by the Israel Science Foundation (grant number 993/17), Tel Aviv University Center for AI and Data Science (TAD), and the Yandex Initiative for Machine Learning at Tel Aviv University.
Abel, D., Arumugam, D., Lehnert, L., & Littman, M. (2018). State abstractions for lifelong reinforcement learning. In International Conference on Machine Learning, (pp. 10-19).
Alharin, A., Doan, T.-N., & Sartipi, M. (2020). Reinforcement learning interpretation methods: A survey. IEEE Access, 8, 171058-171077.
Amir, D., & Amir, O. (2018). Highlights: Summarizing agent behavior to people. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, (pp. 1168-1176).
Amodei, D., Olah, C., Steinhardt, J., Christiano, P. F., Schulman, J., & Mané, D. (2016). Concrete problems in AI safety. CoRR, abs/1606.06565.
Bennett, A., Misra, D., & Kallus, N. (2023). Provable safe reinforcement learning with binary feedback. In International Conference on Artificial Intelligence and Statistics, (pp. 10871-10900).
Blanc, G., Lange, J., & Tan, L.-Y. (2021). Provably efficient, succinct, and precise explanations. Advances in Neural Information Processing Systems, 34.
Brandes, U., Holm, E., & Karrenbauer, A. (2016). Cliques in regular graphs and the core-periphery problem in social networks. In International Conference on Combinatorial Optimization and Applications, (pp. 175-186). Springer.
Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Comput. Surv..
Dong, S., Van Roy, B., & Zhou, Z. (2019). Provably efficient reinforcement learning with aggregated states. arXiv preprint arXiv:1912.06366.
Emam, Y., Glotfelter, P., Kira, Z., & Egerstedt, M. (2021). Safe model-based reinforcement learning using robust control barrier functions. CoRR, abs/2110.05415.
Even-Dar, E., Mannor, S., & Mansour, Y. (2006). Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. JMLR, 7, 1079-1105.
Even-Dar, E., & Mansour, Y. (2003). Approximate equivalence of markov decision processes. In  $\text{COLT}$ .
García, J., & Fernández, F. (2015). A comprehensive survey on safe reinforcement learning. *J. Mach. Learn. Res.*, 16, 1437-1480.
Givan, R., Dean, T., & Greig, M. (2003). Equivalence notions and model minimization in markov decision processes. Artificial Intelligence.
HasanzadeZonuzy, A., Bura, A., Kalathil, D. M., & Shakkottai, S. (2021). Learning with safety constraints: Sample complexity of reinforcement learning for constrained mdps. In AAAI.
Hendrycks, D., Carlini, N., Schulman, J., & Steinhardt, J. (2021). Unsolved problems in ML safety. CoRR, abs/2109.13916.
Hussein, A., Gaber, M. M., Elyan, E., & Jayne, C. (2017). Imitation learning: A survey of learning methods. ACM Comput. Surv..
Koh, P. W., & Liang, P. (2017). Understanding black-box predictions via influence functions. In International Conference on Machine Learning, (pp. 1885-1894).
Li, J., Nagarajan, V., Plumb, G., & Talwalkar, A. (2020). A learning theoretic perspective on local explainability. arXiv preprint arXiv:2011.01205.
Liu, Z., Cen, Z., Isenbaev, V., Liu, W., Wu, Z. S., Li, B., & Zhao, D. (2022). Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, ICML 2022, Proceedings of Machine Learning Research.
Molnar, C. (2019). Interpretable Machine Learning.
Moshkovitz, M., Yang, Y.-Y., & Chaudhuri, K. (2021). Connecting interpretability and robustness in decision trees through separation. arXiv preprint arXiv:2102.07048.
Pfrommer, S., Gautam, T., Zhou, A., & Sojoudi, S. (2021). Safe reinforcement learning with chance-constrained model predictive control. CoRR, abs/2112.13941.
Prajapat, M., Turchetta, M., Zeilinger, M. N., & Krause, A. (2022). Near-optimal multi-agent learning for safe coverage control. In NeurIPS.
Puterman, M. L. (1994). Markov decision processes: Discrete stochastic dynamic programming. In Wiley Series in Probability and Statistics.
Sootla, A., Cowen-Rivers, A. I., Jafferjee, T., Wang, Z., Mguni, D. H., Wang, J., & Ammar, H. (2022). Saute RL: Almost surely safe reinforcement learning using state augmentation. In Proceedings of the 39th International Conference on Machine Learning, Proceedings of Machine Learning Research.
Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction. Cambridge, MA, USA: A Bradford Book.
Vazirani, V. V. (2001). Approximation Algorithms. Berlin, Heidelberg: Springer-Verlag.
Williamson, D. P., & Shmoys, D. B. (2011). The Design of Approximation Algorithms. Cambridge University Press.
Xu, W., Xu, K., Bastani, H., & Bastani, O. (2021). Safely bridging offline and online reinforcement learning. CoRR, abs/2110.13060.
Yang, J., Chen, T., Payne, B., Guo, P., Zhang, Y., & Guo, J. (2020). Generating routes for autonomous driving in vehicle-to-infrastructure communications. Digital Communications and Networks.
A Hardness
In this section, we show that SAFEZONE is NP-hard to solve, and this is why approximation is necessary. Moreover, SAFEZONE is hard even if the MC and the optimal  $\rho$ -safe size,  $k^*$  are known. Our starting point is the NP-hardness of regular cliques. The REGULARCLIENTE  $(G,k_{c})$  problem gets as an input (i) a regular graph  $G$  with  $n$  nodes where each node has degree  $d$ , and (ii) an integer  $k_{c}$ . It returns whether  $G$  contains a clique of size  $k_{c}$ . Whenever  $G$  and  $k_{c}$  are clear from the context we simply write REGULARCLIENTE. The following fact follows, e.g., from Brandes et al. (2016).
Fact A.1. REGULAR CLIQUE is NP-hard.
Markov chain (random walk). Fix a graph  $G = (V, E)$  and a starting vertex  $v_0 \in V$ . The graph induces a Markov Chain (random walk) in the following way. The states of the process correspond to the vertices  $V$  in the graph  $G$ . The transition function is defined as  $P(v|u) = \frac{1}{d} \cdot \mathbb{1}_{(u,v) \in E}$ , where  $d$  is the degree of any node. The process starts from node  $v_0$  and then proceeds according to the transition function  $P$  for  $H$  steps.
Reduction. To prove the hardness of SAFEZONE, we show how to solve REGULARCLIQUE given a solver to SAFEZONE. For each vertex  $v \in V$ , run an algorithm for SAFEZONE with horizon  $H = 2$ ,  $k = k_{c}$ , and  $\rho = 1 - \left(\frac{k_c - 1}{d}\right)^2$ , and  $v$  as the starting state. If there is at least one run of the algorithm that returns YES, then the final answer is YES. Otherwise, the answer is NO. Note that this reduction is efficient.
Theorem A.2. For every graph  $G = (V, E)$  and an integer  $k_c$  there exists a clique of size  $k_c$  in  $G \iff$  there exists  $v \in V$  such that SAFEZONE  $(V, v_0 = v, P, k_c, \rho)$  returns YES.
Proof.  $(\Rightarrow)$  If there is a clique of size  $k_{c}$ , then we can take the corresponding  $k$  states. The probability to remain in this subset is at least  $\left(\frac{k - 1}{d}\right)^2$  (remember that  $H = 2$ ). Thus, an exact solver for SAFEZONE must return YES.
$(\Longleftarrow)$  Suppose there is no clique of size  $k$ . Assume by contradiction that the reduction (algorithm) returns YES. Let  $s_0$  be a vertex which was the starting state from the running instance which the YES came from and let  $\hat{F}$  denote the output of SAFEZONE. We will show that the probability to remain in any subset of size  $k$  is smaller than  $\left(\frac{k - 1}{d}\right)^2$ .
Since there is no clique of size  $k$  in  $G$ , we know that  $\hat{F}$  is not a clique. It therefore follows that there exists at least two vertexes,  $s_a, s_b \in V$  such that  $(s_a, s_b) \notin E$ .
We will now bound the probability of escape from state  $s_0$  by exhaustion.
1. If  $s_0 \neq s_a$ , then