SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lee Cohen  
TTI-Chicago
Yishay Mansour  
Tel-Aviv University  
Google Research
Michal Moshkovitz Bosch Center for AI
Given a policy of a Markov Decision Process, we define a SAFEZONE as a subset of states, such that most of the policy's trajectories are confined to this subset. The quality of a SAFEZONE is parameterized by the number of states and the escape probability, i.e., the probability that a random trajectory will leave the subset. SAFEZONES are especially interesting when they have a small number of states and low escape probability. We study the complexity of finding optimal SAFEZONES, and show that in general, the problem is computationally hard. Our main result is a bi-criteria approximation learning algorithm with a factor of almost 2 approximation for both the escape probability and SAFEZONE size, using a polynomial size sample complexity.
Most research in reinforcement learning (RL) deals with learning an optimal policy for some Markov Decision Process (MDP). One notable exception to that is Safe RL which addresses the concept of safety. Traditional Safe RL focuses on finding the best policy that meets safety requirements, typically by either adjusting the objective to include the safety requirements and then optimizing for it, or incorporating additional safety constraints to the exploration. In both of these cases, the safety requirements should be pre-specified. Anomaly Detection is the problem of identifying patterns in data that are unexpected, i.e., anomalies (see, e.g., Chandola et al. (2009) for survey). This paper introduces the SAFEZONE problem, which addresses the safety of a specific Markov decision process (MDP) policy by detecting anomalous events rather than finding a policy that satisfies some pre-defined safety constraints.
Consider a finite horizon MDP and a policy (a mapping from states to actions). The policy induces a Markov Chain (MC) on the MDP. Given a subset of states, a trajectory escapes it if at least one of its states is not in the subset. The escape probability of a subset is the probability that a randomly sampled trajectory will escape it. A SAFEZONE is a subset of states whose quality is measured by (1) its escape probability and (2) its size. If a SAFEZONE has low escape probability, we consider it safe (hence escaping is the anomaly). We emphasize that safety is policy-dependent and that different policies could have different SAFEZONES.
Trivial solutions for SAFEZONE include the entire states set (minimal escape probability of 0, maximal size), and the empty set (minimal size, maximal escape probability of 1). The goal is to find a SAFEZONE with a good balance: a relatively small size but still safe enough (small escape probability). More precisely, given an upper bound over the escape probability,  $\rho > 0$ , the goal of the learner is to find the smallest SAFEZONE with escape probability at most  $\rho$  using trajectory sampling. We address an unknown environment, by which we mean no prior knowledge of the transition function or the policy used. The learner is only given access to random trajectories generated by the induced Markov chain. For many applications, if a small SAFEZONE exists, it is useful to find it.
One such example is designing a policy for a smaller state space that performs well in most cases but is undefined for some states, or formally, imitation learning with compact policy representation Abel et al. (2018); Dong et al. (2019). Suppose a company would like to automatically generate a 'lite' edition of a software or an app (e.g., Microsoft Office Lite, Facebook Lite) that contains only part of the system's states, finding a SAFEZONE makes a lot of sense—capturing popular users' trajectories. If instead of finding SAFEZONE, one were to simply take the  $90\%$  most popular states of users using Office, they might not include the state that allows for the precious option of saving, which emphasizes the importance of the problem.
Another motivation for the problem is autonomous vehicles and specifically infrastructure design for them. Even though a lot of the progress in the field of autonomous driving is credited to sensors installed on the vehicles, relying solely on the vehicles' sensors has its limitations (e.g., Yang et al. (2020)). In extreme weather, a vehicle might unintentionally deviate from the current lane and the vehicle sensors might not trigger a response in time. Vehicular-to-Infrastructure (V2I) is a type of communication network between vehicles and road infrastructures that are designed to fill the need for an extra layer of safety. $^{1}$  An important part of the V2I communication is based on Road Side Units (RSUs), sensors that are installed alongside roads. Together with the sensors that are installed on the vehicles, they span the V2I communication. As the resources for RSUs distribution are limited and their enhanced safety is key for V2I and the autonomous vehicle adaptation, distributing RSUs in states of a (good) SAFEZONE could enhance the safety of popular commutes efficiently. Namely, given data regarding commutes (trajectories) in an area, installing RSUs in its' SAFEZONE states will ensure increased safety measures of a major part of the commutes, from starting point to destination, and potentially increase the trust in the system. In addition, if regulation were to prevent people from making autonomous commutes outside of the SAFEZONE, having most of the autonomous commutes confined to the SAFEZONE implies that most commutes can still be driverless.
Another useful application is automatic robotic arms that assemble products. If something unusual happened during the assembly of a product, it might result in a malfunctioning product, and in that case, the operator should be notified (anomaly detection). On the other hand, it is not really autonomic if the operator is notified too frequently. If we find a 'safe enough' SAFEZONE, we can make sure that we notify the operator only in the rare event the production process (trajectory) escapes it, which means that something went wrong with the product assembly. Furthermore, if the SAFEZONE is small, the manufacturer can potentially test the SAFEZONE states and verify their compliance, ensuring that the majority of products are well constructed for a significantly lower testing budget.
Finally, the SAFEZONE problem can be viewed through the lens of explainable RL, where the goal is to explain a specific policy. SAFEZONE is a new post-hoc explanation of the summarization type Alharin et al. (2020). For example, for the autonomous vehicle infrastructure design, governments could explain to their citizens the design that was chosen.
Our results include approximation algorithms for the SAFEZONE problem, which we show is NP-hard. We are interested in a good trade-off between the escape probability of the SAFEZONE and its size. Our algorithms are evaluated based on two criteria: their approximation factors (w.r.t. the escape probability bound and the optimal set size for this bound), and their trajectory sample complexity bounds (e.g., Even-Dar et al. (2006)).
Contribution: In Section 2 we formalize the SAFEZONE problem. In Section 3, we explore naive approaches, namely greedy algorithms that select SAFEZONES based on state distributions and trajectory sampling. In addition, we show particular cases in which their solutions are far from optimal, either in terms of high escape probability or significantly larger set size. In Section 4 we design FINDING SAFEZONE, an efficient approximation algorithm with provable guarantees. The algorithm returns a SAFEZONE which is slightly more than twice the size and twice the escape probability compared to the optimal. While the main focus of this work is the introduction of the problem and the aforementioned theoretical guarantees, we do demonstrate the problem empirically, to provide additional intuition to the readers. In Section 5, we compare the performance of the naive approaches to FINDING SAFEZONE and show that different policies might lead to completely different SAFEZONES. In Appendix A, we show that the problem is hard, even for known environment
<sup>1</sup>The 'V2I Deployment Coalition' is an initiative by the U.S. Department of Transportation with the vision of "An integrated national infrastructure that provides the country a connected, safe and secure transportation system taking full advantage of the progress being made in the Connected and Autonomous Vehicle arenas." https://transportationops.org/V2I/V2I-overview
setting, namely even when the induced Markov chain is given, finding a SAFEZONE is NP-hard, even for horizon  $H = 2$ .
For brevity, some algorithms and (full) proofs are relegated to the appendix.
MDPs have been studied extensively in the context of decision making in particular by the Reinforcement Learning (RL) community (see Puterman (1994) for a broad background on MDPs, and Sutton & Barto (2018) for background on reinforcement learning).
Safe RL A related line of research is safe RL, where the goal of the learner is to find the best policy that satisfies safety guarantees. The two main methodologies to handle such problems are: (1) altering the objective to include the safety requirement and optimizing over it, and (2) adding additional safety constraints to the exploration part. See Pfrommer et al. (2021); Emam et al. (2021); Xu et al. (2021); Hendrycks et al. (2021); HasanzadeZonuzy et al. (2021); Bennett et al. (2023); Prajapat et al. (2022); Liu et al. (2022) for recent works and Garcia & Fernandez (2015); Amodei et al. (2016) for surveys. Recent work by Sootla et al. (2022) augments the environment to accommodate some pre-specified safety constraints and thus satisfies them almost surely. In our work, the goal is not to find the optimal policy, but rather, given a policy, finding its SAFEZONE. The SAFEZONE is not characterized by specific requirements, and might not be unique. Moreover, beyond the MDP, the solution very much depends on the policy.
Imitation Learning. In imitation learning, the learner observes a policy behavior and wants to imitate it (see Hussein et al. (2017) for a survey). Similar to imitation learning, we are given access to samples of a given policy. In contrast, rather than imitating the policy we find the policy's SAFEZONE, which is an important property of the policy.
Approximate MDP equivalence. Another related research line is that of finding an (almost) equivalent minimal model for a given MDP, where the goal is that the optimal policy on the (almost) equivalent model induces an (approximately) optimal policy in the original MDP, e.g., Givan et al. (2003); Even-Dar & Mansour (2003). This line of works and ours differ in that we do not try to modify the MDP (e.g., cluster similar states), but rather to find a SAFEZONE, a property that is defined for the existing MDP and a specific policy.
Explainability. In explainability, the goal is to provide a post-hoc explanation to a specific (given) model Molnar (2019), e.g., using decision trees Blanc et al. (2021); Moshkovitz et al. (2021), influential examples Koh & Liang (2017), or local approximation explanations Li et al. (2020). We focus on explainability for reinforcement learning, and specifically, we suggest a new summarization explanation through our SAFEZONE (Amir & Amir, 2018).
2 The Safe Zone Problem
We model the problem using a Markov model with a finite horizon  $H > 1$ . Formally, there is a Markov chain (MC)  $\langle S, P, s_0 \rangle$  where  $S$  is the set of states,  $s_0 \in S$  is the initial state, and  $P: S \times S \to [0,1]$  is the transition function that maps a pair of states into probability by  $P(s, s') = \operatorname*{Pr}[s_{t+1} = s'|s_t = s]$ . We assume the transition function  $P$  is induced by a policy  $\pi: S \to \text{Simplex}^{\mathcal{A}}$  on an MDP  $\langle S, s_0, P', \mathcal{A} \rangle$  with transition function  $P': S \times \mathcal{A} \times S \to [0,1]$  such that  $P(s, s') = \sum_{a \in \mathcal{A}} P'(s, a, s') \cdot \pi(a|s)$  for all  $s, s' \in S$  (though any MC can be generated this way, thus our theoretical guarantees apply for general MCs).
A trajectory  $\tau = (s_0, \dots, s_H)$  starts in the initial state  $s_0$  and followed by a sequence of  $H$  states generated by  $P$ , i.e.,  $\operatorname{Pr}[s_{i+1} = s'|s_i = s] = P(s, s')$  for all  $i \in [H]$ , where  $[H] := \{1, \dots, H\}$ . For example, in the context of autonomous vehicles, the trajectory is a commute. We abuse the notation and regard a trajectory  $\tau$  both as a sequence and a set.
Given a subset of states  $F \subseteq S$ , a trajectory  $\tau$  escapes  $F$  if it contains at least one state  $s \in \tau$  such that  $s \notin F$ , i.e.,  $\tau \nsubseteq F$ . For example, if a commute passes through at least one area (state) that does not have an RSU sensor, it escapes the SAFEZONE. We refer to the probability that a random trajectory escapes  $F$  as escape probability and denote it by  $\Delta(F) = \operatorname{Pr}_{\tau}[\tau \nsubseteq F]$ . We call  $F$  a  $\rho$ -safe (w.r.t. the model  $\langle S, s_0, P \rangle$ ) if its escape probability,  $\Delta(F)$ , is at most  $\rho$ . Formally,
Definition 2.1. A set  $F \subseteq S$  is  $\rho$ -safe if  $\Delta(F) := \operatorname{Pr}_{\tau}[\tau \nsubseteq F] \leq \rho$ , where  $\tau$  is a random trajectory.
A set  $F \subseteq S$  is called  $(\rho, k) - \text{SAFEZONE}$  if  $F$  is  $\rho$ -safe and  $|F| \leq k$ . Given a safety parameter  $\rho \in (0, 1)$ , we denote the smallest size  $\rho$ -safe set by  $k^{*}(\rho)$ :
$$
k^{*}(\rho) = \min_{F\subseteq \mathcal{S}\text{is}\rho -\text{safe}}|F|.
$$
Whenever the discussed parameter  $\rho$  is clear from the context we use  $k^{*}$  instead of  $k^{*}(\rho)$ . We remark that there might be multiple different  $(\rho, k) - \text{SAFEZONE}$  sets. The learner knows the set of states,  $S$ , the initial state,  $s_0$ , and the horizon  $H$ . However, the transition function  $P$  and the minimal size of the  $\rho$ -safe set,  $k^{*}$ , are unknown to the learner. Instead, the learner receives information about the model from sampling trajectories from the distribution induced by  $P$ .
Given  $\rho > 0$ , the ultimate goal of the learner would have been to find a  $(\rho, k^*(\rho)) - \text{SAFEZONE}$ . However, as we show in Appendix A, finding a  $(\rho, k^*(\rho)) - \text{SAFEZONE}$  is NP-hard, even when the transition function  $P$  is known. This is why we loosen the objective to find a bi-criteria approximation  $(\rho', k') - \text{SAFEZONE}$ . (Bi-criteria approximations are widely studied in approximation and online algorithms Vazirani (2001); Williamson & Shmoys (2011).) In our setting, given  $\rho$  the objective is to find a set  $F$  which is  $(\rho', k') - \text{SAFEZONE}$  with minimal size  $k' \geq k^*$  and minimal escape probability  $\rho' \geq \rho$ . In addition, we are interested in minimizing the sample complexity.
Notice that the learner can efficiently verify, with high probability, whether a set  $F$  is approximately  $\rho$ -safe or not, as we formalize in the next proposition. The following proposition follows directly from Lemma C.2.
Proposition 2.2. There exists an efficient algorithm such that for every set  $F \subseteq S$  and parameters  $\epsilon, \lambda > 0$ , the algorithm samples  $O\left(\frac{1}{\epsilon^2} \ln \frac{1}{\lambda}\right)$  random trajectories and returns  $\widehat{\Delta}(F)$ , such that with probability at most  $\lambda$  we have  $|\Delta(F) - \widehat{\Delta}(F)| \geq \epsilon$ .
2.1 A Note on Trajectory Escaping
The SAFEZONE problem deals with escaping trajectories. In particular, given a SAFEZONE, a trajectory escapes it, no matter if only one of its states is outside the SAFEZONE or all of them. A related, yet very different problem, is that of minimizing a subset size, such that the expected number of states outside the set is minimized. This related problem, while significantly easier (as it is solved by returning the most visited states), does not apply to the applications we described earlier. For example, consider the infrastructure design for autonomous vehicles. We want passengers to have a safe experience end-to-end. Hence the entire route must have that extra security layer provided by the RSUs. In Section 3, we show that the solution for the SAFEZONE does not necessarily overlap with the most visited states. Furthermore, simply returning states that appeared in trajectory samples could result in a set size far from optimal.
2.2 A Note on Multiple Policies
Our framework can accommodate an arbitrary number of policies, representing multiple agents. This is made feasible through the implementation of a single mixed policy. This mixed policy is designed to stochastically select a policy from this ensemble of policies, uniformly at random or according to any specified distribution over the participating agents and their respective policies.
2.3 Summary of Contributions
We summarize the results of all the algorithms that appear in the paper in Table 1. The bounds of GREEDY BY THRESHOLD and GREEDY AT EACH STEP require the Markov Chain model as input, and a pre-processing step that takes  $O(|S|^2 H)$  time. Additionally, the bounds for the first three algorithms (the naive approaches) require additional knowledge of  $k^*(\rho)$ . The sample complexities of SIMULATION is bounded by  $poly(k^*, \frac{1}{\rho})$ , and of FINDING SAFEZONE Algorithm is bounded by  $poly(k^*, H, \frac{1}{\epsilon}, \frac{1}{\delta})$  for some parameters  $\epsilon, \delta \in (0, 1)$ .
Beyond the upper bounds, we provide each of the first three algorithms (the naive approaches) instances that show that they are tight up to a constant.
The following theorem is an informal statement of our main theorem, Theorem 4.2.
Theorem 2.3. For every  $\rho, \epsilon, \delta > 0$ , with probability  $\geq 0.99$  there exists an algorithm that returns a set which is  $(2\rho + 2\epsilon, (2 + \delta)k^{*}) - \text{SAFEZONE}$ .
In addition to the sample complexity, the running time of the algorithm is also bounded by  $poly(k^{*},H,\frac{1}{\delta},\frac{1}{\epsilon})$
We empirically evaluate the suggested algorithms on a grid-world instance (where the goal is to reach an absorbing state), showing that FINDING SAFEZONE outperforms the naive approaches. Moreover, we show that different policies have qualitatively different SAFEZONES. Finally, an informal statement of Theorem A.2 which appears in Appendix A due to space limitations.
Theorem 2.4. SAFEZONE is NP-hard.
Table 1: Upper bounds for safety and set size. * Only for layered MDPs.
Algorithm	Safety	Set Size
Greedy by Threshold	2ρ	k*H/ρ
Simulation	2ρ	O(k*H ln k*)
Greedy at Each Step*	ρH	k*
Finding SAFEZONE	2ρ + 2ε	(2 + δ)k*
3 Gentle Start
This section explains and analyzes various naive algorithms to the SAFEZONE problem. We show that even if the transition function is known in advance, these naive algorithms result in outputs that are far from optimal. To describe the algorithms, we define for each state  $s$  the probability to appear in a random trajectory and denote it by  $p(s) = \operatorname{Pr}_{\tau}[s \in \tau] \in [0,1]$ . Note that  $\sum_{s \in S} p(s)$  is a number between 1 and  $H$  (e.g.,  $p(s_0) = 1$ ), and can be estimated efficiently using dynamic programming if the environment and policy are known and sampling otherwise. To be precise, some of the algorithms assume the probabilities  $\{p(s)\}_{s \in S}$  are received as input.
Greedy by Threshold Algorithm. The algorithm gets, in addition to  $\rho$ , the distribution  $p$  and a parameter  $\beta > 0$  as input. It returns a set  $F$  that contains all states  $s$  with probability at least  $\beta$ , i.e.,  $p(s) \geq \beta$ . We formalize this idea as Algorithm 3 in Appendix B. For  $\beta = \frac{\rho}{k^*}$ , the output of the algorithm is  $\left(2\rho, \frac{k^*H}{\rho}\right) - \text{SAFEZONE}$ . More generally, we prove the following lemma.
Lemma 3.1. For any  $\rho, \beta \in (0,1)$ , the GreEDy BY THRESHOLD ALGORITHM returns a set that is  $(\rho + k^{*}\beta, \frac{H}{\beta}) - \text{SAFEZONE}$ . In particular, for  $\beta = \frac{\rho}{k^{*}}$ , this set is  $\left(2\rho, \frac{k^{*}H}{\rho}\right) - \text{SAFEZONE}$ .
While it is clear why there are instances for which the safety is tight, Lemma B.1 in Appendix B shows that the set size is tight as well.
Simulation Algorithm. The algorithm samples  $O\left(\frac{\ln k^*}{\beta}\right)$  random trajectories and returns a set  $F$  with all the states in these trajectories. It is formalized in Appendix B as Algorithm 4.
Lemma 3.2. Fix  $\rho, \beta \in (0,1)$ . With probability at least 0.99, SIMULATION Algorithm returns a set that is  $\left(\rho + k^{*}\beta, O(k^{*} + \frac{\rho H\ln k^{*}}{\beta})\right) - \text{SAFEZONE}$ . In particular, for  $\beta = \frac{\rho}{k^{*}}$ , this set is  $(2\rho, O(k^{*}H\ln k^{*})) - \text{SAFEZONE}$ .
While the algorithm achieves a low escape probability, only  $2\rho$ , in Lemma B.2 in the appendix we prove that the size of  $F$  is tight up to a constant, i.e., we show an MDP instance where  $|F| = \Omega(k^{*}H\ln k^{*})$ . The algorithms presented so far were approximately safe (i.e., low escape probability), but the returned set size was large. Without any further assumptions, the following algorithm provides a  $(\rho H, Hk^{*})$ -SAFEZONE, thus not improving the previous algorithms. However, when considering MDPs with a special structure it provides an optimal sized SAFEZONE, at the price of large escape probability.
Greedy at Each Step Algorithm. For the analysis of the next algorithm, we assume the MDP is layered, i.e., there are no states that appear in more than a single time step and denote  $S = \bigcup_{i=1}^{H} S_i$ . I.e., the transitions  $P(s, s')$  are nonzero only for  $s' \in S_{i+1}$  and  $s \in S_i$ . The GreEDy AT EACH STEP ALGORITHM, sometimes simply called greedy, takes at each time step  $i$  the minimal number of states such that the sum of their probabilities is at least  $1 - \rho$ . It is formalized in Appendix B as Algorithm 5.
Lemma 3.3. For any  $\rho \in (0,1)$ , if the MDP is layered, GreEDy AT EACH STEP ALGORITHM returns a set that is  $(\rho H,k^{*}) - \text{SAFEZONE}$ .
In Lemma B.3 in the appendix we provide a lower bound on the escape probability, matching up to a constant.
Weaknesses of the naive algorithms. We showed algorithms that identify SAFEZONE with escape probability much greater than  $\rho$  or size much greater than  $k^{*}$ , and instances with tight lower bounds for each of them. This holds even when providing extra information about the model or the optimal size of the  $\rho$ -safe set, i.e.,  $k^{*}$ .
4 Algorithm for Detecting Safe Zones
In this section, we suggest a new algorithm that builds upon and improves the added trajectory selection of the SIMULATION Algorithm. One reason for why SIMULATION returns a large set is that it treats every sampled trajectory identically, regardless of how many states are being added, which could be as large as  $H$ . More precisely, fix any  $(\rho, k^*) - \text{SAFEZONE}$  set,  $F^*$ , and consider a trajectory  $\tau$  that escapes it, i.e.,  $\tau \not\subsetneq F^*$ . If  $\tau$  was sampled, its states are added to the constructed set  $F$ , which might increase the size of  $F$  by up to  $H$  states that are not in  $F^*$ , without significantly improving the safety. In contrast, when selecting which trajectory to add to  $F$ , we would consider the number of states it adds to the current set. For the sake of readability, we refer to any state which is not in the current set  $F$  as new, and denote by  $new_F(\tau)$  the number of new states in  $\tau$  w.r.t.  $F$ , i.e.,
$$
\operatorname {n e w} _ {F} (\tau) := | \tau \setminus F |.
$$
Note that for every  $F \subseteq S$ ,  $\operatorname{Pr}_{\tau}[new_F(\tau) \neq 0] = \Delta(F)$ .
The new algorithm does not sample each trajectory uniformly at random, but samples from a new distribution, which will be denoted by  $Q_{F}$ .