Title: Finding Safe Zones of Markov Decision Processes Policies
Venue: NeurIPS 2023
Accepted: yes
Keywords: Theoretical guarantees

algorithms

learning theory

MDP

computational complexity

Interpretability
Abstract: Given a policy of a Markov Decision Process, we define a SafeZone as a subset of states, such that most of the policy's trajectories are confined to this subset. The quality of a SafeZone is parameterized by the number of states and the escape probability, i.e., the probability that a random trajectory will leave the subset. SafeZones are especially interesting when they have a small number of states and low escape probability. We study the complexity of finding optimal SafeZones, and show that in general, the problem is computationally hard. For this reason, we concentrate on finding approximate SafeZones. Our main result is a bi-criteria approximation learning algorithm with a factor of almost $2$  approximation for both the escape probability and \newprob size, using a polynomial size sample complexity.

Finding Safe Zones of Markov Decision Processes Policies

Abstract

Given a policy of a Markov Decision Process, we define a SAFEZONE as a subset of states, such that most of the policy's trajectories are confined to this subset. The quality of a SAFEZONE is parameterized by the number of states and the escape probability, i.e., the probability that a random trajectory will leave the subset. SAFEZONES are especially interesting when they have a small number of states and low escape probability. We study the complexity of finding optimal SAFEZONES, and show that in general, the problem is computationally hard. Our main result is a bi-criteria approximation learning algorithm with a factor of almost 2 approximation for both the escape probability and SAFEZONE size, using a polynomial size sample complexity.

1 Introduction

Most research in reinforcement learning (RL) deals with learning an optimal policy for some Markov Decision Process (MDP). One notable exception to that is Safe RL which addresses the concept of safety. Traditional Safe RL focuses on finding the best policy that meets safety requirements, typically by either adjusting the objective to include the safety requirements and then optimizing for it, or incorporating additional safety constraints to the exploration. In both of these cases, the safety requirements should be pre-specified. Anomaly Detection is the problem of identifying patterns in data that are unexpected, i.e., anomalies (see, e.g., Chandola et al. (2009) for survey). This paper introduces the SAFEZONE problem, which addresses the safety of a specific Markov decision process (MDP) policy by detecting anomalous events rather than finding a policy that satisfies some pre-defined safety constraints.

Consider a finite horizon MDP and a policy (a mapping from states to actions). The policy induces a Markov Chain (MC) on the MDP. Given a subset of states, a trajectory escapes it if at least one of its states is not in the subset. The escape probability of a subset is the probability that a randomly sampled trajectory will escape it. A SAFEZONE is a subset of states whose quality is measured by (1) its escape probability and (2) its size. If a SAFEZONE has low escape probability, we consider it safe (hence escaping is the anomaly). We emphasize that safety is policy-dependent and that different policies could have different SAFEZONES.

Trivial solutions for SAFEZONE include the entire states set (minimal escape probability of 0 , maximal size), and the empty set (minimal size, maximal escape probability of 1 ). The goal is to find a SAFEZONE with a good balance: a relatively small size but still safe enough (small escape probability). More precisely, given an upper bound over the escape probability, ρ > 0 , the goal of the learner is to find the smallest SAFEZONE with escape probability at most ρ using trajectory sampling. We address an unknown environment, by which we mean no prior knowledge of the transition function or the policy used. The learner is only given access to random trajectories generated by the induced Markov chain. For many applications, if a small SAFEZONE exists, it is useful to find it.

One such example is designing a policy for a smaller state space that performs well in most cases but is undefined for some states, or formally, imitation learning with compact policy representation Abel et al. (2018); Dong et al. (2019). Suppose a company would like to automatically generate a 'lite' edition of a software or an app (e.g., Microsoft Office Lite, Facebook Lite) that contains only part of the system's states, finding a SAFEZONE makes a lot of sense- capturing popular users' trajectories. If instead of finding SAFEZONE , one were to simply take the 90% most popular states of users using Office, they might not include the state that allows for the precious option of saving, which emphasizes the importance of the problem.

Another motivation for the problem is autonomous vehicles and specifically infrastructure design for them. Even though a lot of the progress in the field of autonomous driving is credited to sensors installed on the vehicles, relying solely on the vehicles' sensors has its limitations (e.g., Yang et al. (2020)). In extreme weather, a vehicle might unintentionally deviate from the current lane and the vehicle sensors might not trigger a response in time. Vehicular-to-Infrastructure (V2I) is a type of communication network between vehicles and road infrastructures that are designed to fill the need for an extra layer of safety. 1 An important part of the V2I communication is based on Road Side Units (RSUs), sensors that are installed alongside roads. Together with the sensors that are installed on the vehicles, they span the V2I communication. As the resources for RSUs distribution are limited and their enhanced safety is key for V2I and the autonomous vehicle adaptation, distributing RSUs in states of a (good) SAFEZONE could enhance the safety of popular commutes efficiently. Namely, given data regarding commutes (trajectories) in an area, installing RSUs in its' SAFEZONE states will ensure increased safety measures of a major part of the commutes, from starting point to destination, and potentially increase the trust in the system. In addition, if regulation were to prevent people from making autonomous commutes outside of the SAFEZONE , having most of the autonomous commutes confined to the SAFEZONE implies that most commutes can still be driverless.

Another useful application is automatic robotic arms that assemble products. If something unusual happened during the assembly of a product, it might result in a malfunctioning product, and in that case, the operator should be notified (anomaly detection). On the other hand, it is not really autonomic if the operator is notified too frequently. If we find a 'safe enough' SAFEZONE, we can make sure that we notify the operator only in the rare event the production process (trajectory) escapes it, which means that something went wrong with the product assembly. Furthermore, if the SAFEZONE is small, the manufacturer can potentially test the SAFEZONE states and verify their compliance, ensuring that the majority of products are well constructed for a significantly lower testing budget.

Finally, the SAFEZONE problem can be viewed through the lens of explainable RL, where the goal is to explain a specific policy. SAFEZONE is a new post-hoc explanation of the summarization type Alharin et al. (2020). For example, for the autonomous vehicle infrastructure design, governments could explain to their citizens the design that was chosen.

Our results include approximation algorithms for the SAFEZONE problem, which we show is NP-hard. We are interested in a good trade-off between the escape probability of the SAFEZONE and its size. Our algorithms are evaluated based on two criteria: their approximation factors (w.r.t. the escape probability bound and the optimal set size for this bound), and their trajectory sample complexity bounds (e.g., Even-Dar et al. (2006)).

Contribution : In Section 2 we formalize the SAFEZONE problem. In Section 3, we explore naive approaches, namely greedy algorithms that select SAFEZONES based on state distributions and trajectory sampling. In addition, we show particular cases in which their solutions are far from optimal, either in terms of high escape probability or significantly larger set size. In Section 4 we design FINDING SAFEZONE, an efficient approximation algorithm with provable guarantees. The algorithm returns a SAFEZONE which is slightly more than twice the size and twice the escape probability compared to the optimal. While the main focus of this work is the introduction of the problem and the aforementioned theoretical guarantees, we do demonstrate the problem empirically, to provide additional intuition to the readers. In Section 5, we compare the performance of the naive approaches to FINDING SAFEZONE and show that different policies might lead to completely different SAFEZONES. In Appendix A, we show that the problem is hard, even for known environment

setting, namely even when the induced Markov chain is given, finding a SAFEZONE is NP-hard, even for horizon H = 2 .

For brevity, some algorithms and (full) proofs are relegated to the appendix.

1.1 Related Work

MDPs have been studied extensively in the context of decision making in particular by the Reinforcement Learning (RL) community (see Puterman (1994) for a broad background on MDPs, and Sutton &Barto (2018) for background on reinforcement learning).

Safe RL A related line of research is safe RL, where the goal of the learner is to find the best policy that satisfies safety guarantees. The two main methodologies to handle such problems are: (1) altering the objective to include the safety requirement and optimizing over it, and (2) adding additional safety constraints to the exploration part. See Pfrommer et al. (2021); Emam et al. (2021); Xu et al. (2021); Hendrycks et al. (2021); HasanzadeZonuzy et al. (2021); Bennett et al. (2023); Prajapat et al. (2022); Liu et al. (2022) for recent works and García & Fernández (2015); Amodei et al. (2016) for surveys. Recent work by Sootla et al. (2022) augments the environment to accommodate some pre-specified safety constraints and thus satisfies them almost surely. In our work, the goal is not to find the optimal policy, but rather, given a policy, finding its SAFEZONE. The SAFEZONE is not characterized by specific requirements, and might not be unique. Moreover, beyond the MDP, the solution very much depends on the policy.

Imitation Learning. In imitation learning, the learner observes a policy behavior and wants to imitate it (see Hussein et al. (2017) for a survey). Similar to imitation learning, we are given access to samples of a given policy. In contrast, rather than imitating the policy we find the policy's SAFEZONE, which is an important property of the policy.

Approximate MDP equivalence. Another related research line is that of finding an (almost) equivalent minimal model for a given MDP, where the goal is that the optimal policy on the (almost) equivalent model induces an (approximately) optimal policy in the original MDP, e.g., Givan et al. (2003); Even-Dar & Mansour (2003). This line of works and ours differ in that we do not try to modify the MDP (e.g., cluster similar states), but rather to find a SAFEZONE, a property that is defined for the existing MDP and a specific policy.

Explainability. In explainability, the goal is to provide a post-hoc explanation to a specific (given) model Molnar (2019), e.g., using decision trees Blanc et al. (2021); Moshkovitz et al. (2021), influential examples Koh & Liang (2017), or local approximation explanations Li et al. (2020). We focus on explainability for reinforcement learning, and specifically, we suggest a new summarization explanation through our SAFEZONE (Amir & Amir, 2018).

2 The Safe Zone Problem

We model the problem using a Markov model with a finite horizon H > 1 . Formally, there is a Markov chain (MC) ⟨S , P, s 0 ⟩ where S is the set of states, s 0 ∈ S is the initial state, and P : S × S → [0 , 1] is the transition function that maps a pair of states into probability by P ( s, s ′ ) = Pr[ s t +1 = s ′ | s t = s ] . We assume the transition function P is induced by a policy π : S → Simplex A on an MDP ⟨S , s 0 , P ′ , A⟩ with transition function P ′ : S × A × S → [0 , 1] such that P ( s, s ′ ) = ∑ a ∈A P ′ ( s, a, s ′ ) · π ( a | s ) for all s, s ′ ∈ S (though any MC can be generated this way, thus our theoretical guarantees apply for general MCs).

A trajectory τ = ( s 0 , . . . , s H ) starts in the initial state s 0 and followed by a sequence of H states generated by P , i.e., Pr[ s i +1 = s ′ | s i = s ] = P ( s, s ′ ) for all i ∈ [ H ] , where [ H ] := { 1 , . . . , H } . For example, in the context of autonomous vehicles, the trajectory is a commute. We abuse the notation and regard a trajectory τ both as a sequence and a set.

Given a subset of states F ⊆ S , a trajectory τ escapes F if it contains at least one state s ∈ τ such that s / ∈ F , i.e., τ ̸⊆ F . For example, if a commute passes through at least one area (state) that does not have an RSU sensor, it escapes the SAFEZONE . We refer to the probability that a random trajectory escapes F as escape probability and denote it by ∆( F ) = Pr τ [ τ ̸⊆ F ] . We call F a ρ -safe (w.r.t. the model ⟨S , s 0 , P ⟩ ) if its escape probability, ∆( F ) , is at most ρ . Formally,

Definition 2.1. A set F ⊆ S is ρ -safe if ∆( F ) := Pr τ [ τ ̸⊆ F ] ≤ ρ, where τ is a random trajectory.

A set F ⊆ S is called ( ρ, k ) -SAFEZONE if F is ρ -safe and | F | ≤ k . Given a safety parameter ρ ∈ (0 , 1) , we denote the smallest size ρ -safe set by k ∗ ( ρ ) :

Whenever the discussed parameter ρ is clear from the context we use k ∗ instead of k ∗ ( ρ ) . We remark that there might be multiple different ( ρ, k ) -SAFEZONE sets. The learner knows the set of states, S , the initial state, s 0 , and the horizon H . However, the transition function P and the minimal size of the ρ -safe set, k ∗ , are unknown to the learner. Instead, the learner receives information about the model from sampling trajectories from the distribution induced by P .

Given ρ > 0 , the ultimate goal of the learner would have been to find a ( ρ, k ∗ ( ρ )) -SAFEZONE. However, as we show in Appendix A, finding a ( ρ, k ∗ ( ρ )) -SAFEZONE is NP-hard, even when the transition function P is known. This is why we loosen the objective to find a bi-criteria approximation ( ρ ′ , k ′ ) -SAFEZONE . (Bi-criteria approximations are widely studied in approximation and online algorithms Vazirani (2001); Williamson & Shmoys (2011).) In our setting, given ρ the objective is to find a set F which is ( ρ ′ , k ′ ) -SAFEZONE with minimal size k ′ ≥ k ∗ and minimal escape probability ρ ′ ≥ ρ . In addition, we are interested in minimizing the sample complexity.

Notice that the learner can efficiently verify, with high probability, whether a set F is approximately ρ -safe or not, as we formalize in the next proposition. The following proposition follows directly from Lemma C.2.

Proposition 2.2. There exists an efficient algorithm such that for every set F ⊆ S and parameters ϵ, λ > 0 , the algorithm samples O ( 1 ϵ 2 ln 1 λ ) random trajectories and returns ̂ ∆( F ) , such that with probability at most λ we have | ∆( F ) -̂ ∆( F ) | ≥ ϵ .

2.1 A Note on Trajectory Escaping

The SAFEZONE problem deals with escaping trajectories. In particular, given a SAFEZONE, a trajectory escapes it, no matter if only one of its states is outside the SAFEZONE or all of them. A related, yet very different problem, is that of minimizing a subset size, such that the expected number of states outside the set is minimized. This related problem, while significantly easier (as it is solved by returning the most visited states), does not apply to the applications we described earlier. For example, consider the infrastructure design for autonomous vehicles. We want passengers to have a safe experience end-to-end. Hence the entire route must have that extra security layer provided by the RSUs. In Section 3, we show that the solution for the SAFEZONE does not necessarily overlap with the most visited states. Furthermore, simply returning states that appeared in trajectory samples could result in a set size far from optimal.

2.2 A Note on Multiple Policies

Our framework can accommodate an arbitrary number of policies, representing multiple agents. This is made feasible through the implementation of a single mixed policy. This mixed policy is designed to stochastically select a policy from this ensemble of policies, uniformly at random or according to any specified distribution over the participating agents and their respective policies.

2.3 Summary of Contributions

We summarize the results of all the algorithms that appear in the paper in Table 1. The bounds of GREEDY BY THRESHOLD and GREEDY AT EACH STEP require the Markov Chain model as input, and a pre-processing step that takes O ( | S | 2 H ) time. Additionally, the bounds for the first three algorithms (the naive approaches) require additional knowledge of k ∗ ( ρ ) . The sample complexities of SIMULATION is bounded by poly ( k ∗ , 1 ρ ) , and of FINDING SAFEZONE Algorithm is bounded by poly ( k ∗ , H, 1 ϵ , 1 δ ) for some parameters ϵ, δ ∈ (0 , 1) .

Beyond the upper bounds, we provide each of the first three algorithms (the naive approaches) instances that show that they are tight up to a constant.

The following theorem is an informal statement of our main theorem, Theorem 4.2.

Theorem 2.3. For every ρ, ϵ, δ > 0 , with probability ≥ 0 . 99 there exists an algorithm that returns a set which is (2 ρ +2 ϵ, (2 + δ ) k ∗ ) -SAFEZONE .

In addition to the sample complexity, the running time of the algorithm is also bounded by poly ( k ∗ , H, 1 δ , 1 ϵ ) .

We empirically evaluate the suggested algorithms on a grid-world instance (where the goal is to reach an absorbing state), showing that FINDING SAFEZONE outperforms the naive approaches. Moreover, we show that different policies have qualitatively different SAFEZONES. Finally, an informal statement of Theorem A.2 which appears in Appendix A due to space limitations.

Theorem 2.4. SAFEZONE is NP-hard.

3 Gentle Start

This section explains and analyzes various naive algorithms to the SAFEZONE problem. We show that even if the transition function is known in advance, these naive algorithms result in outputs that are far from optimal. To describe the algorithms, we define for each state s the probability to appear in a random trajectory and denote it by p ( s ) = Pr τ [ s ∈ τ ] ∈ [0 , 1] . Note that ∑ s ∈S p ( s ) is a number between 1 and H (e.g., p ( s 0 ) = 1 ), and can be estimated efficiently using dynamic programming if the environment and policy are known and sampling otherwise. To be precise, some of the algorithms assume the probabilities { p ( s ) } s ∈S are received as input.

Greedy by Threshold Algorithm. The algorithm gets, in addition to ρ , the distribution p and a parameter β > 0 as input. It returns a set F that contains all states s with probability at least β , i.e., p ( s ) ≥ β . We formalize this idea as Algorithm 3 in Appendix B. For β = ρ k ∗ , the output of the algorithm is ( 2 ρ, k ∗ H ρ ) -SAFEZONE . More generally, we prove the following lemma.

Lemma 3.1. For any ρ, β ∈ (0 , 1) , the GREEDY BY THRESHOLD ALGORITHM returns a set that is ( ρ + k ∗ β, H β ) -SAFEZONE . In particular, for β = ρ k ∗ , this set is ( 2 ρ, k ∗ H ρ ) -SAFEZONE .

While it is clear why there are instances for which the safety is tight, Lemma B.1 in Appendix B shows that the set size is tight as well.

Simulation Algorithm. The algorithm samples O ( ln k ∗ β ) random trajectories and returns a set F with all the states in these trajectories. It is formalized in Appendix B as Algorithm 4.

Lemma 3.2. Fix ρ, β ∈ (0 , 1) . With probability at least 0 . 99 , SIMULATION Algorithm returns a set that is ( ρ + k ∗ β, O ( k ∗ + ρH ln k ∗ β ) ) -SAFEZONE . In particular, for β = ρ k ∗ , this set is (2 ρ, O ( k ∗ H ln k ∗ )) -SAFEZONE .

While the algorithm achieves a low escape probability, only 2 ρ , in Lemma B.2 in the appendix we prove that the size of F is tight up to a constant, i.e., we show an MDP instance where | F | = Ω( k ∗ H ln k ∗ ). The algorithms presented so far were approximately safe (i.e., low escape probability), but the returned set size was large. Without any further assumptions, the following algorithm provides a ( ρH,Hk ∗ ) -SAFEZONE, thus not improving the previous algorithms. However, when considering MDPs with a special structure it provides an optimal sized SAFEZONE , at the price of large escape probability.

Greedy at Each Step Algorithm. For the analysis of the next algorithm, we assume the MDP is layered , i.e., there are no states that appear in more than a single time step and denote S = ⋃ H i =1 S i . I.e., the transitions P ( s, s ′ ) are nonzero only for s ′ ∈ S i +1 and s ∈ S i . The GREEDY AT EACH STEP ALGORITHM, sometimes simply called greedy, takes at each time step i the minimal number of states such that the sum of their probabilities is at least 1 -ρ . It is formalized in Appendix B as Algorithm 5.

Lemma 3.3. For any ρ ∈ (0 , 1) , if the MDP is layered, GREEDY AT EACH STEP ALGORITHM returns a set that is ( ρH,k ∗ ) -SAFEZONE .

In Lemma B.3 in the appendix we provide a lower bound on the escape probability, matching up to a constant.

Weaknesses of the naive algorithms. We showed algorithms that identify SAFEZONE with escape probability much greater than ρ or size much greater than k ∗ , and instances with tight lower bounds for each of them. This holds even when providing extra information about the model or the optimal size of the ρ -safe set, i.e., k ∗ .

4 Algorithm for Detecting Safe Zones

In this section, we suggest a new algorithm that builds upon and improves the added trajectory selection of the SIMULATION Algorithm. One reason for why SIMULATION returns a large set is that it treats every sampled trajectory identically, regardless of how many states are being added, which could be as large as H . More precisely, fix any ( ρ, k ∗ ) -SAFEZONE set, F ∗ , and consider a trajectory τ that escapes it, i.e., τ ̸⊆ F ∗ . If τ was sampled, its states are added to the constructed set F , which might increase the size of F by up to H states that are not in F ∗ , without significantly improving the safety. In contrast, when selecting which trajectory to add to F , we would consider the number of states it adds to the current set. For the sake of readability, we refer to any state which is not in the current set F as new , and denote by new F ( τ ) the number of new states in τ w.r.t. F , i.e.,

̸

Note that for every F ⊆ S , Pr τ [ new F ( τ ) = 0] = ∆( F ) .

The new algorithm does not sample each trajectory uniformly at random, but samples from a new distribution, which will be denoted by Q F .

While favoring trajectories with higher probabilities, which we already get by the sampling process, another key idea would guide this new distribution: To prefer trajectories that gradually increase the size of F . To implement this idea, we will ensure that the probability of adding a trajectory τ to F should be inversely proportional to new F ( τ ) .

̸

Formally, the support of Q F is the trajectories with new states, i.e., X = { τ | new F ( τ ) = 0 } . For every τ ∈ X

where Pr[ τ ] is the probability of trajectory τ under the Markov Chain with dynamics P . Note that the new distribution depends on the current set F , and changes as we modify it. Intuitively, adding trajectories to F according to Q F instead of adding trajectories sampled directly from the dynamics (as we do in SIMULATION) would increase the expected ratio between the added safety and the number of new states we add to F , thus improving the set size guarantee of the output set. We elaborate on this in Section 4.2.

̸

Our main algorithm is FINDING SAFEZONE, Algorithm 1. The algorithm receives, in addition to the safety parameter ρ , parameters ϵ, λ ∈ (0 , 1) , and maintains a set F that is initiated to { s 0 } . On a high level, to implement the idea of adding trajectories to F according to Q F , we use rejection sampling . Namely, in each iteration of the while-loop we first sample a trajectory τ , and if new F ( τ ) = 0 , we accept it with probability 1 /new F ( τ ) . If the trajectory is accepted, it is added to F . More precisely, if new F ( τ ) = 0 , we sample a Bernoulli random variable, accept ∼ Br (1 /new F ( τ )) . If accept = 1 , we add τ to F . This process of adding trajectories to F generates the desired distribution, Q F . Whenever a trajectory is added to F , we estimate the escape probability ∆( F ) (w.r.t. the updated set, F ).

̸

The algorithm stops adding states to F and returns it as output when it becomes 'safe enough'. To be precise, let ̂ ∆( F ) denote the result of the escape probability estimation (by sampling trajectories as suggested in Proposition 2.2). If ̂ ∆( F ) ≤ 2 ρ + ϵ , it means that F is (2 ρ +2 ϵ ) -safe with probability ≥ 1 -λ j > 1 -λ , in which case the algorithm terminates and returns F as output.

To implement the estimation ̂ ∆( F ) , the algorithm calls EstimateSafety Subroutine. The subroutine samples N j = Θ( 1 ϵ 2 ln 2 λ j ) trajectories, and returns the fraction of trajectories that escaped F .

For cases in which the transition function P is known to the learner, we provide an alternative implementation for EstimateSafety which computes the exact probability ∆( F ) (see Lemma E.1 in Appendix E).

̸

4.1 Algorithm Analysis

We define the event E = {∀ i | ̂ ∆( F i -1 ) -∆( F i -1 ) | ≤ ϵ } , which states that all our EstimateSafety Subroutine estimations are accurate. We show that E holds with high probability using Hoeffding's inequality. In most of the analysis, we condition on E to hold.

The following theorem is the central component in the proof of the main theorem that follows it.

Theorem 4.1. Given ρ, ϵ, λ ∈ (0 , 1) , FINDING SAFEZONE Algorithm returns a subset F ⊆ S such that:

The escape probability is bounded from above by ∆( F ) ≤ 2 ρ +2 ϵ , with probability 1 -λ .

The expected size of F given E is bounded by E [ | F | | E ] ≤ 2 k ∗ .

The sample complexity of the algorithm is bounded by O ( k ∗ λϵ 2 ln k ∗ λ + Hk ∗ ρλ ) , and the running time is bounded by O ( Hk ∗ λϵ 2 ln k ∗ λ + H 2 k ∗ ρλ ) , with probability 1 -λ .

To obtain the main theorem, we run FINDING SAFEZONE Algorithm several times and return the smallest output set, F , see the next section for more details.

Theorem 4.2. (main theorem) Given ϵ, ρ, δ > 0 , if we run FINDING SAFEZONE for Θ( 1 δ ) times and return the smallest output set, F ⊆ S , then with probability ≥ 0 . 99

The escape probability is bounded by ∆( F ) ≤ 2 ρ +2 ϵ.

The size of F is bounded from above by | F | ≤ (2 + δ ) k ∗ .

The total sample complexity and running time are bounded by O ( k ∗ δ 2 ϵ 2 ln k ∗ δ + Hk ∗ ρδ 2 ) , and O ( Hk ∗ δ 2 ϵ 2 ln k ∗ δ + H 2 k ∗ ρδ 2 ) , respectively.

Finding an almost 2 ρ -safe SAFEZONE , nearly 2 k ∗ in size can be valuable. For example, it enhances the safety of popular commuting routes, promotes trust in autonomous vehicles, and aligns with potential regulatory restrictions, making most commutes driverless within this secure zone. Regarding sample complexity, we expect the dependency on k ∗ to be nearly optimal. The reason why is the following. Consider an induced MC with k ∗ -1 trajectories of size 2 , each starting from an initial

state s 0 , ends with a unique corresponding state 1 , . . . , k ∗ -1 , and has a probability of 1 -ρ k ∗ -1 . If the MC has significantly more than k ∗ states with non-zero probability, it would take at least K ∗ -1 samples to find a ( ρ, k ∗ ) -SAFEZONE. As for the parameter δ , this could be treated as a constant. For example, selecting δ = 1 / 3 yields that we need to run the algorithm 6 · ln 300 times and a solution of size 7 / 3 k ∗ w.h.p.

4.2 Proof Technique

Escape Probability Set Size Bounds. To ease the presentation of the proof, we assume that ̂ ∆( F ) = ∆( F ) . For full proofs, we refer to Appendix C. This case is interesting on its own since if the policy and transition function are known, we can compute ∆( F ) efficiently using dynamic programming (see Appendix E). As a result, event E always holds. In addition, it is clear that the termination of the algorithm implies that ̂ ∆( F ) = ∆( F ) ≤ 2 ρ , thus F is (2 ρ +2 ϵ ) -safe. The main challenge is bounding | F | .

A few notations before we start. Let F ∗ denote a minimal ρ -safe set (of size k ∗ ). Consider iteration i inside the while-loop. The random variable G i ( F ) is the number of states in F ∗ that are added to F in iteration i and B i ( F ) is the number of states added to F in iteration i that are not in F ∗ ( G stands for good and B for bad ). For ease of presentation, from here on we write G i and B i instead of G i ( F ) and B i ( F ) , respectively. Notice that the size of the output set is exactly ∑ i B i + G i and that ∑ i G i ≤ k ∗ .

The main idea of the proof technique is to show that by adding trajectories according to the new distribution Q F , we ensure that, in expectation, there are at least as many good states that are added to F as bad states. Suppose the trajectory τ was chosen to be added to F by the algorithm. If τ ⊆ F ∗ , then G i is equal to new F ( τ ) and B i = 0 . If τ ̸⊆ F ∗ , then B i ≤ new F ( τ ) . Summarizing these observations, we have the following bounds

where I [ · ] is the indicator function.

Moreover, a direct consequence of the probability in which τ is added to F is that for any set of trajectories T ,

̸

where Z is the normalization factor of Q F .

To bound the size of F , we want to show that the algorithm does not add too many states outside of F ∗ . We therefore bound E [ B i ] / E [ G i ] , where the expectations are over the trajectory τ that is added to F according to Q F . Applying Equation (1) twice, once with T = { τ | τ ⊆ F ∗ } and once with T = { τ | τ ̸⊆ F ∗ } , we bound the ratio between B i and G i by

̸

̸

̸

We know that Pr τ [ τ ̸⊆ F ∗ ] is always smaller than ρ , so the numerator is ≤ ρ . A lower bound for the denominator is Pr τ [ new F ( τ ) = 0] -Pr τ [ τ ̸⊆ F ∗ ] . In addition, whenever the algorithm is inside the main loop, the safety is at least Pr τ [ new F ( τ ) = 0] = ∆( F ) > 2 ρ . Thus, the denominator is at least ρ. Hence, the RHS of (2) is less or equal to 1 , thus

̸

This completes the proof because we know that the algorithm does not add too many states outside of F ∗ . More precisely,

̸

10

algorithm

safe-zone

8

simulation

coverage percentage

compared to greedy

6

4

2

0

-2

-4

50

100

150

200

250

300

350

k

Sample Complexity. To discuss the sample complexity, we drop the assumption that the MC is known to the learner and use EstimateSafety Subroutine to approximate ∆( F ) . The number of calls to EstimateSafety is bounded by the size of the output set, F . Hence, this part of the sample complexity is bounded by | F | · N | F | and we show that is O ( k ∗ ϵ 2 log k ∗ ) . Another source of sampling is trajectories sampled for purposes of potentially adding them to F . Observe that at any iteration the set F has an escape probability of at least 2 ρ , and each trajectory that escapes F is accepted with a probability of at least 1 /H . This implies a lower bound for the probability that a random trajectory is accepted is 2 ρ/H . This gives an upper bound of 2 | F | ρ H for the expected sample complexity.

Amplification. Theorem 4.1 shows that if E holds, then the set size, | F | , is bounded in expectation by 2 k ∗ . As Pr[ E ] ≥ 1 -λ implies, from Markov's inequality, that the size (2 + δ ) k ∗ with small probability of about δ + λ = O ( δ ) . If we want to make sure that the actual size is at most (2 + δ ) k ∗ with high probability, we can repeat the process about Θ ( 1 δ ) times and take the smallest size set.

5 Empirical Demonstration

This section demonstrates the qualitative and quantitative performance of the described algorithms in the paper. For additional figures, we refer the reader to Appendix D.

The MDP. We focus on a grid of size N × N , for some parameter N . The agent starts off at mid-left state, (0 , ⌊ N 2 ⌋ ) and wishes to reach the (absorbing) goal state at ( N -1 , ⌊ N 2 ⌋ ) with a minimal number of steps. At each step, it can take one of four actions: up, down, right, and left by 1 grid square. With probability 0 . 9 , the intended action is performed and with probability 0 . 1 there is a drift down. The agent stops either way after H = 300 steps.

5.1 FINDING SAFEZONE vs. naive approaches

To compare the FINDING SAFEZONE Algorithm to the naive approaches presented in Section 3 we focus on the policy that first goes to the right and when it reaches the rightmost column, it goes up. The policy is described in the appendix, Figure 6(d). We take N = 30 and 2000 episodes.

We run the FINDING SAFEZONE , GREEDY, and SIMULATION algorithms, and estimate their coverage based on a test set containing 2000 random trajectories. Figure 1 depicts the trajectories coverage of each algorithm minus the coverage of the GREEDY algorithm. For a figure with absolute values, we refer the reader to Figure 6(b) in the appendix. We see that the new algorithm exhibits better performance compared to its competitors. We also see that taking less than 30% of the states ( k = 250 out of 900 states) is enough to get coverage of more than 80% of the trajectories.

Figures 5(a),5(b) show the sets found for k = 60 both by the Finding SAFEZONE Algorithm and GREEDY. We see that GREEDY chooses an unconnected set for this small k , leading to a coverage of 0 . While the new algorithm chooses a few states which consist of several trajectories, thus leading to a coverage larger than 0 .

6 Discussion and Open Problems

In this paper, we have introduced the SAFEZONE problem. We have shown that it is NP-hard, even when the model is known, and designed a nearly (2 ρ, 2 k ∗ ) approximation algorithm for the case where the model and policy are unknown to the algorithm. Beyond improving the approximation factors (or showing that it cannot be done unless P = NP ), a natural direction for future work is the following. Given a small ρ > 0 and a (known or unknown to the learner) MDP, find a policy with a small ρ -safe subset. If the value of the policy, when restricted to the SAFEZONE states, is close to the optimal value of the original MDP, restricting the policy to the SAFEZONE states generates a compact policy representation with a value close to optimal, and most trajectories are completed in the SAFEZONE.

References

Abel, D., Arumugam, D., Lehnert, L., & Littman, M. (2018). State abstractions for lifelong reinforcement learning. In International Conference on Machine Learning , (pp. 10-19).

Alharin, A., Doan, T.-N., & Sartipi, M. (2020). Reinforcement learning interpretation methods: A survey. IEEE Access , 8 , 171058-171077.

Amir, D., & Amir, O. (2018). Highlights: Summarizing agent behavior to people. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems , (pp. 11681176).

Amodei, D., Olah, C., Steinhardt, J., Christiano, P. F., Schulman, J., & Mané, D. (2016). Concrete problems in AI safety. CoRR , abs/1606.06565 .

Bennett, A., Misra, D., & Kallus, N. (2023). Provable safe reinforcement learning with binary feedback. In International Conference on Artificial Intelligence and Statistics , (pp. 10871-10900).

Blanc, G., Lange, J., & Tan, L.-Y. (2021). Provably efficient, succinct, and precise explanations. Advances in Neural Information Processing Systems , 34 .

Brandes, U., Holm, E., & Karrenbauer, A. (2016). Cliques in regular graphs and the core-periphery problem in social networks. In International Conference on Combinatorial Optimization and Applications , (pp. 175-186). Springer.

Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Comput. Surv. .

Dong, S., Van Roy, B., & Zhou, Z. (2019). Provably efficient reinforcement learning with aggregated states. arXiv preprint arXiv:1912.06366 .

Emam, Y., Glotfelter, P., Kira, Z., & Egerstedt, M. (2021). Safe model-based reinforcement learning using robust control barrier functions. CoRR , abs/2110.05415 .

Even-Dar, E., Mannor, S., & Mansour, Y. (2006). Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. JMLR , 7 , 1079-1105.

Even-Dar, E., & Mansour, Y. (2003). Approximate equivalence of markov decision processes. In COLT .

García, J., & Fernández, F. (2015). A comprehensive survey on safe reinforcement learning. J. Mach. Learn. Res. , 16 , 1437-1480.

Givan, R., Dean, T., & Greig, M. (2003). Equivalence notions and model minimization in markov decision processes. Artificial Intelligence .

HasanzadeZonuzy, A., Bura, A., Kalathil, D. M., & Shakkottai, S. (2021). Learning with safety constraints: Sample complexity of reinforcement learning for constrained mdps. In AAAI .

Hendrycks, D., Carlini, N., Schulman, J., & Steinhardt, J. (2021). Unsolved problems in ML safety. CoRR , abs/2109.13916 .

Hussein, A., Gaber, M. M., Elyan, E., & Jayne, C. (2017). Imitation learning: A survey of learning methods. ACM Comput. Surv. .

Koh, P. W., & Liang, P. (2017). Understanding black-box predictions via influence functions. In International Conference on Machine Learning , (pp. 1885-1894).

Li, J., Nagarajan, V., Plumb, G., & Talwalkar, A. (2020). A learning theoretic perspective on local explainability. arXiv preprint arXiv:2011.01205 .

Liu, Z., Cen, Z., Isenbaev, V ., Liu, W., Wu, Z. S., Li, B., & Zhao, D. (2022). Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, ICML 2022 , Proceedings of Machine Learning Research.

Molnar, C. (2019). Interpretable Machine Learning .

Moshkovitz, M., Yang, Y.-Y., & Chaudhuri, K. (2021). Connecting interpretability and robustness in decision trees through separation. arXiv preprint arXiv:2102.07048 .

Pfrommer, S., Gautam, T., Zhou, A., & Sojoudi, S. (2021). Safe reinforcement learning with chance-constrained model predictive control. CoRR , abs/2112.13941 .

Prajapat, M., Turchetta, M., Zeilinger, M. N., & Krause, A. (2022). Near-optimal multi-agent learning for safe coverage control. In NeurIPS .

Puterman, M. L. (1994). Markov decision processes: Discrete stochastic dynamic programming. In Wiley Series in Probability and Statistics .

Sootla, A., Cowen-Rivers, A. I., Jafferjee, T., Wang, Z., Mguni, D. H., Wang, J., & Ammar, H. (2022). Saute RL: Almost surely safe reinforcement learning using state augmentation. In Proceedings of the 39th International Conference on Machine Learning , Proceedings of Machine Learning Research.

Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction . Cambridge, MA, USA: A Bradford Book.

Vazirani, V. V. (2001). Approximation Algorithms . Berlin, Heidelberg: Springer-Verlag.

Williamson, D. P., & Shmoys, D. B. (2011). The Design of Approximation Algorithms . Cambridge University Press.

Xu, W., Xu, K., Bastani, H., & Bastani, O. (2021). Safely bridging offline and online reinforcement learning. CoRR , abs/2110.13060 .

Yang, J., Chen, T., Payne, B., Guo, P., Zhang, Y ., & Guo, J. (2020). Generating routes for autonomous driving in vehicle-to-infrastructure communications. Digital Communications and Networks .

Supplementary Material

A Hardness

In this section, we show that SAFEZONE is NP-hard to solve, and this is why approximation is necessary. Moreover, SAFEZONE is hard even if the MC and the optimal ρ -safe size, k ∗ are known. Our starting point is the NP-hardness of regular cliques. The REGULARCLIQUE( G,k c ) problem gets as an input (i) a regular graph G with n nodes where each node has degree d , and (ii) an integer k c . It returns whether G contains a clique of size k c . Whenever G and k c are clear from the context we simply write REGULARCLIQUE. The following fact follows, e.g., from Brandes et al. (2016).

Fact A.1. REGULARCLIQUE is NP-hard.

Markov chain (random walk). Fix a graph G = ( V, E ) and a starting vertex v 0 ∈ V . The graph induces a Markov Chain (random walk) in the following way. The states of the process correspond to the vertices V in the graph G . The transition function is defined as P ( v | u ) = 1 d · 1 ( u,v ) ∈ E , where d is the degree of any node. The process starts from node v 0 and then proceeds according to the transition function P for H steps.

Reduction. To prove the hardness of SAFEZONE , we show how to solve REGULARCLIQUE given a solver to SAFEZONE. For each vertex v ∈ V , run an algorithm for SAFEZONE with horizon H = 2 , k = k c , and ρ = 1 -( k c -1 d ) 2 , and v as the starting state. If there is at least one run of the algorithm that returns YES, then the final answer is YES. Otherwise, the answer is NO. Note that this reduction is efficient.

Theorem A.2. For every graph G = ( V, E ) and an integer k c there exists a clique of size k c in G ⇐⇒ there exists v ∈ V such that SAFEZONE ( V, v 0 = v, P, k c , ρ ) returns YES.

Proof. (= ⇒ ) If there is a clique of size k c , then we can take the corresponding k states. The probability to remain in this subset is at least ( k -1 d ) 2 (remember that H = 2 ). Thus, an exact solver for SAFEZONE must return YES.

( ⇐ =) Suppose there is no clique of size k . Assume by contradiction that the reduction (algorithm) returns YES. Let s 0 be a vertex which was the starting state from the running instance which the YES came from and let ˆ F denote the output of SAFEZONE . We will show that the probability to remain in any subset of size k is smaller than ( k -1 d ) 2 .

Since there is no clique of size k in G , we know that ˆ F is not a clique. It therefore follows that there exists at least two vertexes, s a , s b ∈ V such that ( s a , s b ) / ∈ E .

We will now bound the probability of escape from state s 0 by exhaustion.

̸

If s 0 = s a , then

Hence

̸

̸

2. If s 0 = s a , then

Hence

Given an environment, a policy, and a SAFEZONE , one could compute exactly how much safe it is (see Appendix E for details), from which we deduce our following corollary.

Corollary A.3. SAFEZONE is NP-complete.

We note that for H = 1 , the GREEDY AT EACH STEP Algorithm is optimal.

B Proofs of Section 3

B.1 Greedy by Threshold Algorithm

A naive approach to the SAFEZONE problem is to return all states s ∈ S with probability p ( s ) ≥ β , for some parameter β > 0 , see Algorithm 3.

Algorithm 3 Greedy by Threshold

Lemma 3.1. For any ρ, β ∈ (0 , 1) , the GREEDY BY THRESHOLD ALGORITHM returns a set that is ( ρ + k ∗ β, H β ) -SAFEZONE . In particular, for β = ρ k ∗ , this set is ( 2 ρ, k ∗ H ρ ) -SAFEZONE .

Proof. There are at most H β states with probability p ( s ) ≥ β. Thus | F | ≤ H β .

Denote by F ∗ an optimal ( ρ, k ∗ ) -SAFEZONE set. By the law of total probability,

Looking at the R.H.S of the inequality, the left term is smaller than ρ by the definition of SAFEZONE. The right term is equal to the probability of reaching a state in F ∗ that its probability is smaller than β , i.e., a state in F ∗ \ F.

Using union bound, this can be bounded by k ∗ β.

Lemma B.1. For every ρ ∈ (0 , 1 / 2 ) , H ∈ N , there exists an MDP and a minimal integer k such that the MDP has a ( ρ, k ) -SAFEZONE , but for β = ρ/k GREEDY BY THRESHOLD Algorithm returns F with escape probability ≤ 2 ρ and of size | F | = Ω( H/β ) .

Proof. Fix ρ ∈ (0 , 1) . For ease of the presentation, we will assume that 1 -ρ β is an integer (if not, it should be rounded to the nearest integer). Define A to contain 1 -ρ β · H states, B to contain k -1 states, and S = { s 0 } ∪ A ∪ B. Consider the following MDP with states S and starting state s 0 . The transition function is defined as follows:

Thus,

For every i ∈ A , Pr[ s A 1 ,i | s 0 ] = β and for every j ∈ [ H -1] , Pr[ s A j +1 ,i | s A j,i ] = 1 .

For s ∈ B , Pr[ s | s 0 ] = 1 -ρ k -1

For s ∈ B , Pr[ s | s ] = 1

The MDP is illustrated in Figure 2. Clearly, { s 0 } ∪ B is a ( ρ, k ) -SAFEZONE . In addition, GREEDY BY THRESHOLD ALGORITHM returns the set of all states, as for every state s ∈ A we have that p ( s ) = β , p ( s 0 ) = 1 > ρ ≥ β , and for every s ∈ B we have that p ( s ) = 1 -ρ k -1 > ρ k = β . Thus the size of the returned set is S is Ω( H/β ) , which completes the proof.

(p,k) - SafeZone

1

B

SA.1

S,1

sB

β

So

A

B.2 Simulation Algorithm

Algorithm 4 Simulation Algorithm

∗

return F

Lemma 3.2. Fix ρ, β ∈ (0 , 1) . With probability at least 0 . 99 , SIMULATION Algorithm returns a set that is ( ρ + k ∗ β, O ( k ∗ + ρH ln k ∗ β ) ) -SAFEZONE . In particular, for β = ρ k ∗ , this set is (2 ρ, O ( k ∗ H ln k ∗ )) -SAFEZONE .

Proof. Denote by F ∗ the optimal ( ρ, k ∗ ) -SAFEZONE set. By the law of total expectation, we can split E [ | F | ] into two parts, depending on whether trajectories are entirely in F ∗ or not:

Trajectories that are entirely in F ∗ contribute at most k ∗ states to F .

A trajectory that is not contained in F ∗ contributes at most H states to F .

We use Markov's inequality to get the desired bound on | F | .

For the safety, we first denote the set of all states in F ∗ with probability at least β as Γ = { s ∈ F ∗ | p ( s ) ≥ β } . We will show that with probability at least 0 . 9995 , it holds that Γ ⊆ F, which will prove our claim, similarly to Lemma 3.1.

For a fixed state s ∈ Γ , the probability that s / ∈ F is bounded by (1 -p ( s )) 1 β ln k ∗ 0 . 005 ≤ e -β β · ln k ∗ 0 . 005 = 0 . 005 k ∗ . Using union bound, the probability that there is a state s ∈ Γ which is not in F is bounded by k ∗ · 0 . 005 k ∗ = 0 . 005 .

In other words, with probability at least 0 . 995 , Γ ⊆ F , thus implementing the greedy approach in Algorithm 3 and proving that the probability that a random trajectory escapes F is bounded by ρ + k ∗ β.

Lemma B.2. For every ρ, γ ∈ (0 , 1) , H,k ∈ N , and β = ρ k , there is an integer r ∈ N and MDP with ( ρ, k ) -SAFEZONE , but with probability ≥ 1 -γ , SIMULATION algorithm returns F of size E [ | F | ] ≥ kH ln k with escape probability ∆( F ) = O ( ρ ) .

Proof. Fix ρ, γ ∈ (0 , 1) . Recall that m = 1 β ln k ∗ 0 . 005 and take r = ⌈ m 2 γ ⌉ . Define A to contain rH states, B to contain k -1 states, and S = { s 0 } ∪ A ∪ B.

Consider the following MDP with states S and starting state s 0 . The transition function is defined as follows:

For every i ∈ A , Pr[ s A 1 ,i | s 0 ] = ρ r and for every j ∈ [ H -1] , Pr[ s A j +1 ,i | s A j,i ] = 1 .

For s ∈ B , Pr[ s | s 0 ] = 1 -ρ k -1

For s ∈ B , Pr[ s | s ] = 1

The MDP is illustrated in Figure 3.

A

(p,k) - SafeZone

1

B

SA.1

SA1

p-r

So

d

k

B

SA,r

1

The set B ∪ { s 0 } is ρ -safe with k states.

We will show that:

After adding ≥ 1 β ln k = k ρ ln k random trajectories, with probability ≥ 1 -γ we have that | F | ≥ kH ln k .

After adding m random trajectories, we have that with high probability F ∗ ⊆ F , thus ∆( F ) ≤ Ω( ρ ) .

̸

To prove the first property, we claim that with probability ≥ 1 -γ , every time we add a trajectory τ such that τ ∩ A = ∅ , we add H new states.

Notice that if we ignore s 0 , trajectories in A are entirely unconnected, and each trajectory is chosen randomly with probability Pr[ s A 1 ,i | s 0 ] = ρ r . This yields that if s A 1 ,i / ∈ F , then s A j,i / ∈ F for every j ∈ [ H ] . As a result, every time we add a new s A 1 ,i to F , we add H -1 more states to F . Let N denote the number of trajectories sampled with states from A . The probability that their intersection contains only s 0 is

From the structure of the MDP, we have that E [ N ] = ρm . Therefore, with probability ≥ 1 -γ ,

The second property follows from Lemma 3.2.

B.3 Greedy at Each Step

Algorithm 5 Greedy at Each Step

Lemma 3.3. For any ρ ∈ (0 , 1) , if the MDP is layered, GREEDY AT EACH STEP ALGORITHM returns a set that is ( ρH,k ∗ ) -SAFEZONE .

Proof. Take a random trajectory τ = ( s 1 , s 2 , . . . ) . For every s i ∈ τ , the probability that s i / ∈ F is bounded by ρ , thus using union bound, the probability that τ has state s i such that s i / ∈ F is at most ρH .

The construction of F guarantees that F is the minimal subset of states such that for every i , the probability that s i is in the subset is at least 1 -ρ . Assume by contradiction that | F | > k ∗ . Then there is a time step i such that Pr[ s i ∈ F ∗ ] < 1 -ρ , which is a contradiction, since Pr[ τ ∈ F ∗ ] ≤ min i Pr[ s i ∈ F ∗ ] .

Lemma B.3. For any ρ ∈ (0 , 1) , there is an MDP and an integer k such that there is a ( ρ, k ) -SAFEZONE , but GREEDY AT EACH STEP Algorithm returns F with escape probability ∆( F ) ≥ Ω( Hρ ) .

Proof. Fix ρ ∈ (0 , 1) and take k = 3 H +1 .

Consider the MDP illustrated in Figure 4. The set { s 0 } ∪ { s i 1 } i ∪ { s i 2 } i ∪ { s i 3 } i form a ( ρ, 3 H + 1) -SAFEZONE .

We will prove by induction that the for every time i ,

p ( s i 1 ) = 1 -2 ρ ,

p ( s i 2 ) = p ( s i 3 ) = p ( s i 4 ) = ρ 2 , and

For every j ∈ { 5 , . . . , k +4 } , p ( s i j ) = ρ 2 k .

So

p/2k

k

p/2

1-2p

p/2

s1

s2

s3

s4

s5

Sk+4

1

1-p

1-2p

1/2

1/2k

s²

s2

s3

s2

SK+4

1/2k

1/2

2p

S1

H-1

S3

H-1

$4

H-1

S'5

SH-1

Sk+4

H-1

It is easy to see that the two properties hold for i = 1 . For i > 1 ,

Similarly, p ( s i 3 ) = ρ 2 .

For every j ∈ { 5 , . . . , k +4 } ,

The algorithm might return { s 0 } ∪ { s i 1 } i ∪ { s i 2 } i ∪ { s i 4 } i , i.e., instead of taking ∪ i { s i 3 } i it takes ∪ i { s i 4 } i . Finally, the observation ∆( { s 0 } ∪ { s i 1 } i ∪ { s i 2 } i ∪ { s i 4 } i ) ≥ ρH 4 completes the proof.

C Proofs of Section 4

For convenience, we state here Hoeffding's inequality.

Lemma C.1. [Hoeffding's Inequality] Let y 1 , . . . , y N be independent random variables such that y i ∈ [ a, b ] for every y i with probability 1 . Then, for any ϵ > 0 ,

C.1 Proof of Theorem 4.1

In this section, we provide a complete proof for Theorem 4.1. Throughout the section, we define a few terms and notions. We will start with proving guarantees regarding a single iteration of the while-loop.

Recall that F ∗ denotes a minimal ρ -safe set (of size k ∗ ). If there are multiple optimal solutions, choose one arbitrarily. For the convince of analysis, we denote the values of the algorithm variables at the end of each iteration i of the while-loop by τ i , F i , accept i . Let j ( i ) denote the value of variable j during the i -th call to EstimateSafety Subroutine. In addition, let N i denote the number of trajectories sampled for the j -th time of calling Subroutine EstimateSafety , i.e., N i = 1 2 ϵ 2 ln 2 λ j ( i ) for j ( i ) ≤ i .

For ease of presentation, we recall some of the definitions from the proof technique description. We say that a trajectory τ is good if all the states in τ are in F ∗ and bad if it escapes it. I.e., a trajectory is good if τ ⊆ F ∗ and bad if τ ̸⊆ F ∗ . Additionally, we say that a state s ∈ S is good if it is in F ∗ and bad otherwise. Namely, a state s is good if s ∈ F ∗ and bad if s / ∈ F ∗ . Let G i ( F i -1 ) and B i ( F i -1 ) be the number of good and bad states added to F i -1 in iteration i , respectively (notice that G i ( F i -1 ) and B i ( F i -1 ) are random variables that depends on F i -1 ). For short, whenever it is clear from the context, we write G i and B i respectively.

The following lemma bounds the error in approximating the escape probability.

Lemma C.2. Let F i -1 ⊆ S be a subset of of states and ϵ, λ j > 0 be some parameters. Let S i be a sample of N i ≥ 1 2 ϵ 2 ln 2 λ j ( i ) i.i.d. random trajectories. Then,

Where the last probability is over all the samples S i made by EstimateSafety Subroutine.

Proof. The first part follows directly from Hoeffding's inequality by taking y i = ■ [ τ ̸⊆ F ] . Assigning λ j = 3 λ 2( πj ) 2 and applying union bound, we get

The inequality marked by ( ∗ ) follows from the fact that ∆( F ) is estimated once for every time j increases.

We define the event that EstimateSafety always provides good estimations by

By the above, we have that Pr[ E ] ≥ 1 -λ/ 4 .

In the following lemma we assume that if the current escape probability is at least 2 ρ , then the fraction of bad trajectories that escape F i -1 is bounded from above by the fraction of good trajectories that escape F i -1 .

Lemma C.3. Let ρ > 0 and assume that ∆( F i -1 ) ≥ 2 ρ . Then,

̸

where the probabilities are over random trajectories.

Also, as λ j = 3 λ 2( πj ) 2 ,

̸

̸

̸

Proof. To prove the lemma, we will bound the probability Pr τ [ new F i -1 ( τ ) = 0 ∧ τ ̸⊆ F ∗ ] from above and the probability Pr τ [ new F i -1 ( τ ) = 0 ∧ τ ⊆ F ∗ ] from below. Since ∆( F ∗ ) ≤ ρ ,

̸

The assumption ∆( F i -1 ) ≥ 2 ρ implies that

̸

hence

̸

Putting (4) and (5) together yields the statement.

Now, as long as the algorithm is inside the while-loop (i.e., the escape probability holds ̂ ∆( F ) > 2 ρ + ϵ ), it follows that ∆( F ) ≥ 2 ρ with high probability from Lemma C.2. Combining it with Lemma C.3 would yield that with high probability over a random trajectory, if the trajectory escapes F then in expectation, it is at least as likely to be good as it is to be bad.

We move on to show the main ingredient of the proof, namely that for any iteration, with high probability, the expected number of good states added to the current set F is larger or equal to the expected number of bad states.

For every iteration i in which we sample τ i both G i and B i depends on the following:

The realizations of the sampled trajectory, τ i , and in particular on new F i -1 ( τ i ) .

The probability of adding it to F , i.e., 1 /new F i -1 ( τ i ) .

Next, we prove Equation (3).

Lemma C.4. Assume event E holds. Thus, for all iterations i inside the while-loop we have

where the expectation is over the trajectory τ that is sampled from the MC dynamics and added to F i -1 according to Q F i -1 .

Proof. Since event E holds, we have that ∆( F i -1 ) ≥ 2 ρ as long as we do not terminate in iteration i . We can use it to bound E τ [ B i | F i -1 ] by

̸

̸

Theorem 4.1. Given ρ, ϵ, λ ∈ (0 , 1) , FINDING SAFEZONE Algorithm returns a subset F ⊆ S such that:

The escape probability is bounded from above by ∆( F ) ≤ 2 ρ +2 ϵ , with probability 1 -λ .

The expected size of F given E is bounded by E [ | F | | E ] ≤ 2 k ∗ .

The sample complexity of the algorithm is bounded by O ( k ∗ λϵ 2 ln k ∗ λ + Hk ∗ ρλ ) , and the running time is bounded by O ( Hk ∗ λϵ 2 ln k ∗ λ + H 2 k ∗ ρλ ) , with probability 1 -λ .

̸

̸

̸

̸

Proof. Assume that the event E holds, and recall that

We start with the first clause. Since the event E holds, Lemma C.2 in particular implies that ∆( F ) ≤ 2 ρ +2 ϵ , hence the first clause holds. For second clause, we will bound E [ | F | | E ] from above by 2 k ∗ . Since E holds, we have that ∆( F i -1 ) ≥ 2 ρ , for every i inside the while-loop, thus Lemma C.4 yields

where the last inequality follows from the definition of G i , as ∑ i G i ≤ | F ∗ | = k ∗ .

We continue with the third clause of the theorem. Let M denote the sample complexity of the algorithm, namely M = M F + M E where M F is the expected total number of trajectories sampled within the FINDING SAFEZONE Algorithm (without the samples made by EstimateSafety Subroutine) and M E is the total number of trajectories sampled using EstimateSafety . We will bound each term separately.

Since E holds, whenever we are inside the while-loop, ∆( F i ) ≥ 2 ρ , which implies that it takes at most 1 / 2 ρ trajectories in expectation to sample a trajectory that escapes F i , and such trajectory is accepted with probability at least 1 /H .

Thus, from Wald's identity, it follows that

From Markov's inequality on the above inequality, with probability at least 1 -λ 4 ,

Moving on to bound M E . Since E holds, it follows from Equation (7) and Markov's inequality that

If | F | ≤ 8 k ∗ λ , the number of calls for Subroutine EstimateSafety is also bounded by 8 πk ∗ /λ (we only call EstimateSafety after we added states to F ). It also implies that 3 λ 3 2(8 πk ∗ ) 2 ≤ λ j for every j ≥ 1 . Thus, if | F | ≤ 8 k ∗ λ ,

Combining the above with Equation (9), we get

As M = M F + M E , union bound over Equation (6), Equation (8) and Equation (10) implies that with probability ≥ 1 -3 λ/ 4 > 1 -λ ,

This implies that

For each trajectory we sample we run in time O ( H ) , e.g., by using a lookup table for maintaining the current set F . Consequently, if the event in Equation (11) holds then the running time of the algorithm is bounded by

Overall, all the clauses in the lemma hold with probability ≥ 1 -λ .

C.2 Proof of Theorem 4.2

Theorem 4.2. (main theorem) Given ϵ, ρ, δ > 0 , if we run FINDING SAFEZONE for Θ( 1 δ ) times and return the smallest output set, F ⊆ S , then with probability ≥ 0 . 99

The escape probability is bounded by ∆( F ) ≤ 2 ρ +2 ϵ.

The size of F is bounded from above by | F | ≤ (2 + δ ) k ∗ .

The total sample complexity and running time are bounded by O ( k ∗ δ 2 ϵ 2 ln k ∗ δ + Hk ∗ ρδ 2 ) , and O ( Hk ∗ δ 2 ϵ 2 ln k ∗ δ + H 2 k ∗ ρδ 2 ) , respectively.

Proof. Assume we run FINDING SAFEZONE Algorithm for m = 2 ln 300 δ times and denote each algorithm output by F i . Return the smallest set F = argmin F i | F i | .

It follows from Theorem 4.1 that for every λ ∈ (0 , 1) , each F i is of expected size E [ | F i | ] ≤ 2 k ∗ , and is (2 ρ +2 ϵ ) -safe with probability ≥ 1 -λ. Choosing λ = 0 . 01 3 m implies

In addition, from Markov's inequality it follows that for every δ > 0 ,

From the independence of the algorithm runs, for m = 2 ln 300 δ ,

As for the sample complexity, let M i denote the (random) sample complexity of the i -th run, and let us denote

Hence

From Theorem 4.1, M i > ¯ M with probability < λ .

By taking the union bound on the sample complexity bound per one run, we get

Where the last inequality follows from Theorem 4.1, and λ = 0 . 01 3 m .

Assigning m = 2 ln 300 δ and λ = 0 . 01 3 m = 0 . 01 δ 6 ln 300 , we get that with probability ≥ 1 -0 . 01 3 ,

Since the algorithm runs in time O ( H ) for every trajectory sampled, if the sample complexity is bounded by the above term, then the total running time is bounded by O ( Hk ∗ δ 2 ϵ 2 ln Hk ∗ δ + Hk ∗ ρδ 2 ) .

Finally, from union bound over Equation (12), Equation (13) and Equation (14) all the theorem properties hold with probability ≥ 0 . 99 .

D Additional Figures for Section 5

D.1 Comparing SAFEZONE of two policies

In this section, we empirically explore the SAFEZONE of two different policies within the same MDP. The first policy, described in the previous section, first goes right and then to the middle, and the second policy first goes to the middle and then goes right. See Figure 6 in the appendix. These seemingly similar policies induce very different SAFEZONES as can be seen in Figure 8 which depicts the number of visits in each state. It shows that the second policy requires fewer states to achieve the same level of safety, even though in terms of minimizing the number of steps to get to the goal state it is outperformed by the first policy (intuitively, the second policy has more fail attempts to go up in expectation since the lowest row of the grid cannot get worst). In Figure 7 we see that already with 14% of the states, all three algorithms achieve trajectory coverage of more than 85% .

Figure 6 depicts the two policies discussed in the paper when N = 7 .

Figure 7 depicts coverage percentage for the different algorithms discussed in the paper when applied to the second policy.

(b) %Coverage: absolute values.

(c) Total number of visits at each state from 2000 episodes. Zero visits in grey.

10

algorithm

safe-zone

8

simulation

coverage percentage

compared to greedy

6

4

2

0

-2

50

100

150

200

250

300

350

k

100

algorithm

safezone

coverage percentage

80

simulation

greedy

60

40

20

0

50

100

150

200

250

300

350

k

3

10

10

2

10

1

10

0

Figure 8 shows the visits of the policies described in the main paper for N = 30 . It is immediately clear that the SAFEZONE of the two policies are fundamentally different. As mentioned, this affects their SAFEZONE sizes. Namely, when trying to go right from a current state in the lowest row it is impossible to get to a square that is lower than that, and the first policy takes advantage of this. In contrast, the second policy keeps trying to go up from the lowest row, which implies that in expectation it goes down more times compared to the first.

E Exact Computation

In this section, we assume that the transition function is known to the algorithm and show how to compute ∆( F ) .

→

(d) Go right and then to the goal state.

→

→

→

↑

(e) Go to the middle and then right.

̸

Given a Markov Chain ⟨S , P, s 0 ⟩ and a set F ⊆ S we create a new Markov Chain ⟨S ′ , P ′ , s 0 ⟩ as follows. We add a new state s sink ̸∈ S , and set S ′ = F ∪{ s sink } . For each transition from a state s ∈ F to a state s ′ ̸∈ F we modify and make the transition in P ′ to the sink s sink . In P ′ , when we are in s sink we always stay in s sink . More formally: (1) if s, s ′ ∈ F then P ′ ( s ′ | s ) = P ( s ′ | s ) , (2) we set P ′ ( s sink | s ) = ∑ s ′ ̸∈ F P ( s ′ | s ) and (3) P ′ ( s sink | s sink ) = 1 and P ′ ( s | s sink ) = 0 for s = s sink .

100

algorithm

safezone

80

simulation

percentage

greedy

60

coveragep

40

20

0

40

60

80

100

120

140

k

Now we claim that ∆( F ) = Pr P ′ [ s H = s sink ] , since any trajectory that reaches a state not in F will reach the sink in P ′ and stay there. We can compute Pr P ′ [ s H = s sink ] using standard dynamics programming.

The running time of constructing ⟨S ′ , P ′ , s 0 ⟩ is O ( |S| 2 ) . Computing the probability of Pr P ′ [ s H = s sink ] takes O ( H |S| 2 ) . Therefore we have established the following.

Lemma E.1. Given a Markov chain ⟨S , P, s 0 ⟩ and a set F ⊆ S we can compute ∆( F ) in time O ( |S| 2 H ) .

Note that the above lemma implements an exact version of the EstimateSafety Subroutine.

3

10

2

10

1

10

0

10

3

10

2

10

1

10

0

10

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work introduces the SafeZone problem for safe reinforcement learning (RL). Instead of learning for the optimal policy under the constrained Markov Decision Process (MDP) as many traditional methods deal with a safe RL problem, the work attempts to search for a SafeZone, a subset of state space that is policy-dependent, balancing between minimizing the number of states contained in the subset and reducing the probability that a random trajectory goes out of the subset. If the probability of escaping the SafeZone is low with a randomly sampled trajectory, it is considered safe. The paper also proposes a new algorithm for detecting the SafeZone. The algorithm uses rejection sampling to decide whether a new random trajectory should be added to the SafeZone and keep updating the safety estimation once a new trajectory is added.\n\nSTRENGTHS:\n1. The work proposes a new way of dealing with the safe reinforcement learning (RL) problem. In traditional safe RL, the problem is usually formulated as a constrained MDP, while in this work, the agent is trained to find the SafeZone for a given policy.\n\n2. The paper detailedly explains the formalization of the new problem setting and compares the new setting with the constrained MDP, it helps with understanding. \n\n3. The paper provides both theoretical analysis and empirical evidence, making the results more sound. Though solving the SafeZone problem is NP-Hard, the paper derives an algorithm and provides empirical results. These results suggest that the idea introduced in the paper is practical.\n\nWEAKNESSES:\nThe main concern comes from the choice of baseline algorithms and the experiment design. The paper empirically tested methods on a $N \\times N$ grid, checking the quality of the SafeZone given by the new algorithm and baselines. A SafeZone with a smaller size but a larger coverage percentage on the trajectory (lower escape probability) is considered better. Although the new method is empirically shown to be better than the baseline algorithms regarding the above standard, I doubt whether the proposed algorithm can be sufficiently tested by the reported experiment, or, whether the advantage and shortage of SafeZone problem setting can be seen from this experiment.\n\nIn the introduction section, the paper points out that a new method is proposed for safe RL. One common idea used by the community is to formulate the problem as constrained MDP to solve the safe RL problem, as the paper points out. The proposed SafeZone method performs as the other path for solving safe RL, besides using constrained MDP. So, one thing remaining unclear is whether SafeZone is a better way of solving the safe RL problem than constrained MDP. \n\nThe paper focuses on comparing the quality of SafeZone detected by the proposed algorithm and baselines, but does not provide any empirical evidence about whether a good SafeZone output helps with improving the learning performance in safe RL problems, or whether a SafeZone method performs better than constrained MDP tasks. As SafeZone is a newly introduced problem setting and the proposed algorithm directly targets searching for a better SafeZone, I am not surprised that the new method outperformed baselines regarding the SafeZone quality. But the lack of learning performance in safe RL tasks makes it hard to say if detecting a SafeZone is a better way of solving the safe RL problem. It could be more convincing if the learning performance in safe RL tasks could be checked and empirically indicate that the SafeZone idea can learn safe RL tasks more efficiently than using the constrained MDP.\n\nQUESTIONS:\nMy question is related to the concerns listed above. I would appreciate it if the authors could discuss whether the proposed method has the potential to outperform the constrained MDP methods, regarding the learning efficiency.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the SafeZone problem: Given an MDP and a policy, find the smallest set of states such that the probability of leaving this set of states (the escape probability) lies below a given threshold using trajectory samples. The authors provide various examples of applications of the SafeZone problem, such as imitation learning with compact policy representation and post hoc explainability of RL.\nThe authors provide proof that the SafeZone problem is NP-complete. The adjusted problem solved in this paper is to instead find a SafeZone with a larger but minimal escape probability than the threshold and a larger but minimal cardinality than the true optimal cardinality given the threshold.\nThree naive approaches are discussed, and one approximation algorithm for finding SafeZones. The authors provide upper bounds on each approach's escape probability and sample complexity. They give provable guarantees on their approximation algorithm. Finally, the paper compares three of the four approaches based on a grid world problem.\n\nPart of this work has been previously presented at the NeurIPS 2022 TSRML Workshop. The work did not appear in any proceedings, journals, or books, so according to NeurIPS' call for papers, this is not considered a dual submission.\n\nSTRENGTHS:\nThe paper introduces a novel problem and gives extensive examples of situations where a solution to this problem could be useful. It shows the need for approximation by proving that the SafeZone problem is NP-complete, even if the induced MC and the minimal cardinality (of the corresponding escape probability threshold) are known.\n\nThe paper analyses four algorithms for finding SafeZones, provides upper bounds on the escape probability and sample complexity, and indicates the limitations of the three naive approaches.\n\nThe paper suggests an interesting future work direction, where the problem is moved from finding SafeZones of an induced MC to finding policies of an MDP with small SafeZones given an escape probability threshold.\n\nWEAKNESSES:\nThe paper is challenging to follow because of two main reasons: a lot of information is only available in the appendices, and few (intuitive) examples are used after introducing the problem.\n\nThe first reason, limited information in the paper itself and reliance on the appendices can, for example, be seen in Section 3, where Appendix B contains the actual algorithms, MDP examples, and proof outlines, and in Section 5, which only contains one figure, with the rest all placed in Appendix E.\nFurthermore, what information is available in the appendices is also not always clear. For example, it is not mentioned in Appendix A that the proof of Theorem A.2 is given in Appendix D. Nor is it mentioned in Section 5 that more figures regarding the empirical demonstration are available in Appendix E, only four subfigures are referenced.\nMoreover, the information about the empirical demonstration section is incomplete. The escape probability threshold is not given for computing the results and the number of repetitions. Also, the paper only compares three of the discussed methods without mentioning the exclusion of the Greedy by Threshold algorithm.\n\nThe second reason, the lack of intuitive examples, concerns the sections after the introduction. The paper gives extensive and intuitive examples in the introduction but does not use these throughout the rest of the paper. Only the autonomous vehicle example is referenced once in Section 2.\nAlthough the paper proves why an approximation of the SafeZone problem is necessary, they give no argumentation as to why an almost 2 approximation is sufficiently tight to have a practical use still. An intuitive example where the size of the SafeZone is compared to an optimal SafeZone could illustrate this usefulness.\nAlso, the problem used in the empirical demonstration does not provide any intuition regarding the applications of the SafeZone problem, as given in the introduction. Some discussion on how this relates would be helpful.\n\nTypo in Section 3, line 208. theses -> these\nIncorrect reference in Section 5, line 355. Section 5 -> Figure 1.\n\nQUESTIONS:\nWhy is Greedy by Threshold excluded from the empirical demonstration?\n\nHow should an escape probability threshold be chosen?\n\nWhat escape probability threshold value was used for the empirical demonstrations?\n\nHow often were the experiments of the empirical demonstration repeated?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new problem that involves finding a SafeZone: given the possibility to interact with an MDP using a fixed policy, the learner as to find a subset of states F of minimal size s.t. the probability that a trajectory visits a state outside F is at most a given parameter \\rho. They show that the problem is NP-hard even when the transition matrix is known. They propose an approximate algorithm that finds near-optimal SafeZones with polynomial sample complexity (in the relevant variables). The approach is numerically tested on a toy problem.\n\nSTRENGTHS:\n1. The paper introduces an interesting problem (though it may lack a bit of \"concrete motivation\", see below). The problem is definitely challenging from a theoretical perspective.\n2. While I haven't checked the proofs, the results seem sound\n3. The authors managed to derive a computationally-efficient algorithm computing near-optimal Safe Zones even though the base problem is NP-hard\n\nWEAKNESSES:\n1. Although the introduction presents some examples, I still wasn't fully convinced about the motivations behind studying this peculiar setting. In particular, it is still unclear to me what problems the proposed algorithm (or in general finding safe zones) enable solving. It would be good to provide a more \"concrete\" example, ideally some numerical results on a real-world problem (or a simulated variant) where it is clear that finding safe zones is the right thing to do\n2. Moreover, even in the context of the examples given in the introduction, the assumption that we can only interact with the MDP with a single policy seems a bit limiting. In those settings, it seems reasonable to observe trajectories collected by multiple policies, mostly because multiple agents are interacting with the environment and each displays learning behavior (so changing policies). How to extend the proposed setting and algorithms to such a context is not clear\n3. I found the core part of the paper (Sec. 3,4) a bit hard to read. In particular, if my understanding is correct, Sec. 3 tries to build some intuition which is later useful to explain the main algorithm in Sec. 4. However, that didn't really work for me: Sec. 4 starts with \"In this section, we suggest a new algorithm that builds upon and improves the added trajectory selection of the SIMULATION Algorithm\", but the SIMULATION algorithm was not explained at all in Sec. 3. The rest of Sec. 4, especially the first paragraphs, are also hard to follow, while Sec. 3 seems mostly to report a bunch of technical results rather than intuitions (so I did not really feel \"a gentle start\" to me).\n4. It is not clear how good the sample complexity of the main algorithm (Th. 4.2) is, especially in terms of dependences on the main variables (k^\\star, \\delta). It would be good to add some discussion about this. Do the authors think that the current dependences are optimal or improvable?\n5. The experiments are conducted on a very toy and low-scale domain. I would have like to see larger domains and also a comparison of the computational requirements (time complexity) of the different algorithms.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nI did not found any discussion about limitations", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper proposes an innovative definition called the SAFEZONE and uses it to describe the escape probability of the sampled trajectory. This paper analyzes several naïve algorithms and proposes an algorithm to overcome the weaknesses in naïve algorithms, especially when considering the size of safety states. Finally, numerical experiments are conducted to evaluate the algorithm's performance compared to mentioned naïve algorithms.

STRENGTHS:
I think the paper is easy to follow.

Strengths:

1. The paper introduces the definition called the SAFEZONE to find the subset of ‘safe states’, i.e., the set that has low escape probability $\rho$ and a small size $k$.
2. Three naïve algorithms are clearly discussed with solid theoretical analysis and specific MDPs examples in the appendix.
3. A new approach for solving an approximate SAFEZONE is proposed with complete analysis, and numerical comparisons show the algorithm's performance.

WEAKNESSES:
No obvious limitations, but this paper is not written well. The motivation for this work should be discussed more, and it would be helpful if the author discuss more concurrent works.

QUESTIONS:
No more questions.

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-17