Title: Compositional Policy Learning in Stochastic Control Systems with Formal Guarantees
Venue: NeurIPS 2023
Accepted: yes
Keywords: Verification

Compositional learning
Abstract: Reinforcement learning has shown promising results in learning neural network policies for complicated control tasks. However, the lack of formal guarantees about the behavior of such policies remains an impediment to their deployment. We propose a novel method for learning a composition of neural network policies in stochastic environments, along with a formal certificate which guarantees that a specification over the policy's behavior is satisfied with the desired probability. Unlike prior work on verifiable RL, our approach leverages the compositional nature of logical specifications provided in SpectRL, to learn over graphs of probabilistic reach-avoid specifications. The formal guarantees are provided by learning neural network policies together with reach-avoid supermartingales (RASM) for the graph’s sub-tasks and then composing them into a global policy. We also derive a tighter lower bound compared to previous work on the probability of reach-avoidance implied by a RASM, which is required to find a compositional policy with an acceptable probabilistic threshold for complex tasks with multiple edge policies. We implement a prototype of our approach and evaluate it on a Stochastic Nine Rooms environment.

Compositional Policy Learning in Stochastic Control Systems with Formal Guarantees

Abstract

Reinforcement learning has shown promising results in learning neural network policies for complicated control tasks. However, the lack of formal guarantees about the behavior of such policies remains an impediment to their deployment. We propose a novel method for learning a composition of neural network policies in stochastic environments, along with a formal certificate which guarantees that a specification over the policy's behavior is satisfied with the desired probability. Unlike prior work on verifiable RL, our approach leverages the compositional nature of logical specifications provided in SPECTRL, to learn over graphs of probabilistic reach-avoid specifications. The formal guarantees are provided by learning neural network policies together with reach-avoid supermartingales (RASM) for the graph's sub-tasks and then composing them into a global policy. We also derive a tighter lower bound compared to previous work on the probability of reach-avoidance implied by a RASM, which is required to find a compositional policy with an acceptable probabilistic threshold for complex tasks with multiple edge policies. We implement a prototype of our approach and evaluate it on a Stochastic Nine Rooms environment.

1 Introduction

Reinforcement learning (RL) has achieved promising results in a variety of control tasks. However, the main objective of RL is to maximize expected reward [56] which does not provide guarantees of the system's safety. A more recent paradigm in safe RL considers constrained Markov decision processes (cMDPs) [5, 26, 3, 21], that in addition to a reward function are also equipped with a cost function. The goal in solving cMDPs is to maximize expected reward while keeping expected cost below some tolerable threshold. Recently, keeping cost below the threshold almost-surely and not only in expectation was considered in [52]. While these methods do enhance safety, they only empirically

try to minimize cost and do not provide guarantees on cost constraint satisfaction. The lack of formal guarantees raises a significant barrier to deployment of these methods in safety-critical applications, where the policies often need to encode complicated, long-horizon reasoning to accomplish a task in an environment in which unsafe behavior can lead to catastrophic consequences [6].

Recent work [13, 32, 33] has explored decomposing high-level logical specifications, belonging to different fragments of Linear Temporal Logic (LTL) [44], into simpler logical specification tasks that RL algorithms can solve more easily. In particular, these methods first decompose the high-level logical specification into a number of simpler logical specifications, then learn policies for these simpler tasks and finally compose the learned policies for each task into a global policy that solves the high-level control problem at hand. The simpler control tasks are solved by designing reward functions which faithfully encode the simpler logical specifications obtained by the decomposition and using an off-the-shelf RL algorithm to solve them.

While these methods present significant advances in deploying RL algorithms for solving complex logical specification tasks, their key limitation is that they also do not provide formal guarantees on the probability of satisfying the logical specification which are imperative for safety-critical applications. This is because reward functions often only 'approximate' logical specification objectives, and furthermore RL algorithms are in general not guaranteed to converge to optimal policies over continuous state spaces. Recently, [30, 62] have proposed verification procedures for formally verifying a composition of neural network policies with respect to a given system dynamics model. However, these works assume that the underlying control systems have deterministic dynamics . This does not allow modelling stochastic disturbances and uncertainty in the underlying system dynamics. Furthermore, [62] assumes that the underlying control system has a linear dynamics function whereas many control tasks have non-linear dynamics .

In this work we propose CLAPS ( C ompositional L e a rning for P robabilistic S pecifications), a new compositional algorithm for solving high-level logical specification tasks with formal guarantees on the probability of the specification being satisfied. We present a method for learning and verifying a composition of neural network policies learned via RL algorithms, which is applicable to stochastic control systems that may be defined via non-linear dynamics functions . Our method only requires that the underlying state space of the system is compact (i.e. closed and bounded) and that the dynamics function of the system is Lipschitz continuous. In terms of the expressiveness of logical specifications, CLAPS considers the SPECTRL specification language [32]. SPECTRL contains all logical specifications that can be obtained by sequential and disjunctive compositions of reach-avoid specifications. Given a target and an unsafe region, a reach-avoid specification requires that the system reaches the target region while avoiding the unsafe region.

Our method learns a policy along with a formal certificate which guarantees that a specification is satisfied with the desired probability. It consists of three key ingredients - (1) high-level planning on a directed acyclic graph (DAG) that decomposes the complex logical specification task into sequentially or disjunctively composed low-level reach-avoid tasks, (2) learning policies that solve low-level reach-avoid tasks with formal guarantees on the probability of satisfying reach-avoidance, and (3) composing the low-level reach-avoid policies into a global policy for the high-level task by traversing edges in the DAG. This yields a fully compositional algorithm which only requires solving control problems with reach-avoid constraints, while performing high-level planning.

To solve the low-level reach-avoid tasks, we leverage and build upon the recent learning algorithm for stochastic control under reach-avoid constraints [68]. This work achieves formal guarantees on the probability of reach-avoidance by learning a policy together with a reach-avoid supermartingale (RASM), which serves as a formal certificate of probabilistic reach-avoidance. When composing such guarantees for a global probabilistic specification we encounter two fundamental challenges, (i) reducing the certification of unnecessary individual reach-avoid constraints that are not critical for global satisfaction, and (ii) reducing the violation probabilities for individual reach-avoid constraints to improve the possibility of satisfaction of the composed policy. We overcome the first challenge by leveraging the DAG decomposition of complex specifications and by performing a forward pass on the DAG which learns edge policies on-demand , and the second one by showing that RASMs prove a strictly tighter lower bound on the probability of reach-avoidance than the bound that was originally proved in [68]. Our novel bound multiplies the bound of [68] by an exponential asymptotic decay term. This is particularly important when composing a large number of edge policies for complex

objectives. Furthermore, our new bound is of independent interest and it advances state of the art in stochastic control under reach-avoid constraints.

Contributions Our contributions can be summarized as follows:

We prove that RASMs imply a strictly tighter lower bound on the probability of reachavoidance compared to the bound originally proved in [68] (Section 4).

We propose a novel method for learning and verifying a composition of neural network policies in stochastic control systems learned in the RL paradigm, which provides formal guarantees on the probability of satisfaction of SPECTRL specifications (Section 5).

We implement a prototype of our approach and evaluate it on the Stochastic Nine Rooms environment, obtained by injecting stochastic disturbances to the environment of [33]. Our experiments demonstrate both the necessity of decomposing high-level logical specifications into simpler control tasks as well as the ability of our algorithm to solve complex specification tasks with formal guarantees (Section 6).

2 Related Work

Learning from logical specifications Using high-level specifications with RL algorithms has been explored in [13, 32, 33, 39, 43, 60] among others. These approaches typically use the specification to guide the reward objective used by a RL algorithm to learn a policy but usually lack a formal guarantee on the satisfaction of the specification. Approaches that provide guarantees on the probability of satisfaction in finite state MDPs have been proposed in [12, 28, 29]. In contrast, we consider stochastic systems with continuous state spaces.

Control with reachability and safety guarantees Several works propose model-based methods for learning control policies that provide formal guarantees with respect to reachability, safety and reach-avoid specifications. The methods of [2, 16, 17, 48, 55, 1] consider deterministic systems and learn a policy together with a Lyapunov or barrier function that guarantees reachability or safety. Control of finite-time horizon stochastic systems with reach-avoid guarantees has been considered in [14, 36, 53, 61]. These methods compute a finite-state MDP abstraction of the system and solve the constrained control problem for the MDP. The works of [9, 10] consider infinite-time horizon stochastic systems with linear dynamics and propose abstraction-based methods that provide probably approximately correct (PAC) guarantees. Control of polynomial stochastic systems via stochastic barrier functions and convex optimization was considered in [46, 49, 65, 42]. Learning-based methods providing formal guarantees for infinite time horizon stochastic systems that are not necessarily polynomial have been proposed in [38, 68, 41, 20]. The work [8] proposes a learning-based method for providing formal guarantees on probability 1 stabilization. To the best of our knowledge, the compositionality of our CLAPS makes it the first method that provides formal guarantees for infinite time horizon stochastic systems with respect to more expressive specifications.

Safe exploration RL algorithms fundamentally depend on exploration in order to learn high performing actions. A common approach to safe exploration RL is to limit exploration within a high probability safety region which is computed by estimating uncertainty bounds. Existing works achieve this by using Gaussian Processes [11, 34, 59], linearized models [23], robust regression [40], Bayesian neural networks [37], shielding [4, 7, 25, 31] and state augmentation [51].

Model-free methods for stochastic systems The recent work of [63] proposed a model-free approach to learning a policy together with a stochastic barrier function, towards enhancing probabilistic safety of the learned policy. Unlike our approach, this method does not formally verify the correctness of the stochastic barrier function hence does not provide any guarantees on safety probability. However, it does not assume the knowledge of the system environment and is applicable in a model-free fashion. Several model-free methods for learning policies and certificate functions in deterministic systems have also been considered, see [24] for a survey.

Statistical methods Statistical methods [67] provide an effective approach to estimating the probability of a specification being satisfied, when satisfaction and violation of the specification can be witnessed via finite execution traces. Consequently, such methods often assume a finite-horizon or the existence of a finite threshold such that the infinite-horizon behavior is captured by traces within that threshold. In environments where these assumptions do not hold, the violation of a reach-avoid (or, more generally, any SPECTRL ) specification can be witnessed only via infinite traces and

hence statistical methods are usually not applicable. In this work, we consider formal verification of infinite-time horizon systems.

Supermartingales for probabilistic programs Supermartingales have also been used for the analysis of probabilistic programs (PPs) for properties such as termination [15], reachability [58] and safety [18, 19].

3 Preliminaries

We consider a discrete-time stochastic dynamical system whose dynamics are defined by the equation

where t ∈ N 0 is a time step, x t ∈ X is a state of the system, u t = π ( x t ) ∈ U is a control action and ω t ∈ W is a stochastic disturbance vector at the time step t . Here, X ⊆ R n is the state space of the system, U ⊆ R m is the action space and W ⊆ R p is the stochastic disturbance space. The system dynamics are defined by the dynamics function f : X × U × W → X , the control policy π : X → U and a probability distribution d over W from which a stochastic disturbance vector is sampled independently at each time step. Together, these define a stochastic feedback loop system .

A sequence ( x t , u t , ω t ) ∞ t =0 of state-action-disturbance triples is a trajectory of the system, if we have that u t = π ( x t ) , ω t ∈ support( d ) and x t +1 = f ( x t , π ( x t ) , ω t ) hold for each time step t ∈ N 0 . For every state x 0 ∈ X , we use Ω x 0 to denote the set of all trajectories that start in the initial state x 0 . The Markov decision process (MDP) semantics of the system define a probability space over the set of all trajectories in Ω x 0 under any fixed policy π of the system [47]. We use P x 0 and E x 0 to denote the probability measure and the expectation operator in this probability space.

Assumptions For system semantics to be mathematically well-defined, we assume that X ⊆ R n , U ⊆ R m and W ⊆ R p are all Borel-measurable and that f : X × U × W → X and π : X → U are continuous functions. Furthermore, we assume that X ⊆ R n is compact (i.e. closed and bounded) and that f and π are continuous functions. These are very general and standard assumptions in control theory. Since any continuous function on a compact domain is Lipschitz continuous, our assumptions also imply that f and π are Lipschitz continuous.

Probabilistic specifications Let Ω denote the set of all trajectories of the system. A specification is a boolean function ϕ : Ω →{ true , false } which for each trajectory specifies whether it satisfies the specification. We write ρ | = ϕ whenever a trajectory ρ satisfies the specification ϕ . A probabilistic specification is then defined as an ordered pair ( ϕ, p ) of a specification ϕ and a probability parameter p ∈ [0 , 1] with which the specification needs to be satisfied. We say that the system satisfies the probabilistic specification ( ϕ, p ) at the initial state x 0 ∈ X if the probability of any trajectory in Ω x 0 satisfying ϕ is at least p , i.e. if P x 0 [ ρ ∈ Ω x 0 | ρ | = ϕ ] ≥ p .

SPECTRL and abstract graphs Reach-avoid specifications are one of the most common and practically relevant specifications appearing in safety-critical applications that generalize both reachability and safety specifications [54]. Given a target region and an unsafe region, a reach-avoid specifications requires that a system controlled by a policy reaches the target region while avoiding the unsafe region. The SPECTRL specification language [32] allows specifications of reachability and avoidance objectives as well as their sequential or disjunctive composition. See Appendix A for the formal definition of the SPECTRL syntax and semantics. 2

It was shown in [33] that a SPECTRL specification can be translated into an abstract graph over reach-avoid specifications. Intuitively, an abstract graph is a directed acyclic graph (DAG) whose vertices represent regions of system states and whose edges are annotated with a safety specification. Hence, each edge can be associated with a reach-avoid specification where the goal is to drive the system from the region corresponding to the source vertex of the edge to the region corresponding to the target vertex of the edge while satisfying the annotated safety specification.

Definition 1 (Abstract graph) . An abstract graph G = ( V, E, β, s, t ) is a DAG, where V is the vertex set, E is the edge set, β : V ∪ E →B ( X ) maps each vertex and each edge to a subgoal region in X , s ∈ V is the source vertex and t ∈ V is the target vertex. Furthermore, we require that β ( s ) = X 0 .

Given a trajectory ρ = ( x t , u t , ω t ) ∞ t =0 of the system and an abstract graph G = ( V, E, β, s, t ) , we say that ρ satisfies abstract reachability for G (written ρ | = G ) if it gives rise to a path in G that traverses G from s to t and satisfies reach-avoid specifications of all traversed edges. We formalize this notion in Appendix B. It was shown in [33, Theorem 3.4] that for each SPECTRL specitication ϕ once can construct an abstract graph G such that for each trajectory ρ we have ρ | = ϕ iff ρ | = G . For completeness, we provide this construction in Appendix B.

Problem Statement We now formally define the problem that we consider in this work. Consider a stochastic feedback loop system defined as above and let X 0 ⊆ X be the set of initial states. Let ( ϕ, p ) be a probabilistic specification with ϕ being a specification formula in SPECTRL and p ∈ [0 , 1] being a probability threshold. Then, our goal is to learn a policy π such that the stochastic feedback loop system controlled by the policy π guarantees satisfaction of the probabilistic specification ( ϕ, p ) at every initial state x 0 ∈ X 0 , i.e. that for every x 0 ∈ X 0 we have P x 0 [ ρ ∈ Ω x 0 | ρ | = ϕ ] ≥ p .

Compositional learning algorithm Our goal is not only to learn policy that guarantees satisfaction of the probabilistic specification, but also to learn such a policy in a compositional manner. Given a SPECTRL specification ϕ , a probability threshold p ∈ [0 , 1] and an abstract graph G of ϕ , we say that a policy π is a compositional policy for the probabilistic specification ( ϕ, p ) if it guarantees satisfaction of ( ϕ, p ) and is obtained by composing a number of edge policies learned for reachavoid tasks associated to edges of G . An algorithm is said to be compositional for a probabilistic specification ( ϕ, p ) if it learns a compositional policy for the probabilistic specification ( ϕ, p ) . In this work, we present a compositional algorithm for the given probabilistic specification ( ϕ, p ) .

4 Improved Bound for Probabilistic Reach-avoidance

In this section, we define reach-avoid supermartingales (RASMs) and derive an improved lower bound on the probability of reach-avoidance that RASMs can be used to formally certify. RASMs, together with a lower bound on the probability of reach-avoidance that they guarantee and a method for learning a control policy and an RASM, were originally proposed in [68]. The novelty in this section is that we prove that RASMs imply a strictly stronger bound on the probability of satisfying reach-avoidance, compared to the bound derived in [68]. We achieve this by proposing an alternative formulation of RASMs, which we call multiplicative RASMs . In contrast to the original definition of RASMs which imposes additive expected decrease condition, our formulation of RASMs imposes multiplicative expected decrease condition. In what follows, we define multiplicative RASMs. Then, we first show in Theorem 1 that they are equivalent to RASMs. Second, we show in Theorem 2 that by analyzing the multiplicative expected decrease, we can derive an exponentially tighter bound on the probability of reach-avoidance. These two results imply that we can utilize the procedure of [68] to learn policies together with multiplicative RASMs that provide strictly better formal guarantees on the probability reach-avoidance.

Prior work - reach-avoid supermartingales RASMs [68] are continuous functions that assign real values to system states, are required to be nonnegative and to satisfy the additive expected decrease condition, i.e. to strictly decrease in expected value by some additive term ϵ > 0 upon every one-step evolution of the system dynamics until either the target set or the unsafe set is reached. Furthermore, the initial value of the RASM is at most 1 while the value of the RASM needs to exceed some λ > 1 in order for the system to reach the unsafe set. Thus, an RASM can intuitively be viewed as an invariant of the system which shows that the system has a tendency to converge either to the target or the unsafe set while also being repulsed away from the unsafe set, where this tendency is formalized by the expected decrease condition. To emphasize the additive expected decrease, we refer to RASMs of [68] as additive RASMs.

Definition 2 (Additive reach-avoid supermartingales [68]) . Let ϵ > 0 and λ > 1 . A continuous function V : X → R is an ( ϵ, λ ) -additive reach-avoid supermartingale ( ( ϵ, λ ) -additive RASM) with respect to X t and X u , if:

Nonnegativity condition. V ( x ) ≥ 0 for each x ∈ X .

Initial condition. V ( x ) ≤ 1 for each x ∈ X 0 .

Safety condition. V ( x ) ≥ λ for each x ∈ X u .

Additive expected decrease condition. For each x ∈ X\X t at which V ( x ) ≤ λ , we have V ( x ) ≥ E ω ∼ d [ V ( f ( x , π ( x ) , ω ))] + ϵ .

Multiplicative RASMs and equivalence In this work we introduce multiplicative RASMs which need to decrease in expected value by at least some multiplicative factor γ ∈ (0 , 1) . We also impose the strict positivity condition outside of the target set X t . Note that any ( ϵ, λ ) -additive RASM satisfies this condition with the lower bound ϵ > 0 .

Definition 3 (Multiplicative reach-avoid supermartingales) . Let γ ∈ (0 , 1) , δ > 0 and λ > 1 . A continuous function V : X → R is a ( γ, δ, λ ) -multiplicative reach-avoid supermartingale ( ( γ, δ, λ ) -multiplicative RASM) with respect to X t and X u , if:

Nonnegativity condition. V ( x ) ≥ 0 for each x ∈ X .

Strict positivity outside X t condition. V ( x ) ≥ δ for each x ∈ X\X t .

Initial condition. V ( x ) ≤ 1 for each x ∈ X 0 .

Safety condition. V ( x ) ≥ λ for each x ∈ X u .

Multiplicative expected decrease condition. For each x ∈ X\X t at which V ( x ) ≤ λ , we have γ · V ( x ) ≥ E ω ∼ d [ V ( f ( x , π ( x ) , ω ))] .

We show that a function V : X → R is an additive RASM if and only if it is a multiplicative RASM.

Theorem 1. [Proof in Apendix C] The following two statements hold:

If a continuous function V : X → R is an ( ϵ, λ ) -additive RASM, then it is also a ( λ -ϵ λ , min { ϵ, λ } , λ ) -multiplicative RASM.

If a continuous function V : X → R is a ( γ, δ, λ ) -multiplicative RASM, then it is also an ((1 -γ ) · δ, λ ) -additive RASM.

Improved bound The following theorem shows that the existence of a multiplicative RASM implies a lower bound on the probability with which the system satisfies the reach-avoid specification.

Theorem 2. [Proof in Appendix D] Let γ ∈ (0 , 1) , δ > 0 and λ > 1 , and suppose that V : X → R is a ( γ, δ, λ ) -multiplicative RASM with respect to X t and X u . Suppose furthermore that V is Lipschitz continuous with a Lipschitz constant L V , and that the system under policy π satisfies the bounded step property , i.e. that there exists ∆ > 0 such that || x -f ( x, π ( x ) , ω ) || 1 ≤ ∆ holds for each x ∈ X and ω ∈ W . Let N = ⌊ ( λ -1) / ( L V · ∆) ⌋ . Then, for every x 0 ∈ X 0 , we have that

Here, N = ⌊ ( λ -1) / ( L V · ∆) ⌋ is the smallest number of time steps in which the system could hypothetically reach the unsafe set X u . This is because, for the system to violate safety, by the Initial and the Safety conditions of multiplicative RASMs the value of V must increase from at most 1 to at least λ . But the value of V can increase by at most L V · ∆ in any single time step since ∆ is the maximal step size of the system and L V is the Lipschitz constant of V . Hence, the system cannot reach X u in less than N = ⌊ ( λ -1) / ( L V · ∆) ⌋ time steps.

We conclude this section by comparing our bound to the bound P x 0 [ ReachAvoid ( X t , X u )] ≥ 1 -1 λ of [68] for additive RASMs. Our bound on the probability of violating reach-avoidance is tighter by a factor of γ N . Notice that this term decays exponentially as λ increases, hence our bound is exponentially tighter in λ . This is particularly relevant if we want to verify reach-avoidance with high probability that is close to 1 , as it allows using a much smaller value of λ and significantly relaxing the Safety condition in Definition 3. To evaluate the quality of the bound improvement on an example, in Table 1 we consider the 9-Rooms environment and compare the bounds for multiple values of λ when γ = 0 . 99 . Note that ∆ is the property of the system and not the RASM.Thus, the value of λ is the key factor for controlling the quality of the bound. Results in Table 1 show that, in order to verify probability 99.9% reach-avoidance for an edge policy in the 9-Room example, our new bound allows using λ ≈ 100 whereas with the old bound the algorithm needs to use λ ≈ 1000 .

Remark 1 (Comparison of RASMs and stochastic barrier functions) . Stochastic barrier functions (SBFs) [45, 46] were introduced for proving probabilistic safety in stochastic dynamical systems, i.e. without the additional reachability condition as in reach-avoid specifications. If one is only interested in probabilistic safety, RASMs of [68] reduce to SBFs by letting X t = ∅ and ϵ = 0 in Definition 2 for additive RASMs, and X t = ∅ and γ = 1 in Definition 3 for multiplicative RASMs. The bound on the probability of reach-avoidance P x 0 [ ReachAvoid ( ∅ , X u )] ≥ 1 -1 λ of [68] for additive RASMs coincides with the bound on the probability of safety implied by SBFs of [45, 46]. Hence, our novel bound in Theorem 2 also provides tighter lower bound on the safety probability guarantees via SBFs.

Exponential stochastic barrier functions (exponential SBFs) have been considered in [49] in order to provide tighter bounds on safety probability guarantees via SBFs for finite time horizon systems , in which the length of the time horizon is fixed and known a priori. Exponential SBFs also consider a multiplicative expected decrease condition, similar to our multiplicative RASMs and provide lower bounds on safety probability which are tighter by a factor which is exponential in the length N of the time horizon [49, Theorem 2]. However, as the length of the time horizon N →∞ , their bound reduces to the bound of [45, 46]. In contrast, Theorem 2 shows that our multiplicative RASMs provide a tighter lower bound on safety (or more generally, reach-avoid) probability even in unbounded (i.e. indefinite) or infinite time horizon systems.

5 Compositional Learning for Probabilistic Specifications

We now present the CLAPS algorithm for learning a control policy that guarantees satisfaction of a probabilistic specification ( ϕ, p ) , where ϕ is a SPECTRL formula. The idea behind our algorithm is as follows. The algorithm first translates ϕ into an abstract graph G = ( V, E, β, s, t ) using the translation discussed in Section 3, in order to decompose the problem into a series of reach-avoid tasks associated to abstract graph edges. The algorithm then solves these reach-avoid tasks by learning policies for abstract graph edges. Each edge policy is learned together with a (multiplicative) RASM which provides formal guarantees on the probability of satisfaction of the reach-avoid specification proved in Theorem 2. Finally, the algorithm combines these edge policies in order to obtain a global policy that guarantees satisfaction of ( ϕ, p ) . The algorithm pseudocode is presented in Algorithm 1. We assume the POLICY+RASM subprocedure which, given a reach-avoid task and given a probability parameter p ′ , learns a control policy together with a RASM that proves that the policy guarantees reach-avoidance with probability at least p ′ . Due to the equivalence of additive and multiplicative RASMs that we proved in Theorem 1, for this we can use the procedure of [68] as an off-the-shelf method. For completeness of our presentation, we also provide details behind the POLICY+RASM subprocedure in Appendix E.

Challenges There are two important challenges in designing an algorithm based on the above idea. First, it is not immediately clear how probabilistic reach-avoid guarantees provided by edge policies may be used to deduce the probability with which the global specification ϕ is satisfied. Second, since SPECTRL formulas allow for disjunctive specifications, it might not be necessary to solve reach-avoid tasks associated to each abstract graph edge and naïvely doing so might lead to inefficiency. Our algorithm solves both of these challenges by first topologically ordering the vertices of the abstract graph and then performing a forward pass in which vertices are processed according to the topological ordering. The forward pass is executed in a way which allows our algorithm to keep track of the cumulative probability with which edge policies that have already been learned might violate the global specification ϕ , therefore also allowing the algorithm to learn edge policies on-demand and to identify abstract graph edges for which solving reach-avoid tasks would be redundant.

Forward pass on the abstract graph Since an abstract graph is a directed acyclic graph (DAG), we perform a topological sort on G in order to produce an ordering s = v 1 , v 2 , . . . , v | V | = t of its vertices (line 4 in Algorithm 1). This ordering satisfies the property that each edge ( v i , v j ) in G must satisfy i < j , i.e. any each edge must be a 'forward edge' with respect to topological ordering. Algorithm 1 now initializes an empty dictionary Prob and sets Prob[ s ] = 1 for the source vertex s (line 5) and performs a forward pass to process the remaining vertices in the abstract graph according to the topological ordering (lines 6-13). The purpose of the dictionary Prob is to store lower bounds on the probability with which previously processed abstract graph vertices are reached by following the computed edge policies while satisfying reach-avoid specifications of the traversed abstract graph edges. For each newly processed vertex v i , Algorithm 1 learns policies for the reach-avoid tasks associated to all abstract graph edges that are incoming to v i . It then combines probabilities with

which the learned edge policies satisfy associated reach-avoid tasks with the probabilities that have been stored for the previously processed vertices in order to compute the lower bound Prob[ v i ] .

More concretely, for each 2 ≤ i ≤ | V | , Algorithm 1 first stores Prob[ v i ] = 0 as a trivial lower bound (line 7). It then computes the set N ( v i ) of all predecessors of v i for which previously learned edge policies ensure reachability with probability at least p while satisfying reach-avoid specification of all traversed abstract graph edged (line 8). Recall, p is the global probability threshold with which the agent needs to satisfy the SPECTRL specification ϕ , and since Algorithm 1 processes vertices according to the topological ordering all vertices in N ( v i ) have already been processed. If N ( v i ) is empty, then the algorithm concludes that v i cannot be reached in the abstract graph while satisfying reach-avoid specifications of traversed edges with probability at least p . In this case, the algorithm does not try to learn edge policies for the edges with the target vertex v i and proceeds to processing the next vertex in the topological ordering. Thus, Algorithm 1 ensures that edge policies are learned on-demand . If N ( v i ) is not empty, then for each v ∈ N ( v i ) the algorithm uses POLICY+RASM together with binary search to find the maximal probability p ( v,v i ) for which POLICY+RASM manages to learn a policy π ( v,v i ) that solves the reach-avoid task with probability at least p ( v,v i ) (line 10). We run the binary search only up to a pre-specified precision, which is a hyperparameter of our algorithm. Since the reach-avoid specification of the edge ( v, v i ) is satisfied with probability at least p ( v,v i ) and since v is reached in the abstract graph while satisfying reachavoid specification of all traversed edges with probability at least Prob[ v ] , Algorithm 1 updates its current lower bound Prob[ v i ] for v i to p ( v,v i ) · Prob[ v ] whenever p ( v,v i ) · Prob[ v ] exceeds the stored bound Prob[ v i ] (line 11). This procedure is repeated for each v ∈ N ( v i ) .

̸

Composing edge policies into a global policy Once all abstract graph vertices have been processed, Algorithm 1 checks if Prob[ t ] ≥ p (line 14). If so, it composes the learned edge policies into a global policy π that guarantees satisfaction of the probabilistic specification ( ϕ, p ) (line 15) and returns the policy π (line 16). The composition is performed as follows. First, note that for each vertex v = s in the abstract graph for which Prob[ v ] > 0 , by lines 7-11 in Algorithm 1 there must exist a vertex v ′ ∈ N ( v ) such that the algorithm has successfully learned a policy π ( v ′ ,v ) for the edge ( v ′ , v ) and such that Prob[ v ] = p ( v ′ ,v ) · Prob[ v ′ ] . Hence, repeatedly picking such predecessors yields a finite path s = v i 0 , v i 1 , . . . , v i k = t in the abstract graph such that, for each 1 ≤ j ≤ k , the algorithm has successfully learned an edge policy π ( v i j -1 ,v i j ) and such that Prob[ v i j ] = p ( v i j -1 ,v i j ) · Prob[ v i j -1 ] holds. The global policy π starts by following the edge policy π ( v i 0 ,v i 1 ) until the target region β ( v ) is reached or until the safety constraint β ( v , v ) associated to the edge is violated. Then, i 1 i 0 i 1

(a) The objective is to navigate from the initial set in the bottom left (in green) to the target set in the top right room (in violet).

(b) Visualization of trajectories of an end-to-end learned policy. The policy hits the walls and cannot be verified as safe.

(c) Visualization of trajectories of the three verifiable compositional policies with intermediate initial/target states shown in grey.

T⊣∫∥ ⌋≀⇕√≀∫⟩⊔⟩≀\ }∇⊣√⟨

V⌉∇⟩⊙⊣⌊↕⌉ ∫⊓⌊⊔⊣∫∥∫

V⌉∇⟩⊙⊣⌊↕⌉ ⌋≀⇕√↕⌉⊔⌉ √⊣⊔⟨

it follows the edge policy π ( v i 1 ,v i 2 ) until the target region β ( v i 2 ) is reached or until the safety constraint β ( v i 1 , v i 1 2 ) associated to the edge is violated. This is repeated for each edge along the path s = v i 0 , v i 1 , . . . , v i k = t until the system reaches a state within the target region β ( t ) . If at any point the safety constraint is violated prior to reaching the target region associated to the edge, policy π follows the current edge policy indefinitely and no longer updates it as the specification ϕ has been violated. As we prove in Theorem 3, violation of the safety constraint associated to some edge or the system never reaching the target region associated to some edge happen with probability at most 1 -p . If Prob[ t ] < p , then Algorithm 1 returns that it could not learn a policy that guarantees satisfaction of ( ϕ, p ) . Theorem 3 establishes the correctness of Algorithm 1 and its proof shows that the global policy π obtained as above indeed satisfies the probabilistic specification ( ϕ, p ) .

Theorem 3. [Proof in Appendix F] Algorithm 1 is compositional, and if it outputs a policy π , then π guarantees the probabilistic specification ( ϕ, p ) .

6 Experiments

We implement a prototype of Algorithm 1 to validate its effectiveness 3 . Motivated by the experiments of [33], we define an environment in which an agent must navigate safely between different rooms. However, different from [33] our Stochastic Nine Rooms environment perturbs the agent by random noise in each step. Concretely, the system is governed by the dynamics function, x t +1 = x t + 0 . 1 min(max(( a t , -1) , 1) + ω t , where ω t is drawn from a triangular noise distribution. The state space of the environment is defined as [0 , 3] × [0 , 3] ⊂ R 2 . The set of the initial states is [0 . 4 , 0 . 6] × [0 . 4 , 0 . 6] , whereas the target states are defined as [2 . 4 , 2 . 6] × [2 . 4 , 2 . 6] . The state space contains unsafe areas that should be avoided, i.e., the red walls shown in Figure 1. Specifically, these unsafe areas cannot be entered by the agent, and coming close to them during the RL training incurs a penalty. Note that this defines a reach-avoid task where the goal is to reach the target region from the initial region while avoiding the unsafe regions defined by red walls.

We first run the POLICY+RASM procedure (i.e. the method of [68]) on this task directly and observe that it is not able to learn a policy with any formal guarantees on probability of reach-avoidance.

We then translate this task into a SPECTRL task which decomposes it into a set of simpler reachavoid tasks by constructing an abstract graph as follows. The vertex set of the abstract graph consists of centers of each of the 9 rooms , and the target regions associated to each room center are [0 . 4 + x, 0 . 6 + x ] × [0 . 4 + y, 0 . 6 + y ] for x, y ∈ { 0 , 1 , 2 } . Edges of this abstract graph are shown in Figure 2 left. To each edge, we associate an unsafe region to be avoided by taking red walls incident to rooms in which the edge is contained.

For each reach-avoid edge task, we run the proximal policy optimization (PPO) [50] reinforcement learning algorithm to initialize policy parameters. We then run the POLICY+RASM procedure to learn a policy and a RASM which proves probabilistic reach-avoidance. We set the timeout to 4 hours for each reach-avoid subtask. In Figure 1 b), we visualize the PPO policy that was trained to reach the target states directly from the initial states. As shown, the policy occasionally hits a wall and cannot be verified, i.e., POLICY+RASM procedure times out. In Figure 2 center, we visualize all successfully verified subtasks in blue. The reach-avoid lower bounds for each of the subtasks are listed in Table 2. Note that the simplest decomposition passing over the bottom right room could not be verified, which shows that the systematic decomposition used in our algorithm has advantages over manual task decompositions. Finally, in Figure 2 right, we highlight the path from the initial states to the target states via safe intermediate sets. In Figure 1 c), we visualize the trajectories of these three compositional policies. As shown, the trajectories never reach an unsafe area.

7 Concluding Remarks

We propose CLAPS , a novel method for learning and verifying a composition of neural network policies for stochastic control systems. Our method considers control tasks under specifications in the SPECTRL lanugange, decomposes the task into an abstract graph of reach-avoid tasks and uses reach-avoid supermartingales to provide formal guarantees on the probability of reach-avoidance in each subtask. We also prove that RASMs imply a strictly tighter lower bound on the probability of reach-avoidance compared to prior work. These results provide confidence in the ability of CLAPS to find meaningful guarantees for global compositional policies, as shown by experimental evaluation in the Stochastic Nine Rooms environment. While our approach has significant conceptual, theoretical and algorithmic novelty, it is only applicable to SPECTRL logical specifications and not to the whole of LTL. Furthermore, our algorithm is not guaranteed to return a policy and does not guarantee tight bounds on the probability of satisfying the logical specification. Overcoming these limitations will provide further confidence in deploying RL based solutions for varied applications.

Acknowledgments

This work was supported in part by the ERC-2020-AdG 101020093 (VAMOS) and the ERC-2020CoG 863818 (FoRM-SMArt).

References

Alessandro Abate, Daniele Ahmed, Alec Edwards, Mirco Giacobbe, and Andrea Peruffo. FOSSIL: a software tool for the formal synthesis of lyapunov functions and barrier certificates using neural networks. In Sergiy Bogomolov and Raphaël M. Jungers, editors, HSCC '21: 24th ACM International Conference on Hybrid Systems: Computation and Control, Nashville,

Tennessee, May 19-21, 2021 , pages 24:1-24:11. ACM, 2021. doi: 10.1145/3447928.3456646. URL https://doi.org/10.1145/3447928.3456646 .

Alessandro Abate, Daniele Ahmed, Mirco Giacobbe, and Andrea Peruffo. Formal synthesis of lyapunov neural networks. IEEE Control. Syst. Lett. , 5(3):773-778, 2021. doi: 10.1109/ LCSYS.2020.3005328. URL https://doi.org/10.1109/LCSYS.2020.3005328 .

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning , pages 22-31. PMLR, 2017.

Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 , pages 2669-2678. AAAI Press, 2018. URL https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17211 .

Eitan Altman. Constrained Markov decision processes , volume 7. CRC Press, 1999.

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. CoRR , abs/1606.06565, 2016. URL http://arxiv.org/ abs/1606.06565 .

Greg Anderson, Abhinav Verma, Isil Dillig, and Swarat Chaudhuri. Neurosymbolic reinforcement learning with formally verified exploration. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips. cc/paper/2020/hash/448d5eda79895153938a8431919f4c9f-Abstract.html .

Matin Ansaripour, Krishnendu Chatterjee, Thomas A. Henzinger, Mathias Lechner, and Ðor ¯ de Žikeli´ c. Learning provably stabilizing neural controllers for discrete-time stochastic systems. In Étienne André and Jun Sun, editors, Automated Technology for Verification and Analysis , pages 357-379, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-45329-8.

Thom S. Badings, Alessandro Abate, Nils Jansen, David Parker, Hasan A. Poonawala, and Mariëlle Stoelinga. Sampling-based robust control of autonomous systems with nongaussian noise. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022 , pages 9669-9678. AAAI Press, 2022. URL https://ojs.aaai.org/index.php/AAAI/article/view/21201 .

Thom S. Badings, Licio Romao, Alessandro Abate, and Nils Jansen. Probabilities are not enough: Formal controller synthesis for stochastic dynamical models with epistemic uncertainty. CoRR , abs/2210.05989, 2022. doi: 10.48550/arXiv.2210.05989. URL https://doi.org/10. 48550/arXiv.2210.05989 .

Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 908-918, 2017. URL https://proceedings.neurips.cc/paper/2017/ hash/766ebcd59621e305170616ba3d3dac32-Abstract.html .

Alper Kamil Bozkurt, Yu Wang, Michael M. Zavlanos, and Miroslav Pajic. Control synthesis from linear temporal logic specifications using model-free reinforcement learning. In 2020 IEEE International Conference on Robotics and Automation, ICRA 2020, Paris, France, May 31 - August 31, 2020 , pages 10349-10355. IEEE, 2020. doi: 10.1109/ICRA40945.2020.9196796. URL https://doi.org/10.1109/ICRA40945.2020.9196796 .

Ronen I. Brafman, Giuseppe De Giacomo, and Fabio Patrizi. Ltlf/ldlf non-markovian rewards. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 , pages 1771-1778. AAAI Press, 2018. URL https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/ view/17342 .

Nathalie Cauchi and Alessandro Abate. Stochy-automated verification and synthesis of stochastic processes. In Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control , pages 258-259, 2019.

Aleksandar Chakarov and Sriram Sankaranarayanan. Probabilistic program analysis with martingales. In Natasha Sharygina and Helmut Veith, editors, Computer Aided Verification - 25th International Conference, CAV 2013, Saint Petersburg, Russia, July 13-19, 2013. Proceedings , volume 8044 of Lecture Notes in Computer Science , pages 511-526. Springer, 2013. doi: 10. 1007/978-3-642-39799-8\_34. URL https://doi.org/10.1007/978-3-642-39799-8_ 34 .

Ya-Chien Chang and Sicun Gao. Stabilizing neural control using self-learned almost lyapunov critics. In IEEE International Conference on Robotics and Automation, ICRA 2021, Xi'an, China, May 30 - June 5, 2021 , pages 1803-1809. IEEE, 2021. doi: 10.1109/ICRA48506.2021.9560886. URL https://doi.org/10.1109/ICRA48506.2021.9560886 .

Ya-Chien Chang, Nima Roohi, and Sicun Gao. Neural lyapunov control. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 3240-3249, 2019. URL https://proceedings.neurips.cc/paper/2019/ hash/2647c1dba23bc0e0f9cdf75339e120d2-Abstract.html .

Krishnendu Chatterjee, Petr Novotný, and Ðor ¯ de Žikeli´ c. Stochastic invariants for probabilistic termination. In Giuseppe Castagna and Andrew D. Gordon, editors, Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages, POPL 2017, Paris, France, January 18-20, 2017 , pages 145-160. ACM, 2017. doi: 10.1145/3009837.3009873. URL https://doi.org/10.1145/3009837.3009873 .

Krishnendu Chatterjee, Amir Kafshdar Goharshady, Tobias Meggendorfer, and Dorde Zikelic. Sound and complete certificates for quantitative termination analysis of probabilistic programs. In Sharon Shoham and Yakir Vizel, editors, Computer Aided Verification - 34th International Conference, CAV 2022, Haifa, Israel, August 7-10, 2022, Proceedings, Part I , volume 13371 of Lecture Notes in Computer Science , pages 55-78. Springer, 2022. doi: 10. 1007/978-3-031-13185-1\_4. URL https://doi.org/10.1007/978-3-031-13185-1_4 .

Krishnendu Chatterjee, Thomas A. Henzinger, Mathias Lechner, and Dorde Zikelic. A learnerverifier framework for neural network controllers and certificates of stochastic systems. In Sriram Sankaranarayanan and Natasha Sharygina, editors, Tools and Algorithms for the Construction and Analysis of Systems - 29th International Conference, TACAS 2023, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2022, Paris, France, April 22-27, 2023, Proceedings, Part I , volume 13993 of Lecture Notes in Computer Science , pages 3-25. Springer, 2023. doi: 10.1007/978-3-031-30823-9\_1. URL https://doi.org/ 10.1007/978-3-031-30823-9_1 .

Yinlam Chow, Ofir Nachum, Edgar A. Duéñez-Guzmán, and Mohammad Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 8103-8112, 2018. URL https://proceedings.neurips.cc/paper/2018/hash/ 4fe5149039b52765bde64beb9f674940-Abstract.html .

Patrick Cousot and Radhia Cousot. Abstract interpretation: A unified lattice model for static analysis of programs by construction or approximation of fixpoints. In Robert M. Graham, Michael A. Harrison, and Ravi Sethi, editors, Conference Record of the Fourth ACM Symposium on Principles of Programming Languages, Los Angeles, California, USA, January 1977 , pages 238-252. ACM, 1977. doi: 10.1145/512950.512973. URL https://doi.org/10.1145/ 512950.512973 .

Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerík, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. ArXiv , abs/1801.08757, 2018.

Charles Dawson, Sicun Gao, and Chuchu Fan. Safe control with learned certificates: A survey of neural lyapunov, barrier, and contraction methods for robotics and control. IEEE Trans. Robotics , 39(3):1749-1767, 2023. doi: 10.1109/TRO.2022.3232542. URL https: //doi.org/10.1109/TRO.2022.3232542 .

Ingy Elsayed-Aly, Suda Bharadwaj, Christopher Amato, Rüdiger Ehlers, Ufuk Topcu, and Lu Feng. Safe multi-agent reinforcement learning via shielding. In Frank Dignum, Alessio Lomuscio, Ulle Endriss, and Ann Nowé, editors, AAMAS '21: 20th International Conference on Autonomous Agents and Multiagent Systems, Virtual Event, United Kingdom, May 3-7, 2021 , pages 483-491. ACM, 2021. URL https://dl.acm.org/doi/10.5555/3463952. 3464013 .

Peter Geibel. Reinforcement learning for mdps with constraints. In Johannes Fürnkranz, Tobias Scheffer, and Myra Spiliopoulou, editors, Machine Learning: ECML 2006, 17th European Conference on Machine Learning, Berlin, Germany, September 18-22, 2006, Proceedings , volume 4212 of Lecture Notes in Computer Science , pages 646-653. Springer, 2006. doi: 10.1007/11871842\_63. URL https://doi.org/10.1007/11871842_63 .

Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy A. Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. CoRR , abs/1810.12715, 2018. URL http://arxiv.org/abs/1810.12715 .

Ernst Moritz Hahn, Mateo Perez, Sven Schewe, Fabio Somenzi, Ashutosh Trivedi, and Dominik Wojtczak. Omega-regular objectives in model-free reinforcement learning. In Tomás Vojnar and Lijun Zhang, editors, Tools and Algorithms for the Construction and Analysis of Systems 25th International Conference, TACAS 2019, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2019, Prague, Czech Republic, April 6-11, 2019, Proceedings, Part I , volume 11427 of Lecture Notes in Computer Science , pages 395-412. Springer, 2019. doi: 10.1007/978-3-030-17462-0\_27. URL https://doi.org/10.1007/ 978-3-030-17462-0_27 .

Mohammadhosein Hasanbeig, Yiannis Kantaros, Alessandro Abate, Daniel Kroening, George J. Pappas, and Insup Lee. Reinforcement learning for temporal logic control synthesis with probabilistic satisfaction guarantees. In 58th IEEE Conference on Decision and Control, CDC 2019, Nice, France, December 11-13, 2019 , pages 5338-5343. IEEE, 2019. doi: 10.1109/ CDC40024.2019.9028919. URL https://doi.org/10.1109/CDC40024.2019.9028919 .

Radoslav Ivanov, Kishor Jothimurugan, Steve Hsu, Shaan Vaidya, Rajeev Alur, and Osbert Bastani. Compositional learning and verification of neural network controllers. ACM Trans. Embed. Comput. Syst. , 20(5s):92:1-92:26, 2021. doi: 10.1145/3477023. URL https://doi. org/10.1145/3477023 .

Nils Jansen, Bettina Könighofer, Sebastian Junges, Alex Serban, and Roderick Bloem. Safe reinforcement learning using probabilistic shields (invited paper). In Igor Konnov and Laura Kovács, editors, 31st International Conference on Concurrency Theory, CONCUR 2020, September 1-4, 2020, Vienna, Austria (Virtual Conference) , volume 171 of LIPIcs , pages 3:1-3:16. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2020. doi: 10.4230/LIPIcs.CONCUR.2020.3. URL https://doi.org/10.4230/LIPIcs.CONCUR.2020.3 .

Kishor Jothimurugan, Rajeev Alur, and Osbert Bastani. A composable specification language for reinforcement learning tasks. In Hanna M. Wallach, Hugo Larochelle, Alina

Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 13021-13030, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ f5aa4bd09c07d8b2f65bad6c7cd3358f-Abstract.html .

Kishor Jothimurugan, Suguman Bansal, Osbert Bastani, and Rajeev Alur. Compositional reinforcement learning from logical specifications. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 10026-10039, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 531db99cb00833bcd414459069dc7387-Abstract.html .

T. Koller, Felix Berkenkamp, Matteo Turchetta, and A. Krause. Learning-based model predictive control for safe exploration. 2018 IEEE Conference on Decision and Control (CDC) , pages 6059-6066, 2018.

Harold J. Kushner. A partial history of the early development of continuous-time nonlinear stochastic systems theory. Autom. , 50(2):303-334, 2014. doi: 10.1016/j.automatica.2013.10.013. URL https://doi.org/10.1016/j.automatica.2013.10.013 .

Abolfazl Lavaei, Mahmoud Khaled, Sadegh Soudjani, and Majid Zamani. AMYTISS: parallelized automated controller synthesis for large-scale stochastic systems. In Shuvendu K. Lahiri and Chao Wang, editors, Computer Aided Verification - 32nd International Conference, CAV 2020, Los Angeles, CA, USA, July 21-24, 2020, Proceedings, Part II , volume 12225 of Lecture Notes in Computer Science , pages 461-474. Springer, 2020. doi: 10.1007/978-3-030-53291-8\ _24. URL https://doi.org/10.1007/978-3-030-53291-8_24 .

Mathias Lechner, Dorde Zikelic, Krishnendu Chatterjee, and Thomas A. Henzinger. Infinite time horizon safety of bayesian neural networks. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 10171-10185, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 544defa9fddff50c53b71c43e0da72be-Abstract.html .

Mathias Lechner, Dorde Zikelic, Krishnendu Chatterjee, and Thomas A. Henzinger. Stability verification in stochastic control systems via neural network supermartingales. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022 , pages 7326-7336. AAAI Press, 2022. URL https://ojs.aaai.org/index.php/AAAI/ article/view/20695 .

Xiao Li, Cristian Ioan Vasile, and Calin Belta. Reinforcement learning with temporal logic rewards. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2017, Vancouver, BC, Canada, September 24-28, 2017 , pages 3834-3839. IEEE, 2017. doi: 10.1109/IROS.2017.8206234. URL https://doi.org/10.1109/IROS.2017.8206234 .

Anqi Liu, Guanya Shi, Soon-Jo Chung, Anima Anandkumar, and Yisong Yue. Robust regression for safe exploration in control. In L4DC , 2020.

Frederik Baymler Mathiesen, Simeon Craig Calvert, and Luca Laurenti. Safety certification for stochastic systems via neural barrier functions. IEEE Control. Syst. Lett. , 7:973-978, 2023. doi: 10.1109/LCSYS.2022.3229865. URL https://doi.org/10.1109/LCSYS.2022.3229865 .

Rayan Mazouz, Karan Muvvala, Akash Ratheesh Babu, Luca Laurenti, and Morteza Lahijanian. Safety guarantees for neural network dynamic systems via stochastic barrier functions. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id= 9XQa6cgLo21 .

Cyrus Neary, Christos K. Verginis, Murat Cubuktepe, and Ufuk Topcu. Verifiable and compositional reinforcement learning systems. In Akshat Kumar, Sylvie Thiébaux, Pradeep Varakantham, and William Yeoh, editors, Proceedings of the Thirty-Second International Conference on Automated Planning and Scheduling, ICAPS 2022, Singapore (virtual), June 13-24, 2022 , pages 615-623. AAAI Press, 2022. URL https://ojs.aaai.org/index.php/ICAPS/article/ view/19849 .

Amir Pnueli. The temporal logic of programs. In 18th Annual Symposium on Foundations of Computer Science, Providence, Rhode Island, USA, 31 October - 1 November 1977 , pages 46-57. IEEE Computer Society, 1977. doi: 10.1109/SFCS.1977.32. URL https://doi.org/ 10.1109/SFCS.1977.32 .

Stephen Prajna, Ali Jadbabaie, and George J. Pappas. Stochastic safety verification using barrier certificates. In 43rd IEEE Conference on Decision and Control, CDC 2004, Nassau, Bahamas, December 14-17, 2004 , pages 929-934. IEEE, 2004. doi: 10.1109/CDC.2004.1428804. URL https://doi.org/10.1109/CDC.2004.1428804 .

Stephen Prajna, Ali Jadbabaie, and George J. Pappas. A framework for worst-case and stochastic safety verification using barrier certificates. IEEE Trans. Autom. Control. , 52(8):14151428, 2007. doi: 10.1109/TAC.2007.902736. URL https://doi.org/10.1109/TAC.2007. 902736 .

Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . Wiley Series in Probability and Statistics. Wiley, 1994. ISBN 978-0-47161977-2. doi: 10.1002/ 9780470316887. URL https://doi.org/10.1002/9780470316887 .

Spencer M. Richards, Felix Berkenkamp, and Andreas Krause. The lyapunov neural network: Adaptive stability certification for safe learning of dynamical systems. In 2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings , volume 87 of Proceedings of Machine Learning Research , pages 466-476. PMLR, 2018. URL http://proceedings.mlr.press/v87/richards18a.html .

Cesar Santoyo, Maxence Dutreix, and Samuel Coogan. A barrier function approach to finite-time stochastic system verification and control. Autom. , 125:109439, 2021. doi: 10.1016/j.automatica. 2020.109439. URL https://doi.org/10.1016/j.automatica.2020.109439 .

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Aivar Sootla, Alexander Cowen-Rivers, Jun Wang, and Haitham Bou Ammar. Enhancing safe exploration using safety state augmentation. Advances in Neural Information Processing Systems , 35:34464-34477, 2022.

Aivar Sootla, Alexander I. Cowen-Rivers, Taher Jafferjee, Ziyan Wang, David Henry Mguni, Jun Wang, and Haitham Ammar. Saute RL: almost surely safe reinforcement learning using state augmentation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 20423-20443. PMLR, 2022. URL https://proceedings.mlr. press/v162/sootla22a.html .

Sadegh Esmaeil Zadeh Soudjani, Caspar Gevaerts, and Alessandro Abate. FAUST 2 : Formal abstractions of uncountable-state stochastic processes. In Christel Baier and Cesare Tinelli, editors, Tools and Algorithms for the Construction and Analysis of Systems - 21st International Conference, TACAS 2015, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2015, London, UK, April 11-18, 2015. Proceedings , volume 9035 of Lecture Notes in Computer Science , pages 272-286. Springer, 2015. doi: 10.1007/ 978-3-662-46681-0\_23. URL https://doi.org/10.1007/978-3-662-46681-0_23 .

Sean Summers and John Lygeros. Verification of discrete time stochastic hybrid systems: A stochastic reach-avoid decision problem. Autom. , 46(12):1951-1961, 2010. doi: 10.1016/j. automatica.2010.08.006. URL https://doi.org/10.1016/j.automatica.2010.08.006 .

Dawei Sun, Susmit Jha, and Chuchu Fan. Learning certified control using contraction metric. In Jens Kober, Fabio Ramos, and Claire J. Tomlin, editors, 4th Conference on Robot Learning, CoRL 2020, 16-18 November 2020, Virtual Event / Cambridge, MA, USA , volume 155 of Proceedings of Machine Learning Research , pages 1519-1539. PMLR, 2020. URL https: //proceedings.mlr.press/v155/sun21b.html .

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014. URL http://arxiv.org/abs/1312.6199 .

Toru Takisaka, Yuichiro Oyabu, Natsuki Urabe, and Ichiro Hasuo. Ranking and repulsing supermartingales for reachability in randomized programs. ACM Trans. Program. Lang. Syst. , 43(2):5:1-5:46, 2021. doi: 10.1145/3450967. URL https://doi.org/10.1145/3450967 .

Matteo Turchetta, Felix Berkenkamp, and A. Krause. Safe exploration for interactive machine learning. In NeurIPS , 2019.

Pashootan Vaezipoor, Andrew C. Li, Rodrigo Toro Icarte, and Sheila A. McIlraith. Ltl2action: Generalizing LTL instructions for multi-task RL. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 1049710508. PMLR, 2021. URL http://proceedings.mlr.press/v139/vaezipoor21a.html .

Abraham P. Vinod, Joseph D. Gleason, and Meeko M. K. Oishi. Sreachtools: a MATLAB stochastic reachability toolbox. In Necmiye Ozay and Pavithra Prabhakar, editors, Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control, HSCC 2019, Montreal, QC, Canada, April 16-18, 2019 , pages 33-38. ACM, 2019. doi: 10.1145/3302504.3311809. URL https://doi.org/10.1145/3302504.3311809 .

Jun Wang, Samarth Kalluraya, and Yiannis Kantaros. Verified compositions of neural network controllers for temporal logic control objectives. In 61st IEEE Conference on Decision and Control, CDC 2022, Cancun, Mexico, December 6-9, 2022 , pages 4004-4009. IEEE, 2022. doi: 10.1109/CDC51059.2022.9993010. URL https://doi.org/10.1109/ CDC51059.2022.9993010 .

Yixuan Wang, Simon Sinong Zhan, Ruochen Jiao, Zhilu Wang, Wanxin Jin, Zhuoran Yang, Zhaoran Wang, Chao Huang, and Qi Zhu. Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 36593-36604. PMLR, 2023. URL https://proceedings.mlr.press/v202/wang23as.html .

David Williams. Probability with Martingales . Cambridge mathematical textbooks. Cambridge University Press, 1991. ISBN 978-0-521-40605-5.

Bai Xue, Renjue Li, Naijun Zhan, and Martin Fränzle. Reach-avoid analysis for stochastic discrete-time systems. In 2021 American Control Conference, ACC 2021, New Orleans, LA, USA, May 25-28, 2021 , pages 4879-4885. IEEE, 2021. doi: 10.23919/ACC50511.2021. 9483095. URL https://doi.org/10.23919/ACC50511.2021.9483095 .

Cambridge Yang, Michael L. Littman, and Michael Carbin. On the (in)tractability of reinforcement learning for LTL objectives. In Luc De Raedt, editor, Proceedings of the ThirtyFirst International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022 , pages 3650-3658. ijcai.org, 2022. doi: 10.24963/ijcai.2022/507. URL https://doi.org/10.24963/ijcai.2022/507 .

Mojtaba Zarei, Yu Wang, and Miroslav Pajic. Statistical verification of learning-based cyberphysical systems. In Aaron D. Ames, Sanjit A. Seshia, and Jyotirmoy Deshmukh, editors, HSCC '20: 23rd ACM International Conference on Hybrid Systems: Computation and Control, Sydney, New South Wales, Australia, April 21-24, 2020 , pages 12:1-12:7. ACM, 2020. doi: 10.1145/3365365.3382209. URL https://doi.org/10.1145/3365365.3382209 .

Dorde Zikelic, Mathias Lechner, Thomas A. Henzinger, and Krishnendu Chatterjee. Learning control policies for stochastic systems with reach-avoid guarantees. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023 , pages 11926-11935. AAAI Press, 2023. doi: 10.1609/aaai.v37i10.26407. URL https://doi.org/10.1609/aaai.v37i10.26407 .

Appendix

A SPECTRL syntax and semantics and the proof of Theorem 4

Syntax A specification in SPECTRL is defined in terms of predicates and specification formulas . An atomic predicate is a boolean function a : X → { true , false } which for each system state specifies whether it satisfies the predicate, and a predicate is defined as a boolean combination of atomic predicates. Specification formulas in SPECTRL are then defined by the grammar

where b is a predicate and ϕ 1 and ϕ 2 are specification formulas. Intuitively, achieve b requires the agent to reach a state in which the predicate b is satisfied and ϕ 1 ensuring b requires the agent to satisfy the specification ϕ while only visiting states in which the predicate b is satisfied. The clause ϕ 1 ; ϕ 2 requires the agent to first satisfy specification ϕ 1 and then satisfy specification ϕ 2 . Finally, ϕ 1 or ϕ 2 requires satisfaction of at least one of the specifications ϕ 1 or ϕ 2 .

Semantics Given a trajectory ρ = ( x t , u t , ω t ) ∞ t =0 and writing ρ K = ( x t , u t , ω t ) K t =0 for its finite prefix of length K , the semantics of each SPECTRL clause are formally defined as follows:

Here, ρ K t denotes the t -th state along ρ K , ρ K [0: t ] denotes the prefix of ρ K consisting of the first t +1 states along ρ K and ρ K [ t : K ] denotes the suffix of ρ K that starts in the ( t +1) -st state along ρ K .

Theorem 4. For each ϕ ∈ Finitary there exists ϕ ′ ∈ SPECTRL such that, for any word w , the word w is accepted by ϕ iff the word w is accepted by ϕ ′ .

Proof. Let ϕ be a finitary specification defined over the set of atomic predicates AP . Since ϕ is finitary, there exist a finite time horizon H and a set L of words over AP of length H such that an infinite word over the alphabet AP is accepted by ϕ iff its prefix of length H is contained in L . Define a SPECTRL formula ϕ ′ via:

where each p ( w i ) is an atomic predicate associated to the i -th letter in the word ( w 1 , . . . , w H ) and ; denotes sequential composition of SPECTRL specifications. Then, an infinite word w is accepted by ϕ if and only if the prefix of w of length H is contained in L , which holds if and only if w is accepted by the SpectRL formula ϕ ′ . This completes our reduction.

B Abstract Reachability Definition and Proof of Theorem 5

Given a trajectory ρ = ( x t , u t , ω t ) ∞ t =0 of the system and an abstract graph G = ( V, E, β, s, t ) , we say that ρ satisfies abstract reachability for G (written ρ | = G ) if it gives rise to a path in G that traverses G from s to t and satisfies reach-avoid specifications of all traversed edges. Formally, we require that there exists a sequence of time steps 0 = i 0 < i 1 < · · · < i k and a finite path s = v 0 , v 1 , . . . , v k = t in G such that

Intuitively, the first condition encodes that the trajectory satisfies reachability specifications of traversed vertices in G while the second condition encodes that it satisfies avoidance specifications of traversed edges in G . We then say that a policy π for the system satisfies abstract reachability for G with probability p ∈ [0 , 1] at an initial state x 0 ∈ X 0 , if we have that P x 0 [ ρ ∈ Ω x 0 | ρ | = G ] ≥ p .

We now provide the proof of Theorem 5.

Theorem 5. Consider a stochastic feedback loop system with an initial set of states X 0 ⊆ X and let ϕ be a SPECTRL specification. Then there exists an abstract graph G = ( V, E, β, s, t ) with | V | in O ( | ϕ | ) such that, for each trajectory ρ of the system, we have ρ | = ϕ if and only if ρ | = G . Hence, for each policy π and initial state x 0 ∈ X 0 , we have P x 0 [ ρ ∈ Ω x 0 | ρ | = ϕ ] = P x 0 [ ρ ∈ Ω x 0 | ρ | = G ] .

Proof. Given a SPECTRL specification ϕ , one can construct an abstract graph G such that for each trajectory ρ of the system we have ρ | = ϕ iff ρ | = G as follows. First, the specification ϕ is parsed according to the grammar of SPECTRL in eq. 1 in order to construct the parse tree of ϕ . We then start by constructing an abstract graph for each leaf formula in the parse tree, and traverse the parse tree bottom-up in order to construct abstract graphs of parent formulas. The abstract graph of the specification ϕ is then obtained by taking the abstract graph constructed for the root in the parse tree. The leaves of the parse tree are formulas of the form achieve p , for which we construct an abstract graph with two vertices s and t , a single edge e = ( s, t ) and set β ( s ) = X 0 , β ( t ) = { x ∈ X | p ( x ) = true } and β ( e ) = X . For a formula ϕ 1 ensuring p , we take an abstract graph ( V, E, β, s, t ) for the specification ϕ 1 which was already constructed for the child node and define the abstract graph G = ( V, E, β ′ , s, t ) by simply modifying the map β via β ′ ( e ) = β ( e ) ∩ { x ∈ X | p ( x ) = true } for each e ∈ E . For a formula ϕ 1 ; ϕ 2 , we take the abstract graph of the specifications ϕ 1 and ϕ 2 which were already constructed for the child nodes and merge them by identifying the target node of ϕ 1 with the source node of ϕ 2 and using the region associated to it by the abstract graph of ϕ 2 . Finally, for a formula ϕ 1 or ϕ 2 , we introduce a novel source node s with β ( s ) = X 0 , take the abstract graph of ϕ 1 and ϕ 2 and connect the novel source node s to them by an edge. Note that this construction yields a graph with V in O ( | ϕ | ) .

Since the above construction soundly encodes the semantics of each SPECTRL grammar element as a reach-avoid specification, it follows by induction on the depth of the parse tree that for each trajectory ρ of the system we have ρ | = ϕ iff ρ | = G . The claim of Theorem 5 follows.

C Proof of Theorem 1

Theorem 1. [Proof in Apendix C] The following two statements hold:

If a continuous function V : X → R is an ( ϵ, λ ) -additive RASM, then it is also a ( λ -ϵ λ , min { ϵ, λ } , λ ) -multiplicative RASM.

If a continuous function V : X → R is a ( γ, δ, λ ) -multiplicative RASM, then it is also an ((1 -γ ) · δ, λ ) -additive RASM.

Proof.

Let δ = min { ϵ, λ } and γ = λ -ϵ λ . To show that V is a ( γ, δ, λ ) -multiplicative RASM, we need to show that the Strict positivity outside X t and the Multiplicative expected decrease conditions hold. By the Additive expected decrease condition, for each x ∈ X\X t at which V ( x ) ≤ λ we have V ( x ) ≥ ϵ . So as δ = min { ϵ, λ } , the Strict positivity outside X t follows. On the other hand, observe that for every x ∈ X\X t at which V ( x ) ≤ λ , we have

where the first inequality follows by the Additive expected decrease condition and the second inequality follows since z -ϵ z is monotonically increasing on the domain z > ϵ . Hence, the Multiplicative expected decrease condition holds.

Let ϵ = (1 -γ ) · δ . To show that V is an ( ϵ, λ ) -additive RASM, we need to show that the Additive expected decrease condition holds. We show this by observing that, for each x ∈ X\X t such that V ( x ) ≤ λ , we have

where the first inequality holds by the Multiplicative expected decrease condition and the last inequality holds by the Strict positivity outside X t condition. Hence, the Additive expected decrease condition is satisfied.

D Proof of Theorem 2

We first provide an overview of definitions and results from martingale theory that we use in the proof. We then present the proof.

Probability theory A probability space is a triple (Ω , F , P ) of a state space Ω , a sigma-algebra F and a probability measure P that satisfies Kolmogorov axioms [64]. A random variable in (Ω , F , P ) is a function X : Ω → R that is F -measurable, i.e. for each a ∈ R we have { ω ∈ Ω | X ( ω ) ≤ a } ∈ F . A (discrete-time) stochastic process is a sequence ( X i ) ∞ i =0 of random variables in (Ω , F , P ) .

Conditional expectation Let X be a random variable in (Ω , F , P ) . Given a sub-sigma-algebra F ′ ⊆ F , a conditional expectation of X given F ′ is an F ′ -measurable random variable Y such that, for each A ∈ F ′ , we have

Here, I ( A ) : Ω → { 0 , 1 } is an indicator function of A , given by I ( A )( ω ) = 1 if ω ∈ A , and I ( A )( ω ) = 0 if ω ̸∈ A . Intuitively, conditional expectation of X given F ′ is an F ′ -measurable random variable that behaves like X upon evaluating its expected value on events in F ′ . It is known that every nonnegative random variable admits a conditional expectation [64]. Moreover, the conditional expectation is almost-surely unique, meaning that for any two F ′ -measurable random variables Y and Y ′ which are conditional expectations of X given F ′ we have P [ Y = Y ′ ] = 1 . Therefore, we pick any such random variable as a canonical conditional expectation and denote it by E [ X | F ′ ] .

Supermartingales Let (Ω , F , P ) be a probability space and ( F i ) ∞ i =0 be an increasing sequence of sub-sigma-algebras in F , i.e. F 0 ⊆ F 1 ⊆ · · · ⊆ F . A nonnegative supermartingale with respect to ( F i ) ∞ i =0 is a stochastic process ( X i ) ∞ i =0 such that each X i is F i -measurable, and X i ( ω ) ≥ 0 and E [ X i +1 | F i ]( ω ) ≤ X i ( ω ) hold for each ω ∈ Ω and i ≥ 0 . Intuitively, the second condition is the expected decrease condition, and it is formally captured via conditional expectation.

We now present two results from martingale theory that will be used in the proof. Let (Ω , F , P ) be a probability space and ( F i ) ∞ i =0 be an increasing sequence of subσ -algebras in F .

Theorem 6 (Supermartingale convergence theorem [64]) . Let ( X i ) ∞ i =0 be a nonnegative supermartingale with respect to ( F i ) ∞ i =0 . Then, there exists a random variable X ∞ in (Ω , F , P ) to which the supermartingale converges to with probability 1 , i.e. P [lim i →∞ X i = X ∞ ] = 1 .

Theorem 7 ( [35]) . Let ( X i ) ∞ i =0 be a nonnegative supermartingale with respect to ( F i ) ∞ i =0 . Then, for every λ > 0 , we have

Theorem 2. [Proof in Appendix D] Let γ ∈ (0 , 1) , δ > 0 and λ > 1 , and suppose that V : X → R is a ( γ, δ, λ ) -multiplicative RASM with respect to X t and X u . Suppose furthermore that V is Lipschitz continuous with a Lipschitz constant L V , and that the system under policy π satisfies the bounded step property , i.e. that there exists ∆ > 0 such that || x -f ( x, π ( x ) , ω ) || 1 ≤ ∆ holds for each x ∈ X and ω ∈ W . Let N = ⌊ ( λ -1) / ( L V · ∆) ⌋ . Then, for every x 0 ∈ X 0 , we have that

Proof. Fix an initial state x 0 ∈ X 0 . We need to show that P x 0 [ ReachAvoid ( X t , X u )] ≥ 1 -1 λ · γ N with N = ⌊ ( λ -1) / ( L V · ∆) ⌋ .

Before proceeding with the proof, we define some notions. Consider the probability space (Ω x 0 , F x 0 , P x 0 ) over the set of all system trajectories that start in x 0 ∈ X 0 . For each time step t ∈ N 0 , define F x 0 ,t ⊆ F x 0 to be a subσ -algebra which contains events that are defined in terms of the first t states along a trajectory. Formally, for each j ∈ N 0 , let C j : Ω x 0 →X assign to each trajectory ρ = ( x t , u t , ω t ) t ∈ N 0 ∈ Ω x 0 the j -th state x j along the trajectory. F i is then defined as the smallest σ -algebra over Ω x 0 with respect to which C 0 , C 1 , . . . , C i are all measurable. The sequence ( F x 0 ,t ) ∞ t =0 defines a filtration in (Ω x 0 , F x 0 , P x 0 ) .

Proceeding with the proof, we show that V induces a supermartingale in the probability space (Ω x 0 , F x 0 , P x 0 ) over the set of all system trajectories that start in x 0 ∈ X 0 . For each t ∈ N 0 , define a random variable X t in (Ω x 0 , F x 0 , P x 0 ) via

for each trajectory ρ = ( x t , u t , ω t ) t ∈ N 0 ∈ Ω x 0 . Intuitively, X t is equal to V at x t until either the target set X t is reached upon which X t is set to 0 , or some V ( x t ) ≥ λ is reached upon which X t is set to λ . We claim that ( X t ) ∞ t =0 is a nonnegative supermartingale with respect to the filtration ( F x 0 ,t ) ∞ t =0 in the probability space (Ω x 0 , F x 0 , P x 0 ) and that it with probability 1 converges to either 0 or to a value that is ≥ λ .

Clearly, each X t is nonnegative. To prove that ( X t ) ∞ t =0 is a supermartingale, note first that each X t is F x 0 ,t measurable as it is defined in terms of the ferst t states along a trajectory. To show that the expected decrease condition is satisfied, we show that E x 0 [ X t +1 | F x 0 ,t ]( ρ ) ≤ X t ( ρ ) holds for each t ∈ N 0 and ρ = ( x t , u t , ω t ) t ∈ N 0 . We consider 3 cases based on the definition of V :

1.

The first equality follows by the law of total probability, the second equality follows by definition of X t , the third inequality follows by observing that V ( x t +1 ) ≥ X t +1 ( ρ ) if x t +1 ∈ X t , the fourth equality is just the sum of expectations over disjoint sets, and finally the fifth inequality follows by the Multiplicative expected decrease condition of V and the assumption that x t ̸∈ X t and V ( x t ) < λ .

If x i ∈ X t for some 0 ≤ i ≤ t and V ( x j ) < λ for all 0 ≤ j ≤ i , then we have E x 0 [ X t +1 | F x 0 ,t ]( ρ ) = γ · X t +1 = X t +1 ( ρ ) = 0 .

Otherwise, we must have V ( x i ) ≥ λ and x 0 , . . . , x i ̸∈ X t for some 0 ≤ i ≤ t , thus E x 0 [ X t +1 | F x 0 ,t ]( ρ ) = X t +1 ( ρ ) = λ .

Hence, we have proved that ( X t ) ∞ t =0 is a nonnegative supermartingale.

By Supermartingale Convergence Theorem, we then know that ( X t ) ∞ t =0 with probability 1 converges to some value. We claim furthermore that this value is either 0 or ≥ λ and that the value is attained. To see this, recall that in Theorem 1 we showed that V is also an ( ϵ, λ ) -additive RASM with ϵ = (1 -γ ) · δ . Then, the same sequence of inequalities as in the case one above shows that E x 0 [ X t +1 | F x 0 ,t ]( ρ ) ≤ X t ( ρ ) -ϵ if x 0 , x 1 , . . . , x t ̸∈ X t and V ( x i ) < λ for each 0 ≤ i ≤ t . Thus, the limit to which ( X t ) ∞ t =0 converges cannot be in the open interval (0 , λ ) and the claim follows.

To prove the theorem claim, we show that ( X t ) ∞ t =0 converges to a value with ≥ λ with probability at most 1 λ · γ N . Then, since ( X t ) ∞ t =0 converging to 0 implies that system reaches a state in which V < δ while never reaching a state in which V ≥ λ , which by the Safety and the Strict positivity outside X t conditions implies that reach-avoidance is satisfied, this will imply the theorem claim. The proof until this point is analogous to the proof of [68, Theorem 1].

The technical novelty of our proof begins in the following step. We define another stochastic process ( Y t ) ∞ t =0 from ( X t ) ∞ t =0 by letting

We claim that ( Y t ) ∞ t =0 is also a nonnegetive supermartingale with respect to the filtration ( F x 0 ,t ) ∞ t =0 . The nonnegativity part of the claim clearly holds since each X t is nonnegative. To check the expected decrease condition of supermartingales, for each t ∈ N 0 and for each ρ ∈ Ω x 0 we have distinguish two cases:

The first equality holds by the law of total probability. The second equality holds by the definition of Y t +1 . The third equality holds by the law of total probability. The fourth inequality holds since above we proved that E x 0 [ X t +1 | F x 0 ,t ]( ρ ) ≤ γ · X t whenever V ( x i ) < λ for all 0 ≤ i ≤ t (cases 1 and 2 above). The fifth equality holds by definition of Y t and the assumption that V ( x i ) < λ for all 0 ≤ i ≤ t . Finally, the sixth inequality follows by observing that in the case when V ( x i ) < λ for all 0 ≤ i ≤ t but V ( x t +1 ) ≥ λ , we have X t +1 /γ t +1 ≥ λ/γ t +1 ≥ λ/γ t ≥ Y t .

Hence, we have proved that ( Y t ) ∞ t =0 is a nonnegative supermartingale.

We conclude the theorem claim by observing that

Three non-trivial steps are the first and the second equality and the last inequality. The first equality holds since, if sup t ≥ 0 X t < λ , then we also have Y t = X t /γ t for each t by the definition of Y t . The second equality holds since the system cannot reach a state in which V ≥ λ and so X t ≥ λ in less than N time steps. On the other hand, for the last inequality we the inequality in Theorem 7. Applying the inequality to ( Y t ) ∞ t =0 and λ γ N and observing that E [ Y 0 ] = E [ X 0 ] ≤ 1 by the Initial

E Learning Policies with Reach-avoid Supermartingales

We now present the POLICY+RASM subprocedure that we use for simultaneously learning a policy π µ and an RASM V θ , both of which are parametrized as neural networks with parameters µ and θ . The subprocedure POLICY+RASM is identical to the algorithm of [68], thus we keep this exposition brief and refer the reader to [68] for details. The reason why we can reuse this algorithm even though it learns additive RASMs is that additive and multiplicative RASMs are equivalent by Thereom 1. As we show below, the algorithm does not need to explicitly set an additive term ϵ or a multiplicative factor γ , thus it is applicable to learning both additive and multiplicative RASMs. We then show how to use V θ to extract the bound in Theorem 2.

Analogously as in [68], the value λ > 1 in Definition 3 is an algorithm parameter and we initialize it to λ = 1 1 -p ′ so that the Theorem 2 bound 1 -1 λ · γ N ≥ 1 -1 λ = p ′ implies satisfaction of the desired probabilistic reach-avoid specification. If the algorithm succeeds in learning π µ and V θ with this value of λ , then the reach-avoid problem is solved. Otherwise, the algorithm gradually decreases the value of λ and tries to relearn π µ and V θ so that the resulting bound in Theorem 2 still exceeds p ′ . Thus, our new bound also yields an improvement in the algorithm.

The algorithm consists of two modules called learner and verifier , which are composed into a loop. In each loop iteration, the learner first learns a policy π µ and an RASM candidate V θ . These are then passed to the verifier which formally checks whether V θ satisfies all conditions in Definition 2. If the verification is successful, the algorithm returns the policy. Otherwise, the verifier identifies counterexample states at which the additive RASM conditions are violated. These are then passed to the learner and are used to fine-tune the previously learned policy and RASM by refining the loss function using the computed counterexamples.

Learner Apolicy π µ and an additive RASM candidate V θ are learned by minimizing the loss function

The first three loss terms are constructed from the sets C init , C unsafe and C dec which are initialized by computing finite discretizations of X 0 , X u and X\X t and are later extended by counterexamples computed by the verifier. The loss terms are used to guide the learner towards learning a true additive RASM which satisfies the Initial, Safety and Expected decrease conditions. Each loss term is designed to incur a loss at a counterexample whenever that counterexample violates the corresponding condition. In order for the Nonnegativity condition to be satisfied by default, the algorithm applies the softplus activation function to the output of V θ . The loss term L Lipschitz ( θ, ν ) is a regularization term that does not enforce any of the defining conditions of additive RASMs, however it helps in decreasing the Lipschitz constants of neural networks. Each loss term is defined as follows:

The last loss term L Lipschitz ( θ, ν ) = t · ( L Lipschitz ( θ ) + L Lipschitz ( ν )) is the regularization term used to guide the learner towards learning neural networks whose Lipschitz constants are below a tolerable threshold ρ , where t > 0 is a regularization constant. By preferring networks with small Lipschitz constants, we allow the verifier to use a wider mesh and thus make verification condition easier to satisfy. We have The regularization term for π θ (and analogously for V ν ) is defined via

where W and b weight matrices and bias vectors for each layer in π θ .

Verifier The verifier checks whether V θ satisfies the defining properties of additive RASMs in Definition 2. Recall, the Nonnegativity condition is satisfied by default due to the softplus activation function applied to the output layer of V θ . Hence, the verifier only needs to check the Initial, Safety and Expected decrease conditions.

Since f , π µ and V θ are continuous functions defined over a compact domain X thus also Lipschitz continuous, the verifier may check the (both Multiplicative and Additive) expected decrease condition by checking a slightly stricter condition at finitely many discretization points. A discretization of the state space X with mesh τ > 0 is a finite set ˜ X ⊆ X such that, for every x ∈ X , there exists ˜ x ∈ ˜ X such that || x -˜ x || 1 < τ . The discretization is computed by taking a grid of mesh τ . Then, to check the expected decrease condition, it was showed in [68] that it suffices to check for each ˜ x ∈ ˜ X whose

adjacent discretization grid cells contain a non-target state and over which V attains a value that is less than or equal to λ that

where K = L V · ( L f · ( L π + 1) + 1) and L f , L π and L V are Lipschitz constants of f , π µ and V θ . It is assumed that L f is provided and L π and L V are computed by the method of [57]. To verify the Initial condition, the verifier collects the set Cells X 0 of all cells of the discretization grid that intersect the initial set X 0 . Then, for each cell ∈ Cells X 0 , it checks if sup x ∈ cell V θ ( x ) ≤ 1 , where the supremum is bounded from above by using interval arithmetic abstract interpretation (IA-AI) [22, 27] to propagate across neural network layers the extreme values that V θ can attain over a cell. Similarly, to verify the Unsafe condition, the verifier collects the set Cells X u of all cells of the discretization grid that intersect the initial set X u . Then, for each cell ∈ Cells X u , it uses IA-AI to check if inf x ∈ cell V θ ( x ) ≥ λ .

If the verifier shows that V θ satisfies the above checks, it concludes that V θ is an additive (and therefore multiplicative) RASM for the system under the policy π µ and returns the policy together with the lower bound on the probability of satisfying the reach-avoid specification as in Theorem 2. The fact that the verifier is correct was proved in [68, Theorem 2]. Otherwise, if a counterexample ˜ x to any of the checks is found, it is added to one of the three counterexample sets C init , C unsafe and C dec that are then used by the learner to fine-tune V θ and π µ .

Soundness and computation of γ . The following theorem establishes that the above is a sound verification procedure and provides a closed-form expression for the values of δ > 0 and γ ∈ (0 , 1) for which V θ is a ( γ, δ, λ ) -multiplicative RASM. Hence, to compute the lower bound on the probability of satisfying reach-avoidance in Theorem 2, one may use the value of γ implied by the theorem together with λ which is fixed by the algorithm, L V which is computed by the algorithm and the maximal step size ∆ which we assume is provided by the user.

Theorem 8. If the verifier returns neural networks π µ and V θ , then V θ is a ( γ, δ, λ ) -multiplicative RASM with

Proof. Since it was shown in [68] that the verifier provides a sound verification procedure for checking that V θ is an additive RASM, by the equivalence in Theorem 1 it follows that it is also sound for checking that V θ is a ( γ, δ, λ ) -multiplicative RASM for some values of γ and δ . Hence, it remains to show that the values of γ and δ in the theorem statement are correct.

First, we show that the Strict positivity outside X t condition is satisfied with the above value of δ . To see this, let x ∈ X\X t . Since δ ≤ λ , suppose without loss of generality that V θ ( x ) ≤ λ . Then, x is contained in a discretization grid cell which contains a non-target state and over which V attains a value that is ≤ λ . Hence, the verifier has shown that E ω ∼ d [ V θ ( f (˜ x , π (˜ x ) , ω ))] < V θ (˜ x ) -τ · K holds for each ˜ x ∈ ˜ X which is a vertex of this cell. Taking a vertex ˜ x ∈ ˜ X of this cell for which || x -˜ x || 1 ≤ τ , by the definition of Lipscthiz constants we have

Hence, by the Nonnegativity condition we also have

since K = L V · ( L f · ( L π +1) + 1) , which concludes the proof.

Second, we show that Multiplicative expected decrease condition with the above value of γ holds. Let x ∈ X\X t be such that V θ ( x ) ≤ λ . We need to show that γ · V ( x ) ≥ E ω ∼ d [ V ( f ( x , π ( x ) , ω ))] . Since we showed in eq. (2) that V ( x ) ≥ V ( x ) -E ω ∼ d [ V ( f ( x , π ( x ) , ω ))] ≥ δ > 0 and since V θ ( x ) ≤ λ , we have

This concludes the proof.

F Proof of Theorem 3

Theorem 3. [Proof in Appendix F] Algorithm 1 is compositional, and if it outputs a policy π , then π guarantees the probabilistic specification ( ϕ, p ) .

Proof. In order to prove that π guarantees satisfaction of the probabilistic specification ( ϕ, p ) , by Theorem 5 it suffices to show that π satisfies abstract reachability for the abstract graph G with probability at least p .

To prove abstract reachability for G with probability at least p , we show that a random trajectory of the system under policy π satisfies reach-avoid specifications of the edges along the finite path s = v i 0 , v i 1 , . . . , v i k = t exhibited above with probability at least p . To prove this, we proceed by induction on 0 ≤ j ≤ k to show that a random trajectory of the system under policy π satisfies reachavoid specifications of each edge along a prefix s = v i 0 , v i 1 , . . . , v i j of this path with probability at least Prob[ v i j ] . Recall, Prob is the dictionary computed by Algorithm 1. Abstract reachability for G with probability at least p then follows if we set j = k , since v j k = t and we must have Prob[ t ] ≥ p for Algorithm 1 to output a policy (lines 15-17).

The base case j = 0 follows trivially since the system starts in the initial region β ( s ) = X 0 and since Prob[ v i 0 ] = Prob[ s ] = 1 by line 6. For the inductive step, suppose that 0 ≤ j ≤ k -1 and that π satisfies reach-avoid specifications of each edge along s = v i 0 , v i 1 , . . . , v i j with probability at least Prob[ v i j ] . The claim for the prefix of length j +1 then follows by our construction of the finite path s = v i 0 , v i 1 , . . . , v i k = t , as it implies that Algorithm 1 has successfully learned an edge policy for the edge ( v i j , v i j +1 ) that ensures satisfaction of the associated reach-avoid specification with probability at least p ( v i j ,v i j +1 ) and for which Prob[ v i j +1 ] = p ( v i j ,v i j +1 ) · Prob[ v i j ] . Since the right-hand-side of this equality is a lower bound on the probability of π satisfying reach-avoid specifications of each edge along s = v i 0 , v i 1 , . . . , v i j multiplied by a lower bound on the probability of it satisfying the reach-avoid specification of the edge ( v i j , v i j +1 ) , the claim follows. This concludes the proof by induction.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a verifiable RL framework that learns a composition of NN policies for stochastic control systems, along with a formal supermartingale certificate for the safety probability of satisfying the reach-avoid specification. It decomposes the global reach-avoid task into a DAG with edges denoting subtasks which are solved by policy+RASM. Finally, it composes the low-level subtask's policy into a global safe policy with probability guarantees. The authors evaluate this framework in a relatively simple Stochastic Nine Room environment.\n\nSTRENGTHS:\nThis paper is greatly written and easy to follow. The authors did a good job on the paper presentation. They study an important safe RL problem with step-wise safety chance constraint, which is suitable for safety-critical systems.\n\nThe proposed compositional safe RL framework is novel to me. \n\nThe proposed algorithm and approach are sound.  \n\nControl policy learning with formal guarantees in probability is significant.\n\nWEAKNESSES:\n1. My biggest concern is the scalability problem of this paper, which might originate from the DAG representation. If the global RL task is too complex, the graph might be too huge to handle. \n\n2. Because of the scalability problem, the experiments in this paper look much simpler than other RL papers. \n\n3. The authors may miss a few recent papers addressing a similar safe RL problem with chance constraints, as in \n\na)  Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments,\n\nas far as I can tell, this reference also considers continuous control problems with safety probability chance constraints, and its guarantees are also based on the supermartingale property, although the approaches are quite different.\n\nQUESTIONS:\n1. What are the relations between (either additive or multiplicative)  RASM and barrier certificate for stochastic (either continuous or discrete) systems? \nThe definition of the barrier certificate can be found in the following references. \n\nb) Stochastic safety verification using barrier certificates. \n\nc) A Barrier Function Approach to Finite-Time Stochastic System Verification and Control. \n\n2. What's the complexity of the proposed algorithm? It should be feasible to conduct the big  O complexity analysis as the algorithm is built on top of topologic sorting, binary searching, etc.\n\nLIMITATIONS:\nThis paper may face a scalability problem.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CLAPS (Compositional Learning for Probabilistic Specifications), a new method for learning a composition of neural network policies in stochastic environments, together with a formal certificate which guarantees that a reach-avoid specification over the policy's behavior is satisfied with the desired probability.\nThe proposed approach is evaluated empirically on a stochastic Nine Rooms environment.\n\nSTRENGTHS:\n1) Different to previous works discussed by the authors, the CLAPS method is applicable to stochastic control systems, that may be defined via non-linear dynamics functions.\n\n2) The approach is compositional. Complex tasks are analysed and solved in terms of simple tasks, that are then combined in order to achieve the overall goal.\n\n3) The literature is discussed in detail. The bibliography, at 61 items, is an excellent overview of current methods in safe RL.\n\nWEAKNESSES:\n1) A lot of material appears in the appendix. I did not check it, but the paper appears to be consistent.\n\n3) This work builds on top of [61] as regards the use of supermartingales. However, the bounds obtained are stricter, which seems to justify the claims to novelty (together with the compositional approach). \n\nMinor, p. 3: When discussing the equation for the stochastic feedback loop system, u_t is described as the control action even though no u_t appears in the equation. On the other hand, function \\pi is not described. I understand that \\pi: X -> U is the policy and \\pi(x_t) = u_t.\n\nQUESTIONS:\np. 4: When the authors say \"if the probability of a random trajectory...\", do they actually mean \"if the probability of any trajectory...\"\n\nLIMITATIONS:\np. 9: The authors mention that \"the systematic decomposition used in our algorithm has advantages over manual task decompositions\". They might discuss these advantages further.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CLAPS, a compositional method designed for learning and verifying neural network policies in stochastic control systems. By considering control tasks with specifications expressed in the SPECTRL language, CLAPS decomposes the task into an abstract graph of reach-avoid tasks. It utilizes reach-avoid supermartingales to offer formal guarantees on the probability of reach-avoidance in each subtask. Additionally, the paper establishes proof demonstrating that RASMs (Reach-Avoid Supermartingales) provide a significantly more stringent lower bound on the probability of reach-avoidance compared to prior approaches. The experimental evaluation conducted in the Stochastic Nine Rooms environment showcases the ability of CLAPS to derive guarantees for global compositional policies.\n\nSTRENGTHS:\n+ Compositional Learning is an important research direction. The approach presented in this paper provides correctness guarantees for individual sub-policies that can be used to collectively ensure the correctness of the global policy, making the learning and verification approach valuable for applications where safety and reliability are critical. The ability to verify and validate each component of the policy offers a robust and trustworthy framework for developing complex control systems.\n+ Taking inspiration from exponential barrier certificates developed in the control theory community, the paper introduces a conceptually similar concept to improve Reach-Avoid Supermartingales, which provides a more strict lower bound on the probability of reach-avoidance guarantees (compared with prior work [61]).\n+ The main algorithm is easy to understand, follow, and implement.\n\nWEAKNESSES:\nThe proposed approach falls short of advancing the state-of-the-art in compositional learning and verification. While Reach-Avoid Supermartingales in prior work makes sense for providing probabilistic correctness guarantees for infinite time horizon systems, the expectation for a global policy composed of sub-policies is that each sub-policy terminates within a finite time horizon. Simpler techniques such as statistical verification, which relies on drawing a large number of samples and employing concentration inequalities, can achieve high-probability correctness guarantees for finite horizons. Consequently, the paper lacks a compelling argument showcasing why their probabilistic verification approach surpasses a straightforward statistical verification approach for compositional policies. Strengthening the paper would involve demonstrating that the probabilistic verification approach indeed outperforms stochastic verification in practice.\n\nIn the specific context of the Stochastic Nine Rooms environment examined in this paper, it seems that the robot can reach the goal within a finite time horizon. Given the convergence of PPO sub-policies, it is reasonable to anticipate that stochastic verification methods can yield substantially higher probabilistic guarantees compared to the reported result of 33% in the paper's probabilistic verification approach.\n\nQUESTIONS:\n+ Can the proposed Reach-Avoid Supermartingales (RASMs) approach be formally characterized in relation to exponential barrier certificates?\n\n+ Have you considered applying Statistical Verification of Learning-Based Cyber-Physical Systems (https://cpsl.pratt.duke.edu/sites/cpsl.pratt.duke.edu/files/docs/zarei_hscc20.pdf) to the Nine Rooms environments?\n\nLIMITATIONS:\nThe paper would benefit from a more comprehensive discussion and extensive experimentation regarding the advantages of the probabilistic verification approach over stochastic verification within the context of compositional policy learning.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents CLAPS, a compositional RL algorithm that also ensures guarantees of correctness when learning from temporal specifications. The core contribution of this work revolves around guarantees. \n\nPrior works in compositional RL with guarantees apply to deterministic environments and/or ones with linear dynamics. This work presents an algorithm that is suitable for stochastic environments with non-linear dynamics. The algorithm builds on two prior works., namely [61] that learns policies for control systems with non-linear dynamics with reach-avoid guarantees and [30] that compositionally learns policies (without guarantees) from temporal specifications. An empirical evaluation on 9-rooms environment over a suite of source and target states shows the efficacy of the approach in learning long-horizon tasks with guarantees.\n\nSTRENGTHS:\nRL from temporal logic to learn long-horizon tasks is a promising and thriving research community. This paper makes a strong contribution in that space by learning policies with guarantees in stochastic systems with non-linear dynamics.\n\nWEAKNESSES:\nThe empirical evaluation can be strengthened. Please find concrete comments below:\n\n1. The evaluations have been presented for one environment only. The 9-room environment is somewhat low-dimensional. I am curious about how the approach may scale to higher dimensional environments.\n2. Is there a comparison between the sample complexity of CLAPS with prior compositional approaches such as DiRL? This could offer a study of tradeoffs for guarantees.   \n3. Would it be possible to compare CLAPS to [27] on the benchmarks that would be common to both? It would be interesting to see how the formal guarantees of both approaches compare.\n\nQUESTIONS:\nI am quite unclear as to how the guarantee extends compositionally. From my understanding, a critical challenge in obtaining guarantees from compositional approaches such as [27] is that one needs to ensure the guarantee at the transition point between two edges, ie., in the region where one policy ends and the other begins. Could you clarify how that is accounted for in the proof of Theorem 5?\n\nThe example I have in mind is as follows: Say, one of the subgoal regions is split by a wall such that a learn policy entering the subgoal region ends before the wall and the policy exiting the subgoal region exits from the other side of the wall. The probability of connecting these two policies is clearly not 1. How is the guarantee in CLAPS accounting for such scenarios?\n\nLIMITATIONS:\nSee above. I can change my score based on the clarification of the question above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}