SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dorde Zikelic*
Klosterneuburg, Austria
djordje.zikelic@ist.ac.at
Mathias Lechner*
mlechner@mit.edu
Abhinay Verma
University Park, PA, USA
verma@psu.edu
Krishnendu Chatterjee
Klosterneburg, Austria
krishnendu.chatterjee@ist.ac.at
Thomas A. Henzinger
tah@ist.ac.at
Reinforcement learning has shown promising results in learning neural network policies for complicated control tasks. However, the lack of formal guarantees about the behavior of such policies remains an impediment to their deployment. We propose a novel method for learning a composition of neural network policies in stochastic environments, along with a formal certificate which guarantees that a specification over the policy's behavior is satisfied with the desired probability. Unlike prior work on verifiable RL, our approach leverages the compositional nature of logical specifications provided in SPECTRL, to learn over graphs of probabilistic reach-avoid specifications. The formal guarantees are provided by learning neural network policies together with reach-avoid supermartingales (RASM) for the graph's sub-tasks and then composing them into a global policy. We also derive a tighter lower bound compared to previous work on the probability of reach-avoidance implied by a RASM, which is required to find a compositional policy with an acceptable probabilistic threshold for complex tasks with multiple edge policies. We implement a prototype of our approach and evaluate it on a Stochastic Nine Rooms environment.
Reinforcement learning (RL) has achieved promising results in a variety of control tasks. However, the main objective of RL is to maximize expected reward [56] which does not provide guarantees of the system's safety. A more recent paradigm in safe RL considers constrained Markov decision processes (cMDPs) [5, 26, 3, 21], that in addition to a reward function are also equipped with a cost function. The goal in solving cMDPs is to maximize expected reward while keeping expected cost below some tolerable threshold. Recently, keeping cost below the threshold almost-surely and not only in expectation was considered in [52]. While these methods do enhance safety, they only empirically
try to minimize cost and do not provide guarantees on cost constraint satisfaction. The lack of formal guarantees raises a significant barrier to deployment of these methods in safety-critical applications, where the policies often need to encode complicated, long-horizon reasoning to accomplish a task in an environment in which unsafe behavior can lead to catastrophic consequences [6].
Recent work [13, 32, 33] has explored decomposing high-level logical specifications, belonging to different fragments of Linear Temporal Logic (LTL) [44], into simpler logical specification tasks that RL algorithms can solve more easily. In particular, these methods first decompose the high-level logical specification into a number of simpler logical specifications, then learn policies for these simpler tasks and finally compose the learned policies for each task into a global policy that solves the high-level control problem at hand. The simpler control tasks are solved by designing reward functions which faithfully encode the simpler logical specifications obtained by the decomposition and using an off-the-shelf RL algorithm to solve them.
While these methods present significant advances in deploying RL algorithms for solving complex logical specification tasks, their key limitation is that they also do not provide formal guarantees on the probability of satisfying the logical specification which are imperative for safety-critical applications. This is because reward functions often only "approximate" logical specification objectives, and furthermore RL algorithms are in general not guaranteed to converge to optimal policies over continuous state spaces. Recently, [30, 62] have proposed verification procedures for formally verifying a composition of neural network policies with respect to a given system dynamics model. However, these works assume that the underlying control systems have deterministic dynamics. This does not allow modelling stochastic disturbances and uncertainty in the underlying system dynamics. Furthermore, [62] assumes that the underlying control system has a linear dynamics function whereas many control tasks have non-linear dynamics.
In this work we propose CLAPS (Compositional Learning for Probabilistic Specifications), a new compositional algorithm for solving high-level logical specification tasks with formal guarantees on the probability of the specification being satisfied. We present a method for learning and verifying a composition of neural network policies learned via RL algorithms, which is applicable to stochastic control systems that may be defined via non-linear dynamics functions. Our method only requires that the underlying state space of the system is compact (i.e. closed and bounded) and that the dynamics function of the system is Lipschitz continuous. In terms of the expressiveness of logical specifications, CLAPS considers the SPECTRL specification language [32]. SPECTRL contains all logical specifications that can be obtained by sequential and disjunctive compositions of reach-avoid specifications. Given a target and an unsafe region, a reach-avoid specification requires that the system reaches the target region while avoiding the unsafe region.
Our method learns a policy along with a formal certificate which guarantees that a specification is satisfied with the desired probability. It consists of three key ingredients – (1) high-level planning on a directed acyclic graph (DAG) that decomposes the complex logical specification task into sequentially or disjunctively composed low-level reach-avoid tasks, (2) learning policies that solve low-level reach-avoid tasks with formal guarantees on the probability of satisfying reach-avoidance, and (3) composing the low-level reach-avoid policies into a global policy for the high-level task by traversing edges in the DAG. This yields a fully compositional algorithm which only requires solving control problems with reach-avoid constraints, while performing high-level planning.
To solve the low-level reach-avoid tasks, we leverage and build upon the recent learning algorithm for stochastic control under reach-avoid constraints [68]. This work achieves formal guarantees on the probability of reach-avoidance by learning a policy together with a reach-avoid supermartingale (RASM), which serves as a formal certificate of probabilistic reach-avoidance. When composing such guarantees for a global probabilistic specification we encounter two fundamental challenges, (i) reducing the certification of unnecessary individual reach-avoid constraints that are not critical for global satisfaction, and (ii) reducing the violation probabilities for individual reach-avoid constraints to improve the possibility of satisfaction of the composed policy. We overcome the first challenge by leveraging the DAG decomposition of complex specifications and by performing a forward pass on the DAG which learns edge policies on-demand, and the second one by showing that RASMs prove a strictly tighter lower bound on the probability of reach-avoidance than the bound that was originally proved in [68]. Our novel bound multiplies the bound of [68] by an exponential asymptotic decay term. This is particularly important when composing a large number of edge policies for complex
objectives. Furthermore, our new bound is of independent interest and it advances state of the art in stochastic control under reach-avoid constraints.
Contributions Our contributions can be summarized as follows:
1. We prove that RASMs imply a strictly tighter lower bound on the probability of reach-avoidance compared to the bound originally proved in [68] (Section 4).
2. We propose a novel method for learning and verifying a composition of neural network policies in stochastic control systems learned in the RL paradigm, which provides formal guarantees on the probability of satisfaction of SPECTRL specifications (Section 5).
3. We implement a prototype of our approach and evaluate it on the Stochastic Nine Rooms environment, obtained by injecting stochastic disturbances to the environment of [33]. Our experiments demonstrate both the necessity of decomposing high-level logical specifications into simpler control tasks as well as the ability of our algorithm to solve complex specification tasks with formal guarantees (Section 6).
Learning from logical specifications Using high-level specifications with RL algorithms has been explored in [13, 32, 33, 39, 43, 60] among others. These approaches typically use the specification to guide the reward objective used by a RL algorithm to learn a policy but usually lack a formal guarantee on the satisfaction of the specification. Approaches that provide guarantees on the probability of satisfaction in finite state MDPs have been proposed in [12, 28, 29]. In contrast, we consider stochastic systems with continuous state spaces.
Control with reachability and safety guarantees Several works propose model-based methods for learning control policies that provide formal guarantees with respect to reachability, safety and reach-avoid specifications. The methods of [2, 16, 17, 48, 55, 1] consider deterministic systems and learn a policy together with a Lyapunov or barrier function that guarantees reachability or safety. Control of finite-time horizon stochastic systems with reach-avoid guarantees has been considered in [14, 36, 53, 61]. These methods compute a finite-state MDP abstraction of the system and solve the constrained control problem for the MDP. The works of [9, 10] consider infinite-time horizon stochastic systems with linear dynamics and propose abstraction-based methods that provide probably approximately correct (PAC) guarantees. Control of polynomial stochastic systems via stochastic barrier functions and convex optimization was considered in [46, 49, 65, 42]. Learning-based methods providing formal guarantees for infinite time horizon stochastic systems that are not necessarily polynomial have been proposed in [38, 68, 41, 20]. The work [8] proposes a learning-based method for providing formal guarantees on probability 1 stabilization. To the best of our knowledge, the compositionality of our CLAPS makes it the first method that provides formal guarantees for infinite time horizon stochastic systems with respect to more expressive specifications.
Safe exploration RL algorithms fundamentally depend on exploration in order to learn high performing actions. A common approach to safe exploration RL is to limit exploration within a high probability safety region which is computed by estimating uncertainty bounds. Existing works achieve this by using Gaussian Processes [11, 34, 59], linearized models [23], robust regression [40], Bayesian neural networks [37], shielding [4, 7, 25, 31] and state augmentation [51].
Model-free methods for stochastic systems The recent work of [63] proposed a model-free approach to learning a policy together with a stochastic barrier function, towards enhancing probabilistic safety of the learned policy. Unlike our approach, this method does not formally verify the correctness of the stochastic barrier function hence does not provide any guarantees on safety probability. However, it does not assume the knowledge of the system environment and is applicable in a model-free fashion. Several model-free methods for learning policies and certificate functions in deterministic systems have also been considered, see [24] for a survey.
Statistical methods Statistical methods [67] provide an effective approach to estimating the probability of a specification being satisfied, when satisfaction and violation of the specification can be witnessed via finite execution traces. Consequently, such methods often assume a finite-horizon or the existence of a finite threshold such that the infinite-horizon behavior is captured by traces within that threshold. In environments where these assumptions do not hold, the violation of a reach-avoid (or, more generally, any SPECTRAL) specification can be witnessed only via infinite traces and
hence statistical methods are usually not applicable. In this work, we consider formal verification of infinite-time horizon systems.
Supermartingales for probabilistic programs Supermartingales have also been used for the analysis of probabilistic programs (PPs) for properties such as termination [15], reachability [58] and safety [18, 19].
We consider a discrete-time stochastic dynamical system whose dynamics are defined by the equation
$$
\mathbf {x} _ {t + 1} = f \left(\mathbf {x} _ {t}, \pi \left(\mathbf {x} _ {t}\right), \omega_ {t}\right),
$$
where  $t \in \mathbb{N}_0$  is a time step,  $\mathbf{x}_t \in \mathcal{X}$  is a state of the system,  $\mathbf{u}_t = \pi(\mathbf{x}_t) \in \mathcal{U}$  is a control action and  $\omega_t \in \mathcal{W}$  is a stochastic disturbance vector at the time step  $t$ . Here,  $\mathcal{X} \subseteq \mathbb{R}^n$  is the state space of the system,  $\mathcal{U} \subseteq \mathbb{R}^m$  is the action space and  $\mathcal{W} \subseteq \mathbb{R}^p$  is the stochastic disturbance space. The system dynamics are defined by the dynamics function  $f: \mathcal{X} \times \mathcal{U} \times \mathcal{W} \to \mathcal{X}$ , the control policy  $\pi: \mathcal{X} \to \mathcal{U}$  and a probability distribution  $d$  over  $\mathcal{W}$  from which a stochastic disturbance vector is sampled independently at each time step. Together, these define a stochastic feedback loop system.
A sequence  $(\mathbf{x}_t, \mathbf{u}_t, \omega_t)_{t=0}^{\infty}$  of state-action-disturbance triples is a trajectory of the system, if we have that  $\mathbf{u}_t = \pi(\mathbf{x}_t)$ ,  $\omega_t \in \mathrm{support}(d)$  and  $\mathbf{x}_{t+1} = f(\mathbf{x}_t, \pi(\mathbf{x}_t), \omega_t)$  hold for each time step  $t \in \mathbb{N}_0$ . For every state  $\mathbf{x}_0 \in \mathcal{X}$ , we use  $\Omega_{\mathbf{x}_0}$  to denote the set of all trajectories that start in the initial state  $\mathbf{x}_0$ . The Markov decision process (MDP) semantics of the system define a probability space over the set of all trajectories in  $\Omega_{\mathbf{x}_0}$  under any fixed policy  $\pi$  of the system [47]. We use  $\mathbb{P}_{\mathbf{x}_0}$  and  $\mathbb{E}_{\mathbf{x}_0}$  to denote the probability measure and the expectation operator in this probability space.
Assumptions For system semantics to be mathematically well-defined, we assume that  $\mathcal{X} \subseteq \mathbb{R}^n$ ,  $\mathcal{U} \subseteq \mathbb{R}^m$  and  $\mathcal{W} \subseteq \mathbb{R}^p$  are all Borel-measurable and that  $f: \mathcal{X} \times \mathcal{U} \times \mathcal{W} \to \mathcal{X}$  and  $\pi: \mathcal{X} \to \mathcal{U}$  are continuous functions. Furthermore, we assume that  $\mathcal{X} \subseteq \mathbb{R}^n$  is compact (i.e. closed and bounded) and that  $f$  and  $\pi$  are continuous functions. These are very general and standard assumptions in control theory. Since any continuous function on a compact domain is Lipschitz continuous, our assumptions also imply that  $f$  and  $\pi$  are Lipschitz continuous.
Probabilistic specifications Let  $\Omega$  denote the set of all trajectories of the system. A specification is a boolean function  $\phi : \Omega \to \{\text{true}, \text{false}\}$  which for each trajectory specifies whether it satisfies the specification. We write  $\rho \models \phi$  whenever a trajectory  $\rho$  satisfies the specification  $\phi$ . A probabilistic specification is then defined as an ordered pair  $(\phi, p)$  of a specification  $\phi$  and a probability parameter  $p \in [0,1]$  with which the specification needs to be satisfied. We say that the system satisfies the probabilistic specification  $(\phi, p)$  at the initial state  $\mathbf{x}_0 \in \mathcal{X}$  if the probability of any trajectory in  $\Omega_{\mathbf{x}_0}$  satisfying  $\phi$  is at least  $p$ , i.e. if  $\mathbb{P}_{\mathbf{x}_0}[\rho \in \Omega_{\mathbf{x}_0} \mid \rho \models \phi] \geq p$ .
SPECTRAL and abstract graphs Reach-avoid specifications are one of the most common and practically relevant specifications appearing in safety-critical applications that generalize both reachability and safety specifications [54]. Given a target region and an unsafe region, a reach-avoid specifications requires that a system controlled by a policy reaches the target region while avoiding the unsafe region. The SPECTRAL specification language [32] allows specifications of reachability and avoidance objectives as well as their sequential or disjunctive composition. See Appendix A for the formal definition of the SPECTRAL syntax and semantics. $^2$
It was shown in [33] that a SPECTRL specification can be translated into an abstract graph over reach-avoid specifications. Intuitively, an abstract graph is a directed acyclic graph (DAG) whose vertices represent regions of system states and whose edges are annotated with a safety specification. Hence, each edge can be associated with a reach-avoid specification where the goal is to drive the system from the region corresponding to the source vertex of the edge to the region corresponding to the target vertex of the edge while satisfying the annotated safety specification.
Definition 1 (Abstract graph). An abstract graph  $G = (V, E, \beta, s, t)$  is a DAG, where  $V$  is the vertex set,  $E$  is the edge set,  $\beta : V \cup E \to \mathcal{B}(\mathcal{X})$  maps each vertex and each edge to a subgoal region in  $\mathcal{X}$ ,  $s \in V$  is the source vertex and  $t \in V$  is the target vertex. Furthermore, we require that  $\beta(s) = \mathcal{X}_0$ .
2In Appendix A, we also show that SPECTRL strictly subsumes all specifications belonging to the Finitary fragment of LTL, that has been shown to be the PAC-MDP-learnable fragment of LTL [66]. However, the converse is not true and SPECTRL is strictly more general since avoidance is not finitary.
Given a trajectory  $\rho = (\mathbf{x}_t, \mathbf{u}_t, \omega_t)_{t=0}^{\infty}$  of the system and an abstract graph  $G = (V, E, \beta, s, t)$ , we say that  $\rho$  satisfies abstract reachability for  $G$  (written  $\rho \models G$ ) if it gives rise to a path in  $G$  that traverses  $G$  from  $s$  to  $t$  and satisfies reach-avoid specifications of all traversed edges. We formalize this notion in Appendix B. It was shown in [33, Theorem 3.4] that for each SPECTRL specification  $\phi$  once can construct an abstract graph  $G$  such that for each trajectory  $\rho$  we have  $\rho \models \phi$  iff  $\rho \models G$ . For completeness, we provide this construction in Appendix B.
Problem Statement We now formally define the problem that we consider in this work. Consider a stochastic feedback loop system defined as above and let  $\mathcal{X}_0 \subseteq \mathcal{X}$  be the set of initial states. Let  $(\phi, p)$  be a probabilistic specification with  $\phi$  being a specification formula in SPECTRL and  $p \in [0,1]$  being a probability threshold. Then, our goal is to learn a policy  $\pi$  such that the stochastic feedback loop system controlled by the policy  $\pi$  guarantees satisfaction of the probabilistic specification  $(\phi, p)$  at every initial state  $\mathbf{x}_0 \in \mathcal{X}_0$ , i.e., that for every  $\mathbf{x}_0 \in \mathcal{X}_0$  we have  $\mathbb{P}_{\mathbf{x}_0}[\rho \in \Omega_{\mathbf{x}_0} \mid \rho \models \phi] \geq p$ .
Compositional learning algorithm Our goal is not only to learn policy that guarantees satisfaction of the probabilistic specification, but also to learn such a policy in a compositional manner. Given a SPECTRL specification  $\phi$ , a probability threshold  $p \in [0,1]$  and an abstract graph  $G$  of  $\phi$ , we say that a policy  $\pi$  is a compositional policy for the probabilistic specification  $(\phi,p)$  if it guarantees satisfaction of  $(\phi,p)$  and is obtained by composing a number of edge policies learned for reach-avoid tasks associated to edges of  $G$ . An algorithm is said to be compositional for a probabilistic specification  $(\phi,p)$  if it learns a compositional policy for the probabilistic specification  $(\phi,p)$ . In this work, we present a compositional algorithm for the given probabilistic specification  $(\phi,p)$ .
4 Improved Bound for Probabilistic Reach-avoidance
In this section, we define reach-avoid supermartingales (RASMs) and derive an improved lower bound on the probability of reach-avoidance that RASMs can be used to formally certify. RASMs, together with a lower bound on the probability of reach-avoidance that they guarantee and a method for learning a control policy and an RASM, were originally proposed in [68]. The novelty in this section is that we prove that RASMs imply a strictly stronger bound on the probability of satisfying reach-avoidance, compared to the bound derived in [68]. We achieve this by proposing an alternative formulation of RASMs, which we call multiplicative RASMs. In contrast to the original definition of RASMs which imposes additive expected decrease condition, our formulation of RASMs imposes multiplicative expected decrease condition. In what follows, we define multiplicative RASMs. Then, we first show in Theorem 1 that they are equivalent to RASMs. Second, we show in Theorem 2 that by analyzing the multiplicative expected decrease, we can derive an exponentially tighter bound on the probability of reach-avoidance. These two results imply that we can utilize the procedure of [68] to learn policies together with multiplicative RASMs that provide strictly better formal guarantees on the probability reach-avoidance.
Prior work - reach-avoid supermartingales RASMs [68] are continuous functions that assign real values to system states, are required to be nonnegative and to satisfy the additive expected decrease condition, i.e. to strictly decrease in expected value by some additive term  $\epsilon > 0$  upon every one-step evolution of the system dynamics until either the target set or the unsafe set is reached. Furthermore, the initial value of the RASM is at most 1 while the value of the RASM needs to exceed some  $\lambda > 1$  in order for the system to reach the unsafe set. Thus, an RASM can intuitively be viewed as an invariant of the system which shows that the system has a tendency to converge either to the target or the unsafe set while also being repulsed away from the unsafe set, where this tendency is formalized by the expected decrease condition. To emphasize the additive expected decrease, we refer to RASMs of [68] as additive RASMs.
Definition 2 (Additive reach-avoid supermartingales [68]). Let  $\epsilon >0$  and  $\lambda >1$ . A continuous function  $V:\mathcal{X}\rightarrow \mathbb{R}$  is an  $(\epsilon ,\lambda)$ -additive reach-avoid supermartingale  $((\epsilon ,\lambda)$ -additive RASM) with respect to  $\mathcal{X}_t$  and  $\mathcal{X}_u$ , if:
1. Nonnegativity condition.  $V(\mathbf{x}) \geq 0$  for each  $\mathbf{x} \in \mathcal{X}$ .
2. Initial condition.  $V(\mathbf{x})\leq 1$  for each  $\mathbf{x}\in \mathcal{X}_0$
3. Safety condition.  $V(\mathbf{x})\geq \lambda$  for each  $\mathbf{x}\in \mathcal{X}_u$
4. Additive expected decrease condition. For each  $\mathbf{x} \in \mathcal{X} \backslash \mathcal{X}_t$  at which  $V(\mathbf{x}) \leq \lambda$ , we have  $V(\mathbf{x}) \geq \mathbb{E}_{\omega \sim d}[V(f(\mathbf{x}, \pi(\mathbf{x}), \omega))] + \epsilon$ .
Table 1: Comparison of the bound in Theorem 2 and the bound in [68] for several values of  $\lambda$  when  $\gamma = 0.99$ . We set  $\Delta = 0.1$  which bounds the step size in the 9-Rooms environment, and  $L_{V} = 5$  which is an upper bound on the Lipschitz constant that we observe in experiments. Thus, the bound in Theorem 2 is  $1 - \frac{1}{\lambda} \cdot \gamma^{N} = 1 - \frac{1}{\lambda} \cdot 0.99^{2\lambda}$  and in [68] is  $1 - \frac{1}{\lambda}$ .
λ	10	100	1000
CLAPS bound (ours)	0.91820	0.99866	0.99999
[68]	0.9	0.99	0.999