Title: Verified Safe Reinforcement Learning  for Neural Network Dynamic Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: neural network

formal verification

safe reinforcement learning
Abstract: Learning reliably safe autonomous control is one of the core problems in trustworthy autonomy. However, training a controller that can be formally verified to be safe remains a major challenge. We introduce a novel approach for learning verified safe control policies in nonlinear neural dynamical systems while maximizing overall performance. Our approach aims to achieve safety in the sense of finite-horizon reachability proofs, and is comprised of three key parts. The first is a novel curriculum learning scheme that iteratively increases the verified safe horizon. The second leverages the iterative nature of gradient-based learning to leverage incremental verification, reusing information from prior verification runs. Finally, we learn multiple verified initial-state-dependent controllers, an idea that is especially valuable for more complex domains where learning a single universal verified safe controller is extremely challenging. Our experiments on five safe control problems demonstrate that our trained controllers can achieve verified safety over horizons that are as much as an order of magnitude longer than state-of-the-art baselines, while maintaining high reward, as well as a perfect safety record over entire episodes. Our code is available at https://github.com/jlwu002/VSRL.

Verified Safe Reinforcement Learning for Neural Network Dynamic Models

Abstract

Learning reliably safe autonomous control is one of the core problems in trustworthy autonomy. However, training a controller that can be formally verified to be safe remains a major challenge. We introduce a novel approach for learning verified safe control policies in nonlinear neural dynamical systems while maximizing overall performance. Our approach aims to achieve safety in the sense of finite-horizon reachability proofs, and is comprised of three key parts. The first is a novel curriculum learning scheme that iteratively increases the verified safe horizon. The second leverages the iterative nature of gradient-based learning to leverage incremental verification, reusing information from prior verification runs. Finally, we learn multiple verified initial-state-dependent controllers, an idea that is especially valuable for more complex domains where learning a single universal verified safe controller is extremely challenging. Our experiments on five safe control problems demonstrate that our trained controllers can achieve verified safety over horizons that are as much as an order of magnitude longer than state-of-the-art baselines, while maintaining high reward, as well as a perfect safety record over entire episodes. Our code is available at https://github.com/jlwu002/VSRL .

1 Introduction

The ability to synthesize safe control policies is one of the core challenges in autonomous systems. This problem has been explored from numerous directions across multiple disciplines, including control theory and AI [Achiam et al., 2017, Dawson et al., 2022]. While considerable progress has been made, particularly when dynamics are linear [Wabersich and Zeilinger, 2018], the ability to synthesize controllers that can be successfully verified to be safe while maintaining high performance in nonlinear dynamical systems remains a major open problem. Indeed, even the subproblem of safety verification in nonlinear systems is viewed in itself as a major challenge and is an active area of research, particularly for neural network controllers [Bastani et al., 2018, Ivanov et al., 2019, Wei and Liu, 2022]. State-of-the-art approaches for safe control synthesis, including most that leverage reinforcement learning [Gu et al., 2022], typically only offer empirical evaluation of safety, and rely on safety proofs that hold either asymptotically (rather than for concrete problems) [Xiong et al., 2024] or under idealized assumptions which do not hold in practice [Berkenkamp et al., 2017].

Two common properties are typically leveraged in safety verification: forward invariance and reachability . The former aims to identify a set of starting subsets of safe states under which one-step

(forward) dynamics remain in this (forward invariant) set. The latter computes the set of states that can possibly be reached after K steps of the dynamics for a given control policy, and checks whether it intersects with the unsafe set. Approaches for synthesizing (including those that do so using learning) safe policies almost exclusively aim to achieve verified safety through forward invariance. However, this has proved extremely challenging to employ beyond the simplest dynamics.

We propose the first (to our knowledge) approach for learning K -step verified safe neural network controllers that also aim to maximize efficiency in systems with neural dynamics. While neural dynamics are clearly not universal, they can capture or effectively approximate a broad range of practical dynamical systems [Nagabandi et al., 2018], and have consequently been the focus of much prior work in safe control and verification [Dai et al., 2021]. For example, consider the scenario of a drone navigating through a series of obstacles to reach a designated goal, requiring K = 50 steps to safely maneuver through the obstacles. We aim to train a controller that can reach the goal as fast as possible, while guaranteeing safety for the initial 50 steps, ensuring 1) the drone does not collide with any obstacles and 2) its angle remains within a predefined safe range.

Our approach combines deep reinforcement learning with state-of-the-art differentiable tools for efficient reachability bound computation, and contains two key novel ingredients. The first is a novel curriculum learning scheme for learning a verified safe controller. This scheme takes advantage of the structure of the K -reachability problem at the root of our safety verification by creating a curriculum sequence with respect to increasing K . An important insight that is specific to the verification setting is that verification must work not merely for a fixed K , but for all steps prior, an issue we address by memorizing subsets of states who either violate, or nearly violate, safety throughout the entire K -step curriculum learning process. Additionally, to maintain both strong empirical and verified performance, we propose a novel loss function that integrates overall reward, as well as both traditional (empirical) safety loss along with the K -reachability bound. Our second innovation is to learn a collection of controllers that depend on the initial state, in contrast to typical approaches that focus on learning a single 'universal' controller. The ability to allow for learning multiple controllers makes the verified learning problem considerably easier, as we can 'save' controllers that work on a subset of initial states, and simply try learning a new controller for the rest, guaranteeing incremental improvement through the learning process. We further improve performance through incremental verification, which leverages information obtained in previous learning iterations.

We evaluate the proposed approach in five control settings. The first two are lane following and obstacle avoidance, both pertaining to autonomous driving. The last three involve drone control with obstacle avoidance. Two of these consider fixed obstacles, while the third aims to avoid even moving obstacles (with known dynamics). We show that the proposed approach outperforms five state-of-the-art safe control baselines in the ability to achieve verified safety without significantly compromising overall reward (efficiency). In particular, our approach learns controllers that can verify K -step safety for K up to an order of magnitude larger than the prior art and maintains a perfect safety record for K far above what we verify, something no baseline can achieve.

In summary, we make the following contributions:

A framework for safe optimal control that combines both finite-horizon verified (worst-case) and empirical (average-case) safety constraints.

A novel curriculum learning approach that leverages memorization, forward reachability analysis, and differentiable reachability overapproximation for efficiently learning verified safe policies.

An approach for learning a collection of control policies that depend on the initial state which enables significant improvements in verified safety horizon over large initial state sets S 0 .

An incremental verification approach that leverages small changes in gradient-based learning to improve verification efficiency during learning.

An extensive experimental evaluation that demonstrates the efficacy of the proposed approach in comparison with five state-of-the-art safe RL baselines.

Related Work: Safe reinforcement learning has been extensively studied through the lens of constrained Markov decision process (CMDP)-based approaches, which represent cost functions as constraints and aim to maximize reward while bounding cost, using approaches such as Lagrangian and penalty methods, and constrained policy optimization [Achiam et al., 2017, Stooke et al., 2020, Ma et al., 2022, Jayant and Bhatnagar, 2022, Yu et al., 2022, So and Fan, 2023, Ganai et al., 2024].

An alternative control-theoretic perspective aims to ensure stability or safety using Lyapunov and control barrier functions. For example, Dawson et al. [2022] used a learning-based approach to find robust control Lyapunov barrier functions; Chow et al. [2018] constructed Lyapunov functions to solve CMDPs; [Wang et al., 2023a] proposed soft barrier functions for unknown and stochastic environments; and Alshiekh et al. [2018] created safety shielding for safe RL agents. These approaches, however, provide no practical formal safety guarantee for neural network controllers. In addition, some work on provably safe RL focuses on the probabilistic setting [Berkenkamp et al., 2017, Jansen et al., 2020, Xiong et al., 2024] and required statistical assumptions, whereas our work aims for strict deterministic safety guarantees over a finite horizon.

Among existing works focusing on safe RL with formal guarantees, Fulton and Platzer [2018] apply a theorem prover for differential dynamic logic to guarantee safety during runtime. Noren et al. [2021] and Wei et al. [2022] consider forward safety invariance for systems with uncertainty. Kochdumper et al. [2023] propose to project actions to safe subspace using zonotope abstraction and mixed-integer programming (MIP). However, these approaches do not readily apply to neural network controllers. For systems involving neural networks, Wei and Liu [2022] applied integer programming formulation for neural networks to solve an MIP problem to find safe control actions satisfying forward invariance; Bastani et al. [2018] extracted decision-tree-based policies for RL to reduce verification complexity; and Ivanov et al. [2019] used hybrid system verification tools to model deep neural networks. Our work differs from these and similar approaches because we consider forward reachability guarantees for neural network controllers in neural nonlinear systems.

We make extensive use of neural network verification tools. Early work in this vein used SMT [Katz et al., 2017, Huang et al., 2017] or MIP-based [Tjeng et al., 2019] approaches to solve this problem, but their scalability is extremely limited. Significant progress has been made in developing techniques to formally verify the properties of large neural networks through overapproximation, such as bound propagation [Zhang et al., 2018, Gowal et al., 2018, Xu et al., 2021], optimization [Qin et al., 2019, Dvijotham et al., 2018, 2020], and abstract interpretation [Gehr et al., 2018, Singh et al., 2019, Katz et al., 2019, Lopez et al., 2023]. Recently, most verifiers have adopted branch-and-bound based approaches to further enhance their performance [Wang et al., 2021, Kouvaros and Lomuscio, 2021, Ferrari et al., 2022, Zhang et al., 2022]. Our approach makes use of differentiable overapproximation methods known collectively as α,β -CROWN [Wang et al., 2021, Zhang et al., 2022] (implemented with the auto_LiRPA package), and takes advantage of the particular structure of these verification approaches in applying incremental verification to significantly speed up safe controller learning.

2 Preliminaries

Constrained Markov Decision Process (CMDP): We consider a deterministic Constrained Markov Decision Process (CMDP) defined by the tuple ( S , A , F, R, γ, C 1 , C 2 , . . . , C m , d 1 , d 2 , . . . , d m ) , where: S is a set of states, A is a set of actions, F : S × A → S is the deterministic state transition function, R : S × A → R is the reward function, C i : S × A → R is the cost function for the i -th constraint, d i is the cost limit for the i -th constraint, and γ ∈ [0 , 1) is the discount factor. A policy π : S → A is a mapping from states to actions. A trajectory is a sequence of states and actions generated by following a policy π from some initial state s 0 ∈ S 0 ⊆ S , which can be represented as a sequence τ = ( s 0 , a 0 , s 1 , a 1 , s 2 , a 2 , . . . ) where s t ∈ S , a t = π ( s t ) for all t , s t +1 = F ( s t , a t ) , a reward r t = R ( s t , a t ) and a cost c t = ∑ i ∈ [ m ] C i ( s t , a t ) are received after each action.

We denote π θ as the policy that is parameterized by the parameter θ . A common goal for CMDP is to learn a policy π θ that maximizes a discounted sum of rewards J ( π θ ) while ensuring that expected discounted costs J C i ( π θ ) do not exceed the cost limit d i , ∀ i ∈ [ m ] . Formally, CMDP is to solve the below optimization problem:

Verified Safe CMDP: We define the state space as the union of predefined safe and unsafe states, denoted as S = S safe ∪ S unsafe . We assume that the transition function F is represented by a ReLU neural network, and is known for verification purposes. This assumption is very general, as many known dynamical systems can be represented exactly or approximately using ReLU neural networks [Gillespie et al., 2018, Pfrommer et al., 2021, Dai et al., 2021, Liu et al., 2024]. Our

objective is to train a controller that not only satisfies safety constraints empirically at decision time, but also ensures verified safety for the first K steps.

Formally, we aim to solve the following optimization problem:

In particular, we aim to solve (2) for high values of K and large sets of verified safe initial states S 0 , while preserving a high objective value. Note that for a given controller, (2c) can also be interpreted as a set of forward reachability verification problems. However, our interest here extends beyond mere verification; we aim to train (synthesize) a controller that can be efficiently verified for safety . For simplicity, we restrict attention to d i = 0 for all i ; however, our approach can be directly applied to arbitrary values of d i .

In this work, we primarily utilize the α,β -CROWN toolbox [Xu et al., 2021, Wang et al., 2021] for neural network (NN) verification; however, our training framework is general and, in principle, can work with any differentiable verification technique. Let F k,π θ denote the k -step forward function (iterative composition of F ) under policy π θ . For example, s 1 = F 1 ,π θ ( s ) = F ( s, π θ ( s )) , F 2 ,π θ ( s ) = F ( s 1 , π θ ( s 1 )) , and so on. Correspondingly, we represent the k -step forward reachable regions returned by the NN verifier for an initial state set S as F k,π θ Bound ( S ) , which is typically represented as a box.

3 Approach

The problem of learning verified safe control over a target horizon K entails three key technical challenges. The first is that as K grows, the differentiable overapproximation techniques for reachability verification become looser, making it difficult to verify K beyond very small horizons. Second, while control policies π depend on state, it is difficult to find a single universal controller that can achieve verified safety for each starting state in S 0 . Our approach addresses these challenges through three technical advances: 1) curriculum learning with memorization and 2) incremental verification, which enable learning verified safe controllers over longer horizons K , and 3) iterative learning a collection of controllers customized for subsets of S 0 , which addresses the third challenge above.

Curriculum Learning with Memorization: Curriculum learning is an iterative training strategy where the difficulty of the task increases as training progresses [Bengio et al., 2009, Wu and V orobeychik, 2022]. At a high level, for a problem targeting K -step verified safety, training can be divided into K phases, with each phase k aiming to achieve verified safety at the corresponding k -th forward step. In the k -th phase, we conduct formal verification against the k -th step safety, filter out regions that cannot be verified, and use them for further training. As k increases, the task difficulty also increases, mainly due to the forward NN F k,π θ becoming deeper. For a deeper NN and a fixed branching budget, the output bounds become looser [Wang et al., 2021], increasing the likelihood of intersections with unsafe regions. However, the ability to verify safety in prior steps enables us to tailor a controller that closely aligns with the fixed NN dynamics, thereby achieving tighter bounds. This process captures the essence of curriculum learning.

Nevertheless, our approach deviates from traditional curriculum learning in a way that is quite consequential for our setting: we aim to ensure that a controller is verified as safe not only for the k -th step but also maintains safety for all prior steps. Consequently, during our curriculum learning process, we store states that are close to being unsafe in each phase in a buffer, effectively memorizing information about regions that potentially violate safety. These states, along with the unverified states at the current phase, are then incorporated into the training process, helping to ensure safety across the entire K -step horizon. Our curriculum training framework is detailed in Algorithm 1.

The process begins by initializing the policy π θ with a pre-trained policy using safe RL algorithms (Line 3). We then split the initial region S 0 into a grid G 0 (Line 4). We assume S 0 is a m -dimensional box centered at s c ∈ R m with a radius r ∈ R m , i.e., S 0 = [ s c -r, s c + r ] . We prioritize splitting the dimensions that are directly implicated in safety constraints, thereby taking advantage of the

̸

typical structure of safety constraints that only pertain to a small subset of state variables. For instance, in drone control for obstacle avoidance, we prioritize splitting the location and angle axes. Next, we design a cost function C R for regions where C R ( S ) = 0 if S ∩ S unsafe = ∅ , and C R ( S ) > 0 otherwise . A positive C R means region S intersects with S unsafe, while C R = 0 indicates S is safe. For example, if the task is to avoid the region [ a, b ] , and the output bounds are given by x B = [ x lb , x ub ] , we can define C R ( x B ) = max( x ub -a, 0) · max( b -x lb , 0) . We then calculate the gradient ∂C R ( F t,π θ Bound ( S 0 )) /∂r for a chosen value of t and proceed to split along the dimensions with the largest gradient values, as a larger gradient indicates a higher likelihood of reducing the cost C R . We continue this process, keeping the total number of grid splits within a predetermined budget, and stop splitting once the budget is reached.

For each training phase k , we monitor the training rounds ( n train) as well as the k -step forward reachable regions returned by the verifier that are identified as unsafe ( S uc ). Each phase is conducted for a maximum of n max rounds or until verified k -step safety is achieved, that is, when S uc = ∅ (Line 8). At the end of each training phase, we also filter out regions S k ⊆ G 0 where F k,π θ Bound ( S k ) are within ϵ distance to the unsafe regions. These regions are then stored in the buffer B (Line 13). We include these critical regions in the training set for each reinforcement learning update to enhance verified safety across the entire horizon. During this process, we optionally use the Branch-and-Bound algorithm [Everett et al., 2020, Wang et al., 2021] to refine G 0 up to a predetermined branching limit, which helps achieve tighter bounds.

For each RL update, we use a loss function that integrates the standard safe RL loss with a k -phase loss for bounds (Line 9), where

Here, L SafeRL is the standard safety RL loss, and L Bound denotes the loss that incentivizes ensuring the output bounds returned by the verifier remain within the safe region. If both S uc is k -step safe and ∀ ( S i , i ) ∈ B , S i is i -step safe, then L Bound ( S uc ∪ B ) = 0 , otherwise, L Bound ( S uc ∪ B ) > 0 . In practice, we clip L Bound ( S uc ∪ B ) to ensure it remains within a reasonable range for training stability. The regularization parameter λ is calculated based on the magnitude of L SafeRL and L Bound, with λ = min( λ max , a r · L SafeRL / L Bound ) , where λ max and a r are hyperparamters. This approach helps maintain the effectiveness of bound training, especially when L Bound is small. Furthermore, we cluster elements in B into categories so that we do not need to construct a computational graph for all i < k . Specifically, we merge all S i for i 1 ≤ i ≤ i 2 into the i 2 category, meaning the elements in B are now ( ∪ i 1 ≤ i ≤ i 2 S i , i 1 , i 2 ) instead of ( S i , i ) .

It is important to note that while our training scheme targets K -step verified safety, the policy returned by Algorithm 1 does not necessarily guarantee it. We address this issue by learning initial-statedependent controllers as described below. Furthermore, the computation of L Bound is computationally intensive. Its backpropagation requires constructing computational graphs for the k -th step forward

NN F k,π θ Bound , as well as for all i -th step forward NNs corresponding to each ( S i , i ) ∈ B . These NNs become increasingly deep as k grows, causing the computational graphs to consume memory beyond the typical GPU memory limits. We will address this next.

Incremental Verification: Above we discussed the challenge presented by the backpropagation of L bound, which is GPU-memory intensive and does not scale efficiently as the target K -step horizon increases. To mitigate these issues, we propose the use of incremental verification to enhance computational efficiency and reduce memory consumption. While incremental verification is wellexplored in the verification literature Wang et al. [2023b], Althoff [2015], to our knowledge, we are the first to apply it in training provably safe controllers.

At a high level, to calculate the reachable region for a k target step, we decompose the verification into multiple phases. We begin by splitting the k target horizon into intervals defined by 0 < k 1 < k 2 < · · · < k n = k target. We first calculate the reachability region for the k i step and then use its output bounds as input to calculate the reachable region for the k i +1 step. This approach ensures that the computational graph is only built for the ( k i +1 -k i ) step horizon when using α,β -CROWN.

Unlike traditional incremental verification, which typically calculates the reachable region from k to k +1 , we incrementally verify and backpropagate several steps ahead in a single training iteration (i.e., from k i to k i +1 , where k i +1 -k i > 1 ). This generalized version of incremental verification is essential for training, as it significantly accelerates the process and reduces the likelihood of becoming trapped in "local optima," where inertia from the policy obtained for k prevents successful verification for k +1 (e.g., due to proximity to the unsafe region with velocity directed toward it).

For the bounds used in neural network training, we effectively build the computational graph and perform backpropagation using a neural network sized for ( k n -k n -1 ) steps' reachability, which is independent of k target. This significantly reduces GPU memory usage. Since F k,π θ is an iterative composition of F under the same policy π θ , the bound for k n -1 steps tends to be tight. Moreover, when training π θ to tighten these bounds, the overall bound for the entire k target horizon becomes increasingly tight.

Initial-State-Dependent Controller: While curriculum learning above includes verification steps, it does not guarantee verified safety for the controller over the entire K -step horizon. In this section, we propose using an initial-state-dependent controller to address this issue. For example, in a vehicle avoidance scenario, different initial conditions, such as varying speeds and positions, may correspond to different control strategies. We introduce a mapping function h : S 0 → Θ , which maps each initial state s 0 ∈ S to a specific policy π h ( s 0 ) . The underlying idea is that training a verifiable safe policy π θ over the entire set of initial states S 0 is inherently challenging. However, by mapping each initial state to a specific set of parameters, we can significantly enhance the expressivity of the policy. This approach is particularly effective in addressing and eliminating corner cases in unverifiable regions.

At a high level, the mapping and parameter set Θ are obtained by first performing comprehensive verification for the controller output from Algorithm 1 over the entire K -step horizon. We then filter unverified regions, cluster them, and fine-tune the controller parameters θ for each cluster. We store these fine-tuned parameters in the parameter set Θ . This iterative refinement process continues until for every s 0 ∈ S 0 , there exists a θ ∈ Θ such that π θ is verified safe for the entire K -step horizon. The detailed algorithm is presented in Algorithm 2.

The algorithm starts with verifying the policy π θ obtained from Algorithm 1. The function VERIFYSAFE ( π θ , S 0 , K ) (Line 4) performs verification of policy π θ for initial states S 0 for the entire horizon K . This verification process identifies and categorizes regions into verified safe areas, S V safe , and areas identified as unsafe, S V unsafe . Notably, the union of these regions covers all initial states, meaning S V safe ∪ S V unsafe = S 0 . After verifying that any state s 0 ∈ S V safe is guaranteed to be safe under policy π θ , we record ( S V unsafe , π θ ) in the mapping dictionary H (Line 5).

Next, we address the unsafe regions S V unsafe that lack a corresponding verified safe controller. We first cluster them based on the type of safety violation (Line 7). The reason for clustering is that regions with similar safety violations are more likely to be effectively verified safe by the same controller. For instance, in a scenario involving navigation around two obstacles, we could potentially identify up to three clusters: the first corresponding to grids that can lead to collisions with obstacle 1, the second includes grids associated with collisions with obstacle 2, and the third is the set of grids that may lead to collisions with both. Given the finite number of safety constraints, the number of possible clusters is also finite. Although the theoretical maximum number of clusters grows exponentially

̸

with the number of safety constraints, in practice, this number is significantly smaller. This is due to the fact that the controller, being pretrained, is less likely to violate multiple or all constraints simultaneously. We then fine-tune the controller for the initial states in each cluster using Algorithm 1. This fine-tuning process is typically fast, as the initial policy is already well-trained. We store each initial state region and its corresponding verified safe policy in the mapping dictionary H . This clustering and fine-tuning process continues until a verified safe policy exists for every s 0 ∈ S 0 .

At decision time, given an initial state s 0 , we first identify the pair ( S V safe , π θ ) in the mapping dictionary H where s 0 ∈ S V safe , then use the corresponding verified safe controller π θ . Note that the soundness of the algorithm directly follows from our use of the sound verification tool α,β -CROWN.

4 Experiments

4.1 Experiment Setup

We evaluate our proposed approach in five control settings: Lane Following, Vehicle Avoidance, 2D Quadrotor (with both fixed and moving obstacles), and 3D Quadrotor [Kong et al., 2015, Dai et al., 2021]. The dynamics of these environments are approximated using NN with ReLU activations. We use a continuous action space for those discrete-time systems. In each experiment, we specify the initial region S 0 for which we wish to achieve verified safety. We then aim to achieve the maximum K for which safety can be verified. We evaluate the approaches using four metrics: 1) VerifiedK : the percentage of regions in S 0 that can be verified for safety over K steps; 2) Verified-Max : the maximum number of steps for which all states in S 0 can be verified as safe; 3) Empk : the percentage of regions in S 0 that are empirically safe for k steps, obtained by sampling 10 7 datapoints from the initial state S 0 . This is evaluated for both k = K (the number of steps we are able to verify safety for) and k = T (total episode length); 4) Avg Reward : the average reward over 10 episodes, with both mean and standard deviations reported. Note that the average reward is computed over the entire episode horizon for each environment, independently of the verification horizon, as in conventional reinforcement learning.

We compare the proposed verified safe RL (VSRL) approach to six baselines: 1) PPO-Lag, which utilizes constrained PPO with the standard Lagrangian penalty [Achiam et al., 2017]; 2) PPO-PID, which employs constrained PPO with PID Lagrangian methods [Stooke et al., 2020]; 3) CAP, which adopts model-based safe RL with an adaptive penalty [Ma et al., 2022]; 4) MBPPO, which applies model-based safe RL with constrained PPO [Jayant and Bhatnagar, 2022]; 5) CBF-RL, which is a Control Barrier Function (CBF)-based safe reinforcement learning approach [Emam et al., 2022]; and 6) RESPO, which implements safe RL using iterative reachability estimation [Ganai et al., 2024].

Next, we describe the four autonomous system environments in which we run our experiments. Further experimental setup details are provided in Appendix A.2.

Lane Following: Our lane following environment follows the discrete-time bicycle model [Kong et al., 2015]. The model inputs are 3-dimensional ( x, θ, v ) , where x is the lateral distance to the center of the lane, θ is the angle relative to the center of the lane, and v represents the speed. The objective is to maintain a constant speed while following the lane, meaning the system equilibrium point is ( x, θ, v ) = (0 , 0 , v target ) . The safety constraints are 1) x stays within a maximum distance from the lane center ( ∥ x ∥ ≤ d max ) , 2) θ remains within a predefined range ( ∥ θ ∥ ≤ θ max ) , and 3) v does not exceed the maximum threshold ( v ≤ v max ) .

Vehicle Avoidance: Our vehicle avoidance environment features a vehicle moving on an x -y plane, with 4-dimensional inputs ( x, y, θ, v ) . Here, ( x, y ) represents the location of the vehicle on the plane, θ is the angle relative to the y -axis, and v is the speed. In this setting, we have five moving obstacles, each moving from one point to another at constant speed. Each obstacle is represented as a square. Additionally, safety constraints are set for the speed ( v ≤ v max ) and angle ( ∥ θ ∥ ≤ θ max ) . The task is to navigate the vehicle to a designated location while following safety constraints.

2D Quadrotor: For the 2D quadrotor environment, we follow the settings in Dai et al. [2021]. The input is 6-dimensional ( y, z, θ, ˙ y, ˙ z, ˙ θ ) , where ( y, z ) represents the position of the quadrotor on the y -z plane, and θ represents the angle. The action space is 2-dimensional and continuous; the actions are clipped within a range to reflect motor constraints. Our safety criteria include an angle constraint ( ∥ θ ∥ ≤ θ max ) and a minimum height constraint to prevent collision with the ground ( y ≥ y min ) . We consider two scenarios for obstacles: fixed and moving. For fixed obstacles, there are five rectangular obstacles positioned in the y -z plane. For moving obstacles, there are five obstacles that moves from one point to another at constant speed, each represented as a square.

3D Quadrotor: Our 3D quadrotor environment features a 12-dimensional input space, represented as ( x, y, z, ϕ, θ, ψ, ˙ x, ˙ y, ˙ z, ω x , ω y , ω z ) . The action space is 4-dimensional and continuous; the actions are clipped within a range to reflect motor constraints. Here, ( x, y, z ) denotes the location of the quadrotor in space, ϕ is the roll angle, θ is the pitch angle, and ψ is the yaw angle, ω x , ω y , ω z represent the angular velocity around the x , y , and z axes, respectively. The task is to navigating towards the goal while adhering to safety constraints, which include avoiding five obstacles represented as 3D rectangles. The details for the environment settings are deferred to the Appendix.

4.2 Results

As shown in Table 1, our approach significantly outperforms all baselines in terms of verified safety, as well as empirical safety over the entire episode horizon. Furthermore, the only environment in which VSRL exhibits a significant decrease in reward compared to baselines is lane following; for the rest, it achieves reward comparable to, or better than the baselines.

Specifically, in the lane following environment, the proposed VSRL approach achieves verified 80 -step safety using a single controller (i.e., | Θ | = 1 ). This is an order of magnitude higher K than all baselines (which only achieve K ≤ 8 ). While all baselines obtain a safety record of over 99.9% over the entire episode ( K = 500 ), our approach empirically achieves perfect safety.

For vehicle avoidance, we achieve verified 50 -step safety using two controllers (i.e., | Θ | = 2 ); in contrast, the best baseline yields only K = 13 . We also observe considerable improvements in both verified and empirical safety over the baseline approaches: for example, the best verified baseline (CAP) violates safety over 10% of the time over the full episode length, whereas VSRL maintains a perfect safety record. In this case, VSRL also achieves the highest reward.

For the 2D Quadrotor environment with fixed and moving obstacles, we are able to achieve verified 50 -step safety using four and two controllers, respectively. The best baseline achieves only K = 14 in the case of fixed and K = 8 in the case of moving obstacles (notably, different baselines are best in these cases).

Finally, in the most complex 3D Quadrotor environment, we achieve verified safety for K = 15 , but empirically maintain a perfect safety record for the entire episode durection. The best baseline achieves verified safety for only K = 4 , but is empirically unsafe over 40% of the time during an episode. Even the best safety record of any baseline is unsafe nearly 12% of the time, and we can only verify its safety over a horizon K = 3 .

Ablation Study: We evaluate the importance of both incremental verification and using multiple initial-state-dependent controllers as part of VSRL. As shown in the Appendix (Section A.1), the

former significantly reduces average verification time during training, whereas the latter enables us to greatly boost the size of the initial state region S 0 for which we are able to achieve verify safety.

5 Conclusion

We present an approach for learning neural network control policies for nonlinear neural dynamical systems. In contrast to conventional methods for safe control synthesis which rely on forward invariance-based proofs, we opt instead for the more pragmatic finite-step reachability verification. This enables us to make use of state-of-the-art differentiable neural network overapproximation tools that we combine with three key innovations. The first is a novel curriculum learning approach for maximizing safety horizon. The second is to learn multiple initial-state-dependent controllers. The third is to leverage small changes in iterative gradient-based learning to enable incremental verification. We show that the proposed approach significantly outperforms state of the art safe RL baselines on several dynamical system environments, accounting for both fixed and moving obstacles. A key limitation of our approach is the clearly weaker safety guarantees it provides compared to forward invariance. Nevertheless, our results demonstrate that finite-step reachability provides a more pragmatic way of achieving verified safety that effectively achieves safety over the entire episode horizon in practice , providing an alternative direction for advances in verified safe RL to the more typical forward-invariance-based synthesis.

Acknowledgments

This research was partially supported by the NSF (grants IIS-1905558, IIS-2214141, CCF-2403758, and IIS-2331967) and NVIDIA. Huan Zhang is supported in part by the AI2050 program at Schmidt Sciences (AI 2050 Early Career Fellowship).

References

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning , pages 22-31. PMLR, 2017.

Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Matthias Althoff. An introduction to CORA 2015. In Proc. of the 1st and 2nd Workshop on Applied Verification for Continuous and Hybrid Systems , pages 120-151. EasyChair, December 2015. doi: 10.29007/zbkv.

Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. Advances in neural information processing systems , 31, 2018.

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning , pages 41-48, 2009.

Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, and Andreas Krause. Safe modelbased reinforcement learning with stability guarantees. In Proceedings of the 31st International Conference on Neural Information Processing Systems , page 908-919, 2017.

Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A lyapunovbased approach to safe reinforcement learning. Advances in neural information processing systems , 31, 2018.

Hongkai Dai, Benoit Landry, Lujie Yang, Marco Pavone, and Russ Tedrake. Lyapunov-stable neuralnetwork control. In Dylan A. Shell, Marc Toussaint, and M. Ani Hsieh, editors, Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021 , 2021.

Charles Dawson, Zengyi Qin, Sicun Gao, and Chuchu Fan. Safe nonlinear control using robust neural lyapunov-barrier functions. In Conference on Robot Learning , pages 1724-1735. PMLR, 2022.

Krishnamurthy Dvijotham, Robert Stanforth, Sven Gowal, Timothy A. Mann, and Pushmeet Kohli. A dual approach to scalable verification of deep networks. In Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018 , pages 550-559, 2018.

Krishnamurthy Dj Dvijotham, Robert Stanforth, Sven Gowal, Chongli Qin, Soham De, and Pushmeet Kohli. Efficient neural network verification with exactness characterization. In Uncertainty in artificial intelligence , pages 497-507. PMLR, 2020.

Yousef Emam, Gennaro Notomista, Paul Glotfelter, Zsolt Kira, and Magnus Egerstedt. Safe reinforcement learning using robust control barrier functions. IEEE Robotics and Automation Letters , 2022.

Michael Everett, Golnaz Habibi, and Jonathan P How. Robustness analysis of neural networks via efficient partitioning with applications in control systems. IEEE Control Systems Letters , 5(6): 2114-2119, 2020.

Claudio Ferrari, Mark Niklas Mueller, Nikola Jovanovi´ c, and Martin Vechev. Complete verification via multi-neuron relaxation guided branch-and-bound. In International Conference on Learning Representations , 2022.

Nathan Fulton and André Platzer. Safe reinforcement learning via formal methods: Toward safe control through proof and learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Milan Ganai, Zheng Gong, Chenning Yu, Sylvia Herbert, and Sicun Gao. Iterative reachability estimation for safe reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin Vechev. Ai2: Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEEE Symposium on Security and Privacy (SP) , pages 3-18, 2018.

Morgan T Gillespie, Charles M Best, Eric C Townsend, David Wingate, and Marc D Killpack. Learning nonlinear dynamic models of soft robots for model predictive control with neural networks. In 2018 IEEE International Conference on Soft Robotics (RoboSoft) , pages 39-45. IEEE, 2018.

Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv:1810.12715 , 2018.

Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330 , 2022.

Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. Safety verification of deep neural networks. In Computer Aided Verification: 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I 30 , pages 3-29. Springer, 2017.

Radoslav Ivanov, James Weimer, Rajeev Alur, George J Pappas, and Insup Lee. Verisig: verifying safety properties of hybrid systems with neural network controllers. In Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control , pages 169-178, 2019.

Nils Jansen, Bettina Könighofer, Sebastian Junges, Alex Serban, and Roderick Bloem. Safe reinforcement learning using probabilistic shields. In 31st International Conference on Concurrency Theory (CONCUR 2020) . Schloss-Dagstuhl-Leibniz Zentrum für Informatik, 2020.

Ashish K Jayant and Shalabh Bhatnagar. Model-based safe deep reinforcement learning via a constrained proximal policy optimization algorithm. Advances in Neural Information Processing Systems , 35:24432-24445, 2022.

Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. In International Conference on Computer Aided Verification , pages 97-117, 2017.

Guy Katz, Derek A Huang, Duligur Ibeling, Kyle Julian, Christopher Lazarus, Rachel Lim, Parth Shah, Shantanu Thakoor, Haoze Wu, Aleksandar Zelji´ c, et al. The marabou framework for verification and analysis of deep neural networks. In Computer Aided Verification: 31st International Conference, CAV 2019, New York City, NY, USA, July 15-18, 2019, Proceedings, Part I 31 , pages 443-452. Springer, 2019.

Niklas Kochdumper, Hanna Krasowski, Xiao Wang, Stanley Bak, and Matthias Althoff. Provably safe reinforcement learning via action projection using reachability analysis and polynomial zonotopes. IEEE Open Journal of Control Systems , 2:79-92, 2023.

Jason Kong, Mark Pfeiffer, Georg Schildbach, and Francesco Borrelli. Kinematic and dynamic vehicle models for autonomous driving control design. In 2015 IEEE intelligent vehicles symposium (IV) , pages 1094-1099. IEEE, 2015.

Panagiotis Kouvaros and Alessio Lomuscio. Towards scalable complete verification of relu neural networks via dependency-based branching. In IJCAI , pages 2643-2650, 2021.

Ziang Liu, Genggeng Zhou, Jeff He, Tobia Marcucci, Fei-Fei Li, Jiajun Wu, and Yunzhu Li. Modelbased control with sparse neural dynamics. Advances in Neural Information Processing Systems , 36, 2024.

Diego Manzanas Lopez, Sung Woo Choi, Hoang-Dung Tran, and Taylor T Johnson. Nnv 2.0: the neural network verification tool. In International Conference on Computer Aided Verification , pages 397-412. Springer, 2023.

Yecheng Jason Ma, Andrew Shen, Osbert Bastani, and Jayaraman Dinesh. Conservative and adaptive penalty for model-based safe reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence , volume 36, pages 5404-5412, 2022.

Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 7559-7566. IEEE, 2018.

Charles Noren, Weiye Zhao, and Changliu Liu. Safe adaptation with multiplicative uncertainties using robust safe set algorithm. IFAC-PapersOnLine , 54(20):360-365, 2021.

Samuel Pfrommer, Mathew Halm, and Michael Posa. Contactnets: Learning discontinuous contact dynamics with smooth, implicit representations. In Conference on Robot Learning , pages 22792291. PMLR, 2021.

Chongli Qin, Krishnamurthy (Dj) Dvijotham, Brendan O'Donoghue, Rudy Bunel, Robert Stanforth, Sven Gowal, Jonathan Uesato, Grzegorz Swirszcz, and Pushmeet Kohli. Verification of non-linear specifications for neural networks. In International Conference on Learning Representations , 2019.

Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. An abstract domain for certifying neural networks. Proceedings of the ACM on Programming Languages , 3(POPL):41, 2019.

Oswin So and Chuchu Fan. Solving stabilize-avoid optimal control via epigraph form and deep reinforcement learning. In Proceedings of Robotics: Science and Systems , 2023.

Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning , pages 9133-9143. PMLR, 2020.

Vincent Tjeng, Kai Y. Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. In International Conference on Learning Representations , 2019.

Kim P Wabersich and Melanie N Zeilinger. Linear model predictive safety certification for learningbased control. In 2018 IEEE Conference on Decision and Control (CDC) , pages 7130-7135. IEEE, 2018.

Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. Beta-CROWN: Efficient bound propagation with per-neuron split constraints for complete and incomplete neural network verification. Advances in Neural Information Processing Systems , 34, 2021.

Yixuan Wang, Simon Sinong Zhan, Ruochen Jiao, Zhilu Wang, Wanxin Jin, Zhuoran Yang, Zhaoran Wang, Chao Huang, and Qi Zhu. Enforcing hard constraints with soft barriers: safe reinforcement learning in unknown stochastic environments. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR, 2023a.

Yixuan Wang, Weichao Zhou, Jiameng Fan, Zhilu Wang, Jiajun Li, Xin Chen, Chao Huang, Wenchao Li, and Qi Zhu. Polar-express: Efficient and precise formal reachability analysis of neural-network controlled systems. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 2023b.

Tianhao Wei and Changliu Liu. Safe control with neural network dynamic models. In Learning for Dynamics and Control Conference , pages 739-750. PMLR, 2022.

Tianhao Wei, Shucheng Kang, Weiye Zhao, and Changliu Liu. Persistently feasible robust safe control by safety index synthesis and convex semi-infinite programming. IEEE Control Systems Letters , 7:1213-1218, 2022.

Junlin Wu and Yevgeniy Vorobeychik. Robust deep reinforcement learning through bootstrapped opportunistic curriculum. In International Conference on Machine Learning , pages 24177-24211. PMLR, 2022.

Nuoya Xiong, Yihan Du, and Longbo Huang. Provably safe reinforcement learning with step-wise violation constraints. Advances in Neural Information Processing Systems , 36, 2024.

Kaidi Xu, Huan Zhang, Shiqi Wang, Yihan Wang, Suman Jana, Xue Lin, and Cho-Jui Hsieh. Fast and Complete: Enabling complete neural network verification with rapid and massively parallel incomplete verifiers. In International Conference on Learning Representations , 2021.

Dongjie Yu, Haitong Ma, Shengbo Li, and Jianyu Chen. Reachability constrained reinforcement learning. In International Conference on Machine Learning , pages 25636-25655. PMLR, 2022.

Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. Efficient neural network robustness certification with general activation functions. In Advances in Neural Information Processing Systems , pages 4944-4953, 2018.

Huan Zhang, Shiqi Wang, Kaidi Xu, Linyi Li, Bo Li, Suman Jana, Cho-Jui Hsieh, and J. Zico Kolter. General cutting planes for bound-propagation-based neural network verification. In Proceedings of the 36th International Conference on Neural Information Processing Systems , 2022.

A Appendix

A.1 Ablation Study

In this section, we conduct an ablation study to evaluate the importance of both incremental verification and the use of multiple initial-state-dependent controllers as part of the VSRL approach.

Table 2 presents the ablation study results for incremental verification. To ensure a fair comparison, we record the runtime for 20 training epochs with only one region from the grid split for all environments. In practice, this process can be run on GPUs in parallel for multiple regions. Given that the neural network structures for the 2D Quadrotor environment with both moving and fixed obstacles are the same, the runtime results are similar; therefore, we report these collectively as 2D Quadrotor. The results indicate that incremental verification significantly reduces the average verification time during training. Without incremental verification, the verification time increases rapidly as the number of steps increases.

Table 3 shows the ablation study results for using multiple initial-state-dependent controllers. We report results for the Vehicle Avoidance environment (Veh. Avoid.), 2D Quadrotor with fixed obstacles (2D-Quad (F)), moving obstacles (2D-Quad (M)), and 3D Quadrotor (3D-Quad (F)). We exclude the Lane Following environment from this comparison, as only one controller was used there to achieve 100% verified safety. The results demonstrate that using multiple controllers significantly enhances the ability to achieve verified safety across a larger initial state region S 0 .

A.2 Experiment Setup

Lane Following Our lane following environment follows the discrete-time bicycle model [Kong et al., 2015]

where we set the wheel base of the vehicle to 2 . 9 m. The model inputs are 3-dimensional ( x, θ, v ) , where x is the lateral distance to the center of the lane, θ is the angle relative to the center of the lane, and v represents the speed. The objective is to maintain a constant speed while following the lane, meaning the system equilibrium point is ( x, θ, v ) = (0 , 0 , v target ) . The safety constraints are

x stays within a maximum distance from the lane center ( ∥ x ∥ ≤ d max ) ,

θ remains within a predefined range ( ∥ θ ∥ ≤ θ max ) ,

v does not exceed the maximum threshold ( v ≤ v max ) .

The parameters are set as d max = 0 . 7 , θ max = π/ 4 , and v max = 5 . 0 . The initial regions S 0 is x ∈ [ -0 . 5 , 0 . 5] , θ ∈ [ -0 . 2 , 0 . 2] , v ∈ [0 . 0 , 0 . 5] . The reward received at each step is measured as the distance to the equilibrium point. More specifically, for a state that is of distance d to the target equilibrium point, the reward is e -d . For VSRL training, our controller is initialized using a controller pretrained with a safe RL algorithm. When training with the bound loss, we add a large penalty on unsafe states to incentivize maintaining safety throughout the entire trajectory. For branch and bound during verification, we set the precision limit as 0 . 025 , which means as soon as the precision of the grid region reaches this precision, we stop branching. For the dynamics approximation, we use an NN with two layers of ReLU each of size 8 .

Vehicle Avoidance Our vehicle avoidance environment features a vehicle moving on an x -y plane, with 4-dimensional inputs ( x, y, θ, v ) . Here, ( x, y ) represents the location of the vehicle on the plane, θ is the angle relative to the y -axis, and v is the speed. In this setting, we have five moving obstacles, each moving from one point to another at a constant speed for the duration of 500 steps. The five obstacles are: 1) moving from ( x, y ) = ( -0 . 6 , 1 . 0) to ( x, y ) = ( -0 . 35 , 2 . 0) ; 2) moving from ( x, y ) = (0 . 6 , 0 . 0) to ( x, y ) = (0 . 75 , 1 . 0) ; 3) moving from ( x, y ) = (0 . 0 , 1 . 0) to ( x, y ) = (0 . 0 , 2 . 0) ; 4) moving from ( x, y ) = ( -0 . 85 , 1 . 0) to ( x, y ) = ( -1 . 6 , 1 . 5) ; 5) moving from ( x, y ) = (0 . 75 , 0 . 0) to ( x, y ) = (0 . 85 , 0 . 0) . Each obstacle is represented as a square with a diameter of 0 . 1 . Additionally, safety constraints are set for the speed ( v ≤ v max ) and angle ( ∥ θ ∥ ≤ θ max ) , where v max = 5 . 0 and θ max = π/ 2 . The task is to navigate the vehicle to a designated location while following safety constraints. The agent starts near the origin within an area defined by x, y ∈ [ -0 . 5 , 0 . 5] , θ ∈ [ -0 . 2 , 0 . 2] , and v ∈ [0 , 0 . 1] , and the goal is ( x target , y target ) = (1 . 0 , 2 . 0) . The branching precision limit is 0 . 025 and for dynamics approximation, we use a NN with two layers of ReLU each of size 10 .

2D Quadrotor For the 2D quadrotor environment, we follow the settings in Dai et al. [2021].

We use a timestep dt = 0 . 02 , the mass of the quadrotor is set to m = 0 . 486 , the length to l = 0 . 25 , the inertia to I = 0 . 00383 , and gravity to g = 9 . 81 . The input is 6-dimensional ( y, z, θ, ˙ y, ˙ z, ˙ θ ) , where ( y, z ) represents the position of the quadrotor on the y -z plane, and θ represents the angle. The action space is 2-dimensional and continuous; the actions are clipped within a range to reflect motor constraints. Our safety criteria are

angle θ remains within a predefined range ( ∥ θ ∥ ≤ θ max ) ,

a minimum height constraint to prevent collision with the ground ( y ≥ y min ) ,

avoid obstacles.

Here we set θ max = π/ 3 and y min = -0 . 2 . The task is for the quadrotor to navigate towards the goal while following safety constraints. We consider two scenarios for obstacles: fixed and moving. For

fixed obstacles, there are five rectangular obstacles positioned in the y -z plane. We use ( x l , x u , y l , y u ) to represent the two dimensional box, and the obstacles are: ( x l , x u , y l , y u ) = ( -0 . 3 , -0 . 1 , 0 . 4 , 0 . 6) , ( x l , x u , y l , y u ) = ( -1 . 2 , -0 . 8 , 0 . 2 , 0 . 4) , ( x l , x u , y l , y u ) = (0 . 0 , 0 . 1 , 0 . 5 , 1 . 0) , ( x l , x u , y l , y u ) = (0 . 6 , 0 . 7 , 0 . 0 , 0 . 2) , ( x l , x u , y l , y u ) = ( -0 . 8 , -0 . 7 , 0 . 7 , 0 . 9) . For moving obstacles, there are five obstacles that moves from one point to another at constant speed for the duration of 500 steps, each represented as a square of diameter 0 . 1 . The obstacles are: 1) moving from ( x, y ) = (0 . 6 , 0 . 0) to ( x, y ) = (0 . 6 , 0 . 1) ; 2) moving from ( x, y ) = ( -0 . 5 , 0 . 2) to ( x, y ) = ( -0 . 4 , 0 . 3) ; 3) moving from ( x, y ) = ( -0 . 3 , 0 . 4) to ( x, y ) = ( -0 . 4 , 0 . 5) ; 4) moving from ( x, y ) = ( -0 . 1 , 0 . 3) to ( x, y ) = (0 . 0 , 0 . 4) ; 5) moving from ( x, y ) = ( -0 . 7 , 0 . 5) to ( x, y ) = ( -0 . 4 , 0 . 6) . The initial region for the quadrotor is defined with x ∈ [ -0 . 5 , 0 . 5] and the remaining state variables within [ -0 . 1 , 0 . 1] . The target goal is set to ( x, y ) = (0 . 6 , 0 . 6) . We set the branching limit to 0 . 0125 and for dynamics approximation we use a NN with two layers of ReLU each of size 6 .

3D Quadrotor Our 3D quadrotor environment features a 12-dimensional input space, represented as ( x, y, z, ϕ, θ, ψ, ˙ x, ˙ y, ˙ z, ω x , ω y , ω z ) . The action space is 4-dimensional and continuous; the actions are clipped within a range to reflect motor constraints. Here, ( x, y, z ) denotes the location of the quadrotor in space, ϕ is the roll angle, θ is the pitch angle, and ψ is the yaw angle, ω x , ω y , ω z represent the angular velocity around the x , y , and z axes, respectively. The environment setting and neural network dynamics approximation follows the setup in Dai et al. [2021], with the modification of using ReLU activations instead of LeakyReLU. The system dynamics is:

The dynamics neural network has two ReLU layers, each with a size of 16 and dt = 0 . 02 . We set the branching precision limit to 0 . 00625 . The task is to navigating towards the goal while avoiding five obstacles represented as 3D rectangles. The locations of the obstacles are ( -0 . 5 , 0 . 5 , -0 . 2 , 0 . 2 , -0 . 65 , -0 . 55) , ( -0 . 7 , -0 . 6 , -0 . 1 , 0 . 1 , -0 . 5 , -0 . 4) , (0 . 5 , 0 . 6 , -0 . 2 , 0 . 2 , -0 . 4 , -0 . 3) , ( -0 . 8 , -0 . 6 , 0 . 2 , 0 . 4 , -0 . 3 , -0 . 2) , ( -0 . 8 , -0 . 6 , -0 . 4 , -0 . 2 , -0 . 2 , -0 . 1) , where the first obstacle is to avoid controller collide with the ground. We set the goal at ( x, y, z ) = (0 . 0 , 0 . 0 , 0 . 0) , and the initial region is defined with x ∈ [ -0 . 5 , 0 . 5] , y ∈ [ -0 . 1 , 0 . 1] , and z ∈ [ -0 . 5 , -0 . 3] , with the remaining variables confined to the range [ -0 . 05 , 0 . 05] . The reward is calculated based on the distance to the goal, where the agent receives a higher reward for being closer to the goal. The environment episodes end if either the magnitude of ϕ or θ exceeds π/ 3 .

A.3 Compute Resources

Our code runs on an AMD Ryzen 9 5900X CPU with a 12-core processor and an NVIDIA GeForce RTX 3090 GPU.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Abstract and Introduction Section

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: we discuss the primary limitation of the work in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide details of the experimental setup both in Section A.2 and in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will provide all code and data on github, along with documentation, to facilitate reproduction of the experiments if the paper is accepted for publication.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide this information in Section 4 and the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report confidence intervals in the results table in the experiments (see Section 4).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Justification: We provide this information in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All aspects of the paper and research conform in all respects to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: As mentioned in the introduction, ability to synthesize safe controllers is a core challenge in autonomous systems (such as autonomous cars). As such, our work is expected to have positive broader impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper is about safe reinforcement learning, and poses no such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper references the Auto_Lirpa codebase and associated research papers describing it.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper proposes methods to learn formally verified neural network control policies for (continuous-space, discrete-time) non-linear dynamical systems, whose dynamics are also represented by a neural network. It builds on existing methods for NN verification, and their application to a k-step composition of NNs for the policy and dynamics. The key novel ideas of the paper are a variant of curriculum learning, built into an incremental policy synthesis approach over increasing time horizons, and a parameterized approach to representing state-dependent policies. Experimental results show improved performance on four benchmarks compared to five comparable techniques.

STRENGTHS:
- Clear and very well written paper.

- Clearly presented and motivated novel ideas to improve performance on a challenging problem.

- Comprehensive empirical evaluation with a good number of meaningful benchmarks and baselines, and a further ablation study in the appendix.

- Impressive gains over baseline implementations.

WEAKNESSES:
- The empirical results focus on the degree of safety achieved by different policies, but there seems to be no discussion of performance, e.g. runtime, or more directly the scalability of the various methods.

QUESTIONS:
1. Relating to the stated weakness above, what is the experimental setup in terms of timeout (if any) used to compare tools, and what is the limiting factor in terms of the techniques' ability to synthesise safe policies?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors primarily propose a novel method to learn verified safe control policies for nonlinear neural dynamical systems, aiming to achieve safety in the sense of bounded reachability. By leveraging memorization, forward reachability analysis, and differentiable reachability over-approximation, the authors effectively learn verified safe policies, and further introduce an incremental verification approach to enhance the efficiency of the learning process, enabling the acquisition of multiple verified initial state-dependent controllers.\n\nSTRENGTHS:\nThe manuscript proposes a method for learning a k-step verified safe neural network controller, aimed at maximizing the efficiency of systems with neural dynamics. In contrast to traditional approaches relying on forward-invariance proofs for safe control synthesis, the manuscript opts for the more practical bounded reachability verification, enabling the leveraging of state-of-the-art differentiable neural network approximation tools. \n\nThe experimental results demonstrate that, across several dynamic system environments considering both static and moving obstacles, the proposed approach significantly outperforms the state-of-the-art safe reinforcement learning baselines. This highlights the advantages of the bounded reachability verification framework over the traditional forward-invariance guarantees, in terms of practical viability and performance when applied to systems with neural dynamics.\n\nWEAKNESSES:\nThe algorithm proposed in this manuscript lacks rationality analysis\n\n(1) Page 3.: “and required statistical assumptions“->”and requires statistical assumptions”\n\n(2) Page 4.: “which enable learning verified safe controllers over longer horizons K”->”which enables learning verified safe controllers over longer horizons K”\n\n(3) Page 4.: “In this work, we primarily utilize the α,β-CROWN toolbox”-> The α,β-CROWN toolbox should give a brief introduction here\n\nQUESTIONS:\n(1) Can the author explain the soundness of the approach proposed in the manuscript?\n\n(2) The rationality of the experimental comparison baseline, why is it more interesting to compare with these methods? What are the maximum dimensions of the system security verification problem that can be solved by this method?\n\nLIMITATIONS:\nThe manuscript thoroughly discusses the limitations of the proposed methods, and the security guarantees provided are weaker compared to forward-invariance.  However, bounded reachability offers a more practical approach to achieving verified safety, effectively realizing the safety of the entire event horizon in practice, and provides an alternative for the development of verified safe reinforcement learning.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of synthesizing verified control policies for dynamical systems with the use of neural networks. The authors focus on nonlinear discrete-time system dynamics for which the use of neural networks is motivated by the challenge of reaching the goal without colliding with obstacles. To guarantee non-collision and maintain reasonable performance, the authors propose an approach of iteratively using an existing verification toolbox with curriculum-based lookahead. The approach is evaluated on the benchmarks from previous work.\n\nSTRENGTHS:\nThe approach of building a curriculum of horizons and re-using verification results from the previous steps is relatively novel and provides an interesting future direction. \n\nIt is a significant improvement over the baselines that the proposed algorithm achieves 100% empirical safety. The decrease in performance is evident for several benchmarks and may exacerbate for a longer horizon. \n\nThe paper is overall well-structured. However, a motivating example to guide the reader through the algorithm would be helpful.\n\nWEAKNESSES:\nAs can be seen from the ablation study, the runtime improvement is achieved with the proposed approach when the lookahead is longer than 5 steps. It is not clear if a long lookahead is required for the considered benchmarks.\nThe authors motivate the choice of verification toolbox (a,b-CROWN) with its suitability for incremental verification. It is not clear how much the proposed technique is constrained by the chosen toolbox or general enough to be adopted for other verification tools under certain assumptions. \n\nThe choice of benchmark problems is not motivated in the evaluation section. The description of the benchmarks is not complete and requires looking into original papers. It would be appreciated to have complete system models stated formally in the appendix.\n\nQUESTIONS:\n1. Could the authors explain the choice of these particular benchmark problems to evaluate the approach on?\n2. Is the state-space continuous or discrete for these benchmarks? (it appears it is continuous but only action-space is explicitly mentioned)\n3. Which of the chosen benchmark problems particularly showcases the benefit of a longer-than-five step lookahead reachability?\n4. How is the average reward computed (with verification over what horizon size)? Is it dependent on the verification horizon?\n5. Is it possible to incorporate the proposed forward K-step reachability into the related reachability-based verification tools which do not yet consider it? What features or assumptions need to be satisfied?\n\nLIMITATIONS:\nThe approach seems to have been specifically designed for verification tools that are suitable for incorporating into gradient-based learning. This can be a significant limitation to generalizability of the approach, since the authors do not discuss or evaluate what requirements the verification tools need to satisfy to be adopt the proposed approach. Based on the current presentation, the approach can only work with a,b-CROWN toolbox. If this is the case, it must be explicitly discussed as a limitation, it is, however, still an improvement over this particular domain.\n\nOpen-access to data and code: the answer is \"yes\" when the data and code are anonymously open-source. If they are not yet open-sourced the answer should be \"no yet\".", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies safe reinforcement/control learning by optimization of long-horizon safety verification and learning multiple initial-state-dependent controllers. The authors propose several novel ideas, including a curriculum learning to increase the verification horizon, incremental verification, and split the initial region for multi-controllers training. A set of control tasks show the effectiveness of the approach, compared to other CMDP-based safe RL methods.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow.\n2. As far as I can tell, the idea of optimizing the verification horizon (curriculum learning) is novel and the experimental result is quite significant. \n3. The approach is sound.\n\nWEAKNESSES:\n1. the reviewer feels that incremental verification in this paper is quite common in the reachable set computation of neural network-controlled systems (NNCS), where the computed reachable set at $k$-th step will become the initial set for $k+1$ set computation. The authors may have to discuss their incremental verification vs. NNCS verification tools. To name a few tools, please refer to POLAR-Express[1], CORA[2], etc.\n2. Splitting the state/initial space set into small grids for verification and controller synthesis/training is also common in the NNCS verification community. However, I have to admit that the design of multi-initial-state-dependent controllers is new to me.  \n3. The experiments only compare to the CMDP-based approach, it is unclear how the proposed approach compares to other control-theoretical methods, for instance, the CBF-based approaches. \n\n[1] Wang, Y., Zhou, W., Fan, J., Wang, Z., Li, J., Chen, X., ... & Zhu, Q. (2023). Polar-express: Efficient and precise formal reachability analysis of neural-network controlled systems. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems. https://github.com/ChaoHuang2018/POLAR_Tool\n[2] https://github.com/TUMcps/CORA?tab=readme-ov-file\n\nQUESTIONS:\n1. How to ensure that Algorithm 2 can terminate?\n\nLIMITATIONS:\nThe authors have addressed the limitations in the final part.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}