Title: Exact Verification of ReLU Neural Control Barrier Functions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Safety

Neural Barrier Function

Verification
Abstract: Control Barrier Functions (CBFs) are a popular approach for safe control of nonlinear systems. In CBF-based control, the desired safety properties of the system are mapped to nonnegativity of a CBF, and the control input is chosen to ensure that the CBF remains nonnegative for all time. Recently, machine learning methods that represent CBFs as neural networks (neural control barrier functions, or NCBFs) have shown great promise due to the universal representability of neural networks. However, verifying that a learned CBF guarantees safety remains a challenging research problem. This paper presents novel exact conditions and algorithms for verifying safety of feedforward NCBFs with ReLU activation functions. The key challenge in doing so is that, due to the piecewise linearity of the ReLU function, the NCBF will be nondifferentiable at certain points, thus invalidating traditional safety verification methods that assume a smooth barrier function. We resolve this issue by leveraging a generalization of Nagumo's theorem for proving invariance of sets with nonsmooth boundaries to derive necessary and sufficient conditions for safety. Based on this condition, we propose an algorithm for safety verification of NCBFs that first decomposes the NCBF into piecewise linear segments and then solves a nonlinear program to verify safety of each segment as well as the intersections of the linear segments. We  mitigate the complexity by only considering the boundary of the safe region and by pruning  the segments with Interval Bound Propagation (IBP) and linear relaxation. We evaluate our approach through numerical studies with comparison to state-of-the-art SMT-based methods. Our code is available at https://github.com/HongchaoZhang-HZ/exactverif-reluncbf-nips23.

Exact Verification of ReLU Neural Control Barrier Functions

Abstract

Control Barrier Functions (CBFs) are a popular approach for safe control of nonlinear systems. In CBF-based control, the desired safety properties of the system are mapped to nonnegativity of a CBF, and the control input is chosen to ensure that the CBF remains nonnegative for all time. Recently, machine learning methods that represent CBFs as neural networks (neural control barrier functions, or NCBFs) have shown great promise due to the universal representability of neural networks. However, verifying that a learned CBF guarantees safety remains a challenging research problem. This paper presents novel exact conditions and algorithms for verifying safety of feedforward NCBFs with ReLU activation functions. The key challenge in doing so is that, due to the piecewise linearity of the ReLU function, the NCBF will be nondifferentiable at certain points, thus invalidating traditional safety verification methods that assume a smooth barrier function. We resolve this issue by leveraging a generalization of Nagumo's theorem for proving invariance of sets with nonsmooth boundaries to derive necessary and sufficient conditions for safety. Based on this condition, we propose an algorithm for safety verification of NCBFs that first decomposes the NCBF into piecewise linear segments and then solves a nonlinear program to verify safety of each segment as well as the intersections of the linear segments. We mitigate the complexity by only considering the boundary of the safe region and by pruning the segments with Interval Bound Propagation (IBP) and linear relaxation. We evaluate our approach through numerical studies with comparison to state-of-the-art SMT-based methods. Our code is available at https: //github.com/HongchaoZhang-HZ/exactverif-reluncbf-nips23 .

1 Introduction

Safety is a critical property for autonomous systems, including unmanned ground, aerial, and space vehicles [1] and robotic manipulators [2]. The importance of safety has motivated extensive research into verification and synthesis of safe control strategies [3, 4, 5, 6, 7]. Control barrier function (CBF)-based algorithms [8] ensure safety by constructing a CBF that is nonnegative if the system is safe and synthesizing a control policy that ensures that the CBF is nonnegative for all time. Control barrier functions provide a high degree of flexibility, since any control policy that satisfies the control barrier function constraint is provably safe, and have been demonstrated in applications including

robotic manipulation [9], vehicle cruise control [10] and space exploration [11]. Recently, CBFs that are parametrized by neural networks ( neural control barrier functions or NCBFs) [12, 13, 14] have been proposed. This novel class of NCBFs is promising due to the universal representability of neural networks (enabling the encoding of complex safety constraints) and the efficiency of learning algorithms, with a comparison in Section A.15 exemplifying numerical difference. However, safety verification of NCBF-based control remains a challenging research problem.

In this paper, we consider the problem of verifying safety of NCBF-based control for nonlinear continuous-time systems. We focus on NCBFs represented by feedforward neural networks with ReLU activation due to their fast convergence in training shown in Section A.16 and widespread use in the safe control literature [12, 13]. The key challenge for this class of NCBFs is that most methodologies for safety verification are based on proving that the derivative of the barrier function is nonnegative at the boundary of the safe region, and hence the barrier function remains nonnegative for all time. Since the ReLU activation function is not continuously differentiable, this approach is inapplicable. We resolve this challenge and derive exact safety conditions for ReLU NCBFs by leveraging a generalization of Nagumo's theorem for proving invariance of sets with nonsmooth boundaries. Based on these conditions, we propose a new class of NCBF-based control policies that do not require the control policy and NCBF to be jointly trained, thus improving flexibility and tunability of the controller design. Our safety conditions also incorporate linear constraints on the control input, which may arise due to limits on actuation.

We propose an algorithm to verify that an NCBF satisfies our derived safety conditions, implying that any control policy that is constrained by the NCBF will be safe. Our approach first decomposes the NCBF into piecewise linear segments. In order to mitigate the complexity of this stage, we show that it suffices to consider linear segments at the boundary of the safe region, and further over-approximate the segments using Interval Bound Propagation and linear relaxation. After decomposing the NCBF, we verify safety by solving a set of nonlinear programs, which check the safety criteria on each linear segment as well as at intersections of the segments. We evaluate our approach through numerical studies, in which we verify NCBFs with our proposed approach and compare to state-of-the-art Satisfiability Modulo Theory (SMT) based methods.

Related Work Energy-based methods have been proposed to guarantee safety by ensuring that a particular energy function remains nonnegative. Barrier certificates for safe control were first proposed in [15]. More recently, CBFs have emerged as promising approaches to safe control, due to their compatibility with a wide variety of control laws [8, 16, 17, 5, 14, 12, 1]. Sum-of-squares (SOS) optimization and other techniques derived from algebraic geometry have been widely used for safety verification of polynomial barrier functions [15, 8, 18, 19, 17]. However, SOS-based approaches for polynomial CBFs cannot be applied directly to NCBF verification, since activation functions used in neural networks are not polynomial and may be non-differentiable.

Neural barrier certificate [20, 21, 22, 23] and NCBFs [12, 13, 14] have been proposed to describe complex safety constraints that cannot be encoded polynomials. Current work, including SMT-based methods [24, 21, 25] and mixed integer programs [26], verify safety by constructing a nominal control policy and proving that it satisfies the NCBF constraints. However, as we show in Section 5, the reliance on a particular control policy may lead to false negatives during safety verification. Another related body of work deals with the problem of verifying neural networks including SMT [27], output reachable set verification [28], polynomial approximations of the barrier function [29], verify input/output relationships [30, 31, 32] and ReLU neural networks focused verification [33, 34]. These methods, however, are not directly applicable to the problem of NCBF verification, which requires joint consideration of the neural network and the underlying nonlinear system dynamics. Methods based on input/output relationship can verify by encode dynamics and control policy with neural networks. However, with approximating error introduced, these methods are not directly applicable for exact verification. Piecewise linear approximations of ReLU neural networks have been used to develop tractable safety verification algorithms using linear [35] and SOS programming [36]. These approaches leads to sound and incomplete verification algorithms and have only be applied for discrete-time systems, whereas the present paper proposes exact verification algorithms for continuous-time systems.

Organization The remainder of the paper is organized as follows. Section 2 gives the system model and background on neural networks and notation. Section 3 presents the problem formulation and

exact conditions for safety. Section 4 presents our approach to verifying that an NCBF satisfies the safety conditions. Section 5 contains simulation results. Section 6 concludes the paper.

2 Model and Preliminaries

In this section, we first present the system model and definition of safety. We then give background and notations of feedforward neural networks.

2.1 System Model and Safety Definition

We consider a continuous-time nonlinear control-affine system with state x ( t ) ∈ X ⊆ R n , control input u ( t ) ∈ U ⊆ R m , and dynamics

where f : R n → R n and g : R n → R n × m are known functions. A control policy is a function µ : R n →U that maps a state x to a control input u .

Safety of dynamical systems requires x ( t ) to remain in a given region C , which we denote as the safe region. We assume the safe region is given by C = { x : h ( x ) ≥ 0 } ⊆ X , for some function h : X → R . Safety is related to the property of positive invariance, which we define as follows.

Definition 1. A set D ⊆ R n is positive invariant under dynamics (1) and control policy µ if x (0) ∈ D and u ( t ) = µ ( x ( t )) ∀ t ≥ 0 imply that x ( t ) ∈ D for all t ≥ 0 .

We define a control policy µ to be safe if there is a set D such that (i) D ⊆ C and (ii) D is positive invariant under dynamics (1) and control policy µ .

One approach to designing safe control policies is to choose a function b : R n → R , denoted as a Control Barrier Function (CBF) , and let D = { x : b ( x ) ≥ 0 } . In the case where b is continuously differentiable, the following result can be used to guarantee positive invariance of D .

Theorem 1 ([8]) . Suppose that b is a CBF, b ( x (0)) ≥ 0 , and u ( t ) satisfies

for all t , where α : R → R is a strictly increasing function with α (0) = 0 . Then the set D = { x : b ( x ) ≥ 0 } is positive invariant.

Theorem 1 implies that, if b is a CBF, then adding (2) as a constraint on the control at each time step suffices to guarantee safety.

Recently, Neural Control Barrier Functions (NCBFs), in which the function b is encoded by a neural network, have been proposed [12, 13, 14]. The advantage of NCBFs arises from the universal representability of neural networks, which allows them to realize a wide variety of safety constraints.

2.2 Neural Network Background and Notation

We give notations to describe a neural network (NN) with L layers and M i neurons in the i -th layer. We let the input to the NN be denoted x , the output at the j -th neuron of the i -th layer be denoted z ij , and the output of the network be denoted y . We let z i denote the vector of neuron outputs at the i -th layer. The outputs are computed as

where σ : R → R is the activation function. The input to the function σ is the pre-activation input to the neuron, and is given by W T 1 j x + r 1 j for the j -th neuron at the first layer and W T ij z i -1 + r ij for the j -th neuron at the i -th hidden layer. W ij has dimensionality n × 1 for i = 1 and M i -1 × 1 for i > 1 . W i is an n × M i matrix. In this paper, we assume that σ is the ReLU function ReLU ( z ) = max { 0 , z } . The output of the network is given by y = Ω T z L + ψ , where Ω ∈ R M L and ψ ∈ R . The j -th neuron at the i -th layer is activated by a particular input x if its pre-activation input is nonnegative, inactivated if the pre-activation input is nonpositive, and unstable if the pre-activation input is zero.

A set of neurons S = ( S 1 , . . . , S L ) , with S i ⊆ { 1 , . . . , M i } denoting a subset of neurons at the i -th layer, is activated by x if all of the neurons in S are activated by x and all the neurons not in S are inactivated by x . A set of neurons T = ( T 1 , . . . , T L ) with T i ⊆ { 1 , . . . , M i } is unstable by x if all of the neurons in T are unstable by x .

For a given set S , if S is activated by x , then the pre-activation input to each neuron and the overall output of the network are affine in x , with the affine mapping determined by S as follows. For the first layer, we define

so that the output of the j -th neuron at the first layer is W 1 j ( S ) T x + r 1 j ( S ) . We recursively define W ij ( S ) and r ij ( S ) by letting W i ( S ) be a matrix with columns W i 1 ( S ) , . . . , W iM i ( S ) and

where r i ( S ) is the vector with elements r ij ( S ) , j = 1 , . . . , M i .

We define W ( S ) = W L ( S )Ω and r ( S ) = Ω T r L ( S ) + ψ . Based on these notations, when an input x activates the set S , z ij = W ij ( S ) T x + r ij ( S ) and y = W ( S ) T x + r ( S ) .

Lemma 1. Let X ( S ) denote the set of inputs x that activate a particular set of neurons S , and let W 0 ( S ) be equal to the identity matrix, and r 0 to be zero vector. Then

A proof can be found in the supplementary material. Note that in (6), a particular input x could belong to multiple activation regions X ( S ) . This is because if the j -th neuron at the i -th layer is unstable, then both ( S 1 , . . . , S i ∪ { j } , . . . , S L ) and ( S 1 , . . . , S i \ { j } , . . . , S L ) can be regarded as activated by x . We let S ( x ) ≜ { S : x ∈ X ( S ) } and let T ( x ) denote the set of unstable neurons produced by input x . Given a collection of activation sets S 1 , . . . , S r , we let

The set T ( S 1 , . . . , S r ) is equal to the set of neurons that must be unstable in order for an input x to belong to X ( S 1 ) ∩ . . . ∩ X ( S r ) .

3 Problem Formulation and Safety Conditions

In this section, we first formally define the problem, and then give necessary and sufficient conditions for verifying NCBFs.

3.1 Problem Formulation

We consider a feedforward neural network (NN) b : X → R with ReLU activation function. Since b is not continuously differentiable due to the piecewise linearity of the ReLU function, the guarantees of Theorem 1 cannot be applied directly. Our overall goal will be to derive analogous conditions to (2) for ReLU-NCBFs, and then ensure that any control policy µ that satisfies the conditions will be safe with D = { x : b ( x ) ≥ 0 } .

Problem 1. Given a nonlinear continuous-time system (1), a neural network function b : X → R , a set of admissible control inputs U = { u : Au ≤ c } for given matrix A ∈ R p × m and vector c ∈ R p , and a safe set C = { x : h ( x ) ≥ 0 } , determine whether (i) D ⊆ C and (ii) there exists a control policy µ such that D is positive invariant under dynamics (1) and control policy µ .

3.2 Exact Conditions for Safety

When a CBF b ( x ) is continuously differentiable, ensuring that (2) is satisfied is equivalent to verifying that there is no x satisfying b ( x ) = 0 , ∂b ∂x g ( x ) = 0 , and ∂b ∂x f ( x ) < 0 [17]. When b is represented by a ReLU neural network, however, b will not be differentiable when the input x leads to neurons having zero pre-activation input, i.e., when T ( x ) = ∅ . Although the set of x with T ( x ) = ∅ has measure zero, such points can nonetheless cause safety violations as illustrated by the following example.

̸

Example: Let x ( t ) ∈ R 2 and suppose the dynamics of x are given by

̸

Suppose that U = R m , the safe region C = { x : x 2 1 + x 2 2 ≤ 9 } , and the candidate CBF is given by b ( x ) = 1 -|| x || 1 . This CBF can be realized by a neural network with a single layer ( L = 1 ) with four neurons ( M 1 = 4 ), weights W 11 = (1 0) T , W 12 = ( -1 0) T , W 13 = (0 1) T , W 14 = (0 -1) T , r 1 j = 0 for j = 1 , . . . , 4 , Ω j = -1 for j = 1 , . . . , 4 , and ψ = 1 . We observe that D = { x : b ( x ) ≥ 0 } ⊆ C . Furthermore, whenever ∂b ∂x exists, we have ∂b ∂x ∈ { (1 1) , (1 -1) , ( -1 1) , ( -1 -1) } , each of which satisfies ∂b ∂x g ( x ) = 0 . On the other hand, the set D is not positive invariant. If x 2 (0) > 1 5 , then | x 1 (0) | ≤ 1 implies that ˙ x 2 (0) > 0 , and indeed, x 2 ( t ) will continue to increase until x ( t ) / ∈ D . More details about this example can be found in Section A.17.

Fundamentally, this safety violation occurs because at x = (0 1) T , we have b ( x ) = 0 , T ( x ) = { (1 , 1) , (1 , 2) } (i.e., the preactivation input to the first and second neurons in the hidden layer is zero), creating a discontinuity in the slope of b ( x ) . For x ′ in the neighborhood of (0 1) T , the value of ∂b ∂x ( x ′ ) g ( x ′ ) will be either 1 or -1 . Since there is no single control input that satisfies (2) for both values of ∂b ∂x g ( x ′ ) , it is impossible to ensure safety of the system in the neighborhood of (0 1) T .

The following lemma addresses this issue by giving exact and general conditions for a NCBF with ReLU activation function to satisfy positive invariance. We let ∂ D denote the boundary of the set D .

Lemma 2. The set D is positive invariant under control policy µ if and only if, for all x ∈ ∂ D , there exist S ∈ S ( x ) such that

The proof is based on the generalized Nagumo's Theorem [37] and a novel characterization of the Clarke tangent cone to ∂ D , and can be found in the supplementary material due to space constraints. Intuitively, conditions (7)-(9) can be interpreted as follows. Condition (9) is similar to (2) when the gradient is given by W ( S ) , i.e., when x is in the interior of the activation region defined by S . Eq. (7)-(8) can be interpreted as choosing the control input to ensure that x remains in the activation region of S (i.e., X ( S ) ).

Lemma 2 can be used to construct NCBF-based safe control policies, analogous to control policies based on continuously differentiable CBFs. Formally, given a nominal control policy µ nom , at each time t , we can choose u ( t ) by solving the optimization problem

where α satisfies the same conditions as in Theorem 1. This problem can be solved by decomposing (10) into | S ( x ) | quadratic programs (one for each S ) and then selecting the value of u that minimizes the objective function across all of the QPs. In particular, if S ( x ) is a singleton, then the constraints on (10) reduce to Eq. (2).

We then give an equivalent condition for b ( x ) to be an NCBF. As a preliminary, we say that a collection of activation sets S 1 , . . . , S r is complete if for any S ′ / ∈ { S 1 , . . . , S r } , we have X ( S 1 ) ∩ · · · ∩ X ( S r ) ∩ X ( S ′ ) = ∅ .

̸

Proposition 1. The function b is a valid CBF iff the following two properties hold:

(i) For all activation sets S 1 , . . . , S r with { S 1 , . . . , S r } complete and any x satisfying b ( x ) = 0 and

there exist l ∈ { 1 , . . . , r } and u ∈ U such that

(ii) For all activation sets S , we have

The proof can be found in the supplementary material. We refer to any x that fails to meet at least one of conditions (i) and (ii) of Proposition 1 as a safety counterexample.

Enumerate

So=[110 ...10]

So=[110 ..10]

1.2

S=[111 ...00]

S=[111 ...00]

Find

0.4

Prune

S=[110..00] intersections

Vo = [So,s1]

S3=[111 ...00]

S=

S3=[111 ...00]

v =V =[s,s]

0.4

S4=[101...00]

S4=[101..00]

Ss=[111...00]

Ss =[111 ...0 0]

-1.2

2.0

2.0

1.2

0.4

0.4

1.2

Foreachcell

Gridifythestatespaceand

ForeachcelluseIBP

Uselinearprogram

containingmultipleSE

findcellsthatcrossO-levelset

tolist allpossible

tofind exact

S,useLinearProgram

b(x)=0

activatedsetsS

activatedsetsS

tofindintersectionV

4 Verification Algorithm

The preceding proposition motivates our overall approach to verifying NCBFs, consisting of the following steps. (Step 1) We conduct coarser-to-finer search by discretizing the state space into hyper-cubes and use linear relaxation based perturbation analysis (LiRPA) to identify grid squares that intersect the boundary { x : b ( x ) = 0 } . (Step 2) We enumerate all possible activation sets within each hyper-cube using Interval Bound Propagation (IBP). We then identify the activation sets and intersections that satisfy b ( x ) = 0 using linear programming. (Step 3) For each activation set and intersection of activation sets, we verify the conditions of Proposition 1. In what follows, we describe each step in detail.

4.1 Enumeration of Activation Set Boundaries

We next present the approach to enumerate activation sets of a given ReLU NCBF that intersect the safety boundary, b ( x ) = 0 . Our approach first conducts a coarse-grained search that overapproximates the collection of activation sets that intersect with the safety boundary. We then prune this collection to remove activation sets that cannot be realized. Specifically, we first identify regions that contain b ( x ) = 0 , then enumerate all possible activation sets within the region and finally identify the sets that intersect the safety boundaries.

We first discretize the given state space X into hyper-cubes. We compute lower and upper bounds of b ( x ) on each hyper-cube, denoted b l and b u respectively, with linear relaxation based perturbation analysis (LiRPA), i.e., auto LiRPA [38]. For each hyper-cube B , we determine B contains b ( x ) = 0 (red squares in Fig 1) if criteria sgn ( b l ) · sgn ( b u ) ≤ 0 holds, where sgn ( ∗ ) is the sign function. For each B contains b ( x ) = 0 , we utilize IBP to over-approximate unstable neurons. Details on this IBP procedure can be found in Section A.7. Let ˜ S denote the over-approximated collection of activation

sets computed by IBP. Note that S ∈ ˜ S may not intersect with b ( x ) = 0 . As shown in Fig. 1, we let S denote the collection of activation sets S ∈ ˜ S that satisfy

for some x ∈ B . The activation set boundaries indicate unstable neurons, which need to be verified separately. Hence, we search for intersections as follows, where 2 S denotes the set of subsets of S .

4.2 Verifying Safety of Each Activation Set

With the activation sets enumerated, the following result gives an equivalent condition for verifying that there is no safety counterexample in the interior of X ( S ) for a given activation set S .

̸

Lemma 3. There is no safety counterexample in the set X ( S ) \ ⋃ S ′ = S X ( S ) if and only if:

The proof can be found in the supplementary material. Conditions 1 and 2 of Lemma 3 can be verified by solving nonlinear programs (see supplementary material for the formulations of these nonlinear programs). Such nonlinear programs are solved for each activation set S in the collection S computed during the enumeration phase. The following corollary describes the special case where there are no constraints on the control, i.e., U = R m .

̸

Corollary 1. If U = R m , then there is no safety counterexample in the set X ( S ) \ ⋃ S ′ = S X ( S ′ ) if and only if (1) there does not exist x ∈ X satisfying x ∈ int ( X ( S )) , b ( x ) = 0 , W ( S ) T g ( x ) = 0 , and W ( S ) T f ( x ) < 0 and (2) there does not exist x ∈ X with b ( x ) = 0 , x ∈ int ( X ( S )) , and h ( x ) < 0 .

̸

Under the conditions of Corollary 1, the complexity of the verification problem can be reduced. If g ( x ) is a constant matrix G , then safety is automatically guaranteed if W ( S ) T G = 0 . If f ( x ) is linear in x as well, then verification can be performed via a linear program.

4.3 Verification of Activation Set Intersections

With the activation set boundaries enumerated, we now describe an approach for verifying safety of intersections between activation sets.

Lemma 4. Suppose that the sets S 1 , . . . , S r satisfy condition (ii) of Lemma 3. There is no safety counterexample in the set

if and only if there do not exist x ∈ X and y 1 , . . . , y r ∈ R T + p +1 , where T = | T ( S 1 , . . . , S r ) | , satisfying (a) x ∈ int ( X ( S 1 )) ∩ · · · ∩ int ( X ( S r )) , (b) b ( x ) = 0 , (c) y l ≥ 0 ∀ l , (d) ∀ l = 1 , . . . , r , y T l Θ l ( S 1 , . . . , S r ( x )) = 0 , where Θ l ( S 1 , . . . , S r , x ) is a ( T + p +1) × m matrix

and (e) ∀ l = 1 , . . . , r , y T l Λ l ( S 1 , . . . , S r , x ) < 0 , where Λ l ( S 1 , . . . , S r , x ) ∈ R T + p +1 is given by

The proof can be found in the supplementary material. The verification problem can be mapped to solving a nonlinear program. This nonlinear program is solved for each ( S 1 , . . . , S r ) ∈ V , where V is computed during the enumeration phase. Note that, if g ( x ) is constant, then the constraints of the nonlinear program are linear, and if f ( x ) is linear then the problem reduces to solving a system of linear and bilinear inequalities. Furthermore, if f ( x ) can be bounded over x ∈ X ( S 1 ) ∩··· ∩ X ( S r ) and b ( x ) = 0 , then this bound can be used to derive a linear relaxation to the objective function of (27), thus relaxing the nonlinear program to a linear program.

The complexity of this approach can also be reduced by utilizing sufficient conditions for safety that are easier to check. For instance, if there exists u ∈ U such that W ( S ) T ( f ( x ) + g ( x ) u ) ≥ 0 for all S ∈ S ( x ) , then the criteria of Lemma 4 are satisfied. In particular, if W ( S ) T f ( x ) ≥ 0 for all S ∈ S ( x ) , then the conditions are satisfied with u = 0 . The following result is a straightforward consequence of Proposition 1, Lemma 3, and Lemma 4.

Theorem 2. Suppose that the conditions of Lemma 3 are satisfied for each S ∈ S and the conditions of Lemma 4 are satisfied for each { S 1 , . . . , S r } ∈ V . Then the NCBF b satisfies the conditions of Problem 1.

5 Experiments

In this section, we evaluate our proposed method to verify neural control barrier functions on three systems, namely Darboux, obstacle avoidance and spacecraft rendezvous. The experiments run on a 64-bit Windows PC with Intel i7-12700 processor, 32GB RAM and NVIDIA GeForce RTX 3080 GPU. We include experiment details in the supplement.

5.1 Experiment Setup

Darboux: We consider the Darboux system [39], a nonlinear open-loop polynomial system that has been widely used as a benchmark for constructing barrier certificates. The dynamic model is given in the supplement. We obtain the trained NCBF by following the method proposed in [24].

Obstacle Avoidance (OA): We evaluate our proposed method on a controlled system [40]. We consider an Unmanned Aerial Vehicles (UAVs) avoiding collision with a tree trunk. We model the system as a Dubins-style [41] aircraft model. The system state consists of 2-D position and aircraft yaw rate x := [ x 1 , x 2 , ψ ] T . We let u denote the control input to manipulate yaw rate and the dynamics defined in the supplement. We train the NCBF via the method proposed in [24] with v assumed to be 1 and the control law u designed as u = µ nom ( x ) = -sin ψ +3 · x 1 · sin ψ + x 2 · cos ψ 0 . 5+ x 2 1 + x 2 2 .

Spacecraft Rendezvous (SR): We evaluate our approach on a spacecraft rendezvous problem from [42]. A station-keeping controller is required to keep the "chaser" satellite within a certain relative distance to the "target" satellite. The state of the chaser is expressed relative to the target using linearized Clohessy-Wiltshire-Hill equations, with state x = [ p x , p y , p z , v x , v y , v z ] T , control input u = [ u x , u y , u z ] T and dynamics defined in the supplement. We train the NCBF as in [13].

hi-ord 8 : We evaluate our approach on an eight-dimensional system that first appeared in [21] to evaluate the scalability of proposed verification method.

5.2 Experiment Results

We first verify that D ⊆ C in the Darboux and obstacle avoidance scenarios. We trained four 1 hidden layer NCBFs with 20 , 5 , 32 , 10 neurons, respectively. NCBF (a) and (c) completed training, (b) terminated after 3 epochs and (d) terminated after 5 epochs. Our verification algorithm identifies safety violations. NCBF (a) and (c) pass the verification while NCBF (b) and (d) fail. As shown in

(a)

(b)

(c)

(d)

2

2

2

2

1

1

1

1

0

0

0

0

-1

-1

-1

-1

-2

2

-1

0

1

2

-2

2

-1

0

1

2

-2

2

-1

0

1

2

-2

=2

-1

0

1

2

Fig. 2a, the fully-trained NCBF (a) separates the safe and unsafe regions while the NCBF (b) with unfinished training has a boundary that intersects the unsafe region in Fig. 2b. As shown in Fig. 2d, the 2-D projection with ψ = -0 . 5 of NCBF (d) shows its boundary overlap the unsafe region.

We next verify the positive invariance of D . All NCBFs for Darboux pass feasibility verification with run-time presented in Table 1. We compare the run-time of proposed verification with SMT-based approaches using translator proposed in [21] and SMT solver dReal [43] and Z3 [44]. The proposed method can verify NCBFs with more ReLU neurons while SMT-based verifier return nothing. All of

the generated NCBFs for obstacle avoidance and spacecraft rendezvous passed the verification. The run-times are presented in Table 2. We find that the activation set sizes grow with the size of neural network and the state dimension n . We conjecture that this growth rate could be reduced by using tighter approximations for the activated sets.

A key feature of our approach is that the verification process does not depend on the control policy µ , but rather we verify that any µ satisfying (7)-(9) for all x ∈ ∂ D is safe. This improves flexibility while reducing false negatives in safety verification, as we illustrate in the following example. The NCBF for obstacle avoidance with one hidden layer and 32 neurons fails the SMT-based verification using dReal given nominal controller µ nom . The counter example is at the point ( -0 . 20 , 0 , -0 . 73) . At this point, we have b ( x ) = 0 . 0033 , ∂b ∂x f ( x ) = 0 . 0095 , which satisfies the sufficient conditions for safety in Lemma 2. However, the nominal controller yields ∂b ∂x ( f ( x ) + g ( x ) u ) = -1 . 30 and hence

fails verification. Hence, by following a safe control policy of the form (10) with the learned value of b and nominal policy µ nom , we can retain the performance of µ nom while still ensuring safety.

The computational complexity of our approach will be determined by several factors including the dimension of the state, the number of layers, the number of neurons in each layer, and the geometry of the 0 -level set of the NCBF. As shown in Table 2, the dimension of the system plays the most important role.

6 Conclusion

This paper studied the problem of verifying safety of a nonlinear control system using a NCBF represented by a feed-forward neural network with ReLU activation function. We leveraged a generalization of Nagumo's theorem for proving invariance of sets with nonsmooth boundaries to derive necessary and sufficient conditions for safety. The exact safety conditions addressed the issue that ReLU NCBFs will be nondifferentiable at certain points, thus invalidating traditional safety verification methods. Based on this condition, we proposed an algorithm for safety verification of NCBFs that first decomposes the NCBF into piecewise linear segments and then solves a nonlinear program to verify safety of each segment as well as the intersections of the linear segments. We mitigated the complexity by only considering the boundary of the safe region and pruning the segments with IBP and LiRPA. We evaluated our approach through numerical studies with comparison to stateof-the-art SMT-based methods. Future extensions of this work could include improving scalability for higher-dimensional systems and deeper neural networks, as well as other activation functions.

References

Devansh R Agrawal and Dimitra Panagou. Safe control synthesis via input constrained control barrier functions. In 2021 60th IEEE Conference on Decision and Control (CDC) , pages 6113-6118. IEEE, 2021.

Shao-Chen Hsu, Xiangru Xu, and Aaron D Ames. Control barrier function based quadratic programs with application to bipedal robotic walking. In 2015 American Control Conference (ACC) , pages 4542-4548. IEEE, 2015.

Xiangru Xu, Jessy W Grizzle, Paulo Tabuada, and Aaron D Ames. Correctness guarantees for the composition of lane keeping and adaptive cruise control. IEEE Transactions on Automation Science and Engineering , 15(3):1216-1229, 2017.

Joseph Breeden and Dimitra Panagou. High relative degree control barrier functions under input constraints. In 2021 60th IEEE Conference on Decision and Control (CDC) , pages 6119-6124. IEEE, 2021.

Hongkai Dai and Frank Permenter. Convex synthesis and verification of control-Lyapunov and barrier functions with input constraints. arXiv preprint arXiv:2210.00629 , 2022.

Shucheng Kang, Yuxiao Chen, Heng Yang, and Marco Pavone. Verification and synthesis of robust control barrier functions: Multilevel polynomial optimization and semidefinite relaxation, 2023.

Nicholas Rober, Michael Everett, Songan Zhang, and Jonathan P How. A hybrid partitioning strategy for backward reachability of neural feedback loops. In 2023 American Control Conference (ACC) , pages 3523-3528. IEEE, 2023.

Aaron D Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control barrier functions: Theory and applications. In 2019 18th European control conference (ECC) , pages 3420-3431. IEEE, 2019.

Wenceslao Shaw Cortez, Denny Oetomo, Chris Manzie, and Peter Choong. Control barrier functions for mechanical systems: Theory and application to robotic grasping. IEEE Transactions on Control Systems Technology , 29(2):530-545, 2019.

Aaron D Ames, Jessy W Grizzle, and Paulo Tabuada. Control barrier function based quadratic programs with application to adaptive cruise control. In 53rd IEEE Conference on Decision and Control , pages 6271-6278. IEEE, 2014.

Joseph Breeden and Dimitra Panagou. Robust control barrier functions under high relative degree and input constraints for satellite trajectories. arXiv preprint arXiv:2107.04094 , 2021.

Charles Dawson, Zengyi Qin, Sicun Gao, and Chuchu Fan. Safe nonlinear control using robust neural Lyapunov-barrier functions. In Conference on Robot Learning , pages 1724-1735. PMLR, 2022.

Charles Dawson, Sicun Gao, and Chuchu Fan. Safe control with learned certificates: A survey of neural Lyapunov, barrier, and contraction methods for robotics and control. IEEE Transactions on Robotics , 2023.

Simin Liu, Changliu Liu, and John Dolan. Safe control under input limits with neural control barrier functions. In Conference on Robot Learning , pages 1970-1980. PMLR, 2023.

Stephen Prajna, Ali Jadbabaie, and George J Pappas. A framework for worst-case and stochastic safety verification using barrier certificates. IEEE Transactions on Automatic Control , 52(8):1415-1428, 2007.

Wei Xiao and Calin Belta. Control barrier functions for systems with high relative degree. In 2019 IEEE 58th conference on decision and control (CDC) , pages 474-479. IEEE, 2019.

Andrew Clark. Verification and synthesis of control barrier functions. In 2021 60th IEEE Conference on Decision and Control (CDC) , pages 6105-6112. IEEE, 2021.

Weiye Zhao, Tairan He, Tianhao Wei, Simin Liu, and Changliu Liu. Safety index synthesis via sum-of-squares programming. arXiv preprint arXiv:2209.09134 , 2022.

Michael Schneeberger, Florian Dörfler, and Silvia Mastellone. SOS construction of compatible control Lyapunov and barrier functions. arXiv preprint arXiv:2305.01222 , 2023.

Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. Advances in Neural Information Processing Systems , 30, 2017.

Alessandro Abate, Daniele Ahmed, Alec Edwards, Mirco Giacobbe, and Andrea Peruffo. Fossil: A software tool for the formal synthesis of Lyapunov functions and barrier certificates using neural networks. In Proceedings of the 24th International Conference on Hybrid Systems: Computation and Control , pages 1-11, 2021.

Zengyi Qin, Kaiqing Zhang, Yuxiao Chen, Jingkai Chen, and Chuchu Fan. Learning safe multiagent control with decentralized neural barrier certificates. arXiv preprint arXiv:2101.05436 , 2021.

Zhizhen Qin, Tsui-Wei Weng, and Sicun Gao. Quantifying safety of learning-based selfdriving control using almost-barrier functions. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 12903-12910. IEEE, 2022.

Hengjun Zhao, Xia Zeng, Taolue Chen, and Zhiming Liu. Synthesizing barrier certificates using neural networks. In Proceedings of the 23rd international conference on hybrid systems: Computation and control , pages 1-11, 2020.

Alessandro Abate, Daniele Ahmed, Mirco Giacobbe, and Andrea Peruffo. Formal synthesis of lyapunov neural networks. IEEE Control Systems Letters , 5(3):773-778, 2020.

Qingye Zhao, Xin Chen, Zhuoyu Zhao, Yifan Zhang, Enyi Tang, and Xuandong Li. Verifying neural network controlled systems using neural networks. In 25th ACM International Conference on Hybrid Systems: Computation and Control , pages 1-11, 2022.

Karsten Scheibler, Stefan Kupferschmid, and Bernd Becker. Recent improvements in the SMT solver iSAT. MBMV , 13:231-241, 2013.

Weiming Xiang, Hoang-Dung Tran, and Taylor T Johnson. Output reachable set estimation and verification for multilayer neural networks. IEEE Transactions on Neural Networks and Learning Systems , 29(11):5777-5783, 2018.

Meng Sha, Xin Chen, Yuzhe Ji, Qingye Zhao, Zhengfeng Yang, Wang Lin, Enyi Tang, Qiguang Chen, and Xuandong Li. Synthesizing barrier certificates of neural network controlled continuous systems via approximations. In 2021 58th ACM/IEEE Design Automation Conference (DAC) , pages 631-636. IEEE, 2021.

Claudio Ferrari, Mark Niklas Muller, Nikola Jovanovic, and Martin Vechev. Complete verification via multi-neuron relaxation guided branch-and-bound. arXiv preprint arXiv:2205.00263 , 2022.

Patrick Henriksen and Alessio Lomuscio. Deepsplit: An efficient splitting method for neural network verification via indirect effect analysis. In IJCAI , pages 2549-2555, 2021.

Huan Zhang, Shiqi Wang, Kaidi Xu, Linyi Li, Bo Li, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. General cutting planes for bound-propagation-based neural network verification. Advances in Neural Information Processing Systems , 35:1656-1670, 2022.

Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. Reluplex: An efficient SMT solver for verifying deep neural networks. In Computer Aided Verification: 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I 30 , pages 97-117. Springer, 2017.

Guy Katz, Derek A Huang, Duligur Ibeling, Kyle Julian, Christopher Lazarus, Rachel Lim, Parth Shah, Shantanu Thakoor, Haoze Wu, Aleksandar Zelji´ c, et al. The Marabou framework for verification and analysis of deep neural networks. In Computer Aided Verification: 31st International Conference, CAV 2019, New York City, NY, USA, July 15-18, 2019, Proceedings, Part I 31 , pages 443-452. Springer, 2019.

Frederik Baymler Mathiesen, Simeon C Calvert, and Luca Laurenti. Safety certification for stochastic systems via neural barrier functions. IEEE Control Systems Letters , 7:973-978, 2022.

Rayan Mazouz, Karan Muvvala, Akash Ratheesh Babu, Luca Laurenti, and Morteza Lahijanian. Safety guarantees for neural network dynamic systems via stochastic barrier functions. Advances in Neural Information Processing Systems , 35:9672-9686, 2022.

Franco Blanchini and Stefano Miani. Set-Theoretic Methods in Control , volume 78. Springer, 2008.

Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai-Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. Automatic perturbation analysis for scalable certified robustness and beyond. Advances in Neural Information Processing Systems , 33:1129-1141, 2020.

Xia Zeng, Wang Lin, Zhengfeng Yang, Xin Chen, and Lilei Wang. Darboux-type barrier certificates for safety verification of nonlinear hybrid systems. In Proceedings of the 13th International Conference on Embedded Software , pages 1-10, 2016.

Andrew J Barry, Anirudha Majumdar, and Russ Tedrake. Safety verification of reactive controllers for uav flight in cluttered environments using barrier certificates. In 2012 IEEE International Conference on Robotics and Automation , pages 484-490. IEEE, 2012.

Lester E Dubins. On curves of minimal length with a constraint on average curvature, and with prescribed initial and terminal positions and tangents. American Journal of mathematics , 79(3):497-516, 1957.

Christopher Jewison and R Scott Erwin. A spacecraft benchmark problem for hybrid control and estimation. In 2016 IEEE 55th Conference on Decision and Control (CDC) , pages 3300-3305. IEEE, 2016.

Sicun Gao, Soonho Kong, and Edmund M Clarke. dreal: An SMT solver for nonlinear theories over the reals. In Automated Deduction-CADE-24: 24th International Conference on Automated Deduction, Lake Placid, NY, USA, June 9-14, 2013. Proceedings 24 , pages 208-214. Springer, 2013.

Leonardo De Moura and Nikolaj Bjørner. Z3: An efficient SMT solver. In Tools and Algorithms for the Construction and Analysis of Systems: 14th International Conference, TACAS 2008, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2008, Budapest, Hungary, March 29-April 6, 2008. Proceedings 14 , pages 337-340. Springer, 2008.

Stephen Prajna, Antonis Papachristodoulou, Peter Seiler, and Pablo A Parrilo. SOSTOOLS and its control applications. Positive polynomials in control , pages 273-292, 2005.

A Supplementary Material

In what follows, we give some details of content omitted in the paper due to space limit. The supplements are organized as follows. We give some proof of Lemma 1, 2, Proposition 1, Lemma 3, 4, and Theorem 2 in Section A.1 -A.6, respectively. We present system dynamics for Darboux, obstacle avoidance, spacecraft rendezvous and hi-ord 8 in Section A.9-A.12. We provide some training details in Section A.13 as well as experiment details and results in Section A.14. We compare polynomial CBFs with NCBF in A.15, compare NCBFs with different activation functions in A.16. We present more details in A.17 on the example in 3.2.

A.1 Proof of Lemma 1

We prove by induction on L . If L = 1 , then x ∈ X ( S ) if the pre-activation input to the (1 , j ) neuron is nonnegative for all j ∈ S 1 and nonpositive for all j / ∈ S 1 . We have that the pre-activation input is equal to W T 1 j x + r 1 j , establishing the result for L = 1 .

Now, inducting on L , we have that x ∈ X ( S 1 , . . . , S L -1 ) if and only if

by induction. If x ∈ X ( S 1 , . . . , S L -1 ) , then x ∈ X ( S L ) if and only if the pre-activation input to the j -th neuron at layer L is nonnegative for all j ∈ S L and nonpositive for j / ∈ S L . The pre-activation input is equal to W T Lj z L -1 + r Lj , which we can expand by induction as

completing the proof.

A.2 Proof of Lemma 2

The proof approach is based on Nagumo's Theorem, which gives necessary and sufficient conditions for positive invariance of a set. We first define the concept of tangent cone, and then present positive invariance conditions based on the tangent cone. The approach of the proof is to characterize the tangent cone to the set D = { x : b ( x ) ≥ 0 } .

Definition 2. Let A be a closed set. The tangent cone to A at x is defined by

The following result gives an approach for constructing the tangent cone.

Lemma 5 ([37]) . Suppose that the set A is defined by

for some collection of differentiable functions q 1 , . . . , q N . For any x , let J ( x ) = { k : q k ( x ) = 0 } . Then

The following is a fundamental preliminary result for establishing positive invariance.

Theorem 3 (Nagumo's Theorem [37], Section 4.2) . A closed set A is controlled positive invariant if and only if, whenever x ( t ) ∈ ∂ A , u ( t ) ∈ U satisfies

The following lemma characterizes the tangent cone to D .

Proposition 2. For any x ∈ ∂ D , we have

Proof. Define X 0 ( S ) = X ( S ) ∩ D . We will first show that, for all x with b ( x ) = 0 ,

We observe that and hence

for τ sufficiently small.

Suppose that z ∈ T X 0 ( S ) ( x ) . Then for any τ ≥ 0 , dist ( x + τz, D ) ≤ dist ( x + τz, X 0 ( S )) since X 0 ( S ) ⊆ D , and hence

We therefore have z ∈ T D ( x ) .

Now, suppose that z ∈ T D ( x ) and yet z / ∈ ⋃ S ∈ S ( x ) T X 0 ( S ) ( x ) . Then for all S ∈ S ( x ) , there exists ϵ S > 0 such that

Let ϵ = min { ϵ S : S ∈ S ( x ) } . For any δ ∈ (0 , ϵ ) , there exists τ > 0 such that τ < τ implies

implying that lim inf τ → 0 dist ( x + τz, D ) τ > 0 and hence z / ∈ T D ( x ) . This contradiction implies (22). It now suffices to show that, for each S ∈ S ( x ) ,

We have that each X 0 ( S ) is given by

thus matching the conditions of Lemma 5 when each g k function is affine. Furthermore, the set J ( x ) is equal to the set of functions that are exactly zero at x , which consists of { ( W i -1 ( S ) W ij ) T x + r ij ( S ) : ( i, j ) ∈ T ( x ) } together with W ( S ) T x + r ( S ) . This observation combined with Lemma 5 gives the desired result.

Lemma 2 is a consequence of Proposition 2. For ease of exposition, we first reproduce the lemma and then present the proof.

Lemma 6. The set D is positive invariant if and only if, for all x ∈ ∂ D , there exist S ∈ S ( x ) and u ∈ U satisfying

Proof. By Theorem 3, the set D is positive invariant if and only if for every x ∈ ∂ D , there exists u such that ( f ( x ) + g ( x ) u ) ∈ T D ( x ) . By Proposition 2, this condition holds iff there exists S ∈ S ( x ) such that

The above condition is equivalent to the conditions of the lemma, completing the proof.

A.3 Proof of Proposition 1

First, suppose that condition (i) holds. Then for any x ∈ D with S ( x ) = { S 1 , . . . , S r } , there exists l ∈ { 1 , . . . , r } and u ∈ U such that x ∈ X ( S l ) and (7)-(8) hold. For this choice of u , we have ( f ( x ) + g ( x ) u ) ∈ T Ψ i ( x ) by Proposition 2. Hence D is positive invariant under any control policy consistent with b by Lemma 2.

Next, suppose that condition (ii) holds. Since D is contained in the union of the activation sets X ( S ) , this condition implies that D ⊆ C .

A.4 Proof of Lemma 3

Suppose that condition 1 holds. Then for any x ∈ ∂ D with S ( x ) = { S 1 , . . . , S r } , there exists l ∈ { 1 , . . . , r } such that x ∈ X ( S l ) and u ∈ U satisfy (7) and (8). For this choice of u , we have ( f ( x ) + g ( x ) u ) ∈ T D ( x ) by Proposition 2. Hence D is positive invariant under any control policy consistent with b by Theorem 3.

If Condition 2 holds, then there is no x with b ( x ) = 0 and x ∈ int ( X ( S )) such that x / ∈ C . Hence, there are no counterexamples to condition (ii) of Proposition 1.

A.5 Proof of Lemma 4

The approach is to prove that condition (ii) of Proposition 1 holds; condition (i) holds automatically if each S 1 , . . . , S r satisfies condition (ii) of Lemma 3. We have that conditions (a) and (b) are equivalent

to b ( x ) = 0 and (11). In order for x to be a safety counterexample, for all l = 1 , . . . , r , at least one of Eqs. (7) and (8) must fail. Equivalently, for all l = 1 , . . . , r , there does not exist u satisfying

By Farkas Lemma, non-existence of such a u is equivalent to existence of y l satisfying y l ≥ 0 as well as (17) and (18).

A.6 Proof of Theorem 2

Suppose that x is a safety counterexample for the NCBF b with b ( x ) = 0 . If x ∈ int X ( S ) for some S , then we have that S ∈ S and hence a contradiction with Lemma 3. If x ∈ X ( S 1 ) ∩ · · · X ( S r ) for some S 1 , . . . , S r , then there is a contradiction with Lemma 4.

A.7 Details on the IBP Procedure

Interval bound propagation aims to compute an interval of possible output values by propagating a range of inputs layer-by-layer, and is integrated into our approach as follows. We first use partition the state space into cells and, for each cell, use LiRPA to derive upper and lower bounds on the value of b(x) when x takes values in that cell. When the interval of possible b(x) values in a cell contains zero, we conclude that that cell may intersect the boundary b(x) = 0. For each neuron, we use IBP to compute the pre-activation input interval for values of x within the cell. When the pre-activation input has a positive upper bound and negative lower bound, we identify the neuron as unstable, i.e., it may be either positive or negative for values of x within the cell. Using this approach, we enumerate a collection of activation sets S . We then identify the activation sets S ∈ ˜ S such that b ( x ) = 0 for some x ∈ X ( S ) by searching for an x that satisfies the linear constraints in (16). This approach uses LiRPA and IBP to identify the activation regions that intersect the boundary { x : b ( x ) = 0 } without enumerating and checking all possible activation sets, which would have exponential runtime in the number of neurons in the network.

A.8 Nonlinear Programming

The condition 2 of Lemma 3 suffices to solve the nonlinear program

and check whether the optimal value is nonnegative (unsafe) or negative (safe).

The verification problem of Lemma 4 can then be mapped to solving the nonlinear program

and checking whether the optimal value is nonnegative (safe) or negative (unsafe).

A.9 Experiment Settings: Darboux

We show the settings of NCBF verification for Darboux system whose dynamic is defined as

We define state space, initial region, and unsafe region as X : { x ∈ R 2 : x ∈ [ -2 , 2] × [ -2 , 2] } , X I : { x ∈ R 2 : 0 ≤ x 1 ≤ 1 , 1 ≤ x 2 ≤ 2 } and x U : { x ∈ R 2 : x 1 + x 2 2 ≤ 0 } respectively.

A.10 Experiment Settings: Obstacle Avoidance

We next evaluate that our proposed method on a controlled system [40]. The system state consists of 2-D position and aircraft yaw rate x := [ x 1 , x 2 , ψ ] T . We let u denote the control input to manipulate yaw rate and define the dynamics as

We define the state space, initial region and unsafe region as X , X I and X U , respectively as

A.11 Experiment Settings: Spacecraft Rendezvous

The state of the chaser is expressed relative to the target using linearized Clohessy-Wiltshire-Hill equations, with state x = [ p x , p y , p z , v x , v y , v z ] T , control input u = [ u x , u y , u z ] T and dynamics defined as follows.

We define the state space and unsafe region as X and X U , respectively as

We obtain the trained NCBF with neural CLBF training in [13] with a nominal model predictive controller.

A.12 Experiment Settings: hi-ord 8

The dynamic model of the system is captured by an ODE as follows.

where we denote the i -th derivative of variable x by x ( i ) . We define the state space and unsafe region as X and X U , respectively as

We obtain the trained NCBFs with training method proposed in [21].

A.13 Training Details

We trained the NCBFs for Darboux and obstacle avoidance via the approach proposed in [24]. Models are trained with their open source code 1 with default settings. Detailed parameters for both cases listed in Table 3a.

We then trained NCBFs for Spacecraft Rendezvous by following approach proposed in [13] with the empirical loss defined in Eq. (5) in [12]. Models are trained with their open source code 2 with default settings. The hyper-parameters are listed in Table 3b.

(a) Hyper-parameters for Darboux and OA

A.14 Experiment Details and Results

We use translators and verifiers proposed in FOSSIL 3 for SMT-based verification with solver dReal and Z3 as baselines. Our proposed enumerating algorithm utilize auto-LiRPA 4 with default settings and linear program with HiGHS solvers provided by SciPy 5 . Detailed settings can be found in our attached code.

We further visualize the trend of the number of activation sets and run-time with respect to the total number of neurons with ReLU activation function in Fig. 3. We can find that the logarithm of the activation sets size grows with the size of the neural network. The dimensionality of the state is the dominant factor in determining the run-time. The logarithm of the run-time is determined by both the state dimension and the number of ReLU hidden layers. The potential result can be cause by loose activation set estimation. Methods deriving tighter bounds than IBP may mitigate the influence of the ReLU hidden layers.

A.15 Comparison of NCBF and SOS-based Synthesis

We compare NCBF with traditional SOS synthesized polynomial CBF for the obstacle avoidance case study in two aspects, namely, training time T t and volume V of the guaranteed safe region. In order to synthesize the polynomial CBF, we adopt the procedure introduced in [45]. This procedure first constructs a nominal controller µ ( x ) , and then uses SOS programming to construct a barrier certificate for the system ˙ x ( t ) = f ( x ) + g ( x ) µ ( x ) . We choose µ ( x ) = -x 3 as the nominal controller and synthesize CBFs of degree 2, 4, 6, 8, and 10 using the Matlab SOSTOOLS toolbox. We compared the result with an NCBF with one hidden layer of 32 neurons trained using the method proposed in [24] with the same nominal controller. The experiment results are shown below. The time of SOS CBF synthesis grows with the degree of the barrier function. Degree 10 CBF takes twice the time compared to NCBF. On the other hand, NCBF outperforms all SOS synthesized CBFs by having the largest safe region volume.

(b) Hyper-parameters for Spacecraft Randezvous

(a)

(b)

LogarithmofActivationSetSizeComparison

LogarithmRun-timeComparison

5

5.00

4

4.00

3

3.00

2

2.00

1.00

1

0.00

0

32

64

96

128

32

64

96

128

-1.00

Darboux1

Darboux2

Obs1

Obs2

Darboux1

Darboux2

Obs1

Obs2

A.16 Comparison between Activation Functions

We considered three case studies, namely, Darboux, obstacle avoidance, and spacecraft rendezvous. For each case study, we trained and verified three NCBFs with the same architecture (2 hidden layers of 32 neurons each) but different activation functions, namely, ReLU, sigmoid, and tanh. We found

that, for the Darboux and obstacle avoidance case studies, the ReLU NCBF completed training faster than both sigmoid and tanh NCBFs. The volume of the safe region was comparable for all three activation functions, with the tanh outperforming the ReLU NCBF in Darboux and the ReLU NCBF providing the largest volume for obstacle avoidance. The most significant difference between the three activation functions was at the verification stage. Our proposed method for verifying ReLU NCBFs terminated within 15 and 274 seconds in the Darboux and obstacle avoidance, respectively, while SMT-based methods did not terminate within three hours for both test cases. In the spacecraft rendezvous example, the ReLU NCBF completed training before the tanh NCBF. Moreover, while our approach verified the correctness of the ReLU NCBF within 4 hours, the tanh NCBF exhibited a safety violation.

A.17 Example Details

Consider the setting of the example in Section 3.2. Let b c denote the NCBF defined in the example, which fails our defined safety conditions. For comparison, we trained an NCBF b θ and verified it using our proposed approach. We then constructed a nominal controller µ nom as a Linear Quadratic Regulator (LQR) controller that drives the system from initial point (0 , 0 . 1) to the origin. We compared the trajectories arising from the optimization-based controller defined by Eq. (10) using the b θ and b c . For the unsafe NCBF b c , the optimization-based controller is unable to satisfy the safety constraints at the boundary point (0 , 1) , resulting in a safety violation. On the other hand, while the NCBF b θ contained multiple non-differentiable points, it is possible to choose u to ensure safety at these points. For example, the point ( -0 . 19 , 2 . 91) is a non-differentiable point on the boundary b θ = 0 . There are four activation sets intersecting at this point, with corresponding values of ∂b c ∂x g ( x ) given by {-0 . 0455 , -0 . 053 , -0 . 025 , -0 . 033 } . Since any control input u with negative sign and sufficiently large magnitude will satisfy ∂b c ∂x ( f ( x ) + g ( x ) u ) ≥ 0 for all of these values, this non-differentiable point does not compromise safety of the system, and the trajectory of the system constrained by b θ remains in the safe region for all time.

Comparison of trajectory of be and bc.

E

2

SafetyBoundaryofbe

1

SafetyBoundaryofbc

0

NCBFTrajectory

CE Trajectory

SafetyBoundaryofh(x)

2

-3

2

0

2

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the exact conditions under which a learned CBF $b$ with ReLU activations yields the positive invariance property. Even though the set of inputs for which the CBF is non-differentiable has measure zero, the paper shows by example that safety can be still violated. This is because the slope of $b(x)$ can be discontinuous at the boundary $b(x) = 0$. To address this, the paper derives a theoretical result (Proposition 1) that provides a necessary and sufficient condition for a valid ReLU CBF. Based on this central result, the paper presents a verification algorithm that consists of state space discretization and search, activation set enumeration, and solving nonlinear programs. The experiments show faster safety verification for low-dimensional systems than SMT-based approaches.\n\nSTRENGTHS:\n* ReLU is a popular activation function that is often used for constructing feedforward neural networks. The paper addresses the important theoretical problem of verifying learned CBFs with ReLU activations.\n* The paper provides a novel, central theoretical result (Proposition 1) that gives necessary and sufficient conditions for a given ReLU neural CBF to be valid.\n* As shown in Section 5, the proposed verification algorithm yields more efficient run-time than SMT-based approaches.\n\nWEAKNESSES:\n* From a practical point of view, an engineer or a researcher can always choose a differentiable activation function to design a neural CBF, such as tanh or softplus. Then, the CBF remains differentiable. This leads to a motivational question of why we should use ReLU activations for modeling CBFs in the first place. Specifically, the proposed approach seems to possess several disadvantages compared to learning differentiable CBFs and verifying them. 1) The verification algorithm involves state-space discretization and activation set enumeration, both of which have inherent poor scalability with respect to the state-space dimensionality and the size of the neural network, respectively. In particular, the enumeration seems to suffer a lot as the network becomes complex, as observed in Table 2. 2) The online optimization problem for control (i.e. equation 10) requires solving possibly multiple quadratic programs, adding complexity to the standard CBF-QP.\n\n* The presentation can be improved so that mathematical descriptions are easier to follow. Especially, it is recommended that the paper 1) clearly define dimensionality of every vector and matrix-valued variables introduced in the paper, such as $W_i$, $W_{ij}$. 2) Define clearly any non-trivial variables in the main statement of mathematical results. For instance, the definition of $\\bar{W}_0$ and $\\bar{r}_0$ should be provided in the main statement of Lemma 1, not in the proofs in the appendix.\n\nQUESTIONS:\n* What are the theoretical and practical advantages of using ReLU activations over differentiable activations (as done in [1], for example) for modeling neural CBFs?\n\n* Shouldn't the last formula in the proof of Lemma 1 use $W_{Lj}$, not $W_{ij}$?\n\n[1] Dawson, Charles, Zengyi Qin, Sicun Gao, and Chuchu Fan. \"Safe nonlinear control using robust neural lyapunov-barrier functions.\" In Conference on Robot Learning, pp. 1724-1735. PMLR, 2022.\n\nLIMITATIONS:\n* The most concerning limitation is the scalability of the verification method. Even if the state space has a low-dimensionality, the usage of more layers or neurons can lead to combinatorial explosion of the runtime complexity.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nNeural control barrier functions offer wider expressive power but do not satisfy the continuously differentiable assumption with ReLU activation functions. In this paper, the authors propose a method to verify that a neural ReLU CBF is a valid CBF and is hence capable of rendering a set forward-invariant. This method derives its foundations by extending the conditions of Nagumo’s theorem. These observations provide a set of necessary and sufficient conditions for verification. Algorithmically, these conditions are verified by a combination of linear relaxation to do a coarse analysis followed by interval bound propagation to fine-tune the regions of violation. Then, two non-linear programs are used to check for the CBF conditions. The final verification engine runs faster than existing SMT methods and works correctly on three benchmarks: Darboux, Obstacle avoidance and Spacecraft.\n\nSTRENGTHS:\nThe theoretical conditions for verification and the proposed algorithm are novel, principled, creative and non-trivial. Previous works typically gloss over the points of non-differentiability in energy functions such as CLF/CBF. Hence, I believe this is an important contribution which can be extended further in future work.\n\nWEAKNESSES:\n1) The presentation needs to be improved and there are several minor mistakes which put off the reader.\nLine 243 - missing reference, Line 181 - mistake in the definition of complete collection, Line 439 -  What definition of distance is used in the definition of tangent cone?, Multiple typos in Lemma 5 in appendix\n\n2) If the set of non-differentiable points has measure zero, it will not actually affect the practical safety filtering application as the system can shoot through those points. \n\n3) For the experiments section, it would be interesting to also plot the points of non-differentiability in figure 2. The current result comparing to improperly trained CBF is rather expected. Showing the enhancement of safety verification/safety filtering by virtue of considering non-differentiable points will enhance the contribution.\n\nQUESTIONS:\n1) In proposition 1, equation (11), is the set difference explicitly needed given that the collection of sets $\\{ S_1, \\dots, S_r \\}$ is defined as complete.\n\n2) What is the meaning of equation (15) in line 186? Why can that not just be written as $\\mathcal{D} \\subset \\mathcal{C}$ ?\n\n3) For the procedure discussed in figure 1, I did not find a lot of background material in the appendix on exactly how IBP is used to enumerate activated sets and which linear program is used for pruning. I recommend a discussion on this in greater detail in the Appendix similar to Section 7.7 for the non-linear program. This is the phase that is seen to be time-consuming from Table 2 and needs to be justified.\n\nLIMITATIONS:\nThe societal impact of this work is likely positive.\nLimitations on the scalability of the approach to higher dimensional systems and deeper networks is discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the problem of synthesising control barrier functions parametrised as ReLU neural networks for non-linear deterministic dynamical systems. The authors first extend standard approaches for synthesising control barrier functions to the case where the barrier function is non-differentiable in a zero-measure set. Then, they encode the synthesis of a neural control barrier function as a nonlinear optimisation problem and illustrate the effectiveness of the approach on three benchmarks\n\nSTRENGTHS:\n-\tThe theory is sound and the extension of standard results for synthesizing control barrier functions to non-differentiable barriers is surely of interest\n-\tPaper is overall well written and the problem considered of interest\n\nWEAKNESSES:\n-\tThe main weakness is the scalability of the approach wrt the complexity of the neural barrier function. While I acknowledge that the algorithm is superior in terms of scalability than SMT based approaches, still experimental results are limited to neural networks of 1 hidden layer and 20 neurons at most \n-\tSome of the statements in the related works are not precise. In fact, the authors claim that it is not possible to use convex programming to synthesize/verify neural barrier functions. This is not accurate, in fact recent approaches rely on piecewise linear uncertain relaxations of neural networks to encode the problem of verifying neural barrier functions using linear programming [Mathiesen, Frederik Baymler, et al. \"Safety certification for stochastic systems via neural barrier functions.\" IEEE Control Systems Letters 7 (2022): 973-978.] or SDP [Mazouz, Rayan, et al. \"Safety guarantees for neural network dynamic systems via stochastic barrier functions.\" Advances in Neural Information Processing Systems 35 (2022): 9672-9686.]. Of course, the resulting approach will be more conservative compared to the one proposed by the authors, but more scalable. Also, please expand the discussion of why approaches employed to synthesize neural Lyapunov functions, e.g. [Abate, Alessandro, et al. \"Formal synthesis of Lyapunov neural networks.\" IEEE Control Systems Letters 5.3 (2020): 773-778], cannot be employed in the setting of this paper\n-\tTo demonstrate the importance of using neural control barrier functions, I believe that in the experiments there should be at least one experiment where the authors compare with the standard approaches commonly employed to synthesise control barrier functions, e.g. parametrising them as a SoS polynomial. This should serve to empirically demonstrate the advantages in being able to make use of the flexibility of neural networks in the context of barrier functions.\n\nQUESTIONS:\nPlease, see Weaknesses Section and in addition consider the following points:\n\n- Equation (7) and (8) quantifies over pairs $(i, j)$ of the set $\\mathbf{T}(x) \\cap \\mathbf{S}$. However, $\\mathbf{T}(x)$ is the set of unstable neurons produced by input $x$, hence it is unclear what pairs the quantification is referring to. Is it any two unstable neurons in the set $\\mathbf{T}(x) \\cap \\mathbf{S}$?\n\n- Set $\\bar{\\mathcal{X}}(\\mathbf{S})$ is defined both inside Lemma 1 and right after the Lemma. Please, be consistent\n\nLIMITATIONS:\nPlease, see Weaknesses Section", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors present a new strategy for verifying neural control barrier functions (NCBFs) to ensure safe control of nonlinear systems. Specifically, the authors address the challenge posed by NCBFs when ReLU activation functions are employed. This renders existing verification strategies inapplicable due to the non-differentiability of resulting NCBFs. The authors tackle this issue by introducing conditions for a ReLU NCBF to satisfy \\emph{positive invariance}, which forms the basis for constructing safe control policies. The proposed verification algorithm enumerates \\emph{activation sets} and individually verifies their positive invariance defined based on the positivity of the activation outputs. The paper demonstrates the effectiveness of the proposed approach by comparing it with Satisfiability Modulo Theory (SMT) based methods across three control problems.\n\nSTRENGTHS:\n- The paper contributes to the field by introducing a new verification algorithm and control strategies that apply to NCBFs using ReLU activation functions. Existing verification approaches are not directly applicable, and the authors address this limitation by introducing a new characterization of active sets and a corresponding verification algorithm. This approach has the potential to broaden the application domain of neural control barrier functions.\n- The paper appears to be technically sound.\n\nWEAKNESSES:\n- While the comparison with SMT-based methods is informative, it would be beneficial to include a more comprehensive performance evaluation of ReLU NCBFs against classical NCBFs with differentiable activations, such as sigmoid. Assessing factors like NCBF training time, verification time, and the expressive capabilities of the resulting NCBFs in real-world control problems would add depth to the analysis. If ReLU NCBFs consistently outperform sigmoid NCBFs, the proposed verification strategy's utility would be significantly enhanced.\n- The authors could provide a thorough analysis of the time complexity of the proposed verification strategy. As the algorithm discretizes the state space into hyper-cubes and calculates corresponding bounds of $b(x)$ individually, it is important to understand how the complexity scales in high-dimensional problems. Also, the authors should provide experiments on more complex, potentially higher-dimensional control problems.\n- (Minor) the manuscript contains several typos that require careful proofreading.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nThe authors should consider the possibility of clearly outlining the limitations of their study and potential negative societal impacts.\n\nAfter the rebuttal phase (the same as the post-rebuttal comments):\nI thank the authors for their response and conducting the additional experiments. The majority of the concerns raised in my initial evaluation have been effectively attended to. While I acknowledge the efforts undertaken by the authors to conduct an additional round of experiments, I still have reservations about how well the proposed algorithm would work for problems with high dimensions. The additional experiments provided are still confined to low-dimensional scenarios. I recommend that the authors explore this aspect in their future research endeavors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a technique to verify the safety of NCBF-based control policies. Usually, the verification depends on the CBF to be continuously differentiable. This is not the case for NNs, however using NCBFs is beneficial as it allows to encode more complex safety constraints.\nTheir approach identifies the piecewise linear segments of the NCBF. The number of segments is reduced by only focusing on those at the boundary of the safe region. The remaining linear segments are overapproximized and verified using nonlinear programs.\nThe authors compare their proposed technique with SOTA SMT based methods and demonstrate that they are able to verify NCBFs that previously resulted in a timeout.\n\nSTRENGTHS:\nThe paper propose a new original technique to prove the safety of NCBFs.\nThe authors do a good job of motivating their new approach by demonstrating the shortcoming of techniques that expect $b$ to be continuously differentiable.\nThe paper is mostly clear in its explanation and formulas, and gives intuitive explanations for many of them.\nIn their experimental evaluation, they use one benchmark to compare their performance against two other SOTA techniques, and demonstrate that they are able to verify instances that would otherwise lead to a timeout. This indicates the significance of their proposed technique, should the results carry over to other benchmarks.\n\nWEAKNESSES:\nThe equations in the paper get increasingly complicated to follow, even with the provided explanations. Especially Lemma 4 is hard to follow. \n\nIn the experimental evaluation, the paper would strongly benefit from exploring more benchmarks. The comparison to SOTA techniques dReal and Z3 is limited to one benchmark, with two more benchmarks that do not include a comparison to those tools. Also, a comparison to an approach based on neural-network-verification (e.g. \"A Hybrid Partitioning Strategy for Backward Reachability of Neural Feedback Loops\" by Nicholas Rober, Michael Everett, Songan Zhang, and Jonathan P. How) would help to demonstrate that the proposed technique can solve previously hard problems.\n\nQUESTIONS:\nMain question: Why can't one use regular verification of safety properties of neural networks? If $D$ is convex (or can be split into a reasonable number of convex subsets), then one could verify that no input in $D$ is mapped to an output outside of $D$. When $f$, $g$ and $\\\\mu$ in Equation 1 are known, that equation could most probably be encoded as a NNs, so all common tools (compare e.g. VNN-COMP 2022 or 2023) should be able to verify this (or time out). You state the main benefit of your approach is that it does not depend on the specific choice of $\\\\mu$ (line 288). However, I do not know if this is often a requirement. If $\\\\mu$ changes, the verification using a technique that depends on $\\\\mu$ could be repeated. Did you do an experimental comparison of your approach with a NN-Verification-Tool-based approach to see how costly this would be?\n\nOther questions:\n\n1) In the text above Proposition 1, is $\\\\overline{X}(S_1) \\\\cap \\\\ldots \\\\cap \\\\overline{X}(S_r) \\\\cap S'$ well-defined? $\\\\overline{X}(S)$ is a set of inputs $x$, but $S'$ is a set of neurons. Should this be $\\\\overline{X}(S')$?\n\n2) In Equation 11, is the term to the right of $ \\\\textbackslash $ obsolete? If $\\\\{S_1, \\ldots, S_r\\\\}$ is complete, then no input that activates all of $\\\\{S_1, \\ldots, S_r\\\\}$ also activates any other $S'$ (based on the text above Proposition 1). So what is removed by the term right of $ \\\\textbackslash $ ?\n\n3) What's the significance of the three different lines in Figure 2? They represent different \"set boundaries\", bu I do not know what that visualization achieves compared to one with just set boundary 0\n\nMinor: What is the missing reference in line 243?\n\nMinor note: Line 268: \"with in\" -> \"within\"\n\nLIMITATIONS:\nThe authors do not provide a list of limitations or potential negative societal impact. However, the potential negative societal impact is probably small, as this technique is developed to increase the provable safety of NCBFs. So the potential negative societal impact is identical to that of any AI-based technology.\nThe paper would benefit from a description of the limitations of their proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-30"}