Infinite Time Horizon Safety of Bayesian Neural Networks

1 Introduction . The success of deep neural networks ( DNNs ) across different domains has created the desire to apply them in safety-critical applications such as autonomous vehicles [ 28 , 30 ] and healthcare systems [ 41 ] . The fundamental challenge for the deployment of DNNs in these domains is certifying their safety [ 3 ] . Thus , formal safety verification of DNNs in isolation and closed control loops [ 26 , 23 , 17 , 43 , 13 ] has become an active research topic . Bayesian neural networks ( BNNs ) are a family of neural networks that place distributions over their weights [ 36 ] . This allows learning uncertainty in the data and the network ’ s prediction , while preserving the strong modelling capabilities of DNNs [ 32 ] . In particular , BNNs can learn arbitrary data distributions from much simpler ( e.g . Gaussian ) weight distributions . This makes BNNs very appealing for robotic and medical applications [ 33 ] where uncertainty is a central component of data . Despite the large body of literature on verifying safety of DNNs , the formal safety verification of BNNs has received less attention . Notably , [ 8 , 45 , 34 ] have proposed sampling-based techniques for obtaining probabilistic guarantees about BNNs . Although these approaches provide some insight ∗Equal Contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . into BNN safety , they suffer from two key limitations . First , sampling provides only bounds on the probability of the BNN ’ s safety which is insufficient for systems with critical safety implications . For instance , having an autonomous vehicle with a 99.9 % safety guarantee is still insufficient for deployment if millions of vehicles are deployed . Second , samples can only simulate the system for a finite time , making it impossible to reason about the system ’ s safety over an unbounded time horizon . In this work , we study the safety verification problem for BNN policies in safety-critical systems over the infinite time horizon . Formally , we consider discrete-time closed-loop systems defined by a dynamical system and a BNN policy . Given a set of initial states and a set of unsafe ( or bad ) states , the goal of the safety verification problem is to verify that no system execution starting in an initial state can reach an unsafe state . Unlike existing literature which considers probability of safety , we verify sure safety , i.e . safety of every system execution of the system . In particular , we present a method for computing safe weight sets for which every system execution is safe as long as the BNN samples its weights from this set . Our approach to restrict the support of the weight distribution is necessary as BNNs with Gaussian weight priors typically produce output posteriors with unbounded support . Conse- quently , there is a low but non-zero probability for the output variable to lie in an unsafe region , see Figure 1 . This implies that BNNs are usually unsafe by default . We therefore consider the more general problem of computing safe weight sets . Verifying that a weight set is safe allows re-calibrating the BNN policy by rejecting unsafe weight samples in order to guarantee safety . As most BNNs employ uni-modal weight priors , e.g . Gaussians , we naturally adopt weight sets in the form of products of intervals centered at the means of the BNN ’ s weight distributions . To verify safety of a weight set , we search for a safety certificate in the form of a safe positive invariant ( also known as safe inductive invariant ) . A safe positive invariant is a set of system states that contains all initial states , is closed under the system dynamics and does not contain any unsafe state . The key advantage of using safe positive invariants is that their existence implies the infinite time horizon safety . We parametrize safe positive invariant candidates by ( deterministic ) neural networks that classify system states for determining set inclusion . Moreover , we phrase the search for an invariant as a learning problem . A separated verifier module then checks if a candidate is indeed a safe positive invariant by checking the required properties via constraint solving . In case the verifier finds a counterexample demonstrating that the candidate violates the safe positive invariant condition , we re-train the candidate on the found counterexample . We repeat this procedure until the verifier concludes that the candidate is a safe positive invariant ensuring that the system is safe . The safe weight set obtained by our method can be used for safe exploration reinforcement learning . In particular , generating rollouts during learning by sampling from the safe weight set allows an exploration of the environment while ensuring safety . Moreover , projecting the ( mean ) weights onto the safe weight set after each gradient update further ensures that the improved policy stays safe . Contributions Our contributions can be summarized as follows : 1 . We define a safety verification problem for BNN policies which overcomes the unbounded posterior issue by computing and verifying safe weight sets . The problem generalizes the sure safety verification of BNNs and solving it allows re-calibrating BNN policies via rejection sampling to guarantee safety . 2 . We introduce a method for computing safe weight sets in BNN policies in the form of products of intervals around the BNN weights ’ means . To verify safety of a weight set , our novel algorithm learns a safe positive invariant in the form of a deterministic neural network . 3 . We evaluate our methodology on a series of benchmark applications , including non-linear systems and non-Lyapunovian safety specifications . 2 Related work . Verification of feed-forward NN Verification of robustness and safety properties in feed-forward DNNs has received much attention but remains an active research topic [ 26 , 23 , 17 , 40 , 7 , 43 ] . As the majority of verification techniques were designed for deterministic NNs , they can not be readily applied to BNNs . The safety verification of feed-forward BNNs has been considered in [ 8 ] by using samples to obtain statistical guarantees on the safety probability . The work of [ 45 ] also presents a sampling-based approach , however it provides certified lower bounds on the safety probability . The literature discussed above considers NNs in isolation , which can provide input-output guarantees on a NN but are unable to reason holistically about the safety of the system that the NN is applied in . Verification methods that concern the safety of NNs interlinked with a system require different approaches than standalone NN verification , which we will discuss in the rest of this section . Finite time horizon safety of BNN policies The work in [ 34 ] extends the method of [ 8 ] to verifying safety in closed-loop systems with BNN policies . However , similar to the standalone setting of [ 8 ] , their method obtains only statistical guarantees on the safety probability and for the system ’ s execution over a finite time horizon . Safe RL Safe reinforcement learning has been primarily studied in the form of constrained Markov decision processes ( CMDPs ) [ 2 , 18 ] . Compared to standard MDPs , an agent acting in a CMDP must satisfy an expected auxiliary cost term aggregated over an episode . The CMDP framework has been the base of several RL algorithms [ 44 ] , notably the Constrained Policy Optimization ( CPO ) [ 1 ] . Despite these algorithms providing a decent performance , the key limitation of CMDPs is that the constraint is satisfied in expectation , which makes violations unlikely but nonetheless possible . Consequently , the CMDP framework is unsuited for systems where constraint violations are critical . Lyapunov-based stability Safety in the context of ” stability ” , i.e . always returning to a ground state , can be proved by Lyapunov functions [ 4 ] . Lyapunov functions have originally been considered to study stability of dynamical systems [ 29 ] . Intuitively , a Lyapunov function assigns a non-negative value to each state , and is required to decrease with respect to the system ’ s dynamics at any state outside of the stable set . A Lyapunov-based method is proposed in [ 11 ] to ensure safety in CMDPs during training . Recently , the work of [ 9 ] presented a method for learning a policy as well as a neural network Lyapunov function which guarantees the stability of the policy . Similarly to our work , their learning procedure is counterexample-based . However , unlike [ 9 ] , our work considers BNN policies and safety definitions that do not require returning to a set of ground states . Barrier functions for dynamical systems Barrier functions can be used to prove infinite time horizon safety in dynamical systems [ 38 , 39 ] . Recent works have considered learning neural network barrier functions [ 47 ] , and a counterexample-based learning procedure is presented in [ 37 ] . Finite time horizon safety of NN policies Safety verification of continuous-time closed-loop systems with deterministic NN policies has been considered in [ 24 , 19 ] , which reduces safety verification to the reachability analysis in hybrid systems [ 10 ] . The work of [ 13 ] presents a method which computes a polynomial approximation of the NN policy to allow an efficient approximation of the reachable state set . Both works consider finite time horizon systems . Our safety certificate most closely resembles inductive invariants for safety analysis in programs [ 14 ] and positive invariants for dynamical systems [ 5 ] . 3 Preliminaries and problem statement . We consider a discrete-time dynamical system xt+1 = f ( xt , ut ) , x0 ∈ X0 . The dynamics are defined by the function f : X × U → X where X ⊆ Rm is the state space and U ⊆ Rn is the control action space , X0 ⊆ X is the set of initial states and t ∈ N≥0 denotes a discretized time . At each time step t , the action is defined by the ( possibly probabilistic ) positional policy π : X → D ( U ) , which maps the current state xt to a distribution π ( xt ) ∈ D ( U ) over the set of actions . We use D ( U ) to denote the set of all probability distributions over U . The next action is then sampled according to ut ∼ π ( xt ) , and together with the current state xt of the system gives rise to the next state xt+1 of the system according to the dynamics f . Thus , the dynamics f together with the policy π form a closed-loop system ( or a feedback loop system ) . The aim of the policy is to maximize the expected cumulative reward ( possibly discounted ) from each starting state . Given a set of initial states X0 of the system , we say that a sequence of state-action pairs ( xt , ut ) ∞t=0 is a trajectory if x0 ∈ X0 and we have ut ∈ supp ( π ( xt ) ) and xt+1 = f ( xt , ut ) for each t ∈ N≥0 . A neural network ( NN ) is a function π : Rm → Rn that consists of several sequentially composed layers π = l1 ◦ · · · ◦ lk . Formally , a NN policy maps each system state to a Dirac-delta distribution which picks a single action with probability 1 . Each layer li is parametrized by learned weight values of the appropriate dimensions and an activation function a , li ( x ) = a ( Wix + bi ) , Wi ∈ Rni×mi , bi ∈ Rni . In this work , we consider ReLU activation functions a ( x ) = ReLU ( x ) = max { x,0 } , although other piecewise linear activation such as the leaky-ReLU [ 25 ] and PReLU [ 21 ] are applicable as well . In Bayesian neural networks ( BNNs ) , weights are random variables and their values are sampled , each according to some distribution . Then each vector of sampled weights gives rise to a ( deterministic ) neural network . Given a training set D , in order to train the BNN we assume a prior distribution p ( w , b ) over the weights . The learning then amounts to computing the posterior distribution p ( w , b | D ) via the application of the Bayes rule . As analytical inference of the posterior is in general infeasible due to non-linearity introduced by the BNN architecture [ 32 ] , practical training algorithms rely on approximate inference , e.g . Hamiltonian Monte Carlo [ 36 ] , variational inference [ 6 ] or dropout [ 15 ] . When the policy in a dynamical system is a BNN , the policy maps each system state xt to a probability distribution π ( xt ) over the action space . Informally , this distribution is defined as follows . First , BNN weights w , b are sampled according to the posterior BNN weight distribution , and the sampled weights give rise to a deterministic NN policy πw , b . The action of the system is then defined as ut = πw , b ( xt ) . Formal definition of the distribution π ( xt ) is straightforward and proceeds by considering the product measure of distributions of all weights . Problem statement We now define the two safety problems that we consider in this work . The first problem considers feed-forward BNNs , and the second problem considers closed-loop systems with BNN policies . While our solution to the first problem will be a subprocedure in our solution to the second problem , the reason why we state it as a separate problem is that we believe that our solution to the first problem is also of independent interest for the safety analysis of feed-forward BNNs . Let π be a BNN . Suppose that the vector ( w , b ) of BNN weights in π has dimension p+ q , where p is the dimension of w and q is the dimension of b . For each 1 ≤ i ≤ p , let µi denote the mean of the random variable wi . Similarly , for each 1 ≤ i ≤ q , let µp+i denote the mean of the random variable bi . Then , for each ∈ [ 0 , ∞ ] , we define the set Wπ of weight vectors via Wπ = p+q∏ i=1 [ µi − , µi + ] ⊆ Rp+q . We now proceed to defining our safety problem for feed-forward BNNs . Suppose that we are given a feed-forward BNN π , a set X0 ⊆ Rm of input points and a set Xu ⊆ Rn of unsafe ( or bad ) output points . For a concrete vector ( w , b ) of weight values , let πw , b to be the ( deterministic ) NN defined by these weight values . We say that πw , b is safe if for each x ∈ X0 we have πw , b ( x ) 6∈ Xu , i.e . if evaluating πw , b on all input points does not lead to an unsafe output . Problem 1 ( Feed-forward BNNs ) . Let π be a feed-forward BNN , X0 ⊆ Rm a set of input points and Xu ⊆ Rn a set of unsafe output points . Let ∈ [ 0 , ∞ ] . Determine whether each deterministic NN in { πw , b | ( w , b ) ∈Wπ } is safe . Next , we define our safety problem for closed-loop systems with BNN policies . Consider a closedloop system defined by a dynamics function f , a BNN policy π and an initial set of states X0 . Let Xu ⊆ X be a set of unsafe ( or bad ) states . We say that a trajectory ( xt , ut ) ∞t=0 is safe if xt 6∈ Xu for all t ∈ N0 , hence if it does not reach any unsafe states . Note that this definition implies infinite time horizon safety of the trajectory . Given ∈ [ 0 , ∞ ] , define the set Trajf , π to be the set of all system trajectories in which each sampled weight vector belongs to Wπ . Problem 2 ( Closed-loop systems with BNN policies ) . Consider a closed-loop system defined by a dynamics function f , a BNN policy π and a set of initial states X0 . Let Xu be a set of unsafe states . Let ∈ [ 0 , ∞ ] . Determine whether each trajectory in Trajf , π is safe . Note that the question of whether the BNN policy π is safe ( i.e . whether each trajectory of the system is safe ) is a special case of the above problem which corresponds to =∞ .

Summary: The paper considers the problem of safely controlling a known, discrete-time system with a stochastic BNN policy with ReLU activations where randomness enters through the policy's weights. Safety is defined in terms of avoiding a set of states that are known to be safe over an infinite time horizon. The paper first considers the problem of determining whether BNN outputs lie in a set of unsafe actions. To this end, the paper proposes to approximate weights by bounding boxes in a set W_epsilon and verifies safety through a corresponding mixed-interger linear program. While the BNN's distribution typically has unbounded support, for any verified epsilon it can be made safe through rejections sampling.  The developed methodology is then used for the safe control problem, where a neural network is used together with a classification problem in order to learn an invariant set under the policy. The algorithm proceeds iteratively by fitting the neural network and then finding counter-examples through a mixed-integer program that are then added to the training set. The result is a trained classifier for safe/unsafe states for a given policy and value epsilon.  The approach is evaluated on two linear and one piecewise-linear dynamic systems with action dimension one and state dimensions between one and three, showing that for these problems the proposed method can verify safety within a reasonable runtime.