Infinite Time Horizon Safety of Bayesian Neural Networks

1 Introduction . The success of deep neural networks ( DNNs ) across different domains has created the desire to apply them in safety-critical applications such as autonomous vehicles [ 28 , 30 ] and healthcare systems [ 41 ] . The fundamental challenge for the deployment of DNNs in these domains is certifying their safety [ 3 ] . Thus , formal safety verification of DNNs in isolation and closed control loops [ 26 , 23 , 17 , 43 , 13 ] has become an active research topic . Bayesian neural networks ( BNNs ) are a family of neural networks that place distributions over their weights [ 36 ] . This allows learning uncertainty in the data and the network ’ s prediction , while preserving the strong modelling capabilities of DNNs [ 32 ] . In particular , BNNs can learn arbitrary data distributions from much simpler ( e.g . Gaussian ) weight distributions . This makes BNNs very appealing for robotic and medical applications [ 33 ] where uncertainty is a central component of data . Despite the large body of literature on verifying safety of DNNs , the formal safety verification of BNNs has received less attention . Notably , [ 8 , 45 , 34 ] have proposed sampling-based techniques for obtaining probabilistic guarantees about BNNs . Although these approaches provide some insight ∗Equal Contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . into BNN safety , they suffer from two key limitations . First , sampling provides only bounds on the probability of the BNN ’ s safety which is insufficient for systems with critical safety implications . For instance , having an autonomous vehicle with a 99.9 % safety guarantee is still insufficient for deployment if millions of vehicles are deployed . Second , samples can only simulate the system for a finite time , making it impossible to reason about the system ’ s safety over an unbounded time horizon . In this work , we study the safety verification problem for BNN policies in safety-critical systems over the infinite time horizon . Formally , we consider discrete-time closed-loop systems defined by a dynamical system and a BNN policy . Given a set of initial states and a set of unsafe ( or bad ) states , the goal of the safety verification problem is to verify that no system execution starting in an initial state can reach an unsafe state . Unlike existing literature which considers probability of safety , we verify sure safety , i.e . safety of every system execution of the system . In particular , we present a method for computing safe weight sets for which every system execution is safe as long as the BNN samples its weights from this set . Our approach to restrict the support of the weight distribution is necessary as BNNs with Gaussian weight priors typically produce output posteriors with unbounded support . Conse- quently , there is a low but non-zero probability for the output variable to lie in an unsafe region , see Figure 1 . This implies that BNNs are usually unsafe by default . We therefore consider the more general problem of computing safe weight sets . Verifying that a weight set is safe allows re-calibrating the BNN policy by rejecting unsafe weight samples in order to guarantee safety . As most BNNs employ uni-modal weight priors , e.g . Gaussians , we naturally adopt weight sets in the form of products of intervals centered at the means of the BNN ’ s weight distributions . To verify safety of a weight set , we search for a safety certificate in the form of a safe positive invariant ( also known as safe inductive invariant ) . A safe positive invariant is a set of system states that contains all initial states , is closed under the system dynamics and does not contain any unsafe state . The key advantage of using safe positive invariants is that their existence implies the infinite time horizon safety . We parametrize safe positive invariant candidates by ( deterministic ) neural networks that classify system states for determining set inclusion . Moreover , we phrase the search for an invariant as a learning problem . A separated verifier module then checks if a candidate is indeed a safe positive invariant by checking the required properties via constraint solving . In case the verifier finds a counterexample demonstrating that the candidate violates the safe positive invariant condition , we re-train the candidate on the found counterexample . We repeat this procedure until the verifier concludes that the candidate is a safe positive invariant ensuring that the system is safe . The safe weight set obtained by our method can be used for safe exploration reinforcement learning . In particular , generating rollouts during learning by sampling from the safe weight set allows an exploration of the environment while ensuring safety . Moreover , projecting the ( mean ) weights onto the safe weight set after each gradient update further ensures that the improved policy stays safe . Contributions Our contributions can be summarized as follows : 1 . We define a safety verification problem for BNN policies which overcomes the unbounded posterior issue by computing and verifying safe weight sets . The problem generalizes the sure safety verification of BNNs and solving it allows re-calibrating BNN policies via rejection sampling to guarantee safety . 2 . We introduce a method for computing safe weight sets in BNN policies in the form of products of intervals around the BNN weights ’ means . To verify safety of a weight set , our novel algorithm learns a safe positive invariant in the form of a deterministic neural network . 3 . We evaluate our methodology on a series of benchmark applications , including non-linear systems and non-Lyapunovian safety specifications . 2 Related work . Verification of feed-forward NN Verification of robustness and safety properties in feed-forward DNNs has received much attention but remains an active research topic [ 26 , 23 , 17 , 40 , 7 , 43 ] . As the majority of verification techniques were designed for deterministic NNs , they can not be readily applied to BNNs . The safety verification of feed-forward BNNs has been considered in [ 8 ] by using samples to obtain statistical guarantees on the safety probability . The work of [ 45 ] also presents a sampling-based approach , however it provides certified lower bounds on the safety probability . The literature discussed above considers NNs in isolation , which can provide input-output guarantees on a NN but are unable to reason holistically about the safety of the system that the NN is applied in . Verification methods that concern the safety of NNs interlinked with a system require different approaches than standalone NN verification , which we will discuss in the rest of this section . Finite time horizon safety of BNN policies The work in [ 34 ] extends the method of [ 8 ] to verifying safety in closed-loop systems with BNN policies . However , similar to the standalone setting of [ 8 ] , their method obtains only statistical guarantees on the safety probability and for the system ’ s execution over a finite time horizon . Safe RL Safe reinforcement learning has been primarily studied in the form of constrained Markov decision processes ( CMDPs ) [ 2 , 18 ] . Compared to standard MDPs , an agent acting in a CMDP must satisfy an expected auxiliary cost term aggregated over an episode . The CMDP framework has been the base of several RL algorithms [ 44 ] , notably the Constrained Policy Optimization ( CPO ) [ 1 ] . Despite these algorithms providing a decent performance , the key limitation of CMDPs is that the constraint is satisfied in expectation , which makes violations unlikely but nonetheless possible . Consequently , the CMDP framework is unsuited for systems where constraint violations are critical . Lyapunov-based stability Safety in the context of ” stability ” , i.e . always returning to a ground state , can be proved by Lyapunov functions [ 4 ] . Lyapunov functions have originally been considered to study stability of dynamical systems [ 29 ] . Intuitively , a Lyapunov function assigns a non-negative value to each state , and is required to decrease with respect to the system ’ s dynamics at any state outside of the stable set . A Lyapunov-based method is proposed in [ 11 ] to ensure safety in CMDPs during training . Recently , the work of [ 9 ] presented a method for learning a policy as well as a neural network Lyapunov function which guarantees the stability of the policy . Similarly to our work , their learning procedure is counterexample-based . However , unlike [ 9 ] , our work considers BNN policies and safety definitions that do not require returning to a set of ground states . Barrier functions for dynamical systems Barrier functions can be used to prove infinite time horizon safety in dynamical systems [ 38 , 39 ] . Recent works have considered learning neural network barrier functions [ 47 ] , and a counterexample-based learning procedure is presented in [ 37 ] . Finite time horizon safety of NN policies Safety verification of continuous-time closed-loop systems with deterministic NN policies has been considered in [ 24 , 19 ] , which reduces safety verification to the reachability analysis in hybrid systems [ 10 ] . The work of [ 13 ] presents a method which computes a polynomial approximation of the NN policy to allow an efficient approximation of the reachable state set . Both works consider finite time horizon systems . Our safety certificate most closely resembles inductive invariants for safety analysis in programs [ 14 ] and positive invariants for dynamical systems [ 5 ] . 3 Preliminaries and problem statement . We consider a discrete-time dynamical system xt+1 = f ( xt , ut ) , x0 ∈ X0 . The dynamics are defined by the function f : X × U → X where X ⊆ Rm is the state space and U ⊆ Rn is the control action space , X0 ⊆ X is the set of initial states and t ∈ N≥0 denotes a discretized time . At each time step t , the action is defined by the ( possibly probabilistic ) positional policy π : X → D ( U ) , which maps the current state xt to a distribution π ( xt ) ∈ D ( U ) over the set of actions . We use D ( U ) to denote the set of all probability distributions over U . The next action is then sampled according to ut ∼ π ( xt ) , and together with the current state xt of the system gives rise to the next state xt+1 of the system according to the dynamics f . Thus , the dynamics f together with the policy π form a closed-loop system ( or a feedback loop system ) . The aim of the policy is to maximize the expected cumulative reward ( possibly discounted ) from each starting state . Given a set of initial states X0 of the system , we say that a sequence of state-action pairs ( xt , ut ) ∞t=0 is a trajectory if x0 ∈ X0 and we have ut ∈ supp ( π ( xt ) ) and xt+1 = f ( xt , ut ) for each t ∈ N≥0 . A neural network ( NN ) is a function π : Rm → Rn that consists of several sequentially composed layers π = l1 ◦ · · · ◦ lk . Formally , a NN policy maps each system state to a Dirac-delta distribution which picks a single action with probability 1 . Each layer li is parametrized by learned weight values of the appropriate dimensions and an activation function a , li ( x ) = a ( Wix + bi ) , Wi ∈ Rni×mi , bi ∈ Rni . In this work , we consider ReLU activation functions a ( x ) = ReLU ( x ) = max { x,0 } , although other piecewise linear activation such as the leaky-ReLU [ 25 ] and PReLU [ 21 ] are applicable as well . In Bayesian neural networks ( BNNs ) , weights are random variables and their values are sampled , each according to some distribution . Then each vector of sampled weights gives rise to a ( deterministic ) neural network . Given a training set D , in order to train the BNN we assume a prior distribution p ( w , b ) over the weights . The learning then amounts to computing the posterior distribution p ( w , b | D ) via the application of the Bayes rule . As analytical inference of the posterior is in general infeasible due to non-linearity introduced by the BNN architecture [ 32 ] , practical training algorithms rely on approximate inference , e.g . Hamiltonian Monte Carlo [ 36 ] , variational inference [ 6 ] or dropout [ 15 ] . When the policy in a dynamical system is a BNN , the policy maps each system state xt to a probability distribution π ( xt ) over the action space . Informally , this distribution is defined as follows . First , BNN weights w , b are sampled according to the posterior BNN weight distribution , and the sampled weights give rise to a deterministic NN policy πw , b . The action of the system is then defined as ut = πw , b ( xt ) . Formal definition of the distribution π ( xt ) is straightforward and proceeds by considering the product measure of distributions of all weights . Problem statement We now define the two safety problems that we consider in this work . The first problem considers feed-forward BNNs , and the second problem considers closed-loop systems with BNN policies . While our solution to the first problem will be a subprocedure in our solution to the second problem , the reason why we state it as a separate problem is that we believe that our solution to the first problem is also of independent interest for the safety analysis of feed-forward BNNs . Let π be a BNN . Suppose that the vector ( w , b ) of BNN weights in π has dimension p+ q , where p is the dimension of w and q is the dimension of b . For each 1 ≤ i ≤ p , let µi denote the mean of the random variable wi . Similarly , for each 1 ≤ i ≤ q , let µp+i denote the mean of the random variable bi . Then , for each ∈ [ 0 , ∞ ] , we define the set Wπ of weight vectors via Wπ = p+q∏ i=1 [ µi − , µi + ] ⊆ Rp+q . We now proceed to defining our safety problem for feed-forward BNNs . Suppose that we are given a feed-forward BNN π , a set X0 ⊆ Rm of input points and a set Xu ⊆ Rn of unsafe ( or bad ) output points . For a concrete vector ( w , b ) of weight values , let πw , b to be the ( deterministic ) NN defined by these weight values . We say that πw , b is safe if for each x ∈ X0 we have πw , b ( x ) 6∈ Xu , i.e . if evaluating πw , b on all input points does not lead to an unsafe output . Problem 1 ( Feed-forward BNNs ) . Let π be a feed-forward BNN , X0 ⊆ Rm a set of input points and Xu ⊆ Rn a set of unsafe output points . Let ∈ [ 0 , ∞ ] . Determine whether each deterministic NN in { πw , b | ( w , b ) ∈Wπ } is safe . Next , we define our safety problem for closed-loop systems with BNN policies . Consider a closedloop system defined by a dynamics function f , a BNN policy π and an initial set of states X0 . Let Xu ⊆ X be a set of unsafe ( or bad ) states . We say that a trajectory ( xt , ut ) ∞t=0 is safe if xt 6∈ Xu for all t ∈ N0 , hence if it does not reach any unsafe states . Note that this definition implies infinite time horizon safety of the trajectory . Given ∈ [ 0 , ∞ ] , define the set Trajf , π to be the set of all system trajectories in which each sampled weight vector belongs to Wπ . Problem 2 ( Closed-loop systems with BNN policies ) . Consider a closed-loop system defined by a dynamics function f , a BNN policy π and a set of initial states X0 . Let Xu be a set of unsafe states . Let ∈ [ 0 , ∞ ] . Determine whether each trajectory in Trajf , π is safe . Note that the question of whether the BNN policy π is safe ( i.e . whether each trajectory of the system is safe ) is a special case of the above problem which corresponds to =∞ .

Summary: In this paper the authors consider the problem of verifying safety in BNNs.  At a high level, they seek a way to compute a set of safe weight sets of a BNN such that its outputs can not fall within some “unsafe set.” They show doing so for two contexts, the first aiding the second. In the first context they take a standard feed forward BNN, while in the second they consider a closed loop system under some dynamics where the policy at each time point is implemented by a BNN.