Make Sure You're Unsure: A Framework for Verifying Probabilistic Specifications

1 Introduction . While neural networks ( NNs ) have shown significant promise in a wide-range of applications ( for e.g. , [ He et al. , 2016 , Yu and Deng , 2014 ] ) , a key-bottleneck towards their wide-spread adoption in safety-critical applications is the lack of formal guarantees regarding safety and performance . In this direction , there has been considerable progress towards developing scalable methods that can provide formal guarantees regarding the conformance of NNs with desired properties [ Katz et al. , 2017 , Dvijotham et al. , 2018b , Raghunathan et al. , 2018 ] . However , much of this progress has been in the setting where the specifications and neural networks do not exhibit any probabilistic behaviour , or is mostly specialized for specific probabilistic specifications [ Weng et al. , 2019 , Wicker et al. , 2020 ] . ∗Equal contribution . Authors listed in alphabetical order . Correspondance to lberrada @ deepmind.com , sdathath @ deepmind.com , dvij @ cs.washington.edu . †DeepMind , London , United Kingdom . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In contrast , we introduce a general framework for verifying specifications of neural networks that are probabilistic . The framework enables us to handle stochastic neural networks such as Bayesian Neural Networks or Monte-Carlo ( MC ) dropout networks , as well as probabilistic properties , such as distributionally robust out-of-distribution ( OOD ) detection . Furthermore , the specification can be defined on the output distribution from the network , which allows us to handle operations such as the expectation on functions of the neural network output . Probabilistic specifications are relevant and natural to many practical problems . For instance , for robotics applications , there is uncertainty arising from noisy measurements from sensors , and uncertainty regarding the actions of uncontrolled agents ( e.g . uncertainty regarding the behaviour of pedestrians for a self-driving vehicle ) . Often these uncertainties are modelled using a probabilistic approach , where a distribution is specified ( or possibly learnt ) over the feasible set of events [ Thrun et al. , 2005 ] . In such cases , we want to provide guarantees regarding the network ’ s conformance to desired properties in the distributional setting ( e.g . given a model of the pedestrian ’ s uncertain behaviour , guarantee that the probability of collision for the autonomous vehicle is small ) . A more general problem includes scenarios where there is uncertainty regarding the parameters of the distribution used to model uncertainty . Here , in this general setting , we seek to verify the property that the network behaviour conforms with the desired specification under uncertainty corresponding to an entire set of distributions . The key to handling the aforementioned complexity in the specifications being verified through our framework is the generalization of the Lagrangian duality . Specifically , instead of using the standard Lagrange duality where the multipliers are linear , we allow for probabilistic constraints ( constraints between distributions ) and use functional multipliers to replace the linear Lagrange multipliers . This allows us to exploit the structure of these probabilistic constraints , enabling us to provide stronger guarantees and facilitates the verification of verification-agnostic networks ( networks that are not designed to be verifiable ) . In our paper , we focus on verification-agnostic networks as this is desirable for many reasons , as noted in Dathathri et al . [ 2020 ] . To summarize , our main contributions are : • We derive a general framework that extends Lagrangian duality to handle a wide range of probabilistic specifications . Our main theoretical result ( Theorem 1 ) shows that our approach ( i ) is always sound and computes an upper bound on the maximum violation of the specification being verified , and ( ii ) is expressive enough to theoretically capture tight verification ( i.e . obtaining both sound and complete verification ) . • We develop novel algorithms for handling specific multipliers and objectives within our framework ( Propositions 1 , 2 ) . This allows us to apply our framework to novel specifications ( such as distributionally robust OOD detection , where input perturbations are drawn from entire sets of distributions ) by better capturing the probabilistic structure of the problem . • We empirically validate our method by verifying neural networks , which are verificationagnostic , on a variety of probabilistic specifications . We demonstrate that even with relatively simple choices for the functional multiplier , our method strongly outperforms prior methods , which sometimes provide vacuous guarantees only . This further points towards the potential for significant improvements to be had by developing tractable optimization techniques for more complex and expressive multipliers within our framework . 2 Probabilistic Specifications . 2.1 Notation . Let us consider a possibly stochastic neural network φ : X → P ( Y ) , where X is the set of possible input values to the model , Y is the set of possible output values , and P ( Y ) is the set of distributions over Y . We assume that Y is a subset of Rl ( unless specified otherwise ) , where l is the number of labels , and the output of the model are logits corresponding to unnormalized log-confidence scores assigned to the labels { 1 , . . . , l } . The model is assumed to be a sequence of K layers , each of them possibly stochastic . For k ∈ { 1 , . . . , K } , πk ( xk|xk−1 ) denotes the probability that the output of layer k takes value xk when its input value is xk−1 . We write xk ∼ πk ( xk−1 ) to denote that xk is drawn from the distribution over outputs of layer k given input xk−1 to layer k. We further assume that each πk ( x ) has the form σ ( w̃x+ b̃ ) , where σ is a non-linear activation function ( e.g. , ReLU , sigmoid , MaxOut ) , and w̃ and b̃ are random variables . The stochasticity for layer πk is assumed to be statistically independent of the stochasticity at other layers . For a BNN , w̃ and b̃ follow a diagonal Gaussian distribution ( i.e. , a Gaussian distribution with a diagonal covariance matrix ) , and for a MC-Dropout network they follow a Bernoulli-like distribution . Given a distribution p0 over the inputs X , we use φ ( p0 ) to denote ( with a slight abuse ) the distribution of the random variable φ ( X0 ) , where X0 ∼ p0 . 2.2 Problem Formulation .. We now introduce the general problem formulation for which we develop the verification framework . Definition 1 ( Probabilistic verification problem ) . Given a ( possibly stochastic ) neural network φ : X → P ( Y ) , a set of distributions over the input P0 and a functional ψ : P ( Y ) 7→ R , the probabilistic verification problem is to check that the following is true : ∀ p0 ∈ P0 , ψ ( φ ( p0 ) ) ≤ 0 . ( 1 ) 2.3 Examples of Specifications . Below we provide examples of probabilistic specifications which are captured by the above problem formulation , and that we further empirically validate our framework on . In Appendix A , we provide further examples of relevant specifications ( e.g. , ensuring reliable uncertainty calibration ) that can be handled by our problem setup . Distributionally Robust OOD Detection . We consider the problem of verifying that a stochastic neural network assigns low confidence scores to all labels for OOD inputs , even in the presence of bounded noise perturbations to the inputs . Given a noise distribution perturbing an OOD image xood , we require that the expected softmax is smaller than a specified confidence threshold pmax for each label i . Since the precise noise distribution is most often unknown , we wish to consider an entire class Pnoise of noise distributions . Denoting by δx the Dirac distribution around x , the problem is then to guarantee that for every p0 in P0 = { δxood + ω : ω ∈ Pnoise } and for each possible label i , ψ ( φ ( p0 ) ) : = Ey∼φ ( p0 ) [ softmax ( y ) i ] − pmax ≤ 0 . Robust OOD detection under bounded ` ∞ perturbations as considered in Bitterwolf et al . [ 2020 ] is a special case of this problem where Pnoise is restricted to a set of δ distributions over points with bounded ` ∞ norm . Robust Classification . We also extend the commonly studied robust classification problem [ Madry et al. , 2017 ] under norm-bounded perturbations , to the setting of probabilistic neural networks ( e.g . BNNs ) . Define P0 to be the set of δ input distributions centered at points within an -ball of a nominal point xnom , with label i ∈ { 1 , . . . , l } : P0 = { δx : ‖x− xnom‖ ≤ } . For every p0 ∈ P0 , we wish to guarantee that the stochastic NN correctly classifies the input , i.e . for each j , ψ ( φ ( p0 ) ) : = Ey∼φ ( p0 ) [ softmax ( y ) i − softmax ( y ) j ] ≤ 0 . Note that it is important to take the expectation of the softmax ( and not logits ) since this is how inference from BNNs is performed . 3 The Functional Lagrangian Framework . We consider the following optimization version : OPT = max p0∈P0 ψ ( φ ( p0 ) ) , ( 2 ) Having OPT ≤ 0 here is equivalent to satisfying specification ( 1 ) . However , solving problem ( 2 ) directly to global optimality is intractable in general , because it can possibly be a challenging nonlinear and stochastic optimization problem . However , to only verify that the specification is satisfied , it may suffice to compute an upper bound on OPT . Here , we describe how the functional Lagrangian framework allows to derive such bounds by decomposing the overall problem into smaller , easier sub-problems . 3.1 General Framework . Let Xk denote the feasible space of activations at layer k , and let pk denote the distribution of activations at layer k when the inputs follow distribution p0 ( so that pK = φ ( p0 ) ) . Assumptions . In order to derive our verification framework , we make the following assumptions : ( A1 ) : ∃ l0 ≤ u0 ∈ Rn such that for each input distribution p0 ∈ P0 , Support ( p0 ) ⊆ X0 = [ l0 , u0 ] . ( A2 ) : Each layer is such that if x ∈ Xk = [ lk , uk ] , then Support ( πk ( x ) ) ⊆ Xk+1 = [ lk+1 , uk+1 ] . Assumption ( A1 ) is natural since the inputs to neural networks are bounded . Assumption ( A2 ) can be restrictive in some cases : it requires that the layer output is bounded with probability 1 , which is not true , for example , if we have a BNN with a Gaussian posterior . However , we can relax this assumption to requiring that the output is bounded with high probability , as in Wicker et al . [ 2020 ] . Functional Lagrangian Dual . In order to derive the dual , we begin by noting that problem ( 2 ) can be equivalently written in the following constrained form : max p0∈P0 , p1 , ... , pK ψ ( pK ) s.t . ∀ k ∈ { 0 , . . . , K − 1 } , ∀ y ∈ Xk+1 , pk+1 ( y ) = ∫ Xk πk ( y|x ) pk ( x ) dx . For the k-th constraint , let us assign a Lagrangian multiplier λk+1 ( y ) to each possible y ∈ Xk+1 . Note that λ ( y ) is chosen independently for each y , hence λ is a functional multiplier . We then integrate over y , which yields the following Lagrangian penalty to be added to the dual objective : − ∫ Xk+1 λk+1 ( y ) pk+1 ( y ) dy + ∫ Xk , Xk+1 λk+1 ( y ) πk ( y|x ) pk ( x ) dxdy . ( 3 ) We now make two observations , which are described here at a high level only and are available in more details in appendix B . First , if we sum these penalties over k and group terms by pk , it can be observed that the objective function decomposes additively over the pk distributions . Second , for k ∈ { 1 , . . . , K − 1 } , each pk can be optimized independently ( since the objective is separable ) , and since the objective is linear in pk , the optimal pk is a Dirac distribution , which means that the search over the probability distribution pk can be simplified to a search over feasible values xk ∈ Xk . This yields the following dual : max pK∈PK ( ψ ( pK ) − ∫ XK λK ( x ) pK ( x ) dx ) + K−1∑ k=1 max x∈Xk ( ∫ Xk+1 λk+1 ( y ) πk ( y|x ) dy − λk ( x ) ) + max p0∈P0 ∫ X0 ( ∫ X1 λ1 ( y ) π0 ( y|x ) dy ) p0 ( x ) dx , ( 4 ) where we define PK , φ ( P0 ) . In the rest of this work , we refer to this dual as g ( λ ) , and we use the following notation to simplify equation ( 4 ) : g ( λ ) = max p0∈P0 g0 ( p0 , λ1 ) + K−1∑ k=1 max xk∈Xk gk ( xk , λk , λk+1 ) + max pK∈PK gK ( pK , λK ) . ( 5 ) The dual g ( λ ) can be seen as a generalization of Lagrangian relaxation [ Bertsekas , 2015 ] with the two key modifications : ( i ) layer outputs are integrated over possible values , and ( ii ) Lagrangian penalties are expressed as arbitrary functions λk ( x ) instead of being restricted to linear functions . Main Result . Here , we relate the functional Lagrangian dual to the specification objective ( 2 ) . Theorem 1 . For any collection of functions λ = ( λ1 , . . . , λK ) ∈ RX1 × . . .× RXK , we have that g ( λ ) ≥ OPT . In particular , if a choice of λ can be found such that g ( λ ) ≤ 0 , then specification ( 1 ) is true . Further , when ψ ( pK ) = Ey∼pK [ c ( y ) ] , the dual becomes tight : g ( λ ? ) = OPT if λ ? is set to : λ ? K ( x ) = c ( x ) ; ∀ k ∈ { K − 1 , . . . , 1 } , λ ? k ( x ) = E y∼πk ( x ) [ λ ? k+1 ( y ) ] . Proof . We give a brief sketch of the proof - the details are in Appendix B . The problem in constrained form is an infinite dimensional optimization with decision variables p0 , p1 , . . . , pK and linear constraints relating pk and pk+1 . The Lagrangian dual of this optimization problem has objective g ( λ ) . By weak duality , we have g ( λ ) ≥ OPT . The second part of the theorem is easily observed by plugging in λ ? in g ( λ ) and observing that the resulting optimization problem is equivalent to ( 2 ) . Example . Let P0 be the set of probability distributions with mean 0 , variance 1 , and support [ −1 , 1 ] , and let N [ a , b ] ( µ , σ2 ) denote the normal distribution with mean µ and variance σ2 with truncated support [ a , b ] . Now consider the following problem , for which we want to compute an upper bound : OPT = max p0∈P0 EX1 [ exp ( −X1 ) ] s.t . X1|X0 ∼ N [ 0,1 ] ( X20 , 1 ) and X0 ∼ p0 . ( 6 ) This problem has two difficulties that prevent us from applying traditional optimization approaches like Lagrangian duality [ Bertsekas , 2015 ] , which has been used in neural network verification Dvijotham et al . [ 2018b ] . The first difficulty is that the constraint linking X1 to X0 is stochastic , and standard approaches can not readily handle that . Second , the optimization variable p0 can take any value in an entire set of probability distributions , while usual methods can only search over sets of real values . Thus standard methods fail to provide the tools to solve such a problem . Since the probability distributions have bounded support , a possible way around this problem is to ignore the stochasticity of the problem , and to optimize over the worst-case realization of the random variable X1 in order to obtain a valid upper bound on OPT as : OPT ≤ maxx1∈ [ 0,1 ] exp ( −x1 ) = 1 . However this is an over-pessimistic modeling of the problem and the resulting upper bound is loose . In contrast , Theorem 1 shows that for any function λ : R→ R , OPT can be upper bounded by : OPT ≤ max x1∈ [ 0,1 ] , p0∈P0 exp ( −x1 ) − λ ( x1 ) + EX0∼p0 [ EX1|X0∼N [ 0,1 ] ( X20 ,1 ) [ λ ( X1 ) ] ] . This inequality holds true in particular for any function λ of the form x 7→ θx where θ ∈ R , and thus : OPT ≤ inf θ∈R max x1∈ [ 0,1 ] , p0∈P0 exp ( −x1 ) − θx1 + EX0∼p0 [ EX1|X0∼N [ 0,1 ] ( X20 ,1 ) [ θX1 ] ] , = inf θ∈R max x1∈ [ 0,1 ] , p0∈P0 exp ( −x1 ) − θx1 + θEX0∼p0 [ X20 ] , = inf θ∈R max x1∈ [ 0,1 ] exp ( −x1 ) − θx1 + θ ≈ 0.37 . Here , our framework lets us tractably compute a bound on OPT that is significantly tighter compared to the naive support-based bound .

Summary: The paper proposes a framework for giving mathematical guarantees on neural networks with probabilistic specifications, e.g. for robustness across distributions or for robustness of models with non-deterministic outputs. The authors first formulate the abstract framework in terms of functional lagrangians and then show how to apply these by using concrete applications and concrete function classes for the lagrangians.