Complete Verification via Multi-Neuron Relaxation Guided Branch-and-Bound

1 INTRODUCTION . Recent years have witnessed substantial interest in new methods for certifying properties of neural networks , ranging from stochastic methods ( Cohen et al. , 2019 ) which construct a robust model from an underlying classifier to deterministic methods ( Gehr et al. , 2018 ; Katz et al. , 2017 ; Xu et al. , 2020a ) that analyze a given network as is ( the focus of our work ) . Key challenge : Scalable and Precise Non-Linearity Handling Deterministic verification methods can be categorized as complete or incomplete . Recent incomplete verification methods based on propagating and refining a single convex region ( Müller et al. , 2021 ; Dathathri et al. , 2020 ; Tjandraatmadja et al. , 2020 ) are limited in precision due to fundamental constraints imposed by convex relaxations . Traditional complete verification approaches based on SMT solvers ( Ehlers , 2017 ) or a single mixed-integer linear programming encoding of a property ( Tjeng et al. , 2017 ; Katz et al. , 2017 ) suffer from worst-case exponential complexity and are often unable to compute sound bounds in reasonable time-frames . To address this issue , a promising recent formulation in the form of a Branch-and-Bound approach ( Bunel et al. , 2020 ) has been popularized , yielding anytime-valid bounds , by recursively splitting the problem domain into easier subproblems and deriving bounds on each of these via cheap and less precise methods ( Xu et al. , 2020b ; Wang et al. , 2021 ; Palma et al. , 2021 ; Henriksen & Lomuscio , 2021 ) . This approach has proven effective on ( smaller ) networks where there are relatively few unstable activations and where splitting can yield substantial improvements . However , for larger networks or those not regularized to be amenable to certification , this strategy becomes increasingly ineffective as the larger number of unstable activations make individual splits less effective ( exacerbated by the relatively loose underlying bounding methods ) . This work : Branch-and-Bound guided by Multi-Neuron Constraints In this work , we propose a new certification method and verifier , called Multi-Neuron Constraint Guided BaB ( MN-BAB ) , which aims to combine the best of both worlds : it builds on the tight multi-neuron constraints proposed by Müller et al . ( 2021 ) and leverages these constraints within a BaB framework to yield an efficient , GPU based dual solver . The key benefit here lies in the fact that exploiting the significantly increased precision of the underlying bounding method substantially reduces the number of domain splits ( carrying exponential cost ) required to certify a property . This benefit is especially pronounced for larger and less regularized networks where additional splits of the problem domain yield diminishing returns . We release all code and scripts to reproduce our experiments at < ANONYMIZED > . Main Contributions : • We present a novel verification framework , MN-BAB , which leverages tight multi-neuron constraints and a GPU-based dual solver in a BaB approach . • We develop a novel branching heuristic based on information obtained from analyzing our multi-neuron constraints . • We propose a new class of branching heuristics , applicable to all BaB-based verifiers , that correct the expected bound improvement of a branching decision for the incurred computational cost . • Our extensive empirical evaluation demonstrates that we improve on the state-of-the-art in terms of certified accuracy by as much as 28 % on challenging networks . 2 BACKGROUND . In this section , we review the necessary background underlying our method ( discussed next ) . 2.1 NEURAL NETWORK VERIFICATION . The neural network verification problem can be defined as follows : given an input domainD ⊆ X , a network f : X → Y , and a linear property P ⊆ Y over the output neurons y ∈ Y , prove f ( x ) ∈ P , ∀x ∈ D. We instantiate this problem with the challenging ` ∞-norm bounded specifications and set D to the ` ∞ ball around an input point x0 of radius : D ( x0 ) = { x | ||x− x0||∞ ≤ } . For ease of presentation , we consider neural networks of L fully connected layers with ReLU activation functions ( we note that MN-BAB can handle a wide range of layers including convolutional , residual , batch-normalization , average-pooling layers ) . We focus on ReLU networks as the BaB framework only yields complete verifiers for piecewise linear activation functions , but remark that our approach is applicable to a wide class of activations including ReLU , Sigmoid , Tanh , MaxPool , and others . We denote the number of neurons in the ith layer as di and the corresponding weights and biases asW ( i ) ∈ Rdi×di−1 and b ( i ) ∈ Rdi for i ∈ { 1 , ... , L } . Further , the neural network is defined as f ( x ) : = ẑ ( L ) ( x ) where ẑ ( i ) ( x ) : =W ( i ) z ( i−1 ) ( x ) + b ( i ) and z ( i ) ( x ) : = max ( 0 , ẑ ( i ) ( x ) ) . For readability , the notation omits the dependency of intermediate activations on x . Since we can encode any linear property over output neurons into an additional affine layer , we can simplify the general formulation f ( x ) ∈ P to f ( x ) > 0 . The property can now be verified by proving that a lower bound is greater 0 via the following optimization problem : min x∈D ( x0 ) f ( x ) = ẑ ( L ) s.t . ẑ ( i ) =W ( i ) z ( i−1 ) + b ( i ) z ( i ) = max ( 0 , ẑ ( i ) ) ( 1 ) A method is called sound if every property it proves actually holds ( no false positives ) . A method is called complete if it can prove every property that actually holds ( no false negatives ) . 2.2 BRANCH-AND-BOUND FOR VERIFICATION . Bunel et al . ( 2020 ) successfully applied the Branch-and-Bound ( BaB ) approach ( Land & Doig , 1960 ) to neural network verification . It consists of a bounding method that computes sound upper and lower bounds on the optimization objective of Eq . ( 1 ) and a branching method that recursively splits the problem into subproblems with added constraints , allowing for increasingly tighter bounds . If an upper bound < 0 is found , this represents an adversarial example and allows to terminate the procedure . If the obtained lower bound is > 0 , the property is verified on the corresponding domain . If a lower bound > 0 is derived on all splits , the property is verified . An ideal splitting procedure minimizes the total time required for bounding , which is often well approximated by the number of considered subproblems . A simple approach is to split the input domain , however , this is inefficient for high-dimensional input spaces . Splitting a ReLU activation node into its positive and negative phases has been shown to be far more efficient ( Bunel et al. , 2020 ) and ultimately yields a complete verifier ( Wang et al. , 2021 ) . Hence , we focus solely on ReLU branching strategies . 2.3 LINEAR CONSTRAINTS The key challenge in neural network verification Eq . ( 1 ) are the non-linear activations . Stable ReLUs , i.e. , those which we can show to be always active ( ẑ ≥ 0 ) or inactive ( ẑ ≤ 0 ) , can be replaced by a linear function . Unstable ReLUs , i.e . those that can be either active or inactive depending on the exact x ∈ D , have to be approximated using a convex relaxation of their´ input-output-set . We build on the convex relaxation introduced in DEEPPOLY ( Singh et al. , 2019b ) and shown in Fig . 1 . Its key property is the single linear upper and lower bound , which allows for efficient bound computation . 2.4 MULTI-NEURON CONSTRAINTS All convex relaxations that consider ReLU neurons individually are fundamentally limited in their precision by the so-called ( single-neuron ) convex relaxation barrier ( Salman et al. , 2019 ) . It can be overcome by considering multiple neurons jointly ( Singh et al. , 2019a ; Tjandraatmadja et al. , 2020 ; Müller et al. , 2021 ; Palma et al. , 2021 ) capturing interactions between these neurons to obtain tighter bounds , illustrated in Fig . 2 . We leverage the multi-neuron constraints from Müller et al . ( 2021 ) , expressed as a conjunction of linear constraints over the joint input and output space of a ReLU layer . 2.5 CONSTRAINED OPTIMIZATION VIA LAGRANGE MULTIPLIERS . To incorporate constraints into an optimization problem we use the technique of Lagrange multipliers . Given a constrained minimization problem minx f ( x ) , s.t . g ( x ) ≤ 0 , we can bound the objective function with : min x f ( x ) ≥ min x max λ≥0 f ( x ) + λg ( x ) If the constraint is satisfied strictly , i.e. , g ( x ) < 0 , we set λ = 0 to maximize the objective , else , i.e. , g ( x ) ≥ 0 , increasing λ would allow the objective to grow unboundedly , shifting the minimum over x until the constraint is satisfied . Hence , if λ > 0 after optimization , the constraint is active , i.e. , it is actively enforced and currently satisfied with equality . 3 A MULTI-NEURON RELAXATION BASED BAB FRAMEWORK . In this section , we describe the two key components of MN-BAB : ( i ) an efficient bounding method leveraging multi-neuron constraints , and ( ii ) a branching method tailored to it . 3.1 EFFICIENT MULTI-NEURON BOUNDING . We build on the approach of Singh et al . ( 2019b ) , extended by Wang et al . ( 2021 ) of deriving a lower bound f as a function of the network inputs and a set of optimizable parameters . Crucially , we tighten these relatively loose bounds significantly by enforcing precise multi-neuron constraints via Lagrange multipliers . To enable this , we develop a method capable of integrating any linear constraint over arbitrary neurons anywhere in the network into the bounding objective . At a high level , we derive linear upper and lower bounds of the form z ( i ) ≥ Az ( i−1 ) + c for every layer ’ s output in terms of its inputs z ( i−1 ) . Then , starting with a linear expression in the last layer ’ s outputs z ( L ) that we aim to bound , we use the linear bounds derived above , to replace z ( L ) with symbolic bounds depending only on the previous layer ’ s values z ( L−1 ) . We proceed in this manner recursively , until we obtain an expression in terms of only the networks inputs . Below we describe this backsubstitution for ReLU and affine layers . Affine Layer Assume any affine layer ẑ ( i ) = W ( i ) z ( i−1 ) +b ( i ) and a lower bound f = â ( i ) ẑ ( i ) + ĉ ( i ) with respect to its outputs . We then substitute the affine expression for ẑ ( i ) to obtain : f = â ( i ) W ( i ) ︸ ︷︷ ︸ a ( i−1 ) z ( i−1 ) + â ( i ) b ( i ) + ĉ ( i ) ︸ ︷︷ ︸ c ( i−1 ) = a ( i−1 ) z ( i−1 ) + c ( i−1 ) ( 2 ) ReLU Layer Let f = a ( i ) z ( i ) + c ( i ) be a lower bound with respect to the output of a ReLU layer z ( i ) = max ( 0 , ẑ ( i ) ) and l ( i ) and u ( i ) bounds on its input s.t . l ( i ) ≤ ẑ ( i ) ≤ u ( i ) , obtained by recursively applying the bounding procedure or using a cheaper but less precise propagationbased bounding method ( Singh et al. , 2018 ; Gowal et al. , 2018 ) . The backsubstitution through a ReLU layer now consists of three distinct steps : 1 ) enforcing multi-neuron constraints , 2 ) enforcing neuron-wise relaxations , and 3 ) enforcing split constraints , which we describe in detail below . Enforcing multi-neuron constraints We compute multi-neuron constraints as described in Müller et al . ( 2021 ) , although our approach is applicable to any linear constraints in the input-output space of ReLU activations , written as : [ P ( i ) P̂ ( i ) −p ( i ) ] z ( i ) ẑ ( i ) 1 ≤ 0 . ( 3 ) where ẑ ( i ) are the pre- and z ( i ) the post-activation values and P ( i ) , P̂ ( i ) , and −p ( i ) the constraint parameters . We enforce these constraints using Lagrange multipliers ( see §2.5 ) , yielding sound lower bounds for all γ ∈ ( R≥0 ) ei , where ei denotes the number of multi-neuron constraints in layer i. a ( i ) z ( i ) + c ( i ) ≥ max γ ( i ) ≥0 a ( i ) z ( i ) + c ( i ) + γ ( i ) > ( P ( i ) z ( i ) + P̂ ( i ) ẑ ( i ) +−pi ) = max γ ( i ) ≥0 ( a ( i ) + γ ( i ) > P ( i ) ) ︸ ︷︷ ︸ a′ ( i ) z ( i ) + γ ( i ) > P̂ ( i ) ẑ ( i ) + γ ( i ) > ( −pi ) + c ( i ) ︸ ︷︷ ︸ c′ ( i ) Note that this approach can be easily extended to linear constraints over any activations in arbitrary layers if applied in the last affine layer at the very beginning of the backsubstitution process . Enforcing single-neuron constraints We now apply the single-neuron DEEPPOLY relaxation with parametrized slopes α collected inD ( see below ) : max γ ( i ) ≥0 a′ ( i ) z ( i ) + c′ ( i ) ≥ max 0≤α ( i ) ≤1 γ ( i ) ≥0 a′ ( i ) ( D ( i ) ẑ ( i ) + b ( i ) ) + c′ ( i ) = max 0≤α ( i ) ≤1 γ ( i ) ≥0 a′ ( i ) D ( i ) ︸ ︷︷ ︸ a′′ ( i ) ẑ ( i ) + a′ ( i ) b ( i ) + c′ ( i ) ︸ ︷︷ ︸ c′′ ( i ) The intercept vector b and the diagonal slope matrixD are defined as : Dj , j = 1 if lj ≥ 0 or node j is positively split 0 if uj ≤ 0 or node j is negatively split αj if lj < 0 < uj and aj ≥ 0 uj uj−uj if lj < 0 < uj and aj < 0 bj = { − uj ljuj−lj if lj < 0 < uj and aj < 0 0 otherwise Where we drop the layer index i for readability and αj is the lower bound slope parameter illustrated in Fig . 1 . Note how , depending on whether the sensitivity a ( i ) j of f w.r.t . z ( i ) j has positive or negative sign , we substitute z ( i ) j for its lower or upper bound , respectively . Enforcing split constraints We encode split constraints of the form ẑ ( i ) j ≥ 0 or ẑ ( i ) j ≤ 0 using the diagonal split matrix S as follows , again dropping the layer index i : Sj , j = −1 positive split : ẑj ≥ 0 1 negative split : ẑj < 0 0 no split S ( i ) ẑ ( i ) ≤ 0 We again enforce these constraints using Lagrange multipliers : max 0≤α ( i ) ≤1 γ ( i ) ≥0 a′′ ( i ) ẑ ( i ) + c′′ ( i ) ≥ max 0≤α ( i ) ≤1 β ( i ) ≥0 γ ( i ) ≥0 ( a′′ ( i ) + β ( i ) > S ( i ) ) ︸ ︷︷ ︸ a′′′ ( i ) ẑ ( i ) + c′′ ( i ) ︸︷︷︸ c′′′ ( i ) Putting everything together , the backsubstitution operation through a ReLU layer is : min x∈D a ( i ) z ( i ) + c ( i ) ≥ min x∈D max 0≤α ( i ) ≤1 β ( i ) ≥0 γ ( i ) ≥0 ( ( a ( i ) + γ ( i ) > P ( i ) ) D ( i ) + β ( i ) > S ( i ) + γ ( i ) > P̂ ( i ) ) ︸ ︷︷ ︸ â ( i ) ẑ ( i ) + ( a ( i ) + γ ( i ) > P ( i ) ) ′b ( i ) + γ ( i ) > -p ( i ) + c ( i ) ︸ ︷︷ ︸ ĉ ( i ) ( 4 ) Full backsubstitution through all layers leads to an optimizable lower bound on f : min x∈D f ( x ) ≥ min x∈D max 0≤α≤1 0≤β 0≤γ a ( 0 ) x+ c ( 0 ) ≥ max 0≤α≤1 0≤β 0≤γ min x∈D a ( 0 ) x+ c ( 0 ) where the second inequality holds due to weak duality . We denote all α ( i ) j from every layer of the backsubstitution process with α and define β and γ analogously . The inner minimization over x has a closed form solution if D is an lp-ball of radius around x0 , given by Hölder ’ s inequality : max 0≤α≤1 0≤β 0≤γ min x∈D a ( 0 ) x+ c ( 0 ) ≥ max 0≤α≤1 0≤β 0≤γ a ( 0 ) x0 − ||a ( 0 ) > ||q + c ( 0 ) ( 5 ) where q is defined s.t . 1p + 1 q = 1 . Since these bounds are sound for any 0 ≤ α ≤ 1 , and β , γ ≥ 0 , they can be tightened by using ( projected ) gradient ascent to optimize these parameters . We compute all intermediate bounds using the same bounding procedure , leading to two full parameter sets for every neuron in the network . To reduce memory requirements , we share parameter sets between all neurons in the same layer , but keep separate sets for upper and lower bounds . Upper Bounding the Minimum Objective Showing an upper bound on the minimum optimization objective precluding verification ( f < 0 ) allows us to terminate the BaB process early . Propagating any input x ∈ D through the network yields a valid upper bound , hence , we propose to use the input that minimizes Eq . ( 5 ) : xi = { ( x0 ) i + if a ( 0 ) i < 0 ( x0 ) i − if a ( 0 ) i ≥ 0 .

Summary: This paper deals with complete verification of neural network, based on a branch and bound framework.  It is essentially a combination of two recent papers: - Beta-crown[1], which optimizes with first order methods the bounds obtained by LIRPA type bounds. Additional constraints on the network activations (such as the ones resulting from branching) are included through lagrange multipliers and optimized over. - Prima[2], which derives linear constraints to impose on the activation of the network to obtain tighter relaxations. These constraints comes from the fact that you can obtain a tighter convex hull by relaxing a group of neurons together, rather than relaxing each neuron independently. In the Prima paper, the constraints are simply encoded into an off-the-shelf LP solver.  The main contribution of this paper comes from adding the linear constraints that comes from the Prima relaxation (using Lagrange multiplier) to the problem optimized by Beta-crown.  In addition to that, the paper also propose new branching heuristics: - Active Constraint Score Branching, which uses the lagrangian variables associated with the constraints of the multi-neuron relaxation to prioritize branching. - Cost Adjusted Branching, which aim to correct the score of the branching heuristic based on the expected computational cost induced by each possible branching choice.  Experiments are ran on MNIST and CIFAR10  [1] Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Complete and Incomplete Neural Network Verification [2] PRIMA: Precise and General Neural Network Certification via Multi-Neuron Convex Relaxations