DeepSplit: Scalable Verification of Deep Neural Networks via Operator Splitting

1 Introduction . Despite their superior performance , neural networks lack formal guarantees , raising serious concerns about their adoption in safety-critical applications such as autonomous vehicles ( Cao et al. , 2019 ) and medical machine learning ( Finlayson et al. , 2019 ) . Motivated by this drawback , there has been an increasing interest in developing tools to verify desirable properties for neural networks , such as robustness to adversarial attacks . Neural network verification refers to the problem of verifying whether the output of a neural network satisfies certain properties for a bounded set of input perturbations . This problem can be framed as optimization problems of the form minimize J ( f ( x ) ) subject to x ∈ X , ( 1 ) where f is given by a deep neural network , J is a real-valued function representing a performance measure ( or a specification ) , and X is a set of inputs to be verified . In this formulation , verifying the neural network amounts to certifying whether the optimal value of ( 1 ) is bounded below by a certain threshold . This problem is large-scale and non-convex , making it extremely difficult to solve efficiently–both in terms of time and memory . For ReLU activation functions and linear objectives , the problem in ( 1 ) can be cast as a MixedInteger Linear Program ( MILP ) ( Lomuscio & Maganti , 2017 ; Cheng et al. , 2017 ; Dutta et al. , 2018 ; Fischetti & Jo , 2018 ) , which can be solved for the global solution via , for example , Branch-and-Bound ( BaB ) methods . While we do not expect these approaches to scale to large problems , for small neural networks they can still be practical . Instead of solving ( 1 ) for its global minimum , one can instead find guaranteed lower bounds on the optimal value via convex relaxations , such as Linear Programming ( LP ) ( Wong & Kolter , 2017 ) and Semidefinite Programming ( SDP ) ( Raghunathan et al. , 2018 ; Fazlyab et al. , 2019 ; 2020 ) . Verification methods based on convex relaxations are sound but incomplete , i.e. , they are guaranteed to detect all false negatives but also produce false positives , whose rate depends on the tightness of the relaxation . Although convex relaxations are polynomial-time solvable ( in terms of number of decision variables ) , in practice they are not computationally tractable for large-scale neural networks . To improve scalability , these relaxations must typically be further relaxed ( Wong & Kolter , 2017 ; Weng et al. , 2018 ; Zhang et al. , 2018 ; Dvijotham et al. , 2018 ; Bunel et al. , 2020a ) . Contributions In this work , we propose an algorithm for solving convex relaxations exactly for large-scale neural networks . Our starting point is to express ( 1 ) as a constrained optimization problem whose constraints are imposed by the forward passes in the network . We then introduce additional decision variables and consensus constraints that naturally split the corresponding problem into independent subproblems , which often have closed-form solutions . Finally , we employ an operator splitting technique based on the Alternating Direction Method of Multipliers ( ADMM ) ( Boyd et al. , 2011 ) , to solve the corresponding Lagrangian relaxation of the problem . This approach has several favorable properties . First , the method requires minimal parameter tuning and relies on simple operations , which scale to very large problems and can achieve a good trade-off between runtime and solution accuracy . Second , all the solver operations are amenable to fast parallelization with GPU acceleration . Third , our method is fully modular and applies to standard network architectures . We employ our method to compute exact solutions to LP relaxations on the worst-case performance of adversarially trained deep networks , with a focus on networks whose convex relaxations were previously impossible to solve exactly due to their size . Specifically , we perform extensive experiments in the ` ∞ perturbation setting , where we verify robustness properties of image classifiers for CIFAR10 and deep Q-networks ( DQNs ) in Atari games ( Zhang et al. , 2020 ) . Our method is able to quickly solve LP relaxations at scales that are too large for exact BaB verifiers , SDP relaxations , or commercial LP solvers such as Gurobi . 1.1 Related Work . Convex relaxations LP relaxations are relatively the most scalable form of convex relaxations ( Ehlers , 2017 ) . However , even solving LPs can become computationally prohibitive for small convolutional networks ( Salman et al. , 2019 ) . One line of work studies computationally cheaper but looser bounds of the LP relaxation ( Wong & Kolter , 2017 ) , which have been extended to larger and more general networks and settings ( Wong et al. , 2018 ; Weng et al. , 2018 ; Zhang et al. , 2018 ; Xu et al. , 2020 ) . These bounds tend to be loose unless optimized during training , which typically comes at a significant cost to standard performance . Further work has aimed to tighten these bounds ( Singh et al. , 2019 ; Tjandraatmadja et al. , 2020 ; de2 ) , however these works focus primarily on small convolutional networks and struggle to scale to more typical deep networks . Other work has studied the limits of these convex relaxations on these small networks using vast amounts of CPU-compute ( Salman et al. , 2019 ) . Recent SDP-based approaches ( Dathathri et al. , 2020 ) can produce much tighter bounds on these small networks . Lagrangian-based bounds Related to our work is that which solves the Lagrangian of the LP relaxation ( Dvijotham et al. , 2018 ; Bunel et al. , 2020a ) , which can tighten the bound but do not aim to solve the LP exactly due to relatively slow convergence . However , these works primarily study small networks whose LP relaxation can still be solved exactly with Gurobi . Although these works could in theory be used on larger networks , only the faster , linear-based bounds ( Xu et al. , 2020 ) have demonstrated applicability to standard deep learning architectures . In our work , we solve the LP relaxation exactly in large network settings that previously have only been studied with loose bounds of the LP relaxation such as LiRPA ( Xu et al. , 2020 ) . Complete verification methods These methods verify properties of deep networks exactly ( without false positives or false negatives ) using methods such as SMT solvers ( Scheibler et al. , 2015 ; Ehlers , 2017 ; Katz et al. , 2017 ) and MILP solvers ( Lomuscio & Maganti , 2017 ; Cheng et al. , 2017 ; Dutta et al. , 2018 ; Fischetti & Jo , 2018 ) . Complete verification methods typically rely on BaB algorithms ( Bunel et al. , 2017 ) , in which the verification problem is divided into subproblems ( branching ) that can be verified using incomplete verification methods ( bounding ) ( Bunel et al. , 2020b ) . However , these methods have a worst-case exponential runtime and have difficulty scaling beyond relatively small convolutional networks . Motivated by this , several recent works have improved the practical running time of BaB methods with custom solvers for medium-sized networks ( Bunel et al. , 2020a ; De Palma et al. , 2021 ; Xu et al. , 2021 ; Wang et al. , 2021 ) , but have yet to scale to larger deep networks . Operator splitting methods Operator splitting , and in particular the ADMM method , is a powerful technique in solving structured convex optimization problems and has applications in numerous settings ranging from optimal control ( O ’ Donoghue et al. , 2013 ) to training neural networks ( Taylor et al. , 2016 ) . These methods scale well with the problem dimensions , can exploit sparsity in the problem data efficiently ( Zheng et al. , 2017 ) , are amenable to parallelization on GPU ( Schubiger et al. , 2020 ) , and have well-understood convergence properties under minimal regularity assumptions ( Boyd et al. , 2011 ) . The benefit of ADMM as an alternative to interior-point solvers has been shown in various classes of optimization problems ( O ’ donoghue et al. , 2016 ) . Our operator splitting method is specifically tailored for neural network verification in order to fully exploit the problem structure . Notation . We denote the set of real numbers by R , the set of nonnegative real numbers by R+ , the set of real n-dimensional vectors by Rn , the set of m× n-dimensional matrices by Rm×n , and the n-dimensional identity matrix by In . The p-norm ( p ≥ 1 ) is denoted by ‖ · ‖p : Rn → R+ . For a set S , we define the indicator function IS ( x ) of S as IS ( x ) = 0 if x ∈ S and IS ( x ) = +∞ otherwise . Given a function f : X → Y , the graph of f is the set Gf = { ( x , f ( x ) ) | x ∈ X } . 2 Scalable Neural Network Verification via Operator Splitting . We consider an ` -layer feed-forward neural network f ( x ) : Rn0 → Rn ` described by the following recursive equations , x0 = x , xk+1 = φk ( xk ) k = 0 , · · · , ` − 1 , f ( x ) = x ` ( 2 ) where x0 ∈ Rn0 is the input to the neural network , xk ∈ Rnk is the input to the k-th layer , and φk : Rnk → Rnk+1 is the operator of the k-th layer , which can represent any commonly-used operator in deep networks , such as linear ( convolutional ) layers , MaxPooling units , and activation functions . Given the neural network f , a specification function J : Rn ` 7→ R , and an input set X ⊂ Rn0 , we say that f satisfies the specification J if J ( f ( x ) ) ≥ 0 for all x ∈ X . This is equivalent to verifying that the optimal value of ( 1 ) is non-negative . We assume X ⊂ Rn0 is a closed convex set and J : Rn ` → R∪ { +∞ } is a convex function , defining a performance measure on the output of the network . Note that our formulation generalizes to arbitrary computational graphs ( e.g . residual blocks ) and general objective functions ( see Appendix B ) . Using the sequential representation of the neural network in ( 2 ) , we may rewrite the optimization problem in ( 1 ) as the following constrained optimization problem , J ? ← minimize J ( x ` ) subject to xk+1 = φk ( xk ) , k = 0 , · · · , ` − 1 , x0 ∈ X , ( 3 ) with n : = ∑ ` k=0 nk decision variables . We can rewrite ( 3 ) equivalently as J ? ← minimize J ( x ` ) subject to ( xk , xk+1 ) ∈ Gφk , k = 0 , · · · , ` − 1 , x0 ∈ X , ( 4 ) where Gφk = { ( xk , xk+1 ) | xk+1 = φk ( xk ) , xk ≤ xk ≤ x̄k } is the graph of φk . Here xk and x̄k are a priori known bounds on xk when x0 ∈ X . The problem in ( 4 ) is non-convex due to presence of nonlinear operators in the network , such as activation layers . By overapproximating Gφk by a convex set ( or ideally by its convex hull ) , we arrive at a direct layer-wise convex relaxation of the problem . However , solving this relaxation directly can not scale to even medium-sized neural networks Salman et al . ( 2019 ) . In this section , by exploiting the sequential structure of the constraints , we propose a reformulation of ( 3 ) whose convex relaxation can be solved efficiently and in a scalable manner .

Summary: The authors present DeepSplit, a novel solver for a popular neural network convex relaxation. Relying on ADMM and on a careful problem re-formulation, the authors achieve both a O(1/T) converge rate and closed-form solutions for the inner problems, differently from previously presented solvers for the same relaxation. Computational results are presented for incomplete neural network verification, showing that DeeSplit scales to a ResNet18 and achieves better bounds in the same time compared to relevant baselines.