DeepSplit: Scalable Verification of Deep Neural Networks via Operator Splitting

1 Introduction . Despite their superior performance , neural networks lack formal guarantees , raising serious concerns about their adoption in safety-critical applications such as autonomous vehicles ( Cao et al. , 2019 ) and medical machine learning ( Finlayson et al. , 2019 ) . Motivated by this drawback , there has been an increasing interest in developing tools to verify desirable properties for neural networks , such as robustness to adversarial attacks . Neural network verification refers to the problem of verifying whether the output of a neural network satisfies certain properties for a bounded set of input perturbations . This problem can be framed as optimization problems of the form minimize J ( f ( x ) ) subject to x ∈ X , ( 1 ) where f is given by a deep neural network , J is a real-valued function representing a performance measure ( or a specification ) , and X is a set of inputs to be verified . In this formulation , verifying the neural network amounts to certifying whether the optimal value of ( 1 ) is bounded below by a certain threshold . This problem is large-scale and non-convex , making it extremely difficult to solve efficiently–both in terms of time and memory . For ReLU activation functions and linear objectives , the problem in ( 1 ) can be cast as a MixedInteger Linear Program ( MILP ) ( Lomuscio & Maganti , 2017 ; Cheng et al. , 2017 ; Dutta et al. , 2018 ; Fischetti & Jo , 2018 ) , which can be solved for the global solution via , for example , Branch-and-Bound ( BaB ) methods . While we do not expect these approaches to scale to large problems , for small neural networks they can still be practical . Instead of solving ( 1 ) for its global minimum , one can instead find guaranteed lower bounds on the optimal value via convex relaxations , such as Linear Programming ( LP ) ( Wong & Kolter , 2017 ) and Semidefinite Programming ( SDP ) ( Raghunathan et al. , 2018 ; Fazlyab et al. , 2019 ; 2020 ) . Verification methods based on convex relaxations are sound but incomplete , i.e. , they are guaranteed to detect all false negatives but also produce false positives , whose rate depends on the tightness of the relaxation . Although convex relaxations are polynomial-time solvable ( in terms of number of decision variables ) , in practice they are not computationally tractable for large-scale neural networks . To improve scalability , these relaxations must typically be further relaxed ( Wong & Kolter , 2017 ; Weng et al. , 2018 ; Zhang et al. , 2018 ; Dvijotham et al. , 2018 ; Bunel et al. , 2020a ) . Contributions In this work , we propose an algorithm for solving convex relaxations exactly for large-scale neural networks . Our starting point is to express ( 1 ) as a constrained optimization problem whose constraints are imposed by the forward passes in the network . We then introduce additional decision variables and consensus constraints that naturally split the corresponding problem into independent subproblems , which often have closed-form solutions . Finally , we employ an operator splitting technique based on the Alternating Direction Method of Multipliers ( ADMM ) ( Boyd et al. , 2011 ) , to solve the corresponding Lagrangian relaxation of the problem . This approach has several favorable properties . First , the method requires minimal parameter tuning and relies on simple operations , which scale to very large problems and can achieve a good trade-off between runtime and solution accuracy . Second , all the solver operations are amenable to fast parallelization with GPU acceleration . Third , our method is fully modular and applies to standard network architectures . We employ our method to compute exact solutions to LP relaxations on the worst-case performance of adversarially trained deep networks , with a focus on networks whose convex relaxations were previously impossible to solve exactly due to their size . Specifically , we perform extensive experiments in the ` ∞ perturbation setting , where we verify robustness properties of image classifiers for CIFAR10 and deep Q-networks ( DQNs ) in Atari games ( Zhang et al. , 2020 ) . Our method is able to quickly solve LP relaxations at scales that are too large for exact BaB verifiers , SDP relaxations , or commercial LP solvers such as Gurobi . 1.1 Related Work . Convex relaxations LP relaxations are relatively the most scalable form of convex relaxations ( Ehlers , 2017 ) . However , even solving LPs can become computationally prohibitive for small convolutional networks ( Salman et al. , 2019 ) . One line of work studies computationally cheaper but looser bounds of the LP relaxation ( Wong & Kolter , 2017 ) , which have been extended to larger and more general networks and settings ( Wong et al. , 2018 ; Weng et al. , 2018 ; Zhang et al. , 2018 ; Xu et al. , 2020 ) . These bounds tend to be loose unless optimized during training , which typically comes at a significant cost to standard performance . Further work has aimed to tighten these bounds ( Singh et al. , 2019 ; Tjandraatmadja et al. , 2020 ; de2 ) , however these works focus primarily on small convolutional networks and struggle to scale to more typical deep networks . Other work has studied the limits of these convex relaxations on these small networks using vast amounts of CPU-compute ( Salman et al. , 2019 ) . Recent SDP-based approaches ( Dathathri et al. , 2020 ) can produce much tighter bounds on these small networks . Lagrangian-based bounds Related to our work is that which solves the Lagrangian of the LP relaxation ( Dvijotham et al. , 2018 ; Bunel et al. , 2020a ) , which can tighten the bound but do not aim to solve the LP exactly due to relatively slow convergence . However , these works primarily study small networks whose LP relaxation can still be solved exactly with Gurobi . Although these works could in theory be used on larger networks , only the faster , linear-based bounds ( Xu et al. , 2020 ) have demonstrated applicability to standard deep learning architectures . In our work , we solve the LP relaxation exactly in large network settings that previously have only been studied with loose bounds of the LP relaxation such as LiRPA ( Xu et al. , 2020 ) . Complete verification methods These methods verify properties of deep networks exactly ( without false positives or false negatives ) using methods such as SMT solvers ( Scheibler et al. , 2015 ; Ehlers , 2017 ; Katz et al. , 2017 ) and MILP solvers ( Lomuscio & Maganti , 2017 ; Cheng et al. , 2017 ; Dutta et al. , 2018 ; Fischetti & Jo , 2018 ) . Complete verification methods typically rely on BaB algorithms ( Bunel et al. , 2017 ) , in which the verification problem is divided into subproblems ( branching ) that can be verified using incomplete verification methods ( bounding ) ( Bunel et al. , 2020b ) . However , these methods have a worst-case exponential runtime and have difficulty scaling beyond relatively small convolutional networks . Motivated by this , several recent works have improved the practical running time of BaB methods with custom solvers for medium-sized networks ( Bunel et al. , 2020a ; De Palma et al. , 2021 ; Xu et al. , 2021 ; Wang et al. , 2021 ) , but have yet to scale to larger deep networks . Operator splitting methods Operator splitting , and in particular the ADMM method , is a powerful technique in solving structured convex optimization problems and has applications in numerous settings ranging from optimal control ( O ’ Donoghue et al. , 2013 ) to training neural networks ( Taylor et al. , 2016 ) . These methods scale well with the problem dimensions , can exploit sparsity in the problem data efficiently ( Zheng et al. , 2017 ) , are amenable to parallelization on GPU ( Schubiger et al. , 2020 ) , and have well-understood convergence properties under minimal regularity assumptions ( Boyd et al. , 2011 ) . The benefit of ADMM as an alternative to interior-point solvers has been shown in various classes of optimization problems ( O ’ donoghue et al. , 2016 ) . Our operator splitting method is specifically tailored for neural network verification in order to fully exploit the problem structure . Notation . We denote the set of real numbers by R , the set of nonnegative real numbers by R+ , the set of real n-dimensional vectors by Rn , the set of m× n-dimensional matrices by Rm×n , and the n-dimensional identity matrix by In . The p-norm ( p ≥ 1 ) is denoted by ‖ · ‖p : Rn → R+ . For a set S , we define the indicator function IS ( x ) of S as IS ( x ) = 0 if x ∈ S and IS ( x ) = +∞ otherwise . Given a function f : X → Y , the graph of f is the set Gf = { ( x , f ( x ) ) | x ∈ X } . 2 Scalable Neural Network Verification via Operator Splitting . We consider an ` -layer feed-forward neural network f ( x ) : Rn0 → Rn ` described by the following recursive equations , x0 = x , xk+1 = φk ( xk ) k = 0 , · · · , ` − 1 , f ( x ) = x ` ( 2 ) where x0 ∈ Rn0 is the input to the neural network , xk ∈ Rnk is the input to the k-th layer , and φk : Rnk → Rnk+1 is the operator of the k-th layer , which can represent any commonly-used operator in deep networks , such as linear ( convolutional ) layers , MaxPooling units , and activation functions . Given the neural network f , a specification function J : Rn ` 7→ R , and an input set X ⊂ Rn0 , we say that f satisfies the specification J if J ( f ( x ) ) ≥ 0 for all x ∈ X . This is equivalent to verifying that the optimal value of ( 1 ) is non-negative . We assume X ⊂ Rn0 is a closed convex set and J : Rn ` → R∪ { +∞ } is a convex function , defining a performance measure on the output of the network . Note that our formulation generalizes to arbitrary computational graphs ( e.g . residual blocks ) and general objective functions ( see Appendix B ) . Using the sequential representation of the neural network in ( 2 ) , we may rewrite the optimization problem in ( 1 ) as the following constrained optimization problem , J ? ← minimize J ( x ` ) subject to xk+1 = φk ( xk ) , k = 0 , · · · , ` − 1 , x0 ∈ X , ( 3 ) with n : = ∑ ` k=0 nk decision variables . We can rewrite ( 3 ) equivalently as J ? ← minimize J ( x ` ) subject to ( xk , xk+1 ) ∈ Gφk , k = 0 , · · · , ` − 1 , x0 ∈ X , ( 4 ) where Gφk = { ( xk , xk+1 ) | xk+1 = φk ( xk ) , xk ≤ xk ≤ x̄k } is the graph of φk . Here xk and x̄k are a priori known bounds on xk when x0 ∈ X . The problem in ( 4 ) is non-convex due to presence of nonlinear operators in the network , such as activation layers . By overapproximating Gφk by a convex set ( or ideally by its convex hull ) , we arrive at a direct layer-wise convex relaxation of the problem . However , solving this relaxation directly can not scale to even medium-sized neural networks Salman et al . ( 2019 ) . In this section , by exploiting the sequential structure of the constraints , we propose a reformulation of ( 3 ) whose convex relaxation can be solved efficiently and in a scalable manner .

Summary: This paper proposes an operator splitting method which solves a convex relaxation of a learge-scale nonconvex optimization problem for analyzing the worst-case performance of deep neural networks against input perturbations. The proposed method is modular, scalable and parallelizable. Experimentally, the authors demonstrate that the proposed method has tighter bounds on the worst-case performance of large CNNs in image classification and RL settings.