Semialgebraic Representation of Monotone Deep Equilibrium Models and Applications to Certification

1 Introduction . With the increasing success of Deep Neural Networks ( DNN ) ( e.g . computer vision , natural language processing ) , one witnesses a significant increase in size and complexity ( topology and activation functions ) . This generates difficulties for theoretical analysis and a posteriori performance evaluation . This is problematic for applications where robustness issues are crucial , for example inverse problems ( IP ) in scientific computing . Indeed such IPs are notoriously ill-posed and as stressed in the March 2021 issue of SIAM News [ 1 ] , “ Yet DL has an Achilles ’ heel . Current implementations can be highly unstable , meaning that a certain small perturbation to the input of a trained neural network can cause substantial change in its output . This phenomenon is both a nuisance and a major concern for the safety and robustness of DL-based systems in critical applications—like healthcare—where reliable computations are essential '' . Indeed , the Instability Theorem [ 1 ] predicts unavoidable lower bound on Lipschitz contants , which may explain the lack of stability of some DNNs , over-performing on training sets . This underlines the need to evaluate precisely a posteriori critical indicators , such as Lipschitz constants of DNNs . However , obtaining an accurate upper bounds on the Lipschitz constant of a DNN is a hard problem , it reduces to prove a global inequality “ Ψ ( x ) ≥ 0 for all x in a domain ” , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . i.e. , to provide a proof of positivity for a function Ψ , which has no simple explicit expression . Even for modest size DNNs this task is practically challenging , previous successful attempts [ 8 , 28 ] were restricted in practice to no more than two hidden layers with less than a hundred nodes . More broadly existing attempts to DNN certification rely either on zonotope calculus , linear programming ( LP ) or hierarchies of SDP based on positivity certificates from algebraic geometry [ 34 ] , which may suffer from the curse of dimensionality . Recently , implicit deep learning [ 12 ] arises as a generalization of the recursive rules of traditional feedforward neural networks . Specifically , Deep Equilibrium Models ( DEQ ) [ 2 ] have emerged as a potential alternative to classical DNNs . With their much simpler layer structure , they provide competitive results on machine learning benchmarks [ 2 , 3 ] . Training DEQs involves solving fix-point equations , which requires further conditions to make the iteration converge . Fortunately , Monotone operator equilibrium network ( monDEQ ) introduced in [ 45 ] satisfies such conditions . Moreover , the authors in [ 33 ] provide explicit bounds on global Lipschitz constant of monDEQs ( w.r.t . the L2-norm ) which can be used for robustness certification . From a certification point of view , DEQs have the definite advantage of having only one implicit layer compared to DNNs and therefore is potentially more amenable to sophisticated techniques ( e.g . algebraic certificates of positivity ) which rapidly face their limit even for classical DNNs with modest size ( but long depth ) . Therefore monDEQs constitute a class of DNNs for which robustness certification , uncertainty propagation or Lipschicity could potentially be investigated in a more satisfactory way than classical networks . Contrary to DNNs , for which a variety of tools have been developped , certification of DEQ modeld is relatively open , the only available tool is the Lipschitz bound in [ 33 ] . Contribution We present three general semialgebraic models of ReLU monDEQ for certification ( p ∈ Z+∪ { +∞ } ) : • Robustness Model for network Lp robustness certification . • Lipschitz Model for network Lipschitz constant estimation with respect to any Lp norm . • Ellipsoid Model for ellipsoidal outer-approximation of the image by the network of a polyhedra or an ellipsoid . All these models can be used for robustness certification , a common task which we consider experimentally . And the main originality of this work is to successfully apply the proposed techniques to monDEQ networks ( especially for the Ellipsoid model which is a novel form of reachability analysis ) , which was not proposed before . Both Lipschitz and Ellipsoid models can in addition be used for further a posteriori analyses . Interestingly , all three models are given by solutions of semidefinite programs ( SDP ) , obtained by Shor relaxation of a common semialgebraic representation of ReLU monDEQs . Our models are all evaluated to simple ReLU monDEQs on MNIST dataset similar as [ 45 , 33 ] on the task of robustness certification . We demonstrate that all three models ourperform the approach of [ 33 ] and the Robustness Model being the most efficient . Our experiments also suggest that DEQs are much less robust to L∞ perturbations than L2 perturbations , in contrast with classical DNNs [ 36 ] which have been shown to be robust on MNIST dataset to the level we are considering ( ε = 0.1 ) . Related works Neural network certification is a challenging topic in machine learning , contributions include : Robustness certification of DNNs Even with high test accuracy , DNNs are very sensitive to tiny input perturbations , see e.g . [ 42 , 16 ] . Robustness to input perturbation has been investigated in many different works with various techniques , including SDP relaxation in SDP-cert [ 36 ] , first-order dual SDP method [ 10 ] , abstract interpretation with ERAN [ 15 , 41 ] , multi-neuron convex relaxations with PRIMA [ 31 ] , GPU-based method GPUPoly [ 30 ] which can scale to large networks with one million neurons and 34 layers , scalable quantitative verification framework with PROVERO [ 4 ] , LP relaxation in Reluplex [ 25 ] , analytical certification with Fast-lin [ 44 ] and CROWN [ 47 ] , and their extension to convolutional neural networks with CNN-Cert [ 7 ] . All these methods are restricted to DNNs or CNNs and do not directly apply to DEQs . Lipschitz constant estimation of DNNs Lipschitz constant of DNNs can be used for robustness certification [ 42 , 18 ] . Existing contributions include naive layer-wise product bound [ 22 , 43 ] , controlled ordinary differential equations ( ODE ) method [ 20 ] , zonotope-based method [ 24 ] which can be further extended to generative models , Mixed-Integer Programming ( MIP ) [ 23 ] for exactly computing the local Lipschitz constant of ReLU networks , the LP relaxations for CNNs [ 48 ] and DNNs [ 28 ] , as well as SDP relaxations [ 35 , 14 , 8 ] . Lipschitz constant estimation of DEQs The authors in [ 33 ] provide an optimization-free upper bound of the Lipschitz constant of monDEQs depending on network weights . This bound is valid for L2 norm and we present a general model for arbitrary Lp norm . Based on the works of [ 33 ] , the authors in [ 38 ] introduced new parameterizations of DEQs that admit a Lipschitz bound during training via unconstrained optimization . 2 Preliminary Background and Notations . We consider the monotone operator equilibrium network ( monDEQ ) [ 45 ] , with a single implicit hidden layer . The main difference between monDEQ and deep equilibrium network ( DEQ ) [ 2 ] is that strong monotonicity is enforced on weight matrix and activation functions to guarantee the convergence of fixed point iterations . The authors in [ 45 ] proposed various structures of implicit layer , we only consider fully-connected layers , investigation of more advanced convolutional layers is in our list of future works . Network description : Denote by F : Rp0 → RK a fully-connected monDEQ for classification , where p0 is the input dimension and K is the number of labels . Let x0 ∈ Rp0 be the input variable and z ∈ Rp be the variable in the implicit layer . We consider the ReLU activation function , which is simply defined as ReLU ( x ) = max { 0 , x } , and the output of the monDEQs can be written as F ( x0 ) = Cz + c , z = ReLU ( Wz + Ux0 + u ) , ( monDEQ ) where W ∈ Rp×p , U ∈ Rp×p0 , u ∈ Rp , C ∈ RK×p , c ∈ RK are parameters of the network . The vector-valued function F ( x0 ) provides a score for each label i ∈ { 1 , . . . , K } associated to the input x0 , the prediction corresponds to the highest score , i.e. , yx0 = arg maxi=1 , ... , K F ( x0 ) i . As in [ 45 ] , the matrix Ip −W is strongly monotone : there is a known m > 0 such that Ip −W mIp , this constraint can be enforced by specific parametrization of the matrix W. With the monotonicity assumption , the solution to equation z = ReLU ( Wz + Ux0 + u ) is unique and can be evaluated using convergent algorithms , see [ 45 ] for more details . Robustness of monDEQs : Given an input x0 ∈ Rp0 , a norm ‖ · ‖ , and a network F : Rp0 → RK , let y0 be the label of input x0 , i.e. , y0 = arg maxi=1 , ... , K F ( x0 ) i . For ε > 0 , denote by E = B ( x0 , ε , ‖ · ‖ ) the ball centered at x0 with radius ε for norm ‖ · ‖ . If for all inputs x ∈ E , the label of x equals y0 , i.e. , y = arg maxi=1 , ... , K F ( x ) i = y0 , then we say that the network F is ε-robust at input x0 for norm ‖ · ‖ . An equivalent way to verify whether the network F is ε-robust is to check that for all labels i 6= y0 , F ( x ) i − F ( x ) y0 < 0 . Semialgebraicity of ReLU function : The key reason why neural networks with ReLU activation function can be tackled using polynomial optimization techniques is semialgebraicity of the ReLU function , i.e. , it can be expressed with a system of polynomial ( in ) equalities . For x , y ∈ R , we have y = ReLU ( x ) = max { 0 , x } if and only if y ( y−x ) = 0 , y ≥ x , y ≥ 0 . For x , y ∈ Rn , we denote by ReLU ( x ) the coordinate-wise evaluation of ReLU function , and by xy the coordinate-wise product of x and y . A subset of Rn defined by a finite conjunction of polynomial ( in ) equalities is called a basic closed semialgebraic set . The graph of the ReLU function is a basic closed semialgebraic set . Going back to equation ( monDEQ ) , we have the following equivalence : z = ReLU ( Wz + Ux0 + u ) ⇔ z ( z−Wz−Ux0 − u ) = 0 , z ≥Wz + Ux0 + u , z ≥ 0 , ( 1 ) where the right hand side is a system of polynomial ( in ) equalities . For the rest of the paper , mention of the ReLU function will refer to the equivalent polynomial system in ( 1 ) . POP and Putinar ’ s positivity certificate : In general , a polynomial optimization problem ( POP ) has the form ρ = max x∈Rn { f ( x ) : gi ( x ) ≥ 0 , i = 1 , . . . , p } , ( POP0 ) where f and gi are polynomials whose degree is denoted by deg . The robustness certification model ( 3.1 ) , Lipischitz constant model ( 3.2 ) and ellipsoid model ( 3.3 ) are all POPs . In most cases , the POPs are non-linear and non-convex problems , which makes them NP-hard . A typical approach to reduce the complexity of these problems is replacing the positivity constraints by Putinar ’ s positivity certificate [ 34 ] . The problem ( POP0 ) is equivalent to ρ = min λ∈R { λ : λ− f ( x ) ≥ 0 , gi ( x ) ≥ 0 , i = 1 , . . . , p , ∀x ∈ Rn } . ( POP ) In order to reduce the size of the feasible set of problem ( POP ) , we replace the positivity constraint λ− f ( x ) ≥ 0 by a weighted sum-of-square ( SOS ) polynomial decomposition , involving the polynomials gi . Let d be a non-negative integer . Denote by σd0 ( x ) , σ d i ( x ) some SOS polynomials of degree at most 2d , for each i = 1 , . . . , p. Note that if d = 0 , such polynomials are non-negative real numbers . Then the positivity of λ− f ( x ) is implied by the following decomposition λ− f ( x ) = σd0 ( x ) + p∑ i=1 σd−ωii ( x ) gi ( x ) , ωi = ddeg gi/2e , ∀x ∈ R n , ( 2 ) for any d ≥ maxi ωi . Equation ( 2 ) is called the order-d Putinar ’ s certificate . By replacing the positivity constraint f ( x ) − λ ≥ 0 in problem ( POP ) by Putinar ’ s certificate ( 2 ) , we have for d ≥ maxi ωi , ρd = min λ∈R { λ : λ− f ( x ) = σd0 ( x ) + p∑ i=1 σd−ωii ( x ) gi ( x ) , ωi = ddeg gi/2e , ∀x ∈ R n } . ( POP-d ) It is obvious that ρd ≥ ρ for all d ≥ maxi ωi . Under certain conditions ( slightly stronger than compactness of the set of constraints ) , it is shown that limd→∞ ρd = ρ [ 27 ] . The main advantage of relaxing problem ( POP ) to ( POP-d ) is that problem ( POP-d ) can be efficiently solved by semidefinite programming ( SDP ) . Indeed a polynomial f of degree at most 2d is SOS if and only if there exists a positive semidefinte ( PSD ) matrix M ( called a Gram matrix ) such that f ( x ) = v ( x ) TMv ( x ) , for all x ∈ Rp , where v ( x ) is the vector of monomials of degree at most d. Problem ( POP-d ) is also called the order-d Lasserre ’ s relaxation . When the input polynomials are quadratic , the order-1 Lasserre ’ s relaxation is also known as Shor ’ s relaxation [ 40 ] . All our models are obtained using variations of Shor ’ s relaxation applied to different POPs , see Section 3.3 for more details .

Summary: Based on semialgebraic of $\text{ReLU}$ and $\partial\text{ReLU}$, this paper proposes three models that are useful for certifying robustness, estimating Lipschitz constants and computing outer approximation ellipsoids.  The authors also use Putinar’s certificates to relax the non-convex problems to convex SDPs which can be solved efficiently. Experiments on MNIST demonstrate the effectiveness of all three models.