Semialgebraic Representation of Monotone Deep Equilibrium Models and Applications to Certification

1 Introduction . With the increasing success of Deep Neural Networks ( DNN ) ( e.g . computer vision , natural language processing ) , one witnesses a significant increase in size and complexity ( topology and activation functions ) . This generates difficulties for theoretical analysis and a posteriori performance evaluation . This is problematic for applications where robustness issues are crucial , for example inverse problems ( IP ) in scientific computing . Indeed such IPs are notoriously ill-posed and as stressed in the March 2021 issue of SIAM News [ 1 ] , “ Yet DL has an Achilles ’ heel . Current implementations can be highly unstable , meaning that a certain small perturbation to the input of a trained neural network can cause substantial change in its output . This phenomenon is both a nuisance and a major concern for the safety and robustness of DL-based systems in critical applications—like healthcare—where reliable computations are essential '' . Indeed , the Instability Theorem [ 1 ] predicts unavoidable lower bound on Lipschitz contants , which may explain the lack of stability of some DNNs , over-performing on training sets . This underlines the need to evaluate precisely a posteriori critical indicators , such as Lipschitz constants of DNNs . However , obtaining an accurate upper bounds on the Lipschitz constant of a DNN is a hard problem , it reduces to prove a global inequality “ Ψ ( x ) ≥ 0 for all x in a domain ” , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . i.e. , to provide a proof of positivity for a function Ψ , which has no simple explicit expression . Even for modest size DNNs this task is practically challenging , previous successful attempts [ 8 , 28 ] were restricted in practice to no more than two hidden layers with less than a hundred nodes . More broadly existing attempts to DNN certification rely either on zonotope calculus , linear programming ( LP ) or hierarchies of SDP based on positivity certificates from algebraic geometry [ 34 ] , which may suffer from the curse of dimensionality . Recently , implicit deep learning [ 12 ] arises as a generalization of the recursive rules of traditional feedforward neural networks . Specifically , Deep Equilibrium Models ( DEQ ) [ 2 ] have emerged as a potential alternative to classical DNNs . With their much simpler layer structure , they provide competitive results on machine learning benchmarks [ 2 , 3 ] . Training DEQs involves solving fix-point equations , which requires further conditions to make the iteration converge . Fortunately , Monotone operator equilibrium network ( monDEQ ) introduced in [ 45 ] satisfies such conditions . Moreover , the authors in [ 33 ] provide explicit bounds on global Lipschitz constant of monDEQs ( w.r.t . the L2-norm ) which can be used for robustness certification . From a certification point of view , DEQs have the definite advantage of having only one implicit layer compared to DNNs and therefore is potentially more amenable to sophisticated techniques ( e.g . algebraic certificates of positivity ) which rapidly face their limit even for classical DNNs with modest size ( but long depth ) . Therefore monDEQs constitute a class of DNNs for which robustness certification , uncertainty propagation or Lipschicity could potentially be investigated in a more satisfactory way than classical networks . Contrary to DNNs , for which a variety of tools have been developped , certification of DEQ modeld is relatively open , the only available tool is the Lipschitz bound in [ 33 ] . Contribution We present three general semialgebraic models of ReLU monDEQ for certification ( p ∈ Z+∪ { +∞ } ) : • Robustness Model for network Lp robustness certification . • Lipschitz Model for network Lipschitz constant estimation with respect to any Lp norm . • Ellipsoid Model for ellipsoidal outer-approximation of the image by the network of a polyhedra or an ellipsoid . All these models can be used for robustness certification , a common task which we consider experimentally . And the main originality of this work is to successfully apply the proposed techniques to monDEQ networks ( especially for the Ellipsoid model which is a novel form of reachability analysis ) , which was not proposed before . Both Lipschitz and Ellipsoid models can in addition be used for further a posteriori analyses . Interestingly , all three models are given by solutions of semidefinite programs ( SDP ) , obtained by Shor relaxation of a common semialgebraic representation of ReLU monDEQs . Our models are all evaluated to simple ReLU monDEQs on MNIST dataset similar as [ 45 , 33 ] on the task of robustness certification . We demonstrate that all three models ourperform the approach of [ 33 ] and the Robustness Model being the most efficient . Our experiments also suggest that DEQs are much less robust to L∞ perturbations than L2 perturbations , in contrast with classical DNNs [ 36 ] which have been shown to be robust on MNIST dataset to the level we are considering ( ε = 0.1 ) . Related works Neural network certification is a challenging topic in machine learning , contributions include : Robustness certification of DNNs Even with high test accuracy , DNNs are very sensitive to tiny input perturbations , see e.g . [ 42 , 16 ] . Robustness to input perturbation has been investigated in many different works with various techniques , including SDP relaxation in SDP-cert [ 36 ] , first-order dual SDP method [ 10 ] , abstract interpretation with ERAN [ 15 , 41 ] , multi-neuron convex relaxations with PRIMA [ 31 ] , GPU-based method GPUPoly [ 30 ] which can scale to large networks with one million neurons and 34 layers , scalable quantitative verification framework with PROVERO [ 4 ] , LP relaxation in Reluplex [ 25 ] , analytical certification with Fast-lin [ 44 ] and CROWN [ 47 ] , and their extension to convolutional neural networks with CNN-Cert [ 7 ] . All these methods are restricted to DNNs or CNNs and do not directly apply to DEQs . Lipschitz constant estimation of DNNs Lipschitz constant of DNNs can be used for robustness certification [ 42 , 18 ] . Existing contributions include naive layer-wise product bound [ 22 , 43 ] , controlled ordinary differential equations ( ODE ) method [ 20 ] , zonotope-based method [ 24 ] which can be further extended to generative models , Mixed-Integer Programming ( MIP ) [ 23 ] for exactly computing the local Lipschitz constant of ReLU networks , the LP relaxations for CNNs [ 48 ] and DNNs [ 28 ] , as well as SDP relaxations [ 35 , 14 , 8 ] . Lipschitz constant estimation of DEQs The authors in [ 33 ] provide an optimization-free upper bound of the Lipschitz constant of monDEQs depending on network weights . This bound is valid for L2 norm and we present a general model for arbitrary Lp norm . Based on the works of [ 33 ] , the authors in [ 38 ] introduced new parameterizations of DEQs that admit a Lipschitz bound during training via unconstrained optimization . 2 Preliminary Background and Notations . We consider the monotone operator equilibrium network ( monDEQ ) [ 45 ] , with a single implicit hidden layer . The main difference between monDEQ and deep equilibrium network ( DEQ ) [ 2 ] is that strong monotonicity is enforced on weight matrix and activation functions to guarantee the convergence of fixed point iterations . The authors in [ 45 ] proposed various structures of implicit layer , we only consider fully-connected layers , investigation of more advanced convolutional layers is in our list of future works . Network description : Denote by F : Rp0 → RK a fully-connected monDEQ for classification , where p0 is the input dimension and K is the number of labels . Let x0 ∈ Rp0 be the input variable and z ∈ Rp be the variable in the implicit layer . We consider the ReLU activation function , which is simply defined as ReLU ( x ) = max { 0 , x } , and the output of the monDEQs can be written as F ( x0 ) = Cz + c , z = ReLU ( Wz + Ux0 + u ) , ( monDEQ ) where W ∈ Rp×p , U ∈ Rp×p0 , u ∈ Rp , C ∈ RK×p , c ∈ RK are parameters of the network . The vector-valued function F ( x0 ) provides a score for each label i ∈ { 1 , . . . , K } associated to the input x0 , the prediction corresponds to the highest score , i.e. , yx0 = arg maxi=1 , ... , K F ( x0 ) i . As in [ 45 ] , the matrix Ip −W is strongly monotone : there is a known m > 0 such that Ip −W mIp , this constraint can be enforced by specific parametrization of the matrix W. With the monotonicity assumption , the solution to equation z = ReLU ( Wz + Ux0 + u ) is unique and can be evaluated using convergent algorithms , see [ 45 ] for more details . Robustness of monDEQs : Given an input x0 ∈ Rp0 , a norm ‖ · ‖ , and a network F : Rp0 → RK , let y0 be the label of input x0 , i.e. , y0 = arg maxi=1 , ... , K F ( x0 ) i . For ε > 0 , denote by E = B ( x0 , ε , ‖ · ‖ ) the ball centered at x0 with radius ε for norm ‖ · ‖ . If for all inputs x ∈ E , the label of x equals y0 , i.e. , y = arg maxi=1 , ... , K F ( x ) i = y0 , then we say that the network F is ε-robust at input x0 for norm ‖ · ‖ . An equivalent way to verify whether the network F is ε-robust is to check that for all labels i 6= y0 , F ( x ) i − F ( x ) y0 < 0 . Semialgebraicity of ReLU function : The key reason why neural networks with ReLU activation function can be tackled using polynomial optimization techniques is semialgebraicity of the ReLU function , i.e. , it can be expressed with a system of polynomial ( in ) equalities . For x , y ∈ R , we have y = ReLU ( x ) = max { 0 , x } if and only if y ( y−x ) = 0 , y ≥ x , y ≥ 0 . For x , y ∈ Rn , we denote by ReLU ( x ) the coordinate-wise evaluation of ReLU function , and by xy the coordinate-wise product of x and y . A subset of Rn defined by a finite conjunction of polynomial ( in ) equalities is called a basic closed semialgebraic set . The graph of the ReLU function is a basic closed semialgebraic set . Going back to equation ( monDEQ ) , we have the following equivalence : z = ReLU ( Wz + Ux0 + u ) ⇔ z ( z−Wz−Ux0 − u ) = 0 , z ≥Wz + Ux0 + u , z ≥ 0 , ( 1 ) where the right hand side is a system of polynomial ( in ) equalities . For the rest of the paper , mention of the ReLU function will refer to the equivalent polynomial system in ( 1 ) . POP and Putinar ’ s positivity certificate : In general , a polynomial optimization problem ( POP ) has the form ρ = max x∈Rn { f ( x ) : gi ( x ) ≥ 0 , i = 1 , . . . , p } , ( POP0 ) where f and gi are polynomials whose degree is denoted by deg . The robustness certification model ( 3.1 ) , Lipischitz constant model ( 3.2 ) and ellipsoid model ( 3.3 ) are all POPs . In most cases , the POPs are non-linear and non-convex problems , which makes them NP-hard . A typical approach to reduce the complexity of these problems is replacing the positivity constraints by Putinar ’ s positivity certificate [ 34 ] . The problem ( POP0 ) is equivalent to ρ = min λ∈R { λ : λ− f ( x ) ≥ 0 , gi ( x ) ≥ 0 , i = 1 , . . . , p , ∀x ∈ Rn } . ( POP ) In order to reduce the size of the feasible set of problem ( POP ) , we replace the positivity constraint λ− f ( x ) ≥ 0 by a weighted sum-of-square ( SOS ) polynomial decomposition , involving the polynomials gi . Let d be a non-negative integer . Denote by σd0 ( x ) , σ d i ( x ) some SOS polynomials of degree at most 2d , for each i = 1 , . . . , p. Note that if d = 0 , such polynomials are non-negative real numbers . Then the positivity of λ− f ( x ) is implied by the following decomposition λ− f ( x ) = σd0 ( x ) + p∑ i=1 σd−ωii ( x ) gi ( x ) , ωi = ddeg gi/2e , ∀x ∈ R n , ( 2 ) for any d ≥ maxi ωi . Equation ( 2 ) is called the order-d Putinar ’ s certificate . By replacing the positivity constraint f ( x ) − λ ≥ 0 in problem ( POP ) by Putinar ’ s certificate ( 2 ) , we have for d ≥ maxi ωi , ρd = min λ∈R { λ : λ− f ( x ) = σd0 ( x ) + p∑ i=1 σd−ωii ( x ) gi ( x ) , ωi = ddeg gi/2e , ∀x ∈ R n } . ( POP-d ) It is obvious that ρd ≥ ρ for all d ≥ maxi ωi . Under certain conditions ( slightly stronger than compactness of the set of constraints ) , it is shown that limd→∞ ρd = ρ [ 27 ] . The main advantage of relaxing problem ( POP ) to ( POP-d ) is that problem ( POP-d ) can be efficiently solved by semidefinite programming ( SDP ) . Indeed a polynomial f of degree at most 2d is SOS if and only if there exists a positive semidefinte ( PSD ) matrix M ( called a Gram matrix ) such that f ( x ) = v ( x ) TMv ( x ) , for all x ∈ Rp , where v ( x ) is the vector of monomials of degree at most d. Problem ( POP-d ) is also called the order-d Lasserre ’ s relaxation . When the input polynomials are quadratic , the order-1 Lasserre ’ s relaxation is also known as Shor ’ s relaxation [ 40 ] . All our models are obtained using variations of Shor ’ s relaxation applied to different POPs , see Section 3.3 for more details .

Summary: The authors propose a framework for the robustness verification of monDEQs (monotone operator deep equilibrium networks). In this framework the paper introduces three different methods: one for direct robustness verification, one for Lipschitz constant estimation and one to compute the ellipsoid over-approximating all possible outputs for a set of inputs (in a norm ball). The authors also show how the Lipschitz constant and ellipsoid in turn can be used for robustness verification.  Mathematically this is enabled encoding the monDEQ as the constraints of a (polynomial) optimization problem. In particular, the fix-point $z = ReLU(Wz + Ux_{0} + u)$ is encoded using the semialgebracity of ReLU, similar to [27]*. This encoding as well as encoding $\ell_{2}$ and $\ell_{\infty}$ inputs yields quadratic constraints. (Other $\ell_{p}$-norms $p>2$ yield higher-order constraints.) The resulting constraints are relaxed via Puntinar's positivity certificate and Lasserre's/Shor's relaxations into a SDP (semidefinite program), e.g. approximated lower-degree polynomials.  For the robustness verification approach the authors follow [27] and maximize the difference between any logit and the target logit. If all of these differences are $< 0$ a robustness certificate is obtained. To make the resulting problem tractable Shor's relaxation is applied. The Lipschitz approach bounds the Lipschitz constant by bounding the supremum of the operator norm of the generalized Jacobian. This in turn, via Shor's relaxation gives an SDP. The Ellipsoid model minimizes the volume of an ellipsoid that covers the output. Via a custom relaxation this also yields an SDP.  All methods are evaluated on an MNIST classifier with a single feed-forward fix-point layer where $z \in \mathbb{R}^{87}$. All three approaches allow for robustness certification (although the robustness formulation performs the best). The computed Lipschitz bound are close to those obtained by [24] for the $\ell_{2}$-norm but much tighter for the $\ell_{\infty}$-norm. Lastly, in an example the authors showcase how the output-ellipsoid method allows insights into the model robustness.