Lipschitz-Bounded Equilibrium Networks

1 INTRODUCTION . Deep neural network models have revolutionized the field of machine learning : their accuracy on practical tasks such as image classification and their scalability have led to an enormous volume of research on different model structures and their properties ( LeCun et al. , 2015 ) . In particular , deep residual networks with skip connections He et al . ( 2016 ) have had a major impact , and neural ODEs have been proposed as an analog with “ implicit depth ” ( Chen et al. , 2018 ) . Recently , a new structure has gained interest : equilibrium networks ( Bai et al. , 2019 ; Winston & Kolter , 2020 ) , a.k.a . implicit deep learning models ( El Ghaoui et al. , 2019 ) , in which model outputs are defined by implicit equations incorporating neural networks . This model class is very flexible : it is easy to show that includes many previous structures as special cases , including standard multi-layer networks , residual networks , and ( in a certain sense ) neural ODEs . However model flexibility in machine learning is always in tension with model regularity or robustness . While deep learning models have exhibited impressive generalisation performance in many contexts it has also been observed that they can be very brittle , especially when targeted with adversarial attacks ( Szegedy et al. , 2014 ) . In response to this , there has been a major research effort to understand and certify robustness properties of deep neural networks , e.g . Raghunathan et al . ( 2018a ) ; Tjeng et al . ( 2018 ) ; Liu et al . ( 2019 ) ; Cohen et al . ( 2019 ) and many others . Global Lipschitz bounds ( a.k.a . incremental gain bounds ) provide a somewhat crude but nevertheless highly useful proxy for robustness ( Tsuzuku et al. , 2018 ; Fazlyab et al. , 2019 ) , and appear in several analyses of generalization ( e.g . ( Bartlett et al. , 2017 ; Zhou & Schoellig , 2019 ) ) . Inspired by both of these lines of research , in this paper we propose new parameterizations of equilibrium networks with guaranteed Lipschitz bounds . We build directly on the monotone operator framework of Winston & Kolter ( 2020 ) and the work of Fazlyab et al . ( 2019 ) on Lipschitz bounds . The main contribution of our paper is the ability to enforce tight bounds on the Lipschitz constant of an equilibrium network during training with essentially no extra computational effort . In addition , we prove existence of solutions with less restrictive conditions on the weight matrix and more natural assumptions on the activation functions via novel connections to convex optimization and contracting dynamical systems . Finally , we show via small-scale image classification experiments that the proposed parameterizations can provide significant improvement in robustness to adversarial attacks with little degradation in nominal accuracy . Furthermore , we observe small gaps between certified Lipschitz upper bounds and observed lower bounds computed via adversarial attack . 2 RELATED WORK . Equilibrium networks , Implicit Deep Models , and Well-Posedness . As mentioned above , it has been recently shown that many existing network architectures can be incorporated into a flexible model set called an equilibrium network ( Bai et al. , 2019 ; Winston & Kolter , 2020 ) or implicit deep model ( El Ghaoui et al. , 2019 ) . In this unified model set , the network predictions are made not by forward computation of sequential hidden layers , but by finding a solution to an implicit equation involving a single layer of all hidden units . One major question for this type of networks is its wellposedness , i.e . the existence and uniqueness of a solution to the implicit equation for all possible inputs . El Ghaoui et al . ( 2019 ) proposed a computationally verifiable but conservative condition on the spectral norm of hidden unit weight . In Winston & Kolter ( 2020 ) , a less conservative condition was developed based on monotone operator theory . Similar monotonicity constraints were previously used to ensure well-posedness of a different class of implicit models in the context of nonlinear system identification ( Tobenkin et al. , 2017 , Theorem 1 ) . On the question of well-posedness , our contribution is a more flexible model set and more natural assumptions on the activation functions : that they are monotone and slope-restricted . Neural Network Robustness and Lipschitz Bounds . The Lipschitz constant of a function measures the worst-case sensitivity of the function , i.e . the maximum “ amplification ” of difference in inputs to differences in outputs . The key features of a good Lipschitz bounded learning approach include a tight estimation for Lipschitz constant and a computationally tractable training method with bounds enforced . For deep networks , Tsuzuku et al . ( 2018 ) proposed a computationally efficient but conservative approach since its Lipschitz constant estimation method is based on composition of estimates for different layers , while Anil et al . ( 2019 ) proposed a combination of a novel activation function and weight constraints . For equilibrium networks , El Ghaoui et al . ( 2019 ) proposed an estimation of Lipschitz bounds via input-to-state ( ISS ) stability analysis . Fazlyab et al . ( 2019 ) estimates for deep networks based on incremental quadratic constraints and semidefinite programming ( SDP ) were shown to give state-of-the-art results , however this was limited to analysis of an already-trained network . The SDP test was incorporated into training via the alternating direction method of multipliers ( ADMM ) in Pauli et al . ( 2020 ) , however due to the complexity of the SDP the training times recorded were almost 50 times longer than for unconstrained networks . Our approach uses a similar condition to Fazlyab et al . ( 2019 ) applied to equilibrium networks , however we introduce a novel direct parameterization method that enables learning robust models via unconstrained optimization , removing the need for computationally-expensive projections or barrier terms . 3 PROBLEM FORMULATION AND PRELIMINARIES . 3.1 PROBLEM STATEMENT . We consider the weight-tied network in which x ∈ Rd denotes the input , and z ∈ Rn denotes the hidden units , y ∈ Rp denotes the output , given by the following implicit equation z = σ ( Wz + Ux+ bz ) , y = Woz + by ( 1 ) where W ∈ Rn×n , U ∈ Rn×d , and Wo ∈ Rp×n are the hidden unit , input , and output weights , respectively , bz ∈ Rn and by ∈ Rp are bias terms . The implicit framework includes most current neural network architectures ( e.g . deep and residual networks ) as special cases . To streamline the presentation we assume that σ : R → R is a single nonlinearity applied elementwise , although our results also apply in the case that each channel has a different activation function , nonlinear or linear . Equation ( 1 ) is called an equilibrium network since its solutions are equilibrium points of the difference equation zk+1 = σ ( Wzk +Ux+ bz ) or the ODE ż ( t ) = −z ( t ) + σ ( Wz ( t ) +Ux+ bz ) . Our goal is to learn equilibrium networks ( 1 ) possessing the following two properties : • Well-posedness : For every input x and bias bz , equation 1 admits a unique solution z . • γ-Lipschitz : It has a finite Lipschitz bound of γ , i.e. , for any input-output pairs ( x1 , y1 ) , ( x2 , y2 ) we have ‖y1 − y2‖2 ≤ γ‖x1 − x2‖2 . 3.2 PRELIMINARIES . Monotone operator theory . The theory of monotone operators on Euclidean space ( see the survey Ryu & Boyd ( 2016 ) ) has been extensively applied in the development of equilibrium networks ( Winston & Kolter , 2020 ) . In this paper , we will use the monotone operator theory on non-Euclidean spaces ( Bauschke et al. , 2011 ) , in particular , we are interested in a finite-dimensional Hilbert space H , which we identify with Rn equipped with a weighted inner product 〈x , y〉Q : = y > Qx where Q 0 . The main benefit is that we can construct a more expressive equilibrium network set . A brief summary or relevant theory can be found in Appendix C.1 ; here we give some definitions that are frequently used throughout the paper . An operator is a set-valued or single-valued function defined by a subset of the space A ⊆ H×H . A function f : H → R ∪ { ∞ } is proper if f ( x ) < ∞ for at least one x . The subdifferential and proximal operators of a proper function f are defined as ∂f ( x ) : = { g ∈ H | f ( y ) ≥ f ( x ) + 〈y − x , g〉Q , ∀y ∈ H } , proxαf ( x ) : = { z ∈ H | z = arg min u 1 2 ‖u− x‖2Q + αf ( u ) } respectively , where ‖x‖Q : = √ 〈x , x〉Q is the induced norm . For n = 1 , we only consider the case ofQ = 1 . An operatorA is monotone if 〈u−v , x−y〉Q ≥ 0 and strongly monotone with parameter m if 〈u− v , x− y〉Q ≥ m‖x− y‖2Q for all ( x , u ) , ( y , v ) ∈ A . The operator splitting problem is that of finding a zero of a sum of two operators A and B , i.e . find an x such that 0 ∈ ( A+B ) ( x ) . Dynamical systems theory . In this paper , we will also treat the solutions of ( 1 ) as equilibrium points of certain dynamical systems ż ( t ) = f ( z ( t ) ) . Then , the well-posedness and robustness properties of ( 1 ) can be guaranteed by corresponding properties of the dynamical system ’ s solution set . A central focus in robust and nonlinear control theory for more than 50 years – and largely unified by the modern theory of integral quadratic constraints ( Megretski & Rantzer , 1997 ) – has been on systems which are interconnections of linear mappings and “ simple ” nonlinearities , i.e . those easily bounded in some sense by quadratic functions . Fortuitously , this characteristic is shared with deep , recurrent , and equilibrium neural networks , a connection that we use heavily in this paper and has previously been exploited by Fazlyab et al . ( 2019 ) ; El Ghaoui et al . ( 2019 ) ; Revay et al . ( 2020 ) and others . A particular property we are interested in is called contraction ( Lohmiller & Slotine , 1998 ) , i.e. , any pair of solutions z1 ( t ) and z2 ( t ) exponentially converge to each other : ‖z1 ( t ) − z2 ( t ) ‖ ≤ α‖z1 ( 0 ) − z2 ( 0 ) ‖e−βt for all t > 0 and some α , β > 0 . Contraction can be established by finding a Riemannian metric with respect to which nearby trajectories converge , which is a differential analog of a Lyapunov function . A nice property of a contracting dynamical system is that if it is time-invariant , a unique equilibrium exists and it possesses a certain level of robustness . Moreover , contraction can also be linked to monotone operators , i.e . a system is contracting w.r.t . to a constant ( state-independent ) metric Q if and only if the operator −f is strongly monotone w.r.t . Q-weighted inner product . We collect some directly relevant results from systems theory in Appendix C.2 .

Summary: The paper introduces a new condition for showing the existence of the solution of a deep equilibrium model (which defines an implicit mapping via the fixed point). The new formulation also comes with a convenient and accurate Lipschitz bound. The proposed condition can be satisfied via reparameterizing an unconstrained set of trainable parameters.