Lipschitz-Bounded Equilibrium Networks

1 INTRODUCTION . Deep neural network models have revolutionized the field of machine learning : their accuracy on practical tasks such as image classification and their scalability have led to an enormous volume of research on different model structures and their properties ( LeCun et al. , 2015 ) . In particular , deep residual networks with skip connections He et al . ( 2016 ) have had a major impact , and neural ODEs have been proposed as an analog with “ implicit depth ” ( Chen et al. , 2018 ) . Recently , a new structure has gained interest : equilibrium networks ( Bai et al. , 2019 ; Winston & Kolter , 2020 ) , a.k.a . implicit deep learning models ( El Ghaoui et al. , 2019 ) , in which model outputs are defined by implicit equations incorporating neural networks . This model class is very flexible : it is easy to show that includes many previous structures as special cases , including standard multi-layer networks , residual networks , and ( in a certain sense ) neural ODEs . However model flexibility in machine learning is always in tension with model regularity or robustness . While deep learning models have exhibited impressive generalisation performance in many contexts it has also been observed that they can be very brittle , especially when targeted with adversarial attacks ( Szegedy et al. , 2014 ) . In response to this , there has been a major research effort to understand and certify robustness properties of deep neural networks , e.g . Raghunathan et al . ( 2018a ) ; Tjeng et al . ( 2018 ) ; Liu et al . ( 2019 ) ; Cohen et al . ( 2019 ) and many others . Global Lipschitz bounds ( a.k.a . incremental gain bounds ) provide a somewhat crude but nevertheless highly useful proxy for robustness ( Tsuzuku et al. , 2018 ; Fazlyab et al. , 2019 ) , and appear in several analyses of generalization ( e.g . ( Bartlett et al. , 2017 ; Zhou & Schoellig , 2019 ) ) . Inspired by both of these lines of research , in this paper we propose new parameterizations of equilibrium networks with guaranteed Lipschitz bounds . We build directly on the monotone operator framework of Winston & Kolter ( 2020 ) and the work of Fazlyab et al . ( 2019 ) on Lipschitz bounds . The main contribution of our paper is the ability to enforce tight bounds on the Lipschitz constant of an equilibrium network during training with essentially no extra computational effort . In addition , we prove existence of solutions with less restrictive conditions on the weight matrix and more natural assumptions on the activation functions via novel connections to convex optimization and contracting dynamical systems . Finally , we show via small-scale image classification experiments that the proposed parameterizations can provide significant improvement in robustness to adversarial attacks with little degradation in nominal accuracy . Furthermore , we observe small gaps between certified Lipschitz upper bounds and observed lower bounds computed via adversarial attack . 2 RELATED WORK . Equilibrium networks , Implicit Deep Models , and Well-Posedness . As mentioned above , it has been recently shown that many existing network architectures can be incorporated into a flexible model set called an equilibrium network ( Bai et al. , 2019 ; Winston & Kolter , 2020 ) or implicit deep model ( El Ghaoui et al. , 2019 ) . In this unified model set , the network predictions are made not by forward computation of sequential hidden layers , but by finding a solution to an implicit equation involving a single layer of all hidden units . One major question for this type of networks is its wellposedness , i.e . the existence and uniqueness of a solution to the implicit equation for all possible inputs . El Ghaoui et al . ( 2019 ) proposed a computationally verifiable but conservative condition on the spectral norm of hidden unit weight . In Winston & Kolter ( 2020 ) , a less conservative condition was developed based on monotone operator theory . Similar monotonicity constraints were previously used to ensure well-posedness of a different class of implicit models in the context of nonlinear system identification ( Tobenkin et al. , 2017 , Theorem 1 ) . On the question of well-posedness , our contribution is a more flexible model set and more natural assumptions on the activation functions : that they are monotone and slope-restricted . Neural Network Robustness and Lipschitz Bounds . The Lipschitz constant of a function measures the worst-case sensitivity of the function , i.e . the maximum “ amplification ” of difference in inputs to differences in outputs . The key features of a good Lipschitz bounded learning approach include a tight estimation for Lipschitz constant and a computationally tractable training method with bounds enforced . For deep networks , Tsuzuku et al . ( 2018 ) proposed a computationally efficient but conservative approach since its Lipschitz constant estimation method is based on composition of estimates for different layers , while Anil et al . ( 2019 ) proposed a combination of a novel activation function and weight constraints . For equilibrium networks , El Ghaoui et al . ( 2019 ) proposed an estimation of Lipschitz bounds via input-to-state ( ISS ) stability analysis . Fazlyab et al . ( 2019 ) estimates for deep networks based on incremental quadratic constraints and semidefinite programming ( SDP ) were shown to give state-of-the-art results , however this was limited to analysis of an already-trained network . The SDP test was incorporated into training via the alternating direction method of multipliers ( ADMM ) in Pauli et al . ( 2020 ) , however due to the complexity of the SDP the training times recorded were almost 50 times longer than for unconstrained networks . Our approach uses a similar condition to Fazlyab et al . ( 2019 ) applied to equilibrium networks , however we introduce a novel direct parameterization method that enables learning robust models via unconstrained optimization , removing the need for computationally-expensive projections or barrier terms . 3 PROBLEM FORMULATION AND PRELIMINARIES . 3.1 PROBLEM STATEMENT . We consider the weight-tied network in which x ∈ Rd denotes the input , and z ∈ Rn denotes the hidden units , y ∈ Rp denotes the output , given by the following implicit equation z = σ ( Wz + Ux+ bz ) , y = Woz + by ( 1 ) where W ∈ Rn×n , U ∈ Rn×d , and Wo ∈ Rp×n are the hidden unit , input , and output weights , respectively , bz ∈ Rn and by ∈ Rp are bias terms . The implicit framework includes most current neural network architectures ( e.g . deep and residual networks ) as special cases . To streamline the presentation we assume that σ : R → R is a single nonlinearity applied elementwise , although our results also apply in the case that each channel has a different activation function , nonlinear or linear . Equation ( 1 ) is called an equilibrium network since its solutions are equilibrium points of the difference equation zk+1 = σ ( Wzk +Ux+ bz ) or the ODE ż ( t ) = −z ( t ) + σ ( Wz ( t ) +Ux+ bz ) . Our goal is to learn equilibrium networks ( 1 ) possessing the following two properties : • Well-posedness : For every input x and bias bz , equation 1 admits a unique solution z . • γ-Lipschitz : It has a finite Lipschitz bound of γ , i.e. , for any input-output pairs ( x1 , y1 ) , ( x2 , y2 ) we have ‖y1 − y2‖2 ≤ γ‖x1 − x2‖2 . 3.2 PRELIMINARIES . Monotone operator theory . The theory of monotone operators on Euclidean space ( see the survey Ryu & Boyd ( 2016 ) ) has been extensively applied in the development of equilibrium networks ( Winston & Kolter , 2020 ) . In this paper , we will use the monotone operator theory on non-Euclidean spaces ( Bauschke et al. , 2011 ) , in particular , we are interested in a finite-dimensional Hilbert space H , which we identify with Rn equipped with a weighted inner product 〈x , y〉Q : = y > Qx where Q 0 . The main benefit is that we can construct a more expressive equilibrium network set . A brief summary or relevant theory can be found in Appendix C.1 ; here we give some definitions that are frequently used throughout the paper . An operator is a set-valued or single-valued function defined by a subset of the space A ⊆ H×H . A function f : H → R ∪ { ∞ } is proper if f ( x ) < ∞ for at least one x . The subdifferential and proximal operators of a proper function f are defined as ∂f ( x ) : = { g ∈ H | f ( y ) ≥ f ( x ) + 〈y − x , g〉Q , ∀y ∈ H } , proxαf ( x ) : = { z ∈ H | z = arg min u 1 2 ‖u− x‖2Q + αf ( u ) } respectively , where ‖x‖Q : = √ 〈x , x〉Q is the induced norm . For n = 1 , we only consider the case ofQ = 1 . An operatorA is monotone if 〈u−v , x−y〉Q ≥ 0 and strongly monotone with parameter m if 〈u− v , x− y〉Q ≥ m‖x− y‖2Q for all ( x , u ) , ( y , v ) ∈ A . The operator splitting problem is that of finding a zero of a sum of two operators A and B , i.e . find an x such that 0 ∈ ( A+B ) ( x ) . Dynamical systems theory . In this paper , we will also treat the solutions of ( 1 ) as equilibrium points of certain dynamical systems ż ( t ) = f ( z ( t ) ) . Then , the well-posedness and robustness properties of ( 1 ) can be guaranteed by corresponding properties of the dynamical system ’ s solution set . A central focus in robust and nonlinear control theory for more than 50 years – and largely unified by the modern theory of integral quadratic constraints ( Megretski & Rantzer , 1997 ) – has been on systems which are interconnections of linear mappings and “ simple ” nonlinearities , i.e . those easily bounded in some sense by quadratic functions . Fortuitously , this characteristic is shared with deep , recurrent , and equilibrium neural networks , a connection that we use heavily in this paper and has previously been exploited by Fazlyab et al . ( 2019 ) ; El Ghaoui et al . ( 2019 ) ; Revay et al . ( 2020 ) and others . A particular property we are interested in is called contraction ( Lohmiller & Slotine , 1998 ) , i.e. , any pair of solutions z1 ( t ) and z2 ( t ) exponentially converge to each other : ‖z1 ( t ) − z2 ( t ) ‖ ≤ α‖z1 ( 0 ) − z2 ( 0 ) ‖e−βt for all t > 0 and some α , β > 0 . Contraction can be established by finding a Riemannian metric with respect to which nearby trajectories converge , which is a differential analog of a Lyapunov function . A nice property of a contracting dynamical system is that if it is time-invariant , a unique equilibrium exists and it possesses a certain level of robustness . Moreover , contraction can also be linked to monotone operators , i.e . a system is contracting w.r.t . to a constant ( state-independent ) metric Q if and only if the operator −f is strongly monotone w.r.t . Q-weighted inner product . We collect some directly relevant results from systems theory in Appendix C.2 .

Summary: > Summary: This paper studies a new and more general way of parameterizing the simplest equilibrium network of the form $\sigma(Wz+Ux+b)$, a form that has been tackled by works like (Winston & Kolter 2020)and (El Ghaoui et al. 2019). The authors provide a computationally (relatively) efficient way of computing Lipschitz-bounded equilibrium networks and a detailed analysis of how the network should be constructed, along with the proof of the existence and uniqueness of the fixed point (and with less restrictive conditions when compared to MON). The empirical results on adversarial robustness shows that the proposed approach is a bit more robust than prior layer-based networks and other implicit networks, and validates most of the theoretical claims made by the authors.