Robust Implicit Networks via Non-Euclidean Contractions

Implicit neural networks , a.k.a. , deep equilibrium networks , are a class of implicitdepth learning models where function evaluation is performed by solving a fixed point equation . They generalize classic feedforward models and are equivalent to infinite-depth weight-tied feedforward networks . While implicit models show improved accuracy and significant reduction in memory consumption , they can suffer from ill-posedness and convergence instability . This paper provides a new framework , which we call Non-Euclidean Monotone Operator Network ( NEMON ) , to design well-posed and robust implicit neural networks based upon contraction theory for the non-Euclidean norm ` ∞ . Our framework includes ( i ) a novel condition for well-posedness based on one-sided Lipschitz constants , ( ii ) an average iteration for computing fixed-points , and ( iii ) explicit estimates on input-output Lipschitz constants . Additionally , we design a training problem with the well-posedness condition and the average iteration as constraints and , to achieve robust models , with the input-output Lipschitz constant as a regularizer . Our ` ∞ well-posedness condition leads to a larger polytopic training search space than existing conditions and our average iteration enjoys accelerated convergence . Finally , we evaluate our framework in image classification through the MNIST and the CIFAR-10 datasets . Our numerical results demonstrate improved accuracy and robustness of the implicit models with smaller input-output Lipschitz bounds . Code is available at https : //github . com/davydovalexander/Non-Euclidean_Mon_Op_Net . 1 Introduction . Implicit neural networks are infinite-depth learning models with layers defined implicitly through a fixed-point equation . Examples of implicit neural networks include deep equilibrium models [ Bai et al. , 2019 ] and implicit deep learning models [ El Ghaoui et al. , 2021 ] . Implicit networks can be considered as generalizations of feedforward neural networks with input-injected weight tying , i.e. , training parameters are transferable between layers . Indeed , in implicit networks , function evaluation is executed by solving a fixed-point equation and backpropagation is implemented by computing gradients using implicit differentiation . Due to these unique features , implicit models enjoy more flexibility and improved memory efficiency compared to traditional neural networks . At the same time , implicit networks can suffer from instability in their training due to the nonlinear nature of their fixed-point equations and can show brittle input-output behaviors due to their model flexibility . ∗These authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . It is known that implicit neural networks require careful tuning and initialization to avoid ill-posed training procedures . Indeed , without additional assumptions , their fixed-point equation may not have a unique solution and the numerical algorithms for finding their solutions might not converge . Several recent works in the literature have focused on studying well-posedness and convergence of the fixedpoint equations of implicit networks using frameworks such as monotone operator theory [ Winston and Kolter , 2020 ] , contraction theory [ El Ghaoui et al. , 2021 ] , and a mixture of both [ Revay et al. , 2020 ] . Despite several insightful results , important questions about conditions for well-posedness of implicit networks and efficient algorithms that converge to their solutions are still open . One of the key features of implicit neural networks is their flexibility , which might come at the cost of low input-output robustness . As first noted in [ Szegedy et al. , 2014 ] , the input-output behavior of deep neural networks can be vulnerable to perturbations ; close enough input data can lead to completely different outputs . This lack of robustness can lead to unreliable performance of neural networks in safety-critical applications . Among several notions of robustness , the Lipschitz constant of a neural network is a coarse but rigorous measure which can be used to estimate input-output sensitivity of the network [ Szegedy et al. , 2014 ] . For this reason , there has been a growing interest in estimating the input-output Lipschitz constant of deep neural networks with respect to the ` 2-norm [ Fazlyab et al. , 2019 , Combettes and Pesquet , 2020 ] . However , it turns out that in some applications , the inputoutput Lipschitz constants with respect to non-Euclidean norms are more informative measures for studying robustness . One such application appears in the robustness analysis of neural networks with large-scale inputs under widely-distributed adversarial perturbations ( examples of these adversarial perturbations can be found in [ Szegedy et al. , 2014 ] ) . For these examples , the input-output ` 2- Lipschitz constant does not provide complete information about robustness of the network ; a neural network with small input-output ` 2-Lipschitz constant can be very sensitive to widespread entrywisesmall perturbations of the input signal . On the other hand , the input-output ` ∞-Lipschitz constant provides a different metric which appears to be well-suited for the analysis of widespread distributed perturbations . Another application is the estimation of input signal confidence intervals from output deviations , where the input-output ` ∞-Lipschitz constant of the network provides more scalable bounds than its ` 2 counterpart . Related works Implicit learning models . Numerous works in learning theory have shown the power of deep learning models with implicit layers . In these learning models , the notion of layers are replaced by a composition rule , which can be either a fixed-point iteration or a solution to a differential equation . Well-known frameworks for deep learning using implicit infinite-depth layers include deep equilibrium networks [ Bai et al. , 2019 ] , implicit deep learning [ El Ghaoui et al. , 2021 ] , and Neural ODEs [ Chen et al. , 2018 ] . In [ Kag et al. , 2020 ] , a class of implicit recurrent neural networks is considered and it is demonstrated that , with this architecture , the models do not suffer from vanishing nor exploding gradients . Implicit layers have also been used to study convex optimization problems [ Agrawal et al. , 2019 ] and to design control strategies [ Amos et al. , 2018 ] . Convergence to global minima of certain classes of implicit networks is studied in [ Kawaguchi , 2021 ] . Well-posedness and numerical algorithms for fixed-point equations . There has been a recent interest in studying well-posedness and numerical stability of implicit-depth learning models . [ El Ghaoui et al. , 2021 ] proposes a sufficient spectral condition for well-posedness and for convergence of the Picard iterations associated with the fixed-point equation of implicit networks . In [ Winston and Kolter , 2020 , Revay et al. , 2020 ] , using monotone operator theory , a suitable parametrization of the weight matrix is proposed which guarantees the stable convergence of suitable fixed-point iterations . A recent influential survey on monotone operators is [ Ryu and Boyd , 2016 ] . A recent survey on fixed point strategies in data science is given by [ Combettes and Pesquet , 2021 ] . Robustness of learning models It is known that neural networks can be vulnerable to adversarial input perturbations [ Szegedy et al. , 2014 ] . A large body of literature is devoted to improve robustness of neural networks using various defense strategies against adversarial examples [ Goodfellow et al. , 2015 , Papernot et al. , 2016 ] . While these strategies are effective in many scenarios , they do not provide formal guarantees for robustness [ Carlini and Wagner , 2017 ] . However , there has been a recent interest in designing classifiers that are provably robust with respect to adversarial perturbations [ Madry et al. , 2018 , Wong and Kolter , 2018 ] . The input-output Lipschitz constant of a neural network is a rigorous metric for its worst-case sensitivity with respect to input perturbations . Several recent works have focused on estimating the Lipschitz constant and enforcing its boundedness . For example , [ Fazlyab et al. , 2019 , 2020 ] propose a convex optimization framework using quadratic constraints and semidefinite programming to obtain upper bounds on Lipschitz constants of deep neural networks . In [ Pauli et al. , 2021 ] , a training algorithm is designed to ensure boundedness of the Lipschitz constant of the neural network via a semidefinite program . Other methods for estimating the Lipschitz constant of deep neural networks include [ Krishnan et al. , 2020 , Revay et al. , 2021 , Combettes and Pesquet , 2020 ] . For implicit neural networks , a sensitivity-based robustness analysis is proposed in [ El Ghaoui et al. , 2021 ] . Lipschitz constants of deep equilibrium networks have also been studied in [ Pabbaraju et al. , 2021 , Revay et al. , 2020 ] using monotone operator theory . Contributions In this paper , using non-Euclidean contraction theory with respect to the ` ∞-norm , we propose our novel framework , Non-Euclidean Monotone Operator Network ( NEMON ) , to design implicit neural networks and study their well-posedness , stability , and robustness . First , we develop elements of a novel non-Euclidean monotone operator theory akin to the frameworks in [ Bauschke and Combettes , 2017 , Ryu and Boyd , 2016 ] . Using the concept of matrix measure , we introduce the essential notion of one-sided Lipschitz constant of a map . Based upon this notion , we prove a general fixed-point theorem with weaker requirements than classical results on Picard and Krasnosel ’ skii–Mann iterations . For maps with one-sided Lipschitz constant less than unity , we show that an average iteration converges for sufficiently small step sizes and optimize its rate of convergence . For the special case of the weighted ` ∞-norm , we show that this average iteration can be accelerated by choosing a larger step size . Additionally , we study perturbed fixed-point equations and establish a bound on the distance between perturbed and nominal equilibrium points as a function of one-sided Lipschitz condition . Second , for implicit neural networks , we use our new fixed-point theorem to ( i ) establish ` ∞-norm conditions for their well-posedness , ( ii ) design accelerated numerical algorithms for computing their solutions , and ( iii ) provide upper bounds on their input-output ` ∞-Lipschitz constants . Third , we propose a parametrization for matrices with appropriate bound on their one-sided Lipschitz constants and use this parametrization with the average iteration to design a training optimization problem . Finally , we perform several numerical experiments illustrating improved performance of NEMON in image classification on the MNIST and the CIFAR-10 datasets compared to the state-of-the-art models in [ El Ghaoui et al. , 2021 , Winston and Kolter , 2020 ] . Additionally , by adding the input-output Lipschitz constant as regularizer in the training problem , we observe improved robustness to some classes of adversarial perturbations . We include all relevant proofs in Appendix C . 2 Review material . Matrix measures Let ‖ · ‖ be a norm on Rn and its induced norm on Rn×n . The matrix measure of A ∈ Rn×n with respect to ‖ · ‖ is defined by µ ( A ) : = limh→0+ ‖In+hA‖−1h , that is , the one-sided directional derivative of the induced norm in direction of A , evaluated at In . Remarkably , the matrix measure is a tighter upper bound on the spectral abscissa of A than ‖A‖ and the set of matrices A ∈ Rn×n satisfying µ ( A ) ≤ 1 is an unbounded subset of Rn×n strictly containing the compact ball ‖A‖ ≤ 1 . We refer to [ Desoer and Haneda , 1972 ] for a list of properties enjoyed by matrix measures . We will be specifically interested in diagonally weighted ` ∞ norms defined by ‖x‖∞ , [ η ] −1 = max i 1 ηi |xi| , ( 1 ) where , given a positive vector η ∈ Rn > 0 , we use [ η ] to denote the diagonal matrix with diagonal entries η . The corresponding matrix norm and measure are ‖A‖∞ , [ η ] −1 = max i∈ { 1 , ... , n } n∑ j=1 ηj ηi |aij | , µ∞ , [ η ] −1 ( A ) = max i∈ { 1 , ... , n } ( aii + n∑ j=1 , j 6=i |aij | ηj ηi ) . ( 2 ) Lipschitz maps Given a norm ‖ · ‖ with induced matrix measure µ ( · ) , a differentiable map F : Rn → Rn is Lipschitz continuous with constant Lip ( F ) ∈ R≥0 if ‖DF ( x ) ‖ ≤ Lip ( F ) for all x ∈ Rn . ( 3 ) For example , for an affine F ( x ) = Ax+ b , the ( smallest ) Lipschitz constant is Lip ( F ) = ‖A‖ . One-sided Lipschitz maps Given a norm ‖ · ‖ , a differentiable map F : Rn → Rn is one-sided Lipschitz continuous with constant osL ( F ) ∈ R if µ ( DF ( x ) ) ≤ osL ( F ) for all x ∈ Rn . ( 4 ) For example , for an affine F ( x ) = Ax+ b , the ( smallest ) one-sided Lipschitz constant is osL ( F ) = µ ( A ) . Note that ( i ) the one-sided Lipschitz constant is upper bounded by the Lipschitz constant , ( ii ) a Lipschitz continuous map is always one-sided Lipschitz continuous , and ( iii ) the one-sided Lipschitz constant may be negative . For a more in-depth review we refer to Appendix A . The notion of onesided Lipschitz continuity unifies several important concepts in dynamical systems and optimization theory . In operator theory , the map F is called a monotone operator if it is one-sided Lipschitz continuous with respect to the ` 2-norm with the constant − osL ( −F ) > 0 [ Ryu and Boyd , 2016 , Bauschke and Combettes , 2017 ] . In control theory , the vector field F is called strongly infinitesimally contracting if it is one-sided Lipschitz continuous with the constant osL ( F ) < 0 [ Desoer and Haneda , 1972 , Lohmiller and Slotine , 1998 , Pavlov et al. , 2004 ] . In what follows , we let osL∞ , [ η ] −1 ( F ) ∈ R denote the one-sided Lipschitz constants with respect to the weighted ` ∞-norm .

Summary: This paper gives a novel characterization of well-posedness for implicit networks based on the one-sided Lipschitz constant (osL). This follows in a line of work of other such characterizations [1,2,3,4], but differs in that well-posedness is guaranteed by a constraint on the osL of the weights matrix, which can be wrt arbitrary norms. The authors focus on the $\ell_\infty$ norm, for which they show accelerated convergence of dampened fixed-point iterations.  They also drive a bound for the Lipschitz constant of the network wrt the $\ell_\infty$ norm on the input space, which is more natural for achieving robustness than the $\ell_2$-based Lipschitz bounds of [3] and [4]. They train networks while enforcing well-posedness and using the Lipschitz bound as a regularizer, and demonstrate the empirical robustness of the models.