Robust Implicit Networks via Non-Euclidean Contractions

Implicit neural networks , a.k.a. , deep equilibrium networks , are a class of implicitdepth learning models where function evaluation is performed by solving a fixed point equation . They generalize classic feedforward models and are equivalent to infinite-depth weight-tied feedforward networks . While implicit models show improved accuracy and significant reduction in memory consumption , they can suffer from ill-posedness and convergence instability . This paper provides a new framework , which we call Non-Euclidean Monotone Operator Network ( NEMON ) , to design well-posed and robust implicit neural networks based upon contraction theory for the non-Euclidean norm ` ∞ . Our framework includes ( i ) a novel condition for well-posedness based on one-sided Lipschitz constants , ( ii ) an average iteration for computing fixed-points , and ( iii ) explicit estimates on input-output Lipschitz constants . Additionally , we design a training problem with the well-posedness condition and the average iteration as constraints and , to achieve robust models , with the input-output Lipschitz constant as a regularizer . Our ` ∞ well-posedness condition leads to a larger polytopic training search space than existing conditions and our average iteration enjoys accelerated convergence . Finally , we evaluate our framework in image classification through the MNIST and the CIFAR-10 datasets . Our numerical results demonstrate improved accuracy and robustness of the implicit models with smaller input-output Lipschitz bounds . Code is available at https : //github . com/davydovalexander/Non-Euclidean_Mon_Op_Net . 1 Introduction . Implicit neural networks are infinite-depth learning models with layers defined implicitly through a fixed-point equation . Examples of implicit neural networks include deep equilibrium models [ Bai et al. , 2019 ] and implicit deep learning models [ El Ghaoui et al. , 2021 ] . Implicit networks can be considered as generalizations of feedforward neural networks with input-injected weight tying , i.e. , training parameters are transferable between layers . Indeed , in implicit networks , function evaluation is executed by solving a fixed-point equation and backpropagation is implemented by computing gradients using implicit differentiation . Due to these unique features , implicit models enjoy more flexibility and improved memory efficiency compared to traditional neural networks . At the same time , implicit networks can suffer from instability in their training due to the nonlinear nature of their fixed-point equations and can show brittle input-output behaviors due to their model flexibility . ∗These authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . It is known that implicit neural networks require careful tuning and initialization to avoid ill-posed training procedures . Indeed , without additional assumptions , their fixed-point equation may not have a unique solution and the numerical algorithms for finding their solutions might not converge . Several recent works in the literature have focused on studying well-posedness and convergence of the fixedpoint equations of implicit networks using frameworks such as monotone operator theory [ Winston and Kolter , 2020 ] , contraction theory [ El Ghaoui et al. , 2021 ] , and a mixture of both [ Revay et al. , 2020 ] . Despite several insightful results , important questions about conditions for well-posedness of implicit networks and efficient algorithms that converge to their solutions are still open . One of the key features of implicit neural networks is their flexibility , which might come at the cost of low input-output robustness . As first noted in [ Szegedy et al. , 2014 ] , the input-output behavior of deep neural networks can be vulnerable to perturbations ; close enough input data can lead to completely different outputs . This lack of robustness can lead to unreliable performance of neural networks in safety-critical applications . Among several notions of robustness , the Lipschitz constant of a neural network is a coarse but rigorous measure which can be used to estimate input-output sensitivity of the network [ Szegedy et al. , 2014 ] . For this reason , there has been a growing interest in estimating the input-output Lipschitz constant of deep neural networks with respect to the ` 2-norm [ Fazlyab et al. , 2019 , Combettes and Pesquet , 2020 ] . However , it turns out that in some applications , the inputoutput Lipschitz constants with respect to non-Euclidean norms are more informative measures for studying robustness . One such application appears in the robustness analysis of neural networks with large-scale inputs under widely-distributed adversarial perturbations ( examples of these adversarial perturbations can be found in [ Szegedy et al. , 2014 ] ) . For these examples , the input-output ` 2- Lipschitz constant does not provide complete information about robustness of the network ; a neural network with small input-output ` 2-Lipschitz constant can be very sensitive to widespread entrywisesmall perturbations of the input signal . On the other hand , the input-output ` ∞-Lipschitz constant provides a different metric which appears to be well-suited for the analysis of widespread distributed perturbations . Another application is the estimation of input signal confidence intervals from output deviations , where the input-output ` ∞-Lipschitz constant of the network provides more scalable bounds than its ` 2 counterpart . Related works Implicit learning models . Numerous works in learning theory have shown the power of deep learning models with implicit layers . In these learning models , the notion of layers are replaced by a composition rule , which can be either a fixed-point iteration or a solution to a differential equation . Well-known frameworks for deep learning using implicit infinite-depth layers include deep equilibrium networks [ Bai et al. , 2019 ] , implicit deep learning [ El Ghaoui et al. , 2021 ] , and Neural ODEs [ Chen et al. , 2018 ] . In [ Kag et al. , 2020 ] , a class of implicit recurrent neural networks is considered and it is demonstrated that , with this architecture , the models do not suffer from vanishing nor exploding gradients . Implicit layers have also been used to study convex optimization problems [ Agrawal et al. , 2019 ] and to design control strategies [ Amos et al. , 2018 ] . Convergence to global minima of certain classes of implicit networks is studied in [ Kawaguchi , 2021 ] . Well-posedness and numerical algorithms for fixed-point equations . There has been a recent interest in studying well-posedness and numerical stability of implicit-depth learning models . [ El Ghaoui et al. , 2021 ] proposes a sufficient spectral condition for well-posedness and for convergence of the Picard iterations associated with the fixed-point equation of implicit networks . In [ Winston and Kolter , 2020 , Revay et al. , 2020 ] , using monotone operator theory , a suitable parametrization of the weight matrix is proposed which guarantees the stable convergence of suitable fixed-point iterations . A recent influential survey on monotone operators is [ Ryu and Boyd , 2016 ] . A recent survey on fixed point strategies in data science is given by [ Combettes and Pesquet , 2021 ] . Robustness of learning models It is known that neural networks can be vulnerable to adversarial input perturbations [ Szegedy et al. , 2014 ] . A large body of literature is devoted to improve robustness of neural networks using various defense strategies against adversarial examples [ Goodfellow et al. , 2015 , Papernot et al. , 2016 ] . While these strategies are effective in many scenarios , they do not provide formal guarantees for robustness [ Carlini and Wagner , 2017 ] . However , there has been a recent interest in designing classifiers that are provably robust with respect to adversarial perturbations [ Madry et al. , 2018 , Wong and Kolter , 2018 ] . The input-output Lipschitz constant of a neural network is a rigorous metric for its worst-case sensitivity with respect to input perturbations . Several recent works have focused on estimating the Lipschitz constant and enforcing its boundedness . For example , [ Fazlyab et al. , 2019 , 2020 ] propose a convex optimization framework using quadratic constraints and semidefinite programming to obtain upper bounds on Lipschitz constants of deep neural networks . In [ Pauli et al. , 2021 ] , a training algorithm is designed to ensure boundedness of the Lipschitz constant of the neural network via a semidefinite program . Other methods for estimating the Lipschitz constant of deep neural networks include [ Krishnan et al. , 2020 , Revay et al. , 2021 , Combettes and Pesquet , 2020 ] . For implicit neural networks , a sensitivity-based robustness analysis is proposed in [ El Ghaoui et al. , 2021 ] . Lipschitz constants of deep equilibrium networks have also been studied in [ Pabbaraju et al. , 2021 , Revay et al. , 2020 ] using monotone operator theory . Contributions In this paper , using non-Euclidean contraction theory with respect to the ` ∞-norm , we propose our novel framework , Non-Euclidean Monotone Operator Network ( NEMON ) , to design implicit neural networks and study their well-posedness , stability , and robustness . First , we develop elements of a novel non-Euclidean monotone operator theory akin to the frameworks in [ Bauschke and Combettes , 2017 , Ryu and Boyd , 2016 ] . Using the concept of matrix measure , we introduce the essential notion of one-sided Lipschitz constant of a map . Based upon this notion , we prove a general fixed-point theorem with weaker requirements than classical results on Picard and Krasnosel ’ skii–Mann iterations . For maps with one-sided Lipschitz constant less than unity , we show that an average iteration converges for sufficiently small step sizes and optimize its rate of convergence . For the special case of the weighted ` ∞-norm , we show that this average iteration can be accelerated by choosing a larger step size . Additionally , we study perturbed fixed-point equations and establish a bound on the distance between perturbed and nominal equilibrium points as a function of one-sided Lipschitz condition . Second , for implicit neural networks , we use our new fixed-point theorem to ( i ) establish ` ∞-norm conditions for their well-posedness , ( ii ) design accelerated numerical algorithms for computing their solutions , and ( iii ) provide upper bounds on their input-output ` ∞-Lipschitz constants . Third , we propose a parametrization for matrices with appropriate bound on their one-sided Lipschitz constants and use this parametrization with the average iteration to design a training optimization problem . Finally , we perform several numerical experiments illustrating improved performance of NEMON in image classification on the MNIST and the CIFAR-10 datasets compared to the state-of-the-art models in [ El Ghaoui et al. , 2021 , Winston and Kolter , 2020 ] . Additionally , by adding the input-output Lipschitz constant as regularizer in the training problem , we observe improved robustness to some classes of adversarial perturbations . We include all relevant proofs in Appendix C . 2 Review material . Matrix measures Let ‖ · ‖ be a norm on Rn and its induced norm on Rn×n . The matrix measure of A ∈ Rn×n with respect to ‖ · ‖ is defined by µ ( A ) : = limh→0+ ‖In+hA‖−1h , that is , the one-sided directional derivative of the induced norm in direction of A , evaluated at In . Remarkably , the matrix measure is a tighter upper bound on the spectral abscissa of A than ‖A‖ and the set of matrices A ∈ Rn×n satisfying µ ( A ) ≤ 1 is an unbounded subset of Rn×n strictly containing the compact ball ‖A‖ ≤ 1 . We refer to [ Desoer and Haneda , 1972 ] for a list of properties enjoyed by matrix measures . We will be specifically interested in diagonally weighted ` ∞ norms defined by ‖x‖∞ , [ η ] −1 = max i 1 ηi |xi| , ( 1 ) where , given a positive vector η ∈ Rn > 0 , we use [ η ] to denote the diagonal matrix with diagonal entries η . The corresponding matrix norm and measure are ‖A‖∞ , [ η ] −1 = max i∈ { 1 , ... , n } n∑ j=1 ηj ηi |aij | , µ∞ , [ η ] −1 ( A ) = max i∈ { 1 , ... , n } ( aii + n∑ j=1 , j 6=i |aij | ηj ηi ) . ( 2 ) Lipschitz maps Given a norm ‖ · ‖ with induced matrix measure µ ( · ) , a differentiable map F : Rn → Rn is Lipschitz continuous with constant Lip ( F ) ∈ R≥0 if ‖DF ( x ) ‖ ≤ Lip ( F ) for all x ∈ Rn . ( 3 ) For example , for an affine F ( x ) = Ax+ b , the ( smallest ) Lipschitz constant is Lip ( F ) = ‖A‖ . One-sided Lipschitz maps Given a norm ‖ · ‖ , a differentiable map F : Rn → Rn is one-sided Lipschitz continuous with constant osL ( F ) ∈ R if µ ( DF ( x ) ) ≤ osL ( F ) for all x ∈ Rn . ( 4 ) For example , for an affine F ( x ) = Ax+ b , the ( smallest ) one-sided Lipschitz constant is osL ( F ) = µ ( A ) . Note that ( i ) the one-sided Lipschitz constant is upper bounded by the Lipschitz constant , ( ii ) a Lipschitz continuous map is always one-sided Lipschitz continuous , and ( iii ) the one-sided Lipschitz constant may be negative . For a more in-depth review we refer to Appendix A . The notion of onesided Lipschitz continuity unifies several important concepts in dynamical systems and optimization theory . In operator theory , the map F is called a monotone operator if it is one-sided Lipschitz continuous with respect to the ` 2-norm with the constant − osL ( −F ) > 0 [ Ryu and Boyd , 2016 , Bauschke and Combettes , 2017 ] . In control theory , the vector field F is called strongly infinitesimally contracting if it is one-sided Lipschitz continuous with the constant osL ( F ) < 0 [ Desoer and Haneda , 1972 , Lohmiller and Slotine , 1998 , Pavlov et al. , 2004 ] . In what follows , we let osL∞ , [ η ] −1 ( F ) ∈ R denote the one-sided Lipschitz constants with respect to the weighted ` ∞-norm .

Summary: This paper presents a thorough analysis of constructing provably convergent equilibrium networks in the non-Euclidean space. Instead of directly requiring Lipschitzness as in prior works (which mostly dealt with $L_2$), the authors show that we can further relax the constraint by considering one-sided Lipschitzness and exploit the contraction theory via semi inner products. The experimental results of the paper seem to demonstrate the validity of the theoretical results and the paper provides empirical evidence on the potential application of the theories in improving implicit networks' adversarial robustness.