Sorting out Lipschitz function approximation

Training neural networks subject to a Lipschitz constraint is useful for generalization bounds , provable adversarial robustness , interpretable gradients , and Wasserstein distance estimation . By the composition property of Lipschitz functions , it suffices to ensure that each individual affine transformation or nonlinear activation function is 1-Lipschitz . The challenge is to do this while maintaining the expressive power . We identify a necessary property for such an architecture : each of the layers must preserve the gradient norm during backpropagation . Based on this , we propose to combine a gradient norm preserving activation function , GroupSort , with norm-constrained weight matrices . We show that norm-constrained GroupSort architectures are universal Lipschitz function approximators . Empirically , we show that norm-constrained GroupSort networks achieve tighter estimates of Wasserstein distance than their ReLU counterparts and can achieve provable adversarial robustness guarantees with little cost to accuracy . Constraining the Lipschitz constant of a neural network ensures that a small change to the input can produce only a small change to the output . For classification , a small Lipschitz constant leads to better generalization ( Sokolić et al. , 2017 ) , improved adversarial robustness ( Cisse et al. , 2017 ; Tsuzuku et al. , 2018 ) , and greater interpretability ( Tsipras et al. , 2018 ) . Additionally , the Wasserstein distance between two probability distributions can be expressed as a maximization problem over Lipschitz functions ( Peyré & Cuturi , 2018 ) . But despite the wide-ranging applications , the question of how to approximate the class of Lipschitz functions with neural networks remains largely unanswered . Existing approaches to enforce Lipschitz constraints broadly fall into two categories : regularization and architectural constraints . Regularization approaches such as double backprop ( Drucker & Le Cun , 1992 ) or the gradient penalty ( Gulrajani et al. , 2017 ) perform well in practice , but do not provably enforce the Lipschitz constraint globally . On the other hand , norm-constrained architectures place limitations on the operator norm ( such as the matrix spectral norm ) of each layer ’ s weight matrix ( Cisse et al. , 2017 ; Yoshida & Miyato , 2017 ) . These techniques provably satisfy the Lipschitz constraint , but this comes at a cost in expressive power . E.g. , norm-constrained ReLU networks are provably unable to approximate simple functions such as absolute value ( Huster et al. , 2018 ) . We first identify a simple property that expressive norm-constrained 1-Lipschitz architectures must satisfy : gradient norm preservation . Specifically , in order to represent a function with slope 1 almost everywhere , each layer must preserve the norm of the gradient during backpropagation . ReLU architectures satisfy this only when the activations are positive ; empirically , this manifests during training of norm-constrained ReLU networks in that the activations are forced to be positive most of the time , reducing the network ’ s capacity to represent nonlinear functions . We make use of an alternative activation function called GroupSort — a variant of which was proposed by Chernodub & Nowicki ( 2016 ) — which sorts groups of activations . GroupSort is both Lipschitz and gradient norm preserving . Using a variant of the Stone-Weierstrass theorem , we show that norm-constrained GroupSort networks are universal Lipschitz function approximators . While we focus our attention , both theoretically and empirically , on fully connected networks , the same general principles hold for convolutional networks where the techniques we introduce could be directly applied . Empirically , we show that ReLU networks are unable to solve even the simplest Wasserstein distance estimation problems which GroupSort can solve completely . Moreover , we observe that norm-constrained ReLU networks must trade non-linear processing for gradient norm leading to less expressive networks . We also train classifiers with provable adversarial robustness guarantees and find that using GroupSort provides improved accuracy and robustness compared to ReLU . Across all of our experiments , we found that norm-constrained GroupSort architectures consistently outperformed their ReLU counterparts . 2 BACKGROUND . Notation We will use x ∈ Rin to denote the input vector to the neural network , y ∈ Rout the output ( or logits ) of the neural network , nl the dimensionality of the lth hidden layer , Wl ∈ Rnl−1×nl and bl ∈ Rnl the weight matrix and the bias of the lth layer . We will denote the pre-activations in layer l with zl and activations with hl . The number of layers in the network will be L with y = zL . We will use φ to denote the activation function used in the neural network . The computation performed by layer l of the network will be : zl = Wlhl−1 + bl hl = φ ( zl ) Network Jacobian Using the chain rule , the Jacobian of a neural network can be expanded as follows : ∂y ∂x = ∂zL ∂hL−1 ∂hL−1 ∂zL−1 . . . ∂z2 ∂h1 ∂h1 ∂z1 ∂z1 ∂x = WLφ ′ ( zL−1 ) . . .W2φ ′ ( z1 ) W1 ( 1 ) 2.1 LIPSCHITZ FUNCTIONS . Given two metric spaces X and Y , a function f : X → Y is Lipschitz continuous if there exists K ∈ R such that for all x1 and x2 in X , dY ( f ( x1 ) , f ( x2 ) ) ≤ KdX ( x1 , x2 ) where dX and dY are metrics ( such as Euclidean distance ) on X and Y respectively . In this work , when we refer to the Lipschitz constant we are referring to the smallest such K for which the above holds under a given dX and dY . Unless otherwise specified , we take X = Rn and Y = Rm throughout . If the Lipschitz constant of a function is K , it is called a K-Lipschitz function . Equivalently , if the function is everywhere differentiable then its Lipschitz constant is bounded by the operator norm of its Jacobian . Throughout this work , we make use of the following definition , Definition 1 . Given a metric space ( X , dX ) where dX denotes the metric onX , we write CL ( X , R ) to denote the space of all 1-Lipschitz functions mapping X to R ( with respect to the Lp metric ) . 2.2 LIPSCHITZ-CONSTRAINED NEURAL NETWORKS . As 1-Lipschitz functions are closed under composition , to build a 1-Lipschitz neural network it suffices to compose 1-Lipschitz affine transformations and activation functions . 1-Lipschitz Linear Transformations : Ensuring that each linear map is 1-Lipschitz is equivalent to ensuring that ||Wx||p ≤ ||x||p for any x ; i.e . constraining the matrix p-norm , ||W||p = sup||x||p=1 ||Wx||p , to be at most 1 . Important examples of matrix p-norms include the matrix 2-norm , which is the largest singular value , and the matrix∞-norm , which can be expressed as : ||W||∞ = max 1≤i≤m m∑ j=1 |wij | . Similarly , we may also define the mixed matrix norm , given by ||W||p , q = sup||x||p=1 ||Wx||q . Enforcing matrix norm constraints naively may be computationally expensive . Fortunately , techniques exist to efficiently ensure that ||W ||p = 1 when p = 2 or p =∞ . We discuss these in Section 4.2 . 1-Lipschitz Activation Functions : Most commonly used activation functions ( such as ReLU ( Krizhevsky et al. , 2012 ) , sigmoid , tanh , maxout ( Goodfellow et al. , 2013 ) ) are 1-Lipschitz , if they are scaled appropriately . 2.3 APPLICATIONS OF LIPSCHITZ NETWORKS . Wasserstein Distance Estimation Wasserstein-1 distance ( also called Earth Mover Distance ) is a principled distance metric between two probability distributions and has found many applications in machine learning in recent years ( Peyré & Cuturi , 2018 ; Genevay et al. , 2017 ) . Using KantorovichRubinstein duality ( Villani , 2008 ) , one can recast the Wasserstein distance estimation problem as a concave maximization problem , defined over 1-Lipschitz functions : W ( P1 , P2 ) = sup f∈CL ( X , R ) ( Ex∼P1 [ f ( x ) ] − Ex∼P2 [ f ( x ) ] ) ( 2 ) Since this dual objective resembles the discriminator objective for generative adversarial networks ( GANs ) , Arjovsky et al . ( 2017 ) proposed the Wasserstein GAN architecture , which uses a neural net architecture to approximate the space of Lipschitz functions . Adversarial Robustness Adversarial examples are inputs to a machine learning system which have been designed to force undesirable behaviour ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ) . Formally , given a classifier f and a data point x , we write an adversarial example as xadv = x + δ such that f ( xadv ) 6= f ( x ) and δ is small . A small Lipschitz constant can guarantee a lower bound on the size of δ ( Tsuzuku et al. , 2018 ) and thus provide robustness guarantees . However , existing approaches have both practical and theoretical limitations ( Huster et al. , 2018 ) . 3 GRADIENT NORM PRESERVATION . When backpropagating through a norm-constrained 1-Lipschitz network , the gradient norm is nonincreasing as it is processed by each layer . This simple fact leads to interesting consequences when we attempt to represent ( scalar-valued ) functions whose input-output gradient has norm 1 almost everywhere . ( Such functions are relevant to Wasserstein distance estimation , where an optimal dual solution always has this property ( Gulrajani et al. , 2017 ) . ) To ensure the input-output gradient norm is 1 , the gradient norm must be preserved by each layer in the network during backpropagation . Unfortunately , norm-constrained networks with common activations are unable to achieve this . Theorem 1 . Consider a neural network , f : Rn → R , built with matrix 2-norm constrained weights ( ||W||2 ≤ 1 ) and 1-Lipschitz , element-wise , monotonically increasing activation functions . If ||∇f ( x ) ||2 = 1 almost everywhere , then f is linear . As a special case , Theorem 1 shows that no 2-norm-constrained neural network with ReLU ( or sigmoid , tanh , etc . ) activations can represent the absolute value function . A full proof can be found in Appendix C. Informally , for ReLU layers , the gradient norm can only be preserved if every activation is positive ( with the exception of units which don ’ t affect the network ’ s output ) . But as this holds for almost all inputs , the network ’ s input-output mapping must be linear . This tension between gradient norm and nonlinear processing is also observed empirically . Figure 5 compares the activation statistics for MNIST classification networks with ReLU activations , with and without matrix norm constraints on the weights . For the network with smallest Lipschitz constant , around 10 % of the units are “ undead ” , or always active ( and hence do not contribute any nonlinear processing ) . This suggests that the network is sacrificing nonlinear capacity in order to maintain adequate gradient norm . Another useful consequence of gradient norm preservation is that we may restrict all of the weight matrices to have singular values of 1 : Theorem 2 . Consider a neural network , f : Rn → R , built with matrix 2-norm constrained weights and with ||∇f ( x ) ||2 = 1 almost everywhere . Then , without changing the computed function , each weight matrix W ∈ Rm×k can be replaced with a matrix W̃ whose singular values all equal 1 . Note that the condition of singular values equaling 1 is equivalent to the following : when m > k , the columns of W̃ are orthonormal ; when m < k , the rows of W̃ are orthonormal ; and when m = k , W̃ is orthogonal . For the remainder of this paper , we abuse terminology slightly and refer to such matrices as orthonormal . The proof of Theorem 2 is given in Appendix C. With these two results in place , we restrict our search for expressive Lipschitz networks to those that contain orthonormal weight matrices ( those with singular values all equal to 1 ) and activations which preserve the gradient norm during backpropagation .

Summary: This paper introduces GroupSort. The motivation is to find a good way to impose Lipschitz constraint to the learning of neural networks. An easy approach is "atomic construction", which imposes a norm constraint to the weight matrix of every network layer. Although it guarantees the network to be a Lipschitz function, not all Lipschitz functions are representable under this strong constraint. The authors point out that this is because the activation function of the network doesn't satisfy the so called Jacobian norm preserving property.