Existing approaches to enforce Lipschitz constraints fall into two categories : regularization and architectural constraints . Regularization approaches

Training neural networks under a strict Lipschitz constraint is useful for provable adversarial robustness, generalization bounds, interpretable gradients, and Wasserstein distance estimation. By the composition property of Lipschitz functions, it suffices to ensure that each individual affine transformation or nonlinear activation is 1-Lipschitz. The challenge is to do this while maintaining the expressive power. We identify a necessary property for such an architecture: each of the layers must preserve the gradient norm during backpropagation. Based on this, we propose to combine a gradient norm preserving activation function, GroupSort, with norm-constrained weight matrices. We show that norm-constrained GroupSort architectures are universal Lipschitz function approximators. Empirically, we show that normconstrained GroupSort networks achieve tighter estimates of Wasserstein distance than their ReLU counterparts and can achieve provable adversarial robustness guarantees with little cost to accuracy. Constraining the Lipschitz constant of a neural network puts a bound on how much its output can change in proportion to a change in its input. For classification tasks, a small Lipschitz constant leads to better generalization (Sokolić et al., 2017), improved adversarial robustness (Cisse et al., 2017; Tsuzuku et al., 2018), and greater interpretability (Tsipras et al., 2018). Additionally, the Wasserstein distance between two probability distributions can be expressed as the solution to a maximization problem over Lipschitz functions (Peyré & Cuturi, 2018). Despite the wide-ranging applications, the question of how to approximate Lipschitz functions with neural networks without sacrificing expressive power has remained largely unanswered. Equal contribution Department of Computer Science, University of Toronto, Toronto, Canada Vector Institute, Toronto, Canada. Correspondence to: Cem Anil , James Lucas . Proceedings of the 36 th International Conference on Machine Learning, Long Beach, California, PMLR 97, 2019. Copyright 2019 by the author(s). Existing approaches to enforce Lipschitz constraints fall into two categories: regularization and architectural constraints. Regularization approaches (Drucker & Le Cun, 1992; Gulrajani et al., 2017) perform well in practice, but do not provably enforce the Lipschitz constraint globally. Approaches based on architectural constraints place limitations on the operator norm (such as the matrix spectral norm) of each layer’s weights (Cisse et al., 2017; Yoshida & Miyato, 2017). These provably satisfy the Lipschitz constraint, but come at a cost in expressive power. E.g., norm-constrained ReLU networks are provably unable to approximate simple functions such as absolute value (Huster et al., 2018). We first identify a simple property that expressive normconstrained Lipschitz architectures must satisfy: gradient norm preservation. Specifically, in order to represent a function with slope 1 almost everywhere, each layer must preserve the norm of the gradient during backpropagation. ReLU architectures satisfy this only when the activations are positive; empirically, this manifests during training of normconstrained ReLU networks in that the activations are forced to be positive most of the time, reducing the network’s capacity to represent nonlinear functions. We make use of an alternative activation function called GroupSort — a variant of which was proposed by Chernodub & Nowicki (2016) — which sorts groups of activations. GroupSort is both Lipschitz and gradient norm preserving. Using a variant of the Stone-Weierstrass theorem, we show that norm-constrained GroupSort networks are universal Lipschitz function approximators. While we focus our attention, both theoretically and empirically, on fully connected networks, the same general principles hold for convolutional networks where the techniques we introduce could be applied. Empirically, we show that ReLU networks are unable to approximate even the simplest Lipschitz functions which GroupSort networks can. We observe that norm-constrained ReLU networks must trade non-linear processing for gradient norm, leading to less expressive networks. Moreover, we obtain tighter lower bounds on the Wasserstein distance between complex, high dimensional distributions using GroupSort architectures. We also train classifiers with provable adversarial robustness guarantees and find that using GroupSort provides improved accuracy and robustness compared to ReLU. Across all of our experiments, we found that norm-constrained GroupSort architectures consistently outperformed their ReLU counterparts. Sorting Out Lipschitz Function Approximation