Lipschitz Continuity and Spectral Normalization | by Wanshun Wong | Towards Data Science

One of the biggest weaknesses of neural networks is that they can be too sensitive to their inputs. The most famous illustration of neural networks’ lack of robustness is probably adversarial examples in computer vision, where small, imperceptible perturbations to the pixels of an image are able to change the neural network output completely. Ideally, if we have inputs that are similar, we want the outputs to be similar as well. Lipschitz continuity is a mathematical property that makes this notion concrete. In this article, we will see how Lipschitz continuity is used in deep learning, and how it motivates a new regularization method known as Spectral Normalization.
Let us begin with the definition of Lipschitz continuity:
A function f : RM → RN is Lipschitz continuous if there is a constant L such that∥f(x) - f(y)∥  ≦  L ∥x - y∥ for every x, y.
Here ∥·∥ denotes the usual Euclidean distance. The smallest such L is the Lipschitz constant of f and is denoted Lip(f). Notice that this definition can be generalized to functions between arbitrary metric spaces.
In our case, f is our neural network, and we want it to be Lipschitz continuous with a small Lip(f). This will provide an upper bound for the perturbations of the outputs. Lipschitz continuity also has the following property:
Let f = g ∘ h. If g and h are Lipschitz continuous, then f is also Lipschitz continuous with Lip(f) ≦ Lip(g) Lip(h).
Therefore, as long as we make each component of a neural network Lipschitz continuous with small Lipschitz constants, the whole neural network will also be Lipschitz continuous with small Lipschitz constants.
As a concrete example, a standard 2-layer feedforward network for binary classification can be written as
f = Sigmoid ∘ FC2 ∘ ReLU ∘ FC1
where FCi(x) = Wi x + bi are fully connected layers. The components of f are FC1, ReLU, FC2, and Sigmoid.
Functions commonly used in neural networks such as ReLU, sigmoid, softmax, tanh, max-pooling, have Lipschitz constant = 1. Hence we can simply keep using them.
Let us consider a fully connected layer. For simplicity we omit the bias term, so FC(x) = Wx for some weight matrix W. It can be shown that FC has Lipschitz constant Lip(FC) = σ(W) the spectral norm of W, which is equivalent to the largest singular value of W. Hence in general, Lip(FC) can be arbitrarily large.
Spectral Normalization comes into play by normalizing the spectral norm of W:
W ↤ W / σ(W)
The normalized matrix has spectral norm = 1, thus Lip(FC) is also 1.
The same idea can be applied to convolution layers as well. Notice that convolutions are linear operations, hence it is possible to rewrite them as matrices of the appropriate sizes and then reuse the above idea. The computation of these matrices and their spectral norms are not straightforward though, and we refer to [3] for details.
TensorFlow: tfa.layers.SpectralNormalization
PyTorch: torch.nn.utils.parametrizations.spectral_norm
Suppose we have a neural network f composed of fully connected layers, convolution layers, and commonly used activation functions. By applying Spectral Normalization to the fully connected layers and the convolution layers, we bound Lip(f) from above by 1.
If our neural network has other kinds of components, then applying Spectral Normalization to the fully connected layers and the convolution layers is not sufficient to bound Lip(f). Nonetheless it might be still be beneficial to do so because Spectral Normalization controls exploding gradients and also vanishing gradients, see [6] for a detailed analysis.
[3] studies Lipschitz continuity for other lp norm in addition to the Euclidean norm.Spectral Normalization was first introduced in [7] to stabilize the training of discriminator of generative adversarial network (GAN). A lot of studies comparing Spectral Normalization to other normalization and regularization methods were done in the GAN framework, see for example [5]. Recently Spectral Normalization has found success in reinforcement learning as well, see [2].In the previous section we have covered fully connected layers and convolution layers. Here we provide references for other architectures: — The Lipschitz constant of residual connections is studied in [3]. — Attention layers is not Lipschitz continuous. [1] and [4] suggest different modifications to make them Lipschitz continuous.
[3] studies Lipschitz continuity for other lp norm in addition to the Euclidean norm.
Spectral Normalization was first introduced in [7] to stabilize the training of discriminator of generative adversarial network (GAN). A lot of studies comparing Spectral Normalization to other normalization and regularization methods were done in the GAN framework, see for example [5]. Recently Spectral Normalization has found success in reinforcement learning as well, see [2].
In the previous section we have covered fully connected layers and convolution layers. Here we provide references for other architectures: — The Lipschitz constant of residual connections is studied in [3]. — Attention layers is not Lipschitz continuous. [1] and [4] suggest different modifications to make them Lipschitz continuous.
G. Dasoulas, K. Scaman, and A. Virmaux. Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networks (2021), ICML 2021.F. Gogianu, T. Berariu, M. Rosca, C. Clopath, L. Busoniu, and R. Pascanu. Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective (2021), ICML 2021.H. Gouk, E. Frank, B. Pfahringer, and M. J. Cree. Regularisation of neural networks by enforcing Lipschitz continuity (2021), Mach Learn 110, 393–416.H. Kim, G. Papamakarios, and A. Mnih. The Lipschitz Constant of Self-Attention (2021), ICML 2021.K. Kurach, M. Lucic, X. Zhai, M. Michalski, and S. Gelly. A Large-Scale Study on Regularization and Normalization in GANs (2019), ICML 2019.Z. Lin, V. Sekar, and G. Fanti. Why Spectral Normalization Stabilizes GANs: Analysis and Improvements (2021), arXiv preprint.T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral Normalization for Generative Adversarial Networks (2018), ICLR 2018.
G. Dasoulas, K. Scaman, and A. Virmaux. Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networks (2021), ICML 2021.
F. Gogianu, T. Berariu, M. Rosca, C. Clopath, L. Busoniu, and R. Pascanu. Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective (2021), ICML 2021.
H. Gouk, E. Frank, B. Pfahringer, and M. J. Cree. Regularisation of neural networks by enforcing Lipschitz continuity (2021), Mach Learn 110, 393–416.
H. Kim, G. Papamakarios, and A. Mnih. The Lipschitz Constant of Self-Attention (2021), ICML 2021.
K. Kurach, M. Lucic, X. Zhai, M. Michalski, and S. Gelly. A Large-Scale Study on Regularization and Normalization in GANs (2019), ICML 2019.
Z. Lin, V. Sekar, and G. Fanti. Why Spectral Normalization Stabilizes GANs: Analysis and Improvements (2021), arXiv preprint.
T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral Normalization for Generative Adversarial Networks (2018), ICLR 2018.