Adversarial Lipschitz Regularization

1 INTRODUCTION . In recent years , Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) have been becoming the state-of-the-art in several generative modeling tasks , ranging from image generation ( Karras et al. , 2018 ) to imitation learning ( Ho and Ermon , 2016 ) . They are based on an idea of a two-player game , in which a discriminator tries to distinguish between real and generated data samples , while a generator tries to fool the discriminator , learning to produce realistic samples on the long run . Wasserstein GAN ( WGAN ) was proposed as a solution to the issues present in the original GAN formulation . Replacing the discriminator , WGAN trains a critic to approximate the Wasserstein distance between the real and generated distributions . This introduced a new challenge , since Wasserstein distance estimation requires the function space of the critic to only consist of 1-Lipschitz functions . To enforce the Lipschitz constraint on the WGAN critic , Arjovsky et al . ( 2017 ) originally used weight clipping , which was soon replaced by the much more effective method of Gradient Penalty ( GP ) ( Gulrajani et al. , 2017 ) , which consists of penalizing the deviation of the critic ’ s gradient norm from 1 at certain input points . Since then , several variants of gradient norm penalization have been introduced ( Petzka et al. , 2018 ; Wei et al. , 2018 ; Adler and Lunz , 2018 ; Zhou et al. , 2019b ) . Virtual Adversarial Training ( VAT ) ( Miyato et al. , 2019 ) is a semi-supervised learning method for improving robustness against local perturbations of the input . Using an iterative method based on power iteration , it approximates the adversarial direction corresponding to certain input points . Perturbing an input towards its adversarial direction changes the network ’ s output the most . Inspired by VAT , we propose a method called Adversarial Lipschitz Regularization ( ALR ) , enabling the training of neural networks with regularization terms penalizing the violation of the Lipschitz constraint explicitly , instead of through the norm of the gradient . It provides means to generate a pair for each input point , for which the Lipschitz constraint is likely to be violated with high probability . In general , enforcing Lipschitz continuity of complex models can be useful for a lot of applications . In this work , we focus on applying ALR to Wasserstein GANs , as regularizing or constraining Lipschitz continuity has proven to have a high impact on training stability and reducing mode collapse . Source code to reproduce the presented experiments is available at https : //github.com/dterjek/adversarial_lipschitz_regularization . Our contributions are as follows : • We propose Adversarial Lipschitz Regularization ( ALR ) and apply it to penalize the violation of the Lipschitz constraint directly , resulting in Adversarial Lipschitz Penalty ( ALP ) . • Applying ALP on the critic in WGAN ( WGAN-ALP ) , we show state-of-the-art performance in terms of Inception Score and Fréchet Inception Distance among non-progressive growing methods trained on CIFAR-10 , and competitive performance in the high-dimensional setting when applied to the critic in Progressive Growing GAN trained on CelebA-HQ . 2 BACKGROUND . 2.1 WASSERSTEIN GENERATIVE ADVERSARIAL NETWORKS . Generative adversarial networks ( GANs ) provide generative modeling by a generator network g that transforms samples of a low-dimensional latent space Z into samples from the data space X , transporting mass from a fixed noise distribution PZ to the generated distribution Pg . The generator is trained simultaneously with another network f called the discriminator , which is trained to distinguish between fake samples drawn from Pg and real samples drawn from the real distribution Pr , which is often represented by a fixed dataset . This network provides the learning signal to the generator , which is trained to generate samples that the discriminator considers real . This iterative process implements the minimax game min g max f Ex∼Pr log ( f ( x ) ) + Ez∼PZ log ( 1− f ( g ( z ) ) ) ( 1 ) played by the networks f and g. This training procedure minimizes the approximate Jensen-Shannon divergence ( JSD ) between Pr and Pg ( Goodfellow et al. , 2014 ) . However , during training these two distributions might differ strongly or even have non-overlapping supports , which might result in gradients received by the generator that are unstable or zero ( Arjovsky and Bottou , 2017 ) . Wasserstein GAN ( WGAN ) ( Arjovsky et al. , 2017 ) was proposed as a solution to this instability . Originating from Optimal Transport theory ( Villani , 2008 ) , the Wasserstein metric provides a distance between probability distributions with much better theoretical and practical properties than the JSD . It provides a smooth optimizable distance even if the two distributions have non-overlapping supports , which is not the case for JSD . It raises a metric dX from the space X of the supports of the probability distributions P1 and P2 to the space of the probability distributions itself . For these purposes , the Wasserstein-p distance requires the probability distributions to be defined on a metric space and is defined as Wp ( P1 , P2 ) = ( inf π∈Π ( P1 , P2 ) E ( x1 , x2 ) ∼πdX ( x1 , x2 ) p ) 1 p , ( 2 ) where Π ( P1 , P2 ) is the set of distributions on the product space X ×X whose marginals are P1 and P2 , respectively . The optimal π achieving the infimum in ( 2 ) is called the optimal coupling of P1 and P2 , and is denoted by π∗ . The case of p = 1 has an equivalent formulation W1 ( P1 , P2 ) = sup ‖f‖L≤1 Ex∼P1f ( x ) − Ex∼P2f ( x ) , ( 3 ) called the Kantorovich-Rubinstein formula ( Villani , 2008 ) , where f : X → R is called the potential function , ‖f‖L ≤ 1 is the set of all functions that are 1-Lipschitz with respect to the ground metric dX , and the Wasserstein-1 distance corresponds to the supremum over all 1-Lipschitz potential functions . The smallest Lipschitz constant for a real-valued function f with the metric space ( X , dX ) as its domain is given by ‖f‖L = sup x , y∈X ; x 6=y |f ( x ) − f ( y ) | dX ( x , y ) . ( 4 ) Based on ( 3 ) , the critic in WGAN ( Arjovsky et al. , 2017 ) implements an approximation of the Wasserstein-1 distance between Pg and Pr . The minimax game played by the critic f and the generator g becomes min g max ‖f‖L≤1 Ez∼PZf ( g ( z ) ) − Ex∼Prf ( x ) , ( 5 ) a formulation that proved to be superior to the standard GAN in practice , with substantially more stable training behaviour and improved sample quality ( Arjovsky et al. , 2017 ) , although recent GAN variants do not always use this objective ( Brock et al. , 2019 ) . With WGAN , the challenge became effectively restricting the smallest Lipschitz constant of the critic f , sparking the birth of a plethora of Lipschitz regularization techniques for neural networks . 2.2 LIPSCHITZ FUNCTION APPROXIMATION . A general definition of the smallest Lipschitz constant of a function f : X → Y is ‖f‖L = sup x , y∈X ; x 6=y dY ( f ( x ) , f ( y ) ) dX ( x , y ) , ( 6 ) where the metric spaces ( X , dX ) and ( Y , dY ) are the domain and codomain of the function f , respectively . The function f is called Lipschitz continuous if there exists a real constant K ≥ 0 for which dY ( f ( x ) , f ( y ) ) ≤ K · dX ( x , y ) for any x , y ∈ X . Then , the function f is also called K-Lipschitz . Theoretical properties of neural networks with low Lipschitz constants were explored in Oberman and Calder ( 2018 ) , Bartlett ( 1998 ) and Drucker and LeCun ( 1992 ) , showing that it induces better generalization . Learning mappings with Lipschitz constraints became prevalent in the field of deep learning with the introduction of WGAN ( Arjovsky et al. , 2017 ) . Enforcing the Lipschitz property on the critic was first done by clipping the weights of the network . This approach achieved superior results compared to the standard GAN formulation , but still sometimes yielded poor quality samples or even failed to converge . While clipping the weights enforces a global Lipschitz constant , it also reduces the function space , which might not include the optimal critic any more . Soon this method has been replaced by a softened one called Gradient Penalty ( GP ) ( Gulrajani et al. , 2017 ) . Motivated by the fact that the optimal critic should have unit gradient norm on lines connecting the coupled points ( x1 , x2 ) ∼ π∗ according to ( 2 ) , they proposed a regularizer that enforces unit gradient norm along these lines , which not only enforces the Lipschitz constraint , but other properties of the optimal solution as well . However , π∗ is not known in practice , which is why Gulrajani et al . ( 2017 ) proposed to apply GP on samples of the induced distribution Pi , by interpolating samples from the marginals P1 and P2 . The critic in the WGAN-GP formulation is regularized with the loss λEx∼Pi ( ‖∇xf ( x ) ‖2 − 1 ) 2 ( 7 ) where Pi denotes the distribution of samples obtained by interpolating pairs of samples drawn from Pr and Pg , and λ is a hyperparameter acting as a Lagrange multiplier . Theoretical arguments against GP were pointed out by Petzka et al . ( 2018 ) and Gemici et al . ( 2018 ) , arguing that unit gradient norm on samples of the distribution Pi is not valid , as the pairs of samples being interpolated are generally not from the optimal coupling π∗ , and thus do not necessarily need to match gradient norm 1 . Furthermore , they point out that differentiability assumptions of the optimal critic are not met . Therefore , the regularizing effect of GP might be too strong . As a solution , Petzka et al . ( 2018 ) suggested using a loss penalizing the violation of the Lipschitz constraint either explicitly with λEx , y∼Pτ ( |f ( x ) − f ( y ) | ‖x− y‖2 − 1 ) 2 + ( 8 ) or implicitly with λEx∼Pτ ( ‖∇xf ( x ) ‖2 − 1 ) 2 + ( 9 ) where in both cases ( a ) + denotes max ( 0 , a ) . The first method has only proved viable when used on toy datasets , and led to considerably worse results on relatively more complex datasets like CIFAR-10 , which is why Petzka et al . ( 2018 ) used the second one , which they termed Lipschitz Penalty ( LP ) . Compared to GP , this term only penalizes the gradient norm when it exceeds 1 . As Pτ they evaluated the interpolation method described above , and also sampling random local perturbations of real and generated samples , but found no significant improvement compared to Pi . Wei et al . ( 2018 ) proposed dropout in the critic as a way for creating perturbed input pairs to evaluate the explicit Lipschitz penalty ( 8 ) , which led to improvements , but still relied on using GP simultaneously . A second family of Lipschitz regularization methods is based on weight normalization , restricting the Lipschitz constant of a network globally instead of only at points of the input space . One such technique is called spectral normalization ( SN ) proposed in Miyato et al . ( 2018 ) , which is a very efficient and simple method for enforcing a Lipschitz constraint with respect to the 2-norm on a per-layer basis , applicable to neural networks consisting of affine layers and K-Lipschitz activation functions . Gouk et al . ( 2018 ) proposed a similar approach , which can be used to enforce a Lipschitz constraint with respect to the 1-norm and ∞-norm in addition to the 2-norm , while also being compatible with batch normalization and dropout . Anil et al . ( 2019 ) argued that any Lipschitzconstrained neural network must preserve the norm of the gradient during backpropagation , and to this end proposed another weight normalization technique ( showing that it compares favorably to SN , which is not gradient norm preserving ) , and an activation function based on sorting .

Summary: Virtual Adversarial Training (Miyato et al., 2017) can be viewed as a form of Lipschitz regularization. Inspired by this, the paper proposes a Lipschitz regularization technique that tries to ensure that the function being regularized doesn’t change a lot in virtual adversarial directions. This method is shown to be effective in training Wasserstein GANs.