Adversarial Lipschitz Regularization

1 INTRODUCTION . In recent years , Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) have been becoming the state-of-the-art in several generative modeling tasks , ranging from image generation ( Karras et al. , 2018 ) to imitation learning ( Ho and Ermon , 2016 ) . They are based on an idea of a two-player game , in which a discriminator tries to distinguish between real and generated data samples , while a generator tries to fool the discriminator , learning to produce realistic samples on the long run . Wasserstein GAN ( WGAN ) was proposed as a solution to the issues present in the original GAN formulation . Replacing the discriminator , WGAN trains a critic to approximate the Wasserstein distance between the real and generated distributions . This introduced a new challenge , since Wasserstein distance estimation requires the function space of the critic to only consist of 1-Lipschitz functions . To enforce the Lipschitz constraint on the WGAN critic , Arjovsky et al . ( 2017 ) originally used weight clipping , which was soon replaced by the much more effective method of Gradient Penalty ( GP ) ( Gulrajani et al. , 2017 ) , which consists of penalizing the deviation of the critic ’ s gradient norm from 1 at certain input points . Since then , several variants of gradient norm penalization have been introduced ( Petzka et al. , 2018 ; Wei et al. , 2018 ; Adler and Lunz , 2018 ; Zhou et al. , 2019b ) . Virtual Adversarial Training ( VAT ) ( Miyato et al. , 2019 ) is a semi-supervised learning method for improving robustness against local perturbations of the input . Using an iterative method based on power iteration , it approximates the adversarial direction corresponding to certain input points . Perturbing an input towards its adversarial direction changes the network ’ s output the most . Inspired by VAT , we propose a method called Adversarial Lipschitz Regularization ( ALR ) , enabling the training of neural networks with regularization terms penalizing the violation of the Lipschitz constraint explicitly , instead of through the norm of the gradient . It provides means to generate a pair for each input point , for which the Lipschitz constraint is likely to be violated with high probability . In general , enforcing Lipschitz continuity of complex models can be useful for a lot of applications . In this work , we focus on applying ALR to Wasserstein GANs , as regularizing or constraining Lipschitz continuity has proven to have a high impact on training stability and reducing mode collapse . Source code to reproduce the presented experiments is available at https : //github.com/dterjek/adversarial_lipschitz_regularization . Our contributions are as follows : • We propose Adversarial Lipschitz Regularization ( ALR ) and apply it to penalize the violation of the Lipschitz constraint directly , resulting in Adversarial Lipschitz Penalty ( ALP ) . • Applying ALP on the critic in WGAN ( WGAN-ALP ) , we show state-of-the-art performance in terms of Inception Score and Fréchet Inception Distance among non-progressive growing methods trained on CIFAR-10 , and competitive performance in the high-dimensional setting when applied to the critic in Progressive Growing GAN trained on CelebA-HQ . 2 BACKGROUND . 2.1 WASSERSTEIN GENERATIVE ADVERSARIAL NETWORKS . Generative adversarial networks ( GANs ) provide generative modeling by a generator network g that transforms samples of a low-dimensional latent space Z into samples from the data space X , transporting mass from a fixed noise distribution PZ to the generated distribution Pg . The generator is trained simultaneously with another network f called the discriminator , which is trained to distinguish between fake samples drawn from Pg and real samples drawn from the real distribution Pr , which is often represented by a fixed dataset . This network provides the learning signal to the generator , which is trained to generate samples that the discriminator considers real . This iterative process implements the minimax game min g max f Ex∼Pr log ( f ( x ) ) + Ez∼PZ log ( 1− f ( g ( z ) ) ) ( 1 ) played by the networks f and g. This training procedure minimizes the approximate Jensen-Shannon divergence ( JSD ) between Pr and Pg ( Goodfellow et al. , 2014 ) . However , during training these two distributions might differ strongly or even have non-overlapping supports , which might result in gradients received by the generator that are unstable or zero ( Arjovsky and Bottou , 2017 ) . Wasserstein GAN ( WGAN ) ( Arjovsky et al. , 2017 ) was proposed as a solution to this instability . Originating from Optimal Transport theory ( Villani , 2008 ) , the Wasserstein metric provides a distance between probability distributions with much better theoretical and practical properties than the JSD . It provides a smooth optimizable distance even if the two distributions have non-overlapping supports , which is not the case for JSD . It raises a metric dX from the space X of the supports of the probability distributions P1 and P2 to the space of the probability distributions itself . For these purposes , the Wasserstein-p distance requires the probability distributions to be defined on a metric space and is defined as Wp ( P1 , P2 ) = ( inf π∈Π ( P1 , P2 ) E ( x1 , x2 ) ∼πdX ( x1 , x2 ) p ) 1 p , ( 2 ) where Π ( P1 , P2 ) is the set of distributions on the product space X ×X whose marginals are P1 and P2 , respectively . The optimal π achieving the infimum in ( 2 ) is called the optimal coupling of P1 and P2 , and is denoted by π∗ . The case of p = 1 has an equivalent formulation W1 ( P1 , P2 ) = sup ‖f‖L≤1 Ex∼P1f ( x ) − Ex∼P2f ( x ) , ( 3 ) called the Kantorovich-Rubinstein formula ( Villani , 2008 ) , where f : X → R is called the potential function , ‖f‖L ≤ 1 is the set of all functions that are 1-Lipschitz with respect to the ground metric dX , and the Wasserstein-1 distance corresponds to the supremum over all 1-Lipschitz potential functions . The smallest Lipschitz constant for a real-valued function f with the metric space ( X , dX ) as its domain is given by ‖f‖L = sup x , y∈X ; x 6=y |f ( x ) − f ( y ) | dX ( x , y ) . ( 4 ) Based on ( 3 ) , the critic in WGAN ( Arjovsky et al. , 2017 ) implements an approximation of the Wasserstein-1 distance between Pg and Pr . The minimax game played by the critic f and the generator g becomes min g max ‖f‖L≤1 Ez∼PZf ( g ( z ) ) − Ex∼Prf ( x ) , ( 5 ) a formulation that proved to be superior to the standard GAN in practice , with substantially more stable training behaviour and improved sample quality ( Arjovsky et al. , 2017 ) , although recent GAN variants do not always use this objective ( Brock et al. , 2019 ) . With WGAN , the challenge became effectively restricting the smallest Lipschitz constant of the critic f , sparking the birth of a plethora of Lipschitz regularization techniques for neural networks . 2.2 LIPSCHITZ FUNCTION APPROXIMATION . A general definition of the smallest Lipschitz constant of a function f : X → Y is ‖f‖L = sup x , y∈X ; x 6=y dY ( f ( x ) , f ( y ) ) dX ( x , y ) , ( 6 ) where the metric spaces ( X , dX ) and ( Y , dY ) are the domain and codomain of the function f , respectively . The function f is called Lipschitz continuous if there exists a real constant K ≥ 0 for which dY ( f ( x ) , f ( y ) ) ≤ K · dX ( x , y ) for any x , y ∈ X . Then , the function f is also called K-Lipschitz . Theoretical properties of neural networks with low Lipschitz constants were explored in Oberman and Calder ( 2018 ) , Bartlett ( 1998 ) and Drucker and LeCun ( 1992 ) , showing that it induces better generalization . Learning mappings with Lipschitz constraints became prevalent in the field of deep learning with the introduction of WGAN ( Arjovsky et al. , 2017 ) . Enforcing the Lipschitz property on the critic was first done by clipping the weights of the network . This approach achieved superior results compared to the standard GAN formulation , but still sometimes yielded poor quality samples or even failed to converge . While clipping the weights enforces a global Lipschitz constant , it also reduces the function space , which might not include the optimal critic any more . Soon this method has been replaced by a softened one called Gradient Penalty ( GP ) ( Gulrajani et al. , 2017 ) . Motivated by the fact that the optimal critic should have unit gradient norm on lines connecting the coupled points ( x1 , x2 ) ∼ π∗ according to ( 2 ) , they proposed a regularizer that enforces unit gradient norm along these lines , which not only enforces the Lipschitz constraint , but other properties of the optimal solution as well . However , π∗ is not known in practice , which is why Gulrajani et al . ( 2017 ) proposed to apply GP on samples of the induced distribution Pi , by interpolating samples from the marginals P1 and P2 . The critic in the WGAN-GP formulation is regularized with the loss λEx∼Pi ( ‖∇xf ( x ) ‖2 − 1 ) 2 ( 7 ) where Pi denotes the distribution of samples obtained by interpolating pairs of samples drawn from Pr and Pg , and λ is a hyperparameter acting as a Lagrange multiplier . Theoretical arguments against GP were pointed out by Petzka et al . ( 2018 ) and Gemici et al . ( 2018 ) , arguing that unit gradient norm on samples of the distribution Pi is not valid , as the pairs of samples being interpolated are generally not from the optimal coupling π∗ , and thus do not necessarily need to match gradient norm 1 . Furthermore , they point out that differentiability assumptions of the optimal critic are not met . Therefore , the regularizing effect of GP might be too strong . As a solution , Petzka et al . ( 2018 ) suggested using a loss penalizing the violation of the Lipschitz constraint either explicitly with λEx , y∼Pτ ( |f ( x ) − f ( y ) | ‖x− y‖2 − 1 ) 2 + ( 8 ) or implicitly with λEx∼Pτ ( ‖∇xf ( x ) ‖2 − 1 ) 2 + ( 9 ) where in both cases ( a ) + denotes max ( 0 , a ) . The first method has only proved viable when used on toy datasets , and led to considerably worse results on relatively more complex datasets like CIFAR-10 , which is why Petzka et al . ( 2018 ) used the second one , which they termed Lipschitz Penalty ( LP ) . Compared to GP , this term only penalizes the gradient norm when it exceeds 1 . As Pτ they evaluated the interpolation method described above , and also sampling random local perturbations of real and generated samples , but found no significant improvement compared to Pi . Wei et al . ( 2018 ) proposed dropout in the critic as a way for creating perturbed input pairs to evaluate the explicit Lipschitz penalty ( 8 ) , which led to improvements , but still relied on using GP simultaneously . A second family of Lipschitz regularization methods is based on weight normalization , restricting the Lipschitz constant of a network globally instead of only at points of the input space . One such technique is called spectral normalization ( SN ) proposed in Miyato et al . ( 2018 ) , which is a very efficient and simple method for enforcing a Lipschitz constraint with respect to the 2-norm on a per-layer basis , applicable to neural networks consisting of affine layers and K-Lipschitz activation functions . Gouk et al . ( 2018 ) proposed a similar approach , which can be used to enforce a Lipschitz constraint with respect to the 1-norm and ∞-norm in addition to the 2-norm , while also being compatible with batch normalization and dropout . Anil et al . ( 2019 ) argued that any Lipschitzconstrained neural network must preserve the norm of the gradient during backpropagation , and to this end proposed another weight normalization technique ( showing that it compares favorably to SN , which is not gradient norm preserving ) , and an activation function based on sorting .

Summary: It is an interesting idea about how to enforce the Lipsthitz constrain in WGAN by using virtual adversarial training. The connection between virtual adversarial and this paper method - ALR is quite simple and clear. In the experiments, the FID score in the table is not complete which can not clearly compare the ability of the Lipschitz regularization to other regularization methods. The paper addresses that the approximation of r_{adv} will affect the performance of ALR. How to balance the quality and computation complexity is quite important. This paper did not provide the reason about why this method can not work better than GP method in high-dimensional setting.