Adversarial Lipschitz Regularization
Authors: Dávid Terjék
Venue:  (2020)
Keywords: generative adversarial networks
wasserstein generative adversarial networks
lipschitz regularization
adversarial training

Abstract: Generative adversarial networks (GANs) are one of the most popular approaches when it comes to training generative models, among which variants of Wasserstein GANs are considered superior to the standard GAN formulation in terms of learning stability and sample quality. However, Wasserstein GANs require the critic to be 1-Lipschitz, which is often enforced implicitly by penalizing the norm of its gradient, or by globally restricting its Lipschitz constant via weight normalization techniques. Training with a regularization term penalizing the violation of the Lipschitz constraint explicitly, instead of through the norm of the gradient, was found to be practically infeasible in most situations. Inspired by Virtual Adversarial Training, we propose a method called Adversarial Lipschitz Regularization, and show that using an explicit Lipschitz penalty is indeed viable and leads to competitive performance when applied to Wasserstein GANs, highlighting an important connection between Lipschitz regularization and adversarial training.

Published as a conference paper at ICLR 2020
ADVERSARIAL LIPSCHITZ REGULARIZATION
Dávid Terjék Robert Bosch Kft. Budapest, Hungary
tevid.terjek@nee www.h.ooochcoooooo com
ABSTRACT
Generative adversarial networks (GANs) are one of the most popular approaches when it comes training generative models, among which variants of Wasserstein GANs are considered superior to the standard GAN formulation in terms of learn- ing stability and sample quality, However, Wasserstein GANs require the critic be I-Lipschitz, which is often enforced implicitly by penalizing the norm of its gradient, by globally restricting its Lipschitz constant via weight normalization techniques. Training with regularization term penalizing the violation of the Lipschitz constraint explicitly, instead of through the norm of the gradient, was found to be practically infeasible in most situations. Inspired by Virtual Adversarial Training, we propose a method called Adversarial Lipschitz Regularization, and show that using an explicit Lipschitz penalty is indeed viable and leads to competi- tive performance when applied to Wasserstein GANs, highlighting an important connection between Lipschitz regularization and adversarial training.
INTRODUCTION
In recent years, Generative adversarial networks (GANs) (Goodfellow et al., 2014) have been becoming the oftte-of-thearr several generative modeling tasks, ranging from image generation (Karras et al. 2018) to imitation learning (Ho and Ermon 2016). They are based on an idea of two-player game, in which a discriminator tries to distinguish between real and generated data samples, while generator tries to fool the discriminator, learning to produce realistic samples on the long run. Wasserstein GAN (WGAN) was proposed as a solution the issues present in the original GAN formulation. Replacing the discriminator, WGAN trains critic to approximate the Wasserstein distance between the real and generated distributions. This introduced a new challenge, since Wasserstein distance estimation requires the function space of the critic to only consist of 1-Lipschitz functions.
To enforce the Lipschitz constraint the WGAN critie, Arjovsky et al. (017)) originally used weight clipping, which was soon replaced by the much more effective method of Gradient Penalty (GP) (Gulrajani al., 2017), which consists of penalizing the deviation of the critic's gradient norm from at certain input points. Since then, several variants of gradient norm penalization have been introduced (Petzka al. 2018 Wei al. 2018 Adler and Lunz 2018 Zhou et 2019b).
Virtual Adversarial Training (VAT) MMiyato et al. 2019) is a semi-supervised learning method for improving robustness against local perturbations of the input. Using iterative method based on power iteration, it approximates the adversarial direction corresponding to certain input points. Perturbing an input towards its adversarial direction changes the network's output the most.
Inspired by VAT, we propose a method called Adversarial Lipschitz Regularization (ALR), en- abling the training of neural networks with regularization terms penalizing the violation of the Lipschitz constraint explicitly, instead of through the norm of the gradient. It provides means to generate a pair for each input point, for which the Lipschitz constraint is likely to be violated with high probability. In general, enforcing Lipschitz continuity of complex models can be useful for a lot of applications. In this work, we focus on applying ALR to Wasserstein GANs, as regular- izing constraining Lipschitz continuity has proven have high impact training stability and reducing mode collapse. Source code to reproduce the presented experiments is available at https://gihhb..co//tteee/aaaaaee

---

Published as a conference paper at ICLR 2020
Our contributions are as follows:
We propose Adversarial Lipschitz Regularization (ALR) and apply it to penalize the violation of the Lipschitz constraint directly, resulting in Adversarial Lipschitz Penalty (ALP).
Applying ALP on the critic in WGAN (WGAN-ALP), we show state-of-the performance in terms of Inception Score and Fréchet Inception Distance among non- on--progressve growing methods trained on CIFAR-10, and competitive performance in the high-dimensional setting when applied to the critic Progressive Growing GAN trained on CelebA-HQ.
2 BACKGROUND
2.1 WASSERSTEIN GENERATIVE ADVERSARIAT. NETWORKS
Generative adversarial networks (GANs) provide generative modeling by a generator network that transforms samples of low-dimensional latent space Z into samples from the data space X, transporting mass from a fixed noise distribution Pz to the generated distribution Pg- The generator is trained simultaneously with another network f called the discriminator, which is trained distinguish between fake samples drawn from Pg and real samples drawn from the real distribution Pr. which is often represented by fixed dataset. This network provides the learning signal to the generator, which is trained to generate samples that the discriminator considers real. This iterative process implements the minimax game
min max max Er lo)(f(x)) + Eg log(1 - f((()))
1
played by the networks and g. This training procedure minimizes the approximate Jensen-Shanme divergence (ISD) between P. and Pg (Goodfellow et al. 2014). However, during training these two distributions might differ strongly or even have non-overlapping supports, which might result in gradients received by the generator that are unstable or zero (Arjovsky and Bottou,[011
Wasserstein GAN (WGAN) (Arjovsky al.) 2017) was proposed as a solution this instability. Originating from Optimal Transport theory /Villani 2008), the Wasserstein metric provides distance between probability distributions with much better theoretical and practical properties than the JSD, It provides smooth optimizable distance even if the two distributions have non-overlapping supports, which is not the case for JSD. It raises metric dx from the space X of the supports of the probability distributions P1 and Pg to the space of the probability distributions itself. For these purposes, the Wasserstein-p distance requires the probability distributions to be defined on a metric space and is defined as
Wp (P,,
inf ((e dx (21, (FEII(P)PP))
(x (21,
(2)
where II(P,, P22) is the set of distributions the product space X x X whose marginals are P1 and P2, respectively. The optimal π achieving the infimum in (2) is called the optimal coupling of P; and Pz, and is denoted by n². The case of p has equivalent formulation
Wi(PI, P2) = sup E Ez ((x),
called the Sectoooovicc-Rubinsseie formula (Villani 2008), where f: X + R is called the potential function, |[fllz < 1 is the set of all functions that are I-Lipschitz with respect to the ground metric dx, and the Wasserstein-1 I distance corresponds to the supremum over all I-Lipschitz potential functions. The smallest Lipschitz constant for a real-valued function f with the metric space (X, dx) as its domain is given
[fxx) [wv)
lff sup x. dx (z. v)
(4)
Based on 3 the critic in WGAN (Arjovsky et al., 2017) implements an approximation of the Wasserstein-1 distance between Pg and P.. The minimax game played by the critic and the generator g becomes
min max fl(r)(((>>)) Ez-r, I
(5)
2

---

Published as a conference paper at ICLR 2020
formulation that proved to be superior to the standard GAN in practice, with substantially more stable training behaviour and improved sample quality (Arjovsky et al., (017)) although recent GAN variants do not always use this objective (Brock et al. 2019). With WGAN, the challenge became effectively restricting the smallest Lipschitz constant of the critic f. sparking the birth of a plethora of Lipschitz regularization techniques for neural networks.
2.2 LIPSCHITZ. FUNCTION APPROXIMATION
A general definition of the smallest Lipschitz constant of a function f: + Y
dy dy()),),
|llfle sup
(6)
dx(2, y)
where the metric spaces (X, dx) and dy, are the domain and codomain of the function f. respectively. The function f is called Lipschitz continuous if there exists a real constant 2 0
for which dy (xx), (() S K dx y) for any I, E X. Then, the function f is also called K-Lipschitz. Theoretical properties of neural networks with low Lipschitz constants were explored in
Oberman and Calder (2018) Bartlett (1998) and Drucker and LeCun 1992). showing that it induces better generalization.
Learning mappings with Lipschitz constraints became prevalent in the field of deep learning with the introduction of WGAN (Arjovsky et al. (017). Enforcing the Lipschitz property on the critic was first done by clipping the weights of the network. This approach achieved superior results compared to the standard GAN formulation, but still sometimes yielded poor quality samples or even failed to converge. While clipping the weights enforces a global Lipschitz constant, it also reduces the function space, which might not include the optimal critic any more. Soon this method has been replaced by a softened one called Gradient Penalty (GP) (Gulrajani et al. 2017). Motivated by the fact that the optimal critic should have unit gradient norm on lines connecting the coupled points (21, .22) ~ T" according to (2). they proposed a regularizer that enforces unit gradient norm along these lines, which not only enforces the I Lipschitz constraint, but other properties of the optimal solution as well. However, π is not known in practice, which is why Gulrajani et al. (2017) proposed to apply GP on samples of the induced distribution P., by interpolating samples from the marginals Pi and Pz. The critic in the WGAN-GP formulation is regularized with the loss
P P. (IIV (x))la 1)²
where Pj denotes the distribution of samples obtained by interpolating pairs of samples drawn from P. and Pg. and À is a hyperparameter acting as Lagrange multiplier.
Theoretical arguments against GP were pointed out by Petzka al. (2018) and Gemici et al. (2018). arguing that unit gradient norm on samples of the distribution P. is not valid, as the pairs of samples being interpolated are generally not from the optimal coupling π., and thus do not necessarily need to match gradient norm 1. Furthermore, they point out that differentiability assumptions of the optimal critic are not met. Therefore, the regularizing effect of GP might be too strong. As a solution, Petzka et al. 2018) suggested using loss penalizing the violation of the Lipschitz constraint either explicitly with
(1(xx) s(w)l -1)
2
AE
| y112
or implicitly with
AE (V))2(()) 1)
where in both cases (a)+ denotes max(0, a). The first method has only proved viable when used on toy datasets, and led to considerably worse results on relatively more complex datasets like 100 which is why |Petzka et al. (2018) used the second one, which they termed I. Lipschitz Penalty (LP). Compared to GP, this term only penalizes the gradient norm when it exceeds 1. As Pr they evaluated the interpolation method described above, and also sampling random local perturbations of real and generated samples, but found no significant improvement compared to P.. Wei et al. (2018) proposed dropout in the critic as a way for creating perturbed input pairs to evaluate the explicit Lipschitz penalty which led to improvements, but still relied on using GP simultaneously.
A second family of Lipschitz regularization methods is based on weight normalization, restricting the Lipschitz constant of network globally instead of only at points of the input space. One such
3

---

Published as a conference paper at ICLR 2020
technique is called spectral normalization (SN) proposed in Miyato al. (2018) which is a very efficient and simple method for enforeing a Lipschitz constraint with respect to the 2-norm on a per-layer basis, applicable to neural networks consisting of affine layers and K-Lipschitz activation functions. /ouu et al. (2018) proposed a similar approach, which can be used to enforce a Lipschitz constraint with respect to the 1-norm and oc-norm in addition to the 2-norm, while also being compatible with batch normalization and dropout. Anil et al. (2019) argued that any Lipschitz- constrained neural network must preserve the norm of the gradient during backpropagation, and to this end proposed another weight normalization technique (showing that it compares favorably to SN, which is not gradient norm preserving). and an activation function based on sorting.
2.3 VIRTUAL ADVERSARIAL TRAINING
VAT (Miyato et al. 2019) is semi-supervised learning method that is able to regularize networks to be robust to local adversarial perturbation. Virtual adversarial perturbation means perturbing input sample points in such way that the change in the output of the network induced by the perturbation maximal in terms of a distance between distributions. This defines a direction for each sample point called the virtual adversarial direction, in which the perturbation is performed. It is called virtual to make the distinction with the adversarial direction introduced in Goodfellow et al. (2015) clear, VAT uses unlabeled data with virtual labels, assigned to the sample points by the network being trained. The regularization term of VAT is called Local Distributional Smoothness (LDS). is defined as
LLDS = D (p(ulæ), Plylæ + rvado)),
(10)
where p is a conditional distribution implemented by a neural network, D(p, p') is divergence between two distributions p and p', for which Miyato et al. (2019) chose the Kullback- divergence (KLD), and
Teade = arg max (p(v)z), +r)
(11)
||llaae
the virtual adversarial perturbation, where c is a hyperparameter. VAT is defined as a training method with the regularizer (10) applied to labeled and unlabeled examples. An important detail is
that (TO) is minimized by keeping p(y(<r fixed and optimizing p(y|æ + Toado) to be close to it.
The adversarial perturbation is approximated by the power iteration Teade eTk, where
V,D D.D p(yl) + r))
Ti+1
r
Er:
(12)
V,D D,D(p(y) (p(ylx), + r))
To is a randomly sampled unit vector and & is another hyperparameter. This iterative scheme is an approximation of the direction at æ that induces the greatest change in the output of p in terms of the divergence D. Miyato et al. (019)) found that l- = iteration is sufficient in practical situations.
3 ADVERSARIAL LIPSCHITZ REGULARIZATION
Adler and Lunz (018) argued that penalizing the norm of the gradient as in 9 more effective than penalizing the Lipschitz quotient directly as in (8), as the former penalizes the slope of f in all spatial directions around v, unlike the latter, which does so only along (x - y). We hypothesize that using the explicit Lipschitz penalty in itself is insufficient because if one takes pairs of samples x, randomly from P,, Po or P, (or just one sample and generates pair for it with random perturbation), the violation of the Lipschitz penalty evaluated at these sample pairs will be far from its maximum, hence a more sophisticated strategy for sampling pairs is required. As we will show, a carefully chosen sampling strategy can in fact make the explicit penalty favorable over the implicit one.
Consider the network f as a mapping from the metric space (X, dx) to the metric space (Y, dy). Let us rewrite (6) with y = x + to get
dy (f dx (x. +r)
|
sup a. XDddxexxddxxx2xx
(13)
given mapping f is K-Lipschitz and only if for any given x E X, taking the supremum over r' in (13 results in value K or smaller. Assuming that this supremum is always achieved for some r,
4

---

Published as a conference paper at ICLR 2020
can define notion of adversarial perturbation with respect to the Lipschitz continuity for given I E X as
dy ((((())
Tade
arg max 2+rE x0cdx (2,0+))
dx(x,x+r)
(14)
and the corresponding maximal violation of the K-Lipschitz constraint
(f(x),),, f(x Fodu)) K
LLLP dx(x, (
(15)
Tado)
We define Adversarial Lipschitz Regularization (ALR) as the method adding (1) as a regularization term to the training objective that penalizes the violation of the Lipschitz constraint evaluated at sample pairs obtained by adversarial perturbation. We call this term Adversarial Lipschitz Penalty (ALP).
To put it in words, ALP measures the deviation of f from being K-Lipschitz evaluated at pairs of the adversarial perturbation of the other. If added to the training objective,
sample points where it makes the learned mapping approximately K- K-Lipschitz around the sample points it is applied at. We found that in the case of the WGAN critic is best minimize (15) without keeping (a) fixed. See Appendix A for the eune-supereieed case and Appendix A.2 for how VAT can be seen as special case of Lipschitz regularization.
3.1 APPROXIMATION OF Tade
In general, computing the adversarial perturbation (14) is nonlinear optimization problem. A crude and cheap approximation is Tado eTk, where
d, (I(x), +r)
T:+1
r=
(16)
v,dy (f(z), r))
the approximated adversarial direction with ro being randomly sampled unit vector. The derivation of this formula is essentially the same as the one described in Miyato et al. (2019), but is included in Appendix A.3 for completeness. Unlike in VAT, we do not fix e, but draw it randomly from predefined distribution P. over R4 to apply the penalty at different scales.
Theoretically, ALR be used with all kinds of metries dx and dy, and any kind of model f. but the approximation of Tade imposes a practical restriction. It approximates the adversarial perturbation of x as a translation with length with respect to the 2-norm in the adversarial direction, which is only a perfect approximation if the ratio in (15) is constant for any 0. This idealized setting is hardly ever the case, which is why we see the search for other approximation schemes as an important
future direction. There is large number of methods for generating adversarial examples besides the one proposed in VAT (Shafahi et al. 2019; Wong et al. 2019 Khrulkov and Oseledets) (018)), which could possibly be combined with ALR either to improve the approximation performance to make it possible with new kinds of metrics. The latter is important since one of the strengths of the Wasserstein distance is that it can be defined with any metric dx. a fact that Adler and Lunz (2018) and Dukler et al.] (2019) built on by extending GP to work with metrics other than the Fuc distance. [dler and Lunz (2018) emphasized the fact that through explicit Lipschitz penalties one could extend WGANs to more general metric spaces as well.
3.2 HYPERPARMETERS
In practice, one adds the Monte Carlo approximation of the expectation (averaged over a minibatch of samples) of either (15) or the square of (15) (or both) to the training objective, multiplied by
Lagrange multiplier x. While VAT adds the expectation of (10) to the training objective, for WGAN we have added the square of the expectation of (15). To train the Progressive GAN, we have added both the expectation and its square. In the semi-supervised setting, we added only the expectation similarly to VAT. We have found these choices to work best in these scenarios, but principled answer to this question is beyond the scope of this paper. The target Lipschitz constant can be tuned by hand, in the presence of labeled data it is possible to caleulate the Lipschitz constant of the dataset
5

---

Published as a conference paper at ICLR 2020
(Oberman and Calder 2018). The hyperparameters of the approximation scheme are k. and those of P..
Choosing the right hyperparameters can be done by monitoring the number of adversarial pertur- bations found by the algorithm for which the Lipschitz constraint is violated (and hence contribute nonzero value the expectation of (15)), and tuning the hyperparameters in order keep this number balanced between its maximum (which is the minibatch size) and its minimum (which is high, means that either K is too small and should be increased, or the regularization effect is too weak, so one should increase A. If it is too low, then either the regularization effect is too strong, or ALR is parametrized in a way that it cannot find Lipschitz constraint violations efficiently. In the former case, one should decrease X. In the latter, one should either decrease K, tune the parameters of P.. or increase the number power iterations for the price of increased runtime. We have not observed any significant effect when changing the value of & in any of the tasks considered.
0). If
3.3 COMPARISON WITH OTHER LIPSCHITZ REGULARIZATION TECHNIQUES
In terms of efficiency when applied to WGANs. ALR compares favorably to the implicit methods penalizing the gradient norm, and weight normalization techniques as well, as demonstrated in the experiments section. See Appendix A.4 for showcase of the differences between weight normalization methods, implicit penalty methods and explicit penalty methods, represented by SN, LP and ALR, respectively. The key takeaways are that
penalty methods result in a softer regularization effect than SN,
ALR is preferable when the regularized network contains batch normalization (BN) layers, and
ALR gives control over the regularization effect, which also means there more hyperparameters to tune.
The performance of ALR mostly depends on the speed of the approximation rade- The current method requires 1 step of backpropagation for each power iteration step, which means that running time will be similar to that of LP and GP with l = 1. SN is much cheaper computationally than cach penalty method, although we believe ALR has the potential to become relatively cheap well by adopting new techniques for obtaining adversarial examples (Shafahi et al. 2019).
4 WGAN-ALP
We specialize the ALP formula 15)
with / being the critic, dx(x, y) = 11l- (x, v) |x- and K = 1, and apply it to the WGAN objective arrive at version with the explicit penalty, which uses adversarial perturbations as sampling strategy. It is formulated as
(f((z)
f(x radv -1) ))
2
PewPes((( - E(x,, + AB
(17)
|| |Fedvll2
where Prog is combination of the real and generated distributions (meaning that sample x can come from both), the Lagrange multiplier, and the adversarial perturbation is defined as
If(x)) f(x + r)
Fadu
arg max
||r|12
(18)
r:0<|||lla
This formulation of WGAN results in stable explicit Lipschitz penalty, overcoming the difficulties experienced when one tries to apply to random sample pairs as shown in Petzka al. (018)).
To evaluate the performance of WGAN-ALP, we trained one CIFAR-10, 10, consisting of 32 x 32 RGB images, using the residual architecture from Guirajani et al. (2077) implemented TensorFlow. Closely following Gulrajani et al. (2017). we used the Adam optimizer [Kingma and Ba 2015) with parameters 31 0, 32 = 0.9 and an imitial learning rate of x 10 decaying linearly to 0 over 100000 iterations, training the critic for 5 steps and the generator for 1 per iteration with minibatches of size 64 (doubled for the generator). We used (I7) as a loss function to optimize the critic. K = 1 was obvious choice, and we found À 100 to be optimal (the training diverged for À 0.1, and

---

Published as a conference paper at ICLR 2020
was stable but performed worse for À = 10 and 1000). The hyperparameters of the approximation of Tade were set to 10, P. being the uniform distribution over [0.1, 10], and h = power iteration. Both batches from P. and Po were used for regularization.
We used Inception Score (Salimans et al., 2016) and FID (Heusel et al. 2017 as our evaluation metrics. The former correlates well with human judgment of image quality and is the most widely used among GAN models, and the latter has been shown to capture model issues such as mode collapse, mode dropping and overfitting, while being robust and efficient metric (Xu et al. 2018). We monitored the Inception Score and FID during training using 10000 samples every 1000 iteration, and evaluated them at the end of training using 50000 samples. We ran the training setting described above 10 times with different random seeds, and calculated the mean and standard deviation of the final Inception Scores and FIDs, while also recording the maximal Inception Score observed during training. We report these values for WGAN-ALP and other relevant GANs (Gulrajani et al.
2017 Petzka et al. 2018 Zhou et al.] 2019a Wei al. 2018 Miyato et al.) 2018 Adler and Lunz 2018 Karras et al. 2018) in Table 1 We did not experiments to evaluate competing models, but included the values reported in the corresponding papers (with the exception of the FID for WGAN-GP, which was taken from Zhou et al. ). They used different methods to arrive the cited results, from which that of Adler and Lunz (2018) the one closest to ours. We show some generated samples in Figure Ta
Table Inception Scores and FIDs on CIFAR-10
Inception Score
Method
Average
Best
FID
WGAN-GP
7.86 + .07
18.86 = .13
WGAN-LP
8.02 + .07
LGAN
8.03 + .03
15.64 + .07
CT-GAN
8.12 + .12
SN-GAN
8.22 + .05
21.70 + .21
BWGAN
8.31 + .07
16.43
Progressive GAN
8.56 + .06
8.80
WGAN-ALP (ours)
8.34 .06
8.59
12.96 .35
(a) No BN in critic
(b) BN in critic
Figure 1: Generated CIFAR-10 10 samples
We also trained WGAN-LP in our implementation. During training, the best observed Inception Score and FID were 8.1 13 and 18.49, while at the end of training the best final Inception Score and
7

---

Published as a conference paper at ICLR 2020
FID were 8.01 and 15.42. To that ALR indeed restricts the Lipschitz constant of the critic, monitored the gradient norms during training, which converged to z 5 with = 100. This was also the case using LP with À 0.1, but the number of Lipschitz constraint violations found by the algorithm were much higher in this case than with ALR.
Our toy example in Appendix/A.4|showeeee showed that when the regularized network contains BN layers, ALR seems to work better than competing methods. In order to see if this still applies in more complex settings, we have trained a variant of WGAN in which the critic contains BN layers (WGAN-BN). Gulrajani et al. (017)) did not use BN in the critic as they argued that GP is not valid in that setting, and indeed when we trained WGAN-BI -BN with GP, the best Inception Score observed during training was only 6.29. When we applied ALP to WGAN-BI BN, the results were nearly on par with the original setting without BN, producing an even better maximal Inception Score 8.71. We leave the question of how BN affects Lipschitz continuity for future work. Generated samples are shown in Figure 1b
Gulrajani et al. (2017) made the distinction between one-sided and two-sided penalties, represented by (2) and 7 The latter is based on the fact that in WGAN, the optimal critic has unit gradient norm on lines connecting points from the optimal coupling π². Petzka et al. (2018) showed that since x is not known in practice, one should use the one-sided penalty, while Gemici et al. (2018) proposed a method to approximate π with an auto auto-encoding scheme. In the limit |r] - 0 the expression inside the arg max operator in (8) is equivalent to the directional derivative of f along r, and the vector Tads corresponding to the maximum value of the directional derivative at x is equivalent to (x)( Since the critic f /corresponds to the potential function in the dual formulation of the optimal transport problem, at optimality its gradient at x points towards its coupling y, where (z, y) T². From this perspective, sampling pairs (ar, x + Tado) using (18) can be seen as an approximation of the optimal coupling प². To test how reasonable this approximation is, we have trained a WGAN variant with the two-sided explicit penalty formulated as
EySPy f(g(2) EZ fx) + AE (15(0) f(x +(x++rade -1)²
(19)
||radell
which performed similarly to the one-sided case with À = 10, but was less stable for other values of X. The findings Petzka et al. (2018) were similar for the case of the implicit penalty. Improving the approximation scheme of Tode might render the formulation using the two-sided penalty (19 preferable in the future.
To show that ALR works in a high-dimensional setting as well, we trained a Progressive GAN on the CelebA-HQ dataset (Karras et al., 2018)) consisting of 1024 x 1024 RGB images. We took the official TensorFlow implementation and replaced the loss function of the critic, which originally used GP, with a version of ALP. Using (T7) as the training objective was stable until the last stage of progressive growing, but to make it work on the highest resolution, we had to replace it with
E f((()) EZxP.f(x)
( ((1re))
f(x [(x+ radu)) -
2 + (1160) )
f(x I(x rado)l ||radulla
-1) (20)
+ AEENPPP
||radullz
meaning that we used the sum of the absolute and squared values of the Lipschitz constraint violation as the penalty. The optimal hyperparameters were À = 0.1, P. being the uniform distribution over 1 step of power iteration. The best FID seen during training with the
[0.1, 100), < 10 and original GP version was 8.69, while for the modified ALP version it was 14.65. The example shows that while ALP did not beat GP in this case (possibly because the implementation was fine-tuned using GP), it does work in the high-dimensional setting as well. For samples generated by the best performing ALR and GP variants see Appendix A.5
CONCLUSIONS
Inspired by VAT, we proposed ALR and shown that it is an efficient and powerful method for learning Lipschitz constrained mappings implemented by neural networks. Resulting in competitive performance when applied to the training of WGANs, ALR a generally applicable regularization method. It draws an important parallel between Lipschitz regularization and adversarial training, which we believe can prove to be a fruitful line of future research.
8

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENTS
The author would like to thank Michael Herman from Bosch Center for Artificial Intelligence (BCAI) for the fruitful discussions, and the Advanced Engineering team Budapest, especially Géza Velkey.
REFERENCES
J. Adler and S. Lunz. Banach wasserstein GAN. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, 3-8 December 2018, Montréal, Canada, pages 6755-6764, 2018. URLhtt:: nnpps
cc/ cc/paper/De 7909- banach wasserstein. b gan
C. Anil, J. Lucas, and R. B. Grosse. Sorting out lipschitz function approximation. In Proceedings of the 36th International Conference on Machine Learming, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pages 291-301, 2019. URL http:: http::/prooceednggg/ mlr 1tt//nnnlla..tt...
/ press/
M. Arjovsky and L. Bottou. Towards principled methods for training generative adversarial networks. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24- 26, 2017, Conference Track Proceedings, 2017. URL /ttt:p://openreeveww.eeeee new.forumtf 1d=Hk4 qw5xe
M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference Machine Learming, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 223, 2017. ar ar arjovskyl wwwpsyylaa.h... 7a.html
P. I.. Bartlett.
The sample complexity of pattern classification with neural networks: The size of the weights more important than the size of the network. IEEE Trans. Information Theory, 44(2): 525-536, 1998. doi: N.1110991866610000 URL htth:tts: /tt/
661502 661502
A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL httpsss//openeeeeew.eeett. wwt.r/ummmeeooooomiccm 099Fm
M. Deza and E. Deza. Encyclopedia of Distances. Encyclopedia of Distances. Springer Berlin Heidelberg, 2009. ISBN 9783642002342.
H. Drucker and Y. LeCun. Improving generalization performance using double backpropagation. IEEE Trans. Neural Networks, 36)199999 1992. doi: 10.1 10.170922.1665000 URL https
7/doi www.rrggo.....o /0. 1109/72. 65600
Y. Dukler, W. Li, A. T. Lin, and G. Montúfar. Wasserstein of wasserstein loss for learning generative models. In Proceedings of the 36th International Conference on Machine Learning, ICMI, 2019, 9-15 June 2019, Long Beach, California, USA, pages 1716-1725, 2019. URL http! //proceedings.oo mlr www.spressse.........
/eeess//9/// /v77 /wwkkkllk..a... html
A. Galloway, A. Golubeva, T. Tanay, M. Moussa, and G. W. Taylor. Batch normalization is cause of adversarial vulnerability. CoRR, abs/1905.02161. 2019. URL/ittp::/axxxivorr///aas
1905. 02161
M. Gemici, Z. Akata, and M. Welling. Primal-dual wasserstein GAN, CoRR, abs/1805.09555, 2018. Uttt:tt:///axxi org/ labs/180... 09575
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza. B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27: Annual Conference Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2672 2680. 2014.
/wtp:/papprr..a/ nips. cc/
paper 542.r5443332 1104423--eneeeeee genereeerreeee rat ve adversa

---

Published as a conference paper at ICLR 2020
I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego. CA, USA. May 7-9, 2015, Conference Track Proceedings, 2015. bttp::////axxi
abs 141. 6572
H. Gouk, E. Frank, B. Pfahringer, and M. Cree. Regularisation of neural networks by enforcing lipschitz continuity. CoRR, abs/ 804.04368, 2018. [tRl:ttt////rx...rr///bbb/
04368
Y. Grandvalet and Y. Bengio. Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems 17 [Neural Information Pro- cessing Systems, NIPS 2004, December 13-18, 2004, Vancouver. British Columbia, Canada]. pages 529-536, 2004.
URL
wwtpsssiiiiscccppppe......e cc/paper/
2740- sem -sueersissseeee wew.wiiedddeeeeieeeeeeeeee meaninggby-eeeee www.byyetyyyyyooo.......oyy mi nimi zation
I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved train- ing of wasserstein gans. In Advances in Neural Information Processing Systems 30: An- nual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA. USA, pages 5767-5777, 2017. URL
ca/paper
7159 improved wewredddttriinnggngg of wasserstein gans
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach,
CA, USA, pages 6626-6637, 2017. URL http:/ http:://paprrssiiii
7240 trains-taineddbb-a two time scale --update rule converge- to a local -ash----ss-euulllii
J. Ho and S. Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems 29: Annual Conference Neural Information Processing Systems 2016, December 5-10. 2016, Barcelona, Spain, pages 4565 4573, 2016. URL http: /papers nips. wtp.scccppper.eooco 6391 -eweeeateieeeeeeeeeeeeeeeeeeeeee www.eevvvvveeviiaaaaaaaaaa imitttiinn wwriioonneeeara learning
A. Householder. The Theory of Matrices in Numerical Analysis. Blaisdell book pure and applied sciences: introduction to higher mathematics. Blaisdell Publishing Company, 1964.
T. Karras, T. Aila, S. Laine, and Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL https://openreviee net. forum
V. Khrulkov and I. V. Oseledets. Art of singular vectors and universal adversarial perturbations. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 8562-8570, 2018 doi: 10.1 URL http://///// http:///ppnaccess. cetenaccesss.hecv...... Art of _t__innulla CVPR 2018 paper.htmi
201 8.00993. wtt.://c//cocccctcoeoeeeo _vpr httpaateenntoprrro1111/eeeaea httppp//2/////aa////ruuuuee
thecvf.ec.... cwt www.tttccocccoottci.otc
D. P. Kingma and J. Ba. Adam: method for stochastic optimization. In 3rd International Conference Learning Representations. ICLR 2015, San Diego. CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. htthttp::/ www.v.oooooooooo abs /1412.141.... 698C
T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL https //openrevie net/ fo um 2id=B d QRgziT-
T. Miyato, S. Maeda, M. Koyama, and S. Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell, 41(8): 1999-1993, 2019. doi: 10. (099TPAM...011.2888282) BtLLhttps://o...oo TPAMI 2018 8 2858821
A. M. Oberman and J. Calder. Lipsehitz regularized deep neural networks converge and generalize. CoRR, abs/1800 hww/1808.0900 808.09540, 2018. http:/ttpe//aaxxi.rrrrabbb////// 09540
10

---

Published as a conference paper at ICLR 2020
H. Petzka, A. Fischer, and D. Lukovnikov. On the regularization of wasserstein gans. In 6th International Conference on Learning Representations. ICLR 2018, Vancouver BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL ?orum? d=B1hhRMMCC
T. Salimans, J. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2226-2234, 2016. URL httpp://ptperrsssststsse//// www.:.i..ppppppeeeee
1125-1mgroved-techhnn ques
A. Shafahi, M. Najibi, A. Ghiasi, Z. Xu, J. P. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein. Adversarial training for free! CoRR, abs//104.. 12843, 2019. URL http://axxx org/abs/1904 12843
C. Villani. Optimal Transport: Old and New. Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg, 2008. ISBN 97835407 10509.
X. Wei, B. Gong, Z. Liu, W. Lu, and L. Wang. Improving the improved training of wasserstein gans: A consistency term and dual effect. 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canuda, April 30 May 3, 2018, Conference Track Proceedings, 2018.
URL httpps://opeereeieet net / forum?
E. Wong, R. Schmidt, and J. Z. Kolter. Wasserstein adversarial examples via projected sinkhorn iterations. In Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pages 6808-6817, 2019. URL http:
/pproceeiings mlr wrwpsprssseee...e. v97 https///////wggg//////m html
Q. Xu, G. Huang, Y. Yuan, C. Guo, Y. Sun, F. Wu, and K. Q. Weinberger. An empirical study on evaluation metrics of generative adversarial networks. CoRR, abs/1806.0777 2018. URL
http://arxis> http://axxiv.oorra/b//aaaa////// hbtp://b////Ba/
07755
Z. Zhou, J. Liang, Y. Song. L. Yu, H. Wang, W. Zhang, Y. Yu, and Z. Zhang. Lipschitz generative adversarial nets. pages 7584-7593, 2019a. URL URL:ttp:://preeett /Rt/ttt://pprceednng... .mrr. /
2hou19c. html
Z. Zhou, J. Shen, Y. Song, W. Zhang, and Y. Yu. Towards efficient and unbiased implementation of lipschitz continuity in gans. CoRR, abs/1994.01 184, 2019b. URL http::/aaxiv..r5// 1904. 01184

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1
SEMI-SUPERVISED LEARNING
Since VAT is a semi-supervised learning method, important to see how ALR fares in that regime. To show this, we have replicated one of the experiments from MMyato et al. (2019). We trained the LangL architecture to classify images from CIFAR-10 with the same setting as described in Miyato al. (2019), except that we did not decay the learning rate, but kept it fixed at 3 x 10 We split the 50000 training examples into 4000 samples for the classification loss, 45000 samples for regularization and 1000 for validation, with equally distributed classes. Test performance was evaluated on the 10000 test examples. We have found that unlike in the unsupervised setting, here it important to assume (x) fixed when minimizing the regularization loss, and also to compiement the smoothing effect with entropy minimization (Grandvalet and Bengio 2004). The baseline VAT method was ALR specialized with K O, dx being the Euclidean metric, dy being the KL divergence, fixed = 8 and À 1. This setting achieved maximal validation performance of 84.2% 2% and test performance 82.46%. After some experimentation, the best performing choice was K = 0, dx being the l2 metric, dy the mean squared difference over the logit space (which parametrize the categorical output distribution over which the KL divergence is computed in the case of VAT), P. being the uniform distribution over [1, 10] and À 1. This way the maximal validation performance was 85.3% and test performance 83. 83.54%. Although this R 1% is improvement is not very significant, it shows that ALR can be competitive choice as semi-supervised learning method as well.
A.2 VIRTUAL ADVERSARIAL TRAINING AS LIPSCHITZ REGULARIZATION
VAT was defined by considering neural networks implementing conditional distributions x)) where the distribution over discrete labels y was conditioned the input image x Miyato al. (2019). To see why LDS (10), the regularization term of VAT, can be seen as special kind of Lipschitz continuity, will and codomain Y, where X is the space of images and Y is the probability simplex (the space of distributions over the finite set of labels).
different perspective. Consider mapping f X + Y with domain X
Since a divergence is in general a premetric (prametric, quasi-distance) on the space of probability measures (Deza and Dezal 2009). and Lipschitz continuity is defined for mappings between metric spaces, let us restrict the divergence D) from the VAT formulation to be metric dy. Miyato et al. (2019) used KLD in their experiments, which is not a metric, but one can use e.g. the square root of JSD or the Hellinger distance, which are metrics. Let us metrize the space of images X with dx being the Euclidean metric. From this perspective, the network f is mapping from the metric space (X, dx) to the metric space (Y, dy.. Let us also assume that we aim to learn mapping ʃ with the smallest possible Il/llzz by setting K O.
To enforce the condition x + E X in (14), we bound the Euclidean norm of r from above by some predefined > 0. If we make the additional assumption that the supremum is always achieved with an of maximal norm the denominator in (14) will be constant, hence the formulas with and
without it will be equivalent up to a scaling factor. With these simplifications, [4] and (1) reduce to
VAT r adv
arg max dy (f()), 0S/1r|12Se
(21)
and
LX = dy (f(z). rade)),
(22)
which equivalent to and 110 respectively. Let consider the question of keeping f(x) fixed when minimizing (22) an implementation detail. With this discrepancy aside, we have recovered VAT special case of Lipschitz regularization.
A.3 DERIVATION OF THE APPROXIMATION OF Tado
We assume that f and dy are both twice differentiable with respect to their arguments almost everywhere, the latter specifically at x = y. Note that one can easily find a dy for which the last assumption does not hold, for example the /1 distance. If dy is translation invariant, meaning that
12

---

Published as a conference paper at ICLR 2020
dy (x. v) dx u. 1..y+ u) for each E Y, then its subderivatives at = y will be independent of x, hence the method described below will still work. Otherwise, one can resort to using a proxy metric in place of dy for the approximation, for example the 12 distance.
We denote dy (f(x), f(æ + r)) by d(r, x) for simplicity. Because d(r,x) 2 0 and d(0, x) = it is easy to that
v,d(r, = 0,
(23)
so that the second-order Taylor approximation of d(r, is d(r;,) 2 H()r, where H(x)
VV,d(r, -)), is the Hessian matrix. The eigenvector with the greatest absolute value is the direction of greatest curvature, which is approximately the
(x)) corresponding its eigenvalue
adversarial direction that we are looking for. The power iteration (Householder 1964) defined by
H(r)r
T++1
|1(()):1l2*
(24)
where To is a randomly sampled unit vector, converges to if and ro are not perpendicular. Calculating (xx) computationally heavy, which why H (x)rr approximated using the finite
differences method as
H V v,d(r,))),
v,d(r,
(25)
where the equality follows from 23
The hyperparameter & + 0 is introduced here. In summary, the adversarial direction approximated by the iterative scheme
v,d(r, 1r=en
Ti+1
(26)
v,d(r, x -=fri
of which one iteration found to be sufficient and necessary in practice.
A.4 TOY EXAMPLE
To showcase the differences between weight normalization methods, implicit penalty methods and explicit penalty methods, represented by SN, LP and ALR, respectively, we devised the following toy example. Suppose that we want to approximate the following real-valued mapping the 2 2-dimensional interval 4|3
f(x, v) = Lo o v o 2
(27)
otherwise
for -45 S 4. addition, we want the approximation to be 1 1-Lipschitz. It easy to see that the optimal approximation with respect to the mean squared error is
if V x0 + 0.5, if < +22 1,5,
1.5 22
fopt (2, y) =
(28)
v 12 3 115
if < v + 2.5.
otherwise.
This example has connections to WGAN, as the optimal critic is I-Lipschitz, and its approximation will provide the learning signal to the generator in the form of gradients. Therefore. it is important to closely approximate the gradient of the optimal critic, which is achieved indirectly by Lipschitz regularization. In this example, we will see how closely the different Lipschitz regularization methods can match the gradient of the optimal approximation fopt-
We implemented the example in PoTcrc.. For the approximation f. we use MLP with 3 hidden layers containing 20, 40 and 20 neurons, respectively, with ReLU activations after the hidden layers, and a variant which also has batch normalization (BN) before the activations. since it has been found that BN hurts adversarial robustness (Galloway et al. 2219),, and hence should also hurt Lipschitz continuity. We trained the networks for 2 iterations, with batches consisting of input,
13

---

Published as a conference paper at ICLR 2020
corresponding output, and additional imput for regularization. The inputs drawn uniformly random from |-4, 4]² and the output defined by (27). The minibatch size was 64 for input-output pairs, and 1024 for regularization inputs. We used heatmaps to visualize the gradient norm surfaces of the optimal and learned mappings, with the color gradient going from black at O to white at 1, see Figure This example is not intended to rank the competing Lipschitz regularization methods, it always depends on the particular application which one is the best suited, but to show that they are fundamentally different and competent their own way.
0.
06
10
(a) colormap
(b) VV fopellz
(c) IV l2 (no regularization)
(d) |V
(e) V=fLA,o=
(f) [V*fLrA=all
(g) Vi
(b) o
/
(i) [=
(k) /V
(m) [
(n) I
(o) |V 2|12
(p) IAAaLaAAAnua
Figure 2: Gradient norm surfaces of optimal and learned approximations of
Without any kind of regularization, the network learned approximate the target function very well, its gradients look nothing like that of fopt: although somehow a better match with BN.
When we apply SN to the MLP layers, the result without BN will be very smooth mapping with maximum gradient norm far below 1. SN is not compatible with BN, the result being only slightly better than the unregularized case. A detail not visible here is that because SN considers weight matrices as linear maps from R" to Rm and normalizes them layer- wise, regularizes globally instead of around actual data samples. In this case, the whole of R2 instead of just [|4,4]2 For WGANs trained on CIFAR-10, 10, the input space consists of 32 x 32 RGB images with pixel values in [-1, 1]. but the trained mapping is regularized 322 x2x3 instead of just [-1, 32xx22x3 (which
14

---

Published as a conference paper at ICLR 2020
contains the supports of the real and fake distributions). This can hurt performance if the optimal mapping implemented by a particular network architecture is K-Lipschitz inside these supports, but not in some other parts of IR 32x332x3
When the network regularized using LP 2 the regularization strength can be controlled by tuning the value of X. We trained with À = 0.1, and 10. Without BN, the highest of these values seems to work the best. With BN, the resulting mapping visibly highly irregular.
With ALR, in addition X, have additional control over the regularization by the hyperparameters of the approximation scheme of Tade- After some experimentation, we have found the best P. for this case was the uniform distribution over [10-6 10 We trained with À 0.1, 1 and 10, and k 0, and power iterations. Arguably, both with and without BN the X = 1 and A: = 5 case seems like the best choice. Without BN, the results quite similar the LP case, but when BN is introduced, the resulting mappings are much smoother than the ones obtained with LP.
15

---

Published as a conference paper at ICLR 2020
A.5 IMAGES GENERATED BY PROGRESSIVE GAN TRAINED ON CELEBA-HO HQ
Figure 3: Images generated using Progressive GAN trained with ALR
16

---

Published as a conference paper at ICLR 2020
Figure 4: Images generated using Progressive GAN trained with GP
17


Reviews:
[{"review_id": "SJlkWrqe5S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "It is an interesting idea about how to enforce the Lipsthitz constrain in WGAN by using virtual adversarial training. The connection between virtual adversarial and this paper method - ALR is quite simple and clear. In the experiments, the FID score in the table is not complete which can not clearly compare the ability of the Lipschitz regularization to other regularization methods. The paper addresses that the approximation of r_{adv} will affect the performance of ALR. How to balance the quality and computation complexity is quite important. This paper did not provide the reason about why this method can not work better than GP method in high-dimensional setting.\nIn general, this paper provides an interesting direction for regularization.\n\nPros:\n1. This paper derived as a generalization of VAT (Virtual Adversarial Training) which provided the new way to think of the regularization.\n2. ALR (Adversarial Lipchitz Regularization) is an new method for learning Lipschitz constrained rather than weight clipping or gradient penalty.\n3. This method provides the connection between Lipschitz regularization and adversarial training.\n\nCons:\n1. The comparison of the experiments was not complete. Some of the Inception Scores and FID were blank in the table.\n2. The results of adding BN were not clearly explained. These included LP and ALR method. Might have some inference about the effect of BN in regularization term.\n3. In high-dimensional setting, the authors did not clearly describe the weakness of ALR method.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkl6hcj55H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: Virtual Adversarial Training (Miyato et al., 2017) can be viewed as a form of Lipschitz regularization. Inspired by this, the paper proposes a Lipschitz regularization technique that tries to ensure that the function being regularized doesn’t change a lot in virtual adversarial directions. This method is shown to be effective in training Wasserstein GANs. \n\nMotivation and placement in literature: Being able to effectively enforce the Lipschitz constraint on neural networks has wide ranging applications. Even though the paper predominantly considers the WGAN setting, the topic at hand is within the scope of NeurIPS and will of interest to the machine learning community at large. \n\nClaimed Contributions and their significance: \n1. Practical method with good performance: The proposed method can be used to train WGANs with high (subjective) sample quality. Although better, quantitative evaluation methods are needed to make stronger claims about the efficacy of this approach for GAN training in general (see below), the method described here will likely be useful for practitioners and GAN community. I’m also convinced that this method has the potential to work for higher dimensions. \n2. VAT as Lipschitz regularization: There is a relatively straightforward connection between the Lipschitz constraint and adversarial robustness - both imply that small changes in the inputs should lead to small changes in the outputs, in their respective space. There are also a number of papers that make strong connections between adversarial training and Lipschitz regularization (Parseval Networks (Cisse et. al, 2017) for example). Therefore, it is perhaps not too surprising that the LDS term from Miyato et. al. can be rephrased as a Lipschitz regularization term by picking suitable input-output (pre-)metrics (in Section 3). I currently don’t see this as a major contribution of this paper, although I’m open to changing my mind if this involves a subtlety that I’m missing. \n\nRelated Work:\nKhrulkov et al (2017) looks like a related work - especially related to how the way the adversarial perturbation is computed and backpropagation is performed. Also Gemici et. al. also discuss the limitations of the original gradient penalty paper (for Section 2.2)\n\nQuestions and Points of Improvement\n1. Better evaluation of GANs: Could you further convince us that this method alleviates common pitfalls of GAN training, such as mode collapse? There are a number of papers that give quantitative metrics for this purpose (such as Xu et. al. 2018). Since the quality of the WGANs presented is one of the biggest strengths of this paper, further evidence in this direction will make the paper stronger. \n\n2. Different tasks: \nThe method described looks flexible enough to be applied on domains other than Wasserstein distance estimation. Did you try other tasks where a Lipschitz penalty might help, such as adversarial robustness? The semi-supervised setting mentioned in the appendix look promising yet perhaps under-explored. \n\n3. Resultant Lipschitz constant: \nSince this paper is about enforcing the Lipschitz constraint through regularization, more experiments on how well the Lipschitz constraint is enforced in practise would be helpful. For example, how much do your WGAN critics violate the 1-Lipschitz constraint? Once this is quantified, how does ALR compare to other Lipschitz regularization techniques? The function approximation task in Section 4.2 seems simple enough that you can probably compute gradient norms on a 2D grid and draw a histogram. How would the histograms look if you did this, for different methods?\n\n4. Sample efficiency: \nSection 4.2 claims that using the explicit Lipschitz penalty is inefficient because violations of the Lipschitz constraint on samples from P_real, P_generated or P_interpolated likely be non-maximal. Could you make a theoretical or empirical case that the additional time spent for finding adversarial directions is actually worth it? If you have a way of quantifying how well the Lipschitz constraint is satisfied (as described above), then doing this empirically should be possible. \n\n5. Problematic baseline for spectral normalization: \nThe way spectral normalization (SN) was used/described in Section 4.1 seem to have some issues. First of all, batch normalization is incompatible with methods that achieve Lipschitz constraint via. architectural constraints, such as spectral normalization. Also, this statement looks problematic: “It can be seen that SN has a very strong regularization effect, which is because SN works by approximating the spectral norm of each layer and then normalizing the layers by dividing their weight matrices by the corresponding spectral norms, thereby resulting in overregularization if the approximation is greater than the actual spectral norm.“ In most practical cases, power iteration used in spectral normalization can get a very close approximation of the spectral norm of the weight matrices with a reasonable number (<20 is a conservative guess) of iterations. The over-regularization effect, however, does exist and is more connected to the loss of gradient norm as described in Anil et. al. than bad approximations to the spectral norm of weight matrices. \n\nWriting: The paper is well-written and easy to understand. \n\nDecision: Weak Accept. \n\nOther, lesser important points of improvement:\n1. The argmax expression in (18) looks problematic - r doesn’t seem bounded, hence can be chosen arbitrarily large. \n2. Equation (25) describes the optimal approximation. According to which metric is this optimal? \n3. Use \\leq for “less than or equal to” in 25. \n4. Consider adding a colormap to Figure 1. \n\n________\nPost-rebuttal edit: The revisions made to the paper address some of the points of improvement listed above. I maintain my initial assessment of weak accept (leaning more towards accept), as I believe the methods discussed in this paper will be of interest to the research community. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xTs6z6qH", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Final Edit:\n\nI have reviewer the final version of the paper and have decided to increase my score to a weak accept. I maintain some concerns around the empirical evaluation in the paper (collating results from multiple sources with different experimental procedures). But my major concerns have been addressed by the authors in their response and changes to the paper.\n\n----------------------------------\nPost rebuttal edit:\n\nFollowing updates to the paper manuscript, which address my concerns around the correctness of the empirical results, I have updated my review score from 1 to 3.\n----------------------------------\n\nSummary:\n\nThis paper draws on connections between virtual adversarial training (VAT) and Lipschitz regularization to utilize VAT techniques in the training of WGAN architectures. While this work may be touching on something quite interesting, I felt that the theoretical exposition of the ideas was lacking. The empirical evidence seemed promising in some direction though lacking in others.\n\nI was requested as an emergency reviewer for this paper.\n\n\nOverview:\n\nThis paper is 9 pages length in total. Unfortunately, I felt that the use of an additional page was unwarranted and that the paper contained unnecessary content.\n\nDue to a concern over correctness of some empirical results and issues with the presented derivations I have opted to reject this paper. I hope that these issues can be addressed by the authors in which case I will reassess my score.\n\n1) Under equation 5, \"with substantially more stable training behaviour and improved sample quality\". A citation should be included for this claim. In fact, recent advances in GAN methods have not required the Wasserstein distance objective [1].\n\n2) I found some issues with Section 2.2.\n\na) First a comment on related work. There is older work studying the generalization properties of Lipschitz neural networks which is not mentioned in this section. For example, [2]. You also write that learning under Lipschitz constraints became prevalent with the introduction of WGAN. While this is probably true, I think it is fair to point out that many older papers also utilized similar bounds in the vein of improving generalization. For example [3], which also aimed to limit the gradient norm of deep neural networks.\n\nb) I felt that this subsection was a little bloated and the content did not fit entirely under the heading. A large chunk of this section is dedicated to discussing potential issues arising with the gradient penalty formulation of WGAN and alternative approaches such as the Banach WGAN. While these are useful additions they did not feel critical to this work and in my opinion did not deserve an extension beyond the 8 page recommendation.\n\n3) I found the discussion in Section~3 a little difficult to follow. I will summarize my key concerns below.\n\na) The authors assume that generalizing Lipschitz continuity to a premetric space is trivial. While the more important results seems believable I am not convinced by the presentation of these results and would prefer to have seen this given more careful treatment. For example, premetrics need not obey symmetry or the triangle inequality (assuming this is the definition used by the authors --- including this would be valuable). It is written (paraphrasing) that a mapping $f$ is $K$-Lipschitz iff for any $x$ the supremum over $r$ is bounded by $K$. However, $r$ only appears on the right-hand side of a potentially asymmetric distance function. Moreover, many properties of Lipschitz continuous functions depend on the triangle inequality holding in the metric space and these would fail to hold here.\n\nb) When connecting ALP to VAT some of the differences in the formulations are hand-waved away in unconvincing ways. Under the trivial metric, the Lipschitz constant is given by the maximum distance in the output space. With this observation, it seems trivial that the VAT formulation will perform a form of Lipschitz regularization. However, the Lipschitz constant does not take into account distance in the input space in a meaningful way and so I am unsure to what extend the connection is really meaningful. Further, the $r < \\epsilon$ constraint in the VAT model is treated as an inconvenient implementation detail but I am not convinced this is sufficient. Indeed, this $\\epsilon$ could be used to bound the input deviations and thus could be seen as affecting the Lipschitz constant under a more reasonable metric.\n\n4) I felt that Section~4.1 presented an important discussion coupled with an interesting toy problem to highlight benefits and shortcomings of the proposed method. However, this section was moderately long and contributed only a little towards understanding the practical settings users of ALR would care about in practice. I did not gain much intuition into how ALR might generalize to high dimensional settings and was concerned by the fact that the distribution $P_\\epsilon$ used was heavily hand-engineered and did not match up with the ones used in later experiments.\n\na) I did not understand the comments that constraining the Lipschitz constant globally may be undesirable in WGANs. The dual optimization problem requires a Lipschitz constraint be enforced over the support of the distributions and we should not care outside of this region in any case (except in cases where the generator might move the support to a currently under-regularized region of the critic domain in which case a global constraint may be advantageous).\n\n5) I am not particularly up to date with evaluation of GAN models but to me the presented results in the main paper looked mostly reasonable. Some major concerns did remain to me which I would appreciate being addressed by the authors.\n\na) I have one question on the reported \"Best\" inception score for the WGAN-ALP and Progressive-GAN models. You stated that each model was trained 5 times and reported the mean, standard deviation and best results. However, the difference between the best and average scores alone would constitute a higher standard deviation: $\\sqrt{(8.80 - 8.56)^2 / 4} = 0.12$. Please can you clarify how exactly each of these values was computed?\n\nb) In the main paper the authors write that ALP is able to work in high dimensional settings (though is not competitive with state of the art). In the appendix however the authors point out that they must make significant modifications to the training objective by including a squared Lipschitz constraint violation term (violating further the comparison to VAT). I do not consider this a huge issue, but it should be discussed in the main paper.\n\nc) Finally, the authors employed a range of different hyperparameter settings through their experiments but gave little guidance on how to choose these settings in practice or how sensitive their proposed method is to changes in these hyperparameters. I believe that this would be a highly-valuable addition to the paper and would help distinguish this method from other training stabilization proposals.\n\n\nMinor comments:\n\n- In paragraph 1, you write that WGAN requires critic to consist of only 1-Lipschitz functions. This is true of the Wasserstein distance estimation problem but the WGAN only requires the correct gradient direction (scaling of the critic is fine).\n- In summary points in intro, \"ALR\" is used before acronym is defined.\n- Equation (12) and (13) are twice normalized (||r_k||^2=1 by definition). Similar issue in (22) and (23).\n- First para of Section 3, you write \"on the space of labels\". Do you mean on the probability simplex?\n\n\nReferences:\n\n[1] \"Large scale GAN training for high fidelity natural image synthesis\", Brock, Donahue, and Simonyan\n[2] \"The sample complexity of pattern classification with neural networks\", Bartlett\n[3] \"Double backpropagation increasing generalization performance\", Drucker and LeCun\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an adversarial approach to enforcing a Lipschitz constraint on neural networks. The idea is intuitively appealing, and the paper is clear and well written. It's not clear from the experiments if this method outperforms competing approaches, but it is at least comparable, which means this is at the very least another useful tool in the toolbox. There was a lot of back-and-forth with the reviewers, mostly over the experiments and some other minor points. The reviewers feel like their concerns have all been addressed, and now agree on acceptance.

Decision: Accept (Poster)