Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Final Edit:

I have reviewer the final version of the paper and have decided to increase my score to a weak accept. I maintain some concerns around the empirical evaluation in the paper (collating results from multiple sources with different experimental procedures). But my major concerns have been addressed by the authors in their response and changes to the paper.

----------------------------------
Post rebuttal edit:

Following updates to the paper manuscript, which address my concerns around the correctness of the empirical results, I have updated my review score from 1 to 3.
----------------------------------

Summary:

This paper draws on connections between virtual adversarial training (VAT) and Lipschitz regularization to utilize VAT techniques in the training of WGAN architectures. While this work may be touching on something quite interesting, I felt that the theoretical exposition of the ideas was lacking. The empirical evidence seemed promising in some direction though lacking in others.

I was requested as an emergency reviewer for this paper.


Overview:

This paper is 9 pages length in total. Unfortunately, I felt that the use of an additional page was unwarranted and that the paper contained unnecessary content.

Due to a concern over correctness of some empirical results and issues with the presented derivations I have opted to reject this paper. I hope that these issues can be addressed by the authors in which case I will reassess my score.

1) Under equation 5, "with substantially more stable training behaviour and improved sample quality". A citation should be included for this claim. In fact, recent advances in GAN methods have not required the Wasserstein distance objective [1].

2) I found some issues with Section 2.2.

a) First a comment on related work. There is older work studying the generalization properties of Lipschitz neural networks which is not mentioned in this section. For example, [2]. You also write that learning under Lipschitz constraints became prevalent with the introduction of WGAN. While this is probably true, I think it is fair to point out that many older papers also utilized similar bounds in the vein of improving generalization. For example [3], which also aimed to limit the gradient norm of deep neural networks.

b) I felt that this subsection was a little bloated and the content did not fit entirely under the heading. A large chunk of this section is dedicated to discussing potential issues arising with the gradient penalty formulation of WGAN and alternative approaches such as the Banach WGAN. While these are useful additions they did not feel critical to this work and in my opinion did not deserve an extension beyond the 8 page recommendation.

3) I found the discussion in Section~3 a little difficult to follow. I will summarize my key concerns below.

a) The authors assume that generalizing Lipschitz continuity to a premetric space is trivial. While the more important results seems believable I am not convinced by the presentation of these results and would prefer to have seen this given more careful treatment. For example, premetrics need not obey symmetry or the triangle inequality (assuming this is the definition used by the authors --- including this would be valuable). It is written (paraphrasing) that a mapping $f$ is $K$-Lipschitz iff for any $x$ the supremum over $r$ is bounded by $K$. However, $r$ only appears on the right-hand side of a potentially asymmetric distance function. Moreover, many properties of Lipschitz continuous functions depend on the triangle inequality holding in the metric space and these would fail to hold here.

b) When connecting ALP to VAT some of the differences in the formulations are hand-waved away in unconvincing ways. Under the trivial metric, the Lipschitz constant is given by the maximum distance in the output space. With this observation, it seems trivial that the VAT formulation will perform a form of Lipschitz regularization. However, the Lipschitz constant does not take into account distance in the input space in a meaningful way and so I am unsure to what extend the connection is really meaningful. Further, the $r < \epsilon$ constraint in the VAT model is treated as an inconvenient implementation detail but I am not convinced this is sufficient. Indeed, this $\epsilon$ could be used to bound the input deviations and thus could be seen as affecting the Lipschitz constant under a more reasonable metric.

4) I felt that Section~4.1 presented an important discussion coupled with an interesting toy problem to highlight benefits and shortcomings of the proposed method. However, this section was moderately long and contributed only a little towards understanding the practical settings users of ALR would care about in practice. I did not gain much intuition into how ALR might generalize to high dimensional settings and was concerned by the fact that the distribution $P_\epsilon$ used was heavily hand-engineered and did not match up with the ones used in later experiments.

a) I did not understand the comments that constraining the Lipschitz constant globally may be undesirable in WGANs. The dual optimization problem requires a Lipschitz constraint be enforced over the support of the distributions and we should not care outside of this region in any case (except in cases where the generator might move the support to a currently under-regularized region of the critic domain in which case a global constraint may be advantageous).

5) I am not particularly up to date with evaluation of GAN models but to me the presented results in the main paper looked mostly reasonable. Some major concerns did remain to me which I would appreciate being addressed by the authors.

a) I have one question on the reported "Best" inception score for the WGAN-ALP and Progressive-GAN models. You stated that each model was trained 5 times and reported the mean, standard deviation and best results. However, the difference between the best and average scores alone would constitute a higher standard deviation: $\sqrt{(8.80 - 8.56)^2 / 4} = 0.12$. Please can you clarify how exactly each of these values was computed?

b) In the main paper the authors write that ALP is able to work in high dimensional settings (though is not competitive with state of the art). In the appendix however the authors point out that they must make significant modifications to the training objective by including a squared Lipschitz constraint violation term (violating further the comparison to VAT). I do not consider this a huge issue, but it should be discussed in the main paper.

c) Finally, the authors employed a range of different hyperparameter settings through their experiments but gave little guidance on how to choose these settings in practice or how sensitive their proposed method is to changes in these hyperparameters. I believe that this would be a highly-valuable addition to the paper and would help distinguish this method from other training stabilization proposals.


Minor comments:

- In paragraph 1, you write that WGAN requires critic to consist of only 1-Lipschitz functions. This is true of the Wasserstein distance estimation problem but the WGAN only requires the correct gradient direction (scaling of the critic is fine).
- In summary points in intro, "ALR" is used before acronym is defined.
- Equation (12) and (13) are twice normalized (||r_k||^2=1 by definition). Similar issue in (22) and (23).
- First para of Section 3, you write "on the space of labels". Do you mean on the probability simplex?


References:

[1] "Large scale GAN training for high fidelity natural image synthesis", Brock, Donahue, and Simonyan
[2] "The sample complexity of pattern classification with neural networks", Bartlett
[3] "Double backpropagation increasing generalization performance", Drucker and LeCun

review_title: Official Blind Review #5

final_score: rating: 6: Weak Accept
review_content: Summary: Virtual Adversarial Training (Miyato et al., 2017) can be viewed as a form of Lipschitz regularization. Inspired by this, the paper proposes a Lipschitz regularization technique that tries to ensure that the function being regularized doesn’t change a lot in virtual adversarial directions. This method is shown to be effective in training Wasserstein GANs. 

Motivation and placement in literature: Being able to effectively enforce the Lipschitz constraint on neural networks has wide ranging applications. Even though the paper predominantly considers the WGAN setting, the topic at hand is within the scope of NeurIPS and will of interest to the machine learning community at large. 

Claimed Contributions and their significance: 
1. Practical method with good performance: The proposed method can be used to train WGANs with high (subjective) sample quality. Although better, quantitative evaluation methods are needed to make stronger claims about the efficacy of this approach for GAN training in general (see below), the method described here will likely be useful for practitioners and GAN community. I’m also convinced that this method has the potential to work for higher dimensions. 
2. VAT as Lipschitz regularization: There is a relatively straightforward connection between the Lipschitz constraint and adversarial robustness - both imply that small changes in the inputs should lead to small changes in the outputs, in their respective space. There are also a number of papers that make strong connections between adversarial training and Lipschitz regularization (Parseval Networks (Cisse et. al, 2017) for example). Therefore, it is perhaps not too surprising that the LDS term from Miyato et. al. can be rephrased as a Lipschitz regularization term by picking suitable input-output (pre-)metrics (in Section 3). I currently don’t see this as a major contribution of this paper, although I’m open to changing my mind if this involves a subtlety that I’m missing. 

Related Work:
Khrulkov et al (2017) looks like a related work - especially related to how the way the adversarial perturbation is computed and backpropagation is performed. Also Gemici et. al. also discuss the limitations of the original gradient penalty paper (for Section 2.2)

Questions and Points of Improvement
1. Better evaluation of GANs: Could you further convince us that this method alleviates common pitfalls of GAN training, such as mode collapse? There are a number of papers that give quantitative metrics for this purpose (such as Xu et. al. 2018). Since the quality of the WGANs presented is one of the biggest strengths of this paper, further evidence in this direction will make the paper stronger. 

2. Different tasks: 
The method described looks flexible enough to be applied on domains other than Wasserstein distance estimation. Did you try other tasks where a Lipschitz penalty might help, such as adversarial robustness? The semi-supervised setting mentioned in the appendix look promising yet perhaps under-explored. 

3. Resultant Lipschitz constant: 
Since this paper is about enforcing the Lipschitz constraint through regularization, more experiments on how well the Lipschitz constraint is enforced in practise would be helpful. For example, how much do your WGAN critics violate the 1-Lipschitz constraint? Once this is quantified, how does ALR compare to other Lipschitz regularization techniques? The function approximation task in Section 4.2 seems simple enough that you can probably compute gradient norms on a 2D grid and draw a histogram. How would the histograms look if you did this, for different methods?

4. Sample efficiency: 
Section 4.2 claims that using the explicit Lipschitz penalty is inefficient because violations of the Lipschitz constraint on samples from P_real, P_generated or P_interpolated likely be non-maximal. Could you make a theoretical or empirical case that the additional time spent for finding adversarial directions is actually worth it? If you have a way of quantifying how well the Lipschitz constraint is satisfied (as described above), then doing this empirically should be possible. 

5. Problematic baseline for spectral normalization: 
The way spectral normalization (SN) was used/described in Section 4.1 seem to have some issues. First of all, batch normalization is incompatible with methods that achieve Lipschitz constraint via. architectural constraints, such as spectral normalization. Also, this statement looks problematic: “It can be seen that SN has a very strong regularization effect, which is because SN works by approximating the spectral norm of each layer and then normalizing the layers by dividing their weight matrices by the corresponding spectral norms, thereby resulting in overregularization if the approximation is greater than the actual spectral norm.“ In most practical cases, power iteration used in spectral normalization can get a very close approximation of the spectral norm of the weight matrices with a reasonable number (<20 is a conservative guess) of iterations. The over-regularization effect, however, does exist and is more connected to the loss of gradient norm as described in Anil et. al. than bad approximations to the spectral norm of weight matrices. 

Writing: The paper is well-written and easy to understand. 

Decision: Weak Accept. 

Other, lesser important points of improvement:
1. The argmax expression in (18) looks problematic - r doesn’t seem bounded, hence can be chosen arbitrarily large. 
2. Equation (25) describes the optimal approximation. According to which metric is this optimal? 
3. Use \leq for “less than or equal to” in 25. 
4. Consider adding a colormap to Figure 1. 

________
Post-rebuttal edit: The revisions made to the paper address some of the points of improvement listed above. I maintain my initial assessment of weak accept (leaning more towards accept), as I believe the methods discussed in this paper will be of interest to the research community. 

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: It is an interesting idea about how to enforce the Lipsthitz constrain in WGAN by using virtual adversarial training. The connection between virtual adversarial and this paper method - ALR is quite simple and clear. In the experiments, the FID score in the table is not complete which can not clearly compare the ability of the Lipschitz regularization to other regularization methods. The paper addresses that the approximation of r_{adv} will affect the performance of ALR. How to balance the quality and computation complexity is quite important. This paper did not provide the reason about why this method can not work better than GP method in high-dimensional setting.
In general, this paper provides an interesting direction for regularization.

Pros:
1. This paper derived as a generalization of VAT (Virtual Adversarial Training) which provided the new way to think of the regularization.
2. ALR (Adversarial Lipchitz Regularization) is an new method for learning Lipschitz constrained rather than weight clipping or gradient penalty.
3. This method provides the connection between Lipschitz regularization and adversarial training.

Cons:
1. The comparison of the experiments was not complete. Some of the Inception Scores and FID were blank in the table.
2. The results of adding BN were not clearly explained. These included LP and ALR method. Might have some inference about the effect of BN in regularization term.
3. In high-dimensional setting, the authors did not clearly describe the weakness of ALR method.
review_title: Official Blind Review #1

Meta-review: This paper introduces an adversarial approach to enforcing a Lipschitz constraint on neural networks. The idea is intuitively appealing, and the paper is clear and well written. It's not clear from the experiments if this method outperforms competing approaches, but it is at least comparable, which means this is at the very least another useful tool in the toolbox. There was a lot of back-and-forth with the reviewers, mostly over the experiments and some other minor points. The reviewers feel like their concerns have all been addressed, and now agree on acceptance.

Decision: Accept (Poster)