LOGAN:  Latent Optimisation for Generative Adversarial Networks

Abstract: Training generative adversarial networks requires balancing of delicate adversarial dynamics. Even with careful tuning, training may diverge or end up in a bad equilibrium with dropped modes. In this work, we introduce a new form of latent optimisation inspired by the CS-GAN and show that it improves adversarial dynamics by enhancing interactions between the discriminator and the generator. We develop supporting theoretical analysis from the perspectives of differentiable games and stochastic approximation. Our experiments demonstrate that latent optimisation can significantly improve GAN training, obtaining state-of-the-art performance for the ImageNet (128 x 128) dataset. Our model achieves an Inception Score (IS) of 148 and an Frechet Inception Distance (FID) of 3.4, an improvement of 17% and 32% in IS and FID respectively, compared with the baseline BigGAN-deep model with the same architecture and number of parameters.

Reviews and author responses:

Thank you for your comments. 

We will open source the CIFAR10 model and training code for the camera-ready version. In the meantime, we would be glad to answer your questions about the implementation, as well as to add any modelling details that might be missing. We appreciate your effort in attempting to replicate the results!

Thank you for your comments.

We have included your suggested changes in the current draft. Here we directly answer some main concerns:

1. Symplectic Gradient Adjustment (SGA)

We have rewritten the section on how LOGAN relates to SGA to make it more clear. The main idea is that the latent step in LOGAN introduces a 3rd player into the game who is on the same side as the generator. 

Note that the “Hessian” of a game differs significantly from the Hessian of a function: it involves second-order derivatives of multiple functions, corresponding to the losses of the players. The game Hessian thus decomposes into blocks labeled by the “type” of interaction: G+G, G+D, G+Z, and so on. SGA makes use of the anti-symmetric component A of the game Hessian, where there are three kinds of blocks corresponding to second-order interactions between G+D, G+z and D+z respectively.

Second-order interactions between G+D are extremely expensive to compute. Thus, LOGAN only adjusts the gradients based on interactions between G+z and D+z. The partial approximation to SGA is well-founded game-theoretically in that it adjusts the dynamics between 2 of 3 possible pairings of the players — which suffice for a substantial performance gain. 

It is currently unclear how approximations like the Gauss-Newton generalize to game Hessians. 


2. Natural Gradients (NG) and SGA: 

Our main motivation is indeed approximating SGA, and NG is the best way we found to implement the gradient updates in SGA. In other words, SGA does not specify the size of gradient steps, and NG provides an efficient way to adapt the step size under our approximation. We realised that the structure of our submission obscured this storyline, and re-organised it in our current draft.

3. About the likelihood interpretation of the hinge loss (Appendix C): This section is relatively independent from the rest of the paper. NG has long been used as an approximate optimisation method without considering the probabilistic meaning of the loss. Similarly, SGA does not require the loss to be a likelihood function. We nevertheless provided this section for readers interested in the probabilistic view and acknowledged the limit of this interpretation in the text.

4. Your comment about the typo is correct, thanks for pointing it out. It should read: “an ideal generator can perfectly fool the discriminator”. This assumption is actually unnecessary, and we removed it. Yes, the likelihood function p(t|z) has the same form as the p in eq. 37. We rewrote this part to make the correspondence more clear.

5. Thank you for pointing out this typo. The clipping should be after updating z. We have not experimented with many other algorithms for optimising z, and would like to investigate other methods including mirror descent in the future. We will also experiment with ELU activation.

The paper proposes a novel training scheme for GANs, which leads to improved scores w.r.t. state-of-the-art. The idea is to update the sampled latent code in a direction improving the inner maximization in the min-max problem. The work considers both, the gradient direction and a direction motivated by the natural gradient which is shown to yield excellent performance. The overall scheme is motivated as an approximation to the (prohibitively expensive) symplectic gradient adjustment method. 

While the work contains some typos, it was easy to read and overall very well written. The experimental results are impressive and clearly validate the usefulness of the approach. Therefore, I recommend acceptance of the paper.  

On the theoretical side, I feel some parts can be improved. I'm willing to further increase my rating if some of the following points are addressed:

1. The connection to SGA is a bit hand-wavy, as terms are dropped or approximated with the only reasoning that they are difficult to compute. In some approximations (e.g. Gauss-Newton approximation of the Hessian) one can argue quite well that certain second-order terms can be dropped under some assumptions (e.g. mild nonlinearity, or vanishing near the optimum).

2. I had some troubles Sec 4 related to the natural gradient.
(a) What is p(t | z)? Is it the same p as in (37), Appendix C? I find the argument that the hinge loss pushing D(G(z)) < 0 during training hand-wavy, as for natural gradient p(t | z) should always be a valid distribution. There are also some typos which made it hard to follow the arguments there. It probably should read "an ideal generator can perfectly fool the discriminator" (not perfectly fool the generator). 

(b) Maybe it would be easier to directly argue that one wishes to approximate SGA as well as possible, rather than taking a detour through the natural gradient.

3. Why is the increment \Delta z clipped in Algorithm 1? Is there a theoretical justification? If the goal of the clipping is to stay inside the support of the uniform distribution, shouldn't it rather be z' = [z + \Delta z]? A soft clipping (e.g. performing a mirror descent step) might give better gradients.

Typos, minor comments (no influence on my rating):
- In Eq. (5), it should be \partial^2 in the last "Hessian-block" multiplication.
- presribed -> prescribed
- Eqs. (7) and (8) might be easier to parse if one uses a different notation to distinguish between total derivative and partial derivative, i.e., write on the left side df(z') / d\theta. Also, I think it is clearer to write in the last terms in (7) and (8) \partial f(z') / \partial \delta z instead of \partial f(z') / \partial z'.
- In Appendix B.1, shouldn't it be \gamma=(1+\eta)/2 instead of  \gamma=\eta (1+\eta)/2?
- I've found ELU activations to work well in GAN models which involve the Jacobian w.r.t. z. Maybe it can stabilize things here as well.


Summary:

LOGAN optimizes the sampled latent generative vector z in conjunction with the generator and discriminator. By exploiting second order updates, z is optimized to allow for better training and performance of the generator and discriminator.

Pros:
+ A relatively efficient way of exploiting second order dynamics in GAN training via latent space optimization.
+ A good set of experiments demonstrating the superior performance of the proposed method on both large and small scale models and datasets.

Cons:
- Lack of code

Comments:
All in all, this appears to be a solid contribution to the GAN literature, addressing some of the limitations of CS-GAN [1]. The lack of open source code accompanying the paper (in this day and age) does it a serious disservice. I have already tried and failed to replicate the cifar10 results. There appears to be some nuance in implementation that would probably clear up if the authors release their code along with the paper.

[1] - http://proceedings.mlr.press/v97/wu19d.html

Meta-review: The authors propose to overcome challenges in GAN training through latent optimization, i.e. updating the latent code, motivated by natural gradients. The authors show improvement over previous methods.  The work is well-motivated, but in my opinion, further experiments and comparisons need to be made before the work can be ready for publication.

The authors write that "Unfortunately, SGA is expensive to scale because computing the second-order derivatives with respect to all parameters is expensive" and further "Crucially, latent optimization approximates SGA using only second-order derivatives with respect to the latent z and parameters of the discriminator and generator separately. The second-order terms involving parameters of both the discriminator and the generator – which are extremely expensive to compute – are not used. For latent z’s with dimensions typically used in GANs (e.g., 128–256, orders of magnitude less than the number of parameters), these can be computed efficiently. In short, latent optimization efficiently couples the gradients of the discriminator and generator, as prescribed by SGA, but using the much lower-dimensional latent source z which makes the adjustment scalable."

However, this is not true. Computing the Hessian vector product is not that expensive. In fact, it can be computed at a cost comparable to gradient evaluations using automatic differentiation (Pearlmutter (1994)). In frameworks such as PyTorch, this can be done efficiently using double backpropagation, so only twice the cost.  Based on the above, one of the main claims of improvement over existing methods, which is furthermore not investigated experimentally, is false. 

It is unacceptable that the authors do not compare with SGA: both in terms of quality and computational cost since that is the premise of the paper. The authors also miss recent works that successfully ran methods with Hessian-vector products: https://arxiv.org/abs/1905.12103 https://arxiv.org/abs/1910.05852

Decision: reject