Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary:

LOGAN optimizes the sampled latent generative vector z in conjunction with the generator and discriminator. By exploiting second order updates, z is optimized to allow for better training and performance of the generator and discriminator.

Pros:
+ A relatively efficient way of exploiting second order dynamics in GAN training via latent space optimization.
+ A good set of experiments demonstrating the superior performance of the proposed method on both large and small scale models and datasets.

Cons:
- Lack of code

Comments:
All in all, this appears to be a solid contribution to the GAN literature, addressing some of the limitations of CS-GAN [1]. The lack of open source code accompanying the paper (in this day and age) does it a serious disservice. I have already tried and failed to replicate the cifar10 results. There appears to be some nuance in implementation that would probably clear up if the authors release their code along with the paper.

[1] - http://proceedings.mlr.press/v97/wu19d.html
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper proposes a novel training scheme for GANs, which leads to improved scores w.r.t. state-of-the-art. The idea is to update the sampled latent code in a direction improving the inner maximization in the min-max problem. The work considers both, the gradient direction and a direction motivated by the natural gradient which is shown to yield excellent performance. The overall scheme is motivated as an approximation to the (prohibitively expensive) symplectic gradient adjustment method. 

While the work contains some typos, it was easy to read and overall very well written. The experimental results are impressive and clearly validate the usefulness of the approach. Therefore, I recommend acceptance of the paper.  

On the theoretical side, I feel some parts can be improved. I'm willing to further increase my rating if some of the following points are addressed:

1. The connection to SGA is a bit hand-wavy, as terms are dropped or approximated with the only reasoning that they are difficult to compute. In some approximations (e.g. Gauss-Newton approximation of the Hessian) one can argue quite well that certain second-order terms can be dropped under some assumptions (e.g. mild nonlinearity, or vanishing near the optimum).

2. I had some troubles Sec 4 related to the natural gradient.
(a) What is p(t | z)? Is it the same p as in (37), Appendix C? I find the argument that the hinge loss pushing D(G(z)) < 0 during training hand-wavy, as for natural gradient p(t | z) should always be a valid distribution. There are also some typos which made it hard to follow the arguments there. It probably should read "an ideal generator can perfectly fool the discriminator" (not perfectly fool the generator). 

(b) Maybe it would be easier to directly argue that one wishes to approximate SGA as well as possible, rather than taking a detour through the natural gradient.

3. Why is the increment \Delta z clipped in Algorithm 1? Is there a theoretical justification? If the goal of the clipping is to stay inside the support of the uniform distribution, shouldn't it rather be z' = [z + \Delta z]? A soft clipping (e.g. performing a mirror descent step) might give better gradients.

Typos, minor comments (no influence on my rating):
- In Eq. (5), it should be \partial^2 in the last "Hessian-block" multiplication.
- presribed -> prescribed
- Eqs. (7) and (8) might be easier to parse if one uses a different notation to distinguish between total derivative and partial derivative, i.e., write on the left side df(z') / d\theta. Also, I think it is clearer to write in the last terms in (7) and (8) \partial f(z') / \partial \delta z instead of \partial f(z') / \partial z'.
- In Appendix B.1, shouldn't it be \gamma=(1+\eta)/2 instead of  \gamma=\eta (1+\eta)/2?
- I've found ELU activations to work well in GAN models which involve the Jacobian w.r.t. z. Maybe it can stabilize things here as well.

review_title: Official Blind Review #1

Meta-review: The authors propose to overcome challenges in GAN training through latent optimization, i.e. updating the latent code, motivated by natural gradients. The authors show improvement over previous methods.  The work is well-motivated, but in my opinion, further experiments and comparisons need to be made before the work can be ready for publication.

The authors write that "Unfortunately, SGA is expensive to scale because computing the second-order derivatives with respect to all parameters is expensive" and further "Crucially, latent optimization approximates SGA using only second-order derivatives with respect to the latent z and parameters of the discriminator and generator separately. The second-order terms involving parameters of both the discriminator and the generator – which are extremely expensive to compute – are not used. For latent z’s with dimensions typically used in GANs (e.g., 128–256, orders of magnitude less than the number of parameters), these can be computed efficiently. In short, latent optimization efficiently couples the gradients of the discriminator and generator, as prescribed by SGA, but using the much lower-dimensional latent source z which makes the adjustment scalable."

However, this is not true. Computing the Hessian vector product is not that expensive. In fact, it can be computed at a cost comparable to gradient evaluations using automatic differentiation (Pearlmutter (1994)). In frameworks such as PyTorch, this can be done efficiently using double backpropagation, so only twice the cost.  Based on the above, one of the main claims of improvement over existing methods, which is furthermore not investigated experimentally, is false. 

It is unacceptable that the authors do not compare with SGA: both in terms of quality and computational cost since that is the premise of the paper. The authors also miss recent works that successfully ran methods with Hessian-vector products: https://arxiv.org/abs/1905.12103 https://arxiv.org/abs/1910.05852
Decision: Reject