Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies how adaptive methods help performance in GANs. The study empirically finds that SGDA with the same vector norm as Adam reaches similar better performance. Based on this observation, normalized SGDA (nSGDA) is proposed as a simpler alternative to Adam. nSGDA is evaluated on several datasets and the results demonstrate that nSGDA is more stable than SGDA. 
main_review: Strengths:
1. This paper finds that the adaptive magnitude of Adam is critical to the performance of GANs. Personally, I think the analysis of the optimizers for GANs is an interesting topic. 
2. In the theoretical aspect, the analysis explains the success of nSGDA.
3. The results show that nSGDA achieves a similar performance compared to Adam using WGAN-GP.

Weaknesses:
1. Although the paper finds that the adaptive magnitude of Adam helps the performance in GANs, the proposed nSGDA is still inferior to Adam. The study does not analyze why Adam is superior to nSGDA. I would expect the study presents how to further improve nSGDA or even presents methods that outperform Adam.
2. The study uses a ResNet WGAN-GP as the network backbone. I would expect to use more recent GANs, such as BigGAN and StyleGAN.
3. For the optimizer, the current experiments are not comprehensive enough. I think the proposed optimizer should be evaluated over different network architectures, different resolutions, and different GANs. 

summary_of_the_review: Although the topic and findings of this paper are interesting, I think the contributions of the current version are limited, especially for the performance of nSGDA and the experiments. 

******************************
Post Rebuttal Comments:

I thank the authors for their efforts in addressing my concerns. However, I still have the major concern about the performance of the proposed method over commonly used GANs such as BigGAN and StyleGAN. The authors respond that simpler with similar performances is better. However, there is no result to show similar performance over BigGAN or StyleGAN. Thus I keep my original rating.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this manuscript the authors investigate the effect of normalized and unormalized gradient updates on the convergence of GANs. In a first experiment a DCGAN model is trained with Adam and it is shown that the best models evaluated with the FID have balanced learning rates but haven't converged while models trained with extremly unbalanced learning rates converge while having high FIDs i.e. didn't learn the training distribution.
In the second experiment it is shown that an adaptive gradient magnitude helps to train a good model compared to an adaptive gradient direction. In a next step using the WGAN-GP model on CIFAR-10, LSUN Churches, STL-10, and CelebA-HQ datasets the optimizers Adam, normalized gradient updates and unnormalized SGDA were compared. The normalized optimizers including Adam outperformed SGDA. Finally it was shown that SGDA is more sensitive to the batch-size. In a final experiment on a small toy dataset it was shown that SGDA suffers from mode collapse while normalized gradient updates guide the generator to learn the modes of the dataset.

main_review: Carefully adjusted update steps in Reinforcement Learning (actor/critic models) and GAN training are vital to reach convergence. E.g. [1] proposed a control variable to level out discriminator and generator. [2] has proven that under some assumptions competing agents converge to an equilibrium if the learning speeds are carefully adjusted. [3] based on the results of [2] have shown that the results are also valid in the GAN context where the slower agent is the generator and the faster agent the discriminator. To reach convergence a faster leader, critic, discriminator guides a slower follower, generator such that the follower, generator is fast enough to follow but in the same time is not too fast to disturb the leader [4]. My main concern with this paper is, that having normalized gradient updates is probably a good starting point to select optimal update steps but according to the research mentioned above it's more likely normalized gradient updates still need to be scaled (for both agents separately).
As the learning speeds of the agents depend not only on the learning rates, e.g. the complexity of the architecture is important as well, one has to be careful to select the correct settings. E.g. in the DCGAN architecture the generator has more parameters and needs a higher learning rate compared to the discriminator to converge cf. Section A5 in [3]. However, in the first experiment the authors train the DCGAN generator with a smaller learning rate which probably leads not to optimal FIDs. I suggest to run additional experiments with higher learning rates for the generator and add them to the results shown in Figure 1. In section 4 i'm not sure if i can see a theoretical proof of theorems 4.1 and 4.2.

[1] Berthelot et al., BEGAN: Boundary Equilibrium Generative Adversarial Networks,  https://arxiv.org/abs/1703.10717 balance
[2] V. S. Borkar. Stochastic approximation with two time scales. Systems & Control Letters,
29(5):291–294, 1997
[3] Heusel et al. GANs trained by a two time-scale update rule converge to a local nash equilibrium. Advances in
neural information processing systems, 30, 2017.
[4] M. W. Hirsch. Convergent activation dynamics in continuous time networks. Neural Networks,
2(5):331–349, 1989.
[5] Tanner Fiez and Lillian Ratliff. Gradient descent-ascent provably converges to strict local minmax
equilibria with a finite timescale separation. arXiv preprint arXiv:2009.14820, 2020.
summary_of_the_review: The importance of balanced updates of the generator and discriminator in GAN training is already well known. Using normalized gradient update steps for convergence is plausible, however not enough as shown in [2,3,4,5].
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper dissects the success of Adam in GAN training by swap the gradient direction/magnitude with that of SGDA. By doing so, authors found the Adam produces higher quality solutions relative to SGDA in GANs mainly due to its adaptive magnitude and not to its adaptive direction. Inspired by this observation, the combination of direction of SGDA and magnitude of Adam yields the use of normalized SGDA (nSGDA) in GAN training, which consistently competes with Adam.
main_review: This paper presents a great study that dissects the success of Adam in GAN training, which covers insightful observations and comprehensive analysis both empirically and theoretically. The findings of this paper, such as "it is the adaptive magnitude of Adam that matters" and "the learning objective of GAN converges does not imply the generator synthesizes high-quality samples" will definitely encourage more exploration along this direction, helping us better understand the optimization process of similar non-convex non-concave min-max problems.
 
Minor concerns:
1. The space is overly adjusted. Related work should be included in a separate section.
2. I suggest avoiding representing both distribution and distriminator as D, which hurts the readability significantly.
3. typos, e.g. autorefalg:adaptive?
4. While the finding of nSGDA is indeed interesting and insightful, it performs comparably with Adam. Can authors discuss the potential advantages of nSGDA over Adam?

summary_of_the_review: I enjoy reading this paper. The findings are insightful with comprehensive empirical and theoretical analysis.
review_title: null

Meta-review: The paper studies how adaptive methods help train GANs to achieve better FID scores. It empirically shows that the adaptive magnitude in ADAM is the reason for ADAM's wide adoption for GAN training. The paper receives three reviews: one ranked the paper "accept, good paper" and two ranked the paper "marginally below the acceptance threshold". The supportive reviewer likes the findings in the paper interesting but does not provide enough explanation on the significance of the findings. On the other hand, the negative reviewers raise several concerns, including the GAN architectures used in the paper are outdated and the achieved performance gain is not major. As the paper focuses on performance instead of convergence, the meta-reviewer feels it would be better to include results on SOTA GAN architectures. The provided rebuttal does not lead to any review score change. Consolidating the review and rebuttal, the meta-reviewer feels the paper needed to be improved to meet the bar and would not recommend its acceptance.
Decision: Reject