Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper analyzes the dynamics of stochastic gradient descent when applied to convex-concave games (motivated by the game used to train GANs, which is typically not convex-concave), as well as the previously proposed GD with optimism and a new anchored GD algorithm that provably converges under weaker assumptions than SGD or SGD with optimism.

This seems like a nice contribution to the GAN theory literature, and the anchored SSSGD algorithm might well have significant practical value (although more thorough experiments would be needed to make this claim). As such I recommend acceptance.

The obvious critique to raise of this kind of work is that the GAN problems that motivate it are clearly not convex-concave, and so it is unclear how or whether the results can or should inform practice. The simplest way to make the case for that kind of relevance is empirically, so I'd recommend that the authors consider what kinds of GAN experiments would support (or falsify) the claim that their theoretical results have some hope of generalizing to the non-convex-concave setting. Figure 4 is suggestive, but it doesn't say much except that anchored Adam might occasionally be a good choice (and sometimes isn't). The samples in Figure 3 are pretty far from the state of the art, and in any case Figure 3 doesn't even say which training method generated them.

Finally, PLEASE don't include acknowledgments in a paper that's under double-blind review; it compromises your anonymity, which in principle could be grounds for a desk rejection.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: *Summary*
This paper provides the analysis of three algorithms in the context of minmax convex concave games: Simultaneous stochastic subgradient method, Simultaneous gradient with optimism and Simultaneous gradient with anchoring. These three algorithm are first analysed in continuous time with an ODE perspective and then leverage these intuitions and techniques to analyze the discrete time versions.

I think that these contributions are of interest of ICLR community, however, I have some concerns regarding the presentations of the results. 

*Decision*
I vote for a weak accept that could move to an accept if the authors improve the presentation of the paper:
The paragraph “Regularized dynamics and convergence’ in Section 3.1 in quite hard to follow. This subsection is basically the proof of the convergence of the continuous version of  GD-O. Stating the result before the proof would help the reader to understand where the authors want to go. 
Very same point for The subsection 4.1
For the stochastic version of SimGD-A a new parameter $\epsilon$ is introduced without any comment or description on why it is necessary. 
A FID above 40 for MNIST is very far from standard results (that are below 1). Thus I am not convinced by the practical advantage of Anchored Adam on these models that have performances results very far from the standard ones.  (for instance between 20 and 25 for CIFAR10 is very reasonable). It is maybe because you do not use convolutional layers in your architecture. I think that using a DGAN architecture (for instance the one from the pytorch tutorial https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html) would give the expected results (i.e. FID close to 0).

*Questions*

- When you cite Lassale Principle you mention that if $z_\infty$ is a limit point of $z(t)$ then starting at $z_\infty$, you stay at a constant distance to $z_*$. But in the bilinear example you give any point in the circle is not a limit point (size no dynamics converge to it). I guess when you said limit point you meant adherent point ?
- Is the condition $\epsilon$ only necessary for the proof ? Does $\epsilon =1$ work in practice ? 
If no, what is the best value for $\epsilon$ ?

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: The paper studies last-iterate convergence of simultaneous gradient descent and related algorithms in (convex-concave) GANs.

The experiments are very weak. Figure 4 shows that anchored Adam outperforms Adam and optimistic Adam in terms of FID on MNIST, but not CIFAR-10. 

The authors cite many of the vast number of algorithms that have been proposed to train GANs (Heusel, Mescheder, Gidel, Gemp, and on and on ... and on…) and discuss some of them in the analysis. However, when it comes to experiments, they only compare against vanilla Adam (!) and Optimism. I would summarize the experiments as *suggesting* that anchoring doesn’t hurt on MNIST or CIFAR-10. The paper doesn’t tune beta and gamma, so it’s hard to be sure. 

The motivation for the paper is GANs, but GANs are not convex-concave. The analysis is therefore not directly relevant. From the experiments, it is not clear *at all* whether anchored Adam is *actually* an improvement in practice over any of the alternative algorithms that the authors discuss or cite. 

In short, the contribution is unclear. The analysis is better suited to a venue like COLT. To be a reasonable ICLR submission, the paper needs to compare against more baselines at a bare minimum.

review_title: Official Blind Review #2

Meta-review: Motivated by GANs, the authors study the convergence of stochastic subgradient                                                     
descent on convex-concave minimax games.                                                                                           
They introduced an improved "anchored" SGD variant, that provably converges                                                        
under milder assumptions that the base algorithm.                                                                                  
It is applied to training GANs on MNIST and CIFAR-10, partially showing                                                            
improvements over alternative training methods.                                                                                    
                                                                                                                                   
A main point of criticism that the reviewers identify is the strength of the                                                       
assumptions needed for the analysis.                                                                                               
Furthermore, the experimental results were deemed weak as the reported scores                                                      
are far away from the SOTA, and only simple baselines were compared against.  
Decision: Reject