user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring  for Minimax Problems and GANs

Paper content:
Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood.
In this work, we analyze last-iterate convergence of simultaneous gradient descent (simGD) and its variants under the assumption of convex-concavity, guided by a continuous-time analysis with differential equations.
First, we show that simGD, as is, converges with stochastic sub-gradients under strict convexity in the primal variable.
Second, we generalize optimistic simGD to accommodate an optimism rate separate from the learning rate and show its convergence with full gradients.
Finally, we present anchored simGD, a new method, and show convergence with stochastic subgradients.
Training of generative adversarial networks (GAN) (Goodfellow et al., 2014) , solving a minimax game using stochastic gradients, is known to be difficult.
Despite the remarkable empirical success of GANs, further understanding the global training dynamics empirically and theoretically is considered a major open problem (Goodfellow, 2016; Radford et al., 2016; Metz et al., 2017; Mescheder et al., 2018; Odena, 2019) .
The local training dynamics of GANs is understood reasonably well.
Several works have analyzed convergence assuming the loss functions have linear gradients and assuming the training uses full (deterministic) gradients.
Although the linear gradient assumption is reasonable for local analysis (even though the loss functions may not be continuously differentiable due to ReLU activation functions) such results say very little about global convergence.
Although the full gradient assumption is reasonable when the learning rate is small, such results say very little about how the randomness affects the training.
This work investigates global convergence of simultaneous gradient descent (simGD) and its variants for zero-sum games with a convex-concave cost using using stochastic subgradients.
We specifically study convergence of the last iterates as opposed to the averaged iterates.
Organization.
Section 2 presents convergence of simGD with stochastic subgradients under strict convexity in the primal variable.
The goal is to establish a minimal sufficient condition of global convergence for simGD without modifications.
Section 3 presents a generalization of optimistic simGD , which allows an optimism rate separate from the learning rate.
We prove the generalized optimistic simGD using full gradients converges, and experimentally demonstrate that the optimism rate must be tuned separately from the learning rate when using stochastic gradients.
However, it is unclear whether optimistic simGD is theoretically compatible with stochastic gradients.
Section 4 presents anchored simGD, a new method, and presents its convergence with stochastic subgradients.
Anchoring represents what we consider to be the strongest contribution of this work.
The presentation and analyses of Sections 2, 3, and 4 are guided by continuous-time firstorder ordinary differential equations (ODE).
In particular, we interpret optimism and anchoring as discretizations of certain regularized dynamics.
Section 5 experimentally demonstrates the benefit of optimism and anchoring for training GANs in some setups.
Prior work.
There are several independent directions for improving the training of GANs such as designing better architectures, choosing good loss functions, or adding appropriate regularizers (Radford et al., 2016; Sønderby et al., 2017; Gulrajani et al., 2017; Wei et al., 2018; Roth et al., 2017; Mescheder et al., 2018; 2017; Miyato et al., 2018) .
In this work, we accept these factors as a given and focus on how to train (optimize) the model effectively.
Optimism is a simple modification to remedy the cycling behavior of simGD, which can occur even under the bilinear convex-concave setup Daskalakis & Panageas, 2018; Mertikopoulos et al., 2019; Gidel et al., 2019a; Liang & Stokes, 2019; Mokhtari et al., 2019; Peng et al., 2019) .
These prior work assume the gradients are linear and use full gradients.
Although the recent name 'optimism' originates from its use in online optimization (Chiang et al., 2012; Rakhlin & Sridharan, 2013a; b; Syrgkanis et al., 2015) , the idea dates back to Popov's work in the 1980s (Popov, 1980) and has been studied independently in the mathematical programming community (Malitsky & Semenov, 2014; Malitsky, 2015; Malitsky & Tam, 2018; Malitsky, 2019; Csetnek et al., 2019) .
We note that there are other mechanisms similar to optimism and anchoring such as "prediction" (Yadav et al., 2018) , "negative momentum" (Gidel et al., 2019b) , and "extragradient" (Korpelevich, 1976; Tseng, 2000; Chavdarova et al., 2019) .
In this work, we focus on optimism and anchoring.
Classical literature analyze convergence of the Polyak-averaged iterates (which assigns less weight to newer iterates) when solving convex-concave saddle point problems using stochastic subgradients (Bruck, 1977; Nemirovski & Yudin, 1978; Nemirovski et al., 2009; Juditsky et al., 2011; Gidel et al., 2019a) .
For GANs, however, last iterates or exponentially averaged iterates (Yazıcı et al., 2019) (which assigns more weight to newer iterates) are used in practice.
Therefore, the classical work with Polyak averaging do not fully explain the empirical success of GANs.
We point out that we are not the first to utilize classical techniques for analyzing the training of GANs.
In particular, the stochastic approximation technique (Heusel et al., 2017; Duchi & Ruan, 2018) , control theoretic techniques (Heusel et al., 2017; Nagarajan & Kolter, 2017) , ideas from variational inequalities and monotone operator theory (Gemp & Mahadevan, 2018; Gidel et al., 2019a) , and continuous-time ODE analysis (Heusel et al., 2017; Csetnek et al., 2019) have been utilized for analyzing GANs.
In this work, we analyzed the convergence of SSSGD, Optimistic simGD, and Anchored SSSGD.
Under the assumption that the cost L is convex-concave, Anchored SSSGD provably converges under the most general setup.
Through experiments, we showed that the practical GAN training benefits from optimism and anchoring in some (but not all) setups.
Generalizing these results to accommodate projections and proximal operators, analogous to projected and proximal gradient methods, is an interesting direction of future work.
Weight clipping and spectral normalization (Miyato et al., 2018) A FURTHER DISCUSSION ON THE CONVERGENCE RESULTS Theorems 1, 2, 3, and 4 use related but different notions of convergence.
Theorems 1 and 4 are asymptotic (has no rate) while Theorems 2 and 3 are non-asymptotic (has a rate).
Theorems 1 and 3 respectively show almost sure and L 2 convergence of the iterates.
Theorems 2 and 3 show convergence of the squared gradient norm for the best and last iterates, respectively.
We did not make these choices.
The choices were dictated by what we can prove based on the analysis.
The discrete-time analysis of SimGD-O of Theorem 2 bounds the squared gradient norm of the best iterate, while the continuous-time analysis bounds the squared gradient norm of the "last iterate" (at terminal time).
The discrepancy comes from the fact that while we have monotonic decrease of g(t) in continuous-time, we have no analogous monotonicity condition on g k in discrete-time.
To the best of our knowledge, there is no result establishing a O(1/k) rate on the squared gradient norm of the last iterate for SimGD-O or the related "extragradient method" Korpelevich (1976) .
Theorem 3 is the first result showing a rate close to O(1/k) on the last literate.
For SimGD-O and Corollary 1, the parameter choices are almost optimal.
The optimal choices that minimize the bound of Theorem 2 are α = 0.124897/R and β = 1.94431α; they provide a factor of 135.771, a very small improvement over the factor 136 of Corollary 1.
For SimGD-A and Theorem 3, there is a discrepancy in the rate between the continuous time analysis O(1/t 2 ) and the discrete time rate O(1/k 2−2p ) for p ∈ (1/2, 1), which is slightly slower than O(1/k).
In discretizing the continuous-time calculations to obtain a discrete proof, errors accumulate and prevent the rate from being better than O(1/k).
This is not an artifact of the proof.
Simple tests on bilinear examples show divergence when p < 1/2.
SSSGD-A and Theorem 4 involves the parameter ε.
While the proof requires ε > 0, we believe this is an artifact of the proof.
In particular, we conjecture that Lemma 17 holds with o(s/τ ) rather than O(s/τ ), and, if so, it is possible to establish convergence with ε = 0.
In Figure 2 , it seems that that the choice ε = 0 and p = 2/3 is optimal for SSSGD-A.
While we do not have a theoretical explanation for this, we point out that this is not surprising as p = 2/3 is known to be optimal in stochastic convex minimization (Moulines & Bach, 2011; Taylor & Bach, 2019) .
Theorems 2, 3, and 4 extend to monotone operators (Ryu & Boyd, 2016; Bauschke & Combettes, 2017) without any modification to their proofs.
In infinite dimensional setups (which is of interest in the field of monotone operators) Theorem 4 establishes strong convergence, while many convergence results (including Theorems 2 and 3) establish weak convergence.
However, Theorem 1 does not extend to monotone operators, as the use of the LaSalle-Krasnovskii principle is particular to convex-concave saddle functions.

assistant: text: Convergence proof of stochastic sub-gradients method and variations on convex-concave minimax problems