ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring  for Minimax Problems and GANs
Despite remarkable empirical success, the training dynamics of generative adversarial networks, which involves solving a minimax game using stochastic gradients, is still poorly understood.In this work, we analyze last-iterate convergence of simultaneous gradient descent and its variants under the assumption of convex-concavity, guided by a continuous-time analysis with differential equations.First, we show that simGD, as is, converges with stochastic sub-gradients under strict convexity in the primal variable.Second, we generalize optimistic simGD to accommodate an optimism rate separate from the learning rate and show its convergence with full gradients.Finally, we present anchored simGD, a new method, and show convergence with stochastic subgradients.
Convergence proof of stochastic sub-gradients method and variations on convex-concave minimax problemsAn anaysis of simultaneous stochastic subgradient, simultaneous gradient with optimism, and simultaneous gradient with anchoring in the context of minmax convex concave games.This paper analyzes the dynamics of stochastic gradient descent when applied to convex-concave games, as well as GD with optimism and a new anchored GD algorithm that converges under weaker assumptions than SGD or SGD with optimism.