Title: TRAINING GANS WITH OPTIMISM

Abstract: We address the issue of limit cycling behavior in training Generative Adversarial Networks and propose the use of Optimistic Mirror Decent (OMD) for training Wasserstein GANs. Recent theoretical results have shown that optimistic mirror decent (OMD) can enjoy faster regret rates in the context of zero-sum games. WGANs is exactly a context of solving a zero-sum game with simultaneous noregret dynamics. Moreover, we show that optimistic mirror decent addresses the limit cycling problem in training WGANs. We formally show that in the case of bi-linear zero-sum games the last iterate of OMD dynamics converges to an equilibrium, in contrast to GD dynamics which are bound to cycle. We also portray the huge qualitative difference between GD and OMD dynamics with toy examples, even when GD is modified with many adaptations proposed in the recent literature, such as gradient penalty or momentum. We apply OMD WGAN training to a bioinformatics problem of generating DNA sequences. We observe that models trained with OMD achieve consistently smaller KL divergence with respect to the true underlying distribution, than models trained with GD variants. Finally, we introduce a new algorithm, Optimistic Adam, which is an optimistic variant of Adam. We apply it to WGAN training on CIFAR10 and observe improved performance in terms of inception score as compared to Adam.

1. INTRODUCTION:

Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have proven a very successful approach for fitting generative models in complex structured spaces, such as distributions over images. GANs frame the question of fitting a generative model from a data set of samples from some distribution as a zero-sum game between a Generator (G) and a discriminator (D). The Generator is represented as a deep neural network which takes as input random noise and outputs a sample in the same space of the sampled data set, trying to approximate a sample from the underlying distribution of data. The discriminator, also modeled as a deep neural network is attempting to discriminate between a true sample and a sample generated by the generator. The hope is that at the equilibrium of this zero-sum game the generator will learn to generate samples in a manner that is indistinguishable from the true samples and hence has essentially learned the underlying data distribution.

Despite their success at generating visually appealing samples when applied to image generation tasks, GANs are very finicky to train. One particular problem, raised for instance in a recent survey as a major issue (Goodfellow, 2017) is the instability of the training process. Typically training of GANs is achieved by solving the zero-sum game via running simultaneously a variant of a Stochastic Gradient Descent algorithm for both players (potentially training the discriminator more frequently than the generator).

The latter amounts essentially to solving the zero-sum game via running no-regret dynamics for each player. However, it is known from results in game theory, that no-regret dynamics in zerosum games can very often lead to limit oscillatory behavior, rather than converge to an equilibrium. Even in convex-concave zero-sum games it is only the average of the weights of the two players that constitutes an equilibrium and not the last-iterate. In fact recent theoretical results of Mertikopoulos et al. (2017) show the strong result that no variant of GD that falls in the large class of Follow-the-Regularized-Leader (FTRL) algorithms can converge to an equilibrium in terms of the last-iterate and are bound to converge to limit cycles around the equilibrium.

Averaging the weights of neural nets is a prohibitive approach in particular because the zero-sum game that is defined by training one deep net against another is not a convex-concave zero-sum game. Thus it seems essential to identify training algorithms that make the last iterate of the training be very close to the equilibrium, rather than only the average.

Contributions.:

In this paper we propose training GANs, and in particular Wasserstein GANs Arjovsky et al. (2017) , via a variant of gradient descent known as Optimistic Mirror Descent. Optimistic Mirror Descent (OMD) takes advantage of the fact that the opponent in a zero-sum game is also training via a similar algorithm and uses the predictability of the strategy of the opponent to achieve faster regret rates. It has been shown in the recent literature that Optimistic Mirror Descent and its generalization of Optimistic Follow-the-Regularized-Leader (OFTRL), achieve faster convergence rates than gradient descent in convex-concave zero-sum games (Rakhlin & Sridharan, 2013a; b) and even in general normal form games (Syrgkanis et al., 2015) . Hence, even from the perspective of faster training, OMD should be preferred over GD due to its better worst-case guarantees and since it is a very small change over GD.

Moreover, we prove the surprising theoretical result that for a large class of zero-sum games (namely bi-linear games), OMD actually converges to an equilibrium in terms of the last iterate. Hence, we give strong theoretical evidence that OMD can help in achieving the long sought-after stability and last-iterate convergence required for GAN training. The latter theoretical result is of independent interest, since solving zero-sum games via no-regret dynamics has found applications in many areas of machine learning, such as boosting (Freund & Schapire, 1996) . Avoiding limit cycles in such approaches could help improve the performance of the resulting solutions.

We complement our theoretical result with toy simulations that portray exactly the large qualitative difference between OMD as opposed to GD (and its many variants, including gradient penalty, momentum, adaptive step size etc.). We show that even in a simple distribution learning setting where the generator simply needs to learn the mean of a multi-variate distribution, GD leads to limit cycles, while OMD converges pointwise.

Moreover, we give a more complex application to the problem of learning to generate distributions of DNA sequences of the same cellular function. DNA sequences that carry out the same function in the genome, such as binding to a specific transcription factor, follow the same nucleotide distribution. Characterizing the DNA distribution of different cellular functions is essential for understanding the functional landscape of the human genome and predicting the clinical consequence of DNA mutations (Zeng et al., 2015; 2016; Zeng & Gifford, 2017) . We perform a simulation study where we generate samples of DNA sequences from a known distribution. Subsequently we train a GAN to attempt to learn this underlying distribution. We show that OMD achieves consistently better performance than GD variants in terms of the Kullback-Leibler (KL) divergence between the distribution learned by the Generator and the true distribution.

Finally, we apply optimism to training GANs for images and introduce the Optimistic Adam algorithm. We show that it achieves better performance than Adam, in terms of inception score, when trained on CIFAR10.

z ∈ F (from some simple distribution F ) and outputs a sample G θ (z) ∈ X. We will focus on addressing this problem via a Generative Adversarial Network (GAN) training strategy.

The GAN training strategy defines as a zero-sum game between a generator deep neural network G θ (•) and a discriminator neural network D w (•). The generator takes as input random noise z ∼ F , and outputs a sample G θ (z) ∈ X. A discriminator takes as input a sample x (either drawn from the true distribution Q or from the generator) and attempts to classify it as real or fake. The goal of the generator is to fool the discriminator.

In the original GAN training strategy Goodfellow et al. (2014) , the discriminator of the zero sum game was formulated as a classifier, i.e. D w (x) ∈ [0, 1] with a multinomial logistic loss. The latter boils down to the following expected zero-sum game (ignoring sampling noise).

inf θ sup w E x∼Q [log(D w (x))] + E z∼F [log(1 -D w (G θ (z)))]

(1)

If the discriminator is very powerful and learns to accurately classify all samples, then the problem of the generator amounts to minimizing the Jensen-Shannon divergence between the true distribution and the generators distribution. However, if the discriminator is very powerful, then in practice the latter leads to vainishing gradients for the generator and inability to train in a stable manner.

The latter problem lead to the formulation of Wasserstein GANs (WGANs) Arjovsky et al. (2017) , where the discriminator rather than being treated as a classifier (equiv. approximating the JS divergence) is instead trying to approximate the Wasserstein-1 or earth-mover metric between the true distribution and the distribution of the generator. In this case, the function D w (x) is not constrained to being a probability in [0, 1] but rather is an arbitrary 1-Lipschitz function of x. This reasoning leads to the following zero-sum game:

EQUATION

If the function space of the discriminator covers all 1-Lipschitz functions of x, then the quantity

sup w E x∼D [D w (x)] -E z∼F [D w (G θ (z))

] that the generator is trying to minimize corresponds to the earth-mover distance between the true distribution Q and the distribution of the generator. Given the success of WGANs we will focus on WGANs in this paper.

2.1. GRADIENT DESCENT VS OPTIMISTIC MIRROR DESCENT:

The standard approach to training WGANs is to train simultaneously the parameters of both networks via stochastic gradient descent. We begin by presenting the most basic version of adversarial training via stochastic gradient descent and then comment on the multiple variants that have been proposed in the literature in the following section, where we compare their performance with our proposed algorithm for a simple example.

Let us start how training a GAN with gradient descent would look like in the absence of sampling error, i.e. if we had access to the true distribution Q. For simplicity of notation, let:

EQUATION

denote the loss in the expected zero-sum game of WGAN, as defined in Equation (2), i.e. inf θ sup w L(θ, w). The classic WGAN approach is to solve this game by running gradient descent (GD) for each player, i.e. for t ∈ {1, . . . , T -1}: with ∇ w,t = ∇ w L(θ t , w t ) and

EQUATION

If the loss function L(θ, w) was convex in θ and concave w, θ and w lie in some bounded convex set and the step size η is chosen of the order 1 √ T , then standard results in game theory and no-regret learning (see e.g. Freund & Schapire (1999) ) imply that the pair ( θ, w) of average parameters, i.e. w = 1 T T t=1 w t and θ = 1 T T t=1 θ t is an -equilibrium of the zero-sum game, for = O 1 √ T . However, no guarantees are known beyond the convex-concave setting and, more importantly for the paper, even in convex-concave games, no guarantees are known for the last-iterate pair (θ T , w T ). (Rakhlin & Sridharan, 2013a) proposed an alternative algorithm for solving zero-sum games in a decentralized manner, namely Optimistic Mirror Descent (OMD), that achieves faster convergence rates to equilibrium of = O1 T for the average of parameters. The algorithm essentially uses the last iterations gradient as a predictor for the next iteration's gradient. This follows from the intuition that if the opponent in the game is using a stable (or regularized) algorithm, then the gradients between the two iterations will not change much. Later Syrgkanis et al. (2015) showed that this intuition extends to show faster convergence of each individual player's regret in general normal form games.

Rakhlin and Sridharan:

Given these favorable properties of OMD when learning in games, we propose replacing GD with OMD when training WGANs. The update rule of a OMD is a small adaptation to GD. OMD is parameterized by a predictor of the next iteration's gradient which could either be simply last iteration's gradient or an average of a window of last gradient or a discounted average of past gradients. In the case where the predictor is simply the last iteration gradient, then the update rule for OMD boils down to the following simple form:

w t+1 = w t + 2η • ∇ w,t -η • ∇ w,t-1 θ t+1 = θ t -2η • ∇ θ.t + η • ∇ θ,t-1

(5)

The simple modification in the GD update rule, is inherently different than any of the existing adaptations used in GAN training, such as Nesterov's momentum, or gradient penalty.

General OMD and intuition. The intuition behind OMD can be more easily understood when GD is viewed through the lens of the Follow-the-Regularized-Leader formulation. In particular, it is well known that GD is equivalent to the Follow-the-Regularized-Leader algorithm with an 2 regularizer (see e.g. Shalev-Shwartz ( 2012)), i.e.:

w t+1 = arg max w η t τ =1 w, ∇ w,τ -w 2 2 θ t+1 = arg min θ η t τ =1 θ, ∇ θ,τ + θ 2 2 (6)

It is known that if the learner knew in advance the gradient at the next iteration, then by adding that to the above optimization would lead to constant regret that comes solely from the regularization term 1 . OMD essentially augments FTRL by adding a predictor M t+1 of the next iterations gradient, i.e.:

w t+1 = arg max w η t τ =1 w, ∇ w,τ + w, M w,t+1 -w 2 2 θ t+1 = arg min θ η t τ =1 θ, ∇ θ,τ + θ, M θ,t+1 + θ 2 2 (7)

For an arbitrary set of predictors, the latter boils down to the following set of update rules:

EQUATION

In the theoretical part of the paper we will focus on the case where the predictor is simply the last iteration gradient, leading to update rules in Equation ( 5). In the experimental section we will also explore performance of other alternatives for predictors.

2.2. STOCHASTIC GRADIENT DESCENT AND STOCHASTIC OPTIMISTIC MIRROR DESCENT:

In practice we don't really have access to the true distribution Q and hence we replace Q with an empirical distribution Q n over samples {x 1 , . . . , x n } and F n of random noise samples {z 1 , . . . , z n }, leading to empirical loss for the zero-sum game of:

EQUATION

Even in this setting it might be impractical to compute the gradient of the expected loss with respect to Q n or F n , e.g.

E x∼Qn [∇ w D w (x)].

However, GD and OMD still leads to small loss if we replace gradients with unbiased estimators of them. Hence, we can replace expectation with respect to Q n or F n , by simply evaluating the gradients at a single sample or on a small batch of B samples. Hence, we can replace the gradients at each iteration with the variants:

EQUATION

Replacing ∇ w,t and ∇ θ,t with the above estimates in Equation ( 4) and ( 5), leads to Stochastic Gradient Descent (SGD) and Stochastic Optimistic Mirror Decent (SOMD) correspondingly.

3. AN ILLUSTRATIVE EXAMPLE: LEARNING THE MEAN OF A DISTRIBUTION:

We consider the following very simple WGAN example: The data are generated by a multivariate normal distribution, i.e. Q N (v, I) for some v ∈ R d . The goal is for the generator to learn the unknown parameter v. In Appendix C we also consider a more complex example where the generator is trying to learn a co-variance matrix.

We consider a WGAN, where the discriminator is a linear function and the generator is a simple additive displacement of the input noise z, which is drawn from F N (0, I), i.e:

D w (x) = w, x G θ (z) = z + θ (11)

The goal of the generator is to figure out the true distribution, i.e. to converge to θ = v. The WGAN loss then takes the simple form:

EQUATION

We first consider the case where we optimize the true expectations above rather than assuming that we only get samples of x and samples of z. Due to linearity of expectation, the expected zero-sum game takes the form:

EQUATION

We see here that the unique equilibrium of the above game is for the generator to choose θ = v and for the discriminator to choose w = 0. For this simple zero sum game, we have ∇ w,t = v -θ t and ∇ θ,t = -w t . Hence, the GD dynamics take the form:

w t+1 =w t + η(v -θ t ) θ t+1 =θ t + ηw t (GD Dynamics for Learning Means)

while the OMD dynamics take the form:

w t+1 =w t + 2η • (v -θ t ) -η • (v -θ t-1 ) θ t+1 =θ t + 2η • w t -η • w t-1 (OMD Dynamics for Learning Means)

We simulated simultaneous training in this zero-sum game under the GD and under OMD dynamics and we find that GD dynamics always lead to a limit cycle irrespective of the step size or other modifications. In Figure 1 we present the behavior of the GD vs OMD dynamics in this game for v = (3, 4). We see that even though GD dynamics leads to a limit cycle (whose average does indeed equal to the true vector), the OMD dynamics converge to v in terms of the last iterate. In Figure 2 we see that the stability of OMD even carries over to the case of Stochastic Gradients, as long as the batch size is of decent size.

In the appendix we also portray the behavior of the GD dynamics even when we add gradient penalty (Gulrajani et al., 2017) to the game loss (instead of weight clipping), adding Nesterov momentum to the GD update rule (Nesterov, 1983) or when we train the discriminator multiple times in between a train iteration of the generator. We see that even though these modifications do improve the stability of the GD dynamics, in the sense that they narrow the band of the limit cycle, they still lead to a non-vanishing limit cycle, unlike OMD.

In the next section, we will in fact prove formally that for a large class of zero-sum games including the one presented in this section, OMD dynamics converge to equilibrium in the sense of last-iterate convergence, as opposed to average-iterate convergence.

4. LAST-ITERATE CONVERGENCE OF OPTIMISTIC ADVERSARIAL TRAINING:

In this section, we show that Optimistic Mirror Descent exhibits final-iterate, rather than only average-iterate convergence to min-max solutions for bilinear functions. More precisely, we consider the problem min x max y x T Ay, for some matrix A, where x and y are unconstrained. In Appendix D, we also show that our convergence result appropriately extends to the general case, where the bi-linear game also contains terms that are linear in the players' individual strategies, i.e. games of the form: inf

EQUATION

In the simpler min x max y x T Ay problem, Optimistic Mirror Descent takes the following form, for all t ≥ 1:

EQUATION

EQUATION

Initialization: For the above iteration to be meaningful we need to specify x 0 , x -1 , y 0 , y -1 . We choose any x 0 ∈ R(A), and y 0 ∈ R(A T ), and set x -1 = 2x 0 and y -1 = 2y 0 , where R(•) represents the column space of A. In particular, our initialization means that the first step taken by the dynamics gives x 1 = x 0 and y 1 = y 0 .

We will analyze Optimistic Mirror Descent under the assumption λ ∞ ≤ 1, where λ ∞ = max{||A||, ||A T ||} and || • || denotes spectral norm of matrices. We can always enforce that λ ∞ ≤ 1 by appropriately scaling A. Scaling A by some positive factor clearly does not change the min-max solutions (x * , y * ), only scales the optimal value x * T Ay * by the same factor.

We remark that the set of equilibrium solutions of this minimax problem are pairs (x, y) such that x is in the null space of A T and y is in the null space of A. In this section we rigorously show that Optimistic Mirror Descent converges to the set of such min-max solutions. This is interesting in light of the fact that Gradient Descent actually diverges, even in the special case where A is the identity matrix, as per the following proposition whose proof is provided in Appendix D.3.

Proposition 1. Gradient descent applied to the problem min x max y x T y diverges starting from any initialization x 0 , y 0 such that x 0 , y 0 = 0.

Next, we state our main result of this section, whose proof can be found in Appendix D, where we also state its appropriate generalization to the general case ( 14).

Theorem 1 (Last Iterate Convergence of OMD). Consider the dynamics of Eq. ( 15) and ( 16) and any initialization 1 2 x -1 = x 0 ∈ R(A), and 1 2 y -1 = y 0 ∈ R(A T ). Let also

γ = AA T + ,

where for a matrix X we denote by X + its generalized inverse and by ||X|| its spectral norm. Suppose that A ≡ λ ∞ ≤ 1 and that η is a small enough constant satisfying η < 1/(3γ 2 ). Letting

∆ t = A T x t 2 2 + ||Ay t || 2 2

, the OMD dynamics satisfy the following:

∆ 1 = ∆ 0 ≥ 1 (1 + η) 2 ∆ 2 ∀t ≥ 3 : ∆ t ≤ 1 - η 2 γ 2 ∆ t-1 + 16η 3 ∆ 0 .

In particular, ∆ t → O(ηγ 2 ∆ 0 ), as t → +∞, and for large enough t, the last iterate of OMD is within O( √ η • γ √ ∆ 0 ) distance from the space of equilibrium points of the game, where √ ∆ 0 is the distance of the initial point (x 0 , y 0 ) from the equilibrium space, and where both distances are taken with respect to the norm x T AA T x + y T A T Ay.

5. EXPERIMENTAL RESULTS FOR GENERATING DNA SEQUENCES:

We take our theoretical intuition to practice, applying OMD to the problem of generating DNA sequences from an observed distribution of sequences. DNA sequences that carry out the same function can be viewed as samples from some distribution. For many important cellular functions, this distribution can be well modeled by a position-weight matrix (PWM) that specifies the probability of different nucleotides occuring at each position (Stormo, 2000) . Thus, training GANs from DNA sequences sampled from a PWM distribution serves as a practically motivated problem where we know the ground truth and can thus quantify the performance of different training methods in terms of the KL divergence between the trained generator distribution and the true distribution.

In our experiments, we generated 40,000 DNA sequences of six nucleotides according to a given position weight matrix. A random 10% of the sequences were held out as the validation set. Each sequence was then embedded into a 4 × 6 matrix by encoding each of the four nucleotides with an one-hot vector. On this dataset, we trained WGANs with different variants of OMD and SGD and evaluated their performance in terms of the KL divergence between the empirical distribution of the WGAN-generated samples and the true distribution described by the position weight matrix. Both the discriminator and generator of the WGAN used in this analysis were chosen to be convolutional neural networks (CNN), given the recent success of CNNs in modeling DNA-protein binding (Zeng et al., 2016; Alipanahi et al., 2015) . The detailed structure of the chosen CNNs can be found in Appendix E.

To account for the impact of learning rate and training epochs, we explored two different ways of model selection when comparing different optimization strategies: (1) using the iteration and learning rate that yields the lowest discriminator loss on the held out test set. This is inspired by the observation in Arjovsky et al. (2017) that the discriminator loss negatively correlates with the quality of the generated samples. (2) using the model obtained after the last epoch of the training. To account for the stochastic nature of the initialization and optimizers, we trained 50 independent models for each learning rate and optimizer, and compared the optimizer strategies by the resulting distribution of KL divergences across 50 runs.

For GD, we used variants of Equation ( 4) to examine the effect of using momentum and an adaptive step size. Specifically, we considered momentum, Nesterov momentum and Adagrad. The specific form of all these modifications is given for reference in Appendix A.

For OMD we used the general predictor version of Equation ( 10) with a fixed step size and with the following variants of the next iteration predictor M t+1 : (v1) Last iteration gradient: M t+1 = ∇f t , (v2) Running average of past gradients: M t+1 = 1 t t i=1 ∇f i , (v3) Hyperbolic discounted average of past gradients: M t+1 = λM t + (1 -λ)∇f t , λ ∈ (0, 1). We explored two training schemes: (1) training the discriminator 5 times for each generator training as suggest in Arjovsky et al. (2017) .

(2) training the discriminator once for each generator training. The latter is inline with the intuition behind the use of optimism: optimism hinges on the fact that the gradient at the next iteration is very predictable since it is coming from another regularized algorithm, and if we train the other algorithm multiple times, then the gradient is not that predictable and the benefits of optimism are lost. where we don't combine the models trained with different learning rates, the learning rate is appended at the end of the method name. For momentum and Nesterov momentum, we used γ = 0.9. For Adagrad, we used the default = 1e -8 . For all afore-described algorithms, we experimented with their stochastic variants. Figure 3 shows the KL divergence between the WGAN-generated samples and the true distribution. When evaluated by the epoch and learning rate that yields the lowest discriminator loss on the validation set, WGAN trained with Stochastic OMD (SOMD) achieves lower KL divergence than the competing SGD variants. Evaluated by the last epoch, the best performance across different learning rates is achieved by optimistic Adam (see Section 6). We note that in both metrics, SOMD with 1:1 generatordiscriminator training ratio yields better KL divergence than the alternative training scheme (1:5 ratio), which validates the intuition behind the use of optimism.

6. GENERATING IMAGES FROM CIFAR10 WITH OPTIMISTIC ADAM:

In this section we applying optimistic WGAN training to generating images, after training on CI-FAR10. Given the success of Adam on training image WGANs we will use an optimistic version of the Adam algorithm, rather than vanilla OMD. We denote the latter by Optimistic Adam. Optimistic Adam could be of independent interest even beyond training WGANs. We present Optimistic Adam for (G) but the analog is also used for training (D). We trained on CIFAR10 images with Algorithm 1 Optimistic ADAM, proposed algorithm for training WGANs on images.

Parameters: stepsize η, exponential decay rates for moment estimates β 1 , β 2 ∈ [0, 1), stochastic loss as a function of weights t (θ), initial parameters θ 0 for each iteration t ∈ {1, . . . , T } do Compute stochastic gradient: ∇ θ,t = ∇ θ t (θ) Update biased estimate of first moment:

m t = β 1 m t-1 + (1 -β 1 ) • ∇ θ,t

Update biased estimate of second moment:

v t = β 2 v t-1 + (1 -β 2 ) • ∇ 2 θ,t

Compute bias corrected first moment:

mt = m t /(1 -β t 1 ) Compute bias corrected second moment: vt = v t /(1 -β t 2 ) Perform optimistic gradient step: θ t = θ t-1 -2η • mt √ vt+ + η mt-1 √ vt-1+

Return θ T Optimistic Adam with the hyper-parameters matched to Gulrajani et al. (2017) , and we observe that it outperforms Adam in terms of inception score (see Figure 14 ), a standard metric of quality of WGANs (Gulrajani et al., 2017; Salimans et al., 2016) . In particular we see that optimistic Adam achieves high numbers of inception scores after very few epochs of training. We observe that for Optimistic Adam, training the discriminator once after one iteration of the generator training, which matches the intuition behind the use of optimism, outperforms the 1:5 generator-discriminator training scheme. We see that vanilla Adam performs poorly when the discriminator is trained only once in between iterations of the generator training. Moreover, even if we use vanilla Adam and train 5 times (D) in between a training of (G), as proposed by Arjovsky et al. (2017) , then performance is again worse than Optimistic Adam with a 1:1 ratio of training. The same learning rate 0.0001 and betas (β 1 = 0.5, β 2 = 0.9) as in Appendix B of Gulrajani et al. (2017) were used for all the methods compared. We also matched other hyper-parameters such as gradient penalty coefficient λ and batch size. For a larger sample of images see Appendix G. 

A VARIANTS OF GD TRAINING:

For ease of reference we briefly describe the exact form of update rules for several modifications of GD training that we have used in our experimental results.

Adagrad:

EQUATION

Momentum:

EQUATION

Nesterov momentum:

EQUATION

B PERSISTENCE OF LIMIT CYCLES IN GD TRAINING:

In Figure 5 we portray example Gradient Descent dynamics in the illustrative example described in Section 3 under multiple adaptations proposed in the literature. We observe that oscillations persist in all such modified GD dynamics, though alleviated by some. We briefly describe the modifications in detail first.

Gradient penalty. The Wasserstein GAN is based on the idea that the discriminator is approximating all 1-Lipschitz functions of the data. Hence, when training the discriminator we need to make sure that the function D w (x) has a bounded gradient with respect to x. One approach to achieving this is weight-clipping, i.e. clipping the weights to lie in some interval. However, the latter might introduce extra instability during training. Gulrajani et al. (2017) introduce an alternative approach by adding a penalty to the loss function of the zero-sum game that is essentially the 2 norm of the gradient of D w (x) with respect to x. In particular they propose the following regularized WGAN loss:

L λ (θ, w) = E x∼Q [D w (x)] -E z∼F [D w (G θ (z))] -λE x∼Q ( ∇ x D w (x) -1) 2

where Q is the distribution of the random vector x + (1 -)G(z) when x ∼ Q and z ∼ F . The expectations in the latter can also be replaced with sample estimates in stochastic variants of the training algorithms.

For our simple example, ∇ x D w (x) = w. Hence, we get the gradient penalty modified WGAN:

L λ (θ, w) = w, v -θ -λ ( w -1) 2 (20)

Hence, the gradient of the modified loss function with respect to θ remains unchanged, but the gradient with respect to w becomes:

EQUATION

Momentum. GD with momentum was defined in Equation ( 18). For the case of the simple illustrative example, these dynamics boil down to:

EQUATION

Nesterov momentum. GD with Nesterov's momentum was defined in Equation ( 19). For the illustrative example, we see that Nesterov's momentum is identical to momentum in the absence of gradient penalty. The reason being that the function is bi-linear. However, with a gradient penalty, Nesterov's momentum boils down to the following update rule.

EQUATION

Asymmetric training. Another approach to reducing cycling is to train the discriminator more frequently than the generator. Observe that if we could exactly solve the supremum problem of the discriminator after every iteration of the generator, then the generator would be simply solving a convex minimization problem and GD should converge point-wise. The latter approach could lead to slow convergence given the finiteness of samples in the case of stochastic training. Hence, we cannot really afford completely solving the discriminators problem. However, training the discriminator for multiple iterations, brings the problem faced by the generator closer to convex minimization rather than solving an equilibrium problem. Hence, asymmetric training could help with cycling. We observe below that asymmetric training is the most effective modification in reducing the range of the cycles and hence making the last-iterate be close to the equilibrium. However, it does not really eliminate the cycles, rather it simply makes their range smaller. 

C ANOTHER EXAMPLE: LEARNING A CO-VARIANCE MATRIX:

We demonstrate the benefits of using OMD over GD in another simple illustrative example. In this case, the example is does not boil down to a bi-linear game and therefore, the simulation results portray that the theoretical results we provided for bi-linear games, carry over qualitatively beyond the linear case.

Consider the case where the data distribution is a mean zero multi-variate normal with an unknown co-variance matrix, i.e., x ∼ N (0, Σ). We will consider the case where the discriminator is the set of all quadratic functions:

EQUATION

)

The generator is a linear function of the random input noise z ∼ N (0, I), of the form:

G V (z) = V z (25)

The parameters W and V are both d × d matrices. The WGAN game loss associated with these functions is then:

L(V, W ) = E x∼N (0,Σ) x T W x -E z∼N (0,I) z T V T W V z (26)

Expanding the latter we get:

L(V, W ) =E x∼N (0,Σ)   ij W ij x i x j   -E z∼N (0,I)   ij W ij k V ik z k m V jm z m   =E x∼N (0,Σ)   ij W ij x i x j   -E z∼N (0,I)   ijkm W ij V ik V jm z k z m   = ij W ij E x∼N (0,Σ) [x i x j ] - ijkm W ij V ik V jm E z∼N (0,I) [z k z m ] = ij W ij Σ ij - ijkm W ij V ik V jm 1{k = m} = ij W ij Σ ij - ijk W ij V ik V jk = ij W ij Σ ij - k V ik V jk

Given that the covariance matrix is symmetric positive definite, we can write it as Σ = U U T . Then the loss simplifies to:

EQUATION

The equilibrium of this game is for the generator to choose V ik = U ik for all i, k, and for the discriminator to pick W ij = 0. For instance, in the case of a single dimension we have

L(V, W ) = W • (σ 2 -V 2 )

, where σ 2 is the variance of the Gaussian. Hence, the equilibrium is for the generator to pick V = σ and the discriminator to pick W = 0.

Dynamics without sampling noise. For the mean GD dynamics the update rules are as follows:

W t ij =W t-1 ij + η Σ ij - k V t-1 ik V t-1 jk V t ij =V t-1 ij + η k W t-1 ik + W t-1 ki V t-1 kj (28)

We can write the latter updates in a simpler matrix form:

W t =W t-1 + η Σ -V t-1 V T t-1 V t =V t-1 + η(W t-1 + W T t-1 )V t-1

(GD for Covariance)

Similarly the OMD dynamics are:

W t =W t-1 + 2η Σ -V t-1 V T t-1 -η Σ -V t-2 V T t-2 V t =V t-1 + 2η(W t-1 + W T t-1 )V t-1 -η(W t-2 + W T t-2 )V t-2

(OMD for Covariance)

Due to the non-convexity of the generators problem and because there might be multiple optimal solutions (e.g. if Σ is not strictly positive definite), it is helpful in this setting to also help dynamics by adding 2 regularization to the loss of the game. The latter simply adds an extra 2λW t at each gradient term ∇ W L(V t , W t ) for the discriminator and a 2λV t at each gradient term ∇ V L(V t , W t ) for the generator. In Figures 7 and 6 we give the weights and the implied covariance matrix Σ G = V V T of the generator's distribution for each of the dynamics for an example setting of the step-size and regularization parameters and for two and three dimensional gaussians respectively. We again see how OMD can stabilize the dynamics to converge pointwise.

Stochastic dynamics.:

In Figure 8 and 9 we also portray the instability of GD and the robustness of the stability of OMD under stochastic dynamics. In the case of stochastic dynamics the gradients are replaced with unbiased estimates or with averages of unbiased estimates over a small minibatch.

In the case of a mini-batch of one, the unbiased estimates of the gradients in this setting take the following form:

∇W,t = x t x T t -V t z t z T t V T t ∇V,t = -(W t + W T t )V t z t z T t (Stochastic Gradients)

where x t , z t are samples drawn from the true distribution and from the random noise distribution respectively. Hence, the stochastic dynamics simply follow by replacing gradients with unbiased estimates: 

W t =W t-1 + η ∇W,t-1 V t =V t-1 -η ∇V,t-1 (SGD for Covariance) W t =W t-1 + 2η ∇W,t-1 -η ∇W,t-2 V t =V t-1 -2η ∇V,t-1 + η ∇V,t-2 (SOMD for

D LAST ITERATE CONVERGENCE OF OMD IN BILINEAR CASE:

The goal of this section is to show that Optimistic Mirror Descent exhibits last iterate convergence to min-max solutions for bilinear functions. In Section D.1, we provide the proof of Theorem 1, that OMD exhibits last iterate convergence to min-max solutions of the following min-max problem

EQUATION

where A is an abitrary matrix and x and y are unconstrained. In Section D.2, we state the appropriate extension of our theorem to the case:

EQUATION

D.1 PROOF OF THEOREM 1

As stated in Section 4, for the min-max problem (29) Optimistic Mirror Descent takes the following form, for all t ≥ 1:

EQUATION

EQUATION

where for the above iterations to be meaningful we need to specify x 0 , x -1 , y 0 , y -1 .

As stated in Section 4 we allow any initialization x 0 ∈ R(A), and y 0 ∈ R(A T ), and set x -1 = 2x 0 and y -1 = 2y 0 , where R(•) represents column space. In particular, our initialization means that the first step taken by the dynamics gives x 1 = x 0 and y 1 = y 0 .

Before giving our proof of Theorem 1, we need some further notation. For all i ∈ N, we set:

M i = A j (A T A) k , N i = A T j AA T k ∆ i t = ||N i Ay t || 2 2 + M i A T x t 2 2

where k ∈ Z and j ∈ {0, 1} are such that: i = 2k + j.

With this notation,

∆ 0 t = A T x t 2 2 + ||Ay t || 2 2 , ∆ 1 t = AA T x t 2 2 + A T Ay t 2 2 , ∆ 2 t = A T AA T x t 2 2 + AA T Ay t 2 2 , etc.

We also use the notation u, v X = u T XX T v, for vectors u, v ∈ R d and square d × d matrices X. We similarly define the norm notation ||u|| X = u, u X . Given our notation, we have the following claim, shown in Appendix D.3. Claim 1. For all matrices A and vectors u, v of the appropriate dimensions:

Au, Av AM T i = u, v A T N T i+1 ; A T u, A T v A T N T i = u, v AM T i+1 ; u, Av AM T i = v, A T u A T N T i .

With our notation in place, we show (through iterated expansion of the update rule), the following lemma, proved in Appendix D.3: Lemma 2. For the dynamics of Eq. ( 31) and (32) and any initialization 1 2 x -1 = x 0 ∈ R(A), and 1 2 y -1 = y 0 ∈ R(A T ) we have the following for all i, t ∈ N such that i ≥ 0 and t ≥ 2:

∆ i t -∆ i t-1 = 4η 2 ∆ i+1 t-1 -5η 2 ∆ i+1 t-2 -2η 3 x t-2 , Ay t-4 AM T i+1 -y t-2 , A T x t-4 A T N T i+1 .

We are ready to prove Theorem 1. Its proof is implied by the following stronger theorem, and Corollary 7. Theorem 3. Consider the dynamics of Eq. ( 31) and (32) and any initialization 1 2 x -1 = x 0 ∈ R(A), and

1 2 y -1 = y 0 ∈ R(A T ). Let γ = max AA T + , A T A + ,

where for a matrix X we denote by X + its generalized inverse and by ||X|| its spectral norm. Suppose that max{||A||, ||A T ||} ≡ λ ∞ ≤ 1 and η is a small enough constant satisfying η < 1/(3γ 2 ). Then, for all i ∈ N:

EQUATION

EQUATION

and, for all i, t ∈ N such that t ≥ 3, the following condition holds:

EQUATION

Proof. Eq. ( 33) holds trivially as under our initialization x 1 = x 0 and y 1 = y 0 . Eq. ( 34) is also easy to show by noticing the following. Given our initialization:

x 2 = x 0 -ηAy 0 y 2 = y 0 + ηAx 0 Hence (using j = i mod 2):

EQUATION

EQUATION

EQUATION

EQUATION

Similarly:

EQUATION

It follows from ( 40) and ( 41) that

∆ i 2 ≤ (1 + η) 2 ∆ i 0 .

We use induction on t to prove (35). We start our proof by showing the inductive step, and postpone establishing the basis of our induction to the end of this proof. For the inductive step, we assume that H(i, τ ) holds for all i ≥ 0 and 1 ≤ τ < t, for some t > 3. Assuming this, we show next that H(i, t) holds for all i. To do this, we make use of a few lemmas, whose proofs are given in Appendix D.3.

Lemma 4. Under the conditions of the theorem, for all i ≥ 0, t ≥ 2:

∆ i t -∆ i t-1 ≤ 4η 2 ∆ i+1 t-1 -5η 2 ∆ i+1 t-2 + η 3 (∆ i+1 t-2 + ∆ i+1 t-4 ).

Lemma 5. Under the conditions of the theorem, for all i, t ≥ 0: ∆ i+1 t ≤ ∆ i t .

Lemma 6. Under the conditions of the theorem, for all i ≥ 0, t ≥ 0:

∆ i+2 t ≥ 1 γ 2 ∆ i t .

Given these lemmas, we show our inductive step. So for t ≥ 4:

EQUATION

)

where for the first inequality we used Lemma 4, for the second inequality we used that ∆ i+1 t-1 ≤ ∆ i+1 t-2 + 16η 3 ∆ 0 0 (which is implied by the induction hypothesis), for the third inequality we used Lemma 6, for the fourth inequality we used Lemma 5, for the fifth inequality we applied the induction hypothesis iteratively, for the sixth inequality we used Eq. ( 34), for the seventh and eighth inequality we used that η is small enough, and for the last inequality we used Lemma 5. Hence:

∆ i t ≤ 1 - η 2 γ 2 ∆ i t-1 + 16η 3 ∆ 0 0 .

This completes the proof of our inductive step.

It remains to show the basis of the induction, namely that H(i, 3) holds for all i ∈ N. From Lemma 4 we have:

EQUATION

where for the second equality we used that 0.5x -1 = x 0 = x 1 and 0.5y -1 = y 0 = y 1 (which follow from our initialization), for the third inequality we used that (34), for the fourth inequality we used Lemma 5, for the fifth inequality we used Lemma 6, and for the last inequality we used Lemma 5. Hence, for small enough η, we have:

∆ i 3 ≤ 1 - η 2 γ 2 ∆ i 2 + 16η 3 ∆ 0 0 .

Corollary 7. Under the conditions of Theorem 3,

∆ 0 t ≡ A T x t 2 2 + ||Ay t || 2 2 → O(ηγ 2 ∆ 0 0 ) as t → +∞.

In particular, for large enough t, the last iterate of OMD is within O √ η • γ ∆ 0 0 distance from the space of equilibrium points of the game, where ∆ 0 0 is the distance of the initial point (x 0 , y 0 ) from the equilibrium space, and both distances are taken with respect to the norm x T AA T x + y T A T Ay.

Proof of Corollary 7: It follows from ( 33), ( 34) and ( 35) that:

∆ 0 t ≤ 1 - η 2 γ 2 t-2 (1 + η) 2 ∆ 0 0 + 16 ∞ t=0 1 - η 2 γ 2 t η 3 ∆ 0 0 = 1 - η 2 γ 2 t-2 (1 + η) 2 ∆ 0 0 + O ηγ 2 ∆ 0 0 ,

which shows the first part of our claim. For the second part of our claim recall that the solutions to ( 29) are all pairs (x, y) such that x is in the null space of A T and y is in the null space of A.

D.2 GENERAL BILINEAR CASE:

Theorem 8. Consider OMD for the min-max problem (30): inf

x sup y x T Ay + b T x + c T y + d .

Under the same conditions as Corollary 7 and whenever (30) is finite, OMD exhibits last iterate convergence in the same sense as in Corollary 7. In particular, for large enough t, the last iterate of OMD is within O √ η • γ ∆ 0 0 distance from the space of equilibrium points of the game, where √ ∆ 0 is the distance of the point (x 0 + (A T ) + c, y 0 + A + b) from the equilibrium space, and where both distances are taken with respect to the norm x T AA T x + y T A T Ay. Whenever (30) is infinite or undefined, the OMD dynamics travels to infinity and we characterize its motion.

Proof of Theorem 8: Trivially, we need only consider functions of the form x T Ay + b T x + c T y. We consider the following decompositions of b and c:

b = b 1 + b 2 where b 1 ∈ R(A), b 2 ∈ N (A T ) c = c 1 + c 2 where c 1 ∈ R(A T ), c 2 ∈ N (A)

Given the above we can also define b 3 and c 3 as follows:

Ac 3 = b 1 feasible since b 1 ∈ R(A) A T b 3 = c 1 feasible since c 1 ∈ R(A T )

Then, we can make the following variable substition:

α t = x t + ηtb 2 + b 3 β t = y t -ηtc 2 + c 3 so that: A T α t = A T x t + ηtA T b 2 + A T b 3 = A T x t + c 1 since b 2 ∈ N (A T ) Aβ t = Ay t -ηtAc 2 + Ac 3 = Ay t + b 1 since c 2 ∈ N (A)

We also state the OMD dynamics for x t and y t for problem (30):

x t = x t-1 -2η(Ay t-1 + b) + η(Ay t-2 + b) = x t-1 -2ηAy t-1 + ηAy t-2 -ηb y t = y t-1 + 2η(A T x t-1 + c) -η(A T x t-2 + c) = y t-1 + 2ηA T x t-1 -ηA T x t-2 + ηc

Note that given this update step:

x t+1 = x t -2ηAy t + ηAy t-1 -ηb x t+1 = x t -ηb 2 -2ηAy t + ηAy t-1 -ηAc 3 x t+1 = x t -ηb 2 -2ηA(y t + c 3 ) + ηA(y t-1 + c 3 ) x t+1 = x t -ηb 2 -2ηA(y t -ηc 2 t + c 3 ) + ηA(y t-1 -ηc 2 (t -1) + c 3 ) x t+1 + ηb 2 (t + 1) = x t + 2 t -2ηA(y t -ηc 2 t + c 3 ) + ηA(y t-1 -ηc 2 (t -1) + c 3 ) x t+1 + ηb 2 (t + 1) + b 3 = x t + ηb 2 t + b 3 -2ηA(y t -ηc 2 t + c 3 ) + ηA(y t-1 -ηc 2 (t -1) + c 3 ) α t+1 = α t -2ηAβ t + ηAβ t-1 Analogously: β t+1 = β t + 2ηA T α t -ηA T α t-1

Note that these are precisely the dynamics for which we proved convergence in Theorem 1. Thus, by invoking Theorem 3 and Corollary 7 on the sequence (α t , β t ) and then substituting back (x t , y t ), we have that for all large enough t:

x

t = -ηb 2 t -b 3 + x (t) y t = ηc 2 t -c 3 + y (t) such that ||A T x (t)|| 2 , ||A y (t)|| 2 ∈ O √ η • γ ∆ 0 0 , where ∆ 0 0 = ||A T (x 0 + b 3 )|| 2 2 + ||A(y 0 + c3)|| 2 2 .

In particular, this shows that, whenever (30) is finite (i.e. b 2 = c 2 = 0), OMD exhibits last iterate convergence. For large enough t, the last iterate of OMD is within O √ η • γ ∆ 0 0 distance from the space of equilibrium points of the game, where ∆ 0 0 is the distance of (x 0 + b 3 , y 0 + c 3 ) from the equilibrium space in the norm x T AA T x + y T A T Ay. Whenever ( 30) is infinite or undefined, the OMD dynamics travels to infinity linearly, with fluctuations around the divergence specified as above.

D.3 OMITTED PROOFS:

Proof of Proposition 1: To show this, we consider the 2 distance of the solution at time t. First, recall the GD update step in the special case of f (x, y) = x T y:

x

t = x t-1 -ηy t-1 y t = y t-1 + ηx t-1

Then, note that the squared 2 distance of the running iterate (x t , y t ) to the unique equilibrium solution (0, 0) is given by d(t) := ||x t || 2 2 + ||y t || 2 2 , which we can calculate:

||x t || 2 2 = ||x t-1 || 2 2 -2ηx T t-1 y t-1 + η 2 ||y t-1 || 2 2 ||y t-1 || 2 2 = ||y t-1 || 2 2 + 2ηy T t-1 x t-1 + η 2 ||x t-1 || 2 2 ∴ d(t) = d(t -1) + η 2 d(t -1) = (1 + η 2 )d(t -1)

This indicates that for any value of η > 0, the running iterate of GD diverges from the equilibrium.

Proof of Claim 1: For our first claim, observe that:

Au, Av AM T i = u T A T AM T i M i A T Av = u T A T A(A T A) k (A j ) T A j (A T A) k A T Av = u T A T (AA T ) k A(A j ) T A j A T (AA T ) k Av = u T A T [(AA T ) k A(A j ) T ][A j A T (AA T ) k ]Av = u T A T N T i+1 N i+1 Av = u, v A T N T i+1 Our second claim, A T u, A T v A T N T i = u, v AM T i+1

, is proven analogously. For our third claim:

u, Av AM T i = u T AM T i M i A T Av = u T A(A T A) k (A T A) j (A T A) k A T Av if j = 0: = u T A(A T A) k (A T A) k A T Av = u T A[A T (AA T ) k ][(AA T ) k A]v = u A[A T N T i ][N i A]v = v, A T u A T N T i otherwise: = u T A(A T A) k A T A(A T A) k A T Av = u T A[(A T A) k A T A][(A T A) k A T A]v = u T A[A T (AA T ) k A][A T (AA T ) k A]v = u T A[A T N T i ][N i A]v = v, A T u A T N T i

Proof of Lemma 2: First, we note the following scaled update rule:

M i A T x t = M i A T x t-1 -2ηA T Ay t-1 + ηA T Ay t-2 N i Ay t = N i Ay t-1 + 2ηAA T x t-1 -ηAA T x t-2

Then, taking the norm of both sides, and using the statements of Claim 1: 

||x t || 2 AM T i = ||x t-1 || 2 AM T i + 4η 2 ||y t-1 || 2 A T N T i+1 + η 2 ||y t-2 || 2 A T N T i+1 -4η x t-

+ ||y t || 2 A T N T i = ∆ i t-1 + 4η 2 ∆ i+1 t-1 + η 2 ∆ i+1 t-2 + 2η( x t-1 , Ay t-2 AM T i -y t-1 , Ax t-2 A T N T i ) -4η 2 ( x t-1 , x t-2 AM T i+1 + y t-1 , y t-2 A T N T i+1 )

Expanding the first pair of inner products above and using Claim 1 again:

x t-1 , Ay t-2 AM T i -y t-1 , A T x t-2 A T N T i = x t-2 -2ηAy t-2 + ηAy t-3 , Ay t-2 AM T i -y t-2 + 2ηA T x t-2 -ηA T x t-3 , A T x t-2 A T N T ) + η( x t-2 , x t-3 AM T i+1 + y t-2 , y t-3 A T N T i+1 )

Then, multiplying by 2η and substituting into the previous derivation yields:

∆ i t -∆ i t-1 = 4η 2 ∆ i+1

t-1 -3η 2 ∆ i+1 t-2 + 2η 2 ( x t-2 , x t-3 AM T i+1 + y t-2 , y t-3 A T N T i+1 ) -4η 2 ( x t-1 , x t-2 AM T i+1 + y t-1 , y t-2 A T N T i+1 )

Now, consider the following inner product:

x t-2 , x t-1 AM T i+1 + y t-2 , y t-1 A T N T i+1 = x t-2 , x t-2 -2ηAy t-2 + ηAy t-3 AM T i+1 + y t-2 , y t-2 + 2ηA T x t-2 -ηA T x t-3 A T N T i+1 = ∆ i+1 t-2 + η x t-2 , Ay t-3 AM T i+1 -y t-2 , A T x t-3 A T N T i+1

Once again, we multiply by -4η 2 and substitute:

∆ i t -∆ i t-1 = 4η 2 ∆ i+1

t-1 7η 2 ∆ i+1 t-2 + 2η 2 ( x t-2 , x t-3 AM T i+1 + y t-2 , y t-3 A T N T i+1 ) + 4η 3 ( y t-2 , A T x t-3 A T N T i+1 -x t-2 , Ay t-3 AM T i+1 )

Now, we use the update step for time t -2. For all t ≥ 1, this is well-defined, since x -1 and y -1 are defined. To ensure that this step is sound for t = 0 requires we define the following, where X + denotes the generalized inverse:

x -2 = 4x 0 + 1 η (A T ) + y 0 y -2 = 4y 0 -

1 η A + x 0

We define these such that: A T x -2 = 4A T x 0 + y0 η and Ay -2 = 4Ay 0 -x0 η (since x 0 ∈ R(A) and y 0 ∈ R(A T ), and thus the following equalities hold:

x 0 = x -1 -2ηAy -1 + ηAy -2 y 0 = y -1 + 2ηA T x -1 -ηA T x -2

This allows us to use the following expansion freely for all t ≥ 2:

x t-2 = x t-3 -2ηAy t-3 + ηAy t-4 =⇒ x t-3 -2ηAy t-3 = x t-2 -ηAy t-4 y t-2 = y t-3 + 2ηA T x t-3 -ηA T x t-4 =⇒ y t-3 + 2ηA T x t-3 = y y-2 + ηA T x t-4

We can gather the inner product terms and use this update rule to get our final desired result:

∆ i t -∆ i t-1 = 4η 2 ∆ i+1

t-1 -7η 2 ∆ i+1 t-2 + 2η 2 ( x t-2 , x t-3 -2ηAy t-3 AM T i+1 + y t-2 , y t-3 + 2ηA T x t-3 A T N T i+1 ) = 4η 2 ∆ i+1 t-1 -5η 2 ∆ i+1 t-2 -2η 3 ( x t-2 , Ay t-4 AM T i+1 -y t-2 , A T x t-4 A T N T i+1 )

Proof of Lemma 4: To prove this, first consider the following trivial inequality: 

y t-2 -A T x t-4 2 A T N T i+1 + ||x t-2 + Ay t-4 || 2 AM T i+1 = ||y t-2 || 2 A T N T i+1 -2 y t-2 , A T x t-4 A T N T i+1 + A T x t-4 2 A T N T i+1 + ||x t-

:

Code for our models is available at https://github.com/vsyrgkanis/optimistic_GAN_ training * These authors contribute equally to this work.

PRELIMINARIES: WGANS AND OPTIMISTIC MIRROR DESCENTWe consider the problem of learning a generative model of a distribution of data points Q ∈ ∆(X). Our goal is given a set of samples from D, to learn an approximation to the distribution Q in the form of a deep neural network G θ (•), with weight parameters θ, that takes as input random noise

The latter is a consequence of the be-the-leader lemmaKalai & Vempala (2005);Rigollet (2015)

Figure 13: Sample of images from Generator of Epoch 19 trained via Adam and 5:1 training ratio.

Figures and Tables:

Figure1: Training GAN with GD converges to a limit cycle that oscilates around the equilibrium (we applied weight-clipping at 10 for the discriminator). On the contrary training with OMD converges to equilibrium in terms of last-iterate convergence.

Figure 2: Robustness of last-iterate convergence of OMD to stochastic gradients.

Figure 3: KL divergence of WGAN trained with different optimization strategies. Methods are ordered by the median KL divergence. Methods in (a) are named by the category and version of the method. "ratio 1" denotes training the discriminator once for every generator training. Otherwise, we performed 5 iterations. For (b)where we don't combine the models trained with different learning rates, the learning rate is appended at the end of the method name. For momentum and Nesterov momentum, we used γ = 0.9. For Adagrad, we used the default = 1e -8 .

Inception score on CIFAR10, when training with Adam and Optimistic Adam. "ratio1" means we performed 1 iteration of training of (D) in between 1 iteration of (G). Otherwise we performed 5 iterations. We further test (averaging over 35 trials) the two top-performing optimizers, Adam (ratio 5) and Optimistic Adam with ratio 1, in Appendix H.(b) Sample of images from Generator of Epoch 94, which had the highest inception score.

Figure 4: Comparison of Adam and Optimistic Adam on CIFAR10.

(a) GD dynamics with a gradient penalty added to the loss. η = 0.1 and λ = 0.1. (b) GD dynamics with momentum. η = 0.1 and γ = 0.5. (c) GD dynamics with momentum and gradient penalty. η = .1, γ = 0.2 and λ = 0.1. (d) GD dynamics with momentum and gradient penalty, training generator every 15 training iterations of the discriminator. η = .1, γ = 0.2 and λ = 0.1. (e) GD dynamics with Nesterov momentum and gradient penalty, training generator every 15 training iterations of the discriminator. η = .1, γ = 0.2 and λ = 0.1.

Figure 5: Persistence of limit cycles in multiple variants of GD training.

Covariance) (a) GD dynamics. η = 0.1, T = 500, λ = 0.3. (b) OMD dynamics. η = 0.1, T = 500, λ = 0.3.

Figure 6: Stability of OMD vs GD in the co-variance learning problem for a two-dimensional gaussian (d = 2). Weight clipping in [-1, 1] was applied in both dynamics.

Figure 7: Stability of OMD vs GD in the co-variance learning problem for a three-dimensional gaussian (d = 3). Weight clipping in [-1, 1] was applied in both dynamics.

Figure 8: Stochastic GD dynamics for covariance learning of a two-dimensional gaussian (d = 2). Weight clipping in [-1, 1] was applied to the discriminator weights.

Figure 9: Stability of OMD with stochastic gradients for covariance learning of a two-dimensional gaussian (d = 2). Weight clipping in [-1, 1] was applied to the discriminator weights.

(a) Sample of images from Generator of Epoch 94, which had the highest inception score.

Figure 10: Samples of images from Generator trained via Optimistic Adam on CIFAR10.

Figure 11: Sample of images from Generator of Epoch 19 trained via Optimistic Adam and 1:1 training ratio.

Figure 12: Sample of images from Generator of Epoch 19 trained via Adam and 1:1 training ratio.

Figure14: The inception scores across epochs for GANs trained with Optimistic Adam (ratio 1) and Adam (ratio 5) on CIFAR10 (the two top-performing optimizers found in Section 6, with 10%-90% confidence intervals. The GANs were trained for 30 epochs and results gathered across 35 runs.

1 , Ay t-1 AM T i + 2η x t-1 , Ay t-2 AM T i -4η 2 y t-1 , y t-2 A T N T , A T x t-1 A T N T i + 2η y t-1 , A T x t-2 A T N T i -4η 2 x t-1 , x t-2 AM T

Table content: <table><tr><td/><td/><td/><td>i+1</td></tr><tr><td>||y t || 2 A T N T i</td><td>= ||y t-1 || 2 A T N T i</td><td>+ 4η 2 ||x t-1 || 2 AM T i+1</td><td>+ η 2 ||x t-2 || 2 AM T i+1 + 4η y t-1 i+1</td></tr><tr><td colspan="2">∴ ∆ i t = ||x t || 2 AM T i</td><td/><td/></tr></table>

2 || , A T x t-4 A T N T i+1 -2 x t-2 , Ay t-4 AM T i+1 ≤ ∆ i+1 t-2 + A T x t-4

Table content: <table><tr><td colspan="2">G CIFAR10 GENERATOR IMAGE SAMPLES</td></tr><tr><td>2 AM T i+1</td><td colspan="2">+ 2 x t-2 , Ay t-4 AM T i+1 + ||Ay t-4 || 2 AM T i+1</td></tr><tr><td>≥ 0</td><td/></tr><tr><td>Rearranging:</td><td/></tr><tr><td colspan="3">2 y t-2 2 A T N T i+1</td><td>2 + ||Ay t-4 || AM T i+1</td></tr><tr><td colspan="2">≤ ∆ i+1 t-2 + λ 2 ∞ ||x t-4 || 2 AM T i+1</td><td>2 + ||y t-4 || A T N T i+1</td></tr><tr><td colspan="2">≤ ∆ i+1 t-2 + λ 2 ∞ ∆ i+1 t-4</td></tr><tr><td colspan="2">≤ ∆ i+1 t-2 + ∆ i+1 t-4</td></tr></table>

Peer review (iclr2018):
 This paper proposes the use of optimistic mirror descent to train Wasserstein Generative Adversarial Networks (WGANS). The authors remark that the current training of GANs, which amounts to solving a zero-sum game between a generator and discriminator, is often unstable, and they argue that one source of instability is due to limit cycles, which can occur for FTRL-based algorithms even in convex-concave zero-sum games. Motivated by recent results that use Optimistic Mirror Descent (OMD) to achieve faster convergence rates (than standard gradient descent) in convex-concave zero-sum games and normal form games, they suggest using these techniques for WGAN training as well. The authors prove that, using OMD, the last iterate converges to an equilibrium and use this as motivation that OMD methods should be more stable for WGAN training. They then compare OMD against GD on both toy simulations and a DNA sequence task before finally introducing an adaptive generalization of OMD, Optimistic Adam, that they test on CIFAR10. This paper is relatively well-written and clear, and the authors do a good job of introducing the problem of GAN training instability as well as the OMD algorithm, in particular highlighting its differences with standard gradient descent as well as discussing existing work that has applied it to zero-sum games. Given the recent work on OMD for zero-sum and normal form games, it is natural to study its effectiveness in training GANs.The issue of last iterate versus average iterate for non convex-concave problems is also presented well. The theoretical result on last-iterate convergence of OMD for bilinear games is interesting, but somewhat wanting as it does not provide an explicit convergence rate as in Rakhlin and Sridharan, 2013. Moreover, the result is only at best a motivation for using OMD in WGAN training since the WGAN optimization problem is not a bilinear game. The experimental results seem to indicate that OMD is at least roughly competitive with GD-based methods, although they seem less compelling than the prior discussion in the paper would suggest. In particular, they are matched by SGD with momentum when evaluated by last epoch performance (albeit while being less sensitive to learning rates). OMD does seem to outperform SGD-based methods when using the lowest discriminator loss, but there doesn*t seem to be even an attempt at explaining this in the paper. I found it a bit odd that Adam was not used as a point of comparison in Section 5, that optimistic Adam was only introduced and tested for CIFAR but not for the DNA sequence problem, and that the discriminator was trained for 5 iterations in Section 5 but only once in Section 6, despite the fact that the reasoning provided in Section 6 seems like it would have also applied for Section 5. This gives the impression that the experimental results might have been at least slightly "gamed". For the reasons above, I give the paper high marks on clarity, and slightly above average marks on originality, significance, and quality. Specific comments: Page 1, "no-regret dynamics in zero-sum games can very often lead to limit cycles": I don*t think limit cycles are actually ever formally defined in the entire paper. Page 3, "standard results in game theory and no-regret learning": These results should be either proven or cited. Page 3: Don*t the parameter spaces need to be bounded for these convergence results to hold? Page 4, "it is well known that GD is equivalent to the Follow-the-Regularized-Leader algorithm": For completeness, this should probably either be (quickly) proven or a reference should be provided. Page 5, "the unique equilibrium of the above game is...for the discriminator to choose w=0": Why is w=0 necessary here? Page 6, "We remark that the set of equilibrium solutions of this minimax problem are pairs (x,y) such that x is in the null space of A^T and y is in the null space of A": Why is this true? This should either be proven or cited. Page 6, Initialization and Theorem 1: It would be good to discuss the necessity of this particular choice of initialization for the theoretical result. In the Initialization section, it appears simply to be out of convenience. Page 6, Theorem 1: It should be explicitly stated that this result doesn*t provide a convergence rate, in contrast to the existing OMD results cited in the paper. Page 7, "we considered momentum, Nesterov momentum and AdaGrad": Why isn*t Adam used in this section if it is used in later experiments? Page 7-8, "When evaluated by....the lowest discriminator loss on the validation set, WGAN trained with Stochastic OMD (SOMD) achieved significantly lower KL divergence than the competing SGD variants.": Can you explain why SOMD outperforms the other methods when using the lowest discriminator loss on the validation set? None of the theoretical arguments presented earlier in the paper seem to even hint at this. The only result that one might expect from the earlier discussion and results is that SOMD would outperform the other methods when evaluating by the last epoch. However, this doesn*t even really hold, since there exist learning rates in which SGD with momentum matches the performance of SOMD. Page 8, "Evaluated by the last epoch, SOMD is much less sensitive to the choice of learning rate than the SGD variants": Learning rate sensitivity doesn*t seem to be touched upon in the earlier discussion. Can these results be explained by theory? Page 8, "we see that optimistic Adam achieves high numbers of inception scores after very few epochs of training": These results don*t mean much without error bars. Page 8, "we only trained the discriminator once after one iteration of generator training. The latter is inline with the intuition behind the use of optimism....": Why didn*t this logic apply to the previous section on DNA sequences, where the discriminator was trained multiple times? After reading the response of the authors (in particular their clarification of some technical results and the extra experiments they carried out during the rebuttal period), I have decided to upgrade my rating of the paper from a 6 to a 7. Just as a note, Figure 3b is now very difficult to read.