Title: THE UNUSUAL EFFECTIVENESS OF AVERAGING IN GAN TRAINING

Abstract: We examine two different techniques for parameter averaging in GAN training. Moving Average (MA) computes the time-average of parameters, whereas Exponential Moving Average (EMA) computes an exponentially discounted sum. Whilst MA is known to lead to convergence in bilinear settings, we provide theto our knowledge -first theoretical arguments in support of EMA. We show that EMA converges to limit cycles around the equilibrium with vanishing amplitude as the discount parameter approaches one for simple bilinear games and also enhances the stability of general GAN training. We establish experimentally that both techniques are strikingly effective in the non-convex-concave GAN setting as well. Both improve inception and FID scores on different architectures and for different GAN objectives. We provide comprehensive experimental results across a range of datasets -mixture of Gaussians, CIFAR-10, STL-10, CelebA and ImageNet -to demonstrate its effectiveness. We achieve state-of-the-art results on CIFAR-10 and produce clean CelebA face images.

1. INTRODUCTION:

Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are two-player zero-sum games. They are known to be notoriously hard to train, unstable, and often don't converge. There has been a lot of recent work to improve the stability of GANs by addressing various underlying issues. Searching for a more stable function family (Radford et al., 2015) and utilizing better objectives (Arjovsky et al., 2017; Gulrajani et al., 2017) have been explored for training and instability issues. For tackling non-convergence, regularizing objective functions (Mescheder et al., 2017; Salimans et al., 2016) , two time-scale techniques (Heusel et al., 2017) and extra-gradient (optimistic) methods (Daskalakis et al., 2018; Mertikopoulos et al., 2019) have been studied.

One reason for non-convergence is cycling around an optimal solution (Mertikopoulos et al., 2018; Papadimitriou & Piliouras, 2018) , or even slow outward spiraling (Bailey & Piliouras, 2018) . There might be multiple reasons for this cycling behavior: (i) eigenvalues of the Jacobian of the gradient vector field have zero real part and a large imaginary part (Mescheder et al., 2017) . (ii) Gradients of the discriminator may not move the optimization in the right direction and may need to be updated many times to do so (Arjovsky et al., 2017) . (iii) Sampling noise causes the discriminator to move in different directions at each iteration, as a result of which the generator fluctuates. (iv) Even if the gradients of the discriminator point in the right direction, the learning rate of the generator might be too large. When this overshooting happens, the gradients of the discriminator lead in the wrong direction due to over-confidence (Mescheder et al., 2018) .

In this work, we explore in detail simple strategies for tackling the cycling behavior without influencing the adversarial game. Our strategies average generator parameters over time outside the training loop. Averaging generator and discriminator parameters is known to be an optimal solution for convex-concave min-max games (Freund & Schapire, 1999) . However, no such guarantees are known (even for bilinear games) if we apply exponential discounting.

Our contributions are the following: (i) We show theoretically that although EMA does not converge to equilibrium, even in bilinear games, it nevertheless helps to stabilize cyclic behavior by shrinking its amplitude. In non-bilinear settings it preserves the stability of locally stable fixed points. (ii) We demonstrate that both averaging techniques consistently improve results for several different datasets, network architectures, and GAN objectives. (i) We compare it with several other methods that try to alleviate the cycling or non-convergence problem and demonstrate its unusual effectiveness.

2. RELATED WORK:

There have been attempts to improve stability and alleviate cycling issues arising in GANs. Salimans et al. (2016) use historical averaging of both generator and discriminator parameters as a regularization term in the objective function: each model deviating from its time average is penalized to improve stability. Ge et al. (2018) update the discriminator by using a mixture of historical generators. Heusel et al. (2017) use two time scale update rules to converge to a local equilibrium. Mescheder et al. (2017) identify the presence of eigenvalues of the Jacobian of the gradient vector with zero real part and large imaginary part as a reason for non-convergence. They address this issue by including squared gradients of the players to the objective. Extra-gradient (optimistic) methods help with convergence (Daskalakis et al., 2018; Mertikopoulos et al., 2019) . Some of the above stated methods such as Salimans et al. (2016) ; Mescheder et al. (2017) influence the min-max game by introducing regularization terms into the objective function. Regularizing the adversarial objective can potentially change the optimal solution of the original objective. Our method on the other hand does not change the game dynamics, by keeping an average over iterates outside of the training loop.

Averaging at maximum likelihood objective: Uniform Average and Exponential Moving Average over model parameters has been used for a long time (Polyak & Juditsky, 1992; van den Oord et al., 2017; Athiwaratkun et al., 2018; Izmailov et al., 2018) . However we would like to emphasize the difference between convex/non-convex optimization and saddle point optimization. GANs are an instance of the latter approach. There is no formal connection between the two approaches, so we cannot extrapolate our understanding of averaging at maximum likelihood objectives to saddle point objectives.

Averaging at minimax objective: Gidel et al. (2018) used uniform averaging similar to our work, however we show its shortcomings and how exponential decay averaging performs better. While this strategy has been used in (Karras et al., 2017) recently, their paper lacks any insights or a detailed analysis of this approach.

3. METHOD:

In case of convex-concave min-max games, it is known that the average of generator/discriminator parameters is an optimal solution (Freund & Schapire, 1999) , but there is no such guarantee for a non-convex/concave setting. Moving Average (MA) over parameters θ is an efficient implementation of uniform averaging without saving iterates at each time point:

EQUATION

When the generator iterates reaches a stationary distribution, Eq.1 approximates its mean. However there are two practical difficulties with this equation. First we do not know at which time point the iterates reach a stationary distribution. Second, and more importantly, iterates may not stay in the same distribution after a while. Averaging over samples coming from different distributions can produce worse results, which we have also observed in our own experiments.

Because of the issues stated above, we use Exponential Moving Average (EMA):

EQUATION

where θ (0)

EM A = θ (0) . EMA has an effective time window over iterates, where early iterates fade at a rate depending on the β value. As β approaches 1, averaging effectively computes longer time windows.

Both EMA and MA offer performance gains, with EMA typically being more robust (for more details see Section 5). Averaging methods are simple to implement and have minimal computation overhead. As they operate outside of the training loop, they do not influence optimal points of the game. Hyperparameter search for EMA can be done with a single training run by simultaneously keeping track of multiple averages with different β parameters.

4. WHY DOES AVERAGING WORK?:

4.1 GLOBAL ANALYSIS FOR SIMPLE BILINEAR GAMES As we will show experimentally, parameter averaging provides significant benefits in terms of GAN training across a wide range of different setups. Although one cannot hope to completely justify this experimental success theoretically in general non-convex-concave settings, it is at least tempting to venture some informed conjectures about the root causes of this phenomenon. To do so, we focus on the simplest possible class of saddle point problems, the class of planar bilinear problems (i.e. zero-sum games such as Matching Pennies). This is a toy, albeit classic, pedagogical example used to explain phenomena arising in GANs. Specifically, similar to Goodfellow (2017) , we also move to a continuous time model to simplify the mathematical analysis.

In terms of the moving average method, it is well known that time-averaging suffices to lead to convergence to equilibria (Freund & Schapire, 1999) . In fact, it has recently been established that the smooth (continuous-time) analogues of first order methods such as online gradient descent (follow-the-regularized leader) in bilinear zero-sum games are recurrent (i.e. effectively periodic) with trajectories cycling back into themselves (Mertikopoulos et al., 2018) . As a result, the timeaverage of the system converges fast, at a O(1/T ) rate, to the solution of the saddle point problem (Mertikopoulos et al., 2018) . These systems have connections to physics (Bailey & Piliouras, 2019) .

In contrast, not much is known about the behavior of EMA methods even in the simplest case of zero-sum games. As we will show by analyzing the simplest case of a bilinear saddle problem max x min y xy, even in this toy case EMA does not lead to convergent behavior, but instead reduces the size of the oscillations, leading to more stable behavior whilst enabling exploration in a close neighborhood around the equilibrium. In this case the gradient descent dynamics are as follows:

dx dt = y dy dt = -x ⇔ dx dt = ∂H ∂y dy dt = -∂H ∂x

This corresponds to a standard Hamiltonian system whose energy function is equal to H = x 2 +y 2 2 . All trajectories are cycles centered at 0. In fact, the solution to the system has the form of a periodic orbit (x(t), y(t)) = (a cos(t), a sin(t)). These connections have recently been exploited by Balduzzi et al. (2018) to design training algorithms for GANs (see also Bailey & Piliouras (2019) ). Here we take a different approach. For simplicity let's assume a = 1. The Exponential Moving Average method now corresponds to an integral of the form C • T 0 β T -t θ(t)dt where θ(t) = (cos(t), sin(t)) and C a normalizing term such that the EMA of a constant function is equal to that constant (i.e.

C = 1/ T 0 β T -t dt). Some simple algebra implies that C • T 0 β T -t sin(t)dt = -ln(β) 1+ln 2 (β) β T -cos(T )-ln(β) sin(T ) 1-β T

. As the terms β T vanish exponentially fast, this function reduces to ln(β) 1+ln 2 (β) (cos(T ) + ln(β) sin(T )). This is a periodic function, similarly to the components of θ(t), but its amplitude is significantly reduced, and as a result it is more concentrated around its mean value (which is the same as its corresponding θ(t) component, i.e. zero). For β very close to 1, e.g. β = 0.999 this periodic function can be further approximated by ln(β) cos(T ). In other words, no matter how close to 1 we choose the discounting factor β, even in the toy case of a planar bilinear saddle problem (i.e. the Matching Pennies game) EMA has a residual non-vanishing periodic component, alas of very small amplitude around the solution of the saddle point problem. 

4.2. LOCAL STABILITY ANALYSIS FOR GANS:

The model of the previous section introduced two simplifications so as to be theoretically tractable: i) it focused on a (small) bilinear game, and ii) it was a continuous time model. Despite these simplifications the global behavior was still rather complex, indicating the difficulty of completely analyzing a more realistic model. Nevertheless, in this section we examine such a model (discretetime non-convex loss functions) and show that EMA preserves the stability of locally stable fixed points (Nash equilibria), providing further theoretical evidence in support of averaging. Due to the complexity of GANs, local stability analysis is often used as a method for obtaining theoretical insights, see e.g. Nagarajan & Kolter (2017); Mescheder et al. (2018) .

Suppose we have a generator G parameterized by θ ∈ R p , and a discriminator D parameterized by φ ∈ R q ; let x = (θ, φ) ∈ R n , n = p + q. Then, during training, G and D minimize loss functions L (θ) (θ, φ) and L (φ) (θ, φ) respectively. Following the same formulation as in Gidel et al. (2018) :

θ * ∈ argmin θ L (θ) (θ, φ * ) φ * ∈ argmin φ L (φ) (θ * , φ)

For specific possible instantiations of these loss functions see Goodfellow et al. (2014) ; Arjovsky et al. (2017) . We define the gradient vector field V (θ, φ) ∈ R n and its Jacobian J V (θ, φ) ∈ R n×n as follows:

V (θ, φ) = ∇ θ L (θ) (θ, φ) ∇ φ L (φ) (θ, φ) J V (θ, φ) = ∇ 2 θ L (θ) (θ, φ) ∇ φ ∇ θ L (θ) (θ, φ) ∇ θ ∇ φ L (φ) (θ, φ) ∇ 2 φ L (φ) (θ, φ)

.

In this analysis, we assume that G and D are trained with some method that can be expressed as the iteration x t = F (x t-1 )

(3) for some function F , where x t = (θ t , φ t ) denotes the parameters obtained after t training steps. This framework covers all training algorithms that rely only on a single past iterate to compute the next iterate, and includes simultaneous gradient descent and alternating gradient descent methods. More concretely, simultaneous gradient descent with a learning rate η can be written in this form as follows

F η (x) = x -ηV (x).

(4)

EMA then averages the iterates x t to produce a new sequence of iterates xt such that

xt = β xt-1 + (1 -β)x t (by definition (2)) = β xt-1 + (1 -β)F (x t-1 )

(Replace x t using Eq. ( 3))

= β xt-1 + (1 -β)F xt-1 -β xt-2 1 -β (Replace x t-1 using definition (2))

We have thus expressed the EMA algorithm purely in terms of a new set of iterates xt . Note that computing the next iterate now requires the past two iterates. Flammarion & Bach (2015) observed that the standard moving average can similarly be expressed as a second-order difference equation. We can then define the EMA operator Fβ (x t , x t-1 ) such that

[ xt xt-1 ] = Fβ (x t-1 , x t-2 ) Fβ (x t , x t-1 ) = βI n 0 n I n 0 n x t x t-1 + (1 -β) F (A[x t x t-1 ] T ) 0 n , for A = 1 1 -β [I n -βI n ].

The Jacobian of Fβ , J Fβ is therefore

J Fβ (x t , x t-1 ) = βI n 0 n I n 0 n + (1 -β) J F (A[x t x t-1 ] T ) • A 0 n = βI n + J F (A[x t x t-1 ] T ) -βJ F (A[x t x t-1 ] T ) I n 0 n

We can now proceed to explore the behavior of the EMA algorithm near a local Nash equilibrium by examining the eigenvalues of its Jacobian J Fβ , which are described by Theorem 1.

Theorem 1. The eigenvalues of the Jacobian J Fβ (x * , x * ) at a local Nash equilibrium x * are β (with multiplicity n) and the eigenvalues of J F (x * ), the Jacobian of the training operator at the equilibrium.

Proof. We solve det(γI 2n -J Fβ ) = 0 to obtain eigenvalues γ. Some algebra reveals that

det(γI 2n -J Fβ ) = (γ -β)I n -J F (x * ) βJ F (x * ) -I n γI n = |γ(γ -β)I n -γJ F (x * ) + βJ F (x * )| = |(γ -β)(γI n -J F (x * ))| = (γ -β) n det(γI n -J F (x * ))

where we used the following equality (e.g. see Lemma 1 in Gidel et al. ( 2018)):

| A B C D | = |AD -BC| if C and D commute. Thus, det(γI 2n -J Fβ ) = 0 if γ = β or det(γI n -J F (x * )) = 0 so γ is an eigenvalue of J F (x * ).

Given that 0 < β < 1 in the EMA algorithm, we have the following corollary that formalizes the intuition that the dynamics of EMA near a local Nash equilibrium are tightly linked to those of the underlying training algorithm. Corollary 1.1. Assuming that 0 < β < 1, the eigenvalues of J F lie within the unit ball if and only if the eigenvalues of J Fβ lie within the unit ball.

Also, we see that EMA could potentially slow down the rate of convergence compared to the underlying training method if β is larger than the magnitude of all other eigenvectors of J F β .

5. EXPERIMENTS:

We use both illustrative examples (i.e. mixtures of Gaussians) as well as four commonly used realworld datasets, namely CIFAR-10 ( Krizhevsky et al., 2009) , STL-10 (Coates et al., 2011) , CelebA (Liu et al., 2015) , and ImageNet (Russakovsky et al., 2015) to show the effectiveness of averaging. For mixtures of Gaussians, the architecture, hyperparameters and other settings are defined in its own section. For CelebA, 64x64 pixel resolution is used, while all other experiments are conducted on 32x32 pixel images.

All dataset samples are scaled to [-1, 1] range. No labels are used in any of the experiments. For the optimizer, we use ADAM (Kingma & Ba, 2014) with α = 0.0002, β 1 = 0.0 and β 2 = 0.9. For the GAN objective, we use the original GAN (Referring to Non-Saturating variant) (Goodfellow et al., 2014) objective and the Wasserstein-1 (Arjovsky et al., 2017) objective, with Lipschitz constraint satisfied by gradient penalty (Gulrajani et al., 2017) . Spectral normalization (Miyato et al., 2018 ) is only used with the original objective. In all experiments, the updates are alternating gradients.

The network architecture is similar to a full version of a progressively growing GAN (Karras et al., 2017) , with details provided in the Appendix. We call this architecture conventional. ResNet from Gulrajani et al. (2017) is used as a secondary architecture. Unless stated otherwise, the objective is the original one, the architecture is conventional, discriminator to generator update ratio is 1, β value is 0.9999 and MA starting point is 100k. Inception (Salimans et al., 2016) and FID (Heusel et al., 2017) scores are used as quantitative measures. Higher inception scores and lower FID scores are better. We have used the online Chainer implementation2 for both scores. For Inception score evaluation, we follow the guideline from Salimans et al. (2016) . The FID score is evaluated on 10k generated images and statistics of data are calculated at the same scale of generation e.g. 32x32 data statistics for 32x32 generation. The maximum number of iterations for any experiment is 500k. For each experiment, we state the training duration. At every 10k iterations, Inception and FID scores are evaluated, and we pick the best point. All experiments are repeated at least 3 times with random initializations to show that the results are consistent across different runs.

In addition to these experiments, we also compare the averaging methods, with Consensus Optimization (Mescheder et al., 2017) , Optimistic Adam (Daskalakis et al., 2018) and Zero Centered Gradient Penalty (Mescheder et al., 2018) on the Mixture of Gaussians data set and CIFAR-10. For CIFAR-10 comparison, we use a DCGAN-like architecture Radford et al. (2015) and follow the same hyperparameter setting as in Mixture of Gaussians except WGAN-GP's λ which is 10.0.

To ensure a fair comparison, we use the same samples from the prior distribution for both averaged and non-averaged generators. In this way, the effect of averaging can be directly observed, as the generator generates similar images. We did not cherry-pick any images: all images generated are from randomly selected samples from the prior. For extended results refer to the Appendix.

5.1. MIXTURE OF GAUSSIANS:

This illustrative example is a two-dimensional mixture of 16 Gaussians, where the mean of each Gaussian lies on the intersection points of a 4x4 grid. Each Gaussian is isotropic with σ = 0.2. Original GAN (Goodfellow et al., 2014) is used as the objective function with Gradient Penalty from Gulrajani et al. (2017) , with 1:1 discriminator/generator update and alternative update rule. We run the experiment for 40k iterations, and measure Wasserstein-13 distance with 2048 samples from both distributions at every 2k iterations. We take the average of the last 10 measurements from the same experiment and repeat it 5 times. We compare this baseline with Optimistic Adam (OMD), Consensus Optimization (CO), and Zero Centered Gradient Penalty (Zero-GP) by using the same architecture, and following the original implementations as closely as possible. We have also combined EMA and MA with OMD, CO and Zero-GP to see whether they can improve over these methods. Detailed settings are listed in the Appendix.

Table 1 shows the distance for various methods. In all cases, EMA outperforms other methods, and interestingly, it also improves OMD, CO and Zero-GP. This indicates that OMD, CO and Zero-GP do not necessarily converge in non-convex/concave settings, but still cycle. MA also improves the results in certain cases, but not as much as EMA. Our observation is that uniform averaging over long time iterates in the non-convex/concave case hurts performance (more on this in later experiments). Because of this, we have started uniform averaging at later stages in training and treat it as a hyper-parameter. For this experiment, the start interval is selected as 20k. 4 and others in the Appendix). We have observed improvements when the starting point of MA is pushed later in training, however for convenience and ease of experimentation our main results are based on the EMA method.

Published as a conference paper at ICLR 2019

5.3. CIFAR-10:

Figure 3 shows images generated with and without EMA for CIFAR-10 after 300k iterations. Objects exhibit fewer artifacts and look visually better with EMA. It is interesting to note that the averaged and non-averaged versions converge to the same object, but can have different color and texture.

Figure 4 shows Inception and FID scores during training. EMA model outperform its non-averaged counter-part consistently by a large margin, which is more or less stable throughout the training. However MA model's FID performance reduces gradually as it considers longer time windows for averaging. This observation also holds in STL10 and ImageNet experiments (See Appendix).

Our intuition is that this phenomenon occurs because generator iterates does not stay in the same parameter region and averaging over parameters of different regions produces worse results. Besides, we have seen clear disagreement between FID and IS in case of MA. We think this happens because IS is not a proper metric to measure difference between two distributions and has many known flaws such as it does not measure intra-class diversity (Barratt & Sharma, 2018; Zhou et al., 2018; Heusel et al., 2017) .

Figure 5 compares EMA, CO and OMD. EMA clearly improves baseline and OMD in both scores, while its improvement on CO is smaller but still consistent. 

5.4. CELEBA:

Figure 6 shows the same 10 samples from the prior fed to the non-averaged generator, the EMA generator with various β values, and the MA generator at 250k iterations in the training. The nonaveraged images show strong artifacts, but as we average over longer time windows, image quality improves noticeably. Interestingly, significant attribute changes occurs as β approaches 1. There is a shift in gender, appearance, hair color, background etc., although images are still similar in pose and composition. EMA results with β = 0.999 and β = 0.9999 also look better than MA results, which are averaged over the last 200k iterates. This shows there is an optimal β/window size to tune for best results. Figure 8 shows images generated with and w/o EMA for STL-10 and ImageNet. Even though quantitative scores are better for EMA, we do not see as clear visual improvements as in previous results but rather small changes. Both models produce images that are unrecognizable to a large degree. This observation strengthens our intuition that averaging brings the cycling generator closer to the local optimal point, but it does not necessarily find a good solution as the local optimum may not be good enough. 

6. CONCLUSION:

We have explored the effect of two different techniques for averaging parameters outside of the GAN training loop, moving average (MA) and exponential moving average (EMA). We have shown that both techniques significantly improve the quality of generated images on various datasets, network architectures and GAN objectives. In the case of the EMA technique, we have provided the first theoretical analysis of its implications, showing that even in simple bilinear settings it converges to stable limit cycles of small amplitude around the solution of the saddle problem. Averaging methods are easy to implement and have minimal computation overhead. As a result, these techniques are readily applicable in a wide range of settings. In the future, we plan to explore their effect on larger scales as well as on conditional GANs.

A ADDITIONAL RESULTS AND EXPERIMENTS:

A.1 PARAMETERIZED ONLINE AVERAGING As an additional method we have used parameterized online averaging:

EQUATION

where α regulates the importance of the terms. When α = 1, the equation corresponds to moving average (Eq. 1). As t-α t might be negative, this algorithm should be restricted to the case of t ≥ α. We have started to apply it when t ≥ α. We have experimented (Table 3 ) with this method on CIFAR-10 with the same setting of the first row of Table 2 . α = 1 has failed as it started averaging from the first iteration which aligns with out earlier observations. α = 10, 100, 1000 improves the scores at varying degree, however none is as good as EMA. In the main body of this paper, we have used β = 0.9999 in all experiments but CelebA as it performs the best. In this section we show the effect of β to quantitative scores. We have experimented different β values ranging from 0.9 to 0.9999 on CIFAR-10 with the same setting of the first row of Table 2 . We have run the experiment four times and provide the mean and standard deviation in Table 4 . Each experiment runs for 500k. IS, FID score are collected with 20k iteration interval and best scores are published. From Table 4 , we can see clear and progressive improvement when β increases from 0.9 to 0.9999 in both IS and FID. 

:

https://github.com/pfnet-research/chainer-gan-lib/blob/master/common/ evaluation.py

Python Optimal Transport package at http://pot.readthedocs.io/en/stable/

Figures and Tables:

Figure 1: MA versus EMA for various values of β.

Figure 3: Generation for CIFAR-10 dataset.

Figure 6: Generation for CelebA dataset for various β values at 250k iterations. From top to bottom: (a) non-averaged generator, (b) β = 0.9, (c) β = 0.99, (d) β = 0.999, (e) β = 0.9999, (f) MA.

Figure7compares the stability of a non-averaged generator and EMA by using the same 2 samples from the prior. Starting from 50k iteration, images are generated from both models with 10k interval until 200k iterations. Images from the non-averaged generator change attributes frequently, while only preserving generic image composition. Meanwhile, the averaged generator produces smoother changes, with attributes changing more slowly.

Figure 7: Generation for CelebA dataset for the same 2 noise samples from 50k to 200k with 10k intervals. (top 2 rows) without averaging, (bottom two rows) with EMA β = 0.9999

Figure 8: Generation for STL-10 & ImageNet dataset after 500k iteration. (Top): STL-10, (Bottom):ImageNet, (Left): w/o averaging (Right): with EMA

Figure 9: Generation for CIFAR-10 dataset after 300k iterations.

Figure 11: Generation for ImageNet dataset after 500k iterations.

Figure 14: Generation for CelebA for various β values at 100k iteration. From top to bottom rows: (a) non-averaged generator, (b) β = 0.9, (c) β = 0.99, (d) β = 0.999, (e) β = 0.9999, (f) MA

Figure 18: Generation for CelebA for various β values at 350k iteration. From top to bottom rows: (a) non-averaged generator, (b) β = 0.9, (c) β = 0.99, (d) β = 0.999, (e) β = 0.9999, (f) MA

Figure 20: CIFAR-10 FID and IS scores during training. Setting: Original GAN objective/ Conventional Architecture/ n dis = 1

Figure 23: CIFAR-10 FID and IS scores during training. Setting: WGAN-GP objective/ Conventional Architecture/ n dis = 5

Wasserstein-1 Distance for non-averaged generator, EMA, MA, Optimistic Adam, Consensus Figure 2 is an illustration of the above results. The blue regions indicate a lack of support, while the red ones indicate a region of support.Without averaging, the generator omits certain modes at different time points, and covers them later while omitting previously covered modes. We observe a clear fluctuation in this case, while the EMA consistently shows more balanced and stable supports across different iterations. OMD, CO and Zero-GP perform better than the baseline, however they seem to be less stable across different iterations than EMA results.

Table content: <table><tr><td>Optimization and Zero-GP</td><td/><td/></tr><tr><td/><td>no-average</td><td>EMA</td><td>MA</td></tr><tr><td>Baseline</td><td colspan="3">0.0419 ± 0.0054 0.0251 ± 0.0026 0.0274 ± 0.0016</td></tr><tr><td>Optimistic Adam</td><td colspan="3">0.0494 ± 0.0041 0.0431 ± 0.0042 0.0525 ± 0.0024</td></tr><tr><td colspan="4">Consensus Optimization 0.0286 ± 0.0034 0.0252 ± 0.0035 0.0390 ± 0.0105</td></tr><tr><td>Zero-GP</td><td colspan="3">0.0363 ± 0.0018 0.0246 ± 0.0068 0.0317 ± 0.0121</td></tr></table>

Inception and FID scores on CIFAR-10, STL-10 and ImageNet. (100k), (250k) etc. refer to number of iterations of the generator. Experiments were repeated 3 times.

Table content: <table><tr><td/><td/><td>IS</td><td/><td/><td>FID</td></tr><tr><td/><td>no average</td><td>EMA</td><td>MA</td><td>no average</td><td>EMA</td><td>MA</td></tr><tr><td>CIFAR-10</td><td/><td/><td/><td/><td/></tr><tr><td>conventional (500k)</td><td colspan="3">8.14 ± 0.01 8.89 ± 0.12 8.74 ± 0.19</td><td colspan="3">16.84 ± 1.00 12.56 ± 0.17 16.59 ± 0.52</td></tr><tr><td colspan="4">conventional WGAN-GP n dis = 1 (250k) 7.18 ± 0.06 7.85 ± 0.08 7.93 ± 0.15</td><td colspan="3">25.51 ± 1.22 19.18 ± 0.95 22.71 ± 0.95</td></tr><tr><td colspan="4">conventional WGAN-GP n dis = 5 (250k) 7.22 ± 0.07 7.92 ± 0.10 8.20 ± 0.06</td><td colspan="3">25.32 ± 0.70 19.72 ± 0.31 20.02 ± 0.75</td></tr><tr><td>ResNet n dis = 5 (120k)</td><td colspan="3">7.86 ± 0.13 8.46 ± 0.13 8.68 ± 0.09</td><td colspan="3">20.64 ± 1.38 17.58 ± 1.84 18.30 ± 1.32</td></tr><tr><td>ResNet WGAN-GP n dis = 1 (250k)</td><td colspan="3">7.63 ± 0.12 8.22 ± 0.05 8.51 ± 0.07</td><td colspan="3">21.49 ± 0.40 15.85 ± 0.27 16.27 ± 0.56</td></tr><tr><td>ResNet WGAN-GP n dis = 5 (250k)</td><td colspan="3">7.38 ± 0.26 7.94 ± 0.31 8.32 ± 0.25</td><td colspan="3">23.20 ± 2.62 19.41 ± 2.57 19.33 ± 3.17</td></tr><tr><td>STL-10</td><td/><td/><td/><td/><td/></tr><tr><td>conventional (250k)</td><td colspan="3">7.96 ± 0.35 8.39 ± 0.10 8.23 ± 0.24</td><td colspan="3">22.33 ± 1.59 19.64 ± 1.38 26.11 ± 2.27</td></tr><tr><td>ImageNet</td><td/><td/><td/><td/><td/></tr><tr><td>conventional (500k)</td><td colspan="3">8.33 ± 0.10 8.88 ± 0.14 8.33 ± 0.03</td><td colspan="3">24.59 ± 0.59 21.83 ± 0.93 26.77 ± 1.12</td></tr><tr><td>5.2 QUALITATIVE SCORES</td><td/><td/><td/><td/><td/></tr></table>

IS and FID on CIFAR-10 (the same setting of the first row of Table2) with various α values for parameterized online averaging.

Table content: <table><tr><td/><td>IS</td><td>FID</td></tr><tr><td>nA</td><td colspan="2">8.02 ± 0.20 17.32 ± 0.53</td></tr><tr><td>α = 1</td><td>failed</td><td>failed</td></tr><tr><td>α = 10</td><td colspan="2">8.58 ± 0.06 14.55 ± 1.01</td></tr><tr><td>α = 100</td><td colspan="2">8.48 ± 0.06 13.85 ± 0.51</td></tr><tr><td>α = 1000</td><td colspan="2">8.28 ± 0.04 14.64 ± 0.51</td></tr><tr><td colspan="3">α = 10000 8.11 ± 0.13 15.60 ± 0.59</td></tr><tr><td colspan="3">A.2 QUANTITATIVE EFFECTS OF DIFFERENT β FOR EMA</td></tr></table>

IS and FID on CIFAR-10 (the same setting of the first row of Table2) with various β values for EMA.

Table content: <table><tr><td/><td>IS</td><td>FID</td></tr><tr><td>nA</td><td colspan="2">8.09 ± 0.07 18.14 ± 1.08</td></tr><tr><td>EMA β = 0.9</td><td colspan="2">8.19 ± 0.09 17.82 ± 0.43</td></tr><tr><td>EMA β = 0.99</td><td colspan="2">8.32 ± 0.03 15.64 ± 0.58</td></tr><tr><td>EMA β = 0.999</td><td colspan="2">8.69 ± 0.11 13.77 ± 0.38</td></tr><tr><td colspan="3">EMA β = 0.9999 9.05 ± 0.09 12.91 ± 0.49</td></tr></table>

Peer review (iclr2019):
 This paper tries to adapt the concept of averaging, well known is the game literature, to GAN training. In a simple min-max example the iterates obtained by gradient method do not converge to the equilibrium of the game but their average does. This work first provides intuitions on the potential benefits of exponential moving average (EMA) on a simple illustrative example and explore the effect of averaging on GAN. In think that the approach of this paper is interesting. I particularly like the experiments on Celeb-A (Fig 6 and 7) that seem to show that the averaged iterates change more smoothly (with respect to the attributes of the faces) during the training procedure. Nevertheless, I have some concerns about the claims of the paper and the experimental process. I*m surprised by the values of the inception score provided in Table 2 which do not seem to correlate with the sample quality in Fig. 3. Why did not you use the standard implementation of the inception score provided in Salimans et al. [2016]*s paper ? I think that the effectiveness of EMA over uniform averaging is a bit overclaimed. - From a theoretical point of view uniform averaging works better (at least in your example in 3.1): If you (uniformly) average the periodic orbit you get a converging iterate. Moreover, concerning to this toy example, note that this continuous analysis has been already introduced in [Goodfellow et al., 2016] and the Hamiltonian interpretation has been already provided in [Balduzzi et al. 2018]. However I think that the intuition on the vanishing magnitude of the oscillation provided by EMA is interesting. - The continuous dynamics is actually different from the discrete one, I think that an analysis on the discrete case that is used in practice might be more insightful. - The comparison with uniform averaging is not fair in the sense that uniform averaging has no hyperparameter to tune: In figure 6 uniform averaging performs better than a not well tuned EMA. A fair comparison would be for instance to propose a parametrized online averaging and to tune it the same way is tuned in EMA. Refs: Salimans, Tim, et al. "Improved techniques for training gans." Advances in Neural Information Processing Systems. 2016. Goodfellow, I. (2016). NIPS 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00160. Balduzzi, David, et al. "The Mechanics of n-Player Differentiable Games." ICML (2018). Minor comments: - In the introduction "gradient vector fields of the game may not be conservative (Mescheder et al. 2017)" and the related work "Mescheder et al. (2017) states that a reason for non-convergence is the non-conservative gradient vector of the players.": the notion of conservative vs. non-conservative vector field is never mentioned in [Mescheder et al. 2017]. I think you are actually referring to the blog post on that paper https://www.inference.vc/my-notes-on-the-numerics-of-gans/ . - In the Related work "can not" - "In fact, it has recently been established that the smooth (continuous-time) analogues of first order methods such as online gradient descent (follow-the-regularized leader) in bilinear zero-sum games are recurrent (i.e. effectively periodic) with trajectories cycling back into themselves. " can you provide a citation ? - Some published papers are refereed as arxiv paper ( for instance (Mescheder et al. 2017) and (Mescheder et al. 2018)), you should cite the published version.