Taming GANs with Lookahead-Minmax
Authors: Tatjana Chavdarova
Matteo Pagliardini
Sebastian U Stich
François Fleuret
Martin Jaggi
Venue: ICLR 2021 Poster (2021)
Keywords: Minmax
Generative Adversarial Networks

Abstract: Generative Adversarial Networks are notoriously challenging to train. The underlying minmax optimization is highly susceptible to the variance of the stochastic gradient and the rotational component of the associated game vector field. To tackle these challenges, we propose the Lookahead algorithm for minmax optimization, originally developed for single objective minimization only. The backtracking step of our Lookahead–minmax naturally handles the rotational game dynamics, a property which was identified to be key for enabling gradient ascent descent methods to converge on challenging examples often analyzed in the literature. Moreover, it implicitly handles high variance without using large mini-batches, known to be essential for reaching state of the art performance. Experimental results on MNIST, SVHN, CIFAR-10, and ImageNet demonstrate a clear advantage of combining Lookahead–minmax with Adam or extragradient, in terms of performance and improved stability, for negligible memory and computational cost. Using 30-fold fewer parameters and 16-fold smaller minibatches we outperform the reported performance of the class-dependent BigGAN on CIFAR-10 by obtaining FID of 12.19 without using the class labels, bringing state-of-the-art GAN training within reach of common computational resources.

Published as a conference paper at ICLR 2021
TAMING GANS WITH BOOKAHEAD-MINANN
Tatjana Chavdarova" EPFL
Mattéo Pagliardini EPFL
Sebastian U. Stich
François Fleuret University of Geneva
Martin Jaggi EPFL
EPFL
ABSTRACT
Generative Adversarial Networks are notoriously challenging to train. The under- lying minmax optimization highly susceptible to the variance of the stochastic gradient and the rotational component of the associated game vector field. To tackle these challenges, we propose Lookahead algorithm for minmax optimization, originally developed for single objective minimization only. The backtracking step our Lookahead-minmen naturally handles the rotational game dynamics, property which was identified to be key for enabling gradient ascent descent methods to converge on challenging examples often analyzed in the literature. Moreover, it implicitly handles high variance without using large Mini-batches, known to be essential for reaching state of the art performance. Experimental results on MNIST, SVHN, CIFAR-10, and ImageNet demonstrate clear advan- tage of combining performance and improved stability, for negligible memory and computational cost. Using 30-fold fewer parameters and 16-fold smaller minibatches we outperform the reported performance of the class- deas-dee BigGAN on CIFAR- by obtaining FID of 12.19 without using the class labels, bringing state-of-the of-thearrr GAN training within reach common computational resources. Our source code available: wtt.cccc//cc/// https: httphhuttcccc///aaaaaaraaaaaaaee Lookaaaa Mi mmxx
mookahead-minmas with Adam or extragradient, in terms of
INTRODUCTION
Gradient-based methods are workhorse of machine learning. These methods optimize the parameters of model with respect to single objective f: X' - R. However, an increasing interest for multi-objective optimization arises various domains such as mathematics, economics, multi- agent reinforcement learning (Omidshatiei et al., where several agents aim at optimizing their own cost function f;: Xi X ... x Xx > simultaneously.
A particularly successful class of algorithms of this kind are the Generative Adversarial Net- works (Goodfellow et al. [014]] (GANs)), which consist of two players referred to as generator and discriminator. GANs were originally formulated minmax optimization f: X' - IR (Von Neu- mann & Morgenstern 1944). where the generator and the discriminator aim at minimizing and maximizing the same value function, see $2 A natural generalization gradient descent for minmax problems is the gradient descent ascent algorithm (GDA), which alternates between gradient descent step for the min-player and a gradient ascent step for the max max-player mlx-el This minmax training aims at finding a Nash equilibrium where no player has the incentive of changing its parameters.
Despite the impressive quality of the samples generated by the GANs relative to classical maximum likeiihood-based generative models-these models remain notoriously difficult train. In particular, poor performance (sometimes manifesting as "mode collapse"), brittle dependency on hyperparam- eters, divergence are often reported. Consequently, obtaining state-of-the-ant performance shown to require large computational resourees (Brock et al. 2019), making well- well-performing models unavailable for common computational budgets.
"Equal contributions. Correspondence fi ristnamee. 1 aastaam@eppf1.co

---

Published as a conference paper at ICLR 2021
25
Fast weights
ww
Slow weights
LA-step every 10k iterations
15
Figure Illustration of Lookahead-minmal [Alg with GDA on: ming maxy x y. with a=0.5. The solution, trajectory {æt, 1/ Y
2
5
Iteration
x X10"
Figure 2: FID (L) wwwoooooad-minnnnne on x2x 32 ImageNet (8 (). see Fig. [12 for IS. We observe significant improvements after each LA-step. what empirically confirms the existence of rotations and thus the intuition of the behaviour of LA- minmax on games, as well as the relevance of the bilincar game for real world applications Fig]]
and the lines between (x" (x".v²) P) and (2t. are shown with red star, blue line, and dashed
green line, resp. The backtracking step of
Alg. [ (lines 10 & [1] allows the otherwise non-converging GDA to converge, see $ 4.2.1
was empirically shown that: (i) GANs often converge to a locally stable stationary point that is not differential Nash equilibrium (Berard et al. 2020); (ii) increased batch size improves GAN performances (Brock et al. 2019) in contrast to minimization (Defazio & Bottou| 2019 (Shallue et al. 2018). A principal reason is attributed the rotations arising due to the adversarial component of the associated vector field of the gradient of the two player's parameters (Mescheder et al. 2018 Balduzzi et al. 2018), which are atypical for minimization. More precisely, the Jacobian of the associated vector field (see def. in $ 2 can be decomposed into a symmetric and antisymmetric component (Balduzzi et al. 2018), which behave as a potential (Monderer & Shapley (996) and Hamiltonian game, resp. Games are often combination of the two, making this general case harder to solve.
In the context of single objective minimization, Zhang et al. (2019) recently proposed the Lookahead algorithm, which intuitively uses an update direction by "looking ahead" at the sequence of parameters that change with higher variance due to the "stochasticity" of the gradient estimates-calle fast weights-generater by an inner optimizer. Lookahead was shown to improve the stability during training and to reduce the variance of the called slow weights.
Contributions. Our contributions can be summarized as follows:
We propose Gookahead-minmax for optimizing minmax problems, that applies extrapolation in the joint parameter space (see T. so as to account for the rotational component of the associated game vector field (defined in $2).
In the context of: (i) single objective minimization: by building on insights of Wang et al. (2020). who argue that Lookahead can be interpreted as instance of local SGD, we derive improved convergence guarantees for the Lookahead algorithm; (ii) two-player games: we elaborate why wookahead-minma suppresses the rotational part simple bilinear game, and prove its convergence for a given converging bast-optimizer, in $ and resp.
We motivate the use of Lookahead minmax for games by considering the extensively studied toy bilinear example (Goodfellow) 2016) and show that: (i) the of lookahead allows for convergence of the otherwise diverging GDA on the classical bilinear game in full-batch setting (see 4.2.11 well (ii) it yields good performance on challenging stochastic variants of this game, despite the high variance (see $ 4.2.2.
We empirically benchmark Lookahead-minmax on GANs on four standard datasets- MNIST, CIF SVHN and ImageNet-on two different models (DCGAN & ResNet), with standard optimization methods for GANs, GDA and Extragradient, called LA-AlIGAN and AA-ExtraGradient resp. We consistently observe both stability and performance improvements at negligible additional cost that does not require additional forward and backward passes, see $
2

---

Published as a conference paper at ICLR 2021
2 BACKGROUND
GAN formulation. Given the data distribution Pd, the generator a mapping G: 2 + 2, where is sampled from a known distribution z Ps and ideally x ~ Pa- The discriminator D: x + D(x) E [0, 1] is a binary classifier whose output represents a conditional probability estimate that an æ sampled from balanced mixture of real data from pa and G-generated data is actually real. The optimization of a GAN is formulated as a differentiable two-player game where the generator G with parameters 0, and the discriminator D with parameters (p, aim at minimizing their own cost function L and LV, respectively, as follows:
0" E arg min L (0, 4p" Bee
and
xp" E arg min (6", (p)
(2P-G)
PE#
When C = _L the game is called a zero-sum and equation 2P-G is minmax problem.
Minmax optimization methods. As GDA does not converge for some simple convex-concave game, Korpelevich 1976) proposed the extragradient method, where a "prediction" step is performed to obtain an extrapolated point (0++ ( using GDA, and the gradients the extrapolated point
are then applied to the current iterate (0,, (P:) as follows:
Extrapolation: {
PP:)
0 [0.100
Update:
(EG)
( (0. sp)
( (, (p+4))
where 1) denotes the step size. In the context of zero-sum games, the extragradient method converges for any convex-concave function C and any closed convex sets è and (Facchinei & Pang 2003).
The joint vector field. Mescheder et al. (017)) and Balduzzi et al. (2018) argue that the vector field obtained by concatenating the gradients of the two players gives more insights of the dynamics than studying the loss surface. The joint vector field (JVF) and the Jacobian of JVF are defined as:
VoLa (0, x)
Vo
v = ( and v'(0,))) = V 4 (0,
resp. (JVF)
V (O,
Rotational component of the game vector field. Berard et al. 2020) show empirically that GANs converge locally stable stationary point (Verhulst 1990 LSSP) that is not differential Nash equilibriun-deffeed as a point where the norm of the Jacobian is zero and where the Hessian of both the players are definite positive, see $ C LSSP is defined point (0* :4p") where:
v(0", ",'"")= =0 and R(A) O, AA E Sp(v'(0", (0",)")),
(LSSP)
where Sp(-) denotes the spectrum of v-(.) and R(-) the real part. In summary, (i) if all the eigenvalues of o' (' (p)) have positive real part the point (0.. (P:) is LSSP, and (ii) if the eigenvalues of 2'(0,, Pp) have imaginary part, the dynamics of the game exhibit rotations.
Impact noise due to the stochastic gradient estimates on games. Chavdarova et al. (2019) point out that relative to minimization, noise impedes more the game optimization, and show that there exists a class of zero-sum games for which the stochastic extragradient method diverges. Intuitively, bounded noise of the stochastic gradient hurts the convergence as with higher probability the noisy gradient points in a direction that makes the algorithm to diverge from the cquilibrium, due to the properties of o' (see 1i Chavdarova et al. 2019
LOOKAHEAD FOR SINGLE OBJECTIVE
In the context of single objective minimization, Zhang et al. 2019) recently proposed the Lookahead algorithm where at every step t: (i) a copy of the current iterate W; is made: Wi <- Wes (ii) W, is
then updated for k > 1 times yielding Witk. and finally (iii) the actual update Wr+ is obtained as a point that lies on a line between the two iterates: the current w, and the predicted one Wepsi
Wh+1 <- wi + alir+k wi),
where [0. 1].
(LA)
3

---

Published as a conference paper at ICLR 2021
Algorithm 1 General Lookahead-Minnaa pseudocode.
GDA
UnrolEXY
GDA 1=10
OGDA
1: Input: Stopping time T, learning rates rje, Th, ini- tial weights Oo. (po, lookahead & and a, losses _ CP, d dc base optimizer updates defined in
ExerreGred
LAJOGDA
LA ExernCinnd
LA-GDA
Unrol-Y
LA-GDA a-0.4
Pa and p3 real and noise-date data distributions, resp.
10
2: 00,0; PP.00
< Pa.0 40
3: for / E 0, T-1 do
Datane to the optinum
100
4: for O.... ......,111 k-1 do
5: Sample x, Pa. Ps
10.2
6: Pi,i+1 = L1, The di ( Pt,i; x;, x)
7: Sample = ~ P=
10-4
8: 0,i+1 = 0r,i+1 110 no ( (e..
9:
end for
10: Pi+1 = P: + axp r k- - 4p:)
11:
Oct1 02 + (0 - O.)
200
Numbeer of possoss
600
800
1.1100 1.+1.0 + 0c+1, 1
Figure 3: Distance the optimum of equa- tion SSB-G using different full-re full-hatch methods, averaged over 5 runs. Unless otherwise speci- fied, the learning rate is 0.3. See $ $44.2.1
13: end for
Output: 0r, PT
Lookahead uses two additional hyperparameters: (i) k-the number of steps used to obtain the prediction Witk, well (ii) a- that controls how large a step make towards the predicted iterate W: the larger the closest, and when a = equation LAjis equivalent to regular optimization (has no impact). Besides the extra hyperparameters, LA was shown to help the used optimizer be more resilient to the choice of its hyperparameters. achieve faster convergence across different tasks, well to reduce the variance of the gradient estimates (Zhang al. 2019).
Theoretical Analysis. Zhang et al. (2019) study [A on quadratic functions and Wang et al. (2020) recently provided an analysis for general smooth non-convex functions. One of their main observa- tions is that L. can be viewed as an instance of local SGD (or parallel SGD, Stich 2019 Koloskoval
ct al. 2020 Woodworth et al., 2020b) which allows us to further tighten prior results.
Theorem 1. Let Rdd R be L-smooth (possibly non-convex) and assume access to unbiased stochastic gradients o² -bounded variance. Then the LA optimizer with hyperparemeters (k, a)
converges to a stationary point V f
)|2 < (for the proof refer to Appendix A), after at most k
0
lovkv111 -
1-a ( a
02
3//2
iterations. Here Wout denotes uniformly at random chosen iterate Lf
Remark 1. When in addition is also quadratic, the complexity estimate improves to O + +
The asymptotically most significant term, O ((²), matches with the corresponding term in the SGD convergence rate for all choices of a E (0, 1]. and when - 1, the same convergence guarantees as for SGD can be attained. When x² = 0 the rate improves to ( in contrast to ( in WWan et al. 2020 For small values of a, the worst-case complexity estimates of LA can in general be k times worse than for SGD (except for quadratic functions, where the rates match). Deriving tighter analyses for LA that corroborate the observed practical advantages is still an open problem.
4 LOOKAHEAD FOR MINMAX OBJECTIVES
We now study the LA optimizer in the context of minmax optimization. We start with an illustrative example, considering the bilinear game: ming max 0 Iςp (see Fig. 1 Observation Gradient Descent Ascent always diverges. The iterates of GDA are:
0 p0+1
0. 8
W1+1
+
0,
'Our results rely optimally tuned stepsize for every choice of a. different ohservation hut keep the stepize y fixed for different choices of
Wang et al. 2020 state slightly
4

---

Published as a conference paper at ICLR 2021
The norm of the iterates || 112 (1 + w, increases for any stepsize 1 0, hence GDA diverges for all choices of 17.
In this bilinear game, taking point on line between two points cyclic trajectory would reduce the distance to the optimum-a as illustrated in Fig. | what motivates extending the Lookahead algorithm to games, while using equation LA in joint parameter space w - ())
Observation 2: Lookahead can converge. The iterates of
LA with A steps of GDA are:
[0+1 [4+1]
[
-2p: n10
W++11
[P:
07)
20, 7/p:
We can again compute the norm:
||++1ll? n2)= + +4 anfa²2 ||0]|? For the choice
(0, +"44
the norm strictly decreases, hence the algorithm converges linearly.
4.1 THE GENERAL LOOKAHEAD-MINMAL ALGORRIHH AND ITS CONVERGENCE
Alg. M summarizes the proposed bankaheaa--iinas algorithm which for clarity does not cover
different update ratios for the two players-see Alg. for this case. At every step f I,..., I, first compute k-step "predicted" iterates (0t,,. (P.,e) also called fast brights-using, an inner "hase" optimizer whose updates for the two players are denoted with de and d For GDA and EG have:
di A V
V (P Pr,,s 2,
dh V C (0
1 Pe,s+ 2, 2,
GDA:
EG:
de Vo (0r, 2),,;)
d V
C +. 9.
where for the latter, 4 02 1.4+ are computed using equation EG The above updates can be combined with Adam (Kingma & Ba 2015). see G detailed descriptions of these algorithms. point on line between (0., (P:) and (0z,A: ( -i.. equation LA. see lines 10 & I1 (herein called backtracking or LA-step). When combined with GDA, Alg. can be equivalently re-written as Alg. and note that it differs from using Lookahead separately per player-see $ G.2 Moreover, Alg. can be applied several times using nested LA-steps with (different) k, see Alg.
The so called slow weights 0 1, Pi+1 are then obtained
Local convergence. We analyze the game dynamics as a discrete dynamical system as in (Wang ct al. 2020 Gidel et al. 2019b). Let (w) denote the operator that an optimization method applies to (0, sp). i.e.
the iterates point wn is called fixed point if F F("*=(()))))) w*, and it is stable if the spectral radius p( ((FF(w))) S 1, show that Lookahead-
Fo... - F(wo). For example, for GDA we have F(w) = (w).
where for GDA for example VF(w") = 178' (w"). In the following, minmax combined with a base optimizer which converges to a stable fixed point, also converges to the same fixed point. On the other hand, its convergence when combined with diverging base-optimizer depends on the choice of its hyper-parameters. Developing a practical algorithm for finding optimal set of hyper- hyper-paramene parameters for realistic setup out of the scope of this work, and in $ 5 we show empirically that Alg. [ outperforms its base-optimizer for all tested hyperparameters.
Theorem 2. (Proposition 4.4. Bertsekas 1999) If the spectral radius pl V (w=")) 1, the w" is point of attraction and for wo in a sufficiently small neighborhood of wt, the distance of We to the stationary point wt converges at a linear rate.
The above theorem guarantees local linear convergence to a fixed point for particular class of operators. The following general theorem (proof in $. (B) further shows that for any such base optimizer (of same operator class) which converges linearly to the solution, Mookahead-Mnnaa converges too, what includes Proximal Point, EG and OGDA but does not give precise rates rates-left for future work.
Theorem 3. (Convergence of Lookahead Minmaxx given converging base optimizer) If the spectral radius of the Jacobian of the operator of the base optimizer satisfies pbaee ggha)e("")) ))= < 1, then for wo in a neighborhood of w", the iterates w, of MiiisaheadMiimmaa converge to wt as t + 00.
Re-visiting the above example. For the GDA operator have p (VFGGA("")) and as À 2 i0 C, with {X(*) ft= re ..... r dke which are diverging and rotating in C. The spectral radius of LA-GDA
| 1+1/² >
1. Applying k times will thus result in eigenvalues
with k = 2 is then - a + ax² (see $ BB) where a selects a particular point that lies between 1 + Oi and 2 in C. allowing for reducing the spectral radius of the GDA base operator.
5

---

Published as a conference paper at ICLR 2021
4.2 MOTIVATING EXAMPLE: THE BILINEAR GAME
We argue that Lookahead-minmax allows for improved stability and performance on minmax prob- lems due main reasons: (i) it handles well rotations, as well (ii) it reduces the noise due to making more conservative steps. In the following, we disentangle the two, and show in $ $ that Lookahead-minnae converges fast in the full-batch setting, without presence of noise as each parameter update uses the full dataset.
In $ 4.2.2 we consider the challenging problem of Chavdarova et al. 2019), designed have high variance. We show that besides therein proposed Stochastic Variance Reduced Extragradient (SVRE), wookahea-miinman is the only method that converges on this experiment, while considering all methods of Gidel et al. 2019a $7.1). More precisely, we consider the following bilinear problem:
min max L(O, (p)
Σ min max-
e b, + 0 A C, ())
(SB-G)
GER spERd
ØER peRd 7
i=1
with 0, sp b,, GER and AERaxdxd n=d=100, and draw [A and [bo]kr [c,l (0,, 1/d), 1 < k, < d. where 6 if k=l=i, and O otherwise. As one pass we count a forward and backward pass D.5 and all results are normalized. See D. for hyperparameters.
4.2.1 THE FULL-BATCH SETTING
In Fig. 3 we compare: (i) GDA with learning rate 10 and 0.3 (in blue), which oscilates around the optimum with small enough step size, and diverges otherwise; (ii) Unroll-Y where the max-player is unrolled & steps, before updating the min player, as in (Metz et al. (017);; (iii) Un- roll-XY where both the players are unrolled steps with fixed opponent, and the actual updates are done with un unrolled opponent (see $ D):
(iv) LA-GDA with a 0.5 and
0..4 (in red and
pink, resp.) which combines Alg. [ with GDA. (v) ExtraGradient -etiieentte..ll EG] (vi) GradEx
which combines Alg. [ with ExtraGradient; (vii) OGDA Optimistic GDA (Rakhlin & Sridharan
2013), as well as (vili LA-OGDA which combines Alg. [Tit OGDA. See $|D description of
the algorithms and their implementation. Interestingly, observe that Lookahead-Minma, allows GDA to converge on this example, and moreover speeds up the convergence of ExtraGradient.
4.2.2 THE STOCHASTIC SETTING
In this section, we show that besides SVRE, Lookahead-minne also converges on equation SB G In addition, we test all the methods of Gidel et al. (2019a] $7.1) using minibatches of several sizes B 1, -1,16, 16, 64, and sampling without replacement. In particular, we tested: (i) the Adam method combined with GDA (shown in blue); (ii) ExtraGradient-Eq EG as well as (iii) ExtraAdam proposed by (Gidel et al. 2019a): (iv) our proposed method LA-GDA (Alg. M combined with GDA; results. See for details on the implementation and choice of hyperparameters. We observe that besides the good performance of LA-GDAA games in the batch setting, it also has the property to cope well large variance of the gradient estimates, and it converges without using restarting.
well as (v) SVRE (Chavdarova et al., 2019, Alg.1) for completeness. Fig. depicts
Adam
Extra-Adam
Extragradient
LA-GDA
SVRE
full-batch
64
B
40
40
40
40
Distance the optimum
30-
30
30
20
20
20
20
10-
10
10
10-
0-
0
500 0
20000 0
20000 0
20000
Number of passes
Figure 4: Convergence of Adam, ExtraAdam, Extragradient, SVRE and LA- GDA, on equation SB G for several minibatch sizes B, averaged over 5 runs with random initialization of both the parameters and the data points (A, b and c). Fast (4p, 0) and slow (sp, 0) weights of LA GDA are shown with solid and dashed lines, resp.
6

---

Published as a conference paper at ICLR 2021
Fréchet Inception distance 4
Inception score ?
(32x32) ImageNet
avg
EMA
EMA-slow
avg
EMA
EMA-slow
AlIGAN
15.63
.46
14.16
+ 27
7.23 +
7.81 +:
LA-AIIGNN
14.37
20
13.06
20
12.53 .06
7.58 =
7.97 = .11
8.42 + 11
NLA-AIIGNN
13.14 = .25
13.07
.25
12.71 = .13
7.85 + .05
7.87 = .01
8.10
ExtraGrad
15.48 +
14.15
7.31 =
7.85 +
LA-EEttaGrad
27 14.53
14.13
14.09 .28
7.62 + .06
7.70 +
7.89
NIA-FxttaGGrad
15.05 .96
14.79 +
13.88 +.45
7.39 + +12
7.48 + .12
7.76
CIFAR-10
AlIGAN
21.37
1.60
16.92 + 1.16
7.41 + .16
8.03 + .13
LA-AIIGAN
16.74 .46
13.98 + .47
12.67 + .57
8.05 + .43
8.19 = .05
8.55 + .04
ExtraGrad
18.49 +.49--99
15.47 1.82
7.61 + .07
8.05 + .09
LA-ExttaGrad
15.25 +.0
14.68 + .30
13.39 .23
7.99 + .03
8.04 + .04
8.40 .05
Unrolled-GAn
21.04 + 1.08
17.51 1.08
7.43 + .07
7.88 + .12
SVHN
AliGAN
7.84 + 1.21
6.83 + 2.88
3.10 + .09
3.19 + .09
LA-AIGGAN
3.87 + .09
3.28 .09
3.21 + 119
3.16 +
3.22 + .08
3.30 + .07
ExtraGrad
4.08 + .11
3.22
+ .99
3.21 + .02
3.16 + .02
LA- A-EExtrarrrd
3.20 .99
3.16 + .14
3.15 1.5::. 31
3.20 + .02
3.19 + .03
3.20 .04
Table 1: Comparison of LA-GAN with its respective baselines AltGAN and ExtraGrad (see 8 for naming). using FID (lower better) and IS (higher better), and best obtained scores. EMA denotes exponential moving average, see $ F All methods use Adam, see for detailed description. Results are averaged over runs. We run cach experiment for 500K iterations. See H and $ for details on architectures and hyperparameters and for discussion on the results, resp. Our overall best obtained FID scores are 12.19 on CIFAR- 10 and 2.823 on SVHN, $ O samples of these generators. Best
obtained for each metric and dataset highlighted yellow. For each column best score in bold along with any score within its standard deviation reach.
Unconditional GANs
Conditional GANs
SNGAN
GrOg.GAN
NCSN
WS-SVRE
ExtraAdam
LA-AlIGAN
SNGAN
BigGAN
Miyato et al. Karras et al Song & Ermon Chavdarova et al. Gidel et al.
(ours)
Miyato et al. Brock et al.
FID
21.7
25.32
16.77
16.78 + .21
12.19
25.5
14.73 9.22
IS
8,22
8.80 .05
8.87
8.47 : .10
8.78
8.60
Table 2: Summary of the recently reported best scores on CIFAR-10 and benchmark with LA GAN, using published results. Note that the architectures are not identical for all the methods see $ 5.2
EXPERIMENTS
In this section, we empirically benchmark Cookahead-minmax--ee for training GANs. For the purpose of fair comparison, as an iteration we count each update of both the players, see Alg.
5.1 EXPERIMENTAL SETUP
Datasets. We used the following image datasets: (i) MNIST (Lecun & Cortes 1998,
(ii) CI-
FAR-10 (Krizhessky) 2009 $3). (iii) SVHN (Netzer et al. 2011). and (iv) ImageNet ILSVRC
2012 (Russakovsky et 2015), using resolution 28 28 MNIST, and 32 x 32 for the rest.
Metrics. We use Inception score (IS, Salimans et al. 2016 and Fréchet Inception distance
(FID, Heusel al., (017) as these are most commonly used metrics for image synthesis, see $ H.] for details. On datasets other than ImageNet, IS is less consistent with the sample quality (see H.I.1).
DNN architectures. For experiments on MNIST, we used the DCGAN architectures (Radford) et al. 2016), described in & H.2.1 For SVHN and CIFAR-10, 10, we used the ResNet architectures, replicating the setup (Miyato et al. 2018 Chavdarova et al.) 2019), described in detail H.2.2
Optimization methods. We use: (i) AltGan: the standard alternating GAN, (ii) ExtraGrad: the
extragradient method, as well as (iii) UnrolledGANN (Metz et al.) 2017). We combine Lookahead- minmax with and and (ii) and we refer to these LA-AItGAN and L.-ExtraGrad, respectively
or for both as GAN for brevity. We denote nested LA-GAN with prefix NLA, see Š G..I All
methods in this section use Adam (Kingma & Bal 2015). We compute Exponential Moving Average (EMA, see def. in $ F of both the fast and slow weights-calll EMA and EMA-slow, resp. See Appendix for results of uniform iterate averaging and RAdam (Liu et al. 2020).
7

---

Published as a conference paper at ICLR 2021
5.2 RESULTS AND DISCUSSION
Comparison with baselines. Table 1 summarizes our comparison of combining Alg. with AltGAN and ExtraGrad. On all datasets, we observe that the iterates (column "no avg") LA- AltGAN and LA-ExtraGrra perform notably better than the corresponding baselines, and using EMA on LA-AlIGAN and LA-ExtraGrad further improves the FID and IS scores obtained with LA-AIIGAN. As the performances improve after each LA-step, see Fig. computing EMA solely on the slow weights further improves the scores. is interesting to note that on most of the datasets, the scores of the iterates of A-GANN (column no-avg) outperform the EMA scores of the respective baselines. In some cases, EMA for AltGAN does not provide improvements, as the iterates diverge relatively early. In our baseline experiments, ExtraGrad outperforms AtllAN-while requiring twice the computational time of the latter per iteration. The addition of Lookahead miniamax stabilizes AtlGAN making it competitive to -A-Extrrrrad while using half of the computational time.
In Table we report our results on MNIST, where- differrnnt from the rest of the datasets- the training of the baselines is stable, to gain insights if LA-GAN still yields improved performances. The best FID scores of the iterates (column "no avg") are obtained with LA-GAN. Interestingly, although we obtain that the last iterates are not LSSP (which could be due to stochasticity), from Fig. 5 --which depicts the eigenvalues of JVF we observe that after convergence LA-GAN shows no rotations.
avg
EMA
EMA-slow
ARGAN
Innginary Part
AlIGAN
.094 .006
.031 1 .002
LA-ARGAN
LA-AIIGAN
.053 + .004
.029 .002
.032 + .002
ExtraGrad
.094 = .013
.032 + .003
LA-ExtraGrad
.053 + .005
.032 + .002
.034 + .001
Unrolled-GAN
.077 + .006
.030 1 0002
0.5
Table 3: FID (lower is better) results on MNIST. averaged over
runs. Each experiment is trained for 100K iterations. Note that Unrolled MrrNlccd-GAiia is computationally more expensive: in the order of the ratio 4 22 we used 20 steps of unrolling what gave best results. See 5. for implementation and discussion, resp.
Real Part
Figure 5: Eigenvalues of v'(0. sp)
at
100K iterations on MNIST, see $ 5.2
20
33
2
LA-AIGAN
LA-AHGAN
LA-AIGAN
15
ARGAN
30
ARGAN
22
ARGAN
20
10-
25-
5-
20-
16-
0-
15
12
3
0
2
Treration
Treration
x1I
Iteration
x10
(a) SVHN dataset.
(b) CIFAR-10 dataset.
(c) ImageNet (32 x 32) dataset.
Figure 6: Improved stability LA- LAIAAA relative its baselines SVHN, CIFAR- 10 and ImageNet, over runs. The median and the individual runs are illustrated with ticker solid lines and with transparent lines.
respectively. See $15.2 for discussion.
Benchmark on CIFAR-10 using reported results. Table 2 summarizes the recently reported best obtained FID and IS scores on CIFAR-10. Although using the class Iabeis-Con GAN known to improve GAN performances (Radford al. 2016), we outperform BigGAN (Brock et al. 2019) on CIFAR-10, 10. Notably, our model and BigGAN have 5.1M and 158.3M parameters in total, respectively, and we use minibatch size of 128, whereas BigGAN uses 2048 samples. The competitive result of (Yazıcı et al. 2019) of average 12.56 FID uses ~ 3.5 times larger model than ours and is omitted from Table 2 as it does not use the standard metrics pipeline.
Additional memory & computational cost. Lookahead-minma requires the same extra memory footprint as EMA and uniform averaging (one additional set of parameters per player)-bott of which are updated each step whereas LA-GAN is updated once every k iterations.
On the choice of a and k. We fixed a = 0.5, and we experimented with several values of whilee once selected, keeping i fixed throughout the training. We observe that all values of & improve the baseline. Using larger k in large part depends on the stability of the base-optimizer: if it quickly diverges, smaller & necessary to stabilize the training. Thus we used k = 5 and t: 5000 for LA-
8

---

Published as a conference paper at ICLR 2021
AltGAN and LA ExtraaGadd, resp. When using larger values of ls we noticed that the obtained scores would periodically drastically improve every k iterations- after an LA step, see Fig Interestingly, complementary to the results of (Berard et al. 2020) locally analyze the vector field, Fig 2 confirms the presence of rotation while taking into account the used optimizer, and empirically validates the geometric justification of aakkahead-minmal illustrated in Fig The necessity of small k for unstable baselines motivates nested LA-Minmax using two different values of k, of which one is relatively small (the inner LA-step) and the other is larger (the outer LA-step). Intuitively, the inner and outer LA- steps in such nested LA-GAN tackle the variance and the rotations, resp.
Stability of convergence. Fig. 6 shows that LA GAN consistently improved the stability of its respective baseline. Despite using 1 5 update ratio for G D known to improve stability, our
baselines always diverge (also reported by [hhvddrovva et al., 2019 Brock et al., 2019). On the other hand, LA-GAN diverged only few times and notably later in the training relative to the same baseline, see additional results |
The stability further improves using nested LA Ltest as Fig
6 RELATED WORK
Parameter averaging. In the context of convex single-objective optimization, taking an arithmetic average of the parameters as by Polyyyk & Juditsky (1992); Ruppert (1988) is well-known to yield faster convergence for convex functions and allowing the use of larger constant step-sizes in the case of stochastic optimization (Dieuleveut et al.) 2017). It recently gained more interest in deep learning in general (Garipov et al. 2018), in natural language processing (Merity et al. 2018), and particularly uniform or exponential
in GANs (Yazieı et al. 2019) where researchers report the performance
moving average of the iterates. Such averaging post-processing after training is fundamentally different from immediately applying averages during training. Lookahead as of our interest here in spirit is closer to extrapolation methods (Korpelevich) 1976) which rely on gradients taken not the current iterate but at an extrapolated point for the current trajectory. For highly complex optimization landscapes such as in deep learning, the effect of using gradients perturbations of the current iterate has a desirable smoothing effect which is known to help training speed and stability in the case of non-convex single-objective optimization (Wen al. 2018 Haruki et al.) 2019).
GANs. Several proposed methods for GANs are motivated by the "recurrent dynamics". Apart from the already introduced works, (i) Yaday al.
(2018) use prediction steps, (ii) Daskalakis et al.)
(2018) propose Optimistic Mirror Decent (OMD)
(iii) Balduzzi et al. al. (2018) propose the Symplectic
Gradient Adjustment (SGA) (iv) Gidel et al. (211b) propose negative momentum, (v) Xu et al.)
(2020) propose closed-loop based method from control theory, among others. Besides its simplicity. the key benefit of LA-GAN is that it handles well both the rotations of the vector field as well as
noise from stochasticity, thus performing well on real-world applications.
CONCLUSION
Motivated by the adversarial component of games and the negative impact of noise on games, we proposed an extension of the Lookahead algorithm to games, called "Lookahead-minmax"" On the bilinear toy example we observe that combining Lookahead-minmax with standard gradient methods converges. and that Lookahead-minnt gradient estimates. Exponential moving averaging of the iterates is known help obtain improved performances for GANs, yet it does not impact the actual iterates, hence does not stop the algorithm from (early) divergence. Lookahead-miinet than rumning averages, and it is straightforward to implement. It can be applied to any optimization method, and in practice it consistently improves the stability of its respective baseline. While we do not aim at obtaining new oftte-of-the-are result, it is remarkable that Lookahead -ainddxxixxxxxxm obtains competitive result on CIFAR-10 10 of 12.19 FID FID- outperforrnng the 30-times larger BigGAN.
copes very well with the high variance of the
goes beyond such averaging, requires less computation
As Lookahead- sookahead-minmax uses two additional hyperparameters, future directions include developing adaptive schemes of obtaining these coefficients throughout training, which could further speed up the convergence of

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
This work was in part done while TC and FF were affiliated with Idiap research institute. TC was funded in part by the grant 100021-169112 from the Swiss National Science Foundation, and MP was funded by the grant 40516.1 from the Swiss Innovation Agency. TC would like to thank Hugo Berard for insightful discussions on the optimization landscape of GANs and sharing their code, as well as David Balduzzi for insightful discussions on n-player differentiable games.
REFERENCES
David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. The mechanics of n--laaee differentiable games. ICML, 2018.
Hugo Berard, Gauthier Gidel, Amjad Almahairi, Pascal Vincent, and Simon Lacoste-Julien. A closer look the optimization landscapes of generative adversarial networks. In ICLR, 2020.
Dimitri P Bertsekas. Nonlinear programming. Athena scientific Belmont, 1999.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
Ronald E Bruck. On the weak convergence of ergodic iteration for the solution of variational inequalities for monotone operators in hilbert space. Journal of Mathematical Analysis and Applications, 1977.
Tatjana Chavdarova, Gauthier Gidel, François Fleuret, and Simon Lacoste-Julien. Reducing noise in GAN training with variance reduced extragradient. In NeurIPS, 2019.
Constantinos Daskalakis, Andrew Iyas,, Vasilis Syrgkanis, and Haoyang Zeng. Training GANs with optimism. In ICLR, 2018.
Constantinos Daskalakis, Stratis Skoulakis, and Manolis Zampetakis. The complexity of constrained min-max optimization. 2oXiv:2009009623 2020.
Aaron Defazio and Léon Bottou. On the ineffectiveness of variance reduced optimization for deep learning. In NeurlPS, 2019.
Popoy Leonid Denisovich. A modification of the arrow-hurwiez method for search of saddle points. Mathematical Notes of the Academy of Sciences ofthe the USSR, 28(5):45-8888 1980.
Aymeric Dieuleveut, Alain Durmus, and Francis Bach. Bridging the gap between constant step size stochastic gradient descent and markov chains.
2017.
Francisco Facchinei and Jong-Shi Pang. Finite-Dimensional Variational Inequalities and Comple- mentarity Problems Vol I. Springer Series in Operations Research and Financial Engineering, Finite-Dimensional Variational Inequalities and Complementarity Problems. Springer- Verlag, 2003.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of DNNs. In NeurIPS, 2018.
Gauthier Gidel, Hugo Berard, Pascal Vincent, and Simon I Lacoste-Julien. variational inequality perspective generative adversarial nets. In ICLR, 2019a.
Gauthier Gidel, Reyhane Askari Hemmat, Mohammad Pezeshki, Rémi Le Priol, Gabriel Huang, Simon Lacoste-Julien, and loannis Mitliagkas. Negative momentum for improved game dynamics. In AISTATS, 2019b.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
Ian Goodfellow. NIPS 2016 tutorial: Generative adversarial networks. arXiv: /00100077000000000 2016.
10

---

Published as a conference paper at ICLR 2021
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Kosuke Haruki, Tajji Suzuki, Yohei Hamakawa, Takeshi Toda, Ryuji Sakai, Masahiro Ozawa, and Mitsuhiro Kimura. Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD. arXiv:1906. 1022,, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to local nash equilibrium. In NIPS, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICMI,, 2015.
Chi Jin, Praneeth Netrapalli, and Michael Jordan. What is local optimality in nonconvex-nonncomave minimax optimization? In JCML, 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian U. Stich. A unified theory of decentralized SGD with changing topology and local updates. In ICML, 2020.
Galina Michailovna Korpelevich. The extragradient method for finding saddle points and other problems. Malecon, 1976.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Master's thesis, 2009.
Yann Lecun and Corinna Cortes. The MNIST database handwritten digits. 1998. URL http:
7/yann. lecun htmm/exxbmmmi....
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In ICLR, 2020.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTM language models. ICLR, 2018
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of GANs. In NIPS, 2017.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin, Which Training Methods for GANs do actually Converge? In ICML., 2018.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. In ICLR, 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. ICLR, 2018.
Aryan Mokhtari, Asuman Ozdaglar, and Sarath Pattathil, A unified analysis of extra-gradient and op- timistic gradient methods for saddle point problems: Proximal point approach. arXiv: 1901.08/1, 2019.
Dov Monderer and Lloyd S Shapley. Potential games. Games and economic behavior, 1996.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits natural images with unsupervised feature learning. 2011. URI. http://ufldi stanford. www.edu//aa wwt.//ouseeuuberr/..
Shayegan Omidshafiei, Jason Pazis, Chris Amato, Jonathan P. How, and John Vian. Deep de- centralized multi-tass task multi-agent reinforeement learning under partial observability. In ICML, 2017.
11

---

Published as a conference paper at ICLR 2021
B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 1992. doi: 10.1137/0330046.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
Alexander Rakhlin and Karthik Sridharan. Online learning with predictable sequences. In COLT, 2013.
David Ruppert. Efficient estimations from slowly convergent Robbins-Monro process. Technical report, Cornell University Operations Research and Industrial Engineering, 1988.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJC,, 115(3):211-252 2015.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In NIPS, 2016.
Christopher Shallue, Jaehoon Lee, Joe Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George Dahl. Measuring the effects of data parallelism neural network training. Stti::8111.3600 2018.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurlPS, pp. 11895-11907, 2019.
Sebastian U. Stich. Local SGD converges fast and communicates little. In ICLR, 2019.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. arXiv: 1r.Xiv/5/2..00 /5l::0/6/2.05 2015.
Fiez Tanner, Chasnov Benjamin, and Ratliff Lillian. Implicit learning dynamics in stackelberg games: Equilibria characterization, convergence analysis, and empirical study. In ICML, 2020.
Ferdinand Verhulst. Nonlinear Differential Equations and Dynamical Systems. Springer-Verlag Verlag Berlin Heidelberg, 1990.
J Von Neumann and O Morgenstern. Theory of games and economic behavior. Princeton University Press. 1944.
J. Wang, V. Tantia, N. Ballas, and M. Rabbat. Lookahead converges to stationary points of smooth non-convex functions. In ICASSP, 2020.
Yuanhao Wang, Guodong Zhang, and Jimmy Ba. On solving minimax optimization locally: A follow-the-ridge approach. ICLR, 2020.
Wei Wen, Yandan Wang, Feng Yan, Cong Xu, Chunpeng Wu, Yiran Chen, and Hai Li. Smoothout: Smoothing out sharp minima to improve generalization in deep learning. arXiv:1805.0780) 2018.
Blake Woodworth, Kumar Kshitij Patel, and Nathan Srebro. Minibatch vs local SGD for heteroge- neous distributed learning. arXiv, 2020a.
Blake Woodworth, Kumar Kshitij Patel, Sebastian U. Stich, Zhen Dai, Brian Bullins, H. Brendan McMahan, Ohad Shamir, and Nathan Srebro. Is local SGD better than minibatch SGD? In ICML, 2020b.
Kun Xu, Chongxuan Li, Huanshu Wei, Jun Zhu, and Bo Zhang. Understanding and stabilizing gans' training dynamics with control theory. In ICML, 2020.
Abhay Yadav, Sohil Shah, Zheng Xu, David Jacobs, and Tom Goldstein. Stabilizing adversarial nets with prediction methods. In /CLR, 2018.
Yasin YazIeı, Chuan-Sheng Foo, Stefan Winkler, Kim-Hui Yap. Georgios Piliouras, and Vijay Chandrasekhar. The unusual effectiveness of averaging in GAN training. In ICLR, 2019,
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey Hinton. Lookahead optimizer: k steps forward, step back. In NeurlPS, 2019.
12

---

Published as a conference paper at ICLR 2021
A THEORETICAL ANALYSIS OF LOOKAHEAD FOR SINGLE OBJECTIVE MINIMIZATION PROOF OF THEOREM
In this section we give the theoretical justification for the convergence results claimed Section 3 in Theorem @aneemaaand. and Remark We consider the LA oooptmmizer, with A SGD prediction steps, that is,
With W. Σ g(
where g: X -> IR is a stochastic gradient estimator, Eg(w) = V((w), with bounded variance Bllg(w)]l o². Further, assume that f is L-smooth for constant L 0 (but not necessar- ily convex). We count the total number of gradient evaluations K. As is standard, we consider output of the algorithm a uniformly at random chosen iterate wont (we prove convergence of E| [([] (wl))) This matches the setting considered in (Wang et al. 2020).
Wang al. (0020 Theorem 2) show that if the stepsize 7 satisfies
on + (1 -
1
then the expected squared gradient norm of the LA iterates after '"" updates of the slow sequence, i.e. K = kT gradient evaluations, can be bounded as
o Fo (1Fx
Lo (1- (1-0)))222220 1)
where Fo =
== f(w)) Jine denotes upper bound the the optimality gap. Now, departing from Wang et al. (2020). we can directly derive the claimed bound in Theorem
by choosing
such to minimze equation 2 while respecting the constraints given equation
(two necessary
constraints are e.g. y2 S (1-0)-14R((-1) and S af.) For this minimization with respect to
y, see for instance (Koloskova et al. 2020 Lemma 15), where we plug in ro Fo = aLo?
= (1- a
and d = min aL (I-a)Lk)+
The improved result for the quadratic case follows from the observations in Woodworth et al. (2020b) who analyze local SGD on quadratic functions (they show that linear update algorithms (such as local SGD on quadratics) benefit from variance reduction, allowing to recover the same convergence guarantees as single-thin theeedd SGD. These observations also hold for non-convex quadratic functions.
We point out, that the the choice of the stepsize
VkT proposed in Theorem 2 in WWag et al. Whist
2020) does not necessarily satisfy their constraint given in equation I for small values their conclusions remain valid when & is a constant (or in the limit, T + oc), the current analyses (including our slightly improved bound) do not show that I.A can general match the performance upper bounds of SGD when & is large. Whilst casting [. as an instance of local SGD was useful to derive the first general convergence guarantees, we believe- in regard to recently derived lower bounds wWoodworth et al.) (2020a) that show limitations on convex functions -tht this approach is limited and more carefully designed analyses are required to derive tighter results for LA future works.
B PROOF OF THEOREM 3
Proof. Let pbuse (w) denote the operator that an optimization method applies to the iterates (0,4)) sp). Let w' be a stable fixed point of plast i.e. O(Vbbaseuss (w+)) < 1. Then, the op- crator of bookahead-MMimt algorithhm FLA (when combined with prase (.)) is:
FLA(W) w + a( & (w) w)
= = - a)w a abbsese (w),
with a E [0,1] and 2.
13

---

Published as a conference paper at ICLR 2021
Let À denote the eigenvalue of Jouse p( (xbuae(w")) = |A], let u be its associated eigenvector: joase u== xu.
V Fhose (w*) with largest modulus, i.e.
Using the fixed point property (F(w*) w*), the Jacobian of Lookahead
w" is thus:
J (FFAA(w)) = (1-a)1 + at aabbsso.k.
By noticing that:
JLău =(11 a)] + a(Juaseyh) =(11- - a) + aat))u
we deduce u is an cigenvector of JLA with eigenvalue - a + at=
We know that p(Jbase) max ( [Auese] } < 1 (by the assumption of this theorem). To prove Theorem 3 by contradiction, let us assume that: p(JLL) 2 1, what implies that there exist an eigenvalue of the spectrum of JTA such that:
AALA Spec(JLA) s.t.
IA -
ax> ((xbasee)) 1.
(3)
As [abase| < 1, we have [A basejjk [(Abaseyhy < Furthermore, the set of points {p Cla E [0, 11. tween (A (A and 1 + Oi (excluded from the segment). As both ends the segment are the unit circle, it follows that the left hand side of the inequality of equation 3 is strictly smaller than 1. By contradiction, this implies p(J < 1, hence it converges linearly.
a(Xbaseys) is describing segment in the complex plane be-
Hence, if w stable fixed point for some inner optimizer whose operator satisfies our assumption, then wn is a stable fixed point for Lookahead-Minmax as well.
C SOLUTIONS OF A MIN-MAX GAME
In $ we briefly mention the "ideal" convergence point of 2-player games and defined LSSP requieed to follow up our result in Fig. S For completeness of this manuscript, in this section we define Nash equilibria formally, and we list relevant works along this line which further question this
notion of optimality in games.
For simplicity, in this section we focus on the case when the two players aim minimizing and =
maximizing the same value function L == L, called zero-sum minimax game. More precisely:
min max L(0,
(ZS-G)
oc8 peb
where : R.
Nash Equilibria. For 2-player games, ideally we would like to converge to point called Nash equilibrium (NE). In the context of game theory, NE is combination of strategies from which, no player has an incentive to deviate unilaterally. More formally, Nash equilibria for continuous games defined as a point (yp*, ,*)) where:
L()", < (((",00) < 0*)
(NE)
Such points are (locally) optimal for both players with respect to their decision variable, i.e. no player has the incentive to unilaterally deviate from it.
Differential Nash Equilibria. In machine learning we interested in differential games where L twice differentiable, in which case such NE needs to satisfy slightly stronger conditions. A point (0+, up*) is Differential Nash Equilibrium (DNE) of zero-sum game iff:
||VoL(0", . = (0" bp*]| = 0,
V3c(0', .p') > 0, and
V20 x²)) x 0,
(DNE)
14

---

Published as a conference paper at ICLR 2021
where A 0 and A < 0 iff A is positive definite and negative definite, respectively. Note that the key difference between DNE and NE is that (-C(-) and V (()) for DNE are required to be definite (instead of semidefinite).
(Differential) Stackelberg Equilibria. recent line works questions the choice of DNEs as a game solution of multiple machine learning applications, including GANs (Tanner al. 2020 Jin et al.) 2020; Wang et al. 2020). Namely, as GANs are optimized sequentially, authors argue that a so called Stackelberg game is more suitable, which consist of leader player and a follower. Namely, the leader can choose her action while knowing the other player plays best-response:
Leader:
0
arg min
[(0, ((0,)):
arg max C(O, bp) PE#
0g
(Stackelberg- -aa
Follower:
arg max (("", :4)
Briefly, such games converge so called Differential Stackelberg Equilibria (DSE), defined as point (0*, (0*) where:
VeL :0)== = = e ,000)""")))0 0, and
(4)
V VoV
:0)) 0.
(DSE)
Remark. (Berard et al. 2020) show that GANs do not converge to DNEs, however, these con- vergence points often have good performances for the generator. Recent works on DSEs give game-theoretic interpretation of these solution concepts-which constitute superset of DNEs. More- over, while DNEs are not guaranteed to exist, approximate NE are guaranteed to exist (Daskalakis) et al. 2020). A more detailed discussion is out of the scope of this paper, and we encourage the interested reader to follow up the given references.
D EXPERIMENTS ON THE BILINEAR EXAMPLE
In this section list the details regarding implementation of the experiments on the bilinear example of equation SB-G that were presented in $ $4.2 In particular: (i) in š [.I] we list the imple- mentational details the benchmarked algorithms. (ii
we list the hyperparameters used in $ 1 and 4.2.2 respectively (iii) in $ D.5 we describe the choice of the x-axis of the
toy experiments presented the paper, and finally (iv) $D.6 we present visualizations in aim to
improve the reader's 's intuition on how aiktahhad-miimman works on games.
D.1 IMPLEMENTATION DETAILS
Gradient Descent Ascent (GDA). We use an alternating implementation of GDA where the players updated sequentially, as follows:
F1+ 9: (V)
0.+1 WVo 4p++1)
(GDA)
ExtraGrad. Our implementation of extragradient follows equation thus:
EG with L --LP(-),
(0+3 0, 1 (e
(0+1 -0:-n/Vo /V
Extrapolation:
Update:
[Pt+] = 4p: + v C(0,,
(9t+1 = = + 1/ V (c+))
(EG-ZS)
OGDA. Optimistic Gradient Descent Ascent (Denisovichi 1980 Rakhlin & Sridharan 2013
Daskalakis et al, 2018) has last iterate convergence guaranties when the objective is linear for both the min and the max player. The update rule is follows:
0.+1 0. 2/ + (Vo
(OGDA)
P::1 sp: V - 1
15

---

Published as a conference paper at ICLR 2021
Contrary to EG ZS
OGDA requires one gradient computation per parameter update, and requires
storing the previously computed gradient at t 1. Interestingly. both EG-ZS and [OGDA can be seen approximations of the proximal point method for min-max problem, see (Mokhtari et al. 2019) for details.
Unroll-Y. Unrolling was introduced by Metz et al. (2017) as way to mitigate mode col- lapse of GANs. It consists of finding an optimal max- max-player sp for fixed min-player 0, i.e. 5p (0) )+)= = arg max, [ sp) through "unrolling" as follows:
pi Pt
Pi (0) Pi v/V oL P("),
8 (0:
lim pi"
In practice m is finite number of unrolling steps, yielding spth The min-player 0,, e.g. the generator, can be updated using the unrolled spt". while the update of 4: is unchanged:
0+t1 02 Vo L (0t, spt
Pi+1 = 4: VV
(UR-X)
Unroll-XY. While Metz al. 2017) only unroll one player (the discriminator in their GAN setup), we extended the concept of unrolling to games and for completeness also considered unrolling both players. For the bilinear experiment we also have that L (0,, (p:) L (p:).
Adam. Adam (Kingma & Ba 2015) computes exponentially decaying average of both past gradients my and squared gradients it for each parameter of the model as follows:
m, = m + (1 - 31)g
(5)
U/ 32 + (1-32)9²²
(6)
where the hyperparameters 31, 82 E [0. 1 mo = 0, nn = 0. and denotes the iteration 1,...... 7.
my and ny respectively the estimates of the first and the second moments of the stochastic gradient.
To compensate the bias toward due to their initialization to me 0, vo = 0, Kingma & Ba 2015 propose use bias-corrected estimates of these first two moments:
mr 1 81
6 1
(8)
Finally, the Adam update rule for all parameters at t-th iteration wi can be described
Wi+1 1/ V0, +
(Adam)
Extra-Adam. Gidel et al. (2019a) adjust Adam for extragradient equation
EG and obtain the
empirically motivated ExtraAdam which re-uses the same running averages of equation Adam when computing the extrapolated point W & as well as when computing the new iterate w (see Alg.4, We used the provided implementation by the authors.
Gidel al. 2019a).
SVRE. Chavdarova et al. (201)) propose SVRE as way to cope with variance in games that may cause divergence otherwise. We used the restarted version of SVRE used for the problem of equation SSB G described in (Alg3, Chavdarova et al. 2019), which we describe in Alg 2 for completeness-we wheree do and dip denote "variance corrected gradient:
(0 0
(05) == thp + V L D Z(nz)) d0 s, 0S .65) = no L (nn))) VoL ss 2/mn)))
(OS, (0000.sss,
2(m=))
(9)
(10)
where 8 and S are the snapshots and μo and μu their respective gradients. D and 2 denote the finite data and noise datasets. With probability (fixed) before the computation of S and μ decide whether to restart SVRE (by using the averaged iterate the new starting point-Alg, Line 6-w,) or computing the batch snapshot at point w; For consistency, we used the provided implementation by the authors.
16

---

Published as a conference paper at ICLR 2021
Algorithm 2 Pseudocode for Restarted SVRE.
1: Input: Stopping time T, learning rates 1jo, lhp losses L and L probability of restart p. dataset
D. noise dataset 2, with [D] = 2 n.
2: Initialize: yp, 1 0
Di is for the online average computation.
3: for 7o
4: Draw restart ~ B(p).
Check if we restart the algorithm.
5: if restart and e 0 then
6:
- 4. + 0 and =1
7:
end if
g + and u - D i= Ve (0, s0o)
8:
9:
os - 0 and 10
- 2
zi== VeL. (OS) s0os)
10:
N ~ Geom (1/n)
D Length of the epoch.
11:
for to N do
12:
Sample no. upp
Te. do extrapolation:
13:
6 - nod sp 0 .65)) - Thp do, sp, 0s, (0.)
b equation
and equation 10
14:
Sample ig ~ To. ip ~ T.p do update:
15:
9 nod 0 3 0s 55)) + 0 The do :. 0s .0.) ,00)
>> equation
and equation
16:
0
- 4+1
0 + F
0 and
4 #I
Online computation of the average. D Increment f. for the online average computation.
17:
1 1
end for
19:
end for
Output: 0, 4
D.2 HYPERPARAMETERS USED FOR THE FULL-BATCH SETTING
Optimal a. In the full-batch bilinear problem, is possible derive the optimal a parameter for small enough Given the optimum w*, the current iterate w, and the "previous" iterate w before k steps. let " + a(w - >") be the next iterate selected be on the interpolated line between wP and w. We aim at finding (or in effect a)) that is closest to w* For an infinitesimally small learning rate, GDA iterate would revolve around w*, hence ||W will !| shortest distance between æ and wt would be according to:
will r. The
2 = llw
x1| + ||
w1 = || x0|2 + | w w11
Hence the optimal æ, for any k, would be obtained for ||W xi|| = ||m x||. which is given for 0.5.
In the case of larger learning rate, for which the GDA iterates diverge, we would have w will = ri || wi|l r2 as we are diverging. Hence the optimal would follow
I x < || æ) which is given for a 0.5. In Fig. 3 we indeed observe LA-GDA with a = 0.4 converging faster than with a = 0.5.
Hy perparammeeers.. Unless otherwise specified the learning rate used is fixed to 0.3. For both Unroll-Y and Unroll-XY, or Extragradient, we use l of 6 and and a 0.5 unless otherwise emphasized.
6 unrolling steps. When combining aakkaaaad-minna with GDA
D.3 PERFORMANCE EMA-ITERATES THE STOCHASTIC SETTING
For the identical experiment presented in $ 4.2.2 in this section depict the corresponding perfor- mance of the used are the same and shown in $ D.4 to the optimum as a function of the number of passes for each method. We observe the high variance of the stochastic bilinear is preventing the convergence of Adam, Adam and Extragradient,
of Adam, Extra-Adam, Extragradient and LA-GDA. The hyperparameters 0.999 for EMA. In Fig, we report the distance
17

---

Published as a conference paper at ICLR 2021
when the batch size B is small, despite computing exponential moving average of the iterates 20k iterations.
RMA-Adam
FMA Extra-Adaan
EMA Extragradient
EMA LA-GDA
full-batch
B
64
B
16
B
40
40
40
Distance to the optimum
30
30-
30
30
20
20
20
20
10
10
0- 0
0-
0%
20000
20000 0
20000
20000
Number of passes
Figure 7: Distance to the optimum as function of the number of passes, for Adam, Extra-Adam, Extragradient, and LA-GAN, all combined with EMA (3 0.999). For small batch sizes, the high variance of the problem preventing convergence for all methods but Lookahead-Minmar
D.4 HYPERPARAMETERS USED FOR THE STOCHASTIC SETTING
The hyperparameters used in the stochastic bilinear experiment of equation SB G are listed in Table [ We tuned the hyperparameters of each method independently, for each batch-size. We tried y ranging from 0.005 to When for all values of 1 the method diverges, we set = 0.005 in Fig. [4] To tune the first moment estimate of Adam 31, we consider values ranging from -1 to 1, as Gidel et al. reported that negative 31 can help in practice. We used a E {0.3, 0.5) and & E [5, 3000].
Batch-size
Parameter
Adam Extra-Adam
Extragradient 0.8
LA-GDA
SVRE
?) Adam 31
0.005
0.02
0.2
-0.9
-0.6
full-batch
Lookahead le
15
Lookahead a
0.3
0.005
0.01
0.005
0.005
Adam $1
-0.6
-0.2
64
Lookahead k
450
Lookahead
0.3
0.005
0.005
0.005
0.01
Adam 31
-.3
0.0
16
Lookahead &
1500
Lookahead a
0.3
" Adam 31
0.005
0.005
0.005
0.05
0.1
0.0
0.0
Lookahead k
2450
Lookahead a
0.3
restart probability p
0.1
Table 4: List of hyperparameters used in Figure 4 and & in Alg.
1 denotes the learning rate, 3n is defined in equation and
Fig. 8 depicts the final performance of Lookahead-minman using different values of k. Note that, the choice of plotting the distance to the optimum at a particular final iteration is causing the frequent oscillations of the depicted performances. since the iterate gets closer to the optimum only after the "backtracking" step. Besides the misleading oscillations, one can notice the trend of how the choice of l affects the final distance to the optimum. Interestingly, the case of B 16 in Fig. 8 captures the
18

---

Published as a conference paper at ICLR 2021
30
0.05, B
Final distance to the optinam
25
") = 0.01, B
20
15-
MRA Meiyy www My
10-
2000
4000
6000
8000
10000
Figure 8: Sensitivity of LA-GDA to the value of the hyperparameter &s in Alg. for combinations of batch sizes and y). The y-axis the distance to optimum 20000 passes. The jolting curves is due to final value being affected by how close it is to the last equation LA step, i... lines 10 and of Alg.
periodicity the rotating vector field, what sheds light on future directions in finding methods with adaptive k.
D.5 NUMBER OF PASSES
In our toy experiments, as x-axis we use the "number of passes" as in (Chavdarova et al. 2019), so as to account for the different computation complexity of the optimization methods being compared. The number of passes is different from the number of iterations, where the former denotes one cycle of forward and backward passes, and alternatively, can be called "gradient queries". More precisely, using parameter updates as the x-axis could be misleading as for example extragradient uses extra passes (gradient queries) per one parameter update. Number of passes thus a good indicator the computation complexity, wall-clock time ceessremeentswwhie depend on the concurrent overhead of the machine at the time of running, can be relatively more noisy.
D.6 ILLUSTRATIONS OF GAN OPTIMIZATION WITH LOOKAHEAD
In Fig. consider 2D bilinear game min, maxy T y. and we illustrate the convergence of Lookahead-Minman Interestingly, Lookahead makes use of the rotations of the game vector field caused by the adversarial component of the game. Although standard-GDA diverges with all three shown learning rates, Lookahead-minnax converges. Moreover, we see Lookahead-minmax with larger learning rate of 7) = 0.4 (and fixed and a) in fact converges faster then the case 7) 0.1, what indicates that Lookahead- mineeaxxmmmmmxxx is also sensitive to the value of besides that it introduces additional hyperparameters and
E EXPERIMENTS ON QUADRATIC FUNCTIONS
In this section experiments on the following 2D quadratic zero-sum problems:
C(x,y) -3x²2 2 + 4xy y2
(QP-1)
C(x,8) 3) x² 5xy y²
(QP-2)
E. EXPERIMENTS ON EQUATION QP-1
In Fig. 10 we compare GDA, LA-GDA, ExtraGrad, and LA-Ea ExtraGrad on the problem defined by
equation QP- One can show that the spectral radius for the Jacobian of the GDA operator is always
19

---

Published as a conference paper at ICLR 2021
0.1, 0.5, k
0.4, 0.5,
1.5, 0.5, k:
55
2-
4
0
-2
-2
-2
-3
-3 -2
-3 -2
10
-2
2
Figure 9: Illustration of Lookahead-minnax the bilinear game min, max, for different values of the learning rate {0.1, 0.4, 1.5], with fixed & and 0.5. The trajectory of the iterates is depicted with green line, whereas the the interpolated line between (w,, (,,)), ................................ T. with wi = (0:, shown with dashed red line. The transparent lines depict the level curves of the loss function, and w = (0.0). See $ D.6 discussion.
larger than one when considering single step size 17 shared by both players and y. Therefore.
we tune each method by performing a grid search over individual step sizes for each player Th and "ly We pick the pair (7e Myy which gives the smallest spectral radius. From Fig. 10 we observe that Lookahead Minmax provides good performances.
10
10-26
Distance to the optimmm
10 46
10 66
10 10-86
10 106
GDA
10 126
LA-GDA
ExtraGrad
10-146
LA- ExtraGrad
20
40
60
80
100
Number of passes
Figure 10: Convergence of GDA, LA-GDA ExtraGrad, and LA-ExtraGrad on equation QP-1
E.2 EXPERIMENTS ON EQUATION QP-2
In Fig. 11 we compare GDA, LA-GDA, ExtraGrad, and LA-ExtraGrad on the problem defined by equation QP-2, This problem better conditioned than equation [P-1] and one can show the spectral radius for the Jacobian of GDA and EG are smaller than one for some step size shared by both players. In order to find the best hyperparameters for each method, we perform grid search the different hyperparameters and pick the set giving the smallest spectral radius. The results are corroborating our previous analysis as we observe a faster convergence when using
20

---

Published as a conference paper at ICLR 2021
10-10
10-35
Distance to the optimmm
10 10-60 60
10-85
10
10 10-135 133
GDA LA-GDA ExtraGrad
10-160
10 185
LA- Exttrrarraa
20
40
60
80
100
Number
passes
Figure 11: Convergence GDA, LA-GDA, ExtraGrad, and LA-ExtraGrad equation QP-2
F PARAMETER AVERAGING
Polyak parameter averaging was shown to give fastest convergence rates among stochastic gradient algorithms for convex functions, by minimizing the asymptotic variance induced by the algorithm (Polyak & Juditsky 1992). This, so called Rupper-Polyak averaging, computed as arithmetic average of the parameters:
T 0 RRP T E
1.
(RP-Avg)
In the context of games, weighted averaging proposed by Bruck (977) as follows:
Lt= 0()
(T) WA
(W-Avg)
ΣT= p(0)
Eq. equation W-Avg can be computed efficiently online as: ( (1 - ((0))(())))))))))))))))))))))) + 0000 with 7 [0, With 7 we obtain the Uniform Moving Averages (UMA) whose performance reported in our experiments in $ and is computed as follows:
0 Uuma (1 (
((()))
t 11 L.....T T.
(UMA)
Analogously, we compute the Exponential Moving Averages (EMA) in an online fashion using 7 1 - 3 < 1, as follows:
(t-1) @ (0((-1) + (1 -1- ())())) t= 1-
(EMA)
our experiments, following related works (Yazıcı et al. 2019 Gidel et al. 2019a Chavdarova et al. 2019). we fix B = 0.9999 for computing equation E.M.A all the fast weights and when computing on the slow weights in the experiments with small & (e.g. when = 5). The remaining case of computing equation EMA on the slow weights of the experiments that use large & produce only few slow weight iterates, thus using such large 3 0.9999 results in largely weighting the parameters at initialization. We used fixed 0.8 for those cases.
21

---

Published as a conference paper at ICLR 2021
G DETAILS ON THE LOOKAHEAD-MINMAN ALGORITHM AND ALTERNATIVES
As Lookahead uses iterates generated by an inner optimizer also refer it is meta-optimizer or wrapper. In the main paper we provide general pseudocode that can be applied to any optimization method.
Alg. reformulates the general Alg. [ when combined with GDA, by replacing the inner fast- weight loop by modulo operation of t-to clearly demonstrate to the reader the negligible modification for LA-MM of the source code of the base-optimizer. Note that instead of the notation of fast weights
0, c we snapshot networks 0 S (past iterates) to denote the slow weights, and the parameters
0, pp for the fast weights. Alg. 3 also covers the possibility of using different update ratio r E
for the two players. For fair comparison, all our empirical LA-GAN results use the convention
of Alg. L.e. as one step we count one update both players (rather than one update of the slow weights- it is in Alg. T.
Algorithm 3 Alternative formulation of Lookahead- (equivalent [ GDA)
1: Input: Stopping time T, learning rates 1e Tip. initial weights 0, sp lookahead hyperparameters k and a, losses L L update ratio r, real-data distribution pa, noise-data distribution ps. (store snapshots)
2: 0 8
0,4, ip
3: for 1..... do
4:
for E 1, ..... do
5:
x ~ Pd. ~ Pz
6:
6 -= 4
npp (0,
(update up times)
7:
end for
8:
z P=
9:
O = 0 V L ( p.
(update O once)
if t%
0 then
11:
+0 1 (4 8)
(backtracking interpolated line up$ ()
12:
0 0s
+ a0 0$)
(backtracking on interpolated line 0S 0)
gs
S
0 sp
(update snapshots)
14:
end if
end for
16: Output: 0 608
Alg. 5 shows in detail how Lookahead-minmax can be combined with Adam equation Adam Alg shows in detail how Lookahead-minmar can be combined with Extragradient. By combining equa- tion Adam with Alg. (analogous to Alg. 5). one could implement the LA--ExtaGraddAA algorithm, used in our experiments MNIST, CIFAR- 10, SVHN and ImageNet. A basic pytorch implementation in Listing.
Minama, including computing EMA on the slow weights, is provided
22

---

Published as a conference paper at ICLR 2021
from collections import defaultdict import torch import copy
class Lookahead (tokh optim. Opt 0ptocc..ppt....ptmmmaa::::
def _nint___sef,, optimizer, alpha-0.5):
self.optimizer optimizer
self.alpha alpha
celi.paran_groupe self. optimizer. param_group
self. defauitdict (dict)
def Lookahead_step(seil1
for group www..param_graeeeee for fast in group["params"] param_state stl..state(fas
if "slow_parame" not paran_state:
param__ttatee""tooo_parrem"""""" torch zeros wwwosrlikoseeeeeeeeeeaaaaeeeelee (fast. data)
(fast.dat))
slow = param_staate(rsloo__aaaa)))
slow siooraappha-<fatt---
slow (fast. slow) self.alpha fast.data.copy_(s111
def step (self, closure-None)
loss halffoptimizer..tte.m (closure) return loss
def update_ema_gen (G, G_ema, beta_ema=0.999 9999)
param 1ist parameters )))
I __em_pparam 1ist coo_ema-paramttrrss// for in range (len (1 (1_pparm))): with torch. corch.no_ggra()::
__ema_param data. (1_ema_____e [______paraam]]]] .aa.a
data. mul (beta_em) _mma) addd [1_paaia data. (1-beta ema)
def (r, D, optimizerD, optimizerG, mata_sampler, noise_ sampler, iterations=100, 5, beta_ema- 9999, Iookahead
$ Wrapping the optimizers with the lockahead optimizer optimizerD = Lookahead (optimizerD) Alpha=llokaahedd_alpa
optimizerG = Lookahead (optimizers, alpha-lookahead _appha alpha)
G_ema None
for in range (iterations):
Update discriminator and generator
di alsriminattorsste (D, opt imizz,, data sata_sampler, noise_sampler) generator_steplee optimizerG, noise_ samplerr
if (i+1) $ lookatead == 0: # Joint lockahead update optimizerG. lookahead_step optimizerD. lookahead_step if G_ema is None:
G_ema = copy deepcopy (G) else:
Update EMA the slow weights
update_cma_ (G, G_ema, pota_ema-beta_ema)
return G_mma
Listing I: Implementation Lookahead minmax in Pytorch. Also includes the computation of EMA on the slow weights.
23

---

Published as a conference paper at ICLR 2021
Unroll-GAN Vs. Lookahead-Minmas (LA-MM). LAGAN differs from UnrolledGAN (and approximated UnrolledGAN which does not backprop through the unrolled steps) as it: (i) does nor fix player update the other player k times. (ii) at each step t, it does not take gradient at future iterate 1. + k to apply this gradient at the current iterate (as extragradient does too). (iii) contrary UnrolledGAN, LA-MM uses point line between two iterates, closeness to which is controlled with parameter hence deals with rotations in a different way. Note that LA-MM can be applied to
UnrolledGAN, however the heavy computational cost associated UnrolledGAN prevented us from running this experiment (see note in Table 3 for computational comparison).
Algorithm Martaahhhd--iimaaa combined with Extragradient (equivalent to Alg. 1 FG)
1: Input: Stopping time T, learning rates 17. Th. initial weights 0, s, lookahead hyperparameters
l: and a, losses L L update ratio real-data distribution pa. noise distribution (store snapshots)
2: 0 50. < 0. &p
3: for 1etE1,....
do
4: Genro 40
5: for E ....,,r.d do
6:
2 ~ Pd. z Ps exttre extra
7:
Tip V L9
L (8,
eau x, 2)
(Compute the extrapolated up)
8:
end for
9:
P= Geara
10:
1 110 V (0,4,))
(Compute the extrapolated 0)
11:
for
E 1 ....., do
12:
Ps
60 = 6
Thp v. P eexnn 40, x, 2)
(update 4p times)
14:
end for
z P=
16:
O = 0
V Co ( aam z)
(update O once)
17:
if 1%
0 then
18:
8 + 10pp (4p )
(backtracking interpolated line up sp) (backtracking on interpolated line 0 O) (update snapshots)
19:
0 0
+ (0 0S)
20:
0s s + 0, 4p
21: end if
22:
end for
23: Output: 08, 8
24

---

Published as a conference paper at ICLR 2021
Algorithm 5 Lookahead (Alg- combined with Adam inner optimizer.
Input: Stopping time T' learning rates Te, Thp, initial weights 0, s, lookahead hyperparameters k and a, losses L LV, update ratio r, real-data distribution ps, noise-data distribution pa Adam parameters m vf, m" v" $1, 52.
2: 05 30000 0.4
(store snapshots)
3: m 19 m 2 t- 0,0,0.
(initialize first and second moments for Adam)
4:
for 1, do
5:
for E
do
6:
2 ~ Pd. = ~ ps
7:
9 t V p (0, 4,
8:
m = mmm + (1 gg
9:
UP 0 + +(1 1 32) ())
m
m
0 --1
11:
8/P
1- 3/ [|x
12:
= 4
Thp V0
(update r times)
13:
end for
14:
2 P=
15:
9d
V LL
0 40 x)
16:
m
9
31m
+ (1 81 81
17:
0 0 + (1 8)
18:
m m
I-3
19:
6 I-
20:
0 0
m
(update O once)
+e
21:
if t% = 0 then
22:
Φ 8 + 0ipp (4 $
(backtracking interpolated line S (p)
23:
0 0
+ ( 05)
(backtracking interpolated line 0 0)
24:
0
4
0, 4p
(update snapshots)
25:
end if
26:
end for
27: Output: 0 S
G.1 NESTED LOOKAHEAD-MINMAN
In our experiments explored the effect of different values of k. On one hand, observed that less stable baselines such as Alt-GAN tend to perform better using small & (e.g. k = 5) when combined with Lookahead-Minmas as it seems be the necessary to prevent early divergence (what in turn achieves better iterate and EMA performances). On the other hand, we observed that stable baselines combined with Lookahead-Minmax with large (e.g. 10000) tend take better advantage of the rotational behavior of games, as indicated by the notable improvement after each LA-step, see Fig. 2 and Fig. 12
This motivates combination of both small "slow" k; and a large "super slow" ke.- as shown in Alg. In this so called "nested" Lookahead mookahead-minimax copies of the each player, one corresponding to the traditional slow weights updated every k, and another for the so called "super slow" " weights updated every liss. When computing EMA on the slow weights for NLA-GAN methods we use the super-slow weights they correspond to the best performing iterates. However, better results could be obtain by computing EMA on the slow weights EMA performs best with larger parameter and averaging relatively small number of super-slow weights). We empirically find Nested Lookahead -inemmmm to be more stable than its non-nested counterpart, Fig
with NLA prefix, we store two
many iterates (whereas obtain
25

---

Published as a conference paper at ICLR 2021
8.0
7.5
2 7.0
6.5
Fast. weights
Slow weights
0
3
Iteration
x10
Figure 12: IS (higher is better) of LA-GAN ImageNet with relatively large & 10000. The backtracking step is significantly improving model's performance every 10000 iterations. This shows how a large can take advantage the rotating gradient vector field.
26
LA-ExtraGrad
24
GLrlExtraGrul
ExtraGrad
FID
20
16
2
5
Iteration
x x10F
Figure 13: Ex-txtrrrr,,, NLA ExttraGrra and Extragrad models trained ImageNet. For LA- LA-Extragrad 5000), the lighter and darker colors represent the fast and weights respectively. For NLA ExtraGrad (k, = 5, k.. = 5000), the lighter and darker colors represent the fast and super-slow weights respectively. In our experiments on ImageNet, while LA-ExtraGrad is more stable than ExtraGradd it still has a tendency to diverge early. Using Alg. 6 we notice clear improvement in stability.
G.2 ALTERNAING-LOOXAIIADD-MMAAA
Lookahead Vs. Lookahead-minnax. Note how Alg. I differs from applying Lookahead to both the players separately. The obvious difference is for the case 1, as the backtracking is done at different number of updates of &p and O. The key difference is in fact that after applying equation LA one of the players, do not use the resulting interpolated point update the parameters of the Entennating-Lookhhaaa
other player-a version we refer to applied to both the players the same time, which we found that outperforms the former. Unless otherwise emphasized,
see G Instead, equation [.A is
focus the "joint" version, described in Alg.
For completeness, in this section we consider an alternative implementation of Lookahead-minmax, which naively applies equation LA each player separately, which we refer to "alternating- lookahead".. This turn uses "backtracked" iterate to update the opponent, rather than performing the "backtracking" step at the same time for both the players. In other words, the fact that line of Alg. is executed before updating 0 in line 14 and vice versa, does not allow for Lookahead to help deal with the rotations typical for games.
26

---

Published as a conference paper at ICLR 2021
Algorithm 6 Pseudocode of Nested Lookahead-Minnal
Input: Stopping time T, learning rales 17., Thp, initial weights 0, sp, lookahead hyperparameters kas has and a, losses _ LV, update ratio r, real-data distribution pa. noise-data distribution ps.
(0,, 0s. Ps: Pss) (0.0,p.4p)
(store copies for slow and super-slow)
3:
f I..... do
4:
for E , do
5:
Pd. P=
6:
4 + 4
Thp V V o (0, 2,
(update up r times)
7:
end for
8:
z ~ p=
9:
0 0 110 V L sp
(update O once)
10:
if t% = 0 then
6 < 0. + oo o0 0.)
9, 02p 9 4p:)
(backtracking on interpolated line ysx, (p) (backtracking on interpolated line 8, 6) (update slow checkpoints)
12:
(0., px) (0.4)
14:
end if'
if
t%
then
16:
a8 + a
pos)
(backtracking on interpolated line s... sp)
.. + ag ma)
(backtracking on interpolated line a... 0) (update super-slow checkpoints) (update slow checkpoints)
18:
(0a, psa) - (0,00 4p)
O0, )x) - (0,4) 4
20:
end if
21:
end for
22: Output: 0ss. P8
Algorithm 7 Alternating Lookahead- minmaax pseudocode.
1: Input: Stopping time T, learning '10, 'pp initial weights 0, 4,, ho, kip, 00, aip. losses L C update ratio r, real-data distribution Pa. noise-data distribution ps. (store copy)
2: 0 0+0
3: 6 y
4: fort E O.... T do
5: for 1...., do
6:
Pd. Pz
7:
+ 4 Tkp Vo L (0) 40, x. 2)
(update up l times)
8:
if (t*r + i)%oki
0 then
9:
t 5 + asp (4 )
(backtracking on line i. s)
10:
5- 4
11:
end
end for
13:
z ~
14:
O 0 110 V VoL (0,40)))
(update O once)
15:
if 1%kg = then
0 0 + ag 0)
(backtracking on line 0, 0)
17:
5-6 0
18:
end if
19:
end for
20:
Output: 0, 40
On SVHN and CIFAR- 10, the joint Lookahead-minng consistently gave the best results, be seen in Figure 14 and [5] On MNIST, the alternating and joint implementations worked equally well, Figure 15
27

---

Published as a conference paper at ICLR 2021
30.0
30.0
ALI-AA-AIGAN
Joint-LA SaanttLA---rraatee ExmraGrrd (K 5000)
27.5
LeaI-LA-
Joint- I.A. A-Fexrrarrd (k 1000)
A M-I.A. Extrrrrrd A-I.A-FExttClii
1030)
25.0
22.5
22.5
20.0
20.0
17.5
17.5
15.0
25000
500000
150000
300000
Iteration
Iteration
(a) LA-AIIGAN.
(b) LA LA-ExtraGrad.
Figure 14: Comparison between different extensions of Lookahcad to games on CIFAR-IO. We use prefix joint and ait to denote Alg. and Alg. respectively of which the former is the presented in the main paper. We can see some significant improvements in FID when using the joint implementation, for both LA-AIIGAN (left) LA-ExtraGrad (right).
12
10
AIcLA-ARGAN
AltGan Alt-LA-AAttam
10
JOLGA-AA-AGGAN
Unrollel-GAN
ExtraGrrad
FID
FID
24
250000
500000
20000
40000
G0000
100000
Iveration
Iteration
(a) SVHN dataset.
(b) MNIST dataset.
Figure 15: (a): Comparison of the joint Lookahead-minnax implementation (Joint prefix, Algorithm I alternating Lookahead-minns. implementation (Alt prefix, see Algorithm 7 on SVHN dataset. (b): Results obtained with the different methods introduced in $5 as well as an alternating implementation of Lookahead miieaaaaddmiimax.... the MNIST dataset. Each curve is obtained averaged over runs. The results of the alternating implementation differ very little from the joint implementation, the curve for Alt-LA-AIGAN matches results in Table
28

---

Published as a conference paper at ICLR 2021
Figure 16: Samples from our generator model with the highest IS score. We can clearly see some unrealistic artefacts. We observed that the IS metric docs not penalize these artefacts, whereas FID does penalize them.
H DETAILS ON THE IMPLEMENTATION
For our experiments, used the PyTorel deep learning framework. For experiments CIFAR-10 and SVHN, we compute the FID and IS metrics using the provided implementations Tensorflow for consistency with related works. For experiments ImageNet use faster-running Py Torch implementation of FID and IS by (Brock et al. 2019) which allows for more frequent evaluation.
H.1 METRICS
We provide more details about the metrics enumerated in $ Both FID and use: (i) the Inception v3 network (Szegedy et al. 2013) that has been trained on the ImageNet dataset consisting of ~ million RGB images of 1000 classes, C' 1000. (ii) sample of generated images æ Pg- where usually m 50000.
H.1.1 INCEPTION SCORE
Given an image x, IS uses the softmax output the Inception network p(y which represents probability that is of class q, i 1... C, i.e., p(y|æ) [0, 1J It then computes the marginal class distribution p(y) = fg p(yl IS measures the the predicted conditional label distribution p(y)a) æ) and the marginal class distribution p(y). More precisely, it is computed as follows:
divergence Dx between
m
C
plye (x,) log P(uelara)) p(ve)
IS( cxp (Ezwp, P (pL((())))
cxp
Σ
Σ
(11)
m
aims at estimating (i) if the samples look realistic i.e., p(y[z) should have low entropy, and (ii) the samples diverse (from different ImageNet classes) i.e., p(y) should have high entropy. As these are combined using the Kullback-Leibler divergence, the higher the score is, the better the performance. Note that the range of IS scores convergence varies across datasets, the Inception network is pretrained on the ImageNet classes, For example, we obtain low IS values on the SVHN dataset as a large fraction of classes are numbers, which typically do not appear in the ImageNet dataset. Since MNIST has greyscale images, we used a classifier trained on this dataset and used m 5000. For CIFAR-14 and SVHN, we used the original implementation" of IS in TensorFlow, and 50000.
As the Inception Score considers the classes as predicted by the Inception network, it can prone not penalize visual artefacts as long as those do not alter the predicted class distribution. In Fig. T6 we show some images generated by our best model according to IS. Those images exhibit some visible unrealistic artifacts, while enough of the image is left for to recognise potential image label. For this reason we consider that the Fréchet Inception Distance is a more reliable estimator of image quality. However, reported IS for completeness.
https::/ppttorch.orrg/
29

---

Published as a conference paper at ICLR 2021
Generator
Discriminator
Input: R 128 (0,
Input: E R 1x28x22
transposed conv. (ker. 3x3, 128 512; stride:
conv. (ker: 4x4,1-> 64; stride: 2: pad:1)
Batch Normalization
LeakyReL. (negative slope: 0.2)
ReLU
conv. (ker: 4x4, 64 128; stride: 2: pad:1)
transposed conv. (ker: 4x4, 512 256,
stride:
Batch Normalization
Batch Normalization
LeakyReLU (negative slope: 0.2)
ReLU
conv. (ker: 4x4, 128 - 256; stride: 2; pad:1)
transposed conv. (ker: 4x4, 256 128, stride:
Batch Normalization
Batch Normalization
LeakyReLU (negative slope: 0.2)
ReLU
transposed conv. (ker: 4x4, 128 1, stride: 2, pad: 1)
conv. (ker: 3x3, 256 - 1; stride: 1)
Sigmoid(()
Tanh(.)
Table 5: DCGAN architectures (Radford et al. 2016) used for experiments on MNIST. We use ker and pad to denotc kernel and padding for the (transposed) convolution layers, respectively. With hx we denote the kemel size. With Con Yout we denote the number of channels of the input and output, for (transposed) convolution layers.
H.1.2 FRÉCHET INCEPTION DISTANCE
Contrary IS, FID aims comparing the synthetic samples Py with those of the training dataset & ~ Pa in feature space. The samples are embedded using the first several layers of the Inception network. Assuming Pg and Pa are multivariate normal distributions, it then estimates the means m, and m. and covariances C and Ca, respectively for and Pa in that feature space. Finally, FID is computed as:
Dri((a. xe d' ((ma, Ca), (mg. C2)) = mm m, Tr(Ca + +C, ((C,C)) (12)
where d' denotes the Fréchet Distance. Note that as this metric is a distance, the lower it is, the better the performance. We used the original implementation of FIL in Tensorflow, along with the provided statistics of the datasets.
H.2 ARCHITECTURES & HYPERPARAMETERS
Description of the architectures. We describe the models we used in the empirical evaluation of Lookahead-minmax by listing the layers they consist of, as adopted in GAN works, e.g. (Miyato ct al. 2018). With "conv." we denote a convolutional layer and "transposed conv" transposed convolution layer (Radford et al.) 2016). The models use Batch Normalization (loffe & Szegedy and Spectral Normalization layers (Myyato et al. 2018).
2015
H.2.1
ARCHITECTURES FOR EXPERIMENTS ON MNIST
For experiments the MNIST dataset, used the DCGAN architectures (Radford al. 2016).
listed in Table and the parameters of the models are initialized using PyTorch default initialization. For experiments on this dataset, we used the non saturating GAN loss proposed (Goodfellow et al.
2014):
CD = Ea-py log (D(x)) + Ez-p, log(D(G(z)))
(13)
LG E log ()G()))))
(14)
where pa and pa denote the data and the latent distributions (the latter to be predefined).
H.2.2 RESNET ARCHITECTURES FOR IMAGENET, CIFAR-10 AND SVHN
We replicate the experimental setup described for CIFAR-10 and SVHN in (Miyato et al. 2018 Chavdarova et al.) 2019), as listed in Table 7 This setup uses the hinge version of the adversarial non-saturating loss, see (Miyato et al. 2018). As reference, our ResNet architectures for CIFAR-10 have approximately 85 layers-in total for G and D, including the non linearity and the normalization layers.
https::/gittub..cmm
30

---

Published as a conference paper at ICLR 2021
D-ResBlock (6-th block)
G-ResBlock
Bypass:
Bypass:
[AvgPool (ker:2x2 )}, if
Upsample(x2)
(ker: 3 / 128; stride: 1)
Spectral Normalization
Feedforward:
AAvgPoo (kcr:2 x2, stride:2) (f #1
Batch Normalization
Feedforwand:
ReLU
ReLU # conv. (ker: 3x3, 3 1 + 128; stride: pad: Spectral Normalization
Upsample(x2)
conv. (ker: 3x3, 256 256; stride: 1: pad:
Batch Normalization
ReLU
ReLU
conv. (ker: 3x3, 128 128; stride: 1; pad: 1)
conv. (ker: 3x3, 256
256;
stride: 1; pad: 1)
Spectral Normalization
AvgPool (ker:2x2)
Table 6: ResNet blocks used for the ResNet architectures (see Table 7). for the Generator (left) and the Discriminator (right). Each ResNet block contains skip connection (bypass), and sequence of convolutional layers, normalization, and the ReLU non linearity. The skip connection of the ResNet blocks for the Generator (left) upsamples the input using factor of 2 (we the default PyTorch upsampling algorithm-nearest neighhor), whose output then added to the one obtained from the ResNet block listed above. For clarity we list the layers sequentially, however, note that the bypass layers operate in parallel with the layers denoted as "feedforward" (He et al. (016). The ResNet block for the Discriminator (right) differs if it is the first block in the subsequent one, l
network (following the imput the Diseriminator), l the ReLU non con-riieyr immediate on the input.
avoid performing
Generator
Discriminator
Input: z E R228 ~ N((,7))
Input: x E R3x32x32
Linear(128 - 4096)
D ResBlock
G-ResBlock
D ResBlock
G G-ResBlock
ResBlock
G-ResBlock
D-ResBlock
Batch Normalization
ReLU
ReLU
AvgPool (ker:8
conv. (ker: 3x3, 256 - 3; stride: 1; pad:1)
Lincar(128 - 1)
T Tanh(-)
Spectral Normalization
Table 7: Deep ResNet architectures used experiments on ImageNet, SVHN CIFAR-10, where G- ResBlock and D-ResBlock for the Generator (left) and the Discriminator (right), respectively. are described in T The models' parameters are initialized using the Xavier initialization (Glorot & Bengio 2010). For ImageNet experiments, the generator's input of dimension 512 instead of 128.
31

---

Published as a conference paper at ICLR 2021
H.2.3 UNROLLING IMPLEMENTATION
In Section D.1 we explained how we implemented unrolling for our full-batch bilinear experiments. Here describe our implementation for MNIST and CIFAR-10 experiments.
Unrolling is computationally intensive, which can become a problem for large architectures. The computation of V (O," spx). with m unrolling steps, requires the computation of higher order derivatives which comes with a xm memory footprint and significant slowdown. Due to limited memory, one can only backpropagate through the last unrolled step, bypassing the computation of higher order derivatives. We empirically see the gradient is small for those derivatives. In this approximate version, unrolling can be seen of the same family as extragradient, computing its extrapolated points using more than single step. We tested both true and approximate unrolling on MNIST, with a number of unrolling steps ranging from to 20, The full unrolling that performs the backpropagation CIFAR-10 10 we only experimented with approximate unrolling over memory footprint of the ResNet architectures used for the generator and discriminator, making the other approach infeasible given our resources.
the unrolled discriminator was implemented using the Higher library. On
steps due to the large
H.2.4 HYPERPARAMETERS USED ON MNIST
Table 8 lists the hyperparameters that we used for our experiments on the MNIST dataset.
Table &: Hyperparameters used on MNIST.
Parameter
AltGAN
LA-AlIGAN
ExtraGrad
LA-ExtraGrad
Unrolled-GAN
Vla
0.001
0.001
0.001
0.001
0.001
AD
0.001
0.001
0.001
0.001
0.001
Adam B1
0.05
0.05
0.05
0.05
0.05
Batch-size
50
50
50
50
50
Update ratio
1
1
Lookahead &
1000
1000
Lookahead
0.5
0.5
Unrolling steps
20
H.2.5 HYPERPARAMETERS USED SVHN
Table 9: Hyperparameters used SVHN.
Parameter
AltGAN
LA-AIIGAN
ExtraGrad
LA-ExtraGrad
1|G
0.0002
0.0002
0.0002
0.0002
TJn
0.0002
0.0002
0.0002
0.0002
Adam 31
0.0
0.0
0.0
0.0
Batch-size
128
128
128
128
Update ratio r
5
5
5
Lookahead k
5
5000
Lookahead a
0.5
0.5
Table 9 lists the hyperparameters used for experiments on SVHN. These values were selected for each algorithm independently after tuning the hyperparameters for the baseline.
32

---

Published as a conference paper at ICLR 2021
H.2.6 HYPERPARAMETERS USED ON CIFAR-10
Table 10: Hyperparameters that we used for our experiments on CIFAR-10.
Parameter
AltGAN
LA-AIGGAN
ExtraGrad
LA-ExtraGrad
Unrolled-GAN
f|
0.0002
0.0002
0.0002
0.0002
0.0002
0.0002
0.0002
0.0002
0.0002
0.0002
Adam 31
0.0
0.0
0.0
0.0
0.0
Batch-size
128
128
128
128
128
Update ratio r
5
Lookahead k
5
5000
Lookahead a
0.5
0.5
Unrolling steps
The reported results on CIFAR-10 obtained using the hyperparameters listed in Table TO These values were selected for each algorithm independently after tuning the hyperparameters. For the baseline methods we selected the hyperparameters giving the best performances. Consistent with the results reported by related works, we also observed that using larger ratio of updates of the discriminator and the generator improves the stability of the baseline, and we used 5. We observed that using learning rate decay delays the divergence, but does not improve the best FID scores, hence we did not use it our reported models.
H.2.7 HYPERPARAMETERS USED IMAGENEET
Table 11: Hyperparameters used for AltGAN experiments ImageNet.
Parameter
AlIGAN
LA-AIGAN
DLA-AIIGAN
GJG
0.0002
0.0002
0.0002
1JD
0.0002
0.0002
0.0002
Adam B1
0.0
0.0
Batch-size
256
256
256
Update ratio r
5
Lookahead k
5
Lookahead
10000
Lookahead
0.5
0.5
Unrolling steps
Table 12: Hyperparameters used for our ExtraGrad experiments on ImageNet.
Parameter
ExtraGrad
LA-ExtraGrad
DLA-ExtraGrad
NG
0.0002 0.0002
0.0002
0.0002
TDD
0.0002
0.0002
Adam 81
0.0
0.0
0.0
Batch-size
256
256
256
Update ratio
5
5
5
Lookahead k
5
5
Lookahead K'
5000
Lookahead a
0.5
0.5
Unrolling steps
The reported results on ImageNet were obtained using the hyperparameters listed in Tables II and
12
33

---

Published as a conference paper at ICLR 2021
I ADDITIONAL EXPERIMENTAL RESULTS
In Fig. 6 we compared the stability of LA-AIGANN methods against their AltGAN baselines on both the CIFAR-10 and SVHN datasets. Analogously, in Fig. [7 we report the comparison between LA-ExtraGrad and ExtraGrad over the iterations. We observe that the experiments on SVHN with ExtraGrad are more stable than those of CIFAR-10. Interestingly, we observe that: (i) LA -rat dient improves both the stability and the performance of the baseline on CIFAR-10, 10, see Fig. 17a and (ii) when the stability the baseline relatively good as on the SVHN dataset, LA Extragradiiet still improves its performances, see Fig. 17b
For completeness Fig. 5 in Fig 18 we report the respective eigenvalue analyses on MNIST, that are summarized in the main paper.
35
15
LA A-ExtraGmdd
LA-ExtmaGadd
30
ExtreGrad
ExtraGrad
10
FID
25
5
15
0
250000
500000
0
250000
500000
Iteration
Iteration
(a) CIFAR-10 dataset.
(b) SVHN dataset.
Figure 17: Improved stability LA-ExttaGrad relative to its ExtraGrad baseline on SVHN CIFAR-10 over 5 runs. The median and the individual runs are illustrated with ticker solid lines and with transparent lines, respectively, See and H for discussion and details on the implementation, resp.
1.1 COMPLETE BASELINE COMPARISON
In $5.2 we omitted uniform averaging of the iterates for clarity of the presented results-selected as down-performs the exponential moving average in our experiments. In this section, for completeness we report the uniform averaging results. Table 13 lists these results, including experiments using the RAdam (Kingma & Ba 2015) optimizer instead of Adam.
AlGGAN
0.5
ALGGNN
Magnitude
Magnitade
AlIGA
LA-AIGANN
Imaginary
-20
100
AlGAN AAGGN
-10
0+
-0.5
10 16 18 20
100
200
Top-20 Eigenvalues
Top-20 Eigenvalues
Real Part
(a) Generator.
(b) Discriminator.
Eigenvalues of v' sp)
Figure 18: Analysis on MNIST at 100K iterations. Fig. 18a
18h
Largest eigenvalues Hessian of
the generator and the discriminator. Fig, 8c Figenvalues of the Jacobian JVF indicating no rotations at the point of convergence of LA-AIIGAN (see
34

---

Published as a conference paper at ICLR 2021
CIFAR-10
Fréchet Inception distance
Inception
Method
no avg
uniform avg
EMA
no avg
uniform avg
EMA
AlIGAN-R
23.27 + 1.65
19.81 + 2.58
17.82
1.31
7.32. + .30
8.14 + .20
7.99 + .13
LA-AIGAN-R
17.31 + .58
16.68 + 1.45
16.68
1.45
7.81 +
8.61 + .08
8.13 + .07
ExtraGrad-R
19.15 + 1.13
16.17 = .63
15.40 :94
7.59
8.55 + .05
8.05 .15
LA-ExtraGrad-R
15.38 .76
14.75
14.99 +:
7.93 .05
8.51 + .08
8.01
.09
AGGGAN-
21.37 1.60
15.02 = 1..25
16.92
7..114.116"
3.23 1.7
8.003
.13
LA-AIIGANAA
16.74
13.98 4.7
8.05
8.45 + .32
8.19 .05
ExtraGrad-A
18.49 .99
16.22 1.50
15.47 1.82
7.61 07
8.46 + .08
8.05
.09
LA- LA-ExtraGrad-A
15.25 +: .30
14.95 = .44
14.68 1: .30
7.99 1 .3
8.13 - .18
8.04 + .04
Unrolled-GAN.
21.04 + 1.08
18.25 + 1.60
17.51 + 1.08
7.43 + .07
8.26 + .15
7.88 + .12
SVHN
ARIGAN-A
7,84
+
2.88
3.12 .14
3.19 .09
LA-AIIGAN-A
3.87 + .09
3.28 .09
3.16
3.38
.09
3.22
.08
HxtraGrra-A
4.08 +
8.89 1.07
3.22 .09
3.21 02
3:21 + .04
3.16
.02
LA LA-ExtraGrad-A
3.20 + .09
7.66 1.54
3.16 .14
3.20 + .2
3.32 + .13
3.19
.03
MNIST
AlIGAN-A
.094 + .006
.167 .033
.011 .002
8.92 .01
8.88 + 002
8.99 .01
LA-AIIGAN-A
.053 .001
.176 .024
.002
8.93
8.92 +
8.96
.02
ExtraGrad-A
.013
.122 024
.003
8.00
8.88
003
8.98
001
LA Ex-ExtrGraadAA
.005
024
.032 .002
8.92 - .02
8.95
.01
Unrolled Unrolled-GAN-A .077 .006
.244 +. .16
.00 = .002
8.91 + .2
8.91 + .02
8.99
.01
Table 13: Comparison of the LA-GAN optimizer with its respective baselines ARGANN and ExtraGrad (see $ $5.1 for naming), using FTD (lower is hetter) and (higher is better). EMA denotes exponential moving average (with fixed B 0.9999, see $(F). With suffix -R and -A we denote Adam Kingma & Ba 2015) optimizer, respectively. Results are averaged over runs. We run each experiment on MNIST for 1O0K ieerations, and for 500K iterations for the rest of the datasets. See and $ 5.2 for details architectures and hyperparameters and for discussion the results, resp.
we use RAdam (Liu et al. 2020) and
1.2 SAMPLES OF LA-GAN GENERATORS
In this section we show random samples of the generators of our LAGAN experiments trained on ImageNet, CIFAR- 10 and SVHN.
The samples in Fig. 19 are generated by our best performing LA-AlIGAN models trained on CIFAR-
10. Similarly, Fig. 20 & 21 depict such samples of generators trained on ImageNet and SVHN, respectively.
35

---

Published as a conference paper at ICLR 2021
Figure 19: Samples generated by best performing trained generator CIFAR-10, using LA-AItGAN and exponential moving average (EMA) on the slow weights. The obtained FID score 12.193.
36

---

Published as a conference paper at ICLR 2021
OPENA
Figure 20: Images generated by best model trained on 32x32 ImageNet, obtained with LA-AIGGAN and EMA of slow weights, yielding FID 12.44.
37

---

Published as a conference paper at ICLR 2021
65
463
7
315
6
6
12
3
24
7
37
33
14
08
12
95
1919
98
5
88
04
5
89
223
6
35
56
80
00
0
33
9
25
39
33
19
M7
77
98
21
5
0
12
3
2
27
$7
0O
1
2
8
33
33
44
1A
2
8
3
66
16
16
26
1
4
37
3
10
12
3
59
5
50
300
85
9
0
1.
0
6
126
15
2
33
2
0
3
6
549
9
28
5
5
2
2
2
5
7
7
14
9
38
2
15
10
11
52
66
125
4
6
2
4
86
2
7
1
3
2
15
4
2
17
1
50
3
Q
9
0
15
13
101
18
5
4
37
04
2
51
12
2
1
56
F0C
8
5
115
51
95
8
54
3
5
19
3
36
2
2
5
20
7
5
5
40
2
6
0
37
2
7
11
2
20
08
46
5
22
2
8
33
2
167
2
8
6
20
5
15
4
6
33
989
19
5
2
3
5
18
5
15
3
166
198
2
04
4
3
6
3
57
6189
16
8
0
2
07
15
3
15
24
2
7
13
51
6
25
36
5
66
130
9
2
14
35
5
2
4
7
43
51
4
28
20
Figure 21: Images generated by one of our best LA-ExtraGrad & EMA model (FID of 2.94) trained on SVHN.
38


Reviews:
[{"review_id": "ujcNKpKJ5Js", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##### Summary:\n\nThis paper proposes a lookahead-minmax algorithm for optimizing minmax problems such as GANs, which updates the parameters (of both the generator and the discriminator) with the extrapolation. With a bilinear example, the authors show that the use of lookahead-minimax allows for convergence in cases where other methods does not, and yields good performance under high variance. Experiments of generative performance on several well-known public datasets demonstrates the effectiveness of the proposed method.\n\n##### Reasons for score:\n\nThe idea is intuitive and easy to follow. My major concern is about the scale of the experiments.\n\n---\n##### Pros: \n \n1. The paper is well-motivated and the idea is easy to understand. Figure 1 clearly demonstrates how lookahead-minmax can address the rotational nature in GAN’s training, which is further confirmed in Figure 5.\n \n2. The advantages of the proposed lookahead-minmax is demonstrated with a bilinear example. For me, this helps to understand the merits of the proposed method.\n\n3. The paper provides comprehensive experiments and describes the experimental settings in details. \n \n##### Cons: \n \n1. The contributions are a little bit confusing. The first contribution listed in the introduction (and Section 3) is an improved convergence guarantee, which seems an independent contribution. What is the relation between this theoretical result and the lookahead-minmax algorithm for GANs? Does the following analysis and conclusions rely on this improvement? After reading the paper, I found the rest of the paper does not necessarily build on the new convergence guarantee. I think it might be clearer to mention the most significant contribution first.\n  \n2. Some related works are missing. Gidel et al. 2019b presents the negative momentum method, which leads to a similar “backtracking step” in the vector field the gradients. CLC-GAN [Xu et al. arXiv:1909.13188] proposes to stabilize GANs’ training dynamics by interpreting the gradient flow as a dynamic system and manage it with closed-loop control, which is also related to this paper.\n \n3. The experiments are conducted on 4 representative datasets. However, the image sizes (32*32) do not match the state-of-the-art GANs, which typically can generate larger images (e.g. Mescheder et al. 2018). One concern is whether the proposed method works well on larger datasets such as LSUN and ImageNet with higher resolutions. It would be more convincing if the authors can provide results on larger datasets.\n  \n---\n##### Some typos: \n(1) In Eqn. (JVF): the second entry in the JVF $\\nabla_{\\phi}\\mathcal{L}^{\\theta}$ -> $\\nabla_{\\phi}\\mathcal{L}^{\\phi}$\n\n(2) In the line below Eqn. (SB-G): It seems that $b$ and $c$ are of dimension $d \\times n$.\n\n(3) In paragraph of “Benchmark on CIFAR-10 …” of Page 7, “3.5 times larger model then ours” -> “3.5 times larger model than ours”\n\n(4) In the same paragraph mentioned in (3), the references to tables are inconsistent. Both “Table 2” and “Tab. 2” are used.\n\n(5) In the last line of Page 7, “LA—GAN” em dash -> en dash\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TjBFmBLybZ_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This work extends the recently proposed lookahead optimizer (which was designed for single-objective optimization) to minimax optimization, particularly GAN training. The authors claim that the backtracking step in lookahead optimizer alleviates the notorious rotational behavior in GAN dynamics. Moreover, the authors argue that the lookahead optimizer implicitly handles the high variance in the small-batch setting. Both arguments are backed up by toy experiments on stochastic bilinear games.  Finally, on standard image datasets, the lookahead minimax algorithm outperforms some popular algorithms and achieves state-of-the-art performance on CIFAR-10.\n\nReview: This paper is well-written and explains the main idea in a clear and effective manner. The empirical results of the introduced lookahead minimax algorithm are quite impressive given its simplicity. However, the lookahead minimax dynamics is not well-understood and no theoretical justification is provided in the paper. The bilinear game might not reflect the real GAN training as standard GDA never diverges in real GAN training. I tend to give a score of 7 or even higher, but I'm not satisfied with the current explanation of the performance gain. I think a better understanding of the dynamics is needed and I will increase my rating if the authors could improve the paper accordingly.\n\nHere are detailed comments:\n\n1. In the third paragraph of the introduction, the authors mentioned that empirically GANs often converge to a locally stable stationary point that is not a differential Nash equilibrium. Actually, this phenomenon occurs even for the simplest quadratic minimax games, see for example [1, 2, 3]. Moreover, Nash equilibrium might not even exist in general, Stackelberg equilibrium [2, 4] is probably a better solution notion.\n\n2. The authors provide some theoretical results of the lookahead optimizer, but for single-objective optimization. I suggest the authors moving this section to the Appendix (as this section seems irrelevant) and replace it with the analysis of lookahead for minimax games. In particular, the authors argue many times in the paper that the lookahead algorithm is able to handle the rotational dynamics well, I think the authors should be able to verify that theoretically on quadratic minimax games if the argument is right.\n\n3. The bilinear game might not reflect the real training dynamics of GANs. In bilinear games, simultaneous GDA with finite step sizes simply diverges, but it is not the case in practice for GAN training. To this end, strongly-convex strongly-concave minimax games might be better to model the underlying GAN dynamics. Strongly-convex strongly-concave quadratic minimax games shall be as simple as the bilinear cases and some theoretical analyses and simulation should be easy to do. \n\n4. In the paper, the authors chose Extragradient as one of their baselines. However, after inspecting the code, I find much fewer iterations are used for Extragradient. I understand the authors would like to keep the total gradient queries the same over different algorithms, but there is a similar algorithm (Optimistic gradient descent ascent [5, 6]) with only one gradient query per step and perform similarly with Extragradient. In this sense, I think it is fair to only use fewer iterations for Extragradient. \n\n5. For stochastic games (especially for SB-G), there is a better version of Extragradient which handles the variance well, see [7] for details.\n\n6. In the original lookahead paper, it was mentioned that the slow weights can be understood as the exponential moving average (EMA) of the fast weights. In particular, the authors of the original lookahead paper did some analysis on a noisy quadratic model and showed lookahead reduces variance. In that simple noisy quadratic model, exponential moving average can effectively reduce variance, see section 3.4 of [8] for details. I think it is probably worth testing EMA on the toy experiments and mentioning it in the paper. I notice that the authors have already included EMA in section 5 and EMA improves the performance a lot. To this end, I think the advantages of lookahead can be potentially decomposed into two parts (though more works need to be done): (1) it suppresses the rotational part in game dynamics (I encourage the authors to analyze the spectrum of the dynamics); (2) it implicitly does exponential moving average and hence reduces the variance.\n\nOverall, I think this paper is very strong on the empirical results. However, analysis on the dynamics of the lookahead algorithm is lacking. I will be happy to increase my rating if the authors could resolve some of my comments.\n\n--------------------\n**In the rebuttal, the authors resolved most of my concerns, therefore I increased my score to 7 as promised.**\n\n\nReference:\n[1] On Finding Local Nash Equilibria (and Only Local Nash Equilibria) in Zero-Sum Games, 2019.\n\n[2] On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach, 2020.\n\n[3] GANs May Have No Nash Equilibria, 2020.\n\n[4] Convergence of learning dynamics in Stackelberg games, 2019.\n\n[5] Training GANs with Optimism, 2018.\n\n[6] A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approach, 2019.\n\n[7] Revisiting Stochastic Extragradient, 2019.\n\n[8] Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model, 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "S3N-JFtTcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors are focusing in this paper on the development of LookAhead mechanism. Their contributions are the following:\n\n1. They provide an improvement upon an already existing convergence rate for the minimization case.\n2. They propose the application of the LookAhead mechanism (under the necessary adjustments) for the more general min/max framework.\n3. These results are extensively justified by various numerical experiments.\n\nThe paper is well-written and easy to follow. Moreover, the core idea of applying the LookAhead for mix-max problems, as far as my knowledge goes, is  novel. In addition, without being an expert, the experiments seem promising and extensive.\nThat said, my concern would be regarding the theoretical part of the paper. I fully understand that the paper is primarily experimentally oriented. However some more theoretical analysis would be useful. More precisely, the theoretical justification for the improved convergence part for the minimization part seems to be more like a sketch rather than a rigorous mathematical proof. Overall, I suggest that it would be very interesting to see some provable theoretical guarantees starting with the  paper's motivation example that of the bilinear game.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mnpk9cNQLXc", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper adapts a recently introduced Lookhead method in optimization to improve the training of minmax problems such as GANs. The main challenge is to address the variance of stochastic gradients and the rotational component in the Jacobian of the gradients. The algorithm proposed in this paper shows improvements over existing methods, in terms of convergence rate. The stability of existing algorithms is a major issue when applying GANs on image dataset, the proposed Lookhead minmax method also improves the stability. I find that the results on the bilinear game are convincing, but the results on GANs are much less convincing.  \n\nThere seems to me some inconsistency in the numerical results in Tab. 1 and Fig. 6. Are the same hyper-parameters used for LA-Alt-GAN on ImageNet? The median diverges in (c) of Fig. 6 while in Table 1, the FID is around 14.37? This is hard for me to interpret as the total number of iterations are the same, which is 500k. \n\nIn terms of writing, I would recommend to focus on the minmax problem. In this sense, the section 3 is a bit distracting as it is about the optimization problem. The Algorithm 5 (page 19) about LA-AltGAN is important to understand the results, so it would be better to explain it more in the main body of the paper. Also what is LA-ExtraGrad algorithm? How Adam is applied in these algorithms? These proposed variants are not very clear to the reader. Also is the number of passes the same as the number of iterations? \n\nTo make the results in Fig 5. more conclusive, I would recommend to zoom into the eigenvalues of LA-AltGAN, to see how small the imagery parts of the eigenvalues are. This is an interesting observation that it seems not very conclusive to show that LA-GAN shows no rotations.\n\nOverall, I think both the results and the writing need to be improved. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews were a bit mixed, with a general consensus towards acceptance. The authors were one of the first to extend lookahead to minimax optimization, and demonstrated its potential through thorough experiments. The theoretical results were not as strong or at least not very well presented. Overall, the authors made interesting contributions and this work is of general interest to the ICLR audience. Please consider further polishing the draft according to the reviewers' comments. The AC would also like to draw the authors' attention to the following issues discovered in an independent assessment:

(a) As the reviewers mentioned, how lookahead-minimax addresses rotational dynamics is not clearly presented. The current justification is a bit handwaving and speculative. 

(b) Please consider rewriting Section 3. If there is some new results on the minimization problem, state the results in a theorem and include all assumptions clearly and precisely. This is also useful for other people to reference your result. As the authors themselves pointed out, this result falls quite short of explaining or motivation lookahead. 

(c) Theorem 1, add e.g. in the citation before (Bertsekas, 1999). Theorem 2, in its current form, is quite weak in two aspects: (a) without checking its proof one can already see how to derive it in 1 line or 2. (b) if the base optimizer already converges, what is the point of having lookahead to converge as well? The potentially different convergence rate should be one's target here. It is certainly fine for the authors to not fully justify their proposed algorithm, as long as the authors (hopefully) are at least aware of the issues.

(d) Section 4 is a bit disappointing as one would have expected the authors to derive some qualitative results here (also raised by some reviewers).
Decision: Accept (Poster)