Implicit competitive regularization in GANs
Authors: Florian Schaefer
Hongkai Zheng
Anima Anandkumar
Venue:  (2020)
Keywords: GAN
competitive optimization
game theory

Abstract: Generative adversarial networks (GANs) are capable of producing high quality samples, but they suffer from numerous issues such as instability and mode collapse during training. To combat this, we propose to model the generator and discriminator as agents acting under local information, uncertainty, and awareness of their opponent. By doing so we achieve stable convergence, even when the underlying game has no  Nash equilibria. We call this mechanism \emph{implicit competitive regularization} (ICR) and show that it is present in the recently proposed \emph{competitive gradient descent} (CGD).
When comparing CGD to Adam using a variety of loss functions and regularizers on CIFAR10, CGD shows a much more consistent performance, which we attribute to ICR.
In our experiments, we achieve the highest inception score when using the WGAN loss (without gradient penalty or weight clipping) together with CGD. This can be interpreted as minimizing a form of integral probability metric based on ICR.

Under review as a conference paper at ICLR 2020
IMPLICIT COMPETITIVE REGULARIZATION IN GANS
Anonymous authors
Paper under double-blind review
ABSTRACT
Generative adversarial networks (GANs) are capable of producing high quality samples, but they suffer from numerous issues such as instability and mode col- lapse during training. To combat this, we propose to model the generator and dis- criminator as agents acting under local information. uncertainty, and awareness of their opponent. By doing so we achieve stable convergence, even when the under- lying game has no Nash equilibria. We call this mechanism implicit competitive regularization (ICR) and show that it is present the recently proposed comper- itive gradient descent (CGD). When comparing CGD to Adam using variety of loss functions and regularizers on CIFARIO 10, CGD shows a much more consis- tent performance, which attribute to ICR. In our experiments, highest inception score when using the WGAN loss (without gradient penalty or weight clipping) together with CGD. This can be interpreted as minimizing form of integral probability metric based on ICR.
achieve the
INTRODUCTION
Generative adversarial networks (GANs): (Goodfellow et al. 2014) are class of generative models based on a competitive game between a generator that tries to generate realistic new data, and discriminator, that tries to distinguish generated from real data. In the original formulation, the two players are playing a zero-sum game with the loss function of the generator given by the binary eross entropy,
1 min max Er Panen [log D(x)] + _E [log (1 D(x)) D
(1)
Here, G is the probability distribution generated by the generator, D the classifier provided by the discriminator, and Paats is the target measure, for example the empirical distribution of the training data. In practice, both players are parameterized by neural networks that are trained simultancously by variant of stochastic gradient descent (SGD).
Instability and mode collapse: GANs constitute the state of the art (SOTA) on many problems, but suffer from variety of issues such as instability and mode collapse, i.c. drastic loss of sample diversity, during training.
Beginning with (Goodfellow et al., 2014), GANs have often been interpreted as the generator per- forming approximate minimization a function defined implicitly by maximization over the dis- criminator. From this perspective, instability and mode collapse have been explained with the fact that for finite amounts of data and any continuous distribution & produced by the generator, a suf- ficiently powerful discriminator will be able to fully maximize the two summands in Equation independently of each other, instead of searching for a trade-off. This is because it can first maxi- mize the second summand and then modify its value in e-neighborhoods of the true data to maximize the first summand. In the limit of small these modifications don't 't influence the second objective, leading to the discriminator maximizing both objectives. While one could hope that this problem vanishes for large data sets, real data will often be concentrated around low-dinensional structure, which can be exploited by the discriminator in a similar fashion. Thus, for any fixed generator expect the weights of the discriminator diverge it fully performs the above trick, causing instability and preventing the generator from receiving useful gradient information.
Gradient penalties need to choose metric sample space: The above observation has mo- tivated restriction of the size of the gradients D (as a function on sample space) (Arjovsky &
Bottou 2017 Arjovsky al. 2017 Gulrajani et al. 2017 Roth al. 2017: Kodali et al. 2017

---

Under review as a conference paper at ICLR 2020
Miyato et al.) 2018). Measuring the size of the gradient of D requires metric the space of samples, typically chosen to be l2. For complex samples like images, this will be a crude measure of similarity. In images of human faces for instance. skin- and hair color will likely dominate over other subtle features like facial expression in terms of the f2-norm. In contrast, the original GAN has the intriguing property of not requiring any explicit choice of distance on sample space.
The game behind GANs:
For convex-concay. games, Nash equilbria provide a natural solution concept. We merely need to choose which algorithm to use when computing them. For non-convex- concave games like GANs however, Nash equilibria need not be the right solution concept (see Be-
rard et al. (2019) and Figure 2 which spurred search for new solution concepts (Jin et 2019
Fiez et al. 2019).
Both the choice of a loss function and of notion of solution is modeling problem. We argue that rather than deriving an algorithm based on these choices, it might be preferable to directly model iterative games between the two players. Just like "the goal is to capture the other player's king" does not fully specify the rules of chess, Equation [ does not fully specify a game between the two players. Rather, we need to specify what information the players have access to, which "moves" allowed, and in what order are they are to be made. Even with the rules completely specified, the behavior of the players will crucially depend on the heuristics that they use to anticipate each other's behavior and choose their own strategy, possibly in the face of uncertainty. Finally, chess player might prefer a certain draw over an uncertain victory depending on the format of the tourna- ment. Similarly. players aiming to minimize an averaged loss might play differently from players that greedily minimize the loss after every move.
Even for the same loss function, different such assumptions lead to different dynamics in GANs. Rather than trying to define a suitable solution concept based on, for instance, the spectrum of the Hessian, it might be more fruitful to directly model the behavior of the two players.
In this work we show that modeling the generator and discriminator as agents making decisions under uncertainty and in awareness of each others' goals can lead to stable be- havior. Surprisingly. this holds even in the absence of Nash equilibria and for games that are mean- ingless in the minimax sense. We show empirically that even generators that produce good samples can incur large losses under discriminator optimized for this particular generator, but such dis- criminators are highly vulnerable to changes of the generator's s strategy (see Figure 2) Therefore, discriminator aware of the competitive nature of the game will try to find trade-off between achieving a low loss and being robust to the actions of the generator, and vice versa. Based on this observation, we propose to mitigate the lack of stability in GAN training by choosing more realistic models of the agents' behavior. One realization of this idea is the recently proposed competitive gradient descent (CGD) algorithm (Schafer & Anandkumar 2019), where players update their pa- rameters using the Nash equilibrium of local approximations of the loss functions that takes into account actions of the other player. We show that it imposes an implicit competitive regularization (ICR) on the updates, leading to moves that are locally robust to the other player's 's response.
Our contributions:
We conclude with an extensive suite of experiments using Adam and CGD on the original GAN loss from Equation [and the bilinear WGAN loss (Arjovsky & Bottou| 2017 Arjovsky et al. 2017), and a variety of different regularizers. We observe that the performance of CGD is much more consis- tent than that of Adam and that the highest inception score is achieved by combining the WGAN loss (without gradient penalty or weight clipping) with CGD. Recall that when combined with a Lipschitz constraint, WGAN minimizes the Wasserstein distance between generator and target dis- tribution. By analogy, the combination of WGAN loss with CGD can be interpreted as minimizing an integral probability metric (Sriperumbudur et al. 2009) based on ICR. Thus, its superior per- formance supports the claim that ICR is the appropriate form of regularization for GANs. We emphasize that in our experiments we did not perform any architecture or hyperparameter tuning, and instead use model intended to be used with WGAN gradient penalty (Gulrajani et al. 2017).
2 THE NASHLESS GAME
What is the solution, if there is no solution? We believe that rather than viewing the loss function of GAN as specifying a well-defined computational problem that is solved by computing local Nash equilibrium using one of the many methods proposed in the literature, the choice of the algo- rithm should part of the modeling problem. In order make this more concrete, we consider the
2

---

Under review as a conference paper at ICLR 2020
X
8
Figure 1: Implicit competitive regularization in the nashless game: Under full information, each player moves towards infinity as quickly as possible (first panel). Under limited information, but without accounting each other's actions, the players oscillate and eventually diverge (second panel). Under both limited information and awareness of the opponent, the trajectories become stable (third panel). (The vertical axis measures "time", each horizontal grid line corresponds to one iteration).
zero-sum game given by
min max- - -ex((=))= azy + exp(y"), RR yER
(2)
which we will refer to as the nashless game. Here, > I is a large but fixed parameter. This game does not have a global Nash equilibrium, since each player can always achieve an exponen- tially increasing reward by moving towards infinity. It furthermore does not have any local Nash equilibria, since the curvature of the objective function is always negative in x and positive in y. One approach would be to interpret it a Stackelberg game where player y gets to make their move after player x. However, since player y can always achieve arbitrarily large rewards by diverging to infinity this does still not result in notion of solution. (Jin et al.] 2019. Definition 14) introduced a notion of local minmax point (x* ,y") that amounts to restricting the movement of the follower to within small distance of y² However they also require the objective to be local minimum with respect to y. which prevents it from being adequate for the game. (Fiez et al. 2019 Definition 4) propose a notion of differential Stackelberg equilibrium. However, since is only based on first or- der information it will implicitly use the worst strategy of the follower. Thus, the leader can always locally improve by moving further towards infinity, preventing the game from having differential Stackelberg equilibrium in the sense of (Fiez et al. 2019, Defininition 4.). While it is possible to define other notions of equilibrium, and hence solutions, for problems like in Equation 2 the above discussion illustrates that it is much harder than in the convex-concave setting.
An alternative way of turning equation 2 into an algorithm is to turn it into an iterative game between the two players, and obtain the algorithm from suitable notion of rational behavior.
Three games for one loss function: To illustrate the different dynamics that can arise, we now equip the loss function of Equation 2 with three different sets of rules for iterative play.
1. The global game: The two players move simultaneously, a distance of at most 1. They have access to the entire optimization landscape and aim to minimize their average loss in the asymptotic limit of infinite time.
2. The myopic game: The two players move simultancously at a distance of at most 1. At the k-th step, the player (y) only has access to objective function on [i I, + 1] {y} ([x) x lyk
1, s) + 11) and tries to minimize the loss at step k + 1.
distance of most 1. At the k-th step, both players have access to the objective function on [Ix - 1, Th + 1] x [y+- 1, + 1)) and try to minimize their loss at step + 1. The players are furthermore aware of each other's 's goals.
3. The predictive game: The two players move simultaneously
In the global game, player y can win the game with a loss of ooo by diverging to infinity quickly as possible. This is similar in spirit to the maximization over D in the original GAN where the global optimization over D results in unstable behavior of the underlying game (see Figure])
Since the global optimization of nonconvex problems is computationally infeasible, the first set of rules is not suitable model for GAN training. In myopic game, model the use of local algorithms by only giving the players access to the loss function in a local neighborhood of their present position. Initially of the exponentially increasing reward at infinity they try to improve their
3

---

Under review as a conference paper at ICLR 2020
L2 ponerato ondenn.
10P
epocho sheckpoint
spochtp chockpoitt
10'
Dischnimabor loos
spochi
22 upec apechhe-ccrttch
Dtecmmmattrr
10'
11 epochs chocspoint 21 spoetrecheepppe spoine
10-3
epecta over-tranno
21 spocte over-traine
11 apochwssratch
11 upec bechkecchekkoin
21 epochs scratch
21 epochs checkpoint
20000 Iteration
800
Iteration
Ieeratoo
Figure 2: Overtraining Discriminators: We begin by training a GAN on MNIST using Adam for 21 (11) epochs, saving the resulting generator and discriminator as a "checkpoint". We then over- train the discriminator while keeping the checkpoint generator fixed, achieving low discriminator loss (second panel). As indicated by the increasing gradient norm of the generator during training (third panel), the resulting discriminator is brittle and incurs large losses as we start training the gen- erator again (fourth panel). Values computed from minibatches and averaged over small z-windows.
reward locally, entering the cycling dynamics also known from simultaneous gradient descent (see Figure D. While the cycling behavior slows down the divergence, the players will eventually diverge far enough for the exponential term to overpower the bilinear term, from where on both players will diverge to infinity, without further cycling behavior.
In predictive game, the two players have access not only to their own, but also to the other player's loss function as well as the effect that the other player's s moves will have on their own loss. Therefore, they don't it blindly aim to increase their own loss function but also try to estimate the actions of the other player and adapt choose their own actions accordingly. In the initial position = 0. y 2, player y that unaware of the presence of x will choose to move to 3, in order to minimize its loss. If is aware of the goals of however, it will know that whatever move y makes, the best move for x will be to move to 1. Thus. player y should make its move based on the assumption that x will move 1 and hence (since >> 1) moves always exist unique optimal strategies for x and resulting in the periodic behavior illustrated in Figure [
instead. For the given starting conditions there
The above example shows that even in the absence of suitable equilibria we can obtain meaningful and stable dynamics by modeling the players as agents acting with limited information and aware- ness of their opponents. Before providing possible framework for this type of modeling, we will provide evidence that key features of the nashless game might indeed be present in GANs.
3 WHY A COMPETITION-WWAR GAN MIGHT BE STABLE
GAN discriminators are not optimal: The key feature of the nashless of the last section that even though either player is incentivized to diverge to infinity, in doing so it enables the other player to "oouteer-attack by means of the strong bilinear term. In order to verify these properties in GANs, first train GAN on MNIST using 21 iterations of Adam, obtaining good generative model. For future reference, we will refer to this generator and discriminator as the checkpoint generator and discriminator. We then overtrain a new discriminator from scratch against the fixed checkpoint generator. Similar to Arrjovsky & Bottou| (2017), we observe that the overtrained discriminator can achieve orders of magnitude lower discriminator loss than the checkpoint discriminator. There- fore, the checkpoint discriminator is far from being globally optimal. It seems conceivable that the checkpoint discriminator might instead be locally optimal. In order to verify this claim, we continue training the checkpoint discriminator while keeping the checkpoint generator fixed. As shown in Figure we observe almost the same behavior as when overtraining the discriminator from scratch. Thus, the checkpoint discriminator was not at local minimum (that is deep enough to affect stochas- tic gradient descent). Recent results by Berard et al. (2019) indicate that many good (in terms of image quality) generators are not local minima, but rather critical points of their loss function. This casts further doubt on the idea that local Nash equilibria are the right notion of solution for GANs.
Optimal discriminators in GANs are brittle: Given that the discriminator could drastically improve over the checkpoint generator we have to ask ourselves why it doesn't do so during GAN training. What, if anything, makes the checkpoint discriminator the better choice over the over-
4

---

Under review as a conference paper at ICLR 2020
trained one? In the nashless game, players were partly discouraged from decreasing their losses since in doing so they became would become more vulnerable to counter play by the other player. To investigate this question now perform a second experiment were we fix the overtrained dis- criminators from the first experiment and start training the checkpoint generator again. As shown in Figure [] the discriminator loss increases by orders of magnitude within just 40 iterations, significantly beyond the loss at the checkpoint. In contrast, training the generator against the fixed checkpoint discriminator the discriminator loss does not show a significant increase over the first 100 iterations. This behavior is already foreshadowed by the increase in norm of the generator's s gradient in the initial stage of the overtraining procedure (see Figure 2). Thus, while the overtrained discriminator can achieve a greatly reduced loss, it is much more vulnerable to changes of the gen- erator. A discriminator that aware of the goals of the generator might therefore choose not to minimize its loss myopically but rather aim for a trade-off between achieving low loss and being robust to actions of the generator.
MODELING THE PLAYERS' ACTIONS
In the previous section we observed that GANs, just like the nashless game, might not be described well by the solution concept of a (local) Nash equilibrium. However, our experiments suggested that a suitable notion of rational play of the two players might indeed lead to stable GAN formulations. As described in the introduction and further illustrated on the nashless game, there are numerous ways to turn a given loss function for the two players into an iterative game. In the following, we will present one possible framework towards this goal. We will restrict our framework towards the case of both players moving simultaneously, forgetting forggggn the history of the game and greedily aiming to minimize their loss after each move. Our framework models the players as choosing optimal actions based on the following three ingredients.
Belief, uncertainty, and anticipation:
1. The belief that the agents have about the loss associated with different actions.
2. The way they incorporate their uncertainty into their decisions.
Their anticipation of the action of the adversary when making a decision.
In the myopic version of the nashless game, both players' model consisted of the knowledge of the loss function in 1-neighborhood of their own position. The agents handled their uncertainty conservatively, by refusing to step outside the region where they are aware of the loss function. The agents did not have enough information to make use of an anticipation the other agent's 's strategy, thus can take them to assume that the other player stays fixed. The next iterate (ZK+1, 3k+1) can therefore be obtained as solution to the optimization problem
min f(z, yk)) + 1211)
(3)
max fIrk v) - 2 (sn)
where f is the loss function and (x(x) = oc for x & and zero everywhere else.
In the predictive version of the nashless game, both players' model consists of the true loss func- tion in 1-neighborhood of the present position of either player. As before, the players treat the uncertainty conservatively by refusing to step outside the region where they have knowledge of the objective function. Differently from the myopic game, they are aware of the objectives of the other player and the way in which the other player's actions affect their own loss. Therefore, their update obtained the Nash equilibrium of
min f(x, y)
max f(x, v) ((vlo--
In the particular case considered in the nashless game the above local game always had pure equilibrium. For general problems, the optimal solution of the local problem might correspond to a randomized strategy, resulting in randomized algorithm for the "solution" of the two-player game.
Computationally feasible instances: In practice, we usually don't have access to the exact loss function in a local neighborhood, but rather to local approximation that we expect to less reliable
5

---

Under review as a conference paper at ICLR 2020
as move away from the present iterate. A natural and computationally feasible way of modelling agent behavior in this setting is to let the agents use a Taylor approximation of the loss function, together with regularizer that penalizes moving too far away from the last iterate, and thus into regions where the Taylor approximation is unreliable. The popular simultaneous gradient descent falls into this framework, with players playing optimal with respect to linear approximation of the loss and quadratic penalty modeling their uncertainty, obtaining the game
1
min f(r, yk) (xr xxm) T V 2 (2 3k3) + x xx
2n
1
max (T 1/a + (v - yk) V y (Tk: - 2ny lly 1x
y
By choosing different models for the player's uncertainty, we could obtain other instances of simul- taneous mirror descent. This game very similar the one in equation3Jant and simultaneous gradient descent can be seen as a computationally feasible version of the myopic game and as such features a similar oscillatory behavior.
In order to mitigate this problem, we can allow the players to anticipate each others actions, for instance by assuming that the other player will perform a gradient descent update. For the players to be able to use this information we have to introduce bilinear approximation, obtaining Learning with eeponnnt-learning awareness (LOLA) (Foerster et al. 2018) as solution to the local game
1
min f y&) + (xr xx) V V 31) + (x xx) f V (IR, ykh +
||
x 1
21
1
max f y1) + (y- -10--10 yk) VV f Mk) (y yka y DT D.y (1r. yk V.. yss
27 11 yk 11?
9
2017) and the extragradient method (Korpelevich) 1977 can similarly be interpreted as based on "ottimistic" predictions of the two players actions.
Optimistic mirror descent (Daskalakis et al.)
Competitive gradient descent: In LOLA, both players make their decision based on the assump- tion that the other player will play gradient descent. In contrast, the predictive variant of the nashless game lets the players choose the Nash equilibrium of the local problem, thus accounting the uncer- tainty on the other player's 's plans. A computationally feasible analogue of this idea given by competitive gradient descent (CGD) (Schäfer & Anandkumar 2019), which lets the players choose the Nash equilibrium of a bilinear approximation the loss function, regularized with a quadratic penalty. The resulting game is given by
1
min f (xx Fk)
V f + (x xK) T yn) + (y 9k) V of +
||
x& 1|
27
1
max f+ f+ xR) v. f + (I (x))))]] Dayy, fy - yk) + (v y V,, f
y/ 113
27
where all derivatives and function values of f in the above expression are evaluated in the last iterate, (Ix, 3k.) Schäfer& & Anandkumar (2019) observe that this game has a pure unique Nash equilibrium resulting in the update rule
(V&f + Dey vff DayfDD f リ yh + (ld +n² D2 DD by A) (V.f - DD / vV
Ik+1 I&k n (Id +n²² D2
Implicit competitive regularization in CGD: While Schäfer & Anandkumar 20119) introduced CGD as a method for finding Nash equilibria, we believe that by modeling the behavior of the two players it also provides a useful notion of implicit regularization. Indeed, while the gradient term Vzf models the goal of x to minimize f. The additional term /Vy is a gradient step for the penalty term m/ V Vgfll? similar to LOLA or symplectic gradient adjustment (SGA) (Gemp & Mahadevan 2018: Balduzzi et al., 2018) Letcher et al.) 2019) which can be interpreted
as attempting to decrease the effect of the other player's action. The term (Id+7) DDe corresponds to choosing updates that are robust to the actions of the other player. If some of the singular values of D are very large, this amounts to approximately restricting the update to the orthogonal complement of the corresponding singular vectors. For smoothly varying singular vectors, this can be though of as approximately constraining the trajectories to manifold of robust

---

Under review as a conference paper at ICLR 2020
P
morm weneraeeeeee graster
10t
Diociminator
10'
11 epochs OGD epochs OSD
/pl
epaiten cherapaire
23 spociao shapoport speccto sctulan
20"1
epochs-Adam
21 epachisAdam
Veuss (Obel)
epochs checkpoiit
23 spours
10
opochs checkpoint
100n 2000
3000 4000
5000
6000
4.00
600
1000
Figure 3: Implicit competitive regularization: First panel: While simple gradient ascent would lead towards the overtrained discriminator " (first arrow), CGD tries also reduce the other play- ers gradient (second arrow) and for large mixed Hessian Df projects the update on the orthog- onal complement of the leading singular vector. We illustrate how this could prevent overtraining, with the thick black curve depicting the "manifold of robust play". Second and third panel: When attempting to overtrain the discriminator using CGD, actually becomes more robust (compare to Figure
play (see Figure 3 for an illustration). Contrary to the explicit regularization using gradient penalties, this implicit competitive regularization does not require choosing a metric on the space of samples but is instead informed by the inductive biases of the generator's and discriminator's architecture. We will now investigate how the use of CGD affects the overtraining in on MNIST presented in Figure 2 To this end, we try to overtrain the diseriminator from the checkpoint, this time using CGD instead of Adam. As shown in F no significant overtraining takes place even after 21 epochs. In fact, the generator gradients are smaller and the discriminator more robust than the checkpoint.
5 EMPIRICAL STUDY ON CIFAR10
Experimental setup: One of the key claims this work that the implicit competitive regulariza- tion due CGD is enough allow stable GAN training. To substantiate this claim, compare Adam and CGD on a wide range of regularizers and loss functions. While it is still unclear what the best way of combining CGD with learning rate heuristics, we opt for naive combination of CGD with RMSprop (referred to as ACGD) to make for a more fair comparison to Adam. As loss functions, we use the original GAN loss (OGAN) of equation T the Wasserstein GAN loss function (WGAN) given by
min max 9 D
Ez~Paun [P(x)] - Eg-Py [D(x)].
When using Adam OGAN, we stick to the common practice of replacing the generator loss by Ezwg [- log [I] as this has been found to improve training stability (Goodfellow et al. 2014 2016). In order the use with WGAN gradient penalty (Gulrajani et al., 2017). As regularizers, we consider no regularization (NOREG), l penalty on the discriminator with different weights (I.2), Dropout (p 0.5) on the discriminator after each convolutional layer (Dropout), 1-centered gradient penalty on the discriminator, following Gulrajani et al.) (0177) (GP). Following the advice in (Goodfellow) et al. 2016) we train generator and discriminator simultaneously with the exception of WGAN-GP and Adam, for which we follow (Gulrajani et al. 2017) in making five discriminator updates per generator update. We the Pytorch implementation of inception score (Salimans al. 2016) to quantitatively compare the quality of the different generators! [
generous to existing methods, we use an existing architecture intended for
Experimental results: ing our attention to the top performing models, we observe that the combination of ACGD with the WGAN loss and without any regularization achieves higher inception score than all other combina- tions tested (see Figure 4 (2:) Many of the other high-performing models are obtained via ACGD
We will summarize main experimental findings. (1:) When restrict-
Note that Pytorch implementation results in slightly different scores compared to Tensorflow imple- mentation. We also report. Tensorflow inception scores for important runs in Appendix 7_] that show that the relative performance of the different models remains largely the same.
7

---

Under review as a conference paper at ICLR 2020
5.0
4.5-
4.5-
Iropptte
Drop
DropcuttAcam
GP+Asam
GP-Asamm
3.5
DGAN-Drap AACGO
3.5-
DSA04-DrpoottAACO
WSAN NOREG HACGD
WGAN RORGGNNO
WSAN GP GP+ACGD
WGAN GP-ACGO
15c
200
200
400
Thousand generator nemr(3003)
gradient callstonn))
Figure 4: We plot the inception score against the number of iterations (first panel) and gradient or Hessian- vector product computation (second panel). The third panel shows final samples of WGAN trained with ACGD and without regularization.
4.5
OSAN
3.0
OSAN
13:
100
420
$50
Thousand generator 10
Thousand genorator IterstL""3
Trousand gensrater Tors210 5)
Figure 5: First panel: We compare the variance over seven runs of the same model with ACGD and Adam. Second panel: We plot the difference between inception scores between ACGD and Adam (positive values correspond The only cases where we observe nonconvergence of ACGD are OGAN without regularization or with weight decay of weight 0.0001. The inception score is however still higher than for the same model trained with Adam.
larger score for ACGD) over all iterations and models. Third panel:
(see Figure 4). (3:) When comparing the number of gradient computations and Hessian- prod- ucts, ACGD is significantly slower than OGAN combined with ADAM and dropout, because of the iterative solution of the matrix inverse in ACGD's update rule (see F (4:) The only instance where we observe erratic behavior with ACGD is when using OGAN without regularization, or with a small le penalty. However, ACGD still outperforms Adam on those cases (see Figure 5) We note that we observed nan values in one case of OGAN-L2 and ACGD, presumably because of underflow in the CG solve as discriminator weights approach zero. (5:) When plotting seven runs of OGAN with dropout using both ACGD and Adam, the results of ACGD consistent while Adam shows large variance (see Figure (6:) When plotting the difference between the inception score obtained by ACGD and Adam for the same model over the number of iterations, for all models, observe that ACGD often performs significantly better, and hardly ever significantly worse (see Figure 5).
6 CONCLUSION AND OUTLOOK
In this work we have argued that difficulties in GAN training might better be addressed by modeling the behavior of generator and discriminator than by just modifying the loss function devising new local solution concepts, like generalizations of Nash or Stackelberg equilibria. We have proposed framework for doing so and illustrated how it relates different algorithms in the literature. For one instance of our framework, competitive gradient descent, we explore in more detail the implicit competitive regularization that it induces. Extensive numerical experiments show that the perfor- mance of competitive gradient descent is significantly more consistent than that of Adam across different loss functions and regularizers. In particular, the highest performing model is the bilinear WGAN loss together with competitive gradient descent and no explicit regularization. Analogous to the Wasserstein distance being the integral probability metric (IPM) (Sriperumbudur et al.) 2009) associated to Lipschitz constraint, the WGAN loss optimized with CGD can be intuitively inter- preted as a form IPM associated to implicit competitive regularization. GANs allow to employ neural networks for generative modeling without having to prescribe notion of similarity between images explicitly, by taking full advantages of their "inductive biases". The present works suggests that their training might be best stabilized by sticking to this approach
8

---

Under review as a conference paper at ICLR 2020
and relying implicit competitive regularization rather than using penalties that make explicit choice of distance function on sample space. We believe that once reliable methods for their training have been established, adversarial methods will greatly increase the scope of applications of deep neural networks.
REFERENCES
Martín Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:
//openzeview. www/ iiiimmiiddd--- __xke_
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223, 2017.
David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Grae- pel. The mechanies of n-player differentiable games. arXiv preprint arXiv:/802.0556 056422 2018.
Hugo Berard, Gauthier Gidel, Amjad Almahairi, Pascal Vincent, and Simon Lacoste-Julien. A closer look the optimization landscapes of generative adversarial networks. arXiv preprint 15t6:0990044448 2019.
Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. Training gans with optimism. arXiv preprint arXiv: 1.000:11,, 2017.
Tanner Fiez, Benjamin Chasnov, and Lillian J Ratliff. Convergence of learning dynamics in stack- elberg games. arXiv preprint M.XX:1996.001//11 2019.
Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-leaming awareness. In Proceedings of the 17th International Conference Autonomous Agents and MultiAgent Systems. pp. 122-130. International Founda- tion for Autonomous Agents and Multiagent Systems, 2018.
Ian Gemp and Sridhar Mahadevan. Global convergence to the equilibrium of gans using variational inequalities. arXiv preprint a8Xiv: 01531, 2018
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, pp- 5767-5777, 2017.
Chi Jin, Praneeth Netrapalli, and Michael I Jordan. Minmax optimization: Stable limit points of gradient descent ascent are locally optimal. arXiv preprint arXiv:1902.00612 2019.
Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. convergence and stability of gans. arXiv preprint arXiv: arXiv: /705.07255. 2017.
GM Korpelevich. Extragradient method for finding saddle points and other problems. Matekon, 13 (4):35-49, 1977.
Alistair Letcher, David Balduzzi, Sébastien Racanière, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. Differentiable game mechanics. Journal of Machine Learning Research, 20 (84):1-40, 2019. http:/// wwtp://ppps/////////
008 html
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks, arXiv preprint arXiv: 18x1v:1802.05957 2018.

---

Under review as a conference paper at ICLR 2020
Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. In Advances in neural information pro- cessing systems, pp- 2018-2022 2017.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234 2234-2242, 2016.
Florian Schäfer and Anima Anandkumar. 1rXiv:190511203,, 2019.
Competitive gradient descent. arXiv preprint
Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert RG Lanckriet. On integral probability metrics, \phi-divergences and binary classification. arXiv preprint arXiv:0901..6688, 2009.
7 APPENDIX
7.1
TENSORFLOW INCEPTION SCORE
We computed Tensorflow inception scores for important runs of our experiments. As reported the figure 6 our results match the ones reported in the literature (Figure in Gulrajani et al. (2017) with ACGD still outperforming WGAN-GP trained with Adam.
6
Inception score
6-
Inception
5
4
4-
NAANNDOORE++AGD WGAN-GP+Adan
GGGN- WGAN-GP+Adam
3
OGAN-Dropoute Dropoutt
OGAN CGAN-DropouttAAaa
2
200 400 600 800 1000 1200 1400
0 1000 2000 3000 4000 5000 6000 70008000
Thousand generator iteration(10
Thousands gradient calls(10~3)
Figure 6: We plot the Tensorflow inception scores against the number of generator iterations(first panel) and gradient or Hessian-vector product computation (second panel).
10


Reviews:
[{"review_id": "HyeCPFQo9r", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "https://openreview.net/pdf?id=SkxaueHFPB\n\nThe paper has some interesting ideas but I don’t think any of them are fully fleshed out.\n\nI find the reporting of Inception Score highly suspect. The authors choose WGAN-GP as a baseline and report scores of ~4.5 vs ~5.5 with their modification. However the WGAN-GP paper reports an IS of 7.86 on CIFAR. Furthermore, current GAN SOTA on CIFAR is approaching IS=9. I am not making the argument that the authors ought to demonstrate SOTA results, however they should at least present results which are consistent with the published results of their chosen baseline.\n\nThe authors then make this statement:\n“Thus, its superior performance supports the claim that ICR is the appropriate form of regularization for GANs. We emphasize that in our experiments we did not perform any architecture or hyperparameter tuning, and instead use a model intended to be used with WGAN gradient penalty”\nThis does not hold, since the numbers reported are far below the actual baseline.\n\nBesides this major point, I am unconvinced by some of the mathematical statements in the paper. Much of the mathematical details are deferred to the original CGD paper. It is not really particularly reader-friendly to defer that to the CGD paper since they are seemingly crucial to the discussion here. Relative to the CGD paper some signs have been flipped and some definitions appear to be used in subtly different ways which makes for a very difficult read. I feel that far too much has been left as an exercise to the reader.\n\nConcretely my concerns refer to the main discussion of the effect of the CGD as a regularizer:\n\nThe authors state:\n“If some of the singular values of Dxy are very large, this amounts to approximately restricting the update to the orthogonal complement of the corresponding singular vectors” \n\nI don’t see how this is the case. The terms Dxy/Dyx aren’t really introduced or defined anywhere in this work. Assuming is the transpose of the other (?) then the update direction is:\nA + B where A=inv(S) grad_x and B = inv(S)Dxy grad_y (and S = I + Dxy Dyx). So we have a term which is being affected by the smallest singular values of S and a term which is the orthogonal projection of grad_y onto Dxy, alternatively the ridge-regression fit of grad_y on Dxy which would attenuate directions corresponding to the *small* singular values (as is well known from the theory of ridge regularizers). I feel like there is much more to say here than what is discussed in the paper in very vague terms. \n\nOf course the effective rank of S, or the rate of decay of its singular values is crucially important. In practise I would assume the smaller SVs of Dxy to be difficult to estimate or the matrix to be rank deficient in which case they would simply be unity in the inverse whereas the directions corresponding to large singular values would be attenuated. So in this case it is the regularized orthogonal complement but its not clear (if the matrix is not full rank) that it is a meaningful direction (and again this is all highly dependent on the effective rank, too).\n\nFurther on it is mentioned: “For smoothly varying singular vectors, this can be though of as approximately constraining the trajectories to a manifold of robust play”.\n\nFirst it is not at all clear to me what “smoothly varying singular vectors” are. Varying with respect to what? Secondly, the “manifold of robust play” has not been defined anywhere. \n\nFinally, figure 3 is quite bizarre to me. None of the quantities have been rigorously defined and so it seems like the relative effect of each of the arrows and the manifold have been drawn arbitrarily in order to fit the story, rather than to actually illuminate the true behaviour in an intuitive manner. \nB (defined above) has a very clear interpretation as a least-squares fit so I figure that any geometric interpretation of the CGD update direction could start from there.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BkeoDK4qqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper analyzes instability in training GANs, relates it to Nash equilibria, and proposes a novel training set-up based on competitive nashless games. The solution is related to other recently proposed work, but the paper brings additional insights into understanding it.\n\nThe analysis on conditions that lead to divergence or convergence, and of the proposed solution, are interesting. I recommend accepting.\n\nI have some basic knowledge of GANs but am not deeply familiar with the field. The paper was accessible to me on a high level. Especially compelling to me were sections 2 and 3. The empirical study also seemed to yield positive results.\n\nSuggestions for improvement:\n- Additional proofreading would be beneficial.\n- The scale of the axes in figure 1 is not clear, making it a little less compelling.\n- Inception score is used as the only evaluation metric for the generators. Perhaps this is standard in the field, although human ratings would seem more reliable to me.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "B1xuDl80YB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a new way of regularization in Generative Adversarial Network (GAN). It is well known that a naive training of GAN can fail to converge. Although GAN is relatively a new concept, many papers tried to introduce a good way of stabilizing GAN training. I believe that this paper is addressing the stability issues in the most fundamental and effective ways. The paper utilizes Competitive Gradient Descent proposed by Schäfer and Anandkumar in 2019 in training GAN. The intuition is that both players should predict what their opponent is going to do. This results in a convergence point where each agent becomes robust against changes of the other agent. The performance of the new method was demonstrated on CIFAR10.\n\nThe paper is definitely interesting. If this method works as well as the authors claim, it can significantly improve the practicality of GAN. The paper is very readable and understandable but many small typos and grammar errors can be found in the text. This can be easily corrected by the authors.\n\nHowever, the contribution of this paper is questionable. The original CGD paper already applies it to train a GAN.\n\nI would also appreciate if the method is tested on multiple other data sets. \n\nOverall, the paper is well-written, technically correct and interesting enough for the venue. However, as I pointed out above, the contribution should be more clearly stated.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1e_HcwAFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a novel training methodology for GANs to improve stability. The resulting regularization, termed implicit competitive regularization, updates the parameters of both the generator and the discriminator to be robust to one another. A framework for practical application of this approach is described -- this is done by a local Taylor approximation of the loss and updating each model’s parameters to this approximate model’s nash equilibrium.  The method is shown to prevent overfitting and produce high-performing models with consistent training.\n\nThe approach and insights are reasonable and the problem is worthwhile to approach. The method is clear and the associated code is appreciated. The results are interesting in terms of describing the ICR property and demonstrating its performance. \n\nThe paper gives background and intuition to solidifying the CGD update. Are there additional algorithmic approaches that are possible and potentially more efficient with this understanding in mind? This I believe would help solidify the paper and build beyond CGD.\n\nSome additional results that could clarify the benefits:\n- A primary contribution of the training approach is training consistency. The distribution over many training runs should be provided in figures.\n- Clearly due to the additional gradient calls the approach is computationally slower, as shown in Figure 4. If each approach is trained for the same amount of time, how does the performance compare?\n- One may expect that the update may result in more conservative updates and thus potentially lower-performing policies in the limit. If there iterations were instead log-scale to show performance in high training iterations, is there any loss of performance in top-performing runs? \n- Could a similar approach be used to allow safe gradient updates according to a risk over the opponent’s possible updates, e.g., via CVar? This may also be a stable training procedure as well with less conservatism.  \n\nThe paper should be proofread, there are several minor typos throughout, e.g.:\n- “generators producing that produce good” -> generators that produce good\n- “This game is very similar similar” -> repeated word\n- “GAN trainin.”\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to study "implicit competitive regularization", a phenomenon borne of taking a more nuanced game theoretic perspective on GAN training, wherein the two competing networks are "model[ed] ... as agents acting with limited information and in awareness of their opponent". The meaning of this is developed through a series of examples using simpler games and didactic experiments on actual GANs. An adversary-aware variant employing a Taylor approximation to the loss. 

Reviewer assessment amounted to 3 relatively light reviews, two of which reported little background in the area, and one more in-depth review, which happened to also be the most critical. R1, R2, R3 all felt the contribution was interesting and valuable. R1 felt the contribution of the paper may be on the light side given the original competitive gradient descent paper, on which this manuscript leans heavily, included GAN training (the authors disagreed); they also felt the paper would be stronger with additional datasets in the empirical evaluation (this was not addressed). R2 felt the work suffered for lack of evidence of consistency via repeated experiments, which the authors explained was due to the resource-intensity of the experiments. 

R5 raised that Inception scores for both the method and being noticeably worse than those reported in the literature, a concern that was resolved in an update and seemed to center on the software implementation of the metric. R5 had several technical concerns, but was generally unhappy with the presentation and finishedness of the manuscript, in particular the degree to which details are deferred to the CGD paper. (The authors maintain that CGD is but one instantiation of a more general framework, but given that the empirical section of the paper relies on this instantiation I would concur that it is under-treated.)

Minor updates were made to the paper, but R5 remains unconvinced (other reviewers did not revisit their reviews at all). In particular: experiments seem promising but not final (repeatability is a concern), the single paragraph "intuitive explanation" and cartoon offered in Figure 3 were viewed as insufficiently rigorous. A great deal of the paper is spent on simple cases, but not much is said about ICR specifically in those cases. 

This appears to have the makings of an important contribution, but I concur with R5 that it is not quite ready for mass consumption. As is, the narrative is locally consistent but quite difficult to follow section after section. It should also be noted that ICLR as a venue has a community that is not as steeped in the game theory literature as the authors clearly are, and the assumed technical background is quite substantial here. For a game theory novice, it is difficult to tell which turns of phrase refer to concepts from game theory and which may be more informally introduced herein. I believe the paper requires redrafting for greater clarity with a more rigorous theoretical and/or empirical characterization of ICR, perhaps involving small scale experiments which clearly demonstrates the effect. I also believe the authors have done themselves a disservice by not availing themselves of 10 pages rather than 8.

I recommend rejection at this time, but hope that the authors view this feedback as valuable and continue to improve their manuscript, as I (and the reviewers) believe this line of work has the potential to be quite impactful.
Decision: Reject