Implicit competitive regularization in GANs

1 INTRODUCTION . Generative adversarial networks ( GANs ) : ( Goodfellow et al. , 2014 ) are a class of generative models based on a competitive game between a generator that tries to generate realistic new data , and a discriminator , that tries to distinguish generated from real data . In the original formulation , the two players are playing a zero-sum game with the loss function of the generator given by the binary cross entropy , min G max D 1 2 Ex∼Pdata [ logD ( x ) ] + 1 2 Ex∼G [ log ( 1−D ( x ) ) ] . ( 1 ) Here , G is the probability distribution generated by the generator , D is the classifier provided by the discriminator , and Pdata is the target measure , for example the empirical distribution of the training data . In practice , both players are parameterized by neural networks that are trained simultaneously by a variant of stochastic gradient descent ( SGD ) . Instability and mode collapse : GANs constitute the state of the art ( SOTA ) on many problems , but suffer from a variety of issues such as instability and mode collapse , i.e . a drastic loss of sample diversity , during training . Beginning with ( Goodfellow et al. , 2014 ) , GANs have often been interpreted as the generator performing approximate minimization of a function defined implicitly by maximization over the discriminator . From this perspective , instability and mode collapse have been explained with the fact that for finite amounts of data and any continuous distribution G produced by the generator , a sufficiently powerful discriminator will be able to fully maximize the two summands in Equation 1 independently of each other , instead of searching for a trade-off . This is because it can first maximize the second summand and then modify its value in -neighborhoods of the true data to maximize the first summand . In the limit of small , these modifications don ’ t influence the second objective , leading to the discriminator maximizing both objectives . While one could hope that this problem vanishes for large data sets , real data will often be concentrated around low-dimensional structure , which can be exploited by the discriminator in a similar fashion . Thus , for any fixed generator we expect the weights of the discriminator to diverge as it fully performs the above trick , causing instability and preventing the generator from receiving useful gradient information . Gradient penalties need to choose a metric on sample space : The above observation has motivated restriction of the size of the gradients of D ( as a function on sample space ) ( Arjovsky & Bottou , 2017 ; Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ; Roth et al. , 2017 ; Kodali et al. , 2017 ; Miyato et al. , 2018 ) . Measuring the size of the gradient of D requires a metric on the space of samples , typically chosen to be ` 2 . For complex samples like images , this will be a crude measure of similarity . In images of human faces for instance , skin– and hair color will likely dominate over other subtle features like facial expression in terms of the ` 2-norm . In contrast , the original GAN has the intriguing property of not requiring any explicit choice of distance on sample space . The game behind GANs : For convex-concave games , Nash equilbria provide a natural solution concept . We merely need to choose which algorithm to use when computing them . For non-convexconcave games like GANs however , Nash equilibria need not be the right solution concept ( see Berard et al . ( 2019 ) and Figure 2 ) , which spurred a search for new solution concepts ( Jin et al. , 2019 ; Fiez et al. , 2019 ) . Both the choice of a loss function and of a notion of solution is a modeling problem . We argue that rather than deriving an algorithm based on these choices , it might be preferable to directly model iterative games between the two players . Just like ” the goal is to capture the other player ’ s king ” does not fully specify the rules of chess , Equation 1 does not fully specify a game between the two players . Rather , we need to specify what information the players have access to , which ” moves ” are allowed , and in what order are they are to be made . Even with the rules completely specified , the behavior of the players will crucially depend on the heuristics that they use to anticipate each other ’ s behavior and choose their own strategy , possibly in the face of uncertainty . Finally , a chess player might prefer a certain draw over an uncertain victory depending on the format of the tournament . Similarly , players aiming to minimize an averaged loss might play differently from players that greedily minimize the loss after every move . Even for the same loss function , different such assumptions lead to different dynamics in GANs . Rather than trying to define a suitable solution concept based on , for instance , the spectrum of the Hessian , it might be more fruitful to directly model the behavior of the two players . Our contributions : In this work we show that modeling the generator and discriminator as agents making decisions under uncertainty and in awareness of each others ’ goals can lead to stable behavior . Surprisingly , this holds even in the absence of Nash equilibria and for games that are meaningless in the minimax sense . We show empirically that even generators that produce good samples can incur large losses under a discriminator optimized for this particular generator , but such discriminators are highly vulnerable to changes of the generator ’ s strategy ( see Figure 2 ) . Therefore , a discriminator aware of the competitive nature of the game will try to find a trade-off between achieving a low loss and being robust to the actions of the generator , and vice versa . Based on this observation , we propose to mitigate the lack of stability in GAN training by choosing more realistic models of the agents ’ behavior . One realization of this idea is the recently proposed competitive gradient descent ( CGD ) algorithm ( Schäfer & Anandkumar , 2019 ) , where players update their parameters using the Nash equilibrium of local approximations of the loss functions that takes into account actions of the other player . We show that it imposes an implicit competitive regularization ( ICR ) on the updates , leading to moves that are locally robust to the other player ’ s response . We conclude with an extensive suite of experiments using Adam and CGD on the original GAN loss from Equation 1 and the bilinear WGAN loss ( Arjovsky & Bottou , 2017 ; Arjovsky et al. , 2017 ) , and a variety of different regularizers . We observe that the performance of CGD is much more consistent than that of Adam and that the highest inception score is achieved by combining the WGAN loss ( without gradient penalty or weight clipping ) with CGD . Recall that when combined with a Lipschitz constraint , WGAN minimizes the Wasserstein distance between generator and target distribution . By analogy , the combination of WGAN loss with CGD can be interpreted as minimizing an integral probability metric ( Sriperumbudur et al. , 2009 ) based on ICR . Thus , its superior performance supports the claim that ICR is the appropriate form of regularization for GANs . We emphasize that in our experiments we did not perform any architecture or hyperparameter tuning , and instead use a model intended to be used with WGAN gradient penalty ( Gulrajani et al. , 2017 ) . 2 THE NASHLESS GAME . What is the solution , if there is no solution ? We believe that rather than viewing the loss function of a GAN as specifying a well-defined computational problem that is solved by computing a local Nash equilibrium using one of the many methods proposed in the literature , the choice of the algorithm should part of the modeling problem . In order to make this more concrete , we consider the zero-sum game given by min x∈R max y∈R − exp ( x2 ) − αxy + exp ( y2 ) , ( 2 ) which we will refer to as the nashless game . Here , α 1 is a large but fixed parameter . This game does not have a global Nash equilibrium , since each player can always achieve an exponentially increasing reward by moving towards infinity . It furthermore does not have any local Nash equilibria , since the curvature of the objective function is always negative in x and positive in y . One approach would be to interpret it as a Stackelberg game where player y gets to make their move after player x . However , since player y can always achieve arbitrarily large rewards by diverging to infinity this does still not result in a notion of solution . ( Jin et al. , 2019 , Definition 14 ) introduced a notion of local minmax point ( x∗ , y∗ ) that amounts to restricting the movement of the follower to within a small distance of y∗ . However they also require the objective to be a local minimum with respect to y , which prevents it from being adequate for the game . ( Fiez et al. , 2019 , Definition 4 ) propose a notion of differential Stackelberg equilibrium . However , since it is only based on first order information it will implicitly use the worst strategy of the follower . Thus , the leader can always locally improve by moving further towards infinity , preventing the game from having a differential Stackelberg equilibrium in the sense of ( Fiez et al. , 2019 , Defininition 4. ) . While it is possible to define other notions of equilibrium , and hence solutions , for problems like in Equation 2 , the above discussion illustrates that it is much harder than in the convex-concave setting . An alternative way of turning equation 2 into an algorithm is to turn it into an iterative game between the two players , and obtain the algorithm from a suitable notion of rational behavior . Three games for one loss function : To illustrate the different dynamics that can arise , we now equip the loss function of Equation 2 with three different sets of rules for iterative play . 1 . The global game : The two players move simultaneously , a distance of at most 1 . They have access to the entire optimization landscape and aim to minimize their average loss in the asymptotic limit of infinite time . 2 . The myopic game : The two players move simultaneously at a distance of at most 1 . At the k-th step , the player x ( y ) only has access to objective function on [ xk − 1 , xk + 1 ] × { y } ( { x } × [ yk − 1 , yk + 1 ] ) and tries to minimize the loss at step k + 1 . 3 . The predictive game : The two players move simultaneously at a distance of at most 1 . At the k-th step , both players have access to the objective function on [ xk − 1 , xk + 1 ] × [ yk − 1 , yk + 1 ] ) and try to minimize their loss at step k+1 . The players are furthermore aware of each other ’ s goals . In the global game , player y can win the game with a loss of −∞ by diverging to infinity as quickly as possible . This is similar in spirit to the maximization overD in the original GAN where the global optimization over D results in unstable behavior of the underlying game ( see Figure1 ) Since the global optimization of nonconvex problems is computationally infeasible , the first set of rules is not a suitable model for GAN training . In myopic game , we model the use of local algorithms by only giving the players access to the loss function in a local neighborhood of their present position . Initially of the exponentially increasing reward at infinity they try to improve their reward locally , entering the cycling dynamics also known from simultaneous gradient descent ( see Figure 1 ) . While the cycling behavior slows down the divergence , the players will eventually diverge far enough for the exponential term to overpower the bilinear term , from where on both players will diverge to infinity , without further cycling behavior . In predictive game , the two players have access not only to their own , but also to the other player ’ s loss function as well as the effect that the other player ’ s moves will have on their own loss . Therefore , they don ’ t blindly aim to increase their own loss function but also try to estimate the actions of the other player and adapt choose their own actions accordingly . In the initial position x = 0 , y = 2 , a player y that is unaware of the presence of x will choose to move to 3 , in order to minimize its loss . If y is aware of the goals of x however , it will know that whatever move y makes , the best move for x will be to move to 1 . Thus , player y should make its move based on the assumption that x will move to 1 and hence ( since α 1 ) moves to 1 instead . For the given starting conditions there always exist unique optimal strategies for x and y resulting in the periodic behavior illustrated in Figure 1 . The above example shows that even in the absence of suitable equilibria we can obtain meaningful and stable dynamics by modeling the players as agents acting with limited information and awareness of their opponents . Before providing a possible framework for this type of modeling , we will provide evidence that key features of the nashless game might indeed be present in GANs .

Summary: The paper presents a novel training methodology for GANs to improve stability. The resulting regularization, termed implicit competitive regularization, updates the parameters of both the generator and the discriminator to be robust to one another. A framework for practical application of this approach is described -- this is done by a local Taylor approximation of the loss and updating each model’s parameters to this approximate model’s nash equilibrium.  The method is shown to prevent overfitting and produce high-performing models with consistent training.