DO-GAN: A Double Oracle Framework for Generative Adversarial Networks

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) have been applied in various domains such as image and video generation , image-to-image translation and text-to-image synthesis ( Liu et al. , 2017 ; Reed et al. , 2016 ) . Various architectures are proposed to generate more realistic samples ( Radford et al. , 2015 ; Mirza & Osindero , 2014 ; Pu et al. , 2016 ) as well as regularization techniques ( Arjovsky et al. , 2017 ; Miyato et al. , 2018b ) . From the game-theoretic perspective , GANs can be viewed as a two-player game where the generator samples the data and the discriminator classifies the data as real or generated . The two networks are alternately trained to maximize their respective utilities until convergence corresponding to a pure Nash Equilibrium ( NE ) . However , pure NE can not be reliably reached by existing algorithms as pure NE may not exist ( Farnia & Ozdaglar , 2020 ; Mescheder et al. , 2017 ) . This also leads to unstable training in GANs depending on the data and the hyperparameters . Therefore , mixed NE is a more suitable solution concept ( Hsieh et al. , 2019 ) . Several recent works propose mixture architectures with multiple generators and discriminators that consider mixed NE such as MIX+GAN ( Arora et al. , 2017 ) and MGAN ( Hoang et al. , 2018 ) . MIX+GAN and MGAN can not guarantee to converge to mixed NE . Mirror-GAN ( Hsieh et al. , 2019 ) finds the mixed NE by sampling over the infinite-dimensional strategy space and proposes provably convergent proximal methods . However , the sampling approach may not be efficient as mixed NE may only have a few strategies in the support set . Double Oracle ( DO ) algorithm ( McMahan et al. , 2003 ) is a powerful framework to compute mixed NE in large-scale games . The algorithm starts with a restricted game with a small set of actions and solves it to get the NE strategies of the restricted game . The algorithm then computes players ’ best-responses using oracles to the NE strategies and add them into the restricted game for the next iteration . DO framework has been applied in various disciplines ( Jain et al. , 2011 ; Bošanský et al. , 2013 ) , as well as Multi-agent Reinforcement Learning ( MARL ) settings ( Lanctot et al. , 2017 ) . Inspired by the successful applications of DO framework , we , for the first time , propose a Double Oracle Framework for Generative Adversarial Networks ( DO-GAN ) . This paper presents four key contributions . First , we treat the generator and the discriminator as players and obtain the best responses from their oracles and add the utilities to a meta-matrix . Second , we propose a linear program to obtain the probability distributions of the players ’ pure strategies ( meta-strategies ) for the respective oracles . The linear program computes an exact mixed NE of the meta-matrix game in polynomial time . Third , we propose a pruning method for the support set of best response strategies to prevent the oracles from becoming intractable as there is a risk of the meta-matrix growing very large with each iteration of oracle training . Finally , we provide comprehensive evaluation on the performance of DO-GAN with different GAN architectures using both synthetic and real-world datasets . Experiment results show that DO-GAN variants have significant improvements in terms of both subjective qualitative evaluation and quantitative metrics . 2 RELATED WORKS . In this section , we briefly introduce existing GAN architectures , double oracle algorithm and its applications such as policy-state response oracles that are related to our work . GAN Architectures . Various GAN architectures have been proposed to improve the performance of GANs . Deep Convolutional GAN ( DCGAN ) ( Radford et al. , 2015 ) replaces fully-connected layers in the generator and the discriminator with deconvolution layer of Convolutional Neural Networks ( CNN ) . Weight normalization techniques such as Spectral Normalization GAN ( SNGAN ) ( Miyato et al. , 2018a ) stabilize the training of the discriminator and reduce the intensive hyperparameters tuning . There are also multi-model architectures such as Stacked Generative Adversarial Networks ( SGAN ) ( Huang et al. , 2017 ) that consist of a top-down stack of generators and a bottom-up discriminator network . Each generator is trained to generate lower-level representations conditioned on higher-level representations that can fool the corresponding representation discriminator . Training GANs is very hard and unstable as pure NE for GANs might not exist and can not be reliably reached by the existing approaches ( Mescheder et al. , 2017 ) . Considering mixed NE , MIX+GAN ( Arora et al. , 2017 ) maintains a mixture of generators and discriminators with the same network architecture but have their own trainable parameters . However , training a mixture of networks without parameter sharing makes the algorithm computationally expensive . Mixture Generative Adversarial Nets ( MGAN ) ( Hoang et al. , 2018 ) propose to capture diverse data modes by formulating GAN as a game between a classifier , a discriminator and multiple generators with parameter sharing . However , MIX+GAN and MGAN can not converge to mixed NE . Mirror-GAN ( Hsieh et al. , 2019 ) finds the mixed NE by sampling over the infinite-dimensional strategy space and proposes provably convergent proximal methods . The sampling approach may be inefficient to compute mixed NE as the mixed NE may only have a few strategies with positive probabilities in the infinite strategy space . Double Oracle Algorithm . Double Oracle ( DO ) algorithm starts with a small restricted game between two players and solves it to get the player strategies at NE of the restricted game . The algorithm then exploits the respective best response oracles for additional strategies of the players . The DO algorithm terminates when the best response utilities are not higher than the equilibrium utility of the current restricted game , hence , finding the NE of the game without enumerating the entire strategy space . Moreover , in two-player zero-sum games , DO converges to a min-max equilibrium ( McMahan et al. , 2003 ) . DO framework is used to solve large-scale normal-form and extensive-form games such as security games ( Tsai et al. , 2012 ; Jain et al. , 2011 ) , poker games ( Waugh et al. , 2009 ) and search games ( Bosansky et al. , 2012 ) . DO framework is also used in MARL settings ( Lanctot et al. , 2017 ; Muller et al. , 2020 ) . Policy-Space Response Oracles ( PSRO ) generalize the double oracle algorithm in a multi-agent reinforcement learning setting ( Lanctot et al. , 2017 ) . PSRO treats the players ’ policies as the best responses from the agents ’ oracles , builds the meta-matrix game and computes the mixed NE but it uses Projected Replicator Dynamics that update the changes in the probability of each player ’ s policy at each iteration . Since the dynamics need to simulate the update for several iterations , the use of dynamics takes a longer time to compute the meta-strategies and does not guarantee to compute an exact NE of the meta-matrix game . However , in DO-GAN , we can use a linear program to compute the players ’ meta-strategies in polynomial time since GAN is a two-player zero-sum game ( Schrijver , 1998 ) . 3 PRELIMINARIES . In this section , we mathematically explain the preliminary works that are needed to explain our DO-GAN approach including generative adversarial networks and game theory concepts such as normal-form game and double oracle algorithm . 3.1 GENERATIVE ADVERSARIAL NETWORKS . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) have become one of the dominant methods for fitting generative models to complicated real-life data . GANs are deep neural net architectures comprised of two neural networks trained in an adversarial manner to generate data that resembles a distribution . The first neural network , a generator G , is given some random distribution pz ( z ) on the input noise z and a real data distribution pdata ( x ) on training data x . The generator is supposed to generate as close as possible to pdata ( x ) . The second neural network , a discriminator D , is to discriminate between two different classes of data ( real or fake ) from the generator . Let the generator ’ s differentiable function be denoted as G ( z , πg ) and similarly D ( x , πd ) for the discriminator , where G and D are two neural networks with parameters πg and πd . Thus , D ( x ) represents the probability that x comes from the real data . The generator lossLG and the discriminator loss LD are defined as : LD = Ex∼pdata ( x ) [ − logD ( x ) ] + Ez∼pz ( z ) [ − log ( 1−D ( G ( z ) ) ] , ( 1 ) LG = Ez∼pz ( z ) [ log ( 1−D ( G ( z ) ) ] . ( 2 ) GAN is then set up as a two-player zero-sum game between G and D as follows : minG maxD Ex∼pdata ( x ) [ logD ( x ) ] + Ez∼pz ( z ) [ log ( 1−D ( G ( z ) ) ] . ( 3 ) During training , the parameters of G and D are updated alternately until we reach the global optimal solution D ( G ( z ) ) = 0.5 . Next , we let Πg and Πd be the set of parameters for G and D , considering the set of probability distributions σg and σd , the mixed strategy formulation ( Hsieh et al. , 2019 ) is : min σg max σd Eπd∼σdEx∼pdata ( x ) [ logD ( x , πd ) ] + Eπd∼σdEπg∼σgEz∼pz ( z ) [ log ( 1−D ( G ( z , πg ) , πd ) ] . ( 4 ) Similarly to GANs , DCGAN , SNGAN and SGAN can also be viewed as two-player zero-sum games with mixed strategies of the players . DCGAN modifies the vanilla GAN by replacing fully-connected layers with the convolutional layers . SGAN trains multiple generators and discriminators using the loss as a linear combination of 3 loss terms : adversarial loss , conditional loss and entropy loss . 3.2 NORMAL FORM GAME AND DOUBLE ORACLE ALGORITHM . A normal-form game is a tuple ( Π , U , n ) where n is the number of players , Π = ( Π1 , . . . , Πn ) is the set of strategies for each player i ∈ N , where N = { 1 , . . . , n } and U : Π→ Rn is a payoff table of utilities R for each joint policy played by all players . Each player chooses the strategy to maximize own expected utility from Πi , or by sampling from a distribution over the set of strategies σi ∈ ∆ ( Πi ) . We can use linear programming , fictitious play ( Berger , 2007 ) or regret minimization ( Roughgarden , 2010 ) to compute the probability distribution over players ’ strategies . In the Double Oracle ( DO ) algorithm ( McMahan et al. , 2003 ) , there are two best response oracles for the row and column player respectively . The algorithm creates restricted games from a subset of strategies at the point of each iteration t for row and column players , i.e. , Πtr ⊂ Πr and Πtc ⊂ Πc as well as a meta-matrix U t at the tth iteration . We then solve the meta-matrix to get the probability distributions on Πtr and Π t c. Given a probability distribution σc of the column player strategies , BRr ( σc ) gives the row player ’ s best response to σc . Similarly , given probability distribution σr of the row player ’ s strategies , BRc ( σr ) is the column player ’ s best response to σr . The best responses are added to the restricted game for the next iteration . The algorithm terminates when the best response utilities are not higher than the equilibrium utility of current restricted game . Although in the worst-case , the entire strategy space may be added to the restricted game , DO is guaranteed to converge to mixed NE in two-player zero-sum games . DO is also extended to the multi-agent reinforcement learning in PSRO ( Lanctot et al. , 2017 ) to approximate the best responses to the mixtures of agents ’ policies , and compute the meta-strategies for the policy selection .

Summary: This paper applies Double-Oracle (DO) / PSRO to training a GAN, a 2-player zero-sum game. DO cannot be applied directly "out-of-the-box". Instead of an exact oracle, the generator and discriminator are trained using local gradient optimizers for a finite number of steps. Also, in DO, the meta-game matrix can grow very large and maintaining and training against a large population of neural networks is expensive, so the population is pruned throughout training.