Provably convergent quasistatic dynamics for mean-field two-player zero-sum games

1 INTRODUCTION . Finding Nash equilibrium has seen many important applications in machine learning , such as generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014a ) and reinforcement learning ( Busoniu et al. , 2008 ) . In these problems , pure Nash equilibria are usually search for a function f ( x , y ) . Yet , the problems arising from machine learning are usually nonconvex in x and nonconcave in y , in which case pure Nash equilibrium may not exist . And even if it exists , there is no guarantee for any optimization algorithm to find it efficiently . This difficulty is reflected in practice , that compared with simple minimization , machine learning applications involving Nash equilibria usually have more complicated behaviors and more subtle dependence on hyper-parameters . For example , stable and efficient training of GANs requires a number of carefully designed tricks ( Gao et al. , 2018 ) . On the other hand , the mixed Nash equilibrium ( MNE ) is known to exist in much more general settings , e.g . when the strategy spaces are compact and the payoff function is continuous ( Glicksberg , 1952 ) . In the mixed Nash equilibrium problem , instead of taking “ pure strategies ” x and y , two ” mixed strategies ” for x and y , in the form of probability distributions , are considered , resulting in the following functional , ∫ f ( x , y ) p ( x ) q ( y ) dxdy , where p and q are density functions of probability distributions of x and y , respectively . Efforts are invested to develop theoretically endorsed algorithms that can efficiently find MNE for high dimensional problems , with applications on the training of mixture of GANs . In Hsieh et al . ( 2019 ) , a mirror-descent algorithm is proposed and its convergence is proven . In Domingo-Enrich et al . ( 2020 ) , theoretical analysis and empirical experiments are conducted for a gradient descent-ascent flow under a Wasserstein-Fisher-Rao metric and its particle discretization . In this paper , we also consider the mixed Nash equilibrium problem , and propose a simple QuasiStatic Wasserstein Gradient Flow ( QSWGF ) for solving the problem . In our dynamics , we treat q as a component with much faster speed than p , hence is always at equilibrium as p moves . With entropy regularization for both p and q ( without requirement on the strength of the regularizations ) , we prove that the QSWGF converges to the unique mixed Nash equilibrium from any initialization ( under mild conditions ) . Furthermore , we show there is a simple way to discretize the QSWGF , regardless of the complexity of the Wasserstein gradient flow of p induced by the fact that q is always at equilibrium . Concretely , a partition function related with p appears in the QSWGF dynamics , and we find an efficient way to approximate the partition function . By discretizing the QSWGF , we derive a particle dynamics with an inner-outer structure , named the QuasiStatic Langevin Gradient Descent algorithm ( QSLGD ) . In QSLGD , after each iteration of the outer problem ( for the x particles ) , the inner loop conducts sufficient iterations to bring the y particles to equilibrium . Numerical experiments show the effectiveness of QSLGD on synthetic examples and training mixture of GANs . Our method outperforms the vanilla Langevin gradient descent-ascent method when the entropy regularization is weak . As a summary , our two major contributions are : 1 . We propose the quasistatic Wasserstein gradient flow dynamics for mixed Nash equilibrium problems , and show its convergence to the unique Nash equilibrium under weak assumptions . Our result neither requires the entropy regularization to be sufficiently strong , nor assumes the dynamics to converge a priori . 2 . We derive a simple while practical quasistatic Langevin gradient descent algorithm by discretizing the quasistatic Wasserstein gradient flow , by finding an efficient way to approximate the partition function appearing in the dynamics of p. The proposed algorithm is applied on several problems including training mixtures of GANs . 2 RELATED WORK . The mixed Nash equilibrium problem has a long history , with the proof of its existence dates back to Morgenstern & Von Neumann ( 1953 ) . It draws new attention in recent years , especially in the machine learning community , due to the development of GANs ( Goodfellow et al. , 2014a ) and adversarial training ( Goodfellow et al. , 2014b ) . Training mixture of GANs is already discussed in paper ( Goodfellow et al. , 2014a ) . Some numerical experiments were conducted in ( Arora et al. , 2017 ) . In Grnarova et al . ( 2017 ) , the authors proposed an online learning approach for training mixture of GANs , and proved its effectiveness for semi-shallow GANs ( GANs whose discriminator is a shallow neural network ) . Yet , rigorous theoretical treatment to an algorithm started from ( Hsieh et al. , 2019 ) , in which a mirror descent method was studied and proven to converge . The implementation of the mirror descent method involves big computational cost that asks for heuristics to alleviate . Later , ( Domingo-Enrich et al. , 2020 ) studied more efficient algorithms under a mixture of Wasserstein and Fisher-Rao metrics . Theoretically , the time average of the dynamics ’ trajectories is shown to converge to the mixed Nash equilibrium . As a comparison , in this work we show the global convergence of the quasistatic Wasserstein gradient flow without the need of taking time average . Meanwhile , the Wasserstein nature of our dynamics makes it easy to implement as well . The Wasserstein gradient flow in the density space has been explored in previous works . For example , ( Wang & Li , 2019 ) studied the Nesterov ’ s accelerated gradient flows for probability distributions under the Wasserstein metric , and ( Arbel et al. , 2019 ) studied practical implementations of the natural gradient method for the Wasserstein metric . Both works focus on minimization problems instead of min-max problems considered in this work . A more related work is Lin et al . ( 2021b ) , where a natural gradient based algorithm is proposed for training GANs . Yet , the method still optimizes one generator and one discriminator , searching for pure Nash equilibrium . Another work that derives algorithms for GANs from a Wasserstein perspective is ( Lin et al. , 2021a ) . Another volume of works that studies the Wasserstein gradient flow in the machine learning context is the mean-field analysis of neural networks . This line of works started from two-layer neural networks ( Mei et al. , 2018 ; Rotskoff & Vanden-Eijnden , 2018 ; Chizat & Bach , 2018 ; Sirignano & Spiliopoulos , 2020 ) , to deep fully-connected networks ( Araújo et al. , 2019 ; Sirignano & Spiliopoulos , 2021 ; Nguyen , 2019 ; Wojtowytsch et al. , 2020 ) , and residual networks ( Lu et al. , 2020 ; E et al. , 2020 ) . The mean-field formulations treat parameters as probability distributions , and the training dynamics are usually the gradient flow under Wasserstein metric . Attempts to prove convergence of the dynamics to global minima are made ( Mei et al. , 2018 ; Chizat & Bach , 2018 ; Rotskoff et al. , 2019 ) , though in the case without entropy regularization a convergence assumption should usually be made a priori . 3 THE QUASISTATIC DYNAMICS . We consider the entropy regularized mixed Nash equilibrium problem , which in our case is equivalent with solving the following minimax problem : min p∈P ( Ω ) max q∈P ( Ω ) ∫ Ω×Ω K ( x , y ) p ( x ) q ( y ) dxdy + β−1 ∫ Ω p log pdx− β−1 ∫ Ω q log qdy . ( 1 ) In ( 1 ) , Ω is a compact Riemannian manifold without boundary , and P ( Ω ) is the set of probability distributions on Ω . Since Ω is compact , any probability distribution in P ( Ω ) naturally has finite moments . Let E ( p , q ) = ∫ Ω×ΩK ( x , y ) p ( dx ) q ( dy ) , and S ( p ) = ∫ Ω p log pdx and S ( q ) = ∫ Ω q log qdy be the ( negative ) entropy of p and q , respectively . Then , the minimax problem ( 1 ) can be written in short as min p∈P ( Ω ) max q∈P ( Ω ) E ( p , q ) + β−1S ( p ) − β−1S ( q ) . ( 2 ) Remark 1 . Strictly speaking , in ( 1 ) we should distinguish probability distributions and their density function ( if exist ) , and the entropy should also be defined using the Radon-Nikodym derivative with canonical measure . In this paper , since p and q indeed have density functions because of the entropy regularization , we shall abuse the notation by using p and q to represent both probability distributions and their density functions . The entropy regularizations in ( 1 ) and ( 2 ) make the problem strongly convex in p and strongly concave in q . Hence , there exists a unique Nash equilibrium for the problem . Such results are shown for example by the following theorem from ( Domingo-Enrich et al. , 2020 ) . Theorem 1 . ( Theorem 4 of ( Domingo-Enrich et al. , 2020 ) ) Assume Ω is a compact Polish metric space equipped with canonical Borel measure , and that K is a continuous function on Ω × Ω . Then , problem ( 2 ) has a unique Nash equilibrium given by the solution of the following fixed-point problem : p ( x ) = 1 Zp exp ( −βU ( x , q ) ) , q ( x ) = 1 Zq exp ( βV ( y , p ) ) , ( 3 ) where Zp and Zq are normalization constants to make sure p and q are probability distributions , and U and V are defined as U ( x , q ) = δE ( p , q ) δp ( x ) = ∫ Ω K ( x , y ) q ( y ) dy , V ( y , p ) = δE ( p , q ) δq ( y ) = ∫ Ω K ( x , y ) p ( x ) dx . Considering the efficiency in high-dimensional cases , a natural dynamics of interest to find the Nash equilibrium for ( 2 ) is the gradient descent-ascent flow under the Wasserstein metric , ∂tpt = ∇ · ( pt∇ ( U ( x , qt ) + β−1 log pt ) ) , ∂tqt = ∇ · ( qt∇ ( −V ( y , pt ) + β−1 log qt ) ) , ( 4 ) because it can be easily discretized into a Langevin gradient descent-ascent method by treating the PDEs as Fokker-Planck equations of SDEs . When β−1 is sufficiently large , ( 4 ) can be proven to converge linearly to the unique MNE of ( 2 ) ( Eberle et al. , 2019 ) . However , when β−1 is small , whether ( 4 ) converges remains open . This hinders the application of ( 4 ) because in practice the entropy terms are usually used as regularization and are kept small . ( We realize that it is proven in Domingo-Enrich & Bruna ( 2022 ) when our work is under review . ) In ( 4 ) , the dynamics of p and q have the same speed . In this work , instead , we study a quasistatic Wasserstein gradient descent dynamics , which can be understood as a limiting dynamics when the speed of q becomes faster and faster compared with that of p. In this case , at any time t , we assume qt reaches at the equilibrium of the maximizing problem instantaneously by fixing p = pt in ( 2 ) . That is to say , at any time t , qt is determined by qt = q [ pt ] : = arg max q∈P ( Ω ) E ( pt , q ) − β−1S ( q ) . ( 5 ) On the other hand , pt follows the Wasserstein gradient descent flow with qt = q [ pt ] at the equilibrium : ∂tpt = ∇ · ( pt∇ ( δ ( E ( pt , q [ pt ] ) − β−1S ( q [ pt ] ) ) δpt + β−1 log pt ) ) . ( 6 ) The following theorem shows qt = q [ pt ] can be explicitly written as a Gibbs distribution depending on pt , and thus the free energy in ( 6 ) can be simplified to depend on a partition function related with pt . Theorem 2 . Assume K is continuous on the compact set Ω and β > 0 . Then , for fixed pt the maximization problem ( 5 ) has a unique solution q [ pt ] ( y ) : = 1 Zq ( pt ) exp ( βV ( y , pt ) ) , ( 7 ) where Zq ( p ) is a normalization factor , Zq ( p ) : = ∫ exp ( βV ( y , p ) ) dy . Moreover , the dynamics ( 6 ) for pt can be written as ∂tpt = ∇ · ( pt∇ ( δβ−1 logZq ( pt ) δpt + β−1 log pt ) ) . ( 8 ) Let Fp , β ( p ) : = β−1 logZq ( p ) +β−1S ( p ) . By Theorem 2 , the dynamics ( 8 ) of pt is the Wasserstein gradient descent flow for minimizing Fp , β ( p ) . By the Proposition 3 below , Fp , β is strongly convex with respect to p. Therefore , it is possible to prove global convergence for the dynamics ( 8 ) , and thus the convergence for the quasistatic Wasserstein gradient flow for the minimax problem ( 2 ) . Proposition 3 . For any probability distributions p1 , p2 in P ( Ω ) , and any λ ∈ [ 0 , 1 ] , we have Fp , β ( λp1 + ( 1− λ ) p2 ) < λFp , β ( p1 ) + ( 1− λ ) Fp , β ( p2 ) . In practice the partition function logZq ( pt ) in ( 8 ) seems hard to approximate , especially when Ω is in high dimensional spaces . However , we show in the following proposition that the variation of the partition function with respect to pt can be written as a simple form involving qt . This property will be used to derive a particle method in Section 5 Proposition 4 . For any p ∈ P ( Ω ) , we have δβ−1 logZq ( p ) δp = U ( · , q [ p ] ) , ( 9 ) where q [ p ] is defined in ( 7 ) . Therefore , the dynamics ( 8 ) is equivalent with ∂tpt = ∇ · ( pt∇ ( U ( x , q [ pt ] ) + β −1 log pt ) ) . ( 10 )

Summary: The paper under review studies the QuasiStatic Wasserstein Gradient Flow and its application to solve two-player zero-sum games. The corresponding Langevin SDE is derived for the Fokker-Planck equation, and the discretized algorithm (GD) is proposed in this new setting.  Example of the min-max problem on 1-d torus and spheres are provided.