Provably convergent quasistatic dynamics for mean-field two-player zero-sum games

1 INTRODUCTION . Finding Nash equilibrium has seen many important applications in machine learning , such as generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014a ) and reinforcement learning ( Busoniu et al. , 2008 ) . In these problems , pure Nash equilibria are usually search for a function f ( x , y ) . Yet , the problems arising from machine learning are usually nonconvex in x and nonconcave in y , in which case pure Nash equilibrium may not exist . And even if it exists , there is no guarantee for any optimization algorithm to find it efficiently . This difficulty is reflected in practice , that compared with simple minimization , machine learning applications involving Nash equilibria usually have more complicated behaviors and more subtle dependence on hyper-parameters . For example , stable and efficient training of GANs requires a number of carefully designed tricks ( Gao et al. , 2018 ) . On the other hand , the mixed Nash equilibrium ( MNE ) is known to exist in much more general settings , e.g . when the strategy spaces are compact and the payoff function is continuous ( Glicksberg , 1952 ) . In the mixed Nash equilibrium problem , instead of taking “ pure strategies ” x and y , two ” mixed strategies ” for x and y , in the form of probability distributions , are considered , resulting in the following functional , ∫ f ( x , y ) p ( x ) q ( y ) dxdy , where p and q are density functions of probability distributions of x and y , respectively . Efforts are invested to develop theoretically endorsed algorithms that can efficiently find MNE for high dimensional problems , with applications on the training of mixture of GANs . In Hsieh et al . ( 2019 ) , a mirror-descent algorithm is proposed and its convergence is proven . In Domingo-Enrich et al . ( 2020 ) , theoretical analysis and empirical experiments are conducted for a gradient descent-ascent flow under a Wasserstein-Fisher-Rao metric and its particle discretization . In this paper , we also consider the mixed Nash equilibrium problem , and propose a simple QuasiStatic Wasserstein Gradient Flow ( QSWGF ) for solving the problem . In our dynamics , we treat q as a component with much faster speed than p , hence is always at equilibrium as p moves . With entropy regularization for both p and q ( without requirement on the strength of the regularizations ) , we prove that the QSWGF converges to the unique mixed Nash equilibrium from any initialization ( under mild conditions ) . Furthermore , we show there is a simple way to discretize the QSWGF , regardless of the complexity of the Wasserstein gradient flow of p induced by the fact that q is always at equilibrium . Concretely , a partition function related with p appears in the QSWGF dynamics , and we find an efficient way to approximate the partition function . By discretizing the QSWGF , we derive a particle dynamics with an inner-outer structure , named the QuasiStatic Langevin Gradient Descent algorithm ( QSLGD ) . In QSLGD , after each iteration of the outer problem ( for the x particles ) , the inner loop conducts sufficient iterations to bring the y particles to equilibrium . Numerical experiments show the effectiveness of QSLGD on synthetic examples and training mixture of GANs . Our method outperforms the vanilla Langevin gradient descent-ascent method when the entropy regularization is weak . As a summary , our two major contributions are : 1 . We propose the quasistatic Wasserstein gradient flow dynamics for mixed Nash equilibrium problems , and show its convergence to the unique Nash equilibrium under weak assumptions . Our result neither requires the entropy regularization to be sufficiently strong , nor assumes the dynamics to converge a priori . 2 . We derive a simple while practical quasistatic Langevin gradient descent algorithm by discretizing the quasistatic Wasserstein gradient flow , by finding an efficient way to approximate the partition function appearing in the dynamics of p. The proposed algorithm is applied on several problems including training mixtures of GANs . 2 RELATED WORK . The mixed Nash equilibrium problem has a long history , with the proof of its existence dates back to Morgenstern & Von Neumann ( 1953 ) . It draws new attention in recent years , especially in the machine learning community , due to the development of GANs ( Goodfellow et al. , 2014a ) and adversarial training ( Goodfellow et al. , 2014b ) . Training mixture of GANs is already discussed in paper ( Goodfellow et al. , 2014a ) . Some numerical experiments were conducted in ( Arora et al. , 2017 ) . In Grnarova et al . ( 2017 ) , the authors proposed an online learning approach for training mixture of GANs , and proved its effectiveness for semi-shallow GANs ( GANs whose discriminator is a shallow neural network ) . Yet , rigorous theoretical treatment to an algorithm started from ( Hsieh et al. , 2019 ) , in which a mirror descent method was studied and proven to converge . The implementation of the mirror descent method involves big computational cost that asks for heuristics to alleviate . Later , ( Domingo-Enrich et al. , 2020 ) studied more efficient algorithms under a mixture of Wasserstein and Fisher-Rao metrics . Theoretically , the time average of the dynamics ’ trajectories is shown to converge to the mixed Nash equilibrium . As a comparison , in this work we show the global convergence of the quasistatic Wasserstein gradient flow without the need of taking time average . Meanwhile , the Wasserstein nature of our dynamics makes it easy to implement as well . The Wasserstein gradient flow in the density space has been explored in previous works . For example , ( Wang & Li , 2019 ) studied the Nesterov ’ s accelerated gradient flows for probability distributions under the Wasserstein metric , and ( Arbel et al. , 2019 ) studied practical implementations of the natural gradient method for the Wasserstein metric . Both works focus on minimization problems instead of min-max problems considered in this work . A more related work is Lin et al . ( 2021b ) , where a natural gradient based algorithm is proposed for training GANs . Yet , the method still optimizes one generator and one discriminator , searching for pure Nash equilibrium . Another work that derives algorithms for GANs from a Wasserstein perspective is ( Lin et al. , 2021a ) . Another volume of works that studies the Wasserstein gradient flow in the machine learning context is the mean-field analysis of neural networks . This line of works started from two-layer neural networks ( Mei et al. , 2018 ; Rotskoff & Vanden-Eijnden , 2018 ; Chizat & Bach , 2018 ; Sirignano & Spiliopoulos , 2020 ) , to deep fully-connected networks ( Araújo et al. , 2019 ; Sirignano & Spiliopoulos , 2021 ; Nguyen , 2019 ; Wojtowytsch et al. , 2020 ) , and residual networks ( Lu et al. , 2020 ; E et al. , 2020 ) . The mean-field formulations treat parameters as probability distributions , and the training dynamics are usually the gradient flow under Wasserstein metric . Attempts to prove convergence of the dynamics to global minima are made ( Mei et al. , 2018 ; Chizat & Bach , 2018 ; Rotskoff et al. , 2019 ) , though in the case without entropy regularization a convergence assumption should usually be made a priori . 3 THE QUASISTATIC DYNAMICS . We consider the entropy regularized mixed Nash equilibrium problem , which in our case is equivalent with solving the following minimax problem : min p∈P ( Ω ) max q∈P ( Ω ) ∫ Ω×Ω K ( x , y ) p ( x ) q ( y ) dxdy + β−1 ∫ Ω p log pdx− β−1 ∫ Ω q log qdy . ( 1 ) In ( 1 ) , Ω is a compact Riemannian manifold without boundary , and P ( Ω ) is the set of probability distributions on Ω . Since Ω is compact , any probability distribution in P ( Ω ) naturally has finite moments . Let E ( p , q ) = ∫ Ω×ΩK ( x , y ) p ( dx ) q ( dy ) , and S ( p ) = ∫ Ω p log pdx and S ( q ) = ∫ Ω q log qdy be the ( negative ) entropy of p and q , respectively . Then , the minimax problem ( 1 ) can be written in short as min p∈P ( Ω ) max q∈P ( Ω ) E ( p , q ) + β−1S ( p ) − β−1S ( q ) . ( 2 ) Remark 1 . Strictly speaking , in ( 1 ) we should distinguish probability distributions and their density function ( if exist ) , and the entropy should also be defined using the Radon-Nikodym derivative with canonical measure . In this paper , since p and q indeed have density functions because of the entropy regularization , we shall abuse the notation by using p and q to represent both probability distributions and their density functions . The entropy regularizations in ( 1 ) and ( 2 ) make the problem strongly convex in p and strongly concave in q . Hence , there exists a unique Nash equilibrium for the problem . Such results are shown for example by the following theorem from ( Domingo-Enrich et al. , 2020 ) . Theorem 1 . ( Theorem 4 of ( Domingo-Enrich et al. , 2020 ) ) Assume Ω is a compact Polish metric space equipped with canonical Borel measure , and that K is a continuous function on Ω × Ω . Then , problem ( 2 ) has a unique Nash equilibrium given by the solution of the following fixed-point problem : p ( x ) = 1 Zp exp ( −βU ( x , q ) ) , q ( x ) = 1 Zq exp ( βV ( y , p ) ) , ( 3 ) where Zp and Zq are normalization constants to make sure p and q are probability distributions , and U and V are defined as U ( x , q ) = δE ( p , q ) δp ( x ) = ∫ Ω K ( x , y ) q ( y ) dy , V ( y , p ) = δE ( p , q ) δq ( y ) = ∫ Ω K ( x , y ) p ( x ) dx . Considering the efficiency in high-dimensional cases , a natural dynamics of interest to find the Nash equilibrium for ( 2 ) is the gradient descent-ascent flow under the Wasserstein metric , ∂tpt = ∇ · ( pt∇ ( U ( x , qt ) + β−1 log pt ) ) , ∂tqt = ∇ · ( qt∇ ( −V ( y , pt ) + β−1 log qt ) ) , ( 4 ) because it can be easily discretized into a Langevin gradient descent-ascent method by treating the PDEs as Fokker-Planck equations of SDEs . When β−1 is sufficiently large , ( 4 ) can be proven to converge linearly to the unique MNE of ( 2 ) ( Eberle et al. , 2019 ) . However , when β−1 is small , whether ( 4 ) converges remains open . This hinders the application of ( 4 ) because in practice the entropy terms are usually used as regularization and are kept small . ( We realize that it is proven in Domingo-Enrich & Bruna ( 2022 ) when our work is under review . ) In ( 4 ) , the dynamics of p and q have the same speed . In this work , instead , we study a quasistatic Wasserstein gradient descent dynamics , which can be understood as a limiting dynamics when the speed of q becomes faster and faster compared with that of p. In this case , at any time t , we assume qt reaches at the equilibrium of the maximizing problem instantaneously by fixing p = pt in ( 2 ) . That is to say , at any time t , qt is determined by qt = q [ pt ] : = arg max q∈P ( Ω ) E ( pt , q ) − β−1S ( q ) . ( 5 ) On the other hand , pt follows the Wasserstein gradient descent flow with qt = q [ pt ] at the equilibrium : ∂tpt = ∇ · ( pt∇ ( δ ( E ( pt , q [ pt ] ) − β−1S ( q [ pt ] ) ) δpt + β−1 log pt ) ) . ( 6 ) The following theorem shows qt = q [ pt ] can be explicitly written as a Gibbs distribution depending on pt , and thus the free energy in ( 6 ) can be simplified to depend on a partition function related with pt . Theorem 2 . Assume K is continuous on the compact set Ω and β > 0 . Then , for fixed pt the maximization problem ( 5 ) has a unique solution q [ pt ] ( y ) : = 1 Zq ( pt ) exp ( βV ( y , pt ) ) , ( 7 ) where Zq ( p ) is a normalization factor , Zq ( p ) : = ∫ exp ( βV ( y , p ) ) dy . Moreover , the dynamics ( 6 ) for pt can be written as ∂tpt = ∇ · ( pt∇ ( δβ−1 logZq ( pt ) δpt + β−1 log pt ) ) . ( 8 ) Let Fp , β ( p ) : = β−1 logZq ( p ) +β−1S ( p ) . By Theorem 2 , the dynamics ( 8 ) of pt is the Wasserstein gradient descent flow for minimizing Fp , β ( p ) . By the Proposition 3 below , Fp , β is strongly convex with respect to p. Therefore , it is possible to prove global convergence for the dynamics ( 8 ) , and thus the convergence for the quasistatic Wasserstein gradient flow for the minimax problem ( 2 ) . Proposition 3 . For any probability distributions p1 , p2 in P ( Ω ) , and any λ ∈ [ 0 , 1 ] , we have Fp , β ( λp1 + ( 1− λ ) p2 ) < λFp , β ( p1 ) + ( 1− λ ) Fp , β ( p2 ) . In practice the partition function logZq ( pt ) in ( 8 ) seems hard to approximate , especially when Ω is in high dimensional spaces . However , we show in the following proposition that the variation of the partition function with respect to pt can be written as a simple form involving qt . This property will be used to derive a particle method in Section 5 Proposition 4 . For any p ∈ P ( Ω ) , we have δβ−1 logZq ( p ) δp = U ( · , q [ p ] ) , ( 9 ) where q [ p ] is defined in ( 7 ) . Therefore , the dynamics ( 8 ) is equivalent with ∂tpt = ∇ · ( pt∇ ( U ( x , q [ pt ] ) + β −1 log pt ) ) . ( 10 )

Summary: In this paper, the authors study the minimax problem for finding the mixed Nash equilibrium for mean-field two-player zero-sum games. This is an inf-sup problem in the Wasserstein space. In this paper, the authors consider the case where the sup problem is solved and apply the gradient descent flow for the result inf sup functional. Some theoretical analyses are conducted on these dynamics. They show its convergence to the mixed Nash equilibrium under some conditions. Some numerical schemes for the quasistatic Langevin gradient descent method with inner-outer iterations are presented. They show that the method works on training the mixture of GANs.