Fast Policy Extragradient Methods for Competitive Games with Entropy Regularization

1 Introduction . Finding the equilibrium of competitive games , which can be viewed as constrained saddle-point optimization problems with probability simplex constraints , lies at the heart of modern machine learning and decision making paradigms such as Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) , competitive reinforcement learning ( RL ) ( Littman , 1994 ) , game theory ( Shapley , 1953 ) , adversarial training ( Mertikopoulos et al. , 2018b ) , to name a few . In this paper , we study one of the most basic forms of competitive games , namely two-player zero-sum games , in both the matrix setting and the Markov setting . Our goal is to find the equilibrium policies of both players in an independent and decentralized manner ( Daskalakis et al. , 2020 ; Wei et al. , 2021a ) with guaranteed last-iterate convergence . Namely , each player will execute symmetric and independent updates iteratively using its own payoff without observing the opponent ’ s actions directly , and the final policies of the iterative process should be a close approximation to the equilibrium up to any prescribed precision . This kind of algorithms is more advantageous and versatile especially in federated environments , as it requires neither prior coordination between the players like twotimescale algorithms , nor a central controller to collect and disseminate the policies of all the players , which are often unavailable due to privacy constraints . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1.1 Last-iterate convergence in competitive games . In recent years , there have been significant progresses in understanding the last-iterate convergence of simple iterative algorithms for unconstrained saddle-point optimization , where one is interested in bounding the sub-optimality of the last iterate of the algorithm , rather than say , the ergodic iterate — which is the average of all the iterations — that are commonly studied in the earlier literature . This shift of focus is motivated , for example , by the infeasibility of averaging large machine learning models in training GANs ( Goodfellow et al. , 2014 ) . While vanilla Gradient Descent / Ascent ( GDA ) may diverge or cycle even for bilinear matrix games ( Daskalakis et al. , 2018 ) , quite remarkably , small modifications lead to guaranteed last-iterate convergence to the equilibrium in a non-asymptotic fashion . A flurry of algorithms is proposed , including Optimistic Gradient Descent Ascent ( OGDA ) ( Rakhlin and Sridharan , 2013 ; Daskalakis and Panageas , 2018b ; Wei et al. , 2021b ) , predictive updates ( Yadav et al. , 2017 ) , implicit updates ( Liang and Stokes , 2019 ) , and more . Several unified analyses of these algorithms have been carried out ( see , e.g . Mokhtari et al . ( 2020a ) ; Liang and Stokes ( 2019 ) and references therein ) , where these methods in principle all make clever extrapolation of the local curvature in a predictive manner to accelerate convergence . With slight abuse of terminology , in this paper , we refer to this ensemble of algorithms as extragradient methods ( Korpelevich , 1976 ; Tseng , 1995 ; Mertikopoulos et al. , 2018a ; Harker and Pang , 1990 ) . However , saddle-point optimization in the constrained setting , which includes competitive games as a special case , remains largely under-explored even for bilinear matrix games . While it is possible to reformulate constrained bilinear games to unconstrained ones using softmax parameterization of the probability simplex , this approach falls short of preserving the bilinear structure and convexconcave properties in the original problem , which are crucial to the convergence of gradient methods . Therefore , there is a strong necessity of understanding and developing improved extragradient methods in the constrained setting . Daskalakis and Panageas ( 2018a ) proposed the optimistic variant of the multiplicative weight updates ( MWU ) method ( Arora et al. , 2012 ) – which is extremely natural and popular for optimizing over probability simplexes – called Optimistic Multiplicative Weight Updates ( OMWU ) , and established the asymptotic last-iterate convergence of OMWU for matrix games . Very recently , Wei et al . ( 2021b ) established non-asymptotic last-iterate convergences of OMWU . However , these last-iterate convergence results require the Nash equilibrium to be unique , and can not be applied to problems with multiple Nash equilibria . 1.2 Our contributions . Motivated by the algorithmic role of entropy regularization in single-agent RL ( Neu et al. , 2017 ; Geist et al. , 2019 ; Cen et al. , 2020 ) as well as its wide use in game theory to account for imperfect and noisy information ( McKelvey and Palfrey , 1995 ; Savas et al. , 2019 ) , we initiate the design and analysis of extragradient algorithms using multiplicative updates for finding the quantal response equilibrium ( QRE ) , which are solutions to competitive games with entropy regularization ( McKelvey and Palfrey , 1995 ) . While finding QRE is of interest in its own right , by controlling the knob of entropy regularization , the QRE provides a close approximation to the Nash equilibrium ( NE ) , and in turn acts as a smoothing scheme for finding the NE . Our contributions are summarized below . • Near dimension-free last-iterate convergence to QRE of entropy-regularized matrix games . We propose two policy extragradient algorithms to solve entropy-regularized matrix games , namely the Predictive Update ( PU ) and OMWU methods , where both players execute symmetric and multiplicative updates without knowing the entire payoff matrix nor the opponent ’ s actions . Encouragingly , we show that the last iterate of the proposed algorithms converges to the unique QRE at a linear rate that is almost independent of the size of the action spaces . Roughly speaking , to find an ϵ-optimal QRE in terms of Kullback-Leibler ( KL ) divergence , it takes no more than Õ ( 1 ητ log ( 1 ϵ ) ) iterations , where Õ ( · ) hides logarithmic dependencies . Here , τ is the regularization parameter , and η is the learning rate of both players . Maximizing the learning rate , the iteration complexity is bounded by Õ ( ( 1 + ∥A∥∞/τ ) log ( 1/ϵ ) ) , where ∥A∥∞ = maxi , j |Ai , j | is the ℓ∞ norm of the payoff matrix A . • Last-iterate convergence to ϵ-NE of unregularized matrix games without uniqueness assumption . The QRE provides an accurate approximation to the NE by setting the entropy regularization τ sufficiently small , therefore our result directly translates to finding a NE with last-iterate convergence guarantee . Roughly speaking , to find an ϵ-NE ( Zhang et al. , 2020 , Definition 2.1 ) , it takes no more than Õ ( 1 + ∥A∥∞ϵ ) iterations with optimized learning rates , which is again independent of the size of the action spaces up to logarithmic factors . Unlike prior literature ( Daskalakis and Panageas , 2018a ; Wei et al. , 2021b ) , our last-iterate convergence guarantee does not require the NE to be unique . • Extensions to two-player zero-sum Markov games . By connecting value iteration with matrix games , we propose a policy extragradient method for solving infinite-horizon discounted entropyregularized zero-sum Markov games , which finds an ϵ-optimal minimax soft Q-function—in terms of ℓ∞ error—in at most Õ ( 1 τ ( 1−γ ) 2 log 2 ( 1 ϵ ) ) iterations , where γ ∈ ( 0 , 1 ) is the discount factor . To the best of our knowledge , our paper is the first that develops policy extragradient algorithms for solving entropy-regularized competitive games with multiplicative updates and dimension-free linear last-iterate convergence , and demonstrates entropy regularization as a smoothing technique to find ϵ-NE without the uniqueness assumption . Table 1 provides detailed comparisons of the proposed methods with prior arts for solving matrix games . Our results highlight the positive role of entropy regularization for accelerating convergence and safeguarding against imperfect information in competitive games . We defer the complete proof of our results to Cen et al . ( 2021 ) . 1.3 Related works . Our work lies at the intersection of saddle-point optimization , game theory , and reinforcement learning . In what follows , we discuss a few topics that are closely related to ours . Unregularized matrix game . Freund and Schapire ( 1999 ) showed that many standard methods such as GDA and MWU have a converging average duality gap at the rate of O ( 1/ √ T ) , which is improved to O ( 1/T ) by considering optimistic variants of these methods , such as OGDA and OMWU ( Rakhlin and Sridharan , 2013 ; Daskalakis et al. , 2011 ; Syrgkanis et al. , 2015 ) . However , the last-iterate convergence of these methods are less understood until recently ( Daskalakis and Panageas , 2018a ; Wei et al. , 2021b ) . In particular , under the assumption that the NE is unique for the unregularized matrix game , Daskalakis and Panageas ( 2018a ) showed the asymptotic convergence of the last iterate of OMWU to the unique equilibrium , and Wei et al . ( 2021b ) showed the last iterate of OMWU achieves a linear rate of convergence after an initial phase of sublinear convergence , however the rates therein can be highly pessimistic in terms of the problem dimension , while our rate for entropy-regularized OMWU is dimension-free up to logarithmic factors . Saddle-point optimization . Considerable progress has been made towards understanding OGDA and extragradient ( EG ) methods in the unconstrained convex-concave saddle-point optimization with general objective functions ( Mokhtari et al. , 2020a , b ; Nemirovski , 2004 ; Liang and Stokes , 2019 ) . However , the last-iterate convergence of constrained convex-concave saddle-point optimization still lacks theoretical understanding in general and most works fall short of characterizing a finite-time convergence result . In particular , Mertikopoulos et al . ( 2018a ) demonstrated the asymptotic lastiterate convergence of EG , and Hsieh et al . ( 2019 ) investigated similar questions for single-call EG algorithms . Lei et al . ( 2021 ) showed that OMWU converges to the equilibrium locally without an explicit rate . Wei et al . ( 2021b ) showed that the last-iterate of OGDA converges linearly for strongly-convex strongly-concave constrained saddle-point optimization with an explicit rate . Entropy regularization in RL and games . In single-agent RL , the role of entropy regularization as an algorithmic mechanism to encourage exploration and accelerate convergence has been investigated extensively ( Neu et al. , 2017 ; Geist et al. , 2019 ; Mei et al. , 2020 ; Cen et al. , 2020 ; Lan , 2021 ; Zhan et al. , 2021 ) . Turning to the game setting , entropy regularization is used to account for imperfect information in the seminal work of McKelvey and Palfrey ( 1995 ) that introduced the QRE , and a few representative works on entropy and more general regularizations in games include Savas et al . ( 2019 ) ; Hofbauer and Sandholm ( 2002 ) ; Mertikopoulos and Sandholm ( 2016 ) . Zero-sum Markov games . There have been a significant recent interest in developing provably efficient self-play algorithms for Markov games , including model-based algorithms ( Perolat et al. , 2015 ; Zhang et al. , 2020 ) , value-based algorithms ( Bai and Jin , 2020 ; Xie et al. , 2020 ) , and policybased algorithms ( Daskalakis et al. , 2020 ; Wei et al. , 2021a ; Zhao et al. , 2021 ) . The iteration complexities in prior works ( Perolat et al. , 2015 ; Daskalakis et al. , 2020 ; Wei et al. , 2021a ; Zhao et al. , 2021 ) depend on various notions of concentrability coefficient and therefore can scale quite pessimistically with the problem dimension . Our approach can be regarded as a policy-based algorithm to approximate value iteration , which can be implemented in a decentralized manner with symmetric and multiplicative updates from both players , and the iteration complexity is almost independent of the size of the state-action space . Notation . We denote by ∆ ( A ) the probability simplex over the set A . We overload the functions such as log ( · ) and exp ( · ) to take vector inputs with the understanding that the function is applied in an entrywise manner . For instance , given any vector z = [ zi ] 1≤i≤n ∈ Rn , the notation exp ( z ) denotes exp ( z ) : = [ exp ( zi ) ] 1≤i≤n ; other functions are defined analogously . Given two probability distributions µ and µ′ over A , the KL divergence from µ′ to µ is defined by KL ( µ ∥µ′ ) : = ∑ a∈A µ ( a ) log µ ( a ) µ′ ( a ) . Given a matrix A , ∥A∥∞ is used to denote entrywise maximum norm , namely , ∥A∥∞ = maxi , j |Ai , j | . The all-one vector is denoted as 1 .

Summary: The authors propose to use mirror descent to solve classic entropy-regularized competitive games. Linear convergence rates are obtained for the proposed algorithm. In addition, the authors propose to use such an algorithm to solve the subproblem in the entropy-regularized Markov game and obtain a linear convergence rate to the regularized equilibrium.