Fast Policy Extragradient Methods for Competitive Games with Entropy Regularization

1 Introduction . Finding the equilibrium of competitive games , which can be viewed as constrained saddle-point optimization problems with probability simplex constraints , lies at the heart of modern machine learning and decision making paradigms such as Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) , competitive reinforcement learning ( RL ) ( Littman , 1994 ) , game theory ( Shapley , 1953 ) , adversarial training ( Mertikopoulos et al. , 2018b ) , to name a few . In this paper , we study one of the most basic forms of competitive games , namely two-player zero-sum games , in both the matrix setting and the Markov setting . Our goal is to find the equilibrium policies of both players in an independent and decentralized manner ( Daskalakis et al. , 2020 ; Wei et al. , 2021a ) with guaranteed last-iterate convergence . Namely , each player will execute symmetric and independent updates iteratively using its own payoff without observing the opponent ’ s actions directly , and the final policies of the iterative process should be a close approximation to the equilibrium up to any prescribed precision . This kind of algorithms is more advantageous and versatile especially in federated environments , as it requires neither prior coordination between the players like twotimescale algorithms , nor a central controller to collect and disseminate the policies of all the players , which are often unavailable due to privacy constraints . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1.1 Last-iterate convergence in competitive games . In recent years , there have been significant progresses in understanding the last-iterate convergence of simple iterative algorithms for unconstrained saddle-point optimization , where one is interested in bounding the sub-optimality of the last iterate of the algorithm , rather than say , the ergodic iterate — which is the average of all the iterations — that are commonly studied in the earlier literature . This shift of focus is motivated , for example , by the infeasibility of averaging large machine learning models in training GANs ( Goodfellow et al. , 2014 ) . While vanilla Gradient Descent / Ascent ( GDA ) may diverge or cycle even for bilinear matrix games ( Daskalakis et al. , 2018 ) , quite remarkably , small modifications lead to guaranteed last-iterate convergence to the equilibrium in a non-asymptotic fashion . A flurry of algorithms is proposed , including Optimistic Gradient Descent Ascent ( OGDA ) ( Rakhlin and Sridharan , 2013 ; Daskalakis and Panageas , 2018b ; Wei et al. , 2021b ) , predictive updates ( Yadav et al. , 2017 ) , implicit updates ( Liang and Stokes , 2019 ) , and more . Several unified analyses of these algorithms have been carried out ( see , e.g . Mokhtari et al . ( 2020a ) ; Liang and Stokes ( 2019 ) and references therein ) , where these methods in principle all make clever extrapolation of the local curvature in a predictive manner to accelerate convergence . With slight abuse of terminology , in this paper , we refer to this ensemble of algorithms as extragradient methods ( Korpelevich , 1976 ; Tseng , 1995 ; Mertikopoulos et al. , 2018a ; Harker and Pang , 1990 ) . However , saddle-point optimization in the constrained setting , which includes competitive games as a special case , remains largely under-explored even for bilinear matrix games . While it is possible to reformulate constrained bilinear games to unconstrained ones using softmax parameterization of the probability simplex , this approach falls short of preserving the bilinear structure and convexconcave properties in the original problem , which are crucial to the convergence of gradient methods . Therefore , there is a strong necessity of understanding and developing improved extragradient methods in the constrained setting . Daskalakis and Panageas ( 2018a ) proposed the optimistic variant of the multiplicative weight updates ( MWU ) method ( Arora et al. , 2012 ) – which is extremely natural and popular for optimizing over probability simplexes – called Optimistic Multiplicative Weight Updates ( OMWU ) , and established the asymptotic last-iterate convergence of OMWU for matrix games . Very recently , Wei et al . ( 2021b ) established non-asymptotic last-iterate convergences of OMWU . However , these last-iterate convergence results require the Nash equilibrium to be unique , and can not be applied to problems with multiple Nash equilibria . 1.2 Our contributions . Motivated by the algorithmic role of entropy regularization in single-agent RL ( Neu et al. , 2017 ; Geist et al. , 2019 ; Cen et al. , 2020 ) as well as its wide use in game theory to account for imperfect and noisy information ( McKelvey and Palfrey , 1995 ; Savas et al. , 2019 ) , we initiate the design and analysis of extragradient algorithms using multiplicative updates for finding the quantal response equilibrium ( QRE ) , which are solutions to competitive games with entropy regularization ( McKelvey and Palfrey , 1995 ) . While finding QRE is of interest in its own right , by controlling the knob of entropy regularization , the QRE provides a close approximation to the Nash equilibrium ( NE ) , and in turn acts as a smoothing scheme for finding the NE . Our contributions are summarized below . • Near dimension-free last-iterate convergence to QRE of entropy-regularized matrix games . We propose two policy extragradient algorithms to solve entropy-regularized matrix games , namely the Predictive Update ( PU ) and OMWU methods , where both players execute symmetric and multiplicative updates without knowing the entire payoff matrix nor the opponent ’ s actions . Encouragingly , we show that the last iterate of the proposed algorithms converges to the unique QRE at a linear rate that is almost independent of the size of the action spaces . Roughly speaking , to find an ϵ-optimal QRE in terms of Kullback-Leibler ( KL ) divergence , it takes no more than Õ ( 1 ητ log ( 1 ϵ ) ) iterations , where Õ ( · ) hides logarithmic dependencies . Here , τ is the regularization parameter , and η is the learning rate of both players . Maximizing the learning rate , the iteration complexity is bounded by Õ ( ( 1 + ∥A∥∞/τ ) log ( 1/ϵ ) ) , where ∥A∥∞ = maxi , j |Ai , j | is the ℓ∞ norm of the payoff matrix A . • Last-iterate convergence to ϵ-NE of unregularized matrix games without uniqueness assumption . The QRE provides an accurate approximation to the NE by setting the entropy regularization τ sufficiently small , therefore our result directly translates to finding a NE with last-iterate convergence guarantee . Roughly speaking , to find an ϵ-NE ( Zhang et al. , 2020 , Definition 2.1 ) , it takes no more than Õ ( 1 + ∥A∥∞ϵ ) iterations with optimized learning rates , which is again independent of the size of the action spaces up to logarithmic factors . Unlike prior literature ( Daskalakis and Panageas , 2018a ; Wei et al. , 2021b ) , our last-iterate convergence guarantee does not require the NE to be unique . • Extensions to two-player zero-sum Markov games . By connecting value iteration with matrix games , we propose a policy extragradient method for solving infinite-horizon discounted entropyregularized zero-sum Markov games , which finds an ϵ-optimal minimax soft Q-function—in terms of ℓ∞ error—in at most Õ ( 1 τ ( 1−γ ) 2 log 2 ( 1 ϵ ) ) iterations , where γ ∈ ( 0 , 1 ) is the discount factor . To the best of our knowledge , our paper is the first that develops policy extragradient algorithms for solving entropy-regularized competitive games with multiplicative updates and dimension-free linear last-iterate convergence , and demonstrates entropy regularization as a smoothing technique to find ϵ-NE without the uniqueness assumption . Table 1 provides detailed comparisons of the proposed methods with prior arts for solving matrix games . Our results highlight the positive role of entropy regularization for accelerating convergence and safeguarding against imperfect information in competitive games . We defer the complete proof of our results to Cen et al . ( 2021 ) . 1.3 Related works . Our work lies at the intersection of saddle-point optimization , game theory , and reinforcement learning . In what follows , we discuss a few topics that are closely related to ours . Unregularized matrix game . Freund and Schapire ( 1999 ) showed that many standard methods such as GDA and MWU have a converging average duality gap at the rate of O ( 1/ √ T ) , which is improved to O ( 1/T ) by considering optimistic variants of these methods , such as OGDA and OMWU ( Rakhlin and Sridharan , 2013 ; Daskalakis et al. , 2011 ; Syrgkanis et al. , 2015 ) . However , the last-iterate convergence of these methods are less understood until recently ( Daskalakis and Panageas , 2018a ; Wei et al. , 2021b ) . In particular , under the assumption that the NE is unique for the unregularized matrix game , Daskalakis and Panageas ( 2018a ) showed the asymptotic convergence of the last iterate of OMWU to the unique equilibrium , and Wei et al . ( 2021b ) showed the last iterate of OMWU achieves a linear rate of convergence after an initial phase of sublinear convergence , however the rates therein can be highly pessimistic in terms of the problem dimension , while our rate for entropy-regularized OMWU is dimension-free up to logarithmic factors . Saddle-point optimization . Considerable progress has been made towards understanding OGDA and extragradient ( EG ) methods in the unconstrained convex-concave saddle-point optimization with general objective functions ( Mokhtari et al. , 2020a , b ; Nemirovski , 2004 ; Liang and Stokes , 2019 ) . However , the last-iterate convergence of constrained convex-concave saddle-point optimization still lacks theoretical understanding in general and most works fall short of characterizing a finite-time convergence result . In particular , Mertikopoulos et al . ( 2018a ) demonstrated the asymptotic lastiterate convergence of EG , and Hsieh et al . ( 2019 ) investigated similar questions for single-call EG algorithms . Lei et al . ( 2021 ) showed that OMWU converges to the equilibrium locally without an explicit rate . Wei et al . ( 2021b ) showed that the last-iterate of OGDA converges linearly for strongly-convex strongly-concave constrained saddle-point optimization with an explicit rate . Entropy regularization in RL and games . In single-agent RL , the role of entropy regularization as an algorithmic mechanism to encourage exploration and accelerate convergence has been investigated extensively ( Neu et al. , 2017 ; Geist et al. , 2019 ; Mei et al. , 2020 ; Cen et al. , 2020 ; Lan , 2021 ; Zhan et al. , 2021 ) . Turning to the game setting , entropy regularization is used to account for imperfect information in the seminal work of McKelvey and Palfrey ( 1995 ) that introduced the QRE , and a few representative works on entropy and more general regularizations in games include Savas et al . ( 2019 ) ; Hofbauer and Sandholm ( 2002 ) ; Mertikopoulos and Sandholm ( 2016 ) . Zero-sum Markov games . There have been a significant recent interest in developing provably efficient self-play algorithms for Markov games , including model-based algorithms ( Perolat et al. , 2015 ; Zhang et al. , 2020 ) , value-based algorithms ( Bai and Jin , 2020 ; Xie et al. , 2020 ) , and policybased algorithms ( Daskalakis et al. , 2020 ; Wei et al. , 2021a ; Zhao et al. , 2021 ) . The iteration complexities in prior works ( Perolat et al. , 2015 ; Daskalakis et al. , 2020 ; Wei et al. , 2021a ; Zhao et al. , 2021 ) depend on various notions of concentrability coefficient and therefore can scale quite pessimistically with the problem dimension . Our approach can be regarded as a policy-based algorithm to approximate value iteration , which can be implemented in a decentralized manner with symmetric and multiplicative updates from both players , and the iteration complexity is almost independent of the size of the state-action space . Notation . We denote by ∆ ( A ) the probability simplex over the set A . We overload the functions such as log ( · ) and exp ( · ) to take vector inputs with the understanding that the function is applied in an entrywise manner . For instance , given any vector z = [ zi ] 1≤i≤n ∈ Rn , the notation exp ( z ) denotes exp ( z ) : = [ exp ( zi ) ] 1≤i≤n ; other functions are defined analogously . Given two probability distributions µ and µ′ over A , the KL divergence from µ′ to µ is defined by KL ( µ ∥µ′ ) : = ∑ a∈A µ ( a ) log µ ( a ) µ′ ( a ) . Given a matrix A , ∥A∥∞ is used to denote entrywise maximum norm , namely , ∥A∥∞ = maxi , j |Ai , j | . The all-one vector is denoted as 1 .

Summary: The paper studies two standard classes of games with entropy regularization: two-player zero-sum matrix games and zero-sum Markov games. For the regularized matrix games, the authors apply the mirror descent approach (using KL distance) to the regularized objective, and propose two first-order descent/ascent algorithms using different optimistic predictions: the Predictive Update (PU) and the Optimistic Multiplicative Weights Update (OMWU). The authors prove that both algorithms converge to the unique Nash equilibrium of the regularized game with constant stepsizes, linearly. The authors characterize the linear convergence via four measures: KL distance, infinity norm, optimality gap, and duality gap. It is featured by last-iterate convergence and almost dimension-free. For the regularized Markov games, the authors apply the proposed PU and OMWU to the $Q$-value functions at each state and prove the convergence in infinite norm to the optimal $Q$-value function of the regularized game. The convergence rate is slightly worse than the linear rate, but it is also almost dimension-free. The authors also provide computational experiments to verify the usefulness of the proposed algorithms.