Policy Optimization in Zero-Sum Markov Games: Fictitious Self-Play Provably Attains Nash Equilibria

1 INTRODUCTION . Multi-agent reinforcement learning ( MARL ) ( Bu et al. , 2008 ; Sutton & Barto , 2018 ) has achieved great empirical success , e.g. , in playing the game of Go ( Silver et al. , 2016 ; 2017 ) , Dota 2 ( Berner et al. , 2019 ) , and StarCraft 2 ( Vinyals et al. , 2019 ) , which are all driven by policy optimization algorithms which iteratively update the policies that are parameterized using deep neural networks . Empirically , the popularity of policy optimization algorithms for MARL is attributed to the observations that they usually converges faster than value-based methods that iteratively update the value functions ( Mnih et al. , 2016 ; O ’ Donoghue et al. , 2016 ) . Compared with their empirical success , the theoretical aspect of policy optimization algorithms in MARL setting ( Littman , 1994 ; Hu & Wellman , 2003 ; Conitzer & Sandholm , 2007 ; Pérolat et al. , 2016 ; Zhang et al. , 2018 ) remains less understood . Although convergence guarantees for various policy optimization algorithms have been established under the single-agent RL setting ( Sutton et al. , 2000 ; Konda & Tsitsiklis , 2000 ; Kakade , 2002 ; Agarwal et al. , 2019 ; Wang et al. , 2019 ) , extending those theoretical guarantees to arguably one of the simplest settings of MARL , two-player zero-sum Markov game , suffers from challenges in the following two aspects . First , in such a Markov game , each agent interact with the opponent as well as the environment . Seen from the perspective of each agent , it belongs to an environment that is altered by the actions of the opponent . As a result , due to the existence of an opponent , the policy optimization problem of each agent has a time-varying objective function , which is in stark contrast with the value-based methods such as value-iteration Shapley ( 1953 ) ; Littman ( 1994 ) , where there is a central controller which specifies the policies of both players . When the joint policy of both players are considered , the problem of solving the optimal value function corresponds to finding the fixed point of the Bellman operator , which is defined independently of the policy of the players . Second , when viewing the policy optimization in zero-sum Markov game as an optimization problem for both players together , although we have a fixed objective function , the problem is minimax optimization with a non-convex non-concave objective . Even for classical optimization , such a kind of optimization problem remains less less understood ( Cherukuri et al. , 2017 ; Rafique et al. , 2018 ; Daskalakis & Panageas , 2018 ; Mertikopoulos et al. , 2018 ) . It is observed that first-order methods such as gradient descent might fail to converge ( Balduzzi et al. , 2018 ; Mazumdar & Ratliff , 2018 ) . As an initial step to study policy optimization for MARL , we propose a novel policy optimization algorithm for any player of a multi-player Markov game , which is dubbed as smooth fictitious selfplay ( FSP ) . Specifically , when a player adopts smooth FSP , in each iteration , it first solves a policy evaluation problem that estimates the value function associate with the current joint policy of all players . Then it update its own policy via an entropy-regularized proximal policy optimization ( PPO ) Schulman et al . ( 2017 ) step , where the update direction is obtained from the estimated value function . This algorithm can be viewed as an extension of the fictitious play ( FP ) algorithm that is designed for normal-form games ( Von Neumann & Morgenstern , 2007 ; Shapley , 1953 ) and extensive-form games ( Heinrich et al. , 2015 ; Perolat et al. , 2018 ) to Markov-games . FP is a general algorithmic framework for solving games where an agent first infer the policy of the opponents and then adopt a policy that best respond to the inferred opponents . When viewing our algorithm as a FP method , instead of estimating the policies of the opponents directly , the agent infers the opponent implicitly by estimating the value function . Besides , policy update corresponds to a smoothed best-response policy Swenson & Poor ( 2019 ) based on the inferred value function . To examine the theoretical merits of the proposed algorithm , we focus on two-player zero-sum Markov games and let both players follow smooth FSP , i.e. , with self-play . Moreover , we restrict to a class of Lipschitz games ( Radanovic et al. , 2019 ) where the impact of each player ’ s policy change on the environment is Lipschitz continuous with respect to the magnitude of policy change . For such a Markov game , we tackle the challenge of non-stationarity by imposing entropy regularization which brings algorithmic stability . In addition , to establish convergence to Nash equilibrium , we explicitly characterize the geometry of the policy optimization problem from a functional perspective . Specifically , we prove that the objective function , as a bivariate function of the two players ’ policies , despite being non-convex and non-concave , satisfies a one-point strong monotonicity condition ( Facchinei & Pang , 2007 ) at a Nash equilibrium . Thanks to such benign geometry , we prove that smooth FSP converges to a neighborhood of a Nash equilibrium at a sublinear Õ ( 1/T ) rate , where T is the number of policy iterations and Õ hides logarithmic factors . Moreover , as a byproduct of our analysis , if any of the two players deviates from the proposed algorithm , it is shown the other player that follows smooth FSP exploits such deviation by finding the best-response policy at a same sublinear rate . Such a Hannan consistency property exhibited in our algorithm is related to Hennes et al . ( 2020 ) , which focus on normal-form games . Thus , our results also serve as a first step towards connecting regret between minimization in normal-form/extensive-form games and Markov games . Contribution . Our contribution is two-fold . First , we propose a novel policy optimization algorithm for Markov games , which can be viewed as a generalization of FP . Second , when applied to a class of two-player zero-sum Markov games satisfying a Lipschitz regularity condition , our algorithm provably enjoys global convergence to a neighborhood of a Nash equilibrium at a sublinear rate . To the best of our knowledge , we propose the first provable FSP-type algorithm with finite time convergence guarantee for zero-sum Markov games . Related Work . There is a large body of literature on the value-based methods to zero-sum Markov games ( Lagoudakis & Parr , 2012 ; Pérolat et al. , 2016 ; Zhang et al. , 2018 ; Zou et al. , 2019 ) . More recently , Perolat et al . ( 2018 ) prove that actor-critic fictitious play asymptotically converges to the Nash equilibrium , while our work provides finite time convergence guarantee to a neighborhood of a Nash equilibrium . In addition , Zhang et al . ( 2020 ) study the sample comlexity of planning algorithm in the model-based MARL settting as opposed to the model-free setting with function approximation in this paper . Closely related to smooth FSP proposed in this paper , there is a line of work in best-response algorithms ( Heinrich et al. , 2015 ; Heinrich & Silver , 2016 ) , which have also shown great empirical performances ( Dudziak , 2006 ; Xiao et al. , 2013 ; Kawamura et al. , 2017 ) . However , they are only applicable to extensive-form games and not directly applicable to stochastic games . Also , our smooth FSP is related to Swenson & Poor ( 2019 ) , which focus on the potential games . It does not enforce entropy-regularization and only provides asymptotic convergence guarantee to a neighborhood of the Nash equilibrium for smooth fictitious play in multi-player two-action potential games . Moreover , our work also falls into the realm of regularizing and smoothing techniques in reinforcement learning ( Dai et al. , 2017 ; Geist et al. , 2019 ; Shani et al. , 2019 ; Cen et al. , 2020 ) , which focus on the single-agent setting . 2 BACKGROUND . In this section , we briefly introduce the general setting of reinforcement learning for two-player zero-sum Markov games . Zero-Sum Markov Games . We consider the two-player zero-sum Markov game ( S , A1 , A2 , P , r , γ ) , where S ⊂ Rd is a compact state space , A1 and A2 are finite action spaces of Player 1 and Player 2 , respectively , P : S × S ×A1 ×A2 → [ 0 , 1 ] is the Markov transition kernel , r : S ×A1×A2 → [ −1 , 1 ] is the reward function of Player 1 , which implies that the reward function of Player 2 is −r , and γ ∈ ( 0 , 1 ) is the discount factor . Let r1 = r and r2 = −r be the reward functions of Player 1 and Player 2 , respectively . For notational simplicity , throughout this paper , we write Player −i as Player i ’ s opponent , where i ∈ { 1 , 2 } . In the rest of this paper , we omit i ∈ { 1 , 2 } where it is clear from the context . Also , we denote by Eπi , π−i [ · ] the expectation over the trajectory induced by the policy pair [ πi ; π−i ] . Given a policy π−i : A−i×S → [ 0 , 1 ] of Player−i , the performance of a policy πi : Ai×S → [ 0 , 1 ] of Player i is evaluated by its state-value function ( Vi-function ) V πi , π−i i : S → R , which is defined as V π i , π−i i ( s ) = Eπi , π−i [ ∞∑ t=0 γt · ri ( st , ait , a−it ) ∣∣∣∣ s0 = s ] . ( 2.1 ) Correspondingly , the performance of a policy πi : Ai × S → [ 0 , 1 ] of Player i is evaluated by its action-value function ( Qi-function ) Q πi , π−i i : S ×Ai×A−i → R , which is defined by the following Bellman equation , Qπ i , π−i i ( s , a i , a−i ) = ri ( s , a i , a−i ) + γ · Es′∼P ( · | s , ai , a−i ) [ V π i , π−i i ( s ′ ) ] . We denote by νπi , π−i ( s ) and σπi , π−i ( s , ai , a−i ) = πi ( ai | s ) · π−i ( a−i | s ) · νπi , π−i ( s ) the stationary state distribution and the stationary state-action distribution associated with the policy pair [ πi ; π−i ] , respectively . Correspondingly , we denote by Eσπi , π−i [ · ] and Eνπi , π−i [ · ] the expectations E ( s , ai , a−i ) ∼σπi , π−i [ · ] and Es∼νπi , π−i [ · ] , respectively . Throughout this paper , we denote by 〈· , ·〉 the inner product between vectors . Let [ π1∗ , π 2 ∗ ] be a Nash equilibrium of the two-player zero-sum Markov game ( S , A1 , A2 , P , r , γ ) , which exists ( Shapley , 1953 ) and satisfies J ( π1 , π2∗ ) ≤ J ( π1∗ , π2∗ ) ≤ J ( π1∗ , π2 ) for all policy pairs [ π1 ; π2 ] . Here we define the performance function as J ( π1 , π2 ) = Eν∗ [ V π 1 , π2 1 ( s ) ] , ( 2.2 ) where ν∗ is the stationary distribution σπ1∗ , π2∗ . Regularized Markov Games . Based on the definition of the two-player zero-sum Markov game ( S , A1 , A2 , P , r , γ ) , we define its entropy-regularized counterpart ( S , A1 , A2 , P , r , γ , λ1 , λ2 ) , where λ1 , λ2 ≥ 0 are the regularization parameters . Specifically , ( S , A1 , A2 , P , r , γ , λ1 , λ2 ) is defined as the two-player general-sum Markov game with the reward function of Player i replaced by its entropy-regularized counterpart rπ i , π−i ( i ) : S ×A i ×A−i → R , which is defined as rπ i , π−i ( i ) ( s , a i , a−i ) = ri ( s , a i , a−i ) − λi · log πi ( ai | s ) . ( 2.3 ) With a slight abuse of notation , we write rπ i , π−i i ( s ) = Eπi , π−i [ ri ( s , a i , a−i ) ] , rπ i , π−i ( i ) ( s ) = Eπi , π−i [ rπ i , π−i ( i ) ( s , a i , a−i ) ] = rπ i , π−i i ( s ) + λi ·H ( πi ( · | s ) ) as the state-reward function and the entropy-regularized state-reward function , respectively . Here H ( πi ( · | s ) ) = − ∑ ai∈Ai π i ( ai | s ) · log πi ( ai | s ) is the Shannon entropy . For Player i , the entropyregularized state-value function ( V ( i ) -function ) V πi , π−i ( i ) : S → R and the entropy-regularized action-value function ( Q ( i ) -function ) Q πi , π−i ( i ) : S ×A i ×A−i → R are defined as V π i , π−i ( i ) ( s ) = Eπi , π−i [ ∞∑ t=0 γt · rπ i , π−i ( i ) ( st , a i t , a −i t ) ∣∣∣∣ s0 = s ] , ( 2.4 ) Qπ i , π−i ( i ) ( s , a i , a−i ) = ri ( s , a i , a−i ) + γ · Es′∼P ( · | s , ai , a−i ) [ V π i , π−i ( i ) ( s ′ ) ] , ( 2.5 ) respectively . By the definition of rπ i , π−i ( i ) in ( 2.3 ) , we have that , for all policy pairs [ π i ; π−i ] and s ∈ S , ∣∣∣Eπi , π−i [ rπi , π−i ( i ) ( s , ai , a−i ) ] ∣∣∣ ≤ 1 + λi · log |Ai| , which , by ( 2.4 ) and ( 2.5 ) implies that , for all policy pairs [ πi ; π−i ] and ( s , ai , a−i ) ∈ S ×Ai ×A−i , ∣∣V πi , π−i ( i ) ( s ) ∣∣ ≤ V max ( i ) = 1 + λi · log |Ai|1− γ , ( 2.6 ) ∣∣Qπi , π−i ( i ) ( s , ai , a−i ) ∣∣ ≤ Qmax ( i ) = 1 + γ · ( 1 + λi · log |Ai| ) 1− γ . ( 2.7 )

Summary: This paper studies the problem of learning to play a Nash equilibrium in two-player, zero-sum Markov games.  This is a longstanding problem, with many algorithms proposed but relatively few theoretical convergence guarantees, and most of those either for quite restricted settings or with strong assumptions.  This is in stark contrast to the stateless setting of Normal Form Games, where we have many strong theoretical convergence guarantees.  The main algorithm is a version of the classic fictitious play algorithm.  Like prior adaptations of fictitious play to Markov Games, it operates on the Q-values, but a key novelty (at least in the stateful setting; similar ideas were recently applied in a special case of normal form games by Swenson and Poor 2019) is the use of a particular form of regularization in the best response process.  The main result is that as long as the game satisfies Lipschitz and Concentratability properties for each player when the other plays optimally and the policy updates are sufficiently accurate then play converges to a Nash equilibrium.