Policy Optimization in Zero-Sum Markov Games: Fictitious Self-Play Provably Attains Nash Equilibria

1 INTRODUCTION . Multi-agent reinforcement learning ( MARL ) ( Bu et al. , 2008 ; Sutton & Barto , 2018 ) has achieved great empirical success , e.g. , in playing the game of Go ( Silver et al. , 2016 ; 2017 ) , Dota 2 ( Berner et al. , 2019 ) , and StarCraft 2 ( Vinyals et al. , 2019 ) , which are all driven by policy optimization algorithms which iteratively update the policies that are parameterized using deep neural networks . Empirically , the popularity of policy optimization algorithms for MARL is attributed to the observations that they usually converges faster than value-based methods that iteratively update the value functions ( Mnih et al. , 2016 ; O ’ Donoghue et al. , 2016 ) . Compared with their empirical success , the theoretical aspect of policy optimization algorithms in MARL setting ( Littman , 1994 ; Hu & Wellman , 2003 ; Conitzer & Sandholm , 2007 ; Pérolat et al. , 2016 ; Zhang et al. , 2018 ) remains less understood . Although convergence guarantees for various policy optimization algorithms have been established under the single-agent RL setting ( Sutton et al. , 2000 ; Konda & Tsitsiklis , 2000 ; Kakade , 2002 ; Agarwal et al. , 2019 ; Wang et al. , 2019 ) , extending those theoretical guarantees to arguably one of the simplest settings of MARL , two-player zero-sum Markov game , suffers from challenges in the following two aspects . First , in such a Markov game , each agent interact with the opponent as well as the environment . Seen from the perspective of each agent , it belongs to an environment that is altered by the actions of the opponent . As a result , due to the existence of an opponent , the policy optimization problem of each agent has a time-varying objective function , which is in stark contrast with the value-based methods such as value-iteration Shapley ( 1953 ) ; Littman ( 1994 ) , where there is a central controller which specifies the policies of both players . When the joint policy of both players are considered , the problem of solving the optimal value function corresponds to finding the fixed point of the Bellman operator , which is defined independently of the policy of the players . Second , when viewing the policy optimization in zero-sum Markov game as an optimization problem for both players together , although we have a fixed objective function , the problem is minimax optimization with a non-convex non-concave objective . Even for classical optimization , such a kind of optimization problem remains less less understood ( Cherukuri et al. , 2017 ; Rafique et al. , 2018 ; Daskalakis & Panageas , 2018 ; Mertikopoulos et al. , 2018 ) . It is observed that first-order methods such as gradient descent might fail to converge ( Balduzzi et al. , 2018 ; Mazumdar & Ratliff , 2018 ) . As an initial step to study policy optimization for MARL , we propose a novel policy optimization algorithm for any player of a multi-player Markov game , which is dubbed as smooth fictitious selfplay ( FSP ) . Specifically , when a player adopts smooth FSP , in each iteration , it first solves a policy evaluation problem that estimates the value function associate with the current joint policy of all players . Then it update its own policy via an entropy-regularized proximal policy optimization ( PPO ) Schulman et al . ( 2017 ) step , where the update direction is obtained from the estimated value function . This algorithm can be viewed as an extension of the fictitious play ( FP ) algorithm that is designed for normal-form games ( Von Neumann & Morgenstern , 2007 ; Shapley , 1953 ) and extensive-form games ( Heinrich et al. , 2015 ; Perolat et al. , 2018 ) to Markov-games . FP is a general algorithmic framework for solving games where an agent first infer the policy of the opponents and then adopt a policy that best respond to the inferred opponents . When viewing our algorithm as a FP method , instead of estimating the policies of the opponents directly , the agent infers the opponent implicitly by estimating the value function . Besides , policy update corresponds to a smoothed best-response policy Swenson & Poor ( 2019 ) based on the inferred value function . To examine the theoretical merits of the proposed algorithm , we focus on two-player zero-sum Markov games and let both players follow smooth FSP , i.e. , with self-play . Moreover , we restrict to a class of Lipschitz games ( Radanovic et al. , 2019 ) where the impact of each player ’ s policy change on the environment is Lipschitz continuous with respect to the magnitude of policy change . For such a Markov game , we tackle the challenge of non-stationarity by imposing entropy regularization which brings algorithmic stability . In addition , to establish convergence to Nash equilibrium , we explicitly characterize the geometry of the policy optimization problem from a functional perspective . Specifically , we prove that the objective function , as a bivariate function of the two players ’ policies , despite being non-convex and non-concave , satisfies a one-point strong monotonicity condition ( Facchinei & Pang , 2007 ) at a Nash equilibrium . Thanks to such benign geometry , we prove that smooth FSP converges to a neighborhood of a Nash equilibrium at a sublinear Õ ( 1/T ) rate , where T is the number of policy iterations and Õ hides logarithmic factors . Moreover , as a byproduct of our analysis , if any of the two players deviates from the proposed algorithm , it is shown the other player that follows smooth FSP exploits such deviation by finding the best-response policy at a same sublinear rate . Such a Hannan consistency property exhibited in our algorithm is related to Hennes et al . ( 2020 ) , which focus on normal-form games . Thus , our results also serve as a first step towards connecting regret between minimization in normal-form/extensive-form games and Markov games . Contribution . Our contribution is two-fold . First , we propose a novel policy optimization algorithm for Markov games , which can be viewed as a generalization of FP . Second , when applied to a class of two-player zero-sum Markov games satisfying a Lipschitz regularity condition , our algorithm provably enjoys global convergence to a neighborhood of a Nash equilibrium at a sublinear rate . To the best of our knowledge , we propose the first provable FSP-type algorithm with finite time convergence guarantee for zero-sum Markov games . Related Work . There is a large body of literature on the value-based methods to zero-sum Markov games ( Lagoudakis & Parr , 2012 ; Pérolat et al. , 2016 ; Zhang et al. , 2018 ; Zou et al. , 2019 ) . More recently , Perolat et al . ( 2018 ) prove that actor-critic fictitious play asymptotically converges to the Nash equilibrium , while our work provides finite time convergence guarantee to a neighborhood of a Nash equilibrium . In addition , Zhang et al . ( 2020 ) study the sample comlexity of planning algorithm in the model-based MARL settting as opposed to the model-free setting with function approximation in this paper . Closely related to smooth FSP proposed in this paper , there is a line of work in best-response algorithms ( Heinrich et al. , 2015 ; Heinrich & Silver , 2016 ) , which have also shown great empirical performances ( Dudziak , 2006 ; Xiao et al. , 2013 ; Kawamura et al. , 2017 ) . However , they are only applicable to extensive-form games and not directly applicable to stochastic games . Also , our smooth FSP is related to Swenson & Poor ( 2019 ) , which focus on the potential games . It does not enforce entropy-regularization and only provides asymptotic convergence guarantee to a neighborhood of the Nash equilibrium for smooth fictitious play in multi-player two-action potential games . Moreover , our work also falls into the realm of regularizing and smoothing techniques in reinforcement learning ( Dai et al. , 2017 ; Geist et al. , 2019 ; Shani et al. , 2019 ; Cen et al. , 2020 ) , which focus on the single-agent setting . 2 BACKGROUND . In this section , we briefly introduce the general setting of reinforcement learning for two-player zero-sum Markov games . Zero-Sum Markov Games . We consider the two-player zero-sum Markov game ( S , A1 , A2 , P , r , γ ) , where S ⊂ Rd is a compact state space , A1 and A2 are finite action spaces of Player 1 and Player 2 , respectively , P : S × S ×A1 ×A2 → [ 0 , 1 ] is the Markov transition kernel , r : S ×A1×A2 → [ −1 , 1 ] is the reward function of Player 1 , which implies that the reward function of Player 2 is −r , and γ ∈ ( 0 , 1 ) is the discount factor . Let r1 = r and r2 = −r be the reward functions of Player 1 and Player 2 , respectively . For notational simplicity , throughout this paper , we write Player −i as Player i ’ s opponent , where i ∈ { 1 , 2 } . In the rest of this paper , we omit i ∈ { 1 , 2 } where it is clear from the context . Also , we denote by Eπi , π−i [ · ] the expectation over the trajectory induced by the policy pair [ πi ; π−i ] . Given a policy π−i : A−i×S → [ 0 , 1 ] of Player−i , the performance of a policy πi : Ai×S → [ 0 , 1 ] of Player i is evaluated by its state-value function ( Vi-function ) V πi , π−i i : S → R , which is defined as V π i , π−i i ( s ) = Eπi , π−i [ ∞∑ t=0 γt · ri ( st , ait , a−it ) ∣∣∣∣ s0 = s ] . ( 2.1 ) Correspondingly , the performance of a policy πi : Ai × S → [ 0 , 1 ] of Player i is evaluated by its action-value function ( Qi-function ) Q πi , π−i i : S ×Ai×A−i → R , which is defined by the following Bellman equation , Qπ i , π−i i ( s , a i , a−i ) = ri ( s , a i , a−i ) + γ · Es′∼P ( · | s , ai , a−i ) [ V π i , π−i i ( s ′ ) ] . We denote by νπi , π−i ( s ) and σπi , π−i ( s , ai , a−i ) = πi ( ai | s ) · π−i ( a−i | s ) · νπi , π−i ( s ) the stationary state distribution and the stationary state-action distribution associated with the policy pair [ πi ; π−i ] , respectively . Correspondingly , we denote by Eσπi , π−i [ · ] and Eνπi , π−i [ · ] the expectations E ( s , ai , a−i ) ∼σπi , π−i [ · ] and Es∼νπi , π−i [ · ] , respectively . Throughout this paper , we denote by 〈· , ·〉 the inner product between vectors . Let [ π1∗ , π 2 ∗ ] be a Nash equilibrium of the two-player zero-sum Markov game ( S , A1 , A2 , P , r , γ ) , which exists ( Shapley , 1953 ) and satisfies J ( π1 , π2∗ ) ≤ J ( π1∗ , π2∗ ) ≤ J ( π1∗ , π2 ) for all policy pairs [ π1 ; π2 ] . Here we define the performance function as J ( π1 , π2 ) = Eν∗ [ V π 1 , π2 1 ( s ) ] , ( 2.2 ) where ν∗ is the stationary distribution σπ1∗ , π2∗ . Regularized Markov Games . Based on the definition of the two-player zero-sum Markov game ( S , A1 , A2 , P , r , γ ) , we define its entropy-regularized counterpart ( S , A1 , A2 , P , r , γ , λ1 , λ2 ) , where λ1 , λ2 ≥ 0 are the regularization parameters . Specifically , ( S , A1 , A2 , P , r , γ , λ1 , λ2 ) is defined as the two-player general-sum Markov game with the reward function of Player i replaced by its entropy-regularized counterpart rπ i , π−i ( i ) : S ×A i ×A−i → R , which is defined as rπ i , π−i ( i ) ( s , a i , a−i ) = ri ( s , a i , a−i ) − λi · log πi ( ai | s ) . ( 2.3 ) With a slight abuse of notation , we write rπ i , π−i i ( s ) = Eπi , π−i [ ri ( s , a i , a−i ) ] , rπ i , π−i ( i ) ( s ) = Eπi , π−i [ rπ i , π−i ( i ) ( s , a i , a−i ) ] = rπ i , π−i i ( s ) + λi ·H ( πi ( · | s ) ) as the state-reward function and the entropy-regularized state-reward function , respectively . Here H ( πi ( · | s ) ) = − ∑ ai∈Ai π i ( ai | s ) · log πi ( ai | s ) is the Shannon entropy . For Player i , the entropyregularized state-value function ( V ( i ) -function ) V πi , π−i ( i ) : S → R and the entropy-regularized action-value function ( Q ( i ) -function ) Q πi , π−i ( i ) : S ×A i ×A−i → R are defined as V π i , π−i ( i ) ( s ) = Eπi , π−i [ ∞∑ t=0 γt · rπ i , π−i ( i ) ( st , a i t , a −i t ) ∣∣∣∣ s0 = s ] , ( 2.4 ) Qπ i , π−i ( i ) ( s , a i , a−i ) = ri ( s , a i , a−i ) + γ · Es′∼P ( · | s , ai , a−i ) [ V π i , π−i ( i ) ( s ′ ) ] , ( 2.5 ) respectively . By the definition of rπ i , π−i ( i ) in ( 2.3 ) , we have that , for all policy pairs [ π i ; π−i ] and s ∈ S , ∣∣∣Eπi , π−i [ rπi , π−i ( i ) ( s , ai , a−i ) ] ∣∣∣ ≤ 1 + λi · log |Ai| , which , by ( 2.4 ) and ( 2.5 ) implies that , for all policy pairs [ πi ; π−i ] and ( s , ai , a−i ) ∈ S ×Ai ×A−i , ∣∣V πi , π−i ( i ) ( s ) ∣∣ ≤ V max ( i ) = 1 + λi · log |Ai|1− γ , ( 2.6 ) ∣∣Qπi , π−i ( i ) ( s , ai , a−i ) ∣∣ ≤ Qmax ( i ) = 1 + γ · ( 1 + λi · log |Ai| ) 1− γ . ( 2.7 )

Summary: The authors consider self-play in zero-sum discounted two-player Markov games with compact state space and finite actions. They present a smooth fictitious self-play algorithm where each player adopts an entropy-regularized policy optimization method with the average of the past generated Q-values. Under appropriate assumptions, among which a Lipschitz regularity of the Markov game, the authors prove that this algorithm approximates the Nash equilibrium at a rate O(1/T) where T is the number of iteration.