A Sharp Analysis of Model-based Reinforcement Learning with Self-Play

1 INTRODUCTION . This paper is concerned with the problem of multi-agent reinforcement learning ( multi-agent RL ) , in which multiple agents learn to make decisions in an unknown environment in order to maximize their ( own ) cumulative rewards . Multi-agent RL has achieved significant recent success in traditionally hard AI challenges including large-scale strategy games ( such as GO ) ( Silver et al. , 2016 ; 2017 ) , real-time video games involving team play such as Starcraft and Dota2 ( OpenAI , 2018 ; Vinyals et al. , 2019 ) , as well as behavior learning in complex social scenarios ( Baker et al. , 2020 ) . Achieving human-like ( or super-human ) performance in these games using multi-agent RL typically requires a large number of samples ( steps of game playing ) due to the necessity of exploration , and how to improve the sample complexity of multi-agent RL has been an important research question . One prevalent approach towards solving multi-agent RL is model-based methods , that is , to use the existing visitation data to build an estimate of the model ( i.e . transition dynamics and rewards ) , run an offline planning algorithm on the estimated model to obtain the policy , and play the policy in the environment . Such a principle underlies some of the earliest single-agent online RL algorithms such as E3 ( Kearns & Singh , 2002 ) and RMax ( Brafman & Tennenholtz , 2002 ) , and is conceptually appealing for multi-agent RL too since the multi-agent structure does not add complexity onto the model estimation part and only requires an appropriate multi-agent planning algorithm ( such as value iteration for games ( Shapley , 1953 ) ) in a black-box fashion . On the other hand , modelfree methods do not directly build estimates of the model , but instead directly estimate the value functions or action-value ( Q ) functions of the problem at the optimal/equilibrium policies , and play the greedy policies with respect to the estimated value functions . Model-free algorithms have also been well developed for multi-agent RL such as friend-or-foe Q-Learning ( Littman , 2001 ) and Nash Q-Learning ( Hu & Wellman , 2003 ) . While both model-based and model-free algorithms have been shown to be provably efficient in multi-agent RL in a recent line of work ( Bai & Jin , 2020 ; Xie et al. , 2020 ; Bai et al. , 2020 ) , a more precise understanding of the optimal sample complexities within these two types of algorithms ( respectively ) is still lacking . In the specific setting of two-player zero-sum Markov games , the current best sample complexity for model-based algorithms is achieved by the VI-ULCB ( Value Iteration with Upper/Lower Confidence Bounds ) algorithm ( Bai & Jin , 2020 ; Xie et al. , 2020 ) : In a tabular Markov game with S states , { A , B } actions for the two players , and horizon length H , VI-ULCB is able to find an -approximate Nash equilibrium policy in Õ ( H4S2AB/ 2 ) episodes of game playing . However , compared with the information-theoretic lower bound Ω ( H3S ( A+B ) / 2 ) , this rate has suboptimal dependencies on all of H , S , and A , B . In contrast , the current best sample complexity for model-free algorithms is achieved by Nash V-Learning ( Bai et al. , 2020 ) , which finds an -approximate Nash policy in Õ ( H6S ( A + B ) / 2 ) episodes . Compared with the lower bound , this is tight except for a poly ( H ) factor , which may seemingly suggest that model-free algorithms could be superior to model-based ones in multi-agent RL . However , such a conclusion would be in stark contrast to the single-agent MDP setting , where it is known that model-based algorithms are able to achieve minimax optimal sample complexities ( Jaksch et al. , 2010 ; Azar et al. , 2017 ) . It naturally arises whether model-free algorithms are indeed superior in multi-agent settings , or whether the existing analyses of model-based algorithms are not tight . This motivates us to ask the following research question : Question : How sample-efficient are model-based algorithms in multi-agent RL ? In this paper , we advance the theoretical understandings of multi-agent RL by presenting a sharp analysis of model-based algorithms on Markov games . Our core contribution is the design of a new model-based algorithm Optimistic Nash Value Iteration ( Nash-VI ) that achieves an almost optimal sample complexity for zero-sum Markov games and improves significantly over existing modelbased approaches . We summarize our main contributions as follows . A comparison between our and prior results can be found in Table 1 . • We design a new model-based algorithm Optimistic Nash Value Iteration ( Nash-VI ) that provably finds -approximate Nash equilibria for Markov games in Õ ( H3SAB/ 2 ) episodes of game playing ( Section 3 ) . This improves over the best existing model-based algorithm by O ( HS ) and is the first algorithm that matches the sample complexity lower bound except for a Õ ( min { A , B } ) factor , showing that model-based algorithms can indeed achieve an almost optimal sample complexity . Further , unlike state-of-the-art model-free algorithms such as Nash V-Learning ( Bai et al. , 2020 ) , this algorithm achieves in addition a Õ ( √ T ) regret bound , and outputs a simple Markov policy ( instead of a nested mixture of Markov policies as returned by Nash V-Learning ) . • We design an alternative algorithm Optimistic Value Iteration with Zero Reward ( VI-Zero ) that is able to perform task-agnostic ( reward-free ) learning for multiple Markov games sharing the same transition ( Section 4 ) . For N > 1 games with the same transition and different ( known ) rewards , VI-Zero can find -approximate Nash policy for all games simultaneously in Õ ( H4SAB logN/ 2 ) episodes of game playing , which scales logarithmically in the number of games . • We design the first line of sample-efficient algorithms for multi-player general-sum Markov games . In a multi-player game with M players and Ai actions per player , we show that an nearoptimal policy can be found in Õ ( H4S2 ∏ i∈ [ M ] Ai/ 2 ) episodes , where the desired optimality can be either one of Nash equilibrium , correlated equilibrium ( CE ) , or coarse correlated equilibrium ( CCE ) . We achieve this guarantee by either a multi-player version of Nash-VI or a multi-player version of reward-free value iteration ( Section 5 & Appendix C ) . Due to space limit , we defer a detailed survey of related works to Appendix A . 2 PRELIMINARIES . In this paper , we consider Markov Games ( MGs , Shapley , 1953 ; Littman , 1994 ) , which are also known as stochastic games in the literature . Markov games are the generalization of standard Markov Decision Processes ( MDPs ) into the multi-player setting , where each player seeks to maximize her own utility . For simplicity , in this section we describe the important special case of twoplayer zero-sum games , and return to the general formulation in Appendix C. Formally , we consider the tabular episodic version of two-player zero-sum Markov game , which we denote as MG ( H , S , A , B , P , r ) . HereH is the number of steps in each episode , S is the set of states with |S| ≤ S , ( A , B ) are the sets of actions of the max-player and the min-player respectively with |A| ≤ A and |B| ≤ B , P = { Ph } h∈ [ H ] is a collection of transition matrices , so that Ph ( ·|s , a , b ) gives the distribution of the next state if action pair ( a , b ) is taken at state s at step h , and r = { rh } h∈ [ H ] is a collection of reward functions , where rh : S × A × B → [ 0 , 1 ] is the deterministic reward function at step h.1 This reward represents both the gain of the max-player and the loss of the min-player , making the problem a zero-sum Markov game . In each episode of this MG , we start with a fixed initial state s1 . At each step h ∈ [ H ] , both players observe state sh ∈ S , and pick their own actions ah ∈ A and bh ∈ B simultaneously . Then , both players observe the actions of their opponent , receive reward rh ( sh , ah , bh ) , and then the environment transitions to the next state sh+1 ∼ Ph ( ·|sh , ah , bh ) . The episode ends when sH+1 is reached . Policy , value function . A ( Markov ) policy µ of the max-player is a collection of H functions { µh : S → ∆A } h∈ [ H ] , each mapping from a state to a distribution over actions . ( Here ∆A is the probability simplex over action set A . ) Similarly , a policy ν of the min-player is a collection of H functions { νh : S → ∆B } h∈ [ H ] . We use the notation µh ( a|s ) and νh ( b|s ) to represent the probability of taking action a or b for state s at step h under Markov policy µ or ν respectively . We use V µ , νh : S → R to denote the value function at step h under policy µ and ν , so that V µ , ν h ( s ) gives the expected cumulative rewards received under policy µ and ν , starting from s at step h : V µ , νh ( s ) : = Eµ , ν [ ∑H h′=h rh′ ( sh′ , ah′ , bh′ ) ∣∣∣ sh = s ] . ( 1 ) We also define Qµ , νh : S × A × B → R to be the Q-value function at step h so that Q µ , ν h ( s , a , b ) gives the cumulative rewards received under policy µ and ν , starting from ( s , a , b ) at step h : Qµ , νh ( s , a , b ) : = Eµ , ν [ ∑H h′=h rh′ ( sh′ , ah′ , bh′ ) ∣∣∣ sh = s , ah = a , bh = b ] . ( 2 ) For simplicity , we define operator Ph as [ PhV ] ( s , a , b ) : = Es′∼Ph ( ·|s , a , b ) V ( s′ ) for any value function V . We also use notation [ DπQ ] ( s ) : = E ( a , b ) ∼π ( · , ·|s ) Q ( s , a , b ) for any action-value function Q . By definition of value functions , we have the Bellman equation Qµ , νh ( s , a , b ) = ( rh + PhV µ , ν h+1 ) ( s , a , b ) , V µ , ν h ( s ) = ( Dµh×νhQ µ , ν h ) ( s ) 1We assume the rewards in [ 0 , 1 ] for normalization . Our results directly generalize to randomized reward functions , since learning the transition is more difficult than learning the reward . for all ( s , a , b , h ) ∈ S ×A×B× [ H ] , and at the ( H + 1 ) th step we have V µ , νH+1 ( s ) = 0 for all s ∈ S. Best response and Nash equilibrium . For any policy of the max-player µ , there exists a best response of the min-player , which is a policy ν† ( µ ) satisfying V µ , ν † ( µ ) h ( s ) = infν V µ , ν h ( s ) for any ( s , h ) ∈ S× [ H ] . We denote V µ , †h : = V µ , ν† ( µ ) h . By symmetry , we can also define µ † ( ν ) and V † , νh . It is further known ( cf . ( Filar & Vrieze , 2012 ) ) that there exist policies µ ? , ν ? that are optimal against the best responses of the opponents , in the sense that V µ ? , † h ( s ) = supµ V µ , † h ( s ) , V † , ν ? h ( s ) = infν V † , ν h ( s ) , for all ( s , h ) . We call these optimal strategies ( µ ? , ν ? ) the Nash equilibrium of the Markov game , which satisfies the following minimax equation 2 : supµ infν V µ , ν h ( s ) = V µ ? , ν ? h ( s ) = infν supµ V µ , ν h ( s ) . Intuitively , a Nash equilibrium gives a solution in which no player has anything to gain by changing only her own policy . We further abbreviate the values of Nash equilibrium V µ ? , ν ? h andQ µ ? , ν ? h as V ? h and Q ? h . We refer readers to Appendix D for Bellman optimality equations for ( the value functions of ) the best responses and the Nash equilibrium . Learning Objective . We measure the suboptimality of any pair of general policies ( µ̂ , ν̂ ) using the gap between their performance and the performance of the optimal strategy ( i.e. , Nash equilibrium ) when playing against the best responses respectively : V † , ν̂1 ( s1 ) − V µ̂ , † 1 ( s1 ) = [ V † , ν̂1 ( s1 ) − V ? 1 ( s1 ) ] + [ V ? 1 ( s1 ) − V µ̂ , † 1 ( s1 ) ] Definition 1 ( -approximate Nash equilibrium ) . A pair of general policies ( µ̂ , ν̂ ) is an - approximate Nash equilibrium , if V † , ν̂1 ( s1 ) − V µ̂ , † 1 ( s1 ) ≤ . Definition 2 ( Regret ) . Let ( µk , νk ) denote the policies deployed by the algorithm in the kth episode . After a total of K episodes , the regret is defined as Regret ( K ) = K∑ k=1 ( V † , ν k 1 − V µk , † 1 ) ( s1 ) . One goal of reinforcement learning is to design algorithms for Markov games that can find an -approximate Nash equilibrium using a number of episodes that is small in its dependency on S , A , B , H as well as 1/ ( PAC sample complexity bound ) . An alternative goal is to design algorithms for Markov games that achieves regret that is sublinear in K , and polynomial in S , A , B , H ( regret bound ) . We remark that any sublinear regret algorithm can be directly converted to a polynomial-sample PAC algorithm via the standard online-to-batch conversion ( see e.g. , Jin et al . ( 2018 ) ) .

Summary: The authors introduce new algorithms to solve two-players zero-sum Markov games, as well as two-players Markov game in the reward-free setting. The approach is model-based, based on successive episods of planning and counting for updating the model estimate. It involves solving a matrix game at each iteration, looking for a notion of equilibria that is computable in polynomial time (unlike Nash equilibria) An extension to multi-player games is proposed for both reward and reward free setting (in the appendix).