A Sharp Analysis of Model-based Reinforcement Learning with Self-Play

1 INTRODUCTION . This paper is concerned with the problem of multi-agent reinforcement learning ( multi-agent RL ) , in which multiple agents learn to make decisions in an unknown environment in order to maximize their ( own ) cumulative rewards . Multi-agent RL has achieved significant recent success in traditionally hard AI challenges including large-scale strategy games ( such as GO ) ( Silver et al. , 2016 ; 2017 ) , real-time video games involving team play such as Starcraft and Dota2 ( OpenAI , 2018 ; Vinyals et al. , 2019 ) , as well as behavior learning in complex social scenarios ( Baker et al. , 2020 ) . Achieving human-like ( or super-human ) performance in these games using multi-agent RL typically requires a large number of samples ( steps of game playing ) due to the necessity of exploration , and how to improve the sample complexity of multi-agent RL has been an important research question . One prevalent approach towards solving multi-agent RL is model-based methods , that is , to use the existing visitation data to build an estimate of the model ( i.e . transition dynamics and rewards ) , run an offline planning algorithm on the estimated model to obtain the policy , and play the policy in the environment . Such a principle underlies some of the earliest single-agent online RL algorithms such as E3 ( Kearns & Singh , 2002 ) and RMax ( Brafman & Tennenholtz , 2002 ) , and is conceptually appealing for multi-agent RL too since the multi-agent structure does not add complexity onto the model estimation part and only requires an appropriate multi-agent planning algorithm ( such as value iteration for games ( Shapley , 1953 ) ) in a black-box fashion . On the other hand , modelfree methods do not directly build estimates of the model , but instead directly estimate the value functions or action-value ( Q ) functions of the problem at the optimal/equilibrium policies , and play the greedy policies with respect to the estimated value functions . Model-free algorithms have also been well developed for multi-agent RL such as friend-or-foe Q-Learning ( Littman , 2001 ) and Nash Q-Learning ( Hu & Wellman , 2003 ) . While both model-based and model-free algorithms have been shown to be provably efficient in multi-agent RL in a recent line of work ( Bai & Jin , 2020 ; Xie et al. , 2020 ; Bai et al. , 2020 ) , a more precise understanding of the optimal sample complexities within these two types of algorithms ( respectively ) is still lacking . In the specific setting of two-player zero-sum Markov games , the current best sample complexity for model-based algorithms is achieved by the VI-ULCB ( Value Iteration with Upper/Lower Confidence Bounds ) algorithm ( Bai & Jin , 2020 ; Xie et al. , 2020 ) : In a tabular Markov game with S states , { A , B } actions for the two players , and horizon length H , VI-ULCB is able to find an -approximate Nash equilibrium policy in Õ ( H4S2AB/ 2 ) episodes of game playing . However , compared with the information-theoretic lower bound Ω ( H3S ( A+B ) / 2 ) , this rate has suboptimal dependencies on all of H , S , and A , B . In contrast , the current best sample complexity for model-free algorithms is achieved by Nash V-Learning ( Bai et al. , 2020 ) , which finds an -approximate Nash policy in Õ ( H6S ( A + B ) / 2 ) episodes . Compared with the lower bound , this is tight except for a poly ( H ) factor , which may seemingly suggest that model-free algorithms could be superior to model-based ones in multi-agent RL . However , such a conclusion would be in stark contrast to the single-agent MDP setting , where it is known that model-based algorithms are able to achieve minimax optimal sample complexities ( Jaksch et al. , 2010 ; Azar et al. , 2017 ) . It naturally arises whether model-free algorithms are indeed superior in multi-agent settings , or whether the existing analyses of model-based algorithms are not tight . This motivates us to ask the following research question : Question : How sample-efficient are model-based algorithms in multi-agent RL ? In this paper , we advance the theoretical understandings of multi-agent RL by presenting a sharp analysis of model-based algorithms on Markov games . Our core contribution is the design of a new model-based algorithm Optimistic Nash Value Iteration ( Nash-VI ) that achieves an almost optimal sample complexity for zero-sum Markov games and improves significantly over existing modelbased approaches . We summarize our main contributions as follows . A comparison between our and prior results can be found in Table 1 . • We design a new model-based algorithm Optimistic Nash Value Iteration ( Nash-VI ) that provably finds -approximate Nash equilibria for Markov games in Õ ( H3SAB/ 2 ) episodes of game playing ( Section 3 ) . This improves over the best existing model-based algorithm by O ( HS ) and is the first algorithm that matches the sample complexity lower bound except for a Õ ( min { A , B } ) factor , showing that model-based algorithms can indeed achieve an almost optimal sample complexity . Further , unlike state-of-the-art model-free algorithms such as Nash V-Learning ( Bai et al. , 2020 ) , this algorithm achieves in addition a Õ ( √ T ) regret bound , and outputs a simple Markov policy ( instead of a nested mixture of Markov policies as returned by Nash V-Learning ) . • We design an alternative algorithm Optimistic Value Iteration with Zero Reward ( VI-Zero ) that is able to perform task-agnostic ( reward-free ) learning for multiple Markov games sharing the same transition ( Section 4 ) . For N > 1 games with the same transition and different ( known ) rewards , VI-Zero can find -approximate Nash policy for all games simultaneously in Õ ( H4SAB logN/ 2 ) episodes of game playing , which scales logarithmically in the number of games . • We design the first line of sample-efficient algorithms for multi-player general-sum Markov games . In a multi-player game with M players and Ai actions per player , we show that an nearoptimal policy can be found in Õ ( H4S2 ∏ i∈ [ M ] Ai/ 2 ) episodes , where the desired optimality can be either one of Nash equilibrium , correlated equilibrium ( CE ) , or coarse correlated equilibrium ( CCE ) . We achieve this guarantee by either a multi-player version of Nash-VI or a multi-player version of reward-free value iteration ( Section 5 & Appendix C ) . Due to space limit , we defer a detailed survey of related works to Appendix A . 2 PRELIMINARIES . In this paper , we consider Markov Games ( MGs , Shapley , 1953 ; Littman , 1994 ) , which are also known as stochastic games in the literature . Markov games are the generalization of standard Markov Decision Processes ( MDPs ) into the multi-player setting , where each player seeks to maximize her own utility . For simplicity , in this section we describe the important special case of twoplayer zero-sum games , and return to the general formulation in Appendix C. Formally , we consider the tabular episodic version of two-player zero-sum Markov game , which we denote as MG ( H , S , A , B , P , r ) . HereH is the number of steps in each episode , S is the set of states with |S| ≤ S , ( A , B ) are the sets of actions of the max-player and the min-player respectively with |A| ≤ A and |B| ≤ B , P = { Ph } h∈ [ H ] is a collection of transition matrices , so that Ph ( ·|s , a , b ) gives the distribution of the next state if action pair ( a , b ) is taken at state s at step h , and r = { rh } h∈ [ H ] is a collection of reward functions , where rh : S × A × B → [ 0 , 1 ] is the deterministic reward function at step h.1 This reward represents both the gain of the max-player and the loss of the min-player , making the problem a zero-sum Markov game . In each episode of this MG , we start with a fixed initial state s1 . At each step h ∈ [ H ] , both players observe state sh ∈ S , and pick their own actions ah ∈ A and bh ∈ B simultaneously . Then , both players observe the actions of their opponent , receive reward rh ( sh , ah , bh ) , and then the environment transitions to the next state sh+1 ∼ Ph ( ·|sh , ah , bh ) . The episode ends when sH+1 is reached . Policy , value function . A ( Markov ) policy µ of the max-player is a collection of H functions { µh : S → ∆A } h∈ [ H ] , each mapping from a state to a distribution over actions . ( Here ∆A is the probability simplex over action set A . ) Similarly , a policy ν of the min-player is a collection of H functions { νh : S → ∆B } h∈ [ H ] . We use the notation µh ( a|s ) and νh ( b|s ) to represent the probability of taking action a or b for state s at step h under Markov policy µ or ν respectively . We use V µ , νh : S → R to denote the value function at step h under policy µ and ν , so that V µ , ν h ( s ) gives the expected cumulative rewards received under policy µ and ν , starting from s at step h : V µ , νh ( s ) : = Eµ , ν [ ∑H h′=h rh′ ( sh′ , ah′ , bh′ ) ∣∣∣ sh = s ] . ( 1 ) We also define Qµ , νh : S × A × B → R to be the Q-value function at step h so that Q µ , ν h ( s , a , b ) gives the cumulative rewards received under policy µ and ν , starting from ( s , a , b ) at step h : Qµ , νh ( s , a , b ) : = Eµ , ν [ ∑H h′=h rh′ ( sh′ , ah′ , bh′ ) ∣∣∣ sh = s , ah = a , bh = b ] . ( 2 ) For simplicity , we define operator Ph as [ PhV ] ( s , a , b ) : = Es′∼Ph ( ·|s , a , b ) V ( s′ ) for any value function V . We also use notation [ DπQ ] ( s ) : = E ( a , b ) ∼π ( · , ·|s ) Q ( s , a , b ) for any action-value function Q . By definition of value functions , we have the Bellman equation Qµ , νh ( s , a , b ) = ( rh + PhV µ , ν h+1 ) ( s , a , b ) , V µ , ν h ( s ) = ( Dµh×νhQ µ , ν h ) ( s ) 1We assume the rewards in [ 0 , 1 ] for normalization . Our results directly generalize to randomized reward functions , since learning the transition is more difficult than learning the reward . for all ( s , a , b , h ) ∈ S ×A×B× [ H ] , and at the ( H + 1 ) th step we have V µ , νH+1 ( s ) = 0 for all s ∈ S. Best response and Nash equilibrium . For any policy of the max-player µ , there exists a best response of the min-player , which is a policy ν† ( µ ) satisfying V µ , ν † ( µ ) h ( s ) = infν V µ , ν h ( s ) for any ( s , h ) ∈ S× [ H ] . We denote V µ , †h : = V µ , ν† ( µ ) h . By symmetry , we can also define µ † ( ν ) and V † , νh . It is further known ( cf . ( Filar & Vrieze , 2012 ) ) that there exist policies µ ? , ν ? that are optimal against the best responses of the opponents , in the sense that V µ ? , † h ( s ) = supµ V µ , † h ( s ) , V † , ν ? h ( s ) = infν V † , ν h ( s ) , for all ( s , h ) . We call these optimal strategies ( µ ? , ν ? ) the Nash equilibrium of the Markov game , which satisfies the following minimax equation 2 : supµ infν V µ , ν h ( s ) = V µ ? , ν ? h ( s ) = infν supµ V µ , ν h ( s ) . Intuitively , a Nash equilibrium gives a solution in which no player has anything to gain by changing only her own policy . We further abbreviate the values of Nash equilibrium V µ ? , ν ? h andQ µ ? , ν ? h as V ? h and Q ? h . We refer readers to Appendix D for Bellman optimality equations for ( the value functions of ) the best responses and the Nash equilibrium . Learning Objective . We measure the suboptimality of any pair of general policies ( µ̂ , ν̂ ) using the gap between their performance and the performance of the optimal strategy ( i.e. , Nash equilibrium ) when playing against the best responses respectively : V † , ν̂1 ( s1 ) − V µ̂ , † 1 ( s1 ) = [ V † , ν̂1 ( s1 ) − V ? 1 ( s1 ) ] + [ V ? 1 ( s1 ) − V µ̂ , † 1 ( s1 ) ] Definition 1 ( -approximate Nash equilibrium ) . A pair of general policies ( µ̂ , ν̂ ) is an - approximate Nash equilibrium , if V † , ν̂1 ( s1 ) − V µ̂ , † 1 ( s1 ) ≤ . Definition 2 ( Regret ) . Let ( µk , νk ) denote the policies deployed by the algorithm in the kth episode . After a total of K episodes , the regret is defined as Regret ( K ) = K∑ k=1 ( V † , ν k 1 − V µk , † 1 ) ( s1 ) . One goal of reinforcement learning is to design algorithms for Markov games that can find an -approximate Nash equilibrium using a number of episodes that is small in its dependency on S , A , B , H as well as 1/ ( PAC sample complexity bound ) . An alternative goal is to design algorithms for Markov games that achieves regret that is sublinear in K , and polynomial in S , A , B , H ( regret bound ) . We remark that any sublinear regret algorithm can be directly converted to a polynomial-sample PAC algorithm via the standard online-to-batch conversion ( see e.g. , Jin et al . ( 2018 ) ) .

Summary: This paper studies learning in stochastic games, which are extensions of Markov decision processes (MDPs) from the single-agent setup to the multi-agent one. Here the objective of each learner is to optimize her own reward function. Similarly to the case of MDPs, here one can devise learning algorithms with controlled sample complexity or regret (or both simultaneously) even when reward and transition functions are unknown.