When Can We Learn General-Sum Markov Games with a Large Number of Players Sample-Efficiently?

∑ i≤mAi/ε 3 ) episodes ( when only highlighting the dependence on S , Ai , and ε ) , which only depends linearly in ∑ i≤mAi and significantly improves over the best known algorithm in the ε dependence . Overall , our results shed light on what equilibria or structural assumptions on the game may enable sample-efficient learning with many players . 1 INTRODUCTION . Multi-agent reinforcement learning ( RL ) has achieved substantial recent successes in solving artificial intelligence challenges such as GO ( Silver et al. , 2016 ; 2018 ) , multi-player games with team play such as Starcraft ( Vinyals et al. , 2019 ) and Dota2 ( Berner et al. , 2019 ) , behavior learning in social interactions ( Baker et al. , 2019 ) , and economic simulation ( Zheng et al. , 2020 ; Trott et al. , 2021 ) . In many applications , multi-agent RL is able to yield high quality policies for multi-player games with a large number of players ( Wang et al. , 2016 ; Yang et al. , 2018 ) . Despite these empirical progresses , theoretical understanding of when we can sample-efficiently solve multi-player games with a large number of players remains elusive , especially in the setting of multi-player Markov games . A main bottleneck here is the exponential blow-up of the joint action space—The total number of joint actions in a generic game with simultaneous plays is equal to the product of the number of actions for each player , which scales exponentially in the number of players . Such an exponential dependence is indeed known to be unavoidable in the worst-case for certain standard problems . For example , for learning an approximate Nash equilibrium from payoff queries in an one-step multi-player general-sum game , the query complexity lower bound of ( Chen et al. , 2015 ; Rubinstein , 2016 ) shows that at least exponentially many queries ( samples ) is required , even when each player only has two possible actions and the query is noiseless . Moreover , for learning Nash equilibrium in Markov games , the best existing sample complexity upper bound also scales with the size of the joint action space ( Liu et al. , 2021 ) . Nevertheless , these exponential lower bounds do not completely rule out interesting theoretical inquiries—there may well be other notions of equilibria or additional structures within the game that allow us to learn with a better sample complexity . This motivates us to ask the following Question : When can we solve general-sum Markov games with sample complexity milder than exponential in the number of players ? This paper makes steps towards answering the above question by considering multi-player generalsum Markov games ( MGs ) withm players , H steps , S states , andAi actions per player . We make two lines of investigations : ( 1 ) Can we learn alternative notions of equilibria with better sample complexity than learning Nash ; ( 2 ) Can the Nash equilibrium be learned with better sample complexity under additional structural assumptions on the game . This paper makes contributions on both ends , which we summarize as follows . • We first design an algorithm that learns the ε-approximate Coarse Correlated Equilibrium ( CCE ) with Õ ( H5Smaxi∈ [ m ] Ai/ε2 ) episodes of play ( Section 3 ) . Our algorithm CCE-VLEARNING is a multi-player adaptation of the Nash V-Learning algorithm of Bai et al . ( 2020 ) . • We design an algorithm CE-V-LEARNING which learns the stricter notion of ε-approximate Correlated Equilibrium ( CE ) with Õ ( H6 maxi∈ [ m ] A2i /ε2 ) episodes of play ( Section 4 ) . For Markov games , these are the first line of sample complexity results for learning CE and CCE that only scales polynomially with maxi∈ [ m ] Ai , and improves significantly in the Ai dependency over the current best algorithm which scales with ∏ i∈ [ m ] Ai . • Technically , our algorithm CE-V-LEARNING makes several major modifications over CCEV-LEARNING in order to learn the CE ( Section 4.2 ) . Notably , inspired by the connection between CE and low swap-regret learning , we use a mixed-expert Follow-The-Regularized Leader algorithm within its inner loop to achieve low swap-regret for a particular adversarial bandit problem . Our analysis also contains new results for adversarial bandits on weighted swap regret and weighted regret with predicable weights , which may be of independent interest . • Finally , we consider learning Nash equilibrium in Markov Potential Games ( MPGs ) , an important subclass of general-sum Markov games . By a reduction to single-agent RL , we design an algorithm NASH-CA that achieves Õ ( ΦmaxH3S ∑ i∈ [ m ] Ai/ε 3 ) sample complexity , where Φmax ≤ Hm is the bound on the potential function ( Section 5 ) . Compared with the recent result of ( Leonardos et al. , 2021 ) , we significantly improves the ε dependence from their 1/ε6 . 1.1 RELATED WORK . Learning equlibria in general-sum games The sample ( query ) complexity of learning Nash , CE , and CCE from samples in one-step ( i.e . normal form ) general-sum names with m players and Ai actions per player has been studied extensively in literature ( Hart & Mas-Colell , 2000 ; Hart , 2005 ; Stoltz , 2005 ; Cesa-Bianchi & Lugosi , 2006 ; Blum & Mansour , 2007 ; Fearnley et al. , 2015 ; Babichenko & Barman , 2015 ; Chen et al. , 2015 ; Fearnley & Savani , 2016 ; Goldberg & Roth , 2016 ; Babichenko , 2016 ; Rubinstein , 2016 ; Hart & Nisan , 2018 ) . It is known that learning Nash equilibrium requires exponential in m samples in the worst case Rubinstein ( 2016 ) , whereas CE and CCE admit efficient poly ( m , maxi≤mAi ) -sample complexity algorithms by independent no-regret learning ( Hart & Mas-Colell , 2000 ; Hart , 2005 ; Syrgkanis et al. , 2015 ; Goldberg & Roth , 2016 ; Chen & Peng , 2020 ; Daskalakis et al. , 2021 ) . Our results for learning CE and CCE can be seen as extension of these works into Markov games . We remark that even when the game is fully known , the computational complexity for finding Nash in general-sum games is still PPAD-hard ( Daskalakis , 2013 ) . Markov games Markov games ( Shapley , 1953 ; Littman , 1994 ) is a widely used framework for game playing with sequential decision making , e.g . in multi-agent reinforcement learning . Algorithms with asymptotic convergence have been proposed in the early works of ( Hu & Wellman , 2003 ; Littman , 2001 ; Hansen et al. , 2013 ) . A recent line of work studies the non-asymptotic sample complexity for learning Nash in two-player zero-sum Markov games ( Bai & Jin , 2020 ; Xie et al. , 2020 ; Bai et al. , 2020 ; Zhang et al. , 2020 ; Liu et al. , 2021 ; Chen et al. , 2021 ; Jin et al. , 2021 ; Huang et al. , 2021 ) and learning various equilibria in general-sum Markov games ( Liu et al. , 2021 ; Bai et al. , 2021 ) , building on techniques for learning single-agent Markov Decision Processes sample-efficiently ( Azar et al. , 2017 ; Jin et al. , 2018 ) . Learning the Nash equilibrium in general-sum Markov games are much harder than that in zero-sum Markov games . Liu et al . ( 2021 ) present the first line of results for learning Nash , CE , and CCE in general-sum Markov games ; however their sample complexity scales with ∏ i≤mAi due to the model-based nature of their algorithm . Algorithms for learning CE in extensive-form games has been studied in ( Celli et al. , 2020 ) , though we remark Markov games and extensive-form games are different frameworks and our results do not imply each other . Markov potential games Lastly , a recent line of works considers Markov potential games ( Macua et al. , 2018 ; Leonardos et al. , 2021 ; Zhang et al. , 2021 ) , a subset of general-sum Markov games in which the Nash equilibrium admits more efficient algorithms . Leonardos et al . ( 2021 ) gives a sample-efficient algorithm based on the policy gradient method ( Agarwal et al. , 2021 ) . The special case of Markov cooperative games is studied empirically in e.g . Lowe et al . ( 2017 ) ; Yu et al . ( 2021 ) . For one step potential games , ( Kleinberg et al. , 2009 ; Palaiopanos et al. , 2017 ; Cohen et al. , 2017a ) show the convergence to Nash equilibria of no-regret dynamics . 2 PRELIMINARIES . We present preliminaries for multi-player general-sum Markov games as well as the solution concept of ( approximate ) Nash equilibrium . Alternative solution concepts and other concrete subclasses of Markov games considered in this paper will be defined in the later sections . Markov games A multi-player general sum Markov game ( MG ; ( Shapley , 1953 ; Littman , 1994 ) ) with m players can be described by a tuple MG ( H , S , { Ai } mi=1 , P , { ri } m i=1 ) , where H is the episode length , S is the state space with |S| = S , Ai is the action space for the ith player with |Ai| = Ai . Without loss of generality , we assume Ai = [ Ai ] . We let a : = ( a1 , · · · , am ) denote the vector of joint actions taken by all the players and A = A1 × · · · × Am denote the joint action space . Throughout this paper we assume that S and Ai are finite . The transition probability P = { Ph } h∈ [ H ] is the collection of transition matrices , where Ph ( ·|s , a ) ∈ ∆S denotes the distribution of the next state when actions a are taken at state s at step h. The rewards ri = { rh , i } h∈ [ H ] , i∈ [ m ] is the collection of reward functions for the ith player , where rh , i ( s , a ) ∈ [ 0 , 1 ] gives the deterministic1 reward of ith player if actions a are taken at state s at step h. Without loss of generality , we assume the initial state s1 is deterministic . A key feature of general-sum games is that the rewards ri are in general different for each player i , and the goal of each player is to maximize her own cumulative reward . Policy , value function A product policy is a collection of m policies π : = { πi } i∈ [ m ] where πi is the ( potentially history-dependent ) policy for the i-th player . We first focus on the case of Markov product policies , in which πi = { πh , i : S → ∆Ai } h∈ [ H ] , and πh , i ( ai|s ) is the probability for the ith player to take action ai at state s at step h. For a policy π and i ∈ [ m ] , we use π−i : = { πj } j∈ [ m ] , j 6=i to denote the policy of all but the ith player . The value function V πh , i ( s ) : S → R is defined as the expected cumulative reward for the ith player when policy π is taken starting from state s and step h : V πh , i ( s ) : = Eπ [ H∑ h′=h rh′ , i ( sh′ , ah′ ) ∣∣∣∣∣sh = s ] . ( 1 ) Definition of the Q function and Bellman equations can be found in Appendix B . Best response & Nash equilibrium For any product policy π = { πi } i∈ [ m ] , the best response for the ith player against π−i is defined as any policy π† such that V π† , π−i 1 , i ( s1 ) = supπ′i V π′i , π−i 1 , i ( s1 ) . For any Markov product policy , this best response is guaranteed to exist as the above maximization problem is equilvalent to solving a Markov Decision Process ( MDP ) for the ith player . We will also use the notation V † , π−i1 , i ( s1 ) to denote the above value function V π† , π−i 1 , i ( s1 ) . We say π is a Nash equilibrium ( e.g . Nash ( 1951 ) ; Pérolat et al . ( 2017 ) ) if all players play the best response against other players , i.e. , for all i ∈ [ m ] , V π1 , i ( s1 ) = V † , π−i 1 , i ( s1 ) . 1Our results can be straightforwardly generalized to Markov games with stochastic rewards . Note that in general-sum MGs , there may exist multiple Nash equilibrium policies with different value functions , unlike in two-player zero-sum MGs ( Shapley , 1953 ) . To measure the suboptimality of any policy π , we define the NE-gap as NE-gap ( π ) : = max i∈ [ m ] [ sup µi V µi , π−i 1 , i ( s1 ) − V π 1 , i ( s1 ) ] . For any ε ≥ 0 , we say π is ε-approximate Nash equilibrium if NE-gap ( π ) ≤ ε . General correlated policy A general correlated policy π is a set of H maps π : = { πh : R × ( S ×A× R ) h−1 × S → ∆A } h∈ [ H ] . The first argument of πh is a random variable z ∈ R , and the other arguments contain all the history information and the current state information ( unlike Markov policies in which the policies only depend on the current state information ) . The output of πh is a general distribution of actions in A = A1 × · · · × Am ( unlike product policies in which the action distribution is a product distribution ) . The random variable z is sampled from some underlying distribution which may be shared among all steps h ∈ [ H ] . For any correlated policy π = { πh } h∈ [ H ] and any player i , we can define a marginal policy π−i as a set of H maps π−i : = { πh , −i : R × ( S × A × R ) h−1 × S → ∆A−i } h∈ [ H ] where A−i : = A1 × · · · ×Ai−1 ×Ai+1 × · · · ×Am , and the output of πh , −i is defined as the marginal distribution of the output of πh restricted to the space A−i . For any general correlated policy π , we can define its initial state value function V π1 , i ( s1 ) similar as ( 1 ) . The best response value of the i th player against π is V † , π−i1 , i ( s1 ) = supµi V µi , π−i 1 , i ( s1 ) , where V µi , π−i 1 , i ( s1 ) is the value function of the policy ( µi , π−i ) ( the ith player plays according to general policy µi , and all other players play according to π−i ) , and the supremum is taken over all general policy µi of the ith player . Learning setting Throughout this paper we consider the interactive learning ( i.e . exploration ) setting where algorithms are able to play episodes within the MG and observe the realized transitions and rewards . Our focus is on the PAC sample complexity ( i.e . number of episodes of play ) for any learning algorithm to output an approximate equilibrium .

Summary: The paper studies general sum episodic Markov Games. They provide an algorithm which is effectively V-learning with Follow the regularized leader subroutine (FTRL is hedge algorithm in their case). They prove convergence of their algorithm to $\epsilon$- CCE (coarse correlated equilibria) and $\epsilon$-CE equilibrium policies in $1/\epsilon^2$ with dependence on \max_{i} A_{i} (cardinality of action space of an individual rather than the whole action space). This happens due to ``independent updates" of the agents. Moreover for the case of potential markov games, they manage to show $1/\epsilon^3$ convergence to $\epsilon$-Nash pure Nash policies, improving Leonardos et al paper and solving an open question about convergence to deterministic policies.  However, the agents do not update simultaneously (i.e., the updates are not concurrent!). They also provide lower bounds.