Decentralized Cooperative Multi-Agent Reinforcement Learning with Exploration

1 INTRODUCTION . A variety of real-world sequential decision-making problems have been successfully addressed with reinforcement learning ( RL ) , such as playing the game of Go ( Silver et al. , 2016 ) , Poker ( Brown & Sandholm , 2018 ) , real-time strategy games ( Vinyals et al. , 2019 ) , autonomous driving ( ShalevShwartz et al. , 2016 ) , and robotics ( Kober et al. , 2013 ) . Many of these applications can be cast as a multi-agent reinforcement learning ( MARL ) problem , where multiple agents are involved in an interactive environment . These successful applications have inspired a surging interest in understanding the theoretical aspects of MARL , which , however , is mostly devoted to the competitive setting , e.g. , two-player zero-sum Markov games . This has left many problems in other important MARL settings , especially cooperative MARL , relatively open . Cooperative MARL generally involves a team of agents collaborating on a common task . We roughly categorize cooperative MARL settings into three classes based on the information available to the agents . The first class is ( 1 ) centralized/joint learning ( Boutilier , 1996 ; Claus & Boutilier , 1998 ) , which assumes that there exists a single coordinator who can access the local information of all the agents , and makes decisions/learns policies jointly for all of them . This centralized training ( though possibly decentralized execution ) approach has become a common practice in empirical MARL ( Oliehoek et al. , 2008 ; Foerster et al. , 2016 ; Lowe et al. , 2017 ; Rashid et al. , 2018 ; Son et al. , 2019 ) . Centralized learning essentially reduces the multi-agent problem to a single-agent one , but such a method does not scale well with the number of agents . In fact , one can show that the computation & sample complexity grows exponentially with the number of agents in the worst case . The second class is ( 2 ) learning with communications ( Kar et al. , 2013 ; Zhang et al. , 2018 ; Dubey & Pentland , 2021 ) , which typically assumes that the agents can communicate by sharing some local information with other agents ( sometimes through a central agent ) . This approach gets around the centralized computation bottleneck , but instead suffers from additional communication overhead . It can also be unrealistic in some real-world scenarios where communication may be expensive and/or unreliable , such as in unmanned aerial vehicle ( UAV ) field coverage ( Pham et al. , 2018 ) . Given the aforementioned limitations , we are more interested , in this paper , in a more practical setting : ( 3 ) decentralized learning1 . We enforce a seemingly daunting restriction : Each agent only has access to its own local information ( e.g. , local actions and rewards ) , and can neither communicate with its teammates nor be coordinated by any central controller during learning . In fact , we require that each agent is completely unaware of the underlying structure of the game ( whether it is cooperative or not ) , or even the presence of other agents . Among the three classes mentioned above , decentralized learning requires the weakest assumption on information availability , which makes it suitable for many practical multi-agent learning scenarios ( Fudenberg et al. , 1998 ; Wang & de Silva , 2008 ; Kuyer et al. , 2008 ) . In addition , decentralized learning is generally more scalable , and does not suffer from the exponential sample & computation complexity in the number of agents . Seemingly tempting , one might wonder how strong , if any , theoretical results can be established in this challenging decentralized setting . In the paper , we investigate the theoretical aspects of decentralized cooperative multi-agent RL in the non-asymptotic regime . A natural and fundamental theoretical question is : Does any independent Q-learning style algorithm lead to Nash equilibria in cooperative MARL ? The answer to this question has remained elusive ever since it was raised in Lauer & Riedmiller ( 2000 ) . To answer this question , we focus on arguably the most basic and fundamental cooperative MARL setting — Markov teams ( Lauer & Riedmiller , 2000 ; Boutilier , 1996 ; Wang & Sandholm , 2002 ) – where the cooperating agents share a common team-reward function , and the state transition in the environment is affected by the joint actions of all agents . In fact , even in the basic Markov team setting , decentralized learning can be highly challenging . First , since both rewards and transitions are affected by the other agents , the environment becomes non-stationary from each agent ’ s own perspective when the other agents also update their policies . Hence , an agent needs to explore the unknown environment efficiently while keeping in mind that the information it gathers now will soon become outdated . This rules out many successful single-agent RL solutions , which critically rely on the assumption that the agent is learning in a stationary environment . Second , compared with RL in two-player zero-sum games , an additional challenge in teams is the existence of possibly multiple Nash equilibria ( NE ) with different values . The agents in teams hence have to additionally coordinate on which NE to converge to . Our contributions in the paper are summarized as follows . Contributions . 1 ) As a warm-up , we start with decentralized matrix teams ( Section 3 ) , which constitute a special class of Markov teams with no state transitions . We show that a simple stochastic gradient descent ( SGD ) algorithm can provably find an ε-approximate NE policy using ∝ O ( 1/ε8 ) samples , which can be improved to ∝ Õ ( 1/ε4 ) by using variance reduction techniques . 2 ) For the general problem of decentralized Markov teams ( Section 4 ) , we propose an algorithm named StageBased V-learning with Stochastic Gradient Descent ( V-learning SGD ) , where V-learning ( a name first coined in Bai et al . ( 2020 ) ) is a simple variant of Q-learning . Our algorithm can be executed in a decentralized way as we defined above . 3 ) We show in Section 5 that if all agents in the team run V-learning SGD , they can find an ε-approximate NE in at most Õ ( 1/ε4 ) · poly ( S , H , Amax ) episodes , where S is the number of states , Amax is the size of the largest action space , and H is the length of an episode . Our results rely on a novel investigation of stage-based V-learning to create a stage-wise stationary environment for the agents . 4 ) We further show that our algorithm can approach the best equilibrium , i.e. , the team-optimality , in a specific class of problems named “ smooth teams ” . 5 ) In Section 6 , we provide numerical results that corroborate our theoretical findings . To the best of our knowledge , our work appears to be the first to provide non-asymptotic guarantees for cooperative MARL with exploration , while having the additional advantage of being decentralized . Related Work . A common mathematical framework of multi-agent RL is stochastic games ( Shapley , 1953 ) , which is also referred to as Markov games . Early attempts to learn NE in Markov games include Littman ( 1994 ; 2001 ) ; Hu & Wellman ( 2003 ) ; Hansen et al . ( 2013 ) , but they either assume the transition kernel and rewards are known , or only yield asymptotic guarantees . Recently , various sample efficient methods have been proposed ( Wei et al. , 2017 ; Bai & Jin , 2020 ; Sidford et al. , 2020 ; 1This setting has been studied under various names in the literature , including individual learning ( Leslie & Collins , 2005 ) , decentralized learning ( Arslan & Yüksel , 2016 ) , agnostic learning ( Tian et al. , 2021 ; Wei et al. , 2021 ) , and independent learning ( Claus & Boutilier , 1998 ; Daskalakis et al. , 2020 ) . It also belongs to a more general category of teams/games with decentralized information structure ( Ho , 1980 ; Nayyar et al. , 2013a ; b ) . Xie et al. , 2020 ; Bai et al. , 2020 ; Liu et al. , 2021 ; Zhao et al. , 2021 ; Guo et al. , 2021 ) , mostly for learning in two-player zero-sum Markov games . Several works have investigated zero-sum games in a decentralized setting as we consider here ( Daskalakis et al. , 2020 ; Tian et al. , 2021 ; Wei et al. , 2021 ; Sayin et al. , 2021 ) , but these results do not carry over in any way to the decentralized cooperative/team setting . We refer the reader to Appendix A for a more detailed discussion on these related works . Another line of research has considered RL in teams . Without enforcing a decentralized environment , Boutilier ( 1996 ) has proposed to coordinate the agents by letting them take actions in a lexicographic order . Wang & Sandholm ( 2002 ) have studied optimal adaptive learning that converges to the optimal NE in Markov teams . These two methods critically rely on communications among the agents ( beforehand ) and/or observing the teammates ’ actions . In contrast , the distributed Q-learning algorithm in Lauer & Riedmiller ( 2000 ) is decentralized and coordination-free , which , however , only works for deterministic tasks , and has no non-asymptotic convergence guarantees . In fact , developing provable decentralized RL for cooperation in general stochastic environments is considered an important open problem in Lauer & Riedmiller ( 2000 ) , which we address in this paper . Arslan & Yüksel ( 2016 ) and Yongacoglu et al . ( 2019 ) have shown that decentralized Q-learning style algorithms can converge to NE in weakly acyclic games , which cover Markov teams as an important special case . Their decentralized settings are most similar to ours in that each agent is oblivious to the presence of the others . However , both of them require a coordinated exploration phase , and only yield asymptotic guarantees . In fact , it is not clear if their methods will converge in polynomial time . We significantly improve their results in this respect , by explicitly showing a polynomial sample complexity upper bound . Two contemporaneous works ( Zhang et al. , 2021b ; Leonardos et al. , 2021 ) have studied gradient play in Markov potential games,2 but they do not consider the aspect of exploring the unknown environment , one of the key issues in RL , and/or have to assume perfect knowledge of the environment . A concurrent work ( Chang et al. , 2021 ) has studied cooperative multi-player multi-armed bandits with information asymmetry , which parallels our matrix team setting . Nevertheless , ( Chang et al. , 2021 ) requires stronger conditions than our decentralized setting as their algorithm relies on playing a predetermined sequence of actions . 2 PRELIMINARIES . An N -player episodic Markov team is defined by a tuple ( N , H , S , { Ai } Ni=1 , r , P ) , where ( 1 ) N = { 1 , 2 , . . . , N } is the set of agents ; ( 2 ) H ∈ N+ is the number of time steps in each episode ; ( 3 ) S is the finite state space ; ( 4 ) Ai is the finite action space for agent i ∈ N ; ( 5 ) r : [ H ] × S ×A → [ 0 , 1 ] is the team reward function for all agents , where A = ×Ni=1Ai is the joint action ( or action profile ) space ; and ( 6 ) P : [ H ] × S ×A → ∆ ( S ) is the transition kernel . We remark that both the reward function and the state transition function depend on the joint actions of all the agents . The agents interact in an unknown environment for K episodes . At each time step h ∈ [ H ] , the agents observe the state sh ∈ S , and take actions aih ∈ Ai , i ∈ N simultaneously . All agents receive the same reward rh ( sh , ah ) , where ah = ( a1h , . . . , a N h ) , and the environment transitions to the next state sh+1 ∼ Ph ( ·|sh , ah ) . We assume for simplicity that the initial state s1 of each episode is fixed . Note that the state transition here is general and not restricted to be deterministic , in contrast to Lauer & Riedmiller ( 2000 ) . This makes decentralized learning considerably more challenging , as the agents can not implicitly coordinate by enumerating/rehearsing all possible states . We focus on the decentralized setting , where each agent only observes the states , rewards , and its own actions , but not the actions of the other agents . We require that each agent is completely oblivious of the existence of the others , and is also not allowed to communicate . This decentralized information structure requires each agent to learn to make decisions based on only its local information . For each agent i , a ( Markov ) policy πi : [ H ] × S → ∆ ( Ai ) is a mapping from the time index and state space to a distribution over its own action space . Each agent seeks to find a policy that maximizes its own reward . A joint policy ( or policy profile ) π = ( π1 , . . . , πN ) induces a probability measure over the sequence of states and joint actions . For convenience , we use the superscript −i to denote the set of agents excluding agent i , i.e. , N\ { i } . For example , we can rewrite π = ( πi , π−i ) 2All our results in this paper immediately generalize to Markov potential games , which include Markov teams as a special case . For simplicity of notations , we illustrate our algorithm and results only in the Markov team setting . A short discussion on Markov potential games is included in Appendix B. using this convention . For a policy profile π , and for any h ∈ [ H ] , s ∈ S , and a ∈ A , we define the value function and the state-action value function ( or Q-function ) as follows : V πh ( s ) def = Eπ [ H∑ h′=h rh′ ( sh′ , ah′ ) | sh = s ] , Qπh ( s , a ) def = Eπ [ H∑ h′=h rh′ ( sh′ , ah′ ) | sh = s , ah = a ] . ( 1 ) For agent i , a policy πi ? is a best response to π−i if V π i ? , π−i h ( s ) = supπi V πi , π−i h ( s ) , for any step h ∈ [ H ] and state s ∈ S. A policy profile π = ( πi , π−i ) is a Nash equilibrium if πi is a best response to π−i for all i ∈ N . One can show that NE always exists in finite Markov teams ( Boutilier , 1996 ) . There can be multiple NE in a Markov team with possibly different values . One desirable equilibrium is the team-optimal one that achieves the highest value . Specifically , a policy profile π ? is team-optimal if V π ? h ( s ) = supπ V π h ( s ) , for any step h ∈ [ H ] and state s ∈ S . It follows immediately that a team-optimal policy profile is always a NE , but the converse is not true in general . For general ( non-Markov ) policies , we can still define their values at step h = 1 in a sense similar to Equation ( 1 ) . A best response πi ? with respect to general policies π−i maximizes player i ’ s value at step 1 , i.e. , V π i ? , π−i 1 ( s1 ) = supπi V πi , π−i 1 ( s1 ) . The best response to the general policies of the opponents is not necessarily Markov . We also have an approximate notion of Nash equilibrium : Definition 1 . ( ε-approximate Nash equilibrium ) . For ε > 0 , a general policy profile π = ( πi , π−i ) is an ε-approximate Nash equilibrium if V π i , π−i 1 ( s1 ) ≥ supπi′ V πi ′ , π−i 1 ( s1 ) − ε , for all i ∈ N , where the supremum is taken over all general policies that are not necessarily Markov . For notational convenience , in most parts of the paper we illustrate our algorithms and results for two-player teams , i.e. , N = 2 . It is straightforward to extend our results to the general N -player team as we defined above . With two players , we use A and B to denote the action spaces of players 1 and 2 , respectively . Let S = |S| , A = |A| , B = |B| . Denote by Amax = max { A , B } , which is known to both players . We also rewrite the policies ( π1 , π2 ) as ( µ , ν ) for notational convenience .

Summary: Authors tackles the cooperative multi-agent reinforcement learning problem with stochastic dynamics. They introduced V-learning Stochastic Gradient Descent (SGD) with theoretical proofs of convergence to the optimal policy with desired e probability. The resulting sample complexity is O(1/e^4) * Poly (A,S,H), where H is the episode horizon. Authors also empirically tested the algorithm in a toy 2 agent matrix game showing convergence to the optimal policy.