Decentralized Cooperative Multi-Agent Reinforcement Learning with Exploration

1 INTRODUCTION . A variety of real-world sequential decision-making problems have been successfully addressed with reinforcement learning ( RL ) , such as playing the game of Go ( Silver et al. , 2016 ) , Poker ( Brown & Sandholm , 2018 ) , real-time strategy games ( Vinyals et al. , 2019 ) , autonomous driving ( ShalevShwartz et al. , 2016 ) , and robotics ( Kober et al. , 2013 ) . Many of these applications can be cast as a multi-agent reinforcement learning ( MARL ) problem , where multiple agents are involved in an interactive environment . These successful applications have inspired a surging interest in understanding the theoretical aspects of MARL , which , however , is mostly devoted to the competitive setting , e.g. , two-player zero-sum Markov games . This has left many problems in other important MARL settings , especially cooperative MARL , relatively open . Cooperative MARL generally involves a team of agents collaborating on a common task . We roughly categorize cooperative MARL settings into three classes based on the information available to the agents . The first class is ( 1 ) centralized/joint learning ( Boutilier , 1996 ; Claus & Boutilier , 1998 ) , which assumes that there exists a single coordinator who can access the local information of all the agents , and makes decisions/learns policies jointly for all of them . This centralized training ( though possibly decentralized execution ) approach has become a common practice in empirical MARL ( Oliehoek et al. , 2008 ; Foerster et al. , 2016 ; Lowe et al. , 2017 ; Rashid et al. , 2018 ; Son et al. , 2019 ) . Centralized learning essentially reduces the multi-agent problem to a single-agent one , but such a method does not scale well with the number of agents . In fact , one can show that the computation & sample complexity grows exponentially with the number of agents in the worst case . The second class is ( 2 ) learning with communications ( Kar et al. , 2013 ; Zhang et al. , 2018 ; Dubey & Pentland , 2021 ) , which typically assumes that the agents can communicate by sharing some local information with other agents ( sometimes through a central agent ) . This approach gets around the centralized computation bottleneck , but instead suffers from additional communication overhead . It can also be unrealistic in some real-world scenarios where communication may be expensive and/or unreliable , such as in unmanned aerial vehicle ( UAV ) field coverage ( Pham et al. , 2018 ) . Given the aforementioned limitations , we are more interested , in this paper , in a more practical setting : ( 3 ) decentralized learning1 . We enforce a seemingly daunting restriction : Each agent only has access to its own local information ( e.g. , local actions and rewards ) , and can neither communicate with its teammates nor be coordinated by any central controller during learning . In fact , we require that each agent is completely unaware of the underlying structure of the game ( whether it is cooperative or not ) , or even the presence of other agents . Among the three classes mentioned above , decentralized learning requires the weakest assumption on information availability , which makes it suitable for many practical multi-agent learning scenarios ( Fudenberg et al. , 1998 ; Wang & de Silva , 2008 ; Kuyer et al. , 2008 ) . In addition , decentralized learning is generally more scalable , and does not suffer from the exponential sample & computation complexity in the number of agents . Seemingly tempting , one might wonder how strong , if any , theoretical results can be established in this challenging decentralized setting . In the paper , we investigate the theoretical aspects of decentralized cooperative multi-agent RL in the non-asymptotic regime . A natural and fundamental theoretical question is : Does any independent Q-learning style algorithm lead to Nash equilibria in cooperative MARL ? The answer to this question has remained elusive ever since it was raised in Lauer & Riedmiller ( 2000 ) . To answer this question , we focus on arguably the most basic and fundamental cooperative MARL setting — Markov teams ( Lauer & Riedmiller , 2000 ; Boutilier , 1996 ; Wang & Sandholm , 2002 ) – where the cooperating agents share a common team-reward function , and the state transition in the environment is affected by the joint actions of all agents . In fact , even in the basic Markov team setting , decentralized learning can be highly challenging . First , since both rewards and transitions are affected by the other agents , the environment becomes non-stationary from each agent ’ s own perspective when the other agents also update their policies . Hence , an agent needs to explore the unknown environment efficiently while keeping in mind that the information it gathers now will soon become outdated . This rules out many successful single-agent RL solutions , which critically rely on the assumption that the agent is learning in a stationary environment . Second , compared with RL in two-player zero-sum games , an additional challenge in teams is the existence of possibly multiple Nash equilibria ( NE ) with different values . The agents in teams hence have to additionally coordinate on which NE to converge to . Our contributions in the paper are summarized as follows . Contributions . 1 ) As a warm-up , we start with decentralized matrix teams ( Section 3 ) , which constitute a special class of Markov teams with no state transitions . We show that a simple stochastic gradient descent ( SGD ) algorithm can provably find an ε-approximate NE policy using ∝ O ( 1/ε8 ) samples , which can be improved to ∝ Õ ( 1/ε4 ) by using variance reduction techniques . 2 ) For the general problem of decentralized Markov teams ( Section 4 ) , we propose an algorithm named StageBased V-learning with Stochastic Gradient Descent ( V-learning SGD ) , where V-learning ( a name first coined in Bai et al . ( 2020 ) ) is a simple variant of Q-learning . Our algorithm can be executed in a decentralized way as we defined above . 3 ) We show in Section 5 that if all agents in the team run V-learning SGD , they can find an ε-approximate NE in at most Õ ( 1/ε4 ) · poly ( S , H , Amax ) episodes , where S is the number of states , Amax is the size of the largest action space , and H is the length of an episode . Our results rely on a novel investigation of stage-based V-learning to create a stage-wise stationary environment for the agents . 4 ) We further show that our algorithm can approach the best equilibrium , i.e. , the team-optimality , in a specific class of problems named “ smooth teams ” . 5 ) In Section 6 , we provide numerical results that corroborate our theoretical findings . To the best of our knowledge , our work appears to be the first to provide non-asymptotic guarantees for cooperative MARL with exploration , while having the additional advantage of being decentralized . Related Work . A common mathematical framework of multi-agent RL is stochastic games ( Shapley , 1953 ) , which is also referred to as Markov games . Early attempts to learn NE in Markov games include Littman ( 1994 ; 2001 ) ; Hu & Wellman ( 2003 ) ; Hansen et al . ( 2013 ) , but they either assume the transition kernel and rewards are known , or only yield asymptotic guarantees . Recently , various sample efficient methods have been proposed ( Wei et al. , 2017 ; Bai & Jin , 2020 ; Sidford et al. , 2020 ; 1This setting has been studied under various names in the literature , including individual learning ( Leslie & Collins , 2005 ) , decentralized learning ( Arslan & Yüksel , 2016 ) , agnostic learning ( Tian et al. , 2021 ; Wei et al. , 2021 ) , and independent learning ( Claus & Boutilier , 1998 ; Daskalakis et al. , 2020 ) . It also belongs to a more general category of teams/games with decentralized information structure ( Ho , 1980 ; Nayyar et al. , 2013a ; b ) . Xie et al. , 2020 ; Bai et al. , 2020 ; Liu et al. , 2021 ; Zhao et al. , 2021 ; Guo et al. , 2021 ) , mostly for learning in two-player zero-sum Markov games . Several works have investigated zero-sum games in a decentralized setting as we consider here ( Daskalakis et al. , 2020 ; Tian et al. , 2021 ; Wei et al. , 2021 ; Sayin et al. , 2021 ) , but these results do not carry over in any way to the decentralized cooperative/team setting . We refer the reader to Appendix A for a more detailed discussion on these related works . Another line of research has considered RL in teams . Without enforcing a decentralized environment , Boutilier ( 1996 ) has proposed to coordinate the agents by letting them take actions in a lexicographic order . Wang & Sandholm ( 2002 ) have studied optimal adaptive learning that converges to the optimal NE in Markov teams . These two methods critically rely on communications among the agents ( beforehand ) and/or observing the teammates ’ actions . In contrast , the distributed Q-learning algorithm in Lauer & Riedmiller ( 2000 ) is decentralized and coordination-free , which , however , only works for deterministic tasks , and has no non-asymptotic convergence guarantees . In fact , developing provable decentralized RL for cooperation in general stochastic environments is considered an important open problem in Lauer & Riedmiller ( 2000 ) , which we address in this paper . Arslan & Yüksel ( 2016 ) and Yongacoglu et al . ( 2019 ) have shown that decentralized Q-learning style algorithms can converge to NE in weakly acyclic games , which cover Markov teams as an important special case . Their decentralized settings are most similar to ours in that each agent is oblivious to the presence of the others . However , both of them require a coordinated exploration phase , and only yield asymptotic guarantees . In fact , it is not clear if their methods will converge in polynomial time . We significantly improve their results in this respect , by explicitly showing a polynomial sample complexity upper bound . Two contemporaneous works ( Zhang et al. , 2021b ; Leonardos et al. , 2021 ) have studied gradient play in Markov potential games,2 but they do not consider the aspect of exploring the unknown environment , one of the key issues in RL , and/or have to assume perfect knowledge of the environment . A concurrent work ( Chang et al. , 2021 ) has studied cooperative multi-player multi-armed bandits with information asymmetry , which parallels our matrix team setting . Nevertheless , ( Chang et al. , 2021 ) requires stronger conditions than our decentralized setting as their algorithm relies on playing a predetermined sequence of actions . 2 PRELIMINARIES . An N -player episodic Markov team is defined by a tuple ( N , H , S , { Ai } Ni=1 , r , P ) , where ( 1 ) N = { 1 , 2 , . . . , N } is the set of agents ; ( 2 ) H ∈ N+ is the number of time steps in each episode ; ( 3 ) S is the finite state space ; ( 4 ) Ai is the finite action space for agent i ∈ N ; ( 5 ) r : [ H ] × S ×A → [ 0 , 1 ] is the team reward function for all agents , where A = ×Ni=1Ai is the joint action ( or action profile ) space ; and ( 6 ) P : [ H ] × S ×A → ∆ ( S ) is the transition kernel . We remark that both the reward function and the state transition function depend on the joint actions of all the agents . The agents interact in an unknown environment for K episodes . At each time step h ∈ [ H ] , the agents observe the state sh ∈ S , and take actions aih ∈ Ai , i ∈ N simultaneously . All agents receive the same reward rh ( sh , ah ) , where ah = ( a1h , . . . , a N h ) , and the environment transitions to the next state sh+1 ∼ Ph ( ·|sh , ah ) . We assume for simplicity that the initial state s1 of each episode is fixed . Note that the state transition here is general and not restricted to be deterministic , in contrast to Lauer & Riedmiller ( 2000 ) . This makes decentralized learning considerably more challenging , as the agents can not implicitly coordinate by enumerating/rehearsing all possible states . We focus on the decentralized setting , where each agent only observes the states , rewards , and its own actions , but not the actions of the other agents . We require that each agent is completely oblivious of the existence of the others , and is also not allowed to communicate . This decentralized information structure requires each agent to learn to make decisions based on only its local information . For each agent i , a ( Markov ) policy πi : [ H ] × S → ∆ ( Ai ) is a mapping from the time index and state space to a distribution over its own action space . Each agent seeks to find a policy that maximizes its own reward . A joint policy ( or policy profile ) π = ( π1 , . . . , πN ) induces a probability measure over the sequence of states and joint actions . For convenience , we use the superscript −i to denote the set of agents excluding agent i , i.e. , N\ { i } . For example , we can rewrite π = ( πi , π−i ) 2All our results in this paper immediately generalize to Markov potential games , which include Markov teams as a special case . For simplicity of notations , we illustrate our algorithm and results only in the Markov team setting . A short discussion on Markov potential games is included in Appendix B. using this convention . For a policy profile π , and for any h ∈ [ H ] , s ∈ S , and a ∈ A , we define the value function and the state-action value function ( or Q-function ) as follows : V πh ( s ) def = Eπ [ H∑ h′=h rh′ ( sh′ , ah′ ) | sh = s ] , Qπh ( s , a ) def = Eπ [ H∑ h′=h rh′ ( sh′ , ah′ ) | sh = s , ah = a ] . ( 1 ) For agent i , a policy πi ? is a best response to π−i if V π i ? , π−i h ( s ) = supπi V πi , π−i h ( s ) , for any step h ∈ [ H ] and state s ∈ S. A policy profile π = ( πi , π−i ) is a Nash equilibrium if πi is a best response to π−i for all i ∈ N . One can show that NE always exists in finite Markov teams ( Boutilier , 1996 ) . There can be multiple NE in a Markov team with possibly different values . One desirable equilibrium is the team-optimal one that achieves the highest value . Specifically , a policy profile π ? is team-optimal if V π ? h ( s ) = supπ V π h ( s ) , for any step h ∈ [ H ] and state s ∈ S . It follows immediately that a team-optimal policy profile is always a NE , but the converse is not true in general . For general ( non-Markov ) policies , we can still define their values at step h = 1 in a sense similar to Equation ( 1 ) . A best response πi ? with respect to general policies π−i maximizes player i ’ s value at step 1 , i.e. , V π i ? , π−i 1 ( s1 ) = supπi V πi , π−i 1 ( s1 ) . The best response to the general policies of the opponents is not necessarily Markov . We also have an approximate notion of Nash equilibrium : Definition 1 . ( ε-approximate Nash equilibrium ) . For ε > 0 , a general policy profile π = ( πi , π−i ) is an ε-approximate Nash equilibrium if V π i , π−i 1 ( s1 ) ≥ supπi′ V πi ′ , π−i 1 ( s1 ) − ε , for all i ∈ N , where the supremum is taken over all general policies that are not necessarily Markov . For notational convenience , in most parts of the paper we illustrate our algorithms and results for two-player teams , i.e. , N = 2 . It is straightforward to extend our results to the general N -player team as we defined above . With two players , we use A and B to denote the action spaces of players 1 and 2 , respectively . Let S = |S| , A = |A| , B = |B| . Denote by Amax = max { A , B } , which is known to both players . We also rewrite the policies ( π1 , π2 ) as ( µ , ν ) for notational convenience .

Summary: The authors establish a variety of novel theoretical results for Markov team problems. First of all, they present a novel non-asymptotic convergence bound for single-step Markov teams, using a no-regret independent learning variant of stochastic gradient descent (SGD-IX). Secondly, they extend SGD-IX with a stage-based design to arrive at a novel, decentralized MARL algorithm (Stage-based V-learning). For Stage-based V-learning, the authors additionally proceed to establish a novel non-asymptotic convergence bounds that do not depend on the size of the centralised joint policy, but instead just the largest individual action space. The authors also show how their algorithm sidesteps technical difficulties inherent to establishing convergence bounds for Optimistic Q-learning [Jin et al, 2018]. Using a simple matrix team task, as well as a Markov team task as an example, the authors demonstrate empirically that their algorithms can end up selecting favourable Nash equilibria.