CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning

1 INTRODUCTION . Many real-world scenarios that require cooperation among multiple autonomous agents are multi-goal multi-agent control problems : each agent needs to achieve its own individual goal , but the global optimum where all agents succeed is only attained when agents cooperate to allow the success of other agents . In autonomous driving , multiple vehicles must execute cooperative maneuvers when their individual goal locations and nominal trajectories are in conflict ( e.g. , double lane merges ) ( Cao et al. , 2013 ) . In social dilemmas , mutual cooperation has higher global payoff but agents ’ individual goals may lead to defection out of fear or greed ( Van Lange et al. , 2013 ) . Even settings with a global objective that seem unfactorizable can be formulated as multi-goal problems : in Starcraft II micromanagement , a unit that gathers resources must not accidentally jeopardize a teammate ’ s attempt to scout the opponent base ( Blizzard Entertainment , 2019 ) ; in traffic flow optimization , different intersection controllers may have local throughput goals but must cooperate for high global performance ( Zhang et al. , 2019 ) . While the framework of multi-agent reinforcement learning ( MARL ) ( Littman , 1994 ; Stone and Veloso , 2000 ; Shoham et al. , 2003 ) has been equipped with methods in deep reinforcement learning ( RL ) ( Mnih et al. , 2015 ; Lillicrap et al. , 2016 ) and shown promise on high-dimensional problems with complex agent interactions ( Lowe et al. , 2017 ; Mordatch and Abbeel , 2018 ; Foerster et al. , 2018 ; Lin et al. , 2018 ; Srinivasan et al. , 2018 ) , learning multi-agent cooperation in the multi-goal scenario involves significant open challenges . First , given that exploration is crucial for RL ( Thrun , 1992 ) and even more so in MARL with larger state and joint action spaces , how should agents explore to learn both individual goal attainment and cooperation for others ’ success ? Uniform random exploration is common in deep MARL ( Hernandez-Leal et al. , 2018 ) but can be highly inefficient as the value of cooperative actions may be discoverable only in small regions of state space where cooperation is needed . Furthermore , the conceptual difference between attaining one ’ s own goal and cooperating for others ’ success calls for ∗Correspondence to jiachen.yang @ gatech.edu †Work done at HRI more modularized and targeted approaches . Second , while there are methods for multi-agent credit assignment when all agents share a single goal ( i.e. , a global reward ) ( Chang et al. , 2004 ; Foerster et al. , 2018 ; Nguyen et al. , 2018 ) , and while one could treat the cooperative multi-goal scenario as a problem with a single joint goal , this coarse approach makes it extremely difficult to evaluate the impact of an agent ’ s action on another agent ’ s success . Instead , the multi-goal scenario can benefit from fine-grained credit assignment that leverages available structure in action-goal interactions , such as local interactions where only few agents affect another agent ’ s goal attainment at any time . Given these open challenges , our paper focuses on the cooperative multi-goal multi-agent setting where each agent is assigned a goal1 and must learn to cooperate with other agents with possibly different goals . To tackle the problems of efficient exploration and credit assignment in this complex problem setting , we develop CM3 , a novel general framework involving three synergistic components : 1 . We approach the difficulty of multi-agent exploration from a novel curriculum learning perspective , by first training an actor-critic pair to achieve different goals in an induced single-agent setting ( Stage 1 ) , then using them to initialize all agents in the multi-agent environment ( Stage 2 ) . The key insight is that agents who can already act toward individual objectives are better prepared for discovery of cooperative solutions with additional exploration once other agents are introduced . In contrast to hierarchical learning where sub-goals are selected sequentially in time ( Sutton et al. , 1999 ) , all agents act toward their goals simultaneously in Stage 2 of our curriculum . 2 . Observing that a wide array of complex MARL problems permit a decomposition of agents ’ observations and state vectors into components of self , others , and non-agent specific environment information ( Hernandez-Leal et al. , 2018 ) , we employ function augmentation to bridge Stages 1-2 : we reduce the number of trainable parameters of the actor-critic in Stage 1 by limiting their input space to the part that is sufficient for single-agent training , then augment the architecture in Stage 2 with additional inputs and trainable parameters for learning in the multi-agent environment . 3 . We propose a credit function , which is an action-value function that specifically evaluates actiongoal pairs , for localized credit assignment in multi-goal MARL . We use it to derive a multi-goal multi-agent policy gradient for Stage 2 . In synergy with the curriculum , the credit function is constructed via function augmentation from the critic in Stage 1 . We evaluate our method on challenging multi-goal multi-agent environments with high-dimensional state spaces : cooperative navigation with difficult formations , double lane merges in the SUMO simulator ( Lopez et al. , 2018 ) , and strategic teamwork in a Checkers game . CM3 solved all domains significantly faster than IAC and COMA ( Tan , 1993 ; Foerster et al. , 2018 ) , and solved four out of five environments significantly faster than QMIX ( Rashid et al. , 2018 ) . Exhaustive ablation experiments show that the combination of all three components is crucial for CM3 ’ s overall high performance . 2 RELATED WORK . While early theoretical work analyzed Markov games in discrete state and action spaces ( Tan , 1993 ; Littman , 1994 ; Hu and Wellman , 2003 ) , recent literature have leveraged techniques from deep RL to develop general algorithms for high dimensional environments with complex agent interactions ( Tampuu et al. , 2017 ; Mordatch and Abbeel , 2018 ; Lowe et al. , 2017 ) , which pose difficulty for traditional methods that do not generalize by learning interactions ( Bhattacharya et al. , 2010 ) . Cooperative multi-agent learning is important since many real-world problems can be formulated as distributed systems in which decentralized agents must coordinate to achieve shared objectives ( Panait and Luke , 2005 ) . The multi-agent credit assignment problem arises when agents share a global reward ( Chang et al. , 2004 ) . While credit assignment be resolved when independent individual rewards are available ( Singh et al. , 2019 ) , this may not be suitable for the fully cooperative setting : Austerweil et al . ( 2016 ) showed that agents whose rewards depend on the success of other agents can cooperate better than agents who optimize for their own success . In the special case when all agents have a single goal and share a global reward , COMA ( Foerster et al. , 2018 ) uses a counterfactual baseline , while Nguyen et al . ( 2018 ) employs count-based variance reduction limited to discrete-state 1Goal discovery and assignment are challenges for MARL . However , many practical multi-agent problems have clear goal assignments , such as in autonomous driving and soccer . Our work is specific to known goal assignment and is complementary to methods such as ( Carion et al. , 2019 ) for the unknown case . environments . However , their centralized critic does not evaluate the specific impact of an agent ’ s action on another ’ s success in the general multi-goal setting . When a global objective is the sum of agents ’ individual objectives , value-decomposition methods optimize a centralized Q-function while preserving scalable decentralized execution ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ) , but do not address credit assignment . While MADDPG ( Lowe et al. , 2017 ) and M3DDPG ( Li et al. , 2019 ) apply to agents with different rewards , they do not address multi-goal cooperation as they do not distinguish between cooperation and competition , despite the fundamental difference . Multi-goal MARL was considered in Zhang et al . ( 2018 ) , who analyzed convergence in a special networked setting restricted to fully-decentralized training , while we conduct centralized training with decentralized execution ( Oliehoek et al. , 2008 ) . In contrast to multi-task MARL , which aims for generalization among non-simultaneous tasks ( Omidshafiei et al. , 2017 ) , and in contrast to hierarchical methods that sequentially select subtasks ( Vezhnevets et al. , 2017 ; Shu and Tian , 2019 ) , our decentralized agents must cooperate concurrently to attain all goals . Methods for optimizing high-level agent-task assignment policies in a hierarchical framework ( Carion et al. , 2019 ) are complementary to our work , as we focus on learning low-level cooperation after goals are assigned . Prior application of curriculum learning ( Bengio et al. , 2009 ) to MARL include a single cooperative task defined by the number of agents ( Gupta et al. , 2017 ) and the probability of agent appearance ( Sukhbaatar et al. , 2016 ) , without explicit individual goals . Rusu et al . ( 2016 ) instantiate new neural network columns for task transfer in single-agent RL . Techniques in transfer learning ( Pan and Yang , 2010 ) are complementary to our novel curriculum approach to MARL . 3 PRELIMINARIES . In multi-goal MARL , each agent should achieve a goal drawn from a finite set , cooperate with other agents for collective success , and act independently with limited local observations . We formalize the problem as an episodic multi-goal Markov game , review an actor-critic approach to centralized training of decentralized policies , and summarize counterfactual-based multi-agent credit assignment . Multi-goal Markov games . A multi-goal Markov game is a tuple 〈S , { On } , { An } , P , R , G , N , γ〉 with N agents labeled by n ∈ [ N ] . In each episode , each agent n has one fixed goal gn ∈ G that is known only to itself . At time t and global state st ∈ S , each agent n receives an observation ont : = o n ( st ) ∈ On and chooses an action ant ∈ An . The environment moves to st+1 due to joint action at : = { a1t , . . . , aNt } , according to transition probability P ( st+1|st , at ) . Each agent receives a reward Rnt : = R ( st , at , g n ) , and the learning task is to find stochastic decentralized policies πn : On × G ×An → [ 0 , 1 ] , conditioned only on local observations and goals , to maximize J ( π ) : = Eπ [ ∑∞ t=0 γ t ∑N n=1R ( st , at , g n ) ] , where γ ∈ ( 0 , 1 ) and joint policy π factorizes as π ( a|s , g ) : = ∏N n=1 π n ( an|on , gn ) due to decentralization . Let a−n and g−n denote all agents ’ actions and goals , respectively , except that of agent n. Let boldface a and g denote the joint action and joint goals , respectively . For brevity , let π ( an ) : = πn ( an|on , gn ) . This model covers a diverse set of cooperation problems in the literature ( Hernandez-Leal et al. , 2018 ) , without constraining how the attainability of a goal depends on other agents : at a traffic intersection , each vehicle can easily reach its target location if not for the presence of other vehicles ; in contrast , agents in a strategic game may not be able to maximize their rewards in the absence of cooperators ( Sunehag et al. , 2018 ) . Centralized learning of decentralized policies . A centralized critic that receives full state-action information can speed up training of decentralized actors that receive only local information ( Lowe et al. , 2017 ; Foerster et al. , 2018 ) . Directly extending the single-goal case , for each n ∈ [ 1 .. N ] in a multigoal Markov game , critics are represented by the value function V πn ( s ) : = Eπ [ ∑∞ t=0 γ tRnt ∣∣ s0 = s ] and the action-value function Qπn ( s , a ) : = Eπ [ ∑∞ t=0 γ tRnt ∣∣ s0 = s , a0 = a ] , which evaluate the joint policy π against the reward Rn for each goal gn . Multi-agent credit assignment . In MARL with a single team objective , COMA addresses credit assignment by using a counterfactual baseline in an advantage function An ( s , a ) : = Qπ ( s , a ) −∑ ân π n ( ân|on ) Qπ ( s , ( ân , a−n ) ) ( Foerster et al. , 2018 , Lemma 1 ) , which evaluates the contribution of a chosen action an versus the average of all possible counterfactuals ân , keeping a−n fixed . The analysis in Wu et al . ( 2018 ) for a formally equivalent action-dependent baseline in RL suggests that COMA is a low-variance estimator for single-goal MARL . We derive its variance in Appendix C.1 . However , COMA is unsuitable for credit assignment in multi-goal MARL , as it would treat the collection of goals g as a global goal and only learn from total reward , making it extremely difficult to disentangle each agent ’ s impact on other agents ’ goal attainment . Furthermore , a global Q-function does not explicitly capture structure in agents ’ interactions , such as local interactions involving a limited number of agents . We substantiate these arguments by experimental results in Section 6 .

Summary: The paper presents a new method CM3 for multi-goal multi-agent settings where agent must care about the rewards of others as well as theirs. The method restructures the problem into two-stage curriculum where individual agents solve the problem in the individual setting of the environment in the first stage and then more agents are introduced in the multi-agent setting in second stage. The method also uses function augmentation to only learn parameters which are necessary for single agent in first stage and then more parameters are introduced in the second stage. The results are shown on three environments which show that CM3 outperforms the baselines.