Multi-Agent Collaboration via Reward Attribution Decomposition

1 INTRODUCTION . In recent years , multi-agent deep reinforcement learning ( MARL ) has drawn increasing interest from the research community . MARL algorithms have shown super-human level performance in various games like Dota 2 ( Berner et al. , 2019 ) , Quake 3 Arena ( Jaderberg et al. , 2019 ) , and StarCraft ( Samvelyan et al. , 2019 ) . However , the algorithms ( Schulman et al. , 2017 ; Mnih et al. , 2013 ) are far less sample efficient than humans . For example , in Hide and Seek ( Baker et al. , 2019 ) , it takes agents 2.69− 8.62 million episodes to learn a simple strategy of door blocking , while it only takes human several rounds to learn this behavior . One of the key reasons for the slow learning is that the number of joint states grows exponentially with the number of agents . Moreover , many real-world situations require agents to adapt to new configurations of teams . This can be modeled as ad hoc multi-agent reinforcement learning ( Stone et al. , 2010 ) ( Ad-hoc MARL ) settings , in which agents must adapt to different team sizes and configurations at test time . In contrast to the MARL setting where agents can learn a fixed and team-dependent policy , in the Ad-hoc MARL setting agents must assess and adapt to the capabilities of others to behave optimally . Existing work in ad hoc team play either require sophisticated online learning at test time ( Barrett et al. , 2011 ) or prior knowledge about teammate behaviors ( Barrett and Stone , 2015 ) . As a result , they do not generalize to complex real-world scenarios . Most existing works either focus on improving generalization towards different opponent strategies ( Lanctot et al. , 2017 ; Hu et al. , 2020 ) or simple ad-hoc setting like varying number of test-time teammates ( Schwab et al. , 2018 ; Long et al. , 2020 ) . We consider a more general setting where test-time teammates may have different capabilities . The need to reason about different team configurations in the Ad-hoc MARL results in an additional exponential increase ( Stone et al. , 2010 ) in representational complexity comparing to the MARL setting . In the situation of collaboration , one way to address the complexity of the ad hoc team play setting is to explicitly model and address how agents collaborate . In this paper , one key observation is that when collaborating with different agents , an agent changes their behavior because she realizes that the team could function better if she focuses on some of the rewards while leaving other rewards to other teammates . Inspired by this principle , we formulate multi-agent collaboration as a joint optimization over an implicit reward assignment among agents . Because the rewards are assigned differently for different team configurations , the behavior of an agent changes and adaptation follows . While solving this optimization directly requires centralization at test time , we make an interesting theoretical finding that each agent has a decentralized policy that is ( 1 ) approximately optimal for the joint optimization , and ( 2 ) only depends on the local configuration of other agents . This enables us to learn a direct mapping from states of nearby agents ( or “ observation ” of agent i ) to its Q-function using deep neural network . Furthermore , this finding also suggests that the Q-function of agent i should be decomposed into two terms : Qalonei that only depends on agent i ’ s own state si , andQ collab i that depends on nearby agents but vanishes if no other agents nearby . To enforce this semantics , we regularize Qcollabi ( si , · ) = 0 in training via a novel Multi-Agent Reward Attribution ( MARA ) loss . The resulting algorithm , Collaborative Q-learning ( CollaQ ) , achieves a 40 % improvement in win rates over state-of-the-art techniques for the StarCraft multi-agent challenge . We show that ( 1 ) the MARA Loss is critical for strong performance and ( 2 ) both Qalone and Qcollab are interpretable via visualization . Furthermore , CollaQ agents can achieve ad hoc team play without retraining or fine-tuning . We propose three tasks to evaluate ad hoc team play performance : at test time , ( a ) assign a new VIP unit whose survival matters , ( b ) swap different units in and out , and ( c ) add or remove units . Results show that CollaQ outperforms baselines by an average of 30 % in all these settings . Related Works . The most straightforward way to train such a MARL task is to learn individual agent ’ s value function Qi independently ( IQL ) ( Tan , 1993 ) . However , the environment becomes non-stationary from the perspective of an individual agent thus this performs poorly in practice . Recent works , e.g. , VDN ( Sunehag et al. , 2017 ) , QMIX ( Rashid et al. , 2018 ) , QTRAN ( Son et al. , 2019 ) , adopt centralized training with decentralized execution to solve this problem . They propose to write the joint value function as Qπ ( s , a ) = φ ( s , Q1 ( o1 , a1 ) , ... , QK ( oK , aK ) ) but the formulation of φ differs in each method . These methods successfully utilize the centralized training technique to alleviate the non-stationary issue . However , none of the above methods generalize well to ad-hoc team play since learned Qi functions highly depend on the existence of other agents . 2 COLLABORATIVE MULTI-AGENT REWARD ASSIGNMENT . Basic Setting . A multi-agent extension of Markov Decision Process called collaborative partially observable Markov Games ( Littman , 1994 ) , is defined by a set of states S describing the possible configurations of allK agents , a set of possible actionsA1 , . . . , AK , and a set of possible observations O1 , . . . , OK . At every step , each agent i chooses its action ai by a stochastic policy πi : Oi ×Ai → [ 0 , 1 ] . The joint action a produces the next state by a transition function P : S×A1×· · ·×AK → S. All agents share the same reward r : S × A1 × · · · × AK → R and with a joint value function Qπ = Est+1 : ∞ , at+1 : ∞ [ Rt|st , at ] where Rt = ∑∞ j=0 γ jrt+j is the discounted return . In Sec . 2.1 , we first model multi-agent collaboration as a joint optimization on reward assignment : instead of acting based on the joint state s , each agent i is acting independently on its own state si , following its own optimal value Vi , which is a function of the perceived reward assignment ri . While the optimal perceived reward assignment r∗i ( s ) depends on the joint state of all agents and requires centralization , in Sec . 2.2 , we prove that there exists an approximate optimal solution r̂i that only depends on the local observation slocali of agent i , and thus enabling decentralized execution . Lastly in Sec . 2.3 , we distill the theoretical insights into a practical algorithm CollaQ , by directly learning the compositional mapping slocali 7→ r̂i 7→ Vi in an end-to-end fashion , while keeping the decomposition structure of self state and local observations . 2.1 BASIC ASSUMPTION . A naive modeling of multi-agent collaboration is to estimate a joint value function Vjoint : = Vjoint ( s1 , s2 , . . . , sK ) , and find the best action for agent i to maximize Vjoint according to the current joint state s = ( s1 , s2 , . . . , sK ) . However , it has three fundamental drawbacks : ( 1 ) Vjoint generally requires exponential number of samples to learn ; ( 2 ) in order to evaluate this function , a full observation of the states of all agents is required , which disallows decentralized execution , one key preference of multi-agent RL ; and ( 3 ) for any environment/team changes ( e.g. , teaming with different agents ) , Vjoint needs to be relearned for all agents and renders ad hoc team play impossible . Our CollaQ addresses the three issues with a novel theoretical framework that decouples the interactions between agents . Instead of using Vjoint that bundles all the agent interactions together , we consider the underlying mechanism how they interact : in a fully collaborative setting , the reason why agent i takes actions towards a state , is not only because that state is rewarding to agent i , but also because it is more rewarding to agent i than other agents in the team , from agent i ’ s point of view . This is the concept of perceived reward of agent i . Then each agent acts independently following its own value function Vi , which is the optimal solution to the Bellman equation conditioned on the assigned perceived reward , and is a function of it . This naturally leads to collaboration . We build a mathematical framework to model such behaviors . Specifically , we make the following assumption on the behavior of each agent : Assumption 1 . Each agent i has a perceived reward assignment ri ∈ R|Si||Ai|+ that may depend on the joint state s = ( s1 , . . . , sK ) . Agent i acts according to its own state si and individual optimal value Vi = Vi ( si ; ri ) ( and associated Qi ( si , ai ; ri ) ) , which is a function of ri . Note that the perceived reward assignment ri ∈ R|Si||Ai|+ is a non-negative vector containing the assignment of scalar reward at each state-action pair ( hence its length is |Si||Ai| ) . We might also equivalently write it as a function : ri ( x , a ) : Si × Ai 7→ R , where x ∈ Si and a ∈ Ai . Here x is a dummy variable that runs through all states of agent i , while si refers to its current state . Given the perceived rewards assignment { ri } , the values and actions of agents become decoupled . Due to the fully collaborative nature , a natural choice of { ri } is the optimal solution of the following objective J ( r1 , r2 , . . . , rK ) . Here re is the external rewards of the environment , wi ≥ 0 is the preference of agent i and is the Hadamard ( element-wise ) product : J ( r1 , . . . , rK ) : = K∑ i=1 Vi ( si ; ri ) s.t . K∑ i=1 wi ri ≤ re ( 1 ) Note that the constraint ensures that the objective has bounded solution . Without this constraints , we could easily take each perceived reward ri to +∞ , since each value function Vi ( si ; ri ) monotonously increases with respect to ri . Intuitively , Eqn . 1 means that we “ assign ” the external rewards re optimally to K agents as perceived rewards , so that their overall values are the highest . In the case of sparse reward , most of the state-action pair ( x , a ) , re ( x , a ) = 0 . By Eqn . 1 , for all agent i , their perceived reward ri ( x , a ) = 0 . Then we only focus on nonzero entries for each ri . Define M to be the number of state-action pairs with positive reward : M = ∑ ai∈Ai 1 { ri ( x , ai ) > 0 } . Discarding zero-entries , we could regard all ri as M -dimensional vector . Finally , we define the reward matrix R = [ r1 , . . . , rK ] ∈ RM×K . Clarification on Rewards . There are two kinds of rewards here : external reward re and perceived reward for each agent ri . re is defined to be the environmental reward shared by all the agents : re : S×A1×· · ·×Ak → R. Given this external reward , depending on a specific reward assignment , each agent can receive a perceived reward ri that drives its behavior . If the reward assignment is properly defined/optimized , then all the agents can act based on the perceived reward to jointly optimize ( maximize ) the shared external reward . 2.2 LEARN TO PREDICT THE OPTIMAL ASSIGNED REWARD r∗i ( s ) The optimal reward assignments R∗ of Eq . 1 , as well as its i-th assignment r∗i , is a function of the joint states s = { s1 , s2 , . . . , sK } . Once the optimization is done , each agent can get the best action a∗i = argmaxai Qi ( si , ai ; r ∗ i ( s ) ) independently from the reconstructed Q function . The formulation Vi ( si ; ri ) avoids learning the value function of statistically infeasible joint states Vi ( s ) . Since an agent acts solely based on ri , ad hoc team play becomes possible if the correct ri is assigned . However , there are still issues . First , since each Vi is a convex function regarding ri , maximizing Eqn . 1 is a summation of convex functions under linear constraints optimization , and is hard computationally . Furthermore , to obtain actions for each agent , we need to solve Eqn . 1 at every step , which still requires centralization at test time , preventing us from decentralized execution . To overcome optimization complexity and enable decentralized execution , we consider learning a direct mapping from the joint state s to optimally assigned reward r∗i ( s ) . However , since s is a joint state , learning such a mapping can be as hard as modeling Vi ( s ) . Fortunately , Vi ( si ; ri ( s ) ) is not an arbitrary function , but the optimal value function that satisfies Bellman equation . Due to the speciality of Vi , we could find an approximate assignment r̂i for each agent i , so that r̂i only depends on a local observation slocali of the states of nearby other agents observed by agent i : r̂i ( s ) = r̂i ( slocali ) . At the same time , these approximate reward assignments { r̂i } achieve approximate optimal for the joint optimization ( Eqn . 1 ) with bounded error : Theorem 1 . For all i ∈ { 1 , . . . , K } , all si ∈ Si , there exists a reward assignment r̂i that ( 1 ) only depends on slocali and ( 2 ) r̂i is the i-th column of a feasible global reward assignment R̂ such that J ( R̂ ) ≥ J ( R∗ ) − ( γC + γD ) RmaxMK , ( 2 ) where C and D are constants related to distances between agents/rewards ( details in Appendix ) . Since r̂i only depends on the local observation of agent i ( i.e. , agent ’ s own state si as well as the states of nearby agents ) , it enables decentralized execution : for each agent i , the local observation is sufficient for an agent to act near optimally . Limitation . One limitation of Theorem 1 is that the optimality gap of r̂i heavily depends on the size of slocali . If the local observation of agent i covers more agents , then the gap is smaller but the cost to learn such a mapping is higher , since the mapping has more input states and becomes higher-dimensional . In practice , we found that the observation oi of agent i covers slocali works sufficiently well , as shown in the experiments ( Sec . 4 ) .

Summary: To address the ad hoc team play, the authors propose a residual term of Q function, which additionally considers the states of nearby agents. A novel MARA loss is introduced to the residual term as a regularization to achieve the reward assignment implicitly. The proposed CollaQ could be easily built on QMIX and trained end-to-end. CollaQ outperforms other baselines on various tasks with the ad hoc team play setting.