Cooperative Multi-Agent Reinforcement Learning with Sequential Credit Assignment

Centralized training with decentralized execution is a standard paradigm for coop-1 erative multi-agent reinforcement learning ( MARL ) , with credit assignment being2 a major challenge . In this paper , we propose a cooperative MARL method with3 sequential credit assignment ( SeCA ) that deduces each agent ’ s contribution to the4 team ’ s success one by one to learn better cooperation . We first present a sequential5 MARL framework , under which we introduce a new counterfactual advantage to6 evaluate each agent based on its preceding agents ’ actions in a specific sequence.7 As this credit assignment sequence tremendously impacts the performance , we8 further present a sequence adjustment algorithm utilizing integrated gradients . It9 dynamically modifies the sequence among agents according to their contribution10 to the team . SeCA employs a network which either estimates the Q value for11 training the centralized critic or deduces the proposed advantage of each agent for12 decentralized policy learning . Our method is evaluated on a challenging set of13 StarCraft II micromanagement tasks and achieves state-of-the-art performance.14 1 Introduction15 Cooperative multi-agent reinforcement learning ( MARL ) is a helpful tool in numerous applications16 such as robot swarm control [ 9 ] , autonomous vehicle coordination [ 3 ] , network routing [ 36 ] , and17 productivity optimization [ 37 ] . This kind of problem where agents learn coordinated policies to18 optimize the global reward has been extensively studied in recent years [ 7 , 19 , 18 , 38 , 8 ] .19 One natural way of addressing the cooperative MARL problem is the centralized approach , which20 treats the team as a single actor with a joint action space . Although we can trivially apply single-agent21 reinforcement learning algorithms to such settings , it usually does not scale well because the size of22 the joint action space grows exponentially with the number of agents . Besides , it is not applicable23 in real-world settings due to the inherent constraints on agent observability and communication.24 An alternative approach is to learn decentralized policies by independently training agents based25 on their local observations , but simultaneous exploration often brings non-stationarity that causes26 unstable learning and difficulties in convergence . As a result , the majority of work on MARL27 follows the centralized training with decentralized execution ( CTDE ) paradigm [ 17 , 10 , 22 , 6 ] , where28 decentralized policies can access extra state information during training.29 A crucial challenge of the CTDE paradigm in cooperative settings is to correctly deduce each agent ’ s30 contribution to the team ’ s success , also known as the multi-agent credit assignment problem [ 4 ] .31 Existing methods can be classified as implicit and explicit credit assignment [ 39 ] . Previous implicit32 methods often deduce all agents ’ contributions by representing the global state-action value as an33 aggregation of each agent ’ s state-action value [ 26 , 22 , 12 , 24 , 21 , 29 ] and assigning the shared rewards34 to each agent according to the joint action at one time . In this way , these methods avoid the complex35 interaction analysis and instead fit these cooperation relationships by neural networks . However,36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . implicit methods often face limitations in expressiveness , and their extensions to continuous action37 spaces may require additional strategies [ 39 ] .38 On the other hand , recognized explicit approaches calculate difference rewards [ 34 ] against a certain39 reward baseline [ 28 , 20 , 6 ] . However , in cooperative MARL , evaluating any agent ’ s action requires40 considering the actions of all agents , so it is often difficult to determine the impact a particular agent ’ s41 behavior has on the team when we have not assessed other agents ’ actions . In other words , we can42 not say that a single agent ’ s action is bad if the team receives a small reward because the shared43 reward is not decided only by this agent ’ s behavior . Maybe its action is actually good in that state.44 This paper presents a sequential credit assignment SeCA to evaluate individual agent actions explicitly45 and sequentially . Our motivation is to address the drawbacks of implicit methods that neglect the46 cooperation between agents or simply leave it to neural networks and further improve explicit credit47 assignment . In summary , we face two main challenges to learn a better explicit credit assignment : ( 1 ) 48 how to alleviate the problem that it is hard to accurately deduce the contribution of one agent without49 previously assessing all the others ’ action , and ( 2 ) how to evaluate agents better in an explicit way.50 To deal with ( 1 ) , we introduce a sequential MARL framework . As mentioned above , without assessing51 the behaviors of other agents , we would never be able to evaluate a given agent ’ s action accurately.52 However , we point out in this paper that some agents are less affected by such influences than others,53 and we can first assign credit to them . For instance , evaluating a staff ’ s action needs to take the54 CEO ’ s command or action into consideration , while the former has little importance in assessing the55 CEO . Thus , we could evaluate the CEO first without considering the staff ’ s behavior and then analyze56 the staff based on the CEO ’ s action . We fully consider the action coordination between agents and57 explicitly deduce contribution to them one by one according to a particular order , so as to make up58 for the disadvantage of implicit methods that the cooperation is only inexplicably fitted by neural59 networks . Intuitively , the order significantly impacts the overall performance , so we further propose60 an algorithm to adjust the sequence dynamically through integrated gradients [ 25 ] .61 As for ( 2 ) , we compute an advantage function for each agent to attribute agent contributions explicitly.62 COMA [ 6 ] is a representative method that computes a baseline for each agent to reason about63 counterfactuals in which only one agent a ’ s action changes , so its evaluation of a ’ s action is based on64 the joint action u−a of other agents . In other words , the policy gradient of COMA only encourages65 agent a to learn in the direction that benefits the team while other agents are acting u−a , but the66 others ’ actions are not necessarily u−a when executing . Unlike COMA , we focus more on the action67 coordination among agents and propose a new advantage under the proposed sequential framework.68 We summarize the contributions of this paper as follows : ( 1 ) We propose a sequential MARL69 framework in Section 3.2 ; ( 2 ) Under this framework , we introduce a sequential advantage function70 for each agent to guide their learning explicitly in Section 3.3 . We further prove that the sequential71 credit assignment we proposed achieves additive advantage-decomposition . ( 3 ) We present a sequence72 adjustment algorithm based on integrated gradients to modify the credit assignment order dynamically73 in Section 3.4 . This algorithm alleviates the impact caused by the sequence ’ s randomness and helps74 achieve competitive performance on a challenging set of StarCraft II micromanagement tasks [ 23 ] .75 2 Related Work76 Explicit credit assignment gives valuable insights into agent actions ’ contributions to the shared77 team reward and substantially promotes policy optimization . The representative method COMA [ 6 ] 78 utilizes a counterfactual baseline that marginalizes out a single agent ’ s action while keeping the other79 agents ’ actions fixed to calculate the advantage function . However , the advantage evaluates a single80 agent ’ s action based on the other agents ’ current behaviors and ignores different action combinations.81 SQDDPG [ 30 ] distributes the global reward reflecting each agent ’ s contribution through Shapley82 Value . Although SQDDPG provides a theoretically justified framework , its assumption on the83 observability and convex game makes it impractical and performs poorly in complex environments.84 Implicit methods are a more common way when addressing the credit assignment challenge . Among85 them , LICA [ 39 ] is a policy-based method , which learns an end-to-end differentiable optimization86 where it trains a hypernetwork that maps the state into a set of weights which , in turn , maps the87 action policies into the Q estimate . On the other hand , value-based methods often represent the88 global state-action value as an aggregation of the individual values . The value decomposition is linear89 in the earlier work VDN [ 26 ] , and it ignores the state information . QMIX [ 22 ] learns a non-linear90 mixing network with the global state and maps the individual state-action values into the joint Q value91 estimate . Although QMIX performs well in various environments , it still faces the mixing network ’ s92 monotonicity constraint limitation . QTRAN [ 24 ] further avoids the representation limitations by93 using linear constraints between individual utilities and the global state-action value . It guarantees94 optimal decentralization , but its constraints are computationally intractable , and the relaxations often95 lead to unsatisfied performance . QPLEX [ 29 ] decomposes Q values following the dueling structure,96 transferring the monotonicity condition from Q values to advantage values . QPD [ 35 ] leverages the97 integrated gradient attribution technique to decompose global Q values along trajectory paths based98 on the assumption that an agent ’ s local reward is linearly correlated with its contribution to the team.99 3 Methods100 3.1 Preliminaries101 Notations . This work considers a fully cooperative multi-agent task with n agents A = { 1 , ... , n } 102 as a Dec-POMDP [ 16 ] defined by a tuple G = ( S , U , P , r , Z , O , n , γ ) . The environment has a103 true state s ∈ S. Each agent a chooses an action uat from its action space U at each timestep104 t and forms a joint action ut that induces a transition in the environment according to the state105 transition function P ( st+1|st , ut ) : S×Un×S → [ 0 , 1 ] . The agents share the same reward function106 r ( s , u ) : S × un → R , and γ ∈ [ 0 , 1 ) is the discount factor . We consider partially observable107 scenarios in which agent a acquires its local observation za ∈ Z drawn from O ( st , a ) : S ×A → Z.108 Each agent has an action-observation history τa ∈ T ≡ ( Z × U ) ∗ , on which it conditions a policy109 πa ( ua|τa ) : T × U → [ 0 , 1 ] . We denote joint quantities over agents in bold and joint quantities over110 agents other than a given agent a with the superscript −a.111 Integrated Gradients . Many works aim to attribute the predictions of deep networks to their input112 features [ 1 , 15 , 2 ] . As one of them , integrated gradients [ 25 ] aggregates the gradients along the inputs113 that fall on the lines between the baseline ~b and the input ~x = ( x1 , ... , xj , ... , xd ) . It explains how114 much one feature affects the deep network output F while changing from F ( ~b ) to F ( ~x ) along a115 path between ~b and ~x . Given a path function τ ( α ) with α ∈ [ 0 , 1 ] specifying a path from baseline116 τ ( 0 ) = ~b to the input τ ( 1 ) = ~x , then integrated gradients along the jth dimension is acquired by:117 cj = PathIG τ j ( ~x ) : := ∫ 1 0 ∂F ( τ ( α ) ) ∂τj ( α ) ∂τj ( α ) ∂α dα , ( 1 ) where cj represents xj ’ s contribution to the difference between baseline prediction F ( ~b ) and F ( ~x ) .118 In this work , we leverage the integrated gradients technique to dynamically adjust the order of our119 proposed sequential credit assignment according to each agent ’ s contribution to the team.120 3.2 Sequential MARL Framework121 The relationship in a multi-agent system is complicated , as every agent makes decisions based on122 the environment interfered with by the other agents . If we model each agent as a node and model123 the cooperations between them as edges , the cooperative relationship will be built as a complicated124 web-like graph shown in Figure 1 ( a ) . Evaluating the actions of any agent should take into account125 the behaviors of other agents in this situation . It is hard to judge whether an agent ’ s current action is126 beneficial to the team when we have not evaluated other agents ’ actions . If we can not determine an127 analysis order , we can only analyze all the agents implicitly as most existing methods did , and the128 cooperation is often fitted only by deep neural networks , leading to unsatisfactory results.129 This section presents a sequential framework for cooperative MARL , which aims to analyze agents ’ 130 actions one by one . Our key assumption is that evaluations of some agents in a team are less affected131 than others . Thus we can study these less-affected agents first and then analyze the others based on132 the actions of these already-studied agents . For instance , when evaluating a staff ’ s action , the CEO ’ s133 decision plays a vital role because we have to judge whether the staff obeys the command or not . On134 the contrary , the staff intuitively has little impact on evaluating the CEO ’ s decision . In assessing the135 CEO , we often consider external factors such as market situation , modeled as state s in MARL.136 We introduce a variable Oi to help model this sequential MARL framework . This additional variable137 represents a random event that our cooperation study ( e.g. , credit assignment ) on agent ai is optimal or138 precise . Then the probability p ( Oi ) denotes the accuracy of our research on agent ai . For illustration139 and understanding convenience , we discuss a simple multi-agent system with three agents as an140 example , in which agents are identified by ai ( i ∈ { 1 , 2 , 3 } ) . In original MARL , the evaluation of141 agent ai will influence all the other agents ’ assessments . Thus events O1 , O2 and O3 are mutually142 dependent , as shown in Figure 1 ( b ) . We calculate the probability of studying the system accurately143 by computing conditional probabilities:144 p ( O1 , O2 , O3 ) = p ( O1 ) · p ( O2|O1 ) · p ( O3|O1 , O2 ) ( 2a ) ... = p ( O3 ) · p ( O2|O3 ) · p ( O1|O2 , O3 ) ( 2b ) where p ( Oj |Oi ) denotes the probability of agent aj ’ s accurate analysis under the condition of145 conducting a precise study on agent ai . It also indicates the accuracy of aj ’ s analysis conditions on146 precisely assess ai . We then conclude that:147 p ( O1 , O2 , O3 ) = p ( Oi ) · p ( Oj |Oi ) · p ( Ok|Oi , Oj ) ( 3 ) where i , j , k ∈ { 1 , 2 , 3 } , i 6= j , k 6= i , j.148 We take Equ . ( 2a ) as an example . To study the cooperation of this multi-agent system precisely ( i.e.,149 big p ( O1 , O2 , O3 ) ) , we can first analyze a1 as accurately as possible ( i.e. , big p ( O1 ) ) and then go150 on to investigate a2 and a3 respectively with the best possible accuracy ( i.e. , big p ( O2|O1 ) and151 p ( O3|O1 , O2 ) ) under the condition of preceding agents ’ precise analysis.152 The sequential MARL framework reduces the complexity of the model with six dotted arrows that153 indicate correlations between agents ’ evaluations in Figure 1 ( b ) by half , as those three dotted lines in154 Figure 1 ( c ) show . Equ . ( 3 ) suggests that we can analyze the cooperation of a multi-agent system in155 any order , but from the CEO-Staff example , we can see that the difficulty of analyzing in various156 orders is not the same . Further discussion on the sequence will show in Section 3.4.157 In general , we specify an order to analyze the cooperation in the sequential MARL framework . We158 fix an agent ’ s actions after assessing it and study a particular agent based on the fixed actions of its159 preceding agents , reflecting the intuition that a CEO ’ s decision has a strong influence on evaluating160 the staff in the example mentioned earlier . This sequential MARL framework significantly alleviates161 the correlations in studying agents and helps us assess their cooperation more directly.162 3.3 Sequential Credit Assignment163 Following the CTDE paradigm , we utilize a centralized critic for each actor to follow a gradient164 based on an advantage function A estimated from this critic:165 g = ∇θπ log π ( u|τat ) A . ( 4 ) The advantage function A for each actor explicitly deduces how that particular agent contributes to166 the team . COMA [ 6 ] introduced a counterfactual baseline inspired by difference rewards [ 34 ] . For167 each agent a , COMA computes an advantage function that compares the Q-value for the action ua to168 a counterfactual baseline that marginalizes out ua while keeping the others ’ actions u−a fixed:169 AaCOMA ( s , u ) = Q ( s , ( ua , u−a ) ) − ∑ u′a πa ( u′a|τa ) ·Q ( s , ( u−a , u′a ) ) . ( 5 ) COMA avoids expensive calculations through careful network design . However , each agent ’ s170 contribution deduced by COMA is still imperfect . The evaluation of ua is based on the fixed u−a171 in Equ . ( 5 ) , so agent a will learn a policy that works better with u−a in this way . It ignores the joint172 actions ( ua , u−a′ ) with u−a′ 6= u−a that may lead to unexpected results when assessing ua.173 To analyze each agent a ’ s contribution more objectively , we consider the influence of all joint actions174 with ua . Considering all potential action combinations , we calculate a counterfactual advantage for175 each agent ’ s action , derived by computing the expectation on all the actions of other agents:176 Aa ( s , u ) = Eu−a [ Q ( s , ( ua , u−a ) ) ] − Eu−a [ ∑ u′a πa ( u′a|τa ) ·Q ( s , ( u−a , u′a ) ) ] . ( 6 ) Under our proposed sequential MARL framework , we carry out credit assignment according to a177 specific order , and there is no need to consider all the possible joint actions . After assessing agent a,178 we fix its action and evaluate agents after it based on a ’ s fixed action , so the following agents ’ credit179 assignments do not have to compute the expectation on ua anymore.180 We now give the detailed sequential credit assignment for a team with n agents identified by181 ai ( i ∈ { 1 , ... , n } ) under one specific sequence { a1 , a2 , ... , an } , and it can also be concluded from the182 rest ( n ! − 1 ) orders in the same way . Here we denote uai−1a1 = [ ua1 , ua2 , ... , uai−1 ] ( i = 2 , 3 , ... , n ) .183 As for agent ai ( i 6= 1 ) in the sequence , the contribution of its leading agents a1 , a2 , ... , ai−1 has184 been deduced . We fix the leading agents ’ actions and assess agent ai ’ s action based on u ai−1 a1 , so185 there is no need to calculate the expectations on [ ua1 , ua2 , ... , uai−1 ] , simplifying Equ . ( 6 ) to:186 Aai ( s , u ) = ∑ u′ai+1 · · · ∑ u′an πai+1 ( u′ai+1 |τai+1 ) · · ·πan ( u′an |τan ) ·Q ( s , ( uaia1 , u ′ai+1 , · · · , u′an ) ) − ∑ u′ai · · · ∑ u′an πai ( u′ai |τai ) · · ·πan ( u′an |τan ) ·Q ( s , ( uai−1a1 , u ′ai , · · · , u′an ) ) . ( 7 ) Then the first agent a1 ’ s advantage is:187 Aa1 ( s , u ) = ∑ u′a2 · · · ∑ u′an πa2 ( u′a2 |τa2 ) · · ·πan ( u′an |τan ) ·Q ( s , ( ua1 , u′a2 , · · · , u′an ) ) − ∑ u′a1 · · · ∑ u′an πa1 ( u′a1 |τa1 ) · · ·πan ( u′an |τan ) ·Q ( s , ( u′a1 , u′a2 , · · · , u′an ) ) ( 8 ) To illustrate the effectiveness of our sequential counterfactual advantage , we conduct a simple188 but illuminating test in two common multi-agent particle environments [ 11 ] , Predator-Prey and189 Cooperative Navigation . We train both methods with 5 random seeds , and agents are trained for 5000190 episodes . We provide detailed information on the environments and experiments in the Appendix . As191 shown in Figure 2 , our sequential advantage functions help agents handle the task faster and better.192 Our sequential advantage for each agent achieves an additive decomposition of the total advantage193 function , which to some extent explains the soundness and superiority of our advantage over COMA ’ s.194 Claim 1 . The proposed sequential credit assignment achieves additive advantage-decomposition.195 Proof . See Appendix A.196 Facing the same problem as COMA that those evaluations are expensive , we model the first term197 in Equ . ( 7 ) as a function fφ of ( ua1 , ua2 , ... , uai , πai+1 , ... , πan ) to address this issue , and the second198 term is a similar function of ( ua1 , ua2 , ... , uai−1 , πai , ... , πan ) . Thus , we rewrite Equ . ( 7 ) as:199 Aai = fφ ( s ; u a1 , ua2 , ... , uai , πai+1 , ... , πan ) − fφ ( s ; ua1 , ua2 , ... , uai−1 , πai , ... , πan ) . ( 9 ) Here fφ is a function evaluating agents ’ action-policy vectors , where fφ ( ua1 , ua2 , ... , uan ) = Q and200 fφ ( π a1 , πa2 , ... , πan ) = V . We design the complete setup for SeCA , which is illustrated in Figure 3.201 Critic Learning . We train critic fφ on-policy to estimateQ , utilizing a practical variant of TD ( λ ) [ 27 ] 202 adapted for use with deep neural networks . In particular , the critic parameter φ is updated by minibatch203 gradient descent to minimize the following loss:204 Lt ( φ ) = ( y ( λ ) t − fφ ( st , ut ) ) 2 , where y ( λ ) t = rt + γ ( λy ( λ ) t+1 + ( 1− λ ) fφ− ( st+1 , ut+1 ) ) . ( 10 ) We utilize a target critic fφ− [ 14 ] to improve learning stability and update φ− ← φ periodically . The205 critic learning flow is shown at the top of Figure 3 ( c ) . The input for critic training is the state s and206 the action vector u = [ u1 , u2 , ... , un ] denoted as v1 : n.207 Policy Learning . We optimize each agent a ’ s policy parameter θa by maximizing the following208 objective , which contains our proposed advantage function and an entropy regularization termH:209 ga = Eτ∼π [ ∇θa log πa ( ua|τa ) Aa ( s , u ) +H ( πa ( ·|τa ) ) ] , ( 11 ) where the derivative of the adaptive entropy regularization termH ( πa ( ·|τa ) ) [ 39 ] with respect to the210 i-th action probability pai is given by:211 dHi : = −ξ · ( log pai + 1 ) /H ( πa ( ·|τa ) ) , where H ( πa ( ·|τa ) ) = Eua∼πa [ − log πa ( ua|τa ) ] . ( 12 ) We share parameters among agents , and the gradient we use to train the actor shared by all agents is:212 g = Eτ∼π [ ∑ a ( ∇θa log πa ( ua|τa ) Aa ( s , u ) +H ( πa ( ·|τa ) ) ) ] . ( 13 ) The inputs of the centralized critic fφ to compute the advantage function are the state s and two213 action-policy vectors v1 : i = [ u1 , ... , ui , πi+1 , ... , πn ] and v1 : i−1 = [ u1 , ... , ui−1 , πi , ... , πn ] . The214 bottom of Figure 3 ( c ) demonstrates the policy learning flow.215 3.4 Sequence Adjustment Through Integrated Gradients216 We apply integrated gradients to adjust the credit assignment sequence dynamically . Reviewing the217 enlightening and straightforward CEO-Staff example discussed in Section 3.2 , we can evaluate the218 staff ’ s behavior based on the CEO ’ s decision , but assessing the CEO does not require much attention219 to the staff ’ s action . Therefore , we would analyze the CEO first and then evaluate the staff based on220 the CEO ’ s current action . However , this example is not generalized for two reasons : ( 1 ) There are221 often multiple agents taking the same role in a system with superior-subordinate relationships , and the222 sequence of these agents is hard to determine ; ( 2 ) Not all scenarios have such superior-subordinate223 relationships . The agents often do not need to follow others ’ commands in many applications.224 We generalize the CEO-Staff example to propose a universal model . Instead of focusing on the roles225 among the agents as in [ 31 , 32 ] , we are more interested in agents ’ contributions . Although the CEO226 and the staff have a superior-subordinate relationship , they are essentially employees of an enterprise.227 The staff plays an auxiliary role and acts based on the CEO ’ s decision . The staff ’ s work is meaningful228 only if the CEO ’ s decision is correct . Therefore , we often intuitively assume that an enterprise ’ s229 leader is paid more and contributes more . Based on this , we transform the roles of the CEO and staff230 into employees with different contributions to the enterprise . In the sequential MARL framework , we231 first assign credit to the agent with a higher contribution to the team.232 The attribution method is a powerful way to determine the influence of input features ’ each component233 on the network output value [ 2 ] . Among them , integrated gradients [ 25 ] leverages path integral to234 aggregate gradients along the inputs that fall on the lines between the baseline and the input , which235 is a natural tool for measuring each agent ’ s contribution . QPD [ 35 ] utilizes the integrated gradient236 attribution technique to decompose shared rewards along trajectory paths , revealing how much each237 agent ’ s observation and action contributes to the global Q value . However , it remains unclear whether238 individual Q value should be linearly correlated to or approximated by the agent ’ s contribution , as in239 the case of QPD . The proper connection between agents ’ contributions and their individual Q values240 in a cooperative team is worth well studied for the community.241 Here we avoid detailed analysis on the relationship between agents ’ contributions and their individual242 rewards . Instead , we use integrated gradients to measure agents ’ contributions to the state transition243 and adjust the credit assignment sequence based on their contributions . In particular , we estimate244 agent a ’ s contribution ca in the trajectory path τ t2t1 from time t1 to t2 based on its policy vector π a:245 ca = ∑ xj∈πa PathIG τ t2 t1 j ( π a ) , ( 14 ) where xj is j-th dimension of the policy vector πa . The computation for PathIG is shown in Equ . ( 1 ) .246 We compute each agent ’ s contribution c to the state transition from st1 to st2 and analyze the agent247 with higher c first . We further study the adjustment frequency and its effectiveness in Section 4.2248 4 Experiments and Analysis249 4.1 Experimental Setup250 We consider a challenging set of cooperative StarCraft II maps from the SMAC benchmark [ 23 ] 251 classified as Easy , Hard , and Super Hard scenarios according to the baseline algorithms ’ performance.252 The inherent differences among various methods and their training procedure ( e.g. , on/off-policy253 learning for value-based/policy-based methods ) bring difficulties when comparing methods in a254 reasonably fair manner without introducing additional components ( e.g. , importance sampling [ 13 , 33 ] 255 for off-policy methods ) . To attribute any poor performance of policy-based methods to potential256 algorithmic limitations or poor training conditions ( in particular , high variance due to small batch257 sizes or insufficient gradient steps ) , we follow [ 5 , 39 ] , training all methods with 32 parallel runners258 to generate trajectories and using batches of 32 episodes . We evaluate each method every 320K259 steps with 32 episodes and report the 1st , median , and 3rd quartile win rates across 5 random seeds.260 Detailed information about the scenarios and the experimental setup is shown in the Appendix.261 4.2 Ablation Studies262 We first carry out ablation experiments on a Super Hard map MMM2 to validate key elements of SeCA.263 Proposed Advantage and Architecture . In Section 3.3 , we compare our sequential advantage264 with COMA ’ s in two simple multi-agent particle environments and show our superiority in Figure 2.265 Afterward , we introduce a fφ approximation and a corresponding network architecture . Here we apply266 the same approximation and architecture for COMA ’ s counterfactual advantage ( COMA-newArchi ) 267 and compare it with the original COMA and our method SeCA to show the effects of our advantage268 function , approximation , and network architecture . The result is illustrated in Figure 4 ( a ) . COMA269 performs poorly on this Super Hard map but acquires significant improvement with our approximation270 and architecture . Our sequential advantage further accelerates and stabilizes the training.271 Sequence Adjustment Algorithm . SeCA ’ s credit assignment sequence is dynamic . We compare272 our method with some intuitive adjustments to validate its effects . One could first evaluate agents273 with higher current-action probability ( SeCA-Prob ) or lower policy entropy ( SeCA-Entro ) , as these274 agents are more confident in their acts , and we can assess other agents based on their behaviors . Since275 SeCA-Prob and Entro get a new order at each step , to be fair , we set the path length in Equ . ( 14 ) to one,276 i.e. , consider agents ’ contributions based on the transition from st to st+1 ( SeCA-IG-1 ) . Figure 4 ( b ) 277 illustrates that SeCA-Prob and Entro learn better than the fixed method ( SeCA-Fixed ) , but Prob has a278 larger variance than Entro . Fixed is better than expected , which we believe is because that the fixed279 sequence acquires adequate training . Our integrated-gradients-adjustment performs the best in win280 rates and stability , and the others have inferior performance and incredibly high variance.281 Sequence Adjustment Frequency . We next consider how the sequence adjustment frequency in282 SeCA-IG affects the performance . Except per step adjustment ( i.e. , SeCA-IG-1 ) , one could also283 update the sequence after a stage or an episode . If we change the credit assignment order for every284 episode during training ( SeCA-IG-episode ) , then τ t2t1 in Equ . ( 14 ) represents a whole episode . As for285 stage adjustment , it is hard to define a stage in these tasks , and the stage length varies in diverse maps.286 Here we set stage length to 10 and 20 , respectively denoted as SeCA-IG-10 and SeCA-IG-20 . As the287 results in Figure 4 ( c ) show , IG-1 and IG-episode have similar final win rates . However , IG-episode288 converges more quickly with smaller variance . The reason for IG-10 ( 20 ) ’ s mediocre performance289 and high variance may be because the stage length needs to be dynamically adjusted . Inappropriate290 adjustment frequency fails to adapt to the stage changes in the task and causes insufficient training291 for each sequence . We utilized SeCA-IG-episode in other experiments and will investigate dynamic292 stage learning in the future to improve stage adjustment.293 4.3 Comparisons with State-of-the-arts294 We compare SeCA with some competitive algorithms , including the representative explicit credit295 assignment method COMA , the policy-based implicit method LICA , the common-used baseline296 QMIX and QTRAN . Methods are evaluated on 6 scenarios , including 2 Easy ones ( 2s3z , 1c3s5z ) ,297 2 Hard ones ( 2c_vs_64zg , 3s_vs_5z ) , and 2 Super Hard ones ( MMM2 , 3s5z_vs_3s6z ) . We train298 all methods for 32 million steps in Easy maps and 64 million steps in Hard and Super Hard maps.299 These scenarios involve homogeneous and heterogeneous teams , symmetric and asymmetric battles,300 allowing a holistic study on all methods . Our experiments are based on the latest PyMARL [ 23 ] 301 utilizing SC2.4.10 . Performance is not always comparable between versions , so the results may be302 subtly different from the original papers.303 As we can see in Figure 5 , SeCA demonstrates its robustness by achieving good performances in304 scenarios with various characteristics . All methods except COMA and QTRAN solve two Easy305 scenarios , and SeCA performs better in convergence speed and stability . SeCA ’ s advantage is further306 extended in the Hard map 2c_vs_64zg , and it converges significantly faster than other methods.307 Although classified only as Hard , 3s_vs_5z invalidates most algorithms except QMIX and SeCA , as308 Stalkers have to learn dispersing and making enemies give chase while maintaining enough distance309 ( `` kiting '' technique ) in this map . SeCA has a higher variance than QMIX . This is possibly because310 the Stalkers ’ scattering prioritizes individual performance over cooperation which is more in line311 with QMIX ’ s monotonicity constraint . Nevertheless , SeCA ’ s performance improvements on the312 Super Hard scenarios MMM2 and 3s5z_vs_3s6z demonstrate the effectiveness of our method . LICA ’ s313 performance in 3s5z_vs_3s6z here is different from the original paper , as the original results for314 this map are obtained by using a different entropy coefficient , which is explained in its open-source315 implementation.1 This parameter tuning is unfair when comparing methods , so all experiments in this316 paper use the fixed entropy coefficient . We also visualize the learned sequences in different battles of317 3s_vs_5z to provide insights into our sequence adjustment in the Appendix.318 We are supposed to compare our method with QPD that also utilizes integrated gradients to show319 our improvement . However , QPD modifies the original SMAC environment to acquire additional320 information for policy training , which is mentioned in its open-source implementation.2 Therefore , it321 is unfair to compare QPD ’ s learning curves in the modified environment with other methods , and322 QPD ’ s authors did not provide methods ’ learning curves comparison in the original paper . We follow323 them , providing a win rate table in the Appendix to show our superiority over QPD.324 5 Conclusions and Future Work325 This paper presents SeCA , a cooperative MARL framework with sequential credit assignment.326 SeCA computes counterfactual advantage functions to evaluate each agent based on the actions of327 the preceding agents under a specific sequence . The sequence is adjusted dynamically according328 to agents ’ contributions to the team deduced by integrated gradients . SeCA accelerates policy329 convergence and improves the final performance over existing recognized methods in practice . In the330 future , we will further investigate stage learning in an episode and adjust the sequence per stage to331 improve SeCA and achieve adaptive cooperation in various task situations.332 1https : //github.com/mzho7212/LICA 2https : //github.com/QPD-NeurIPS2019/QPD References333 [ 1 ] Marco Ancona , Enea Ceolini , Cengiz Öztireli , and Markus Gross . Towards better understanding334 of gradient-based attribution methods for deep neural networks . In International Conference on335 Learning Representations , 2018.336 [ 2 ] Guillem Brasó Andilla . Attribution methods for deep convolutional networks.337 [ 3 ] Yongcan Cao , Wenwu Yu , Wei Ren , and Guanrong Chen . An overview of recent progress in338 the study of distributed multi-agent coordination . IEEE Transactions on Industrial informatics,339 9 ( 1 ) :427–438 , 2012.340 [ 4 ] Yu-han Chang , Tracey Ho , and Leslie Kaelbling . All learning is local : Multi-agent learning in341 global reward games . In Advances in Neural Information Processing Systems , pages 808–814,342 2004.343 [ 5 ] Yali Du , Lei Han , Meng Fang , Ji Liu , Tianhong Dai , and Dacheng Tao . Liir : Learning individual344 intrinsic reward in multi-agent reinforcement learning . In Advances in Neural Information345 Processing Systems , pages 4403–4414 , 2019.346 [ 6 ] Jakob Foerster , Gregory Farquhar , Triantafyllos Afouras , Nantas Nardelli , and Shimon Whiteson.347 Counterfactual multi-agent policy gradients . In AAAI Conference on Artificial Intelligence,348 pages 2974–2982 , 2018.349 [ 7 ] Jayesh K Gupta , Maxim Egorov , and Mykel Kochenderfer . Cooperative multi-agent control350 using deep reinforcement learning . In International Conference on Autonomous Agents and351 Multiagent Systems , pages 66–83 , 2017.352 [ 8 ] Pablo Hernandez-Leal , Bilal Kartal , and Matthew E Taylor . A survey and critique of multiagent353 deep reinforcement learning . Autonomous Agents and Multi-Agent Systems , 33 ( 6 ) :750–797,354 2019.355 [ 9 ] Maximilian Hüttenrauch , Adrian Šošić , and Gerhard Neumann . Guided deep reinforcement356 learning for swarm systems . In AAMAS Autonomous Robots and Multirobot Systems ( ARMS ) 357 Workshop , 2017.358 [ 10 ] Landon Kraemer and Bikramjit Banerjee . Multi-agent reinforcement learning as a rehearsal for359 decentralized planning . Neurocomputing , 190:82–94 , 2016.360 [ 11 ] Ryan Lowe , Yi Wu , Aviv Tamar , Jean Harb , Pieter Abbeel , and Igor Mordatch . Multi-agent361 actor-critic for mixed cooperative-competitive environments . In Advances in Neural Information362 Processing Systems , pages 6382–6393 , 2017.363 [ 12 ] Anuj Mahajan , Tabish Rashid , Mikayel Samvelyan , and Shimon Whiteson . Maven : Multi-364 agent variational exploration . In Advances in Neural Information Processing Systems , pages365 7611–7622 , 2019.366 [ 13 ] A Rupam Mahmood , Hado van Hasselt , and Richard S Sutton . Weighted importance sampling367 for off-policy learning with linear function approximation . In Advances in Neural Information368 Processing Systems , pages 3014–3022 , 2014.369 [ 14 ] Volodymyr Mnih , Koray Kavukcuoglu , David Silver , Andrei A Rusu , Joel Veness , Marc G370 Bellemare , Alex Graves , Martin Riedmiller , Andreas K Fidjeland , Georg Ostrovski , et al.371 Human-level control through deep reinforcement learning . Nature , 518 ( 7540 ) :529–533 , 2015.372 [ 15 ] Grégoire Montavon , Wojciech Samek , and Klaus-Robert Müller . Methods for interpreting and373 understanding deep neural networks . Digital Signal Processing , 73:1–15 , 2018.374 [ 16 ] Frans A Oliehoek and Christopher Amato . A concise introduction to decentralized POMDPs.375 Springer , 2016.376 [ 17 ] Frans A Oliehoek , Matthijs TJ Spaan , and Nikos Vlassis . Optimal and approximate q-value377 functions for decentralized pomdps . Journal of Artificial Intelligence Research , 32:289–353,378 2008.379 [ 18 ] Afshin OroojlooyJadid and Davood Hajinezhad . A review of cooperative multi-agent deep380 reinforcement learning . arXiv preprint arXiv:1908.03963 , 2019.381 [ 19 ] Gregory Palmer , Karl Tuyls , Daan Bloembergen , and Rahul Savani . Lenient multi-agent deep382 reinforcement learning . In International Conference on Autonomous Agents and MultiAgent383 Systems , pages 443–451 , 2018.384 [ 20 ] Scott Proper and Kagan Tumer . Modeling difference rewards for multiagent learning . In385 International Conference on Autonomous Agents and Multi-Agent Systems , pages 1397–1398,386 2012.387 [ 21 ] Tabish Rashid , Gregory Farquhar , Bei Peng , and Shimon Whiteson . Weighted qmix : Expanding388 monotonic value function factorisation . In Advances in Neural Information Processing Systems,389 pages 10199–10210 , 2020.390 [ 22 ] Tabish Rashid , Mikayel Samvelyan , Christian Schroeder , Gregory Farquhar , Jakob Foerster,391 and Shimon Whiteson . Qmix : Monotonic value function factorisation for deep multi-agent392 reinforcement learning . In International Conference on Machine Learning , pages 4295–4304,393 2018.394 [ 23 ] Mikayel Samvelyan , Tabish Rashid , Christian Schroeder de Witt , Gregory Farquhar , Nantas395 Nardelli , Tim G. J. Rudner , Chia-Man Hung , Philiph H. S. Torr , Jakob Foerster , and Shimon396 Whiteson . The starcraft multi-agent challenge . CoRR , abs/1902.04043 , 2019.397 [ 24 ] Kyunghwan Son , Daewoo Kim , Wan Ju Kang , David Earl Hostallero , and Yung Yi . Qtran:398 Learning to factorize with transformation for cooperative multi-agent reinforcement learning.399 In International Conference on Machine Learning , pages 5887–5896 , 2019.400 [ 25 ] Mukund Sundararajan , Ankur Taly , and Qiqi Yan . Axiomatic attribution for deep networks . In401 International Conference on Machine Learning , pages 3319–3328 , 2017.402 [ 26 ] Peter Sunehag , Guy Lever , Audrunas Gruslys , Wojciech Marian Czarnecki , Vinicius Zambaldi,403 Max Jaderberg , Marc Lanctot , Nicolas Sonnerat , Joel Z Leibo , Karl Tuyls , et al . Value-404 decomposition networks for cooperative multi-agent learning based on team reward . In In-405 ternational Conference on Autonomous Agents and MultiAgent Systems , pages 2085–2087,406 2018.407 [ 27 ] Richard S Sutton . Learning to predict by the methods of temporal differences . Machine learning,408 3 ( 1 ) :9–44 , 1988.409 [ 28 ] Kagan Tumer and Adrian Agogino . Distributed agent-based air traffic flow management . In410 International Joint Conference on Autonomous Agents and Multiagent Systems , pages 1–8,411 2007.412 [ 29 ] Jianhao Wang , Zhizhou Ren , Terry Liu , Yang Yu , and Chongjie Zhang . Qplex : Duplex dueling413 multi-agent q-learning . arXiv preprint arXiv:2008.01062 , 2020.414 [ 30 ] Jianhong Wang , Yuan Zhang , Tae-Kyun Kim , and Yunjie Gu . Shapley q-value : A local reward415 approach to solve global reward games . In AAAI Conference on Artificial Intelligence , pages416 7285–7292 , 2020.417 [ 31 ] Tonghan Wang , Heng Dong , Victor Lesser , and Chongjie Zhang . Roma : Multi-agent reinforce-418 ment learning with emergent roles . In International Conference on Machine Learning , pages419 9876–9886 , 2020.420 [ 32 ] Tonghan Wang , Tarun Gupta , Anuj Mahajan , Bei Peng , Shimon Whiteson , and Chongjie Zhang.421 Rode : Learning roles to decompose multi-agent tasks . arXiv preprint arXiv:2010.01523 , 2020.422 [ 33 ] Ziyu Wang , Victor Bapst , Nicolas Heess , Volodymyr Mnih , Remi Munos , Koray Kavukcuoglu,423 and Nando de Freitas . Sample efficient actor-critic with experience replay . arXiv preprint424 arXiv:1611.01224 , 2016.425 [ 34 ] David H Wolpert and Kagan Tumer . Optimal payoff functions for members of collectives . In426 Modeling complexity in economic and social systems , pages 355–369 . World Scientific , 2002.427 [ 35 ] Yaodong Yang , Jianye Hao , Guangyong Chen , Hongyao Tang , Yingfeng Chen , Yujing Hu,428 Changjie Fan , and Zhongyu Wei . Q-value path decomposition for deep multiagent reinforcement429 learning . In International Conference on Machine Learning , pages 10706–10715 , 2020.430 [ 36 ] Dayong Ye , Minjie Zhang , and Yun Yang . A multi-agent framework for packet routing in431 wireless sensor networks . Sensors , 15 ( 5 ) :10026–10047 , 2015.432 [ 37 ] Wang Ying and Sang Dayong . Multi-agent framework for third party logistics in e-commerce.433 Expert Systems with Applications , 29 ( 2 ) :431–436 , 2005.434 [ 38 ] Kaiqing Zhang , Zhuoran Yang , and Tamer Başar . Multi-agent reinforcement learning : A435 selective overview of theories and algorithms . arXiv preprint arXiv:1911.10635 , 2019.436 [ 39 ] Meng Zhou , Ziyu Liu , Pengwei Sui , Yixuan Li , and Yuk Ying Chung . Learning implicit437 credit assignment for cooperative multi-agent reinforcement learning . In Advances in Neural438 Information Processing Systems , 2020.439 Checklist440 1 . For all authors ... 441 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s442 contributions and scope ? [ Yes ] 443 ( b ) Did you describe the limitations of your work ? [ Yes ] We discussed it in the experiment444 analysis in Section 4.3 and future work in Section 5.445 ( c ) Did you discuss any potential negative societal impacts of your work ? [ N/A ] 446 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to447 them ? [ Yes ] 448 2 . If you are including theoretical results ... 449 ( a ) Did you state the full set of assumptions of all theoretical results ? [ Yes ] 450 ( b ) Did you include complete proofs of all theoretical results ? [ Yes ] We provided the proof451 of our Claim in the supplemental material.452 3 . If you ran experiments ... 453 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-454 mental results ( either in the supplemental material or as a URL ) ? [ Yes ] We provided455 our code and instructions in the supplemental material.456 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they457 were chosen ) ? [ Yes ] We described the training details in the supplemental material.458 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-459 ments multiple times ) ? [ Yes ] See Figure 2 , 4 and 5.460 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type461 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] We described it in the supplemental462 material.463 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 464 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] 465 ( b ) Did you mention the license of the assets ? [ Yes ] 466 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 467 We provided our code in the supplemental material.468 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re469 using/curating ? [ N/A ] 470 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable471 information or offensive content ? [ N/A ] 472 5 . If you used crowdsourcing or conducted research with human subjects ... 473 ( a ) Did you include the full text of instructions given to participants and screenshots , if474 applicable ? [ N/A ] 475 ( b ) Did you describe any potential participant risks , with links to Institutional Review476 Board ( IRB ) approvals , if applicable ? [ N/A ] 477 ( c ) Did you include the estimated hourly wage paid to participants and the total amount478 spent on participant compensation ? [ N/A ] 479

Summary: The authors are studying cooperative marl with global reward,  and in particular the credit assignment problem. This is considering the sequential approach where one assign credit one agent at a time by marginalizing out its action akin to difference rewards. When there is a natural sequence of importance of the agents and whose contribution depends on whose actions. They use CEO and staff as an illustrative example of when it makes sense.