Cooperative Multi-Agent Reinforcement Learning with Sequential Credit Assignment
Year: 2021

Abstract:
Centralized training with decentralized execution is a standard paradigm for coop1 erative multi-agent reinforcement learning (MARL), with credit assignment being 2 a major challenge. In this paper, we propose a cooperative MARL method with 3 sequential credit assignment (SeCA) that deduces each agent’s contribution to the 4 team’s success one by one to learn better cooperation. We first present a sequential 5 MARL framework, under which we introduce a new counterfactual advantage to 6 evaluate each agent based on its preceding agents’ actions in a specific sequence. 7 As this credit assignment sequence tremendously impacts the performance, we 8 further present a sequence adjustment algorithm utilizing integrated gradients. It 9 dynamically modifies the sequence among agents according to their contribution 10 to the team. SeCA employs a network which either estimates the Q value for 11 training the centralized critic or deduces the proposed advantage of each agent for 12 decentralized policy learning. Our method is evaluated on a challenging set of 13 StarCraft II micromanagement tasks and achieves state-of-the-art performance. 14

Centralized training with decentralized execution is a standard paradigm for coop-1 erative multi-agent reinforcement learning (MARL), with credit assignment being2 a major challenge. In this paper, we propose a cooperative MARL method with3 sequential credit assignment (SeCA) that deduces each agent’s contribution to the4 team’s success one by one to learn better cooperation. We first present a sequential5 MARL framework, under which we introduce a new counterfactual advantage to6 evaluate each agent based on its preceding agents’ actions in a specific sequence.7 As this credit assignment sequence tremendously impacts the performance, we8 further present a sequence adjustment algorithm utilizing integrated gradients. It9 dynamically modifies the sequence among agents according to their contribution10 to the team. SeCA employs a network which either estimates the Q value for11 training the centralized critic or deduces the proposed advantage of each agent for12 decentralized policy learning. Our method is evaluated on a challenging set of13 StarCraft II micromanagement tasks and achieves state-of-the-art performance.14
1 Introduction15
Cooperative multi-agent reinforcement learning (MARL) is a helpful tool in numerous applications16 such as robot swarm control [9], autonomous vehicle coordination [3], network routing [36], and17 productivity optimization [37]. This kind of problem where agents learn coordinated policies to18 optimize the global reward has been extensively studied in recent years [7, 19, 18, 38, 8].19
One natural way of addressing the cooperative MARL problem is the centralized approach, which20 treats the team as a single actor with a joint action space. Although we can trivially apply single-agent21 reinforcement learning algorithms to such settings, it usually does not scale well because the size of22 the joint action space grows exponentially with the number of agents. Besides, it is not applicable23 in real-world settings due to the inherent constraints on agent observability and communication.24 An alternative approach is to learn decentralized policies by independently training agents based25 on their local observations, but simultaneous exploration often brings non-stationarity that causes26 unstable learning and difficulties in convergence. As a result, the majority of work on MARL27 follows the centralized training with decentralized execution (CTDE) paradigm [17, 10, 22, 6], where28 decentralized policies can access extra state information during training.29
A crucial challenge of the CTDE paradigm in cooperative settings is to correctly deduce each agent’s30 contribution to the team’s success, also known as the multi-agent credit assignment problem [4].31 Existing methods can be classified as implicit and explicit credit assignment [39]. Previous implicit32 methods often deduce all agents’ contributions by representing the global state-action value as an33 aggregation of each agent’s state-action value [26, 22, 12, 24, 21, 29] and assigning the shared rewards34 to each agent according to the joint action at one time. In this way, these methods avoid the complex35 interaction analysis and instead fit these cooperation relationships by neural networks. However,36
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
implicit methods often face limitations in expressiveness, and their extensions to continuous action37 spaces may require additional strategies [39].38
On the other hand, recognized explicit approaches calculate difference rewards [34] against a certain39 reward baseline [28, 20, 6]. However, in cooperative MARL, evaluating any agent’s action requires40 considering the actions of all agents, so it is often difficult to determine the impact a particular agent’s41 behavior has on the team when we have not assessed other agents’ actions. In other words, we can42 not say that a single agent’s action is bad if the team receives a small reward because the shared43 reward is not decided only by this agent’s behavior. Maybe its action is actually good in that state.44
This paper presents a sequential credit assignment SeCA to evaluate individual agent actions explicitly45 and sequentially. Our motivation is to address the drawbacks of implicit methods that neglect the46 cooperation between agents or simply leave it to neural networks and further improve explicit credit47 assignment. In summary, we face two main challenges to learn a better explicit credit assignment: (1)48 how to alleviate the problem that it is hard to accurately deduce the contribution of one agent without49 previously assessing all the others’ action, and (2) how to evaluate agents better in an explicit way.50
To deal with (1), we introduce a sequential MARL framework. As mentioned above, without assessing51 the behaviors of other agents, we would never be able to evaluate a given agent’s action accurately.52 However, we point out in this paper that some agents are less affected by such influences than others,53 and we can first assign credit to them. For instance, evaluating a staff’s action needs to take the54 CEO’s command or action into consideration, while the former has little importance in assessing the55 CEO. Thus, we could evaluate the CEO first without considering the staff’s behavior and then analyze56 the staff based on the CEO’s action. We fully consider the action coordination between agents and57 explicitly deduce contribution to them one by one according to a particular order, so as to make up58 for the disadvantage of implicit methods that the cooperation is only inexplicably fitted by neural59 networks. Intuitively, the order significantly impacts the overall performance, so we further propose60 an algorithm to adjust the sequence dynamically through integrated gradients [25].61
As for (2), we compute an advantage function for each agent to attribute agent contributions explicitly.62 COMA [6] is a representative method that computes a baseline for each agent to reason about63 counterfactuals in which only one agent a’s action changes, so its evaluation of a’s action is based on64 the joint action u−a of other agents. In other words, the policy gradient of COMA only encourages65 agent a to learn in the direction that benefits the team while other agents are acting u−a, but the66 others’ actions are not necessarily u−a when executing. Unlike COMA, we focus more on the action67 coordination among agents and propose a new advantage under the proposed sequential framework.68
We summarize the contributions of this paper as follows: (1) We propose a sequential MARL69 framework in Section 3.2; (2) Under this framework, we introduce a sequential advantage function70 for each agent to guide their learning explicitly in Section 3.3. We further prove that the sequential71 credit assignment we proposed achieves additive advantage-decomposition. (3) We present a sequence72 adjustment algorithm based on integrated gradients to modify the credit assignment order dynamically73 in Section 3.4. This algorithm alleviates the impact caused by the sequence’s randomness and helps74 achieve competitive performance on a challenging set of StarCraft II micromanagement tasks [23].75
2 Related Work76
Explicit credit assignment gives valuable insights into agent actions’ contributions to the shared77 team reward and substantially promotes policy optimization. The representative method COMA [6]78 utilizes a counterfactual baseline that marginalizes out a single agent’s action while keeping the other79 agents’ actions fixed to calculate the advantage function. However, the advantage evaluates a single80 agent’s action based on the other agents’ current behaviors and ignores different action combinations.81 SQDDPG [30] distributes the global reward reflecting each agent’s contribution through Shapley82 Value. Although SQDDPG provides a theoretically justified framework, its assumption on the83 observability and convex game makes it impractical and performs poorly in complex environments.84
Implicit methods are a more common way when addressing the credit assignment challenge. Among85 them, LICA [39] is a policy-based method, which learns an end-to-end differentiable optimization86 where it trains a hypernetwork that maps the state into a set of weights which, in turn, maps the87 action policies into the Q estimate. On the other hand, value-based methods often represent the88 global state-action value as an aggregation of the individual values. The value decomposition is linear89
in the earlier work VDN [26], and it ignores the state information. QMIX [22] learns a non-linear90 mixing network with the global state and maps the individual state-action values into the joint Q value91 estimate. Although QMIX performs well in various environments, it still faces the mixing network’s92 monotonicity constraint limitation. QTRAN [24] further avoids the representation limitations by93 using linear constraints between individual utilities and the global state-action value. It guarantees94 optimal decentralization, but its constraints are computationally intractable, and the relaxations often95 lead to unsatisfied performance. QPLEX [29] decomposes Q values following the dueling structure,96 transferring the monotonicity condition from Q values to advantage values. QPD [35] leverages the97 integrated gradient attribution technique to decompose global Q values along trajectory paths based98 on the assumption that an agent’s local reward is linearly correlated with its contribution to the team.99
3 Methods100
3.1 Preliminaries101
Notations. This work considers a fully cooperative multi-agent task with n agents A = {1, ..., n}102 as a Dec-POMDP [16] defined by a tuple G = (S,U, P, r, Z,O, n, γ). The environment has a103 true state s ∈ S. Each agent a chooses an action uat from its action space U at each timestep104 t and forms a joint action ut that induces a transition in the environment according to the state105 transition function P (st+1|st,ut) : S×Un×S → [0, 1]. The agents share the same reward function106 r(s,u) : S × un → R, and γ ∈ [0, 1) is the discount factor. We consider partially observable107 scenarios in which agent a acquires its local observation za ∈ Z drawn from O(st, a) : S ×A → Z.108 Each agent has an action-observation history τa ∈ T ≡ (Z × U)∗, on which it conditions a policy109 πa(ua|τa) : T × U → [0, 1]. We denote joint quantities over agents in bold and joint quantities over110 agents other than a given agent a with the superscript −a.111 Integrated Gradients. Many works aim to attribute the predictions of deep networks to their input112 features [1, 15, 2]. As one of them, integrated gradients [25] aggregates the gradients along the inputs113 that fall on the lines between the baseline ~b and the input ~x = (x1, ..., xj , ..., xd). It explains how114 much one feature affects the deep network output F while changing from F (~b) to F (~x) along a115 path between ~b and ~x. Given a path function τ(α) with α ∈ [0, 1] specifying a path from baseline116 τ(0) = ~b to the input τ(1) = ~x, then integrated gradients along the jth dimension is acquired by:117
cj = PathIG τ j (~x) ::= ∫ 1 0 ∂F (τ(α)) ∂τj(α) ∂τj(α) ∂α dα, (1)
where cj represents xj’s contribution to the difference between baseline prediction F (~b) and F (~x).118 In this work, we leverage the integrated gradients technique to dynamically adjust the order of our119 proposed sequential credit assignment according to each agent’s contribution to the team.120
3.2 Sequential MARL Framework121
The relationship in a multi-agent system is complicated, as every agent makes decisions based on122 the environment interfered with by the other agents. If we model each agent as a node and model123 the cooperations between them as edges, the cooperative relationship will be built as a complicated124 web-like graph shown in Figure 1(a). Evaluating the actions of any agent should take into account125 the behaviors of other agents in this situation. It is hard to judge whether an agent’s current action is126 beneficial to the team when we have not evaluated other agents’ actions. If we cannot determine an127 analysis order, we can only analyze all the agents implicitly as most existing methods did, and the128 cooperation is often fitted only by deep neural networks, leading to unsatisfactory results.129
This section presents a sequential framework for cooperative MARL, which aims to analyze agents’130 actions one by one. Our key assumption is that evaluations of some agents in a team are less affected131 than others. Thus we can study these less-affected agents first and then analyze the others based on132 the actions of these already-studied agents. For instance, when evaluating a staff’s action, the CEO’s133 decision plays a vital role because we have to judge whether the staff obeys the command or not. On134 the contrary, the staff intuitively has little impact on evaluating the CEO’s decision. In assessing the135 CEO, we often consider external factors such as market situation, modeled as state s in MARL.136
We introduce a variable Oi to help model this sequential MARL framework. This additional variable137 represents a random event that our cooperation study (e.g., credit assignment) on agent ai is optimal or138 precise. Then the probability p(Oi) denotes the accuracy of our research on agent ai. For illustration139 and understanding convenience, we discuss a simple multi-agent system with three agents as an140 example, in which agents are identified by ai (i ∈ {1, 2, 3}). In original MARL, the evaluation of141 agent ai will influence all the other agents’ assessments. Thus events O1, O2 and O3 are mutually142 dependent, as shown in Figure 1(b). We calculate the probability of studying the system accurately143 by computing conditional probabilities:144
p(O1,O2,O3) = p(O1) · p(O2|O1) · p(O3|O1,O2) (2a) ...
= p(O3) · p(O2|O3) · p(O1|O2,O3) (2b) where p(Oj |Oi) denotes the probability of agent aj’s accurate analysis under the condition of145 conducting a precise study on agent ai. It also indicates the accuracy of aj’s analysis conditions on146 precisely assess ai. We then conclude that:147
p(O1,O2,O3) = p(Oi) · p(Oj |Oi) · p(Ok|Oi,Oj) (3) where i, j, k ∈ {1, 2, 3}, i 6= j, k 6= i, j.148 We take Equ.(2a) as an example. To study the cooperation of this multi-agent system precisely (i.e.,149 big p(O1,O2,O3)), we can first analyze a1 as accurately as possible (i.e., big p(O1)) and then go150 on to investigate a2 and a3 respectively with the best possible accuracy (i.e., big p(O2|O1) and151 p(O3|O1,O2)) under the condition of preceding agents’ precise analysis.152 The sequential MARL framework reduces the complexity of the model with six dotted arrows that153 indicate correlations between agents’ evaluations in Figure 1(b) by half, as those three dotted lines in154 Figure 1(c) show. Equ.(3) suggests that we can analyze the cooperation of a multi-agent system in155 any order, but from the CEO-Staff example, we can see that the difficulty of analyzing in various156 orders is not the same. Further discussion on the sequence will show in Section 3.4.157
In general, we specify an order to analyze the cooperation in the sequential MARL framework. We158 fix an agent’s actions after assessing it and study a particular agent based on the fixed actions of its159 preceding agents, reflecting the intuition that a CEO’s decision has a strong influence on evaluating160 the staff in the example mentioned earlier. This sequential MARL framework significantly alleviates161 the correlations in studying agents and helps us assess their cooperation more directly.162
3.3 Sequential Credit Assignment163
Following the CTDE paradigm, we utilize a centralized critic for each actor to follow a gradient164 based on an advantage function A estimated from this critic:165
g = ∇θπ log π (u|τat )A. (4)
The advantage function A for each actor explicitly deduces how that particular agent contributes to166 the team. COMA [6] introduced a counterfactual baseline inspired by difference rewards [34]. For167 each agent a, COMA computes an advantage function that compares the Q-value for the action ua to168 a counterfactual baseline that marginalizes out ua while keeping the others’ actions u−a fixed:169
AaCOMA(s,u) = Q ( s, ( ua,u−a )) − ∑ u′a πa (u′a|τa) ·Q ( s, ( u−a, u′a )) . (5)
COMA avoids expensive calculations through careful network design. However, each agent’s170 contribution deduced by COMA is still imperfect. The evaluation of ua is based on the fixed u−a171 in Equ.(5), so agent a will learn a policy that works better with u−a in this way. It ignores the joint172 actions (ua,u−a′) with u−a′ 6= u−a that may lead to unexpected results when assessing ua.173 To analyze each agent a’s contribution more objectively, we consider the influence of all joint actions174 with ua. Considering all potential action combinations, we calculate a counterfactual advantage for175 each agent’s action, derived by computing the expectation on all the actions of other agents:176
Aa(s,u) = Eu−a [ Q ( s, ( ua,u−a ))] − Eu−a [∑ u′a πa (u′a|τa) ·Q ( s, ( u−a, u′a ))] . (6)
Under our proposed sequential MARL framework, we carry out credit assignment according to a177 specific order, and there is no need to consider all the possible joint actions. After assessing agent a,178 we fix its action and evaluate agents after it based on a’s fixed action, so the following agents’ credit179 assignments do not have to compute the expectation on ua anymore.180
We now give the detailed sequential credit assignment for a team with n agents identified by181 ai(i ∈ {1, ..., n}) under one specific sequence {a1, a2, ..., an}, and it can also be concluded from the182 rest (n!− 1) orders in the same way. Here we denote uai−1a1 = [ua1 , ua2 , ..., uai−1 ] (i = 2, 3, ..., n).183 As for agent ai (i 6= 1) in the sequence, the contribution of its leading agents a1, a2, ..., ai−1 has184 been deduced. We fix the leading agents’ actions and assess agent ai’s action based on u ai−1 a1 , so185 there is no need to calculate the expectations on [ua1 , ua2 , ..., uai−1 ], simplifying Equ.(6) to:186
Aai (s,u) = ∑ u′ai+1 · · · ∑ u′an πai+1 (u′ai+1 |τai+1)· · ·πan (u′an |τan)·Q ( s, ( uaia1 , u ′ai+1 ,· · ·, u′an ))
− ∑ u′ai · · · ∑ u′an πai(u′ai |τai) · · ·πan (u′an |τan) ·Q ( s, ( uai−1a1 , u ′ai , · · · , u′an )) . (7)
Then the first agent a1’s advantage is:187 Aa1 (s,u) = ∑ u′a2 · · · ∑ u′an πa2 (u′a2 |τa2) · · ·πan (u′an |τan) ·Q (s, (ua1 , u′a2 , · · · , u′an))
− ∑ u′a1 · · · ∑ u′an πa1 (u′a1 |τa1) · · ·πan (u′an |τan) ·Q (s, (u′a1 , u′a2 , · · · , u′an)) (8)
To illustrate the effectiveness of our sequential counterfactual advantage, we conduct a simple188 but illuminating test in two common multi-agent particle environments [11], Predator-Prey and189 Cooperative Navigation. We train both methods with 5 random seeds, and agents are trained for 5000190 episodes. We provide detailed information on the environments and experiments in the Appendix. As191 shown in Figure 2, our sequential advantage functions help agents handle the task faster and better.192
Our sequential advantage for each agent achieves an additive decomposition of the total advantage193 function, which to some extent explains the soundness and superiority of our advantage over COMA’s.194 Claim 1. The proposed sequential credit assignment achieves additive advantage-decomposition.195
Proof. See Appendix A.196
Facing the same problem as COMA that those evaluations are expensive, we model the first term197 in Equ.(7) as a function fφ of (ua1 , ua2 , ..., uai , πai+1 , ..., πan) to address this issue, and the second198 term is a similar function of (ua1 , ua2 , ..., uai−1 , πai , ..., πan). Thus, we rewrite Equ.(7) as:199
Aai = fφ (s;u a1 , ua2 , ..., uai , πai+1 , ..., πan)− fφ (s;ua1 , ua2 , ..., uai−1 , πai , ..., πan) . (9)
Here fφ is a function evaluating agents’ action-policy vectors, where fφ (ua1 , ua2 , ..., uan) = Q and200 fφ (π a1 , πa2 , ..., πan) = V . We design the complete setup for SeCA, which is illustrated in Figure 3.201
Critic Learning. We train critic fφ on-policy to estimateQ, utilizing a practical variant of TD(λ) [27]202 adapted for use with deep neural networks. In particular, the critic parameter φ is updated by minibatch203 gradient descent to minimize the following loss:204
Lt(φ) = ( y (λ) t − fφ(st,ut) )2 , where y(λ)t = rt + γ ( λy (λ) t+1 + (1− λ)fφ−(st+1,ut+1) ) . (10)
We utilize a target critic fφ− [14] to improve learning stability and update φ− ← φ periodically. The205 critic learning flow is shown at the top of Figure 3(c). The input for critic training is the state s and206 the action vector u = [ u1, u2, ..., un ] denoted as v1:n.207
Policy Learning. We optimize each agent a’s policy parameter θa by maximizing the following208 objective, which contains our proposed advantage function and an entropy regularization termH:209
ga = Eτ∼π [∇θa log πa(ua|τa)Aa(s,u) +H (πa(·|τa))] , (11) where the derivative of the adaptive entropy regularization termH(πa(·|τa)) [39] with respect to the210 i-th action probability pai is given by:211
dHi := −ξ · (log pai + 1)/H(πa(·|τa)), where H(πa(·|τa)) = Eua∼πa [− log πa(ua|τa)] . (12) We share parameters among agents, and the gradient we use to train the actor shared by all agents is:212
g = Eτ∼π [∑
a (∇θa log πa(ua|τa)Aa(s,u) +H (πa(·|τa)))
] . (13)
The inputs of the centralized critic fφ to compute the advantage function are the state s and two213 action-policy vectors v1:i = [ u1, ..., ui, πi+1, ..., πn ] and v1:i−1 = [ u1, ..., ui−1, πi, ..., πn ] . The214 bottom of Figure 3(c) demonstrates the policy learning flow.215
3.4 Sequence Adjustment Through Integrated Gradients216
We apply integrated gradients to adjust the credit assignment sequence dynamically. Reviewing the217 enlightening and straightforward CEO-Staff example discussed in Section 3.2, we can evaluate the218 staff’s behavior based on the CEO’s decision, but assessing the CEO does not require much attention219 to the staff’s action. Therefore, we would analyze the CEO first and then evaluate the staff based on220 the CEO’s current action. However, this example is not generalized for two reasons: (1) There are221 often multiple agents taking the same role in a system with superior-subordinate relationships, and the222 sequence of these agents is hard to determine; (2) Not all scenarios have such superior-subordinate223 relationships. The agents often do not need to follow others’ commands in many applications.224
We generalize the CEO-Staff example to propose a universal model. Instead of focusing on the roles225 among the agents as in [31, 32], we are more interested in agents’ contributions. Although the CEO226 and the staff have a superior-subordinate relationship, they are essentially employees of an enterprise.227 The staff plays an auxiliary role and acts based on the CEO’s decision. The staff’s work is meaningful228 only if the CEO’s decision is correct. Therefore, we often intuitively assume that an enterprise’s229 leader is paid more and contributes more. Based on this, we transform the roles of the CEO and staff230 into employees with different contributions to the enterprise. In the sequential MARL framework, we231 first assign credit to the agent with a higher contribution to the team.232
The attribution method is a powerful way to determine the influence of input features’ each component233 on the network output value [2]. Among them, integrated gradients [25] leverages path integral to234 aggregate gradients along the inputs that fall on the lines between the baseline and the input, which235 is a natural tool for measuring each agent’s contribution. QPD [35] utilizes the integrated gradient236 attribution technique to decompose shared rewards along trajectory paths, revealing how much each237 agent’s observation and action contributes to the global Q value. However, it remains unclear whether238 individual Q value should be linearly correlated to or approximated by the agent’s contribution, as in239 the case of QPD. The proper connection between agents’ contributions and their individual Q values240 in a cooperative team is worth well studied for the community.241
Here we avoid detailed analysis on the relationship between agents’ contributions and their individual242 rewards. Instead, we use integrated gradients to measure agents’ contributions to the state transition243 and adjust the credit assignment sequence based on their contributions. In particular, we estimate244 agent a’s contribution ca in the trajectory path τ t2t1 from time t1 to t2 based on its policy vector π a:245
ca = ∑ xj∈πa PathIG τ t2 t1 j (π a), (14)
where xj is j-th dimension of the policy vector πa. The computation for PathIG is shown in Equ.(1).246 We compute each agent’s contribution c to the state transition from st1 to st2 and analyze the agent247 with higher c first. We further study the adjustment frequency and its effectiveness in Section 4.2248
4 Experiments and Analysis249
4.1 Experimental Setup250
We consider a challenging set of cooperative StarCraft II maps from the SMAC benchmark [23]251 classified as Easy, Hard, and Super Hard scenarios according to the baseline algorithms’ performance.252 The inherent differences among various methods and their training procedure (e.g., on/off-policy253 learning for value-based/policy-based methods) bring difficulties when comparing methods in a254 reasonably fair manner without introducing additional components (e.g., importance sampling [13, 33]255 for off-policy methods). To attribute any poor performance of policy-based methods to potential256 algorithmic limitations or poor training conditions (in particular, high variance due to small batch257 sizes or insufficient gradient steps), we follow [5, 39], training all methods with 32 parallel runners258 to generate trajectories and using batches of 32 episodes. We evaluate each method every 320K259 steps with 32 episodes and report the 1st, median, and 3rd quartile win rates across 5 random seeds.260 Detailed information about the scenarios and the experimental setup is shown in the Appendix.261
4.2 Ablation Studies262
We first carry out ablation experiments on a Super Hard map MMM2 to validate key elements of SeCA.263
Proposed Advantage and Architecture. In Section 3.3, we compare our sequential advantage264 with COMA’s in two simple multi-agent particle environments and show our superiority in Figure 2.265 Afterward, we introduce a fφ approximation and a corresponding network architecture. Here we apply266 the same approximation and architecture for COMA’s counterfactual advantage (COMA-newArchi)267 and compare it with the original COMA and our method SeCA to show the effects of our advantage268 function, approximation, and network architecture. The result is illustrated in Figure 4(a). COMA269 performs poorly on this Super Hard map but acquires significant improvement with our approximation270 and architecture. Our sequential advantage further accelerates and stabilizes the training.271
Sequence Adjustment Algorithm. SeCA’s credit assignment sequence is dynamic. We compare272 our method with some intuitive adjustments to validate its effects. One could first evaluate agents273 with higher current-action probability (SeCA-Prob) or lower policy entropy (SeCA-Entro), as these274 agents are more confident in their acts, and we can assess other agents based on their behaviors. Since275 SeCA-Prob and Entro get a new order at each step, to be fair, we set the path length in Equ.(14) to one,276 i.e., consider agents’ contributions based on the transition from st to st+1 (SeCA-IG-1). Figure 4(b)277 illustrates that SeCA-Prob and Entro learn better than the fixed method (SeCA-Fixed), but Prob has a278 larger variance than Entro. Fixed is better than expected, which we believe is because that the fixed279 sequence acquires adequate training. Our integrated-gradients-adjustment performs the best in win280 rates and stability, and the others have inferior performance and incredibly high variance.281
Sequence Adjustment Frequency. We next consider how the sequence adjustment frequency in282 SeCA-IG affects the performance. Except per step adjustment (i.e., SeCA-IG-1), one could also283 update the sequence after a stage or an episode. If we change the credit assignment order for every284 episode during training (SeCA-IG-episode), then τ t2t1 in Equ.(14) represents a whole episode. As for285 stage adjustment, it is hard to define a stage in these tasks, and the stage length varies in diverse maps.286 Here we set stage length to 10 and 20, respectively denoted as SeCA-IG-10 and SeCA-IG-20. As the287 results in Figure 4(c) show, IG-1 and IG-episode have similar final win rates. However, IG-episode288 converges more quickly with smaller variance. The reason for IG-10(20)’s mediocre performance289 and high variance may be because the stage length needs to be dynamically adjusted. Inappropriate290 adjustment frequency fails to adapt to the stage changes in the task and causes insufficient training291 for each sequence. We utilized SeCA-IG-episode in other experiments and will investigate dynamic292 stage learning in the future to improve stage adjustment.293
4.3 Comparisons with State-of-the-arts294
We compare SeCA with some competitive algorithms, including the representative explicit credit295 assignment method COMA, the policy-based implicit method LICA, the common-used baseline296 QMIX and QTRAN. Methods are evaluated on 6 scenarios, including 2 Easy ones (2s3z, 1c3s5z),297 2 Hard ones (2c_vs_64zg, 3s_vs_5z), and 2 Super Hard ones (MMM2, 3s5z_vs_3s6z). We train298 all methods for 32 million steps in Easy maps and 64 million steps in Hard and Super Hard maps.299 These scenarios involve homogeneous and heterogeneous teams, symmetric and asymmetric battles,300 allowing a holistic study on all methods. Our experiments are based on the latest PyMARL [23]301
utilizing SC2.4.10. Performance is not always comparable between versions, so the results may be302 subtly different from the original papers.303
As we can see in Figure 5, SeCA demonstrates its robustness by achieving good performances in304 scenarios with various characteristics. All methods except COMA and QTRAN solve two Easy305 scenarios, and SeCA performs better in convergence speed and stability. SeCA’s advantage is further306 extended in the Hard map 2c_vs_64zg, and it converges significantly faster than other methods.307 Although classified only as Hard, 3s_vs_5z invalidates most algorithms except QMIX and SeCA, as308 Stalkers have to learn dispersing and making enemies give chase while maintaining enough distance309 ("kiting" technique) in this map. SeCA has a higher variance than QMIX. This is possibly because310 the Stalkers’ scattering prioritizes individual performance over cooperation which is more in line311 with QMIX’s monotonicity constraint. Nevertheless, SeCA’s performance improvements on the312 Super Hard scenarios MMM2 and 3s5z_vs_3s6z demonstrate the effectiveness of our method. LICA’s313 performance in 3s5z_vs_3s6z here is different from the original paper, as the original results for314 this map are obtained by using a different entropy coefficient, which is explained in its open-source315 implementation.1 This parameter tuning is unfair when comparing methods, so all experiments in this316 paper use the fixed entropy coefficient. We also visualize the learned sequences in different battles of317 3s_vs_5z to provide insights into our sequence adjustment in the Appendix.318
We are supposed to compare our method with QPD that also utilizes integrated gradients to show319 our improvement. However, QPD modifies the original SMAC environment to acquire additional320 information for policy training, which is mentioned in its open-source implementation.2 Therefore, it321 is unfair to compare QPD’s learning curves in the modified environment with other methods, and322 QPD’s authors did not provide methods’ learning curves comparison in the original paper. We follow323 them, providing a win rate table in the Appendix to show our superiority over QPD.324
5 Conclusions and Future Work325
This paper presents SeCA, a cooperative MARL framework with sequential credit assignment.326 SeCA computes counterfactual advantage functions to evaluate each agent based on the actions of327 the preceding agents under a specific sequence. The sequence is adjusted dynamically according328 to agents’ contributions to the team deduced by integrated gradients. SeCA accelerates policy329 convergence and improves the final performance over existing recognized methods in practice. In the330 future, we will further investigate stage learning in an episode and adjust the sequence per stage to331 improve SeCA and achieve adaptive cooperation in various task situations.332
1https://github.com/mzho7212/LICA 2https://github.com/QPD-NeurIPS2019/QPD
References333 [1] Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding334 of gradient-based attribution methods for deep neural networks. In International Conference on335 Learning Representations, 2018.336
[2] Guillem Brasó Andilla. Attribution methods for deep convolutional networks.337
[3] Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in338 the study of distributed multi-agent coordination. IEEE Transactions on Industrial informatics,339 9(1):427–438, 2012.340
[4] Yu-han Chang, Tracey Ho, and Leslie Kaelbling. All learning is local: Multi-agent learning in341 global reward games. In Advances in Neural Information Processing Systems, pages 808–814,342 2004.343
[5] Yali Du, Lei Han, Meng Fang, Ji Liu, Tianhong Dai, and Dacheng Tao. Liir: Learning individual344 intrinsic reward in multi-agent reinforcement learning. In Advances in Neural Information345 Processing Systems, pages 4403–4414, 2019.346
[6] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson.347 Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence,348 pages 2974–2982, 2018.349
[7] Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control350 using deep reinforcement learning. In International Conference on Autonomous Agents and351 Multiagent Systems, pages 66–83, 2017.352
[8] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. A survey and critique of multiagent353 deep reinforcement learning. Autonomous Agents and Multi-Agent Systems, 33(6):750–797,354 2019.355
[9] Maximilian Hüttenrauch, Adrian Šošić, and Gerhard Neumann. Guided deep reinforcement356 learning for swarm systems. In AAMAS Autonomous Robots and Multirobot Systems (ARMS)357 Workshop, 2017.358
[10] Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for359 decentralized planning. Neurocomputing, 190:82–94, 2016.360
[11] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent361 actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information362 Processing Systems, pages 6382–6393, 2017.363
[12] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-364 agent variational exploration. In Advances in Neural Information Processing Systems, pages365 7611–7622, 2019.366
[13] A Rupam Mahmood, Hado van Hasselt, and Richard S Sutton. Weighted importance sampling367 for off-policy learning with linear function approximation. In Advances in Neural Information368 Processing Systems, pages 3014–3022, 2014.369
[14] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G370 Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.371 Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.372
[15] Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for interpreting and373 understanding deep neural networks. Digital Signal Processing, 73:1–15, 2018.374
[16] Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs.375 Springer, 2016.376
[17] Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value377 functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289–353,378 2008.379
[18] Afshin OroojlooyJadid and Davood Hajinezhad. A review of cooperative multi-agent deep380 reinforcement learning. arXiv preprint arXiv:1908.03963, 2019.381
[19] Gregory Palmer, Karl Tuyls, Daan Bloembergen, and Rahul Savani. Lenient multi-agent deep382 reinforcement learning. In International Conference on Autonomous Agents and MultiAgent383 Systems, pages 443–451, 2018.384
[20] Scott Proper and Kagan Tumer. Modeling difference rewards for multiagent learning. In385 International Conference on Autonomous Agents and Multi-Agent Systems, pages 1397–1398,386 2012.387
[21] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding388 monotonic value function factorisation. In Advances in Neural Information Processing Systems,389 pages 10199–10210, 2020.390
[22] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster,391 and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent392 reinforcement learning. In International Conference on Machine Learning, pages 4295–4304,393 2018.394
[23] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas395 Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon396 Whiteson. The starcraft multi-agent challenge. CoRR, abs/1902.04043, 2019.397
[24] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran:398 Learning to factorize with transformation for cooperative multi-agent reinforcement learning.399 In International Conference on Machine Learning, pages 5887–5896, 2019.400
[25] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In401 International Conference on Machine Learning, pages 3319–3328, 2017.402
[26] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi,403 Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-404 decomposition networks for cooperative multi-agent learning based on team reward. In In-405 ternational Conference on Autonomous Agents and MultiAgent Systems, pages 2085–2087,406 2018.407
[27] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning,408 3(1):9–44, 1988.409
[28] Kagan Tumer and Adrian Agogino. Distributed agent-based air traffic flow management. In410 International Joint Conference on Autonomous Agents and Multiagent Systems, pages 1–8,411 2007.412
[29] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling413 multi-agent q-learning. arXiv preprint arXiv:2008.01062, 2020.414
[30] Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu. Shapley q-value: A local reward415 approach to solve global reward games. In AAAI Conference on Artificial Intelligence, pages416 7285–7292, 2020.417
[31] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforce-418 ment learning with emergent roles. In International Conference on Machine Learning, pages419 9876–9886, 2020.420
[32] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang.421 Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523, 2020.422
[33] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu,423 and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint424 arXiv:1611.01224, 2016.425
[34] David H Wolpert and Kagan Tumer. Optimal payoff functions for members of collectives. In426 Modeling complexity in economic and social systems, pages 355–369. World Scientific, 2002.427
[35] Yaodong Yang, Jianye Hao, Guangyong Chen, Hongyao Tang, Yingfeng Chen, Yujing Hu,428 Changjie Fan, and Zhongyu Wei. Q-value path decomposition for deep multiagent reinforcement429 learning. In International Conference on Machine Learning, pages 10706–10715, 2020.430
[36] Dayong Ye, Minjie Zhang, and Yun Yang. A multi-agent framework for packet routing in431 wireless sensor networks. Sensors, 15(5):10026–10047, 2015.432
[37] Wang Ying and Sang Dayong. Multi-agent framework for third party logistics in e-commerce.433 Expert Systems with Applications, 29(2):431–436, 2005.434
[38] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A435 selective overview of theories and algorithms. arXiv preprint arXiv:1911.10635, 2019.436
[39] Meng Zhou, Ziyu Liu, Pengwei Sui, Yixuan Li, and Yuk Ying Chung. Learning implicit437 credit assignment for cooperative multi-agent reinforcement learning. In Advances in Neural438 Information Processing Systems, 2020.439
Checklist440
1. For all authors...441
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s442 contributions and scope? [Yes]443
(b) Did you describe the limitations of your work? [Yes] We discussed it in the experiment444 analysis in Section 4.3 and future work in Section 5.445
(c) Did you discuss any potential negative societal impacts of your work? [N/A]446 (d) Have you read the ethics review guidelines and ensured that your paper conforms to447 them? [Yes]448
2. If you are including theoretical results...449
(a) Did you state the full set of assumptions of all theoretical results? [Yes]450 (b) Did you include complete proofs of all theoretical results? [Yes] We provided the proof451 of our Claim in the supplemental material.452
3. If you ran experiments...453
(a) Did you include the code, data, and instructions needed to reproduce the main experi-454 mental results (either in the supplemental material or as a URL)? [Yes] We provided455 our code and instructions in the supplemental material.456
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they457 were chosen)? [Yes] We described the training details in the supplemental material.458
(c) Did you report error bars (e.g., with respect to the random seed after running experi-459 ments multiple times)? [Yes] See Figure 2, 4 and 5.460
(d) Did you include the total amount of compute and the type of resources used (e.g., type461 of GPUs, internal cluster, or cloud provider)? [Yes] We described it in the supplemental462 material.463
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...464
(a) If your work uses existing assets, did you cite the creators? [Yes]465 (b) Did you mention the license of the assets? [Yes]466 (c) Did you include any new assets either in the supplemental material or as a URL? [Yes]467 We provided our code in the supplemental material.468 (d) Did you discuss whether and how consent was obtained from people whose data you’re469 using/curating? [N/A]470 (e) Did you discuss whether the data you are using/curating contains personally identifiable471 information or offensive content? [N/A]472
5. If you used crowdsourcing or conducted research with human subjects...473
(a) Did you include the full text of instructions given to participants and screenshots, if474 applicable? [N/A]475
(b) Did you describe any potential participant risks, with links to Institutional Review476 Board (IRB) approvals, if applicable? [N/A]477
(c) Did you include the estimated hourly wage paid to participants and the total amount478 spent on participant compensation? [N/A]479

Review summaries:
This paper focuses on the problem of multi-agent credit assignment for cooperative multi-agent reinforcement learning. The problem is framed in a leader-follower manner, so it sees the agent actions in a sequential manner. The sequence is updated dynamically using integrated gradients as a measure of agent's contribution to state transition. The paper claims that this allows them to achieve state-of-the-art results on many star-craft (SMAC) benchmark tasks.  

Base on an observation that it is often difficult to determine the contribution a particular agent’s behavior has on the team when we have not well assessed other agents’ actions, this paper proposes a sequential credit assignment method (SeCA) which deduces each agent's contribution to the team's success one by one according to a particular order. Futher, according to this order, SeCA simplifies the computation of the advantage function of COMA (specifically, the expectation over other agents' actions) by fixing the preceding agents' actions. Finally, the sequence order is dynamically adjusted according to agents' contributions to the team deduced by integrated gradients.

The authors are studying cooperative marl with global reward,  and in particular the credit assignment problem. This is considering the sequential approach where one assign credit one agent at a time by marginalizing out its action akin to difference rewards. When there is a natural sequence of importance of the agents and whose contribution depends on whose actions. They use CEO and staff as an illustrative example of when it makes sense.