Cooperative Multi-Agent Reinforcement Learning with Sequential Credit Assignment
Authors: Yifan Zang
Jinmin He
Kai Li
Lily Cao
Haobo Fu
QIANG FU
Junliang Xing
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Cooperative Multi-Agent Reinforcement Learning
Credit Assignment

Abstract: Centralized training with decentralized execution is a standard paradigm for cooperative multi-agent reinforcement learning (MARL), with credit assignment being a major challenge. In this paper, we propose a cooperative MARL method with sequential credit assignment (SeCA) that deduces each agent's contribution to the team's success one by one to learn better cooperation. We first present a sequential MARL framework, under which we introduce a new counterfactual advantage to evaluate each agent based on its preceding agents' actions in a specific sequence. As this credit assignment sequence tremendously impacts the performance, we further present a sequence adjustment algorithm utilizing integrated gradients. It dynamically modifies the sequence among agents according to their contribution to the team. SeCA employs a network which either estimates the Q value for training the centralized critic or deduces the proposed advantage of each agent for decentralized policy learning. Our method is evaluated on a challenging set of StarCraft II micromanagement tasks and achieves state-of-the-art performance. 

Cooperative Multi-Agent Reinforcement Learning with Sequential Credit Assignment
Anonymous Author(s)
Affiliation Address
email
Abstract
Centralized training with decentralized execution is standard paradigm for coop- erative multi-agent reinforcement learning (MARL), with credit assignment being a major challenge. In this paper, we propose a cooperative MARL method with sequential credit assignment (SeCA) that deduces each agent's contribution the team's ' success one by one to learn better cooperation. We first present sequential MARL framework, under which we introduce new counterfactual advantage evaluate each agent based on its preceding agents' actions in specific sequence. As this credit assignment sequence tremendously impacts the performance, we further present sequence adjustment algorithm utilizing integrated gradients. It dynamically modifies the sequence among agents according to their contribution to the team. SeCA employs network which either estimates the Q value for training the centralized critic or deduces the proposed advantage each agent for decentralized policy learning. Our method is evaluated on challenging set of StarCraft micromanagement tasks and achieves state-of-the-art performance.
10
11
12
14
Introduction
16
Cooperative multi-agent reinforcement learning (MARL) is a helpful tool in numerous applications such as robot swarm control 191. autonomous vehicle coordination [3], network routing [36], and
18
productivity optimization (37]. This kind of problem where agents learn coordinated policies to
optimize the global reward has been extensively studied in recent years 7 19 18
38
g)
20 One natural way of addressing the cooperative MARL problem is the centralized approach, which treats the team as single actor with joint action space. Although we can trivially apply single-agent reinforcement learning algorithms to such settings, usually does not scale well because the size of
21
22
23 the joint action space grows exponentially with the number of agents. Besides, it is not applicable
24 in real-world settings due to the inherent constraints on agent observability and communication. An alternative approach is to learn decentralized policies by independently training agents based 26 on their local observations, but simultaneous exploration often brings non non-stationarity that causes unstable learning and difficulties in convergence. As a result, the majority of work MARL follows the centralized training with decentralized execution (CTDE) paradigm [17 10 22 6), where decentralized policies
28
access extra state information during training.
30 crucial challenge of the CTDE paradigm in cooperative settings is to correctly deduce each agent's contribution to the team's 's success, also known as the multi-agent credit assignment problem [4] Existing methods can be classified as implicit and explicit credit assignment (39]. Previous implicit methods often deduce all agents' contributions by representing the global state-action value as aggregation of each agent's ss state-action value 126 12 21 20] and assigning the shared rewards to each agent according to the joint action at one time. In this way, these methods avoid the complex interaction analysis and instead fit these cooperation relationships by neural networks, However,
31
32
34
36
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

37 implicit methods often face limitations in expressiveness, and their extensions to continuous action spaces may require additional strategies (39]
38
39 On the other hand, recognized explicit approaches calculate difference rewards 34 against a certain reward baseline [8 20 b However, in cooperative MARL., evaluating any agent's action requires considering the actions of all agents, so it is often difficult to determine the impact a particular agent's behavior has on the team when we have not assessed other agents' actions. In other words, we can not say that a single agent's action is bad if the team receives a small reward because the shared reward is not decided only by this agent's 's behavior. Maybe its action is actually good in that state.
41
42
43
44
45
This paper presents a sequential credit assignment SeCA to evaluate individual agent actions explicitly and sequentially. Our motivation is to address the drawbacks of implicit methods that neglect the cooperation between agents or simply leave it to neural networks and further improve explicit credit assignment. In summary, we face two main challenges learn better explicit credit assignment: (1) how to alleviate the problem that it is hard to accurately deduce the contribution of one agent without previously assessing all the others' action, and (2) how to evaluate agents better in an explicit way.
46
47
48
49
50
To deal with (I), we introduce a sequential MARL framework. As mentioned above, without assessing the behaviors of other agents, we would never be able to evaluate a given agent's action accurately. However, we point out in this paper that some agents are less affected by such influences than others, and we can first assign credit to them. For instance, evaluating staff's ' action needs to take the CEO's command or action into consideration, while the former has little importance in assessing the CEO. Thus, we could evaluate the CEO first without considering the staff"s behavior and then analyze the staff based on the CEO's action. We fully consider the action coordination between agents and explicitly deduce contribution them one by one according to particular order, so as to make up for the disadvantage of implicit methods that the cooperation is only incxplicably fitted by neural networks. Intuitively, the order significantly impacts the overall performance, so we further propose 61 an algorithm to adjust the sequence dynamically through integrated gradients (25]
51
53
54
55
56
57
59
80
As for (2), we compute advantage function for each agent attribute agent contributions explicitly. 63
COMA [6] is a representative method that computes a baseline for each agent to reason about counterfactuals in which only one agent a's action changes, so its evaluation of a's ''s action is based on the joint action u of other agents. In other words, the policy gradient of COMA only encourages agent learn in the direction that benefits the team while other agents are acting u -o but the others' actions are not necessarily u when executing. Unlike COMA, we focus more on the action coordination among agents and propose new advantage under the proposed sequential framework.
65
66
67
68
69 We summarize the contributions of this paper as follows: (1) We propose a sequential MARL framework in Section 3.2 (2) Under this framework, we introduce sequential advantage function for each agent to guide their learning explicitly in Section 3.3, We further prove that the sequential (3) We present sequence 73 adjustment algorithm based on integrated gradients to modify the credit assignment order dynamically
71
72 credit assignment we proposed achieves additive divaatage-decomposee
74 in Section 3.4 This algorithm alleviates the impact caused by the sequence's 's randomness and helps
75
achieve competitive performance on a challenging set of StarCraft micromanagement tasks 23
2 Related Work
76
77 Explicit credit assignment gives valuable insights into agent actions' contributions to the shared 78 team reward and substantially promotes policy optimization. The representative method COMA (
79 utilizes a counterfactual baseline that marginalizes out single agent's action while keeping the other agents' actions fixed to calculate the advantage function. However, the advantage evaluates a single agent's action based on the other agents' current behaviors and ignores different action combinations.
80
81
82 SQDDPG (30] distributes the global reward reflecting each agent's contribution through Shapley Value. Although SQDDPG provides a theoretically justified framework, its assumption the observability and convex game makes it impractical and performs poorly in complex environments.
84
85 Implicit methods are more common way when addressing the credit assignment challenge. Among
86 them, LICA (39] is a policy-based method, which learns an end-to-end differentiable optimization
87 where it trains a hypernetwork that maps the state into a set of weights which, in turn, maps the
88 action policies into the Q estimate. On the other hand, value-based methods often represent the global state-action value an aggregation of the individual values. The value decomposition is linear

---

90
in the earlier work VDN 266, and it ignores the state information. QMIX [22 learns non -inear mixing network with the global state and maps the individual state-action values into the joint Q value estimate. Although QMIX performs well in various environments, it still faces the mixing network's monotonicity constraint limitation. QTRAN (24] further avoids the representation limitations by using linear constraints between individual utilities and the global state-action value. It guarantees optimal decentralization, but its constraints are computationally intractable, and the relaxations often lead to unsatisfied performance. QPLEX [2 decomposes Q values following the dueling structure, transferring the monotonicity condition from Q values to advantage values. QPD 35 leverages the integrated gradient attribution technique to decompose global Q values along trajectory paths based on the assumption that an agent's 's local reward is linearly correlated with its contribution to the team.
91
92
93
94
95
97
99
100
3
Methods
101
3.1
Preliminaries
102
Notations. This work considers fully cooperative multi-agent task with n agents = {1,..., [......,)] n as a Dec-POMDP [16] defined by a tuple G = (S,U, P.r, Z, 7,, The environment has a true state E S. Each agent a chooses an action uf from its action space U at each timestep I and forms a joint action u, that induces a transition in the environment according to the state
103
104
105
108
transition function P(s++1 St, uz) S x U" x - [0, 1]. The agents share the same reward function rss, u) S x nn - R, and y E [0, 1) is the discount factor. We consider partially observable scenarios in which agent a acquires its local observation 2t E Z drawn from (sg, a) :S x A - Z. history 7" E T = (Z x U)*, on which it conditions a policy TO (u"|77) (r)) T x U + [0, 1]. We denote joint quantities over agents in bold and joint quantities over agents other than a given agent a with the superscript
107
108
109
Each agent has an cetion-observation
110
112
Integrated Gradients. Many works aim to attribute the predictions of deep networks to their input features 15. As one of them, integrated gradients (25) aggregates the gradients along the inputs
113
that fall on the lines between the baseline i and the input . = (71, ..... F....., ...., TT). It explains how much one feature affects the deep network output F while changing from F(b) F( along a path between and E. Given a path function r(a) with a E [o, 1] specifying a path from baseline
114
115
116
T(0)= = b to the input (1) = I. then integrated gradients along the jth dimension is acquired by:
117
Cj (1) : L- (F(T(()) Øry(a) da, Ør,(a) da
118 where Gj represents 2y's contribution to the difference between baseline prediction F(b) and F(T). 119 In this work, we leverage the integrated gradients technique to dynamically adjust the order of our proposed sequential credit assignment according to each agent's 's contribution to the team.
120
3.2
Sequential MARL Framework
122
The relationship in multi-agent system is complicated, as every agent makes decisions based on the environment interfered with by the other agents. If we model each agent as a node and model the cooperations between them as edges, the cooperative rel relationship will be built as a complicated web-like graph shown in Figure I a). Evaluating the actions of any agent should take into account the behaviors of other agents in this situation. It is hard to judge whether an agent's current action is beneficial to the team when we have not evaluated other agents' actions. If we cannot determine an analysis order, we can only analyze all the agents implicitly most existing methods did, and the cooperation is often fitted only by deep neural networks, leading to unsatisfactory results.
123
124
125
126
127
129
130
This section presents a sequential framework for cooperative MARL, which aims to analyze agents' actions one by one. Our key assumption is that evaluations of some agents in team are less affected than others. Thus we can study these less-affected agents first and then analyze the others based on
131
132
135
the actions of these already-studied agents. For instance, when evaluating a staff's action, the CEO's
decision plays a vital role because we have to judge whether the staff obeys the command or not. On
135
the contrary, the staff intuitively has little impact on evaluating the CEO's decision. assessing the
CEO, often consider external factors such as market situation, modeled as state in MARL.
3

---

01
Z Ug
Q
Q
Q
M2
My
0₂
a2
Us
M2
Ms
(a)
(b)
(c)
Figure 1: A toy example with three agents. (a) Agents affect each other they choose actions based on the state interfered with by the others' actions. the other agents' assessments in the original MARL framework. Agent's cooperation analyses interrelated. (c) Each agent's s cooperation study in the proposed sequential MARL framework. Dotted arrows representing correlations decrease from 6 in (b) to in (c), reducing the complexity by half. This merit also holds for systems with other numbers of agents.
(b) The study on one agent will influence all
We introduce variable O₂ to help model this sequential MARL framework. This additional variable represents a random event that our cooperation study (e.g.. credit assignment) on agent a; is optimal or precise. Then the probability p(O,) denotes the accuracy of our research on agent dg. For illustration and understanding convenience, we discuss a simple multi-agent system with three agents as an example, in which agents are identified by a; (i {1, 2, 3}). In original MARL, the evaluation of agent 0; will influence all the other agents' assessments. Thus events O, O₂ and Og are mutually dependent, as shown in Figure (bb). We calculate the probability of studying the system accurately
138
140
142
143
144
by computing conditional probabilities:
p(O,, O2, O₂) - p((0) p(02)01)) -(00)0)))(00000000000
(2a)
=(0s)) p(22)0,) P(00100))((0110000000
(2b)
145
where (O;) denotes the probability of agent aj's accurate analysis under the condition of conducting precise study on agent a, It also indicates the accuracy of aj 's analysis conditions on precisely assess az. We then conclude that:
148
147
p(O,, O2, O3) p(O,) P(0,)0,) (000)10))-P0O0O..0000
where i,j, k E (1,2,3), j,
i.j.
149
We take Equ. 2ap as an example. To study the cooperation of this multi-agent system precisely (i.e.. big 10 O2, Oz)), we can first analyze
150
accurately as possible (i.e., big p(Or)) and then go on to investigate a2 and 03 respectively with the best possible accuracy (i.e., big p(O))O;; and
151
152
pl O0 O2)) under the condition of preceding agents' precise analysis.
153
The sequential MARL framework reduces the complexity of the model with six dotted arrows that indicate correlations between agents' evaluations in Figure by half, as those three dotted lines in Figure Tee) show. Equ.(3) suggests that can analyze the cooperation of multi-agent system in any order, but from the CEO-Staff examplc, we can see that the difficulty of analyzing in various orders is not the same. Further discussion on the sequence will show in Section 3.4
154
155
156
157
158 In general, we specify an order to analyze the cooperation in the sequential MARL framework. We fix agent's 's actions after assessing it and study a particular agent based the fixed actions of its preceding agents, reflecting the intuition that a CEO's ' decision has a strong influence on evaluating the staff in the example mentioned earlier. This sequential MARL framework significantly alleviates the correlations in studying agents and helps us assess their cooperation more directly.
160
162
163
3.3
Sequential Credit Assignment
184
Following the CTDE paradigm, we utilize a centralized critic for each actor to follow a gradient based on an advantage function A estimated from this critie:
185
g V log (ul 7d)=") A.
(4)

---

Predator-Pree Prey
Cooperative Navigation
1:00
COMA
Mean Reward Per Episodee
Steps Per Eppoond
1.20
E00
-12
COMA
1900
2000
500C
4000
1000
2000
3000
4500
Training Episodes
Training Episodes
Figure 2: Performances between COMA's counterfactual advantage and ours in two environments. (Left) Predator- Predator-Prey. Three predators cooperate to chase a faster prey that acts randomly in an area containing two obstacles. The game terminates when a predator captures the prey, and then shared reward is given. The predators trained by our advantage capture the prey faster. (Right) Cooperative Navigation initializes three agents and three landmarks with random locations. Agents cooperate to cover all the landmarks, and the shared reward is the negative sum of displacements between each landmark and its nearest agent. Our method helps the team gain bigger rewards than COMA.
186 The advantage function A for each actor explicitly deduces how that particular agent contributes to the team. COMA (6] introduced a counterfactual baseline inspired by difference rewards (34). For each agent a, COMA computes an ad advantage function that compares the Q-value for the action u" to counterfactual bascline that marginalizes out ua while keeping the others' actions fixed:
168
AComs u) Q (s, (u", u u)))) Σ T ("
10 (7 Q (s, (u- n)"))
(5)
COMA avoids expensive calculations through careful network design. However, each agent's contribution deduced by COMA is still imperfect. The evaluation of u based on the fixed u ue in Equ. so agent will learn a policy that works better with u in this way. It ignores the joint actions (u", (a))) with u a fu that may lead to unexpected results when assessing ua.
171
172
173
174
To analyze each agent a's contribution more objectively, we consider the influence of all joint actions with u². Considering all potential action combinations, we calculate a counterfactual advantage for each agent's 's action, derived by computing the expectation on all the actions of other agents:
175
178
(s, u) Fn [Q (s, (u
9)]] F
Σ n (u(4179) Q (s, n x99)
(6)
Under our proposed sequential MARL framework, we carry out credit assignment according to specific order, and there is no need to consider all the possible joint actions. After assessing agent a, we fix its action and evaluate agents after it based on a's fixed action, so the following agents' credit assignments do not have to compute the expectation on ua anymore.
178
180
181
We now give the detailed sequential credit assignment for a team with n agents identified by ai E (1, ..., n)) under one specific sequence {a1, 02 ... an), and can also be concluded from the af rest (n! - 1) orders in the same way. Here we denote us = [u"1, "e,,...,utt111 (i 2,3...., n.)
182
183
As for agent a; (i * 1) in the sequence, the contribution of its leading agents a a2 .... a; has been deduced. We fix the leading agents' actions and assess agent a; 's action based on uai so there is no need to calculate the expectations on [u=), " ,..,, u"="), simplifying Equ.
185
186
A
(s
Σ a fe111
n o (u (04 (7 ... ..77% (u (00) Q(s, (us; u 10.11 u u(x))
(1d (00)) .. ann (nl (n Q (s. (u da u (m)) (7)
187
Then the first agent 01 advantage is:
A (s, u) Σ
T (2/02/7 (-02 - (m (aa)) Q (s, (u u0 u maa)))
T (u (r T ( Q (m" n0 u ma)) (8)
5

---

92
fcr learning
①
Q
Critic
MLP
MLP
MLP
MLP
Critic
for Policy learning
MF
W
state
Input vector Ve
Critic
2
A
AP-1
ME
A'
=f
MI.F
GRU
MLP
(+(x))
Cr"C (xt)
x (-1t"))
x((-<x) (=(z))
x (af)
Critic
Actor
Actor
Actor
Actor
Actor
polcy
(b)
Figure 3: (a) A centralized mixing critic network that maps the state into a set of weights (top) and the decentralized agent network structure (bottom). (b) The overall SeCA architecture. (c) Critic learning (top) and policy learning (bottom) flow. View color if possible for better understanding.
188
To illustrate the effectiveness of our sequential counterfactual advantage, we conduct a simple but illuminating test in two common multi-agent particle environments [11, Predator-Prey and Cooperative Navigation. We train both methods with 5 random seeds, and agents are trained for 5000 episodes. We provide detailed information the environments and experiments in the Appendix. As shown in Figure 2
189
190
192
our sequential advantage functions help agents handle the task faster and better.
193
Our sequential advantage for each agent achieves an additive decomposition of the total advantage function, which to some extent explains the soundness and superiority of our advantage over COMA's.
194
195
Claim 1. The proposed sequential credit assignment achieves additive advantage denptppedecoooooiiiionn
196
Proof. See Appendix
197
Facing the same problem as COMA that those evaluations are expensive, we model the first term in Equ. as function fo of (u u² udi " Th, address this issue, and the second term is a similar function of (u w02 ui 2= on Thus, we rewrite Equ. as:
198
A = (s; ut u2 ...., u
TT") fo (s; u 22 ..... u 7:
T"x)
200
Here fo a function evaluating agents' action-policy vectors, where fo (w u ..... u²") Q and fø (1 T2 xn)) V. We design the complete setup for SeCA, which is illustrated in Figure 3
Critic Learning. We train critic fe on-policy estimate Q, utilizing practical variant of TD A) adapted for use with deep neural networks. In particular, the critic parameter ø is updated by minibatch gradient descent minimize the following loss:
203
204
(2) - Sa(se: u.) (e) = -(((
where yi (A) = r. +7 (AuLA) + ((--)/e A)So (#141 (s141, (10)
We utilize target critic fø- 14 to improve learning stability and update x- <- & periodically. The critic learning flow is shown at the top of Figure c).. The input for critic training is the state s and
206
the action vector = [u' u2 u u²] denoted as ylm.
207
208
Policy Learning. We optimize each agent a's policy parameter 0, by maximizing the following objective, which contains our proposed advantage function and an entropy regularization term H:
209
g" = Vo. log n " ( u) + H (T (-17")),
(11)
210
where the derivative of the adaptive entropy regularization term (1 39 with respect the i-th action probability pt is given by:
211
dH, == - (log + (T (-Ir"), where H ((T"(+7")) = E [- log n (u"| [r"]. (12)
We share parameters among agents, and the gradient use to train the actor shared by all agents is:
g = Exax Σ (Ve. log T" ("' + (M (+179)))
(13)

---

The inputs of the centralized critic f& to compute the advantage function are the state s and two 213
214 action-polic -plicyy vectors y = [u',... u', n' ,..., -" and vl:i-1 wu' .. u u'-
IT ..... π The
215
bottom of Figure ( demonstrates the policy learning flow.
3.4
Sequence Adjustment Through Integrated Gradients
We apply integrated gradients to adjust the credit assignment sequence dynamically. Reviewing the enlightening and straightforward CEO-Staff example discussed in Section 3.2] we can evaluate the staff's 's behavior based on the CEO's decision, but assessing the CEO does not require much attention 219
217
to the staff's action. Therefore, we would analyze the CEO first and then evaluate the staff based on
221
the CEO's current action. However, this example is not generalized for two reasons: (1) There are
222
often multiple agents taking the same role in system with superior-subordinal relationships, and the
223
sequence of these agents is hard to determine; (2) Not all scenarios have such superior-subordinat relationships. The agents often do not need to follow others' commands in many applications.
224
225
We generalize the CEO-Staff example to propose a universal model. Instead of focusing on the roles among the agents as in 31 32,, we are more interested in agents' contributions. Although the CEO and the staff have a The staff plays an auxiliary role and acts based the CEO's decision. The staff's work is meaningful only if the CEO's decision is correct. Therefore, we often intuitively assume that an enterprise's s leader is paid more and contributes more. Based on this, we transform the roles of the CEO and staff
227
relationship, they are essentially employees of an enterprise.
228
230
231
into employees with different contributions to the enterprise. In the sequential MARL framework, we first assign credit the agent with higher contribution the team.
232
233
The attribution method is a powerful way to determine the influence of input features' each component on the network output value 22] Among them, integrated gradients [25] leverages path integral to aggregate gradients along the inputs that fall on the lines between the baseline and the input, which natural tool for measuring each agent's 's contribution. QPD (35 utilizes the integrated gradient attribution technique to decompose shared rewards along trajectory paths, revealing how much each agent's 's observation and action contributes to the global Q value. However, it remains unclear whether individual Q value should be linearly correlated to or approximated by the agent's 's contribution, as in the case of QPD. The proper connection between agents' contributions and their individual Q values in a cooperative team is worth well studied for the community.
235
237
239
241
Here we avoid detailed analysis on the relationship between agents' contributions and their individual rewards. Instead, we use integrated gradients to measure agents' contributions to the state transition and adjust the credit assignment sequence based their contributions. In particular, we estimate agent a's contribution ca in the trajectory path f from time t1 to t2 based on its policy vector n":
243
245
Σ E
PathIG "I (TT),
(14)
where zj is j-th dimension of the policy vector π.. The computation for PathIG is shown in Equ. ( We compute each agent's 's contribution c to the state transition from st, to 5t, and analyze the agent with higher first. We further study the adjustment frequency and its effectiveness in Section 4.2
247
4
Experiments and Analysis
250
4.1
Experimental Setup
251
We consider a challenging set of cooperative StarCraft II maps from the SMAC benchmark [23 classified as Easy, Hard, and Super Hard scenarios according the baseline algorithms' performance.
252
253
The inherent differences among various methods and their training procedure (c.g., on/off-policy learning for galue-based/policy--iiee
254
methods) bring difficulties when comparing methods in reasonably fair manner without introducing additional components (e.g., importance sampling [13 [3
255
256
for off-policy methods). To attribute any poor performance of posicy-based methods to potential algorithmic limitations or poor training conditions (in particular, high variance due to small batch
257
sizes or insufficient gradient steps), we follow 39 training all methods with 32 parallel runners 258
to generate trajectories and using batches of 32 episodes. We evaluate each method every 320K
260
steps with 32 episodes and report the 1st, median, and 3rd quartile win rates across 5 random seeds. Detailed information about the scenarios and the experimental setup is shown in the Appendix.
7

---

Ablation on Advantage & Architecture
Ablation on Sequence Adjusteneus Algurithmms
Ablution on Adjestment Froquency
103
100
Sar4 FF
DOXChertahee
00055 wadldd
SCHDD
(Dam))
19
20 Steps (x:10))
20
50
(4)
(b) Stops (x:IPP
Steps (xclot)
Figure 4: Ablations for SeCA's key elements scenario MMM2 (Super Hard). (a) investigates the effects of our sequential advantage and network architecture. (b) validates our sequence adjustment through integrated gradients. (c) shows the test win percentage with various adjustment frequencies.
282
4.2
Ablation Studies
We first carry out ablation experiments on a Super Hard map MMM2 to validate key elements of SeCA.
264
Proposed Advantage and Architecture. In Section 3.3 we compare our sequential advantage with COMA's in two simple multi-agent particle environments and show our superiority in Figure 2 Afterward, we introduce fa approximation and a corresponding network architecture. Here we apply the same approximation and architecture for COMA's counterfactual advantage (cMM--newArroe and compare it with the original COMA and our method ScCA to show the effects of our advantage function, approximation, and network architecture. The result is illustrated in Figure 4a). COMA performs poorly on this Super Hard map but acquires significant improvement with our approximation and architecture. Our sequential advantage further accelerates and stabilizes the training.
266
268
270
272
Sequence Adjustment Algorithm. SeCA's credit assignment sequence dynamic. We compare our method with some intuitive adjustments to validate its effects. One could first evaluate agents with higher current-action probability (SeCA-Prot (rob)) or lower policy entropy (SeCA-Fnnro), as these agents are more confident in their acts, and we can assess other agents based on their behaviors. Since SeCA-Prob and Entro get a new order at each step, to fair, we set the path length Equ.(T4) to one, i.e. consider agents' contributions based on the transition from St to 8 (SeCA-IG-1). Figure (b) illustrates that SeCA-Prob and Entro learn better than the fixed method (SeCA-Fixed), but Prob has a larger variance than Entro. Fixed is better than expected, which we believe is because that the fixed sequence acquires adequate training. Our ineegratd-grrdiene gradients-adjustmen adjustment performs the best in win rates and stability, and the others have inferior performance and incredibly high variance.
274
276
278
280
282
Sequence Adjustment Frequency. We next consider how the sequence adjustment frequency in SeCA-IG affects the performance. Except per step adjustment (i.e., SeCA-IG-1), one could also update the sequence after a stage or an episode. If we change the credit assignment order for every episode during training (SeCA-IG-episode then T/ in Equ. represents a whole episode. As for stage adjustment, it is hard to define a stage in these tasks, and the stage length varies in diverse maps. Here we set stage length to 10 and 20, respectively denoted as SeCA-IG-10 10 and SeCA-IG-20. As the results in Figure 4 c) show, IG-1 and IG-episode have similar final win rates. However, IG-episode converges more quickly with smaller variance. The reason for IG- I0-10('')'s mediocre performance and high variance may be because the stage length needs to be dynamically adjusted. Inappropriate adjustment frequency fails to adapt to the stage changes in the task and causes insufficient training for each sequence. We utilized SeCA-IG-episode in other experiments and will investigate dynamic stage learning in the future to improve stage adjustment.
284
286
286
290
282
283
4.3
Comparisons with State-of-the-ars
295 We compare SeCA with some competitive algorithms, including the representative explicit credit assignment method COMA, the policy-based implicit method LICA, the common -ueed baseline
296
297
QMIX and QTRAN. Methods are evaluated on 6 scenarios, including 2 Easy ones (2s32, 1c3s52),
298
2 Hard ones (2c_vs_66 vs__4zg, 3s_vs_ _s_2,, and 2 Super Hard ones (MMM2, 3s5z_ve vs v__3s6z). We train
all methods for 32 million steps in Easy maps and 64 million steps in Hard and Super Hard maps.
300
These scenarios involve homogeneous and heterogeneous teams, symmetric and asymmetric battles, allowing holistic study all methods. Our experiments are based on the latest PyMARL 23
8

---

2nde (Easy]
Lehobe (Baey)
De. Vs. Be.. (Hard)
10:
100
COML
Steps (xII))
Steps (xUP)
Stope (xlif)
vs. (Hurd)
(Super Hard)
Joda (Super Hard)
100
100
QMIX
QURAN
QUTRAN
SOCA
Stepe (xtif]
Stepe (x1if)
Stepe (xiof)
Figure 5: The comparison of SeCA against various baseline algorithms on six SMAC maps.
utilizing SC2.4.10. Performance is not always comparable between versions, so the results may be
303
subtly different from the original papers.
304
As we can see in Figure 5 SeCA demonstrates its robustness by achieving good performances in scenarios with various characteristics. All methods except COMA and QTRAN solve two Easy scenarios, and SeCA performs better in convergence speed and stability. SeCA's advantage is further _0__v__644p0 ___g, and converges significantly faster than other methods. Although classified only as Hard, 3s_vs_5z invalidates most algorithms except QMIX and SeCA, as
305
308
307
extended in the Hard map 2c_ vs_
308
309
Stalkers have learn dispersing and making enemies give chase while maintaining enough distance ("kiting" technique) in this map. SeCA has higher variance than QMIX. This is possibly because Stalkers' scattering prioritizes individual performance over cooperation which more in line with QMIX's monotonicity constraint. Nevertheless, SeCA's performance improvements on the Super Hard scenarios MMM2 and 3s5z_ve. __662 demonstrate the effectiveness of our method. LICA's performance in 3s5z_vs. _s_666 here is different from the original paper, as the original results for this map are obtained by using a different entropy coefficient, which is explained in its open-source implementation This parameter tuning is unfair when comparing methods, all experiments in this paper use the fixed entropy coefficient. We also visualize the learned sequences in different battles of 3s_ as_vv___ vs_5z to provide insights into sequence adjustment in the Appendix.
311
313
315
317
319
We are supposed to compare our method with QPD that also utilizes integrated gradients show our improvement. However, QPD modifies the original SMAC environment to acquire additional information for policy training, which mentioned its open-source implementation Therefore, is unfair to compare QPD's learning curves in the modified environment with other methods, and QPD's s authors did not provide methods' learning curves comparison in the original paper. We follow them, providing win rate table in the Appendix to show our superiority over QPD.
321
323
5
Conclusions and Future Work
325
326
This paper presents SeCA, a cooperative MARL framework with sequential credit assignment. SeCA computes counterfactual advantage functions evaluate each agent based the actions of the preceding agents under a specific sequence. The sequence is adjusted dynamically according agents' contributions the team deduced by integrated gradients. SeCA accelerates policy convergence and improves the final performance over existing recognized methods in practice. In the
328
329
330
331
future, we will further investigate stage learning in an episode and adjust the sequence per stage to improve SeCA and achieve adaptive cooperation in various task situations.
332

---

References
333
334
[1] Marco Ancona, Enea Ccolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In International Conference on Learning Representations, 2018.
336
337
[2] Guillem Brasó Andilla. Attribution methods for deep convolutional networks.
338
[3] Yongean Cao. Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial informatics, 9(1 (1)):477443, 2012.
339
[4] Yu-han Chang, Tracey Ho, and Leslie Kaelbling. All learning is local: Multi-agent learning in global reward games. In Advances in Neural Information Processing Systems, pages 808-814 2004.
342
344
[5] Yali Du, Lei Han, Meng Fang. Ji Liu, Tianhong Dai, and Dacheng Tao. Lir: Learning individual intrinsic reward in multi-agen reinforcement learning. In Advances in Neural Information Processing Systems, pages 4403 04144414,220 2019.
345
348
347
[6] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence, pages 2974-2982, 2018.
349
350
[7] Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pages 66-83, 2017.
352
353
[8] Pablo Hernandez-Leat Bilal Kartal, and Matthew E Taylor. A survey and critique of multiagent
354
deep reinforcement learning. Autonomous Agents and Multi-Agent Systems, (3(6):550-797, 2019.
355
[9] Maximilian Hüttennaach, Adrian Šošić, and Gerhard Neumann. Guided deep reinforcement learning for swarm systems. In AAMAS Autonomous Robots and Multirobot Systems (ARMS) Workshop, 2017.
357
[10] Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82-94, 2016.
360
361
[11] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettive environments. In Advances in Neural Information Processing Systems, pages 6382-6399 2017.
362
[12] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi- agent variational exploration. In Advances in Neural Information Processing Systems, pages 7611-7622 2019.
365
367
[13] A Rupam Mahmood, Hado van Hasselt, and Richard Sutton. Weighted importance sampling for off-policy learning with linear function approximation. In Advances in Neural Information Processing Systems, pages 3014-3023 2014.
368
389
370
[14] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51877500::59-533, 2015.
372
373
[15] Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73:1-15, 2018.
375
[16] Frans A Oliehock and Christopher Amato. concise introduction to decentralized POMDPs. Springer, 2016
378
[17] Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008.
378
10

---

380
[18] Afshin OroojlooyJadid and Davood Hajinezhad. A review of cooperative multi-agent deep reinforcement learning, arXiv preprint arXiv:1908.03965 2019.
381
382
[19] Gregory Palmer, Karl Tuyls, Daan Bloembergen, and Rahul Savani. Lenient multi-agent deep reinforcement learning. In International Conference Autonomous Agents and MultiAgent Systems, pages 443-451, 2018.
384
385
[20] Scott Proper and Kagan Tumer. Modeling difference rewards for multiagent learning. In International Conference Autonomous Agents and Multi-Agent Systems, pages 1397-1398, 2012.
387
388
[21] Tabish Rashid, Gregory Farquhar, Bei Peng. and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation. In Advances in Neural Information Processing Systems, pages 10210, 2020.
390
381
[22]
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4295-4304,
392
393
394
2018.
395
[23] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. CoRR,
386
387
2019.
398
[24] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5887-5896, 2019.
400
401
[25]
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In 2017.
402
International Conference Machine Learning, pages
[26]
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning based on team reward. In In- lernational Conference on Autonomous Agents and MultiAgent Systems, pages 2085-2087, 2018.
404
405
406
407
408
[27] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9 44, 1988.
409
410
[28] Kagan Tumer and Adrian Agogino. Distributed agent-based air traffic flow management. In International Joint Conference on Autonomous Agents and Multiagent Systems, pages 1-8, 2007.
411
412
413
[29] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent g-learning. arXiv preprint arXiv:2008. 01062, 2020.
414
415
[30] Jianhong Wang, Yuan Zhang, Kyu-Kyun Kim, and Yunjie Gu. Shapley v-value: A local reward approach to solve global reward games. In AAAI Conference on Artificial Intelligence, pages 2020.
417
418
[3]] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforce- ment learning with emergent roles. In International Conference on Machine Learning, pages 9876-9886, 2020.
419
420
421
[32] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang Rode: Learning roles decompose multi-agent tasks. arXiv preprint arXiv:2010.01523, 2020.
422
423
[33] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint 2016.
424
425
426
[34] David H Wolpert and Kagan Tumer. Optimal payoff functions for members of collectives. In Modeling complexity in economic and social systems, pages 355-369, World Scientific, 2002.
11

---

428
[35] Yaodong Yang, Jianye Hao, Guangyong Chen, Hongyao Tang, Yingfeng Chen, Yujing Hu, Changjie Fan, and Zhongyu Wei. Q-value path decomposition for deep multiagent reinforcement learning. In International Conference on Machine Learning, pages 10706-10715, 2020.
429
[36] Dayong Ye, Minjie Zhang, and Yun Yang. A multi-agent framework for packet routing in wireless sensor networks. Sensors, 15(5::1002610000
432
2015.
433
[37] Wang Ying and Sang Dayong. Multi-agen aagitt framework for third party logistics in e-commerce. Expert Systems with Applications, 29(2):431 4314433,1161111 2005.
434
435
[38] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. arXiv preprint arXiv: 2019.
437
[39] Meng Zhou, Ziyu Liu, Pengwei Sui. Yixuan Li, and Yuk Ying Chung. Learning implicit credit assignment for cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
438
439
Checklist
440
441
1. For all authors...
442
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
444
(b) Did you describe the limitations of your work? [Yes] We discussed it in the experiment analysis in Section 4.3 and future work in Section S
445
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
448
449
2. If you are including theoretical results...
450
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
451
(b) Did you include complete proofs of all theoretical results? [Yes] We provided the proof of our Claim in the supplemental material.
452
3. If you experiments...
454
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We provided our code and instructions in the supplemental material.
455
456
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We described the training details in the supplemental material.
458
459
(c) Did you report error bars (e.g.. with respect to the random seed after running experi-
460
ments multiple times)? [Yes] See Figure 2 and 5
461
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] We described it in the supplemental material.
463
464
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
465
(a) If your work uses existing assets, did you cite the creators? [Yes]
486
(b) Did you mention the license of the assets? [Yes]
457
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
468
We provided our code in the supplemental material.
469
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
472
473
5. If you used crowdsourcing or conducted research with human subjects...
474
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
12

---

478
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
477
478
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "lpHaYQ_A6VK", "reviewer": "Reviewer_8jFv", "summary": "This paper focuses on the problem of multi-agent credit assignment for cooperative multi-agent reinforcement learning. The problem is framed in a leader-follower manner, so it sees the agent actions in a sequential manner. The sequence is updated dynamically using integrated gradients as a measure of agent's contribution to state transition. The paper claims that this allows them to achieve state-of-the-art results on many star-craft (SMAC) benchmark tasks. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nThe prior of a specific ordering in the form of leader-follower definitely makes sense as a way to simplify the problem of credit assignment. Using integrated gradients to figure out the ordering dynamically is an even nicer trick. One important detail about the results that is not clarified in the paper itself is that these state of the art results are at the cost of more than an order of magnitude more samples than existing works. But it's a good exploration of how much data might be required for a lot of these methods to converge. The paper is also very clearly written (although gets a little repetitive with Boss-Staff analogies) and exploring alternative ideas for credit assignment in multi-agent reinforcement learning is quite pertinent to making these more practical.\n\n**Weaknesses**\n\nAs [1,2] shows that these many samples are not necessary to achieve the performance that is being achieved. For example, [1] achieves 80% win rate on \"3s vs 5z\" with only 2million samples, while this paper seems to require 20 times more samples to reach 40% win rate. But improved credit assignment should definitely show up as lower sample complexity, so I am worried if the differences between the algorithms are not all just because of sub-optimal hyperparameters and difference in credit assignment is immaterial (I understand that there might be version differences in SMAC, but the other works [1,2] seem recent enough that it's unlikely that it's just a matter of version difference. I am not sure if following refs[5,39] leads to the best comparisons against other algorithms). Other figures only do ablation studies or comparison against COMA. In my own experience with these algorithms, COMA tends to underperform compared to just using a centralized baseline on most tasks. Moreover the proposed approach inherits the problem from COMA of not being able to scale to teams with larger number of agents.\n\n[1] Witt et al. \"Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?\" https://arxiv.org/abs/2011.09533\n\n[2] Hu et al. \"Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning\" https://arxiv.org/abs/2102.03479\n\nEdit: Thanks to the authors for their comments. I'm still not convinced of the comparisons with off-policy methods but I'm not changing my positive impression of the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper doesn't really discuss its own limitations explicitly. Adding a section would be quite helpful. There might be [more stochastic] scenarios that the sequential ordering may not make sense and might require leveraging ideas like coordination graphs [A,B,C] instead.\n\n[A] W Böhmer et al. \"Deep Coordination Graphs\" https://arxiv.org/abs/1910.00091\n[B] S Li et al. \"Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning\" https://arxiv.org/abs/2006.11438\n[C] N Naderializadeh et al. \"Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learning\" https://arxiv.org/abs/2010.04740\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "gl2s3mymH-o", "reviewer": "Reviewer_E7Ut", "summary": "This work proposes a novel approach for addressing the challenge of credit assignment in the context of centralised training with decentralised execution (CTDE): sequential credit assignment (SeCA). The method proposes to calculate the counterfactual advantage for each agent in a sequential manner, as well as an approach based on integrated gradients to dynamically determine the order for evaluating the agents (i.e., in descending order of the agents' contributions to the state transition).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nThe work addresses an important problem in MARL, namely credit assignment. The work extends the idea of calculating counterfactual advantage values and also incorporates the integrated gradients method for a novel purpose.\n\nQuality:\nI do not agree with the presentation and some of the statements made in Section 3.2. At lines 137-138 the work introduces a random variable to model the event that the credit assignment study is precise. You then proceed to unfold the joint probability distribution according to the product rule, on the 3 agents example, which is fine. But in lines 153-155 it is stated that: \"The sequential MARL framework reduces the complexity of the model with six dotted arrows that indicate correlations between agents’ evaluations in Figure 1(b) by half, as those three dotted lines in Figure 1(c)\". This is incorrect and misleading. First of all, these are 'dependencies' and not 'correlations'. Secondly, Figure 1(c) is not correct. The top nodes are no longer just the variables O1, O2 and O3, but rather: O1, O2|O1 and O3|O2,O1, obtained from the sequential calculation of the terms in the product rule. \n\nClarity: \nThe paper is nicely written and organised. I find that perhaps the recurring CEO-Staff example takes too much space for the utility it brings to the paper. It might have been more interesting to present for example a short discussion on a testing episode (from the supplementary material). This would show on a real example how exactly the idea of agent contributions translates in practice.\n\nLines 178-179 state 'After assessing agent a, we fix its action and evaluate agents after it based on a’s fixed action', can you clarify how or to what do you fix the action of each agent during the sequential assessment.\n\nSignificance: \nI appreciate the ablation studies and comparisons with SOTA approaches. The study of the sequence adjustment frequency was interesting. I was wondering how the 10 and 20 values were selected for the stage adjustment? The experimental results show that SeCA performs well, on par or even better than the other considered approaches. \n\n----Post-rebuttal----\nI thank the authors for the diligent clarifications. I have raised my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "_fpKn_Gk8zj", "reviewer": "Reviewer_oyDk", "summary": "Base on an observation that it is often difficult to determine the contribution a particular agent’s behavior has on the team when we have not well assessed other agents’ actions, this paper proposes a sequential credit assignment method (SeCA) which deduces each agent's contribution to the team's success one by one according to a particular order. Futher, according to this order, SeCA simplifies the computation of the advantage function of COMA (specifically, the expectation over other agents' actions) by fixing the preceding agents' actions. Finally, the sequence order is dynamically adjusted according to agents' contributions to the team deduced by integrated gradients.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n$\\textbf{Pros:}$\n(1) The motivation is very clear. The paper clearly argues that a successful framework for sequential credit assignment has been a missing element of prior work on multi-agent RL. I encourage the authors to continue pursuing this study as it should make a valuable contribution.\n\n(2) The paper is well organized and clearly written. \n\n\n$\\textbf{Cons:}$\n\n$\\textbf{(1)} \\text{ [Fatal error]}$ The implementation of SeCA has fatal errors. Specifically, as illustrated in Figure 3, when computing the expectation over agents' policies of the advantage function, i.e., $A^{a_{i}}=f_{\\phi}\\left(s ; u^{a_{1}}, u^{a_{2}}, \\ldots, u^{a_{i}}, {\\color{red}\\pi^{a_{i+1}}, \\ldots, \\pi^{a_{n}}}\\right)-f_{\\phi}\\left(s ; u^{a_{1}}, u^{a_{2}}, \\ldots, u^{a_{i-1}}, {\\color{red}\\pi^{a_{i}}, \\ldots, \\pi^{a_{n}}}\\right)$ (marked in red), the concrete implementation is absolutely wrong. \n\nAfter checking the code provided in the Appendix, I found the author tries to compute the expectation over agents' policies by simply taking the probabilities of selecting all valid actions as input. Directly takeing the probabilities of selecting all valid actions (called policy vectors in the paper) as the input of the Critic neural network doesn't mean you're calculating the expected value!\n\nThis incorrect implementation will lead to wrong conclusion which will seriously affect the later studies. \n\nI could understand that the author's purpose is to improve the efficiency of the expectation calculation. Correctness, however, must come first.\n\n$\\textbf{(2)}$ Based on the analysis in (1), the experimental evaluation results are not convincing.\n\n$\\textbf{(3)}$ [About the experimental results in SMAC] The experimental evaluation in SMAC is not significant. Most recently, the paper RIIT[1] shows that QMIX could consistantly achieve SOTA performance among all SMAC maps with some code level modification only, e.g., changing optimizer, incorporating TD($\\lambda$) and enlarging exploration. So, I recommend the authors to re-implement SeCA based on pymarl-v2 (I call the optimized version of pymarl by Jian Hu in paper RIIT[1] as pymarl-v2) and show how it performs.\n\nOverall, I vote for a rejection for current version of this paper. I may modify my rating if my concerns mentioned above are addressed.\n\n\n$\\textbf{Reference:}$\n  * [1] Hu J, Jiang S, Harding S A, et al. RIIT: Rethinking the Importance of Implementation Tricks in Multi-Agent Reinforcement Learning[J]. arXiv preprint arXiv:2102.03479, 2021.\n  * [2] Christianos F, Schäfer L, Albrecht S V. Shared experience actor-critic for multi-agent reinforcement learning[J]. arXiv preprint arXiv:2006.07169, 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the main review above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "NvOCnFrIsun", "reviewer": "Reviewer_c8rv", "summary": "The authors are studying cooperative marl with global reward,  and in particular the credit assignment problem. This is considering the sequential approach where one assign credit one agent at a time by marginalizing out its action akin to difference rewards. When there is a natural sequence of importance of the agents and whose contribution depends on whose actions. They use CEO and staff as an illustrative example of when it makes sense.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The key novelty in this paper is that they adjusts the sequence based on a particular way of measuring the contribution of each agent , integrated gradients, that has previously been introduced for this purposes (decomposing global Q into ind agents) in QPD under the extra assumption of a local reward being linearly related to the contribution to the total. Here such local rewards and assumptions about them are not relied on and instead this value is used to rank the agents for sequential credit assignment.\n\nThe empirical evaluation on SCII micromanagement tasks looks promising.\n\nThe paper is combing a few existing techniques in a particular manner that feels slightly ad-hoc, though not without some intuition.\n\nIt would be interesting to know if the sequential credit assignment also reinforces the chosen ranking of contributions in some way and make the learning, in any way, such that the lower rank learns to \"consider\" the higher ranked more than the other way around.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It is not clear to me if the sequential approach can be bad for all orderings in some situations, but I would guess that such exists.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers and AC discussed the paper. There was agreement that the approach is promising for dealing with sequential credit assignment in MARL. Nevertheless, there are concerns about the fairness of the experimental results and some of the technical details (e.g., the advantage function). It is great that the authors will do (or have already done) some additional experiments, but those are not currently visible. These results are necessary to understand the performance of the approach in the context of other state-of-the-art methods. Similarly, while the authors provide additional details about the approximation of the advantage function, this approach should be better motivated and clarified in the paper. In order to properly evaluate the paper, it needs significant updating and re-review. 
Decision: Reject