Learning Explicit Credit Assignment for Multi-agent Joint Q-learning
Authors: Hangyu Mao
Jianye HAO
Dong Li
Jun Wang
Weixun Wang
Xiaotian Hao
Bin Wang
Kun Shao
Zhen Xiao
Wulong Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: Mutli-agent Credit Assignment
Mutli-agent Joint Q-learning

Abstract: Multi-agent joint Q-learning based on Centralized Training with Decentralized Execution (CTDE) has become an effective technique for multi-agent cooperation. During centralized training, these methods are essentially addressing the multi-agent credit assignment problem. However, most of the existing methods \emph{implicitly} learn the credit assignment just by ensuring that the joint Q-value satisfies the Bellman optimality equation. In contrast, we formulate an \emph{explicit} credit assignment problem where each agent gives its suggestion about how to weight individual Q-values to explicitly maximize the joint Q-value, besides guaranteeing the Bellman optimality of the joint Q-value. In this way, we can conduct credit assignment among multiple agents and along the time horizon. Theoretically, we give a gradient ascent solution for this problem. Empirically, we instantiate the core idea with deep neural networks and propose Explicit Credit Assignment joint Q-learning (ECAQ) to facilitate multi-agent cooperation in complex problems. Extensive experiments justify that ECAQ achieves interpretable credit assignment and superior performance compared to several advanced baselines.

Many real-world problems such as robot swarm control can be naturally modeled as cooperative multi-agent systems where each agent can only observe parts of the systems' state and all agents share the same global reward. Recently, the IGM-based multi-agent joint Q-learning has become means the Under review as a conference paper at ICLR 2022
LEARNING EXPLICIT CREDIT ASSIGNMENT FOR MULTI-AGENT JOINT Q-LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-agent joint Q-learning based on Centralized Training with Decentralized Execution (CTDE) has become an effective technique for multi-agent coopera- tion. During centralized training, these methods are essentially addressing the multi-agent credit assignment problem. However, most of the existing methods implicitly learn the credit assignment just by ensuring that the joint Q-value satis- fies the Bellman optimality equation. In contrast, we formulate an explicil credit assignment problem where each agent gives its suggestion about how to weight individual Q-values to explicitly maximize the joint Q-value, besides guarantee- ing the Bellman optimality of the joint Q-value. In this way, assignment among multiple agents and along the time horizon. Theoretically, we give gradient ascent solution for this problem. Empirically, core idea with deep neural networks and propose Explicit Credit Assignment joint Q-learning (ECAQ) facilitate multi-agent cooperation in complex problems. Extensive experiments justify that ECAQ achieves interpretable credit assignment and superior performance compared to several advanced baselines.
conduct credit
instantiate the
INTRODUCTION
an effective technique to solve such problems, where IGM (i.e., Individual- Settvidual-Global---ae consistency between individual and joint greedy action selections.
During training, most IGM-based methods (Rashid et al. 2018 [Sunehag et al. 2018 Wang et al., 2020b, Yang et al.) [2020a b) are essentially addressing the multi-agent eredit assignment problem as pointed out by Yang et al. (2020a); Zhou et al.) (2020). Specifically, they try to learn assign- ment function f parameterized by w to align the joint Q-value Qtotal shared by all agents with the individual Q-values Qi belonging agent i.e., Qiotal =((Q1.... QN; w). Typically, these methods mainly apply temporal difference learning (TD-learning) to extract the parameter w, which represents specific credit assignment. However, TD-leaming does not explicitly optimize the credit assignment among multiple agents at a given timestep ! and the existing methods are often called the implicil multi-agent credit assignment mentioned by ZZhou et al. (2020); Wang al. (2020a); Naderializadeh et al. (2020); Li et al. (2021b)a) a). Besides, the extracted parameter w is usually lack of interpretability in terms of multi-agent credit assignment. Finally, the performance may be poor due to unsuitable credit assignment (Zhou et al. 2020 Yang et al. (2020a).
In contrast, conducting explicit nulti-agent credit assignment could distribute the global reward to each agent based on its contribution to the agent group, thus it may substantially facilitate policy optimization and promote learning performance as pointed out by many previous methods (Proper & Tumer, 2012, Tumer & Agogino 2007; Wang et al.) 2020c). More importantly, it can figure out which agent is critical according to the assigned credits, so as to achieve better interpretabilityy which makes up for the defect that deep neural networks are unexplainable. Inspired by these, investigate explicit multi-agent credit assignment for the IGM-based joint Q-learning methods.
In general, TD-learning is considered to explicitly assign the credit along the time horizon, namely, dis- trihuting the future credit (i.c., the delayed reward) to previous timesteps.

---

Under review as a conference paper at ICLR 2022
Our contributions are three-fold. First, we propose a criterion measure assignment function so that we can find better assignments by explicitly optimizing this criterion. Specifically, a good as- signment function should be helpful for agent cooperation to maximize the reward, so we define the criterion as the maximization of Qottal (i.e., the expected long-term cumulative reward) Second, introduce exact solution optimize the defined criterion. Theoretically, our solution can find the optimal Qrotol with mild conditions. Empirically, we approximate the core idea with deep neural networks and propose Explicit Credit Assignment joint Q-learning (ECAQ) facilitate multi-agent cooperation in complex scenarios. Third, we evaluate ECAQ on several challenging tasks. The results demonstrate that ECAQ achieves interpretable credit assignment and superior performance compared to advanced baselines.
2 BACKGROUND
DEC-POMDP. We consider a fully cooperative multi-agent setting that can be formulated as
DEC-POMDP (Bernstein et al. 2002). It is formally defined tuple (N, S, A, T, R, O, 7) where N the number agents; S is the set of state; A = A x ... x AN represents the set of joint action, and A, is the set of local action that agent can take; T a) S x A x S [0.
represents the state transition function; R S x A + R is the reward function; O = [O ON] is
the set of joint observation controlled by the observation function Z S x A O; and y E [0,1] is the discount factor.
In given state each agent i generates an action a; based its observation 0₁- The joint action a = (a,, am results in a new state s' (i.e., the state of the next timestep) and a global reward r, where a is the joint action of teammates of agent i. The agent aims at learning policy ma(a;) /o;) that can maximize Eo,- [G] where G the discount return defined as G = Σn' = 7ep² and H is the time horizon. In partially observable scenarios, the action is typically generated based on the entire observation-ace history Tú, i.e., (T(a: [T)., rather than the current observation O,.
Multi-agent Joint Q-learning. The multi-agent joint Q-learning is a notable approach to solve DEC-POMDP problems. The idea is to coordinate all agents by the joint Q-value Qooin a) where (T,, T-i) is the joint history of all agents, then the best joint action can be derived by a' = argmax, Qjosne(T. a). In practice, the true but unknown joint Q-value Qjoint (T, a) is approximated by Qtotal a). which in turn is implemented using deep neural network Qroai((T a; w) parameterized by w. Typically, Qtatal(T ,, a; w) is optimized by minimizing the following TD-loss with temporal difference learning (TD-learning):
L(w) = -E(aarr>)) + + Qtotal a'; (T) Qtotal(TT a))]
(1)
where D is the replay buffer containing recent experience tuples (T, a, T'), and Qiora((T, a; w the target network whose parameter w: is periodically updated by copying w.
However, vanilla joint Q-learingg has some disadvantages. First, the scalability is poor for large- scale agents because it needs to search the whole joint action space to find the optimal one.
Second, the agent cannot interact with the environment based on its own information Tr since
the optimal action also relies on the teammates' information T- namely, of readout a"
argmax a Qiotal ((T;, a;;))
To remedy these disadvantages, the Centralized Training with Decentralized Execution (CTDE) paradigm is applied (Lowe et al.] 2017 Foerster et al. 2018, Sunehag et al.) 2018 Rashid et al. 2018). During centralized training, agents are granted access to other agents' information T (and possibly the global state if available) to estimate the joint Q-value Qrozal a; w) in a stationary way, while during decentralized execution, the agent makes decision independently based on indi- vidual Q-value a; = argmax,, Qe(Ti, aq; 0:) where 0; is the policy parameter of agent i. In order to achieve effective value-based CTDE, critical to ensure the consistency between individual and joint greedy action selections, which induces the Individual-Global-Me (IGM) principle (Son let al. 2019): aargmax,,, QI(T),))) ; argmax,
) argmax, Qjoina (T, a)
We notice that several previous methods (Zhou al. 2020 Wang et al. 2020e also take the maximization of Qrocal as a target measurement for good credit assignment. The differences are discussed in Section
2

---

Under review as a conference paper at ICLR 2022
The IGM principle is very effective train large-scale agents because it has a linear (rather than exponential) search space for the optimal joint action (compared to the vanilla joint Qllearning). Recently, QTRAN (Son et al. 2019) proposes sufficient and necessary condition for IGM:
a = [argmax Qi az)]
DN oz (T a:) Qjoint (T, Vjoint (T)
(3)
0
otherwise
where 0; > O, and (Toine(T) = maxa joinn TT. a) D Ot max., QT a,) could be interpreted
as baseline function to correct for the diserepancy between the optimal joint Q-value and the
weighted summation of the optimal individual Q-values. Note that Equation (2) and (3) are defined under specifie state (equally, under specific joint history T) because it is hard to satisfy IGM for
all states. Nevertheless, if we could always find a corresponding a satisfying Equation (3) for each possible T. we say the task itself is factorizable (Son et al. 2019).
RELATED WORK
IGM-based Credit Assignment. During centralized training, the IGM-based joint Q-learning methods are essentially addressing the multi-agent credit assignment problem (Yang et al. 2020a
Zhou et al. 2020), namely, aligning the joint Q-value Qtota! with individual Q-values Q Qiotal(T. a: u) ([ ag; 8,)], : The major difference lies in the detailed implemen- tation of the credit assignment function f. For example, VDN (Sunehag al., 2018) proposes simple additivity assignment function Qrotal a) = ΣN Qi(Ti, (2;(Tr,;;00) O;; and it works pretty well. QMIX (Rashid et al. 2018) applies nonlinear assignment function increase representation
pressiveness, but with the constraint of monotonic improvement Vi, 9Quotai(T.an)
to satisfy the
IGM. Recent methods such as WQMIX (Rashid et al.] 2020), QTRAN (Son et al. 2019). QPLEX [Wang et al., 2020b), Qatten (Yang al., (200b) and QPD (Yang al. 2000aa propose more phisticated assignment functions to enhance the representation expressiveness. c.g., the multi-head attention function (Yang et al., 2020bb and the duplex dueling function (Wang al.
2020b
Policy-based Credit Assignment. A well-known method is COMA (Foerster et al. 2018 which conducts credit assignment by counterfactual baseline. We argue that maximizing Q &total is one of the effective ways to learn a good credit assignment function. For example, LICA (Zhou et al. 2020) and DOP (Wang al. 2020e) take this as the target or measurement of good credit assignment. The key differences between our ECAQ and these methods are two-fold: first, ECAQ is Q-learning method, while LICA and DOP are actor-critic methods; second, ECAQ explicitly optimizes a credit assignment criterion, while LICA and DOP learn the decomposed credit assignment implicitly.
Other Methods. There are other types of multi-agent credit assignment methods (Proper & Tumer 2012; Tumer& & Agogino/ 2007 Wang et al. 2020c Zhang et al. 2020 Zhou et al. 202 1) and multi-age agent cooperation methods (Nguyen et al. 2018 Zhang et | 2020 Mahajan et al. 2019;
Wang et al. 2020d). However, they are beyond the scope of this paper. Due space limitation, provide a brief review for these methods in the Appendix.
4
OUR METHOD
Explicit credit assignment is important for achieving better performance and interpretability, but most IGM-based methods only implicitly learn the credit assignment function, resulting in non- interpretable assignment and possibly poor performance (Zhou et al. 2020) Yang et al., (202a)). In this section, we investigate explicit credit assignment for the IGM-based joint Q-learning. Specifi- cally, Se.tion[4.]] definne the considered problem; Section 4.2 proposes an exact solution to assign credit among multiple agents at a given timestep/state; Section approximates the exact solution to handle complex problems; Scction 4.4 combines TD-learnin with our solution, so the integrated approach can conduct credit assignment among multiple agents and between different timesteps.
4.1 PROBLEM FORMULATION
In this paper, the true but unknown joint Q-value Qjoine a) is approximated by Quatai(T a), which in tum is implemented using deep neural network Qtotal (T, a; w) parameterized by w. Us-
3

---

Under review as a conference paper at ICLR 2022
ing these notations, the considered multi-agent credit assignment function is formulated
follows:
Qrotai (T, a) = (a)) 0.) ((T) 2 Qjoint a)
where ai (T:) > is the weight of Qi, and b(T) (T)) We choose this formulation due to two important reasons. First, is a sufficient and necessary condition for IGM under some conditions demonstrated by Equation ( so it has good fitting ability theoretically (Son et al., 2019 Wang et al., 2020b). Second, it allows us to intuitively interpret the weight as (T,) as the importance of Qi, which can be analyzed to understand the concrete credit assignment (please the experiments).
There are infinite possible assignment solutions satisfying Equation (4). In order to find the best one, can define criterion justify how good an assignment function is, then explicitly optimize such a criterion. We call this kind of methods the explicil mulli-agent credit assignment (MACA).
Recall that the ultimate goal of MACA is to boost learning performance, which is measured by the
maximization of Qrotal (i.e., the expected long-term system-level rewards). Thus, we propose an explicit multi- auett credit assignment criterion as follows:
(Ti, 073 (7)))
argmax Qtotal (T, (T,a) a)
(5)
S.L. Qiotal (T, = aT) az; 0.) + b(T) and ou(T:) > 0 and EN a;
The additional constraint E (T,)) = makes sure that au (Tx) is bounded, so we cannot maximize Qtotal by simply using an infinite o; (T,).
4.2 AN EXACT SOLUTION FOR MACA
In this section, introduce exact solution of Equation (5) under the single-state/timeste setting- This will guide us to design the approximated solution for multi-state problems in Section 4.3 In the following, we omit unnecessary notations (i.e., and [r]] due to single state) for simplicity.
Specifically, Equation 5 is two-objective (i.e., a = [atl and 0" = [0 optimization problem. The Generalized Expectation Maximization (GEM) (Fessler & Hero 1994) is a popular technique to solve such problems. We adopt this idea and propose two-stage optimization method. At the initial weighting stage, each agent proposes an initial weighting vector a; = (o!..., xg that assigns af weights (i.e., credits) to Q: to show its preference about how much contribution agent has made to the agent team. Here, a; [a is agent i's estimation of the optimal a' [ag] At the optimization stage, each agent i iteratively optimizes its weighting vector a; and its policy parameter 0; as follows:
af + af + 31 E p (a) a!)
0: - 0 + 32 2z Vo
(7)
where 31 and 32 are the learning rates. Equation (6) guarantees that all agents eventually reach consistent weighting vectors based on the difference of them. Equation (7) makes sure that Qiotal can be maximized by gradient ascent given the weighting vectors. Interlacing the above updates could be seen kind of GEM algorithms, and the advantages are two-fold (Blondin & Hale b). First, this "decentralized negotiation" will be more robust to the weighting disagreement among agents compared centralized weighting mechanism. Second, it can guarantee that the convergent values (at"" 0,1,11 are optimal under mild assumptions shown by Proposition D
Proposition Under assumption that the individual Q-value functions [Q:] were continuously differentiable and convex, interlacing Equation (6) and 7 enough times will resull in convergent =aaj
laf" ON where a."" == =a =a = ai, and the joint Q-value Qtotat(a) = EN 12 Qa(a;; 67) will converge to the optimal value Qiotal
Proof. First. Olfati-Saber et al. (007)) have proved that updating Equation 6 enough times will make all agents reach the same convergent weighting vector, i.c., af a'y Second, assuming all [QII were continuously differentiable and convex (this is common assumption, and it is true if the Q-values are linear in "features" " as pointed out by (iivvr et al.) (2014). it is easy to prove that Qtotat will also be convex given specific weighting vector; therefore, gradient ascent in Equation
4

---

Under review as a conference paper at ICLR 2022
(r. 4.)l ((x)
TD-loss ....
(T. *.... ECA-loss
relu
W2
D+
[1,t aol ((t)
1:
relu
[-1
W1
relu
Transformation Network
Consistency Network
[0: ag]i=1
a = [e]
KL-loss
-. (1))
Guplueheat head
Weighting head
Q. 9.)
(((T,) (at. a
MLP GRU MLP
MLP MLP
MLP
MLP
h4
GRU
hy
Agent Network
Agent Network
MLP
L2-loss
VAE
ON
(a)
(b)
(c)
Figure I: (a) The structures of Agent Network and Transformation Network. (b) The overall ECAQ architecture. (c) The Consistency Network structure.
7 can always find the maximum point Qioeal with proper learning rate (since the first term does not affect the gradient direction in expectation). Finally, the convergence theory of Expectation
Maximization (Wul 1983 Xu & Jordan 1996) tells us that interlacing Equation (6) and (7) will not
affect the final convergence properties, so we will eventually find the corresponding [a 1 k=1- 0
4.3 APPROXIMATING THE EXACT SOLUTION
The real-world problems often consist of multiple states, but the above solution can hardly learn set of [a; 81 [=1 that is optimal for all states because the optimal [a," 0, pe is stateddependent. In general, policy with strong fitting ability can alleviate this problem, so we approximate the exact solution using deep neural networks (DNN) as the function approximator.
Overall Design. Our DNN-based method is called ECAQ, which consists of three sub-networks as shown in Figure [ The Agent Network first applies GRU to encode the local observation o, into the history т₁. Then, it uses the Q-value head and the weighting head to generate the individual Q- value Qi(T, a;: 0.) and the weighting vector a; (T;; 0,) !a....... a.N], respectively. The weighting head adopts a Softmax activation function to guarantee ΣN of
The Consistency Network is proposed to optimize the weighting vector. It takes as input the original weighting vectors from all agents, and optimizes them to the same converged value [a (r;; 0,]] (i.e., mimicking the effect of Equation (6 The details will be introduced in the following section.
The Transformation Network proposed to increase BCAQ's s representation expressiveness, As pre- vious methods, it uses the joint history T (or the state if available) generate a two-layer hypemett- work (Ha et al. 2017), then transforms individual Q-value [Q, (T,, a;; f,]] to [Q,(T, a; 0r, w)] using this hypemeeworr. Besides, it also generates baseline function b(T;
Finally, we form the joint Q-value as Qtotal(T a; w) = ا at (TA Qi(T ag 02 w) + b(T; w). Afterwards, we can optimize the policy parameter to the converged value [0:]," (i.e., mimicking the effect of Equation )7). The details will be introduced in the following section.
Optimizing the Weighting Vector. As mentioned before, the Consistency Network is proposed to optimize the weighting vectors |as] to the same converged value. The main difficulty is how to apply DNN to achieve this goal. Here, we adopt the variational inference technique (Mao et al. 2020b with the following key idea: assuming that the optimal weighting vector is a =< _ but it is unknown; each agent i would like to infer a' based on o; by p(a" o:); all agents' weighting 2 could really converge to the same a", would achieve goal.
vectors [a:] N
plo: "orla")pe p(("") In practice, directly computing poa" = p(o; "pta is quite difficult, so we approximate p((" (o:) using another tractable distribution q(a" Jo:) by minimizing the KL.-divergence between
5

---

Under review as a conference paper at ICLR 2022
them, namely, min " which equals to: (o))(a)))))p((a"))
max [oz)) log p(o
(8)
8 can be modeled by variational autoencoder (VAE), which the main part of the Consistency Network. The encoder of this VAE learns mapping ((cy,oo;; 02) from to a,, and the decoder learns a mapping p(oi 0,) from a, back to o,. The loss function to train this VAE is:
Equation
Liue (04) = (2( o;: 0.) + [Liq(as (,))la
where the first term represents the reconstruction error of observatiooss/taatss, and minimizing this error makes that the weighting vector (,; 0z) is state-dependent as better handle real- world problems consisting of multiple states; the second term ensures that the learned distribution g(o:/or; 04) is similar to the true prior distribution p(a*). However, the true prior p(a") )) in Equation ( is unknown. One could assume that p(a²) follows a unit Gaussian distribution as previous methods, but it cannot be true for all states. In practice, we find that other agents' weighting vector a(ay oj: fg) is a good surrogate for p(xx", namely, we approximate Equation (9) by:
1 EN T.q0 (0,) v T,2 oe; 02) + N -= KTL_K los 0:)llg(as) 0j))
(10)
Provably, Proposition 2 shows that Equation (TO) has the same effect as Equation (6) under single- state setting. Intuitively, this approximation punishes any pair of agents (i.j) with inconsistent weighting vectors, namely, with large K Lgg(a, lo,; ( oj: 02)). Therefore, it is helpful for converging to the same weighting vector. In practice, the above optimization cannot be iterated infinitely, so we make sure that the weighting vectors are eventually consistent by averaging them of ~ 1 E.N 10/-
N 1.
Proposition 2. Under single-state setting, Equation (T0) has the same effect as Equation O
Proof. For single-state, some common assumptions hold: 1) there is no need to recover observation, so the first term of Equation (10) is removed; 2) g(os) and g(<x)) are Gaussians with equal stan-
dard deviation; therefore, minimizing , (Lsll))))) = N -ررة log Zi +
20
N 1 (a:(as) - My (az)) has the same effect as minimizing N Mلح (ay - a;) in Equation 6 N
Optimizing the Policy Parameter. In practice, we share policy parameters among agents as the previous methods (e.g.. VDN, QMIX and QTRAN). Besides accelerating convergence, the special advantage is that the first term of Equation can be simplified as Mn a 0;. therefore Equation can be rewritten as:
0, -
0, + B2 Qo, Qrotai(T, ar Vo, Qi 0;)
(11)
The loss function of Equation 11 is L (f:) Qrotai- We call it Explicit Credit Assignment loss (ECA-loss) because it explicitly maximizes Qtoeat (i.e., the criterion of good credit assignment).
4.4 PUTTING IT ALL TOGETHER
The exact solution proposed in Section 4.2 is summarized by Equation (6) and 7). In Section 1.3
ECAQ adopts the VAE-loss (i.e., Equation (T0) and ECA-loss (i.e., Equation (IT)) to approxi- mate Equation (6) and 77 respectively. Nevertheless, VAE-loss and ECA-loss
mainly used to optimize the credit assignment among multiple agents at a given timestep/state. For the sequen- tial decision-making problems consisting of multiple timesteps/states, it is also critical to do credit assignment along the time horizon (i.e., assigning the delayed reward to previous timesteps). There- fore, ECAQ also minimizes the following TD-loss:
ld (w) = B(rrarrr">>>(>>>>> + max Qtoral aa; " Qrorat (To a; w))
(12)
a'
ensures the Bellman optimality of Qtotal, namely, Qtotal can approximate the true but unknown Qjoint very closely. Putting it all together, the total loss to train ECAQ is:
L(w, 0.) = I'((w) + (ENN,LLe
+ EN L (0.))
(13)
where 77 is the hyperparameter to balance the credit assignment among multiple agents and between different timesteps. The detailed training algorithm is provided in the Appendix.

---

Under review as a conference paper at ICLR 2022
(a) The easy 2s3z map.
(b) The easy lc3s5z map.
(c) The hard 2c vs 64zg map.
(d) The hard Sm 66m map.
(e) Super hard 3s5z.vs (i) The super hard MMM2 map.
The test win rate on different StarCraft II maps. ECAQ achieves the best performance on four maps (i.e., the easy Ic3s5z, the hard 2c_ vs_64zg and the super hard 3s5z_ vs. vs.3s6z and MMM2), and performs as good as QPLEX on two maps (i.e., the easy 2s3z and the hard 5m.vs.6m).
Figure 2:
5 EXPERIMENT
Environment. To guarantee fair comparison, the decentralized StarCraft II micromanagement problem (Samvelyan et al. 2019) is used since it is usually considered as the official testbed for the IGM-based methods. Besides, previous works show that StarCraft II is suitable for credit assignment study (Zhou et al.) 2020 Yang et al. 2020a Wang et al.] 2020e Foerster et al. 2018). Specifically, six maps with different configurations (e.g.. easy, hard, and super hard settings, homogeneous and heterogeneous agents) are used to guarantee that ECAQ does not overfit to one specific scenario.
We also evaluate ECAQ on the cooperative navigation problem (Lowe et al. 2017: Mordatch & Abbeel 2018). which is a simple yet popular multi-agent environment. Specifically, there are N agents and N landmarks on 10-by-10 2D plane. The agents are controlled by our methods, and they try to cover all landmarks. The observation is the relative positions and velocities of other agents and landmarks. The action is the velocity of agents. The reward is the negative distance of any agent to each landmark. We test three scenarios where N = 4, 6 and 10, respectively.
Baseline. Since we study the explicit credit assignment for the IGM-based joint Q-learning, here we mainly compare with these IGM-based methods. Specifically, we consider the most relevant VDN (Sunehag et al. 2018), QMIX (Rashid et al. (0188) and QTRAN (Son et al. 2019). We also adopt COMA (Foerster et al., 2018) and IDQN (Tampuu al. 2017). COMA applies counterfactual baseline to do credit assignment; in contrast, there is no credit assignment in IDQN. The advanced methods like QPLEX (Wang et al. 2020b), WQMIX (Rashid et al. 2020), LICA (Zhou al. 2020).
DOP Wang et al. 2020e) and Qatten (Yang et al.) 2020b) are also reported.
Implementation. Since most baselines are officially provided in the PyMARL framework also use this framework to implement ECAQ and to conduct the experiments. We do not modify any of PyMARL to guarantee a fair comparison. For the
of the default wttpgguuraiiinn/yyyypprraaeeeeee special hyperparameter r) of ECAQ, we decrease it from 1.0 to 0.05 gradually as training goes on.
5.1 MAIN RESULT
Result of StarCraft II. The average test win rates of five independent runs are shown in Figure As can be observed, ECAQ achieves the best performance four maps (i.e., the easy lc3s5z, the hard 2c vs 642g, and the super hard 3s5z. vs 3s,3862 and MMM2), and performs as good as QPLEX on two maps (i.e., the easy 2s3z and the hard 5m_vs_6m) Notably, the performance of COMA is unstable: it works well in some scenarios but it is even worse than IDQN in other scenarios. These results demonstrate that good credit assignment is very necessary for consistent multi-agent cooperation. In order to compare ECAQ with advanced methods, we show the results on hard and super hard maps
7

---

Under review as a conference paper at ICLR 2022
Table 1: The test win rate on the hard and super hard StarCraft II maps.
Map
Qatten
QPLEX
WOMIX
LICA
DOP
ECAQ (ours)
hard oc.vs_64zg
66
55
67
68
84
85
hard 5m vs. 6m
72
70
60
60
63
73
super hard 3s5z_ve vs 3s6z
17
12
6
0
0
15
super hard MMM2
79
72
84
50
80
average test win rate
58.5
52.25
39.0
53.0
49.25
63.25
in Table As can seen, BCAQ's performance is better than or as good as many advanced methods (e.g.. Qatten, QPLEX and WQMIX) in specific maps, while its average performance is the best, although ECAQ does not involve advanced DNN architectures like duplex dueling or multi-head attention critie. This is because ECAQ can directly learn a good credit assignment that maximizes the long-term rewards, which is highly positive for the test win rate. For example, in the super hard 3s5z_vs_3s66 scenario, ECAQ assigns high credit to ally's Zealots because the learned policy is that ally's ' Zealots hold enemy's ' Zealots and attack enemy's Stalkers at the same time.
Result of Cooperative Navigation. The average rewards of independent runs are shown in Figure 3 It can be observed that VDN outperforms QMIX in cooperative navigation, which in contrast to the results of StarCraft II where QMIX is better than VDN. This highlights the re- lationship among method performance, method complex- ity and task complexity: complex methods do not always get better performance in simple tasks. Nevertheless, as can be seen, ECAQ obtains more rewards than other base- lines in most scenarios. It seems that the complexity of the evaluated tasks does not influence ECAQ too much.
Figure 3: The average rewards in coop- erative navigation. Lower bar is better.
5.2 FURTHER ANALYSIS
Ablation Study. The average test win rate is shown in Figure 4 where TD, ECA and VAE represent the three loss functions defined before (namely, TD+ECA+VAE stands for ECAQ). Surprisingly, can be observed that neither TD+ECA nor TD+VAE performs well, and they are somehow worse than simply applying a single TD-loss. The reason may be that 1) TD+ECA cannot reach consistent weighting vectors due to the lack of VAE-loss, so the optimization of ECA-loss may be incorrect; 2) TD+VAE does not optimize the credit assignment at all due to the lack of ECA-loss. Therefore, both TD+ECA and TD+ VAE deteriorate the solution. These results assert conclusion that both VAE- loss and ECA-loss are necessary for ECAQ's good performance. We also find the same conclusion in the matrix games, and the details are shown in the Appendix.
15
T2
(a) The hard 2c_vs_647g.
(b) The super hard 3x57_v8_3s6iz
(c) The super hard MMM2.
Figure 4: The ablation results for (super) hard scenarios. Other scenarios are shown in the Appendix.
Credit Assignment Study. To make clear analysis, we should make the environment simple as possible, so we adopt the matrix games in this study. Firstly, we use the asymmetric monotonic game G as shown in Table 2(a) There are two agents in G and each agent i has three actions a;A. a,B, and acc. The asymmetricity means that different agents have different impact on the payoff. Specifically. the column agent (i.e., agent 2) has larger impact compared to the row agent (i.c., agent 1) in game G, since the payoff changes by two units in column while by one unit in row. We train ECAQ through full exploration conducted over 20,000 steps QTRAN (Son et al. 2019). We
8

---

Under review as a conference paper at ICLR 2022
Table 2: The asymmetric monotonic game and the converged results on the game.
Payoff of the matrix game G.
(b) Q1, Q2 and Qtotal learned by ECAQ under setting #1.
a1
02
42a
028
a2c
Qi -22
4.828(a24)
2.107(a2s)
-0.61l(a2c)
014
7
5
3
6,639(114)
7.004
5.005
3,007
018
2
2.854(a18)
6.000
4.001
2.003
a1c
5
3
1
-0.927(a1c)
4.997
2.998
1.000
(c) The converged values. ECAQ has learned the optimal policy (i.e., 01A and 02A)
Setting
aq
Qi
Q2
Qintat
#1: G
0.26523
0.73477
6,639(a1
4.828()24)
7.004
#2: GT
0.72562
0.27438
4.755(a A)
6.435(a24)
6.982
#3: C 10
5.83e-08
1.00e+00
55.900(a14)
33.236(a2A)
63.738
Table 3: The symmetric non-monotonic game and the converged results on the game.
(a) Payoff of the matrix game.
(b) The converged values in different settings.
01
d2
02A
d2B
02c
Setting
Q2
Piotal
8
-12
-12
#1: full exploration
0.55601
0.44399
-3,568
018
-12
0
0
#2: high probability for 01
1.35c-07
9,99c-01
0.666
01C
-12
0
0
#3: high probability for 02a
9.99e-01
9.09e-08
0.151
check how different payoff settings will affect the credit assignment. As shown in (ablee((c) ECAQ assigns large weight to agent 2 in game G (i.e., o3 R 0.735 in setting #1) because agent 2 has larger impact on the payoff. In contrast, when we transpose the payoff of G, denoted by G the weight of agent 2 will be small (i.e., as R 0.274 in setting #2) since agent 2 is less influential on the payoff in G We further increase the payoff of G by ten times, denoted by G * 10, and the weight of agent 2 changes to very large value (i.e., 02 = 1.0 in setting #3). This is because the payoff changes by twenty units (rather than the original two units) in column, and the absolute impact of agent 2 becomes much larger. These analyses have demonstrated the good credit assignment ability of ECAQ. Consequently, ECAQ can easily find the optimal decentralized policy (i.e., 014 A and (2 in all games observed from Table 2 Finally, comparing Table 2(b) with 2(a) it can be seen that ECAQ fits the payoff matrix very accurately, which asserts the good fitting ability of ECAQ.
Secondly, use the symmetric non- monoooic game as shown in Table (aa) This game is proposed by QTRAN (Son et al., 2019). We check how different exploration settings will affect the credit assignment. As shown in Table ((b) the full exploration (i.e., setting #1 QTRAN) will result in almost random credit assignment values (i.e., oz. = 0.5). This is expected because the payoff matrix symmetric and the exploration is full, so there is difference between the two agents. In contrast, when we make one agent more stable (e.g., raising the probability of α1 in setting #2), ECAQ will assign a large weight to focus on the other agent (e.g., a¹ 1.0 in setting #2). The reason is that the other agent is more random, and it has greater impact on the obtained reward. The results of setting #3 are similar to these of setting #2, and both settings can find the optimal policy as shown in the Appendix. Overall, these analyses have demonstrated the good credit assignment ability of ECAQ.
CONCLUSION
Multi-agent credit assignment has long been fundamental issue for multi-agen cooperation. This paper presented the explicit multi-agent credit assignment for the IGM-based joint Q-learning, which not only ensures the Bellman optimality of Qtota! to do credit assignment along the time horizon, but also optimizes a criterion to do credit assignment among different agents explicitly. We instantiate this idea with deep neural networks and propose ECAQ to facilitate multi-agent cooperation in more realistic scenarios. Extensive experiments justify the superior performance of ECAQ. Furthermore, the detailed analyses show that ECAQ has really learned interpretable credit assignment values. To our best knowledge, the explicit credit assignment is complementary yet novel to the existing IGM- based studies. We believe that it is basic for building effective learning- baasd multi-agent systems.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Adrian Agogino and Kagan Turner. Multi-agent reward analysis for learning in noisy domains. In Proceedings of the fourth international joint conference Autonomous agents and multiagent systems, pp. 81-88, 2005.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4): 119-840,22 2002.
Maude J Blondin and Matthew Hale. An algorithm for multi-objective multi-agent optimization. In 2020 American Control Conference (ACC), pp. 1489-1494, 2020a. doi: 10.23919/ACC4553 2020.9148017.
Maude J Blondin and MT Hale. A decentralized multi-objectivve optimization algorithm. arXiv preprint arXiv:201000781,, 2020b.
J.A. Fessler and A.O. Hero. Space-alternating generalized expectation-maxxiizaaiio algorithm. IEEE Transactions Signal Processing, 42(10):264--2677 1994. doi: 10.1 10.1109/78.327322
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7219-7226, 2018.
John J Grefenstette. Lamarckian learning in multi-agent environments. Technical report, NAVY CENTER FOR APPLIED RESEARCH IN ARTIFICIAL INTELLIEENE WASHINGTON DC, 1995.
David Ha. Andrew Dai, and Quoc V Le. Hypernetorrss. In Proceedings of the International Conference on Learning Representations, 2017.
Kenneth E. Kinnear (ed.). Advances in Genetic Programming. MIT Press, Cambridge, MA, USA, 1994. ISBN 02621 11888
Jiahui Li, Kun Kuang, Baoxiang Wang. Furui Liu, Long Chen, Fei Wu, and Jun Xiao. Shapley counterfactual credits for multi-agent reinforcement learning. In Feida Zhu, Beng Chin Ovi, and Chunyan Miao (eds.), KDD '21: The 27th ACM SIGKDI) Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021, pp. 934 942. ACM, 2021a. doi: /0045//4444553 414444444488347777020 3467420
11114//444548.344422222
TtRLhhtps::///c /tttps://doi.oor//1...
Wenhao Li, Xiangfeng Wang, Bo Jin, Dijun Luo, and Hongyuan Zha. Structured cooperative re- inforcement learning with tame-varying composite action space. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021b.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed pooperative-competiivv environments. In Advances in neural infor- mation processing systems, pp. 6379-6390, 2017.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. MAVEN: multi- agent variational exploration. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Buee Emily B. Fox, and Roman Garnett (eds.). Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cexsing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada,, pp. 7611-7622, 2019. URL https::/ www.:ewwww...................... www.eeewwipsscpppppeeeeeeee wwtpppper//219//aa.... EBl:aCDeefaac7/86641992222222222 Absttttttbbtttttt.... .tml
Patrick Mannion, Sam Devlin, Jim Duggan, and Enda Howley. Multi-agent credit assignment in stochastic resource management games. The Knowledge Engineering Review, 32, 2017.
Hangyu Mao, Zhibo Gong, and Zhen Xiao. Reward design in cooperative multi-agent reinforcement learning for packet routing. arXiv preprint arXiv:2003. 2003.0333,.,, 2020a.
Hangyu Mao, Wulong Liu, Jianye Hao, Jun Luo, Dong Li, Zhengchao Zhang. Jun Wang, and Zhen Xiao. Neighborhood cognition consistent multi-agent reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 7219-7226, 2020b.
10

---

Under review as a conference paper at ICLR 2022
Maja J Mataric. Learning to behave socially. In Third international conference simulation of adaptive behavior, volume 617, pp. 453-444422 Citeseer, 1994.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Navid Naderializadch, Fan H Hung. Sean Soleyman, and Deepak Khosla. Graph convolutional value decomposition in multi-agent reinforcement learning, arXiv preprint (xXiv:2010004740, 2020.
Duc Thien Nguyen, Akshat Kumar, and Hoong Chuin Lau. Credit assignment for collective multi- agent with global rewards. In Advances in Neural Information Processing Systems, pp. 8102- 8113, 2018.
Reza Olfati-Saber, J. Alex Fax, and Richard M. Murray. Consensus and cooperation in networked multi-agent systems. Proceedings of the IEEE, 95(1):215-22) 233, 2007. doi: 10.1009//PO...2006 887293.
Scott Proper and Kagan Tumer. Modeling difference rewards for multiagent learning. In Pro- ceedings of the 1 11th International Conference on Autonomous Agents and Multiagent Systems Volume 3, AAMAS '2, pp. 1397-1398, Richland, SC, 2012. International Foundation for Au- tonomous Agents and Multiagent Systems. ISBN 0981 1 738133.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value functiin factorisation for deep multi-agent rein- forcement learning. In International Conference on Machine Learning, pp. 4292-4301, 2018.
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr. Jakob Foerster, and Shimon White- son. The starcraft multi-agent challenge. Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 21 86 2188, 2019.
Lloyd S. Shapley. A value for n-person games. In The Shapley Value, pp. 31-40. Cambridge University Press, 1988. doi: (0.10077cco077951115284888 10. 1017 cbo 97 378051111111 1281162244666 003
URL/httss: /oi. org/
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, pp. 387-395, 2014.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. Interna- tional Conference on Machine Learning, pp. 5887-5896, 2019.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanetot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning based team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2085-2087, Inter- national Foundation for Autonomous Agents and Multiagent Systems, 2018
Ardi Tampuu, Tambet Matüsen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4):e0172395, 2017.
Kagan Tumer and Adrian Agogino. Distributed agent- based air traffic flow management. In Pro- ceedings of the 6th international joint conference on Autonomous agents and multiagent systems, pp. 1-8, 2007.
Jianhao Wang. Zhizhou Ren, Beining Han, Jianing Ye, and Chongjie Zhang, Towards un- derstanding linear value decomposition in cooperative multi multi-agent q-learning. arXiv preprint arXiv:2006.00587 2020a.
11

---

Under review as a conference paper at ICLR 2022
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling
multi-agent q-learning. arXiv preprint arXiv:2008.01062. 2020b.
Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu. Shapley q-value: A local reward approach to solve global reward games. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7285-7292, 2020c.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. ROMA: Multi-agent reinforcement neltt-geenteinforreenntt learning with emergent roles. Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9876-9886, PMLR, 13-18 Jul 2020d. URL htttp:/pprcceeiing mml.
Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Dop: Off-policy multi-agent decomposed policy gradients. In International Conference on Learning Representa- lions, 2020e.
C. Jeff Wu. On the Convergence Properties of the EM Algorithm. The Annals of Statistics, 11(1): 95 - 103, 1983. doi: 10. 10.1214/ass/1 176346667771 URL
1176346060
Lei Xu and Michael Jordan. On convergence properties of the em algorithm for gaussian mixtures. Neural Computation, 8(1):129-151, 1996. doi: 1wt.///e..............
Yaodong Yang, Jianye Hao, Guangyong Chen, Hongyao Tang, Yingfeng Chen, Yujing Hu, Changjie Fan, and Zhongyu Wei. Q-value path decomposition for deep multiagent reinforcement learning. arXiv preprint vrXv:20002.0395. 2020a.
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03939, 2020b.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuan- dong Tian. Multi-agent collaboration via reward attribution decomposition. arXiv preprint arXiv:2010.0833/ 2020.
Meng Zhou, Ziyu Liu, Pengwei Sui, Yixuan Li, and Yuk Ying Chung, Learning implicit credit as- signment for cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Tianze Zhou, Fubiao Zhang, Kun Shao, Kai Li, Wenhan Huang, Jun Luo, Weixun Wang, Yaodong Yang, Hangyu Mao, Bin Wang, et al. Cooperative multi-agent transfer learning with level- adaptive credit assignment. arXiv preprint 10Xxv:210660051111 2021.
12

---

Under review as a conference paper at ICLR 2022
A THE TRAINING ALGORITHM
ECAQ adopts the Centralized Training with Decentralized Execution (CTDE) paradigm. The train- ing algorithm for ECAQ is shown in Algorithm
Algorithm 1: Training Algorithm for ECAQ
Input: Randomly initialized 8 and w for the policy networks (i.e., individual Q-value functions
Qi(Ti, ag; 0:)) and the mixing critic (i.e., the joint Q-value function Qtatai (T, a; w)). Output: Converged individual Q-value QJ (Ti, 0;; ( for future decentralized execution. (Total(T a; w
(T, w);
while not terminated do
The agents interact with the environment based on Qi 0s; 0z). and put the experience tuples (s, T, r, T') into replay buffer D;
Sample batch size of tuples T, a, T') from replay buffer D;
for each luple (8, T; 7') do
Each agent i extracts observation o;:
The Agent Network generates individual Q-values Qi(Ti, a,; 0x) and weighting vector
(T; 0.) .........))) based Os, and further generates the decoded observation
o; based on (T;);
Caleulate the VAE-loss
(,a0(0.) L2(0,, 0.) + N , log ( 0,)): The Consistency Network takes as input [as and outputs the converged
[or 00)
10
The Transformation Network takes as input [Qi(Ts, 0;; 0 and state s, then generates (Q, a;; 0; w)] and b(T;w);; Calculate the joint Q-value as
11
Qrotal (T, a w) 1, az (T): as; 0, w) - b(T; wr); Calculate the ECA-loss a (w) = -Qootal (T, a;; w);
13
Calculate the TD-loss as
L (w) = (r + max Qroual(T', d'; ) Qrotal (T, a; w))²
Calculate the total loss as L(w, 02) = (w) + uEN Li + vV _(ee(())
Train the network parameters [0,]]11 and w by back-propagation based on the total loss; end
15
16
17
In practice, the above for iteration is processed in mini-batch manner;
18
if at target update interval then
19
Update the target mixing critic by Qrotat (T, a; w +- Qtotai(T, a; w);
end
end
B CREDIT ASSIGNMENT ANALYSIS
B.1 CREDIT ASSIGNMENT ANALYSIS FOR COOPERATIVE NAVIGATION
We analyze the training behaviors of the weighting value (i.e., credit assignment value) a; (i 1, 2, 3) using the 3_vs_3 cooperative navigation. We first analyze the dynamics of ai EN a and agent i's estimation af As shown by the black, green and blue lines, although the values are very different at the beginning
2, 3), which are shown by the four solid lines Figure (a)
of training, different af (i = 2, 3) tend to be consistent as training goes on. We then analyze whether the converged
(i 1, is meaningful. Figure ((b) shows the learned policies by the red arrows, and the credit assignment values (which can be estimated from Figure 5(a)) are as follows ai 0.47 since agent Al is far from the landmark and its policy has the largest influence on the reward: 02 Ps 0.35 and ay ss 0.18 since agent A2 and A3 are near the landmark and their policies have little influence on the reward. The results are consistent with these shown in the main paper: the most influential agent usually corresponds to the largest assignment value. These analyses indicate that the assignment values o; (i 1,2, 3) are meaningful identify critical agents.
13

---

Under review as a conference paper at ICLR 2022
0.7
0.6
0.5
0.4
alpha 1 21
0.3
alpha_22
alpha 301
0.2
alpha 1
0.1
alpha_2
alpha
2000
4000
6000
8000
10000
12000
14000
training step
(a) The training dynamic of on and agent i's estimation at of a1.
A3
A2
A1
(b) ax meaningful for MACA. Ai represents agent
Figure The credit assignment analysis using the v.v..3 cooperative navigation task.
Table 4: The payoff matrix of the one-step game and the converged results on the game,
(a) Payoff of the matrix game.
Q1, Q2 and Qrotal learned by ECAQ under setting #3.
a1
a2
a2a
028
020
Qi Q2
-0.014(azA)
-2.497
-3.062(a2c)
01A
8
-12
-12
1.674(a1A)
0.1509
0.1505
0.1505
81B
-12
0
-,.40(a118)
-6.9395
-6,9395
-6.9300
41C
-12
0
-6.487(ncc)
-8.0108
-8.0108
-8.0109
The converged values for different settings. Note that both #2 and find the optimal policy.
Setting
ag
az
Qi
Qi
Viotal
#1: fully exploration
0.56
0.44
0.0403(a1c)
0.6296(a2c)
-3.568
#2: high probability for a1a
1.35e-07
9.99e-01
0.4702((1A)
1.7616(az4)
0.666
#3: high probability 0244
9,99e-01
9.09e-08
1.6739(a1A)
-0.0140(a24)
0.151
#4: training only by TD-loss
9.99c-01
2.58e-06
0.4889(014)
-0.8592(az8)
-0.695
14

---

Under review as a conference paper at ICLR 2022
B.2 CREDIT ASSIGNMENT ANALYSIS FOR MATRIX GAME
We also adopt a matrix game (Son et al. 2019) to check the credit assignment ability of ECAQ. As shown Table 4[a) there are two agents and each agent has three actions aiA, a, and a.c- We check how different settings will affect the credit assignment. As shown in Table [[c] the fully exploration (i.e,, setting #1 as QTRAN (Son et at.) 2019)) will result in almost random credit assignment value (i.e., of x 0.5). This is expected because the payoff matrix symmetric and the exploration is full, so there is no difference between the two agents. In contrast, when we make one agent more stable (e.g., raising the probability of 014 in setting #2). ECAQ will assign large weights to focus on the other agent (e.g., 02 ~ 1.0 in setting #2). The reason is that the other agent is more random, and it has greater impact on the obtained reward. The results of setting #3 are similar to these of setting #2, and both settings can find the optimal decentralized policy (i.e., 4144 and (2 as observed. Comparing the results shown in Table (bb) with these shown (Son et al. 2019 Wang ee al., (2020b), it can be seen that ECAQ fits the optimal payoff more accurately than VDN, QMIX and Qatten. Overall, these analyses have demonstrated the good credit assignment ability and fitting ability of ECAQ.
C MORE ABLATION STUDY
We conduct ablation study on the easy maps (e.g.. 2s3z), finding that there is no significant perfor- mance difference for different ablation models as shown by Figure(6(a) The reason that the maps are too easy, and all methods can get good results. Therefore, we focus on doing ablation study on the hard and super hard maps. As shown in the main paper, neither TD+ECA nor TD+ VAE performs well, and they are somehow worse than simply applying a single TD-loss in the hard 2c_vs_ -66z,, super hard 3s5z vs 3s6z and super hard MMM2 maps. However, for the hard 5m vs 6m scenarios, TD+ECA, TD+ VAE and TD have almost similar performance as shown by Figure[6(b) but they are worse than TD+ECA+VAC (i.e., ECAQ). In the future, we will do more ablation study on different maps and draw more detailed conclusions when the computing resources are available.
1000
VAE
TD DCA
VAE
10 VAE
1D BUA
10 BCA
TD
TO
20
23
meerpe 10,000
Smeette 10,000
102
150
275
200
(a) The easy 2s32 map.
(b) The hard 5m vs 6m map.
Figure 6: The ablation results on StarCraft II.
We also conduct ablation study on the matrix game shown in Table 4(a) The payoff matrix is symmetric, so there will be no difference between the two agents under full exploration. Therefore, we raise the probability of 024 (i.e., setting #3 in Table [(c) and ECAQ can find the optimal decentralized policy (i.e., 414 and a24) observed. In contrast, if we train ECAQ only by TD-loss with the same exploration (i.e., setting #4), agent can just find a non-optimal policy a2g, which asserts the necessity of both VAE-loss and ECA-loss.
D RELATED WORK
D.1 MULTI-AGENT CREDIT ASSIGNMENT APPROACH
In this section, we provide a brief review about the multi-agent credit assignment approaches that are closely related to the proposed ECAQ.
15

---

Under review as a conference paper at ICLR 2022
Explicit Credit Assignment. In general, the explicit methods attribute agent contributions that are at least locally optimal (Kinnear 1994). A notable approach is to assess an action by calculating difference reward against certain reward baseline (Tumer & Agogino 2007 Agogino & Tumeer 2005] Proper & 'Tumer 2012). The key idea is that the true contribution of agent i can be approxi- mated by the difference between rewards induced by a and [mc, a.1), where a, is counterfactual action (i.e., not the true action agent i has taken). For example, Agogino & Turner (2005) and
Tumeer & Agogino (2007) adopt specific ae to calculate the difference reward or CLEAN reward to do credit assignment. COMA (Foerster et al. 2018)) and SQDDPG (Wang et al., 2020c) extends this idea from reward difference to Q-value difference, and calculates the counterfactual baseline or Shapley Q-value to do credit assignment. Specifically, COMA (Foerster et al.] 2018) uses a cen- tralized critic to estimate the counterfuctual advantage of an action. SQDDPG (Wang et al. (020c) applies the Shapley-value framework (Shapley) 1988) to do credit assignment based on an agent's marginal contribution as it is sequentially added to possible agent groups. However, the explicit mentioned here is different from the explicit mentioned by ECAQ.
The IGM-based Approach. The representative methods are VDN (Sunehag et al. 2018). QMIX (Rashid et al., 2018). QTRAN (Son et al. 2019), Qatten (Yang et al., 2020b), QPLEX (Wang al., 2020b), etc. We have reviewed these methods in the main paper. They are often called the implicir credit assignment methods as noted by [Zhou et al. (2020). In contrast, ECAQ applies explicit credit assignment training signal to optimize the IGM-hased approaches.
The Attribution Approach. The key idea is that split the final reward into two kinds of rewards: self-reward and attributed reward; and the attributed reward is assumed to be redistributed by other agents based on the agent's 's contribution. Methods following this idea are (Nguyen al., 2018 Zhang et al. 2020 Mao et al. (020a)), and they can also be seen as explicit credit assignment.
Other Approach. There are many other multi-agent credit assignment methods that can be hardly classified clearly, for example, the implicit credit assignment (Zhou et al., 2020), the social reward credit assignment (Mataric 1994) and others (Mannion al. Grefensttttt 1995).
D.2 MULTI-AGENT COOPERATION APPROACH
Please note that we aim at giving a complementary thought for the multi-agent credit assignment problem rather than beating all methods (with hyperparameter tuning), so we implement ECAQ based on the basic QMIX instead of the advanced approaches like Qatten (Yang et al. 2020b) and QPLEX (Wang et al. 2020b). Therefore, we do not intend to give detailed review for all recent deep MARL methods. Nevertheless, there are many topics facilitate multi-agent cooperations, for example, the role-based methods (Mahajan et al. 2019), the coordinative exploration methods (Wang et al. (2020d). the graph-based methods (Blondin & Hale 2020a by Mao et al. (020bb and so on. However, these methods are beyond the scope of this paper.
E IMPLEMENTATION DETAILS OF ECAQ
As mentioned in the main paper, we use the up-to-date PyMARL framework to conduct the ex- periments. We do not modify any of the default configurations of PyMARL to guarantee a fair comparison. We do not tune the hyperparameters of ECAQ too much: the weights of VAE-loss and ECA-loss are directly set equal, and they decrease from 1.0 to 0.05 gradually as training goes on, all of the other hyperparameters (c.g., exploration and activation functions) keep the same as these of PyMARI. The code is available on the submission system. Therefore, the main experimental results can be easily reproduced. In addition, we train our methods the 64-core CPU machine with total memory of 128G. The training time is 4 hours to 10 hours for different StarCraft II maps. This not too computing heavy for MARL applications.
hhtps://githh.co//www//t//// com/oxwhirl/pymar.. The code licensed under the Apache License v2.0, can use it freely for research purpose.
16

---

Under review as a conference paper at ICLR 2022
F LIMITATIONS OF ECAQ
ECAQ is implemented based on deep neural networks, so it faces the "black box problem" where the behaviors of individual agents may be interpretable from the perspective of human. Fortunately, ECAQ adopts an explicit manner to do multi-agent credit assignment, and the learned weighting vectors are interpretable some extend.
ECAQ focuses on the fully cooperative setting, so the goal is set as the maximization of the long- term shared global reward. As result, the multi-agent credit assignment (MACA) may raise ethical issues when the optimal joint actions require sacrificing certain agents, Certainly, we can define the "fair-criterion" for MAČA, then optimize this criterion to guarantee fairness between agents, and this is our future work.
ECAQ is an IGM-based method. The main limitation is that it cannot fit the non monooootonic pay- off perfectly, as shown in Table 4(b) The reason is that both the Transformation Network and the weighting vector Og are nonnegative. However, as mentioned before, we aim at giving a comple- mentary thought for the multi-agent credit assignment problem rather than beating all methods, so we implement ECAQ by an casy-to-cxplain joint Q-value function. If we implemented ECAQ based more advanced approaches like Qatten, QTRAN+ QPLEX, it will be much easier fit the non-monotonic payoff.
17


Reviews:
[{"review_id": "tzpr3d-RX1X", "reviewer": "Reviewer_49nJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is interesting to read, and it advances the state-of-the-art multi-agent RL literature by introducing a new MARL method that considers a novel form of credit assignment. That said, I have several concerts about the current results and their exposition, listed below together with the strengths of the paper. \n\n**Strengths:**\n\n- *The paper studies an important problem.* It explores one of the fundamental challenges in multi-agent RL, the credit assignment problem, and it contributes novel ideas relevant for making cooperative multi-agent RL more effective. It studies an explicit approach to the multi-agent credit assignment problem, in which agents estimate their contributions to the joint performance. \n\n- *The novel algorithmic approach outperforms baselines in standard multi-agent RL testbeds.* Based on the presented results, the proposed method, ECAQ, seems to outperform baselines from prior work. More specifically, ECAQ's performance is either comparable to or better than recent MARL methods. The proposed MARL design (NN architecture in Figure 1) seems novel. \n\n- *Related work is covered well and the paper explains how its main algorithmic contribution is grounded in theory.* The paper extensively surveys the prior work on the credit assignment problem in  MARL, providing satisfactory explanations for the most important references. It also develops the proposed MARL method by reasoning about the exact solutions to the *explicit* credit assignment problem for simple domains.    \n\n**Weaknesses:** \n\n- *The contributions of the paper are somewhat incremental, while the paper could be considerably improved in terms of presentation.* It is hard to disentangle the core novel ideas that the paper introduces from existing works. While I believe I understood  the main contributions of the paper, I generally found the details hard to follow. In my opinion, the paper is not written very well and it contains quite a few typos. \n\n- *Theoretical results are not rigorous enough.* For example, the proof of Proposition 1 is actually not a formal proof. This is best demonstrated by the sentence: *...it is easy to prove that $Q_{total}$ will also be convex given a specific weighting vector;...*. Generally, some of the theoretical claims look quite informal and/or are not rigorously stated. For example,  in Proposition 2, what does *the same effect* mean? Some of the modeling assumptions are not clearly explained. For example, in Eq. 7, we have that policy parameters $\\theta_i$ are updated based on the policy parameters of other agents, i.e., $\\theta_j$, but this update rule does not seem to be justified in the text.   \n \n- *Experimental results are not clearly explained and look somewhat incomplete.* Regarding the experiments, I've found the main results interesting, but this whole section could be considerably improved. a) There are quite a few formatting issues. For example, the exact meaning of the bold numbers in Table 1 does not seem to be explained, or in Figure 2, x and y axis are barely visible.  b) It is not clear what the baselines selection procedure is for different environments. For example, in Figure 3, the results are obtained for a cooperative environment from Lowe et al. 2017. How does the approach from Lowe et al. 2017 perform compared to the proposed approach? Why are some baselines (QPLEX, WQMIX, LICA, etc.) missing in Figure 3?  c) Some of the  paragraphs in the experimental section are ambiguous. For example, the game from the last paragraph in Section 5.2 is taken from Son et al. 2019. However, the 3 settings mentioned in this paragraph are not clearly explained; perhaps explaining how these results relate to the ones in Son et al 2019 could help. It is also not clear why these results show *the good credit assignment ability* of ECAQ, especially since the discussion is not focused on comparing ECAQ to other approaches. \n", "summary_of_the_paper": "The paper studies the credit assignment problem in multi-agent reinforcement learning (MARL), focusing on the centralized training with decentralized executing paradigm. The paper introduces a new MARL method based on value function factorization, and it experimental showcases its benefits over existing approaches. This new method aims to more explicitly account for agents' contributions to the joint performance, and the paper aims to justify its components using a simple theoretical analysis. The core evaluation is experimental, and it is based on standard MARL experimental testbeds from prior work. ", "summary_of_the_review": "The paper contains interesting contributions, however its theoretical and experimental results could be improved: formal claims should be more rigorously stated, while experimental results could be expanded. The presentation of the paper could be considerably improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_iNxJ3HrQiB", "reviewer": "Reviewer_56iQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall speaking, I think this paper is written clearly and easy to follow, and I think it is quite valuable to the research area of Multi-agent Cooperative Learning.\n\nThe authors introduced a simple but very helpful idea of \"explicit credit assignment\" into existing IGM-based joint Q-learning studies, by using maximizing Q_{total} as a criterion for realizing efficient credit assignment. To the best of my knowledge the proposed technique is novel, and the empirical experiment evaluation results appear to be convincing and demonstrate that the proposed method achieve superior performance compared to several advanced SOTA baselines (e.g., WQMIX, DOP). \n\nOn the other hand, although the current experimental results are already somewhat exciting, it would be better and even more convincing if we could see it scaled up (to the extent possible) in the other envs, to make sure the NNs don't break anything at scale. Such scale testing of the proposed approach would be more valuable to understand better the power and limits of the proposed novel method.\n\nMoreover, for better clarity and fair comparison as well as better replicatibility,  it would be good to for the authors to explicitly clarify in the paper how much tuning efforts (e.g., tune the architecture) were spent there to get these results, for the proposed ECAQ method as well as other baseline methods.\n\n\n\n", "summary_of_the_paper": "In this paper, the authors proposes a novel scheme named ECAQ for learning an explicit credit-assignment scheme for CTDE tasks, which in its theoretical form should converge to the optimal solution and also empirically evaluates the DNN form. The main idea is to use maximizing Q_{total} as a criterion for realizing efficient credit assignment for  for multi-agent joint Q-learning. The authors conducted experiments to demonstrate that the proposed ECAQ technique achieves interpretable credit assignment and superior performance compared to several advanced baselines.", "summary_of_the_review": "Briefly speaking, I think the paper is well-written and easy to follow, and he results seems to be technically solid based on my check. The ideas and contents in the paper seems to be interesting and novel, and the experimental results are also providing some reasonable justification of the values of the proposed new approach.\n\nThere are some minor areas that might be improved about the empirical experiment evaluations part, as described in the main review section, but generally speaking, I think the paper is valuable to the research community of  of Multi-agent Cooperative Learning, and I don't have too much concern about accepting the paper for publication on ICLR.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Yk1OQW0qY0K", "reviewer": "Reviewer_trcZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My score is completely based on the fact that the proposed method does not optimize credit assignment in any way for the MARL objective. If the authors can convince me otherwise, I would be glad to change my review.\n\nThe paper gives the impression that it aims to do optimal credit assignment. The abstract says \"we formulate an explicit credit assignment problem where each agent gives its suggestion about how to weight individual Q-values to explicitly maximize the joint Q-value,\". This is then formally written as the optimization problem in equation (5), where the weighs $\\alpha$ are supposed to be optimized so as to maximize $Q_{\\text{total}}$. But then, in the proposed solution in equations (6) and (7), the $\\alpha$ update does not depend on any notion of performance. The $\\theta_i$ update depends on $Q$, but the $\\alpha$ updates are not affected by $Q$ at all. Even if the initial set of $\\alpha^l_i$ converge to some consistent values, they are not the \"optimal\" weights since the update rule doesn't explicitly optimize any objective. This discrepancy between the intention of the method and what it actually does is also shown later in writing, where the paper says \"the Consistency Network is proposed to optimize the weighting vectors $[\\alpha_i]_{i=1}^N$ to the same converged value\". Creating a process for a set of values to converge doesn't mean that the process optimizes anything. A possible counterargument is that the updates (6) and (7) causes the \"optimality\" to be contained in the $Q_i$ weights $\\theta_i$ rather than in $\\alpha_i$. But that is not at all explicit.\n\nAnother flaw is that the so-called 'explicit multi-agent credit assignment criterion\" in equation (5) is problematic. Essentially, part of it says: given a fixed functional form of $Q_{\\text{total}}$, which is some function of $\\theta_i$ via the $Q_i$ but not a function of the Markov dynamics (since that is absent in the expressions), find $\\theta_i$ that maximizes $Q_{\\text{total}}$. So one can just choose $\\theta_i = \\pm \\infty$, depending on signs. Perhaps the authors meant to use an RL objective, like the expected cumulative return, but that is not what equation (5) says.\n\nI do like the overall intention to do explicit multi-agent credit assignment. I believe that some adjustments to the approach will make this work suitable for another submission.\n\nMinor points:\n- grammar, e.g. \"Besides, the extracted parameter w is usually lack of interpretability\"\n- \"First, this “decentralized negotiation” will be more robust to the weighting disagreement among agents compared to a centralized weighting mechanism.\" In a centralized weighting scheme, there is no such thing as \"disagreement\", so what is being compared? The decentralized one has non-zero disagreement.\n- equation (9), is the \"L2\" supposed to be the $l_2$ error norm? How is this equivalent to maximizing the first term in equation (8)? Why wasn't the standard VAE reconstruction loss used?\n- The colorbars in the legend in Figure 2 are too thin, I can't distinguish them and match them to the curves easily even at 200% magnification.", "summary_of_the_paper": "This paper tries to tackle the multi-agent credit assignment problem by an explicit method. Specifically, the method is based on individual-global-max criteria for value decomposition methods for MARL, whereby the joint $Q$-value depends on a weighted sum of individual $Q_i$. The method interprets the individual weight $\\alpha_i$ as the \"credit\" (i.e, \"importance\") of agent $i$, then lets each agent $i$ have its own guess $\\alpha^l_i$ of the importance of another agent $l$, then proposes an update rule to achieve consistency among all agents' guesses. This is first formulated for the stateless case, and then extended to the temporal setting with function approximation. Experiments were conducted on the StarCraft micromanagement benchmark and cooperative navigation to show performance versus baselines, and in matrix games to show the dependence of weights $\\alpha$ on payoff asymmetry.", "summary_of_the_review": "This paper purports to optimize credit assignment for the MARL objective but the method does not do so.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0u_ioJ2dpN", "reviewer": "Reviewer_wZYm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I believe this paper aims to tackle a fundamental problem for MARL, i.e., the credit assignment problem. However, I find that most discussions and presentations in this paper are rather vague, which is very hard to understand for me.\n\nI have many questions on the main content:\n\n- Eq.(4) is a VDN-like value factorization. Why do the citations refer to QTRAN and QPLEX?\n\n- In Eq.(4), why $Q_{total}\\geq Q_{joint}$?\n\n- The ground-truth objective is $Q_{joint}$. Why do we need to maximize $Q_{total}$ in Eq.(5)?\n\n- I do not understand the constraint $\\sum_{i=1}^N \\alpha_i=1$. When all agents have full observations, the value of $\\alpha$ seems to be a one-hot vector (i.e., a greedy maximization).\n\n- I do not understand why we need to introduce GEM. A trivial maximization solution of Eq.(5) seems to be $Q_i\\equiv\\infty$.\n\n- Given that $\\alpha$ represents an explicit credit assignment, why these values are generated by a VAE? Proposition 2 definitely ignores the reconstruction loss.", "summary_of_the_paper": "This paper aims to establish an explicit multi-agent credit assignment approach with interpretability.\n\nThe proposed method, Explicit Credit Assignment joint Q-learning (ECAQ), is an end-to-end solution to the designed problem formulation. It can leverage the advanced deep-learning-based implementation of MARL based on CTDE.", "summary_of_the_review": "I vote for rejection since I cannot capture the main ideas of the proposed method from the current version. Most presented arguments do not make sense for me.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides an interesting method to address the CTDE problem in MARL. While the experiments are promising, the theory is either insufficient or not rigorous. One of the reviewers believe that there is a flaw in the paper. There was an extensive discussion among the authors and the reviewer. The authors could not convince the reviewer for the apparent flaw.
Decision: Reject