Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper presents an approach towards multi-agent reinforcement learning (MARL) in the context of centralized training with decentralized execution (CTDE). This paper considers the Individual Global Max (IGM) principle that involves consistency between the local action of every agent and joint action of all agents. 

The paper presents an approach that enforces IGM through an appropriate selection of a neural network architecture that transforms the IGM principle into constraints on the output range of an appropriate neural network. These constraints enable enforcing consistency that is necessary for enforcing the IGM principle.

Overall, I find the paper to be well written, and is well motivated. I went through the theoretical claims but haven’t been able to completely verify the claims of Proposition 2. In terms of experimental results, the paper presents results on a variant of a matrix game considered by prior work, two state MMDP and in both online and offline settings of a micro-management benchmark of StarCraft. 

While I believe the paper presents good contributions to MARL, I am not an expert in this area, so, I will mark my review with low confidence since I am unable to offer constructive pointers on possible improvements etc.
review_title: Interesting paper.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################
 
Summary:
 
The paper proposed a multi-agent Q Learning algorithm with an entire IGM function class for cooperative games. The key idea is to leverage a duplex dueling network architecture to factorize the joint action-value function into individual action-value functions. The main contributions of the work lie in that the proposed method offered an highly scalable algorithms for cooperative tasks. Empirical results show that the method could achieve significant improvement in StarCraftII tasks.
##########################################################################

Reasons for score: 
 
Overall, I'd vote for acceptance to the paper. I like the idea that is using a Transformer to learn to decouple the relationship between different agents. However, the main concerns of mine lie in the details of the proposed model(see cons below).
##########################################################################
Pros: 
 
Overall, the paper is well written. The motivation of the proposed method is straightforward and sound. This paper provides comprehensive experiments, including didactic problems and complex benchmarks, to demonstrate the efficacy and efficiency of the proposed methods.
The scalability of previous methods is one of the most critical issues of a cooperative multi-agent setting. The proposed methods give a practical solution for that.
The credit assignment problem is another exciting problem of a multi-agent system.
 
 
 
##########################################################################
Cons: 
 Although the main motivation of the proposed method is to address the scalability of previous methods, some details are unclear to me
Why does \lambda take joint-action and joint-observation as a parameter in eq 10. 
What is the computation cost to calculate of \lambda for every agent?
Although the proposed method conducts a few experiments, I still suggest the authors conduct the following ablation studies to enhance the quality of the paper: 
Why do the choice of layer and the number of heads are not aligned on matrix and StarCraft II tasks?
The ablation studies on Qatten
 
 
##########################################################################

Questions during the rebuttal period: 
 
Please address and clarify the cons above 
 
review_title: Recommendation to accept

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a new value-based method in cooperative multi-agent reinforcement learning. This paper proposes a duplex dueling network that can solve both the limited representation power problem existing in QMIX, and the approximation error problem due to neural network separation, existing in QTRAN. The authors show the novelty of the proposed method through theoretical proof and experimental results. This paper is well written, but there are two significant concerns:

1. The authors argue that QTRAN does not perform well in complex domains because it uses soft constraints, but QPLEX needs to demonstrate this more precisely. QPLEX uses two additional neural network technologies, and It seems the performance gains through these technologies are significant. First, QTRAN uses summation of individual action-values, while QPLEX uses summation of transformed action-values, and QPLEX constructs an advantage function using a scalable multi-head attention module. The authors need to show that performance is better than QTRAN even when these factors are removed. Also, in figure 2(b), QTRAN learns the global optimal policy quickly, but QPLEX learns slowly, so it seems necessary to explain. On the graph, in a matrix game, QTRAN learns more than five times faster than QPLEX.

2. An additional explanation of the authors' environmental selection criteria is needed. In the online data collection setting, the authors use environments such as 5s10z and 7sz, which seem to be environments that were not well used in existing MARL papers, and most environments only use stalker and zealot as agents. Also, the online data collection experiment and the offline data collection experiment use different environments. If the authors use a unified environment, the experiment results will be more enjoyable.

review_title: Need for further discussion on comparison with QTRAN

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a novel value decomposition approach to learn decentralized Q function in multi-agent setting. This idea is to follow Individual-Global-Max (IGM) principle. The main contribution is to use dueling structure (Q_i = V_i+A_i) for each agent i, and separately combining advantage/value terms respectively to form a centralized Q and A terms for training. Such combinations keep the positive correlation constraint between Q_tot and individual agent Q_i in QMix (i.e,. \partial Q_{tot} / \partial Q_i > 0) via positive trainable module in neural network, implemented by multi-head attention, etc. 

While this paper seems to be a reasonable contribution, a bunch of critical issues exist in the experiments so I vote for rejection. 

1. The idea is quite similar/incremental to QMIX, but instead of a mixing network, the authors use two-stages (first a linear weight that depends on all states, then a linear mixture over all agents with attention) to enforce the positive correlation how an individual Q would affect Q_tot. I wonder what’s the key component that makes QPLEX much better than QMIX? Is that purely because you have more parameters in the mixing? There don’t seem to be ablation studies (the video has sub-captions called “ablation study” but no sentence on what’s going on in the two videos) and discussions. 

2. Qatten uses attention while the remaining baselines (e.g., QMix, VDN, QTRAN) do not have attention. Is QPLEX working because of attention rather than the dueling structure? How is QPLEX compared with a version of itself without attention?

3. In StarCraft Multi-agent Challenge, 5s10z is neither a hard or a super hard map. The hard maps are 2c_vs_64zg, bane_vs_bane, 5m_vs_6m and 3s_vs_5z, and the super hard maps are MMM2, corridor, 3s5z_vs_3s6z, 6h_vs_8z and 27m_vs_30m (See Tbl. 1, Fig. 5 and Fig. 6 in SMAC paper). I wonder why Fig. 4 and Fig. 5 are tested on different set of maps and how does QPLEX perform in hard/super hard maps (e.g., no comparison for online setting in 2c_vs_64zg, and for offline setting, QPLEX doesn’t see to perform well on 2c_vs_64zg). Note that the Qatten paper shows performance on these hard/super hard tasks. 

Minor: 
1. What does the strikethrough mean in Fig. 2a (pay off matrix)?
review_title: Review

Meta-review: The majority of reviewers recommend acceptance for this paper, and the average score is in the acceptance rate. Only one reviewer (reviewer 2) recommend rejection, and from the reading the review, the authors answer, and the paper, I think it is possible that the reviewer missed the motivation behind this architecture, which is partly reason for rejection. Unfortunately the reviewer did not answer the authors so I cannot be sure if the reviewer is aware of that. Therefore, I am confident in my recommendation to follow the majority of the reviewers.

The reviewers generally believe this paper is well written. The paper has a good structure and although quite technical, is still easy to follow.

Concerns were raised about the scale of the experiments and motivations for some experimental choices. 
I appreciate that the authors have extended the set of evaluations on the Starcraft task, and added ablations studies, which partly address some of these concerns.

Decision: Accept (Poster)