Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
This paper presents an improved version of QTRAN [1]. The design is based on new loss function design, as well as new action-value estimator designs. The paper claims superior perfromance gains compared to previous methods on the Starcraft Multi-Agent Challenge (SMAC) environment.

### Strengths
+ The ideas proposed to improve the previous QTRAN (or might be applied to other MARL algorithms as well) seems novel and general.
+ The authors perform comprehensive ablation studies for different components they proposed.
+ The empirical performance on the SMAC benchmark is better and more stable across different runs.
+ The writing is clear and easy to follow.

### Weaknesses
- Only one environment is evaluated, which might not be that convincing. It would be good to see more results on different benchmarks.
- In Figure 3, some results seem to be not converged yet. Since the metric is the win rate, which is bounded, it would be interesting to see given enough training steps, whether all methods can actually converge to similar winning rate, or inherently the proposed scheme can lead to better results.

### Minor issues
Typo: Page 6, "... for being “selfish.” This ..." -> "... for being “selfish”. This ..."


Considering all the aspects, I tend to accept the paper in the current stage.


### Reference
1. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. 2019.

----------------------------------------------------------------------

**Updates**: After reading the other reviews and the rebuttal, I still maintain my current score. The additional experiments on the converged results are good to me. As I'm not very familiar with the performance in MARL literatures, I have decreased my confidence from 3 to 2 to reflect some of the concerns of other reviewers involving the performance for the baselines.
review_title: Interesting work; evaluation can be improved

final_score: rating: 7: Good paper, accept
review_content: ## Summary

This paper addresses the domain of cooperative multiagent learning with centralised learning and decentralised execution. Specifically, it improves on the QTRAN algorithm, a theoretically justified algorithm which previously had not produced strong learning performance. With these improvements, QTRAN++ outperforms baselines on the SMAC environments.

I recommend accepting this paper. It delivers strong performance on a popular benchmark for complex, cooperative multiagent learning (SMAC). While the algorithmic contribution is incremental, it still delivers insight into how to improve the empirical performance of a theoretically interesting and well-justified algorithm.

 ## Positives

The problem addressed - cooperative multiagent environments with CTDE - is a widely studied and important one. It is well set up in the paper, including discussion of related algorithms.

The base algorithm - QTRAN - should theoretically perform well in a wider variety of environments than other algorithms for these problems, so improving its performance is particularly valuable. The improvements made to the algorithm are clear and well motivated; section 3.1 in particular explains clearly the difference the modified loss is intended to make.

The empirical studies in the paper are strong. They show that QTRAN++ outperforms several baselines in data efficiency and final performance across a variety of domains. Further, a comprehensive ablation study shows that each of the improvements made to QTRAN is independently important (in at least some domains).

## Negatives

The algorithmic contribution of the paper is relatively minor, since it provides fairly simple modifications to an existing algorithm.

Experimentally, it would be good to see experiments on similar domains to those addressed in the original QTRAN paper, which are designed to probe the advantages QTRAN has over related algorithms. This would demonstrate that QTRAN++ retains the benefits of QTRAN in non-monotonic factorisable environments.

## Sources of reviewer uncertainty

I am not knowledgeable enough in this domain to be certain of the coverage of the baselines and domains in the paper. Since the empirical performance of the algorithm is central to the paper, this is important.
review_title: Good paper - simple fixes to an important algorithm, yielding SOTA performance.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary and claims

This work proposes a MARL (multi-agent reinforcement learning) algorithm.
In the MARL setting, multiple agents have to make choices based on independent information to maximize a common objective. An existing algorithm in this space is QTRAN.
The authors propose several modifications to QTRAN: changing the architecture, adding two additional constraints to the loss function and also allowing gradients to flow from the QTRAN objective into the "true" action-value estimator.
The claims of the paper are:
1) QTRAN++ achieves better performance than QTRAN
2) The modifications introduced stabilize training compared to QTRAN

It took me some time to fully understand all of the components that go into QTRAN++ and I find the complexity of the overall algorithm pretty surprising, especially considering that other algorithms in this space are as simple as "add up all the Q values of the individual agents".
I think the proposed changes consist of:
 - Rather than directly training action-value networks, a QMIX-like hypernet approach is used
 - The network that estimates the "true" (combined) action-value is implemented through what the authors call a "semi-monotonic mixing network", which is the sum of a non-monotonic (regular) hypernet and a monotonic hypernet as used in QMIX. This seems pretty arbitrary. Isn't the original idea behind QTRAN that this would accurately track the true values?
 - In QTRAN the separate network that aggregates the Q values of the individual agents is trained to track the "true" action-value. In QTRAN++ this is done through multiple hypernetworks (the authors call these "heads").
 - The loss function is modified to impose two additional constraints on the transforming value function.
 - Gradients are now also backpropagated from the "tracking loss" into the "true" action-value estimator, which makes it somewhat unclear what it is actually representing.

### Relation to prior work

The paper is positioned sufficiently with respect to prior work. I've noticed that there is a larger section on related work in the appendix. I'm not sure what the purpose of moving the related work into the appendix is, especially if some of the papers mentioned there are not actually related to the work presented in this paper. I think it would be good to try to move as much as possible of that section into the main text, leaving out prior work that is not sufficiently related.
Some of the additions in QTRAN++ seem very similar to ideas proposed in QMIX, but this is not directly acknowledged as far as I can tell. It would be good to point out which parts of the architecture come from QMIX.

### Are the claims supported?

The experiments presented in the paper are reasonably thorough and show that
QTRAN++ consistently outperforms QTRAN on the tasks that were tested. SMAC (StarCraft Multiagent Challenge) is a nontrivial benchmark, so I would agree that claim 1) has been shown sufficiently. But I'm not sure whether the ablation studies are thorough enough to really demonstrate that all of the components of the (rather complex) proposed algorithm are really needed.

The authors often make claims about improved stability and other properties of the algorithm throughout the paper, but these are not supported by any empirical evidence. If the authors want to claim that QTRAN++ outperforms QTRAN because of a specific mechanism then it would be good to provide some sort of empirical evidence or proof (the proof in appendix A doesn't count since it doesn't make any statements about stability). Therefore I think that claim 2) is currently not well supported and it would be good to either support it better or soften the statements in the paper to make statements in the form of "we believe that the algorithm has improved stability".

### Presentation and clarity

The paper is reasonably clear and understandable. There are some cases where incorrect grammar or word choice made a sentence difficult to understand. For example the choice of "affluent" to describe a class of estimators. It would be good to address cases like this to improve the clarity of the paper.

### Conclusions

The main claim of the paper (that QTRAN++ is an improvement over QTRAN) has been demonstrated sufficiently. But the high complexity of the approach (with several additions to the algorithm feeling somewhat arbitrary) and the fact that the paper "merely" presents an upgrade to QTRAN could be potential arguments against accepting it.


### *Edit after author comments:*

I have read the author comments and the latest paper revision. The authors have noticeably improved the clarity of the paper in several places and adjusted their claims about the stability of the algorithm, and the improved ablation studies are appreciated. Unfortunately, after thinking it through very carefully and despite the author comments, I have not been able to understand some aspects of the model, for example why gradients from the tracking loss are backpropagated into the value function that is supposed to track the "true" action-values. Several parts of the architecture seem to have a complicated dual purpose, which makes it difficult to understand what is going on and why the model is performing better. I suspect that other readers might also encounter similar issues, which makes it difficult for me to raise my rating. I've decided to leave the rating at 6 (marginal accept).
review_title: Improvement over QTRAN is demonstrated. Approach is complex. Feels somewhat incremental.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper provides good improvements that make QTRAN more practical and can be applied to problems other than matrix games. Since QTRAN is a significant improvement for value-based multi-agent reinforcement learning after QMIX, the practical implementation of QTRAN is expected for a long time in the community. However, after the publication of QTRAN, especially in 2020, some other works have explored the question of how to extend QMIX to the full IGM function class. Due to these works (QPLEX is the major concern, given that weighted-QMIX has been compared in the experiments), the expectation of the advanced version of QTRAN is higher than before. I have several concerns regarding several core contributions of QTRAN++.

QTRAN++ relies heavily on the true joint action-value function. (1.1) However, learning joint action-value functions is not an adorable choice in multi-agent problems. (1.2) To ease the training and representation of joint action-value functions, the authors condition $Q_{jt}$ on individual q values and use a semi-monotonic structure. However, it is difficult to tell the contribution of the monotonic part. It has been shown that monotonic functions can not represent some Q-values. Why should this part be included? I expect that I can find the answer from ablation studies, but on two out of three scenarios, FC-QTRAN++ is very similar to QTRAN++. The authors can provide a more serious discussion of this part to make their paper stronger.

About the training of $Q_{tran}^{i}$. I have two questions about the training of this value function. (2.1) When training $Q_{tran}^{i}$, whether local utility function of agent $j$ ($q_j$ using the notation from the paper) is updated? (2.2) The training scheme is a midpoint between VDN and QMIX, which is similar to an attention mechanism that has been explored in multi-agent value decomposition settings. The formulation is quite different from previous papers (DOP [Wang et al., 2020] and REFIL [Iqbal et al. 2020]), but based on the results from these previous work, I think the multi-head structure may not improve the performance. Although the authors use a matrix game to illustrate their idea, which I appreciate, I can hardly tell whether this example is specially designed. I was expecting a convincing ablation study on SMAC, but I do not find them sufficient: (1) The authors did not record how many random seeds did they test, and SMAC tasks are typically sensitive to random seeds. (2) The gap between QTRAN and QTRAN++ is not significant. If the authors can provide results with more random seeds on more maps, I will consider revising my rating.

My last concern is about QPLEX, as cited by the authors. Similar to QTRAN, QPLEX provides full expressivity for the IGM function class. Nevertheless, the implementation of QPLEX seems to be much more lightweight than QTRAN++. Since QPLEX has provided codes that can be freely tested on the SMAC benchmark, I was wondering why the authors cited this paper but did not compare to it. At least, a detailed discussion of the differences can make the contribution of QTRAN++ clearer.

** A minor concern about experiments.
It seems that the authors are using an older version of QMIX. In the latest version, QMIX can achieve a win rate pf 80% on MMM2. This fact is unknown for many, because the journal version of QMIX reports the same win rate as in this paper.

[Wang et al., 2020] Wang, Y., Han, B., Wang, T., Dong, H. and Zhang, C., 2020. Off-Policy Multi-Agent Decomposed Policy Gradients. arXiv preprint arXiv:2007.12322.

[Iqbal et al. 2020] Iqbal, S., de Witt, C.A.S., Peng, B., Böhmer, W., Whiteson, S. and Sha, F., 2020. AI-QMIX: Attention and Imagination for Dynamic Multi-Agent Reinforcement Learning. arXiv preprint arXiv:2006.04222.



##### ======================== UPDATE =========================

Thanks for the authors' clarifications.

After a careful re-evaluation of the paper, I have many concerns about the performance of baselines on the StarCraft II benchmark tasks. The reported performance is not consistent with those reported in the SMAC benchmark paper (see Figure 4,5,6 in [1]) and QPLEX paper (Figure 5,8,19 in [2]). Moreover, I also evaluate the available GitHub codes of baselines on my own, which is consistent with [1,2].

Using results in [1,2], QTRAN++ significantly underperforms the baselines on the StarCraft II benchmark tasks. Moreover, the paper claims that it uses the standard StarCraft II benchmark, the latest version of SC2, and the default baseline codes.

Due to these concerns, I tend to lower my rating.

[1] Samvelyan M, Rashid T, de Witt C S, et al. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043, 2019.

[2] Jianhao Wang, Zhizhou Ren, Terry Liu, Yu Yang, and Chongjie Zhang. Qplex: Duplex dueling multi-agent Q-learning. ICLR submission. https://openreview.net/forum?id=Rcmk0xxIQV
review_title: QTRAN++ is a good improvement for QTRAN, but the expectation of the advanced version of QTRAN is higher than in 2019.

Meta-review: This paper proposes practical improvements to theoretically well founded QTRAN, which is a state-of-the-art technique of cooperative multi-agent reinforcement learning.  The improvements include new designs of loss function and action-value estimator, which might be widely applicable beyond QTRAN.  However, it is not obvious if the proposed improvements actually improves the performance of QTRAN, and experimental evaluation is essential to this work.  After the discussion, there remain some major concerns about the experimental results.  In particular, the performance of baselines in the experiments is not consistent with those reported in the prior work.
Decision: Reject