Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper describes a new method for learning factored value functions in cooperative multi-agent reinforcement learning. The approach uses energy-based policies to generate this factorization. The method is presented and experiments are given for smaller domains as well as starcraft. 

The idea of learning factored value functions is promising for learning separate value functions for each agent that allow them to learn in a centralized manner and execute in a decentralized manner (centralized training and decentralized execution). Several methods have been proposed along these lines, but as the paper points out, they have limitations that makes them perform poorly in some problems. 

The proposed approach in this paper has some promising experimental results, but there are questions about the novelty and significance of the method. Furthermore, evaluating these contributions is difficult due to the lack of clear details in the paper. 

In particular, the details of the approach itself in 3 are not clear. Starting with Definition 1, it seems like IGO is using an optimal *centralized* policy. Is this what is meant? If so, why is this needed (as opposed to an optimal decentralized policy). It will typically be impossible to achieve a centralized policy with decentralized information. Furthermore, the energy-based policies are defined in 3.2, but 'key' ideas such as approximating the weight vector aren't fully explained making the exact approach hard to determine. Also, it is beneficial that the current theorems and proofs are included, but the lack of sufficient detail makes it hard to parse and evaluate them. 

There are also similar max entropy approaches, such as the paper below. 

Iqbal, S. & Sha, F.. (2019). Actor-Attention-Critic for Multi-Agent Reinforcement Learning. Proceedings of the 36th International Conference on Machine Learning, in PMLR 97:2961-2970

As well as other factorized methods, such as the papers below (which are admittedly new). 

Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. Tabish Rashid, Gregory Farquhar, Bei Peng, Shimon Whiteson. NeurIPS 2020.

de Witt, Christian Schroeder, et al. "Deep Multi-Agent Reinforcement Learning for Decentralized Continuous Cooperative Control." arXiv preprint arXiv:2003.06709 (2020).

The paper should discuss how the proposed method is an improvement over this other work and have a more comprehensive related work section. 

The experiments are promising, but the relevant related work is not included and there isn't sufficient detail describing how the methods were run and discussing the results. In terms of comparisons, the paper should also need to compare with non-factored state-of-the-art methods. It is, of course, natural to compare with other factored methods, but what matters is general state-of-the-art performance of the domains. 

As noted, the clarity and writing of the paper should be improved. Beyond the examples above, some other instances are below. 

- If the reader doesn't already understand the relative overgeneralization problem, Section 2.3 probably isn't sufficient. Figure 1 is helpful, but it should be described in the text to make the issue clear. 

- The connection between the overgeneralization problem and factored representations isn't completely clear. Factored representations have problems because they typically cannot represent the optimal value function (or policy). That is a separate issue than getting stuck in a local optimum (which can happen with any type of method).
review_title: Blind review

final_score: rating: 2: Strong rejection
review_content: The paper proposes a Q-factorization method by assuming an energy-based policies model. Q-functions are formulated as soft value functions with the energy parameters, and this adoption renders the function factorization more flexible compared to existing ones. The proposed solution applies to continuous-action tasks, a feat left unconquered by some of the existing methods. Authors exhibit that FSV outperforms others in various environments characterized by local optima.

Strengths:

+ The formulation of Q-functions as soft functions, despite appearing simple, shows some effectiveness in a number of MARL tasks.

+ The network architecture is intuitive.


Major Concerns:

- Neither energy-based policies nor soft value functions is an original contribution of this work. True, the authors do not claim so. But the reviewer is left unsure as to what then the primary contribution of the paper would be.

- The method generalizes IGM to IGO but in doing so, foregoes the simplicity of the IGM condition. The reviewer would then expect to be met with a somewhat strong guarantee, but is instead presented with approximations on \lambda_i. It is not clear from the paper how much insightful value the method has, when its criticism of a previous work (QTRAN) was based on intractability but the FSV method itself still relies on approximations. It would seem as though QTRAN and FSV each chose different paths to approximate different components of an MARL training scheme - the former takes may stronger assumption on the value functions while the latter takes assumptions on the nature of value functions being parametrized by approximated weights.

- The effectiveness of the proposed method is not yet well-accounted for. Issues are raised, but little explanation (or any attempt thereof) is provided. For example, the reviewer would have very much liked to gain an understanding of the relevance between IGO and its ability to alleviate relative overgeneralization. How does taking on greedy policies (which makes IGO collapse into IGM) make MARL agents more prone to overgeneralize with respect to each other? What kinds of findings would the authors present? What evidence could support those findings? The evaluation, while illustrating great performance gaps, needs a careful redesign so as to construct solid grounds for the soft value function factorization under IGO to be "explainably" better than existing works.

- The paper could be better positioned. The Related Works section could be put to better use to clearly distinguish two very different lines of research: value function factorizing MARL works and maximum entropy principle.

- There needs to be some justification about multi-head attention being used to "enable efficient learning" in Section 3.3. The reviewer is left hanging as to why and how such a choice was made.


Minor Concerns:

* A few parts of the paper were difficult to follow. For example, there is an unfinished sentence in Related Works. In Section 2.1, there is an incomplete clause beginning with "the reward function [...] shared by all agents". Under Theorem 1, "any distributions" --> "any distribution". Also, what is meant by "correct architecture" in that same paragraph?
review_title: Q-functions and Policies

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a novel MARL framework named FSV, which incorporates the idea of energy-based policies and an efficient linear decomposition architecture in the joint action-value function with multi-agent maximum entropy reinforcement learning. Besides, the authors propose the IGO, which extends the IGM in stochastic policy cases. FSV suits in both the discrete and continuous action space scenarios. Experiments conducted on two simple examples with discrete and continuous action settings show that FSV could overcome the relative overgeneralization problem with the proper temperature setting. Furthermore, FSV in the challenging SMAC benchmark outperforms VDN, QMIX, and QTRAN in three scenarios.

Overall, this paper is well-organized and easy to read. The authors present interesting ideas that combine the energy-based policy and maximum entropy reinforcement learning into the centralized Q-value mixing network to obtain a better expression ability than VDN and QMIX and overcome the relative overgeneralization problem.

There are some questions.

Q1: Does the linear decomposition of Qtot of Qi and Vi limit the representation ability of FSV? It seems that FSV cannot represent the non-linear formation of Qtot and Qi.

Q2: In Section 5.1, it is better to show the estimated \lambda_i’s performance compared with the different \alpha_0 settings.

Q3: In Section 5.3, there lacks analysis of FSV and other methods. Especially, the ablation of FSV should be considered. In these three scenarios, which part (soft RL, critic’s structure, or others) contributes to FSV most and improves its performance steadily?

There are some typos.
In section 5.2, “on others’ policies (?).”
In section 5.3, “exploration efficiency” -> “exploration efficiency.”

As pointed out by other reviewers, this paper shares much similarity with QPLEX but without reference and discussions, thus I would be inclined to reduce the score as well.

review_title: A novel paper which factorizes soft value function with stochastic policies. 

final_score: rating: 2: Strong rejection
review_content: This paper is concerned with the problem of cooperative multi-agent reinforcement learning for CTDE scenario which is well studied in recent literature. The authors propose a factorisation method based on soft value functions. I found that the paper is extremely poorly written which makes it very difficult to understand the overall method. The presentation is also quite arbitrary with discussion around results that seem unnecessary. There is little novelty as most of the paper borrows from SAC paper by Harnooja et al, albeit with gross errors in copying. Here are some of the  major issues:

1. The authors discuss the IGO decentralisability, however what is the relation between IGO optimal policy and soft policy when the former is not representable by latter?

2. How does the local soft policy iteration guarantee joint policy improvement? 

3. Why is the * being arbitrarily switched in sec 3.2? What does eq 12 even imply? isn't the KL minimiser $\pi_i^*$ itself? Where is $\Pi$ defined?

4. How did eq 18 come about?

5. The paper is full of unbacked blanket statements like: " Although energy based distribution is very general which has the representation ability of most tasks," "Our method are a member of them but out of the deterministic policy" etc.

6. There are many unintelligible sentences like: "we need to extend the function class into any distributions", "IGO is more generality than IGM", "The individual value network is trained by minimize", "relative overgeneralization, where finding a
suboptimal Nash Equilibrium, which is a well-known game-theoretic pathology" etc.

7. In proof for Theorem 2, $\epsilon$-greedy eq 26 cannot be matched by a soft policy in general, thus the rest of the proof cant follow without corrections.



review_title: Poor writing with incomplete/inconsistent proofs

final_score: rating: 3: Clear rejection
review_content: In this paper, the authors use a method in soft RL style and a QPLEX factorization to propose the first multi-agent value function decomposition (VFD) method for stochastic policies. I appreciate the efforts to extend VFD to a larger function class.

However, I am confused about the similarity to QPLEX [https://arxiv.org/abs/2008.01062v1 ], which was published several months ago. One of the main contributions of the paper is the value function factorization stated in Theorem 2. However, it is identical to QPLEX, including notations. Specifically, Eq. 7 and 13 of FSV are the same as Eq. 8, 10, and 12 in QPLEX, including notations.

Moreover, some statements in the paper and details of the figures are very similar to those in the QPLEX paper.  Eq. 33 of FSV is the same as Eq. 50 of QPLEX. These equations describe an implementation detail where QPLEX and FSV stop gradients at the same variables. Stopping gradients is reasonable in the context of QPLEX, and the reason for using this trick is well-motivated there. By contrast, why this trick is necessary for FSV remains largely unclear. I encourage the authors to explain why they stop gradients here in the context of soft reinforcement learning.

In summary, the similarity including theorems, equations, and notations (and even figures). Despite these similarities, the authors did not cite QPLEX, which has been online for several months before FSV is published.

Additionally, about the integration of QPLEX and soft Q-learning, I also have some concerns. The definition of soft value functions depends on the specific selection of the temperature parameter. In this paper, the temperature parameters are end-to-end learned by minimizing the temporal-difference error of Q-learning. Could the authors explain why this makes sense in the framework of soft Q-learning? The counterpart in QPLEX ($\lambda$ in Eq. 10) is end-to-end learned to ensure the rich expressivity of QPLEX so that it can represent the complete IGM function class. This design is well-motivated in the original QPLEX paper.


** Minor points
Some of the claims in the paper need refinements. For example, in Para. 2 of the introduction, "Centralized training with decentralized execution (CTDE) (Oliehoek et al. (2011)) is a common paradigm to address the partial observability". This claim is not solid. Partial observability can still induce miscoordination in decentralized execution [Wang et al. ICLR 2020, https://openreview.net/forum?id=HJx-3grYDB]. That is to say, the framework of CTDE itself cannot solve the problem of partial observability. It is better to say something like "CTDE with communication." I also find the claim "Value function factorization methods have been an increasingly popular paradigm for solving the scalability in CTDE" not convincing. In fact, QMIX, which is a VFD method, generally can not work very well in tasks with more than 20 agents.

Typos: Last paragraph in the introduction: "it significantly outperforms other baselines, SMAC (Samvelyan et al. (2019))." SMAC is not an algorithm.
review_title: Without citation to an earlier paper, QPLEX [https://arxiv.org/abs/2008.01062], but the factorization and many details are identical.

Meta-review: Although the paper presents some interesting ideas, in general the reviewers agree that the paper lacks clear results and is not an easy read. The paper proposes a factorisation of value functions, a topic that has received quite some attention in the literature (e.g. QPLEX), and it seems that their is not sufficient innovation in the proposed method in the paper. There are also a number of claims in the paper (e.g. partial observability etc.) with which some of the reviewers disagree, and should be discussed more carefully in a revised version of the article, that all in all seems to need more work.
Decision: Reject