Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper takes a hierarchical approach rather than the now fairly common flat approach to factorizing joint value function for multiple agents in multi-agent reinforcement learning. Sub-team assignments can change for agents over time. Additionally, the paper explores different sub-team assignment strategies and proposes a meta-gradient based approach to optiize these assignments. The proposed method is compared against baselines that have a flat factorization on multiple grid-based domains and shown to outperform them especially as with increasing number of agents.
main_review: **Strengths**

The paper is clearly written and the experiments are well done with results reported over 30 runs. Moreover the experiments go to about 80 coordinating agents unlike most other works in this domain. Unlike many papers, the delta difference at the end of training is quite substantial as the number of agents increases. Ablations with respect to different assignment strategies are quite useful even though they don't seem to matter always. The shared source code is fairly easy to follow as well, so kudos for that.

**Weaknesses**

Biggest weakness is the paper's lack of recognition of alternative value function factorizations based on ideas of coordination graphs [1,2,3]. These are not exactly flat, but can't be claimed to be hierarchical in this manner. But given the paper uses a different set of task domains than the other papers, it's difficult to infer the pros and cons of these different approaches. Nevertheless, at least some discussion is warranted. From the code it seems the assignment learning is another stochastic optimization problem that would likely become difficult with larger number of sub-teams and is not _just_ about linear scaling. That is possibly what is happening with Battle[80]. Some more discussion would be useful here. In general, the paper reports a bunch of results but doesn't try to figure out or given any intuition for why it could be happening. I understand that this is difficult and often not feasible with deep neural network based methods, but still some hypotheses would be useful!

[1] W Böhmer et al. "Deep Coordination Graphs" https://arxiv.org/abs/1910.00091

[2] S Li et al. "Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning" https://arxiv.org/abs/2006.11438

[3] N Naderializadeh et al. "Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learning" https://arxiv.org/abs/2010.04740
limitations_and_societal_impact: The paper goes in quite a bit detail in Sec. 8 for potential societal impact. As mentioned above an important thread from the literature is missing but otherwise the limitations are interspersed through the text.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces VAST, a framework to train decentralised MARL agents to each learn local policies from a single global reward. In contrast to VFF, where each agent calculates a local Q function which is then aggregated in a flat way into a total Q function to which Deep Q-Learning may be applied, VAST first assigns agents into variable-agent sub-teams, linearly aggregates the local Q function of each sub-team into a group Q function, and then aggregates the group function into a total Q function in the manner of VFF. The authors hypothesise that VFF underperforms with large numbers of agents due to a performance bottleneck that limits the information received by each agent, and empirically demonstrate a range of experiments where VAST achieves superior performance to flat versions of VFF when training large numbers of agents. They further discuss different ways of assigning agents to sub-teams, which may vary between time steps, and introduce an effective meta-gradient based approach.
main_review: Originality: This is a good, original work. It provides a novel variant of existing VFF techniques, building upon previous work while citing clearly.

Quality: It is technically sound, and the performance of VAST is well-supported with clear experimental results. 

A major limitation is that there is insufficient discussion of the group ratio, and how this might be chosen. A key contribution of the work is the ability to scale VFF to larger numbers of agents, but the smallest group ratio considered is ¼, only a 4x decrease in the number of entities considered. It is likely that in settings with many agents we would want a significantly larger decrease. Eg, it would be instructive to see the case with just 2 or 3 sub-teams explored.

Clarity: The work is overall fairly well-written, well-organised and easy to follow. In particular, the background section was helpful and informative for putting the paper’s contribution into context. The experiments were well presented and clearly explained. A few points were unclear, as discussed below.

Significance: VAST outperforms previous approaches to VFF on MARL problems with large numbers of agents, a valuable contribution to an important problem. 

The theoretical basis given for this work is shaky. The relative performance increases with larger numbers of agents, providing suggestive but not conclusive evidence for the hypothesis that this is due to a performance bottleneck. The authors should give more consideration to alternative explanations. Nevertheless, the empirical evidence for VAST is clear and persuasive.

Detailed comments:
- In 4.1 the ideal group ratio is described as $\frac{1}{N} < \eta \ll N$. As $\eta < 1 < N$, this is unclear, and $N$ should be replaced with $1$
- The phrase “performance bottleneck” feels misleading, and when reading the abstract I interpreted the authors as arguing that VFF was computationally intractable with large numbers of agents, and that VAST would resolve this.
- In 4.2 the discussion of the meta-gradients method for assigning teams was confusing, and could do with further detail, especially as we see that team assignment has a significant effect on performance. In particular: What does J mean, and how was this chosen? What are other reasonable choices of J, and how might this affect performance? How is $R_t$ included in calculations when this depends on the unknown future reward?
- Theorem 1 seems correctly proven, but is rigorously stating and proving the simple statement that “as team Q functions are linear aggregations of agent Q functions, they are maximised iff agent Q functions are maximised, so the total Q function is maximised iff each agent Q function is maximised”. To aid in reader understanding, the theorem should be accompanied by a more intuitive description.
- In Figure 6a-6c, $K=20$ yet only 2-4 groups are shown. The paper should clarify whether sub-teams are non-empty, and the impact on train time and performance of having many sub-teams with most empty vs having few sub-teams all non-empty.

limitations_and_societal_impact: The authors discuss and acknowledge a range of possible negative societal problems, including use in autonomous weapons and traffic jams. I consider the most significant possible negative societal impact to be use in autonomous weapons, and would want to see a more detailed discussion of this. Eg, how often is MARL used today in autonomous weapons, and how much could this work further enable that?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a novel value factorization method for multi-agent Q-learning based on team organization.
To enable automatic self-organization, this paper presents a meta-gradient-based approach to explore team structures.
The experiments demonstrate the proposed method can outperform baseline value factorization structures.
main_review: **Originality**: VAST is well embedded in the widely-used CTDE  paradigm, and its novelty is clearly discussed. To my knowledge, the proposed factorization method is novel.

**Quality and Clarity**: The paper is well organized and easy to understand. Especially, Figure 1 is very clear and helpful to understand the proposed factorization structure.

**Significance**: Overall, this paper is a good start and attempt to study team coordination in multi-agent reinforcement learning. However, I have a few technical concerns (see questions).
limitations_and_societal_impact: I am willing to increase my score if my concerns are addressed. (1) and (2) are my major concerns, and (3) is minor.

(1) From my perspective, the concept of team coordination is a little bit weak in VAST. In general, VAST represents a subset of IGM factorization which focuses on individual execution. The selection of teams seems to only impact centralized training. In addition, I think it would be better to discuss some related work of coordination, such as deep coordination graph [1], and remark the differences from this paper.

(2) I am not sure whether I understand section 4.2 correctly. The proposed meta-gradient-based approach seems to require on-policy training. Whether this component will limit the sample efficiency and scalability?

(3) It is known that QTRAN may become unscalable in hard tasks, such as the StarCraft benchmark. It would be better to consider some more advanced techniques [2,3] and conduct more complex environments.

*References*

[1] Böhmer et al., Deep Coordination Graphs, ICML 2020.

[2] Rashid et al., Weighted qmix: Expanding monotonic value function factorisation, NeurIPS 2020.

[3] Wang et al., Qplex: Duplex dueling multi-agent q-learning, ICLR 2021.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This method proposes a hierarchical decomposition for value function factorization methods which enables significantly greater scalability in terms of the number of agents.
main_review: # Strengths
* The paper successfully addresses a significant shortcoming of a popular class of algorithms for deep MARL.
* The scalability of the method with respect to agent quantities is particularly commendable.

# Weaknesses
* Section 4.2, on the meta-gradient method for learning $\chi$, should be significantly expanded. Whereas the rest of the method is relatively straightforward, this part is a bit more subtle, and it would be good to read more on the motivation for and methodology behind this choice. In fact, the approach to learning $\chi$ as described in this paper looks more like a straightforward policy gradient approach, rather than the cited meta-gradient method. The cited method is concerned with meta-learning a set of parameters that directly influence the objective function of the policy/value function; whereas, this paper is concerned with learning the parameters of $\chi$, which can just be seen as additional parameters of the value function.
It's also not clear to me why $\chi$ cannot simply be learned in an end-to-end manner where it is implemented, for example, as a differentiable sampling procedure via Gumbel-Softmax.
* The authors do not evaluate on the StarCraft Multi-Agent Challenge [1], the standard benchmark for value function factorization-based cooperative MARL methods, nor do they compare to the most recent state-of-the-art in this field (QPLEX [2]). This omission suggests a lack of scalability to more complex non-gridworld domains.

## Minor Points
* Using $\mathcal{D}$ to represent the set of agents is slightly confusing, as it is typically used to denote a replay buffer or dataset.
* The methods section is a bit dense with respect to notation.

# Questions/Comments
* A recent approach, REFIL [3], proposes a method for extending QMIX to variable-sized teams via attention mechanisms as part of their approach. This may be worth looking into for future extensions of this work, such that the sub-teams may use a more expressive operator than VDN for their factorization scheme.
* Lines 207-209 "Since value-based algorithms are highly sensitive w.r.t. the exploration decay schedule, we completely omit exploration tuning and use $Q_i$ as critic for local actor-critic learning to realize $\pi_i$ instead " - this connection isn't quite clear to me. Actor-critic methods do not implicitly alleviate problems with exploration, they just deal with them differently (e.g. entropy penalty on the policies).

# Overall Thoughts
I like the main idea and motivation behind this paper, as scalability is a major concern for current MARL methods; however, I am a bit troubled by the lack of description in section 4.2. I am not very familiar with the cited meta-gradient method, and upon reading that paper, it is not clear how what is described in section 4.2 relates to that work. As such, I hope the authors can provide a more thorough explanation of this section in the rebuttal, such that I can evaluate their work more effectively.

[1] Samvelyan, Mikayel, et al. "The starcraft multi-agent challenge." arXiv preprint arXiv:1902.04043 (2019).

[2] Wang, Jianhao, et al. "QPLEX: Duplex Dueling Multi-Agent Q-Learning." International Conference on Learning Representations. 2020.

[3] Iqbal, S., Witt, C. A. S. D., Peng, B., Böhmer, W., Whiteson, S., & Sha, F. Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021
limitations_and_societal_impact: The authors could spend some time explaining why the approach does not scale to more complex domains (e.g. StarCraft).
review_title: null

Meta-review: The reviewers in most part recognise the novelty/originality of the presented approach.  The experiments, through which a strong  case is made for the scalability of the method to large number of agents, seem solid and competitive with state of the art. Some concerns raised by  Reviewer t76M about the clarity of section 4.2  on metagradient subroutine which needs to be addressed in the final version.
Decision: Accept (Poster)