Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes QSCAN, which considers local coordination within a subset of the agents and dynamically solves each task by sub-team coordination. In order to incorporate sub-team coordination, QSCAN extends the monotonic mixing network in QPLEX and achieves the full representational capacity of IGM. The experimental results showed that QSCAN can outperform QPLEX and QMIX in a number of benchmarks, especially Predator-Prey and Switch4.

strengths_and_weaknesses: Strengths
- The authors provide a good motivation for coordinating sub-teams, which can be beneficial for tasks where agents should divide into groups and focus on solving each sub-task. 
- The paper provides a nice sub-team generalization of some MARL algorithms where  QPLEX can be interpreted as QSCAN_n and QMIX as QSCAN_0. They provide a thorough theoretical analysis for various properties of QPLEX and QMIX such as the equivalence between QMIX and IGM within individual sub-team assignments (Proposition 1). 
- The experimental results for Predator Prey and Switch4 are interesting. The environment states and attention maps given in Figure 6 provide interesting insight into the potential reasons for choosing a particular action. This can be viewed as one evidence for communication among agents via actions


Weaknesses:
- There is a lack of adequate comparisons with relevant benchmarks in sub-team coordination, especially VAST[18 (supp.)], which also proposes variable sub-teams beyond pairwise coordination and retains the IGM consistency. Sub-team coordination itself is not a novel concept, and I believe it is necessary to emphasize the differences between QSCAN, VAST [18], and other sub-team formulation algorithms (if any), both qualitatively and quantitatively as benchmarks.  VAST [18] is also working in cooperative MARL so it could be misleading to categorize them in the same group as the ones using coalitions and Shapley values.
- Related to the first point, the paper has a brief discussion of GNN-based approaches which are a different approach with similar motivations. The authors write in the supplementary material A that with GNNs, “it is difficult for them to extract hierarchical coordination patterns.” However, this claim was not verified in the experiments. 
- With the lack of benchmarks against similar sub-team formulation algorithms and GNN-based ones, it is hard to agree with the statement that QSCAN “significantly outperforms” other benchmarks in the predator-prey and Switch4 tasks. Moreover, QSCAN only performs on-par with QPLEX on the SMAC tasks, and the improvement over QPLEX in the matrix game also appears marginal. So overall, the experimental results are not entirely convincing.
- There is mention of “relative overgeneralization”, “credit assignment”, “faster learning” scattered around the paper, but it is not clear what the main problem is that QSCAN aims to solve. I understand the intuitive motivation for sub-team formulation, but there is no characterization of the types of tasks in which the proposed algorithm is beneficial.
- Related to the previous point, the illustrative example given in Figure 1 is confusing since it only shows an example of when a sub-team is sufficient for solving the task. On the other hand, the experimental results seem to suggest that the main point of using sub-team formulation is to have a more flexible algorithm which can solve tasks which QPLEX/QMIX cannot, i.e. tasks in which sub-team formulation is necessary. Again, the main argument of the paper is ambiguous, especially with respect to why the extension of QPLEX is specifically the better method.
- VAST [1] shows that sub-team formulation can be useful for when the number of agents is sufficiently large. This is not necessarily a weakness of QSCAN but again, the main benefit of using QSCAN is not entirely clear. The benefit of using sub-team coordination (e.g. scalability to # of agents, fast convergence, etc.) and the benefit of using QSCAN in particular has to be properly addressed.
- The algorithm is derived from a fairly simple extension of QPLEX, where the architectural figures look almost identical. 
- (Minor) Figure 6 is ineligible without zooming in.

EDIT: Much of my initial concerns in the review were clarified by the responses by the authors. The authors have shown that the flexible sub-team coordination is a novel problem which was not tackled in previous work. Furthermore, they have added additional experiments to show that the performance is comparable to VAST and scalable to a larger number of agents. To reflect this, I’ve raised my overall score (3→5) as well as the soundness (2→3) and contribution scores (2→3). 

questions: 1. What is the main benefit of using (1) sub-team coordination in general and (2) QSCAN in particular?
2. What is the main difference between QSCAN and VAST ?
3. How does QSCAN overcome the relative overgeneralization mentioned as an issue for QMIX?
4. Right before Proposition 1, what is the meaning of “satisfy the IGM Condition consistently during the course of training”?
5. Can we really say QSCAN significantly outperforms the benchmarks in the matrix game? Also, what do you mean by poor generalization for QPLEX in 5.1? QPAIR seems to have the same performance as QPLEX in the Matrix Game, around a reward value of 22. Thus, I think it would be misleading to say pairwise coordination patterns provide better generalization. Also, I am not convinced that the reward difference between 22 (QPAIR/QPLEX) and 23 (QSCAN) is significant. Is there a crucial point worth mentioning for this gap?
6. In the matrix game, what is the k used for QSCAN? Why is it able to perform (slightly) better than QPLEX/QPAIR? If the grouping is 2 sub-teams with 2 agents and 1 agent for QSCAN, are there some groups (e.g. player 1-2, 3) that can do well while other groupings don’t? 

limitations: The authors outline the limitations of the work in the Conclusion (Section 6). I agree that going beyond advantage factorization is one of the main limitations in this work because QSCAN as it stands is somewhat limited by IGM and QPLEX.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Authors present QSCAN, an attention-based, sub-team-level organization, task allocation, and credit assignment algorithm to outperform certain special cases of MARL algorithms that fall under the sub-team organization, namely, QMIX (k=1) and QPLEX (k=n). QSCAN is evaluated in three environments against other factorization methods and shows better performance consistently.
strengths_and_weaknesses: QSCAN ties in well with the literature, and the evaluation is carried out under varied settings.

The paper's weakness lies in its lack of proof for the initial claim of more effective factorization and faster learning. Sub-team organization and attention are additions to the computational complexity, so the number of steps in Fig. 5 cannot vouch for "faster" learning. For the SMAC evaluation, even the number of steps data is omitted for reasons unknown. On a separate note, some explanation on role assignment line of MARL research (e.g., RODE) could make the analyses more comprehensive, as role assignments (with no explicit team formation) and sub-team formation (with no explicit role definition) could be positioned interestingly as duals.
questions: Q1: How important is the addition of self-attention? It would be nice if the authors could share some ablated evaluation results, which are probably available if they begun initially without attention.

Q2: What is the effect of altering k but for the same evaluation setting? What is the (expected) effect of forming different-size sub-teams?
limitations: Please have a look into the effects of different k as well as the effects of attention. Other than that, the limitations of QSCAN have already been clearly stated in Section 6.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors proposed a method called QSCAN to perform value-based decomposition on a subset of agents in multi-agent reinforcement learning. For the given task, the idea is to arrive at weights by which Q-functions are scaled to get an idea of sub-team coordination. The proposed technique is tested on different environments and obtained good performance when compared with different existing methods. 
strengths_and_weaknesses: The paper is structured well and written well. The paper deals with an interesting problem that I believe no one is dealt with before. The problem is of good importance in industries where there will be multiple tasks and only a subset of agents participate in a single task. However, below are some of the points which I think may improve the paper. 

1. How the weights $\lambda_i$ are learned over time. Do authors use the regular experience replay buffer for this case? I suggest incorporating more details here. 

2. What about the scalability of the proposed approach? I see the computational complexity is so enormous (especially when the number of combinations to try out is huge). 

3. I am confused on the execution part other proposed approach. Are weights calculated for every time step? 

questions: Please refer to my points in the weakness of the paper. 
limitations: There are no limitations addressed in the proposed approach. There are some typos in the paper which the authors need to correct.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a hierarchical approach to value function factorization called QSCAN, where the multi-agent team is divided into sub-teams of fixed size, which solve different sub-tasks, in order to exploit local coordination. The architecture of QSCAN is based on QPLEX, where a sub-team coordination module determines the contribution weights of each agent for all potential sub-teams. The paper presents two instantiations of QSCAN called QPAIR, enumerating all sub-teams of size 2, and QSCAN, using self-attention on the agents themselves, which are evaluated against QMIX and QPLEX in various benchmark domains.
strengths_and_weaknesses: - **Originality**: A similar idea about sub-team based hierarchical value factorization was published in NeurIPS 2021 [1]. However, the introduced coordination hierarchy with QMIX and QPLEX as special cases is interesting and novel in my view.
- **Quality**: The paper seems technically sound and the performance of QSCAN is supported in the experiments. The number of agents in all benchmarks is rather small, which would make an evaluation with $k > 2$ teams feasible. Regarding the claim about focusing on local coordination, I recommend to evaluate in SMAC scenarios with > 10 agents like ``bane_vs_bane`` rather than ``2c_vs_64zg``, where the single sub-team is obvious.
- **Clarity**: The approach is well explained. I did not check the attention heat-maps in Figure 6 because the plots are too small, making it very hard to follow the descriptions in the text.
- **Significance**: QPAIR and QSCAN seem to clearly outperform QMIX and QPLEX in most settings. However, I could not find any information about the number of training runs. The error/deviation bars in Table 2 regarding the SMAC results are missing. Thus, I cannot assess the significance of the results.

Minor comments:
- Line 72: QPAIR is mentioned for the first time without further description, which is confusing. The QSCAN variant should be named differently from its actual framework as it is not obvious that QSCAN is used in different contexts in the same parapgraph.
- Line 91: $r: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ denotes __the__ global reward function
- The text and labels on all figures are very small and hard to read when printed (the white texts in Figure 4 are completely unreadable)
- Line 283: „several state-of-the-art MARL approaches“ are just two approaches: QMIX and QPLEX

[1] Phan et al., "VAST: Value Function Factorization with Variable Agent Sub-Teams", NeurIPS 2021
questions: - Line 225: Is the importance weight $g^{\textit{ST}}_i$ assumed to be non-negative?
- How many training runs were conducted per task? I could not find any information in the paper or appendix.
limitations: The provided checklist refers to Section 6, but I did not find any explicit limitations. There is a statement about a challenge of designing more effective hierarchical structures but the actual limitation is unclear to me, given that the results are mostly in favor of QPAIR/QSCAN anyway.

Potential negative societal impact is not discussed.
review_title: null

Meta-review: metareview: This paper presents QSCAN, a hierarchical approach to value function decomposition where the agents are grouped into sub-teams to solve different sub-tasks via local coordination. QSCAN extends the monotonic mixing network in QPLEX in order to represent sub-team coordination, and shown to outperform QPLEX and QMIX in a number of benchmark tasks. Reviewers were all in favor of accepting the paper based on the sound contribution of the proposed approach. However, figures are illegible due to font sizes. Please fix the figures in the final version of the paper.
confidence: Certain
Decision: Accept