Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors explore coordination and heterogeneity levels of cooperative MARL environments. In particular, they formalize the notions of coordination level and heterogeneity level of an environment and present HECOGrid, a suite of multi-agent RL environments that facilitates empirical evaluation of different MARL approaches across different levels of coordination and environmental heterogeneity by providing a quantitative control over coordination and heterogeneity levels of the environment. They also proposed a new method named SAF (Stateful Active Faciliator) to enable agents to perform well in difficult environments with high levels of coordination and heterogeneity. They evaluated SAF and compare its performance against baselines IPPO and MAPPO on HECOGrid, and demonstrate the advantages of  the proposed SAF method.
strength_and_weaknesses: Strength:

[1] The problem of studying coordination and heterogeneity levels of cooperative MARL environments is interesting, and the developed new environments HECOGrid are helpful for better quantitative understanding of different MARL approaches across different levels of coordination and environmental heterogeneity, which could be valuable for future RL research.

[2] The proposed SAF method seems to be valid, and the provided empirical evaluations help demonstrate the advantages of the proposed SAF method.

[3] The paper is written reasonably clearly and I find it easy to follow.

Weakness/suggestions:

[1] As mentioned in the paper, environmental heterogeneity has been less explored in the literature. To better justify the significance of the contribution & better understand the motivation, I'd love to see some more explanations/discussions how important/essential for such separate concepts of "coordination" and "environmental heterogeneity" for MARL (rather than a single concept of "difficulty level" in some previous MARL environments, which implicitly supports various levels of  "coordination" and "environmental heterogeneity"), and why this is super important from MARL research perspective. Such explanation/clarification would be helpful to justify that the contribution here is significant enough.

[2] It would be valuable/helpful if the authors could elaborate a bit more about the supported scale & complexity for developed new environments HECOGrid, against previous MARL environments. Is  HECOGrid appropriate for all different scales (from small to super large) and complexities (from simple to super complicated) of MARL experiments, and how about the comparison with previous MARL environments? Table 1 shows that HECOGrid has some advantage over all the other previous MARL environments by enabling explicit control of "environmental heterogeneity", but I'd like to also see some more comprehensive/objective discussions on pros and cons of HECOGrid against other MARL environments. E.g., are there any scenarios where other MARL environments would be more appropriate/preferred for the experiments rather than HECOGrid? Is the main advantage for HECOGrid being that it allows explicit control of "environmental heterogeneity", and it'd be not that relevant for future research that is not focusing on "environmental heterogeneity"?
clarity,_quality,_novelty_and_reproducibility: The paper is written reasonably clear, and it contains some novel valuable results.
summary_of_the_review: The authors formalized two MARL concepts ("coordination" and "environmental heterogeneity"), developed new MARL environments HECOGrid incorporating these two concepts, and proposed a novel algorithm SAF whose advantages were demonstrated in empirical evaluations in HECOGrid. The proposed method is heuristic in nature, and the contribution is mainly on empirical side and minor from theoretical side. 

I found the paper is generally written clear, and it does contain some valuable novel results which might be appropriate for publication on ICLR, though I'd like to see some more clarifications/explanations from the authors to better justify the contributions & motivation, as described in above "strength and weakness" section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose HECOGrid, a suite of MARL environments. The authors claim that two properties that affect the performance of MARL algorithms in the environment are coordination and heterogeneity. The authors formally define these two metrics. HECOGrid allows the user to choose the level of coordination and heterogeneity in the environment. The authors also propose Stateful Active Facilitator (SAF), a cooperative MARL algorithm. They evaluate SAF in HECOGrid, for varying levels of coordination and heterogeneity against IPPO and MAPPO.
strength_and_weaknesses: Strengths 

- The paper proposes an interesting suite of environments that would be useful to the MARL community.
- The evaluation of SAF is detailed and the evaluation baselines are appropriate.
 
-------------------------------------------------------------------------------
Weaknesses 
- In Table 1, MPE has scenarios with partial observability, e.g. simple_speaker_listener
- Page 3, paragraph Notation: Why are the policies of the agents conditioned on the observation and not the history of observations?
- SAF is not properly described in the text. What is the actor's loss? Is it trained using the MAPPO loss? Is it an on-policy or an off-policy algorithm?
- The used hyperparameters are not mentioned anywhere in the paper.
- How does SAF compare to SOTA Value Decomposition methods, for example, QPLEX?
- Do you use parameter sharing for the algorithms? If not, why? It has been shown in several studies that parameter sharing significantly improves the achieved returns of cooperative MARL with homogeneous agents.
- The paper claims that one of its main contributions is HECOGrid, a suite of MARL environments. However, the authors have not included the source code of HECOGrid.
- While state/observation heterogeneity is a feature currently missing from many MARL environments, I believe that agent heterogeneity is a more interesting and fundamental challenge (for the MARL community).
- It seems like some IPPO results are missing from Figure 4.
- Most of the experiments are performed for coordination level 1, which means that only a single agent is required to solve the task. That primarily shows that SAF does not "enables agents to work efficiently in high-coordination and high-heterogeneity environments". It seems that the main benefits of SAF come from the centralized computation of the value function. Can the authors elaborate more on why they believe that SAF outperforms the related baselines?
- In Figure 6, what do KS, and KS-PP mean? Does KS mean SAF without PP? What is KS-PP? Is it just SAF? Or is it SAF without KS? If this is the case, how much is the contribution of KS to SAF?
- I would expect to see more clarifications on why SAF is better compared to baselines. This is not presented in the paper. I can only see the algorithm description without much justification behind the design choices.


clarity,_quality,_novelty_and_reproducibility: **Reproducibility**: The reproducibility of the paper is very poor. Very little information about the algorithm is included. The source code is not included. The expectation is also that when the authors propose a benchmark, they would include the source code.

**Clarity**: Most of the paper is clearly written (in terms of the use of language). However, it lacks organization in several places.
Several details are missing about SAF. I would also recommend adding information on how to use the new environment suite in the appendix.

**Quality**: While the contributions of the paper are worthwhile the quality of the work cannot be fully assessed with the lack of reproducibility.

**Novelty**: HECOGrid is a very welcoming contribution since it is currently missing from the literature an environment with these properties. SAF seems like a standard CTDE algorithm with an extra attention module and some additional implementation tricks. It is not clearly explained in the text, why we expect SAF to perform better compared to other baselines.
summary_of_the_review: Overall, I believe that the paper presents some worthy contributions. However, I have several concerns about the reproducibility of this work. Additionally, the writing seems a bit rushed, since the paper is not well organized. Additionally, the lack of some results in the figures, as well as, the lack of the source code shows that this work is in its early stage. Therefore, I recommend rejection. I would happily increase my score if the authors address my concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This explores training MARL under heterogeneous environments with different coordination difficulties. The authors first define heterogeneous environments and different levels of coordination. Then three Grid environments are designed to support different levels of coordination. Then the author proposed an attention-based method to solve the designed tasks. 
strength_and_weaknesses: **Strength**
- This work tries to study and define the level of coordination in MARL, which can be a good perspective to evaluate various algorithms. 
- The paper is overall easy to follow. 

**Weakness**
- For the definition of the coordination level, it seems each time step is considered in isolation, what about coordinations that are temporal, such as the actions that need to be taken by one agent after another agent has taken a certain action?
- A high-level diagram should be provided for the proposed algorithm which would make it simpler for the readers to understand the overall network architecture. 
- It would a good idea to further motivate how heterogeneous environments can happen in a multi-agent environment. Why would different agents have different transition functions (designed as the change of action order)?  How does that map to a real-world environment?
- The topic of the work is related to coordination and hetergeneous environment, but it is not so well motivated why these two are explored simultaneously in a single paper. The designed environments seem to focus more on the coordination level while the designed algorithm (dynamically selects model) seems to focus on the hetergeneous property.
- From the evaluation result, it seems the designed solution performs better for different hetergeneous environments, but has less improvement when the coordination level is higher. How does the proposed algorithm improve coordination?

clarity,_quality,_novelty_and_reproducibility: *Clarity: The paper is overall easy to follow
*Novolty: This work is somewhat novel, but should be better motivated
*Reproducibiliy: Code not provided so this is unknown. 
summary_of_the_review: The work investigates the training of MARL under different heterogeneous environments and coordination levels. The authors have a good attempt to define the coordination level of MARL, though it seems incomplete. The relationship between the environment heterogeneity and coordination and motivation should be discussed more in the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of Multi-agent Reinforcement Learning (MARL). This paper specifically addresses two components of MARL. First, the paper designs and presents three benchmark domains to explore the limits of modern MARL algorithms in terms of their ability to handle environment heterogeneity and the need to actually cooperate (rather than independently acting to complete the team's task). Second, the paper develops a new algorithm for MARL specifically designed for heterogeneity and cooperation. The paper evaluates their approach against baseline methods in the new benchmark domains, presenting numerous, positive results.
strength_and_weaknesses: Strengths:
-This paper clearly states its contributions on Page 3 (and in the abstract).
-Unlike many papers in MARL, this paper contributes not only an algorithm but also a set of MARL environments that push MARL algorithms' ability to handle environment heterogeneity and test the need of these algorithms to truly coordinate.
-The domains presented (TeamTgoether, TeamSupport and KeyForTreasure) are intuitive and do, indeed, appear to require coordination (unlike some MARL domains). Though, it is important to note that these are not the only domains that *benefit* from coordination (e.g., Predator-Prey [1]). 
-The paper shows positive results for SAF vs. MAPPO and IPPO on the proposed benchmark environments while varying the degree of coordination (Figure 3) and heterogeneity (Figure 4).
-The paper also provides generally positive (but somewhat mixed results) for out-of-distribution generalization (Figure 5).
-The breadth and depth of the empirical results in this paper (not including the appendices) is impressive.

Weaknesses:
-The paper could be improved with a DEC-POMDP setup formalism to help explain the terminology/line-of-inquiry.
-The description of the Stateful Active Facilitator (SAF) is relatively short. While more details are in the appendix, it would be helpful if pseudocode was provided (Algorithm 1) in the main paper. Figure 2 does not help much in describing the approach and could be removed to make room for a better illustration or more text-based details. Many approaches work by generating and integrating messages -- how does this approach really differ? The technical uniqueness is unclear here in the algorithmic formulation. The selection of predefined(?) policies appears unique, but details are missing about how these policies are enumerated and trained.
-The paper is missing important baselines in the MARL literature, including [2], [3], [4], and [5]

[1] Singh, A., Jain, T. and Sukhbaatar, S., 2018. Learning when to communicate at scale in multiagent cooperative and competitive tasks. arXiv preprint arXiv:1812.09755.

[2] Rashid, T., Samvelyan, M., Schroeder, C., Farquhar, G., Foerster, J. and Whiteson, S., 2018, July. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International conference on machine learning (pp. 4295-4304). PMLR.

[3] Seraj, E., Wang, Z., Paleja, R., Martin, D., Sklar, M., Patel, A. and Gombolay, M., 2022, May. Learning efficient diverse communication for cooperative heterogeneous teaming. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems (pp. 1173-1182).

[4] Iqbal, S. and Sha, F., 2019, May. Actor-attention-critic for multi-agent reinforcement learning. In International conference on machine learning (pp. 2961-2970). PMLR.

[5] Mao, W., Yang, L., Zhang, K. and Basar, T., 2022, June. On improving model-free algorithms for decentralized multi-agent reinforcement learning. In International Conference on Machine Learning (pp. 15007-15049). PMLR.
clarity,_quality,_novelty_and_reproducibility: The paper seems to be attempting to do too much in a single paper. The authors are commended for developing a set of three new environments to test the efficacy of MARL algorithms along dimensions of heterogeneity and cooperation in terms of the environment. The authors are also commended for additionally developing a new MARL approach. However, doing both in one paper means that a significant amount of detail is left to appendices/supplementary detail. As such, it is difficult to fully understand the details of the technical approach. I would recommend either figuring out a way to move some of the ablation studies (and/or Figure 2) to the appendix and instead moving more details about the algorithm and its motivation/intuition and psuedocode to the main paper.

The novelty is clear in terms of a new set of domains, but the novelty w.r.t. MARL algorithms is unclear without further empirical analysis against more baselines (or better discussion of why those baslines are not necessary).

Code for reproducibility appears missing.
summary_of_the_review: This paper contributes three new domains to explore the limits of MARL in terms of heterogeneity and cooperation. These domains will likely be high-impact for the community. The paper also contributes an algorithm. The algorithm is under-described, unfortunately. Either, the paper must make more room for details about the technical algorithm (and adding baselines to demonstrate the technical contribution), or the paper needs to be refocused on evaluating MARL algorithms on these domains to show which methods are capable of heterogeneity/cooperation and why/why not.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper considers coordination in heterogeneous multi-agent RL (MARL).  It formalizes these notions of coordination and heterogeneity and argues that heterogeneity has been understudied in MARL.  The paper presents HECOGrid, which is a new suite of MARL environments that allows varying the amount of heterogeneity faced across agents and the level of coordination required. It also presents a new method, Stateful Active Facilitator (SAF), designed for settings with high heterogeneity and a high level of coordination required.  SAF outperforms two baselines on HECOGrid.

Strengths

- Coordination and heterogeneity are both important in MARL

- The benchmark suite HECOGrid seems like a useful new way of quantifying performance while varying coordination requirements and heterogeneity

- It seems useful to pull out heterogeneity and coordination level as specific problem characteristics and to be able to study them differently from other aspects of an environment's difficulty

- Strong empirical evaluation 

Weaknesses

- The definition of the coordination level considers each time step in isolation. This does not allow capturing coordination that is temporal, where one agent needs to take an action after another agent takes its action.

- It is possible that this paper is trying to do too much, (tackling coordination, heterogeneity, and both providing an algorithm and a new suite of environments) as articulated by reviewers y8q8 and PAoT and as is evident by some of the details left out articulated by reviewer UJ6m. It seems possible to do this, but it will require the appendix to be longer than usual and will put a burden on the authors to be especially clear and organized.

- Opportunities to improve clarity and motivation, as called out in the individual reviews from all reviewers

Other comments

- Initial experiments used observations rather than histories as the input to agents. Follow-up experiments addressed this, using RNNs that take the history as input. Additional experiments also added new baselines.
justification_for_why_not_higher_score: - Paper lacks focus at times and does not always clearly articulate arguments --- it tries to do too much
- Concern that the definition of coordination is not sufficiently general
justification_for_why_not_lower_score: - Strong empirical evaluation
- Suite of environments seems quite useful
- Value in pulling out coordination and heterogeneity as distinct concepts
Decision: Accept: poster