Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces MATE, a multi-agent learning environment that models an asymmetric game between two teams of players: the "cameras" and the "targets". The cameras are fixed in their location but can change direction and zoom, with the goal of observing the targets. Their sensors can not see through obstacles in the environment. The targets are mobile agents with sensors that can see through obstacles, with the goal of moving items to warehouses. In terms of code, this environment provides an OpenAI Gym interface for interacting with the environment and training agents.
strengths: This paper introduces an interesting and novel learning environment. It incorporates a few features that I have not seen combined in other environments, including teaming, partial observability, and non-zero-sum reward. The code is compatible with the standard OpenAI Gym interface and with RLLib. It has the potential to open up new research directions in algorithm development for this setting.
weaknesses: - The entire setup is an important step towards a general learning environment for studying this problem, but presently quite narrow in scope. For example, lines 139-141 state that "there are two kinds of vehicles for targets, one with high speed and small capacity, and the other with low speed and large capacity. The former is as twice fast as the latter but with a halved carrying capacity" - why not have parameters for speed, capacity, size, etc.? Why not let the agents set their own speed via an action? These suggestions are about this as a learning environment that others can use, as opposed to a single stylized simulation for a specific learning task. 
- Table 1, while containing helpful comparisons, can be misleading in the conclusions that it implies by its use of colors (green implying good). Specifically, it appears to imply that the MATE environment is the only environment that has all of the desirable features compared to the other environments listed. The more accurate conclusion that should be drawn from this table is that, along the indicated axes, MATE has a unique set of attributes relative to other learning environments. However, given its relatively narrow focus, it should be clearer that MATE does not supersede these environments but rather provides a new learning environment with a novel set of features.
- The "obstacle" defined in Section 3.1 is really an environment feature. A more general version of this environment would allow arbitrary obstacles and layouts with different topologies.
- Depending on whether the authors would expand the Experiments section, this may be a more appropriate submission for the NeurIPS Main Track than the Datasets & Benchmarks track.
correctness: There are no concerns about correctness, although I am not specifically evaluating the experiments from Section 5 (treating them more as a proof-of-concept that this learning environment can be used for training agents in different scenarios). I would suggest deemphasizing this section and putting more emphasis on how users of this learning environment can expand it and provide modifications to the base scenario.
clarity: The paper is written clearly. There are some sections that could be rephrased but everything is generally easy-to-read. 
relation to prior work: The related work section (Section 2) is quite narrow in scope. Especially in the context of learning environments, there could be more detail on what this kind of learning environment provides.
additional feedback: Minor comments:
- Line 197: it would be helpful to be concrete about what "modern CPU" means (especially if this paper is read in the future)
review_title: Review of "MATE: Benchmarking Multi-Agent Reinforcement Learning in Distributed Target Coverage Control"

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper introduces a new dataset for multi-agent reinforcement learning in tracking tasks. The authors present not only tracking tasks but also tasks conveyed by mobile agents, enabling cooperative, competitive, and mixed games to be played with ease. Contributions are as follows:
 * Multi-Agent tracking tasks combined with the shipments tasks carried by mobile agents with properties like communications, heterogeneous, and asymmetric composition of agents.
 * Cooperative, Competitive and mixed tasks
 * Easy to use, requiring low memory for CPU, which enables sample efficiency by distributed experiments.
 * Can present researchers with a basic environment for multi-camera tracking tasks.
strengths: 1. It is not original to track some targets because there are environments for tracking active objects [1]. In some ways, however, it appears novel. Tracking multiple agents (targets) with multiple agents (cameras) initially appears novel. Second, it appears novel to combine tracking tasks with cargo shipments, which enables a zero-sum game. 
2. I believe the MATE environment has reached completion. 
3. As soon as I saw the environment, I found it intriguing that it could serve as a basic simulator for real-world applications. It could provide a guideline for algorithm and application researchers in the field of tracking and cargoes that are not exposed to anything. 
weaknesses: 1. The GUI closely resembles Multi-Agent Particle Environment (MPE), which contradicts the authors' claims of realism. Therefore, it is insufficient for authors to advocate realistic characteristics. 
2. Baselines are not enough to show the validity of the environment, as shown in Fig 5. Lines are so flat that it doesn't looks like being under training for 10 million steps.
3. So short Related Work section that I felt hard to understand in some sections like 5.3.
correctness: The structure of the state, observation, action, entities, and reward looks well constructed. But seeing Fig 5(b), the communication property seems to not affect training. It makes me wonder whether the individual and public observation or states are well constructed or not.
clarity: The paper is straightforward and well-written. However, I regret that certain points, such as the communication mechanism (TarMAC, I2C) in section 5, were not understood. Also difficult to comprehend completely is section 5.3. It would be courteous to display additional relevant work for the paper in section 2. 
relation to prior work: The authors mentioned the prior work focusing on the Multi-Camera Multi-Target Tracking.
additional feedback: 1. What effect does 'minimizing the exposure time to cameras' have on target agents in terms of real-world applications? 
2. Env SMAC is written to be non-scalable in table 1, but I believe it is scalable because it can produce more units depending on the settings. 
3. In Figure 3, I wonder how to choose between peer-to-peer and broadcast methods for passing messages. How do agents choose a peer when using the peer-to-peer method? 
4. Do you have any experience with curriculum learning for training the algorithms in Figure 5 that were not trained? 
5. In section 5, the HRL model receives a high-level signal from the MARL algorithm and selects a rule-based action. However, since the signal may be a latent vector, isn't it necessary to train the HRL model with the signal as an input? 
6. I want to know the selection criteria for baseline algorithms (IPPO, MAPPO, QMIX, MADDPG). There are SOTA multi-agent algorithms that are policy gradient and value-based in cooperative and competitive environments. 
7. I have a strong suspicion regarding the training camera experiments in the fully cooperative game. Despite 10 million training steps, the learning curves in Fig. 5 do not have particularly steep lines, but are rather flat. Is the training environment valid for tracking tasks?
review_title: Well-written paper, interesting environment but not enough valid experiments for training.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors propose a new OpenAI Gym environment for the partially observable two-team target coverage problem. The environment comes with wrappers for built-in rule-based strategies to test against, three types of reward signals, and various configuration options for the number of agents in each team, number of obstacles, radii parameters, etc. The authors also benchmark various existing multi-agent reinforcement learning solutions in this environment for the fully cooperative settings where the opposing team strategy is fixed as well as the mixed cooperative-competitive setting where both agents are learning. 
strengths: * This is an interesting new benchmark for mixed cooperative-competitive multi-agent reinforcement learning with sufficient customizability options and ease of use due to its OpenAI Gym interface. The problem it models also seems relevant and I am not aware of another such benchmark for multi-agent tracking and coverage.
* There are various built-in opponent strategies for quick testing as well as multiple reward structures
* Various communication options are available to agents, thereby enabling full communication settings as well as restricted communication settings
* The agents in a team can be heterogeneous and the teams can play asymmetric games
* The performance of various popular multi-agent reinforcement learning solutions are benchmarked against the proposed environment, thereby providing a nice benchmarking in the context of multi-agent coverage and tracking
weaknesses: * I do not see any substantial weaknesses outside of the writing, which can be improved.
correctness: The results seem correct and the reward metrics being adopted are reasonable for the fully cooperative and mixed cooperative-competitive settings in multi-agent reinforcement learning.
clarity: The writing can use some work. While the paper is comprehensible, the writing is clumsy enough to hinder the flow of the paper. Some examples, though there are many more:
* "fully-competition" -> fully-competitive
* "gained many interests" -> gained much interest
* "The competitive game hosted in MATE involve two-team of heterogeneous agents, More importantly" -> ...involves two teams of heterogeneous agents. More importantly
relation to prior work: Prior work is properly referenced and put in the context of this work in Table 1.
additional feedback: N/A
review_title: A useful benchmark for mixed cooperative-competitive multi-agent reinforcement learning

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a multi-agent tracking environment (MATE) where a  group of camera agents trying to cover target agents as much as possible, while another group of target agents try to escape from the coverage of camera agents. Then the authors perform several MARL algorithms (MAPPO, IPPO, QMIX and MADDPG) for cooperative tasks in MATE. Different communication protocols (TarMAC, I2C) are also tested in cooperative tasks. Some other experiments such as effectiveness of fictitious play in competitive tasks is investigated. This new MARL environment is provided with high-quality docs.
strengths: 1. In my opinion, this is the first open-sourced and standardized multi-agent environment specifically for target coverage problem. It provides detailed docs and clear coding tutorials for target coverage problem-related researchers. It is integrated with the OpenAI Gym API and can be used with RLib, Tianshou, Stable-Baselines-3, etc as the authors claimed. 

2. Dislike other multi-agent environments using broadcast communication regime (which is naïve), MATE provides peer-to-peer communication options. I believe such features provide convenience and some inspires for studying communication problems in multi-agent system.


weaknesses: 1. Though this work make up the gap that there is no standard environment for the target coverage control problems, this paper does not clarify the importance in the field of target coverage control problems. The references about the target coverage control problems are supposed to be added. 

2. Though Some features of META are highlighted such as the Game Type, Communication Method, Agent Type and the Scalability, I did not see the superiority of META compared to other multi-agent environments. META is specifically designed for the target coverage control problem. All these highlighted features are from the problem setting not from the META itself. 

3. META is more like an extension work of this paper: "Learning Multi-Agent Coordination for Enhancing Target Coverage in Directional Sensor Networks" which provided some environments resource. Do you think is it an incremental work?

4. The paper "Learning Multi-Agent Coordination for Enhancing Target Coverage in Directional Sensor Networks" has a good method to solve the problem. The HiT-MAC method should be performed as well in the experiments. If the HRL model is some method similar to Hit-MAC, the author should cite the paper and explain the difference and similarity between them.
correctness: 1. Line 196: "MATE is a multi-agent environment based on numerical simulation and implemented in pure Python with minimal dependencies."

If you claim "minimal", you should provide a table or some references to verify it. And the interpretations of "dependencies" can be varied. This claim seems incorrect to me unless you add some more constraints.

2. Line 238: "newly-developed MAL algorithms"

What is MAL algorithm?

3. Line 246: "We use and customize the RLlib implementations of QMIX, MADDPG, PPO, TarMAC and I2C algorithms in all of our experiments."

I do not understand how the PPO of RLlib can be naturally extended to MAPPO. 

4. I am wondering why HRL was not applied to training targets in a fully-cooperative game?

5. I do not understand the necessity of subsection 5.3 in Experiments. Did these experiments show any novelty of META?
clarity: Yes, this paper is well written and well organized.
relation to prior work: No.

It did not include the method in "Learning Multi-Agent Coordination for Enhancing Target Coverage in Directional Sensor Networks" as a baseline. It neither discusses the difference/improvement compared to that paper. 

MATE has some similarities to the multi-agent environment PuckWorld and WaterWorld. But it did not mention them. There are a lot of multi-agent environments, the authors should better show MATE's novelty or show some new marl algorithms in the paper.
additional feedback: The novelty is not obvious, but I am happy to see this high-quality engineering work contributing to our research society. To make this benchmark paper more attractive,  I think the authors should more focus on the target coverage problem itself instead of enthusiastically promoting its general comparison with other multi-agent environments.
review_title: I can see authors' ambitions in this new MARL environment but this paper needs more novelty

Meta-review: metareview: This paper introduces an interesting cooperative-competitive MARL environment involving groups of camera and targets with the objective of transportation. The benchmark environment appears to be scalable with novel components including partial observability and general-sum rewards in an interesting environment. The authors provide extensive baselines for MARL algorithms as well. The majority of reviewers appraised the work positively, and the authors adequately addressed the concerns brought up in the review along with a revision incorporating some components of feedback. I think this is a good addition to the MARL literature, and the authors should be sure to incorporate all the remaining feedback in the final version of the paper.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept