Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper aims to tackle the problem of large discrete action spaces in multi-agent MDPs (MMDPs) where the number of actions rises exponentially with the number of agents. The paper proposes to embed agreed priority conventions (agent priorities, agent ID, action priorities) into reinforcement learning (PCRL) to tackle large action spaces. The paper leverages RNNs to generate Q-values for the system one dimension of the action vector at a time sequentially (while maintaining priority conventions), and proposes an equality auxiliary loss to make sure that the V*(s) are equal for each RNN output corresponding to each action dimension. The paper claims that the priority conventions do not miss the optimal actions. The authors evaluate their proposed approach on coverage planning and pong domains.
strengths_and_weaknesses: ## General Comments 
- The proposed approach seems sound and it is interesting to see authors approach match up to the performance of DQN with significantly lesser number of parameters required.

- The paper is a bit hard to understand at places with grammatical mistakes. Figure 2 makes it easier to understand the paper. It would be great if the authors could take some time to focus on the writing of the paper. Some points to consider:

  - For example:  “some researchers resorted to independent reinforcement learning (RL) regardless of other agents, but that will form loose cooperation and yield suboptimal solutions” → Independent MARL has shown to be a surprisingly good baseline in recently published papers in cooperative MARL. [6]
  - Some minor points:
    - Line 161: "where h_0 = C_0 is the featured and supplied state s" → I am not sure what the authors mean by featured and supplied state?
    - Line 166: ungaurenteed → unguaranteed; inconsitent → incosistent
    - Line 198: (??) reference missing

- The paper is missing several key citations from recent MARL literature which especially focuses on improving exploration in MARL large action spaces with role-based learning (RODE: [1]) as well as papers leveraging the centralised training decentralised execution (CTDE) framework (an intermediary between fully decentralised learning and centralised learning) [2,3,4,5]

- The idea of leveraging RNNs to predict one dimension of the action vector at a time seems to be very similar to the other works [7,8] with similar equality constraints as proposed by authors in Eq. 11. Can the authors comment on the novelty of their approach?

- The considered problem in the paper seems to be difficult from the perspective of a large number of agents (thereby large action spaces), and therefore the proposed approach seems to be only useful for scenarios with full observability and homogeneous agents which is not commonly observed in real-world multi-agent problems.

## Comments on Experimental Section

- The authors only compare their approach against the joint-action DQN based approach. There have been several recent works on tackling the large action spaces in MARL with role-based learning [1], and other CTDE based approaches [2,3,4,5] which can tackle large action spaces pretty effectively especially on the coverage and pong domains in the paper. The authors should compare against these approaches to fully showcase the effectiveness of their approach.

- The authors does not evaluate on any large-scale multi-agent RL domains like Starcraft II (SMAC) [7] (with some maps having 27 agents and each agent allowed to take 8 actions) which would help prove the efficacy of their method on large action space domains.

[1] Wang, Tonghan, et al. "Rode: Learning roles to decompose multi-agent tasks." ICLR 2020.

[2] Son, Kyunghwan, et al. "Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning." International conference on machine learning. PMLR, 2019.

[3] Mahajan, Anuj, et al. "Maven: Multi-agent variational exploration." Advances in Neural Information Processing Systems 32 (2019). APA  

[4] Rashid, Tabish, et al. "Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning." Advances in neural information processing systems 33 (2020): 10199-10210. APA  

[5] Son, Kyunghwan, et al. "QTRAN++: improved value transformation for cooperative multi-agent reinforcement learning." arXiv preprint arXiv:2006.12010 (2020).

[6] de Witt, Christian Schroeder, et al. "Is independent learning all you need in the starcraft multi-agent challenge?." arXiv preprint arXiv:2011.09533 (2020).

[7] Samvelyan, Mikayel, et al. "The starcraft multi-agent challenge." arXiv preprint arXiv:1902.04043 (2019).

questions: - Can the authors answer questions from the weakness/strengths section?

- Additional Questions

  - It seems like the different priority conventions (agent priorities, agent ID, action priorities) conventions are predefined before learning. Would it affect the convergence if conventions are changed? For example, if action priorities are changed, would it change the final solution? How would one define such priority conventions for large scale domains like Starcraft II [7]?

  - Line 235: the authors mentions that “PCRL network can successfully learn the conventions”. Aren’t the conventions already predefined for the domains? 
limitations: Yes, the authors does mention the limitations of their work via a future work paragraph. Additionally, the authors should consider partial observable domains and possibly learning priority conventions.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper looks to tackle multi-agent reinforcement learning problems operating in large-discrete-action spaces. The authors do so by introducing a priority convention into the RL loop. The priority convention results in generating a sequence of actions that are somehow better. The authors demonstrate the effectiveness of their algorithm by demonstrating experiments on various multi-robot problems. 
strengths_and_weaknesses: Strengths
1. The paper is thorough with its literature review. 

Weaknesses
1. The paper is extremely poorly written and throws around a lot of words that carry no meaning scientifically. For example in the abstract alone "In this work, we propose to embed agreed priority conventions into reinforcement learning (PCRL) to directly tackle the microscopically sequencable multi-agent LDAS problems" To the best of my knowledge I have never heard the term microscopically sequencable (and after reading the paper I still do not know what the authors are trying to say with the term microscopically sequencable means in a rigid mathematical sense.) Further phrases such as "can help exploitation around the lexicographic-smallest optimal action vector" make no sense to me. Such phrases and terms occur in the paper at multiple places and either serve very little purpose or only stand to confuse the reviewer when not rigidly defined. The phrase cargmax or consistent argmax at a first glance only confuses the reader. I would very sincerely request the authors to overhaul Section 3 and define everything instead of vaguely throwing out terms that either mean nothing or only confuse the reader which is to say nothing of the poor choice of notation (writing out "one hot", ? in Line 198, writing out "length") that only makes things worse.
2. Now to come to the actual methodology devised in the paper here; the authors say nothing about what the actual priority convention is anywhere in the paper or in their experiments. How is the priority convention decided? How is it set. What are the parameters. How does it affect the training. In the experiments the robots all look identical and how does interchangeable convention set precedence to anything; i.e if Robot 1 is the same as Robot N then how does it matter if I switch priority from 1 to N and vice-versa. 
3. Further, the paper itself doesn't say anything about what happens if you force in a priority convention/ie why do you still need RL. In most cases in multi-robot problems, setting a priority convention is akin to setting a heuristic and removes the need for any RL. One chooses multi-agent RL if there exists a hard to define objective function for each robot but there exists a global objective function we wish to optimize for. If one is able to discretize this and say R1 is more important than Rn, then it is fairly trivial (especially in the grid world situations) to design a controller without any RL or machine learning that simply optimizes for the R1 > Rn objective. If one thinks back about the problem in Fig 1c, if, there exists a priority convention for the 18 robots, you do not need a learning based solution and can simply write a dfs solution with some edges carrying more weights (corresponding to the priority convention) and have the optimal solution. In the case where you have stochasticity it still seems to me that one can get away without doing any RL and setup an objective function that accounts for the stochasticity.
4. Lastly, the experiments are extremely poorly designed. The proposed method needs to be compared with a relevant algorithm, By biasing the proposed algorithm and not providing the baseline DQN any reward based on the priority convention or information about the priority convention, the baseline is handicapped. Further, the experiments themselves are not compelling enough to determine that the proposed method here outperforms any standard non heuristic enabled multi-agent RL algorithm. 

questions: I do not think this paper in its current form meets the high bar for publication at NeuRIPS. The paper is weak in many areas from writing. novelty, actual methodology, experimentation. 

I recommend the authors take a step back and re evaluate the proposal here and what introducing priorities does to the problem and if it even needs RL if one were to introduce a heuristic like priority convention. 
limitations: Please refer above.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors attempt to tackle the problem of RL in settings with large action spaces caused by there being many agents in the system. The authors provide a mechanism to pick an ordering over which actions for each agent are generated by a planner and show that they can pick the action that is "lexicographically smallest"
strengths_and_weaknesses: Unfortunately, this paper is written in a manner that makes it not possible to assess the quality of the paper. The main figure in the paper is unreadable, basic definitions are missing, spelling errors are abundant in almost every paragraph, and the figures are low enough pixel density that I have trouble reading them on my laptop (they appear to be screenshots). This is unfortunate because the underlying idea, to the extent that I can figure out what it is, that actions can be selected sequentially in centralized, cooperative multi-agent problems is interesting and the related work suggests the authors have expertise on this topic. I would ask the authors to do a more thorough job in preparing their paper for submission and provide some suggestions on how to do so.
questions: What is the connection between the results in this work and the results in “Multi-Agent Reinforcement Learning is a Sequence Modeling Problem”? and how does this paper go beyond the results there?

What is the importance of tie-braking and the connection to social conventions posited in the introduction?

What is the importance of small lexicographic orderings?


## PCRL: Priority Convention Reinforcement Learning for Microscopically Sequencable Multi-agent Problems

### Grammar + sundry things:

- I think in the title the word should be sequenceable?
- In abstract it should be “existing work has mainly tackled”
- Figure 2 is too low resolution and has to be improved to be legible. This is essential.
- The paper is full of what I would call “unbacked opinions”. The authors will say things like “the probability is hard to model” (line 104). Hard how? Why? The reviewers are asked to believe this on faith rather than with some backing. Or in line 178 the authors say “the approximation has physical meaning” but don’t state what it is.
- The paper should probably have a reference to “**Multi-Agent Reinforcement Learning is a Sequence Modeling Problem”** which contains quite similar ideas.
- What is the relevance of the reward hypothesis in lines 185-186? Also, there’s a paper to cite for this rather than a footnote.

### Writing:

- There are many components in the introduction that need to be rewritten to make the contributions of the paper clearer:
    - What do social conventions have to do with microscopic and macroscopic sequenceable models?
    - Why are priority conventions important?
    - Why is the sequencing a solution to action space blow-up?
    - More intuition on what microscopic sequenceability is
- This paper is not written in standard academic form and employs colloquialisms in many places such as lines 129-130
- There are spelling mistakes through the paper. Please employ a spell-checker.
limitations: The authors, in the conclusion, point out that they investigate their results on a relatively limited number of systems.
review_title: null

Meta-review: metareview: While the ideas in this paper are promising, there are issues with the paper's presentation and experimental results. The paper needs to be (further) updated to clarify the proposed method and discuss additional related work. More extensive experimental results are also needed to show the benefits of the proposed approach. 
confidence: Certain
Decision: Reject