Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new algorithm called TOM2C to solve the multi-agent reinforcement learning problem. To achieve goals in the MARL problem, communication between agents is important. However, it is often challenging due to scalability and communication costs. To solve this problem, the authors adopt the Theory of Mind to multi-agents. The agent infers the mental states and intentions of others upon partial observation. TOM2C has two kinds of agents: a planner that decides sub-goal and reaches a consensus, and a low-level executor that takes actions. The authors also provide a communication reduction method based on CTDE.
main_review: Strength
Overall model design to address the problem is interesting. In addition, performing two tasks (CN and MSMTC) for performance evaluation and choice of the baseline is sound. The ablation studies show how well the proposed model is structured. Code is provided to improve reproducibility.

Concerns
1.	As written in section 3, the proposed TOM2C receives a local partial observation. However, obtaining the current pose (guess that it is a ‘position’) of all agents does not seem to be a partial observation setting. More clarification should be provided about the partial observation. 
 2.	The authors claim that TOM2C can provide fully distributed execution. What about the learning process? Does TOM2C consist of centralized learning? 
3.	The paper does not state “who” provides rewards to all agents. In section 4.1, the authors ‘punish’ the team with a penalty, which implies the existence of a coordinator. Does TOM2C have a coordinator who gives rewards to all agents? Does it mean a centralized training framework? 
4.	In section 3.1, the global information is an input to the attention module and partial observation is an output. Is the attention model a part of the agent? If then, can we still say the partial observation? The input to the agent will be the global information. If the attention is a part of the environment, does the environment have an individual attention module for each agent? How can it be explained in a real-world scenario?  
5.	In section 4, the simulation results include the case of 3 agents. What happens if the number of agents increases? Does the proposed solution work in any multiagent system? 
6.	Section 4.1 describes the cooperative navigation task and multi-sensor target coverage task. However, the details of tasks are missing. How much do the targets move in a single step? It would be much convincing if results of other benchmarks are provided. For example, Hanabi could be a great candidate to evaluate the performances of the MARL algorithm.
7.	In research on Theory of Mind, inference performance of ToM net is a significant factor. So, it would be beneficial to show the accuracy of the ToM net as an individual model, not as a part of ToM2C. In addition, it would be interesting to show the performance change of the entire ToM net according to the number of agents.

summary_of_the_review: The approach that combines MARL and Theory of Mind is interesting. The proposed TOM2C can be applied to multiple MARL problems which need cooperation between agents. But the proposed algorithms should be explained more in detail. Also, more experiments are needed to persuade the reviewers regarding its performance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new method for communication and cooperation in multi-agent settings. The method relies on modelling other agents' intentions and internal states using Theory of Mind based neural nets. The predictions from the ToM model are used to decide how to communicate and coordinate with other agents. The authors test the method on two common multi-agent cooperation tasks to achieve SOTA communication efficiency and reward performance. The authors also show the utility of modelling mental states and using communication through ablation studies. Finally, the model shows flexibility in generalization, with consistent performance across different settings.
main_review: The paper was clearly written and easy to understand. The authors presented their motivations well. While using ToM inspired models has been explored in 2-agent tasks, integrating these models in a multi-agent setting to modulate communication was quite neat. Correspondingly, I found the ablation studies showing the utilities of ToM and the communication reduction to be very useful. I was particularly impressed by the model's generalization across different task settings and scales (different numbers of cameras and targets).  

One of the weaker points of the paper was the use of privileged information to learn the ToM module; the ToMnet uses supervision from ground truth observations from other agents and requires supervision from their goal states. Access to others' true observations and goal states is unrealistic. Further, this reduces the extent to which the model is decentralized during training (as observation and goal states are directly used for supervision). Finally, the ToM model is limited in its novelty, but its use to influence communication is novel.

The authors evaluate the approach on two synthetic tasks: cooperative navigation and multi-sensor target coverage. I would have liked the method to be evaluated in more complex settings (as the authors themselves note in the conclusion) like SMAC [1] or the Hanabi Challenge [2](that specifically test for decentralized cooperation). If the authors believe that their method is inapplicable to the domains, then I would have liked to see a discussion on how this approach could be adapted. 

One of the other lacking aspects of the paper was a discussion of other hierarchical baselines for modelling agents. I would have liked to see a discussion on such approaches. For example, [5] use a similar hierarchical approach for cooperation in a two-agent setting.

Finally, what do the authors think about approaches in IRL/ meta-IRL;  that directly try to predict the goals/ rewards of an agent?

I have added a few relevant references that work with agency reasoning and cooperation; I believe they are related to theory of mind inferences and collaboration. The authors might include them if they think that they are relevant.

Agency reasoning benchmarks: AGENT [3] BIB [4]

Cooperation benchmarks: Watch2Help [5] Overcooked [6] PHASE [7]

[1] Samvelyan, M., Rashid, T., De Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G., ... & Whiteson, S. (2019). The starcraft multi-agent challenge. *arXiv preprint arXiv:1902.04043*.

[2] Bard, N., Foerster, J. N., Chandar, S., Burch, N., Lanctot, M., Song, H. F., ... & Bowling, M. (2020). The Hanabi challenge: A new frontier for ai research. *Artificial Intelligence*, *280*, 103216.

[3] Shu, T., Bhandwaldar, A., Gan, C., Smith, K. A., Liu, S., Gutfreund, D., ... & Ullman, T. D. (2021). AGENT: A Benchmark for Core Psychological Reasoning. *arXiv preprint arXiv:2102.12321*.

[4] Gandhi, K., Stojnic, G., Lake, B. M., & Dillon, M. R. (2021). Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others. *arXiv preprint arXiv:2102.11938*.

[5] Puig, X., Shu, T., Li, S., Wang, Z., Liao, Y. H., Tenenbaum, J. B., ... & Torralba, A. (2020). Watch-and-help: A challenge for social perception and human-AI collaboration. *arXiv preprint arXiv:2010.09890*.

[6] Carroll, M., Shah, R., Ho, M. K., Griffiths, T., Seshia, S., Abbeel, P., & Dragan, A. (2019). On the utility of learning about humans for human-ai coordination. *Advances in Neural Information Processing Systems*, *32*, 5174-5185.

[7] Netanyahu, A., Shu, T., Katz, B., Barbu, A., & Tenenbaum, J. B. (2021). PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception. *arXiv preprint arXiv:2103.01933*.
summary_of_the_review: The use of a TOM model to facilitate communication for cooperation was impressive. I believe the paper is slightly below the acceptance threshold in its current state due to a lack of tests on more complex benchmarks that test for decentralized cooperation (SMAC/ Hanabi) and limited applicability because of access to unrealistic privileged information during training.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper designs a new agent with the ability to estimate the local observations and potential goals of other agents, conduct message passing of such local estimates with neighboring agents, then use the outcome of message passing in a two-level hierarchical policy to select its own goal and take primitive actions conditioned on that goal. Experiments and ablations were done in the cooperative navigation benchmark and a multi-sensor target coverage task, in comparison to previous multi-agent communciation baselines.
main_review: I commend the authors for designing and implementing this highly complex agent architecture, which clearly exceeds the complexity of many previous multi-agent algorithms (having seen the supplementary material, I see the agent model is thousands of lines of code). Precisely because of the complexity, it is imperative that the technical writing is organized perfectly to ensure that readers appreciate the work and are not put off due to the complexity. In its present form, this paper hasn't met that standard. I believe the architecture is a valuable contribution to the advancement of agent design, especially for cooperative MARL, so I encourage the authors to take the following criticism in a positive way to improve the writing.

1. Referring to the abstract, do agents really "reach a consensus"? By definition, a consensus is achieved when many agents all converge to the same decision or estimation of a quantity. This paper is concerned with multi-goal problems, so clearly agents don't all choose the same goal. If the authors intend to say that agent $i$'s prediction of agent $j$'s goal matches agent $j$'s actual chosen goal, then they authors need to verify this in their experiments (I don't see such a test) and describe it as so. Calling that "consensus" is confusing.
2. Still in the abstract, authors say agents chooose "sub-goals". Does this mean an agent chooses many different sub-goals at various steps in an episode? How many primitive actions are taken per sub-goal? Is each sub-goals drawn from a predefined set of possible goals? Or is a sub-goal a latent variable whose meaning emerges via some kind of unsupervised learning? Or are there hand-designed rewards for each sub-goal? I have these questions even after multiple readings of the paper. Reference [1] below is an example of such details that one expects from hierarchical MARL methods.
3. Page 3, related work section, the authors claim that CTDE methods are flawed becaue "coooperation collapse in spite of a slight change in the team formation,..." Which work has shown this? What is meant by "change in team formation"? Plenty of works have shown excellent generalization, especially since many multi-agent environments have localized interactions, so centralized training with few agents is sufficient for generalizing to decentralized settings with many more agents.
4. Page 4, "pose" $(\phi_1,\dotsc, \phi_n)$ is undefined. It also seems way too specific to particular applications.
5. Page 4, "The final communication connection is sampled according to the computed graph edge features." How are edge features defined? Even after reading to the end of the methods section, I still can't find the answer.
6. Section 3.2, authors say "Therefore, the entire ToM net of agent i is actually composed of n-1 separate ToM nets", but then later say it's actually just a single model because agents are homogeneous and one model can be used to process all other agents' info. Be consistent.
7. Each agent infers the intended goal of other agents. The accuracy of this prediction needs to be shown in an experiment.
8. Page 4, last line, authors mention an "auxiliary task" for agent $i$ to estimate the observation of agent $j$. It turns out, two pages later in page 6, that this auxiliary task is that agent $i$ predicts whether agent $j$ can see a landmark. Authors should define that auxiliary task immediately after introducing it, not 2 pages later. Also, the accuracy of this prediction needs to be shown in an experiment.
9. Page 5, in the paragraph "connection choice", what is an "effect", as in "node effect" and "edge effect"? This is not a standard term in the graph neural network literature.
10. Page 6, in the "Communication Reduction (CR)" paragraph, there's a model that trims the edges of the communication graph, and it is "supervised by the generated pseudo labels". What exactly is this thing? What are the inputs and outputs? Is it a binary classifier that decides whether to cut or not cut an edge? This is completely unclear.
11. Section 3.4, first paragraph, what do authors mean by an "actor-critic feature"? The observation input to the policy network?
12. Section 3.4, first paragraph, authors say "The actor decides its goals $g_i$ according to $\eta_i$. How does actor "decide $g_i$ according to $\eta_i$"? What is the reward for the goal-selector? What is the reward for the low-level executor? By goal $g_i$ and "sub-goal", do you mean the same thing? What is the reward corresponding to each sub-goal?

Of course, some of these questions may be addressed by the paper somewhere, but the fact that I still have these questions after multiple readings shows that the paper's organization really needs improvement (not just minor local changes) and the responsibility isn't entirely on the reader's part.

There are multiple typos/grammar mistakes that the authors should fix.
"to decomposes the", "in complexer scenarios"

[1] Yang et al. "Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery." AAMAS. 2020.
summary_of_the_review: Technical and experimental work is impressive but the organization of writing and lack of clarity in certain key parts makes this paper fall under the necessary standards.

During rebuttal: authors have provided detailed answers to my questions and revised the manuscript to improve clarity.
review_title: null

Meta-review: The current paper presents a new method for communication and cooperation in multi-agent settings. Specifically, the authors propose to model other agents' intentions and internal states using ToM nets and using these predictions to then decide how to communicate/coordinate. The authors present experiments in two multi-agent cooperation tasks (multi-sensor multi-target coverage and cooperative navigation), compare against 4 previous methods (TarMAC, I2C, MAPPO and HiT-MAC) and perform the necessary ablations studies and find that their method achieve better rewards in both environments.
All reviewers have found the present study to be novel with convincing experimental findings. Reviewers have raised some concerns however a great deal of those have been addressed by the authors during the rebuttal and many of these points have now been incorporated in the paper. 

Having read the paper and considering the reviews I agree with the reviewers that this manuscript will make a good addition to the program of ICLR and as such I recommend its acceptance.
Decision: Accept (Poster)