Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: * The paper proposes a Target-oriented Multi-agent Communication and Cooperation mechanism (ToM2C) using the Theory of Mind. 

* It uses an observation encoder, ToM net, message sender, and decision-maker. The ToM net is designed for estimating the observation and inferring the goals (intentions). The message sender uses a graph neural network using inferred state and intention to generate local graphs. Once the agent receives the messages, it decides its goals using the inferred goals of others and received messages using the actor-critic. 

* The empirical results of the model on the multi-sensor multi-target covering scenario is presented and compared with 4 different baselines from state-of-the-art MARL methods - HiT-MAC, I2C, A2C  and Heuristic Search algorithm. 

main_review: Originality
* There is lot of research around Theory of Mind such as Machine Theory of Mind, multi-agent cooperation using theory of mind and ToM Net for two-player scenarios where an agent model another agent to infer its goal and preferences. The famework presented in this paper provides the modeling for multi-agent scenario consisting of multiple agents in a team where each agent is required to choose a subset of interesting targets and reaching them to contribute to the team goal.

* The communication reduction method used by the agents in the team that uses centralized training decentralized execution is interesting and makes the contributions novel. The authors train the connection choice network which provides the communication protocol for ’when’, ’who’ and ’what’ in multi-agent communication. 


Quality
* The scenarios used in the multi-sensor multi-target tracking environment makes the environment much more realistic for a goal-oriented setting especially as agent can only obtain the information from its local observation.

* The training strategies 1) curriculum learning strategy to increase episode length L and γ factor during the training and 2) to split the optimization of the ToM and RL model to avoid the nested loop of influence among  the ToM net and the policy network is clearly defined and implemented in the code attached to the paper. The environment and model are implemented in Python.The model is built on PyTorch and was trained on a machine with 7 Nvidia GPUs (Titan Xp) and 72 Intel CPU Cores.

* The literature review is thorough and helps in understanding the current state of the art to find the missing gaps. 


Clarity

Overall the paper is well organized and clearly written. The implementation details are clear. However, there are elements of the paper which do not provide enough information, such as:
* In order to obtain edge and compute the probabilistic distribution over the type of the  edge the authors propose to use the Interaction Networks (IN) by propagating the node and edge features spatially. However, the details about the marshalling function, m, that computes the matrix with interaction terms is missing? Additionally, which were the learnable parameters used in the objective function?  

* To sample the discrete edge type in the local communication graph for the gradients to be back-propagated for training, the Gumbel-Softmax trick, is used in the message sender module. The authors need to provide as to what was the softmax temperature, tao, used in the model? It is needed because softmax temperature is a learned parameter as it can be interpreted as entropy regularization where the Gumbel-Softmax distribution can adaptively adjust the confidence of proposed samples during the training process.

* In the Inferred-goal Filter, the authors propose to generate local graphs using inferred state and intentions for agent i. The filtered feature is concatenated with the estimated observation representation to form the estimated node feature (equation 1) to form the estimated node feature. However the probability threshold for the inferred goal is not defined and needs to be explained.  Furthermore, for the agent i, in the observation estimation, how would the observation be made by i,  if the other agents are not facing(pose) towards the target? 


Significance
* The submission provides the empirical results and compares them against the baselines from the existing MARL architectures:HiT-MAC, I2C, A2C  and Heuristic Search algorithm. They also provide the framework for modeling in multi-agent scenario consisting of n(>3) agents in a team where each agent is required to choose a subset of interesting targets and reaching them to contribute to the team goal.

* The pytorch implementation for the environment and the model is clear. 
limitations_and_societal_impact: * The authors present the limitations of the research by describing that their model was evaluated in a simulated scenarios only which can be extended to real world environments for the robotic agents. 
* It is acknowledged by the authors that their framework generated pseudo labels for communication reduction which can be optimized in future work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper addresses the problem of reasoning about the mental states of other agents in multiagent reinforcement learning using Theory of Mind, focused on cooperative MAS domains such as multi-sensor target tracking.  A deep learning architecture is proposed that combines four networks to create a centralized solution: (1) one network that uses attention to encode observations for communication messages, (2) another than estimates the observations and goals of the agents in the environment (generalizing prior work to more than two cooperating agents), (3) a third that learns a graph determining who should communicate with whom, and when communication should happen, and (4) an actor-critic network for determining the policies of agents.  Experimental results varying the number of sensors and targets in the environment demonstrate the performance of the proposed solution against baselines.
main_review: ===Originality===

The problems of multi-sensor target tracking in particular, and cooperative multiagent reinforcement learning are popular, important problems in the MAS and reinforcement learning communities.  The solution is an incremental combination of different types of neural networks to produce a new solution that has some differences from existing work.  I didn't fully understand the novelty of the solution as I was reading the paper.  The generalization to environments with more than two agents is important (albeit straightforward), and prior work seems to focus individually individually on the networks that make up the components of the proposed solution, so I think the proposed solution is more holistic than prior work, but I am not certain.

===Quality===

The related work seemed mostly complete, although I was left wondering why ToMnet wasn't chosen as a baseline, given the similarities with the focus of this work.  I also wasn't sure why A2C was chosen as the underlying algorithm and baseline, when PPO is typically used instead.

Another question I had about the model while reading was in Eq. 1, why is \delta used?  Wouldn't argmax G^* make more sense?  What happens if 2+ goals have G^* higher than \delta?  What happens if 0 goals have G^* higher than delta?

===Clarity===

Overall, the paper was relatively easy to follow, and I think I understood most of the details (I do have some questions below in the Significance section of this review).  Some clarifications would be helpful, such as more explicitly explaining the novelty of the solution -- what is unique vs. what is simply a combination of known techniques?  Also, a running illustrative example in Section 3 would help the reader understand what is the terms refer to and what is being learned.  Since you only have one test domain, it would be a great choice here.

===Significance===

My greatest uncertainty after reading the paper is about the significance of its contributions to the literature.  As mentioned above, I'm not exactly sure what the novelty in the work is.  Is it just a new holistic deep architecture?  Are some of the parts meaningful advances over the state-of-the-art beyond considering more than 2 agents?  To me, it appears to be a combination of known techniques with some incremental advances.

Also, only one domain was considered, so it is difficult to assess how the benefits of the approach might generalize.  Is it only useful in target-tracking domains?  Could it be applied to other cooperative MAS, which are a hot topic in deep MARL right now?  This is especially complicated by the fact that the domain appears to be relatively simple to solve since (1) simple rule-based policies were appropriate instead of needing reinforcement learning to learn low-level policies, and (2) a heuristic search algorithm outperformed all of the reinforcement learning solutions.  In real-world applications, we choose RL-based solutions when learning is necessary, possibly because the system needs to adapt to the dynamic environment, or when insufficient information is available a priori to know how to operate.  In this case, RL wasn't necessary, so it's difficult to know if the proposed approach outperformed the others because it achieved superior learning, or if it exploited some of the same reasons that heuristic search did well without learning.  So I'm left wondering if I'm misunderstanding something -- did heuristic search have access to information that wasn't available to the RL methods so that it is an appropriate upper bound on performance?  More details would help clarify here, which changes how the significance of the work might be viewed by the reader.

I was also unsure exactly how the scalability experiments were conducted.  You state that "Note that the model is only trained in the setting of 4 sensors and 5 targets, so this could be regarded as a zero-shot transfer".  How then did you conduct the experiments whose results are presented in Figure 6b?  If you have a different number of sensors than 4, wouldn't you need a different Theory of Mind Network, since it has components for each of the n agents in the environment?  Wouldn't this also affect the other components of the network since the communication graphs would be different?
limitations_and_societal_impact: The broader impacts were discussed in the paper, although also addressing limitations of the research would strengthen the overall work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a multi-agent communication and cooperation framework, ToM2C, where agents need to decide who and when to communicate with, and how to allocate subtasks (specifically, subtasks are defined by physical locations that agents need to navigate to / cover). The key idea is to train the agents to infer others' observations and goals, which may in turn helps achieve more effective and efficient communication/cooperation. The method adopts ToMnet-like architecture for a ToM module, and uses GNNs to fuse information and inference from multiple channels. Compared to the previous GNN architectures for multi-agent policies, it proposes two new designs, inferred-goal filter and communication reduction, which are enabled by the goal inference. The evaluation was done in a 2D continuous environment simulating a target coverage task. Compared to recent approaches, HiT-MAC and I2C, the proposed method reaches a higher reward with slightly fewer messages in this single task.
main_review: Strengths:
+ The work is well motivated. Indeed, ToM is a key ability for human cooperation and communication, which may also help machine agents' interactions.

+ The method shows improvement over 2 recent baselines, in terms of rewards and communication costs, which is encouraging.

Weaknesses:
- It is unclear to me how ToM2C as a problem setup differs from TarMAC [19] and I2C [13]. If it is conceptually the same setup, where an agent will send messages directed to individual agents, then I do not see how this is a different framework. 

- Training ToMnet module requires other agents' ground-truth observations (what entities they can see) and goals.

- This is not the first paper that introduces ToM (goal inference, belief inference) to multi-agent RL, e.g., the suggested references [1,2,3].

- Many design choices such as how goal is represented are restricted to target navigation / coverage type of tasks. Generalizability seems quite limited compared to existing methods (e.g., I2C). This also raises the question -- how much benefit we see in the experiment comes from these domain knowledge (decomposing the task into specific subtasks defined by the target locations) and additional GT supervision that are not available in existing methods. The proposed method does seem to enjoy an unfair leg up when considering existing methods need to train a policy from scratch without knowing what subtasks are.

- Related to the above point, what if we do not have manually defined subtasks?

- Given manually defined subtasks, the suggested reference [4] proposed a Bayesian approach for ToM-based cooperation, which is related and should be discussed in the paper.

- The evaluation was done in only one experiment. It is fine if the scope of the work is to solve the target coverage problem, but this intended scope should be clarified. Otherwise, it is unclear how this method could be applied to other tasks and environments, since there are domain-specific knowledge and supervision used for model design and training.

- I do not see the number of runs conducted to produce the results. I would also expect to see variance in Figure 5 & 6.

Minor comments:
- fig.x -> Fig. x

Suggested references:

[1] Mordatch & Abbeel, Emergence of Grounded Compositional Language in Multi-Agent Populations, AAAI 2018.

[2] Qi & Zhu, Intent-Aware Multi-Agent Reinforcement Learning, ICRA 2018.

[3] Shu & Tian, M^3RL: Mind-aware Multi-agent Management Reinforcement Learning. ICLR 2019.

[4] Wu et al., Too many cooks: Bayesian inference for coordinating multi-agent collaboration. CogSci 2020.

Post Rebuttal:
I would raise my rating to 5. I think a major revision is necessary so the submission is not ready in its current form.
limitations_and_societal_impact: I would like to see more discussions on the limitations of the proposed method and how it is grounded in prior literature.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper develops an approach to MARL that is inspired by theory of mind from psychology.
The authors claim that in problems like multi-sensor coverage that a better outcome (in terms of sensor coverage of
targets can be achieved) with such an explicit model that allows agents to communicate their "intentions".

main_review: # Originality
- The authors claim they introduce theory of mind (ToM) to multi-agent systems.
  There are a number of problems with this claim. 

- First, the theory of mind is a theory from psychology that makes the assumptions that other
  have mental attitudes (such as beliefs, desires and intentions) such as our own, and by making
  this assumption we can better coordinate/communicate with other humans/agents.

- So the statement that the paper incorporates the theory of mind into a multi-agent system is not quite accurate
  and is perhaps misleading at worst. 
  A more accurate statement would be that that the authors have incorporated ideas from the psychological theory of mind
  into a multi-agent system/architecture; or that they have developed a model of agent interaction that is **inspired** by
  the theory of mind from psychology. A such, the name of the model should be something else to distinguish it from the
  psychological theory. 

- Second, there many other multi-agent systems (see the literature on multi-agent systems, BDI agents and multi-agent
  team work) that incorporate modelling other agent's beliefs, desires and intentions and in the case of team work
  modelling joint-beliefs, joint-goals and joint intentions. 

- In any case the problem of having multiple sensors covering multiple targets is a well known problem that has had
  various approaches thrown at it. It has been addressed in both MARL and non-MARL (i.e. broader agent papers for a
  number of years). So it is isn't an original problem. Many agent systems have looked modelling the mental attitudes
  of team-mate agents. So, it isn't clear what the novel contribution is in this paper?
  Is it the combination MARL with ideas inspired by theory of mind? Explicitly stating this in the abstract/intro and
  conclusion would be a good idea.

- In a number of places you mention that agents share their "intentions". What is actually meant by this in this paper?
  Are agents sharing their policies?

- Note again that there is a body of work from the multi-agent systems community (specifically the field of BDI agents -
  that is agents based on the beliefs, desires and intentions model of agent reasoning) that have a very well defined
  meaning of intentionality. See works by Wooldridge, Roa/Georgeff on BDI agents for example. 

## Quality
- Overall the paper is well structured, the paper reads well and there are informative diagrams.
- As mentioned above, the omission of broader references from work in multi-agent systems community on modelling shared
  mental attitudes is an issue but otherwise the related work section seems ok.
- The method section seems ok. However, it would seem to me that the proposed solution seems overly complex for the
  examples used in the experiments. I have a suspicion that a simpler approach could have garnered similar results for
  the simple domains shown. What would be of more interest would be a domain where it is obvious that a could outcome
  could not be achieved without explicit cooperation/communication.
- So a possible critique would be, could the problems be solved witout a ToM-Net; or why do you need a neural network
  implementation that is inspired by ToM?
- The experiments section is good. The explanation of the environment is good, with explicit descriptions of the
  reward, action spaces and observation space. 
- While the baseline comparison is good, I think a better comparison would be your own architecture with and without a
  model inspired by theory of mind. 
- Should there be error bars on the bar charts in Figure 5?
- Also good to see the results shown in Figure 6 where the number of targets and the number of sensors is increased.
  A couple of questions here:
  1) It seems the heuristic search outperforms the ToM2C model; I was wondering if this was the case (as per my previous
  comment) have you over engineered the architecture where a simpler model achieves a better result. What is the
  advantage or contribution of your approach then?
  2) Should there be error bars in Figure 6 (how many cases were run)?
  3) The numbers of targets and sensors is still within the same order of magnitude. A more impressive scalability
     metric would be if you scaled the number of sensors and targets by orders of magnitude (e.g. 10, 100, 1000 etc)
     (or perhaps 5, 10, 50, 100, 500, 1000). You could then plot number of sensors vs number of targets as a 2D plot
     and have the % coverage rate as a heatmap. That would be interesting to see. 
- It was good to see Section 5 that includes a Conclusion and Discussion and even a broader impact section. 
  My recommendation for these sections would be to reorganise them slightly. Rename Section 5 Discussion and 
  rename "Broader Impact" to Conclusions and cover the following:
- 5. Discussion (discussion as is).
- 6. Conclusion (include a small discussion on limitations, future work and broader impact). 


## Clarity
- There are some parts of the paper where sentences read somewhat awkwardly, but nothing major in my opinion.
- I found the paper fairly clearly to read, but like all papers can also benefit through another edit just to make sure
  everything is clear for the reader. 

## Significance
- Multi-sensor target coverage is well studied program across many different AI literatures.
- I'm not sure the results here are significantly better than heuristic search or some of the other baselines
  to conclusively state that the approach taken will make a significant impact on the field.
- Note, a variation on my previous critique - this isn't to say that incorporating models inspired by the theory of mind is a
  bad idea in multi-agent systems. On the contrary, I and many others believe that it is a good idea.
  However, a neural network might not be the best way to do it. 

limitations_and_societal_impact: - A more in depth discussion of the limitations of the approach would be desirable. 
- I think the fruits collection example from Figure 1 actually detracts from the paper.
- In my opinion if the target coverage environment described in Section 4.1 was used 
  from the start of the paper, it would be a more compelling and "real-world" example.
- The fruit example doesn't really add anything to the paper in my opinion.
review_title: null

Meta-review: Overall there is not enough support from reviews for me to recommend acceptance.

During the discussion, two reviewers bumped their respective scores by a point, but the overall scores remain in the borderline range.

Reviewers agreed that there were clear strengths to the paper, and that author responses cleared up some (but not all) issues. The main remaining concerns with the submission are
1. discussion of limitations, especially around the ToM modeled and how it compares to ideas in the literature,
2. providing experiment details/extensions discussed in the review responses, and
3. that making these revisions (and others suggested in the discussion) would constitute a major revision (and thus perhaps require another set of reviews).

On #1:
* Reviewer koJ6 wrote "the concepts of theory of mind [...] are treated somewhat casually and some examination (and citing) of the literature in this area would improve the paper".
* Reviewer 2f7u agreed "the ToM modeled by this work is quite rudimentary" and the fix would be "a good statement of limitation and a reframing of the contributions".
* Reviewer 2f7u also wrote "I would like to see more discussions on the limitations of the proposed method and how it is grounded in prior literature", and "the additional discussions on prior work and the experimental results provided in the authors' responses are valuable".

On #2, there were new results and many experiment details provided in the authors' rebuttal which should be added to the submission text, as reviewers suggested and agreed by the authors.

On #3, Reviewer 2f7u summarized "the additional experimental results are very important for people to really understand how this work is different from the prior work, what were the settings for the experiments, and how general this approach is (i.e., what kinds of problems could benefit from it). There should be a major revision to address these points, both in literature review and in experiments." I agree that including the content covered in the discussion period would require significant revisions to the paper.

Additionally:
* Reviewer ADoL reamined concerned about the treatment of scalability experiments, though these concerns were not highlighted by other reviewers.
* The main concerns of Reviewer RvLm, around learnable parameters and observation estimation, were addressed, and that score was raised to a 7 (the highest among the reviewers).

These issues must be weighed against the significance of the contributions in the paper, which reviewers found to be valuable but limited:
* Reviewer koJ6: "I'm not sure the results here are significantly better than heuristic search or some of the other baselines to conclusively state that the approach taken will make a significant impact on the field"
* Reviewer ADoL: "My greatest uncertainty after reading the paper is about the significance of its contributions to the literature. [...] Also, only one domain was considered, so it is difficult to assess how the benefits of the approach might generalize"
* Reviewer 2f7u: "The evaluation was done in only one experiment"


To summarize, reviewers agreed that the underlying work has strengths and that the paper is close to acceptance, but that to push the manuscript to be a clear acceptance would require substantial revisions.

Because substantial revisions are required, this submission doesn't quite meet the bar given the other strong submissions.
Decision: Reject