ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind
Authors: Yuanfei Wang
fangwei zhong
Jing Xu
Yizhou Wang
Venue: ICLR 2022 Poster (2022)
Keywords: Theory of Mind
Target-oriented Multi-Agent Cooperation
Multi-agent Communication

Abstract: Being able to predict the mental states of others is a key factor to effective social interaction. It is also crucial for distributed multi-agent systems, where agents are required to communicate and cooperate. In this paper, we introduce such an important social-cognitive skill, i.e. Theory of Mind (ToM), to build socially intelligent agents who are able to communicate and cooperate effectively to accomplish challenging tasks. With ToM, each agent is capable of inferring the mental states and intentions of others according to its (local) observation. Based on the inferred states, the agents decide "when'' and with "whom'' to share their intentions. With the information observed, inferred, and received, the agents decide their sub-goals and reach a consensus among the team. In the end, the low-level executors independently take primitive actions to accomplish the sub-goals. We demonstrate the idea in two typical target-oriented multi-agent tasks: cooperative navigation and multi-sensor target coverage. The experiments show that the proposed model not only outperforms the state-of-the-art methods on reward and communication efficiency, but also shows good generalization across different scales of the environment.


Published as a conference paper at ICLR 2022
ToM2C: TARGET- MULTI-AGENT COMMU- NICATION AND COOPERATION WITH THEORY OF MIND
Yuanfei Wang" Fangwei Zhong 2,5, Jing Xu', Yizhou Wang?
1 Center for Data Science, Peking University
2 School of Artificial Intelligence, Peking University
3 Center on Frontiers of Computing Studies, School of Computer Science, Peking University
4 Adv. Inst. of Info. Tech, Peking University
$ Beijing Institute for General Artificial Intelligence (BIGAI)
yuanfei_wanl wang, zfw, jing.xu, xu, y zhou. , wang }@pku. edu. cn
ABSTRACT
Being able to predict the mental states of others is key factor to effective social interaction. is also crucial for distributed multi-agent systems, where agents are required to communicate and cooperate. In this paper, we introduce such an important social-cognitive skill, i.e. Theory of Mind (ToM), to build socially intel- ligent agents who are able to communicate and cooperate effectively accomplish challenging tasks. With ToM, each agent is capable of inferring the mental states and intentions of others according its (local) observation. Based on the inferred states, the agents decide "when" and with "whom" to share their intentions. With the information observed, inferred, and received, the agents decide their sub-goals and reach consensus among the team. In the end, the low-level executors indepen- dently take primitive actions to accomplish the sub-goals. We demonstrate the idea in two typical target-oriented multi-agent tasks: cooperative navigation and multi- sensor target coverage. The experiments show that the proposed model not only outperforms the state-of-the-ud methods on reward and communication efficiency, but also shows good generalization across different scales of the environment.
INTRODUCTION
Cooperation is key component of human society, which enables people divide labor and achieve common goals that could not be accomplished independently. In particular, humans are able to form an ad-hoc team with partners and communicate cooperatively with one another (Tomasello 2014). Cognitive studies (Sher et al. [014]] [annfyy et al. (2155; Etel & Slaughter 2019) show that the ability to model others' mental states (intentions, beliefs, and desires), called Theory of Mind (ToM) (Premack & Woodruff 1978). is important for such social interaction. Consider a simple real-world scenario (Fig. D). where three people (Alice, Bob and Carol) are required to take the fruits (apple, orange and pear) following the shortest path. To achieve it, the individual should take four steps sequentially: 1) observe their surrounding: 2) infer the observation and intention of others; 3) communicate with others to share the local observation or intention if necessary; 4) make decision and take action to get the chosen fruits without conflict. In this process, the ToM is naturally adopted in inferring others (Step 2) and also guides the communication (Step 3).
In this paper, we focus on the Target-oriented Multi-Agent Cooperation (ToMAC) problem, where agents need to cooperatively adjust the relations among the agents and targets reach the expectation, egg. covering all the targets (Xu et al.) 2020). Such problem setting widely exists in real-world applications, e.g., collecting multiple objects (Fig. D. navigating to multiple landmarks (Lowe et al.) 2017), following pedestrian (Zhong et al. 202 and transporting objects (Tuci et al.) 2018). While running, the distributed agents are required to concurrently choose subset of interesting targets and optimize the relation to them to contribute to the team goal. In this case, the key to realizing high-quality cooperation is to reach a consensus among agents to avoid the inner-team conflict. However, it is still difficult for the off-the-shelf multi-agent reinforcement learning methods, as they only implicitly model others in the hidden state and are inefficient in communication.
Here we propose Target-oriented Multi-agent Communication and Cooperation mechanism (ToM2C) using Theory of Mind. Shown as Fig. 2 each agent of a two-level hierarchy. The
indicates equal contribution

---

Published as a conference paper at ICLR 2022
An
Oob can sne apple morer to.t.
lunt une
Ganal is rearer to APp meee har.
apple
An orongg and pasr hame
HO shuskl take ios
Step 1. Observing
Step 2. Inferring
Applls so tuke
Bob
takn crongn
poir bahind the wall
Step Communicating
Step Decision making
Figure I:
A fruits collection example. The agents are required to cooperatively collect the three target objects (apple, pear, and orange) in the room fast possible. The whole process can be divided into 4 steps. In Step 1, 3 agents observe the environment and obtain the state of the visible targets. In Step 2, each agent tries infer what other agents have seen, and which targets they shall choose as goals. In Step 3. each agent decides whom to communicate with according to the previous inference. In Step each agent decides its own goal of target based on what observed, inferred, and received.
high-level policy (plamner) needs cooperatively choose certain interesting targets sub-goal to deal with, such as tracking certain moving objects or navigating to a specific landmark. Then, the low-level policy (executor) takes primitive actions reach the selected goals for l steps. Concretely, each agent receives local observation and estimates the observation of others in the Theory of Mind Network (ToM Net). Combining the observed and inferred states, the ToM Net will predict/infer the target choices (intentions) of other agents, After that, each agent decides 'whom' to communicate with according to the local observation filtered by the inferred goals of others. The message the predicted goals of the message receiver, inferred by the sender. In the end, all the agents decide their own goals by leveraging the observed, inferred, and received information. With the inferring and sharing of intentions, the agents can easily reach consensus to cooperatively adjust the target-agent relations by taking a sequence of actions.
Furthermore, we also introduce a communication reduction method remove the redundant message passing among agents. Thanks to the Centralized Training and Decentralized Execution (CTDE) paradigm, measure the effect of the received messages each agent, by comparing the output of the planner with and without messages. Hence, we can figure out the unnecessary connection among agents. Then we train the connection choice network channels in supervised manner. Eventually, we argue that ToM2C systemically solves the problem of 'when', 'who" and 'what' in multi-agent communication, providing an efficient and interpretable communication protocol for multi-agent cooperation.
these dispensable
The experiments are conducted in two environments. First, in the cooperative navigation sce- nario (Lowe et al.) 2017). the team goal is to occupy landmarks (static targets) and avoid collision. Then we evaluate our method in a more complex scenario, multi-sensor nario (Xu et al. 2020). The team goal of the sensors is to adjust their orientation to cover as many moving targets as possible. The results show that our method achieves the best performance (the highest reward and the lowest communication cost) among the fate-offthe-aa MARI. methods, e.g., HrT-MAC (Xu et al., 2020), 12C (Ding et al., 2020), MAPPO (Yu et al. [222]] and TarMAC (Das) al. 2019). Moreover, we further show the good scalability of ToM2C and conduct ablation study to evaluate the contribution each key component in ToM2C.
tarret covering sce-
Our contributions are in three-folds: 1) We introduce rognition-inspieed social agent with the ability to infer the mental states of others for enhancing multi-agent cooperation in target-oriented tasks. 2) We provide ToM-based communication mechanism and communication reduction method to improve the efficiency of multi-agent communication. 3) We conduct experiments in two typical target-oriented tasks: the cooperative navigation and multi-sensor multi-target coverage problem.
2

---

Published as a conference paper at ICLR 2022
Message Sender
Theory of Mind Net
Decision Maker
f F(lter(si,.E)) Mej (Fig.
Critic
Goal Inference
8ix
C Connection Choice
hservation Estimation
ToM/
ToM k
TOME
Planner
0i
$ R
%
Executor
E
PrimitiveAction a;
O bservaaton Encoder
Target
Concatenate
10
The architecture of ToM2C for each individual. There are four key components: Observation encoder, Theory of Mind net, Message sender, and Decision maker. Each agent first receives local observation and encodes it with the encoder. Then it performs Theory of Mind inference to estimate the observation of others and predict their goals. Next, it decides 'whom' to communicate with according to local observation filtered by the inferred goals of others. In the end, the planner in decision maker outputs the sub-goal according to what it observes, infers, and receives. The low-level exccutor takes primitive actions to reach the chosen sub-goal independently.
Figure 2:
2 RELATED WORK
Multi-agent Cooperation and Communication. The cooperation of multiple agents is crucial yet challenging in distributed systems. Agents' policies continue to shift during training, leading to a non-stationary environment and difficulty in model convergence. To mitigate the non-stationarity. the centralized training decentralized execution (CTDE) paradigm is widely employed in the recent multi-agent learning works (Lowe et al. 2017] Foerster al.) 2018 /Sun/hha al. 2018 Rashid et al., 2018 Iqbal & Sha 2019). However, these methods only implicitly guide agents to overfit certain policy patterns of others. Without communication mechanism, agents lack the ability to negotiate with each other and avoid conflicts. As a result, it is hard for the individual agent to quickly adapt to unseen cooperators/environmentt Learning to communicate (Sukhbaatar et al.) 2016) is a feasible way to promote efficient multi-agent cooperation. Unfortunately. most previous works (Sukhbaatar al., 2016 Das et al. 2019 (Singh et al. 2019 require broadcast communication channel, leading to huge pressure on bandwidth. Besides, even though 12C (Ding et al.] (020) proposes an individual communication method, the message is just the encoding of observation, which is not only costly but also uninterpretable. (Li et al.) (2020a) designs mechanism to share the camera poses according to the visibility of the target, by exploiting the behavior consistency among agents. In this paper, we will investigate more efficient pecr-to- pecr-to-peer communication, based on theory- offory-of-mind. Hierarchical frameworks (Yang et al. 2019: Kim et al.) 2020 Xu et al.] 2020) are also investigated to promote multi-agent cooperaation/coordiaatoo HFT-MAC (Xu al., 2020) is the closest work to ours. It proposes a hierarchical multi-agen coordination framework to decompose the target coverage problem into two-level tasks: assigning targets by the centralized coordinator and tracking assigned targets by decentralized executors. The agents in ToM2C are also of a two-level hierarchy. Considering the natural structure of ToMAC, also decompose the cooperation tasks in this way, rather than learning skills in an unsupervised manner (Yang et al. 2019). Differently, both levels in ToM2C are enabled perform distributively, thanks to the use of ToM and communication mechanism.
Theory of Mind. Theory of Mind is a long-studied concept in cognitive science (Sher et al. 2014
Sanfey et al. [2015] Etel & Slaughter, 2019). However, how to apply the discovery in cognitive science to build cooperative multi-agent systems still remains a challenge. Most previous works make use of Theory of Mind to interpret agent behaviors, but fail to take a step forward to enhance cooperation. For example, Machine Theory of Mind (Rabinowitz et al., 2018) proposes meta-learning method to learn a ToMnet that predicts the behaviors or characteristics of a single agent. Besides, (Shum) let al. 2019) studies how to apply Bayesian inference to understand the behaviors of group and predict the group structure. (Track et al. (2018) introduces the concept of Satisficing Theory of Mind, which refers to the sufficing and satisfying model of others. (Shu et al. 2021) and (Gandhi et al.
3

---

Published as a conference paper at ICLR 2022
introduce benchmarks for evaluating machine mental reasoning. These works mainly focus on how to accurately infer the mental state of others, rather than interactive cooperation. (Piig et al.) Carroll et al. 2019 Netanyahu et al. 2021) studies the human-Al cooperation with mental inference. (Lim et al., (020) considers a 2-player scenario and employs Bayesian Theory of Mind to promote collaboration. Nevertheless, the task is relatively simple and it requires the model of other agents to do the inference. M'RL (Shu & Tian 2019) proposes to train a manager that infers the minds of agents and assigns sub-tasks them, which benefits from the centralized mechanism and is not comparable with decentralized methods. (Wu et al. (02T) introduces Bayesian approach for ToM-based cooperation, yet assumes that the environment has a partially ordered set of sub-tasks, Differently, we study how to leverage ToM to guide efficient decentralized communication to further enhance multi-agent cooperation.
2021
2021:
Opponent Modeling. Opponent modeling (He et al. 2016 Raileanu et al. 2018 Grover et al. 2018, Zhong et al., 2021a) is another kind of method comparable with Theory of Mind. Agents endowed with opponent modeling can explicitly represent the model of others, and therefore plan with awareness of the current status of others. Nevertheless, these methods rely on the access to the observation of others, which means they not truly decentralized paradigms. Compared with existing methods, ToM2C applies ToM not only to explicitly model intentions and mental states but also improve the efficiency of communication to further promote cooperation.
3 METHODS
In this section, we will explain how to build a target-oriented social agent to achieve efficient multi-agent communication and cooperation. We formulate the target-oriented cooperative task as Dec-POMDP (Bernstein et al.] (002). The aim of all agents is to maximize the team reward. The overall network architecture is shown in Fig. 2 from the perspective of agent i. The model is mainly composed of four functional networks: observation encoder, Theory of Mind network (ToM Net), message sender, and decision maker. First, the raw observation o,, indicating the states of observed targets, will be encoded into E; by an attention- based encoder. After that, the ToM Net takes Theory of Mind inference ToM (G; E, 4 estimate the joint intention (sub-goals) of others Gi. according to the encoded observation F; and the poses of agents + (Ø1,... 0..) In details, taking 7 ToM.,j as an example, it uses the estimated observation G. infers the probability of agent j choosing these targets as its goals, denoted as D.,- The estimation of Ci is based on the pose Φj. In general, pose éj indicates the location and rotation of the agent j. Specifically, it can be represented as a 6D vector (ar, y, z, roll, yow, pitch) in 3D space and a 3D vector (z, V, yaw) in 2D plane. After the ToM inference, the message sender decides whom to communicate with. Here we employ a graph neural network to model the connection among agents. The node feature of agent is the concatenation of f.j and Fg filtered by -h The final communication connection is sampled according to the computed graph edge features. Agent will send HT, to agent j if there exists communication edge from to j. In the end, we aggregate G E, and received messages Σ 9h.5 as 7); for the decision making. The planner n.. (gi (m)) guided by the team reward, chooses the sub-goal g to the low-level executor n.'¹ (a,lo,,9) 9t). which takes K steps primitive actions to reach the sub-goal.
In the following sections, we will illustrate the key components of ToM2C in detail.
3.1 OBSERVATION ENCODER
We employ an attention module (Vaswani et al. 2017) to encode the local observation. There are two prominent advantages of this module. On one hand, it oopulation-ivvarian and order-invariant, which is crucial for scalability. On the other hand, multi-target information can be encoded into a single feature due to the weighted sum mechanism. In this paper, we use scaled dot-product self- attention similar to (Xu et al., 2020). The local observation is first transformed to key K,, query
Q: and value V; through 3 different neural networks. Then the output F, so sof'tmaa fotmmar( Q.K:" T + Vi.
Vdc where du is the dimension of one key. oz,q and Eigg represent the raw and encoded feature of target agent i respectively.
3.2 THEORY OF MIND NETWORK (ToM NET)
Inspired by the Machine Theory of Mind (Rabinowitz al. 2018), introduce the ToM Net that enables agents to infer the observation and intentions of others. For agent i, the ToM Net takes the poses of agents & and the encoded local observation F, input. Then it infers the observation 6
4

---

Published as a conference paper at ICLR 2022
and goals G2 of others. Most previous work (He et al. 2016 Raileanu al. 2018 Lim al. 2020
consider two-player scenarios, where the agent only needs to model one other agent. Instead, we take step forward to run our model in more complex multi-agent scenario consisting of n (>>) agents. Therefore, the entire ToM Net of agent is conceptually composed of n separate sub-modules, which foeus on modeling the mind of one agent respectively. Shown Fig. ToM,; is an ensemble of To ToM,, k. I T.,,, In practice, we simply let these models share parameters since all the agents are homogeneous. In this way, we can accordingly adjust the number of sub-modules in ToM Nets with the change of agent number. A single ToM Net is made up of two functional modules: Observation Estimation and Goal Inference.
Observation Estimation. The first step of ToM inference is to estimate the observation representation of the other agent. Intuitively, when an agent tries to infer the intention of others, should first infer which targets are seen by them. Here, we take Bob in Fig. [ an example. Before he tries to infer the goals of Alice and Carol, he first infers that Alice cannot observe the apple but Carol can. Similarly, agent i infers the observation of agent j. denoted as Ei with the pose Φj. Note that Ei is only representation of the estimated observation. To better learn this representation, we further introduce an auxiliary task, where agents should infer the relation between the targets and other agents, e.g.. which fruits are visible or closest to Alice. In practice, we employ the Gated Recurrent Units (Cho) eet al. 2014) (GRUs) to model the observation of others on time series. In target coverage task, agent infers which targets are in the observation field of agent j and in cooperative navigation task infers which landmark is closest to agent j.
After agent i finishes the observation estimation of others, it is able to predict which targets will be chosen by them at this step. Just like humans, the agent infers the intentions of others based on what it sees and what it thinks that others see. If we denote this goal inference network as a function GI, then the process can formulated as gf GI(E, ft where 9i.j.g stands for the probability of agent j choosing target q, inferred by 2. Since there are total of agents and m targets in the environment, G E (n-1))m With ToM Net, each agent holds a belief on the observation and goal intentions G* of others. Such belief is not only taken into account for decision making, but also serves as an indispensable component in communication choice.
Goal Inference.
Learning ToM Net. We introduce two classification tasks to learn the ToM Net, which is param- eterized by @'oM First, the ToM Net infers the goals G of others. Note that g. indicates the probability of agent choosing target g, inferred by i. Meanwhile, agent j decides its real goals gj- Therefore, 9j can be the label of gig. The Goal Inference loss is the binary cross entropy loss of this classification task:
1 G = N
Los, log(() + (1 9i.n) ) log(1 - 96.5
(1)
J#L 4
Secondly, the estimated observation is additionally guided by the auxiliary task mentioned before. The agent i infers the relation between the targets and agent j. denoted as ci The ground truth is the real observation field cy. Ggg- = 1 indicates that agent observes target q or landmark q is closest to agent j. Similar to the previous Goal Inference task, this Observation Estimation learning also adopts binary cross entropy loss:
1O
1sga log((i.ja)) + (1 - (1-cag) llog11 - G.
(2)
N
L(OTMM) 11 + LOR
3.3 MESSAGE SENDER
The message sender leverages the inferred mental state of others from ToM Net to decide 'when' and with 'whom' to communicate, independently. During communication, the message sent from agent i to is just the inferred intention Ii.j- Moreover, further propose communication reduction method, which can remove useless connections to improve communication efficiency.
In practice, we use Graph Neural Network (GNN), similar to (Li et al. 2020b; Battaglia et al. 2016), to model the communication network in an end-to-end manner. Each agent computes its own graph based on its observation and inferred information of others. Specifically, in the perspective of agent i, we make use of the inferred intention of to filter the observation E, to generate graph node features. Edge features computed based on node features, which are further transferred into a probabilistic
5

---

Published as a conference paper at ICLR 2022
distribution over the type of edges(cut retain). Communication connections are finally sampled according to the probabilistic distribution.
Inferred-goal Filter. The feature of agent j in the graph of agent is the target features filtered by the inferred goals Gi as follows. is a probability threshold, which we set to 0.5 in this paper. If Pi.. 6, then agent i will consider target q as the goal that will be chosen by agent j. Then we concatenate the filtered feature E Ex= (9i5, > 6) Es.g with the estimated observation representation Ei to form the estimated node feature un, = (E)) 4.J). For agent i itself, W1, = (2g Ei.g. ez), where e; is also computed by Observation Estimation with the pose of i.
Connection Choice, For scenario consisting of n agents, there is total of n directed graphs Ei) is the local graph for agent to compute the communication connection from agent i. The vertices V. = {f (u.,))) where f is node feature encoder. Edges & (us (. where a is an edge feature encoder. Like the Interaction Networks (IN) (Battaglia) et al. 2016). we propagate the node and edge features spatially to obtaín node and edge effects. For convenience, we will describe only graph G; in the following formula and omit the index i. Let V be the encoded node feature of j. and h2 be the node effect. Similarly, let &g,& be the encoded edge feature, hg.4 be the edge effect. Initially, hy = Vj. hy,k Eg,-- Then the graph iterates several times to propagate the effect:
g = (91, ...) 9;
h, nndoo (V,.hj, Chk.))
(4)
hj.A gedge (hi. Dus, hz.k)
(5)
In the end, we obtain the final edge feature (Er,,, bi.)) and compute the probabilistic distribution over i.j the type of the edge (pout + Pretain 1). Here we apply the Gumbel-Softmax trick (Jang et al. 2016
Maddison et al. 2016) to sample the discrete edge type, so the gradients can be back-propagated in end-to-end training. Considering that it is the local communication graph of agent only the types of E are sampled. If edge Er, is retained, agent will send Jī, to j.
Communication Reduction (CR). In practice, the GNN tends to learn densely connected graph, even if applying a sparsity regularization while learning. However, we observe that the decisions made by receivers are not always influenced by the received messages, indicating that some connections are actually redundant in the GNN. Therefore, it is necessary for us to figure out the really valuable connections from the densely connected networks. To this end, we measure the effectiveness of each connection by taking an ablative analysis. To be specific, we observe the non-communicative sub-goals 9, of agent by removing the received message in T): We can estimate the effect of the received messages to agent by measuring the KL-divergence, referred as X D l99;). Here we set constant threshold J to generate a binary pscudo label for learning to remove redundant connections. Specifically, if x < T, we regard that the messages are redundant to agent i. Thus the edges pointing at i will be labeled as 'cut", l,,; = O. Otherwise (x > T), labeled as 'retain', L,,; = 1. Then the tuning of message-sender network follows the binary cross entropy loss:
M Σ > Ihis log(pietan)) + (1 kt.)) log(p'd)))
1
3.4 DECISION MAKER
Once the agent receives all the messages, can decide its own sub-goals of targets based on its observation, inferred intentions of others and received messages. Therefore, the actor feature ?7 = (Eg max; Figz E. is the input to the actor network. The second term max; gi., refers to the max inferred probability of a target to be chosen by another agent. The third term Σ, 97, refers to the sum of the messages from others, indicating how much certain others infer that agent should choose the target. The actor decides its goals y; according to 7/:- The centralized critic obtain global feature (11.....%) to compute value. The low-level executor n. [ax jo,, 9:) takes primitive actions to accomplish the sub-goal. Similar to HiT-MAC (Xu et al. 2020). the high-level planner is guided by the team reward. The low-level executor guided by the soal-conditioned reward, measuring the quality of the achievement of the sub-goals.
3.5 TRAINING
Following the Centralized Training Decentralized Execution (CTDE) paradigm (Rashid et al. 2018). also train our model in centralized manner, ie.. feeding the global observation the critic and
6

---

Published as a conference paper at ICLR 2022
using the actual observation and goals of all agents to supervise the ToM Net. The overall network is trained by Reinforcement Learning (RL) in an end-to-end manner. We adopt standard A2C (Mnih et al. 2016) the RL algorithm, while any MARL method with CTDE framework is also applicable, such as PPO (Schulman et al.; 2017 Yu et al. 202T). Besides, as we mentioned in Sec. [.2 we provide the actual observation and goals of other agents supervise the training of the ToM net. We also apply communication reduction(sec. 3.3 to tune the Message-Sendee network.
Training Strategy. We find that it is hard for an agent to learn long-term planning from scratch. Therefore, we set the initial episode length L and discount factor to a low value, forcing agents to learn short-term planning first. During training, the episode length and discount factor y increase gradually, leading the agents to estimate the value on a longer horizon. Furthermore, we freeze the ToM net while the other parts of the model (parameterized by gother) are updated through RI. The reason is that the ToM net infers the goals of others, and the policy network is continuously updated during RL training. Meanwhile, the output of ToM net is a part of the input to the policy network. If we train them simultaneously, they are likely to influence each other in nested loop. Therefore, we only collect the ToM inferred data into a batch during RL training. Once the batch is large enough, stop RL and start ToM training to minimize ToM loss in Eq. 3. More details about the model and training strategy can be found in Appendix
EXPERIMENTS
We evaluate ToM2C in two typical target-oriented multi-agent tasks: cooperative navigation (CN) (Lowe et al. 2017) and multi-sensor multi- target coverage (MSMTC) (Xu et al.) 2020). CN is the simplest environment, where targets (landmarks) are static and the agent only needs to choose one target as sub-goal. In CN, n agents need to occupy n landmarks (targets) so as to minimize the distances of all landmarks to their nearest agents. All agents share a team reward, which is the sum of the negative distance of each landmark to its nearest agent. If two agents collide, the team will get a penalty -1. In MSMTC, agents need to cooperatively adjust the sensing orientation to maximize the coverage rate of m moving targets. As is shown in Fig each sensor can only see the targets that within the radius and not blocked by any obstacle. The reward is the coverage rate of targets. If there is no target covered by sensors, we punish the team with a reward r = -0.1. For MSMTC, all targets are dynamic and each agent chooses multiple targets concurrently its sub-goals. Therefore, CN and MSMTC respectively represent two typical settings of ToMAC: static target & oonecchice sub-goal and dynamic targets & multi-choice sub-goal. More details can be found in Appendix. A
In the following, we compare ToM2C with baselines in the two scenarios. Then, in MSMTC, we conduct an ablation study, analyze the communication, and evaluate the scalability of ToM2C.
4.1 COMPARE WITH BASELINES
We compare our methods with 4 baselines. TarMAC (Das ctt al. 2019) is a multi-agent communication method that requires broadcast channel. I2C (Ding et al. (0200) pro- poses an individual communication mechanism, which is also achieved by ToM2C. HiT-MAC (Xu et al. (0200) is hierarchical method that uses a centralized coordinator to
organize the agents. MAPPO (Yu et al.) (022) is variant of PPO which is specialized for multi-agen -agent settings, run- ning without communication. we also implement global heuristic search algorithm in MSMTC. as a reference. This policy searches in one step for the primitive actions of all the sensors to minimize the sum of minimum angle distance of a target to a sensor. Note that the heuristic search policy exploits the global state to do the global search, while all the other methods are restricted to local observation. Therefore the heuristic search policy can serve as reference 'upper bound' that evaluates all the MARL basclines. Please refer to Appendix. C for more details.
Figure 3: An example of the target cov- erage environment with obstacles.
ToM2C is a hicrarchical method, while some of the basclines are not. In MSMTC, we let all of the methods share the lowelevel rule-based policy adopted in ToM2C. Therefore, these baselines are only used for training the high-level policy, same as ToM2C. In CN, ToM2C and HíT-MAC share the low-level policy and other methods keep non-hierarchical.
Quantitative Results. In CN, as Tab. I shows, ToM2C and HiT-MAC are far better than other methods. Here we also add MADDPG as a baseline method, because this environment is early adopted in the original paper of MADDPG (Lowe et al. 2017). The performance of HiT-MAC is
7

---

Published as a conference paper at ICLR 2022
80.0
80.0
70.0
70.0
Rewards
Rewards
ToM2C
80.0
50.0
ToM2C-ToM
HIT-MAC
A2C
50.0
40.0
MAPPO
TarMAG
40.0
30.0
0.0M
0.5M
1.0M
1.5M
2.0M
2.5M
3.0M
0.0M
0.5M
1.0M
1.5M
2.0M
2.5M
3.0M
of Iterations
# of Iterations
(a)
The learning curve of our method with baselines and reference policies in the MSMTC scenario. The learning-base methods are all trained in the environment with 4 sensors and 5 targets. (a) comparing ours with baselines; (b) comparing ours with its ablations.
Figure 4:
Table I: Quantitative Results in Cooperative Navigation (n 7).
ToM2C
TarMAC
12C
HiT-MAC
MADDPG
MAPPO
Reward
-0.79 + 0.39
-2.14 + 0.24 -1.59 + 0.40 -0.61 = 0.14 -2.75 + 0.61 -2.47 + 0.21
slightly better than ToM2C. It because the hierarchical structures of ToM2C and HiT-MAC are different. For HiT-MAC, there is centralized coordinator that collects all the local observations and assigns the goals for each agent. On the contrary, ToM2C agents run in fully decentralized manner. Agents only have access to local observations and decide their own goals. The hierarchy structure is only used for separate goal selections and primitive actions. Due the centralized manner, HiT-MAC is likely to degrade in more complex scenarios (e.g., large-scale MSMTC), Moreover, HiT-MAC is also more costly in communication (Sec. 4.3) and weaker in scalability (Sec. 4.4). Therefore, ToM2C not inferior to HiT-MAC.
In MSMTC, as Fig ((a) shows, ToM2C achieves the second highest reward (75) in the setting sensors and targets, only lower than the searching policy (80). The reward performance of I2C, MAPPO and HiT-MAC are all around 66. TarMAC reaches a reward of 71, which is superior to the other three baselines. This could be attributed to the broadcast channel adopted by TarMAC. Since all the methods leverage the hierarchy structure, the superiority of ToM2C comes from the model and training strategies.
4.2 ABLATION STUDY
We conduct this study to evaluate the contribution two key components our model: ToM net and Message sender. The ToM2C-Comm model abandons communication, so the actor makes decisions only based on local observation and inferred goals of others. The ToM2C-ToM abandons ToM net, but keeps the Messages sender. However, explained before, the local graph node feature is computed based on the ToM net output. To deal with this problem, we use the encoded observation Ej to replace the original node feature Mz In this way, the local graphs degrade into global graph, so the ToM2C-ToM model actually breaks the local communication mechanism. If we remove both the ToM net and Message sender, ToM2C will become pure A2C. We show in Fig 4(b) that we abandon one of the key components, the performance will drop. Considering that ToM2C-Comm outperforms ToM2C- ToM and ToM is actually essential for communication, we argue that ToM net mainly contributes to our method.
4.3 COMMUNICATION ANALYSIS
We compare method with several candidates in regard to communication expenses. There 2 metrics here: the number communication edges and communication bandwidth. The latter metric considers both the count of edges and the length of single message. TarMAC utilizes a broadcast channel, so there are two communication edges between each pair. The communi- cation in HiT-MAC is between the executors and the coordinator. ToM2C w/o CR refers to the ToM2C model without communication reduction. Apparently, communication bandwidth is more significant in real applications, so we place the figure of communication bandwidth the main text.
8

---

Published as a conference paper at ICLR 2022
The statistics of communication edges can found in Appendix. D As is shown in Fig 5 the communication bandwidth of ToM2C and ToM2C without CR much lower than Tar- MAC. I2C and HiT-MAC. It is be- cause in ToM2C the message is only the inferred goals, while TarMAC, I2C and HiT-MAC have to send the local observation. Therefore, the sin-
(b)
Figure 5: The communication bandwidth of models in (a)
Cooperative navigation and (b) MSMTC. gle message ToM2C much simpler than that of TarMAC, I2C and HiT-MAC. a result, the communication cost of ToM2C is extremely less than existing methods.
4.4 SCALABILITY
We evaluate the scalability of our method different number of sen- sors and targets in the MSMTC task. Note that the model is only trained in the setting of 4 sensors and 5 targets, so this could be regarded as zero-shot transfer. Since the difficulty of the task changes with the scale of cam- eras and targets, we make use of the heuristic search policy, whose target coverage rate in each setting can serve as referential value. We adopt such a metric that divides Repaa C. /CHS, where CLoM refers to the coverage rate of ToM2C in the setting of cameras and q targets and Chis refers to the coverage rate of heuristic search. In this way, we may reduce the fluctuation induced by the change of task difficulty. To better show the advantage of our method, we further test the scalability of HiT-MAC. The experiments are conducted where cameras and targets both range from 2 to 10. We compute R ToM and R RM (ratio of HiT-MAC and heuristic search) in all the settings and draw the heatmap in Fig. 6 that the ratio RTM stays near 1, which means that the performance of ToM2C is close to heuristic search in all the settings. Furthermore, if we compare heatmap of ToM2C and HAT-MAC, obvious that ToM2C is more stable than HiT-MAC when transferred different scales. In this way, we show that ToM2C has a rather strong generalization.
004
Figure 6: Analyzing the scalability in scenarios with different sizes of cameras and targets. The left heatmap shows R RooM The right heatmap shows RHM
coverage rate of the model by the coverage rate the search policy:
total of settings,
shown in the left figure
5 CONCLUSION AND DISCUSSION
In this work, we study the target-oriented multi-agent cooperation (ToMAC) problem. Inspired by the cognitive study in Theory of Mind (ToM), we propose an effective Target-orient Multi-agent Cooperation and Communication mechanism (ToM2C) for ToMAC. For each agent, ToM2C is composed of an observation encoder, ToM net, message sender, and decision maker. The ToM net designed for estimating the observation and inferring the goals (intentions) of others. It is also deeply used by the message sender and decision maker. Besides, a communication reduction method proposed further improve the efficiency of the communication. Empirical results demonstrated that our method can deal with challenging cases and outperform the state-of-the-art MARL methods.
Although impressive improvements have been achieved, there are still some limitations this work leaving for addressed by future works. 1) It is necessary to further evaluate the model on other applications. As each component the model general, we are confident to apply ToM2C to other target-oriented tasks (e.g. SMAC (Samvelyan et al.) 2019)) in the future. It is also interesting to extend ToM2C to non-target environments (e.g. Hanabi (Bard et al. 2020 which requires a further definition of sub-gone -oblo automatic goal generation. 2) Besides, the communication reduction method can also be further optimized, as the pseudo labels we generated for communication reduction noisy in some cases.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
The code is available at httpss//gihhub.com//nnall haw.aonnoo/naaaaaaacnnno//2222 The details of the environments be found in Appendix. A The hyper-parameters used in ToM22C are listed in Tab. 2
ACKNOWLEDGEMENTS
This work was supported by MOST-2016AAA0000000 ASFC-62061136000 China National Post- doctoral Program for Innovative Talents (Grant No. BX2021008), Qualcomm University Research Grant.
REFERENCES
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: new frontier for ai research. Artificial Intelligence, 280: 103216, 2020.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray kavukcuoglu. Interaction networks for learning about objects, relations and physics. In Proceedings of the 30th International Conference Neural Information Processing Systems, pp. 4509-4517, 2016.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo 7ilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27)4): 819-840, 2002.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in Neural Information Processing Systems, 32:5174-51855 2019.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1078, 2014.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning, pp. 1538 1546. PMLR, 2019.
gang Ding, Tiejun Huang, and Zongqing Lu. Learning individually inferred communication for multi-agent cooperation. In Advances in Neural Information Processing Systems, volume 33, pp. (0069-20079, 2020.
Evren Etel and Virginia Slaughter. Theory of mind and peer cooperation in two play contexts. Journal of Applied Developmental Psychology, 60:87-95, 2019.
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Pecroy-Seconn AAAI Conference on Artificial Intelligence, 2018
Kanishk Gandhi, Gala Stojnic, Brenden M Lake, and Moira R Dillon. Baby intuitions benchmark (bib): Discerning the goals, preferences, and actions of others. arXiv preprint 112. 2021.
Aditya Grover, Maruan AI-Shedivat, Jayesh Gupta, Yuri Burda, and Harrison Edwards. Learning policy representations in multiagent systems. In International Conference Machine Learning, pp. 1802-1811. PMLR, 2018.
He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep rein- forcement learning. In International Conference Machine Learning, pp. 1804-1813. PMLR, 2016.
Shariq Iqbal and Fei Sha. Actor-atteation-crite for multi-agent reinforcement learning. In Interna- fional Conference on Machine Learning, pp. 2961-2970. PMLR, 2019.
10

---

Published as a conference paper at ICLR 2022
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softmax. arXiv preprint 1r1.v:1611.01 144, 2016.
Dong-Ki Ki Kim, Miao Liu, Shayegan Omidshafiei, Sebastian Lopez-Cot, Matthew Riemer, Golnaz Habibi, Gerald Tesauro, Sami Mourad, Murray Campbell, and Jonathan How. Learning hierar- chical teaching in cooperative multiagent reinforcement learning. In AAMAS, 2020.
Jing Li, Jing Xu, Fangwei Zhong, Xiangyu Kong, Yu Qiao, and Yizhou Wang. Pose-assisted multi- the AAAI Conference on Artificial
camera collaboration for active object tracking. In Proceedings the Intelligence, volume 34, pp. 759-766, 2020a.
Yunzhu Li, Antonio Torralba, Anima Anandkumar, Dieter Fox, and Animesh Garg. Causal discovery in physical systems from videos. Advances in Neural Information Processing Systems, 33, 2020b.
Terence Lim, Sidney Tio, and Desmond C Ong. Improving multi-agent cooperation using theory of mind. arXiv preprint arXiv:2007. 157003 2020.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent conperttiieeoopeiiiiiie environments. In Advances in Neural Information
actor-critic for mixed Processing Systems, pp. 1379-6390) 2017.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712 2016.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International Conference on Machine Learning, pp. 1928-1937, 2016.
Aviv Netanyahu, Tianmin Shu, Boris Katz, Andrei Barbu, and Joshua B Tenenbaum. Phase: Physically-grounded abstract social events for machine social perception. arXiv preprint arXiv:2103.01933, 2021.
David Premack and Guy Woodruff. Does the chimpanzee have theory of mind? Behavioral and brain sciences, 1(4):515-526, 1978.
Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang. Yuan- Liao, Joshua B Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-help: A challenge for social perception and human-ai collaboration. In International conference learning representations, 2021.
Neil Rabinowitz, Frank Perbet, Francis Song. Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In International conference machine learning, pp. 2218-4227. PMLR, 2018.
Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforcement learning. International Conference on Machine Learning, pp. 4257-4266, PMLR, 2018.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. International Conference on Machine Learning, pp. 4295-4304. PMLR, 2018
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung. Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: / /00p.09020044333 2019.
Alan G Sanfey, Claudia Civai, and Peter Vavra. Predicting the other in cooperative interactions.
Trends in cognitive sciences, 19(7)3344-365, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X1v:1707.06347, 2017.
Itai Sher, Melissa Koenig, and Aldo Rustichini. Children's strategic theory mind. Proceedings of the National Academy of Sciences, 111677):1300--3312, 2014.

---

Published as a conference paper at ICLR 2022
Tianmin Shu and Yuandong Tian. M
Mind-aware multi-agent management reinforcement learnings. In International conference on learning representations, 2019.
Tianmin Shu, Abhishek Bhandwaldar, Chuang Gan, Kevin A Smith, Shari Liu, Dan Gutfreund, Elizabeth Spelke, Joshua B Tenenbaum, and Tomer Ullman. Agent: A benchmark for core psychological reasoning. arXiv preprint arXiv:2102. 12321, 2021.
Michael Shum, Max Kleiman-Weiner, Michael L Littman, and Joshua B Tenenbaum. Theory of minds: Understanding behavior in groups through inverse planning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 6163-6170, 2019.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Individualized controlled continuous munication model for multiagent cooperative and competitive tasks. In International conference on learning representations, 2019.
Sainbayar Sukhbaatar, arthur szlam, and Rob Fergus. Learning multiagent communication with backpropagation. 29, 2016.
Peter Sunehag. Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinícius Flores Zam- baldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel 7. Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning based on team reward. In AAMAS. pp. 2085-2087, 2018.
Michael Tomasello. A natural history of human thinking. Harvard University Press, 2014.
Socially Interactive Agents Track, Jan Pöppel, and Stefan Kopp. Satisficing models of bayesian theory of mind for explaining behavior of differently uncertain agents. In Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems, Stockholm, Sweden, pp.
10-15, 2018
Elio Tuci, Muhanad HM Alkilabi, and Otar Akanyeti. Cooperative object transport in multi-robot systems: A review of the state-of-the- art. Frontiers in Robotics and AI, 5:59, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:
2017.
Sarah A Wu, Rose E Wang, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Kleiman-Weiner. Too many cooks: Bayesian inference for coordinating multi-agent collaboration. Topics in Cognitive Science, 13(2):414 432, 2021.
Jing Xu, Fangwei Zhong, and Yizhou Wang. Learning multi-agent coordination for enhancing target coverage in directional sensor networks. In Advances in Neural Information Processing Systems, volume 33, pp. 10053- 10064, 2020.
Jiachen Yang, Igor Borovikov, and Hongyuan Zha. Hierarchical cooperative multi-agent reinforce- ment learning with skill discovery. In AAMAS, 2019.
Chao Yu. Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of mappo cooperative, multi-agent games. arXiv preprint arXiv:2103.01 2021.
Fangwei Zhong, Peng Sun, Wenhan Luo, Tingyun Yan, and Yizhou Wang. Ad-vat+: An asymmetric dueling mechanism for learning and understanding visual active tracking. IEEE Transactions
Pattern Analysis and Machine Intelligence, 43(5): -487-1828. 2021a.
Fangwei Zhong, Peng Sun, Wenhan Luo, Tingyun Yan, and Yizhou Wang. Towards distraction-robust active visual tracking. In International Conference on Machine Learning, pp. 12782-12792. PMLR, 2021b.
12

---

Published as a conference paper at ICLR 2022
A DETAILS OF ENVIRONMENT
A.1 MULTI-SENSOR TARGET COVERAGE
The multi-sensor multi-targel tracking environment is developed based on the previous version in HiT-MAC(XE (Xu et al., 2020), and it inherits most of the characters. It is 2D environment that simulates the real target coverage problem in directional sensor networks. Each sensor can only see the targets that are within the radius and blocked by any obstacle. There types of target: iitiinationnnavigatte
and random walking. The former one moves in the shortest path to reach a previously sampled destination. The latter one moves randomly each time step. At the beginning of each episode, the location of sensors, targets and obstacles are randomly sampled. Besides, the types of the targets are also sampled according to pre-defined probability. The length of an episode is 100 steps.
Observation Space. each time step, the local observation o; a set of agent-target pairs: (0.,1,..0i If target g is visible to agent i, then 0i,q (i, q, di,g aig)), where di,q is the distance and 0iiy is the relative angle. If target is not visible to i, then 0i,g = (0, 0, 0, 0). Therefore, Og E Rmmx4-
Action Space. The primitive action for a sensor is to stay or rotate +5/-5 degrees. For our method, the high-level action is the chosen goals gs. which is binary vector of length It.q 1 means the agent chooses target as one of its goals. Fi.g = means not. Although the low-level executor can be trained by reinforcement learning (RL) as HiT-MAC, we find simple rule-based policy can also work well in most cases. Therefore we only train the high-level policy.
Reward. Reward is the coverage rate of targets:
1 Σ, In where I, 1 if is covered by any m
sensor. If there is no target covered by sensors, punish the team with reward -0.1.
Compared with the version in HiT-MAC, we add some new features to make it more complex and realistic:
First, HiT-MAC assumes that the agents can see all targets the environment, which is unreasonable for real applications. Therefore, we change this setting into Dec-POMDP, in which an agent can only obtain the information in its local observation. For those targets that are out of view, the corresponding observation will be zero vector.
Secondly, we add another kind of objects, obstacles, into the environment. The obstacles all circles in this 2D plain simulator, varying in the radius. The targets within the observation radius will still be invisible if it is shadowed by an obstacle.
Finally, in the original environment all the targets move in goal-oriented manner. The targets sample their destinations at the beginming of an episode and navigate themselves to the destinations. Nevertheless, not all targets in real world follow the same action pattern. Therefore, we fill the environment with a mixed-type population of targets. The target can either be goal-oriented or random-walking, When the target random- waakinng. will randomly sample a primitive action to take at each step. In this way, the movement of targets is harder to predict, raising the difficulty of the planning.
Figure 7: Snapshots of the multi-sensor multi-target tracking environments with different scales. From the upper left figure, in the clockwise direction, they are 4 vs 3, 4 vs 7, 2 vs 5, and 6 vs 12, respectively.
13

---

Published as a conference paper at ICLR 2022
A.2 COOPERATIVE NAVIGATION
The environment is similar the used in MADDPG (Lowe al. 2017). agents need to cooperatively reach n. landmarks. In this task, we set n = 7. The length of each episode is 100 steps. For hierarchical methods, the high-level episode length is set K 10. Thus, each high-level step consists of 10 low-level steps.
Observation Space. At each time step, the observation og is the concatenation of self location, self velocity, location and velocity of visible agents, location of visible landmarks.
Action Space. The primitive action for an agent high level action is the chosen goal 9:- Similar to the multi-sensor target coverage task, ToM2C and HiT-MAC again make use rule-based low-level policy.
move up/dowwnleft/rigtt For our method, the
Reward. The team reward is the sum of the negative distance of each landmark to its nearest agent. two agents collide, the team will get penalty r 1.
B TOM2C DETAILS
Network Architecture and flyper-parameters for ToM2C. The observation encoder consists of 2-layer multilayer perceptron (MLP) and an attention module: att1. The ToM net consists of a Gated Recurrent Unit (GRU) and 2-layer MLP. The message sender is Graph Neural Network (GNN) and the actor consists of one fully connected layer. The critic consists of an attention module att, that can handle different numbers of agents. As mentioned before, the basic RL. training algorithm A2C, and the hyper--arameters are detailed in Tab.
Table 2: Hyper-parameters for ToM2C
Hyper-pane sapee--arrmeters
Description
GRU hidden units
32
the # of hidden units for GRU the of hidden units for att; the of hidden units for att; maximum environment steps sampled in workers
att; hidden units
64
atta hidden units
192
steps
3M
episode length
100
maximum time steps per episode discount factor for rewards parameter for entropy regularization learning rate for all networks the # of workers for sampling the network updates every # steps in A2C
discount factor
0.9
entropy weight
0.005
learning rate
1e-3
workers
6
update frequency
20
ToM Frozen
5
the ToM net is frozen for every times of RI. training the increasing rate of discounting factor y
gamma rate
0.002
Training Strategy. There are two training strategies adopted to accelerate training and stabilize the result. As mentioned in See S.c3.5 one is increase episode length L and y factor gradually during training, the other one is to split the optimization of the ToM and RL model.
In this paper, we propose this curriculum learning strategy that gradually increases episode length L and discounting factor Usually, the discounting factor is set larger than 0.9 to encourage long-term planning RL algorithms. Furthermore, the length of an episode usually determined by the environment. We notice that if using the default inefficient and unstable while learning. In our experiments, we set T. 20 and 7 = 0.1 initially. After 2000 episodes of warm-up, the factor will be updated according to pre-set rate 3. Each time the network optimized through reinforcement learning, 7 = 7 * (1 + 3). where 3 = 0,002 in this paper. Simultaneously. the episode length is updated with 7. In fact, = 7+0.11 x 20. In the end, 7 0.9 and L 100. By doing so, the agents learn short-term planning first, and then adapt to a longer horizon. We find in experiments that such strategy accelerates the training process, leading faster convergence and better performance.
the agents are sample
0.2
14

---

Published as a conference paper at ICLR 2022
Furthermore, separate the optimizations of the ToM and RL model in implementation. Before the training process starts, the parameters of our model are split into two parts: 8TMM and @ather Each part is optimized individually by different optimizer. Since we adopt A2C the base RL training algorithm, we collect trajectories data from different worker processes and send them to the training process when all the running episodes end. After that, gother is optimized with regard to the A2C loss. Meanwhile, the trajectories data for ToM training are saved instead of being used for training ToM net immediately. In this way, the ToM net is 'frozen'. OTohl will be optimized with regard to ToM loss after gother has been optimized for Tp times. Here we choose TF = 5. Just like the discussion before, the separation of ToM and RL training avoids the nested loop of influence among the ToM net and the policy network.
The environment and model are implemented in Python. The model is built on PyTorch and is trained on machine with 7 Nvidia GPUs (Titan Xp) and 72 Intel CPU Cores.
C BASELINES
Heuristic Search Method. To evaluate the performance of our ToM2C model, we choose to implement heuristic search policy to serve as reference. This search policy is applied to select low- level sensor action(Stay, Turn Left/Right.. At each step, the policy searches all the 3 possibilities of combination of actions, where is the number of sensors. The goal is to find the action combination that minimizes the angle distance of targets to sensors. Specifically, we denote the angle distance of target j to sensor as Qzj. Then the objective is minimize ΣØ= min,fas3 }- It is obvious that step, thus not the optimal policy. However, we show that this naive heuristic search can reach 80% target coverage. As a result, it can serve as a reference 'upper bound' that evaluates all the MARL baselines.
such searching policy only considers
The code HiT-MAC and I2C are from their official repositories. We follow the default seper-parametee in their code, except that change the learning rate, discounting factor y and episode length to be the same as ToM2C. TarMAC is implemented on our own because no official code is released. Moreover, HiT-MAC is hierarchical method, we simply train the high-level coordinator and use the same rule-based low-level policy utilized in ToM2C. On the other hand, I2C not hicrarchical method and it is not target-oriented. As result, we concatenate all the target information into one vector as the observation for 12C. The action space is modified as the set of choice of all the targets, so the space size is 2mh where m is the number of targets. In this way, the output action of I2C agent is the selection of goal targets, same as HiT-MAC and ToM2C. Once the goal target is selected, the primitive actions will be chosen by the rule-based policy.
MARL Baselines.
D QUANTITATIVE RESULTS
For MSMTC task, We list the coverage rate achieved by different methods in Tab. 3 The mean and standard deviation are computed based on the data collected 1000 episodes. The performance in cooperative navigation is listed in Tab. [
Table 3: Coverage Rate in 4 sensors vs targets scenario
Methods
Coverage Rate(%)t
A2C
38.44 + 0.54
HiT-MAC
61.48 = 1.45
12C
66.29 = 1.40
MAPPO
66.87 = 0.69
ToM2C-TOM
67.66 - 0.63
TarMAC
70.56 + 0.81
ToM2C-Comm
71.61 = 0.31
ToM2C(Ours)
75.38 + 0.57
Apart from the coverage rate, we analyze the communication efficiency of different methods. There are 2 metrics introduced in this paper. Communication edges refer to the count of directed communi-
15

---

Published as a conference paper at ICLR 2022
Table 4: Communication Statistics of MSMTC
Methods
Communication Edges
Communication Bandwidth.]
TarMAC
12.00 + 0.00
384.00 = 0.00
I2C
7.19 + 0.13
258.84 + 4.16
HiT-MAC
8.00 + 0.00
164.00 + 0.00
ToM2C w/o CR
9.39 + 0.19
46.93 + 0.97
ToM2C(Ours)
6.03 + 0.20
30.15 + 1.02
Table 5: Communication Statistics of CN
Methods
Communication Edges Communication Bandwidth.]
TarMAC
42.00 + 0.00
1344.00 + 0.00
I2C
14.21 + 1.65
511.41 + 59.44
HiT-MAC
14.00 + 0.00
231.00 + 0.00
ToM2C w/o CR
42.00 + 0.00
126.00 + 0.00
ToM2C(Ours)
22.48 + 0.91
67.44 + 2.74
cation pairs. One edge from to j means that agent i sends a message to agent j. Communication bandwidth refers the total volume of messages. As we mentioned in the experiment section, is the volume of messages that has decisive effect on the cost of communication. Since the messages are all float-type vectors, we use the length volume of a single message. For I2C, the message from agent is the local observation the information of all the targets. For HiT-MAC, communication happens between the executors and the coordinator. The executors send their local observation to the coordinator, and the coordinator returns the goal assignment. For ToM2C w/o CR and ToM2C, the message is simply the inferred goals of the receiver. ToM2C w/o CR means that the trained ToM2C model is not further optimized to reduce communication.
message instead of the number of bits to represent the
containing
The experiments conducted in the 4-sensors- and----r--an-----rreeee scenario and 7v7 cooperative navigation. As shown in Tab, 4 and Tab. []] our method achieves the lowest communication cost.
ToM Accuracy. We further tested the accuracy of ToM prediction, including goal inference and observation estimation. In MSMTC, the accuracy of goal inference 80.2% = 1.4%. The accuracy of observation estimation is 98. 18.1% + 0.3%. For comparison, the accuracy of random prediction 50.1% + 0.3% and 50.4% + 0.6% respectively.
Pose Accessibility. In this paper, enable the agents have access to the poses of all the other agents, while restricting the observability of targets. To further validate the performance of ToM2C in the of restricted pose access, we add observation distance between agents. If agent is out of the view of agent i, we mask the ToM inference and communication to agent from agent i. Under such partially observable agent poses, we test our model in MSMTC. The average rewards among 100 episodes are listed in Tab. 6, comparing with the case of using all agent poses. It is shown that the performance of ToM2C is still comparable to the original version (use all agent poses) when the poses of agents are partially observable.
Table 6: Comparison between partially and fully observable poses in MSMTC.
sensors Vs. targets
Partially observable agent poses
All agent poses
10 vs. 10
75.96 +: 0.81
77.26 + 0.52
10 vs. 5
85.06 = 0.32
84.69 0.77
4 vs. 5
74.73 = 0.95
75.38 + 0.57
16

---

Published as a conference paper at ICLR 2022
E DEMO SEQUENCE
To better understand the learned behavior, we render the target coverage environment and show typical demo sequence in Fig- 8 consists of consecutive keyframes one episode. The arrows between sensors indicate communication connections. Note that communication only happens every 10 steps. In step 16, sensor D can track target 1, and 4. However when comes to step 22, sensor D can no longer track all the three targets, so it starts to hesitate about which targets to track. Then step 24, sends message to D, and D inferred that A would track target and Therefore, re-plans its own goal to be target 4. In the end, we can see that sensor D really abandons target 1 and and focuses on target 4.
Step 16
Step 22
Step 21
Step 29
Figure 8: An exemplar sequence in 4 sensors and targets MSMTC environment. The gray circle indicates the obstacle. The arrows are rendered as solid only when the communication happens, and transparent at other times.
17


Reviews:
[{"review_id": "N03NH422AzR", "reviewer": "Reviewer_YARZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\nOverall model design to address the problem is interesting. In addition, performing two tasks (CN and MSMTC) for performance evaluation and choice of the baseline is sound. The ablation studies show how well the proposed model is structured. Code is provided to improve reproducibility.\n\nConcerns\n1.\tAs written in section 3, the proposed TOM2C receives a local partial observation. However, obtaining the current pose (guess that it is a ‘position’) of all agents does not seem to be a partial observation setting. More clarification should be provided about the partial observation. \n 2.\tThe authors claim that TOM2C can provide fully distributed execution. What about the learning process? Does TOM2C consist of centralized learning? \n3.\tThe paper does not state “who” provides rewards to all agents. In section 4.1, the authors ‘punish’ the team with a penalty, which implies the existence of a coordinator. Does TOM2C have a coordinator who gives rewards to all agents? Does it mean a centralized training framework? \n4.\tIn section 3.1, the global information is an input to the attention module and partial observation is an output. Is the attention model a part of the agent? If then, can we still say the partial observation? The input to the agent will be the global information. If the attention is a part of the environment, does the environment have an individual attention module for each agent? How can it be explained in a real-world scenario?  \n5.\tIn section 4, the simulation results include the case of 3 agents. What happens if the number of agents increases? Does the proposed solution work in any multiagent system? \n6.\tSection 4.1 describes the cooperative navigation task and multi-sensor target coverage task. However, the details of tasks are missing. How much do the targets move in a single step? It would be much convincing if results of other benchmarks are provided. For example, Hanabi could be a great candidate to evaluate the performances of the MARL algorithm.\n7.\tIn research on Theory of Mind, inference performance of ToM net is a significant factor. So, it would be beneficial to show the accuracy of the ToM net as an individual model, not as a part of ToM2C. In addition, it would be interesting to show the performance change of the entire ToM net according to the number of agents.\n", "summary_of_the_paper": "This paper proposes a new algorithm called TOM2C to solve the multi-agent reinforcement learning problem. To achieve goals in the MARL problem, communication between agents is important. However, it is often challenging due to scalability and communication costs. To solve this problem, the authors adopt the Theory of Mind to multi-agents. The agent infers the mental states and intentions of others upon partial observation. TOM2C has two kinds of agents: a planner that decides sub-goal and reaches a consensus, and a low-level executor that takes actions. The authors also provide a communication reduction method based on CTDE.", "summary_of_the_review": "The approach that combines MARL and Theory of Mind is interesting. The proposed TOM2C can be applied to multiple MARL problems which need cooperation between agents. But the proposed algorithms should be explained more in detail. Also, more experiments are needed to persuade the reviewers regarding its performance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ByljMoI51jV", "reviewer": "Reviewer_V4LB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I commend the authors for designing and implementing this highly complex agent architecture, which clearly exceeds the complexity of many previous multi-agent algorithms (having seen the supplementary material, I see the agent model is thousands of lines of code). Precisely because of the complexity, it is imperative that the technical writing is organized perfectly to ensure that readers appreciate the work and are not put off due to the complexity. In its present form, this paper hasn't met that standard. I believe the architecture is a valuable contribution to the advancement of agent design, especially for cooperative MARL, so I encourage the authors to take the following criticism in a positive way to improve the writing.\n\n1. Referring to the abstract, do agents really \"reach a consensus\"? By definition, a consensus is achieved when many agents all converge to the same decision or estimation of a quantity. This paper is concerned with multi-goal problems, so clearly agents don't all choose the same goal. If the authors intend to say that agent $i$'s prediction of agent $j$'s goal matches agent $j$'s actual chosen goal, then they authors need to verify this in their experiments (I don't see such a test) and describe it as so. Calling that \"consensus\" is confusing.\n2. Still in the abstract, authors say agents chooose \"sub-goals\". Does this mean an agent chooses many different sub-goals at various steps in an episode? How many primitive actions are taken per sub-goal? Is each sub-goals drawn from a predefined set of possible goals? Or is a sub-goal a latent variable whose meaning emerges via some kind of unsupervised learning? Or are there hand-designed rewards for each sub-goal? I have these questions even after multiple readings of the paper. Reference [1] below is an example of such details that one expects from hierarchical MARL methods.\n3. Page 3, related work section, the authors claim that CTDE methods are flawed becaue \"coooperation collapse in spite of a slight change in the team formation,...\" Which work has shown this? What is meant by \"change in team formation\"? Plenty of works have shown excellent generalization, especially since many multi-agent environments have localized interactions, so centralized training with few agents is sufficient for generalizing to decentralized settings with many more agents.\n4. Page 4, \"pose\" $(\\phi_1,\\dotsc, \\phi_n)$ is undefined. It also seems way too specific to particular applications.\n5. Page 4, \"The final communication connection is sampled according to the computed graph edge features.\" How are edge features defined? Even after reading to the end of the methods section, I still can't find the answer.\n6. Section 3.2, authors say \"Therefore, the entire ToM net of agent i is actually composed of n-1 separate ToM nets\", but then later say it's actually just a single model because agents are homogeneous and one model can be used to process all other agents' info. Be consistent.\n7. Each agent infers the intended goal of other agents. The accuracy of this prediction needs to be shown in an experiment.\n8. Page 4, last line, authors mention an \"auxiliary task\" for agent $i$ to estimate the observation of agent $j$. It turns out, two pages later in page 6, that this auxiliary task is that agent $i$ predicts whether agent $j$ can see a landmark. Authors should define that auxiliary task immediately after introducing it, not 2 pages later. Also, the accuracy of this prediction needs to be shown in an experiment.\n9. Page 5, in the paragraph \"connection choice\", what is an \"effect\", as in \"node effect\" and \"edge effect\"? This is not a standard term in the graph neural network literature.\n10. Page 6, in the \"Communication Reduction (CR)\" paragraph, there's a model that trims the edges of the communication graph, and it is \"supervised by the generated pseudo labels\". What exactly is this thing? What are the inputs and outputs? Is it a binary classifier that decides whether to cut or not cut an edge? This is completely unclear.\n11. Section 3.4, first paragraph, what do authors mean by an \"actor-critic feature\"? The observation input to the policy network?\n12. Section 3.4, first paragraph, authors say \"The actor decides its goals $g_i$ according to $\\eta_i$. How does actor \"decide $g_i$ according to $\\eta_i$\"? What is the reward for the goal-selector? What is the reward for the low-level executor? By goal $g_i$ and \"sub-goal\", do you mean the same thing? What is the reward corresponding to each sub-goal?\n\nOf course, some of these questions may be addressed by the paper somewhere, but the fact that I still have these questions after multiple readings shows that the paper's organization really needs improvement (not just minor local changes) and the responsibility isn't entirely on the reader's part.\n\nThere are multiple typos/grammar mistakes that the authors should fix.\n\"to decomposes the\", \"in complexer scenarios\"\n\n[1] Yang et al. \"Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery.\" AAMAS. 2020.", "summary_of_the_paper": "This paper designs a new agent with the ability to estimate the local observations and potential goals of other agents, conduct message passing of such local estimates with neighboring agents, then use the outcome of message passing in a two-level hierarchical policy to select its own goal and take primitive actions conditioned on that goal. Experiments and ablations were done in the cooperative navigation benchmark and a multi-sensor target coverage task, in comparison to previous multi-agent communciation baselines.", "summary_of_the_review": "Technical and experimental work is impressive but the organization of writing and lack of clarity in certain key parts makes this paper fall under the necessary standards.\n\nDuring rebuttal: authors have provided detailed answers to my questions and revised the manuscript to improve clarity.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BqIzW-Mpzg7", "reviewer": "Reviewer_jxPG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper was clearly written and easy to understand. The authors presented their motivations well. While using ToM inspired models has been explored in 2-agent tasks, integrating these models in a multi-agent setting to modulate communication was quite neat. Correspondingly, I found the ablation studies showing the utilities of ToM and the communication reduction to be very useful. I was particularly impressed by the model's generalization across different task settings and scales (different numbers of cameras and targets).  \n\nOne of the weaker points of the paper was the use of privileged information to learn the ToM module; the ToMnet uses supervision from ground truth observations from other agents and requires supervision from their goal states. Access to others' true observations and goal states is unrealistic. Further, this reduces the extent to which the model is decentralized during training (as observation and goal states are directly used for supervision). Finally, the ToM model is limited in its novelty, but its use to influence communication is novel.\n\nThe authors evaluate the approach on two synthetic tasks: cooperative navigation and multi-sensor target coverage. I would have liked the method to be evaluated in more complex settings (as the authors themselves note in the conclusion) like SMAC [1] or the Hanabi Challenge [2](that specifically test for decentralized cooperation). If the authors believe that their method is inapplicable to the domains, then I would have liked to see a discussion on how this approach could be adapted. \n\nOne of the other lacking aspects of the paper was a discussion of other hierarchical baselines for modelling agents. I would have liked to see a discussion on such approaches. For example, [5] use a similar hierarchical approach for cooperation in a two-agent setting.\n\nFinally, what do the authors think about approaches in IRL/ meta-IRL;  that directly try to predict the goals/ rewards of an agent?\n\nI have added a few relevant references that work with agency reasoning and cooperation; I believe they are related to theory of mind inferences and collaboration. The authors might include them if they think that they are relevant.\n\nAgency reasoning benchmarks: AGENT [3] BIB [4]\n\nCooperation benchmarks: Watch2Help [5] Overcooked [6] PHASE [7]\n\n[1] Samvelyan, M., Rashid, T., De Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G., ... & Whiteson, S. (2019). The starcraft multi-agent challenge. *arXiv preprint arXiv:1902.04043*.\n\n[2] Bard, N., Foerster, J. N., Chandar, S., Burch, N., Lanctot, M., Song, H. F., ... & Bowling, M. (2020). The Hanabi challenge: A new frontier for ai research. *Artificial Intelligence*, *280*, 103216.\n\n[3] Shu, T., Bhandwaldar, A., Gan, C., Smith, K. A., Liu, S., Gutfreund, D., ... & Ullman, T. D. (2021). AGENT: A Benchmark for Core Psychological Reasoning. *arXiv preprint arXiv:2102.12321*.\n\n[4] Gandhi, K., Stojnic, G., Lake, B. M., & Dillon, M. R. (2021). Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others. *arXiv preprint arXiv:2102.11938*.\n\n[5] Puig, X., Shu, T., Li, S., Wang, Z., Liao, Y. H., Tenenbaum, J. B., ... & Torralba, A. (2020). Watch-and-help: A challenge for social perception and human-AI collaboration. *arXiv preprint arXiv:2010.09890*.\n\n[6] Carroll, M., Shah, R., Ho, M. K., Griffiths, T., Seshia, S., Abbeel, P., & Dragan, A. (2019). On the utility of learning about humans for human-ai coordination. *Advances in Neural Information Processing Systems*, *32*, 5174-5185.\n\n[7] Netanyahu, A., Shu, T., Katz, B., Barbu, A., & Tenenbaum, J. B. (2021). PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception. *arXiv preprint arXiv:2103.01933*.", "summary_of_the_paper": "The paper presents a new method for communication and cooperation in multi-agent settings. The method relies on modelling other agents' intentions and internal states using Theory of Mind based neural nets. The predictions from the ToM model are used to decide how to communicate and coordinate with other agents. The authors test the method on two common multi-agent cooperation tasks to achieve SOTA communication efficiency and reward performance. The authors also show the utility of modelling mental states and using communication through ablation studies. Finally, the model shows flexibility in generalization, with consistent performance across different settings.", "summary_of_the_review": "The use of a TOM model to facilitate communication for cooperation was impressive. I believe the paper is slightly below the acceptance threshold in its current state due to a lack of tests on more complex benchmarks that test for decentralized cooperation (SMAC/ Hanabi) and limited applicability because of access to unrealistic privileged information during training.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The current paper presents a new method for communication and cooperation in multi-agent settings. Specifically, the authors propose to model other agents' intentions and internal states using ToM nets and using these predictions to then decide how to communicate/coordinate. The authors present experiments in two multi-agent cooperation tasks (multi-sensor multi-target coverage and cooperative navigation), compare against 4 previous methods (TarMAC, I2C, MAPPO and HiT-MAC) and perform the necessary ablations studies and find that their method achieve better rewards in both environments.
All reviewers have found the present study to be novel with convincing experimental findings. Reviewers have raised some concerns however a great deal of those have been addressed by the authors during the rebuttal and many of these points have now been incorporated in the paper. 

Having read the paper and considering the reviews I agree with the reviewers that this manuscript will make a good addition to the program of ICLR and as such I recommend its acceptance.
Decision: Accept (Poster)