ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind
Authors: Yuanfei Wang
fangwei zhong
Jing Xu
Yizhou Wang
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Theory of Mind
Target-oriented Multi-Agent Cooperation
Multi-agent Communication

Abstract: Being able to predict the mental states of others is a key factor to effective social interaction. It is also crucial to distributed multi-agent systems, where agents are required to communicate and cooperate with others. In this paper, we introduce such an important social-cognitive skill, i.e. Theory of Mind (ToM), to build socially intelligent agents who are able to communicate and cooperate effectively to accomplish challenging tasks. With ToM, each agent is able to infer the mental states and intentions of others according to its (local) observation. Based on the inferred states, the agents decide when and with whom to share their intentions. With the information observed, inferred, and received, the agents decide their sub-goals and reach a consensus among the team. In the end, the low-level executors independently take primitive actions according to the sub-goals. We demonstrate the idea in a typical target-oriented multi-agent task, namely multi-sensor target coverage problems. The experiments show that the proposed model not only outperforms the state-of-the-art methods in sample efficiency and target coverage rate but also has good generalization across different scales of the environment.

ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind
Anonymous Author(s)
Affiliation Address email
Abstract
Being able to predict the mental states of others is key factor to effective social interaction. It is also crucial to distributed multi-agent systems, where agents are required to communicate and cooperate with others. In this paper, we introduce such an important social-cognitive skill, i.e. Theory of Mind (ToM), to build socially intelligent agents who are able communicate and cooperate effectively to accomplish challenging tasks. With ToM, each agent is able to infer the mental states and intentions of others according to its (local) observation. Based on the inferred states, the agents decide "when" and with "whom" to share their intentions. With the information observed, inferred, and received, the agents decide their sub- goals and reach a consensus among the team. In the end, the low-level executors independently take primitive actions according to the sub-goals We demonstrate the idea in typical target-oriented multi-agent task, namely multi-sensor target coverage problems. The experiments show that the proposed model not only outperforms the state-of-the-art methods in sample efficiency and target coverage rate but also has good generalization across different scales the environment.
2
10
11
13
15
Introduction
16
Cooperation is key component of human society, which enables people to divide labor and achieve common goals that no individual can reach on his/her own. In particular, human are able to form an ad-hoc team with partners and communicate cooperatively with one another [I] Cognitive show that the ability to model others' mental states (intentions, beliefs, and desires), called Theory of Mind (ToM) [5] is important for such social interaction. Considering a simple real-world scenario (Fig. D. where three people (Alice, Bob, and Carol) are required to take the fruits (apple, orange, and pear) with shortest path. To achieve it, the individual will take four steps sequentially: 1) observing their surrounding: 2) Inferring the observation and intention of others; 3) communicate with others to share the local observation intention if necessary; 4) making decision and taking action to get the chosen fruits without conflict. In this process, the ToM is naturally adopted in inferring others (Step 2) and also guides the communication among agents (Step 3).
18
19
20
studies 2 3
21
22
23
24
26
27
28 Motivated by this, machine learning researchers have takes efforts on developing the machine ToM 6 or modeling opponents 77] for multi-agent learning [8] (0). But most of the existing computing models are only used in toy environments, where are only a few agents (two or three) performing
29
30
31 simple tasks. It is still challenging to implement such thinking mechanism for social agents,
32 especially in cases of many agents. That because the mental state of one agent will be impacted by
33 many other agents, leading to the accuracy and efficiency of the ToM drop.
34 In this paper, we study the Target-oriented Multi-Agent Cooperation problem (ToMAC). In ToMAC.
35 the agents need to cooperatively reach and keep specific relations among the agents and targets. Such
36 problem setting widely exists real-world applications, e.g. collecting multiple objects (Fig. [
37
navigating to multiple landmarks [1], monitoring a group of pedestrians [2] When running,
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

An apple here!
Bob spole
Alice can't see the upple but Carol apple haave
An apole hone!
An crange and
Bob in clsse ha
Step Observing
Step Inferring
FB caloe ROVAA
Carol wi take eppll so
apple pogours
pelr
Bob will take pest,
Bola you cin calo the pour bahind the wall
Step Communicating
Step Decision making
Figure I: A fruits collection example. The agents are required to cooperatively collect the three target objects (apple, pear, and orange) in the room as fast as possible. The whole process can be divided into 4 steps. In the first step, 3 agents observes the environment and obtains the state of the visible targets. In the second step, each agent tries to infer what other agents have seen, and which targets they shall choose as goals. In the third step, each agent decides whom to communicate with according to the previous inference. In the fourth step, each agent decides its own goal of target based on what it observed, inferred, and received.
38 each agent is required choose a subset of interesting targets and reaching them to contribute 39 to the team goal. In this case, the key to realizing high-quality cooperation is how to reach consensus among agents to avoid the inner conflict the team. However, the existing multi-agent reinforcement learning methods still do not handle it well, as they only implicitly model others in the state representation and are inefficient in communication.
40
41
42
43
Here we propose Target-oriented Multi-agent Communication and Cooperation mechanism (ToM2CC using the Theory of Mind, shown as Fig. 2 In ToM2C, each agent of two-level hierarchy. The high level policy (planner) needs to cooperatively choose certain interesting targets as a sub-goal to deal with, such as tracking certain moving objects or navigating to a specific landmark. Then low level policy (executor) takes primitive actions to reach the selected goals for k steps. To be more specific, each agent receives local observation of targets, and estimate the local observation of others in the ToM Net. Combining the observed and inferred state, the ToM net will predict/infer the target choices (intentions) of other agents. After that, each agent decide 'whom' to communicate with according to local observation filtered by the inferred goals and the estimated observation of others. The message is rather simple and comprchensible, which is only the predicted goals of the message receiver, inferred by the sender. In the end, all the agents decide its own goals by leveraging the observed, inferred, and received information. Thanks to the inferring and sharing of intentions, the agents easily reach consensus to cooperatively adjust the target-agent relations to the expected.
44
45
46
47
48
50
51
52
54
Furthermore, we also introduce communication reduction method remove the redundant message passing among agents. Take the advantage of the centralized training decentralized execution (CTDE) 58 paradigm, we measure the effect of the received messages on each agents, by comparing the output of the planner with and without messages. Hence, we can figure out the unnecessary connection among agents. Then we train the connection choice network to cut these dispensable channels in supervised manner. Eventually, ToM2C systemically solves the problem of 'when', 'who" and 'what' in multi-agent communication, providing a compact, efficient and interpretable communication protocol.
56
59
60
61
82
64
The experiments are conducted in a challenging multi-sensoo multi-targe murgi-target covering scenario. The team goal of sensors is to adjust their orientation to cover as many targets as possible. It is shown that our method achieves the highest coverage ratio among several state-of-the-art MARL methods [3. 12) in the case of 4 sensors and 5 targets. Moreover, we also demonstrate the strong scalability of ToM2C
85
86
68 in different populations of sensors and targets. We further take an ablation study to evaluate the contribution of each key component of our model.

---

2
Related Work
70
71
Multi-agent Cooperation. The cooperation of multiple agents is crucial yet challenging in dis- tributed systems. Agents' policies continue to shift during training. leading to non-stationary en- vironment and difficulty in model convergence. Recent work [1] 14
72
17 in multi agent 74 reinforcement learning (MARL) mainly adopts centralized training decentralized execution (CTDE)
73
15
16
75 paradigm mitigate non-stationarity. However, such training method only implicitly guides agents
76 to adapt to certain policy patterns of others. As a result, cooperation collapses even if there is only a slight change in the team formation, making the model extremely impractical and poor of scalability. Furthermore, some existing work tries to make use of communication to promote cooperation, such
77
78
79 as [8] 19 20]. Unfortunately, they all require a broadcast communication channel that pose huge
80 pressure on bandwidth. Besides, even though I2C [33 proposes a individual communication method,
81 the message just the encoding of observation, which not only costly but also uninterpretable. Compared with existing methods, ToM2C does not only apply ToM to explicitly model intentions and mental states but also to improve the efficiency of communication to further promote cooperation. For the target-oriented multi-agent cooperation, HiT-MAC [12] propose hierarchical multi-agent coordination framework to decomposes the target coverage problem into two-level tasks: assigning targets by centralized coordinator and tracking assigned targets by decentralized executors. The agents in ToM2C are also of a two-level hierarchy. Differently, thanks to the use of ToM, both levels enabled to perform distributedly.
82
83
84
85
86
87
However, 90 how to apply the discover in cognitive science to building cooperative multi-agent systems still
Theory of Mind. Theory of Mind is long-studied conception in cognitive science 89
91 remains a challenge. Most previous work make use of Theory of Mind to interpret agent behaviours,
92 but fail to take a step forward to enhance cooperation. For example, Machine Theory of Mind [6]
93 proposes a meta-learning method to learn a ToMnet that predicts the behaviours or characteristies of
94 single agent. Besides, [21] studies how to apply Bayesian inference to understand the behaviours of group and predict the group structure. [22] introduces the concept of Satisficing Theory of Mind, which means the sufficing and satisfying model of others. None of these work looks into the problem
95
96
97
of multi-agent cooperation. [10] considers a 2-player scenario and employs Bayesian Theory of
Mind to promote collaboration. Nevertheless, the task is too simple and it requires the model of other agents to do the inference. On the other hand, opponent modeling 7 g g is another kind of methods
99
100
comparable with Theory of Mind. Agents endowed with opponent modeling can explicitly represent the model of others, and therefore plan with awareness of current status of others. Nevertheless, these methods rely on the access the observation of others, which means they are not truly decentralized paradigms.
101
102
103
3 Methods
104
105
In this section, we will explain how to build a target-oriented social agent to realize efficient multi- agent communication and cooperation. We formulate the target-oriented cooperative task as a Dec-POMDP (23) The aim of all agents is to maximize the team reward. Furthermore, agents are allowed to communicate with each other to enhance cooperation. The overall network architecture is shown in Fig. Firom the perspective of agent i. The model is mainly composed of four functional networks: Observation encoder, ToM net, Communication choice net, and actor-critic net. To be specific. it receives a local partial observation oG, which includes the information of visible targets.
106
107
108
109
110
112
What's more, it obtains the current pose (ø1,... On) of all the agents, where n is the number of agents. The raw observation will be encoded into F, by an iitention-based encoder. Then the agent starts to do Theory of Mind inference with the ToM net. It first estimates the observation representation c of each other agents according to their poses. Cj can be used for inferring the current visible targets of agent j. which is an auxiliary task that will be discussed later. Based on C, and E,, agent infers the probability of agent j choosing these targets its goals, denoted as Gi,y-- After that, agent i decides whom to communicate with. We employ graph neural network here as the communication choice net. The node feature of agent j is the concatenation of ej and E, filtered by Gi The final communication connection is sampled according to computed graph edge features. Agent i will send Gity to agent> if there exists a communication edge from to j. Finally, G;, E, and received messages is concatenated as 7, for planner(actor) and critic. Planner !! (gi lo:) is the high level policy
113
114
116
117
118
119
120
121
122
that chooses the goals gi, which guides the low-level executor =f (a, (0s, 9i) to perform primitive actions.
123
3

---

Message Sender
T Theory of Mind Net
Decision Maker
Wes (E E (G))er((,.EE)
City Pa-
Hii Goal Inference
Critic -
C Connection Choice
TeM
ToM
ToN
Planner
Oseevvatio Estimation
gi
Σ
Pa. F.
F..
Executor
0
E2
PrimitiveActiion d
O bservation Encoder
Target
D;
Concatenate
The architecture of ToM2C for each individual. There are 4 key components: Observation encoder, Theory of Mind net, Message sender and Decision maker.
Figure 2:
125
In the following sections, we will illustrate the key components of ToM2C in details.
126
3.1
Observation Encoder
127 We employ the attention module 24 to encode the local observation. There are two prominent advantages of this module. On one hand, crucial for scalability. On the other hand, global information can be encoded into single feature due to the weighted sum mechanism. In this paper, we use scaled dot-product self-attention similar to 12). is the number of targets. The input is the local observation o; E Im xdoi, and the output E. E Bmxdue where oi.q and Er., represent the raw and encoded feature of target y to agent 132
128
appulation-ivvarian and order-invariant, which is
129
130
133
respectively.
134
3.2
Theory of Mind Network (ToM Net)
Inspired by the Machine Theory of Mind (6), introduce ToM net that enables agents infer the observation and intentions of others. Most previous work [7] [0] consider two-player scenarios, where the agent only needs to model one other agent. Instead, we take step forward to evaluate model in a more complex multi-agent scenario consisting of n(>3) agents. Therefore, the entire ToM of agent is actually composed of n-1 separate ToM nets, each utilized model the corresponding agent. The single ToM net is made up of two functional modules: Observation Estimation and Goal Inference. The overall ToM net takes the poses of agents and local observation input. Then outputs the inferred observation representation and goals of others.
136
138
139
140
141
142
Observation Estimation. The first step of ToM inference is to estimate the observation representation of the other agent. The term refers to the visibility of the environment. Intuitively, when an agent tries to infer the intention of others, it should first infer which targets seen by them. Take Bob in fig. as an example. Before he tries to infer the goals of Alice and Carol, he first infers that Alice cannot observe the apple but Carol can. Similarly, Agent infers the observation of agent j. denoted
144
146
147
148
as j, with the pose Φj. Note that Ej is only a representation of the observation. To better learn this representation, we introduce an auxiliary task here. Agent i needs to infer which targets are in the observation field of agent j, based on this representation ej and local observation E,, In practice, we employ GRU to model the observation of others on time series.
149
150
151
152
Goal Inference. After agent finishes the observation estimation of others, it is able predict which targets will be chosen by them at this step. Just like human, the agent infers the intentions of others based on what it sees and what it thinks that others see. If we denote this goal inference network as a function GI, then the process can be formulated as :Gi GI ((E.g C)). Gi,j. stands for the probability of agent choosing target q, inferred by i. Since there are a total of n agents and m targets
153
154
155
156
in the environment, G. e (In--)xm-

---

158
With ToM net, each agent holds a belief on the observation and goal intentions of others. Such belief not only taken into account for final self decision, but also serves as a indispensable component in communication choice. The details will be discussed in the next section.
159
160
161
3.3
Message Sender
162
Learning to communicate has been studied in a number of multi-agent reinforcement learning works. However, most of them either require a public communication channel or a centralized mechanism to decide the communication connection, which is definitely unrealistic for real aaglti systems. Moreover, the message is usually an encoded feature, making it both uninterpretable and lengthy.
169
184
185
166
Instead, introduce message sender by leveraging the information inferred by ToM net. Each agent decides 'when' and with 'whom' to communicate completely on its own. And the message is the inferred ToM goals of the receiver. To achieve this, use graph neural network similar to [6] The details is in the next paragraph. After the model is trained, we further propose a communication reduction method remove useless connections and improve the efficiency of the communication network.
167
168
169
170
171
172
As stated before, use Graph Neural Network (GNN) to learn the connection in an end-to-end manner. In previous works [27]. there is only one global graph that collects all the observation node features. Such implementation breaks the individuality. Instead, we propose
Inferred-goal Filter.
173
175
method to make use of the inferred state and intention to generate local graphs. Specifically, in the
176
perspective of agent the feature of agent j is the target features filtered by the inferred goals G² as
follows. 5 is a probability threshold, if G > 8, then agent considers it as the goal that will be
177
178
chosen by agent j.
m
E
(G. > 6) Eigg
179
Then we concatenate the filtered feature Di,5 with the estimated observation representation ej, to form the estimated node feature u, Cy). For agent i itself, u,,i = CE Esg. 6)), where e; is also computed by Observation Estimation module with the pose of i.
180
181
Connection Choice. For scenario consisting of agents, there is total of n directed graphs G = (G1,92 G2)... G; = (V,,&) is the local graph for agent to compute the communication connection from agent i. The vertices V, (f(u..)), where is a node feature encoder. Edges & (u.,, u,,k)) ). where a is an edge feature encoder. Like the Interaction Networks (IN) (26), propagate the node and edge features spatially to obtain node and edge effect. For convenience, we will describe only graph G; in the following formula and omit the index i. Let Vj be the encoded
183
184
185
187
188
node feature of j. and hy be the node effect. Similarly, let & be the encoded edge feature, hJ,& be the edge effect. Initially, h2 = VG, h1g = Ej,k- Then the graph iterates for several times to propagate
189
190
the effect:
h, = modee (Vjs h1g Σ ha.s)
(2)
&
hg,k wedge (hzj bi, hz3k)
(3)
192
In the end, we obtain edge (Ei,, ha.)), and compute the probabilistic distribution over the type of the edge (cut or retain). Here we apply the Gumbel-Softman trick (28) 29] to sample the discrete edge type, so the gradients can be back-propagated in end-to-end training. Considering that it is the local communication graph of agent i, only the types of E._,, are sampled. edge Eú, is retained, agent i
193
195
196
will send the inferred goals of to it.
197
Communication Reduction (CR). The communication choice network learns in an end-to-end man- ner. If no regularization is applied here, the network tends to learn a relatively dense communication connection graph. However, some of these connections are actually redundant. In fact, some receivers choose the same goals with and without these messages. Therefore, we can figure out the necessity of certain communication edges. Formally, we estimate the effect of the received messages to agent
198
199
200
201
DKi(9; I[9:). Note that Di denotes the probability distribution over the goals of agent i when all the messages sent to i are masked. If x < T, we regard that the messages are redundant to agent i. Thus the edges pointing at i will be 'cut'. Otherwise (x > T), we 'retain' all the edges to agent i. Here 7 is constant, regarded
i by measuring the KL-divergence between 9i and 9, referred as X
204
5

---

208 as a threshold. Supervised by the generated pseudo labels, the model learns to cut the redundant
207 connections easily, leading to a more efficient communication network.
3.4 Decision Making
Once the agent receives all the messages, it can decide its own goals of targets based on its observation, inferred goals of others and received messages. Therefore, the actor-critic feature Y/ (E,, maxe Ci Σ. G Cr The second term refers to the max inferred probability of an target be chosen by another agent. The third term refers to the sum of the messages from others, indicating how much certain others infer that agent i should choose the target. The actor decides its goals 9: according to 7):- The centralized critic obtain global feature (?)....))) to compute value. The low level executor
x (aglos
209
takes primitive action accomplish the sub-goal. Although this executor also be trained by reinforcement learning (RL) as [12. we find a simple rule-based policy can also work well in most cases. this way, other methods without a hierarchical structure only need to learn the high-level
210
211
212
policy, so we can compare them with our method fairly.
213
3.5
Training
The model can be divided ToM and other parts. ToM net is trained in supervised learning with the true state of others. Other parts are trained by reinforcement learning (RL). We adopt standard
215
216
A2C [30] as the RL training algorithm, while any MARL method with CTDE framework is also applicable, such as PPO
217
31 32
Learning ToM Net. We introduce two classification tasks for learning the ToM Net, which is parameterized by ToM First, the ToM net infers the goals G of others. Note that D indicates the probability of agent j choosing target g. inferred by i. Mcanwhile, agent j decides its real goals gj- Therefore, 9j can be the label of Ki The Goal Inference loss is the binary cross entropy loss of this classification task:
220
222
G
ΣΣΣΣ J
Σ
1ssn -log(11g)) + (i.g) log(1 - 9
(4)
N
223
Secondly, the estimated observation representation c is trained in the auxiliary task mentioned before.
224
The agent: infers which targets are the observation j. denoted as cf,y- The ground truth
real observation field Cj. Cj.g 1 indicates that agent observes target q. Similar to the previous
226 Goal Inference task, this Observation Estimation learning also adopts binary cross entropy loss:
LO ΣΣΣ
(os log(cl..o) + (1-d) logg1 G.
(5)
joti
L(0TbM) = I + LO
(6)
227
Training Strategy. We find that it is hard for an agent to learn long-term planning from scratch. Therefore, set the initialize episode length L and discount factor y to low value, forcing agents to learn short-term planning first. During training, the episode length and discount factor y increase gradually, leading the agents to estimate the value a longer horizon.
229
231
Furthermore, we freeze the ToM net while the other parts of the model is updated through RL. The reason is that the ToM net infers the goals of others, and the policy network is continuously updated during RL training. Meanwhile, the output of ToM net is part of the input to policy network. If we train them simultaneously, they are likely to influence each other in nest loop. Therefore, we only collect the ToM inferred data into batch during RI. training. Once the batch is large enough, we
232
233
235
stop RL and start ToM training minimize ToM loss in Eq.

---

4
Experiments
237
238
We evaluate ToM2C in the challenging multi-sensor target coverage problem. Sensors need to cooperate with others to reach a maximum target coverage rate. We compare our method with 3 state-of-the-ar MARL methods: I2C [T3, HÎT-MAC [2]. A2C [30,, and reference greedy search
239
240
241
policy. We also conduct an ablation study to validate the contribution of ToM net and message sender. Finally, we show that our model can generalize to different size of agents and targets.
242
243
4.1
Environment
244
The environment is modified based on the one used in HiT-MAC [12]. and it inherits most of the characters. As is shown in Fig it is a 2D en- vironment that simulates the real target coverage
245
246
Seew
247
problem in directional sensor networks. Each sensor can only see the targets the sector, if not blocked by any obstacle. There 2 types of target: destination -itinatton-naviaation and random walk- ing. The former one moves in the shortest path to reach a previously sampled destination. The latter one moves randomly at each time step. At the beginning of each episode, the location of sensors, targets and obstacles are randomly sam- pled. Besides, the targets type is also sampled according to a pre-defined probability.
249
251
253
255
Figure 3: An example of the target coverage envi- ronment with obstacles.
257
258
259
each time step, the local observation or a set of agent-targer pairs: (o, (o.,1....i.ml If target g is visible to agent i, then 0t,q (f, q, di.g: (gg)) where di, is the distance and as,y is the relative angle. If target is not visible to i, then 0ū = (0, 0, 0). Therefore,
Observation Space.
261
O E 12x
263
Action Space. The primitive action for a sensor is to stay or rotate +5/-5 degrees. For our method, the high level action is the chosen goals 9:, which is binary vector of length m. Ji.g = 1 means the means not. Although the low-level executor can be trained by reinforcement learning (RL) as [12. we find a simple rule-based policy can also work well in most cases. Therefore we only train the high-level policy. this way, other methods without hierarchical structure are comparable with our method.
264
265
agent chooses target q as one of its goals. Fi.g
257
Reward. Reward is the coverage rate of targets: r 1 E, Lq- where Lg 1 if q is covered by any m sensor. If there is no target covered by sensors, punish the team with reward -0.1.
269
4.2
Baselines
272
We compare our methods with 4 baselines. HiT-MAC [12] is a hierarchical method that uses a coordinator to enhance cooperation. I2C [13] proposes individual communication mechanism, which is also achieved by ToM2C. A2C [30 is a standard reinforcement learning algorithm. Here we employ A2C to train single agent that selects the goals for all the sensors. Finally, we implement heuristic search algorithm as a reference policy. This policy searches in one step for the primitive actions of all the sensors to minimize the sum of minimum angle distance of target to sensor.
274
276
4.3
Results
279 As fig 4(a) shows, ToM2C achieves the second highest reward (75) in the setting of 4 sensors and targets, only lower than the searching policy (80). The vanilla A2C shows similar performance to random policy, indicating that the task is not trivial. The reward performance of HiT-MAC is around
281
62. lower than the result presented in the original paper. This could be attributed to the addition of obstacles. 12C reaches a fair reward of 66, but we will show that such performance still lower than
283
284
our ablation models.
285
Ablation Study. We conduct this study to evaluate the key components of our model: ToM net and
Message sender. The ToM2C-Comm model abandons communication, the actor makes decisions
7

---

80.0
80.0
60.0
Rewards
Rewards
40.0
40.0
A2C
20.0
HIT-MAC
TcABC
0.0
Hourisoo Search
ToM77C-TMM
Random
A2C
0.0M
0.5M
1.0M
1. .MM
2.0M
2 2.5M
3.0M
0.0M
0.9M
1.0M
1.5M
2.0M
2.5M
3.0M
# Iterations
of Iterations
(a)
(b)
Figure The learning curve of our method with baselines and reference policies. The learning- -base methods are all trained in environment with sensors and 5 targets. (a) comparing ours with baselines; (b) comparing ours with its ablations.
# comiiunication edges
Communication bandwidth
199
12C
HIT-MAC
ToW2C
ToM2C w/c CR
FC
120
HTMAC
TokeC
ToM22 we CR
(a)
(b)
Figure 5: Communication performance analysis of our method compared with other algorithms. (a)
comparison of the communication edges numbers; (b) comparison of the communication bandwidth.
only based on local observation and inferred goals of others. The ToM2C-ToM. abandons ToM net, but keeps Messages sender. However, as explained before, the local graph node feature is computed based the ToM net output. To deal with this problem, we use the encoded observation Eş to replace original node feature Wi - this way, the local graphs degrades into one global graph, the ToM2C-ToM model actually breaks the local communication mechanism. We show in fig 4(b) that if we abandon one of key components, the performance will drop. Specifically, ToM2C-Comm reaches 72, and ToM2C-TOM reaches 68, both higher than I2C. Considering that ToM2C-Comn outperforms ToM2C- ToM and ToM net is actually essential for communication, we argue that ToM mainly contributes to method.
286
290
292
294
296
Communication Analysis. We compare our method with several candidates in regard of communi- cation expense. There are 2 metrics here: the number of communication edges and communication bandwidth. The latter metric considers both the count of edges and the length of a single message. There are candidates here. FC refers to fully connected communication in ToM2C. ToM2C w/o CR refers to the ToM2C model without communication reduction. The communication in HiT-MAC between the exccutors and the coordinator. As is shown in fig 5(a) ToM2C performs the least communication regard of edge count, but this doesn't fully demonstrate the advantage of ToM2C over other methods. In fig (bb) the communication bandwidth of ToM2C, ToM2C without CR, and because in ToM2C the message only the inferred goals, while I2C and HiT-MAC have send the local observation. Therefore, the single
296
300
302
304
even FC much lower than 12C and HiT-MAC.
306
message in ToM2C is much simpler than that 12C and HiT-MAC. As result, the communication cost of ToM2C is extremely less than existing methods.
308
Scalability. We evaluate the scalability our method to different number of sensors and targets.
Note that the model is only trained in the setting of 4 sensors and 5 targets, so this could be regarded

---

80
80
Coverase Rate (K)
Rata
70
70
80
80
50
80
ToM2C
ToNZC
Heuriste Search
40
40
5
2
targets
(a)
(b)
Figure 6: Analyzing the scalability our method in scenarios with different sizes of sensors and largets. (a) n= 4, m from to 7: (b) m= 5, n is from 2 to 6
310 as zero-shot transfer. In 6(a) the number of sensors is fixed to 4, and in 6(b) the number of targets fixed 5. We also report the result of heuristic search because it not learnt policy and has good generalization in different settings. It is clear in the figures that the variation of coverage rate ToM2C follows the trend of heuristic search when the difficulty of the setting changes. this way, we show that ToM2C has rather stable generalization among different sizes of sensors and targets.
311
313
Conclusion and Discussion
315
In this work, study the target-oriented multi-agent cooperation (ToMAC) problem. Inspired by the cognitive study in Theory of Mind (ToM), we propose an effective Target-orient Multi-agent Cooperation and Communication mechanism (ToM2C) for ToMAC. For each agent, ToM2C is composed an observation encoder, ToM net, message sender, and decision- maker.. The ToM net is designed for estimating the observation and inferring the goals (intentions) of others. It is also deeply used by the message sender and decision-ne makiin... Besides, an communication reduction method is proposed further improve the efficiency of the communication. Empirical results demonstrated
317
319
321
323
that our method can deal with challenging scenes and outperform the state-of-the-art MARI. methods (12C, HiT-MAC).
325
Although impressive improvements have achieved, there is still a number limitations this work leaving for addressed by future works. 1) The model is only evaluated in simulated scenario. But the environment we used contains most features that other applications, e.g. partial observation, team reward structure. And each component in the model is general. So
327
confident to extend ToM2C in other application scenarios, c.g. cooperative searching in the future. 2) Besides, the
329
communication reeducation method can also be further optimized, as the pseudo labels generated for communication reduction are noisy in some cases.
331
Broader Impact
332
The target-oriented multi-agent cooperation problem widely exists in lot of real-world applications. So a great number of robot tasks will benefit from our work, e.g. cooperatively searching for disaster victims, cleaning trash, scene reconstruction, actively capturing sports videos. They all will make
334
335
336
our life more convenient and better. The use of ToM in multi-agent cooperation and communication will also promote the intersection of multi-agent systems and cognitive science, making them mutual benefit. But there is also the potential of being misused in the military field, e.g. using directional controlling unmanned vehicles to attacks targets. If our method fails, some targets in the corner would be neglected by the agents.
338
339
radars monitor missiles/aircraft
340
References
341
[1] M. Tomasello, A natural history of human thinking. Harvard University Press, 2014.

---

343
[2] I. Sher, M. Koenig, and A. Rustichini, "Children's is strategic theory of mind," Proceedings of the National Academy of Sciences, vol. 111, no. 37, pp. 13307-13312, 2014.
2014.
345 [3] A. G. Sanfey, C. Civai, and P. Vavra, "Predicting the other in cooperative interactions," Trends in cognitive sciences, vol. 19, no. 7, pp. 364-365, 2015.
346
347 [4] E. Etel and V. Slaughter, "Theory of mind and peer cooperation in two play contexts," Journal Applied Developmental Psychology. vol. 60, pp. 87-95, 2019.
348
349
[5] D. Premack and G. Woodruff. "Does the chimpanzee have a theory of mind?." Behavioral and brain sciences, vol. 1, no. 4. pp. 515-526, 1978.
350
351
[6] N. Rabinowitz, F. Perbet, F. Song, Zhang, S. A. Eslami, and M. Botvinick, "Machine theory of mind," in International conference on machine learning. pp. 421844218444 4227, PMLR, 2018.
352
Boyd-Graber, [7] H. He, J. Boyd Graber, K. Kwok, and H. Daumé III, "Opponent modeling in deep reinforcement learning."
354
in International Conference on Machine Learning, pp. 1804-1813, PMLR, 2016.
355
[8] R. Raileanu, E. Denton, A. Szlam, and R. Fergus, "Modeling others using oneself in multi-agent reinforce- ment learning," " in International Conference on Machine Learning, pp. 4257-4266, PMLR, 2018.
357
191 A. Grover, M. Al-Shedivat, J. Gupta, Y. Burda, and H. Edwards, "Learning policy representations in
multiagent systems," in International Conference on Machine Learning, pp. 1802-1811 I, PMLR, 2018.
359
[10] T. X. Lim, S. Tio, and D. C. Ong, "Improving multi-agent cooperation using theory of mind," arXiv preprint arXiv:2007.15703 2020.
380
361
[11] R. Lowe, Y. Wu, A. Tamar, J. Harb, O.P. Abbeel, and Mordatch, "Multi-agent actor-critic for mixed
362
cooperatie-competiiiv environments," in Advances in Neural Information Processing Systems, pp. 6379- 6390, 2017.
363
364
[12] J. Xu, F. Zhong, and Y. Wang. "Learning multi-agent coordination for enhancing target coverage in directional sensor networks," in Advances in Neural Information Processing Systems, vol. 33, pp. 10053- 10064, 2020.
385
367
[13] g. Ding, T. Huang, and Z. Lu, "Learning individually inferred communication for aueti- cooperation." in Advances in Neural Information Processing Systems, vol. 33, pp. 22069-2207777 2020.
388
369
[14] J. N. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and Whiteson, "Counterfactual multi-agent policy gradients." in Shirty-Second AAAI Conference on Artificial Intelligence, 2018.
370
371
[15] P. Sunchag, G. Lever, A. Gruslys, W. M. Czarnecki, V. F. Zambaldi, M. Jaderberg, M. Lanetot, N. Sonnerat,
372
J. Z. Leibo, K. Tuyls, et al., decoppsdcoooooittioon networks for cooperative multi-agent learning based on team reward.," " in AAMAS, pp. 2085-2087, 2018.
373
374
[16] T. Rashid, M. Samvelyan. C. Schroeder, G. Farquhar, J. Foerster, and S. Whitesoo, "Qmix: Monotonic value function factorisation for deep multi-agen reinforcement learning in International Conference on Machine Learning, pp. -295-4304 PMLR. 2018
375
378
377
[17] S. Iqbal and F. Sha, "Actor-attenioon-re
for multi-agent reinforcement learning." in International Conference on Machine Learning, pp. 2961-2970, PMLR, 2019.
378
379
[18] S. Sukhbaatar, a. szlam, and R. Fergus, "Learning multiagent communication with backpropagation." vol. 29, 2016.
380
[19] A. Das, T. Gervet, J. Romoff, D. Batra, D. Parikh, M. Rabbat, and Pineau, "Tarmac: Targeted multi-agent communication." " International Conference on Machine Learning, pp. 1538 1546, PMLR, 2019.
382
[20] A. Singh, T. Jain, and S. Sukhbaatar, "Individualized controlled continuous communication model for multiagent cooperative and competitive tasks," in International conference on learning representations, 2019.
385
[21] M. Shum, M. Kleiman- Weineer M. L. Littman, and J. B. Tenenbaum, "Theory of minds: behavior in groups through inverse planning." " in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 6163-6170, 2019.
Understanding
388
[22] S. S.L.A A. Track, J. Pöppel, and S. Kopp, "Satisficing models of bayesian theory of mind for explaining behav- ior of differently uncertain agents." in Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems, Stockholm, Sweden, pp. 10-15, 2018
390
391
[23] D. S. Bernstein, R. Givan, N. Immerman, and S. Zilberstein, "The complexity of decentralized control of markov decision processes," Mathematics of operations research. vol. 27, no. 4, pp. 819-840, 2002.
393
384
[24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," arXiv preprint
2017.
396
[25] Y. Li, A. Torralba, A. Anandkumar, D. Fox, and A. Garg, "Causal discovery in physical systems from videos." Advances in Neural Information Processing Systems, vol. 33, 2020.
10

---

398
[26] P. Battaglia, R. Pascanu, M. Lai, D. Rezende, and K. kavukcuoglu, "Interaction networks for learning about objects, relations and physics." " in Proceedings the 30th International Conference on Neural Information Procexsing Systems. pp. 4509-4517, 2016.
400
401
[27] J. Jiang, C. Dun, T. Huang, and Z. Lu, "Graph convolutional reinforcement learning," in International Conference on Learning Representations, 2019.
403
[28] E. Jang, S. Gu, and B. Poole. "Categorical reparameterization with gumbel-softmax." arXiv preprint
404
arXiv: 16Xiv:66110011 1.01 144, 2016.
405
[29] C. J. Maddison, Mnih, and W. Tch, "The conerete distribution: continuous relaxation of discrete random variables," arXiv preprint arXiv:1611.000 2016.
406
407
[30] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, "Asyn-
chronous methods for deep reinforcement learning." in International Conference on Machine Learning,
409
pp. 1928-1937, 2016.
[31] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint 177.v:1707.06347, 2017.
412
[32] C. Yu, A. Velu, E. Vinitsky, Y. Wang. A. Bayen, and Y. Wu, "The surprising effectiveness of mappo in cooperative, multi-agent games," arXiv preprint arXi:: 2021.
413
Checklist
415
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. . You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
417
419
Did you include the license to the code and datasets? [Yes] See Section ??.
420
Did you include the license the code and datasets? [No] The code and the data are proprietary.
422
Did you include the license to the code and datasets? [N/A]
423
Please do not modify the questions and only use the provided macros for your answers. Note that the
424
Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
425
426
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
428
429
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethies review guidelines and ensured that your paper conforms to them? [Yes]
432
433
2. If you are including theoretical results...
434
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
435
(b) Did you include complete proofs of all theoretical results? [N/A]
436
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (cither in the supplemental material or as a URL)? [Yes] See supplemen- tal material
438
440
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See supplemental material.
441
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
443
444
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplemental material.
4. If you using existing assets (c.g., code, data, models) or curating/releasing new assets...
11

---

447
(a) If your work uses existing assets, did you cite the creators? [Yes] The environments
448
are based on [[2]. and the baselines are from the official repository [2]
448
(b) Did you mention the license of the assets? [N/A]
450
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] In the supplemental material.
452
(d) Did you discuss whether and how consent was obtained from people whose data you' using/curating? [No] The environment we used are from open source repository. According to the license, allowed to modify and use the environments for research
454
purposes.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No] The experiments are conducted on a simulator.
457
458
5. If you used crowdsourcing or conducted research with human subjects...
459
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
480
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
462
463
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "dAirgpHvp2V", "reviewer": "Reviewer_2f7u", "summary": "This paper proposes a multi-agent communication and cooperation framework, ToM2C, where agents need to decide who and when to communicate with, and how to allocate subtasks (specifically, subtasks are defined by physical locations that agents need to navigate to / cover). The key idea is to train the agents to infer others' observations and goals, which may in turn helps achieve more effective and efficient communication/cooperation. The method adopts ToMnet-like architecture for a ToM module, and uses GNNs to fuse information and inference from multiple channels. Compared to the previous GNN architectures for multi-agent policies, it proposes two new designs, inferred-goal filter and communication reduction, which are enabled by the goal inference. The evaluation was done in a 2D continuous environment simulating a target coverage task. Compared to recent approaches, HiT-MAC and I2C, the proposed method reaches a higher reward with slightly fewer messages in this single task.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The work is well motivated. Indeed, ToM is a key ability for human cooperation and communication, which may also help machine agents' interactions.\n\n+ The method shows improvement over 2 recent baselines, in terms of rewards and communication costs, which is encouraging.\n\nWeaknesses:\n- It is unclear to me how ToM2C as a problem setup differs from TarMAC [19] and I2C [13]. If it is conceptually the same setup, where an agent will send messages directed to individual agents, then I do not see how this is a different framework. \n\n- Training ToMnet module requires other agents' ground-truth observations (what entities they can see) and goals.\n\n- This is not the first paper that introduces ToM (goal inference, belief inference) to multi-agent RL, e.g., the suggested references [1,2,3].\n\n- Many design choices such as how goal is represented are restricted to target navigation / coverage type of tasks. Generalizability seems quite limited compared to existing methods (e.g., I2C). This also raises the question -- how much benefit we see in the experiment comes from these domain knowledge (decomposing the task into specific subtasks defined by the target locations) and additional GT supervision that are not available in existing methods. The proposed method does seem to enjoy an unfair leg up when considering existing methods need to train a policy from scratch without knowing what subtasks are.\n\n- Related to the above point, what if we do not have manually defined subtasks?\n\n- Given manually defined subtasks, the suggested reference [4] proposed a Bayesian approach for ToM-based cooperation, which is related and should be discussed in the paper.\n\n- The evaluation was done in only one experiment. It is fine if the scope of the work is to solve the target coverage problem, but this intended scope should be clarified. Otherwise, it is unclear how this method could be applied to other tasks and environments, since there are domain-specific knowledge and supervision used for model design and training.\n\n- I do not see the number of runs conducted to produce the results. I would also expect to see variance in Figure 5 & 6.\n\nMinor comments:\n- fig.x -> Fig. x\n\nSuggested references:\n\n[1] Mordatch & Abbeel, Emergence of Grounded Compositional Language in Multi-Agent Populations, AAAI 2018.\n\n[2] Qi & Zhu, Intent-Aware Multi-Agent Reinforcement Learning, ICRA 2018.\n\n[3] Shu & Tian, M^3RL: Mind-aware Multi-agent Management Reinforcement Learning. ICLR 2019.\n\n[4] Wu et al., Too many cooks: Bayesian inference for coordinating multi-agent collaboration. CogSci 2020.\n\nPost Rebuttal:\nI would raise my rating to 5. I think a major revision is necessary so the submission is not ready in its current form.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would like to see more discussions on the limitations of the proposed method and how it is grounded in prior literature.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5 hours"}, "extra_text": {}}, {"review_id": "W0tGmrp7bDH", "reviewer": "Reviewer_RvLm", "summary": "* The paper proposes a Target-oriented Multi-agent Communication and Cooperation mechanism (ToM2C) using the Theory of Mind. \n\n* It uses an observation encoder, ToM net, message sender, and decision-maker. The ToM net is designed for estimating the observation and inferring the goals (intentions). The message sender uses a graph neural network using inferred state and intention to generate local graphs. Once the agent receives the messages, it decides its goals using the inferred goals of others and received messages using the actor-critic. \n\n* The empirical results of the model on the multi-sensor multi-target covering scenario is presented and compared with 4 different baselines from state-of-the-art MARL methods - HiT-MAC, I2C, A2C  and Heuristic Search algorithm. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n* There is lot of research around Theory of Mind such as Machine Theory of Mind, multi-agent cooperation using theory of mind and ToM Net for two-player scenarios where an agent model another agent to infer its goal and preferences. The famework presented in this paper provides the modeling for multi-agent scenario consisting of multiple agents in a team where each agent is required to choose a subset of interesting targets and reaching them to contribute to the team goal.\n\n* The communication reduction method used by the agents in the team that uses centralized training decentralized execution is interesting and makes the contributions novel. The authors train the connection choice network which provides the communication protocol for ’when’, ’who’ and ’what’ in multi-agent communication. \n\n\nQuality\n* The scenarios used in the multi-sensor multi-target tracking environment makes the environment much more realistic for a goal-oriented setting especially as agent can only obtain the information from its local observation.\n\n* The training strategies 1) curriculum learning strategy to increase episode length L and γ factor during the training and 2) to split the optimization of the ToM and RL model to avoid the nested loop of influence among  the ToM net and the policy network is clearly defined and implemented in the code attached to the paper. The environment and model are implemented in Python.The model is built on PyTorch and was trained on a machine with 7 Nvidia GPUs (Titan Xp) and 72 Intel CPU Cores.\n\n* The literature review is thorough and helps in understanding the current state of the art to find the missing gaps. \n\n\nClarity\n\nOverall the paper is well organized and clearly written. The implementation details are clear. However, there are elements of the paper which do not provide enough information, such as:\n* In order to obtain edge and compute the probabilistic distribution over the type of the  edge the authors propose to use the Interaction Networks (IN) by propagating the node and edge features spatially. However, the details about the marshalling function, m, that computes the matrix with interaction terms is missing? Additionally, which were the learnable parameters used in the objective function?  \n\n* To sample the discrete edge type in the local communication graph for the gradients to be back-propagated for training, the Gumbel-Softmax trick, is used in the message sender module. The authors need to provide as to what was the softmax temperature, tao, used in the model? It is needed because softmax temperature is a learned parameter as it can be interpreted as entropy regularization where the Gumbel-Softmax distribution can adaptively adjust the confidence of proposed samples during the training process.\n\n* In the Inferred-goal Filter, the authors propose to generate local graphs using inferred state and intentions for agent i. The filtered feature is concatenated with the estimated observation representation to form the estimated node feature (equation 1) to form the estimated node feature. However the probability threshold for the inferred goal is not defined and needs to be explained.  Furthermore, for the agent i, in the observation estimation, how would the observation be made by i,  if the other agents are not facing(pose) towards the target? \n\n\nSignificance\n* The submission provides the empirical results and compares them against the baselines from the existing MARL architectures:HiT-MAC, I2C, A2C  and Heuristic Search algorithm. They also provide the framework for modeling in multi-agent scenario consisting of n(>3) agents in a team where each agent is required to choose a subset of interesting targets and reaching them to contribute to the team goal.\n\n* The pytorch implementation for the environment and the model is clear. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "* The authors present the limitations of the research by describing that their model was evaluated in a simulated scenarios only which can be extended to real world environments for the robotic agents. \n* It is acknowledged by the authors that their framework generated pseudo labels for communication reduction which can be optimized in future work. \n", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "24"}, "extra_text": {}}, {"review_id": "UZ2gKmbfsCx", "reviewer": "Reviewer_koJ6", "summary": "The paper develops an approach to MARL that is inspired by theory of mind from psychology.\nThe authors claim that in problems like multi-sensor coverage that a better outcome (in terms of sensor coverage of\ntargets can be achieved) with such an explicit model that allows agents to communicate their \"intentions\".\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality\n- The authors claim they introduce theory of mind (ToM) to multi-agent systems.\n  There are a number of problems with this claim. \n\n- First, the theory of mind is a theory from psychology that makes the assumptions that other\n  have mental attitudes (such as beliefs, desires and intentions) such as our own, and by making\n  this assumption we can better coordinate/communicate with other humans/agents.\n\n- So the statement that the paper incorporates the theory of mind into a multi-agent system is not quite accurate\n  and is perhaps misleading at worst. \n  A more accurate statement would be that that the authors have incorporated ideas from the psychological theory of mind\n  into a multi-agent system/architecture; or that they have developed a model of agent interaction that is **inspired** by\n  the theory of mind from psychology. A such, the name of the model should be something else to distinguish it from the\n  psychological theory. \n\n- Second, there many other multi-agent systems (see the literature on multi-agent systems, BDI agents and multi-agent\n  team work) that incorporate modelling other agent's beliefs, desires and intentions and in the case of team work\n  modelling joint-beliefs, joint-goals and joint intentions. \n\n- In any case the problem of having multiple sensors covering multiple targets is a well known problem that has had\n  various approaches thrown at it. It has been addressed in both MARL and non-MARL (i.e. broader agent papers for a\n  number of years). So it is isn't an original problem. Many agent systems have looked modelling the mental attitudes\n  of team-mate agents. So, it isn't clear what the novel contribution is in this paper?\n  Is it the combination MARL with ideas inspired by theory of mind? Explicitly stating this in the abstract/intro and\n  conclusion would be a good idea.\n\n- In a number of places you mention that agents share their \"intentions\". What is actually meant by this in this paper?\n  Are agents sharing their policies?\n\n- Note again that there is a body of work from the multi-agent systems community (specifically the field of BDI agents -\n  that is agents based on the beliefs, desires and intentions model of agent reasoning) that have a very well defined\n  meaning of intentionality. See works by Wooldridge, Roa/Georgeff on BDI agents for example. \n\n## Quality\n- Overall the paper is well structured, the paper reads well and there are informative diagrams.\n- As mentioned above, the omission of broader references from work in multi-agent systems community on modelling shared\n  mental attitudes is an issue but otherwise the related work section seems ok.\n- The method section seems ok. However, it would seem to me that the proposed solution seems overly complex for the\n  examples used in the experiments. I have a suspicion that a simpler approach could have garnered similar results for\n  the simple domains shown. What would be of more interest would be a domain where it is obvious that a could outcome\n  could not be achieved without explicit cooperation/communication.\n- So a possible critique would be, could the problems be solved witout a ToM-Net; or why do you need a neural network\n  implementation that is inspired by ToM?\n- The experiments section is good. The explanation of the environment is good, with explicit descriptions of the\n  reward, action spaces and observation space. \n- While the baseline comparison is good, I think a better comparison would be your own architecture with and without a\n  model inspired by theory of mind. \n- Should there be error bars on the bar charts in Figure 5?\n- Also good to see the results shown in Figure 6 where the number of targets and the number of sensors is increased.\n  A couple of questions here:\n  1) It seems the heuristic search outperforms the ToM2C model; I was wondering if this was the case (as per my previous\n  comment) have you over engineered the architecture where a simpler model achieves a better result. What is the\n  advantage or contribution of your approach then?\n  2) Should there be error bars in Figure 6 (how many cases were run)?\n  3) The numbers of targets and sensors is still within the same order of magnitude. A more impressive scalability\n     metric would be if you scaled the number of sensors and targets by orders of magnitude (e.g. 10, 100, 1000 etc)\n     (or perhaps 5, 10, 50, 100, 500, 1000). You could then plot number of sensors vs number of targets as a 2D plot\n     and have the % coverage rate as a heatmap. That would be interesting to see. \n- It was good to see Section 5 that includes a Conclusion and Discussion and even a broader impact section. \n  My recommendation for these sections would be to reorganise them slightly. Rename Section 5 Discussion and \n  rename \"Broader Impact\" to Conclusions and cover the following:\n- 5. Discussion (discussion as is).\n- 6. Conclusion (include a small discussion on limitations, future work and broader impact). \n\n\n## Clarity\n- There are some parts of the paper where sentences read somewhat awkwardly, but nothing major in my opinion.\n- I found the paper fairly clearly to read, but like all papers can also benefit through another edit just to make sure\n  everything is clear for the reader. \n\n## Significance\n- Multi-sensor target coverage is well studied program across many different AI literatures.\n- I'm not sure the results here are significantly better than heuristic search or some of the other baselines\n  to conclusively state that the approach taken will make a significant impact on the field.\n- Note, a variation on my previous critique - this isn't to say that incorporating models inspired by the theory of mind is a\n  bad idea in multi-agent systems. On the contrary, I and many others believe that it is a good idea.\n  However, a neural network might not be the best way to do it. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- A more in depth discussion of the limitations of the approach would be desirable. \n- I think the fruits collection example from Figure 1 actually detracts from the paper.\n- In my opinion if the target coverage environment described in Section 4.1 was used \n  from the start of the paper, it would be a more compelling and \"real-world\" example.\n- The fruit example doesn't really add anything to the paper in my opinion.  \n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {"ethical_concerns": "- The authors finish their paper discussing potential misuse of the approach in military applications.\n  This is not necessarily a misuse as military uses can be used to defend populations as well.\n  Its worth thinking more broadly in terms of societal impact - these multi-agent UAV sensor coverage problems have many\n  potential useful applications ranging from disaster management/rescue, to sports (think about motor sports or cycling\n  or even many sports that will use UAV camers in the Olympic games), to law enforcement applications etc. "}}, {"review_id": "FcQDpnfzvX7", "reviewer": "Reviewer_ADoL", "summary": "This paper addresses the problem of reasoning about the mental states of other agents in multiagent reinforcement learning using Theory of Mind, focused on cooperative MAS domains such as multi-sensor target tracking.  A deep learning architecture is proposed that combines four networks to create a centralized solution: (1) one network that uses attention to encode observations for communication messages, (2) another than estimates the observations and goals of the agents in the environment (generalizing prior work to more than two cooperating agents), (3) a third that learns a graph determining who should communicate with whom, and when communication should happen, and (4) an actor-critic network for determining the policies of agents.  Experimental results varying the number of sensors and targets in the environment demonstrate the performance of the proposed solution against baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "===Originality===\n\nThe problems of multi-sensor target tracking in particular, and cooperative multiagent reinforcement learning are popular, important problems in the MAS and reinforcement learning communities.  The solution is an incremental combination of different types of neural networks to produce a new solution that has some differences from existing work.  I didn't fully understand the novelty of the solution as I was reading the paper.  The generalization to environments with more than two agents is important (albeit straightforward), and prior work seems to focus individually individually on the networks that make up the components of the proposed solution, so I think the proposed solution is more holistic than prior work, but I am not certain.\n\n===Quality===\n\nThe related work seemed mostly complete, although I was left wondering why ToMnet wasn't chosen as a baseline, given the similarities with the focus of this work.  I also wasn't sure why A2C was chosen as the underlying algorithm and baseline, when PPO is typically used instead.\n\nAnother question I had about the model while reading was in Eq. 1, why is \\delta used?  Wouldn't argmax G^* make more sense?  What happens if 2+ goals have G^* higher than \\delta?  What happens if 0 goals have G^* higher than delta?\n\n===Clarity===\n\nOverall, the paper was relatively easy to follow, and I think I understood most of the details (I do have some questions below in the Significance section of this review).  Some clarifications would be helpful, such as more explicitly explaining the novelty of the solution -- what is unique vs. what is simply a combination of known techniques?  Also, a running illustrative example in Section 3 would help the reader understand what is the terms refer to and what is being learned.  Since you only have one test domain, it would be a great choice here.\n\n===Significance===\n\nMy greatest uncertainty after reading the paper is about the significance of its contributions to the literature.  As mentioned above, I'm not exactly sure what the novelty in the work is.  Is it just a new holistic deep architecture?  Are some of the parts meaningful advances over the state-of-the-art beyond considering more than 2 agents?  To me, it appears to be a combination of known techniques with some incremental advances.\n\nAlso, only one domain was considered, so it is difficult to assess how the benefits of the approach might generalize.  Is it only useful in target-tracking domains?  Could it be applied to other cooperative MAS, which are a hot topic in deep MARL right now?  This is especially complicated by the fact that the domain appears to be relatively simple to solve since (1) simple rule-based policies were appropriate instead of needing reinforcement learning to learn low-level policies, and (2) a heuristic search algorithm outperformed all of the reinforcement learning solutions.  In real-world applications, we choose RL-based solutions when learning is necessary, possibly because the system needs to adapt to the dynamic environment, or when insufficient information is available a priori to know how to operate.  In this case, RL wasn't necessary, so it's difficult to know if the proposed approach outperformed the others because it achieved superior learning, or if it exploited some of the same reasons that heuristic search did well without learning.  So I'm left wondering if I'm misunderstanding something -- did heuristic search have access to information that wasn't available to the RL methods so that it is an appropriate upper bound on performance?  More details would help clarify here, which changes how the significance of the work might be viewed by the reader.\n\nI was also unsure exactly how the scalability experiments were conducted.  You state that \"Note that the model is only trained in the setting of 4 sensors and 5 targets, so this could be regarded as a zero-shot transfer\".  How then did you conduct the experiments whose results are presented in Figure 6b?  If you have a different number of sensors than 4, wouldn't you need a different Theory of Mind Network, since it has components for each of the n agents in the environment?  Wouldn't this also affect the other components of the network since the communication graphs would be different?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The broader impacts were discussed in the paper, although also addressing limitations of the research would strengthen the overall work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall there is not enough support from reviews for me to recommend acceptance.

During the discussion, two reviewers bumped their respective scores by a point, but the overall scores remain in the borderline range.

Reviewers agreed that there were clear strengths to the paper, and that author responses cleared up some (but not all) issues. The main remaining concerns with the submission are
1. discussion of limitations, especially around the ToM modeled and how it compares to ideas in the literature,
2. providing experiment details/extensions discussed in the review responses, and
3. that making these revisions (and others suggested in the discussion) would constitute a major revision (and thus perhaps require another set of reviews).

On #1:
* Reviewer koJ6 wrote "the concepts of theory of mind [...] are treated somewhat casually and some examination (and citing) of the literature in this area would improve the paper".
* Reviewer 2f7u agreed "the ToM modeled by this work is quite rudimentary" and the fix would be "a good statement of limitation and a reframing of the contributions".
* Reviewer 2f7u also wrote "I would like to see more discussions on the limitations of the proposed method and how it is grounded in prior literature", and "the additional discussions on prior work and the experimental results provided in the authors' responses are valuable".

On #2, there were new results and many experiment details provided in the authors' rebuttal which should be added to the submission text, as reviewers suggested and agreed by the authors.

On #3, Reviewer 2f7u summarized "the additional experimental results are very important for people to really understand how this work is different from the prior work, what were the settings for the experiments, and how general this approach is (i.e., what kinds of problems could benefit from it). There should be a major revision to address these points, both in literature review and in experiments." I agree that including the content covered in the discussion period would require significant revisions to the paper.

Additionally:
* Reviewer ADoL reamined concerned about the treatment of scalability experiments, though these concerns were not highlighted by other reviewers.
* The main concerns of Reviewer RvLm, around learnable parameters and observation estimation, were addressed, and that score was raised to a 7 (the highest among the reviewers).

These issues must be weighed against the significance of the contributions in the paper, which reviewers found to be valuable but limited:
* Reviewer koJ6: "I'm not sure the results here are significantly better than heuristic search or some of the other baselines to conclusively state that the approach taken will make a significant impact on the field"
* Reviewer ADoL: "My greatest uncertainty after reading the paper is about the significance of its contributions to the literature. [...] Also, only one domain was considered, so it is difficult to assess how the benefits of the approach might generalize"
* Reviewer 2f7u: "The evaluation was done in only one experiment"


To summarize, reviewers agreed that the underlying work has strengths and that the paper is close to acceptance, but that to push the manuscript to be a clear acceptance would require substantial revisions.

Because substantial revisions are required, this submission doesn't quite meet the bar given the other strong submissions.
Decision: Reject