ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind

1 INTRODUCTION . Cooperation is a key component of human society , which enables people to divide labor and achieve common goals that could not be accomplished independently . In particular , humans are able to form an ad-hoc team with partners and communicate cooperatively with one another ( Tomasello , 2014 ) . Cognitive studies ( Sher et al. , 2014 ; Sanfey et al. , 2015 ; Etel & Slaughter , 2019 ) show that the ability to model others ’ mental states ( intentions , beliefs , and desires ) , called Theory of Mind ( ToM ) ( Premack & Woodruff , 1978 ) , is important for such social interaction . Consider a simple real-world scenario ( Fig . 1 ) , where three people ( Alice , Bob and Carol ) are required to take the fruits ( apple , orange and pear ) following the shortest path . To achieve it , the individual should take four steps sequentially : 1 ) observe their surrounding ; 2 ) infer the observation and intention of others ; 3 ) communicate with others to share the local observation or intention if necessary ; 4 ) make a decision and take action to get the chosen fruits without conflict . In this process , the ToM is naturally adopted in inferring others ( Step 2 ) and also guides the communication ( Step 3 ) . In this paper , we focus on the Target-oriented Multi-Agent Cooperation ( ToMAC ) problem , where agents need to cooperatively adjust the relations among the agents and targets to reach the expectation , e.g. , covering all the targets . Such problem setting widely exists in real-world applications , e.g. , collecting multiple objects ( Fig . 1 ) , navigating to multiple landmarks ( Lowe et al. , 2017 ) , transporting objects ( Tuci et al. , 2018 ) , and monitoring a group of pedestrians ( Xu et al. , 2020 ) . While running , the distributed agents are required to concurrently choose a subset of interesting targets and optimize the relation to them to contribute to the team goal . In this case , the key to realizing high-quality cooperation is to reach a consensus among agents to avoid the inner conflict in the team . However , the existing multi-agent reinforcement learning methods still do not handle it well , as they only implicitly model others in the state representation and are inefficient in communication . Here we propose a Target-oriented Multi-agent Communication and Cooperation mechanism ( ToM2C ) using Theory of Mind . Shown as Fig . 2 , each agent is of a two-level hierarchy . The high-level policy ( planner ) needs to cooperatively choose certain interesting targets as a sub-goal to deal with , such as tracking certain moving objects or navigating to a specific landmark . Then low-level policy ( executor ) takes primitive actions to reach the selected goals for k steps . To be more specific , each agent receives local observation of targets , and estimates the local observation of others in the ToM Net . Combining the observed and inferred states , the ToM net will predict/infer the target choices ( intentions ) of other agents . After that , each agent decides ‘ whom ’ to communicate with according to local observation filtered by the inferred goals of others . The message is the predicted goals of the message receiver , inferred by the sender . In the end , all the agents decide their own goals by leveraging the observed , inferred , and received information . With the inferring and sharing of intentions , the agents can easily reach a consensus to cooperatively adjust the target-agent relations . Furthermore , we also introduce a communication reduction method to remove the redundant message passing among agents . Thanks to the Centralized Training and Decentralized Execution ( CTDE ) paradigm , we measure the effect of the received messages on each agent , by comparing the output of the planner with and without messages . Hence , we can figure out the unnecessary connection among agents . Then we train the connection choice network to cut these dispensable channels in a supervised manner . Eventually , we argue that ToM2C systemically solves the problem of ‘ when ’ , ‘ who ’ and ‘ what ’ in multi-agent communication , providing an efficient and interpretable communication protocol for multi-agent cooperation . The experiments are conducted in two environments . First in the cooperative navigation scenario ( Lowe et al. , 2017 ) , the team goal is to occupy landmarks ( static targets ) and avoid collision . Then we evaluate our method in a more complex scenario , multi-sensor multi-target covering scenario ( Xu et al. , 2020 ) . The team goal of the sensors is to adjust their orientation to cover as many moving targets as possible . The results show that our method achieves the best performance ( the highest reward and the lowest communication cost ) among the state-of-the-art MARL methods , e.g. , HiT-MAC ( Xu et al. , 2020 ) , I2C ( Ding et al. , 2020 ) , MAPPO ( Yu et al. , 2021 ) and TarMAC ( Das et al. , 2019 ) . Moreover , we further show the good scalability of ToM2C and conduct an ablation study to evaluate the contribution of each key component of our model . Our contributions can be summarized in three-folds : 1 ) We introduce a cognition-inspired social agent with the ability to infer the mental states of others . Such ability enhances multi-agent cooperation in target-oriented tasks . 2 ) We provide a ToM-based communication mechanism , which is fully decentralized in execution . We also propose a communication reduction method to remove redundant connections among agents . 3 ) We conduct experiments in two representative target-oriented tasks : the cooperative navigation and multi-sensor multi-target coverage problem . The results show that ToM2C not only outperforms the state-of-the-art MARL methods on reward and communication efficiency but also is of good scalability across scenes of different populations . 2 RELATED WORK . Multi-agent Cooperation and Communication . The cooperation of multiple agents is crucial yet challenging in distributed systems . Agents ’ policies continue to shift during training , leading to a non-stationary environment and difficulty in model convergence . To mitigate the non-stationarity , the centralized training decentralized execution ( CTDE ) paradigm are widely employed in the recent multi-agent learning works ( Lowe et al. , 2017 ; Foerster et al. , 2018 ; Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Iqbal & Sha , 2019 ) . However , these methods only implicitly guide agents to overfit to certain policy patterns of others . Without communication mechanism , agents lack the ability to negotiate with each other and avoid conflicts . As a result , it is hard for the individual agent to quickly adapt to unseen cooperators/environments . Learning to communicate ( Sukhbaatar et al. , 2016 ) is a feasible way to promote efficient multi-agent cooperation . Unfortunately , most of previous works ( Sukhbaatar et al. , 2016 ; Das et al. , 2019 ; Singh et al. , 2019 ) require a broadcast communication channel , leading to huge pressure on bandwidth . Besides , even though I2C ( Ding et al. , 2020 ) proposes an individual communication method , the message is just the encoding of observation , which is not only costly but also uninterpretable . In this paper , we will investigate a more efficient peer-to-peer communication , based on theory-of-mind . Hierarchical frameworks Yang et al . ( 2019 ) ; Kim et al . ( 2020 ) ; Xu et al . ( 2020 ) are also investigated to promote multi-agent cooperation/coordination . HiT-MAC ( Xu et al. , 2020 ) is the closest work to ours . It proposes a hierarchical multi-agent coordination framework to decompose the target coverage problem into two-level tasks : assigning targets by the centralized coordinator and tracking assigned targets by decentralized executors . The agents in ToM2C are also of a two-level hierarchy . Considering the natural structure of ToMAC , we also decompose the cooperation tasks in this way , rather than learning skills in an unsupervised manner ( Yang et al. , 2019 ) . Differently , both levels in ToM2C are enabled to perform distributively , thanks to the use of ToM and communication mechanism . Theory of Mind . Theory of Mind is a long-studied concept in cognitive science ( Sher et al. , 2014 ; Sanfey et al. , 2015 ; Etel & Slaughter , 2019 ) . However , how to apply the discovery in cognitive science to build cooperative multi-agent systems still remains a challenge . Most previous works make use of Theory of Mind to interpret agent behaviors , but fail to take a step forward to enhance cooperation . For example , Machine Theory of Mind ( Rabinowitz et al. , 2018 ) proposes a meta-learning method to learn a ToMnet that predicts the behaviors or characteristics of a single agent . Besides , Shum et al . ( 2019 ) studies how to apply Bayesian inference to understand the behaviors of a group and predict the group structure . Track et al . ( 2018 ) introduces the concept of Satisficing Theory of Mind , which refers to the sufficing and satisfying model of others . Shu et al . ( 2021 ) and Gandhi et al . ( 2021 ) introduce benchmarks for evaluating machine mental reasoning . These works mainly focus on how to accurately infer the mental state of others , rather than interactive cooperation . Puig et al . ( 2021 ) ; Carroll et al . ( 2019 ) ; Netanyahu et al . ( 2021 ) studies the human-AI cooperation with mental inference . Lim et al . ( 2020 ) considers a 2-player scenario and employs Bayesian Theory of Mind to promote collaboration . Nevertheless , the task is relative simple and it requires the model of other agents to do the inference . M3RL ( Shu & Tian , 2019 ) proposes to train a manager that infers the minds of agents and assigns sub-tasks to them , which benefits from the centralized mechanism and is not comparable with decentralized methods . Wu et al . ( 2021 ) introduce Bayesian approach for ToM-based cooperation , yet assuming that the environment has a partially ordered set of sub-tasks . Differently , we study how to leverage ToM to guide efficient decentralized communication to further enhance the multi-agent cooperation . Opponent Modeling . Opponent modeling ( He et al. , 2016 ; Raileanu et al. , 2018 ; Grover et al. , 2018 ) is another kind of method comparable with Theory of Mind . Agents endowed with opponent modeling can explicitly represent the model of others , and therefore plan with awareness of the current status of others . Nevertheless , these methods rely on the access to the observation of others , which means they are not truly decentralized paradigms . Compared with existing methods , ToM2C applies ToM not only to explicitly model intentions and mental states but also to improve the efficiency of communication to further promote cooperation . 3 METHODS . In this section , we will explain how to build a target-oriented social agent to achieve efficient multi-agent communication and cooperation . We formulate the target-oriented cooperative task as a Dec-POMDP ( Bernstein et al. , 2002 ) . The aim of all agents is to maximize the team reward . The overall network architecture is shown in Fig . 2 , from the perspective of agent i . The model is mainly composed of four functional networks : observation encoder , Theory of Mind network ( ToM Net ) , message sender , and decision maker . First , the raw observation oi , indicating the states of observed targets , will be encoded into Ei by an attention-based encoder . After that , the ToM Net takes Theory of Mind inference ToMi ( G∗i |Ei , Φ ) to estimate the joint intention ( sub-goals ) of others G∗i , according to the encoded observation Ei and the poses of agents Φ = ( φ1 , ... , φn ) . In details , taking ToMi , j as an example , it uses the estimated observation i , j infers the probability of agent j choosing these targets as its goals , denoted as g∗i , j . The estimation of i , j is based on the pose φj . In general , pose φj indicates the location and rotation of the agent j . Specifically , it can be represented as a 6D vector ( x , y , z , roll , yaw , pitch ) in 3D space and a 3D vector ( x , y , yaw ) in 2D plane . After the ToM inference , the message sender decides whom to communicate with . Here we employ a graph neural network to model the connection among agents . The node feature of agent j is the concatenation of i , j and Ei filtered by g∗i , j . The final communication connection is sampled according to the computed graph edge features . Agent i will send g∗i , j to agent j if there exists a communication edge from i to j . At the end , we aggregate G∗i , Ei and received messages ∑ g∗ ? , j as ηi for the decision making . The planner πHi ( gi|ηi ) , guided by the team reward , chooses the sub-goal gi to the low-level executor πLi ( ai|oi , gi ) , which takes K steps primitive actions to reach the sub-goal . In the following sections , we will illustrate the key components of ToM2C in detail .

Summary: This paper addresses the problem of reasoning about the mental states of other agents in multiagent reinforcement learning using Theory of Mind, focused on cooperative MAS domains such as multi-sensor target tracking.  A deep learning architecture is proposed that combines four networks to create a centralized solution: (1) one network that uses attention to encode observations for communication messages, (2) another than estimates the observations and goals of the agents in the environment (generalizing prior work to more than two cooperating agents), (3) a third that learns a graph determining who should communicate with whom, and when communication should happen, and (4) an actor-critic network for determining the policies of agents.  Experimental results varying the number of sensors and targets in the environment demonstrate the performance of the proposed solution against baselines.