# Multiagent Hierarchical Cognition Difference Policy for Multiagent Cooperation
## 1 Introduction  
Grouping and effective communication are important methods to promote multiagent cooperative behavior. Agents in nature such as ants, social animals, and humans tend to cooperate and generate complex cooperative strategies by grouping and exchanging information. Naturally, the behaviors of grouping and exchanging information also apply to multiagent systems, especially in scenarios that require cooperation, such as smart grid control [1], resource management [2], and games [3, 4]. Recently, deep reinforcement learning (DRL) has shown great potential in many domains, such as games [5, 6] and robotics [7, 8]. Inspired by the powerful perception and learning ability of DRL, researchers have made continuous attempts to apply DRL to multiagent reinforcement learning (MARL) to promote multiagent cooperative behaviors in environments with many agents [9, 10, 11, 12, 13, 14, 15]. Based on the common paradigm of centralized learning with decentralized execution, some MARL algorithms learn centralized critics for multiple agents and determine the decentralized action. However, when these methods are applied to environments with a large number of agents, they have their limitations. Some MARL algorithms [9, 10] ignore different underlying influences brought by different groups of observation. Although some algorithms based on attention mechanism [16, 17, 18] partially consider the influences of groups, they do not take communication relationship among the agents into consideration. Furthermore, some algorithms [14, 19, 20] deal with the communication relationship between agents, but they do not consider the problem ofcommunication redundancy, making it unsuitable for environments with a large number of agents. To address the limitations above, we propose a novel method called multiagent hierarchical cognition difference policy (MA-HCDP) to handle state representations of different group of agents and filter irrelevant agents to promote agents' cooperation behavior. In general, MA-HCDP can divide agents into different groups, filter out irrelevant agents in these groups, and handle the interactions among the remaining agents, in that order. Corresponding to the three functions is three networks in MA-HCDP, including a hierarchical group network (HGN), a cognition difference network (CDN), and a soft communication network (SCN). Specifically, HGN is responsible for extracting high-dimensional state representations of different groups including friend groups, enemy groups, and object groups. Then, the agents' understanding of the environment is extracted based on the group-level state representations obtained from SCN and posterior distributions of variational autoencoder (VAE) [21] in CDN. Next, the differences between distributions of the agents are calculated with Kullback-Leibler (KL) divergence [22]. If the difference is large, these agents are defined as irrelevant agents and filtered out. SCN is responsible for the weight distribution of the agents to handle the influence of different neighbors. SCN expands the agents' communication field with the chain propagation characteristics of graph neural networks (GNN). The main contributions and novelties are summarized as follows:  
* A novel method, called MA-HCDP, is proposed to promote cooperation behaviors in environments with many agents. * A hierarchical group network based on prior knowledge is designed to extract high-dimensional group-level state representation. * A cognition difference network based on a variational autoencoder is designed to allow each agent to choose its neighbors adaptively to communicate. * The effectiveness of MA-HCDP is evaluated in different tasks including cooperative navigation and group containment. Compared with existing methods, MA-HCDP shows a significant improvement in all the tasks, especially for the tasks with numerous agents. The rest of this paper is organized as follows. In Section 2, we present the related works. In Section 3, we describe the background.