# Multiagent Hierarchical Cognition Difference Policy for Multiagent Cooperation
## 2 Related Works  
The multiagent deep deterministic policy gradient (MADDPG) [9] is extended from the deep deterministic policy gradient (DDPG) [7] to multiagent systems for mixed cooperative-competitive environments. A counterfactual multiagent (COMA) [10] computes a counterfactual advantage function to handle the problem of multiagent credit assignment. They adopt a common paradigm of centralized learning with decentralized execution (CTDE) to enhance cooperative behaviors of agents. Although MADDPG and COMA can improve the agents' cooperation ability, they do not consider the complex interactions among the agents. They aggregate observations of all the agents and never distinguish different groups of agents, thus limiting the cooperation of agents.  
To address the limitation, the agent grouping method [12] employs a two-level graph neural network to model the interagent and intergroup relationships effectively. However, it ignores the communication relationship between the agents in the same group. The authors of [13] designed a two-level attention network to distinguish the different semantic meanings of observation. Nonetheless, it does not consider the communication contents.  
To deal with the problem of communication, a feedforward deep neural network is adopted in [14] to map all agents' inputs to their actions. Each agent can have access to an implicit communication channel to receive other agents' states. A bidirectionallycoordinated network (BiCNet) [19] based on the actor-critic model adopts bidirectional recurrent networks to achieve mutual communication between agents. Master-slave [20] is a communication architecture for real-time strategy games, where the action of each slave agent consists of information from the slave agents and master agent. Although these methods can promote cooperation behaviors of agents, they need the global states of the environment during the training, which is not applicable in partially observable environments. Furthermore, these methods take all the communication among the agents into consideration, but they ignore the influence of redundant communication caused by too many communication objects.  
To address this issue, some methods based on the attention mechanism appeared [15; 16; 17; 18]. Nonetheless, each agent within the communication range will be assigned an attention coefficient with attention mechanism. The total amount of communication is not decreased since irrelevant agents can still obtain the attention coefficients and be included in the total communication amount. The soft attention mechanism usually assigns small but nonzero attention coefficients to irrelevant agents, which weakens the attention assigned to the significant agents.  
Based on the above analyses, MA-HCDP is proposed to handle state representations of different group of agents and filter irrelevant agents to promote agents' cooperation behavior. To the best of our knowledge, none of existing work in MARL tackles simultaneously the problem of agents grouping, redundant communication information processing, and agent interaction processing like our MA-HCDP.