# Multiagent Hierarchical Cognition Difference Policy for Multiagent Cooperation
## 4 Multiagent Hierarchical Cognition Difference Policy
### Cognition Difference Network  
In this part, we first analyze the importance of redundant agents. Then, the design process of the cognition difference network is presented. In environments with a large number of agents, if an agent communicates with all of its neighbors of the groups, redundant information will affect the agent's decision. When the number of agents in the environment increases, the communication among agents become more complicated, which makes the influence of redundant information more serious. Therefore, it is necessary for the agents to process redundant information. Artificial rules [11] or a soft attention mechanism [15; 16; 17; 18] are often used to process communication among the agents. The methods with artificial rules require strong prior knowledge of the environment and may not be suitable for application in complex environments. The methods with a soft attention mechanism assign attention coefficients to all agents within the communication range. The total amount of communication is not decreased since irrelevant agents can still obtain the attention coefficient and be included in the total communication amount. The soft attention mechanism usually assigns small but nonzero attention coefficients to irrelevant agents, which essentially weakens the attention assigned to the significant agents. Therefore, a cognition difference network (CDN) is designed to filter redundant agents to reduce redundant information. First, we construct the communication status between agents as a graph, where each node represents a single agent, and all nodes are connected in pairs by default. \(G^{t}=\left(V,E^{t}\right)\) represents the communication status between the agents. In particular, \(V=\left\{1,\ldots,N\right\}\) is a set of the agents. \(E^{t}\subseteq V\times V\) denotes the edge set at time \(t\). \(h^{t,group}\) is a set of group state representations at time \(t\), \(h^{t,group}=\left\{\vec{h}_{1}^{t,group},\vec{h}_{2}^{t,group},...,\vec{h}_{ N}^{t,group}\right\}\)\(\vec{h}_{i}^{t,group}\in\mathbb{R}^{L}\), where \(L\) represents the state representation dimension of each node. Moreover, \(N_{i}^{t}\) represents a set of neighbors of node \(i\) at time \(t\) in the graph. Agent \(j\in N_{i}^{t}\) if \(\left\|p_{i}^{t}-p_{j}^{t}\right\|_{2}<r_{c}\) where \(r_{c}\) is communication range and \(\left\|\cdot\right\|_{2}\) is a two-dimensional Euclidean norm. As indicated by (9), there is a time-varying adjacency matrix \(A^{t}\) where \(a_{ij}^{t}=1\) if \(j\in N_{i}^{t}\) otherwise \(a_{ij}^{t}=0\). \[a_{ij}^{t}=\left\{ \begin{array}{ll}1&if\ dist\left(p_{i}^{t},p_{j}^{t} \right)\leq r_{c}\ or\ i=j\\ 0&if\ dist\left(p_{i}^{t},p_{j}^{t}\right)>r_{c}\end{array} \right. \tag{9}\]  
where \(dist\) denotes a two-dimensional Euclidean norm that can calculate the distance between two agents. Then, we introduce definitions of cognitive difference based on an assumption below to filter neighbor agents more effectively. Definition 1: The cognition of an agent is its understanding of the local environment. It contains the states of all entities within its observation range, or the high-level hidden states or distributions extracted from these states (e.g., learned with deep neural networks (DNNs)). Definition 2: The cognition difference is the difference of high-level hidden states or distributions between the agents measured by \(n\)-dimensional norm or distribution measures. **Assumption 1**. : _The cognition of each agent can be represented by a vector \(C{=}[c_{1},\ldots,c_{k}]\) or a distribution \(p\left(C_{i}\big{|}h_{i}^{group}\right)\) obeying Gaussian distribution._  
Under the above definitions and assumption, if the cognition difference between agent \(i\) and agent \(j\) is relatively large, agents' perceptions of the environment are quite different. In other words, the state of agent \(j\) is noise or disturbance to agent \(i\), which will affect the policy of agent \(i\), so agent \(i\) does not need to communicate with agent \(j\).