# Multiagent Hierarchical Cognition Difference Policy for Multiagent Cooperation
## 4 Multiagent Hierarchical Cognition Difference Policy
### Soft Communication Network  
In this part, the influence of filtered neighbor agents is analyzed. Then, details of the soft communication network are presented. Furthermore, the process of enlarging the communication field is described.  
Actually, in addition to processing redundant agents, the filtered neighbor agents of agent \(i\) need to be treated differently to promote cooperation. Different neighbors have different cognition of the environment, and this will have different influences on agent \(i\). For instance, to an agent, agents closer to it may have more influence than agents further  
Figure 3: Architecture of variational autoencoder. The agent understanding for the environment is obtained via variational autoencoder (VAE). L2-loss is the reconstruction loss function and KL-Loss is the loss function for VAE training.  
away, which means the agent should take into the influence of different agents. Therefore, the graph attention mechanism [33] is adopted in SCN to enable agents to handle the filtered neighbor agents' states differently.  
Specifically, SCN operates on graph-structured data and obtains the features of each graph node by aggregating the states of its neighbors. As shown in Figure 4, the attention coefficient \(e_{ij}\) from agent \(j\) to agent \(i\) and its normalized form \(\alpha_{ij}\) are calculated with the hidden states of the agents:  
\[e_{ij}=a_{\mathbb{G}}^{k}\left(W_{\mathbb{G}}^{k}\widehat{C}_{i},W_{\mathbb{G} }^{k}\widehat{C}_{j}\right) \tag{16}\]  
\[\alpha_{ij}=\text{softmax}(e_{ij})=\frac{\exp\bigl{(}\text{LeakyReLU}\bigl{(}e _{ij}\bigr{)}\bigr{)}}{\sum_{k\in N_{i}}\exp\bigl{(}\text{LeakyReLU}\bigl{(}e_ {ij}\bigr{)}\bigr{)}} \tag{17}\]  
where \(W_{\mathbb{G}}^{k}\) is a linear learnable matrix, \(a_{\mathbb{G}}^{k}\) is a single-layer feedforward neural network, and \(LeakyReLU\) is a nonlinear activation function. [33] shows that the import of multihead attention is helpful to stabilize the learning process of the attention coefficients. Therefore, the aggregation states of agent \(i\) with multihead attention at \(t\) is given by:  
\[h_{i}^{s}=\left\|\frac{M}{m=1}\sigma\Bigl{(}\sum_{j\in N_{i}}a_{ij}^{m}W^{m} \widehat{C}_{j}\Bigr{)}\right. \tag{18}\]  
where \(N_{i}\) denotes the filtered neighbor agents for agent \(i\). \(||\) is the concatenation operator and \(M\) denotes the number of the heads. \(W^{m}\) is the weight matrix of the \(m\)th linear transformation and \(\alpha_{ij}^{m}\) is the normalized attention coefficient of the \(m\)th attention head.  
Moreover, the chain propagation characteristic of the graph convolution is adopted in SCN to enlarge the communication field. As described in Figure 5, agent 4 can get the states from its neighbors (agent 3 and agent 5) with one SCN layer. With two layers, agent 4 further obtains the states of its neighbors' neighbors (agent 1 and agent 6). By stacking a third layer, agent 4 can finally obtain all the agents' states. Therefore, multiple SCN layers can be utilized to enlarge the communication field of agent 4. Although multiple stacked SCN layers can take more additional structure information from the neighbor nodes of  
Figure 4: Architecture of attention. Different states of neighbors of agent \(i\) (\(\widehat{C_{j}},\widehat{C_{k}},\widehat{C_{l}},\widehat{C_{m}}\)) are concentrated with agent \(i\); then, the attention coefficient \(e_{ij}\) is calculated. The normalized form of \(e_{ij}\) is \(\alpha_{ij}\), and it is used as the weight of state concentration.  
the center node into consideration, the cost of computing is greatly increased. Therefore, the number of SCN layers is set to three in this paper.  
With SCN, each agent can handle different cognition of different neighbor agents. The communication field is enlarged with the chain propagation characteristic of graph, which promotes cooperation behaviors of the agents.