# Multiagent Hierarchical Cognition Difference Policy for Multiagent Cooperation
## 4 Multiagent Hierarchical Cognition Difference Policy
### Cognition Difference Network
After analyzing the role of the cognition difference, we need to solve two problems, including the representation of the cognition and the measurement of the cognition difference. The representation of the cognition of agent \(i\) is denoted as cognition vector \(C_{i}\); it is based on the states of all entities within its observation range. The common methods are to adopt directly multilayer perception (MLP) [9; 10; 11] or a graph convolution network (GCN) [15; 29; 30] to extract features of the observations as the cognition vector \(C_{i}\). However, the cognitive vectors extracted by these methods are essentially the result of single-valued mapping from vector to vector. These methods cannot fully decouple the observations' factors, such as position or velocity. Therefore, the cognition vector extracted by MLP or GCN is not appropriate for the cognition of the environment. In order to effectively represent the cognitive vector, we adopt a probability distribution method. Specifically, the cognition vector \(C_{i}\) is sampled with posterior distribution from group states \(h_{i}^{group}\) and \(C_{i}\) is inferred with:  
\[p\left(C_{i}\big{|}h_{i}^{group}\right)=\frac{p\left(h_{i}^{group}|C_{i} \right)p(C_{i})}{p\left(h_{i}^{group}\right)}=\frac{p\left(h_{i}^{group}|C_{i} \right)p(C_{i})}{\int p\left(h_{i}^{group}|C_{i}\right)p(C_{i})dC_{i}} \tag{10}\]  
where \(p\left(h_{i}^{group}|C_{i}\right)\) is a reconstruction process from the cognition vector \(C_{i}\): \(p\left(C_{i}\big{|}h_{i}^{group}\right)\) is the cognition representation for agent \(i\). However, the above equation is difficult to calculate directly. Hence, we approximate \(p\left(C_{i}\big{|}h_{i}^{group}\right)\) by another tractable distribution \(q\left(C_{i}\big{|}h_{i}^{group}\right)\). The restriction is that \(q\left(C_{i}\big{|}h_{i}^{group}\right)\) needs to be close to \(p\left(C_{i}\big{|}h_{i}^{group}\right)\). We achieve it by minimizing the following KL divergence:  
\[\min KL\left(q\left(C_{i}\big{|}h_{i}^{group}\right)\big{\|}p\left(C_{i} \big{|}h_{i}^{group}\right)\right) \tag{11}\]  
which is equal to the maximum of the evidence lower bound (ELBO) [31]:  
\[\max E_{q\left(C_{i}\big{|}h_{i}^{group}\right)}\,\log p\left(C_{i}\big{|}h_{ i}^{group}\right)-KL\left(q\left(C_{i}\big{|}h_{i}^{group}\right)\big{\|}p \left(C_{i}\right)\right) \tag{12}\]  
Note that the former is the reconstruction likelihood, and the latter is used to ensure that \(q\left(C_{i}\big{|}h_{i}^{group}\right)\) and the true prior distribution \(p\left(C_{i}\right)\) are as similar as possible. This process can be modeled as a VAE [21] as shown in Figure 3. Specifically, a mapping \(q\left(\widehat{C}_{i}\big{|}h_{i}^{group};\chi\right)\) from \(h_{i}^{group}\) to \(\widehat{C}_{i}\) is obtained in the encoder of VAE where \(\chi\) are the parameters of DNNs. Next, we adopt the \"reparameterization trick\" to sample \(\epsilon\) from a unit Gaussian. Then, \(\widehat{C}_{i}\) with mean \(\mu_{C_{i}}\) and variance \(\sigma_{C_{i}}\) is generated: \(\widehat{C}_{i}=\mu_{C_{i}}+\sigma_{C_{i}}\odot\epsilon\) where \(\epsilon\sim N(0,1)\). A distribution mapping \(p\left(\widehat{h_{i}^{group}}\big{|}\widehat{C_{i}};\chi\right)\) from \(\widehat{C}_{i}\) back to \(\widehat{h_{i}^{group}}\) is learned in the decoder of the VAE. Based on the above analyses, the loss function for training the VAE is:  
\[\min L2\left(h_{i}^{group},\widehat{h_{i}^{group}};\chi\right)+KL\left(q\left( \widehat{C}_{i}\big{|}h_{i}^{group};\chi\right)\big{\|}p\left(C\right)\right) \tag{13}\]  
where \(\widehat{C}_{i}\) is the result of the cognition representation of agent \(i\) and the learned distribution \(q\left(\widehat{C}_{i}\big{|}h_{i}^{group};\chi\right)\) is an approximation of \(p\left(C_{i}\big{|}h_{i}^{group}\right)\). For the measurement of difference, since \(\widehat{C}_{i}\) is sampled from the learned distribution, the distribution is more general in the environment's cognition than \(\widehat{C}_{i}\).