# Multiagent Hierarchical Cognition Difference Policy for Multiagent Cooperation
## 4 Multiagent Hierarchical Cognition Difference Policy
### Training Method  
After extraction by SCN, \(h_{i}^{s}\) is utilized to optimize the policy of agents. PPO is adopted to train the agents based on an actor-critic framework. With HGN, the agent can obtain high-dimensional state representation of different groups. Owing to CDN and SCN, each agent can choose the agents to communicate with and assign different weights to them. The objective function of PPO as shown in (4) is changed as (21) after \(h_{i}^{s}\) extracted from CDN and SCN. Then, PPO is trained by minimizing a total loss \(L_{total}\), which is conducted by the weighted summation of value loss \(L_{V}(w)\), action loss \(L_{\pi}(\theta)\), action entropy \(H(\theta)\), and cognition difference loss \(L_{cd}(\chi)\):  
\[L_{V}(w)=E\bigg{[}\bigg{(}\begin{array}{c}Q(O_{1},O_{2},\cdots,O_{N},a)\\ -Q(O_{1},O_{2},\cdots,O_{N},a;w)\end{array}\bigg{)}^{2}\bigg{]} \tag{19}\]  
\[l_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|(O_{1},O_{2},\cdots,O_{N}))}{\pi_{ \theta^{\dagger}}(a_{t}|(O_{1},O_{2},\cdots,O_{N}))} \tag{20}\]  
\[L_{\pi}(\theta)=E[\min(l_{t}(\theta)\hat{A}_{t}^{\theta^{k}}(O_{1},O_{2}, \cdots,O_{N}),\ clip(l_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}^{\theta^{ k}}(O_{1},O_{2},\cdots,O_{N})] \tag{21}\]  
\[H(\theta)=-\sum\pi_{\theta}(a_{t}|O_{1},O_{2},\cdots,O_{N})\log(\pi_{\theta}( a_{t}|O_{1},O_{2},\cdots,O_{N})) \tag{22}\]  
\[L_{cd}(\chi)=E\bigg{[}\bigg{(}L2\bigg{(}h_{i}^{group},\overline{h_{i}^{group}}; \chi\bigg{)}+KL\Big{(}q\Big{(}\widehat{C}_{i}\Big{|}h_{i}^{group};\chi\Big{)} \|p(C)\Big{)}\bigg{)}\bigg{]} \tag{23}\]  
\[L_{total}=\beta_{1}L_{V}(w)+\beta_{2}L_{\pi}(\theta)-\beta_{3}H(\theta)+\beta _{4}L_{cd}(\chi) \tag{24}\]  
where \(\beta_{i}\) is the weight coefficient of the loss function. The action entropy \(H(\theta)\) is specially designed to encourage exploration for agents by penalizing the entropy of actor \(\pi_{\theta}(a_{t}|O_{1},O_{2},\cdots,O_{N})\). The implementation details are presented in Algorithm 1.  
Figure 5: Enlarging respective fields. With stacked layers, agent 4 first obtains the information of agents 3 and agent 5, then obtains the information of agents 1 and 6, and finally obtains the information of agent 2.  
```
0: agent's state \(o_{i}\) Initialization: Initialize actor \(\theta_{a}\), critic \(\theta_{c}\), and old actor \(\theta_{a}^{old}\) network
1:for Episode 1 to M do
2: Run policy \(\tau_{\theta e}(o_{1},\dots,o_{N})\) for \(T\) time-steps for each agent, collecting \(\{o,a,r\}\) where \(o=(o_{1},\cdots,o_{N})\), \(a=(a_{1},\cdots a_{N})\) and \(r=(r_{1},\cdots,r_{N})\) Estimate and advantages function \(A_{t}\) \(\tau_{\theta a}^{old}=\tau_{\theta e}\)
3:for k = 1 to 4 do
4: Calculate value loss: \(L_{o}(w)\) \(L_{o}(w)=\left(Q-Q_{final}^{t}(w)\right)^{2}\)  Calculate action loss: \(L_{\pi}(\theta)\) \(L_{\pi}(\theta)=E[\min(l_{t}(\theta)\tilde{A}_{t}^{pv}(x,a_{1},\dots,a_{N}),\) \(clip(l_{t}(\theta),1-\varepsilon,1+\varepsilon)\tilde{A}_{t}^{pv}(x_{t_{1}}, \dots,a_{N})]\) Calculate entropy loss: \(H(\theta)\) \(H(\theta)=-\sum\tau_{\theta}\log(\pi_{\theta})\) Calculate cognition difference loss: \(L_{cd}(\chi)\) \(L_{cd}(\chi)=E\left[\left(L2\left(h_{i}^{group},\widehat{h_{i}^{group}};\chi \right)+KL\left(q\left(\tilde{C}_{i}\middle|h_{i}^{group};\chi\right)\middle\| p(C)\right)\right)\right]\) Calculate total loss: \(L_{total}\) \(L_{total}=\beta_{1}L_{V}(w)+\beta_{2}L_{\pi}(\theta)-\beta_{3}H(\theta)+\beta_ {4}L_{cd}(\chi)\) Update actor and critic network by minimizing \(L_{total}\)
5:endfor
6:endfor
```  
**Algorithm 1** MA-HCDP.