# Multiagent Hierarchical Cognition Difference Policy for Multiagent Cooperation
## 4 Multiagent Hierarchical Cognition Difference Policy
### Hierarchical Group Network  
In this part, the influence of neighbor agents is analyzed firstly. Then, details of the hierarchical group network are presented. In partially observable environments with many agents, there are many neighbors within the agents' observation range. In general, these neighbors can be divided into different groups based on prior knowledge, including a friend group, enemy group, and object group (obstacles or common goal). For instance, agents can be categorized into two groups, the friend group (agents) and the object group (obstacles or common goals), in a cooperative task. Different state representations of different groups have different influences on agents' policies. If an agent does not distinguish between states of the different groups, the influence of the states of different groups on the agent will be stacked together, thereby affecting its policy. Therefore, a hierarchical group network (HGN) is proposed to handle the different influences of state representations of different groups on the agent, which is shown in Figure 1.  
Figure 1: Architecture of MA-HCDP. Suppose there are nine entities around agent \(i\). HGN divides the entities into different groups and adopts attention mechanism to extract high-dimensional state representations. Then, CDN is responsible for filtering irrelevant agents to reduce redundant information. Next, the remaining agents are modeled as a graph \(G\). SCN is responsible for the weight distribution of the remaining agents. Finally, the captured states of different groups are subsequently used to update critic and actor networks. In this paper, the other entities such as agents or obstacles within observation range of each agent \(i\)\((i=1,\ldots,N)\) are categorized into \(H\) groups, and agent \(i\) is set as the center of each group. Note that agent \(i\) has a local observation \(o_{i}^{t}=\left[o_{i}^{t1},\ldots,o_{i}^{th},\ldots,o_{i}^{tH}\right]\) where \(o_{i}^{th}=\left[o_{i0}^{th},o_{i1}^{th},\ldots,o_{ij}^{th},\ldots,o_{iN^{th}}^{ th}\right]\), \(N^{h}\) is the number of entities in group \(h\), and \(h=1,\ldots,H\). \(o_{i0}^{th}\) is the own state of agent \(i\). Then, the attention mechanism is adopted by agent \(i\) to obtain different state representations \(h_{i}^{h}\) of different groups. It can enable agents to handle the states of the other entities in a group effectively. As shown in Figure 2, by using linear weight matrices \(W^{Q}\), \(W^{K}\), \(W^{V}\), the states \(o_{ij}^{th}\) of entity \(j\) in group \(h\) for agent \(i\) are transformed to a different space including _query_\(Q_{j}=W^{Q}o_{ij}^{th}\), _key_\(K_{j}=W^{K}o_{ij}^{th}\) and _value_\(V_{j}=W^{V}o_{ij}^{th}\). After receiving the \((query,value)\) pair from the entities in group \(h\), the attention coefficient \(a_{ij}^{h}\) from entity \(j\) in group \(h\) to agent \(i\) is computed. Then, the state representations of entities in group \(h\) for agent \(i\) are aggregated according to attention coefficients \(a_{ij}^{h}\):  
\[e_{ij}^{h}=\frac{\left(W^{Q}o_{ij}^{th}\right)\left(W^{K}o_{i0}^{th}\right)}{d _{K}},\ a_{ij}^{h}=\frac{\exp\left(e_{ij}^{h}\right)}{\sum_{j\in N^{h}}\exp \left(e_{ij}^{h}\right)} \tag{6}\]  
\[h_{i}^{h}=\sigma\left(\sum_{j\in N^{h}}a_{ij}^{h}V_{j}\right) \tag{7}\]  
where \(d_{K}\) is the dimensionality of _key_ and \(h_{i}^{h}\) is the aggregated states from other entities in group \(h\) with nonlinear activation function \(\sigma\). Figure 2: Architecture of one group attention. Suppose there are three agents around agent \(i\) in a group. State representations of agents in group \(h\) for agent \(i\) are aggregated according to attention coefficients \(a_{i}^{j}\). After state representation of all groups are obtained, they are aggregated to form a high-dimensional group-level state representation for agents' policy learning. \[h_{i}^{group}=\sum_{d=1}^{H}h_{i}^{h} \tag{8}\]  
With HGN, the entities are divided into different groups, which enables the agents to handle the state of different groups separately.