# Target-Oriented Multi-Agent Coordination with Hierarchical Reinforcement Learning
## 4 Method
### Low-Level Primitive Action  
Upon receiving the target assignment from the high-level policy, each agent performs the task independently based on the assigned target condition. The aim of the low-level policy is to maximize the local reward under the given target assignment. Therefore, while the high-level policy handles coordination, the low-level policy executes the assigned target task. The input to the low-level policy \(\mu_{\theta}(o,g)\) in agent \(i\) consists of two parts: one is its observation of states \(o_{i}=(o_{i1},o_{i2},\dots,o_{in},\dots,o_{in+m})\), and the other is the assigned target \(g_{i}\) transmitted by the high-level policy. Note that the observation states of agents are different in different environments, so we use the most observation states in the model.  
In the two-level architecture, however, irrelevant information can impede efficient learning of the low-level policies. For example, for agent \(i\), the targets not assigned to it are not supposed to be concerned, but there is a large amount of state information of irrelevant targets during the training process that affects its learning efficiency. The target condition filter sets the unassigned target information to 0 to reduce its disturbance. That is, for the  
Figure 2: The frame of high-level.  
assigned target \(g_{i}=(g_{i1},g_{i2},\dots,g_{im})\) and \(o_{i}\), if \(g_{ij}\) is 0, then the corresponding \(o_{i}(n+j)\) information will be changed to 0 before input. The filter layer can be represented as the following function:  
\[filter\big{(}o_{ij}\big{)}=\left\{\begin{array}{l}o_{ij},j\in[1,n]\\ 0,g_{i}(j-n)\\ o_{ij},other\end{array}\right.=0 \tag{10}\]  
For each low-level agent, it adopts an actor-critic architecture, where the actor network \(\mu_{\theta}\) outputs primitive actions, and the critic network \(Q_{\varphi}\) will collect all the state \(x\) in the environment including target and agent to calculates the value and updates the network parameters to optimize the actor network. During each episode, the agent interacts with the environment to produce trajectory data. Therefore, in agent \(i\), the loss function of \(\varphi^{i}\) is calculated as follows:  
\[L\Big{(}\varphi^{i}\Big{)}=\underset{(x,a,r_{l},x^{\prime})\in D_{l}}{(E)} \Bigg{\|}Q_{\varphi^{i}}(x,a)-r_{l}^{i}-\gamma_{l}Q_{\varphi^{i}_{target}} \big{(}x^{\prime},d^{\prime}\big{)}\bigg{|}_{d^{\prime}_{i}=\mu_{\theta^{i}_{ target}}\big{(}o^{i}_{i}\big{)}}\Bigg{\|}^{2} \tag{11}\]  
where \(D_{l}\) is a subset of replay buffer \(B_{l}\) at a low level, and the \(\mu_{\theta^{i}_{target}}\) and \(Q_{\varphi^{i}_{target}}\) target are the same as the target network at a high level. Noted that the \(a=(a_{1},a_{2},\dots,a_{n})\) is the actions performed by all agents and \(r_{l}=\big{(}r_{l}^{1},r_{l}^{2},\dots,r_{l}^{n})\) is the rewards for all agents.  
\[\begin{array}{l}\theta^{i}_{target}=\rho\theta^{i}_{target}+(1-\rho)\theta ^{i}\\ \varphi^{i}_{target}=\rho\varphi^{i}_{target}+(1-\rho)\varphi^{i}\end{array} \tag{12}\]  
Also, the loss function of the parameter \(\theta^{i}\) of the policy network in agent \(i\) is:  
\[L\Big{(}\theta^{i}\Big{)}=\underset{x\in D_{l}}{E}Q_{\varphi^{i}}(x,a)\big{|} _{a_{i}=\mu_{\varphi^{i}}(o_{i})} \tag{13}\]  
Below, we give the model diagram of the low-level network.