# Target-Oriented Multi-Agent Coordination with Hierarchical Reinforcement Learning
## 4 Method
### High-Level Target Assignment  
The goal of the high-level policy \(\mu_{g}(s)\) is to learn an optimal policy to maximize the global reward. The high-level policy determines the target assignment for each agent. It is implemented through the construction of a deep neural network, which consists of two components: actor and critic. According to the joint state \(s=(s_{1},s_{2},\ldots,s_{n},\ldots,s_{n+m})\) where m is the number of target and \(n\) is the low-level agent, the actor outputs the target assignment \(g=(g_{1},g_{2},\ldots,g_{n})\). An actor receives the joint state as an input and calculates the probability \(g_{ij}\) of each assignment through a fully connected layer where \(g_{ij}\) is a binary value indicating whether or not to let agent \(i\) track target \(j\). Therefore, the actor will output the target assignment of all agents \(g=(g_{1},g_{2},\ldots,g_{n})\), where \(g_{i}=(g_{i1},g_{i2},\ldots,g_{im})\) denotes the target assignment of agent \(i\).  
The high-level reward function is shown as follows:  
\[r_{h}=\sum_{j=1}^{m}\left(\frac{p_{j}}{k_{j}}-\alpha_{1}q_{i}\right)\left| \frac{p_{j}}{k_{j}}_{=0\text{ \ \ when }k_{j}=0}\right. \tag{6}\]  
where m is the number of targets, \(p_{j}\) represents whether the assigned target task \(i\) is completed \((0:No,1:Yes)\), \(k_{j}=\sum_{i=1}^{n}g_{ij}\) represents the number of agents tracked to the assigned target \(j\), \(q_{j}\) represents whether the assigned target \(i\) is not completed \((0:No,1:Yes)\), and \(\alpha_{1}\) is a punishment factor that ranges from 0 to 1. p and q motivate the agents to complete more assigned target tasks, the larger the number of k the smaller the reward, avoiding targets conflict in agent coordination.  
The critic network calculates loss function and updates actor network parameters. The critic calculates the optimal value of the state through the neural network, and the actor uses the optimal value to update the parameters of the policy function, then selects the action and receives feedback and the next state. The loss function of the critic is calculated as follows:  
\[L(\phi)=\underset{(s,g,\mathcal{L}_{h},s^{\prime})\in D_{h}}{E}\|Q_{\phi}(s,g)-r_ {h}+\gamma_{h}Q_{\phi target}\big{(}s^{\prime},\mu_{\beta target}\big{(}s^{ \prime}\big{)}\big{)}\|^{2} \tag{7}\]  
where the \(D_{h}\) is a subset of the replay buffer \(B_{h}\) at a high level. Since the \(Q\) network and policy function are constantly changing in the process of training, which will lead to the instability of the environment, we need to introduce the objective function. The \(\beta_{\text{target}}\) and \(\phi_{\text{target}}\) are the parameter the target policy network and the \(Q\) network, respectively, which change slowly.  
\[\beta_{\text{target}}=\rho\beta_{\text{target}}+(1-\rho)\beta\phi_{\text{ target}}=\rho\phi_{\text{target}}+(1-\rho)\phi \tag{8}\]  
where \(\rho\) is a number close to one. Therefore, the target network will update very slowly. By using the policy gradient theorem, the policy network update is as follows:  
\[\beta=\beta+\alpha_{2}\underset{s\in D_{h}}{E}\
abla\beta\mu\beta(s)\
abla_{ g}Q_{\phi}(s,g) \tag{9}\]  
The model of the high-level structure is showed in Figure 2.