# Target-Oriented Multi-Agent Coordination with Hierarchical Reinforcement Learning
## 4 Method
### Regulation Mechanism  
High-level exploration has a considerable impact on the overall exploration process. However, the non-stationarity of high-level transitions affects the overall learning efficiency. The high-level policy has the responsibility of outputting target assignments and receiving reward feedback, where the reward measures the value of high-level actions. The high-level policy expects the maximum reward that the low-level policy can receive under the given target assignment, representing the upper limit of the assignment and its true value. Nevertheless, during the training process, the low-level policy may not initially perform well under the given target assignment due to its ongoing learning exploration process and its model that has not yet been trained. This discrepancy can result in a mismatch between the reward feedback received by the high-level policy and the true value of its actions. Therefore, a good assignment of high-level action may yield poor rewards, which is not conducive to the high-level policy's learning. The regulation mechanism lessens the impact of the low-level policy through reward regulation. The high-level problem is modeled as follows: under the specified target assignment \(g\), there are n agents, and each agent can take action \(a\). For achieving their respective tasks as per the target assignment, every agent is rewarded. The main challenge is to coordinate the actions of all the agents to maximize the total reward. The regulatory mechanism compares the dynamic programming's evaluation reward with the low-level execution reward, and the higher reward serves as feedback for the high-level action. In the initialization setting, n is the number of agents, \(d\) is the number of agent actions, and \(r_{h}\) is the reward feedback under the action of the low-level policy. The action function, \(act(. )\), can calculate the states of assigned target after the action taken by the taking agent. The reward function, \(r(. )\), can calculate the reward value corresponding to the states of assigned target using the function 6. The DP matrix \(dp[n][d]\) has n rows and \(d\) columns. \(dp[i][j]\) is a binary sequence, indicating the maximum number of targets that can be accomplished by the first \(i\) agents taking action, and the selected action of agent \(i\) is \(j\). In the binary sequence of \(dp[i][j]\), there is a 1 in position \(t\) which indicates that the target \(t\) is tracked. The \(cnt(. )\) function calculates the number of 1 in the binary sequence, which coincides with the number of assigned targets tracked. We give the proof of the optimality of the reward regulation algorithm below. Theorem 3.1: _Both in the case of discrete and continuous action space, Algorithm 1 can calculate the optimal high-level reward value._  
```
1:the state information \(s\) and target assignation \(g\)
2:estimated high-level reward \(r_{h}\)
3:Initialize the number of the action \(d\), reward \(r_{h}\)
4:Initialize the dynamic programming array \(dp\) Given the action function \(act(. )\), counting function \(cnt(. )\) and reward function \(r(. )\)
5:for\(i=1\to n\)do
6:for\(j:=1\to d\)do
7: Compute the tracking status mask in \(act(i,j)\)
8:for\(k:=1\to d\)do
9:if\(i=0\)then
10:\(dp[i][j]\Leftarrow mask\)
11:else
12:if\(cnt(dp[i][j])<cnt(dp[i-1][k]mask)\)then
13:\(dp[i][j]\Leftarrow[i-1][k]|mask\)
14:endif
15:endif
16:endfor
17:endfor
18:for\(i:=1\to n\)do
19:\(r_{n}\Leftarrow max(r_{h},r(dp[n][i]))\)
20:endfor
21:return rh
```  
**Algorithm 1** Regulation Mechanism  
**Proof of Theorem 3.1. ** We first give the proof in discrete action space. In the high-level reward function (6), when the target assignment policy \(g\) is given, the k value is determined because of it definition \(k_{j}=\sum_{i=1}^{n}\delta_{ir}\). Therefore, we can deduce that as long as we allow the agent to accomplish as many tasks as possible, the more high-level rewards will be received.