# Target-Oriented Multi-Agent Coordination with Hierarchical Reinforcement Learning
## 4 Method
### Regulation Mechanism
We use the \(\Delta cnt_{k}(act(i,j))\) to indicate the amount of change that will occur if the \(i\) agent adopts action \(j\), assuming that the first \(i-1\) agents have reached the maximum target number and the \(i-1\)-th agent choose action k. Therefore, we can describe the above problem as the following: In this model, we transform the problem of regulating the reward into the problem of finding the maximum path in the abovementioned DAG. The path length before two nodes is defined as the amount of change in the completed targets resulting from the action taken to reach the end of the current path. For example, the length of the path between two nodes of agent \((i,d)\) and agent \((i+1,j)\) is \(\Delta cnt_{d}(act(i+1,j))\), where \((i,d)\) means agent \(i\) takes action \(d\). Each point in the model represents the maximum number of acquisition targets that can be achieved if the current layer performs this action. For example, the node \((i,d)\) means \(Max(cnt(dp[i][d]))\). Note that once the action of the first \(i-1\) agent is determined, the amount of change of the \(i\)-th agent is also determined. Therefore, each path in the diagram has been determined. In this model, the following optimized substructures exist  
\[Max(cnt(dp[i][j]))=Max(Max(cnt(dp[i-1][k]))+\Delta cnt_{k}(act(i,j))) \tag{14}\]  
where \(k=1,2,\ldots,d\). Note that in different \(k\), the \(\Delta cnt_{k}(act(i+1,j))\) is different because of its definition. Therefore, we find the optimal substructure of this problem, which proves that the optimal solution can be obtained by using dynamic programming algorithm. As for the continuous action space, we can find that every For every action, we ultimately examine the condition of its target. In other words, for a continuous action space \(\Omega\), the function \(act(. )\) can be mapped. \[act:\Omega\times S\to B \tag{15}\]  
where \(B\) is the binary sequence, whose length is the number of targets. Therefore, the continuous action space \(\Omega\) can be divided into intervals that map to different binary sequences \(B\). For example, for cases where the number of targets is 3, all actions in a subinterval \(\Omega_{1}\) of the continuous action space achieve a target capture of 010, while all actions in another subinterval \(\Omega_{2}\) of the continuous action space achieve a target capture of 110. Consequently, the continuous action space can be divided into at most \(2^{m}\) subspaces and can be treated as a case of discrete action space. We start by considering the discrete action space. In the high-level reward function (6), when the target assignment policy \(\pi\) is given, the value of \(k\) is determined by its definition. Therefore, we can deduce that the more tasks an agent completes, the higher the reward received by the high-level policy. Let \(\Delta_{ij}\) denote the change in the number of completed tasks when agent iii adopts action \(j\), assuming the first \(i-1\) agents have already maximized their task completions and agent \(i-1\) chose action \(k\). This problem can be modeled as finding the maximum path in a directed acyclic graph (DAG), where each node represents the maximum number of completed tasks if the current agent performs a specific action. The path length between two nodes indicates the change in the number of completed tasks resulting from the action taken. For example, the path length between nodes \((i,d)\) and \((i+1,j)\) is \(\Delta_{ij}\). The goal is to find the path that maximizes the total reward, which can be achieved through dynamic programming. Define the optimal substructure for the problem as follows:  
\[\Delta_{ij}=\max\left(\sum_{k=1}^{d}\Delta_{ij}\right) \tag{16}\]  
where, \(j=1,2,\ldots,d\). Here, the optimal path is determined by the maximum number of tasks completed by the agents. Use dynamic programming to solve the optimization problem. Let \(f(i,j)\) be the maximum number of tasks completed if the first iii agents choose actions that lead to node \(j\).