## 2 Background  
We define the interaction process between the agent and the environment as a Markov decision process (MDP) [21]. In a multi-agent system, MDP can be represented by a quintuple as \(M=(S,A,R,T,\gamma)\). Among them, \(S\) is the state space of the agent, \(A\) is the action space, \(R:S\times A\times S\rightarrow\mathbb{R}\) is the reward function, \(T:S\times A\times S\rightarrow[0,1]\) is the state transition function, and \(\gamma\in[0,1]\) is the discount factor. In a partially observable environment, each agent can only observe the environment around itself to obtain an observation \(O:S\to O\). Each agent uses their policy to obtain an action by observing the environment \(\pi_{\theta}:S\to A\). After performing the action, the environment will reward each agent. Accordingly, the purpose of the agent is to maximize the cumulative reward \(R_{\theta}=E_{\tau\sim p_{\theta}(\tau)}[R(\tau)]\).  
In the mainstream algorithm of reinforcement learning, the value-based DQN fits the state value function through the neural network, thereby solving the dimensional disaster problem of the state space. The loss function of the network is:  
\[\mathcal{L}(\theta_{i})=E_{s,a,r,s^{\prime}}[(r+\gamma max_{a^{\prime}}Q(s^{ \prime},a^{\prime};\theta_{i}^{-})-Q(s,a;\theta_{i}))^{2}] \tag{1}\]  
The policy-based PG algorithm [8] directly fits the policy, and the update function of the network is:  
\[\
abla_{\theta}J(\pi_{\theta})=E[\
abla_{\theta}\log\pi(a|s)Q^{\pi}(s,a)] \tag{2}\]  
The proposal of DDPG [22] solves the continuous action space problem. Unlike DQN and PG output action distribution, DDPG directly outputs a deterministic action. At the same time, DDPG uses a generalized AC architecture [23], where the actor aims to find the action that maximizes the Q value, and the critic aims to estimate the action value. The loss function of the critic net is:  
\[\mathcal{L}(\phi)=\
abla_{\phi}\frac{1}{|B|}\sum_{(s,a,r,s^{\prime},a^{\prime} )\in B}(Q_{\phi}(s,a)-y)^{2} \tag{3}\]  
where  
\[y=r+\gamma Q_{\phi}(s^{\prime},\mu_{\theta}(s^{\prime})) \tag{4}\]  
MADDPG, on the other hand, extends DDPG to a multi-agent environment. The algorithm first predicts the opponent's actions by fitting their policies. Then, a centralized action value network is constructed, which combines the joint action and the environment state to obtain its action state value. The algorithm solves the non-stationary problem by introducing the actions of other agents. Critic can be updated by minimizing the loss:  
\[\mathcal{L}(\theta_{i})=\mathbb{E}_{x,a,r,x^{\prime}}[(Q_{i}^{\mu}(x,a_{1}, \dots,a_{N})-y)^{2},y=r_{i}+\gamma Q_{i}^{\mu^{\prime}}(x^{\prime},a_{1}^{ \prime},\dots,a_{N}^{\prime})|_{a_{i}^{\prime}=\mu_{i}^{\prime}(o_{j})} \tag{5}\]  
and the actor's policy gradient is computed by:  
\[\
abla_{\theta_{i}}J(\mathbf{\mu}_{i})=\mathbb{E}_{x,a\sim D}[\
abla_{\theta_{i}} \mathbf{\mu}_{i}(a_{i}|o_{i})\
abla_{a_{i}}Q_{i}^{\mu}(x,a_{1},\dots,a_{N})|_{a_{i }}=\mathbf{\mu}_{i}(o_{i})] \tag{6}\]  
Hierarchical reinforcement learning is very effective in solving complex or reward-sparse environments. This method uses the idea of divide and conquer to decompose the problem: the upper layer makes long-term decisions, and the lower layer makes real-time decisions. Hierarchical reinforcement learning includes option based [24; 25] and goal based [26; 27; 28]. The former will pre-train multiple sub-policies, and then the upper layer will switch the sub-policies according to the environment state. The latter formulates long-term goals for the lower layer, and the lower layer achieves this goal through its policy.