## 4 Method
### Learning Process  
In ordinary reinforcement learning, the agent performs a complete round of observation, decision making, feedback, and learning at every moment. However, in hierarchical reinforcement learning, the goal given by the manager to the worker cannot be completed in one time step. The goal requires the worker to achieve through a period of hard work. Therefore, the decision frequency of managers and workers is different. The manager makes a decision at every \(d\) time step, and the worker still makes an instant decision. Figure 1 shows the network structure of managers and workers. We assign each manager a DDPG algorithm to make decisions. At time step t (if it is time to make a decision), manager \(M_{i}\) observes the environment and obtains local state \(LS\). Through communication, \(M_{i}\) can obtain the states of its neighbors \(N_{i}^{M}\) so that \(M_{i}\) has a wider range of environmental conditions \(LS\)+\(NS\). Since workers are distributed differently across regions, we expect managers to set specific goals for each worker. Therefore, we added the transformer structure to DDPG. According to the location relationship, we assign different attention weights to different workers. In this way, we can obtain the action output \(A\) of the manager, which is the goal \(LG\) of the worker. DDPG is designed based on actor-critic architecture. The actor network is parameterized as \(\mu_{\theta}^{M}\), the critic network is parameterized as \(Q_{\phi}^{M}\). Therefore, the loss function of its actor network is:  
\[\
abla_{\theta^{M}}J(\mu_{\theta^{M}})=E[\
abla_{\theta^{M}}\mu_{\theta^{M}}(s) Q^{\mu}(s,a)|a=\mu_{\theta^{M}}] \tag{7}\]  
The loss function of critic network is Equation (3), where the network's parameter is named \(\phi^{M}\). Moreover, DDPG uses soft-update to update the parameter of the target network:  
\[\theta^{\prime} \leftarrow\tau\theta+(1-\tau)\theta^{\prime} \tag{8}\] \[\phi^{\prime} \leftarrow\tau\phi+(1-\tau)\phi^{\prime}\]  
Likewise, we assign the MADDPG algorithm to each worker where we use \(\pi_{\theta}^{W}\) and \(Q_{\phi}^{W}\) to represent actor net and critic net. Workers decide how to switch traffic signals by observing the environment. As shown in Figure 1 at time t, workers will first collect information, including their local state \(LS\), neighbors' states \(NS\), their own goals \(LG\), and their neighbors' goals \(NG\). Using convolutional neural networks, we can extract features from these data. Since the instantaneous state \(S\) does not reflect the continuous change in the environment, we use long short-term memory (LSTM) to memorize the state of the environment to obtain the hidden state. In this way, we can make decisions based on the hidden state, and, thus, obtain the action. The updated formula of the actor network and critic network is  
\[\
abla_{\theta_{i}}J(\pi_{i}^{M})=\mathbb{E}_{o,a,S\sim D}[\
abla_{\theta_{i} }\pi_{i}^{M}(a_{i}|o_{i},g_{i})\
abla_{a_{i}}Q_{i}^{\pi^{M}}(x,a_{1},\dots,a_{ N})|_{a_{i}}=\pi_{i}^{M}(o_{i},g_{i})] \tag{9}\]  
where \(Q\) is a goal-based action-value function which uses state, goal and the actions of all agents as input and outputs the Q-value of \(W_{i}\). The original MADDPG algorithm speculates on other agents' actions by fitting their policies. However, in the TSC problem, each worker has the same task. Therefore, we use our own policy to predict the actions \(a_{j}=\pi_{i_{i}}^{W}(O_{j}^{M})\) of other agents. The update method of the action value function is  
\[\mathcal{L}(\phi_{i})=\mathbb{E}_{D}[(Q_{i}^{\phi^{M}}(x,a_{1},\dots,a_{N})-y )^{2},y=r_{i}+\gamma Q_{i}^{\phi^{\prime}}(x^{\prime},a^{\prime}_{1},\dots,a^{ \prime}_{N})|_{a^{\prime}_{j}=\phi^{\prime}_{j}(o_{j},g_{j})} \tag{10}\]  
We train managers to obtain a better task allocation policy and train workers to obtain a better traffic-signal switching policy. At the same time, the upper and lower layers realize  
Figure 1: The architecture of our goal-based multi-agent hierarchical model. The red part is manager’s network.