## 4 Sequence-to-Sequence Multi-Agent Deep Deterministic Policy Gradient
### Local Network  
From the framework of SMADDPG, it can be seen that the local network is composed of multiple DDPG networks. Each agent has its own DDPG network. The DDPG structure is shown in Figure 3. In the DDPG algorithm, there are four networks: online police network with parameter \(\theta^{\mu}\), target police network with parameter \(\theta^{\mu^{\prime}}\), online Q network  
Figure 2: Framework of SMADDPG. with parameter \(\theta^{Q}\), target Q network with parameter \(\theta^{Q^{\prime}}\). Online police network and target police network constitute actor \(\mu\big{(}o_{j},\theta^{\mu}\big{)}\) directly maps observation \(o_{j}\) to the action \(a_{j}\), which maximize the total reward. Online Q network and target Q network constitute critic \(Q\big{(}o_{j},a_{j},\theta^{Q}\big{)}\) which means the long-term reward will be received in the future after action \(a_{i}\) is taken under observation \(o_{i}\). The actor and critic constitute DDPG, in which the environments send the observations to the actor and critic. The actor selects the appropriate action according to the observation and sends it to the critic. Then, the critic estimates the Q value of this simulation and chooses the action corresponding to the maximum Q value under the current condition. The update method of actor and critic in DDPG is essentially an iterative method, that is, each version is updated from the previous version. Therefore, the most important role of target policy network \(\mu^{\prime}\big{(}o_{j},\theta^{\mu^{\prime}}\big{)}\) and target Q network \(Q^{\prime}\Big{(}o_{j},a_{j},\theta^{Q^{\prime}}\Big{)}\) is to fix the previous version and maintain the stability of the learning process. The parameter \(\theta^{\mu^{\prime}}\) and \(\theta^{Q^{\prime}}\) are updated by the equations below:  
\[\theta^{\mu^{\prime}}\leftarrow\tau\theta^{\mu}+(1-\tau)\theta^{\mu^{\prime}} \,,\,\theta^{Q^{\prime}}\leftarrow\tau\theta^{Q}+(1-\tau)\theta^{Q^{\prime}}, \tag{3}\]  
Critic updates by minimizing the loss between target \(y\) and Q value through the following equation:  
\[\min_{\theta^{Q}}\ L=\frac{1}{B}\sum_{j}\Big{(}y_{j}-Q\Big{(}o_{j},a_{j},\theta ^{Q}\Big{)}\Big{)}^{2}, \tag{4}\]  
In Equation (4), \(B\) is the mini batch size, \(\big{(}o_{j},a_{j},r_{j},o_{j+1}\big{)}\) is the data sampled from the experience replay buffer and \(y_{j}\) is calculated as the following, where \(\gamma\) is the discount factor:  
\[y_{j}=r_{j}+\gamma Q^{\prime}\Big{(}o_{j+1},\mu^{\prime}\Big{(}o_{j+1},\theta^ {\mu^{\prime}}\Big{)},\theta^{Q^{\prime}}\Big{)}, \tag{5}\]  
The actor network is updated in the way of policy gradient, as shown in the following equation:  
\[\
abla_{\theta^{\mu}}J\approx\frac{1}{B}\sum_{j}\
abla_{a}Q\Big{(}o_{j},a, \theta^{Q}\Big{)}\Big{|}_{a=\mu(o,\beta^{\mu})}\cdot\
abla_{\theta^{\mu}}\mu \big{(}o_{j},\theta^{\mu}\big{)}, \tag{6}\]  
Meanwhile, it can be seen in Figure 3 that stochastic noise \(N_{i}\) is added to the action selected by the actor according to \(o_{t}\), which is to increase the exploration, as shown in the following formula. \[a_{t}=\mu(o_{t},\theta^{\mu})+N_{i}, \tag{7}\]  
Figure 3: Framework of DDPG. The framework of the local network is shown in the Figure 4. Each agent has its own DDPG network. The parameter of the policy network is \(\theta_{i}^{\mu}\), the parameter of the target policy network is \(\theta_{i}^{\mu^{\prime}}\), the parameter of the Q network is \(\theta_{i}^{Q}\), and the parameter of the target Q network is \(\theta_{i}^{Q^{\prime}};i=0,1,\cdots,N-1\). A single DDPG network only receives local observations from its corresponding agent with itself as the \"coordinate origin\". At this point, it is meaningful to use a single strategy to control the actions of all agents. Therefore, this paper refers to asynchronous advantage actor-critic (A3C) [45] and additionally sets a central parameter network in the local network without gradient update, the parameter of policy network is \(\theta_{N}^{\mu}\), the parameter of Q network is \(\theta_{N}^{Q}\).