## 4 Sequence-to-Sequence Multi-Agent Deep Deterministic Policy Gradient
### Global Network
```
1:Initialize policy networks and Q networks with \(\theta_{i}^{u}\) and \(\theta_{i}^{Q}\), \(i=0,1,\cdots,N-1\)
2:Initialize target policy network and target Q network with \(\theta_{i}^{u^{\prime}}\leftarrow\theta_{i}^{u}\) and \(\theta_{i}^{Q^{\prime}}\leftarrow\theta_{i}^{Q}\)
3:Initialize central parameter network with \(\theta_{N}^{u}\) and \(\theta_{N}^{Q}\)
4:Initialize sequence-to-sequence network with \(\delta\)
5:Set the max number of episodes \(E_{\text{max}}\), the max step of every episode \(S_{\text{max}}\) and mini batch size \(B\)
6:for\(e=0\) to \(E_{\text{max}}\)do
7:for\(t=0\) to \(S_{\text{max}}\)do
8: Receive observation of all agents \(O_{t}^{ddgg}=O_{t}^{seq}=o_{t,0},o_{t,1},\cdots,o_{t,N-1}\)
9: Input \(O_{t}^{ddgg}\) into actor networks and obtain \(A_{t}^{ddgg}=a_{t,0}^{d},a_{t,1}^{d},\cdots,a_{t,N-1}^{d}\)
10: Input \(O_{t}^{seq}\) into sequence-to-sequence network and obtain \(A_{t}^{seq}=a_{t,0}^{d},a_{t,1}^{d},\cdots,a_{t,N-1}^{d}\)
11: Execute actions \(A_{t}=A_{t}^{dgg}+A_{t}^{seq}\)
12: Receive rewards \(R_{t}^{ddgg}=R_{t}^{seq}=r_{t,0},r_{t,1},\cdots,r_{t,N-1}\)
13: Receive new observations \(O_{t+1}^{dgg}=O_{t+1}^{seq}=o_{t+1,0},o_{t,1},\cdots,o_{t,N-1}\)
14: Store \(\left(O_{t}^{ddgg},A_{t}^{ddgg},R_{t}^{ddgg},O_{t+1}^{dgg}\right)\) in replay buffer \(D\)
15: Store \(\left(O_{t}^{seq},A_{t}^{seq},R_{t}^{seq}\right)\) in memory \(M\)
16:for\(i=0\) to \(N-1\)do
17: Sample a random batch of \(B\) samples \(\left(o_{j},a_{i},r_{j},o_{j+1}\right)\) from \(D\)
18: Compute \(y_{j}=r_{j}+\gamma Q^{\prime}\Big{(}o_{j+1},u^{\prime}\left(o_{j+1},\theta_{i }^{u^{\prime}}\right),\theta_{i}^{Q^{\prime}}\Big{)}\)
19: Update the critic network for agent i using \(\min\limits_{\theta_{i}^{Q}}\ L=\frac{1}{B}\sum\limits_{j}\Big{(}y_{j}-Q\Big{(}o _{j},a_{j},\theta_{i}^{Q}\Big{)}\Big{)}^{2}\)
20: Update the actor network for agent i using \(\
abla_{\theta_{i}^{u^{\prime}}}f\approx\frac{1}{B}\sum\limits_{j}\
abla_{a}Q \Big{(}o_{j},a,\theta_{i}^{Q}\Big{)}\Big{|}_{a=\mu(o_{i},\theta_{i}^{u^{\prime }})}\cdot\
abla_{\theta_{i}^{u^{\prime}}}\mu\left(o_{j},\theta_{i}^{u^{\prime }}\right)\)
21:endfor
22: Update the local network for each agent i:
23:\(\theta_{i}^{u^{\prime}}\gets\tau\theta_{i}^{u^{\prime}}+(1-\tau)\theta_{i }^{u^{\prime}},\theta_{i}^{Q^{\prime}}\gets\tau\theta_{i}^{Q}+(1-\tau) \theta_{i}^{Q^{\prime}}\)
24:\(\theta_{N}^{u}\gets\tau\theta_{i}^{u^{\prime}}+(1-\tau)\theta_{N}^{u}, \theta_{N}^{Q}\leftarrow\tau\theta_{i}^{u^{\prime}}+(1-\tau)\theta_{N}^{Q}\)
25:\(\theta_{i}^{u^{\prime}}\gets\tau\theta_{N}^{u}+(1-\tau)\theta_{i}^{u^{ \prime}},\theta_{i}^{Q^{\prime}}\leftarrow\tau\theta_{N}^{Q}+(1-\tau)\theta_{i }^{Q^{\prime}}\)
26:endfor
27: Select \(\left(O_{m}^{seq},A_{m}^{seq},R_{m}^{seq}\right)\) by max \(R^{seq}\), and then generate collection
28: Train the sequence-to-sequence network by this data set
29: The core equation in sequence to sequence
30:\(\max\limits_{\delta}\ L_{seq}=\frac{1}{N}\sum\limits_{n=0}^{N-1}\ln\left(a_{0} ^{s},a_{1}^{s},\cdots,a_{N-1}^{s}\big{|}o_{0}^{s},o_{1}^{s},\cdots,o_{N-1}^{s},\delta\right)\)
31:endfor
```  
**Algorithm 1** SMADDPG  
The settings of each key parameter in the algorithm are shown in the Table 1.