## 3 Active Guidance Controller Design for IRWs Based on MARL
### Improved MADDPG with PER
In the centralized training of the MADDPG framework, each agent trains value and policy networks, in which the value network of each agent can access information from all other agents when evaluating the potential benefits of actions. Once the training is complete, the policy network alone is sufficient to guide the independent execution of each agent, and the value network is no longer needed. Figure 4: PER-MADDPG algorithm framework. ```
0: Maximum time steps \(N_{slaps}\); maximum learning episodes \(E_{m}\); discount factor \(\gamma\); priority parameters \(\alpha\), \(\beta\) for PER sampling; soft update rate \(\xi\); experience replay buffer \(D\); mini-batch size \(\mid D\mid.\) Output: Trained actor networks \(\{\mu_{1},\ldots,\mu_{N}\}\) and critic networks \(\{Q_{1},\ldots,Q_{N}\}\). 01: Initialize the online actor networks \(\{\mu_{1}(\theta_{1}),\ldots,\mu_{N}(\theta_{N})\}\) with random parameters \(\{\theta_{1},\ldots,\theta_{N}\}\). 02: Initialize the online critic networks \(\{Q_{1}(\omega_{1}),\ldots,Q_{N}(\omega_{N})\}\) with random parameters \(\{\omega_{1},\ldots,\omega_{N}\}\). 03: Initialize the weights of target actor and critic networks with \(\omega_{i}^{\prime}\leftarrow\omega_{i}\) and \(\theta_{i}^{\prime}\leftarrow\theta_{i}\) (\(i=1,\ldots,N\)). 04: Initialize the SumTree structure for the experience replay buffer. 05:for episode = 1 to \(E_{m}\)do
06: Initialize the Ornstein-Uhlenbeck noise process \(N_{OU}\). 07: Initialize the environment and obtain initial state observation \(s=(o_{1,0},\ldots,o_{N,0})\). 08:for each time step do
09:for each agent \(i=1\) to \(N\)do
10: Select action \(a_{i,t}=\mu_{i,t}(o_{i,t},\theta_{i})+N_{OU}(t)\) for exploration. 11:endfor
12: Execute \(a_{t}=\{a_{1,t},\ldots,a_{N,t}\}\). 13: Observe the individual reward \(r_{t}=\{r_{1,t},\ldots,r_{N,t}\}\), new state \(s_{t+1}=\{o_{1,t+1},\ldots,o_{N,t+1}\}\) and done flag. 14: Store \((s_{t},a_{t},s_{t+1},r_{t})\) with initial high priority in \(D\) using SumTree. 15: Update state: \(s_{t}\leftarrow\delta_{t+1}\). 16:for each agent \(i=1\) to \(N\)do
17: Sample a mini-batch of \(\mid D_{s}\mid\) from \(D\) using PER. 18:for each sampled tuple do
19: Compute the TD error: \(\delta_{i}^{j}=r_{i}^{j}+\gamma Q_{i}^{\prime}\Big{(}s^{j+1},\mu_{i}^{\prime}( s^{j+1};\theta_{i}^{\prime});\omega_{i}^{\prime}\Big{)}-Q_{i}(s^{j},a^{j}; \omega_{i})\). 20: Update the sample priority \(p_{j}\) in SumTree based on \(\left|\delta_{i}^{j}\right|\). 21: Compute the sampling probability \(P_{j}\) using priority \(p_{j}\). 22: Compute the normalized IS weight \(w_{j}\) based on \(P_{j}\). 23:endfor
24: Compute the TD target \(y_{j}\) for each sample using TD error. 25: Update \(Q_{i}\) by minimizing the loss function via \(w_{j}\) weighting: \(L(\omega_{i})=\frac{1}{\mid D_{i}\mid}\sum\limits_{j\in D_{i}}w_{j}\cdot(y_{i }^{j}-Q_{i}(s^{j},a^{j};\omega_{i}))^{2}\). 26: Update \(\mu_{i}\) by optimizing with the \(w_{j}\)-weighted policy gradient: \(\
abla_{\theta_{i}}(J)(\theta_{i})\approx\frac{1}{\mid D_{i}\mid}\sum\limits_{j \in D_{i}}w_{j}\cdot\
abla_{\theta_{i}}a_{i}(o_{i}^{j};\theta_{i})\
abla_{a _{i}=\mu_{i}(o_{i}^{j})}Q_{i}(s^{j},a^{j};\omega_{i})\). 27:endfor
28: Update \(\omega_{i}^{\prime}\) and \(\theta_{i}^{\prime}\) for each agent \(i\) using a soft update: \(\omega_{i}^{\prime}\leftarrow\xi\cdot\omega_{i}+(1-\xi)\cdot\omega_{i}^{\prime}\) for target critic network; \(\theta_{i}^{\prime}\leftarrow\xi\cdot\theta_{i}+(1-\xi)\cdot\theta_{i}^{\prime}\) for target actor network. 29:if done flag is true, then
30:break
31:endif
32:endfor
33:endfor
```  
**Algorithm 1**Training pseudocode of PER-MADDPG algorithm with \(N\) agents