## 4 Training Method of COMA Policy Network Based on Differential Evolution
### Main Procedure of DE-COMA  
The core of DE-COMA lies in optimizing the agent's policy network with DE to enhance its convergence. The pseudocode of DE-COMA is given in Algorithm 1, and the flowchart is shown in Figure 5.  
```
1. Initialization
2. \(\theta^{n}\): the actor network of agents
3. \(\theta^{\tilde{t}}\): the critic network
4. \(\tilde{\theta}^{\tilde{t}}_{t}\): the target network
5. \(NP\): the size of the population
6. \(D\): the dimensions of each individual
7. \(Bestidx\): the index of best individual
8. \(F\): the mutation factor
9. \(CR\): the crossover factor
10. Calculate the fitness of population
11. Sort the fitness, and return \(Bestidx\)
12. select best individual by fitness, \(\theta^{n}=population[Bestidx]\)
13. for each training episode \(e\)do: for\(e_{i}=1\)to\(\frac{\text{BactSize}}{n}\)do
14. \(s_{1}=initial\)\(state,\ t=0,\ h^{\tilde{q}}_{1}=0\) for each agent \(a\)
16. while\(s_{1}\
eq terminal\) and \(t<T\)do
17. \(t=t+1\)
18. for each agent \(a\)do
19. \(h^{a}_{t}=Actor(\theta^{n}_{t},h^{a}_{t-1},u^{a}_{t-1},a,u;\theta_{t})\)
20. Sample \(u^{a}_{t}\) from \(\pi(h^{a}_{t},e(e))\)
21. endfor
22. Get reward \(r_{t}\) and next state \(s_{t+1}\)
23. endwhile
24. Add episode to buffer
25. endfor
26. Collate episodes in buffer into single batch
27. for\(t=1\)to\(T\)do// from now processing all agents in parallel via single batch
28. Batch enroll RUN using states, actions, and rewards
29. Calculate TD(\(\lambda\)) targets using \(y^{\alpha}_{t}\) using \(\hat{\theta}^{\tilde{t}}_{t}\)
30. endfor
31. for\(t=T\)do// \(t=T\)do// \(\lambda\theta^{n}=y^{\alpha}_{t}-Q(\tilde{\sigma}^{\tilde{\tau}},u)\)
32. \(\Delta\theta^{\tilde{\sigma}}=\
abla_{\theta^{\tilde{\sigma}}}(\Delta\theta^{ \tilde{\sigma}}_{t})^{2}\)// calculate critic gradient
33. \(\theta^{\tilde{\sigma}}_{t+1}=\theta^{\tilde{\sigma}}_{t}-\Delta\theta^{ \tilde{\sigma}}_{t}\)// update critic weights
34. Every C steps reset \(\theta^{\tilde{t}}_{t}=\theta^{\tilde{t}}_{t}\)
35. endfor
36. for\(i=1\)to\(NP\)do
37. generate random index \(r_{t}\), \(r_{t}\), \(r_{t}\) mutually different
38. \(v_{t\in e1}=x_{t,\ell}+F\big{(}x_{t,\ell}-x_{\tau,\delta_{\ell}}\big{)}/\text{ mutation}\)
39. for\(j=1\)to\(D\)do
40. if\(rand(j)\leq CR\) or \(j=rnbr(i)\)then
41. \(u_{j,\ell\in e1}=v_{j,\ell\in e1}\)
42. else
43. \(u_{j,\ell\in e1}=x_{ji}\)
44. endif
45. endfor
46. get fitness according to Equation (3)
47. if\(f\big{(}u_{i,\ell\in e1}\big{)}<f\big{(}x_{i,\ell}\big{)}\)then
48. \(x_{i,\ell\in e1}=x_{i,\ell}\)
49. else
50. \(x_{i,\ell\in e1}=u_{i,\ell\in e1}\)
51. endif
52. endfor
53. for\(t=T\)do
54. \(A^{\text{BactSize}}\big{(}s^{\text{BactSize}}_{t},u\big{)}=Q(s^{\text{BactSize}}_{t}, u\big{)}-x_{u}Q(s^{\text{BactSize}}_{t},u,u^{-\text{BactSize}})\pi(u|h^{\text{BactSize}}_{t})\)
55. endfor
56. \(\theta^{\tilde{\sigma}}_{t+1}=\theta^{\tilde{\sigma}}_{t}+\alpha\lambda\theta ^{\tilde{\sigma}}\)// update actor weights
57. endfor
60. return the best policy
```  
**Algorithm 1** DE-COMA  
Figure 6 represents the initialization phase of the entire algorithm. Here, the agent's policy is abstracted into individuals within the population, and the fitness values of the population are initialized. Figure 7 illustrates the training process of the reinforcement learning component. It is important to note that the actor shares a set of parameters, with the only distinction being the addition of the agent's ID during input to differentiate the action outputs of different agents. Furthermore, to efficiently compute the counterfactual baseline, the input to the critic network includes the actions of other agents \(u_{t}^{-a}\), the global state \(s_{t}\), the local observations of this agent \(o_{t}^{a}\), the agent's ID, and the actions of all agents at the previous time step \(u_{t-1}\). This allows the critic network to directly output the counterfactual Q-values for each action of this agent. Figure 5: Flowchart of DE-COMA. Figure 6: DE-COMA initialization module.