## 1 Introduction
However, this parameter sharing gives rise to a series of issues, which includes a lack of diversity and suboptimal convergence in the policy network parameters. Although an intuitive solution might be to allow each agent to possess different policy network parameters, as illustrated in Figure 3, this approach introduces its own set of problems. On one hand, in CTDE architecture, agents share a common set of network parameters to better learn global information, including the strategies and states of other agents, and it may lead to improved optimization of collaborative performance. The multiple sets of policy network parameters may complicate the global information and potentially affect the agents' collaborative objectives. Therefore, parameter sharing demonstrates a clear advantage in the transmission of global information. On the other hand, in a multi-agent environment, the policies of agents are often highly correlated. If each agent has its own independent network parameters and needs to learn similar policies, it may require a larger number of samples to  
Figure 1: CTDE framework vs. DTDE framework. train each agent's network and increases the required sample size for training. Conversely, by utilizing the similarity between agents, the method of sharing network parameters can effectively reduce the required training sample size and improve sampling efficiency. The trade-off between the sharing of network parameters and personalization is an ongoing research topic. The sharing of network parameters can effectively improve sampling efficiency and learning effectiveness, but it may lead to a lack of diversity and poorer convergence of policies. On the other hand, independent parameters may increase training complexity and sample requirements. To address the trade-off between the sharing of network parameters and personalization in MARL, researchers continuously explore various parameterization schemes  
Figure 3: Parameter independence model. Figure 2: Parameter sharing model. and mechanisms to promote policy diversity. Among these, the evolutionary algorithm (EA) [11; 12] is a family of random search algorithms developed based on the principles of natural evolution. The main goal of EA [13] is to iteratively explore the solution space through randomness and population-based search to seek the global optimum for optimization problems. The core characteristic of EA lies in its randomness to avoid getting trapped in local optima. By using adaptive selection and genetic operations, EA passes down excellent individuals and continuously improves and optimizes the current solution. This parallelism and global search characteristic make EA perform well in solving complex problems and situations. Therefore, EA can be a valuable method to increase policy diversity, optimize collaboration effects, and balance the relationship between parameter sharing and personalization. Based on the above analysis, we propose a policy network training method based on the differential evolution (DE) algorithm, named DE-COMA, which utilizes the classic DE algorithm to optimize the policy networks of multiple agents in the COMA algorithm. DE-COMA encodes the policy network parameter model with floating-point vectors, generates an evolutionary population, and enriches the representation of the policy network through mutation and crossover operations. In each generation, the best agent policy network parameters are evolved through the selection operation. By combining evolution and learning, DE-COMA aims to enhance the diversity of agent policies and improve the training convergence. The result provides strong support for the application of EAs to MARL. In this paper, we conducted comparative experiments between DE-COMA and other multi-agent algorithms in the StarCraft II challenge environment. The experimental results validate that DE-COMA exhibits higher win rates and more stable convergence performance compared to the original COMA algorithm in the SMAC environment.