## 1 Introduction  
Many complex reinforcement learning (RL) problems, such as drone control, distributed computing resource management, and video games, are naturally modeled as cooperative multi-agent systems [1, 2, 3]. RL is a primary approach to solving sequential decision-making problems, which requires agents to continuously interact and experiment with the environment. As agents interact with the environment through actions, the actions of agents are evaluated in the environment by providing immediate rewards [4]. Positive rewards increase the probability of taking the corresponding action, while negative rewards decrease this probability. Additionally, the actions of agents can modify the environment, and it requires repeated interactions to find the optimal strategy by maximizing the expected cumulative reward. In a single-agent task, RL only needs to consider the actions and states of a single agent. In contrast, multi-agent reinforcement learning (MARL) needs to consider a much larger action and state space. The reward of each agent is not only related to the environment but also related to the actions of other agents, therefore the multi-agent learning tasks become more complex [5]. To address the complexity of multi-agent systems, scholars have conducted extensive research on training and execution architectures. In the training architecture of multi-agent reinforcement learning, there are mainly two approaches. One is centralized training decentralized execution (CTDE) and the other is decentralized training decentralized execution (DTDE) [6]. As shown in Figure 1, CTDE architecture involves all agents sharinga common set of network parameters during the training process. The advantage of CTDE is that it can learn global information well, such as the strategies and states of other agents, and it can represent collaborative performance more effectively. However, during the execution phase, each agent operates independently by utilizing its own strategy without the need to share parameters to achieve decentralized execution. This method partially resolves the consistency issue between training and execution while achieving high execution speed and efficiency. Consequently, CTDE architecture has become the most commonly cited structure for MARL [7]. COMA [8] is a MARL algorithm based on an actor-critic (AC) framework [9]. Specifically, COMA introduces improvements and innovations in several aspects. Firstly, COMA utilizes a centralized critic and a decentralized actor. The former is only used during the learning process, and the latter is only needed during execution. During the learning phase, COMA can leverage a critic that depends on joint actions and all available state information. The policy of each agent depends only on its action-observation historical information. This centralized learning mode can better utilize global information to optimize collaboration effects. Secondly, COMA employs a counterfactual baseline [10]. By using a centralized critic to calculate agent-specific advantage functions, it compares the expected return of the current joint action with a counterfactual baseline that marginalizes over a counterfactual action of one agent while maintaining the actions of other fixed agents. Then, a separate baseline is computed for each agent, relying on the centralized critic to infer counterfactuals in which only the action of the agent is changed. COMA can estimate more accurately the contribution of each agent's policy to the global return. Thirdly, a critic representation is used to enable efficient computation of counterfactual baselines. In a single forward pass, Q-values are computed for all different actions of a specific agent, which are under the joint conditions of actions of other agents. These improvements make COMA perform exceptionally well in solving cooperative tasks in multi-agent scenarios. COMA shares a set of policy network parameters, as depicted in Figure 2, which improves sampling efficiency and learning performance to some extent.