###### Abstract  
The prevalent utilization of deterministic strategy algorithms in Multi-Agent Deep Reinforcement Learning (MADRL) for collaborative tasks has posed a significant challenge in achieving stable and high-performance cooperative behavior. Addressing the need for the balanced exploration and exploitation of multi-agent ant robots within a partially observable continuous action space, this study introduces a multi-agent centralized strategy gradient algorithm grounded in a local state transition mechanism. In order to solve this challenge, the algorithm learns local state and local state-action representation from local observations and action values, thereby establishing a \"local state transition\" mechanism autonomously. As the input of the actor network, the automatically extracted local observation representation reduces the input state dimension, enhances the local state features closely related to the local state transition, and promotes the agent to use the local state features that affect the next observation state. To mitigate non-stationarity and reliability assignment issues in multi-agent environments, a centralized critic network evaluates the current joint strategy. The proposed algorithm, NST-FACMAC, is evaluated alongside other multi-agent deterministic strategy algorithms in a continuous control simulation environment using a multi-agent ant robot. The experimental results indicate accelerated convergence and higher average reward values in cooperative multi-agent ant simulation environments. Notably, in four simulated environments named Ant-v2 (2 \(\times\) 4), Ant-v2 (2 \(\times\) 4d), Ant-v2 (4 \(\times\) 2), and Manyant (2 \(\times\) 3), the algorithm demonstrates performance improvements of approximately 1.9%, 4.8%, 11.9%, and 36.1%, respectively, compared to the best baseline algorithm. These findings underscore the algorithm's effectiveness in enhancing the stability of multi-agent ant robot control within dynamic environments. automatic representation; state transition; exploration; exploitation; multi-agent reinforcement learning; deterministic strategy gradient algorithm; actor-critic methods 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol., pp. 2018-20  
multiple joints and bodies to ensure stable movement in a dynamic environment, and the single-agent DRL algorithm performs poorly in the task of cooperative multi-agent systems (MASs). The MADRL method can effectively solve the problem of multi-agent collaboration under the framework of collaborative learning and provide a useful reference for MAS applications in the future [14; 15]. In constructing a robot, each agent observes only a subset of body segments and joints. The joint state-action space, influenced by the degrees of freedom of each joint, expands exponentially with the number of agents. The challenges of partial observability and scalability in the state-action space pose obstacles to MADRL algorithms [16]. In recent years, the collaborative learning framework of centralized Training and Decentralized Execution (CTDE) has been widely used in MARL to address these challenges. In this framework, agents act decentrally, accessing global information from centralized critics and relying solely on local historical observations. Various MADRL methods, such as Factorized Multi-Agent Fitted Q-Iteration (FMA-FQI) [17], maximum mean discrepancy (Mmd-mix) [18], and variable agent sub-teams (Vast) [19], aim to learn more accurate joint action \(Q\)-value function in the CTDE framework. Nonetheless, these methods cannot solve the problem that value function decomposition methods are not suitable for high-dimensional continuous actions, making it unsuitable for multi-agent robot control tasks. Among deterministic strategy gradient-based MARL algorithms like MADDPG [20], MATD3 [21], and FACMAC [22], they demonstrate cooperation abilities in high-dimensional continuous action spaces.