## 3 Proposed Methods
### Algorithmic Pseudo-Code  
NST-FACMAC algorithm is a deterministic strategy gradient algorithm based on CTDE framework with AC structure, in which actor input is characterized by local state and explores policy according to its centralized policy novelty. Critic network includes individual critic and a global critic. The input of individual critic is the local observation state representation, the action output of actor and the local state-action representation, and the input of global critic is the \(Q\)-value of individual critic, whose pseudo-code is shown in Algorithm 1:```
Initialize: Encode local state network \(f_{i}(o_{i})\) with the parameter \(\omega_{i}\); Encode local state-action network \(g_{i}(o_{i},a_{i})\) with the parameter \(\underline{\omega}_{i}\); actor network \(\pi_{i}\) with the parameter \(\theta_{i}\) of each agent \(i\); critic Network \(Q_{i}\) with the parameter \(\phi_{i}\); mix network parameter \(\varphi\); Experience Replay Buffer. Initialize: the target local state network with weights \(\omega_{i}^{-}=\omega_{i}\); the target local state-action network with weights \(\underline{\omega}_{i}^{-}=\underline{\omega}_{i}\); the target actor network with weights \(\theta_{i}^{-}=\theta_{i}\); the target critic network with weights \(\phi_{i}^{-}=\phi_{i}\); the target mix network with weights \(\varphi^{-}=\varphi\). Initialize: Hyperparameters such as dimension \(z\_dim\) of the potential observed state
1: for episode = 1, M do
2: Initialize the environment state \(s_{0}\), Local observed state space of the all agent \(o^{0}=\left\{o_{1}^{0},o_{2}^{0},\cdots,o_{n}^{0}\right\}\)
3: Encode local state representations (Equation (3))
4: Actors randomly generate initial joint actions space \(a^{0}=\left\{a_{1}^{0},a_{2}^{0},\cdots,a_{n}^{0}\right\}\)
5: Encode local state-action representations (Equation (4))
6: Updates the environment state \(s_{1}\), local observed state space of the all agent \(o^{1}=\left\{o_{1}^{1},o_{2}^{1},\cdots,o_{n}^{1}\right\}\) and joint reward value \(r^{0}\)
7: Store the sample data \(\left\{s^{0},o^{0},z^{0},a^{0},r^{0},o^{1},z^{1}\right\}\) in the replay buffer
8: for \(t=1\), T do
9: for \(i=1\), N do
10: Sample minibatch of samples \(\left\{s^{t},o^{t},z^{t},a^{t},r^{t},o^{t+1},z^{t+1}\right\}\) from replay buffer
11: Train encoder pair (Equation (5))
12: Execute actions \(a_{i}^{t}=\pi_{\theta_{i}}(z_{i}^{t})\)
13: Calculate each critic network value \(\text{Q}_{i}\)
14: End for
15: Calculate Centralized critic (Equation (6))
16: Calculate strategy novelty \(D\left(\pi_{\theta}\big{(}z^{t}\big{)},\pi_{\theta}\big{(}z^{t+1}\big{)}\right)\)
17: Update all critics by minimizing loss function (Equation (7))
18: Update all actors by minimizing loss function (Equation (9))
19: Update all target networks by soft update: \(\theta^{-}\leftarrow\tau\theta+(1-\tau)\theta^{-}\) every \(T^{-}\) steps
20: End for
21: End for
```

**Algorithm 1**NST-FACMAC algorithm

## 4 Experiments

This paper designed an experiment to answer the following questions:

Can local observation based on local state transition mechanism improve the sampling efficiency of multi-agent deterministic strategy gradient enhancement algorithm, thereby speeding up the convergence rate? What is the influence of the dimensionality of the local observation state automatically extracted on the algorithm? Can novelty exploration of concentrated gradient improve the agent's tendency to fall into local optimality, thereby improving the overall performance? ### Simulation Environment

Simulation experiments were carried out in a single multi-legged ant robot in MAMuJoCo environment, a locally observable continuous cooperative multi-agent robot control benchmark. Each agent could only observe its own joint and body shape, and a joint reward value could only be obtained after multiple agents completed cooperation. The simulation environment is shown in the Figure 3, where square brackets indicate [(number of agents) \(\times\) (joints per agent)]. Colors indicate agent partitionings.