## 3 Proposed Methods
### Algorithmic Pseudo-Code
For details, see [[https://github.com/schroederdewitt/multiagent_mujoco](https://github.com/schroederdewitt/multiagent_mujoco)]([https://github.com/schroederdewitt/multiagent_mujoco](https://github.com/schroederdewitt/multiagent_mujoco)) (accessed on 14 October 2024). ### Experimental Parameters

Parameter batch size and buffer size are commonly used parameter names in neural networks, respectively referring to the size of the sampled data during a training and the size of the replay buffer. Hidden_dim refers to the dimension of the hidden layer in the neural network, Lr refers to the learning rate of gradient descent in the case of backpropagation, and Target_update_tau refers to the target update speed in the case of soft update in the RL algorithm. The term z_dim refers to the dimension of the local observed state output by the last layer in each encoder neural network. To ensure consistency, the hyperparameters of the algorithm and their values are shown in the following Table 1:

### Experimental Results and Analysis

In the realm of MADRL, the collective reward for a team is contingent upon agents collaborating to fulfill a task, offering a means to assess algorithmic efficacy. A higher \begin{table}
\begin{tabular}{c c c} \hline \hline
**Algorithm** & **Hyperparameter** & **Value** \\ \hline All algorithm & Batch size & 100 \\  & Buffer size & 1,000,000 \\  & Buffer warmup & 1000 \\  & Gamma & 0.99 \\  & Grad\_norm\_clip & 10 \\  & Hidden\_dim & 64 \\  & Lr & 0.0005 \\  & Target\_update\_tau & 0.001 \\ \hline Except for MADDPG, MATD3 & Hypernet\_layers & 2 \\  & Mixing\_embed\_dim & 64 \\ \hline Except for NST-FACMAC & Exploration noise & \(N(0,0.1^{2})\) \\ \hline NST-FACMAC & z\_dim & 2, 4, 8 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The hyperparameters of the algorithm and their values. Figure 3: MAMuJoCo-ant simulation environment. reward signifies superior algorithm performance, indicating that agents have acquired strategies conducive to cooperative teamwork. The ensuing section scrutinizes and compares experimental outcomes, where the abscissa denotes the time step in agent learning, and the ordinate represents the average reward per episode. #### 4.3.1 Comparison and Analysis with Baseline Algorithm

In order to verify the effectiveness of NST-FACMAC, it is compared with MADDPG, FACMAC, and MATD3 multi-agent deterministic strategy algorithms, and the experimental results are in Figure 4:

Evidently, in four scenarios involving ant machine crawling, NST-FACMAC exhibits the most stable strategy execution, as denoted by its consistently straight curve. Notably, although it has two more neural networks and some hyperparameters, it achieves the highest reward and swiftest convergence, signifying heightened sampling efficiency and superior cooperative abilities compared to three alternative multi-agent deterministic strategy gradient algorithms. Therefore, NST-FACMAC has great prospects in the practical application of multi-legged crawling robots. In order to make the comparison of experimental results more clearly, the average value and standard deviation of the 10 K returned values after convergence were calculated, and the following Table 2 was made, where the standard deviation was in brackets:

Figure 4: Comparative experimental diagram of the algorithm in four simulation environments. #### 4.3.2 Ablation Experiment Verification Analysis

To ascertain the impact of locally extracted observation dimensions on the state transition mechanism, experiments were conducted on dimensions 2, 4 and 8 respectively, and the experimental results are shown in Figure 5:

The results reveal similar convergence speeds for dimensions 2 and 4, while dimension 8 exhibits slightly slower convergence. This discrepancy suggests that an excessively large dimension introduces extraneous observation states, impeding agent learning speed.