## 3 Proposed Methods
### Learn Local State and Local State-Action Representations  
In the context of a multi-ant robot navigating a locally observable environment with continuous actions and high-dimensional states, the excessive dimensionality of the state poses a challenge for the agent to effectively learn crucial observation information. Consequently, the agent's efficiency in employing effective strategies within this high-dimensional space is diminished. This paper introduced a solution involving a pair of encoders \((f,\ g)\) designed to learn local state and local state-action representations in order to capture the relevant structures in the local observation state space of each agent and establish a mechanism for local state transitions. To elaborate, the neural network \(f_{i}(O)\) encodes the local observation state of agent \(i\) into local observation embeddings. Each input local observation embedding vector undergoes normalization by dividing it by its average absolute value in each dimension. The procedural steps are depicted in Figure 2a, and the specific formula is expressed as follows:  
\[z_{i}^{o}=AvgL1 Norm(f_{i}(o_{i})),\textit{where}\ AvgL1 Norm(z_{i})=\frac{z_{i}}{ \frac{1}{N}\sum_{i}^{N}|z_{i}|} \tag{3}\]  
The neural network \(g_{i}(Z_{i},\ A_{i})\) concurrently encodes the local observation state embeddings and action of agent \(i\) into a state-action pair embedding, as illustrated in Figure 2b. The specific formula is articulated as follows:  
\[z_{i}^{oa}=g(z_{i}^{o},a_{i}) \tag{4}\]  
Figure 1: The NST-FACMAC framework: (**a**) the mechanism of learning local state transitions; (**b**) the decentralized policy and its novelty exploration; and (**c**) the centralized but factored critic. Figure 2: The coding process of the encoder pair. (**a**) The process of encoding local state. (**b**) The process of encoding local state-action. Encoders are divided into local state and local state-action components. In this case, each agent's encoders can be trained under dynamic prediction loss conditions, where the loss is solely dependent on the subsequent local observation state \(o^{\prime}\) and remains independent of the subsequent action or current policy. Therefore, each agent's encoders are trained by minimizing the mean square error between the embedding of the local state-action \(z_{i}^{oa}\) and the embedding of the next local observation state \(z_{i}^{o^{\prime}}\). The formula for this dynamic prediction loss is expressed as follows:  
\[L(f_{i},g_{i})=\left(g_{i}(f_{i}(o_{i}),a_{i})-\left|f_{i}(o_{i}^{\prime}) \right|_{stop}\right)^{2}=\left(z_{i}^{oa}-\left|z_{i}^{o^{\prime}}\right|_{ stop}\right)^{2} \tag{5}\]  
where \(\left|\cdot\right|_{stop}\) is a deep learning technique for controlling gradient flow in the process of backpropagation, which is used to prevent the updating of some parameters. Specifically, the \"\(\left|\cdot\right|_{stop}\)\" operation generates a tensor with the same values as the original but without gradient computation during backpropagation, halting gradient flow. \(z_{i}^{o}\) refers to the normalized local observation state, which can prevent the instability caused by minimizing the monotonous growth of the embedded space distance or collapsing into the redundant representation [51] so as to keep the relative scale of the embedding unchanged throughout the learning process. The output of the encoding pair used to model the local state transition mechanism \((o,a)\to o^{\prime}\) serves as input for the actor network so that the actor can effectively exploit the local observation state related to the local state transition mechanism, which contains the potential strategy information \(\pi(o)\to a\) from the observation state of each agent to the execution action; that is, some local observation states unrelated to the strategy are eliminated.