## 5 Proposed Method
### Observation and Action Space Representation  
For the cooperative searching and tracking problem in our paper, each UAV in the multi-UAV system keeps its individual cognitive information about the mission region to guide its action in a fully distributed decision-making approach. However, if the UAV utilizes the cognitive data of the entire mission area, an excessive amount of incorrect information can lead to confusion in the agent's decision-making. Thus, assuming that each UAV has a field of observation, it will only utilize the cognitive information within the restricted scope of observation. By decreasing the input dimension of policy network, the training speed can be enhanced, and the fixed input dimension can also enhance generalization to mission regions of any size.  
Define the observation field as a square area (with side length \(L_{o}\)) centered in the UAV's current cell. At each timestep, the UAV extract the portion of the local cognitive information about the mission region as input of the policy network. The observation of UAV \(U_{i}\) at timestep \(t\) is composed of three parts:  
\[o_{i,t}=[p_{i,t},e_{i,t},l_{i,t}], \tag{33}\]  
* Target Probability Information \(p_{i,t}\): It is extracted from target probability map \(\mathcal{P}_{i,t}\) and the target existence probability out of mission region is assumed 0;
* Environment Search Status Information \(e_{i,t}\): It is extracted from environment search status map \(\mathcal{E}_{i,t}\) and \(\mathcal{E}_{i,t}^{x,y}\) should be divided by \(t\) for normalization. The environment search status information out of the mission region is assumed 1, which means that there is no need to explore;
* UAV Location Information \(l_{i,t}\): It is extracted from UAV location map \(\mathcal{L}_{i,t}\). And the value of \(\mathcal{L}_{i,t}^{x,y}\) out of the mission region is assumed 0, which means that there is no UAV.  
At each timestep, UAVs have at most nine available actions based on their current position. The action space comprises a collection of target cells encircling the UAV, meaning that each UAV has the option to remain in its present cell or transition into any of its eight adjacent cells. In the event that an action causes the UAV to exit the mission region during the subsequent timestep, it will be excluded from the list of potential actions.  
In our paper, policy network \(\pi_{\theta}\) and value network \(V_{\phi}\) are constructed to process observation or state inputs. Their outputs are the probability distribution over available actions and estimate of the expected value, respectively. The state input to value network \(V_{\theta}\) also contains the same three parts: target location information, environment search status information and UAV location information as the observation \(o_{i,t}\) mentioned above. But the state input represents the global information of environment rather than local observation of a specific agent.  
There are two kinds of network architectures used in our method: one utilizes CNN (convolutional neural network) and fully connected layers to process information, while the other one incorporates RNN (recurrent neural network) in addition to that. The network architecture of recurrent-MAPPO is shown in Figure 2. Policy and value network first use CNN to process the input information. Then the output is flattened to obtain a one-dimensional feature vector, followed by two fully-connected layers and a gated recurrent unit (GRU) network. The usage of RNN is aimed at mining useful information hidden in historical observations. Finally, policy network uses softmax function to obtain the selection probabilities for each action and value network outputs the estimate of the expected value. We also construct the network architecture without RNN to train the agent in our paper. It omits the GRU layer while keeping the rest of the network architecture unchanged.