## 5 Proposed Method
### Preliminaries
#### 5.1.2 Multi-Agent Proximal Policy Optimization  
The performance of single agent reinforcement learning algorithms are usually unsatisfactory in multi-agent system because each agent is influenced by environment and other agents at the same time. Therefore, it is necessary to use MARL to train the agents in DecPOMDP. As a variant of proximal policy optimization (PPO) specialized for multi-agent settings, Multi-Agent Proximal Policy Optimization (MAPPO) is one of the state-of-the-art MARL algorithms [35]. The algorithm adopts centralized training with decentralized execution (CTDE) architecture and has high learning efficiency with limited computing resource. In our paper, we adopt MAPPO as the learning method for multi-UAV cooperative searching and tracking problem. PPO is an on-policy reinforcement learning algorithm based on Actor-Critic framework. It originates from Trust Region Policy Optimization (TRPO) and addressing the challenge posed by policy gradient algorithm's sensitivity to step size and the difficulty in determining an appropriate step size. The objective of RL is to learn the optimal policy \(\pi^{*}\) to maximize the cumulative discounted reward \(R_{t}\):  
\[R_{t}=r_{t+1}+\gamma r_{t+2}+\gamma^{2}r_{t+3}+\gamma^{3}r_{t+4}+\cdots=\sum_{ k=0}^{\infty}\gamma^{k}r_{t+k+1}, \tag{25}\]  
where \(\gamma\in[0,1]\) is the discount factor. The policy \(\pi\) can be measured by the state-value function \(V^{\pi}(s)\) and the action-value function \(Q^{\pi}(s,a)\). State-value function \(V^{\pi}(s)\) refers to the expected cumulative discounted reward by following the policy \(\pi\) under state \(s\):  
\[V^{\pi}(s_{t})=\mathbb{E}_{\pi}[R_{t}\mid s_{t}]=\mathbb{E}_{\pi}\Bigg{[}\sum_ {k=0}^{\infty}\gamma^{k}r_{k+1}\mid s_{t}\Bigg{]}. \tag{26}\]  
And action-value function \(Q^{\pi}(s,a)\) refers to the expected reward by following the policy \(\pi\) under state \(s\) and action \(a\):  
\[Q^{\pi}(s_{t},a_{t})=\mathbb{E}_{\pi}\big{[}R_{t}\mid s_{t},a_{t}\big{]}= \mathbb{E}_{\pi}\Bigg{[}\sum_{k=0}^{\infty}\gamma^{k}r_{k+1}\mid s_{t},a_{t} \Bigg{]}. \tag{27}\]  
To improve the learning efficiency and enhance the stability of training process, the advantage function \(A^{\pi}(s,a)\) is introduced:  
\[A^{\pi}(s_{t},a_{t})=Q^{\pi}(s_{t},a_{t})-V^{\pi}(s_{t}). \tag{28}\]  
The advantage function \(A^{\pi}(s,a)\) gives the advantage of taking a given action over following the policy. Define \(\hat{A}_{t}\) as the estimate of advantage function \(A^{\pi}(s,a)\) at timestep \(t\):  
\[\hat{A}_{t}=-V(s_{t})+r_{t}+\gamma r_{t+1}+...+\gamma^{T-t+1}r_{T-1}+\gamma^{ T-t}V(s_{T}), \tag{29}\]  
where \(T\) is the trajectory length. Policy gradient method is an approach to solve reinforcement learning problem by directly optimizing the policy. The loss function of policy gradient method used in PPO is denoted as  
\[L^{CLIP}(\theta)=\hat{\mathbb{E}}_{t}\bigg{[}\min(r_{t}(\theta)\hat{A}_{t}, \text{clip}(r_{t}(\theta),1-\varepsilon,1+\varepsilon)\hat{A}_{t})\bigg{]}, \tag{30}\]  
where \(r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta}(a_{t}|s_{t})}\) represents the probability ratio. The clip function in Equation (30) is used to restrict \(r_{t}(\theta)\) to the interval \([1-\varepsilon,1+\varepsilon]\). [PERSON] et al. [35] proposed MAPPO algorithm for POMDP based on PPO method. The method proposed in our paper uses MAPPO to train two independent networks, a policy network \(\pi_{\theta}\) with parameter \(\theta\) and a value network \(V_{\phi}\) with parameter \(\phi\), which can be shared among all homogeneous agents. The policy network \(\pi_{\theta}\) maps the observation \(o_{t}\) of the agent to the discrete action space. And the value network \(V_{\phi}\) maps the state \(s_{t}\) of environment to current value.