## 5 Multi-UAV Reconnaissance Proximal Policy Optimization Algorithm
### Reinforcement Learning Algorithm  
Reinforcement learning leverages the framework of MDP, yet it does not require prior knowledge of a complete model [31]. In our multi-UAV radiation source search scenario, the objective is to find an optimal policy that guides UAVs to efficiently locate the radiation source. To address the complex challenges in multi-UAV radiation source search, we propose MURPPO, a novel variant of Proximal Policy Optimization (PPO) algorithm. The PPO framework is particularly suitable for our scenario as it provides stable policy updates while handling the uncertainties in radiation measurements and complex multi-UAV coordination. To train the value network effectively, we need a loss function that measures the accuracy of our value predictions. The mean squared temporal difference (TD) error serves as an ideal loss function as it quantifies the discrepancy between predicted and actual value estimates. The value network loss function is as follows [32]:  
\[\mathcal{L}(\omega)=\frac{1}{2}(\delta_{l})^{2}. \tag{17}\]The core component of this loss function is the TD error \(\delta_{t}\), which measures the difference between our current value estimate and a more accurate target value. The TD error is calculated by comparing the current value estimate with the sum of the immediate reward and the discounted value of the next state [33]:  
\[\delta_{t}=r+\gamma V(s_{t+1})-V(s_{t}), \tag{18}\]  
where \(r\) represents the immediate reward, \(\gamma\) is the discount factor, \(V(s_{t})\) is the value estimate of the current state \(s_{t}\), and \(V(s_{t+1})\) is the value estimate of the next state \(s_{t+1}\). This TD error enables the UAV to learn from immediate radiation measurements while accounting for potential future detections. During the policy optimization process, large policy updates can lead to performance collapse, especially in scenarios with noisy measurements like radiation detection. To address this challenge, the PPO algorithm [34] introduces a trust region-based approach that limits the policy change in each update step. This stability is achieved through a clipped objective function:  
\[L^{CLIP}(\theta)=\mathbb{E}_{t}[\min(r_{t}(\theta)A_{t},\text{clip}(r_{t}( \theta),1-\epsilon,1+\epsilon)A_{t})], \tag{19}\]  
where \(r_{t}(\theta)=\frac{\pi_{\theta}^{max}(a_{t}|s_{t})}{\pi_{\theta}^{d}(a_{t}| s_{t})}\) represents the probability ratio of the new policy to the old policy, \(A_{t}\) is the advantage function evaluating the relative benefit of actions under the current policy, \(\epsilon\) is a small constant controlling the magnitude of policy updates, and \(\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\) constrains the ratio \(r_{t}(\theta)\) to the interval \([1-\epsilon,1+\epsilon]\) to prevent excessive policy changes. This clipping mechanism helps stabilize the training process by preventing large policy updates when the UAV receives varying radiation measurements. The advantage function \(A_{t}\) represents the relative superiority of an action in a given state. It is calculated as the difference between the TD target and the current value estimate. In the PPO algorithm, temporal difference methods are used to estimate the advantage function, which then guides policy optimization. The formula is as follows [35]:  
\[A_{t}\approx\delta_{t}=r_{t}+\gamma V(s_{t+1})-V(s_{t}). \tag{20}\]  
This advantage estimation helps the UAV evaluate whether its current actions contribute to effective source detection compared to the estimated state value. The overall objective of the PPO algorithm is to maximize the weighted sum of policy loss, value function loss, and entropy bonus.