## 5 Multi-UAV Reconnaissance Proximal Policy Optimization Algorithm
### MURPPO Based on Branch Training
As defined in Equations (23) and (24), the policy loss is calculated separately for movement and scanning actions:  
\[L^{CLIP}_{\text{move}}(\theta_{i})=\mathbb{E}_{t}[\min(r_{t,\text{move}}( \theta_{i})A^{\text{norm}}_{t,i},\text{clip}(r_{t,\text{move}}(\theta_{i}),1- \epsilon,1+\epsilon)A^{\text{norm}}_{t,i})], \tag{23}\]  
\[L^{CLIP}_{\text{scan}}(\theta_{i})=\mathbb{E}_{t}[\min(r_{t,\text{scan}}( \theta_{i})A^{\text{norm}}_{t,i},\text{clip}(r_{t,\text{scan}}(\theta_{i}),1- \epsilon,1+\epsilon)A^{\text{norm}}_{t,i})], \tag{24}\]  
where \(r_{t,\text{action}}(\theta_{i})=\frac{\pi^{\text{no}}_{\theta_{i}}(a_{t,\text{ action}}|s_{t})}{\pi^{\text{old}}_{\theta_{i}}(a_{t,\text{action}}|s_{t})}\) represents the policy ratio for the \(i\)-th UAV, with action representing either movement (move) or scanning (scan) actions. The clipping parameter \(\epsilon\) in policy loss function limits the magnitude of policy updates. When the probability ratio between new and old policies exceeds \(1+\epsilon\) or falls below \(1-\epsilon\), the clipping mechanism prevents excessive changes. This mechanism is applied to both movement and scanning policy branches independently, ensuring stable updates across both action spaces. The final policy loss for the \(i\)-th UAV combines Equations (23) and (24):  
\[L^{CLIP}_{total}(\theta_{i})=L^{CLIP}_{\text{move}}(\theta_{i})+L^{CLIP}_{ \text{scan}}(\theta_{i}). \tag{25}\]  
To maximize immediate rewards, a greedy policy often selects the currently known best action, but it can become stuck in local optima. Therefore, this paper adopts a decaying \(\epsilon\)-greedy policy [31] that primarily selects the best action based on the current policy, while with a small probability \(\epsilon\), it chooses a random action to promote exploration. Initially, \(\epsilon\) is set to 1 and decays over time, allowing for extensive early-stage exploration while gradually focusing on exploitation as the policy improves. As shown in Equation (26), the action selection process follows a probability-based mechanism:  
\[\text{Action}= \begin{cases}\text{output from the Actor},&\text{with probability }1-\epsilon,\\ \text{output a random action},&\text{with probability }\epsilon.\end{cases}  \tag{26}\]  
Figure 4: Network structure of the critic. Incorporating task-specific knowledge, when a target is detected, the UAV maintains its position and scanning direction to avoid losing the target. If no target is detected, the UAV follows the action determined by the Actor network. With probability \(\epsilon\), a random action is selected, regardless of whether a target was detected. This design ensures sufficient exploration during early training while optimizing performance in later stages by leveraging both learned strategies and domain knowledge. In MURPPO, to ensure effective exploration of both movement and scanning actions, we calculate the entropy loss separately for each action branch of the \(i\)-th UAV. Equations (27) and (28) define the entropy losses for movement and scanning policies, respectively:  
\[S[\pi_{\theta_{i},\text{move}}](s_{t}) =\mathbb{E}_{t}[\mathcal{H}(\pi_{\theta_{i},\text{move}}(\cdot|s _{t}))], \tag{27}\] \[S[\pi_{\theta_{i},\text{scan}}](s_{t}) =\mathbb{E}_{t}[\mathcal{H}(\pi_{\theta_{i},\text{scan}}(\cdot|s _{t}))], \tag{28}\]  
where \(\mathcal{H}\) is the entropy of the policy. The entropy loss encourages exploration by preventing the policy from becoming overly deterministic. For our radiation detection scenario, this is particularly important as it helps discover diverse movement patterns and scanning strategies.