## 5 Multi-UAV Reconnaissance Proximal Policy Optimization Algorithm
### MURPPO Based on Branch Training  
Red team UAVs need to efficiently perceive the reconnaissance environment using multiple devices simultaneously. In this scenario, multiple UAVs interact with the environment concurrently, known as multi-UAV reinforcement learning. System state transitions are determined by the collective actions of UAVs, while rewards depend on their joint actions. Multi-UAV systems can be categorized into centralized and distributed control. Centralized control involves a central entity for information gathering and decision-making,with all UAVs following its instructions. This approach enables global optimization but suffers from poor scalability. In contrast, distributed control allows each UAV to make decisions independently, offering better scalability and fault tolerance. This paper employs a MURPPO algorithm with distributed control and branched training to learn complex decision-making tasks. Red team UAVs' behavior is controlled by the algorithm. Each UAV uses an independent algorithm model, and cooperation in perception and localization is achieved through reward design. When three UAVs simultaneously detect an anomalous radiation source, it triggers a three-point localization function (i.e., \(|VM(t)|\geq 3\)), enabling the reconnaissance and localization of the anomalous radiation source. To implement our distributed reinforcement learning framework effectively, we design the policy network with a dual-branch structure to reduce decision complexity and improve learning efficiency. The policy network is improved through branched training, enabling a single network to output both movement direction and detection suppression direction actions. The branched structure enables separate optimization of movement and detection tasks, where the first branch outputs the movement direction action in the movement action space. The second branch outputs the detection direction action in the detection action space. Since these two actions serve different purposes and have different gradient directions, separate network parameters are required for each branch. The policy network structure is illustrated in Figure 3.  
Additionally, a value network is designed that takes state information as input and outputs a value to evaluate the policy network. The structure of this network is shown in Figure 4. Based on Equation (17) and TD error in Equation (18), the value loss \(L^{\mathrm{VF}}(\omega_{i})\) is designed for each UAV to optimize its value estimation. This network architecture, working in conjunction with our task-specific reward design, forms the foundation of our MURPPO algorithm's capability in handling complex radiation source detection tasks. In MURPPO, to enhance training stability, we normalize the advantage function from Equation (20) for the \(i\)-th UAV by subtracting the mean and dividing by the standard deviation. The formula for the normalized advantage function is defined as follows:  
\[A_{t,i}^{\mathrm{norm}}=\frac{A_{t,i}-\mu_{A,i}}{\sigma_{A,i}+1\times 10^{-8}}, \tag{22}\]  
where \(\mu_{A,i}\) represents the mean of all advantage values, calculated as \(\frac{1}{N}\sum_{t=1}^{N}A_{t,i}\), while \(\sigma_{A,i}\) is the standard deviation, computed as \(\sqrt{\frac{1}{N}\sum_{t=1}^{N}(A_{t,i}-\mu_{A,i})^{2}}\). By subtracting the mean and dividing by the standard deviation, the advantages are scaled to have zero mean and unit variance across the batch of experiences. The small constant \(1\times 10^{-8}\) added to  
Figure 3: Network structure of the actor. the denominator prevents potential division by zero, ensuring numerical stability. This normalization is particularly important in radiation detection tasks as it helps stabilize the training process. Based on Equation (19), we modify the policy loss in MURPPO to account for the dual-branch structure.