## 4 Solutions
### Algorithm Structure
In our implementation, we use a shared replay buffer in which all UAVs store their experiences. During training, mini-batches of experiences are randomly sampled from this buffer to update the network weights, which ensures that the UAVs learn from a diverse set of experiences. After each training session, the replay buffer is cleared to collect new information and retrain. * Parameter sharing: Parameter sharing across UAVs can significantly reduce the number of parameters to be learned and thus enhance the learning process. In our approach, we share the parameters of the actor networks among all UAVs. This not only accelerates training but also ensures that the UAVs learn a coordinated strategy for maximizing coverage and minimizing energy consumption. * Parallel training: To further speed up training, we utilize parallel training by running multiple simulations concurrently. Each simulation runs on a separate environment instance, which enables the UAVs to collect more experience in less time. The experiences from all parallel simulations are aggregated and used for updating the policy and value networks. ### Reward Function

To meet our goals, we design a reward function that is strongly correlated with the system coverage and energy consumption of the drones. If a UE is covered by drone \(u\) in time slot \(t\), and the number of UEs served by the drone does not reach the upper limit, the coverage factor \(C_{n}^{t}\) is 1--that is,

\[C_{n}^{t}= \begin{cases}1,&SR_{n}u\leq SR_{th}\\ 0,&\text{otherwise}.\end{cases}  \tag{7}\]Therefore, the total number of users served in the time slot \(t\) is

\[C^{t}=\sum_{n=1}^{N}C_{n}^{t}. \tag{8}\]

According to Equations (4) and (5), the energy consumed by drone \(u\) between time slot \(t\) can be calculated as

\[E_{t}=P_{\circ}(v_{c})\frac{H_{as}^{u}}{v_{c}}+P_{h}(v_{h})\frac{d^{u}}{v_{h}}+ P_{h}(v_{hov})\Delta t, \tag{9}\]

where \(H_{as}^{u}\) and \(d^{u}\) imply ascending or descending flying distance and horizontal flying distance of drone \(u\). The reward function that we designed aims to balance the trade-off between maximizing user coverage and minimizing energy consumption. According to Equations (8) and (9), it is expressed as

\[r_{t}=C_{t}-\xi\cdot E_{t}, \tag{10}\]

where \(C_{t}\) is the user coverage at time step \(t\), \(E_{t}\) is the energy consumption of all drones in time slot \(t\), and \(\xi\) is the weighting factor. ## 5 Simulation Results

To evaluate the performance of our MAPPO-based UAV control system, we conduct extensive simulations in a realistic disaster scenario. This section details the simulation setup, the parameters used, the evaluation metrics, and the results obtained from our experiments. ### Simulation Setup

The simulation environment is designed to mimic a typical post-disaster area with the following characteristics:

* Area: A region measuring 3 km \(\times\) 3 km is used to simulate the disaster area. * UE distribution: As illustrated in Figure 3, the rescuers are randomly distributed among 10 rescue centers in the area; each group of rescuers is assigned a part of the area and each rescue center is assigned the same number of rescuers to conduct an undifferentiated manual search of the disaster scene. * Dynamic conditions: The locations of rescuers and communication demands change over time to simulate the dynamic nature of real-world disaster scenarios. Figure 3: Rescue workers’ search route map. We deploy a different number of drones in the simulation. The drones are modeled with quad-copter dynamics, including constraints on speed (10 m/s), altitude (16 m), and maneuverability. The energy consumption of UAVs is calculated based on their speed, distance traveled, and hovering activities. Each drone-UE connection has a signal strength threshold within which it can provide services to its users. Using path loss exponential modeling, it is revealed that the communication service quality degrades with the distance from the UAV. Additionally, the interference between different drones to UEs must be considered.