## 4 Solutions
### Algorithm Structure  
MAPPO is an extension of proximal policy optimization (PPO) [22] designed for multi-agent environments. It optimizes the policies of multiple agents (drones in our case) in a centralized manner while enabling decentralized execution. The following are the key components of the MAPPO algorithm:  
* Centralized critic: A single critic evaluates the joint actions of all agents, thereby providing a more stable learning process. * Decentralized actors: Each UAV has its own actor network, making decisions based on local observations. * Clipped objective: Similar to PPO, MAPPO uses a clipped surrogate objective to ensure stable policy updates, thus preventing large deviations from the current policy. The MAPPO optimization objective is provided by \[L^{\text{CLIP}}(\theta)=\mathbb{E}_{t}\big{[}\min\big{(}r_{t}(\theta)\hat{A}_ {t},\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\big{)}\big{]},\] (6) where \(r_{t}(\theta)\) is the probability ratio between the new and old policies, \(\hat{A}_{t}\) is the advantage estimate, and \(\epsilon\) is a hyperparameter that controls the clipping range. Algorithm 1 outlines an MAPPO-based MARL algorithm designed for post-disaster drone-UE communication scenarios. The MAPPO algorithm is implemented using a centralized training approach with decentralized execution. The training process involves simulating multiple episodes, where the UAVs learn to optimize their trajectories through interaction with the environment. ```
1:Orthogonally initialize actor networks \(\pi_{u}(\theta_{u})\) for each drone \(u\). 2:Initialize a shared critic network \(V(\phi)\). 3:Initialize replay buffer \(B\). 4:Set hyperparameters: learning rate \(\alpha\), \(\gamma\), clip range \(\epsilon\), batch size, replay buffer size, update interval. 5:for episode \(=1,2,\ldots\)do
6:for\(t=1\) to update interval do
7: Obtain current states \(s_{t}=\{o_{u,t}\}\) for all drones. 8:for each drone \(u\)do
9: Sample action \(a_{u,t}\) from \(\pi_{u}(o_{u,t}|\theta_{u})\). 10:endfor
11: Execute actions \(a_{u,t}\) and observe rewards \(r_{t}\) and next states \(s_{t+1}\). 12: Store experiences \((s_{t},\{a_{u,t}\},r_{t},s_{t+1})\) in replay buffer \(B\). 13:if episode is done then
14: Reset environment. 15:endif
16:endfor
17:if size of \(B\geq\) replay_buffer_size then
18:for each update step do
19: Sample mini-batch of experiences from \(B\). ```  
**Algorithm 1** MAPPO-based drones trajectory algorithm```
0: Compute advantage estimates and targets using \(V(\phi)\). 21:for each mini-batch do
22: Compute critic loss: \(L_{V}=(targets-V(s))^{2}\). 23: Update critic network: \(\phi\leftarrow\phi-\alpha\cdot\
abla_{\phi}L_{V}\). 24:endfor
25:for each drone \(u\)do
26: Compute ratio: \(r_{t}(\theta_{u})=\exp(\log\pi_{u}(a_{u,t}|s_{u,t})-\log\pi_{u}^{old}(a_{u,t}|s _{u,t}))\). 27: Compute clipped objective: \(L^{CLIP}=\min(r_{t}(\theta_{u})\cdot A_{t},\text{clip}(r_{t}(\theta_{u}),1-e,1+ e)\cdot A_{t})\). 28: Compute actor loss: \(L_{\pi}=-\mathbb{E}[L^{CLIP}]\). 29: Update actor network: \(\theta_{u}\leftarrow\theta_{u}-\alpha\cdot\
abla_{\theta_{u}}L_{\pi}\). 30:endfor
31:endfor
32: Clear replay buffer \(B\). 33:endif
34:endfor
```

**Algorithm 1**_Cont._

### Training Process

Training deep reinforcement learning models, particularly in a multi-agent setting like MAPPO, can be computationally intensive and time-consuming. To accelerate the training process and improve the efficiency of learning, we employ several techniques:

* Input normalization: Normalization ensures that all features contribute equally to the learning process and prevents issues related to varying scales of input data. By normalizing the input data, we ensure that our MAPPO-based UAV control system operates on a stable and consistent input space, thereby leading to more efficient and effective learning. * Experience replay: Experience replay helps in breaking the correlation between consecutive training samples, which can lead to more stable learning.