# Maximizing UAV Coverage in Maritime Wireless Networks: A Multiagent Reinforcement Learning Approach
## 5 Numerical Result
### Convergence Analysis  
We compared the performance of three algorithms: MATD3, MADDPG, and our proposed MATD3-Stacked_LSTM. A scenario with four UAVs was used for evaluation. As shown in Figure 5, the reward curves of MATD3 and MADDPG gradually increased and stabilized over training, but MADDPG required about 10,000 episodes to converge, while MATD3-Stacked_LSTM achieved convergence within around 5000 episodes and attained higher final rewards. This suggests that MATD3-Stacked_LSTM outperforms the other two algorithms in terms of faster convergence and superior performance. This superior performance of MATD3-Stacked_LSTM can be attributed to its use of an RNN-based LSTM network for the modeling of joint state representations. LSTM excels at capturing temporal dependencies, and in this multiagent task with long-term dependencies, the LSTM network can more comprehensively learn complex interactions among agents, leading to enhanced state representations. As shown in Figure 6, the loss values of both the actor and critic networks decrease significantly at the beginning of the training, and the decrease rate gradually slows down with the increase in training steps, eventually approaching stability. This indicates that the MATD3-Stacked_LSTM model gradually learns the optimal policy and value function during the training process and converges to the optimal solution. This also proves the effectiveness and reliability of the model. \begin{table}
\begin{tabular}{c c} \hline \hline
**Notation** & **Definition** \\ \hline Actor learning rate & \(1\times 10^{-6}\) \\ Critic learning rate & \(1\times 10^{-3}\) \\ \(\gamma\) & 0.95 \\ \(\tau\) & \(1\times 10^{-5}\) \\ Memory capacity & 100,000 \\ Batch size & 512 \\ Policy noise & 0.6 \\ Noise clip & 0.5 \\ Policy delay & 2 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Summary of notations used in the paper. The primary requirement of this coverage task is to cover all areas except for obstacles and no-fly zones; hence, the coverage rate is the most important metric. The coverage rate is calculated as the number of covered cells divided by the total number of cells, that is:  
\[Cover=\frac{\sum\limits_{i=1}^{m}\sum\limits_{j=1}^{n}Area(i,j)}{m\times n} \tag{36}\]  
where \(m\) and \(n\) are the numbers of rows and columns of the map, respectively; \(Area(i,j)\) is the coverage status at the \(i\)th row and \(j\)th column and takes a value of 0 or 1; 1 means covered; and 0 means not covered. The range of the summation is all positions of the whole map. In this experiment, we found:  
\[\frac{\sum\limits_{i,j}[Area_{i,j}==1]}{100} \tag{37}\]  
where 100 represents the total number of grids. Figure 5: Average reward trends with training episodes (smooth = 0.85). Figure 6: Average loss trends with training steps (smooth = 0.85). As shown in Figure 7, the coverage rates obtained by the three algorithms increase rapidly with the increase in the number of episodes, indicating that the three algorithms can quickly adapt to the task environment and gradually improve their exploration and exploitation abilities. MATD3-Stacked_LSTM models temporal dependencies in states using an LSTM network. At each time step, it aggregates observations across agents to learn coordinated joint actions. MATD3 directly operates on raw states without modeling interdependencies. MADDPG extends the single-agent DDPG to a multiagent setting but does not explicitly consider interactions between agents. The coverage rates based on the MATD3-Stacked_LSTM and MATD3 algorithms reach around 80% and 70%, respectively, while the MADDPG algorithm can only reach 60%. This underscores the benefits of explicitly incorporating temporal modeling capabilities via LSTM. It should be noted that since the no-fly zone and obstacles are not coverable, these areas need to be excluded when calculating the coverage rate, so the coverage rate cannot reach 100%. In order to achieve collaborative data monitoring in a target area using multiple UAVs, we conducted trajectory planning for the UAVs.