# Maximizing UAV Coverage in Maritime Wireless Networks: A Multiagent Reinforcement Learning Approach
## 4 Design
### Matd3
To capture the experience gained by UAVs during environmental interactions, we employed  
Figure 2: A framework diagram based on MATD3-Stacked_LSTM for the implementation of multi-UAV coverage planning. a two-layer stacked LSTM network. As illustrated in Figure 3, in the first LSTM layer, the UAV's position, velocity, and direction serve as inputs. The outputs from this layer, along with the rewards and actions obtained by the UAV in the previous time step, are then fed into the second LSTM layer. Moreover, LSTM can store and utilize previous states through its memory function, which is beneficial for solving path-planning problems under partially observable conditions. The output of the LSTM is given by:  
\[h_{t}=LSTM(s_{t},h_{t-1}) \tag{29}\]  
where \(s_{t}\) represents the state at time \(t\), and \(h_{t}\) denotes the output state of the LSTM. Finally, the output of the LSTM is combined with the observation vector in MATD3 to obtain the agent's state representation:  
\[s_{t}^{i}=\left[ \begin{array}{c}o_{t}^{i}\\ h_{t}^{i}\end{array} \right] \tag{30}\]  
where \(o_{t}^{i}\) is the observation vector of agent \(i\) at time \(t\), and \(h_{t}^{i}\) is the LSTM output state of agent \(i\) at time \(t\). Thus, as shown in Figure 2, the MATD3-Stacked_LSTM algorithm leverages LSTM networks to model state representations, capturing temporal information within time series data, as well as sequential dependencies between UAVs. The agent selects actions through an actor network and incorporates noise upon execution. Subsequently, a batch of data is randomly sampled from the experience replay buffer comprising states, actions, rewards, and next states. A critic network then computes target Q values based on this sampled batch. The TD target is calculated using the target Q values to minimize the loss function, thereby updating the critic network. Concurrently, the actor network parameters are updated through policy gradient estimation based on the sampled policies. Finally, the target networks are softly updated. As presented in Algorithm 1, we can obtain the MATD3-Stacked_LSTM-based path-planning algorithm for multiple UAVs. Figure 3: Stacked LSTM structure. ```
1:Initialize critic network parameters \(\theta_{Q}\), actor network parameters \(\theta_{\mu}\), set discount factor \(\gamma_{t}\), experience replay buffer \(D\)
2:for t = 1 to T do
3: Get the current state \(s_{t}\) representation from LSTM;
4: Select action \(a_{i}\)
5: According to the noise \(\epsilon_{i}\sim\mathcal{N}(0,\sigma)\) get the execution action \(a_{i}\sim\mu(s_{t})+\epsilon\), observe the \(r\) value
6:for agent i = 1 to N do
7: Randomly fetch a batch of data \((s_{i},a_{i},r_{i},s_{i+1})\) from experience playback cache \(D\)
8: Process via LSTM network;
9: Calculate the target Q value: \(Q^{\prime}(s_{i+1},u^{\prime})\)
10: Calculate TD target: \(\textbf{y}_{i}=r_{i}+\gamma Q^{\prime}(s_{i+1},u^{\prime}+\epsilon)\),where \(\epsilon_{i}\sim clip(\textbf{N}(0,\sigma))\)
11: Update critic network by minimizing the loss \(L=(y_{i}-Q(s_{i},a_{i}))^{2}\)
12: Update actor network using the sampled policy gradient: \(\
abla_{\theta_{\mu}}\prime\approx\frac{1}{N}\sum_{i=1}^{N}\
abla_{a}Q(s_{i}, a;\theta_{Q})|_{a=\mu(s_{i};\theta_{\mu})}\
abla_{\theta_{\mu}}\mu(s_{i};\theta_{\mu})\)
13: Update target networks: \(\theta_{Q^{\prime}}\leftarrow\theta_{Q^{\prime}}\tau+\theta_{Q^{\prime}}(1-\tau)\) \(\theta_{\mu^{\prime}}\leftarrow\theta_{\mu}\tau+\theta_{\mu^{\prime}}(1-\tau)\)
14:endfor
15:endfor
```  
**Algorithm 1** Multi-UAV coverage path planning based on MATD3-Stacked_LSTM algorithm