# Maximizing UAV Coverage in Maritime Wireless Networks: A Multiagent Reinforcement Learning Approach
## 4 Design  
### Matd3  
The MADDPG algorithm tends to overestimate Q value, especially during early stages of training [23]. It should be noted that most real-world applications are stochastic, which has been shown to lead to higher value-function overestimation due to added noise from function approximator errors. The MATD3 algorithm extends TD3 to the multiagent setting in a way similar to how DDPG is extended to MADDPG. This algorithm addresses the stability problem and non-convex optimization objective in collaborative multiagent decision making. It improves upon the DDPG algorithm by optimizing policies, introducing interactions among multiple agents and using double Q networks to evaluate action values. In Figure 2, 3 and 3 illustrate the update process of the actor and critic networks, respectively. The Q-function update formula in the MATD3 algorithm is expressed as:  
\[\mathcal{L}_{i}^{\text{Q}}(\theta)=\mathbb{E}_{(s,a,r,s^{\prime})\sim U(D)} \Big{[}\big{(}y_{i}-Q\big{(}s,a;\theta^{-}\big{)}\big{)}^{2}\Big{]} \tag{25}\]  
where \(\mathcal{L}_{i}^{\text{Q}}(\theta)\) is the loss function of the critic network for agent \(i\) with its parameters \((\theta)\); \(\mathbb{E}_{(s,a,r,s^{\prime})\sim U(D)}\) denotes the expectation over the replay buffer (\(D\)) containing samples of state-action-reward-next state tuples; and \(y_{i}\) is the target value for updating the critic, representing the TD target for the MATD3 algorithm. The TD target (\(y_{t}=r_{t}+\gamma Q^{\prime}(s_{t+1},\mu^{\prime}(s_{t+1}|\theta^{\mu^{\prime}})| \theta^{Q^{\prime}})\)) in the MATD3 algorithm is computed by the target critic network. To enhance exploration and accelerate convergence, a clipped Gaussian noise is introduced for action exploration. Specifically, a noise vector (\(\epsilon\)) is generated based on the deterministic policy (\(\mu(s|\theta^{\mu})\)), then multiplied by a learnable parameter (\(\sigma\)) and added to the selected action; that is:  
\[a=\mu(s|\theta^{\mu})+\epsilon,\epsilon\sim\mathbf{N}(0,\sigma^{2}) \tag{26}\]  
Therefore, the noisy Q value can be written as \(Q(s_{t},a_{i}|\theta^{Q})+\epsilon\), where \(\theta^{Q}\) represents the parameters of the critic network. The clipped Gaussian noise can be bounded using the clip function, restricting it to a certain range, such as \([-c,c]\). Thus, the noisy target (\(y_{t}\)) can be expressed as:  
\[y_{t}=r_{t}+\gamma Q^{\prime}(s_{t+1},\mu^{\prime}(s_{t+1}|\theta^{\mu^{\prime }})|\theta^{Q^{\prime}})+clip(\epsilon,-c,c) \tag{27}\]  
The actor network update in MATD3 is as follows:  
\[\
abla_{\theta_{i}}\mathcal{J}\approx\frac{1}{\mathbf{N}}\sum_{j}\
abla_{ \theta_{i}}Q_{i}(s,a_{i};\theta_{i})|_{s=s_{j};a_{i}=\mathbf{f}_{i}}\big{(} \mathbf{o}_{j}\theta_{i_{j}}\big{)} \tag{28}\]  
where \(f_{i}\) represents the actor network of the \(i\)-th agent, \(\mathbf{o}_{j}\) represents the observation vector of the \(j\)-th agent, and \(\mathbf{N}\) denotes the number of training samples. \(\
abla_{\theta_{i}}\mathcal{J}\) is the gradient of the actor network's objective function for agent \(i\) with respect to its parameters (\(\theta_{i}\)). The update is approximated using samples from the replay buffer, where \(s_{j}\) is the state observation of the \(j\)-th sample, and \(\mathbf{f}_{i}\) is the actor network function for agent \(i\) with its parameters (\(\theta_{\mathbf{f}_{i}}\)). The MATD3 algorithm exhibits excellent performance in multiagent cooperation problems, but it has certain limitations when dealing with high-dimensional state spaces, as it cannot fully exploit the dynamic relationships and patterns between state sequences. LSTM neural networks, on the other hand, can capture the dynamic relationships and patterns within state sequences, providing better state representations. Combining MATD3 and LSTM can enable agents to better understand and adapt to changes in the environment, further improving the performance of multiagent systems. A single LSTM layer consistently underperforms the stacked LSTM structure.