# NOMA-Based Rate Optimization for Multi-UAV-Assisted D2D Communication Networks
## 4 Problem Solution
### MDQN-Based Trajectory Design and Power Control
#### 4.2.2 MDQN Algorithm  
In this section, the MDQN-based trajectory design and power control allocation algorithm is designed with the training and operation modes shown in Figure 5.The MDQN algorithm uses two neural networks for Q-value estimation and updating: the evaluation network and the target network. It accepts the current state as input and then outputs the Q-value for each action. The parameters of the Q network are updated as training progresses to approximate the true Q-value function. The target-value network is also a deep neural network that is used to estimate the target Q-value, which is the expected maximum cumulative reward. The parameters of the target-value network are copied from the Q network by periodic replication and remain constant during training. After the estimated value network obtains all the Q-values in that state, the target network selects the maximum among the obtained Q-values, whose Q-value is the output of the target network plus the rewards of the samples, and its updated expression is shown in Equation (36). \[Q(S,A)\gets Q(S,A)+\alpha[R+\beta\max Q(S^{\prime},A^{\prime})-Q(S,A)] \tag{36}\]  
In order to determine the optimal action for the current state, the UAV inputs the current state information into the evaluation network through the neural network and calculates the reward \(R\) by executing and completing the action \(A\). The state information \(S^{\prime}\) is obtained at the next moment, and the system is updated to the next state. Meanwhile, this paper adopts a hybrid selection strategy based on randomness and experience accumulation to enhance the neural network's ability to explore the action selection; specifically, the neural network effectively reduces the sampling correlation of deep reinforcement learning by storing and playing back randomly sampled data samples. In the early stage of training, the UAV relies more on randomly selected actions and continuously enriches the experience pool through interaction with the environment \(\epsilon_{total}\). In order to gradually improve the quality of the strategy in the later stages of training, the actions are selected using a parameter \(\delta(t)\) that gradually converges to determinism, and the values of the probability of exploration and the corresponding action selection expressions are shown in (37) and (38) when the action is selected for the \(N_{t}-\)th time. \[\delta(t)=\delta_{\text{final}}+(\delta_{\text{start}}-\delta_{\text{final}}) e^{-N_{t}/\delta_{\text{decay}}} \tag{37}\]  
Figure 4: DQN neural network connection. \[A= \begin{cases}\text{random action},&\delta(t)\\ \operatorname*{argmax}_{A}Q(S,A,w_{e}),&1-\delta(t)\end{cases}  \tag{38}\]  
where \(\delta_{\text{start}}\) denotes the initial value of the exploration probability, \(\delta_{\text{final}}\) denotes the minimum value of the exploration probability, and \(\delta_{\text{decay}}\) denotes the decay rate of the exploration probability. The neural network parameters are optimized using the mean square error (MSE) as a loss function. The update formula for the target network and the loss function definition are shown in Equations (39) and (40):  
\[q_{target}=R+\beta\max_{A^{\prime}}Q(S^{\prime},A^{\prime},\omega_{t}) \tag{39}\]  
\[MSE=\frac{1}{n}\sum_{i=1}^{n}\left(q_{target}-Q(S,A,\omega_{e})\right)^{2} \tag{40}\]  
Based on the above analysis, the MDQN-based trajectory design and power control algorithm is shown in Algorithm 2.