## 4 Proposed Trajectory Optimization Approach
### Proposed H-DQN Algorithm
```
0: initial \(Q\) network parameter \(\theta\) and target network parameter \(\theta^{\prime}\)
0: trained network parameters
1: initialize the experience playback pool \(\mathcal{D}\) and the greedy coefficient \(\varepsilon=1\)
2:for\(episode\in\{1,2,3...\}\)do
3: initialize the system state \(t=0\)
4:for\(i\in\{1,2,...,I\}\)do
5: the UAV agent gets the current system state \(s_{i}\)
6: choose an option based on \(\varepsilon\)-greedy strategy \(o_{i}\)
7: execute the middle strategy \(\pi_{o}\) of option
8: get the new system state \(s_{i+1}\) and reward \(r_{o,i}\)
9: save \(\langle s_{i},o_{i},s_{i+1},r_{o,i}\rangle\) into \(\mathcal{D}\)
10: update the greedy coefficient \(\varepsilon\) according to (13), update \(t\gets t+T_{i}\)
11: randomly sample \(K\) samples from \(\mathcal{D}\): \(\langle



reinforcement learning environment is realized based on the Gym toolkit, andit implements the H-DQN algorithm code based on PyTorch. We consider using a 500 m x 500 m area as a simulation scenario. Due to natural disasters or other reasons, UAVs are needed to assist in real-time data collection tasks in this area, and a certain number of sensor devices are distributed. The data convergence center is located at the edge of the area \([0,0]\), and the UAVs take off or land from here. During the execution of the data collection task, the UAV can return to the data convergence center for charging. In the simulation environment, the UAV flies at a fixed speed of 20 m/s and the maximum energy it can carry is 2 \(\times 10^{4}\) J. The experimental analysis is carried out according to the percentage of residual energy. More detailed environmental parameters and UAV parameters are shown in Table 1.

In our H-DQN method, both the Q network and the target Q network use four-layer artificial neural network architecture. The activation function is unified using the ReLu function, the output layer is added with a layer of the Softmax function to obtain the probability distribution of the output action, and the learning rate is set to 0.001. Other hyperparameters related to the H-DQN training are shown in Table 2.

### Training Results

First, we verify the convergence effect of the algorithm when the learning rate is 0.001, and we compare it with the traditional DQN algorithm based on rasterization processing and the Markov decision process modeling. As shown in Figure 3, it can be seen that the layered deep reinforcement learning algorithm H-DQN proposed in this paper converges around the 100 th training round, and the reward remains between 140-150. In comparison, the traditional DQN algorithm has an upward trend, but the convergence process is much slower than the H-DQN method proposed in this paper. This is due to the large action state space of the traditional DQN algorithm, while the H-DQN method is modeled through \begin{table}
\begin{tabular}{c c c} \hline \hline
**Symbol** & **Parameter Description** & **Value** \\ \hline \(\beta_{0}\) & Channel gain when the distance is 1 m & \(-\)50 dB \\ \(B\) & Channel bandwidth & 1 MHz \\ \(\sigma^{2}\) & Gaussian white noise power & \(-\)100 dBm \\ \(v^{u}\) & The flying speed of UAV & 20 m/s \\ \(a\) & Utility function parameter & 0.8 \\ \(w\) & Utility function parameter & 1 \\ \(p_{m}\) & Transmitting power of sensor equipment & 0.5 W \\ \(P_{0}\) & Blade contour energy consumption in hovering state & 99.66 W \\ \(P_{1}\) & Induction energy consumption in hovering state & 120.16 W \\ \(U^{trip}\) & Tip speed & 120 m/s \\ \(\delta\) & The time length of each time slot & 1 s \\ \(v_{0}\) & Average rotor-induced speed in hovering state & 0.002 m/s \\ \(d_{0}\) & Airframe drag ratio & 0.48 \\ \(s_{0}\) & Rotor stability & 0.0001 \\ \(A_{r}\) & Rotor area & 0.5 s\({}^{2}\) \\ \(E_{max}\) & The maximum energy that a drone can carry & 2 \(\times 10^{4}\) J \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The environmental parameters and the UAV parameters.