## 4 Proposed Trajectory Optimization Approach
### Proposed H-DQN Algorithm
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Number of training round episodes & 1000 \\ Target network update frequency C & 100 \\ Size of reply buffer \(B\) & 5000 \\ Mini-batch size \(K\) & 128 \\ Discount factor \(\gamma\) & 0.9 \\ The number of neurons in the first hidden layer & 128 \\ The number of neurons in the second hidden layer & 64 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The hyperparameters of the H-DQN. the SMDP and the option, which greatly reduces the action state space, making it easier to learn the optimal strategy. In addition, in the last 200 training rounds, the DQN method has a certain convergence trend, but the reward is still lower than the H-DQN method. This is because the rasterized flight mode of the DQN method causes more energy loss. The energy consumption weight in the reward function will affect the performance of the algorithm. Figure 4 represents the remaining energy change curve of the H-DQN algorithm when the energy consumption weight \(\beta_{e}=1,1.5,2\). The horizontal axis represents the decision sequence, and the vertical axis represents the current remaining energy consumption as a percentage of the initial energy consumption. It can be seen that when \(\beta_{e}=1\), the percentage of the lowest remaining energy is 2-10%,while 2% of the remaining energy is a relatively dangerous situation, which is close to energy exhaustion, and the safety factor is low, which may lead to the UAV not being able to complete the return flight, making it difficult to deal with emergencies. With the increase in \(\beta_{e}\), it can be seen that the charging frequency of the UAV also increases, and the remaining energy can also be maintained at a safer level. When \(\beta_{e}=1.5\), the remaining energy of the UAV is always maintained above 18%. When \(\beta_{e}=2\), the remaining energy of the UAV is basically maintained at more than 40%, but frequent charging may cause more QoS losses. As shown in Figure 5, when \(\beta_{e}=1\), the average QoS of the system is maintained at around 0.25, which is significantly higher than the QoS when \(\beta_{e}=1.5\) and \(\beta_{e}=2\). Therefore, users can set \(\beta_{e}\) according to the application requirements to achieve a tradeoff between the QoS and the safe power state. ### Comparing with Baselines

In this section, we test the effect of the H-DQN algorithm and other benchmark algorithms. The evaluation metric is the average QoS. As shown in (7), the QoS is a utility function about the AoI. The benchmark algorithms include the DQN algorithm and the only AoI selection method. The DQN algorithm uses the traditional MDP to model the path-planning problem, in which the target area is divided into \(20\times 20\) square sub-areas of equal area, and the center point of each sub-area is specified as the waypoint where the UAV can hover. The action space of the UAV is to move to an adjacent sub-area, and perform data collection tasks and charging tasks in this area. The instantaneous reward of the UAV is set to the AoI QoS only, and the energy reward of the time slot. In the AoI only method, each time the UAV selects the sensor device with the largest AoI in the current system for data collection, it then returns for recharging when the remaining power is less than a certain threshold. Figure 3: Convergence of the H-DQN method at different learning rates. Figure 6 describes the performance of the H-DQN and other benchmark algorithms with different numbers of sensing devices. It can be seen that when the number of ground sensors is 10, 20, and 30, the performance of the H-DQN algorithm is better than the other two benchmark algorithms. First of all, the selection algorithm based on the AoI is a greedy strategy, and the effect in long-term tasks is not as good as reinforcement learning methods, while reinforcement learning methods focus on long-term cumulative benefits.