## 4 Proposed Trajectory Optimization Approach
### Proposed H-DQN Algorithm
In addition, the performance of the H-DQN algorithm, based on the SMDP modeling, is also better than the traditional DQN method which is based on the MDP modeling. When the number of ground sensors is 10, 20, and 30, the performance of the H-DQN method is improved by 18.36%, 37%, and 21%, respectively, compared with the DQN method. This is because the H-DQN method, based on the SMDP modeling, uses the concept of time abstraction, which greatly reduces the state-action space, making it easier to learn better strategies. However, as the number of sensing devices increases, the performance of the H-DQN algorithm is better than the other benchmark algorithms. Figure 4: Change curve of residual energy consumption of the UAV. Figure 5: QoS change curve of the UAV-assisted data collection system. gradually declines. Therefore, when the number of sensing devices is large, the use of the multi-UAV collaboration for data collection should be considered. ## 6 Conclusions

In this paper, we have studied the path-planning problem in UAV-assisted data collection. In order to enable UAVs to perform long-term data collection tasks in dynamic scenarios with a constantly changing AoI and their own power levels, we propose a method based on the QoS maximization problem of the AoI, and we use a semi-Markov decision process (SMDP) to describe this problem. In addition, we also propose a path-planning algorithm based on hierarchical DQN to learn the optimal strategy, and conduct simulation experiments to verify the convergence of the proposed algorithm. We test the effect of the energy consumption weight \(\beta_{e}\) on the remaining power and the QoS influence. Experiments show that the tradeoff between the system QoS and the safe power state can be achieved by adjusting the parameters \(\beta_{e}\). At the same time, it is compared with two benchmark algorithms, i.e., the DQN and the AoI only algorithm. The results show that this algorithm is better than the other two benchmark algorithms. In the H-DQN method in this paper, the bottom-level strategy option uses the established strategy. In future work, further improvements can be made, such as using the DQN for high-level strategies and the DDPG for low-level strategies. Conceptualization, Z.Q. and X.Z. ([PERSON]); methodology, Z.Q., X.Z. ([PERSON]) and B.L. ; validation, Z.Q., X.Z. ([PERSON]) and X.Z. ([PERSON]); formal analysis, B.L. ; investigation, Z.L. and L.G. ; resources, B.L. ; data curation, Z.L. ; writing--original draft preparation, Z.Q. ; writing--review and editing, X.Z. ([PERSON]); visualization, L.G. ; project administration, B.L. and L.G. All authors have read and agreed to the published version of the manuscript. The work was supported by \"National Natural Science Foundation of China\" with No. 61902052, \"Science and Technology MajorIndustrial Project of Liaoning Province\" with No. 2020 JH1/10100013, \"Dalian Science and Technology Innovation Fund\" with No. 2019J11 CY004 and 2020 JJ26 GX037, and \"the Fundamental Research Funds for the Central Universities\" with No. DUT20 ZD210 and DUT20 TD107. Not applicable. The relevant data of simulation experiment can be found in [[https://github.com/wilna-lab/H-DQN](https://github.com/wilna-lab/H-DQN)]([https://github.com/wilna-lab/H-DQN](https://github.com/wilna-lab/H-DQN)). Figure 6: Comparison of the H-DQN and benchmark algorithms. ## Conflicts of Interest: The authors declare no conflict of interest. ## References

* [PERSON] et al. (2016) [PERSON]; [PERSON]; [PERSON] Wireless Communications with Unmanned Aerial Vehicles: Opportunities and Challenges. _IEEE Commun. Mag._**2016**, _54_, 36-42. * [PERSON] et al. (2019) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] Fundamental Tradeoffs in Communication and Trajectory Design for UAV-Enabled Wireless Network. _IEEE Wirel. Commun._**2019**, _26_, 36-44. * [PERSON] et al. (2017) [PERSON]; [PERSON]; [PERSON] [PERSON] Coverage, Capacity and Interference Analysis for An Aerial Base Station in Different Environments.