# An Intelligent Cluster-Based Routing Scheme in 5G Flying Ad Hoc Networks
## 2 Related Work
DDPG aims to achieve the optimal policy which has the highest accumulated reward, rather than the highest Q-value in each state, which may not be optimal; and (b) double DQN [27; 65] uses two identical neural networks, whereby one learns during experience replay, just like DQN does, and the other one is a copy of the last episode of the first network. It solves the overestimation of Q-value caused by selecting actions with the highest Q-values at all times. Addressing overestimation helps double DQN to converge to the most favorable route with reduced computational complexity. Third, a mini-batch from the replay memory consists of experiences with higher occurrences rather than distinctive experiences. This can cause overfitting in which DNN fits \"too well\" to the limited set of the training data, causing sub-optimal actions to be selected. Redundant experiences in a mini batch can be removed to address this so that distinctive experiences can be selected with equal chances. Fourth, further investigation can be carried out in different scenarios with different amounts of white spaces, types of terrain typologies, and types of obstacles (e.g., natural and human-made). Conceptualization, M.F.K., K.-L.A.Y., M.H.L., M.A.I. and Y.-W.C.; methodology, M.F.K., K.-L.A.Y., M.H.L., M.A.I. and Y.-W.C.; investigation, M.F.K. ; resources, K.-L.A.Y., M.H.L., M.A.I. and Y.-W.C.; data curation, M.F.K. ; writing--original draft preparation, M.F.K. ; writing--review and editing, K.-L.A.Y., M.H.L., M.A.I. and Y.-W.C.; visualization, M.F.K., K.-L.A.Y., M.H.L., M.A.I. and Y.-W.C.; project administration, K.-L.A.Y. ; funding acquisition, K.-L.A.Y. and M.H.L. All authors have read and agreed to the published version of the manuscript. This work was supported by the Malaysian Ministry of Education under Fundamental Research Grant Scheme through the \"A Novel Clustering Algorithm based on Reinforcement Learning for the Optimization of Global and Local Network Performances in Mobile Networks\" project with the grant number FRGS/1/2019/ICT03/SYUC/01/1. Data Availability Statement: Data sharing not applicable. The authors declare no conflict of interest. The following abbreviations are used in this manuscript:

5G Fifth generation. CC Centralized controller. CG Cluster gateway. CH Cluster head. CM Cluster member. D2D Device-to-device. DC Distributed controller. DNN Deep neural network. DQN Deep Q-network. DRL Deep reinforcement learning. Flying ad hoc networks. LET Link expiration time. QoS Quality of service. RL Reinforcement learning. UAVs Unmanned aerial vehicles. UE User equipment. VCG Vertical cluster gateway. ## References

* _[PERSON] et al. (2018)_ [PERSON]; [PERSON]; [PERSON] Machine learning and intelligent communications. _Mob. Netws. Appl._**2018**, _23_, 68-70. [CrossRef]
* _[PERSON] et al. (2020)_ [PERSON]; [PERSON]; [PERSON]; [PERSON] Survey and taxonomy of clustering algorithms in 5G. _J. Netw. Comput. Appl._**2020**, _154_, 201-221. [CrossRef]
* _[PERSON] et al. (2019)_ [PERSON]; [PERSON]; [PERSON] Completion time and energy consumption minimization for UAV-enabled multicasting. _IEEE Wirel. Commun. Lett._**2019**, \(8\), 821-824. [CrossRef]
* _[PERSON] et al. (2020)_ [PERSON]; [PERSON] [PERSON]; [PERSON]; [PERSON] Softwarization of UAV networks: A survey of applications and future trends. _IEEE Access_**2020**, \(8\), 98073-98125. [CrossRef]
* _[PERSON] et al. (2020)_ [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] UAV assistance paradigm: State-of-the-art in applications and challenges. _J. Netw. Comput. Appl._**2020**, _166_, 102706. [CrossRef]
* _[PERSON] et al. (2019)_ [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] ECaD: Energy-efficient routing in flying ad hoc networks. _Int. J. Commun. Syst._**2019**, _32_, e4156. [CrossRef]
* _[PERSON] et al. (2016)_ [PERSON]; [PERSON]; [PERSON] Ultra-dense networks: A survey. _IEEE Commun. Surv. Tutor._**2016**, _18_, 2522-2545. [CrossRef]
* _[PERSON] et al. (2018)_ [PERSON]; [PERSON] Routing in multi-hop cellular device-to-device (D2D) networks: A survey. _IEEE Commun.