# An Intelligent Cluster-Based Routing Scheme in 5G Flying Ad Hoc Networks
## 2 Related Work
_Storage complexity_ defines the number of connections between a pair of neurons in DNN used for calculating the state-action values. In DQN (see Figure 3), the number of neurons in the \(n\)th layer is \(H_{n}\), and the number of layers is \(N\). Therefore, the storage complexity of the \(n\)th layer is \(O(H_{n-1}H_{n}+H_{n}H_{n+1}+H_{n+1}H_{n+2}+\cdots+H_{N-1}H_{N})\), which represents the storage required for connections between layers (i.e., input, hidden, and output layers), and so the storage complexity is \(O(\sum_{n=2}^{N}(H_{n-1}H_{n}))\) can be denoted as \(|H|\) for simplicity (see steps 10 and 11 in Algorithm 1). The agent-wise storage complexity is \(O(|S||A||H|)\). Thus, the network-wide storage complexity is \(O(|I||S||A||H|)\) in a network with \(|I|\) agents. Meanwhile, in the traditional RL approach (see Algorithm 2), the algorithm-wise complexity is 1 whenever an agent \(i\) stores the Q-value of a state-action pair (see step 9 in Algorithm 2). The agent-wise complexity is \(\leq(|S||A|)\) whenever an agent \(i\) updates its Q-values for all the state-action pairs. The network-wide complexity is \(O(|I||S||A|)\) in a network with \(|I|\) agents. ## 6 Conclusions

Deep Q-network (DQN), which is based on reinforcement learning (i.e., Q-learning) and deep learning (i.e., deep neural network (DNN)), enables an agent to select the best possible action under a particular state. This article presents an intelligent cluster-based routing scheme to improve network stability, network lifetime, and energy efficiency in 5G-based flying ad hoc networks. Our proposed scheme ensures the recency of data among central controller (CC) and distributed controller (DC) in order to achieve a balanced enhancement between global and local network performances. DNN enables agents to learn about states (i.e., residual energy and the mobility rate) of agents to predict state-action values. Mini batches of experiences are used for the run-time learning of DNN. There are three features that help to achieve a higher convergence rate towards the most favorable route with higher residual energy and lower mobility. First, the delayed reward, which is part of an experience from the replay memory is used to perform training in order to improve the prediction of state-action values in a dynamic environment. Second, the decaying variable \(\varepsilon_{decay}\) is used to tend towards exploitation from exploration as the number of episodes increases. Thirdly, mini batches of run-time values of states from the replay memory are used to training and minimizing a loss function. Our proposed scheme is compared with the traditional reinforcement learning and the random approaches and has shown to improve energy efficiency by up to 20% and 100% and network lifetime by up to 60% and 120%, and to reduce the rate of link breakages by up to 50% and 80%, respectively. ## 7 Future Work

Further research can be pursued to investigate the following open issues. First, vertical routing can be further enhanced to reduce routing overhead incurred to establish and maintain inter- and intra-cluster, and inter- and intra-plane, routes by enabling clusters to adjust their cluster sizes for achieving the optimal number of nodes in a cluster (or the optimal cluster size). This helps the clusters to prolong the cluster lifetime for providing robust data transmission, as well as self-organize traffic load for achieving load balancing among themselves. With improved cluster stability and scalability, vertical routing is expected to improve its route stability. Second, other variants of DQN [63] can be adopted: (a) deep deterministic policy gradient (DDPG) [64] is an actor-critic approach that improves the stability of learning in continuous action space, which is preferred for our vertical routing approach with continuous action space.