# An Intelligent Cluster-Based Routing Scheme in 5G Flying Ad Hoc Networks
## 2 Related Work
At time \(t\), an agent \(i\) in CC observes state \(s^{i}_{t}\), which includes information from DCs, and selects a random action \(a^{i}_{t}\) (during exploration) or an optimal action \(a^{i,*}_{t}\) (during exploitation) as follows (Step 6):

\[a^{i,*}_{t}=\operatorname*{argmax}_{a\in A}Q^{i}_{t}(s^{i}_{t},a) \tag{1}\]

The agent \(i\) receives a positive or negative delayed reward \(r^{i}_{t+1}(s^{i}_{t+1},a^{i}_{t+1})\) from the operating environment at the next time instant \(t+1\) (Step 8). The agent \(i\) explores each possible combination of state-action pair to update its Q-values with respect to time \(t=1,2,\dots\) using Equation (2) as follows (Step 9):

\[Q^{i}_{t+1}(s^{i}_{t},a^{i}_{t})=(1-\alpha)Q_{t}(s^{i}_{t},a^{i} _{t})+\alpha[r_{t+1}(s_{t+1})\] \[+\gamma\operatorname*{argmax}_{a}Q_{t}(s^{i}_{t+1},a)] \tag{2}\]

where learning rate is \(0\leq\alpha\leq 1\) and discount factor is \(0\leq\gamma\leq 1\). ## 5 Performance Evaluation, Results, and Discussion

DQN-based vertical routing scheme, which is embedded in the CC, selects a route with low energy consumption and mobility rate from a source node to a destination node. The proposed scheme reduces energy consumption and the rate of link breakages, and increases network lifetime. Ultimately, it contributes to a higher network stability. The main focus of research on routing schemes in FANETs has been focusing on catering to the dynamicity of UAVs [29]. Swarm intelligence based algorithms have been used for clustering to ensure scalability [30; 31]. To the best of our knowledge, there is no routing scheme designed for FANETs in the context of 5G access networks in the literature. 5G access networks contain controllers (i.e., CC and DCs) and network planes (i.e., macro-, pico-, and femtolplanes), which are unique compared to traditional access networks. Moreover, the proposed vertical routing approach for multiple network planes in 5G access networks is first of its kind. As there is lack of state-of-the-art vertical routing schemes for comparison with our proposed scheme, the traditional RL, random, and optimal approaches are selected. These approaches are chosen because the optimal approach provides the best possible results, and the random approach provides the worst possible results. The DQN approach is compared with its predecessor, namely, RL, and it is investigated with different learning rates. The learning rate of RL is a hyperparameter that controls how quickly the RL approach adapts to the dynamicity of the operating environment. DQN has shown to outperform RL. Similar to [44], UAVs are deployed following the distributive motion, so we consider three-dimensional predictable motions with uniformly distributed directions and random velocity that after an iteration. The source and destination UAVs are selected randomly [45]. The rest of this section explains our simulation platforms, baseline and optimal approaches, simulation parameters, performance measures, analysis, simulation results and discussions, and complexity analysis. ### Simulation Platforms

Simulation is performed using MATLAB (i.e., version 2019b) [46] and Python (i.e., version 3.6) [47], which are the preferred tools for similar investigations in the literature [48, 49]. MATLAB (or matrix laboratory) uses mathematical modeling to develop algorithms, compute large arrays and matrices, as well as accumulate and record statistics. In Python, deep learning is implemented using the Keras library [50] in the TensorFlow framework (i.e., version 1.1). RL algorithms are implemented and compared using the _Gym_ toolkit [47], and the network topology that consists of flying nodes are implemented using the _Nx_ toolkit [51]. ### Baseline and Optimal Approaches

Our proposed scheme is compared with two approaches. First, in the _random approach_, which serves as the baseline approach, an agent \(i\in N\) selects and takes a random action from a set of potential actions (i.e., a set of available next-hop nodes) at all times.