# An Intelligent Cluster-Based Routing Scheme in 5G Flying Ad Hoc Networks
## 2 Related Work
For instance, a lower rate of link breakages improves network stability and lifetime because UAVs can transmit data over their respective routes for a longer time duration without incurring clustering and routing overheads, leading to a lower packet loss rate, a higher throughput and a lower end-to-end delay. ### Analysis

This section presents an analysis of the two main approaches, namely, RL and DQN, via simulation. The analytical outcomes presented in this section help to analyze the effects of various parameters (i.e., learning rate \(\alpha\), exploration rate \(\varepsilon\), and decaying variable \(\varepsilon_{decay}\)) to the learning capability of RL and DQN [60, 61, 62]. The best possible parameters are identified, and simulations are performed based on these parameters for performance comparison between RL and DQN. In this research, the \(\tau\) value is set to a single episode \(z\), and so long-lifetime data does not change within episode \(z\). #### 5.6.1 RL

Figures 4 and 5 present the analytical results of the RL approach based on the state, and reward representations of our model (see Section 4.2). As the learning rate increases from \(\alpha=0.1\) to \(\alpha=1.0\), energy consumption reduces (see Figure 4a), the rate of link breakages reduces (see Figure 4b), and the network lifetime increases (see Figure 4c) with network density. At learning rate \(\alpha=1.0\), energy consumption and the rate of link breakages are the lowest, and the network lifetime is the highest. This is because a higher learning rate enables UAVs to (a) use delayed reward based on recent geographical location of UAVs, which helps to form links between UAVs in a highly dynamic environment in which UAVs are moving at high speed, and (b) select routes with higher residual energy and lower mobility in order to enhance network lifetime. Similar trend is observed in our investigation with respect to mobility as shown in Figure 5. In Figure 4(a), energy consumption increases significantly due to an increased packet re-transmission as a result of packet loss when mobility reaches 80 m/s. Figure 4: _Cont_. Figure 4: Graphical results for the effects of network density on RL. Learning rate \(\alpha=1.0\) achieves the lowest energy consumption and rate of link breakages, as well as the highest network lifetime. Figure 5: _Cont._

#### 5.6.2 DQN

Figures 6 and 7 present the analytical results of DQN. The state, action, and reward representations of our model are presented in Section 4.2. As the learning rate increases from \(\alpha=0.0001\) to \(\alpha=0.001\), energy consumption reduces (see Figure 5(a)), the rate of link breakages reduces (see Figure 5(b)), and the network lifetime increases (see Figure 5(c)) with network density. At learning rate \(\alpha=0.001\), energy consumption and the rate of link breakages are the lowest, and the network lifetime is the highest. This is because a higher learning rate enables UAVs to (a) use delayed reward based on experiences from replay memory \(R_{mem}\). These experiences (\(s_{t}^{i}\), \(a_{t}^{i}\), \(r_{t}^{i}\), \(s_{t+1}^{i}\)) contain current state values and next state values, which are important to select a favorable route in a dynamic environment. Experiences generated using higher learning rates provide better learning for DNN as compared to experiences generated using lower learning rates because these experiences contain more recent and fresher experiences; (b) use decaying exploration rate \(\varepsilon_{decay}\) that tends towards exploitation from exploration as the number of episodes increases; and (c) represent the recent angle of arrival as state, which is trained using mini batches from replay memory \(R_{mem}\), which helps in convergence towards the most favorable route (i.e., the route with a higher residual energy and a lower mobility) in order to enhance network lifetime. Figure 5: Graphical results for the effects of mobility on RL. Learning rate \(\alpha=1.0\) achieves the lowest energy consumption and rate of link breakages, as well as the highest network lifetime.