# An Intelligent Cluster-Based Routing Scheme in 5G Flying Ad Hoc Networks
## 2 Related Work
By considering the long-term reward, DQN may not change its selection of actions (or policy) after every single variation in the network. This is because the best possible action may remain optimal from the long-term perspective; specifically, it continues to achieve the highest state-action value compared to the rest of the potential actions. Therefore, nodes carrying real-time data can still select optimal action, which is the forwarding decision, based on its state-action values while avoiding the delay incurred in receiving forwarding decisions from the CC. * The DQN agent represents two aspects of mobility, namely speed (which is the short-lifetime data) and direction or predictable mobility paths (which is the long-lifetime data), as the state, and so it learns the predictable mobility patterns of UAV nodes. This helps to reduce the rate of link breakages (i.e., disconnectivity) between nodes. ### DQN-Based Vertical Routing Scheme

DQN is embedded in CC (or agent). The CC contains global information and selects a favorable route from a source UAV to a destination UAV based on routing metrics, including the mobility and residual energy of UAVs, in order to prolong route lifetime. Figure 1 depicts the access network. The entities that an agent interacts with are external to the agent, and they are conveniently called the operating environment. In Figure 3, DQN, which is embedded in CC, is applied to the operating environment for improving network lifetime, as well as reducing energy consumption and link breakages. There are three main representations in an agent. First, _state_ represents the decision making factors. The state of an agent \(i\) at time \(t\) is \(s_{t}^{i}=(m_{t}^{i},e_{t}^{i})\in S\), where \(m_{t}^{i}\in M\) represents mobility, and \(e_{t}^{i}\in E_{r}\) represents the residual energy level. Second, _action_ affects the reward under the state. The action of an agent \(i\) at time \(t\) is \(a_{t}^{i}\in A=\{x_{M,E_{r}}^{h}\in\mathfrak{X}\}\), which represents the selection of a next-hop node out of a set of available next-hops nodes \(\mathfrak{X}\) towards the destination node. The next-hop node can be a CH, CM, CG, VCG, or BS. A route cannot be established when there is a lack of an available next-hop, which is considered a link breakage. Third, _delay reward_ represents the performance measures. The delayed reward received by an agent \(i\) at time \(t\) is \(r_{t}(s_{t}^{i},a_{t}^{i},s_{t+1}^{i})=w(r_{t}^{i,j})+(1-w)(c_{t}^{i,j})\), where the weight factor is \(0\leq w\leq 1\), and both \(r_{t}^{i,j}\) and \(c_{t}^{i,j}\) are normalized to \([0,1]\). Therefore, the delayed reward has two components: (a) \(r_{t}^{i,j}\) represents the successful transmission rate of packets towards the destination from node \(i\) to node \(j\) at time \(t\) when both nodes are moving; and (b) \(c_{t}^{i,j}\) represents the network congestion level between nodes \(i\) and \(j\). The delayed reward helps a node \(i\) to find a stable route to increase the route lifetime and data traffic offload from macro-plane to pico- or femto-plane. Figure 3 presents an overview of DQN and its composition. DQN is a value function-based approach that estimates the Q-value of its possible actions. A DQN uses a deep neural network (DNN), which is characterized by network parameter (or weight) \(\theta\), to approximate an action-value function (or Q function) [43]. In our proposed solution, DQN is embedded in the CC. In addition, DQN has another three important components. First, replay memory is used to store experiences for training DNN. Second, the main network, which is characterized by the main network parameter (or weight) \(\theta\), provides Q-values \(Q(s^{t}_{i},a^{t}_{i};\theta_{i})\) that defines the agent's policy.