# An Intelligent Cluster-Based Routing Scheme in 5G Flying Ad Hoc Networks
## 2 Related Work
Third, the target Q-network, which is characterized by target Q-network parameter (or weight) \(\tilde{\theta}\), provides the target Q-values \(Q(s^{t}_{i},a^{t}_{i};\tilde{\theta}_{i})\) used to establish a predefine estimated value for updating network parameter \(\theta\) while minimizing a loss function. Algorithm 1 shows the DQN algorithm. At each episode \(z\), an agent \(i\in N\) observes state \(s^{i}_{t}=(m^{i}_{t},e^{i}_{t})\) and feeds it into its DNN (Step 5). The agent \(i\) selects either an exploitation or an exploration action using the \(\varepsilon\)-greedy approach (Step 7). The decaying variable \(\varepsilon_{decay}\) helps to tend towards exploitation from exploration as episode increases. During exploitation, the agent selects a next-hop node based on the state \(s^{i}_{t}\). During exploration, the agent explores the possible actions. The output Q-value \(Q(s^{i}_{t},a^{t}_{i};\theta_{i})\) is selected based on policy \(\mu_{\theta}(s^{i}_{t},a^{i}_{t})\) (Step 8). At the next time instant \(t+1\), the agent \(i\) in CC observes the next state \(s^{i}_{t+1}\), which includes information from DCs, and receives a reward \(r_{t}(s^{i}_{t},a^{i}_{t},s^{i}_{t+1})\) (Step 9). The agent \(i\) stores this experience, and so it has experiences \(\{(s^{1}_{t},a^{1}_{t},r^{1}_{t},s^{1}_{t+1}),\ldots,(s^{k}_{t},a^{k}_{t},r^{k} _{t},s^{k}_{t+1})\}\) up to this time instant stored in the _replay memory_\(R_{mem}\) (Step 10). A _mini batch_ of experience samples \((s^{t}_{t},a^{t}_{t},r_{t},s^{l}_{t+1})\) are selected randomly from the replay memory \(R_{mem}\) to train the DNN (Step 11). The mini batch of samples has two characteristics: (a) _independent_ because the samples are selected randomly to calculate a

Figure 3: An overview of DQN. Main network, characterized by main network parameters \(\theta_{i}\), provides Q-value \(Q(s^{t}_{i},a^{t}_{i};\theta_{i})\) that defines the behavior policy. Target Q-network, characterized by target network parameters \(\tilde{\theta}_{i}\), generates target Q-value \(Q(s^{i}_{i},a^{t}_{i};\tilde{\theta}_{i})\). Both \(Q(s^{t}_{i},a^{t}_{i};\theta_{i})\) and \(Q(s^{t}_{i},a^{t}_{i};\tilde{\theta}_{i})\) are used to calculate a loss function \(L_{i}(\theta_{i})\) minimized using the gradient descent approach during training. Experiences \((s^{k}_{t},a^{k}_{t},r^{k}_{t},s^{k}_{t+1})\) are stored in the replay memory and they are used during training. A mini batch of random experiences from the replay memory are fed into the main and target networks. The DQN agent is incorporated in CC as shown in Figure 1.

desired target function \(y_{j}\) (i.e., pre-estimated value for training) (Step 13), which is then used to update the network parameter \(\theta\) using gradient descent based on a loss function \(y_{j}-Q(s^{i}_{t},a^{i}_{t};\theta)\) (Step 14), and (b) _stable_ because real experiences are used to update network parameters. In the gradient descent approach, the loss function is differentiated with respect to \(\theta_{i}\) using \(\
abla_{\theta_{i}}L_{i}(\theta_{i})=[(y_{i}-Q(s^{i}_{t},a^{i}_{t};\theta_{i} ))\
abla_{\theta_{i}}Q(s^{i}_{t},a^{i}_{t};\theta_{i})]\), and this process is repeated until it reaches the minimum value of the loss function (Steps 15 and 16). The target network parameters \(\bar{\theta}\) is copied from the main network parameters \(\theta\) every \(C\) steps, specifically \(\bar{\theta}=\theta\) (Step 17). The CC receives short-lifetime data (i.e., neighboring nodes of the source and destination UAVs) from DCs regularly to form a global NS, and establishes a route, which consists of intra- and inter-cluster, as well as intra- and inter-plane communications, between a source UAV and a destination UAV. ### Reinforcement Learning

Reinforcement learning (RL), as shown in Algorithm 2, is also embedded in CC for comparison.