# Deep-Reinforcement-Learning-Based Intelligent Routing Strategy for FANETs
## 4 Proposed Intelligent Routing Method
### Other Details  
The Residual Network (ResNet) which can effectively avoid the problem of degradation of traditional network structures was adopted in the proposed DRL-FIR. The Q values were estimated by an 8-layer ResNet. Moreover, the Adam optimizer and Relu activation function were employed in the training process of the ResNet. The input and outputs of the network, respectively, corresponded to the summative state of the current node and the Q values of the one-hop nodes.  
Algorithm 1 summarizes the proposed intelligent routing strategy. The algorithm mainly shows the training process of the model of the proposed method. In order to avoid violent fluctuations in the process of training, the algorithm adopted the dual neural network (Step 12), replay memory (Step 9), and small batch learning (Step 11) approaches. It is worth noting that the complexity of the proposed strategy is mainly in the training stage of the model, while the complexity of the testing or application stage is quite low, as the method only needs relatively simple forward linear and nonlinear calculations in these stages. Considering that the training of the proposed method can be carried out by ground equipment, and the UAV nodes should only record the historical status and action decisions, thus, the method is computationally feasible for FANETs.  
We note that the proposed DRL-FIR method is forward-compatible, which means that the model of the method can be applied to other scenes that are more simplified than the corresponding scene during training. For the latter, it only needs to set the parameters of the model to zero or replace them with simple values. Meanwhile, it is worth adding that the proposed method does not require the information of all the two-hop nodes around it, as the state set by the method contains relative values rather than absolute values. This ensures the practicality of the method to a great extent, considering that the link sustainability between nodes may be unstable in the dynamic FANETs. The method proposed in this paper is applicable to all nodes in FANETs. The current node obtains and calculates the asymmetry information of surrounding nodes for rapid decision-making, and then sends data packets to the next symmetric node until the data routing process is completed.  
```
1:Input: state \(\mathbf{S}\), action \(\mathbf{A}\), learning rate \(\alpha\), discount factor \(\gamma\), update frequency of the target network \(F_{t}\), the source UAV node, and the destination UAV node.
2:Initialize replay memory \(\mathbf{M_{R}}\), main network \(Q(s,a|\mathbf{\theta})\) with random weights \(\mathbf{\theta}\), target network \(Q(s,a|\mathbf{\theta^{\prime}})\) with weights \(\mathbf{\theta^{\prime}}=\mathbf{\theta},\varepsilon,\varepsilon_{decay}\), \(\varepsilon_{\text{min}}\), training start threshold \(Z_{s}\), \(n=0\), \(r_{-}=0\).
3:For\(k\gets 1,\dots,K\)Do
4:if\(\varepsilon>\varepsilon_{\text{min}}\)then\(\varepsilon\leftarrow\varepsilon\cdot\varepsilon_{decay}\)
5:For\(z\gets 1,\dots,Z\)Do
6:Generate a random number \(p\) from 0 to 1;
7:\(a\leftarrow\left\{\begin{array}{ll}\text{random select from }\mathbf{A}&0\leq p<\varepsilon_{\text{min}}\\ \text{arg max}_{k\in\mathbf{A}}Q(s,a|\mathbf{\theta})&\varepsilon_{\text{min}}\leq p<1 \end{array}\right.\);
8:Take action \(a\) and make state transition \(s\to s^{\prime}\); calculate\(r\gets R(s,a,s^{\prime})\);
9:Save\((s,a,r,s^{\prime})\) into replay memory \(\mathbf{M_{R}}\); \(r_{-}\gets r_{-}+r\);
10:if\(z>Z_{s}\)then
11: Randomly sample a batch of experiences from \(\mathbf{M_{R}}\); calculate\(\text{L}(\mathbf{\theta},\mathbf{\theta^{\prime}})\); update weights \(\mathbf{\theta}\); \(n\gets n+1\);
12:if\(n\) mod \(C=0\)then\(\mathbf{\theta^{\prime}}\leftarrow\mathbf{\theta},s\gets s^{\prime}\)
13:if\(s^{\prime}\) is the destination then Save \(r_{-}\) into action list.
14:End
15:End
```  
**Algorithm 1**The proposed DRL-FIR strategy.