# Deep-Reinforcement-Learning-Based Intelligent Routing Strategy for FANETs
## 4 Proposed Intelligent Routing Method
### State Space  
As the input of the DRL model, the state should be able to objectively and comprehensively reflect the environment of the agent (i.e., the current UAV). In order to enable the agent to grasp more information about the environment, this paper adopted four routing-related parameters of neighboring nodes within two hops of the current node as the composition of the state space, which are introduced below. The signal-to-interference-noise ratio (SINR) can well reflect the quality of the channel, and this paper utilized \(\eta_{c,i}^{I}(t)\) and \(\eta_{i,j}^{II}(t)\) to represent the SINR of the channel between the current node \(c\) and one-hop node \(i\), and the SINR of the channel between one-hop node \(i\) and two-hop node \(j\) at time \(t\), respectively. We have:  
\[\eta_{c,i}^{I}(t) =\frac{g_{c,i}(t)p_{c,i}(t)}{\sigma_{i}^{2}(t)} \tag{7}\] \[\eta_{i,j}^{II}(t) =\frac{g_{i,j}(t)p_{i,j}(t)}{\sigma_{j}^{2}(t)} \tag{8}\]  
where \(g_{c,i}(t)\) and \(g_{i,j}(t)\) denote the average channel gain of the two links (i.e., \(c\to i\) and \(i\to j\)), respectively, \(p_{c,i}(t)\) and \(p_{i,j}(t)\) are the transmission power of node \(c\) and node \(i\), respectively, while \(\sigma_{i}^{2}(t)\) and \(\sigma_{j}^{2}(t)\) represent the variance of a Gaussian white noise at routing node \(i\) and node \(j\), respectively. For the sake of clarity, the time identification in the equations is omitted in the following part of this paper. We could then acquire the channel capacity by the Shannon formula  
\[C_{c,i}^{I} =B_{c,i}\text{log}_{2}\Big{(}1+\eta_{c,i}^{I}\Big{)} \tag{9}\] \[C_{i,j}^{II} =B_{i,j}\text{log}_{2}\Big{(}1+\eta_{i,j}^{II}\Big{)} \tag{10}\]  
with \(B_{c,i}\) and \(B_{i,j}\) denoting the available bandwidth of the two links, respectively. In order to fairly compare the potential of different candidate nodes, we took the channel capacity ratio as the part of the state, which could be, respectively, denoted as  
\[\tilde{C}_{c,i}^{I} =\frac{C_{c,i}^{I}}{\sum\limits_{n\in\mathbf{N}_{\ell}}\tilde{C}_ {c,n}^{I}/\big{|}\mathbf{N}_{\mathcal{C}}|} \tag{11}\] \[\tilde{C}_{c,i}^{II} =\frac{|\mathbf{M_{c}}-\mathbf{M_{c}}\cap\mathbf{N}_{\mathcal{C}}|\times \sum\limits_{j\in(\mathbf{N}_{\mathbf{i}}-\mathbf{N}_{\mathcal{C}}|\times\mathbf{ N}_{\mathcal{C}})}C_{i,j}^{II}}{|\mathbf{N}_{\mathbf{i}}-\mathbf{N}_{\mathcal{C}}| \times\sum\limits_{n,m\in(\mathbf{M_{c}}-\mathbf{M}_{\mathcal{C}}\cap\mathbf{N}_{ \mathcal{C}})}C_{n,m}^{II}} \tag{12}\]  
where \(\mathbf{N}_{i}\) represents the set of one-hop neighboring nodes for node \(i\), which is a single-hop neighbor node of \(c\), while \(|\cdot|\) means the cardinality of set \"\(\cdot\)\". Note that those nodes simultaneously belonging to one hop were excluded when calculating the ratio of two-hop nodes. The second element of the state set in the proposed method was distance, which plays a key role in almost all location-based routing algorithms. With the positioning ability,the distance between nodes could be easily calculated. The following distance ratios were adopted in DRL-FIR:  
\[\bar{D}_{c,i}^{I} =\frac{D_{c,i}^{I}}{\sum\limits_{n\in N_{c}}D_{c,n}^{I}/\left|N_{c} \right|} \tag{13}\] \[\bar{D}_{c,i}^{II} =\frac{\left|\mathbf{M_{c}}-\mathbf{M_{c}}\cap\mathbf{N_{c}}\right|\times \sum\limits_{ \begin{subarray}{c}j\in(\mathbf{N_{i}}-\mathbf{N_{i}}\cap\mathbf{N_{c}})\\ \left|\mathbf{N_{i}}-\mathbf{N_{i}}\cap\mathbf{N_{c}}\right|\times\sum\limits_{n,m\in(\mathbf{ M_{c}}-\mathbf{M_{c}}\cap\mathbf{N_{c}})}D_{n,m}^{II}\end{subarray} } \tag{14}\]  
where \(D_{c,i}^{I}\) and \(D_{i,j}^{II}\) denote the distance between the current node \(c\) and the one-hop node \(i\), and the distance between node \(i\) and the two-hop node \(j\), respectively.