# Deep-Reinforcement-Learning-Based Intelligent Routing Strategy for FANETs
## 3 Reinforcement Learning  
Reinforcement learning is different from other machine learning methods in the process of model training [29; 30; 31; 32]. Its agent relies on the interactive process with the environment to update the evaluation of decision-making actions under different environmental states (see Figure 2). The optimal strategy \(\pi^{*}\) can be acquired in a process of continuous interaction, with which the reward \(R\) is maximized, i.e.,  
\[R{=}\sum_{t=0}^{\infty}\gamma^{t}r_{t+1} \tag{1}\]  
where \(r_{t}\) and \(\gamma^{t}\), respectively, denote the instantaneous reward and decay value of the future reward at time \(t\). Among the RL algorithms, Q-learning is commonly utilized. The action-value function of Q-learning can be mathematically expressed as [29; 33]  
\[Q^{*}(s,a){=}\mathrm{E}_{g^{*}}\left[r_{t+1}+\gamma\max_{a^{\prime}}Q^{*}(s^{ \prime},a^{\prime})|s_{t}=s,a_{t}=a\right] \tag{2}\]  
Figure 1: Schematic diagram of the routing model in the FANET. Blue, green, and brown circles represent the current node, one-hop neighboring nodes, and two-hop neighboring nodes, respectively. where \(s\) and \(a\) are, respectively, the current state and action, while \(s^{\prime}\) and \(a^{\prime}\) denote the next state and action, respectively. Therefore, \(Q^{*}(s,a)\) can be understood as the expected reward when taking action \(a\) under state \(s\). Considering that it is difficult to obtain the optimal strategy \(\pi^{*}\) straightforwardly, the optimal strategy \(\pi^{*}\) is obtained through updating the action-value function \(Q^{\pi}(s,a)\) continuously in Q-learning. Concretely, the following rule is utilized to update the Q-table which stores the action-value function  
\[Q^{\pi}\!(s,a)\!\leftarrow\!\!Q^{\pi}\!(s,a)\!+\!\!\alpha\left(r(s,a)\!+\!\! \gamma\max_{a^{\prime}}Q^{\pi}\!(s^{\prime},a^{\prime})\!-\!\!Q^{\pi}(s,a)\right) \tag{3}\]  
where \(r(s,a)\) denotes the reward for performing action \(a\) under state \(s\), \(\alpha\) represents the learning rate, and \(\max\limits_{a^{\prime}}Q^{\pi}(s^{\prime},a^{\prime})\) means the max \(Q\) value under the next state \(s\). To balance the exploration and exploitation processes in the continuously interactive Q-learning method, and to avoid the learning process obtaining a suboptimal solution with unsatisfactory performance, the \(\varepsilon\)-greedy algorithm is commonly adopted, which can be described as  
\[a=\left\{ \begin{array}{c}\text{randomly action with probability}\quad \varepsilon\\ \arg\max_{a\in A}Q^{\pi}(s,a)\text{with probability}\ \ (1-\varepsilon)\end{array} \right. \tag{4}\]  
Although the Q-learning method based on a Q-table has the advantages of an intuitive principle logic, simplicity, and effectiveness, the method becomes extremely inefficient when the state space or action space is huge, as the increase of storage entries significantly reduces the efficiency of reading from and saving to the table. The deep reinforcement learning (DRL) [34] avoids this problem to a great extent, it introduces a deep learning that is based on a deep neural network (i.e., Deep Q Network, DQN) to perceive the logical relationship between the environmental state and agent action, which ensures that DRL does not need to traverse all state-action pairs as in Q-learning. The typical realization of DRL is the DQN algorithm that updates the action-value function through  
\[Q^{*}(s,a)\approx Q(s,a|\boldsymbol{\theta}) \tag{5}\]  
where \(\boldsymbol{\theta}\) denotes the parameter matrix of the neural network. In order to avoid the performance turbulence of the DQN in the training process, both the main network and the target network are adopted.