# A Reinforcement Learning Routing Protocol for UAV Aided Public Safety Networks
## 3 Reinforcement Learning-Based Routing
### Routing  
For routing, the node that needs to transmit will forward its packet to its respective CH. The CH will then forward this packet to the destination based on its routing table \(H_{i,s_{i}}\). In this paper, we apply RL to update the entries of the routing table to improve EE. Similar to [51], we use a linear functional approximation for the cost function. The proposed cost function is defined as:  
\[Cost=\beta\bigg{(}\alpha\bigg{(}\frac{d_{mn}}{max(d_{mn})}\bigg{)}+(1-\alpha) \bigg{(}\frac{d_{n(\text{UAV}/\text{CC})}}{max(d_{n(\text{UAV}/\text{CC})})} \bigg{)}\bigg{)}+(1-\beta)\Big{(}\frac{E_{mn}}{max(E_{mn})}\Big{)} \tag{15}\]  
where, \(\alpha\) is the weighting factor for shortest distance to the UAV and the next hop and \(\beta\) is the weighting factor to provide a balance between the distance and energy, and \(m\) and \(n\) are sender and receiver respectively. \(d_{mn}\) is the distance between \(m\) and \(n\) while \(max(d_{mn})\) is the maximum distance between \(m\) and \(n\). Similarly \(E_{mn}\) is the transmission energy between \(m\) and \(n\) while \(max(E_{mn})\) is the maximum energy cost for the next hop. The above cost function consists of two sections, the distance and energy. This cost is only checked at a CH. The distance is balanced by \(\alpha\) which considers the weight of the next \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Scheme Name** & **Clustering Type** & **Clustering Overlapping** & **Gate Way** & **Location Awareness** \\ \hline Clustering-energy & Energy based & High & No & Not required \\ Clustering-energy-GW & Energy based & High & Yes & Not required \\ Clustering-k mean & K mean & Low & No & Required \\ Clustering-k mean-GW & K mean & Low & Yes & Required \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Comparison between clustering schemes. GW: Gateway. hop and the corresponding distance of the next hop to UAV/CC. Choosing a minimum distance path to the UAV/CC can cause significant load balancing issues. The energy consumption of forwarding CHs in such a path will also increase significantly. On the contrary, only choosing the minimum distance path for the next hop may not always be feasible as it can increase the number of hops or the distance to the UAV. The value of \(\alpha\) finds the tradeoff in terms of distance which is subsequently used by the parameter \(\beta\) to manage the tradeoff among immediate energy expenditure in the next hop and distance. For example, if a CH has connectivity to multiple CHs through which it can forward its data. The scaling factor \(\beta\) is proportionate to the immediate energy cost of the next hop and distance cost of different routes. The terms \(\max(E_{mn})\), \(\max(d_{mn})\) and \(\max(d_{nULAV/CC})\) normalize the energy and distance of all the possible routing paths in the next hop. Once the costs are calculated in the current epoch, the routing is carried out. All the steps will remain the same except after the cost function RL is applied by using the equation given below:  
\[RL_{mn}=(1-\gamma)RL^{{}^{\prime}}_{mm}+\gamma(Rw_{mn}+V(max(RL_{m^{\prime}n^{ \prime}}))). \tag{16}\]  
Here \(RL^{{}^{\prime}}_{mm}\) is the previous value of \(RL_{mn}\), \(Rw_{mn}\) is the reward obtained from communication which in this case is 0 for an unsuccessful transmission and 1 for a successful transmission, \(\gamma\) is the discount factor varying between \(0\leq\gamma\leq 1\) which tells us how much importance we want to give to the current rewards, and \(V\) is the learning rate varying between \(0\leq V\leq 1\) that tells us to what extent these RL values are updated after each iteration. \(max(RL_{m^{\prime}n^{\prime}})\) is the maximum calculated cost for the next hop. \((1-\gamma)RL^{{}^{\prime}}_{mm}\) takes a weight of the old \(RL^{{}^{\prime}}_{mm}\) value and then by adding the learned value which is the combination of \(Rw_{mn}\) and current \(max(RL_{m^{\prime}n^{\prime}})\). This means an action is taken after looking at the old, current, and future rewards as shown in Algorithm 1.