# A Reinforcement Learning Routing Protocol for UAV Aided Public Safety Networks
## 1 Introduction  
Man-made disasters such as terrorism can result in both the loss of life and critical infrastructure. It is estimated that an underdeveloped country like Pakistan incurred direct losses of 127 billion dollars in the last 17 years or so due to terrorism [1]. In addition, the attacks like Army Public School (APS) Peshawar, in which 150 students lost their lives, left a huge social and psychological impact on society [2]. This event has resulted in the complete overhaul of the security infrastructure and caused indirect losses to the economy. Numerous other terrorist incidents such as on the Pakistan Navy Ship (PNS) Mehran, and General Headquarter (GHQ) are examples in which terrorists attacked a building, took hostages, and/or destroyed critical infrastructure. In these scenarios, to disrupt coordination, communication/cellular infrastructure is often taken out either by the authorities or terrorists. The on-scene devices carried by the trapped victims are unable to communicate to first responders or law enforcement agencies. The information (location and number of victims, audio, video, and images)captured by on-scene available devices can provide timely information to first responders for carrying out a coordinated rescue operation [3,4]. In these situations, a Device to Device (D2D) network provides an alternative method of communication and connectivity among devices [5,6]. The D2D network along with the presence of an Unmanned Aerial Vehicle (UAV) can ensure the information collected by the devices to reach the Command Center (CC). This situation is depicted in Figure 1. Since the transmission power of the devices are limited, it may be possible that some of the devices are unable to reach the UAV. It therefore becomes imperative to cluster the devices. Clustering is a process in which the network is divided into small substructures called clusters, based on node degree, mobility, weights, etc., [7]. This small substructure consists of Cluster Members (CMs) and a Cluster Head (CH). In the network, the CMs communicate with their respective CHs to forward their data to the UAV. The CHs can also rely on each other in forwarding data to the UAV and subsequently to the CC. Multiple clustering techniques are studied in the literature based on different applications [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]. For example, the Link Cluster Algorithm (LCA) introduces the concept of a Gateway (GW) node to provide better connectivity among neighboring CHs [8]. However, this scheme is unstable due to numerous ID exchanges between nodes. The Least Cluster Change (LCC) [9] reduces the cost of re-clustering, which gives stability to the clusters. In [10], the authors proposed a new underlay clustering-based D2D network for the partial and disconnected network. It forms dynamic clusters using adhoc base stations or mobile devices. In [11], low frame-sized beacons signals are broadcasted by CHs to decrease signaling overhead. CMs are declared based on the SINR values. The \(\alpha\)-Stability Structure Clustering algorithm (\(\alpha\)-SSCA) is proposed in [12] in which CH is selected based on a score function calculated by the exchange of hello messages between neighboring nodes. The concept of quasi clusters, a special cluster within a cluster, are introduced in [13] to help in reducing transmission power resulting in a longer network lifetime. In [15], the authors proposed a new clustering scheme in which first clusters are formed by dividing the area into multiple small partitions and then selecting the CH in those partitions. These will be selected based on energy. To prolong network lifetime and to avoid a blind spot issue in the scheme, re-clustering is performed when the energy of the existing CH reaches a certain threshold. This scheme suffer from scalability issues. In our earlier work [16], we compared different basic clustering schemes such as Clustering without GW (CG), Clustering with GW (CWG), and No Clustering (NC) in terms of throughput and energy.