# Deep Q-Learning for Two-Hop Communications of Drone Base Stations
## 1 Introduction  
Recent advancements in Unmanned Ariel Vehicles (UAVs) or Drones, along with the development of communication technology, have changed and led to the emergence of various civilian applications such as emergency medical equipment deliveries [1], real-time monitoring in mining environmental [2], precision agriculture and crop monitoring [3] and traffic monitoring in smart cities [4]. Moreover, one of the most important applications of drones is in cellular networks, where the drones can act as a flying base station to enhance coverage and improve the quality of communication services for the ground users.  
Despite all the advantages that can be brought into a cellular network by drones, they still face many challenges for integrating and operating in networks. Both research and industrial communities showed interest in working on overcoming these challenges. Multiple study items on the integration of UAVs into future cellular networks have been released recently by the 3 rd Generation Partnership Project (3 GPP) [5, 6]. Various features of integration of drones such as deployment, trajectory planning, power consumption, security, and so forth are studied by many researchers as well ([7, 8, 9]).  
One of the main important challenges in the application of drones is providing backhaul link and ensuring its performance. The backhaul link is established between a drone base station (DBS) and a macrocell BS, where the the access link is between a user and the DBS. Although the efficiency of the backhaul link must be considered in the drone deployments, most of the existing works ignored this issue and have only focused on the access link performance. That is why, in this work, we study a two-hop communication model (backhaul link and access link) to improve the system's performance. We consider a practical scenario where a flying DBS is serving mobile users in the environment through a two-hop communication model. Q-learning and Deep Q-learning based algorithms are proposed to optimize the navigation of the drone. The performance of the proposed algorithms is evaluated in different network conditions. Additionally, a simple algorithm that targets maximizing the access link capacity is studied in order to point out the importance of the proposed algorithms and the two-hop communication model.  
In our previous work [10], we modeled the network as a simple reinforcement learning (RL) problem, in which each drone base station tries to learn its navigation assuming that there is a high probability for the other agents to follow their previous direction. In this work, we have considered a complex model, where the drone learns its trajectory based on Q-leaning (QL) and Deep Q-learning network (DQN), taking into consideration the mobility of the ground user. Our contributions are summarized as follows:  
* Considering two-hop communication ( i.e., the access link and the backhaul link) to optimize the drone base station trajectory and to improve network performance,
* Designing Q-learning and Deep Q-learning models to solve the joint two-hop communication scenario.
* Taking into account the mobility of the ground user  
The rest of the paper is structured as follows--Section 2 reviews the related work, and Section 3 presents the system model and the problem formulation. Then the Q-learning and Deep Q-learning concepts are briefly described in Section 4. Section 5 introduces the proposed drone mobility algorithms. Section 6 presents the performance evaluation and simulation results, and the conclusion is provided in Section 7.