# Deep Q-Learning for Two-Hop Communications of Drone Base Stations
## 3 System Model  
We assume that there is a closed square form environment. The Drone Base Station (DBS) starts above an arbitrary location in the environment to provide services to the ground user. A 2D mobility model is considered for the drone where it is flying around at a fixed altitude, \(h_{d}\), and a constant speed. The drone moves continuously, as a result, the proposed model can be applied to both fixed-wing and rotary drones. The fixed-wing drones move continuously and can not hold themselves up in the air, while the rotary ones are able to stop flying, stand still, and start to fly again. The user is moving in the target environment according to a mobility model. The user also follows a full buffer traffic model with continuous downloads.  
A Macro Base Station (MBS) is placed out of our considered environment at height \(h_{m}\), and distance \(r\) from the center of the service area. The MBS is only for backhaul purposes and the communication between the user and the DBS will be offloaded to the MBS. The system model is illustrated in Figure 1.  
Moreover, we suppose that the MBS transmits using a fixed transmission power of \(p_{m}\), with a frequency of \(f_{m}\). The DBS also communicates with a power of \(p_{d}\) at frequency of \(f_{d}\).  
The DBS aims to improve the network capacity for the mobile user taking into account the two-hop communication. Therefore, the DBS needs to adapt its trajectory autonomously according to the movements of the user and the location of MBS, in order to achieve its goal. In this paper, we propose Reinforcement Learning based models for the drone to learn its trajectory by finding a heading direction at each time step.  
Following [PERSON] theorem, the capacity (in bits/sec/Hz) of each link in our two-hop model is given by:  
\[\mathcal{C}_{d,u}=\log_{2}(1+\Gamma_{d,u}) \tag{1}\] \[\mathcal{C}_{m,d}=\log_{2}(1+\Gamma_{m,d}) \tag{2}\]  
where \(C_{d,u}\) and \(C_{m,d}\) represent the capacity of the access link and the backhaul link, respectively. The signal to noise ratio (SNR) of a transmitter and receiver would be as follows:  
\[\Gamma_{d,u}=\frac{p_{d}\times pl_{d,u}}{N_{0}} \tag{3}\] \[\Gamma_{m,d}=\frac{p_{m}\times pl_{m,d}}{N_{0}}, \tag{4}\]  
where \(pl\) is the path loss between the transmitter and the receiver and \(N_{0}\) is the power of the additive white Gaussian noise.  
We divide the whole operation time into \(T\) sequential equal time slots, where the overall two-hop capacity that can be delivered to the user at time slot \(t\in[1,...,T]\) is defined as the minimum of the access link and backhaul link capacity, as follows:  
\[\mathcal{C}^{t}=\min(C^{t}_{d,u^{\prime}},C^{t}_{m,d});\qquad t\in[1,...,T]. \tag{5}\]  
The drone must find a heading direction at each time slot in order to optimize the total two-hop capacity during its mission, hence, the problem is formulated as follows:  
\[\mathcal{A}^{*}=\operatorname*{argmax}_{x^{*}}\frac{\sum_{t=0}^{t=0}\big{(} \min(C^{t}_{d,u^{\prime}},C^{t}_{m,d})\big{)}}{T}, \tag{6}\]  
where \(A^{*}=\{{\alpha_{1}}^{*},...,{\alpha_{T}}^{*}\},t\in[1,...,T]\), is the set of drone's heading direction during the operation.  
We assume that the drone knows its position and the position of the user at any moment.