# Deep Q-Learning for Two-Hop Communications of Drone Base Stations
## 6 Simulation and Results  
To evaluate the performance of the proposed algorithms, a square area with a side length of 240 m, composed of 144 square-cells are considered in our simulation, each cell having a length of 20 m. There is one mobile user in the environment, and one DBS to serve the user. The MBS is located out of the environment at the location of [1000 m, 1000 m]. The user is moving according to the Random Way Point (RWP) model [35] with the speed range of [0.2-5] m/s. The drone located above a cell can move towards any of its neighbor cells by choosing any of eight different directions. However, as mentioned earlier, any invalid movement towards outside of the environment border will be penalized and will keep the drone above its current location. Additionally, we assume that the drone and the macro base station work at different frequencies, and there is no interference between their transmissions. All the parameters regarding the network model are shown in Table 1.  
For the Q-learning and DQN model, we have trained the model with different parameters, as listed in Table 2 in order to obtain Q-values in the training process. The agent is trained during 2000 episodes, each of them composed of 1000 transitions. Afterwards, the trained model is evaluated in both Q-learning and DQN model over 500 independent missions of 1000-transitions. Note that the number of transitions reflects the lifetime of a drone. During its lifetime the drone navigates over the environment in order to improve the user performance. In DQN, an experience replay buffer of size 5000 event is considered and the batch size is 64. Additionally, the parameters of the target network will be updated at each 25 intervals. As mentioned above, the network consists of three dense hidden layers of 128, 128, and 64 neurons, respectively. In the following, we first review the training process of Q-learning and DQN. Figure 3 shows the epsilon decay strategy where the models select the best action through exploitation first, and gradually the exploration rate increases to benefit the learned Q-values. The same epsilon decay strategy are applied to both Q-learning and DQN models. \begin{table}
\begin{tabular}{l l} \hline \hline
**Parameters** & **Value** \\ \hline Drone Height & 20 m \\ MBS Height & 30 m \\ Edge Length of a Square Cell & 20 m \\ Are size & 240 m \\ Drone Working Frequency & 2 GHz \\ MBS Working Frequency & 1.8 GHz \\ Drone Transmission Power & 30 dBm \\ MBS Transmission Power & 32 dBm \\ Noise Figure & 9 dB \\ Thermal Noise Density & \(-\)174 dBm/Hz \\ Transmitter’s/receiver’s antenna gain & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Network and Communications parameters. \begin{table}
\begin{tabular}{l l} \hline \hline
**Parameters** & **Value** \\ \hline Learning rate (\(\alpha\)) & 0.01 \\ Discount factor (\(\lambda\)) & 0.9 \\ Epsilon-start & 1 \\ Epsilon-Decay-rate & 0.0001 \\ Training Episodes & 2000 \\ Replay Buffer Size & 5000 evens \\ Batch Size & 64 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Q-learning and DQN training parameters. The number of penalties during the training process for both models is illustrated in Figure 4. As presented in this figure, during the training, at first the agent made mistakes, but over time it learns to avoid invalid actions. The left figure depicted the penalties during all training episodes, and to have a better understanding we zoomed out the number of penalties during the first 50 episodes at the right side. We can observe that, on average, the number of penalties in DQN is larger than in the Q-learning model, however, after a while they both learn from their experiences and the number of penalties tends to zero. The average reward during 2000 episodes of training is also shown in Figure 5. Similar to the penalty figure, we have shown the rewards of the first 50 episodes at right to have a better perception. As expected, the reward is increasing during the time, and converges after a while, for both models.