# Intelligent Interference Management in UAV-Based HetNets +
## Machine Learning Approach for UAV ICIC and Placement  
### Q Learning  
The Q learning algorithm is one of the commonly used reinforcement learning (RL) algorithms [23]. In reinforcement learning, an agent interacts with an environment by taking actions in different states and observing the costs or rewards of the actions. The agent starts out with random actions and, eventually, by observing the rewards and by exploring different states, it learns the best action to take in each state in order to optimize the cumulative reward. Figure 5 shows the interaction of the MBS and the UABS agent with the UAV HetNet environment considered in this paper. Since RL and deep-learning techniques are relevant and are known to be effective in solving problems with large search spaces, it is instructive to investigate their performance in solving our UAV HetNet optimization problem. Q learning allows the agents to act and learn as follows: being in a state \(s\) after selecting action \(a\), and receiving the immediate cost \(c\), a UABS or MBS agent \(n\) updates its knowledge \(Q_{n}(s,a)\) for the particular state-action pair through the following operation:  
\[Q_{n}(s,a)\leftarrow(1-\eta)Q_{n}(s,a)+\eta\Big{[}c_{n}+\lambda\min_{a}Q_{n}(s ^{\prime},a)\Big{]}, \tag{13}\]  
where \(\eta\), the learning rate, is the agent's willingness to learn from the UAV HetNet environment, \(\lambda\) is the discount factor, and \(s^{\prime}\) is the next state. Lower values of \(\lambda\) give more importance to immediate rewards. We note that, conventionally, \(\alpha\), rather than \(\eta\), is used to denote the agent's learning rate. We use \(\eta\), as, in our work, we have already used \(\alpha\) to denote the ICIC power reduction factor. After the Q learning agent has been trained, and when it is being evaluated, the exploration-exploitation factor, \(\epsilon\), determines the probability of the agent choosing a random action at a given state, (i.e., the agent explores), rather than choosing the optimal action, as learned by it (i.e., the agent exploits). The state space is a sufficient representation of the environment at a point in time, containing all the information required by the agent to choose its next action. In our case, the state space is the set:  
\[[\mathbf{X}_{\text{ue}},\mathbf{X}_{\text{mbs}},\mathbf{X}_{\text{uabs}}, \mathbf{S}_{\text{mbs}}^{\text{ICIC}},\mathbf{s}_{\text{uabs}}^{\text{ICIC}}], \tag{14}\]  
which captures the value of each ICIC parameter for each UABS and MBS, and also the location of each MBS, UABS and UE. The action space is the union of the possible actions for UABSs and MBSs in the system. At each state, along with its position, a UABS may choose any value for its ICIC parameters, \(\tau\) and \(\rho^{\prime}\). An MBS, similarly, may choose any value for its ICIC parameters, \(\alpha,\ \beta\) and \(\rho\). Specifically, an MBS's action space is the following set:  
\[\{\alpha,\beta,\rho\}\mid \alpha\in\{0,\Delta_{\alpha},2\Delta_{\alpha},\ldots 1\},\] \[\beta\in\{0,\Delta_{\beta},2\Delta_{\beta},\ldots 1\},\] \[\rho\in\{\rho_{\text{low}},\rho_{\text{low}}+\Delta_{\rho},\ldots,\rho_{\text{high}}\}, \tag{15}\]  
Figure 5: Interaction of the MBS or the UABS agent with the UAV HetNet environment for Q-learning. and a UABS's action space is the set:  
\[\{\tau,\rho^{\prime},\delta_{x},\delta_{y}\}\mid\tau\in\{0,\Delta_{ \tau},\ldots\tau_{\text{high}}+\Delta_{\tau},\tau_{\text{high}}\},\] \[\rho^{\prime}\in\{\rho^{\prime}_{\text{low}},\rho^{\prime}_{\text{ low}}+\Delta_{\rho^{\prime}},\ldots\rho^{\prime}_{\text{high}}\},\] \[x_{\text{uabs}}\in\{-l/2,-l/2+\Delta_{x},-l/2+2\Delta_{x},\ldots l /2\},\] \[y_{\text{uabs}}\in\{-l/2,-l/2+\Delta_{x},-l/2+2\Delta_{x},\ldots l /2\}, \tag{16}\]  
where \(x_{\text{uabs}}\) and \(y_{\text{uabs}}\) represent the X and Y coordinate, respectively, of the UABS, as a result of the action. Our chosen reward function is the median or mean SE.