# AI-Enabled Interference Mitigation for Autonomous Aerial Vehicles in Urban 5G Networks +
## 5 Reinforcement Learning
### Q-Learning  
Q-learning is a commonly used reinforcement learning algorithm for making optimal decisions in Markov decision processes (MDPs). In Q-learning, the agent learns a Q-function that maps a state-action pair to a value representing the expected cumulative reward obtained by taking that action from that state and following an optimal policy thereafter. Thus, the Q-table is the mapping of state-action pairs with a Q-value relating to the reward. Table 3 illustrates a Q-table. \begin{table}
\begin{tabular}{l p{284.5 pt}} \hline \hline
**Elements** & **Definition** \\ \hline Observations & Observations are measurements of characteristics of the environment. They are expressed as \(\mathcal{O}e\mathbb{R}^{P}\), where p is the no. of characteristics. \\ States & A state is defined as the discretised form of the observations at time step t, and is denoted as \(s_{t}\) e S, where \(s_{t}\) represents the current state. \\ Actions & An action is a valid choice that can be made at a given time step t. It changes the state from \(s\) to the target state \(s^{\prime}\). It is denoted by \(a_{t}\in\mathcal{A}\). \\ Policy & The relationship between the state and the action is governed by a policy, denoted as \(\pi(\cdot)\). The policy determines how the agent selects a new state by taking an action. \\ Rewards & After the agent takes ‘\(a^{\prime}\) when it is in ‘\(s^{\prime}\) at \(t\) and moves to \(s^{\prime}\), it receives a reward. The reward signal obtained is represented as \(w_{s^{\prime},a^{\prime}}[t;q]\). \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Elements of reinforcement learning. Figure 7: Agent-environment interaction. (**a**) RL elements; (**b**) DQL algorithm–agent-environment interaction. The Q-value is not calculated in a fixed manner; instead, it is implemented in the Q-table as an iterative approach. This is known as the training phase. The Q-table is a fundamental component in the Q-learning algorithm, serving as a means of storing and updating information regarding the states, actions, and their associated expected rewards. It is implemented as a mapping between state-action pairs and their corresponding Q-values, which represent the estimated optimal future rewards of taking a particular action from a specific state. The Q-learning algorithm works through the following steps:  
_Initialise Q-table:_ At the initiation of the Q-learning process, the Q-table is initialised to an array of zeros, indicating a lack of prior knowledge about the environment. As the agent interacts with the environment through trial and error, it updates its understanding of the state-action pairs and their corresponding Q-values, which it uses to optimise its future actions and maximise its expected reward. _Choose an action:_ To choose an action in Q-learning, the agent can use an exploration-exploitation strategy. During the exploration phase, the agent randomly chooses actions to gain information about the environment. During the exploitation phase, the agent chooses actions based on its current estimate of the optimal action-value function. One common exploration-exploitation strategy is the epsilon-greedy strategy. The agent chooses a random action with probability \(e\), and the optimal action (i.e., the action with the highest Q-value) with probability \(1-e\). As the agent gains more experience, it can gradually decrease the value of \(e\), which allows it to explore less and exploit more. _Update Q-table:_ The Bellman Equation is a mathematical expression that facilitates the determination of updates to the Q-table following each step taken by the agent. The equation effectively integrates the current perception of value with the anticipated optimal reward, which is premised on the selection of the optimal action as known at that moment. In a practical implementation, the agent assesses all feasible actions for a given state and chooses the state-action pair with the maximum Q-value. The Bellman Equation [31] is formulated as in (1).