## 4 Proposed \(Q\)-Learning Based UAV Positioning Mechanism  
RL has a special place in machine learning, as it is structurally quite different than supervised and unsupervised learning methods. RL consists of a set of policy-based goal-seeking algorithms, where an agent takes actions in a given environment to maximize its reward or minimize the penalty, and therefore RL is predominantly used in optimization problems rather than time-series analysis, classification, or clustering as supervised and unsupervised learning algorithms do.  
RL has unique advantageous characteristics making it more preferable than other types of optimization methodologies, including heuristics. Firstly, RL algorithms, such as \(Q\)-learning and state-action-reward-state-action (SARSA), are predominantly model-free, meaning that they do not require the model of the environment-of-interest in advance, they instead interact with the environment to capture its dynamics [52]. Moreover, since RL algorithms include learning in their body, they do not have to start from scratch every time there is a change in the environment, they rather adapt themselves to the changes, giving them a strength of optimization with reasonable time complexity. This is an essential feature for an optimization algorithm especially for dynamic scenarios, where network conditions change rapidly and frequently. To this end, we employ \(Q\)-learning, one of the most common RL algorithms, in this work to take the benefit of above-mentioned features.  
In RL, there is an agent taking actions to find the optimum policy for a given problem. Based on the action of the agent, first, corresponding state is observed, followed by evaluating the subsequent penalty/reward function. Then, the action-value function, storing calculated penalty/reward values for all the states and actions, is updated [52]. The agent takes action in two different ways: explore and exploit. In the initial phases of the implementation, the agent is expected to explore more to discover the environment better. However, after a sufficient exploration, the agent should start exploiting the available information to be able to focus on finding the best policy.  
We adopted OpenAI Gym [53] tool for building environment for this study. It is based on episodic RL, where experience of each agent is divided into episodes. In initial state of each episode, we randomly localize the UAV BS and the users in a grid, and learning proceeds until the environment reaches one of the stopping criteria (this will be detailed in the following paragraphs). The main goal here is to maximize the total reward per episode and to decrease the number of episodes for achieving desired performance. RL steps in each episode are given in Algorithm 1, where \(s_{t}\) and \(s_{t+1}\) are the current and next states, respectively, and \(a_{t}\) is the current state while \(R_{t+1}\) is the expected value of the reward function.  
In this study, states refer to the position of the UAV in the grid. The agent in the developed \(Q\)-learning algorithm has seven action values for each state, which denote the agent action \(a_{t}\) in UAV state of \(s_{t}\) at time \(t\). The possible actions for each state \(s_{t}\) are _hold_, _move up_, _move down_, _move left_, _move right_, _move forward_, _and move backward_. The agent follows \(\epsilon\)-greedy [52] policy to take random actions initially--which is referred to as exploring-- and decays \(\epsilon\) through iterations--which is referred to as exploiting-- for decreasing random actions. Given that the main goal of this study is to optimize energy consumption of the UAV along with maximizing user coverage, the reward function in the proposed method is inline with the objective function in (23), and depends on the energy consumption and coverage.  
The components of the developed \(Q\)-learning algorithm for the problem of UAV BS positioning are detailed in the following paragraphs.