# Deep Reinforcement Learning for UAV-Based SDWSN Data Collection
## 4 Proposed RL Algorithm Model  
The model has been proposed based on using Reinforcement Learning (RL) to improve the cost functions (3) by designing an optimal policy. RL is a branch of the machine learning paradigm that deals with the multi-state decision process of a software agent (UAV in our case) while interacting with an environment. The Markov Decision Process (MDP) model is the foundation of Reinforcement Learning (RL) algorithms, particularly in solving path-planning problems. Generally, an MDP for UAV path planning consists of four essential components:  
S: The set of all possible states  
A: The set of all possible actions  
P: The transition probability, which determines the likelihood of moving from one state to another  
R: The reward function, which evaluates the performance of an action in each state. These elements are presented in a typical MDP model, as shown in Figure 2. At a given moment, the agent is in state \"s\". The decision-making mechanism can select any available action \"a\" while in state \"s\". Upon executing action \"a\", the agent transitions to a new state \"s\" and receives a reward \"R(s,a)\" for its action. The reward \"R\" is a function of the current state \"s\" and the chosen action \"a\". The reward is defined as:  
\[\text{R(s,a)}=\sum_{\text{a}\in\text{A}}\text{R(s,a)}\text{p(s,a)} \tag{4}\]  
where \(\text{Q(s, a)}\) is defined as a reward value when the agent takes action a at state s. The values of \(\text{Q(s, a)}\) can be generated using the Bellman equations. \[\text{Q(s,a)}=\text{R(s,a)}+\ \gamma\sum_{\text{a}\in\text{A}}\text{p(s,a)} \sum_{\text{a}\ \prime\in\text{A}}\text{Q(s\ \prime,\ a\ \prime)} \tag{5}\]  
Based on the proposed model, we assume that the system has multiple states S (UAV's location, velocity), where at each state, the agent has a finite number of actions a (i.e., moving to neighboring locations within the fuzzy path or adjusting a different velocity for the UAV within the bounded UAV velocity) to choose from. After choosing an action, the agent (UAV) receives a reward \(\text{r(s}_{\text{t}},\text{a}_{\text{t}})\), and moves to the next state \(\text{s}_{\text{t+1}}\). The goal of RL is to learn the best route and find an optimal policy \(\pi^{*}\) that maximizes the cumulative sum of all rewards. The proposed policy \(\pi^{*}=\{\text{a}_{1},\text{a}_{2},\dots,\text{a}_{\text{T}}\}\) defines multiple optimal actions in the whole path that should offer maximum rewards. There are different methods to obtain the RL-optimized path (best policy). One of the most practical ones is utilizinga table called the Q-Learning method, which is based on updating a table in each epoch (time slot) called Q-table. The proposed cost function given in Equation (3) requires a subset of policies to minimize the UAV energy consumption while maximizing the communication throughput. Agent (UAV) states S at each epoch considering the bounded fuzzy regions are UAV position \((\mathrm{X},\mathrm{Y})\) and UAV velocity. To define the environment based on the proposed RL model, the fuzzy boundary is required to be divided into small cells called RL cells in which each cell should be covered by the communication range of the ground gateway. As shown in Figure 3 left, the position of each center can be adjusted based on the predefined gateway position. In the learning policy, the initial path will be designed considering the red points, which are assumed to be within the communication range of gateways (but essentially, they are not receiving the optimal throughput) to train the UAV and get the information from the environment. In the second phase (training phase), as seen in the right figure, each cell is divided into 4 cells (can be divided into more cells, which may impose more computation and complexity on the algorithm). Once the agent (UAV) reaches each location with different states, the UAV should decide the best action (either moving to left, right, north, or south and declining or increasing the velocity within the range) to gain the best rewards.