# Reinforcement Learning-Based Routing Protocols in Flying Ad Hoc Networks (FANET): A Review
## 3 Fundamentals of Reinforcement Learning (RL)
### Reinforcement Learning  
It is known as a powerful tool for learning optimal policy by interacting between an agent and the environment. In reinforcement learning, the agent employs a trial-and-error technique to increase the reward obtained from the environment [16; 17]. In each step, it obtains its state (\(s_{t}\)) from the environment and chooses an action \(a_{t}\). Next, the environment determines the reward \(r_{t}\) and the new state \(s_{t+1}\) based on the selected action. If this action is good, the environment has positive feedback, that is, the agent receives a positive reward. Otherwise, it has negative feedback. The agent continues this process until it maximizesthe expected discounted feedback for any state. More precisely, a RL-based system includes four components:  
* **Policy:** This component includes a set of stimulus-action rules that map each state of the surrounding environment to a list of the allowed actions. In the simplest case, a lookup table can be used to implement this policy. However, searching in this table requires a high computational cost [18; 19]. * **Reward:** This component is a function such as a random function, which depends on state and the action selected by the agent. In fact, it indicates the response of the environment with regard to the selected action. After taking the action, the environment changes its state and produces a reward. The purpose of the agent is to increase the sum of rewards obtained from the interaction with the environment because the reward reflects the main mechanism for modifying the policy. Now, if the agent finds out that the selected action leads to a small reward, it changes its selection and chooses another action when there is a similar condition in the future. This helps the agent to explore different probabilities [19; 29]. * **Value function:** This component is also called the value-action function. Although, the reward represents whether the current selected action is suitable. However, the agent must follow solutions that make more profit in the middle and long term. The value of a certain state represents the sum of received rewards by passing from that state. Thus, actions are selected by searching for the maximum value and not the highest reward. However, it is difficult to calculate the value compared to the reward because the reward is immediately received from the environment. In contrast, the agent must estimate the value by searching for previous interactions with the environment. In all RL algorithms, it is a very challenging issue to estimate the value function [29]. * **Model:** This component describes the performance of the environment. This model estimates the future state and immediate reward according to a specific state and the selected action. Based on this view, two reinforcement learning methods can be defined: model-based and free-model. Model-based methods design a model for solving RL problems. However, free-model methods learn the optimal policy based on trial and error [18; 29]. In the following, important challenges in reinforcement learning algorithms are described:
* **Trade-off between exploration and exploitation:** Exploration and exploitation are two important concepts in reinforcement learning. In exploration, the agent searches for unknown actions to obtain new knowledge. In exploitation, the agent utilizes the existing knowledge and uses the explored actions to produce high feedback. In the learning process, it is necessary to make a trade-off between exploration and exploitation. The agent can exploit its existing knowledge to achieve the suitable value and can explore new actions to increase its knowledge and obtain more valuable rewards in the future. As a result, the agent should not focus only on exploration or exploitation and must experience various actions to gradually obtain the actions with the highest value.