## 3 Overview of Reinforcement Learning (RL)
### Reinforcement Learning  
According to this technique, an agent finds the most suitable policy by interacting with the environment. In each iteration, the agent selects an action \(a_{t}\) in accordance with its state (\(s_{t}\)), gets reward \(r_{t}\) from the environment, and shifts to the latter state \(s_{t+1}\). This process is continues until the agent accumulates the rewards received from the environment and maximizes the expected discounted return from any situation [19]. A reinforcement learning system consists of four main parts:  
* **Policy:** It indicates a set of stimulus-action rules. According to these rules, each state is mapped to a group of actions that can be chosen by the agent. The simplest case is to implement this policy by a lookup table. However, it has high computational complexity for search processes [19; 20].
* **Reward signal:** It can be a random function that indicates the response of environment proportional to the action and state of the agent. After responding to actions, the environment creates a new state of agent and reward. The agent attempts to maximize the total reward resulting from relationship between itself and environment. Reward is an important element for improving the policy. If the selected action causes a weak reward, the agent can select another action in the same condition in the future to get other possibilities [19; 20].
* **Value function:** Note that the reward signal indicates whether the last performed action is good, while the value function calculates the value of a specific state with regard to the sum of rewards collected by the agent when placing in that state. Thus, actions are chosen in accordance with the highest values, not the maximum rewards. Note that the computation of values is much more complicated than rewards because the agent obtains rewards immediately from the environment. However, values must be predicted by searching for past interactions between the agent and the environment. Value estimation is a challenging issue in all RL algorithms [19; 20].
* **Model:** It describes the environment's function. This means that model can predict the latter state and the obtained reward by looking and choosing one of the allowedactions in a particular state. According to this concept, there are two model-based and model-free RL techniques. In model-based approaches, a model is created to solve the desired issue. In contrast, model-free RL approaches utilize a trial and error methodology to learn the most suitable policy [19; 20]. In this area, there are various challenges that are expressed as follows:
* **Exploration and exploitation:** It is an important challenge in RL problems. Exploration means searching for unknown actions to acquire new knowledge. On the other hand, exploitation means utilizing the discovered actions that can produce high feedback. The agent can perform both operations. This means that it can perform the exploitation operation based on its current knowledge and obtains the suitable value. Also, it can perform the exploration operation to learn actions that have never been experienced so far to enhance its knowledge and earn better rewards in the future. Thus, this is not correct to focus only on exploration or extraction. The agent needs to experience different actions to gradually select those with the maximum value [21].
* **Uncertainty:** It is another challenge in RL problems because the agent and the environment work in relation to each other, and this interaction can cause uncertainty when changing the state and obtaining the reward. Whereas, a RL problem must learn a policy that increases the collected rewards (value function) over time [21].