## 3 Base Concepts
### Reinforcement Learning  
In artificial intelligence (AI), there is an important and useful tool called reinforcement learning (RL), which defines two main components, agent and environment. The agent chooses an action to interact with the environment. The aim of this interaction is to achieve an optimal solution for a certain issue. RL supports a specific framework called the Markov decision process (MDP) for solving optimization issues [30; 31]. MDP manages the optimization problem using a random manner and defines four parameters such as \((S,A,p,r)\). The first parameter is \(S\) and indicates the finite state space. The second parameter (i.e., \(A\)) is the finite action space. The third parameter (i.e., \(P\)) represents the transition function. This function determines the next state \(s^{{}^{\prime}}\) after doing the action \(a\) in the current state \(s\). The last parameter is \(r\), which indicates the reward function in the learning issue. This function determines the reward obtained from the learning environment after taking the action \(a_{t}\) by the agent in the state \(s_{t}\). Note that \(t\) indicates time. Figure 2 displays reinforcement learning. In this process, the agent searches the environment by taking the action \(a_{t}\) in the current state \(s_{t}\). Now, the environment calculates the reward value (\(r\)) and the next state \(s_{t+1}\) by considering the performed action and the current state. The purpose of this learning framework is to reach the most suitable policy \(\pi\) and achieve the maximum reward. The long-term purpose of the agent is to boost the expected discounted reward (i.e., \(\max\biggl{[}\sum\limits_{t=0}^{T}\delta r_{t}(s_{t},\pi(s_{t}))\biggr{]}\)). \(\delta\) illustrates the discount factor and indicates the effect of the reward on the Q-value. It is adjustable in \([0,1]\). If \(\delta=1\), the agent is only dependent on previous experiences. In contrast, if \(\delta=0\), the agent considers only the last immediate reward received from the environment [30; 31]. After determining reward values \begin{table}
\begin{tabular}{c c c} \hline \hline
**Method** & **Strengths** & **Weaknesses** \\ \hline IV2 XQ [28] & Determining the discount factor with regard to the density and distance of vehicles on the road section, reducing communication overhead, designing a congestion control mechanism, appropriate convergence speed, reducing the number of states in the Q-learning algorithm & Not considering factors, like speed, movement direction, and link lifetime in the routing process, not relying on new traffic information in the network, introducing a centralized reinforcement learning-based routing algorithm & Not considering factors, like speed, movement direction, and link lifetime in the routing process, not relying on new traffic information in the network, introducing a centralized reinforcement learning-based routing algorithm & Not considering factors, like speed, movement direction, and link lifetime in the routing process, not relying on new traffic information in the network, introducing a centralized reinforcement learning-based routing algorithm \\ \hline IRQ [29] & Adjusting the discount factor with regard to the vehicle density and distance, high PDR, reducing latency in the routing process, presenting a evaluation mechanism for controlling the congested paths, suitable convergence speed, reducing the number of states in the Q-learning algorithm, considering new traffic information in the network & High routing overhead, introducing a centralized reinforcement learning-based routing algorithm \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Cont. and transition probabilities, the Q-value is obtained from the Bellman function presented in Equation (1):  
\[Q(s_{t},a_{t})=(1-\alpha)Q(s_{t},a_{t})+\alpha[r+\delta(\max Q(s_{t+1},a_{t}))] \tag{1}\]  
In the Bellman equation, \(\alpha\) is the learning rate. It is adjustable in \([0,1]\) and creates a tradeoff between exploration and exploitation. It determines whether the agent focuses on new or old information.