## 5 Proposed Classification
### Classification of RL-Based Routing Methods with Regard to Routing Algorithm  
According to the presented classification, RL-based routing approaches are categorized into three main groups based on routing algorithms: position-based, cluster-based, and topology-based. Table 7 expresses the strengths and weaknesses of different routing algorithms. * **Position-based routing schemes:** These approaches use the geographical information of nodes. Thus, each node connects to a positioning system, like the global positioning system (GPS) to obtain its spatial information at any time [1; 6]. These routing methods do not require all information of the network and utilizes local information. This \begin{table}
\begin{tabular}{c c c} \hline \hline
**Routing Scheme** & **Advantages** & **Disadvantages** \\ \hline \multirow{3}{*}{\begin{tabular}{c} Centralized routing \\ method \\ \end{tabular} } & \begin{tabular}{c} Suitable for small VANETs, low \\ computational costs and \\ communication overhead \\ \end{tabular} & \begin{tabular}{c} Inappropriate for large scale \\ networks, single point of failure, \\ not adapting to dynamic \\ network topology \\ \end{tabular} \\ \hline \multirow{2}{*}{\begin{tabular}{c} Distributed routing \\ method \\ \end{tabular} } & \
\begin{tabular}{c} Suitable for large-scale networks, \\ Adaptability to dynamic \\ network environment \\ \end{tabular} &
\begin{tabular}{c} High communication overhead \\ and computational costs \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Centralized and distributed routing methods. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Routing Scheme** & **Advantages** & **Disadvantages** \\ \hline \multirow{2}{*}{\begin{tabular}{c} RL-based routing \\ method \\ \end{tabular} } & \begin{tabular}{c} Suitable for small VANETs \\ with small state and action \\ spaces \\ \end{tabular} & \begin{tabular}{c} Inappropriate for large scale \\ networks, low convergence \\ speed, poor learning ability \\ \end{tabular} \\ \hline \multirow{3}{*}{\begin{tabular}{c} DRL-based routing \\ Method \\ \end{tabular} } & \
\begin{tabular}{c} Suitable for large-scale \\ networks, Good convergence \\ speed, High learning ability \\ \end{tabular} &
\begin{tabular}{c} The problem of instability in \\ the model, sensitive to low \\ changes in Q-value and \\ consequently sudden \\ changes in the policy \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table} 
Table 5: RL and DRL-based routing schemes. improves communication costs, bandwidth, and consumed energy in these methods. Therefore, they are very suitable for highly dynamic networks such as VANET. These approaches are classified into two groups: - _Delay-tolerant networking (DTN) routing approaches:_ These methods can manage the challenges caused by frequent disconnections in VANETs. This problem leads to breaking the paths created to the destination node. In most cases, these approaches use the store-carry-forward technique when the node cannot select a routing path to other nodes [44; 45]. This technique greatly reduces communication overhead because it does not use any additional control packet. However, it increases delay when transferring data process [46; 47]. - _Non-delay tolerant networking (non-DTN) routing methods:_ These protocols are utilized in networks with high connectivity so that the density of the nodes is relatively high. However, if the connectivity is not guaranteed in the network, the performance of these protocols will be weakened. Also, they use a greedy forwarding technique for the data transmission process [48]. According to this technique, the transmitters send data packets to the neighbor closest to the destination. However, if the sender does not find a neighbor close to the destination compared to itself, the data delivery process fails and a recovery strategy is used to manage this condition. These methods have a good performance in high-density networks. In addition, they have a low communication overhead, high scalability, and low memory requirement.