## 7 Discussion
Due to creating the network model, RRPV can estimate the value function with less interaction with the environment. Also, this routing method is more flexible against sudden changes in the environment. However, model-based techniques have higher computational complexity and require a lot of computational resources, and have poor performance for large-scale networks. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Scheme** & **Single-Agent Learning** & **Multi-Agent Learning** \\ \hline IV2 XQ [54] & ✓ & \(\times\) \\ \hline QAGR [55] & ✓ & \(\times\) \\ \hline CEPF [56] & ✓ & \(\times\) \\ \hline PFQ-AODV [57] & ✓ & \(\times\) \\ \hline QGrid [58] & ✓ & \(\times\) \\ \hline RRPV [59] & \(\times\) & ✓ \\ \hline QTAR [60] & ✓ & \(\times\) \\ \hline Q-LBR [61] & ✓ & \(\times\) \\ \hline ECTS [62] & ✓ & \(\times\) \\ \hline RLRC [63] & ✓ & \(\times\) \\ \hline GLS [64] & ✓ & \(\times\) \\ \hline RSAR [65] & ✓ & \(\times\) \\ \hline Wu et al. [66] & ✓ & \(\times\) \\ \hline RHR [67] & ✓ & \(\times\) \\ \hline SeScR [68] & ✓ & \(\times\) \\ \hline IRQ [69] & ✓ & \(\times\) \\ \hline QFHR [70] & ✓ & \(\times\) \\ \hline \hline \end{tabular}
\end{table} 
Table 12: Comparison of RL-based routing methods in terms of learning framework. Table 14 compares routing approaches with regard to learning algorithms. According to this table, IV2 XQ, QAGR, CEPF, PFQ-ADDV, QGrid, QTAR, Q-LBR, RRPV, ECTS, RLRC, GLS, RSAR, Wu et al., RHR, IRQ, and QFHR use traditional RL techniques. These methods have a good performance and find the optimal response with an acceptable convergence speed if the state and action sets are small. However, large state and action sets lead to slow convergence and more time to find an optimal response. SeScR utilizes a deep reinforcement learning technique to improve the learning rate. This method can solve complex computational operations in complex (large-scale) environments, such as VANETs, and speeds up the agent ability to optimize this policy. These routing protocols have a suitable performance for large networks, and their learning speed is good. There are few routing methods that use deep reinforcement learning techniques to improve the routing process. Due to rapid progress of technology and the emergence of new networks such as Internet of Vehicles (IoV), it is essential to design routing techniques that use deep reinforcement learning because these methods are successful in terms of convergence speed in larger state and action spaces. In addition, Table 15 compares various routing approaches with regard to the learning process. In this table, in IV2 XQ, QGrid, Q-LBR, ECTS, GLS, and IRQ, the reinforcement learning algorithm is executed by a central agent (for example a central server or a UAV) in the network. Then, the agent sends information about this learning process to the network nodes to begin the data transfer operation. This can reduce communication and computational overhead in the network because these methods do not need to exchange control messages between nodes. However, they may be involved with the single point of failure problem, meaning that, if the central agent cannot properly execute the RL-based routing algorithm, the network will be disrupted. Also, the central agent cannot adapt itself to the topology changes in a real-time manner. Therefore, when occurring events like accidents, these algorithms cannot predict real-time traffic conditions and adapt to sudden changes in the network. Also, based on Table 15, it can be found that in some methods like QAGR, CEPF, PFQ-AODV, RRPV, QTAR, RLRC, RSAR, Wu et al., RHR, SeScR, and QFHR, RL-based routing algorithms are locally executed in the network. However,