## 8 Challenges and Open Issues  
Designing routing schemes based on reinforcement learning is a serious research topic that must be regarded by researchers in the future. There are various challenges in designing and improving these routing methods in VANETs. Here, we express some of the most important challenges in this area. * **Designing the broadcast control mechanisms:** Exchanging control messages periodically and flooding routing messages increase the use of bandwidth and lead to high overhead. Thus, designing broadcast control mechanisms is an essential need, which must be considered when presenting RL-based routing schemes. For example, researchers can dynamically adjust the broadcast time interval of control messages or filter some nodes with regard to factors such as link quality and movement information to control the flooding process of routing messages. * **Limiting the state space:** Large state and action sets significantly lower the convergence rate and increase delay in the routing process. Therefore, researchers must focus on strategies to restrict state and action spaces such as filtering some states in accordance with specific criteria and employing clustering approaches. * **Q-table:** Q-values obtained from a reinforcement learning algorithm are recorded in Q-table. The size of this table depends on the size of the state and action sets. Therefore, large state and action sets will sharply grow the dimensions of Q-table. This greatly increases the need for memory to maintain Q-table. Also, updating this table needs high latency, computational costs, and overhead. Thus, the management of Q-table size should be considered by in the future, for example, limiting the state space mentioned above can be considered a potential solution. Also, focusing on DRL-based solutions are useful. * **Tradeoff between exploration and exploitation:** In the RL-based routing process, it is very important to dynamically adjust learning parameters to make a balance between exploration and exploitation. It can be considered by researchers in the future. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Scheme**} & \multicolumn{2}{c}{**Position-Based Routing**} & \multirow{2}{*}{**Cluster-Based Routing**} & \multicolumn{3}{c}{**Topology-Based Routing**} \\ \cline{2-6}  & **DTN** & **Non-DTN** & & **Proactive** & **Reactive** & **Hybrid** \\ \hline IV2 XQ [54] & \(\times\) & ✓ & \(\times\) & ✓ & \(\times\) & \(\times\) \\ \hline QAGR [55] & \(\times\) & ✓ & \(\times\) & \(\times\) & ✓ & \(\times\) \\ \hline CEPF [56] & \(\times\) & ✓ & \(\times\) & \(\times\) & ✓ & \(\times\) \\ \hline PFQ-AODV [57] & \(\times\) & \(\times\) & \(\times\) & \(\times\) & ✓ & \(\times\) \\ \hline QGrid [58] & ✓ & ✓ & \(\times\) & ✓ & \(\times\) & \(\times\) \\ \hline RRPV [59] & ✓ & \(\times\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ \hline QTAR [60] & ✓ & \(\times\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ \hline Q-LBR [61] & \(\times\) & \(\times\) & \(\times\) & \(\times\) & ✓ & \(\times\) \\ \hline ECTS [62] & \(\times\) & \(\times\) & \(\times\) & \(\times\) & ✓ & \(\times\) \\ \hline RLRC [63] & \(\times\) & \(\times\) & ✓ & \(\times\) & ✓ & \(\times\) \\ \hline GLS [64] & \(\times\) & ✓ & \(\times\) & ✓ & \(\times\) & \(\times\) \\ \hline RSAR [65] & ✓ & \(\times\) & \(\times\) & \(\times\) & ✓ & \(\times\) \\ \hline Wu et al. [66] & \(\times\) & \(\times\) & ✓ & \(\times\) & \(\times\) & \(\times\) \\ \hline RHR [67] & \(\times\) & \(\times\) & \(\times\) & ✓ & ✓ & ✓ \\ \hline SeScR [68] & \(\times\) & \(\times\) & ✓ & \(\times\) & \(\times\) & \(\times\) \\ \hline IRQ [69] & \(\times\) & ✓ & \(\times\) & ✓ & \(\times\) & \(\times\) \\ \hline QFHR [70] & \(\times\) & ✓ & \(\times\) & \(\times\) & ✓ & \(\times\) \\ \hline \hline \end{tabular}
\end{table} 
Table 16: Comparison of RL-based routing methods in terms of routing techniques. * **Multi-objective RL-based routing methods:** Most RL-based routing protocols focus on a quality of service (QoS) requirement.