## 6 Investigating Several RL-Based Routing Methods
### Iv2 Xq  
[PERSON] et al. in [54] have introduced the intersection-based V2X routing using Q-learning (IV2 XQ) for VANETs. IV2 XQ is a geographic routing scheme, which utilizes a hierarchical routing framework and consists of two main parts: multi-dimensional Q-learning-based routing strategy for selecting the best road segments at intersections and the greedy routing strategy for selecting the best relay node in the selected road segment. In this scheme, the central server acts as the agent, and the learning environment is the network. The central server uses historical traffic data to explore the network environment to choose the most suitable path. In IV2 XQ, the state space represents all network intersections. Thus, this scheme can well lower the number of states of Q-learning and makes it easier to implement. As a result, the algorithm has a high convergence speed. The action set consists of the road segments connected to one intersection, namely north road segment, south road segment, east road segment, and west road segment. The environment allocates the reward \(R\) to the agent after taking an action (i.e., selecting a road segment for forwarding a data packet from the previous intersection to the next intersection). If the data packet reaches the destination intersection, then \(R=\varphi\) (where, \(\varphi>0\)); otherwise, \(R=0\). In IV2 XQ, the learning rate is empirically selected while, the discount factor is dynamically determined in accordance with the density of vehicles and distance to the desired node. Figure 9 shows the learning structure in IV2 XQ. Additionally, in this method, the RSUs at intersections are equipped with monitoring modules to record the number of packets and calculate the network load locally. Therefore, if the network load is more than a load threshold, then the packet is sent to an alternative road segment. This congestion control mechanism prevents network  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Routing Scheme** & **Advantages** & **Disadvantages** \\ \hline \multirow{6}{*}{Position-based routing algorithm} & \multirow{3}{*}{DTN} & Low routing overhead, high scalability, low bandwidth consumption, low memory & High delay in the routing process \\  & & delivery rate & \\ \cline{2-3}  & & Suitable for high-density and dynamic networks, low routing overhead, low bandwidth consumption, low memory & Needing accurate location computation of nodes, poor performance for networks with low density \\ \hline \multirow{3}{*}{Cluster-based routing algorithm} & \multirow{3}{*}{Conductance} & Low routing overhead, managing network congestion, high scalability & Difficulty in cluster management and CH selection \\ \hline \multirow{6}{*}{Proactive} & \multirow{3}{*}{Conductance} & Low scalability, inefficiency in resource consumption such as bandwidth, high memory consumption, high routing overhead or updating routing tables, unsuitable for VANET, not adaptability with repeated changes in topology due to frequent link breakages \\ \cline{2-3}  & & & High delay in the route discovery process, flooding control messages, network congestion, and high bandwidth consumption \\ \cline{2-3}  & & & \\ \cline{2-3}  & & & \\ \cline{2-3}  & & & \\ \cline{2-3}  & & & \\ \hline \hline \end{tabular}
\end{table}
Table 7: Comparison of different routing algorithms.  
congestion and increases the packet delivery rate. When discovering paths at each road segment, the enhanced greedy forwarding methodology is also applied to discover the most suitable relays in each road section. This algorithm is defined for both V2V and V2I communication. The greedy forwarding strategy selects the neighboring vehicle nearest to the desired target as a relay node. If the forwarding strategy fails and the algorithm is involved with the local optimum problem, the carry-and-forward technique will be used to recover the route.