## 6 Investigating Several RL-Based Routing Methods
### [PERSON] et al. in [69] have suggested an intersection-based routing technique using Q-learning (IRQ) for VANET. In this approach, a novel structure is proposed to broadcast traffic information in the network. According to this structure, the central server and other network nodes (i.e., vehicles and RSUs) can access the updated traffic information. They search for routes beads on both refreshed and historical traffic information and do not rely only on the old information. To create such a structure, two beacon and traffic messages are used. Beacon messages are regularly exchanged among all network nodes while traffic messages transfer traffic condition from RSUs to the central server. According to this traffic structure, IRQ obtains two global and local perspectives in the network. The global view shows the general traffic status in the road sections and is used by the central server to build a routing solution based on Q learning. This solution builds the most suitable routes at different network intersections. In this learning operation, the central server acts as an agent and uses its traffic table that contains the traffic status of different roads, to search the network environment for obtaining optimal paths between intersections. Also, the network acts as a learning environment, cooperates with the central server (i.e., the agent), responds to the actions performed by the agent and calculates the latter state and the reward corresponding to the performed action. The state set consists of all intersections, and the action set contains the road sections connected to the current intersection. In this method, the reward value is defined with regard to the density of vehicles, connection time, and delay. See Figure 24. In the learning model, the discount factor is a dynamic value obtained from the density of the road sections and the distance to the destination, and the learning rate has a fixed value based on experience. In this operation, the central server prevents congestion in the routes in accordance with a suitable solution to reduce collision and packet loss. This solution applies information about the traffic situation, i.e., delay and density of vehicles. Whenever the network server finds a congested path based on the congestion detection solution, it reduces the reward value of this path and prevents the selection of the path in future. Ultimately, IRQ employs a greedy routing approach for selecting paths in road sections. This approach is dependent on the local view. This forwarding solution is used for two parts such as vehicle-to-vehicle (V2V) routing and vehicle-to-RSU (V2I) routing. In V2V routing, each node picks out the closest vehicle to the target and forwards the data packet to the node. If the local optimum problem occurs in the routing process, the current vehicle gets a rank for its neighbors with regard to the Euclidean distance between itself and the next node, the one-hop delay, and the connection time, and transmits the packet to a vehicle with the best rank. In the V2I forwarding strategy, RSU at the intersection forwards the data packet to the corresponding road section using a greedy approach. In this process, if there is no vehicle to reach the target point, the RSU keeps this packet until a suitable next-hop node is found.
#### [PERSON] et al. in [70] have offered a Q-learning and fuzzy-based hierarchical routing solution (QFHR) for VANETs. It includes three parts, namely traffic pattern recognition, Intersection-to-Intersection (I2I) routing, and routing at road sections. In the first part, each network node (vehicle or RSU) considers a neighbor table to keep the traffic conditions of road sections. This table is obtained from hello messages transmitted by the network nodes in the road sections. This message is disseminated on the network at a fixed time interval. In addition, RSUs also maintain a traffic table to be aware of the traffic pattern of road sections connected to their intersections at any moment. The task of RSUs is to implement a routing method based on Q-learning to explore different paths between network intersections. In this method, the intersections form the state set. This decreases its size and boosts the convergence rate. In this operation, each message acts as the agent, and the whole network expresses the learning environment. The action set also represents the four road sections connected to each intersection, i.e., the north road, the south road, the east road, and the west road. The reward function considers the density of vehicles, connection quality (obtained from the connection time and the hello reception rate), the congestion level, the road traveling time. See Figure 25. This solution gets a dynamic discount factor according to the density of vehicles and the distance to the destination while the proposed method considers a constant learning rate. Ultimately, vehicles in each road section use a greedy technique to choose the most suitable path in each road section. If this algorithm fails due to the local optimum problem, a fuzzy solution performs route recovery and selects the next node. This algorithm analyzes the chances of neighboring vehicles based on congestion level, connection quality and distance to the destination. This analysis is used for selecting relay node. In the last step, each RSU uses a greedy approach to transmit data packets to the corresponding road section. If the RSU cannot find any vehicle, then the RSU will hold the packet until a suitable next-hop node is found.