## 6 Investigating Several RL-Based Routing Methods
### [PERSON] et al. in [69] have suggested an intersection-based routing technique using Q-learning (IRQ) for VANET. In this approach, a novel structure is proposed to broadcast traffic information in the network. According to this structure, the central server and other network nodes (i.e., vehicles and RSUs) can access the updated traffic information. They search for routes beads on both refreshed and historical traffic information and do not rely only on the old information. To create such a structure, two beacon and traffic messages are used. Beacon messages are regularly exchanged among all network nodes while traffic messages transfer traffic condition from RSUs to the central server. According to this traffic structure, IRQ obtains two global and local perspectives in the network. The global view shows the general traffic status in the road sections and is used by the central server to build a routing solution based on Q learning. This solution builds the most suitable routes at different network intersections. In this learning operation, the central server acts as an agent and uses its traffic table that contains the traffic status of different roads, to search the network environment for obtaining optimal paths between intersections. Also, the network acts as a learning environment, cooperates with the central server (i.e., the agent), responds to the actions performed by the agent and calculates the latter state and the reward corresponding to the performed action. The state set consists of all intersections, and the action set contains the road sections connected to the current intersection. In this method, the reward value is defined with regard to the density of vehicles, connection time, and delay. See Figure 24. In the learning model, the discount factor is a dynamic value obtained from the density of the road sections and the distance to the destination, and the learning rate has a fixed value based on experience. In this operation, the central server prevents congestion in the routes in accordance with a suitable solution to reduce collision and packet loss. This solution applies information about the traffic situation, i.e., delay and density of vehicles. Whenever the network server finds a congested path based on the congestion detection solution, it reduces the reward value of this path and prevents the selection of the path in future. Ultimately, IRQ employs a greedy routing approach for selecting paths in road sections. This approach is dependent on the local view. This forwarding solution is used for two parts such as vehicle-to-vehicle (V2V) routing and vehicle-to-RSU (V2I) routing. In V2V routing, each node picks out the closest vehicle to the target and forwards the data packet to the node. If the local optimum problem occurs in the routing process, the current vehicle gets a rank for its neighbors with regard to the Euclidean distance between itself and the next node, the one-hop delay, and the connection time, and transmits the packet to a vehicle with the best rank. In the V2I forwarding strategy, RSU at the intersection forwards the data packet to the corresponding road section using a greedy approach. In this process, if there is no vehicle to reach the target point, the RSU keeps this packet until a suitable next-hop node is found.  
Figure 24: Learning structure in IRQ.