## 6 Investigating Several RL-Based Routing Methods
### [PERSON] et al. in [56] have presented a context-aware edge-based packet forwarding approach (CEPF) for VANETs. It supports both unicast and broadcast communication types. This method includes two main steps: selecting edge nodes and learning the best route for the last two-hop communication using reinforcement learning. Vehicles selected as the edge nodes are responsible for transmitting, processing, and storing data. The advantage of edge nodes in CEPF is that the number of forwarder nodes is reduced in the routing process. This enhances resource efficiency and improves the performance of this routing scheme. Edge nodes are selected in a decentralized fuzzy logic-based manner. This fuzzy system has the three input parameters, including vehicle speed, the number of nodes moving in the same direction, and link quality to estimate the fitness value of each vehicle for choosing the edge vehicle. After receiving each hello message, each vehicle computes its fitness value using the fuzzy mechanism and compares it with the fitness values of other neighboring vehicles. If the node has more fitness than the neighboring vehicles, the vehicle will be the edge node. In the reinforcement learning-based route discovery operation, the learning environment is the whole network, and each packet is the agent. The state space includes all vehicles in the network and the action set contains the choice of a single-hop neighbor as the next-hop node. Q-table consists of Q-values that are shared through hello messages with other neighbors. Note that in the Q-value updating process, the link status parameter is considered to create stable routes. It is computed with regard to the hello reception rate. The learning structure of CEPF is represented in Figure 11.  
[PERSON] et al. in [57] have offered a fuzzy Q-learning technique called PFQ-AODV for VANETs. The purpose of this method is to obtain the most suitable path between network nodes. In general, PFQ-AODV utilizes a fuzzy system to calculate connection links and utilizes a Q-learning algorithm to learn the most suitable paths. In PFQ-AODV, each node regularly transfers a hello message including bandwidth information with its neighboring  
Figure 11: Learning structure in CEPF.  
Figure 10: Learning structure in QAGR.  
nodes. After receiving this message, vehicles store the information about their single-hop and two-hop neighboring nodes in a neighboring table. In PFQ-AODV, nodes are independent of a positioning system. This has improved bandwidth consumption in this method. PFQ-AODV has designed a fuzzy logic-based link evaluation technique to analyze the link status between a vehicle and its neighbors. This link evaluation is applied to refresh the Q-value in PFQ-AODV. In this routing algorithm, each packet acts as the agent, and the network illustrates the learning environment. The state set involves all network nodes, and the action set consists of all single-hop neighbors that can be selected by the agent in the current state. Figure 12 displays the learning structure in PFQ-AODV. When creating a path, the source vehicle generates a route request (RREQ) packet and broadcasts it to its neighbors. Each node that receives RREQ for the first time re-disseminates this message. The operation continues until RREQ reaches the desired vehicle. In PFQ-AODV, after receiving any hello or RREQ, each node refreshes Q-table. The Q-value is used to evaluate nodes for participating in the data transmission process. After receiving RREQ, the desired vehicle searches its Q-table and choose a vehicle with the highest Q-value, and transfers the route reply (RREP) message to this node. The operation continues until RREP reaches the source vehicle. Ultimately, the created route can be applied for transmitting data packets to the destination node.