# Environment-Aware Adaptive Reinforcement Learning-Based Routing for Vehicular Ad Hoc Networks
## 2 Related Work
### Machine Learning-Based Routing  
Machine learning is now widely applied in various research domains and has yielded numerous achievements. The authors of [36] employed the support vector machine (SVM) to optimize node selection, thereby utilizing the SVM to process vehicular data and categorize nodes into excellent next hops and ordinary next hops. And they generated routing metrics to determine routing indicators and their corresponding weight factors. In [37], a context-aware reliable routing protocol was proposed, which combines k-means clustering and the support vector machine (SVM). The k-means clustering divides routes into two categories: those with a high mean square error (MSE) are labeled as BAD, while those with a low MSE are labeled as GOOD. After training on routing data, the SVM enhances the performance of various metrics and routing efficiency. The SVM is a supervised machine-learning method that requires the training of data. However, because vehicle states are constantly changing, the training dataset needs to be large, thereby making the algorithm relatively complex. Reinforcement learning is a branch of machine learning that infers the external environment through a series of decision processes. In routing algorithms that introduce reinforcement learning, each data packet acts as an agent, and the network serves as a learning environment. The state set involves all network nodes, and the action set consists of all single-hop neighbors that the agent can choose in the current state. In PFQ-AODV [38], a link evaluation technique based on fuzzy logic was designed to analyze the link status between vehicles and neighboring vehicles. This link assessment was used to update the Q value in PFQ-AODV. The neighbor node with the highest Q-value is selected as the best next hop. As the number of vehicles increases, updating and maintaining the Q table requires significant communication overhead and computational resources. In [39], a VANET routing approach based on Q-learning and Qgrid was proposed. This method operates at both the macro and micro levels. At the macro level, the network is divided into smaller grids. The Q-learning algorithm is then used to learn traffic-flow characteristics within these grids, and Qgrid explores the most suitable next grid in the target direction. It utilizes policies provided at the micro level to select relay vehicles within each grid. QtAR [16] is a novel traffic-aware routing protocol based on RSU-assisted Q-learning. It employs two Q-learning-based routing algorithms for sending data packets between vehicles at the road and between RSUs at the intersection. During the routing process, vehicles broadcast HelloV2V messages, including their speed and position. Additionally, RSUs exchange HelloR2R messages with each other. Each vehicle maintains a V2V Q table for sending data packets on road segments. This Q table is refreshed upon receiving any HelloV2V message. In the routing algorithm based on R2R Q-learning, each data packet serves as a learning agent, and the state set represents neighboring RSUs. Furthermore, the action space encompasses all the adjacent intersections. Each RSU maintains two Q tables: a V2V Q table for sending data packets on road segments and an R2R Q table for sending data packets at intersections. In [40], a context-aware edge-based vehicular network packet forwarding scheme was proposed. It utilizes a fuzzy logic-based edge-node selection protocol to search for optimal edge nodes in a distributed manner, thereby facilitating the efficient utilization of wireless resources for data-packet forwarding through the edge. Reinforcement learning algorithms are employed to optimize communication within the last two hops, thereby enhancing the adaptive capabilities of communication routing. Ref. [20] presents a geographic routing solution within a layered routing framework.