# Environment-Aware Adaptive Reinforcement Learning-Based Routing for Vehicular Ad Hoc Networks
## 1 Introduction
Vehicle F travels in the opposite direction of vehicle D, thereby resulting in short link life and unstable communication. Therefore, an intermediate vehicle should be selected to forward the data. For the limitations of traditional routing, many studies have proposed traffic-aware routing protocols [14; 15; 16] aimed at enhancing adaptability in urban routing scenarios. In urban traffic environments, vehicular networks are influenced by road structures and the distribution of buildings. These protocols evaluate traffic flows based on road segments and select the next relay at intersection areas. Traffic-aware routing can adapt to various network and traffic conditions, thereby offering higher efficiency. It can avoid forwarding packets to roads with low vehicle density or poor communication-link quality. Vehicles at intersections send collector packets (CPs) to gather network and traffic conditions on different road segments, which is a widely used approach in current traffic-aware routing. However, the continuous measurement process introduces additional communication overhead, thereby increasing expenses and delays. Moreover, the accuracy and update frequency of link assessments significantly impact the performance of traffic-aware routing protocols. The development of artificial intelligence is driving advancements across various research domains, with machine learning, deep learning, and reinforcement learning serving as crucial areas within the artificial intelligence field. These research domains have had far-reaching impacts across various sectors. Notably, significant achievements have been realized in fields such as transportation systems, autonomous driving, medical-image analysis, and natural-language processing [17]. To address the challenges of dynamic routing, an increasing number of studies have turned to the use of reinforcement learning (RL), because it can autonomously learn the dynamic characteristics of networks to improve system performance while optimizing available network resources. However, vehicle behavior is characterized by a large number of states and high uncertainty, thereby making it challenging to train network models for accurate predictions. As a result, algorithms have become increasingly complex and difficult to converge. Q-learning [18] stands out as a classic algorithm in the realm of reinforcement learning and has found widespread application in packet routing. The Q-learning algorithm has been effectively employed in the design of various types of ad hoc networks, thus encompassing VANETs, among others [19; 20; 21; 22]. It possesses the capability to make optimal decisions through continuous interaction with the environment, even in the absence of prior knowledge about the environment. In the routing process, to ensure the selection of the correct relay nodes, a Q-Learning-based routing protocol needs to incorporate certain constraints in its design, such as speed, delay, network load, link lifespan, and others. In VANETs, the entire network can be modeled as an environment, and vehicles or data packets can be regarded as agents based on design requirements. Whether it is the forwarding of control packets or data packets, it can be seen as an interaction between agents and the environment. Through this interaction, the protocol learns the network's latest state and continuously adjusts its forwarding strategy to ensure the reliability of link communication. This study introduces a novel environment-aware reinforcement learning-based routing (EARR) protocol. It effectively addresses the high mobility of vehicular nodes and environmental variability. In the EARR protocol, we not only depart from the fixed-parameter form of traditional Q-learning, but also abandon the convergence method of using the optimal trajectory. The proposed protocol adjusts its behavior based on environmental feedback, thus taking into consideration metrics such as the position, speed, bandwidth, and power of the next-hop relay nodes.