# Environment-Aware Adaptive Reinforcement Learning-Based Routing for Vehicular Ad Hoc Networks
## 1 Introduction
Most Q-learning-based approaches employ fixed learning rates and discount factors, which may prove inadequate in unstable working environments. Consequently, the accuracy of their routing selection decreases,  
Figure 1: An example of part of the problem with VANETs. and the links established with selected neighboring nodes become exceedingly fragile. Therefore, the EARR protocol adaptively adjusts the learning rate and discount factor based on relevant metrics. In a dynamic network environment, the faster the network topology is changing, the more sensitive it is to environmental variations, and therefore, the learning rate should be larger. For the discount factor, which reflects future expectations, it should be smaller in more unstable environments. Additionally, traditional Q-learning methods require a significant amount of time to explore and exploit the environment when searching for the optimal trajectory, thereby leading to the exploration-exploitation dilemma. The exploration-exploitation dilemma in reinforcement learning refers to the challenge of finding the right balance between exploring new possibilities and exploiting known information to maximize cumulative rewards. Therefore, this study leverages a beacon mechanism and the acquired neighbor information to rapidly explore potential optimal or suboptimal paths. The main contributions of the proposed environment-aware reinforcement learning-based routing Protocol in this paper are as follows:  
* Q tables instead of routing tables: This study proposes an environment-aware reinforcement learning-based routing (EARR) protocol for VANETs. EARR employs Q-learning for multimetric routing optimization and does not rely on fixed routing vector tables. Instead, it utilizes a continuously adaptive updated Q-value table. As a result, there is no need to maintain end-to-end links, thereby reducing the significant network overhead caused by routing maintenance. Given the highly dynamic topology of VANETs, ensuring route stability in a timely manner often leads to prohibitively high maintenance costs. By selecting the next-hop relay node based on the Q value, only local one-hop links need to be maintained, thereby forming optimal or suboptimal routes in the process of selecting the best next-hop for each hop. * Adaptive Q-learning: In highly dynamic VANETs, link stability is extremely fragile. The Q-learning approach rapidly comprehends the network environment through beacons and utilizes neighbor information alongside adaptive Q-learning techniques to autonomously adjust the learning rate and discount factor. By altering the learning rate and discount factor, the routing adapts itself to the current environmental changes. * Broadcast updates: In this study, EARR updates Q-table values by using a special broadcast beacon. The feasibility of the scheme is proved by building a simple vehicle Q-value update model. It also explains the features of the routing scheme, such as solving the dilemma of Q-learning development and exploration and adjusting the parameters in time under the change in environment. * Shadow fading and residual bandwidth: In the unique working environment of VANETs, factors such as urban vegetation and buildings significantly affect signal propagation, thereby resulting in substantial signal loss and reduced link lifetimes. To mitigate data packet loss caused by low-quality communication links, this study introduces the received signal-strength indication (RSSI) into the discount factor. Due to the nature of wireless communication, signals need to be transmitted in specific frequency bands. The network is susceptible to congestion due to excessive traffic, thereby hindering network performance improvement. Therefore, we incorporate residual bandwidth estimation during the link establishment process to assess network conditions and balance network load. The rest of this paper is ordered in the following way. Section 2 reviews the existing research work on routing protocols in VANETs.