# Environment-Aware Adaptive Reinforcement Learning-Based Routing for Vehicular Ad Hoc Networks
## 5 Simulation Results and Analysis  
### Simulation Environment and Parameters  
This study conducted extensive simulations using the SUMO, Veins, and OMNeT++ toolset. OMNeT++ is an object-oriented discrete event network simulator that focuses on simulating wireless network communication. It allows us to model and analyze various communication protocols and network topologies, thereby making it particularly suitable for researching wireless communication systems. SUMO is a traffic simulator that encompasses vehicle driving patterns, behaviors, and route planning. It enables us to generate realistic traffic scenarios that can be used in conjunction with other external programs. Veins is an open-source framework for simulating vehicular networks. It is built on top of OMNeT++ and SUMO. The simulator instantiates an OMNeT++ node for each vehicle in the simulation and pairs the node's movement with vehicle movements in the road-traffic simulator (i.e., SUMO). Veins provides a well-developed underlying structure for vehicular wireless networks, including the physical layer and MAC layer based on the 802.11p protocol. This makes it convenient for secondary development, such as improving MAC layer protocols or researching routing protocols. The simulation scenario covers a 2 km \(\times\) 2 km urban area, consisting of 40 bidirectional two-lane road segments and 25 intersections. In the algorithm, vehicles adapt to their environment using Q-learning to obtain the routing Q table. In our approach, the lower limit of the learning rate for Q-learning was set to 0.6, and the upper limit was 0.95. The upper limit for the discount factor was 0.7. These values needed to be adjusted based on the operational environment of the vehicles. The parameters used in the simulations are listed in Table 2. To accommodate the characteristics of vehicle mobility, this study did not impose distance restrictions between vehicles. Vehicleflow was generated as random traffic by SUMO, with vehicle speeds ranging from 0 to 50 km/h, which aligns with the typical characteristics of urban traffic. In the simulation, to observe the convergence capability of each region, the relationship between the location and \(\mathcal{Q}_{max}\) is presented in Figure 8. At a randomly chosen moment, the \(\mathcal{Q}_{max}\) values were retained for all the vehicles in the entire simulated map. The average \(\mathcal{Q}_{max}\) for all the vehicles within a fixed radius was computed as the \(\mathcal{Q}_{max}\) for that region. From the graph, we can observe that as the training process of the Q table for the entire network progressed, the closer a region was to the destination area, the larger the average \(\mathcal{Q}_{max}\) for that region, thereby ensuring the formation of links. The projections in the x and y directions also clearly show the variations in \(\mathcal{Q}_{max}\). An interesting phenomenon can be observed in the graph along the periphery of the entire area; when vehicles maintained a straight-line road connection with the destination region, their \(\mathcal{Q}_{max}\) value was maximized at these edge points, such as at coordinates (2000, 1000) in the graph. This is understandable because maintaining a straight-line road to the destination region minimizes the number of hops in routing, thereby resulting in minimal convergence loss. In contrast, in other areas with urban road environments, the number of hops in routing is bound to increase.