# Environment-Aware Adaptive Reinforcement Learning-Based Routing for Vehicular Ad Hoc Networks
## 4 Algorithm Design
### EARR Routing Selection  
In the proposed enhanced adaptive Q-learning-based routing (EARR) approach, the vehicle nodes' position can be obtained by GPS. EARR utilizes Algorithm 1 to manage a neighbor table that contains status information about neighboring nodes. This table is regularly updated through the exchange of beacon messages to adapt to the highly dynamic network environment. EARR also divides routing paths based on the vehicle environment.  
Finally, Q-learning is used to make routing decisions. Additionally, Algorithm 2 is employed to construct the EARR neighbor Q table, which takes into account parameters such as link stability based on the learning rate, link duration based on the discount factor, signal strength, and effective bandwidth. Furthermore, a forward-oriented reward is incorporated into the reward function.  
```
0: Neighbor tables
0: Q-tables
1: Initialize the Q-tables for each vehicle
2:for each \(v_{n_{i}}\in\mathcal{V},\ i=1,2,\cdots,|\mathcal{V}|\)do
3:for each \(v_{n_{j}}\in\mathcal{N}_{v_{n_{j}}},\ j=1,2,\cdots,|\mathcal{N}_{v_{n_{j}}}|\)do
4:if\(|Current\_time-BRT(v_{n_{j}})|\geq\tau_{\text{B}}\)then
5: Delete neighbor record
6:else
7: Calculate \(\alpha\) using Equation (12)
8: Calculate \(\gamma\) using Equation (13)
9: Calculate reward \(r_{t}\)
10: Extracted the \(\mathcal{Q}_{max}(v_{n_{j}})\) from neighbor \(v_{n_{j}}\)
11:if\(v_{n_{j}}\) was recorded for the first timethen
12:\(\mathcal{Q}(v_{n_{i}},v_{n_{j}})=0\)
13:else
14:\(\mathcal{Q}(v_{n_{i}},v_{n_{j}})=\mathcal{Q}_{old}(v_{n_{i}},v_{n_{j}})\), Records from the Q-tables
15:endif
16: Update q values as follows:
17:\(\mathcal{Q}(v_{n_{i}},v_{n_{j}})=\mathcal{Q}(v_{n_{i}},v_{n_{j}})+\alpha\times (r_{t}+\gamma\times\mathcal{Q}_{max}(v_{n_{j}})-\mathcal{Q}(v_{n_{i}},v_{n_{j}}))\)
18:endif
19:endfor
20:endfor
```  
**Algorithm 2** Updating Q-table values for each vehicle  
The source vehicle generates data packets and determines the destination's location. If the destination node (_dest_) is within the range of vehicle \(v\), the \(\mathcal{Q}(v,dest)\) is greater than the Q value of any other neighboring node, and the record of the destination node is not expired, then the data packet is transmitted to the destination node. Otherwise, the source vehicle will select and forward the data packet among its neighboring nodes. To limit data-packet transmission, a time-to-live (TTL) mechanism is introduced. If TTL equals 0, the data packet is discarded. If the neighbor table is empty, the data packet is also discarded. If the neighbor table is not empty, the node uses Q-learning to forward the data packet. To ensure that data packets are not transmitted redundantly, EARR recommends maintaining a path recording mechanism. If the neighbor node with the highest Q value has already been recorded, then the second-highest Q value neighbor is chosen. If that neighbor is also recorded, the process continues iteratively. This is shown in Algorithm 3.  
```
0: Q-tables
0: The next hop neighbor vehicle node is selected
1:whileData packets DP that need to be transmitteddo
2:if\(D\) is in its own transmission range and \(\mathcal{Q}(v,\textit{dest})\geq\max_{j}(\mathcal{Q}(v,v_{n_{j}}),\ j=1,2,\cdots,| \mathcal{N}_{v}|\) and \(|\textit{Current\_time}-\textit{BRT}(v_{n_{j}})|<\tau_{B}\)then
3: Transmit data packets \(DP\) to \(\textit{dest}\)
4:else
5:if\(TTL(DP)-1=0\)then
6: Delete data packets \(DP\)
7:elseif\(|\mathcal{N}_{v}|=0\)then
8: Delete data packets \(DP\)
9:else
10:for\(j=1,2,\cdots,|\mathcal{N}_{v}|\)do
11: Obtain the address of the next hop neighbor node with Q size \(j\)-th
12:if\(v_{j\text{-}th}\in\mathcal{RP}(\textit{src},v)\)then
13: Continue
14:else
15: Record \(v_{j\text{-}th}\) to \(\mathcal{RP}(\textit{src},v)\)
16: Transmit data packets to \(v_{j\text{-}th}\)
17:endif
18:endfor
19:if all\(v_{n_{j}}\in\mathcal{RP}(\textit{src},v),\ j=1,2,\cdots,|\mathcal{N}_{v}|\)then
20: Delete data packets \(DP\)
21:endif
22:endif
23:endif
24:endwhile
```  
**Algorithm 3** Routing-path selection model for each vehicle