# Environment-Aware Adaptive Reinforcement Learning-Based Routing for Vehicular Ad Hoc Networks
## 3 Preliminaries
### Network Model  
In the network model, an adaptive framework was employed to implement packet routing in VANET scenarios. The BSs connected to upper-level servers are deployed at the central locations within the urban area. V2V packet forwarding is utilized in the intersection areas. Vehicles are equipped with OBUs for data forwarding and assumed to have GPS systems or digital maps that provide real-time geographic location information for each vehicle. The abbreviations and notations used in this paper are listed in Table 1.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Notation** & **Description** \\ \hline MANET & Mobile ad hoc network \\ VANET & Vehicular ad hoc network \\ RSSI & Received signal-strength indication \\ OBU & On-board unit \\ \(\mathcal{V}\) & Set of vehicle nodes \\ \(\mathcal{L}\) & Set of links \\ \(\mathcal{RP}(\textit{src},\textit{dest})\) & Routing path from the \(S\) to the \(D\) \\ \#\textit{hops}(\mathcal{RP})\) & Number of hops of the route \(\mathcal{RP}\) \\ \(v,w\) & Vehicle node in the network \\ \(v_{w}\) & Vehicle with serial number \(n_{i}\) \\ \(N_{v}\) & Neighbor set of vehicle node \(v\) \\ \(\mathcal{P}_{v}^{l}\), \(\mathcal{P}_{w}^{l_{i}}\) & The position of vehicle node \(v\), \(w\) at time \(t_{i}\): \\ \(\mathcal{P}_{v}^{l_{v}}\) & \(\mathcal{P}_{w}^{l_{v}}=(x_{v}^{l_{v}},y_{v}^{l_{v}})\), \(\mathcal{P}_{w}^{l_{w}}=(x_{v}^{l_{w}},y_{v}^{l_{w}})\) \\ \(DIST(\cdot)\) & Euclidean distance \\ \(LSF(\cdot)\) & Link stability factor \\ \(LD(\cdot),LDF(\cdot)\) & Link duration and link duration factor \\ \(RSSF(\cdot)\) & Received signal-strength factor \\ \(COF(\cdot)\) & Channel occupation factor \\ \(ABWF(\cdot)\) & Available bandwidth factor \\ \(RV(\cdot)\) & Relative velocity \\ \(BRT(\cdot)\) & Beacon reception time \\ \(T_{B}\) & Neighbor information expiration interval \\ \(T_{B}\) & Beacon interval \\ \(ADDR(v)\) & The address of vehicle \(v\) \\ \(\alpha,\gamma\) & Learning rate and discount factor \\ \hline \hline \end{tabular}
\end{table}
Table 1: Table of abbreviations and notations.  
The vehicle network can be considered as a directed graph \(\mathcal{G}=(\mathcal{V},\mathcal{L})\), where \(\mathcal{V}\) is a set of vehicle nodes, and \(\mathcal{L}\) is a set of directed links. A link between vehicle node pairs \((v,w)\) indicates that \(v\) is a sender, \(w\) is a receiver, and vehicle node \(w\) is in the transmission range of vehicle node \(v\). Here, link \(l=(v,w)\), with \(l\in\mathcal{L}\). For a source \(S\) and a destination \(D\), a route from the \(S\) to the \(D\) is denoted as  
\[\mathcal{RP}(src,dest)=\{src=v_{n_{1}}\to v_{n_{2}}\rightarrow\cdots \to v_{n_{m}}=dest\}. \tag{1}\]  
Herein, \((v_{n_{i}},v_{n_{i+1}})\in\mathcal{L}\), \(\forall i=1,\,\ldots,m-1\). For convenience of presentation, sometimes we use the notation \(\mathcal{RP}\) instead of \(\mathcal{RP}(src,dest)\). The number of hops of the route \(\mathcal{RP}\) is denoted as \(\#hops(\mathcal{RP})=m-1\).  
When vehicles receive beacons, the Q values are adjusted based on the Q-learning training methodology. This adjustment depends on factors such as relative vehicle movement, signal strength, and network bandwidth. The path sequence is determined by selecting the neighbor node with the highest Q value in the Q table.