## 5 Proposed Method
### Routing Algorithm at Intersection Level
```
0:\(\varepsilon,\alpha\), \(\gamma\): Q-learning parameters \(I=\{\mathit{Intersect}_{1},\mathit{Intersect}_{2},...,\mathit{Intersect}_{p}\}\) \(\mathit{Road}=\{R_{\mathit{North}},R_{\mathit{South}},R_{\mathit{East}},R_{ \mathit{West}}\}\)
```
0: Q-table Begin
1:while the convergence condition is not met do
2:for episode \(=1\) to \(M\)do
3:\(\mathbf{RSU_{k}}\): Choose an intersection as initial state \(\mathit{Intersect}_{i}^{t}\);
4:for\(t=1\) to \(N\)do
5:\(\mathbf{RSU_{k}}\): Select a random number \(\mathit{num}_{\mathit{rand}}\) in \([0,1]\);
6:if\(\mathit{num}_{\mathit{rand}}\leq\varepsilon\)then
7:\(\mathbf{RSU_{k}}\): Choose an action from \(\mathit{Road}=\{R_{\mathit{North}},R_{\mathit{South}},R_{\mathit{East}},R_{ \mathit{West}}\}\) randomly;
8:elseif\(\mathit{num}_{\mathit{rand}}>\varepsilon\)then
9:\(\mathbf{RSU_{k}}\): Choose the desired action with maximum Q-value from Q-table;
10:endif
11:if\(\mathit{Intersect}_{i}^{t+1}\) is destination then
12:\(R_{t}=R_{\max}\)
13:elseif\(\mathit{Intersect}_{i}^{t+1}\) is a local minimum then
14:\(R_{t}=R_{\min}\)
15:else
16:\(R_{t}=\frac{D_{\mathit{R_{\mathit{\mathit{\mathit{\mathit{\mathit{\mathit{ \mathit{\mathit{\mathit{\mathit{\mathit{\mathit{\mathit{\mathit{\mathit{\mathit{ \mathit{   }}}}}}}}}}}}}}(l)}{\max\limits_{R \in\mathit{Intersect}_{i}}D_{\mathit{R}}(l)}\ +\ \frac{C_{\mathit{R_{\mathit{\mathit{\mathit{\mathit{\mathit{\mathit{ \mathit{\mathit{\mathit{\mathit{\mathit{\mathit{\mathit{\mathit{\mathit{\mathit{\mathit{   \mathit{   }}}}}}}}}}}}}}(l)}{\max\limits_{R \in\mathit{Intersect}_{i}}\overline{\mathcal{Q}_{\mathit{R}}(l)}}\ +\ \left(1-\frac{ \overline{\mathcal{Q}_{\mathit{\mathit{\mathit{\mathit{\mathit{\mathit{ \mathit{\mathit{\mathit{\mathit{\mathit{\mathit{\mathit{\mathit{\mathit{\mathit{ \mathit{ \mathit{   \mathit{   }}}}}}}}}}}}(l)}}}{\max \limits_{R \in\mathit{Intersect}_{i}}\overline{\mathcal{I}_{\mathit{R}}(l)}}\right)\) ```
17:endif
18:\(\mathbf{RSU_{k}}\): Update Q-value in Q-table according to the reward value;
19:endfor
20:endfor
21:endwhile End
```

**Algorithm 2** Q-learning-based routing algorithm (I2I routing)

### Routing Algorithm at the Road Level

This section describes the QFHR routing algorithm at road sections. As stated in Section 5.1, each vehicle shares its position, speed, and queue status through hello messages with other neighboring vehicles. This information is stored in their neighbor table and used in the routing process. In QFHR, the routing algorithm at the road level includes three phases:

* Vehicle-to-vehicle (V2V) routing algorithm;
* Route recovery algorithm;
* Vehicle-to-infrastructure (V2I) routing algorithm. In the following, each of these three phases is explained exactly. #### 5.3.1 Vehicle-to-Vehicle (V2V) Routing Algorithm

In QFHR, each vehicle utilizes a greedy routing technique to single out a relay vehicle in the road section. Algorithm 3 shows the pseudo-code related to the V2V routing scheme. If \(\mathit{Vehicle}_{S}\) wants to send a packet to \(\mathit{Vehicle}_{D}\), this vehicle first determines the location of \(\mathit{Vehicle}_{D}\). There are three modes:

* **First mode:**_Vehicle\({}_{S}\)_ and \(\mathit{Vehicle}_{D}\)_ move on the same road section. In this case, \(\mathit{Vehicle}_{D}\) is determined as a target point (\(P_{\mathit{Target}}\)). * **Second mode:**_Vehicle\({}_{S}\)_ and \(\mathit{Vehicle}_{D}\)_ move in different road sections. \(\mathit{Vehicle}_{S}\), which means the source vehicle in this case, considers \(\mathit{Intersect}_{S}\) as \(P_{\mathit{Target}}\). * **Third mode:**_Vehicle\({}_{5}\)_ and _Vehicle\({}_{D}\)_ move in different road sections. _Vehicles_, which is an intermediate vehicle in this case, considers the intersection obtained from Algorithm 2 (i.e., _Interest\({}_{j}^{t+1}\)_) as _P\({}_{Target}\)_. Now, _Vehicles\({}_{5}\)_ checks its neighbor table to determine whether _P\({}_{Target}\)_ is its neighbor or not. If _P\({}_{Target}\)_ is in the neighbor table, _Vehicles_ send its data packet to _P\({}_{Target}\)_ directly.