## 5 Proposed Method
### Routing Algorithm at Intersection Level  
In this routing process, the source vehicle first calculates the positions of itself and the destination vehicle using GPS and a digital map. Then, it must specify two intersections namely the source intersection and the destination intersection. Note that each road section is connected to two intersections. Thus, the source vehicle should determine the source intersection among the two intersections before starting the routing process. The source vehicle selects the intersection that has less distance from the destination as the source intersection (\(Intersect_{S}\)). Moreover, the destination intersection should be determined among two intersections connected to the destination road section. The destination vehicle chooses the intersection that is closer to the source vehicle as the destination intersection (\(Intersect_{D}\)). Now, the source vehicle must send the data to \(Intersect_{S}\) by using the routing algorithm at the road level described in Section 5.3. At this step, a Q-learning-based distributed routing protocol is introduced to obtain the most suitable route between various intersections using traffic information. This algorithm is also called the intersection-to-intersection (I2I) routing algorithm. Algorithm 2 describes the pseudo-code related to this routing process in QFHR. In the I2I routing model, each message is the agent and the entire network expresses the learning environment. The agent must discover the learning environment by performing different actions and experiencing various states. The state space contains a set of intersections, namely \(I=\big{\{}Intersect_{1},Intersect_{2},\ldots,Intersect_{p}\big{\}}\). The action set represents the four road sections connected to each intersection, namely \(Road=\{R_{North},R_{South},R_{East},R_{West}\}\). See Figure 7. After selecting a road section, the packets are sent from \(Intersect_{i}^{t}\) to \(Intersect_{j}^{t+1}\). Now, the environment gives a reward to the learning agent based on the reward function presented in Equation (15). \[R_{t}=\left\{ \begin{array}{ll}R_{\max},&\text{\emph{Intersect}}_{j}^{t+1} \text{\emph{is destination}}\\ R_{\min},&\text{\emph{Intersect}}_{j}^{t+1}\text{\emph{is local minimum}}\\ \frac{D_{R_{current}}(l)}{\max\limits_{R\in\text{\emph{Intersect}}_{i}}^{ \max}D_{R}(l)}+\frac{CQ_{R_{current}}(l)}{\max\limits_{R\in\text{\emph{Intersect }}_{i}}^{\max}CQ_{R}(l)}+\left(1-\frac{\overline{Q}_{R_{current}}(l)}{\max \limits_{R\in\text{\emph{Intersect}}_{i}}^{\max}\overline{Q}_{R}(l)}\right)+ \left(1-\frac{\overline{R}_{R_{current}}(l)}{R_{\text{\emph{Intersect}}_{i}}^ {\max}\overline{T}_{R}(l)}\right),&\text{\emph{Otherwise}}\end{array} \right. \tag{15}\]  
where \(D_{R_{current}}(l)\), \(CQ_{R_{current}}(l)\), \(\overline{Q}_{R_{current}}(l)\), and \(T_{R_{current}}(l)\) are the density of vehicles, the average connection quality, the average congestion status, and the average road time in the current road section \(R_{current}\). Based on the reward function, if the selected intersection is the desired intersection meaning that the packet reaches the destination intersection, the corresponding road section achieves the maximum reward (\(R_{\max}\)). On the other hand, when the next intersection is a local minimum, meaning that the selected intersection has the minimum distance from the destination compared to other neighboring intersections, it receives the minimum reward (\(R_{\min}\)). In other conditions, the reward function depends on vehicle density, connection quality, congestion status, and road traveling time. QFHR estimates the discount factor in accordance with the network conditions because if this parameter is considered a constant value, the routing algorithm cannot adapt to the dynamic network. In addition, QFHR empirically considers \(\alpha\) as a default value (\(\alpha=0.1\)) according to [28].