## 2 Related Work
### Position-Based Routing Algorithms with Reinforcement Learning  
In the following, we introduce the concept of reinforcement learning, Q-learning, and then compare several position-based routing algorithms that use Q-learning as the basis for decision making. First, Q-learning is briefly described. The reinforcement learning model is composed of several elements. We will first define a few terms below. Definition: a state is called a Markov state if and only if:  
\[P[S_{t+1}|S_{t}]=P[S_{t+1}|S_{1},\ S_{2},\ S_{3},\ \ldots,\ S_{t}]\]  
Definition: For a Markov state \(s\) and its successor state \(s^{\prime}\), define the State Transition Probability as \(P_{sS^{\prime}}=P[S_{t+1}=s^{\prime}|S_{t}=S]\). The State Transition Matrix P is the transition probability for all states \(s\) to all successor states \(s^{\prime}\). Definition: a policy \(\pi\) is the probability distribution of action \(a\) after a given state \(s\). \[\pi(s)=P[A_{t}=a|S_{t}=s]\]  
Definition: the reward \(G_{t}\) is the sum of accumulated depreciation rewards starting at time \(t\). \[G_{t}=R_{t+1}+rR_{t+2}+\cdots,\ where\ r\in[0,1]\]  
Definition: the action-value function \(q_{\pi}(s,\ a)\) is the expected reward obtained by starting from state \(s\), selecting action \(a\), and then following policy \(\pi\). \[q_{\pi}(s,\ a)=E_{\pi}\left[S_{t}=s,\ A_{t}=a\right]\]  
According to the Bellman equation, we can finally rewrite the above formula as:  
\[q_{\pi}(s,\ a)=R_{s}^{a}+r\sum_{s^{\prime}\in S}P_{sS^{\prime}}^{a},\ \ s^{\prime}\in A\ \pi \big{(}s^{\prime}\big{)}q_{\pi}\big{(}s^{\prime},\ a^{\prime}\big{)} \tag{1}\]Finally, we will discuss Q-learning training, for which the core update formula is:  
\[Q(s_{t},\ a_{t})\leftarrow(1-\alpha)*Q(s_{t},\ a_{t})+\alpha*\big{(}R_{t+1}+ \gamma*max_{\alpha}Q\big{(}s_{t+1},\ a^{\prime}\big{)}-Q(s_{t},\ a_{t})\big{)} \tag{2}\]  
\(\alpha\) is the learning rate (0 < \(\alpha\) < 1). \(\gamma\) is the discount factor (0 < \(\gamma\) < 1). When the value of \(\gamma\) is larger, the long-term rewards obtained in the future will receive more attention; the smaller the value, the more the current rewards will be considered. \(R\) is the reward; it acts on each state and will receive the corresponding reward. Position-based routing algorithms use Q-learning as the basis for decision making:  
1. The ITAR-FQ [25] algorithm architecture consists of two main parts: the real-time traffic aware process and road evaluation (RTAP-RE) and the routing decision process with fuzzy Q-learning (RDP-FQ). The RTAP-RE designs a road evaluation method to process traffic information and estimate road quality (RQ), with reference to the number of vehicles moving in the same/a different direction, the length of the road, packet generation time, current time, etc. RDP-FQ divides the entire routing process into multiple routing processes. As the intersection is divided into multiple routing processes, the packet continues to look for the next intersection and selects a new road segment until the packet reaches the destination. Facing the intersection, it calculates the road score (RC) by combining RQ and the Manhattan distance (MDnormal), and selects the road with the highest RC as the next forward road segment. The reward function is modified by referring to the benchmark reward and the corresponding value given by Fuzzy Logic. The advantage of this paper is that the Fuzzy Logic indicator is added to Q-learning for routing decisions in road segments. The disadvantage is that the directionality of the relative position of the transmitter and the destination are not considered. If the road segment with the destination end is the next hop at the intersection, it may cause the packet to be transmitted over a long distance or repeated on the road segment, and the historical movement trajectory of the vehicle is not used for training to obtain better routing decisions. 2. ADOPEL [14] assumes that two kinds of packets are exchanged between vehicle nodes: beacons and event-driven messages.