# Investigating Routing in the VANET Network: Review and Classification of Approaches
## 2 A Review of Past Research (Related Work)
To get the action value, you need to take the discounted state values with the \(P_{\text{ss}}\) probabilities in all possible states (in this case just two states) and add the immediate reward:  
\[\boldsymbol{v_{\pi}(s)}=\mathbb{E}[\boldsymbol{R_{t+1}}+\gamma\boldsymbol{v_{ \pi}(S_{t+1})}|\boldsymbol{S_{t}}=\boldsymbol{s}] \tag{8}\]3  
\[\boldsymbol{v_{\pi}(s)}=\mathbb{E}[\boldsymbol{R_{t+1}}+\gamma\boldsymbol{v_{ \pi}(S_{t+1})}|\boldsymbol{S_{t}}=\boldsymbol{s}] \tag{8}\]4  
This recursion relation can be visualized in another binary tree, with q(s,a) ending in the next state '\(s\)' with a given probability \(P_{\text{ss}}\), as an action 'a' is performed with the probability \(\pi\) and ends with the action value q(s',a'). Figure 11 shows the tree design with node graph. **Optimal policy**  
The most important issue in deep reinforcement learning is to find the optimal action-value function q*. Finding q* means that the agent knows exactly the quality of an action in each given state. Additionally, the AI agent can decide which action to take based on the desired quality. If we want to define q*. The best possible action-value function is the one that follows the policy that maximizes the action values:  
\[\boldsymbol{v_{\pi}(s)}=\mathbb{E}[\boldsymbol{R_{t+1}}+\gamma\boldsymbol{v_{ \pi}(S_{t+1})}|\boldsymbol{S_{t}}=\boldsymbol{s}] \tag{8}\]5  
\[\boldsymbol{v_{\pi}(s)}=\mathbb{E}[\boldsymbol{R_{t+1}}+\gamma\boldsymbol{v_{ \pi}(S_{t+1})}|\boldsymbol{S_{t}}=\boldsymbol{s}] \tag{8}\]6 Drawing tree with node graph. \[\boldsymbol{v_{\pi}(s)}=\mathbb{E}[\boldsymbol{R_{t+1}}+\gamma\boldsymbol{v_{ \pi}(S_{t+1})}|\boldsymbol{S_{t}}=\boldsymbol{s}] \tag{8}\]7 Visualization with node graph. We need to maximize q(s,a) to find the best possible policy. Maximization means to select only the action an among all possible actions for which q(s,a) has the highest value. This gives the following definition for the optimal policy \(\pi\):  
\[\mathbf{\pi}_{*}(\mathbf{s},\mathbf{a})=\left\{ \begin{aligned} \mathbf{1},& \textbf{if }\mathbf{a}=\textbf{arg max }\mathbf{q}_{*}(\mathbf{s},\mathbf{a})\\ & 0,&\textbf{Otherwise}\end{aligned} \right. \tag{19}\]  
**Optimal Bellman equation**  
If the AI agent can solve this equation, it means that it has solved the problem in the given environment. The agent in any state or situation knows the quality of any possible action according to the goal and can act accordingly. Solving [PERSON]'s optimality equation will be the subject of future research in this field. \[\mathbf{q}^{*}(\mathbf{s},\mathbf{a})=\mathbf{R}_{\mathbf{s}}^{\mathbf{a}}+\mathbf{\gamma}\mathbf{ \sum}_{\mathbf{s}^{\prime}\in\mathcal{S}}\mathbf{P}_{\textit{ss}}\textbf{ max }\mathbf{q}_{*}\big{(}\mathbf{s}^{\prime},\mathbf{a}^{\prime}\big{)} \tag{20}\]  
One of the DRL method's limitations is that researchers must completely understand their environment. Not all systems can be made completely and without defects using deep reinforcement learning. In systems where the consequences of wrong decisions can be dire, deep reinforcement learning cannot work on its own. Table 4 examines the articles that are used to control the VANET network using deep reinforcement learning methods. In 2019, [PERSON] et al. [14] proposed a routing mechanism for VANETs using artificial intelligence methods and deep reinforcement learning algorithm. In this method, the vehicle's speed and position is predicted and appropriate routing is chosen, which increases the performance of VANETs network. According to the results, this routing method has reduced the transmission delay and has been effective for routing in VANETs. In 2018, [PERSON] et al. [18] proposed a deep reinforcement learning method for connected vehicle communication. Centralized SDN controller is also used for VANETs network routing. This controller has also been used as a learning agent to learn routing in VANETs network. In addition, a deep Q-learning algorithm has been used to determine the routing policy.