## 5 Deep Reinforcement Learning-Based Caching Solution
### DRL-Based Caching Algorithm  
The high mobility of vehicle users leads to environmental information dynamic changes in vehicular networks. To cope with the challenge, we consider a novel algorithm that effectively solves excessive state and action space in a dynamic environment by combining deep learning and reinforcement learning to solve the complex caching problem. Especially, as the update rule of the DQN does not require knowledge about the transition and reward functions. Therefore, we proposed the P3 DVQC to solve the caching problem. The parameters of the P3 DVQC main networks update formula is expressed as  
\[\theta_{t_{y^{1}}^{x}}=\alpha\left(\gamma Q\left(s_{t_{y^{1}}^{x}},\max_{a_{t_{y ^{1}}^{x}}^{x}=1}Q\left(s_{t_{y^{1}}^{x}},a_{t_{y^{1}}^{x}};\theta_{t_{y}^{x}} ^{x}\right);\theta_{t_{y}^{x}}^{x}\right)+\theta_{t_{y}^{x}}-Q\left(s_{t_{y^{ \overline{y}}^{x}}},a_{t_{y^{\overline{y}}}^{x}};\theta_{t_{y^{\overline{y}}} ^{x}}\right)\right)\
abla Q\left(s_{t_{y^{\overline{y}}}^{x}},a_{t_{y^{ \overline{y}}}^{x}};\theta_{t_{y^{\overline{y}}}^{x}}\right)+\mathcal{R}(t_{ y}^{x}), \tag{46}\]  
where \(\alpha\) is learning rate, \(\gamma\) is discount factor, \(\
abla\) is gradient operator. The P3 DVQC is integrated with prioritized experience replay (PER) technology to achieve priority sampling by changing sampling distribution to improve the performance of the DQN. The P3 DVQC sampling weight formula is expressed as  
\[P_{PER}^{i}{=}\frac{\rho_{PER}^{i}}{\sum\limits_{i\in\Omega}\rho_{PER}^{i}}, \tag{47}\]  
where \(\rho_{PER}^{i}\) is priority parameter of experience pool sampling \(i\), it can be expressed as  
\[\rho_{PER}^{i}=\left(\triangle E_{TD}(t_{y+1}^{x})+\varepsilon\right)^{ \mathrm{e}}, \tag{48}\]  
where \(\varepsilon\) is disturbance coefficient, \(\triangle E_{TD}(t_{y+1}^{x})\) is a distance of \(<\)\(s_{t_{y^{\overline{y}}}},a_{t_{y^{\overline{y}}}^{x}},\mathcal{R}(t_{y}^{x}),s_ {t_{y+1}^{x}}\)\(>\) between output and target value. It is calculated as  
\[\triangle E_{TD}(t_{y+1}^{x})=\left|\gamma Q\left(s_{t_{y^{\overline{y}}}^{x} },\max_{a_{t_{y^{\overline{y}}}^{x}}^{x}=1}Q\left(s_{t_{y^{\overline{y}}}^{x} },a_{t_{y^{\overline{y}}}^{x}};\theta_{t_{y^{\overline{y}}}^{x}}\right); \theta_{t_{y^{\overline{y}}}^{x}}^{x}\right)+\mathcal{R}(t_{y}^{x})-Q\left(s_ {t_{y^{\overline{y}}}^{x}},a_{t_{y^{\overline{y}}}^{x}};\theta_{t_{y^{\overline {y}}}^{x}}\right)\right|, \tag{49}\]  
where \(\theta_{t_{y^{\overline{y}}}^{x}}\) is the parameter of P3 DVQC main network, \(\theta_{t_{y^{\overline{y}}}^{x}}^{x}\) is parameter of P3 DVQC target network, \(|.|\) is absolute operation. In addition, P3 DVQC also integrated dueling technology to improve the deep Q-learning algorithm. The framework of the proposed P3 DVQC is shown in Figure 3. The target network output function is expression as  
\[Q\left(s_{t_{y^{\overline{y}}}^{x}},a_{t_{y^{\overline{y}}}^{x}},\alpha_{t_{y ^{\overline{y}}}^{x}},\beta_{t_{y^{\overline{y}}}^{x}}\right)=V\left(s_{t_{y^ {\overline{y}}}^{x}},\theta_{t_{y^{\overline{y}}}^{x}},\beta_{t_{y^{\overline{ y}}}^{x}}\right)+A\left(s_{t_{y^{\overline{y}}}^{x}},a_{t_{y^{\overline{y}}}^{x}}; \theta_{t_{y^{\overline{y}}}^{x}},\alpha_{t_{y^{\overline{y}}}^{x}}\right), \tag{50}\]  
where \(\alpha_{t_{y^{\overline{y}}}^{x}}\) is parameter of advantage function \(A\left(s_{t_{y^{\overline{y}}}^{x}},a_{t_{y^{\overline{y}}}^{x}};\theta_{t_{y^ {\overline{y}}}^{x}},\alpha_{t_{y^{\overline{y}}}^{x}}\right)\), \(\beta_{t_{y^{\overline{y}}}^{x}}\) is parameter of value function \(V\left(s_{t_{y^{\overline{y}}}^{x}},\theta_{t_{y^{\overline{y}}}^{x}},\beta_{ t_{y^{\overline{y}}}^{x}}\right)\). The pseudo code of the proposed DRL-based caching algorithm is provided in Algorithm 1.