## 3 Intelligent Congestion Control Model Based on DRL in Heterogeneous V2X
### Policy Function  
The paper uses the deep neural network algorithm PPO2 to obtain the approximate optimal policy function \(\pi(a_{t}|s_{t})\). The neural network parameters are denoted as \(\theta\), and the optimization objective function for reinforcement learning is defined as Equation (17)  
\[D(\theta)=E_{\pi_{\theta}}\big{(}\hat{R}_{t}-Q(s_{t})\big{)}^{2} \tag{17}\]  
where \(\hat{R}_{t}\) denotes the expected value of the reward, \(Q(s_{t})\) is the value function of state \(s_{t}\), and \(D(\theta)\) uses the least squares method to calculate a biased estimate of the value function \(Q(s_{t})\), which represents the cumulative reward predicted to be obtained at the end of the round at state \(s_{t}\). Therefore, the least squares method is often used to define the objective function, and the square operation ensures the non-negativity of the objective function.  
The optimization objective function \(D(\theta)\) wants to maximize the reward value obtained by executing the corresponding action \(a_{t}\) by adjusting the parameter \(\theta\) of the policy function. However, if the parameter \(\theta\) is adjusted too much, frequent oscillations during the gradient rise can cause PPO2 to not converge quickly. Therefore, PPO2 introduces \(clip()\) into the objective function \(D(\theta)\), which is defined as Equation (18)  
\[D^{clip}(\theta)=\sum\limits_{(s_{t},a_{t})}min\bigg{(}\frac{\pi_{\theta}(a_{t }|s_{t})}{\pi_{\theta_{k}}(a_{t}|s_{t})}\hat{A}_{t},clip\bigg{(}\frac{\pi_{ \theta}(a_{t}|s_{t})}{\pi_{\theta_{k}}(a_{t}|s_{t})},1-\varepsilon,1+ \varepsilon\bigg{)}\hat{A}_{t}\bigg{)} \tag{18}\]where \(clip()\) is the truncation function, defined as Equation (19)  
\[clip\bigg{(}\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{k}}(a_{t}|s_{t})},1- \varepsilon,1+\varepsilon\bigg{)}=\left\{\begin{array}{cc}\frac{\pi_{\theta}(a _{t}|s_{t})}{\pi_{\theta_{k}}(a_{t}|s_{t})}&1-\varepsilon<\frac{\pi_{\theta}(a _{t}|s_{t})}{\pi_{\theta_{k}}(a_{t}|s_{t})}<1+\varepsilon\\ 1-\varepsilon&\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{k}}(a_{t}|s_{t})} \leq 1-\varepsilon\\ 1+\varepsilon&\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{k}}(a_{t}|s_{t}) }\geq 1+\varepsilon\end{array}\right. \tag{19}\]  
where \(r=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{k}}(a_{t}|s_{t})}\), and \(r\) reflects the magnitude of the change in the parameter update; the larger the value, the larger the update magnitude and vice versa. If the result of \(D^{clip}(\theta)\) is positive, it means that the reward obtained by performing the current action is higher than the mean value and it is desired to select such actions as much as possible. On the contrary, if the result of \(D^{clip}(\theta)\) is negative, it means that the reward for performing the current action is lower than the average value, so the function should try to avoid performing such actions later. With the help of the intercept function \(clip()\), the function value of \(D^{clip}(\theta)\) is limited to the interval \([1-\varepsilon,\,1+\varepsilon]\) to avoid excessive fluctuations in the update. The schematic diagram of the \(D^{clip}(\theta)\) function is shown in Figure 4.  
When the optimization objective function \(D>0\), if \(r>1+\varepsilon\), it is truncated so that it is not too large. Similarly, when \(D<0\), if \(r>1-\varepsilon\), it is also truncated so that it is not too small. \(D^{clip}(\theta)\) ensures that \(r\) does not fluctuate drastically.