## 3 Intelligent Congestion Control Model Based on DRL in Heterogeneous V2X
### Design of State Space
#### 3.3.1 Probability Distribution
\[v_{i}=[v[i,\,1],\,v[i,\,2],\,\cdots,\,v[i,\,N]] \tag{9}\]  
As can be seen from Formula (8), taking \(v_{5}\) as an example, when the system is transferred from \(S_{cv_{6}}\) to \(S_{cv_{5}}\), the probability of congestion loss due to random loss probability \(P_{x}(S_{cv_{5}})\) is \((1-P_{x}(S_{cv_{5}}))\)P. Therefore, the total probability is going to be \(P_{x}(S_{cv_{5}})+1-P_{x}(S_{cv_{5}})=1\). Moreover, in matrix \(Z\), the sum of the probabilities of each row vector must always be equal to 1, as shown in Formula (10). \[\sum_{j=1}^{N}v[i,j]=1,\ \ \ \ \ \forall i\in\{1,\,2,\,\cdots,\,N\} \tag{10}\]  
Now, assume that \(v^{(t)}\) is the probability distribution of the system state at time \(t\), where \(v^{(t)}\) is equal to the product of the probability distribution of the system state at the previous time \((t-1)\) and the matrix \(Z\), as shown in Formula (11). \[v^{(t)}=v^{(t-1)}\,\times\,Z,\ \ \ \ \ \forall t\in\{1,\,2,\,\cdots,\,\infty\} \tag{11}\]  
where \(v^{(0)}\) is the initial state and represents the transition probability distribution of the system state at moment zero. The initial value of _cvnd_ is set to \(\lceil\lambda W\rceil\), indicating that the system is in the loss degradation state of the congestion avoidance phase. In addition, the initial state row vector of Markov is expressed as \(v^{(0)}\), for which the probability distribution is shown in Formula (12)  
\[v^{(0)}[j]=\left\{\begin{array}{ll}1,&j=\lceil\lambda W\rceil\\ 0,&\mbox{\of service of network services indirectly reflects the operating conditions such as whether the network is congested. Therefore, in order to accurately and objectively perform effective congestion control for vehicle communications, the paper defines a congestion sensitivity factor \(\lambda_{ODL}\), which is based on QoS and driven on-demand. The congestion-sensitive factor is a parameter for adjusting and controlling the size of the congestion window under the premise that the QoS indicators meet the service standards and the lowest cost of the normal operation of the network. According to the QoS parameter values of different services, the minimum cost required is calculated first, and then effective congestion control is achieved through the congestion-sensitive factor. First, we express the service capability evaluation index of network service as cost, which is defined as \(U\) and has \(U=[u_{o}\quad u_{d}\quad u_{l}](\sum u_{i}=1,\quad i=o,\,d,\,l)\), where \(u_{i}\) represents the cost caused by selecting and using \(i\) as an important indicator. Furthermore, \(o,\,d\) and \(l\) represent throughput, delay and packet loss rate, respectively. Usually, network performance can be evaluated by two types of metrics: positive metrics that are positively proportional to performance, and negative metrics that are inversely proportional to performance. Obviously, higher throughput implies better system performance, while higher latency and packet loss implies poorer system performance and may lead to congestion. Therefore, throughput \(o\) is a positive metric, while the other two, \(d\) and \(l\), are negative metrics. A higher cost indicates a higher spend to maintain the operation of the service. In summary, the cost function \(\mu_{i}\) is obtained by normalizing the evaluation index domain \(U\) according to the positive and negative indicators, as shown in Equation (13). \[\mu_{i}=\left\{ \begin{array}{ccc}\frac{u_{i}-u^{\min}}{u_{i}^{\max}-u_{i}^{ \min}}&,&\mbox{ \emph{is positive metric}}\\ \frac{u_{i}^{\max}-u_{i}}{u_{i}^{\max}-u_{i}^{\min}}&,&\mbox{ \emph{is negative metric}}\end{array} \right. \tag{13}\]  
For a given \(i\in\{o,\,d,\,l\}\), \(u_{i}\) is the actual measured value of evaluation index \(i\), \(u_{i}^{\min}\) is the non-permissible value of evaluation index \(i\), \(u_{i}^{\max}\) is the satisfactory value of evaluation index \(i\), and \(\mu_{i}\) is the value obtained after dimensionless processing of the index.