## 3 Intelligent Congestion Control Model Based on DRL in Heterogeneous V2X
### Design of State Space
#### 3.3.1 Probability Distribution  
\(Z\) represents the Markov transition probability matrix with \(N\) states, as shown in Formula (5)  
\[Z=\left[ \begin{array}{ccccc}v[1,1]&v[1,2]&\cdots&v[1,N-1]&v[1,N]\\ v[2,1]&v[2,2]&\cdots&v[2,N-1]&v[1,N]\\ \vdots&\vdots&\ddots&\vdots&\vdots\\ v[N-1,1]&v[N-1,2]&\cdots&v[N-1,N-1]&v[N-1,N]\\ v[N,1]&v[N,2]&\cdots&v[N,N-1]&v[N,N]\end{array} \right] \tag{5}\]  
where \(v[i,\ j]\) represents the transition probability of the system from state \(i\)th to state \(j\)th, \(i\), \(j\in\{1,\ 2,\ \cdots,N\}\). \(S_{cw}\) is used to represent the size of the congestion window _cvnd_, and the size of the congestion window _cvnd_ in the \(i\)th state is represented by \(S_{cw_{i}}\). \(\{S_{cw_{1}},\ S_{cw_{2}},\ \cdots,\ S_{cw_{N}}\}\) represents the range of congestion window sizes, which represents a system with \(N\) states. Where, \(S_{cw_{1}}=cvnd_{\min}\) and \(S_{cw_{N}}=cvnd_{\max}\), state \(N\) can be obtained from Formula (6)  
\[N=cvnd_{\max}-cvnd_{\min}\ +1 \tag{6}\]  
where \(cvnd_{\max}\) represents the maximum size of the allowed congestion window, and \(cvnd_{\min}\) represents the minimum size of the congestion window. Assume that there are congestion windows \(cvnd_{\max}=7\) and \(cvnd_{\min}=2\) in the network of vehicle communications. Therefore, according to Formula (6), the system has only 6 states, and the set of system states is represented as \(S_{cw_{i}}\in\{2,\ 3,\ 4,\ 5,\ 6,\ 7\}\), \(i\in\{1,\ 2,\ \cdots,N\}\). Moreover, the state transition probability \(P_{x}(S_{cw})\) of the system from state \(i\)th to state \(j\)th can be calculated by the probability distribution \(v[i,\ j]\) described by Formula (7)  
\[v[i,\ j]=\left\{ \begin{array}{ll}P_{x}(S_{cw}),&i=1,\quad j=\lceil\lambda \times i\rceil\\ P_{x}(S_{cw}),&i=[2,\ \cdots,N-1]\,\ j=\lfloor\lambda\times i\rfloor\\ 1-P_{x}(S_{cw}),&i=[1,\ \cdots,N-1]\,\ j=i+1\\ 1,&i=N,\ j=\lfloor\lambda\times i\rfloor\\ 0,&\text{otherwise}\end{array} \right. \tag{7}\]  
where, \(P_{x}(S_{cw})\) represents the probability of the transition from state \(i\)th to state \(j\)th, and \(i\), \(j\in\{1,\ 2,\ \cdots,N\}\). In addition, \(\lambda\) is the congestion-sensitive factor, indicating in the control function that congestion may be caused by QoS parameters such as throughput, delay and packet loss in different network service application requirements. The congestion sensitivity factor \(\lambda\) is described in detail in Section 3.3.2. Assume \(i\), \(j\in\{1,\ 2,\ \cdots,N\}\)\(N=6\), \(\lambda=0.8\), \(S_{cw_{i}}\in\{2,\ 3,\ 4,\ 5,\ 6,\ 7\}\) here, and the state transition diagram of the system is shown in Figure 3.  
All transfer probabilities shown in Figure 3 are put into the corresponding relevant positions of transfer matrix \(Z\), where zero represents no transfer, and the transfer matrix is obtained as shown in Formula (8). Assume \(N=6\), \(\lambda=0.8\), \(S_{cw_{i}}\in\{2,\ 3,\ 4,\ 5,\ 6,\ 7\}\) and \(i\in\{1,\ 2,\ \cdots,N\}\) here. Figure 3: Example of [PERSON]’s State Transition Diagram. \[Z=\left[ \begin{array}{cccccc}j&1&2&3&4&5&6&i\\ P_{x}(S_{cv_{1}})&1-P_{x}(S_{cv_{1}})&0&0&0&0\\ P_{x}(S_{cv_{2}})&0&1-P_{x}(S_{cv_{2}})&0&0&0\\ 0&P_{x}(S_{cv_{3}})&0&1-P_{x}(S_{cv_{3}})&0&0\\ 0&0&P_{x}(S_{cv_{4}})&0&1-P_{x}(S_{cv_{4}})&0\\ 0&0&0&P_{x}(S_{cv_{5}})&0&1-P_{x}(S_{cv_{5}})\\ 0&0&0&0&1&0\\ \end{array} \right] \tag{8}\]  
where \(v_{i}\) is the vector of the \(i\)th row in the matrix \(Z\), representing the probability of the system transferring from state \(i\)th to all other \(N\) states of the system. As shown in Formula (9), each element represents the probability of the system transferring from state \(i\)th to state \(j\)th.