## 5 RL Elements
### State
#### 5.3.1 State Definition  
A state observation zone that is too long may slow training because it introduces too much information irrelevant to short-term state transitions; one that is too short may result in the deficiency of essential information for state and reward predictions.  
On each intersection leg, the state observation zone should cover the entire approach lanes, the entire approach taper, and a portion of the upstream link, as illustrated in Figure 3. In this study, we set the length of the state observation zone as 150 m.  
We represent state in a grid-based mode. The state observation zone is classified into phase-related grids and phase-unrelated grids. Each grid is _LEN_ meters long and is as wide as the lane width. In this study, _LEN_ equals 4 to ensure that a grid cannot be occupied by two vehicles at the same time. As shown in Figure 3, the phase-related grids start from the stop line, and cover the portion of the approach lane length that is divisible by _LEN_. The phase-unrelated grids follow the phase-related grids and cover the rest of the state observation zone.  
On the \(b\)th intersection leg, the state features of the phase-related grids on the \(u\)th approach lane group are organized as a three-dimensional state matrix \(MAT^{b,u}\), given by:  
\[MAT^{b,u}=\left(\begin{array}{cccc}f^{b,u}_{1,1}&f^{b,u}_{1,2}&\cdots&f^{b,u }_{1,C^{b,u}}\\ f^{b,u}_{2,1}&f^{b,u}_{2,2}&\cdots&f^{b,u}_{2,C^{b,u}}\\ \vdots&\vdots&\ddots&\vdots\\ f^{b,u}_{H^{b,u},1}&f^{b,u}_{H^{b,u},2}&\cdots&f^{b,u}_{H^{b,u},C^{b,u}}\\ \end{array}\right) \tag{3}\]  
where \(H^{b,u}\) and \(C^{b,u}\) are the numbers of rows and columns in the matrix, respectively; and \(f^{b,u}_{h,c}\) is the state feature of the phase-related grid at the \(h^{\text{th}}\) row and \(c^{\text{th}}\) column, which is a 3-tuple given by:  
\[f^{b,u}_{h,c}=\left\langle{o^{b,u}_{h,c}v^{b,u}_{h,c}\cdot s^{b,u}_{h,c}}\right\rangle \tag{4}\]  
If the front end of a vehicle is present in a phase-related grid, \(o^{b,u}_{h,c}\) is set to 1 and \(v^{b,u}_{h,c}\) is set to its immediate speed normalized by the posted speed limit. Otherwise, \(o^{b,u}_{h,c}\) and \(v^{b,u}_{h,c}\) are both set to 0. If the vehicle phase related to the grid displays green, \(g^{b,u}_{h,c}\) is set to 1, otherwise 0.  
Similarly, the state features of the phase-unrelated grids on the \(b\)th intersection leg are organized as a three-dimensional state matrix, given by:  
\[MAT^{b}=\left(\begin{array}{cccc}f^{b}_{1,1}&f^{b}_{1,2}&\cdots&f^{b}_{1,C^{b}} \\ f^{b}_{2,1}&f^{b}_{2,2}&\cdots&f^{b}_{2,C^{b}}\\ \vdots&\vdots&\ddots&\vdots\\ f^{b}_{H^{b},1}&f^{b}_{H^{b},2}&\cdots&f^{b}_{H^{b},C^{b}}\end{array}\right) \tag{5}\]  
where \(H^{b}\) and \(C^{b}\) are the numbers of rows and columns in the matrix, respectively; and \(f^{b}_{h,c}\) is the state feature of the phase-unrelated grid at the \(h\)th row and \(c\)th column, which is a 2-tuple given by:  
\[f^{b}_{h,c}=\left\langle{o^{b}_{h,c},v^{b}_{h,c}}\right\rangle \tag{6}\]  
\(o^{b}_{h,c}\) and \(v^{b}_{h,c}\) are set in the same way as \(o^{b,u}_{h,c}\) and \(v^{b,u}_{h,c}\) for the phase-related grids.  
Lastly, our state consists of the state matrices for the phase-related and phase-unrelated grids on all the intersection legs, given by:  
Figure 3: State representation and neural network structure.  
\[\begin{split} S_{i}=\Big{(}MAT^{1,1},MAT^{1,2},\ldots,MAT^{1,UI^{1}}, MAT^{1},\\ MAT^{2,1},MAT^{2,2},\ldots,MAT^{B,UI^{B}},MAT^{B}\Big{)}\end{split} \tag{7}\]  
where \(S_{i}\) is the state at time step \(i\); \(U^{b}\) is the number of approach lane groups on the \(b\)th intersection leg; and \(B\) is the number of intersection legs.