## 4 Design of the DRL Algorithm for the Traffic Signal Control
### State Representation  
The DNN, as the essential part of DRL, reacquires a pre-processing step for obtained raw data. These data must be structured as the image-like matrices (2D matrices) or sounds-like vectors (i.e., 1D signals) to be digested by the DNN input layer (usually, it is convolution layer). Transformation of the obtained raw traffic data into the mentioned data structures can be difficult since the natural representation of the traffic flow measurements is done by the use of a labelled graph. The common approach for mentioned data transformation is to _pixelate_ or segmentate the surface of the road (and sometimes its nearby road surrounding) into the small segments, partitions, or cells. The next step is to assign obtained raw traffic data to those segments with respect to their measurement location [5; 13; 25]. The height of each segment is usually specified by the width of the traffic lane (or several lanes which are close to each other), and its length depends on the technique that is used for obtaining the traffic data. When these data contain parameters of each vehicle such as position, acceleration, speed, etc., as described by the early applications in [4; 16; 50], the length of a segment corresponds to the average or minimum length of the vehicle, as illustrated in Figure 3a. Furthermore, the same road segmentation approach can be used for traffic data such as the number of passengers per road segment [13] and the direction of the moving vehicles [17], or to conduct raw vehicle speeds post-processing in the form of their normalized values. It is also possible to use a larger uniform length of a segment if the traffic data are aggregated to some extent, e.g. the number of halting vehicles, occupancy of road segments, cumulative waiting time per segment, and percentage of the vehicles that have achieved maximum allowed speed [25]. The above-mentioned road segmentation is illustrated at Figure 3b. It is possible to conclude that both approaches conduct spatial discretization of each intersection bound in a _cellular automata_ fashion. Furthermore, a novel approach based on the aggregated traffic data and variable road segmentation is proposed in [51], which is illustrated in Figure 3c. Incoming lanes at each intersection bound are divided into segments with different sizes. The further the segment is from the stop line, the longer it is since it is less important for the traffic control. This approach enables longer lane length coverage for each intersection bound [51]. The segment is marked as occupied if it is at least one vehicle in it. Additionally, several image-like matrices can be stacked as the single multi-channel input sequence or tensor-like object for the DNN model digestion. It is possible to stack several image-like matrices for the current time step, but with different content regrading the traffic data or it is possible to stack several image-like matrices with the same content but measured in several consecutive time intervals within the one action step. There are several methodologies for coding traffic scenarios through the use of image-like matrices. Figure 4a illustrates the most basic approach where the segmented quadratic region corresponds to the intersection and its nearby surrounding. Segments that correspond with the intersection bounds are highlighted in grey. If they are occupied by vehicles, they have value 1; otherwise, they have value 0. The drawback of this approach is the appearance of the redundant parts in the image-like matrix denoted by the white color in the aforementioned figure. They represent areas that are not part of the particular intersection road surface thus they are not significant for learning  
[PERSON] and [PERSON] [13] proposed segmentation into the smaller uniform segments of each intersection bound per traffic lanes starting from the stop line at each bound. All those segmented lanes are stacked below each other into the one image-like matrix according to their intersection-bound affiliation.