## 4 Design of the DRL Algorithm for the Traffic Signal Control
### Action Representation  
Actions for the DRL approach applied in ATSC are configured to tackle two major problems related to traffic signal plan design: (a) scheduling strategy for Traffic Signal Phases (TSP); and (b) computation of each TSP duration. Most of the current DRL approaches applied in ATSC design use a discrete action space. That approach ensures easy implementation of the mentioned traffic signal control approach in simulation and real-world environment. The TSPs are usually denoted by the numerical indexes in the DRL framework. Compared to the active one, all other TSPs are in the red state. Scheduling of the TSPs sequencing can be fixed/cyclic (e.g., [18; 21]) or acyclic (e.g., [46]). In cyclic TSP sequencing, it is possible to conclude that there are only two possible actions: (1) switch to the next phase in the predefined sequence order; or (2) stay on the same TSP. In the DRL environment that control strategy is known as the binary control decision since there are only two possible outputs of the DNN model. Furthermore, this approach usually requires additional intermediate TSP with a yellow and red light when the currently active TSP is about to change. However, the binary control decision does not clearly indicate which specific traffic streams the agent can control in the case of a multi-phase signalized intersection [52]. In binary control decisions, it is possible to get different rewards for similar state-action pairs what can lead to non-stationary distributions and consequently induce instability in the learning process. In the acyclic TSP approach, it is necessary to index each TSP with a unique value, since the agent chooses a TSP from predefined TSP space [9; 18]. In this case, TSPs scheduling is done exclusively according to the current state without any need for following predefined TSP sequencing. Furthermore, one TSP can have numerous versions of itself regarding its green light duration. Action space in the aforementioned case contains a set of predefined feasible green light duration setups for each TSP, as described in [9]. Since there is no need for computation of each TSP green light duration this approach is suitable for the DRL frameworks which evaluates each TSP modification as the one possible action. Furthermore, there is usually an additional yellow phase with a fixed time duration which is activated if the traffic light is switched from green to red and vice versa. The drawback of this approach is an extensive enlargement of the state-action space, which can dramatically increase computational cost. Furthermore, this approach in some cases can violate the traffic regulations which proscribe a fixed phasing sequence. Figure 5: Most used traffic state configurations for the large network of intersections. Duration of each TSP can be presented by \(n\)-tuple, where \(n\) denotes the number of TSPs. Each element of this tuple represents one TSP, and value of this element represents duration of green light in that particular TSP. Furthermore, these values can represent scaling factors for each existing TSP what preserves total duration of the cycle [20]. [PERSON] et al. [17] introduced a significance of the distance between adjacent intersections. The impact of interactions between different intersections will weaken as the distance becomes larger between them. The same study proposes a concept of Adjusting Matrix of Traffic Signal Phase Control (AMTSPC). Firstly,\(MTSPC_{t}\) matrix for all agents' control action at the time step \(t\) is given as follows \(MTSPC_{t}=\left[A_{1t},A_{2t},...,A_{Nt}\right]^{T}\) where \(N\) is the number of agents. Adjusted action matrix \(MTSPC_{it}\) is created by applying the impact coefficient computed as \(\left(1-\frac{d_{k}}{D}\right)\) at the particular row of \(MTSPC_{t}\), where \(d_{ik}\) is the distance from intersection \(i\) to the adjacent intersection \(k\) and \(D\) is a radius within which the other intersections are relevant for particular intersection \(i\).