## 4 Design of the DRL Algorithm for the Traffic Signal Control
### Reward Function  
The reward function in RL is used to evaluate the impact of an action on its environment [54]. It is represented by the set of scoring values computed for each action interval. During the learning process, RL must maximize its cumulative reward that agents receive in the long run. In early studies regarding the application of DRL for the ATSC algorithm design, the reward function for one intersection was defined as the change in the cumulative vehicle delay between the actions [16]. Furthermore, in some early studies, the reward was computed as the cumulative waiting time of vehicles measured from the point of time when those vehicles have entered particular intersection bound until the start of green light [21]. [PERSON] and [PERSON] [48] extended the previously mentioned reward design by using delay instead of waiting time, and simultaneously introduced the concept of _team reward_ for several intersections. The mentioned approach is described as follows:  
\[r_{t}=\mathbb{R}(S_{t},A_{t})=W_{t}-W_{t+1}. \tag{7}\]  
\[W_{t}=\sum_{i=1}^{N}w_{t}^{i},W_{t+1}=\sum_{i=1}^{N}w_{t+1}^{i}, \tag{8}\]  
\[w_{t}^{i}=\sum_{j_{t}=1}^{M_{t}}w_{j_{t},t}^{i},w_{t+1}^{i}=\sum_{j_{t}=1}^{M_ {t}}w_{j_{t+1},t+1}^{i}, \tag{9}\]  
where \(j_{t}\) is the the \(j\)th observed vehicle in the \(t\)th time step, \(M_{t}\) is the total number of vehicles until the end of \(t\)th time step, \(w_{j_{t},t}^{i}\) is the computed delay of the vehicle \(j\) at the \(i\)th intersection until the end of \(t\)th time step, \(N\) is the number of intersection at the one region (under condition that traffic network is divided into the several regions), \(w_{t}^{i}\) is the the sum of delay of all vehicles at the \(i\)th intersection until the \(t\)th time step, and \(W_{t}\) is the sum of cumulative delay of all vehicles at all intersections until \(t\)th time step. The latest reward functions used for the complex traffic networks are commonly based on the following Key Performance Indicators (KPIs); (1) the average delay, which is computed for all vehicles according to the average waiting time of each vehicle at a single of multiple intersections; (2) the throughput, which is computed as the total traffic flow at one or multi-intersection scenarios; (3) the Average Travelling Time (ATT) of vehicles, which denotes the average traveling time through the one or more intersections; (4) the number of arrival vehicles, which indicates the number of vehicles that have arrived at their destination through the controlled area in a given period of time [18]; and (5) the number of queuing vehicle at each intersection bound [54]. [PERSON] et al. [55] addressed the problem related to global vehicle delay minimization since the DRL based agent can favor specific TSPs affected by the large traffic load. That might cause an unusual long extension of red signal time for TSP with lower traffic load. To alleviate that problem, the reward function, inspired by the _U.S. Bureau of Public Roads (BPR)_ function used for transportation planning, is introduced as follows:  
\[r_{t}=\frac{1}{N}\sum_{i=1}^{N}T_{u}\left[\eta-\eta\left(\frac{w^{i}}{C} \right)^{\tau}\right]. \tag{10}\]  
where \(w^{i}\) is current waiting time for vehicle \(i\), \(T_{u}\) is one unit of time, \(C\) is the generally acceptable waiting time, and \(\eta\) and \(\tau\) are constants which are set as \(\eta=0,15\) and \(\tau=2\), respectively, according to the BPR. multi-objective reward function. [PERSON] et al. [52] concluded that it is hard to compute average delay in real-life traffic environment without the presence of Connected and Autonomous Vehicles (CAVs). It is very difficult to measure sufficiently accurately waiting time for each vehicle at the intersection. The same study proposes a novel and simple approach for reward computation based on the difference between the number of vehicles that passed through the stop line during the last time interval and those that could not pass the intersection and therefore they stayed in the queue.