# A Novel Deep Reinforcement Learning Approach to Traffic Signal Control with Connected Vehicles +
## 4 Simulation Results
### Comparison with Baselines  
The proposed method was compared with existing methods by implementing the DRL-based traffic signal controller in [28] under the same simulation conditions. The DRL training algorithm used is similar to the one described in Algorithm 1. It is important \begin{table}
\begin{tabular}{l r} \hline \hline
**Parameter** & **Value** \\ \hline Lane length & 160 m \\ Vehicle length & 5 m \\ Time step & 1 s \\ Maximum vehicle speed & 20 (m/s) \\ Maximum vehicle acceleration & 3 (m/s\({}^{2}\)) \\ Maximum vehicle deceleration & 4.5 (m/s\({}^{2}\)) \\ Minimum gap between vehicles & 2 m \\ Car following model & Krauss Following Model [51] \\ Duration of yellow phase & 3 s \\ Traffic volume & 480 vehicles per lane and per hour \\ Left turning vehicles ratio & 25\% of total \\ Right turning vehicles ratio & 25\% of total \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Adopted parameters of simulation environment. \begin{table}
\begin{tabular}{l r} \hline \hline
**Parameter** & **Value** \\ \hline Simulated time steps for each episode & 3600 \\ Replay memory size & 20,000 \\ Minibatch size & 64 \\ Pre-train steps & 2000 \\ Target network update interval & 64 control cycles \\ Target network update rate & 0.001 \\ Discount factor & 0.99 \\ Optimizer & Adam [49] \\ Learning rate & \(1\times 10^{-4}\) \\ Initial probability of exploration & 1 \\ Final probability of exploration & 0.01 \\ Ending step for exploration probability & 40,000 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Hyper-parameters of deep reinforcement learning network. to note that the reward signal in [28] is the average waiting time and their signal control strategy involves adding or subtracting 5 s from the duration of one of the current phases. As shown in Figure 9, the performance of both average waiting time and average vehicle speed improve as the training process progresses, but its convergence is slower and more fluctuated compared to our proposed method in this paper. In addition to the proposed DRL-based traffic signal controller, two baseline methods have been implemented for comparison. These include a fixed-timing signal controller and an actuated signal controller. To determine the best fixed-timing strategy, various combinations of cycle lengths and phase duration were explored and the one with the best performance was selected. The selected cycle length was 60 s, and the four phase duration were 20 s, 10 s, 20 s, and 10 s. The actuated signal controller operates on a time-gap basis, allowing the green phase to be extended if there is a continuous flow of traffic. When the time gap between successive vehicles satisfies a predefined criterion, the signal switches to the next phase. Actuated controllers are known to perform better than fixed-timing controllers in dynamic traffic conditions. However, the traffic flow was steady and thus the actuated signal controller performed similarly to the fix-timing controller. In addition, the maximum phase duration was set to match the fix-timing controller, while the minimum phase duration was set at 5 s.  
To evaluate the performance of the proposed DRL-based control method, five common traffic mobility metrics were selected: (1) average vehicle speed, which reflects the overall mobility of the intersection, in the present moment and over a certain period of time, and represents the average travel time of all vehicles to complete the trip when computed over an episode; (2) average waiting time, which is calculated by dividing the total waiting time of vehicles by the number of vehicles present at each time step, providing an insight into travel delays from the perspective of road users; (3) average queue length, which is the average of the total number of lanes and measures the congestion level of the intersection  
Figure 8: Convergence of the proposed DRL network. Figure 9: Training history of a reference DRL-based traffic signal controller.