# Traffic Signal Control with State-Optimizing Deep Reinforcement Learning and Fuzzy Logic
## 4 Simulation Design
### Environment Setting  
The simulation environment was built using SUMO (Simulation of Urban Mobility) software version 1.18.0 and Python 3.9.17 [46]. SUMO software primarily employs microscopic models for traffic simulation. This model focuses on the behavior and interactions of individual vehicles within the traffic system. The driving actions of the vehicle in front, such as accelerating, decelerating, and stopping, affect the driving behavior of the following vehicles. The movement of each vehicle in the traffic flow is based on the following:  
* Car-Following Models: The models define the distance and speed of vehicles according to changing traffic conditions.
* Lane-Changing Models: The models determine the lane-changing behavior of vehicles based on rules such as gap and safety considerations.  
In the simulation, we used a random seed in Python to randomly generate vehicles in each direction, ensuring that each episode would always have the same traffic flow situation when repeated for comparison with other methods or to verify the results, whereas each episode has different traffic flow situations when training 500 episodes and testing 25 episodes. Vehicles built in the given situation passed the intersection only once, and the number of vehicles coming from each intersection was nearly equal.  
We provided the probability of a vehicle turning left or right instead of being straight. This was an approximate ratio of 1 to 3, respectively. One thousand vehicles were built continuously for 5400 s, which was one episode. The probability of releasing one car per second was determined by using a Weibull distribution with a shape parameter of 2.  
Figure 7 shows the number of vehicles that were built according to the settings over a single episode.  
The parameter settings for the environment are listed in Table 4. We assumed a situation for the simulation with the idea that 1000 cars would pass through the intersection in 5400 s or 90 min. Using this number of cars is appropriate and covers as it generates reasonably complex traffic situations while keeping system complexity and resource demands within manageable limits. This balance facilitates effective learning and control improvement.  
For the simulation, vehicles were classified into four categories: personal cars, minibuses, buses, and ambulances. Personal cars had a length of 4.5 m, a maximum speed of 80 km/h, and an acceleration of 1 m/s\({}^{2}\). Regarding minibuses and buses, the minibus was 7 m long, whereas the bus was 12 m long. Both had a maximum speed limit of 60 km/h and an acceleration of 1 m/s\({}^{2}\). The ambulance had a length of 6.5 m, a maximum velocity of 120 km/h, and an acceleration of 2 m/s\({}^{2}\). All vehicles had a deceleration of 4.5 m/s\({}^{2}\), and a departure speed of 36 km/h. The minimum gap between any two vehicles was 2.5 m, and a sigma (driver imperfection) value of 0.5 was used.  
We set a speed limit for general vehicles according to local driving regulations in urban areas. However, ambulances may need to drive at high speed in emergencies. Therefore, the ambulance speed limit was set higher.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Max steps (s) & 5400 \\ \hline Total cars generated & 1000 \\ \hline Total episodes & 500 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Parameter setting of the environment.  
We built ambulances with an approximately 1% probability of spotting this vehicle type, whereas buses and minibuses also had this possibility. There is an approximately 20% probability of being found, and most of the remaining probabilities apply to personal cars.