# Traffic Signal Control with State-Optimizing Deep Reinforcement Learning and Fuzzy Logic
## 1 Introduction
It effectively handles large, intricate state spaces and is widely adopted in research due to its simplicity and effectiveness, where it improves learning by using experience replay. In this method, a neural network replaces the Q-table, which simplifies configuration and tuning without requiring an overly complex design, predicts Q-values, and adjusts weights in the neural network to approximate and adapt to the most suitable Q-values for a given situation. This contrasts with traditional Q-learning, in which learning occurs from the beginning in every situation, thus making the Deep Q-network more flexible [19]. However, their fundamental principles are similar. In this method, the state refers to the collection of situational data, such as the car's position [20], speed [21; 22], and the number of vehicles in each lane [23]. In the action of a Deep Q-network, the optimal traffic signal phase for a given situation is selected. However, the duration of the green signal is predefined and does not dynamically adapt to the current situation [23; 24]. The calculation of reward values varies across studies, with the predominant factor being waiting time [24; 25], and some studies alternatively use queue length and car speed [21]. In traffic control using reinforcement learning (RL), in addition to the widely popular DQN (Deep Q-network), there are other methods such as the TD3 (Twin Delayed Deep Deterministic Policy Gradient) and DDPG (Deep Deterministic Policy Gradient). These methods show high potential in handling problems with a continuous action space or more complex control requirements, such as having continuous values for autonomous vehicles, which allows the vehicle to adapt smoothly to its environment and temperature control, where inputs must be adjusted continuously to maintain the desired state. For DDPG [26], the Actor-Critic approach is employed, where the Actor determines the control policy and the Critic evaluates this policy, making it effective for continuous control scenarios. To address problems in the DDPG, the TD3 [27] was developed by implementing delayed policy updates and using a clipped double Q-learning technique, which enhances the model's stability. However, the DQN is more suitable for problems that require decision-making in discrete action spaces [28], such as adjusting traffic signal phases where a new phase is selected only after the previous one has finished. It handles these problems more effectively and straightforwardly and is also simpler in structure and easier to implement. In their study, [PERSON] and [PERSON] [29] used a combination of deep Q-learning and fuzzy logic control to change the traffic light phase and duration of the green light according to the current situation. They employed Deep Q-learning to select the optimal traffic signal phase, while using fuzzy logic control to compute the duration of the green signal. The results showed that this approach outperformed other methods, such as traditional fixed-time control, Deep Q-learning, and fuzzy logic control. However, one problem discovered in that study is related to the right-hand traffic environment. In particular, the reward function that is used may calculate an unsuitable reward. Consequently, vehicles wishing to turn left experience excessive waiting times at traffic signals. Owing to the reward process, it is reasonable to assume that greater efficiency can be achieved by giving vehicles a signal to go straight. Therefore, in our study, we use a modified Deep Q-network and fuzzy logic control method to optimize the management of traffic lights at single intersections and enhance efficacy. The following characteristics have been improved. * State improvement: We have edited this to reflect the perspective of the people who use it to make traffic light control decisions. The state collects information regarding the waiting time, divided into ten intervals, and the vehicle's position.