# Traffic Signal Control with State-Optimizing Deep Reinforcement Learning and Fuzzy Logic
## 4 Simulation Design
### Network Setting  
Regarding the neural network structure, a Deep Q-network was constructed using Keras and TensorFlow. The input layer of the neural network receives the state obtained from the environment. The input comprised 80 nodes with two hidden layers, each consisting of 100 nodes. The activation function used in each hidden layer is a Rectified Linear Unit (ReLU). Each value obtained from each node in the preceding layer is weighted before further processing, which the first hidden layer receives from the input layer. The output of the first hidden layer is passed to the second hidden layer. The outputs of the second hidden layer are sent to the output layer, which calculates the Q-values for the four action phases. The learning parameters used in the experiments are listed in Table 3.  
We describe the system's learning process in Algorithm 1. At the beginning of each episode, the epsilon value is calculated to determine whether to explore or exploit. Subsequently, the system controls the traffic light based on the epsilon value and stores the state, action, reward, and next state in memory. We utilized the deque data structure from Python's collections module to efficiently manage this memory. The process continues until the maximum step is reached. Afterward, the episode statistics are saved, and the training process known as experience replay is initiated. This important training phase involves using data on the state, action, reward, and next state to enhance learning from experience. Finally, it iterates until the total number of episodes is completed, at which point the model is saved and prepared for testing.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Category** & **Necessary Information** & **Conditions** & **Cost** \\ \hline \multirow{2}{*}{Data collection devices} & Number and position & Installation of sensors, & Cost for installation, \\  & of vehicles & cameras, or detection devices & maintenance, and time required \\  & & & for data processing \\ \hline \multirow{2}{*}{Processing systems and servers} & Data storage, processing, & Data management, data & Cost for hardware, software, \\  & model training & processing, setting up servers, & and time required for system \\  & & and security & development and management \\ \hline \multirow{2}{*}{Traffic signal control systems} & Current signal, phase & Integration with existing & \\  & duration, and control. & traffic control systems or new & Cost for integration or \\ \multicolumn{4}{c}{system installation} \\ \hline \hline \end{tabular}
\end{table}
Table 2: Necessary information for deep reinforcement learning-based traffic signal control systems.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Hidden layers & 2 \\ \hline Nodes in hidden layers & 100 \\ \hline Batch size & 64 \\ \hline Learning rate & 0.001 \\ \hline Training epochs & 500 \\ \hline Memory size & 50,000 \\ \hline Discount factor & 0.75 \\ \hline Number of states (Input) & 80 \\ \hline Number of actions (Output) & 4 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Parameter setting of neural network.