## 5 Performance Evaluation  
This section evaluates the comparative performance between DQN and FLDQN within the SUMO simulation environment. The assessment will focus on the key performance metrics, including travel time, running time, and the effect of the number of agents in FLDQN, where the running time is defined as the duration required to execute experiments and specific episodes. By analyzing these aspects, we aim to provide a detailed understanding of how FLDQN performs in optimizing travel time and running time in the SUMO environment. **Evaluation Setup:** FLDQN is implemented using Python v3.8, Flower v1.4.0 [45] is used for FL, and TensorFlow v2.12.0 [46] serves as the underlying Deep Learning library. For simulation purposes, Eclipse SUMO Version v1.14.1 [40] was utilized. In our experiments, five agents were deployed, with a federated server, serving as the central point for training and coordination. All experiments were conducted on a machine equipped with an Intel i7 processor and 64 GB of RAM. To ensure consistent and reliable results, each experiment was run for three trials with different random seeds while keeping other hyperparameters the same across all the trials for both DQN and FLDQN. The final values were taken as the average of the values of the three trials. Due to the large number of episodes (5000), significant fluctuations in performance were observed. To address this, a moving average with a window size of 100 was applied to smooth the results. The detailed hyperparameters used can be found in Table 1.  
**Model Architecture:** We employ a five-layer feed-forward neural network with three hidden layers, one input layer, and one output layer. The activation function used throughout the hidden layers is the Rectified Linear Unit (ReLU) [47]. The first dense layer has 150 units, the second dense layer has 100 units, and the third dense layer, using linear activation, has the same number of units as there are actions in the environment. The architecture employs a dueling network structure, characterized by two streams of fully connected layers: the advantage stream and the value stream. The advantage stream, which represents all of the connected roads in the urban network, generates an equal number of outputs as actions. The value stream, in contrast, has a single output. In terms of hyperparameters, for DQN, we set the number of episodes to 5000. For FLDQN, configured for FL setting with a round concept [48], the number of episodes is set to 100 per round with a total of 50 rounds, which is equivalent to a total of 5000 episodes. These hyperparameters provide the framework for training and evaluating the model's performance. All the hyperparameters that we used in our experiments are shown in Table 1. Here, \"Fraction-fit\" determines the fraction of available agents used for model updates, set to 1.0 for full participation. \"Min-fit-clients\" specifies the minimum number of agents required for model fitting or training, set to 5. \"Min-available-clients\" sets the minimum number of available clients needed for FL rounds, set to 5, \"Episodes\" defines the number of training episodes per learning iteration, set to 100. \"Rounds\" specifies the number of FL rounds, set to 50. \"Discount-factor\" is set to 0.99. \"Learning-rate\" defines the step size for updating the model parameters during training, set to 0.001. \"Epsilon\" controls \begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameters** & **Values** \\ \hline Fraction-fit & 1.0 \\ \hline Min-fit-clients & 5 \\ \hline Min-available-clients & 5 \\ \hline Episodes & 100 \\ \hline Rounds & 50 \\ \hline Discount-factor & 0.99 \\ \hline Learning-rate & 0.001 \\ \hline Epsilon & 1.0 \\ \hline Epsilon-Decay & 0.95 \\ \hline Epsilon-min & 0.05 \\ \hline Batch-size & 32 \\ \hline Train-starter & 200 \\ \hline Reply-Memory & 100,000 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Table displaying the comprehensive set of hyperparameters utilized in our proposed FLDQN experiments.