## 4 Fldqn
### Architecture  
In this subsection, we delve into a detailed explanation of the FLDQN algorithm, explaining each key aspect and component. Figure 3 illustrates the architectural diagram for a clearer understanding.  
1. **Federated Server:** The federated server initiates the process by distributing an initial global model \(W_{global}^{(t)}\) to all connected agents. Each agent will receive \(W_{global}^{(t)}\) and start training the model on local data, after local training, each agent transmits their updated weights \(w_{i}^{(t)}\) back to the federated server. After receiving the local updated weights from all the connected agents, the server then averages these local updated weights to construct an updated global model \(W_{global}^{(t+1)}\). This iterative process continues until predefined criteria are met. This weight-sharing approach allows agents to undergo simultaneous training by leveraging the collective experience of all agents, even though each may have distinct observations. Throughout training, each agent utilizes its own DQN model, evolving its own hidden states \(S_{t}\), making independent action selections, and communicating solely through the federated server. This distributed approach ensures autonomy for each agent while contributing to the cooperative  
Figure 3: The proposed architecture of FLDQN involves a federated server that coordinates the actions of multiple agents in an SUMO simulation environment. The agents operate within the SUMO environment. After individually training models in the SUMO environment, each agent transmits its trained model back to the server for subsequent processing.  
behavior of the entire system. This promotes knowledge sharing and cooperation, aiming for higher rewards \(R_{t}\) compared to non-cooperative approaches.
2. **Agents:** Agents are SUMO-based intelligent vehicles equipped with a DQN model. They autonomously take actions based on dynamic road and traffic conditions (states). The role of agents is to train their DQN models and frequently send these model updates to the federated server.
3. **SUMO:** The SUMO simulator serves as the environment for generating realistic traffic scenarios. In our scenario, SUMO facilitates the creation of the road network traffic and provides an interface for interaction with the environment using Python APIs.
4. **States:** States are crucial in any RL-based approach, serving as input to the algorithm during the training phase. In our case, the state space at any time-step \(t\) is represented as \[S_{t}=\{N_{vch},A_{spead},L_{road},A_{loc},A_{des}\}\] This representation consists of five elements, each serving a specific purpose: * \(N_{vch}\): Number of vehicles on the current road. * \(A_{spead}\): Maximum allowed speed. * \(L_{road}\): Length of the current road. * \(A_{loc}\): Current position of the agent. * \(A_{des}\): Final destination of the agent.
5. **Reward:** FLDQN employs a unique reward mechanism, which is denoted in Equation (8): \[R_{s_{t}}=-(T_{s_{t+1}}-T_{s_{t}})\] (8) Here, an agent receives a total reward, \(R_{s_{t}}\), for executing a specific action or reaching a particular state at a given moment. Additionally, \(T_{s_{t+1}}\) denotes the total travel time to state \(s_{t+1}\), while \(T_{s_{t}}\) indicates the total travel time to state \(s_{t}\). This approach differs from traditional methods of setting rewards in sequential decision-making processes, which can suffer from long delays. This reward mechanism aims to address this issue, and as a result, it significantly improves the convergence performance of the FLDQN as compared with other discounted reward schemes.