## 4 Fldqn
### Algorithm Description  
This subsection introduces and explains the proposed algorithm, FLDQN (Algorithm 1). The FLDQN is a distributed RL approach designed to foster collaboration among multiple agents, enabling them to learn from shared experiences and enhance decision-making. The algorithm unfolds in two main phases: server- and agent-side execution. In the foregoing discussion, we delve into the specifics of each phase:  
**Server side:** In FLDQN, the server plays a crucial role in coordinating the training process and facilitating communication between agents. On the server side, denoted in lines 1 to 8 of Algorithm 1, the global model \(W_{global}^{(t)}\) is shared with all agents through the UpdateAgent() function. This initializes a baseline for all connected agents, ensuring consistent starting points. The server orchestrates the synchronization of local model updates by waiting for all agents to submit their changes. Upon receiving these updates, the server aggregates and averages them, creating the new global model \(W_{global}^{(t+1)}\). This process, known as aggregation, fosters collaboration and knowledge sharing among agents. Subsequently, the updated global model \(W_{global}^{(t+1)}\) is broadcast back to all agents, maintaining consistency for future learning rounds. ```
1 Initialize \(W_{global}^{(t)}\), episodes, and rounds
2 for each round \(t=1,2,\ldots\)do
3 for\(n=1\)to\(N\)do
4\(w_{t+1}^{n}\gets UpdateAgent(w,episodes)\)
5 end for
6\(W_{global}^{(t+1)}\leftarrow\sum_{t=1}^{N}w_{t+1}^{n}\)
7 Server broadcast \(W_{global}^{(t+1)}\) to all agents. 8
9
10 end for
11
12 functionUpdateAgent(\(w,episodes\)):
13 Initialize \(\epsilon\)
14\(D\leftarrow(s_{t},a_{t},r_{t}(s_{t},a_{t}),s_{t+1})\) /* initialize the replay memory \(D\) to store experience. Equation (6) */
15 Q-function \(\leftarrow\)random(\(\Theta\)) /* initialize the \(Q\) function with random weight \(\Theta\). (2) */
16\(\Theta^{\prime}\leftarrow\Theta\) /* initialize the target network action-value function \(Q^{\prime}\) with weight \(\Theta^{\prime}\). Equation (3) */
17 foreachepisode \(\in\)episodesdo
18 for\(step\in\)episodedo
19 if\(rand<\epsilon\)then
20\(a_{t}\gets random(A)\) /* select a random action \(a_{t}\) from action space (A). */
21 else
22\(a_{t}\gets argmaxQ(s,a,\Theta)\) /* select the action \(a_{t}\) that maximizes the Q-value. */
23
24 end for
25 Execute \(a_{t}\), in SUMO environment, obtain reward \(r_{t}(s_{t},a_{t})\) and next state \(s_{t+1}\). \(D\leftarrow(s_{t},a_{t},r_{t}(s_{t},a_{t}),s_{t+1})\) /* store the transition in a replay memory D. Equation (6) */
26 sample random mini batch of transition \((s_{t},a_{t},r_{t}(s_{t},a_{t}),s_{t+1})\) from \(D\) /* Equation (6) */
27 if\(s_{t+1}\) is terminalthen
28\(y\longleftarrow r\)
29 else
30\(y\longleftarrow r+\gamma maxQ(s,a,\Theta)\)
31 end if
32\((y-Q(s,a,\Theta))^{2}\) /* update model parameters. Equation (5) */
33 end for
34
35 end for
36 returnupdated weights to the server
37
38 end function
```  
**Algorithm 1**FLDQN algorithm  
**Agent side:** The UpdateAgent() function (lines 9-33) is executed independently by each agent and takes two parameters, \"\(w\)\" and \"\(episodes\)\" as input. Here, \"\(w\)\" represents the global model parameters received from the server, and \"\(episodes\)\" indicates the number of iterations that an agent will undergo in each round. During the initialization phase (lines 10-13), the agent first initializes the \(e\), which is used later to determine the actions taken by the agent. Then, the replay memory to store experiences is initialized (line 11). The Q-function is initialized to set action values with random weights using Equation (2) (line 12). Next, the agent initializes the target network to provide stable target values during training; \(\Theta^{\prime}\) is the weight of the target network and \(\Theta\) is the weight of the main network, and it indicates to set the weight of the main network to the target network using Equation (3) (line 13).