## 2 Background and Related Work
### DRL Framework for Traffic Signal Control  
To manage traffic efficiently with data collected via VLC, a Multi-Agent Reinforcement Learning (MARL) system was implemented. Three agents (C0, C1, C2) were considered. The traffic scenario analyzed in this study is depicted in Figure 2. It includes three uniform 4-arm junctions, spaced at varying distances. The lane between junctions C0 and C1 is 400 m long, while the lane between C1 and C2 is only 200 m long. Each intersection is managed by an agent that maps its environment into cells, gathering data on vehicles and pedestrians. Communication between the infrastructure and vehicles ensures a cooperative information-sharing setup. Traffic data collected by the agents are stored in a shared memory, allowing a single network to be trained for controlling all intersections and selecting the best phase for each. A Deep-Q Network (DQN), trained using deep Q-learning, optimizes the system. Instead of traditional tabular Q-learning, which stores Q-values for each state-action pair, the DQN uses a neural network (NN) to predict Q-values based on the current state. This approach handles real-world scenarios with large, continuous state-action spaces. Figure 3 illustrates the simulation and training process. Figure 2: Environment of the arterial scenario. At each time step \(t\), the network receives the current state \(st\), executes an action \(at\), and transitions to the next state \(s_{t+1}\). A reward \(r_{t}\) evaluates the action's effectiveness based on vehicle and pedestrian waiting times (Equation (1)). Waiting time (\(wt_{vch,t}/wt_{ped,t}\)) is the duration a vehicle or pedestrian moves at less than 0.1 m/s since entering the environment. \(n\) represents the total number of vehicles/pedestrians in the environment in \(t\). This value, \(atwt_{t}\), accumulates until they cross the intersection, ensuring the reward reflects both traffic flow and pedestrian efficiency. \[atwt_{vch,t}=\sum_{vch=1}^{n}wt_{(vch,t)}\quad atwt_{ped,t}=\sum_{ped=1}^{n}wt _{(pot,t)} \tag{1}\]  
The final reward equation, \(r_{t}\), is defined in Equation (2), where \(atwt_{t}\) and \(atwt_{t-1}\) are the accumulated total waiting time of all the cars/pedestrians in the intersection captured, respectively, at \(t\) and \(t-1\). The weights of the \(p_{vch}\) and \(p_{ped}\) are set based on the desired priority that the agent should have towards vehicles and pedestrians during network training. The agent will learn a policy that benefits one more than the other or keeps the system balanced if the weights are equal. \[r_{t}=p_{vch}(atwt_{vch,t-1}-atwt_{vch,t})+p_{ped}\Big{(}atwt_{ped,t-1}-atwt_{ped,t}\Big{)} \tag{2}\]  
This experience ex = (\(s_{t}\), \(a_{t}\), \(r_{t}\), \(s_{t+1}\)) will be stored in the replay memory, to be used in the future to train the agent. The replay memory is a dataset of an agent's experiences Dt = (\(e_{1}\), \(e_{2}\), \(\ldots\), \(e_{t}\)), which are gathered when the agent interact with the environment as time goes by (\(t=1\), \(2\ldots\), \(n\)). In training, a mini batch of random samples is selected to break the temporal correlation between consecutive experiences, preventing inefficient learning from highly correlated data. The replay memory buffer fills to a specific length, and, when full, older experiences are overwritten by new ones. The neural network architecture implemented consists of a fully connected layer network (FCLN), and the weights \(\theta\)k of the FCLN are used to approximate its Q-values Q(s, a; \(\theta\)k). The first layer of the network is the input layer, formed by an input layer of 164 neurons, representing the state of the environment. Following this, there are 5 hidden  
Figure 3: Flowchart during simulation and training. layers, each one with 400 neurons and each with rectified linear unit (ReLu), an activation function commonly used in deep neural networks, with the ability to introduce non-linearity to the network, allowing the NN to learn complex patterns and representations in the data.