## 5 Reward and State Definitions  
The definitions of (1) reward, (2) state, and (3) neural network structure are similar to those in [25], except for a few modifications. We describe them as follows:  
(1) Reward is defined as the normalized total number of vehicles departing from the intersection between two consecutive time steps:  
\[R_{t}=\frac{1}{LN}\sum_{l=1}^{LN}VS_{t,t+1}^{l}\,, \tag{2}\]  
where \(R_{t}\) is the reward for time step \(t\); \(VS_{t,t+1}^{l}\) is the count of vehicles crossing the stop line on the \(l\)-th approach lane between time steps \(t\) and \((t+1)\); \(LN\) is the number of approach lanes at the intersection. The agent follows the standard objective in MFDRL, which is to maximize the expectations of cumulative discounted rewards:  
\[\mathbb{E}\Big{[}R_{t+1}+\gamma R_{t+2}+\gamma^{2}R_{t+3}+\dots\Big{]}, \tag{3}\]  
where \(\gamma\in(0,1]\) is the discount factor. This objective amounts to encouraging the agent to serve more vehicles as early as possible to lower the vehicle delay at the intersection [25]. (2) To construct the state representation shown in Figure 5, we take a 120-meter-long portion of each intersection leg as the detection zone. It is divided into grids that are 4 m in length and are as wide as the corresponding lane widths. The state feature of a grid is a 4-tuple if it is on the approach lanes for a phase, or a 2-tuple if it is upstream from approach lanes:  
\[f= \begin{cases}(o,v,g,c)&\text{for approach lanes},\\ (o,v)&\text{otherwise},\end{cases}  \tag{4}\]  
where \(o\), \(v\), \(g\), and \(c\) denote the presence of the vehicle on the grid, the normalized speed of the vehicle, whether the corresponding phase is green, and whether the phase is a lagging phase, respectively. The state features of the grids on the approach lanes for each phase, and those of the other grids upstream from approach lanes, are separately organized as image-like state arrays, as follows:\[s= \begin{pmatrix}f_{1,1}&f_{1,2}&\cdots&f_{1,W}\\ f_{2,1}&f_{2,2}&\cdots&f_{2,W}\\ \vdots&\vdots&\ddots&\vdots\\ f_{H,1}&f_{H,2}&\cdots&f_{H,W}\end{pmatrix} , \tag{5}\]  
where \(f_{h,w}\) is the state feature at the \(h\)-th row and \(w\)-th column; and \(H\) and \(W\) are the number of rows and columns in the corresponding state array, respectively. Lastly, the state at time step \(t\) is represented as a multi-element tuple that includes all the state arrays for the intersection:  
\[S_{t}=\left\langle s^{1}_{\text{LT}},s^{1}_{\text{LT}},s^{1}_{\text{US}},s^{2 }_{\text{LT}},\ldots,s^{B}_{\text{US}}\right\rangle, \tag{6}\]  
where \(s^{b}_{\text{LT}}\) and \(s^{b}_{\text{TH}}\) are the state arrays for the approach lanes on the \(b\)-th intersection leg for the left and through phases, respectively, \(s^{b}_{\text{US}}\) is the state array for the lanes upstream from the approach lanes on the \(b\)-th intersection leg, and \(B\) is the number of legs at the intersection. For the test-bed signalized intersection in our experiments, the state can be represented as a high-dimensional array with \(4\cdot 30\cdot 4\cdot 4=1920\) elements. (3) Each state array is input into a convolutional neural network with \"ReLU\" activations. Its first three layers convolve 32, 64, and 64 filters with stride 1 with zero padding, respectively. The filter size is \(3\times 3\), except when the state array corresponds to a single lane, in which case the filter size is \(3\times 1\). The convolving results for all the state arrays are  
Figure 5: State representation and neural network structure. Information about vehicles and signals on each intersection leg is encoded in three-dimensional arrays and used as the input to the network. concatenated as a vector. But unlike in [25], we feed this vector into several branches of fully connected layers (see Figure 5). Each branch consists of a layer with 128 units and a layer outputting multi-dimensional sub-action values. This modification is required by the training algorithm proposed in the next section.