## 6 Training Algorithm with Multi-Dimensional Aligned Q-Fusion
### Bootstrap Action Value  
Now, we adapt the bootstrap action value \(QB_{t}\) in Equation (12) for multi-dimensional sub-actions. As recommended in [47], we set a global \(QB_{t}^{*}\) used by all sub-action dimensions. It is the mean of the bootstrap value of the sub-action per dimension, given by the following:  
\[QB_{t}^{*}=\left\{\begin{array}{l}\frac{1}{D}\sum_{d=1}^{D}Q_{\theta^{-}}^{d} \left(S_{t+n_{t}},\,sa_{t+n_{t},i+1}^{*}\right),\\ \quad\quad\text{if next phases are lagging};\\ \frac{1}{D}\sum_{d=1}^{D}Q_{\theta^{-}}^{d}\left(S_{t+n_{t}},\,\arg\max_{a_{ t+n_{t},i+1}^{d,\varepsilon}}Q_{\theta}^{d}\left(S_{t+n_{t}},\,a_{t+n_{t},i+1}^{d, \varepsilon}\right)\right),\\ \quad\quad\text{if next phases are leading}.\end{array}\right. \tag{18}\]  
Then, we have a loss function modified from Equation (10). It takes the mean squared error across sub-action dimensions to form an aggregated scalar loss \(L_{t}^{*}\) below, from which the gradients are backpropagated to all the values of the selected sub-actions:  
\[L_{t}^{*}=\frac{1}{D}\sum_{d=1}^{D}\left[\left(RS_{t}+\gamma^{(n_{t})}QB_{t}^{ *}\right)-Q_{\theta}^{d}\left(S_{t},a_{t,i}^{d}\right)\right]^{2}. \tag{19}\]  
We need to optimize \(\mathcal{L}\), the expectation of the per-sample loss over the replay buffer \(\Omega\). It is estimated using the mean of \(L_{t}^{*}\) across a mini-batch including \(M\) samples that are uniformly sampled from \(\Omega\). This adds another averaging operator to Equation (19), as follows:  
Figure 6: Aligned Q-fusion. The two-dimensional Q values output from the network are marked blue. The values of the same sub-action in different dimensions are horizontally aligned and are averaged to calculate the fused values in the center purple column. The sub-action maximizing the fused value (marked with a bold and underlined number) will be selected and shared by all dimensions.  
\[\begin{split}\mathcal{L}&=\mathbb{E}_{\left(S_{t},A_{i,t}, RS_{t},S_{t+n_{t}}\right)\sim\Omega}\left(L_{t}^{*}\right)\\ &=\frac{1}{M}\sum_{m=1}^{M}\frac{1}{D}\sum_{d=1}^{D}\Big{[}\left( RS_{t}+\gamma^{(n_{t})}QB_{t}^{*}\right)-Q_{\theta}^{d}\Big{(}S_{t},a_{t,i}^{d} \Big{)}\Big{]}^{2},\end{split} \tag{20}\]  
where \(m\) denotes the ordinal number of a sample in the mini-batch.  
Given the formula above, we can eventually sum up the training algorithm for MD4-AQF in a pseudo-code, as shown in Algorithm 1.  
```
1 Initialize behavior Q-network with random weights \(\theta\);
2 Initialize target Q-network with weights \(\theta^{-}=\theta\);
3 Initialize empty replay buffer \(\Omega\);
4 Initialize time step \(t=0\), and action step \(i=0\);
5 repeat
6 Sample vehicle volumes, turning ratios, and pedestrian volumes;
7 Execute a new simulation run to its first ADP at time step \(t\);
8 Get initial state \(S_{t}\);
9 // see Section 8.1 for details about \(\epsilon\), \(\mu\), \(\rho\)
10 repeat
11 Select action \(A_{t,i}=\langle a_{t,i}^{1},a_{t,i}^{2}\rangle\) randomly with probability \(\epsilon\), otherwise, select \(a_{t,i}^{1}\) and \(a_{t,i}^{2}\) by Equation (17);
12 Execute current simulation run to next ADP at \((t+n_{t})\);
13 Get next state \(S_{t+n_{t}}\);
14 Get cumulative discounted rewards \(RS_{t}\) by Equation (11);
15 Store sample \(\langle S_{t},A_{t,i},RS_{t},S_{t+n_{t}}\rangle\) in \(\Omega\);
16 Sample random mini-batch of \(\langle S_{t},A_{t,i},RS_{t},S_{t+n_{t}}\rangle\) from \(\Omega\);
17 Compute average loss \(\mathcal{L}\) for mini-batch by Equation (20);
18 Perform SGD on \(\mathcal{L}\) with respect to network weights \(\theta\);
19 Update network weights \(\theta^{-}\) by Equation (21);
20\(t=t+n_{t}\), \(i=i+1\);
21
22 until reach the end of the simulation duration \(\mu\);
23 Stop the current simulation run;
24
25 until evaluation metric has not improved for \(\rho\) action steps;
```  
**Algorithm 1**Training algorithm for MD4-AQF