## 6 Training Algorithm with Multi-Dimensional Aligned Q-Fusion
### Dynamic N-Step Update  
D3 ynQN is a DQN-style training algorithm proposed in [25]. It has already been used in solving a single-ring RTSC problem with a fixed phase sequence. In this section, we introduce and improve D3 ynQN to render it suitable for our action definition.  
We follow the D3 ynQN's suggestion of setting a constant time-step size of **one second**. This ensures the rewards are discounted evenly as time goes on, to reflect the uncertainty arising from the time-varying RTSC environment [25]. Consequently, the actions will be selected every several time steps, instead of every time step. This decouples the action step from the time step. So, we will add another subscript \(i\) to the following action:  
\[A_{t,i}=\langle a_{t,i}^{1},a_{t,i}^{2}\rangle, \tag{7}\]  
to denote the action step, representing the \(i\)-th time of action selection.  
Next, we will conduct the dynamic \(n\)-step temporal-difference bootstrapping derived from D3 ynQN. It adjusts the interval parameter \(n\) to match the time span between two actions so that an action value can exploit the next action's value to bootstrap and update itself.  
We begin by recalculating the aforementioned parameter, \(n\), for DRBPS, which should equal the number of time steps between \(A_{t,i}\) and its next action [25]. We add the subscript \(t\) to \(n_{t}\), as it is dynamic and depends on \(A_{t,i}\). According to the definition of action in Section 4, \(n_{t}\) can be given by the following:  
\[n_{t}=\begin{cases}\max\Big{[}G_{\min}^{1}-G_{\min}^{2}+GTG_{t}^{1},\quad GTG _{t}^{2}\Big{]},\\ \quad\text{if current phases are leading and }G_{\min}^{1}\geq G_{\min}^{2};\\ \max\Big{[}GTG_{t}^{1},\quad G_{\min}^{2}-G_{\min}^{1}+GTG_{t}^{2}\Big{]}, \\ \quad\text{if current phases are leading and }G_{\min}^{1}<G_{\min}^{2};\\ \min\Big{[}GTG_{t}^{1},\quad GTG_{t}^{2}\Big{]},\\ \quad\text{if current phases are lagging.}\end{cases} \tag{8}\]where \(GTG_{t}^{k}\) is the time from the end of the minimum green for the current phase (at time step \(t\)) to the end of the minimum green for the next phase in the \(k\)-th ring:  
\[GTG_{t}=a_{t,i}^{k}+YC^{k}+RC^{k}+NG_{\text{min}}^{k}\,. \tag{9}\]  
Note that \(G_{\text{min}}^{k}\) and \(NG_{\text{min}}^{k}\) are the minimum greens for the current phase and its next phase in the \(k\)-th ring, respectively; \(YC^{k}\) and \(RC^{k}\) are the yellow change and red clearance intervals for the current phase in the \(k\)-th ring, respectively. Some examples of time lengths represented by \(n_{t}\) are visually shown in Figure 4.  
With \(n_{t}\) being known, we can write down the general loss formula used to update the action value \(Q_{\theta}(S_{t},A_{t,i})\) at time step \(t\), given by the following:  
\[L_{t}=\Big{[}\Big{(}RS_{t}+\gamma^{(n_{t})}QB_{t}\Big{)}-Q_{\theta}(S_{t},A_{t,i})\Big{]}^{2}, \tag{10}\]  
where \(RS_{t}\) is the sum of discounted rewards in subsequent \(n_{t}\) time steps:  
\[RS_{t}=\sum_{k=t}^{t+n_{t}-1}\gamma^{k-t}R_{k}\,, \tag{11}\]  
and \(QB_{t}\) is the bootstrap action value after the \(n_{t}\) time steps:  
\[QB_{t}=Q_{\theta^{-}}\Bigg{(}S_{t+n_{t}},\,\arg\max_{A_{t+n_{t},i+1}}Q_{\theta }(S_{t+n_{t}},\,A_{t+n_{t},i+1})\Bigg{)}. \tag{12}\]  
Note that \(\theta\) and \(\theta^{-}\) are the weights of the behavior and target Q-networks, respectively. In the following subsections, we will propose more improvements to the loss formula (10) to further accommodate DRBPS.