## 8 Discussion
### Experiment 2: Versus D3 ynQN Method (Ablation Study)  
We contrasted MD4-AQF with a variant of the double DQN method (D3 ynQN) in terms of training progress. D3 ynQN is the same as MD4-AQF but removes the components of the \"multi-dimensional Q-network\" and subsequent \"Aligned Q-fusion\" for lagging phases. Instead, D3 ynQN directly learns each uni-dimensional combinatorial action for the dual ring. This requires \(256\cdot 26^{2}=160,256\) weights in the network output layer (comparedto \(128\cdot 2\cdot 26=6656\) for MD4-AQF). For lagging phases, D3 ynQN masks the invalid actions that cause the phases to end non-simultaneously. Meanwhile, D3 ynQN retains the practice of dynamic N-step update and uses the same hyperparameters of MD4-AQF. In this sense, Experiment 2 is also an ablation study on the proposed method.  
D3 ynQN [25] is an MFDRL method that has been demonstrated to control signals within a single-ring phase structure while satisfying the same signal timing constraints as in this study. Its success was achieved under the same conditions as in this study, including the same simulation software, road geometry, and traffic demand. Compared to MD4-AQF, D3 ynQN is identical in all factors except for the two major innovations proposed in this study for DRBPS. Thus, D3 ynQN serves as a competitive and comparable ablation baseline against MD4-AQF.  
Figure 8 shows the learning curve for D3 ynQN (marked green), compared with that for MD4-AQF. D3 ynQN improved the median of the average delay to only about 80 s before the 40,000 th action step. The performance exhibited severe fluctuations ranging from 65 s to 95 s until the 120,000 th step. This indicated the difficulty in learning hundreds of combinatorial actions directly. As training progressed to around the 150,000 th step, the oscillation of the median delay began to alleviate. The median dropped below 60 s at the 320,000 th step. During subsequent training, the agent eventually achieved the best performance (55.15 s) at the 465,181 st step. This result lagged behind that of MD4-AQF (47.89 s) by 7.26 s or 15.2%. Overall, D3 ynQN utilized significantly more network parameters than MD4-AQF but experienced greater instability and consistently underperformed compared to MD4-AQF throughout the training process.  
To further compare the agents trained via D3 ynQN and MD4-AQF, we followed the manner in [25] to stochastically generate 2000 traffic demand scenarios for the intersection via the D-optimal design. In each scenario, we performed two simulation runs, with signals being timed by D3 ynQN and MD4-AQF, respectively. We then recorded and compared their average vehicle delay at the intersection.  
As shown in box plots in Figure 9, the medians of the average vehicle delay caused by D3 ynQN and MD4-AQF, across the 2000 traffic demand scenarios, were 56.78 s and 49.62 s respectively. In comparison, the performance disadvantage of D3 ynQN was 7.16 s or 14.4% in terms of the median value. Additionally, the top and bottom quartiles (as well as the upper and lower whiskers) produced by D3 ynQN were larger than those of MD4-AQF. The inter-quartile range (i.e., the difference between the 75 th and 25 th percentiles) produced by D3 ynQN was \(66.91-50.42=16.49\) s, whereas that of MD4-AQF was \(57.69-44.31=13.38\) s. This indicated that the variance in average vehicle delay caused by D3 ynQN was larger than that of MD4-AQF. A similar inference could be drawn by comparing their whisker ranges (i.e., the difference between the upper and lower bounds in the box plots). The whisker range was \(91.56-38.51=53.05\) s from D3 ynQN, versus \(77.70-34.14=43.56\) s from MD4-AQF. These signified that the policy quality of D3 ynQN was inferior to that of MD4-AQF across various traffic load scales. Generally, as an ablated version of MD4-AQF, D3 ynQN's performance was worse than that of MD4-AQF. This result validated the significant impact of \"multi-dimensional Q-network\" and \"Aligned Q-fusion\" components in the proposed method.