## 8 Discussion  
### Experiment 1: Agent Learning  
We conducted almost no hyperparameter fine-tuning for MD4-AQF but took the values of the hyperparameters used in some original MFDRL algorithms. We found that this simple approach had already yielded very satisfactory performance. The minimal need for hyperparameter tuning demonstrated the ease of use of MD4-AQF. Given this competitive edge, we had little motivation to invest time in fine-tuning hyperparameters. It is possible that better performance could be achieved if more effort is devoted to hyperparameter tuning in the future. \begin{table}
\begin{tabular}{l c c c} \hline \hline
**FASC – MD4-AQF** & **N** & **Mean Rank** & **Sum of Ranks** \\ \hline Negative Ranks & 49 1  & 530.35 & 25,987.00 \\ Positive Ranks & 1951  & 1012.31 & 1,975,013.00 \\ Ties & 0 1  & \\ \hline Z & \(-37.729\) & & \\ Sig. (two-tailed) & 0.000 ** & \\ \hline \hline \end{tabular}  
* FASC \(<\) MD4-AQF, 3 FASC \(>\) MD4-AQF, 4 FASC \(=\) MD4-AQF. ** at 0.05 significance level.  
\end{table} 
Table 2: Wilcoxon matched-pair signed rank test on the average vehicle delay caused by MD4-AQF and FASC. Figure 10: Scatter plot of the average vehicle delay caused by MD4-AQF and FASC in different traffic demand scenarios. Each dot corresponds to a specific scenario, with the x-coordinate and y-coordinate denoting the results for FASC and MD4-AQF, respectively. The greater the distance of a dot from the anti-diagonal line, the larger the difference in average vehicle delay between the two methods. Specifically, the stochastic gradient descent (SGD) optimizer used was Adam, with a learning rate of \(6\cdot 10^{-5}\), which had been tuned and used by the authors of \"Rainbow\" (one of the best-performing DQN-style algorithms) to stabilize training [45]. Training started after randomly selecting actions 20,000 times, again as tuned and used in Rainbow [45]. A mini-batch included 64 samples, a commonly used value in many previous studies, and in the \"BDQ\" algorithm, where the Q-dimensionality expansion technique in this study was involved [47]. The target network weights were smoothly updated toward the exponentially moving average of the behavior network weights:  
\[\theta^{-}\leftarrow\tau\theta+(1-\tau)\theta^{-} \tag{21}\]  
where \(\tau=5\cdot 10^{-3}\). This practice was applied by the authors of the \"SAC\" algorithm consistently across various tasks [49]. \(\gamma\) was set to 0.99375 such that the agent's time horizon \(1/(1-\gamma)\) equaled the maximum possible cycle length. Such empirical practice was proven effective in prior work on MFDRL-based RTSC with an environment similar to that of this study [25]. The replay buffer held at most 200,000 samples, which was the maximum we could set, due to the limitation of memory capacity. The exploration probability \(\epsilon\) in the \(e\)-greedy behavior policy was 0.1 throughout the training and 0 for the evaluation. The value of 0.1 was the final value of \(\epsilon\) set by DQN and its many variant algorithms in the later stages of training [44]. To track the agent's learning progress, we evaluated the behavior network every 10 simulation runs during training. For each evaluation, we performed 100 simulation runs and recorded the quartiles of average vehicle delay at the intersection. In principle, more frequent evaluations allowed for more timely tracking of policy improvements, and more simulation runs per evaluation provided more accurate statistics. The training continued until the median of the average delay had not improved for \(\rho=200,000\) action steps. This is essentially the \"early stopping\" tolerance commonly used in deep learning, where training is terminated after waiting a sufficient amount of time without improving the best score. Strictly speaking, \(\rho\) is not considered a hyperparameter, as it does not affect the policy changes during training. As illustrated in Figure 8, the median of delay for MD4-AQF (marked red) oscillated around 135 s in the first 20,000 action steps.