## 7 Experiment
### Results and Analysis
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Policy** & \multicolumn{4}{c}{**Evaluation Metrics**} \\ \hline  & **mean\_length** & **mean\_reward** & **success\_rate** & **response\_time (s)** \\ \hline
**DRLNDT + APRL** & 187.3512 & 43.6059 & 0.5901 & 0.0051 \\
**DRLNDT (Baseline)** & 106.5013 & 32.1885 & 0.11 & 0.0053 \\
**SAC + APRL** & 149.5733 & 48.3275 & 0.411 & 0.0026 \\
**Standard SAC** & 116.5532 & 43.1276 & 0.05 & 0.0027 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison of experimental evaluation processes. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Policy** & \multicolumn{4}{c}{**Training Metrics**} & \multicolumn{2}{c}{**Description**} \\ \hline  & **mean\_length** & **mean\_reward** & **success\_rate** & \\ \hline
**DRLNDT + APRL** & 106.9938 & 22.7781 & 0.5654 & DRLNDT with Active Preference Learning \\
**DRLNDT** & 80.4047 & \(-\)23.9450 & 0.0406 & Standard DRLNDT (Baseline) \\
**SAC + APRL** & 150.5300 & 45.3424 & 0.4933 & Soft Actor-Critic with Active Preference Learning \\
**SAC** & 136.9227 & 3.7050 & 0.0087 & Standard Soft Actor-Critic \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Comparison of Experimental Training Processes. The timestep serves as a discrete unit of time that characterizes the interaction between the environment and the agent. It is crucial for tracking learning progress, including the frequency of policy or value function updates, as well as for evaluating the agent's performance. Analyzing the average reward and success rate across various timesteps offers insights into the learning process. We approach the experiment primarily as a reinforcement learning task, with the performance evaluation focusing on the reinforcement learning aspect and disregarding traditional autonomous driving evaluation metrics. Figure 6 illustrates the outcomes of diverse active preference learning systems. Cosine similarity is used to gauge the effectiveness of active preference learning; a higher value indicates a closer resemblance between the user model and the belief model, which, in turn, implies that the belief model captures the user's preferences more accurately. Consequently, the reward weights derived from the belief model are more reflective of human intent. We conducted four experiments using distinct query optimizers and belief model updating methods. The mutual information optimizer and the NUTS (No-U-Turn Sampler) approach were applied in novel ways. Mutual information methods reduce the agent's uncertainty by gleaning informative knowledge, resulting in simpler yet insightful questions that consider human response capabilities. The NUTS algorithm enhances  
Figure 8: This figure presents the training (**a**,**c**,**e**) and evaluation (**b**,**d**,**f**) results of the four policies: DRLNDT + APRL, DRLNDT (Baseline), SAC + APRL, and Standard SAC. The policies are assessed based on their _mean_length_, _mean_reward_, _andsuccess_rate_. The x-axis represents the timestep, while the y-axis corresponds to the _mean_reward_, _mean_length_, and _success_rate_, respectively. the sample generation during belief model updates, obviating the need for parameter tuning and proving more efficient and user-friendly. Figure 6 demonstrates that our algorithm performs better than alternative methods, with stability analysis revealing the preference approach's advantage. Our method tends to stabilize after approximately 20 iterations. While the mutual information approach also acquires queries effectively, the classic Metropolis-Hastings method achieves a relatively higher cosine similarity after several updates. The figure suggests that the orange curve is less smooth than the red, indicating that the competitor algorithm's stability is inferior to ours. Based on this analysis, we selected the mutual information optimizer and NUTS to calculate the reward weights for active preference learning. Table 2 shows the reward weights derived from preference learning, which differ from those designed manually.