## 5 Conclusions  
This paper has presented an off-policy RL training method to address environments characterized by imbalanced sub-goal distributions. Our work demonstrates how incorporating CER into off-policy algorithms can significantly improve performance. The key contributions and findings of this research are summarized as follows:  
1. Imbalanced Sub-Goal Distribution and CER Technique: In Section 3, we utilized the twin cliff walk example to demonstrate how disproportional goal distributions in multi-goal environments can adversely impact training efficiency and optimality. Our adaptation of the CER technique to off-policy algorithms showed improved performance compared with methods that do not incorporate CER, demonstrating the validity of employing our method for multi-goal tasks. We adapt this approach to highway driving scenarios in the subsequent section. 2. Hierarchical Framework for Highway Driving Task: We proposed a hierarchical framework suitable for dynamic highway environments and trained a motion planner using our suggested training method. By employing separate modules, our agent proficiently generates control inputs across various scenarios without necessitating re-training or transfer learning. This flexibility allows for the easy modification of the decision-making module, empowering the agent to accommodate changes in traffic regulations or individual driving preferences. Simplifying the reward function by synchronizing the online MPC trajectory led to two notable advantages: a more straightforward design of the reward function and maneuvering capabilities of the RL-trained motion planner comparable to an optimized MPC path planner, with computational demands reduced to less than one-tenth of the original. 3. Discussions and Future Work: The incorporation of CER into the training of the motion planner demonstrated substantial improvements in both sample efficiency and performance. Our proposed method fosters the generation of uniform data in the training process, departing from uniform sampling to yield optimal solutions. Our method consistently outperformed other approaches without CER integration, underscoring its efficacy in addressing tasks, even in continuous control problems where traditional RL methods encounter difficulties in training. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Domain**} & \multirow{2}{*}{**Evaluation**} & \multicolumn{3}{c}{**Method**} \\ \cline{3-5}  & & **TD3** & **TD3 + CER[M]** & **TD3 + CER** \\ \hline \multirow{4}{*}{2-lane} & Avg. Deviation & 0.31 m & 0.12 m & 0.09 m \\  & Max. Deviation & 0.49 m & 0.18 m & 0.20 m \\  & Min. Deviation & 0.09 m & 0.03 m & 0.03 m \\  & Avg. Velocity & 19.36 m/s & 24.61 m/s & 23.33 m/s \\ \hline \multirow{4}{*}{3-lane} & Avg. Deviation & 0.32 m & 0.11 m & 0.08 m \\  & Max. Deviation & 0.99 m & 0.21 m & 0.28 m \\  & Min. Deviation & 0.11 m & 0.04 m & 0.02 m \\  & Avg. Velocity & 20.87 m/s & 26.87 m/s & 25.28 m/s \\ \hline \multirow{4}{*}{4-lane} & Avg. Deviation & 0.35 m & 0.10 m & 0.09 m \\  & Max. Deviation & 0.90 m & 0.20 m & 0.23 m \\  & Min. Deviation & 0.11 m & 0.06 m & 0.02 m \\  & Avg. Velocity & 21.53 m/s & 27.97 m/s & 25.49 m/s \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Quantitative analysis of driving performance, including lane-following accuracy and maintenance of desired velocity. Given that the target speed range is 30 m/s, TD3 + CER[M] shows maintaining the most approximate value to the target speed. However, given the deviations from the target lanes, TD3 + CER exhibits the smallest values. Notably, suggested methods are superior to baseline methods in every evaluation criterion. However, there are limitations to the proposed framework. As replay buffers are defined independently for individual sub-goals, the method assumes that the sub-goals are known in advance, necessitating some degree of prior knowledge of the task.