## 1 Introduction  
Reinforcement Learning (RL) has made remarkable strides across various domains, particularly in addressing sequential decision-making problems. Initial successes stemmed from simulated game environments [1, 2, 3], primarily in discrete action spaces, marking milestones in RL's development. Recent advancements have extended RL applications to continuous action spaces, encompassing control tasks like manipulator robot control [4, 5, 6], mobile robot navigation [7], and legged robot locomotion [8, 9]. However, when applied to the domain of autonomous driving, RL faces formidable challenges owing to the intricacies of traffic environments. Autonomous driving involves navigating at high speeds while adhering to traffic regulations and responsively reacting to nearby vehicles. This complexity has led to the adoption of goal-conditioned RL in path planning approaches, aiming to separate decision making and trajectory generation [10, 11, 12]. A universal value function [13] serves as a standard framework for goal-conditioned RL, particularly useful in environments with multiple sub-tasks [14, 15]. However, the irregular occurrence and dominance of certain sub-goals hamper optimal policy learning in such settings, adversely impacting training performance. This imbalance in sub-goal distribution is not unique to RL but is also pervasive in data-centric Artificial Intelligence (AI) systems [16, 17] and real-world applications. The highway environment exemplifiesthis imbalance, where the sub-goal of maintaining the current lane outweighs other sub-goals, such as changing to an adjacent lane. This highlights the necessity of addressing this imbalance to achieve high-performance agents. Experience replay [18], applied in the deep Q-network algorithm [19], marked a significant advancement in off-policy RL. The experience replay method stores transition samples in a buffer, enabling the training of RL networks using uniformly randomly sampled minibatches from this buffer. However, the uniform sampling method poses limitations in effectively learning optimal policies, especially in environments characterized by imbalanced sub-goal distributions across multiple goals. Addressing this challenge has been the focus of research efforts aiming to enhance training efficiency within experience replay. Prioritizing temporal difference error [20] and reinterpreting unsuccessful experiences using methods like Hindsight Experience Replay (HER) [21] have shown effectiveness in sparse reward environments, yet they struggle to fully resolve severe imbalances inherent in practical applications. To tackle this fundamental issue, we propose Classified Experience Replay (CER), specifically designed for multi-goal tasks. CER optimizes sample selection by adjusting the probability of choosing transitions associated with each sub-goal. Unlike traditional uniform sampling, CER's mechanism strategically alters the selection probabilities, mitigating the impact of imbalanced sub-goal distributions during training. Our empirical evaluations show the detrimental effects of imbalanced sub-goal distributions on the training policy. Through experimentation, we demonstrate that CER significantly improves sample efficiency and enhances the trained agent's performance in handling the challenges posed by imbalanced sub-goal distributions in practical scenarios. Another challenge in highway driving tasks lies in designing an effective reward function. Unlike discrete success or fail task completion in game environments, the continuous trajectory generation in driving tasks demands meticulous consideration at each step due to multifaceted constraints. Designing an appropriate reward function for RL in such constraint-rich environments proves challenging, often leading to overlapping reward components and unexpected outcomes [22; 23; 24; 25]. Achieving a comprehensive reward function encompassing all objectives and regulations necessitates extensive iterative reward shaping, a process prone to trial and error.