## 3 Classified Experience Replay and Driving Framework
### Classified Experience Replay  
In environments where the sub-goal \(g\in\mathcal{G}\) is manually switched, the quantity of transitions stored in the replay buffer varies depending on the distribution of sub-goals. While a uniform distribution is typically used for sampling the minibatch, this approach may lead to an uneven utilization of each sub-goal \(g\) and its corresponding transitions, thereby adversely affecting training performance. This effect is exacerbated in cases of more imbalanced sub-goal distributions. To mitigate this issue, we propose classified experience replay. Unlike conventional methods that use a single buffer, we employ multiple buffers, each having the same dimensions as the sub-goal space \(\mathcal{G}\in\mathbb{R}^{n}\). Every transition is stored in the buffer corresponding to its associated \(g\). During policy updates, transitions are equally sampled from all buffers, thereby mitigating the disproportionate sub-goal experience during training and preventing the agent from becoming overly reinforced towards specific goals. Figure 1 illustrates the process of storing and utilizing transitions in CER. Figure 1: Visualization of classified experience replay. Transitions are stored in the classified buffer and uniformly sampled to train the agent, ensuring a balanced selection of transition samples. To evaluate the impact of an imbalanced sub-goal distribution on the agent's training, we devised a straightforward cliff walk scenario featuring two cliffs (see Figure 2). Here, the agent aims to navigate from the start state \(\mathbb{S}\) to the goal state \(\mathbb{G}\) under different sub-goals: \"Straight\", \"Up\", and \"Down\". The optimal trajectory to the goal state varies depending on these sub-goals, which are manually designated before each episode. The agent receives rewards for reaching the goal, with specific rewards varying based on the sub-goal. Figure 2 depicts the reward settings in the environment for each sub-goal, with colors representing the one-step reward received by the agent for stepping on each cell. To demonstrate the effect of sub-goal distribution on training performance, we trained and tested the agent in 10 cases with varying probabilities for \(g_{down}\). The probability distribution is shown in Figure 3. Initially, the probabilities of the sub-goals were uniform, with \(p(g_{straight})\!:\!p(g_{up})\!:\!p(g_{down})=1\):\(1\):\(1\). Subsequently, we gradually increased the probabilities of \(g_{straight}\) and \(g_{up}\) until reaching a ratio of \(p(g_{straight})\!:\!p(g_{up})\!:\!p(g_{down})=10\):\(10\):\(1\). We compared the performances of agents trained using DQN and DQN with CER. The total buffer sizes used for DQN + CER were equivalent to the single buffer size used in DQN. A test was considered successful only if the agent found an optimal path for any \(g\), denoted by \(O\) for optimal path, \(SO\) for sub-optimal path, and \(F\) for failure to reach the goal state. Figure 3 presents the test results, with the optimality test indicating whether the agent found an optimal path to the final destination under a given sub-goal. The agent trained with DQN frequently failed to reach the goal state as the probabilities became biased. In contrast, the agent trained with DQN with CER consistently found the optimal path under any sub-goal, irrespective of the non-uniformity in probabilities. Figure 3: (a) shows the probability distribution of each sub-goal from case #1 to case #10. In case #1, the distribution of three sub-goals is uniform, but as the cases progress, the distribution becomes biased towards two sub-goals other than \(g_{down}\). (b) demonstrates the success or failure of the trained agent. The symbol in each cell indicates whether the agent found a path from the start cell to the goal cell. \(O\) denotes an optimal result, \(SO\) denotes a sub-optimal result, and \(F\) denotes failure in finding a path. Figure 2: The twin cliff walk environment and reward configuration are depicted.