## 3 Classified Experience Replay and Driving Framework
### Two-Stage Hierarchical Framework for Driving Task
#### 3.2.2 Motion Planning  
The decision output from the DM module is transmitted to the RL agent by concatenating it with the observation, subsequently generating the control input of the vehicle. Given the inherent uncertainty in driving scenarios, it is safer to constrain movements within an expected range through decision making rather than directly generating a control signal. The corresponding MDP formulation is presented below. (1) State Space and Action Space: The state space comprises absolute and relative observations. Absolute observation encompasses the ego vehicle's odometric information and the odometric information of four surrounding vehicles. Odometric information, denoted as \(\mathbf{O}_{k}\in\mathbb{R}^{5}\), includes two-dimensional position and velocity information, as well as the heading angle: \(\mathbf{O}_{k}=[s_{x},s_{y},v_{x},v_{y},\psi]_{k}^{T}\). Here, \(k\) represents the index of the vehicle, with 0 assigned to the ego vehicle and 1, 2, 3, and 4 representing nearby vehicles in clockwise order from the ego vehicle's center of mass. Relative information includes the time-to-collision value \(ttc\) for nearby states and the derived target lane \(l_{\text{target}}\) from high-level decisions and the current ego vehicle's position. The state accumulates four consecutive decision time steps to ensure the Markov property. The action space \(\mathbf{a}\in\mathbb{R}^{2}\) comprises the steering angle \(\beta\) and acceleration \(\alpha\), which collectively serve as the control input for the system dynamics. (2) Reward Function: The core component of the reward function involves comparing the ego vehicle's odometric states with those generated by synchronized MPC. To facilitate MPC for vehicle control, we employed a linearized kinematic bicycle model as described in [37]. In this model, we introduced the longitudinal distance \(d_{f}\) between the ego vehicle and the vehicle in front as an additional state variable. This incorporation enables us to  
Figure 5: Decision-making process visualization in simulation. (**a**) The simulation environment showing multiple lanes and nearby vehicles. (**b**) Front and rear TTC values are depicted as orange and yellow dashed lines. (**c**) The guidance lane is indicated by red lines. (**d**) The green stripe represents the decision made by the ego vehicle. TTC values at (**b**) and the guidance lane at (**c**) determine the sub-goal of the current state of the ego vehicle. As depicted in (**d**), the DM module can decide to change lanes even when the vehicle is on the guidance lane if the TTC value is too small. impose a constraint on maintaining a safe front gap. MPC shares the same action space and state space with an RL agent. The objective of the driving task is to follow a reference velocity and desired lateral position while adhering to inequality conditions that satisfy the vehicle's dynamics. Figure 4 illustrates the synchronization process of MPC, where decisions from the DM module serve as the reference for the MPC trajectory, and the MPC trajectory, in turn, serves as the reference for the RL reward function. This approach enables the trained agent to achieve trajectory optimization performance comparable to that of an MPC controller with reduced computation time. The MPC reference trajectory is computed at every \(m\) step from the ego vehicle's current state to ensure reference stability. These \(m\) steps of the reference trajectory are calculated within a cloned environment and then transferred to the original environment for reward calculation. In addition to the odometric reward, the change of action is also factored in to prevent oscillation or abrupt changes in the control input. The running reward is calculated as follows:  
\[r_{running}=\frac{1}{2}(\mathbf{O}_{0}-\mathbf{O}_{0}^{ref})^{T}\mathbf{H}( \mathbf{O}_{0}-\mathbf{O}_{0}^{ref})+\frac{1}{2}\mathbf{a}^{T}\mathbf{R} \mathbf{a}, \tag{1}\]  
where \(\mathbf{H}\) and \(\mathbf{R}\) are diagonal matrices used to weigh each element.