## 3 Dpd3 qn
### Priority Experience Replay
#### 3.2.3 Dual Priority Calculation
This priority sampling method, which combines TD error and segment weights, namely, dual-priority sampling, can more effectively utilize experience playback data in unmanned highway scenarios, especially when dealing with rare but important scenarios (such as high-speed lane changes and emergency braking), which can help the DPD3 QN better learn these key decisions and ensure the stability and convergence speed of the training process. Based on the integration of the dueling network and the DDQN structure, this paper uses dual-priority experience replay to update the experience pool and update the parameters of the neural network, as shown in Figure 4.  
During the autonomous driving process, the system first selects the current action \(a_{t}\) from the main network using the \(\epsilon\)-greedy strategy [22]. This strategy strikes a balance between exploration (selecting random actions) and exploitation (selecting the best action), exploring new strategies by randomly selecting actions with a small probability and selecting the best action evaluated by the main network at other times. Once the action has been chosen, the system executes \(a_{t}\) and then engages with the simulated driving scenario and interacts within it to generate the next state \(s_{t+1}\) and reward value \(r\). This information (current state \(s_{t}\), action \(a_{t}\), reward \(r\), and next state \(s_{t+1}\)) is stored in the experience pool. The experience pool is used to record multiple historical data for experience playback during training. The system extracts samples from the experience pool and calculates the TD error of these samples through the target network to calculate the loss function. The TD error \begin{table}
\begin{tabular}{c} \hline \hline
**Example: Dual-Priority Calculation** \\ \hline \hline Consider a sample with the following attributes: \\ TD error: \(p_{i}^{TD}=0.75\); \\ Position segment weight: \(w_{i}^{pos}=0.20\); \\ Velocity segment weight: \(w_{i}^{rel}=0.15\); \\ Given hyperparameters: \(\alpha=0.6\), \(\beta=0.4\). \\ Then, the comprehensive priority is as follows: \\ \(P_{i}=0.6\times 0.75+0.4\times(0.20+0.15)=0.45+0.14=0.59\). \\ If the sum of priorities in the current batch is \(\sum_{j}P_{j}=5.9\), \\ then, the normalized sampling probability is as follows: \\ \(P_{i}^{sample}=5.9/0.59=0.10\). \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Example calculation. reflects the gap between the prediction and the actual reward and is the core optimization target of training. The main network's parameters are refined via the gradient descent method, employing the backpropagation algorithm to reduce the loss function, thereby enhancing the network's predictive performance progressively. To ensure stability, the target network's parameters are not adjusted in real time but are periodically updated to match those of the main network. This structure helps prevent the target value from fluctuating too much, thereby improving the stability of training. The system repeats the above process continuously, gradually updating the network parameters and improving the strategy, thereby achieving more intelligent decision-making in unmanned driving scenarios. The \(\epsilon\)-greedy [22] strategy facilitates the equilibrium between exploration and exploitation throughout the reinforcement learning training phase, ensuring that the agent both investigates novel states and actions and leverages accumulated knowledge to choose the most advantageous action. The implementation proceeds as follows: at each time step, an action is chosen randomly with probability \(\epsilon\), while the action deemed optimal by the current Q-network is selected with probability \(1-\epsilon\). Following a set number of training iterations, the approach is assessed, and adjustments are made to the exploration strategy or algorithm parameters in accordance with the evaluation outcomes.