## 3 Dpd3 qn
### Priority Experience Replay  
#### 3.2.1 Prioritization Based on TD Error  
Experience replay was first proposed by [PERSON] [13] and applied to DQN. This method determines the importance of each sample based on temporal difference (TD). Samples with higher priority are more likely to be selected for training. This priority can be calculated based on TD error. TD error signifies the discrepancy between the anticipated value and the actual reward value. Its calculation formula is shown in Equation (17):  
\[\delta=\left|r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime};\theta^{-})-Q(s, a;\theta)\right| \tag{17}\]  
where \(r\) is the immediate reward, \(\gamma\) is the discount factor, \(Q(s^{\prime},a^{\prime};\theta^{-})\) is the target network output, and \(Q(s,a;\theta)\) is the output of the current network. The sum of the two TD errors generated by the dual networks and the immediate reward is used as the basis for dividing the importance. The empirical priority value \(p(i)\) can be defined as Equation (18):  
\[p(i)=(\delta_{i}+\epsilon)^{a} \tag{18}\]  
where \(\epsilon\) is a small constant that prevents the priority of the data from being zero. The hyperparameter \(a\) determines the degree of influence of the TD error \(\delta_{i}\) on the sample selection and controls the sensitivity to the error during sampling. Experience with high priority will have a greater chance of being sampled. The larger the TD error, the higher the priority. In the early training process, the TD error may be inaccurate. In light of the aforementioned factors, it is necessary to decrease the priority weight when selecting training data during the initial phase of the training process. At the same time, in order to ensure that even samples with extremely low priority can be selected, we add a bias to the priority compared to the traditional priority approximate distribution \(b\)[13]. In our experiments, the value of the bias term \(b\) is set to 0.01, which helps avoid zero probability and ensures sufficient sample diversity. The approximate distribution of an experience sample being taken is improved to Equation (19):  
\[P(i)=\frac{p_{i}^{\alpha}}{\sum\limits_{i=1}^{k}p_{i}^{\alpha}}+b \tag{19}\]  
The sampling probability for the i-th experience is determined by the parameter \(\alpha\) through the priority mechanism, where \(\alpha\) serves as a balancing coefficient between prioritized and uniform sampling. Specifically, \(\alpha\in\) [0,1] controls the degree of priority emphasis in the probability distribution \(p_{i}\), which represents the priority of the i-th transition--typically defined based on the magnitude of its temporal difference (TD) error, such as \(p_{i}=|\delta i|+\epsilon\), where \(\delta\)i is the TD error and \(\epsilon\) is a small positive constant to ensure a non-zero probability. When \(\alpha\) approaches 1, the selection process predominantly follows the temporal difference error ranking, while values near 0 implement conventional random sampling. To mitigate potential overfitting caused by repeatedly selecting similar high-priority transitions, the algorithm incorporates a baseline probability adjustment through hyperparameter \(\varepsilon\). This safeguard prevents the sampling distribution from becoming excessively skewed toward historically high-priority experiences, thereby maintaining the diversity of training samples and improving generalization capability. The parameter \(\alpha\) essentially operates as a continuous interpolation regulator between two sampling paradigms: complete priority dominance (\(\alpha=1\)) and pure stochastic selection (\(\alpha=0\)). Since the agent tends to replay experiences with higher TD errors more frequently, it will undoubtedly change the frequency of state visits. This change may cause the training process of the neural network to oscillate or even diverge [19].