## 4 Algorithm
### Double Deep Q-Network with Prioritized Experience Replay
The sampling probability \(P(i)\) is computed based on the priority \(p_{i}\) of each experience, and \(\alpha\) controls the bias in the prioritized sampling:  
\[P(i)=\frac{p_{i}^{\alpha}}{\sum_{k}p_{k}^{\alpha}} \tag{34}\]  
After sampling a batch of data from the replay buffer, the next step is to calculate the importance sampling weight \(\omega_{i}\) for each sampled experience, which is calculated as follows:  
\[\omega_{i}=\left(\frac{1}{N}\cdot\frac{1}{P(i)}\right)^{\beta} \tag{35}\]  
where \(N\) denotes the size of the replay buffer, which also represents the total number of experiences in the experience pool. \(P(i)\) is the probability of the current experience being sampled. \(\beta\) is the parameter that controls the bias of the importance sampling weights. Unlike DQN, which directly utilizes the Target Q-Network to both select and evaluate the optimal action, DDQN introduces a dual-network structure, where separate networks are employed for action selection and action evaluation. This mechanism not only effectively alleviates the overestimation of Q-values but also allows the agent to learn action values more stably in the presence of delayed rewards. In practice, DDQN first selects an action using the Q-Network and then evaluates this action using the Target Q-Network. The computation of the target Q-value \(y_{t}\) proceeds as follows. First, the current Q-Network (with parameters \(\theta\)) selects the action \(a_{\max}\) that maximizes the Q-value given the next state \(s_{t+1}\):  
\[a_{\max}=\arg\max_{a}Q(s_{t+1},a;\theta) \tag{36}\]  
where \(Q(s_{t+1},a;\theta)\) denotes the Q-value of action \(a\) in state \(s_{t+1}\) estimated by the Q-Network, where \(\theta\) represents the parameters of the main network. Then, the Target Q-Network (with parameters \(\theta^{-}\)) is used to evaluate the Q-value of the selected optimal action \(a_{\max}\) in state \(s_{t+1}\):  
\[Que_{\text{target}}=Q(s_{t+1},a_{\max};\theta^{-}) \tag{37}\]  
where \(\theta^{-}\) is the parameter set of the target network. This value is a key component in estimating the target Q-value of the current state-action pair. Finally, the target Q-value \(y_{t}\) is computed using the Bellman expectation equation, which is used to update the Q-Network:  
\[y_{t}=r_{t}+\gamma Que_{\text{target}} \tag{38}\]  
where \(r_{t}\) denotes the immediate reward received at time step \(t\), and \(\gamma\) is the discount factor (\(0\leq\gamma\leq 1\)), which controls the influence of future rewards. Through this three-step process, DDQN effectively decouples action selection from evaluation, mitigating Q-value overestimation and improving learning stability and policy performance. DDQN uses the Mean Squared Error (MSE) loss function to calculate the error \(L(\theta)\) between the predicted Q-values of the Q-Network and the target Q-values. The parameters \(\theta\) of the Q-Network are optimized using backpropagation and gradient descent, with \(\alpha\) being the learning rate:  
\[L(\theta)=\mathbb{E}\Big{[}\omega_{i}(y_{t}-Q(s_{t},a_{t};\theta))^{2}\Big{]} \tag{39}\]  
\[\theta\leftarrow\theta-\alpha\
abla_{\theta}L(\theta) \tag{40}\]Since directly using the Q-Network for target computation may lead to training instability, DDQN introduces the mechanism of the Target Q-Network, whose parameters \(\theta^{-}\) are updated in a regulated manner: instead of being synchronized during every gradient update, \(\theta^{-}\) is periodically copied from the main network parameters \(\theta\) at fixed training intervals. This mechanism enhances the stability of the training process, allowing the agent to obtain more reliable value estimations in long-horizon decision-making, thereby supporting policy optimization under delayed reward scenarios. \[p_{i}=|\delta_{t}|+\epsilon \tag{33}\]0  
The detailed process is shown in Algorithm 3.