## 4 Algorithm
### Double Deep Q-Network with Prioritized Experience Replay  
In the in situ data management process, the agent often needs to undergo multiple interactions before receiving effective rewards related to its decisions, resulting in low training efficiency for traditional Q-learning in environments with delayed rewards. To address this issue, this study introduces the Double Deep Q-Network with Prioritized Experience Replay (DDQN-PER) algorithm for decision optimization of personalized access demand queues. The core idea is to gradually learn the optimal queue selection policy through continuous interaction with the entire system environment. Figure 3 illustrates the complete process and framework structure of the DDQN-PER algorithm. This study integrates the Double DQN mechanism proposed by [PERSON] et al. [27] and the Prioritized Experience Replay (PER) method introduced by [PERSON] et al. [28]. The following section begins with a detailed description of the algorithm's computational process. At each decision moment \(t\), the agent perceives the current system state \(s_{t}\) and, based on the current Q-Network, selects the action \(a_{t}\) to execute. This process follows the \(\epsilon\)-greedy policy, where with a certain probability \(\epsilon\), a random action is chosen, and the action with the highest Q-value is selected for the remaining time:  
\[a_{t}= \begin{cases}\text{random action},&\text{with probability }\epsilon\\ \text{arg}\max_{a}Q(s_{t},a;\theta),&\text{otherwise}\end{cases}  \tag{31}\]The agent applies the action \(a_{t}\) to the environment, and the environment returns the immediate reward \(r_{t}\) and the next state \(s_{t+1}\). These interaction data \((s_{t},a_{t},r_{t},s_{t+1})\) are stored in the experience replay buffer (Replay Memory) for subsequent training. Even if the rewards of certain decisions are manifested after multiple steps, the associated experiences can still be repeatedly sampled, thereby enhancing the learning of delayed rewards. The introduced Prioritized Experience Replay mechanism not only requires storing experiences in the buffer but also calculating the TD error \(\delta_{t}\) for each experience:  
\[\delta_{t}=y_{t}-Q(s_{t},a_{t};\theta) \tag{32}\]  
In addition, the Prioritized Experience Replay mechanism calculates the priority \(p_{i}\) for each experience based on the Temporal Difference error (TD error). The TD error reflects the impact of the current experience on Q-Network training, and thus, through priority-weighted sampling, the agent learns more from experiences with larger errors. The priority \(p_{i}\) can be expressed as follows:  
\[p_{i}=|\delta_{t}|+\epsilon \tag{33}\]  
where \(\epsilon\) is a small constant used to avoid the situation where the priority is zero. This mechanism is well-suited to scenarios with delayed rewards, as such experiences are more likely to yield larger TD errors in the future, thereby being assigned higher priorities and sampled more frequently during training. During training of the DDQN-PER model, the Prioritized Experience Replay mechanism performs weighted random sampling from the experience replay buffer, based on  
Figure 3: DDQN-PER Algorithm Structure. This figure illustrates in detail the workflow of the DDQN-ER algorithm, clearly depicting the complete process from environment state perception, action selection by the policy and target networks, to the experience replay mechanism and network parameter updates. It comprehensively reflects the execution logic and core mechanisms of the algorithm in decision-making optimization. the priority of each experience \((s_{t},a_{t},r_{t},s_{t+1})\). Experiences with higher priorities are more likely to be sampled.