## 5 Experimental Analysis
### Convergence Comparison  
To enhance algorithm performance in complex scheduling scenarios, this study focuses on several key factors affecting the convergence of reinforcement learning algorithms. Based on the theoretical work by [PERSON] [30], the empirical convergence of the algorithm during training is analyzed from multiple perspectives. In the in situ data response management environment constructed in this study, the state space is discrete, with each round comprising 3 discrete indicators. The action space is also a discrete and finite set, where the agent can only select one of three queues for response decisions in each round. Therefore, the system satisfies the basic requirements of Q-learning theory regarding a finite state-action space. Meanwhile, the algorithm employs an \(\epsilon\)-greedy strategy for exploration, where \(\epsilon\) decays from an initial value of 1.0 at a rate of 0.995, effectively ensuring sufficient early exploration and later policy convergence. The learning rate \(\alpha\) is fixed at 0.001, which, although not strictly satisfying the Robbins-Monro conditions, has been widely validated for good empirical convergence in small-scale discrete spaces. In addition, the reward function is a negatively weighted combination of indicators with clear upper and lower bounds, ensuring the stability of the value function update process. In the in situ data management task of the InS system, following a clear algorithm comparison and theoretical convergence analysis, simulation experiments were conducted without limiting the number of training rounds. By observing the variation trends of reward values across training rounds for the three reinforcement learning algorithms, it was found that after a sufficient number of rounds, reward values gradually stabilized with limited fluctuations, and the marginal benefit of increasing the number of training rounds further became negligible. On this basis, this study sets the maximum number of training rounds to \(max_{e}pochs=5000\), expecting the reinforcement learning algorithms to achieve policy optimization within a limited number of rounds. Meanwhile, to improve the robustness of the experimental results, each algorithm was independently executed \(N=10\) times under the premise of fixed training rounds, and the final results were averaged for performance analysis. During training, the adjustment parameter \(\zeta\) was varied to further simulate the system's performance under normal and overloaded conditions. The convergence process of the reward function is shown in Figure 4. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Parameter** & **Baseline** & **ABC** & **DQN** & **DDQN** & **DDQN + PER** \\ \hline max epochs & 5000 & 5000 & 5000 & 5000 & 5000 \\ Adjustment Factor (\(\zeta\)) & 1 / 1.5 & 1 / 1.5 & 1 / 1.5 & 1 / 1.5 & 1 / 1.5 \\ Discount Factor (\(\gamma\)) & - & - & 0.99 & 0.99 & 0.99 \\ Epsilon (\(\epsilon\)) & - & - & 1.0 & 1.0 & 1.0 \\ Epsilon Decay (\(\epsilon_{\text{decay}}\)) & - & - & 0.995 & 0.995 & 0.995 \\ Learning Rate (\(\alpha\)) & - & - & 0.001 & 0.001 & 0.001 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Simulation parameters for algorithms comparison. This table lists the parameter settings adopted by various algorithms during the simulation training process. Each subplot illustrates the reward trends of different reinforcement learning algorithms under varying system load levels (\(\zeta=1.0\) and \(\zeta=1.5\)). Since the reward function is dominated by penalty terms, a reward value closer to 0 indicates a better policy selection. Throughout continuous interaction with the environment, all three algorithms exhibit a certain degree of convergence. However, there are noticeable differences in terms of convergence speed and stability. DQN shows considerable fluctuation even in the later training stages, with more pronounced instability under higher load conditions (\(\zeta=1.5\)).