# Optimization of High-Performance Computing Job Scheduling Based on Offline Reinforcement Learning
## 2 Related Work  
This section reviews key advancements in scheduling algorithms for HPC systems, focusing on heuristic-based, reinforcement learning (RL)-based, and offline RL-based approaches. Heuristic methods prioritize simplicity and interpretability, employing rule-based strategies for job scheduling. RL-based techniques leverage online learning to optimize scheduling decisions dynamically but face challenges during early-stage exploration. Recently, offline RL has emerged as a promising paradigm, utilizing pre-collected data to train models, offering improved stability and efficiency for scheduling in complex HPC environments. **Heuristic-Based Scheduling Algorithms:** Heuristic scheduling algorithms typically prioritize jobs based on specific characteristics, with experts designing rules that map jobs to computational resources [19; 20]. These methods are intuitive and easy to deploy, often tailored to achieve specific optimization objectives. Simple, classic strategies include First-Come-First-Served (FCFS) and Shortest Job First (SJF). [PERSON] et al. [13] introduced scheduling algorithms like FAT, WFP3, and UNICEF, which rely on simple nonlinear operations to prioritize jobs based on characteristics such as arrival time, user request, and parallelism. These algorithms effectively reduce average waiting time by ranking job priorities. However, the handcrafted nonlinear operations fail to capture the complex relationships between priorities. [PERSON] et al. [11] introduced F1, F2, F3, and F4 scheduling algorithms, incorporating nonlinear regression to explore priority optimization. Yet, these algorithms remain dependent on basic nonlinear operations, limiting their effectiveness in more complex scheduling scenarios. **Reinforcement Learning-Based Scheduling Algorithms:** Although heuristic methods are widely used in industry, they are inherently suboptimal. To address this, DeepRM [21] models system states with multiple resources (e.g., CPU and memory) as a bitmap and uses the policy gradient (PG) method to optimize resource scheduling. RLScheduler [22] applies proximal policy optimization (PPO) to automatically learn high-quality scheduling policies for different workloads and optimization objectives. GARLSched [23] introduces an expert-guided scheduling algorithm, improving job scheduling through adversarial learning. Decima [24] employs graph neural networks (GNNs) to represent job dependency graphs and designs scalable RL models for continuous, stochastic job arrivals. DeepWeave [25] leverages GNNs to process directed acyclic graph (DAG) information, rewarding solutions that achieve faster job completion times. Similarly, DREAM [26] adopts a GNN-RL encoder-decoder framework to schedule jobs based on dependency relationships. DeepJS [27] autonomously learns fitness computation by solving a multi-dimensional bin-packing problem, optimizing scheduling decisions from experience to reduce job completion times. SchedInspector [1] optimizes job efficiency by reviewing scheduling decisions in real-time, using runtime factors to improve job wait times and slowdown ratios. CHROME [28] leverages reinforcement learning to dynamically optimize cache decisions and integrates cache replacement, bypassing, and prefetching to handle diverse and concurrent memory access patterns. RLSchert [29] leverages deep reinforcement learning and remaining time prediction to dynamically adjust resource allocation and job prioritization, significantly enhancing scheduling performance under varying workloads. However, these online reinforcement learning methods rely heavily on continuous trial-and-error processes to refine their scheduling policies. During the early learning stages, this approach can degrade system performance and waste resources, as the system is directly impacted by immature strategies. **Offline Reinforcement Learning:** In recent years, offline reinforcement learning has made significant progress in various domains, especially in settings where real-time data collection is expensive or impractical.