# Optimization of High-Performance Computing Job Scheduling Based on Offline Reinforcement Learning
## 1 Introduction  
Batch job scheduling is a critical mechanism for allocating resources and managing the execution of jobs in large-scale computing systems. Users submit batch jobs via scripts that specify job attributes, such as resource requests and execution time. These jobs enter a queue and are scheduled by a centralized job scheduler, which applies predefined strategies like first-come-first-served (FCFS) or shortest job first (SJF) [1, 2]. The scheduler allocates the necessary resources for the jobs to run, and once initiated, the jobs run to completion, failure, or termination [3]. Effective batch job scheduling aims to optimize system performance by improving resource utilization, reducing job wait times, and increasing job throughput [4]. In theory, job scheduling is an NP-hard problem [5, 6, 7]. In practice, heuristic-based scheduling strategies have been extensively studied [8, 9, 10, 11, 12, 13, 14]. However, as modern high-performance computing (HPC) systems grow in size, diversity in job types, and increase in system dynamics, traditional heuristic methods--often fine-tuned by experts--struggle to adapt automatically to changing environments and are prone to human bias [15]. These approaches are typically optimized for specific workloads, limiting their flexibility. Recently, data-driven methods like deep reinforcement learning (DRL) [16] have shown potential in addressing scheduling challenges. DRL learns scheduling policies through interactions between an agent and its environment, enabling adaptive performance optimization in dynamic, uncertain settings. These methods are particularly attractive due to their ability to self-improve over time by exploring and exploiting various schedulingstrategies. However, DRL methods face significant hurdles, including job externality, inefficient sample utilization, and unstable learning processes. The agent typically requires extensive training data and time, and performance in large-scale systems is often sub-optimal. This is largely because DRL starts with randomly initialized networks, lacking effective prior knowledge, which leads to a prolonged exploration phase before the agent can develop useful scheduling strategies [17]. Offline reinforcement learning (Offline RL) [18] presents a promising solution for large-scale HPC systems. By learning from fixed historical datasets, offline RL avoids the costly process of online exploration, making it particularly well-suited for large-scale HPC systems. In our approach, we leverage existing expert scheduling strategies as prior knowledge for offline RL and combine them with early-stage trial data from online strategies to create a hybrid dataset. This dataset, which includes a variety of job characteristics and system states, provides diverse experiences that help the model learn more robust scheduling policies. Additionally, using rich historical experience to constrain the strategy exploration space reduces the impact of random search and improves learning efficiency. Combining expert policies with offline RL approaches enables scheduling agents to learn high-quality scheduling policies directly from data. This method effectively handles complex jobs and resource allocation problems in HPC systems. The contributions of this study are as follows:  
* Based on offline reinforcement learning, a batch job scheduler for HPC systems is built, which to the best of our knowledge is the first batch job scheduler based on offline reinforcement learning. By learning from historical data, we can avoid using immature strategies in real environments and reduce the possible risks of real-time job scheduling in real environments. * A hybrid dataset is constructed that integrates various job characteristics and system states, providing diverse experiences. This helps the model learn more robust policies, leading to more efficient strategy optimization. * Extensive evaluations were conducted, comparing the model's performance across different historical datasets and applying the converged model to unseen datasets.