# Optimization of High-Performance Computing Job Scheduling Based on Offline Reinforcement Learning
## 3 Methods
### Historical Data Collection for Training  
Expert policies provide valuable examples for our model, allowing us to recover optimal decision-making patterns from these examples. In large-scale, dynamic scheduling for HPC systems, different expert policies may make varying decisions in the same state, leading to diverse performance outcomes. As shown in Figure 3, the x-axis represents the test sequence number, where each test consists of 1024 randomly selected consecutive jobs. The y-axis represents the performance metric, average bounded slowdown (AVGbsId), which is calculated using the formula provided.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Name** & **Symbol** & **Description** \\ \hline Job ID & \(j_{id}\) & The ID of the job \\ User ID & \(j_{u}\) & The user’s ID \\ Group ID & \(j_{g}\) & The group’s ID \\ Submit Time & \(j_{st}\) & Job submission time \\ Request Time & \(j_{t}\) & Job’s runtime estimation (or upper bound) from users \\ Request Processors & \(j_{n}\) & The number of processors that a job requests \\ Request Memory & \(j_{mem}\) & The requested memory per processor \\ \hline \hline \end{tabular}
\end{table}
Table 1: Description of job attributes.  
Figure 2: The overall architecture of our method.  
\[AVGbsld=\frac{1}{M}\sum_{v}\max\left(\frac{t_{wait}^{v}+t_{real}^{v}}{\max\left(t_{ real}^{v},\tau\right)},1\right) \tag{1}\]  
AVGbsld measures the ratio of a job's total time from submission to completion to its actual runtime. A lower ratio indicates that waiting time constitutes a smaller portion of the total time, making it a key metric for evaluating scheduling efficiency. In this formula, \(M\) is the total number of jobs in the test sequence (in this case, 1024 jobs per test), \(v\) indexes over each job in the test sequence, \(t_{real}^{v}\) denotes the actual runtime of job \(j\), and \(\tau\) is a constant used to prevent very short jobs from disproportionately influencing the results. Previous studies commonly set \(\tau\) to 10 s [1,22,23]. Across the 10 tests, the optimal scheduling strategy varies for each.  
In order to effectively utilize the optimal domain knowledge of existing experts, we combine multiple existing scheduling algorithms, including WFP, FCFS, UNI, SJF, and F1, and call the existing scheduling algorithms to make scheduling decisions by specifying an initial state and the length of the job sequence. Specifically, expert strategies interact with the environment to generate a series of state-action-reward-next state tuples \((S,A,R,S^{\prime})\). Once a job sequence is fully scheduled, these tuples are organized into a complete trajectory. By evaluating each expert's performance in terms of AVGbsld across the entire sequence, we can identify the optimal trajectory. We denote a complete trajectory of optimal scheduling decisions as \(tr\), meaning \(tr=\{(S_{1},A,R,S^{\prime}),(S_{2},A,R,S^{\prime}),\ldots,(S_{H},A,R,S^{ \prime})\}\). Thus, an expert dataset \(\mathcal{D}\) consisting of \(m\) optimal trajectories is represented as \(\mathcal{D}=\{\mathrm{tr}_{i}\}_{i=1}^{m}\).  
The trial trajectories, on the other hand, are sourced from the early exploratory phase of RLScheduler [22], capturing the model's initial decision-making process. These diverse, less optimal experiences help the model generalize better across different scheduling scenarios, complementing the expert knowledge.  
We conduct a hybrid dataset that combines expert-guided trajectories with early-stage trial trajectories from the RLScheduler. The hybrid dataset, specifically, consists of 2500 trajectories derived from the Lublin-256 job trace, with 80% expert-guided trajectories and 20% trial trajectories. Each trajectory represents a sequence of scheduling decisions over 256 jobs. Each job in the dataset includes relevant attributes such as CPU and memory requests, runtime estimates, and submission times, reflecting the resource and timing characteristics of HPC workloads.  
Figure 3: The scheduling performance comparison of different expert policies.