# Optimization of High-Performance Computing Job Scheduling Based on Offline Reinforcement Learning
## 3 Methods
### Overview  
In the reinforcement learning framework, an agent interacts with the environment to learn a policy that maximizes cumulative rewards [16]. At each time step \(t\), the agent observes the current state and takes an action based on its policy \(\pi\). This action leads to a transition from state \(S_{t}\) to \(S_{t+1}\), with the agent receiving a corresponding reward \(R_{t+1}\). In most cases, the agent is unaware of the environment's transition dynamics or reward mechanisms and must learn them through continuous interaction. The goal of RL is to maximize the expected cumulative discounted reward, represented as \(G_{t}=R_{t}+\gamma R_{t+1}+\gamma^{2}R_{t+2}+\ldots\), where \(\gamma\) is the discount factor that balances short-term and long-term rewards. As shown in Figure 1, online reinforcement learning involves the agent interacting with the environment in real-time, updating its policy as it collects new data. The data are generated through continuous interaction with the environment, rather than from a pre-existing dataset. Online RL methods can be categorized into two types: On-policy and off-policy [35]. On-policy methods improve the agent's current policy \(\pi\) by using data generated by that same policy. In this approach, the agent uses its current policy to generate behavior data and simultaneously uses that data to optimize the policy, requiring constant generation of new data for learning. In contrast, the off-policy method generates data during training using a different policy than the current one. Typically, the off-policy method generates data using a behavior policy \(\mu\) (i.e., different from the learned target policy \(\pi\)) but uses generated data to optimize the target policy \(\pi\). This allows Off-Policy methods to reuse past data through techniques like experience replay, making them more sample-efficient, especially in environments where data collection is expensive. Offline reinforcement learning takes this a step further by learning solely from a static dataset, without real-time interaction with the environment [31]. In Offline RL, the training dataset typically consists of tuples of state, action, reward, and next state \((S,A,R,S^{\prime})\) collected from one or more behavior policies. The goal is to learn a policy from this fixed dataset that performs well in the actual environment, maximizing future rewards. In HPC platforms, jobs typically have multiple attributes, such as user ID, group ID, requested processor count, and submission time, as summarized in Table 1. The full list of attributes can be found in the standard workload format (SWF) [36]. An HPC system can be modeled as a collection of \(p\) compute servers, denoted \(\{S_{1},S_{2},S_{3},\ldots,S_{p}\}\). When a job is submitted, it enters a central queue \(Q\) of length \(q\), waiting to be scheduled. Each job consists of a certain number of subprocesses, denoted \(j_{server}\). The number of idle servers in the system is represented by \(n_{server}\), and \(j_{mem}\) and \(n_{mem}\) denote the memory required by the job and the available memory on the servers, respectively. A job can only begin execution when \(n_{server}\geq j_{server}\) and \(n_{mem}\geq j_{mem}\). Figure 1: Difference between on-policy reinforcement learning, off-policy reinforcement learning, and offline reinforcement learning. Figure 2 illustrates the architecture of our proposed method and its key steps. The training process begins with the creation of a hybrid dataset comprising 2500 optimal scheduling trajectories generated by expert algorithms and 500 early-stage trial trajectories collected using the online reinforcement learning algorithm RLScheduler [22]. These trajectories are selected to capture diverse scheduling scenarios and highlight both expert-level decision-making and exploratory trial behaviors. The combined dataset enables the model to balance stability and adaptability during training.