# Optimization of High-Performance Computing Job Scheduling Based on Offline Reinforcement Learning
## 3 Methods
### Reduction of Extrapolation Error  
Extrapolation error arises when the current policy encounters state-action-reward-next state \((S,A,R,S^{\prime})\) tuples that deviate from those sampled in the dataset [31]. Since the offline dataset is fixed, the agent can only learn from a limited set of trajectories and cannot explore unseen or infrequent state-action pairs. For such unseen pairs, the model must estimate based on available data, potentially introducing significant errors and degrading policy quality. Moreover, Q-learning aims to maximize the Q-values, which may leadthe model to incorrectly assign high rewards to unobserved actions, thereby skewing the training process toward these erroneous estimates.  
To effectively learn the optimal scheduling policy, we define a loss function composed of several components, which mainly include the \(Q\)-value loss and the action-matching loss. The \(Q\)-value loss optimizes the action-value estimates of the policy, the action-matching loss ensures that the actions selected by the model align with those observed in historical data. Specifically, the \(Q\)-value loss is computed using a smooth L1 loss function, defined as follows:  
\[L_{q}(q,q_{target})=\begin{cases}\frac{1}{2}(q-q_{target})^{2}&\text{if }|q-q_{target}|<1\\ |q-q_{target}|-\frac{1}{2}&\text{otherwise}\end{cases} \tag{3}\]  
Here, \(q\) represents the Q-value predicted by the current policy network for a given state and action, while \(q_{target}\) is the Q-value target computed via the target network, which is a combination of the immediate reward \(\tau\) and the discounted future Q-value, calculated as follows:  
\[q=Q(s,a) \tag{4}\] \[q_{targets}=\tau+\gamma\text{max}Q_{target}(s^{\prime},a^{\prime})\]  
In this formula, \(Q(s,a)\) represents the action-value function, which estimates the expected cumulative reward when taking action \(a\) in state \(s\) under the current policy. \(q_{target}\) is the target Q-value, which combines the immediate reward \(r\) received after taking action \(a\) in state \(s\), and the discounted maximum Q-value of the next state \(s^{\prime}\) (denoted \(Q_{\text{target}}(s^{\prime},a^{\prime})\)) for the next possible action \(a^{\prime}\). The discount factor \(\gamma\) (with \(0\leq\gamma\leq 1\)) controls the weight of future rewards relative to immediate rewards. The \(\max_{a^{\prime}}Q_{\text{target}}(s^{\prime},a^{\prime})\) term represents the maximum expected future reward obtainable from the next state \(s^{\prime}\) across all possible actions \(a^{\prime}\).  
Figure 4: The DNN-based RL agent reads \((S,A,R,S^{\prime})\) tuples and computes the score of each job.  
The action-matching loss \(L_{i}\) is calculated using the negative log-likelihood loss function \(NLL\_Loss\), ensuring that the selected actions follow the distribution observed in the dataset:  
\[L_{i}(p,a)=-\sum_{i=1}^{C}a_{i}\log(p_{i}) \tag{5}\]  
where \(p_{i}\) is the probability of selecting the \(i\)-th action, obtained by applying softmax to the \(Q\)-value vector, and \(a_{i}\) is the true action label. Thus, the overall loss function for the algorithm is expressed as follows:  
\[L=L_{q}(q,q_{target})+L_{i}(p,a) \tag{6}\]