# Optimization of High-Performance Computing Job Scheduling Based on Offline Reinforcement Learning
## 3 Methods
### Network Structure  
In cases where the state space is large, storing all possible states becomes impractical. Instead, deep neural networks (DNNs [37]) can be utilized as function approximators to enhance the agent's learning capability in complex, high-dimensional environments. Our algorithm adopts a value-based learning approach [38]. In reinforcement learning, the advantage function is defined as the difference between the state-action-value function and the state-value function, i.e., \(A(s,a)=Q(s,a)-V(s)\). For a given state, the sum of the advantage values of all actions equals zero, as the expected action-value of all actions should equal the state-value. In our approach, the \(Q\) network is modeled as follows:  
\[Q_{\alpha,\beta}(s,a)=V_{\alpha}(s)+A_{\beta}(s,a)-\frac{1}{|\mathcal{A}|} \sum_{a^{\prime}}A_{\beta}\big{(}s,a^{\prime}\big{)} \tag{2}\]  
where \(V_{\alpha}(s)\) represents the state-value function, and \(A_{\beta}(s,a)\) denotes the advantage function, capturing the difference between actions in the same state. The parameters \(\alpha\) and \(\beta\) are used for the state-value and advantage functions, respectively. The term \(\frac{1}{|\mathcal{A}|}\sum_{a^{\prime}}A_{\beta}(s,a^{\prime})\) in the equation represents the average advantage across all possible actions for a given state. Subtracting this term helps center the advantage values around zero for each state, ensuring that the advantage values are normalized relative to the average action-value. This normalization fulfills the requirement that the action-value function \(Q(s,a)\) should, on average, equal the state-value \(V(s)\) across actions. This centering helps stabilize learning, preventing large shifts in \(Q\) values by maintaining a consistent advantage function across actions.  
To ensure stable training, we use another \(Q\) network, called the target network, denoted by \(Q_{\text{target}}(s,a)\). While the training network \(Q(s,a)\) is updated at every step, the target network \(Q_{\text{target}}(s,a)\) parameters are synchronized with the training network only at fixed intervals. This technique improves the stability of the target network relative to the training network. Normalized job attributes, along with information about available system resources, are embedded for each job (\(j\)). The embeddings are concatenated to form a vector \(v_{j}\), which is then input into the model.  
As shown in Figure 4, the advantage function network \(A_{\beta}(s,a)\) is a three-layer fully connected network, structured as a three-layer multilayer perceptron (MLP [39]) with 64, 32, and 16 neurons in each layer. For each pending job, the network computes a \"score\", and the scores of all waiting jobs form a vector. By subtracting the average score from each job's score, we derive the advantage of each action. Positive advantage values indicate an action better than the average, while negative values indicate poorer performance. Using this advantage allows the network to learn the differences between actions more effectively, helping the agent evaluate actions and make decisions, while also improving learning efficiency and reducing variance for more stable learning.  
The state-value function network \(V_{\alpha}(s)\) has six layers, with 32, 16, 8, 1, 64, and 32 neurons. It takes as input the concatenated and flattened vector of jobs in the queue, representing the state \(S\). The network outputs a single value, which assesses the weighted average of all possible action values in the given state \(S\). By combining this with the advantage function, the final \(Q\)-values for all possible actions are obtained, guiding the agent's decision-making process.