# [PERSON]: An HPC Job Scheduler Using Deep Reinforcement Learning and Remaining Time Prediction
## 3 RLSchert
### Job Scheduler Based on Deep Reinforcement Learning  
This section describes the job scheduler based on deep reinforcement learning. We modeled job scheduling as a sequential decision problem and formally defined the basic elements of the Markov decision process (MDP). We derived the policy gradient (PG) based on the MDP and used PPO to train the agent. Specifically, we designed a scheduler with the job remaining runtime and a job termination rescheduling policy. At last, we propose an efficient training algorithm to learn the policy. The HPC simulator designed in this work is close to the real-world environment. As Figure 1 shows, the simulator contains the model of clusters, queues, and jobs. The idea of RLSchert is in both the time and resource dimensions. Assuming that we can accurately predict the job runtime, the scheduling becomes a two-dimensional bin packing (BP) problem. It is possible to obtain the optimal solution through DRL, which is better than the heuristic schedulers such as SJF. In the simulator, the cluster is modeled as a CPU pool composed of finite cores. Jobs arrive step by step in discrete time and wait in the queue. The scheduler selects one or more jobs from the waiting queue to the cluster at each time step. The following parts are formal definitions of the main modules. **Job:** The job is the basic unit of scheduling, and a job has a corresponding feature description when the job arrives. The formal definition of job \(j\) is shown in Equation (6), where \(ID\) is the identification of the job, which is unique in the simulator. _res_ represents the requested CPU cores. _enter_time_ is the timestamps when the job arrives. _start_time_ is the timestamps when the job starts to run. _len_real_ represents the actual runtime of the job, and _len_param_ is the predicted runtime based on input parameters before the job runs. \([run_{1},run_{2},\cdots,run_{n}]\) represents the predicted remaining runtime at each time step. \(finish\_time\) is the timestamps when the job finishes, and \(finish\_time=start\_time+len\_real\). \(finish\_time\_hat\) is the predicted finish time. \(finish\_time\_hat=start\_time+len\_param\) be \begin{table}
\begin{tabular}{l l} \hline \hline
**Feature** & **Description** \\ \hline NELECT & total number of electrons \\ NSW & number of steps for ionic upd \\ SystemVolume & volume of the system \\ AtomNumber & number of atoms \\ ElecNumber & number of electrons \\ NELM & maximum steps of electronic self-consistent iteration \\ ElementType Number & number of element types that appear \\ ENCUT & energy cutoff in eV \\ ALGO Index & algorithm: normal (Davidson), fast, very fast \\ EDIFFG & stopping criterion for ionic upd \\ EDIF & stopping criterion for electronic upd \\ Exhosts group & the group of exhausts \\ numExhosts & number of exhausts \\ PREC index & precession: medium, high, or low \\ cores & number of cores \\ NELMIN & No. of electronic steps \\ ISPIN & whether to perform spin polarization calculation \\ GGA index & xc-type: PW PB LM or 91 \\ SIGMA & broadening in eV: -4-tet-1-fermi 0-gaus \\ numExhosts & number of exhausts \\ KPointMethod & the type of KPoint \\ ICIARGE & initialized charge density method \\ ISYM & symmetry: 0-nonsym 1-usesym \\ ISMEAR & part. occupancies \\ LREAL index & nonlocal projectors in real space \\ IBRION & ionic relaxation \\ ISTART & starjob: 0-new 1-cont 2-samecut \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Features extracted from the VASP input files and system logs. fore the job runs, and \(finish\_time\_hat=start\_time+run_{i}\) during the job runs. For the scheduler, if a job finishes before the predicted time, the rest resources are released immediately. If a job exceeds the predicted time, the simulator will add resources to it until the job finishes. Based on the prediction of the remaining runtime, the predicted finish time of the job must be re-estimated at each time step, and resources will be added or deleted in advance. \[