# Improvements to Supercomputing Service Availability Based on Data Analysis
## 4 Method
### Workflow with K-Hook  
We designed and implemented the workflow shown in Figure 3. Users submit jobs through the command line interface and submit job requests to the login nodes. Each request is sent to the daemon based on the server_priv configuration file and communicates with the compute nodes through the scheduler daemon and communication daemon based on the scheduling policy written in sched_priv and communication policy written in comm_priv on the PBS scheduler node. Based on the resource request transmitted through the communication daemon, the user's job is executed by exec_job on the resource-allocated compute nodes through the MoM daemon based on the mom_priv configuration file. We added K-hook to MoM running on the compute nodes rather than Server and Scheduler on PBS Pro scheduler nodes. The reason was to reduce the load of the scheduler nodes, which handle thousands of compute nodes in a huge HPC environment. As our supercomputer scheduled over 8000 compute nodes, we added K-hook to MoM on compute nodes. When a job is submitted, it is checked with the functions defined through K-hook, and if it does not pass then the node is processed offline, and the Server is notified through Communication. Server again attempts to allocate this job request to another node through Scheduler. This principle increases the success rate of job submission. The offline nodes are rebooted while attempting to recover through the self-recovery script we wrote.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Item** & **The Cause of Job Failure** & **Percentage (\%)** \\ \hline \multirow{8}{*}{PBS error} & Job was requeued (if rerunnable) or deleted (if not) & 0.01 \\  & Job execution failed, do retry & 0.04 \\  & Job execution failed, before files, no retry & 0.12 \\  & Job deletion with qedeltoudes walltime limit & **36.37** \\  & Licensed CPUs exceeded & 0.09 \\  & Undefined attribute & 0.36 \\  & PBS etc. & 0.001 \\ \hline \multirow{8}{*}{Program error} & **Operation not permitted** & **41.01** \\  & Argument list was too long & 0.04 \\  & Exec format error & 0.47 \\  & Command not found & 1.91 \\  & Stack overflow & 0.56 \\  & Segmentation fault & 0.28 \\  & Floating-point exception & 0.07 \\  & Illegal instruction & 0.1 \\ \hline \multirow{8}{*}{I/O error} & Operation requires sequential file organization and access & 0.41 \\  & 1/O procedure was truncated & 0.005 \\  & **No such file or directory** & **10.37** \\  & Input/output error & 0.2 \\  & Bad file descriptor & 1.34 \\  & Too many open files & 0.24 \\ \hline \multirow{8}{*}{H/W error} & No such device or address & 0.31 \\  & No child processes & 0.01 \\ \cline{1-1}  & Resource temporarily unavailable & 0.26 \\ \cline{1-1}  & Cannot allocate memory & 0.03 \\ \cline{1-1}  & Bus error & 0.01 \\ \hline \multirow{3}{*}{User error} & Abort signal & 0.18 \\ \cline{1-1}  & Kill signal & 0.09 \\ \cline{1-1}  & Termination signal & 2.05 \\ \hline \multirow{2}{*}{etc.} & etc. & 3.07 \\ \end{tabular}
\end{table}
Table 2: Detailed description of the causes of job failures.  
In this process, adding K-hooks may increase the user's waiting time. If a job fails, the user does not have to wait but must resubmit the job instead. We attempted to solve the problem of increased waiting time caused by adding a K-hook in various ways. One approach involved setting the maximum number of submitted jobs and executed jobs in the job submission policy. As our supercomputer sets different policies for each queue, we verified that waiting time does not increase with K-hooks by each queue.  
K-hook uses Python plugins provided by PBSPro hook. The main functions of K-hook are implemented in a bash script, which is called through PBSPro hooks in Python. The bash script can be extensible to any other batch schedulers.