# Improvements to Supercomputing Service Availability Based on Data Analysis
## 4 Method
### Flowchart with K-Hook  
Scheduling flows in large-scale, high-performance computing systems are complex. In particular, resource and job scheduling for thousands of computing nodes in our supercomputing environment is even more complex. Figure 4 shows a flowchart of scheduling with K-hooks. This flowchart is based on the default settings of PBS Pro scheduling. When a job is submitted, this job will be in a queue with queue_job by Server. After Server queues the job, it is moved or modified depending on the configurations with modify_job and move_job. When a job is moved, it turns to run_job and is sent by Server to the MoM daemon in the compute nodes.  
When the MoM receives a job, the user's job execution starts from this moment with execjob_begin. We applied K-hook with a memory size check function to execjob_begin. Before executing execjob_prologue, we implemented and applied three functions to execjob_begin. First, we checked the communication with the LDAP server. Communication failure causes an error in which the submitted job does not operate normally because of ownership and r/w/x permission issues. This function solves the error message of \"Operation not permitted\" in Program Error of classification (Table 2). Second, when the job is executed on a multi-node, communication between nodes is checked. If communication between nodes does not work properly within a limited time, in the worst case, the job process will be killed. This function solves the error message of \"No such device or address\" in H/W error of classification (Table 2). Third, the memory space of computing nodes is checked. The computing node in Nurion is a diskless system, and all files stored on the local node are loaded in memory. Therefore, the local file (in the/tmp directory) created by the previous user's job might run out of memory, which can cause memory errors notified as \"Resource temporarily unavailable\" or \"Cannot allocate memory\" in H/W error of classification (Table 2).  
When the execution of execjob_begin with K-hook is applied and completed, MoM runs the job as a top shell and user program. If the user's program is a parallel process, such  
Figure 3: An overall workflow of scheduling with K-hook.  
as MPI, it is executed through tm_spawn and pbs_attach. After the job runs successfully, MoM kills the job and, through execjob_epilogue, the job is cleaned up. After this, we added K-hook functions to execjob_end to help increase the success rate.  
We implemented and applied three functions to execjob_end. First, we checked the mount points of the Lustre storage. All computing nodes in Nurion mount the Lustre storage, consisting of a home directory (/home01), an application directory (/apps), and a scratch directory (/scratch), to perform a user's job. If any of these are not mounted properly, problems may occur with the jobs on these nodes. This function solves the error message, \"No such file or directory\", which occurred the most in I/O error classification (Table 2). Second, a check was performed for zombie processes, which can occur after a job ends. Zombie processes, which remain \"dead\" on computing nodes, can degrade the computational performance when the next user's job is assigned. Finally, the STREAM benchmark test was conducted to measure memory bandwidth. By applying K-hook to execjob_end, it is possible to understand comprehensively the status of the computing node and perform offline procedures when an error occurs consistently, and therefore for the administrator to check the status of the node and take appropriate action.  
Figure 4: A flowchart of scheduling with K-hook.