# Improvements to Supercomputing Service Availability Based on Data Analysis
## 3 Problem Statement  
When a user's job fails, the scheduler notifies the user through a failure message by mail. This message contains only a simple summary; for detailed error messages, the *.log  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Type** & **Queue** & **Total Nodes** & **Total CPU Cores** & **Wall Clock Limit (Hours)** & **Max. Submit jobs** & **Max. Running Jobs** \\ \hline \multirow{4}{*}{KNL} & exclusive & 2600 & 176,800 & unlimited & 100 & 100 \\  & normal & 4970 & 337,960 & 48 & 40 & 20 \\  & long & 300 & 20,400 & 120 & 20 & 10 \\  & flat & 180 & 12,240 & 48 & 20 & 10 \\  & debug & 20 & 1360 & 48 & 2 & 2 \\ \hline \multirow{2}{*}{SKL} & commercial & 118 & 4720 & 48 & 6 & 2 \\  & norm\_skl & 118 & 4720 & 48 & 10 & 5 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Job-submit policies by queues.  
and *err files are referred to in the user home directory. We collected the log data from the scheduler server and output the statistics as follows.  
The main factors that influence the job failure rate are summarized in Figure 2. Failure due to program errors accounted for the largest percentage, 44.4%, of the total. Next, PBS scheduler errors, such as job script errors and WCT limit errors, accounted for 37%; and I/O errors, due to incorrect file permissions, incorrect files, or directory specifications, accounted for 12.6%. Failures due to hardware errors such as memory allocation errors or system bus errors accounted for 0.6%, errors due to the user's forced termination signal accounted for 2.3%, and other unidentifiable errors accounted for 3.1%. These statistics were obtained from analyzing the error codes of job failures.  
A detailed analysis of each item is shown in Table 2. For PBS error, \"job deletion with qdel includes walltime limit\" occurred most frequently, at 36.37%. In this case, most of the jobs submitted by users either exceed the walltime limit in the queue policy or are canceled by the user arbitrarily with the qdel command. Therefore, we excluded this cause of job failure from the target to be solved. For program errors, \"operation not permitted\" occurred most frequently, at 41.01%. By further analyzing this case, we found that when communication with the LDAP server fails, the operation does not work normally because of I/O ownership and r/w/x permission issues of the process. For I/O errors, \"no such file or directory\" occurred most frequently, at 10.37%. In this case, analysis showed that the cause of the problem was a file system mount error (OPA connection). For H/W errors, \"no such device or directory\" occurred the most (0.31%). One cause of this problem is that MPI communication occurs between compute nodes. Another cause is that most of it overuses the memory resource. The compute nodes are of the diskless type to ensure that all computation data will be allocated on the memory. If the data are loaded into memory because of previously executed jobs, an I/O message indicating insufficient memory resources is returned. For user errors, \"termination signal\" occurred most frequently, at 2.05%. From this statistic, we attempted to add a prologue hook before scheduling a resource and an epilogue hook after a job is completed.  
Figure 2: Classification of the cause of failed jobs by error codes.