# Log Analysis-Based Resource and Execution Time Improvement in HPC: A Case Study
## 2 System Configuration
### Analysis of Job Execution Logs  
This section describes the way to optimize resource utilization through the analysis of large-scale jobs. It was mentioned that the Tachyon2 system is composed of 3200 compute nodes and has 8 cores per node. In total, it consists of 25,600 cores. The scheduler divides resource groups through queues. In this system, queue groups are largely divided into two types: public queues and private queues [21]. The public queue is used by various researchers to share computational resources through competition, and the exclusive queue is assigned to groups for specific mission-oriented research. The size of the computational resources allocated to the public queue and the exclusive queue can be changed depending on the purpose of the study. This experiment focused on log data from public queues used through resource competition and conducted experiments for statistical analysis of the data. The first approach extracted the number of large-scale jobs using more than 2000 cores over a two-year period on a monthly basis. As shown in Figure 3, the number of large-scale jobs is being gradually increased and more than 4000 cores were concentrated from November 2016 to March 2017. Figure 4 shows the trend of available resources during the same period. The _All Available Resources (core)_ shows the size of the resources allocated to the public queue. As discussed in Section 2.3, the exclusive policy was used in the Tachyon2 system. Therefore, the difference between _Idle Resources (cores)_ and _Avail Resources (Cores)_ represents the sum of the total number of unused resources and the number of cores of unused nodes. The usage statistics of the Tachyon2 system during the period are as follows. - Statistic results of the resource usage (January 2016-December 2017)  
* Average waiting time (_Tradv_): 9.3 h
* Average turnaround time (_Tradv_): 15.3 h
* Average node usage rate: 85.69%  
As shown in the summary, the average waiting time for public queues reaches 9.3 hours. However, the actual overall node utilization is only about 85%. This means that even though about 15% of the nodes were available, resources could not be used efficiently due to scheduling load. In other words, as shown in Figures 3 and 4, when a large-scale job is submitted, it waits until the required resources are available. In the meantime, even if a small job can be executed, it should wait if the priority is low. Therefore, even if a job requires a small number of resources and is performed shortly if the job priority is low, it must wait until the large-scale job is finished, even if there is an available resource. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**No.** & **Property** & **Value(Example)** & **No.** & **Property** & **Value(Example)** \\ \hline
1 & DATE & 20181029 & 12 & CPUS (cores) & 128 \\
2 & JOBID & 2310609 & 13 & CPU USAGE (s) & 2521538.58 \\
3 & GID & na0*** & 14 & MEM USAGE (KB) & 2269836.14 \\
4 & UID & r00*** & 15 & MAX VMEM (KB) & 345363.75 \\
5 & JOBNAME & mpi_solver & 16 & STATUS & D \\
6 & QNAME & normal & 17 & E-CPU(cores) & 256 \\
7 & SUBMIT(DATE) & 20181026111256 & 18 & E-RUN(s) & 11343872 \\
8 & START(DATE) & 20181028191152 & 19 & EXIT-CODE & 0(128) \\
9 & END(DATE) & 20181029050924 & 20 & FAILED & 0(128) \\
10 & WAIT(s) & 444543 & 21 & OMP_NUM_THREADS & 1 \\
11 & RUN(s) & 35852 & 22 & TASK_NUM & 0 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Job executed information of the converted Sun Grid Engine (SGE) log. The algorithm adopted by default in the scheduler is the First Come First Serve (FCFS) method, where the first arriving task in time is executed first. FCFS is the best way to ensure fairness of job order, but fragmentation occurs as the size of the computed resource increases, limiting the efficient use of resources. [22]. All jobs have their priorities. Such priorities are given, in general, based on arrival time. In addition, all jobs have different resource requirements.