# Log Analysis-Based Resource and Execution Time Improvement in HPC: A Case Study
## 2 System Configuration
### Analysis of Job Execution Logs
Therefore, it should be emphasized that actual available resources are not used if a high priority job holds those resources. Moreover, it is also noted that such available resources cannot be fragmented and shared for lower priority jobs. Figures 3 and 4 show the trend of increasing available resources as the number of large-scale operations increases towards the end of 2016 and early 2017. The most extreme way to reduce resource fragmentation is to allocate the shortest job first, which is called the Shortest Job First (SJF) algorithm [23]. Such an algorithm prioritizes small jobs according to fragmented resources. This can improve resource utilization and improve overall performance, but it does not guarantee the fairness of job order. Therefore, it is worth noting that, in the worst case, the algorithm might result in the starvation problem for large-scale jobs. In that sense, the scheduling policy should be used in a way that combines FCFS and SJF considering resource size and job characteristics [24]. Most job scheduling policies simply use an FCFS method in which jobs submitted first to the queue are executed first. The backfilling algorithm is an excellent method for solving the resource fragmentation, but only simple and passive techniques are applied in a site-level service environment. Figure 4: Statistics of the public queue available nodes. Figure 3: Statistics of the large-scale jobs counts. Nurion system, the next generation of the Tachyon2 system that was the background of this study, uses Portable Batch System (PBS) as a scheduler [25]. PBS uses the _backfill_depth_ parameter, which is the number of backfill targets, for the top jobs with the highest priority among the queued job list [26]. PBS can backfill only a few lower priority jobs (_backfill_depth_) based on the highest priority parent job in the list of held jobs. In addition, when a specific event occurs, the scheduler updates the resource status and the job profile, which is called a scheduling cycle [27]. The events that trigger the scheduling cycle include a certain period of time when submitting and terminating a job, when a scheduling server starts, and when a new configuration of a scheduler is applied, and so on. In the batch scheduler, the job size consists of the number of processors and the execution time, where the execution time is estimated by the user and specified in the job script and then submitted to the scheduler. Therefore, the actual job execution time can often be terminated earlier than the expected time. In this case, if the job ends earlier than expected, resource fragmentation occurs again and the backfill target (_backfill_depth_) job fills up. If this situation repeats and small jobs continue to be backfilled, eventually, the top job will not run and will lead to starvation. In general, to avoid this starvation, production level systems set the _backfill_depth_ value very low to adjust the number of backfill operations. In the case of the Nurion system, the value is set to 3 or less. Because of this weakness, if the scheduler limits the number of backfill jobs to a setting like _backfill_depth_, it will not be able to take full advantage of fragmented resource utilization. Therefore, in this study, the simulation is performed by utilizing all fragmentation resources without limiting the number of jobs that can be backfilled as follows.