HyperQueue: Overcoming Limitations of HPC Job Managers Stanislav Böhm stanislav.bohm@vsb.cz IT4Innovations, VSB – Technical University of Ostrava Czech Republic Jakub Beránek jakub.beranek@vsb.cz IT4Innovations, VSB – Technical University of Ostrava Czech Republic Vojtěch Cima vojtech.cima@vsb.cz IT4Innovations, VSB – Technical University of Ostrava Czech Republic Roman Macháček roman.machacek@vsb.cz IT4Innovations, VSB – Technical University of Ostrava Czech Republic Vyomkesh Jha jha0007@vsb.cz IT4Innovations, VSB – Technical University of Ostrava Czech Republic Alfréd Kočí alfred.koci@vsb.cz IT4Innovations, VSB – Technical University of Ostrava Czech Republic Branislav Jansík branislav.jansik@vsb.cz IT4Innovations, VSB – Technical University of Ostrava Czech Republic Jan Martinovič jan.martinovic@vsb.cz IT4Innovations, VSB – Technical University of Ostrava Czech Republic ABSTRACT In recent years, HPC workloads and communities have undergone substantial paradigm shifts. There is an increasing amount of users that want to leverage HPC clusters to execute many simple and embarrassingly parallel tasks as easily as possible. However, due to the limitations of traditional HPC job managers, these users must often resort to manual aggregation of tasks into a smaller number of jobs to reduce job manager overhead. This approach is both labour-intensive and inefficient, as it lacks dynamic load balancing required to fully utilize computational nodes with tens or hundreds of cores. We introduce HyperQueue - a task scheduling runtime that can execute a large amount of tasks on top of an HPC job manager by automatically aggregating tasks into jobs and dynamically load balancing them across all allocated nodes and CPU cores. HyperQueue is an open-source tool that is designed for ease of use and deployment. KEYWORDS scheduling, task management, resource management, cluster 1 INTRODUCTION Modern HPC systems contain a large number of computational nodes, with each node having tens or hundreds of CPU cores. There is an increasing number of HPC users that want to leverage all this computational power with quite simple workloads, for example by running a large number of single-node, relatively short-lived tasks in an embarrassingly parallel manner [1]. While HPC systems are prepared for almost arbitrarily complex distributed computations, it can be surprisingly difficult to execute a large number of such simple tasks on them efficiently. HPC job managers such as Slurm [2] or PBS are almost ubiquitously used to manage computational resources of HPC clusters. The most straightforward approach in this scenario is thus to simply map each task directly to a single job. However, this is usually infeasible, because the manager intro- duces nontrivial overhead and thus severely limits the amount of jobs that can be submitted by a single user. The manager is also usually configured in a way that does not allow allocating sub-node resources (running multiple jobs on a single node concurrently), either for security or performance stability reasons. Therefore, if the tasks only use a few cores, they cannot fully utilize the provided computational nodes. These limitations can pose a barrier to entry for new HPC users, since they cannot submit tasks to the cluster in a straightforward way. To overcome these limitations, we are introducing HyperQueue, a task execution runtime that can efficiently execute many single- node tasks on top of a Slurm/PBS cluster. Users can submit a large amount of tasks into HyperQueue in a simple way; it will then automatically aggregate tasks into a smaller amount of jobs. Once the jobs are started, it will distribute and dynamically load balance the tasks across all allocated jobs, nodes and cores to efficiently utilize available resources. HyperQueue is an open-source Rust project released under the MIT licence: https://github.com/It4innovations/hyperqueue. 2 RELATED WORK There are several other approaches that can be used inst