# A Case Study on Virtual HPC Container Clusters and Machine Learning Applications
## 2 Related Work
### Docker in HPC
The purpose of such a cluster is not necessarily to gain high computational power, but to provide a special environment for comfortably programming parallel applications, testing them, and generally becoming familiar with the functioning of a cluster, without the need to have access to a real supercomputer. The problem with virtual machines lies more in the complexity of their configuration, difficulties in managing them, and potentially in how much they burden computer resources (CPU, memory, disk space). By analyzing instructions for launching a cluster with virtual machines [8; 9], it was found that the presented installation and configuration process seemed time-consuming and prone to errors. The earliest attempt to create a virtual cluster on Docker containers was most likely documented in a 2014 paper [13], around the time of the rapid rise in popularity of this technology. Containers allow for the delivery of varied software stacks (including selected Linux distributions and MPI implementations) according to user needs, simplifying the administration of HPC systems, as there is no need to compromise on globally installed dependencies. The goal of the work was to demonstrate that Docker performs well in HPC clusters based on the MPI standard and InfiniBand communication interfaces. However, the proposal in [13] more closely resembles a concept and does not present a practical way of implementing a virtual cluster [20]. Soon after, descriptions of practical methods for building a virtual cluster on Docker containers appeared. The authors of [21] aimed to solve the issue of different user preferences regarding dependencies installed in the cluster system, which led to problems related to the stability of the operating system and software portability. They designed the architecture of a virtual cluster to allocate isolated environments on a physical cluster, each containing specific dependencies and individually assigned to users. Such a configured virtual cluster is characterized by ease of scalability, as expanding it only requires the addition of more containers. Unfortunately, no performance tests of the containerized cluster were conducted in the work [21]. The authors also did not mention the standard Docker Compose tool, which would greatly simplify the reproduction and management of the virtual cluster on a local machine, not to mention other container orchestration platforms essential for managing the overall infrastructure (e.g., Docker Swarm, Kubernetes). Building on earlier ideas [13; 21], the paper in [22] refined the method of containerizing a cluster using Docker. Compared to previous proposals, this approach seems to be the most user-friendly. It was recognized that too much time is spent on installing and configuring the system, network, development tools, and dependencies before programming and deploying MPI applications. Therefore, a ready-to-use environment was prepared, where programmers could immediately start their work, including writing code and building and testing MPI-based programs. Container orchestration was entrusted to the integrated Docker Swarm platform. With its help, several physical machines with Docker services were connected into a network. Docker Swarm automatically balanced the load by distributing containers among physical nodes. The virtual cluster in the work [22] was not subjected to performance tests. Additionally, the proposal offers a rather limited set of configuration options. While Docker has been successful in the area of cloud and internet technologies and often forms an integral part of microservice architectures, over time it has been found that it does not meet the requirements of many production HPC environments [23; 24]. Among the distinctive features of this platform that hinder its application in supercomputers are the lack of native mechanisms for user authentication, authorization, and accounting;its dependence on disk space; and isolation from host resources [24].