<figure><img src="image_1.png" /></figure>

HAL
open science

# Performance Evaluation of Container-based Virtualization for High Performance Computing Environments

Carlos Arango, Rémy Dernat, John Sanabria

► To cite this version:

Carlos Arango, Rémy Dernat, John Sanabria. Performance Evaluation of Container-based Virtualization for High Performance Computing Environments. 2024. hal-04795161

HAL Id: hal-04795161

https://hal.science/hal-04795161v1

Preprint submitted on 21 Nov 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
Performance Evaluation of Container-based Virtualization for High Performance Computing Environments

Carlos Arango<fnref n="1" />, Rémy Dernat<fnref n="3" />, John Sanabria<fnref n="2" />

**Abstract**— Virtualization technologies have evolved along with the development of computational environments since virtualization offered needed features at that time such as isolation, accountability, resource allocation, resource fair sharing and so on. Novel processor technologies bring to commodity computers the possibility to emulate diverse environments where a wide range of computational scenarios can be run. Along with processors evolution, system developers have created different virtualization mechanisms where each new development enhanced the performance of previous virtualized environments. Recently, operating system-based virtualization technologies captured the attention of communities abroad (from industry to academy and research) because their important improvements on performance area.

In this paper, the features of three container-based operating systems virtualization tools (LXC, Docker and Singularity) are presented. LXC, Docker, Singularity and bare metal are put under test through a customized single node HPL-Benchmark and a MPI-based application for the multi node testbed. Also the disk I/O performance, Memory (RAM) performance, Network bandwidth and GPU performance are tested for the COS technologies vs bare metal. Preliminary results and conclusions around them are presented and discussed.

**Keywords:** Container-based virtualization; Linux containers; Singularity-Containers; Docker; High performance computing.

# I. INTRODUCTION

Computational tools are key elements in the development of different areas of knowledge such as industry, research and academy. Simulations and modeling are important computational techniques used to reduce waiting times and money budgets bringing novel and effective solutions to challenging problems.

New solutions usually required to be obtained through processor-intensive applications which demand specialized infrastructures to perform on acceptable time. High Performance Computing (HPC) is the name given to those processor-intensive applications to take advantage of massive parallel infrastructures known as computational clusters.

Computational clusters fulfill most of the processor-intensive applications requirements, tackling novel problems and presenting foreseeable solutions. However, more challenging problems surpass the capacity of one computational cluster and federations of scattered clusters are necessary

to meet the needs of these problems. Those federations of clusters are known as Grid systems.

Grid systems offer virtual organizations which integrate users and computational resources abroad. Thus, multiple virtual organizations are consolidated world wide tackling diverse problems (e.g. cancer cure, search for fundamental particles and sequencing genomes, among others) then requiring diverse services and applications.

This babel of tools presents a challenging problem for system administrators who have to deal with library versions, dependencies and software compatibility.

Virtualization is not a new technology [36] but it has been recently reactivated because of the advantages that it exhibits. Nowadays, off the shelf processors incorporate optimized virtualization instructions to support the deployment of secure and isolated computational environments bringing power efficient computational environments able to run several services in one single box [39], [43].

Cloud computing then emerges as a new infrastructure to borrow the best of Grid Computing and Virtualization in such a way that several users and projects are able to share computational resources in an isolated fashion, [9]. Cloud computing additionally exhibits other characteristics such as ubiquitous access, scalability on-demand and pay for consumed resources, [28]. Infrastructure, development platforms and software services have took advantage of it and a new economy around to Cloud computing infrastructures have emerged [14].

However, HPC is one of the few scenarios where Cloud computing has fall short on providing the performance expected by HPC applications. Although important milestones have been reached in the virtualization context and some cloud providers make available tailored virtual computational tools, the performance of virtualized contexts are very slow when they are compared with their bare metal counterpart [18].

Many scientific and academic applications taking advantage of native and optimized processor instructions which are penalized when they are executed on top of hypervisor tools. Hypervisors present a simplified view of the native hardware to the virtual machines then they can barely access to the optimized set of instructions of actual processors.

An alternative approach to the hypervisor-based solution to virtualized environments has gained traction and attention. Containers [3] subtract the hypervisor layer of the virtualization equations and relies on namespaces and cgroups in order to provide isolation and accounting of the consumed resources by the container instances.

<footnote n="1">¹ Facultad de Ingeniería, Escuela de Ingeniería de Sistemas y Computación, Universidad del Valle, Colombia
*carlos.arango.gutierrez@correounivalle.edu.co*</footnote>

<footnote n="2">²P. Facultad de Ingeniería, Escuela de Ingeniería de Sistemas y Computación, Universidad del Valle, Colombia
*john.sanabria@correounivalle.edu.co*</footnote>

<footnote n="3">³ ISEM, CNRS, Univ. Montpellier, IRD, EPHE, Montpellier France
*remy.dernat@umontpellier.fr*</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1. Container (blue) vs Virtual machines (red) interest over time. [2]</figcaption></figure>

Then, the rapid development of container-based technologies is getting attention of Internet users because containers accelerates the development process, eases distribution and deployment of applications, Figure 1. Leaders of such development are Docker<fnref n="1" /> [29] and Linux Containers (LXC[17]). Nevertheless its implications for scientific computing including HPC are still on doubt.

Containers are proving to be an extremely valuable technology for science delivering portability and reproducibility to the users. Containers can provide the requirements of a program and execute it directly, without the overhead that comes with hypervisor-based approaches. “Singularity-containers” from [23] is a container-based approach which focuses on providing portable environments which could leverage the migration of computational science to the cloud. Singularity integrates seamlessly with existing workload managers such as Slurm, HTCondor or Torque; fact that could ease its adoption of HPC facilities.

At the distributed systems and networks laboratory, at Universidad del Valle, we are working on the deployment of container-based software infrastructures to support the research process on different areas of knowledge. We have tested diverse operating system-based virtualization technologies running single node and multi-node applications getting important results which show that this kind of virtualization is prime time ready to support research processes. This paper presents a set of benchmarks that stress different aspects such as compute, memory bandwidth, memory latency, network bandwidth, and I/O bandwidth. We will present and compare three container-based operating systems (Docker, LXC and Singularity) in section II. Then, we will describe the methodology used alongside the results in order to evaluate performance overhead of container-based technologies versus the bare metal in section III. Related works will be addressed in section IV.

## II. CONTAINER-BASED OPERATING SYSTEM VIRTUALIZATION TECHNOLOGIES

Containers are software components to enable the execution of applications on isolated environments. Container-based operating systems (COS), also known as lightweight virtual machines, provide isolation of system resources (file system, network communications) in such a way that every container has its own set of processes ids, user identifiers, filesystem namespace and so on. Containers have a closer access to operating system services than their counterparts

virtualization tools such as native virtualization, paravirtualization and hypervisors. Figure 2-a shows that containerized applications run almost at the same level of native applications. In contrast, classical virtualization approaches (Figure 2-b) propose several layers between applications in virtualized environments and the hardware where virtual machines are actually running. In fact, these layers impose a big overhead in virtualized applications when they are compared with applications running on top of bare metal systems. Therefore COS technologies are now very attractive not only because they provide experimental reproducibility and platform portability but also because they exhibit a performance close to the performance exhibited on top of native environments [37].

COS have been around for awhile and there are numerous implementations of it. On 2000, FreeBSD (4.0) featured the Jails system which focused on providing an isolated filesystem (an enhanced version of the `chroot` command). Solaris goes a step further with its operating system OpenSolaris providing not only isolation services but also mechanisms related to snapshots and cloning. These aforementioned projects were mostly supported by BSD operating systems. On 2005 OpenVZ was announced as a COS implementation for Linux systems. Despite it was an open source project there was not too much interest in the Linux community then it was barely included into the Kernel main stream. OpenVZ never gets enough track amongst Linux community.

LXC (Linux Containers) took advantage of the namespace concept. Different from previous approaches where file system isolation was provided, LXC extended the isolation property to users, processes and networking. On 2001, Linux supported the first file system namespace known as the mount namespace. Since then, other namespaces have been supported, UTS, IPC, PID, user and network namespaces. In addition to isolation, on 2006, Google project (process containers) implemented a functionality to limit the resource usage, e.g. CPU, memory, disk I/O, network). This project was later merged into the Linux kernel and it was named cgroups (control groups). From that, cgroups capabilities have been extended to firewalling and unified hierarchy, amongst others.

Docker released on 2013, was basically an additional layer on top of LXC exposing additional features such as mounted storage, network port redirection, and container catalog management. These features made Docker a prime time product for the industry.

Singularity is a project developed at Lawrence Berkeley National Laboratory (LBNL) and it is mainly focused on experimental reproducibility and isolation.

### II-A. LXC

LXC is built on top of kernel namespaces which is a Linux kernel feature that isolates and virtualizes system resources such as processes, network, filesystems, network stack, among others<fnref n="2" />. These capabilities allow a fully operational

<footnote n="1">¹http://www.docker.com</footnote>

<footnote n="2">²http://haifux.org/lectures/320/netLec8_final.pdf</footnote>
<figure><img src="image_3.png" /><figcaption>Fig. 2. (a) Architecture of hypervisor-based virtual environments. (b) Architecture of container-based virtual environments. (c) Singularity Architecture</figcaption></figure>

container-based environments exhibiting interesting features such as exposure of network services from containers, containers live migration, and a complete set of accountability mechanisms[43]. In the networking context, LXC supports route- and bridge-based networking which allow the communication with the outside world but these features add a virtual network layer over the host which imposes an overhead to the network performance. Mechanisms based on cgroups are used for restraining the amount of resources that a container can consume, e.g. CPU, memory, number of opened files and so on. The container scheduling follows two level CPU scheduler which tries to promote fair scheduling among containers. First level scheduler determines which container will run, the second level determines which process in that container actually will run. For I/O bandwidth there is also a two-level scheduling mechanism known as Completely Fair Queuing (CFQ) scheduler. Each container has a priority and inside of it an I/O bandwidth is given according to priorities.

## II-B. Docker

Docker basically extends LXC with a kernel-and application level API [8] and mainly focusing on network service virtualization. Through the libcontainer library, Docker provides access to virtualization facilities provided by the Linux kernel along with some abstracted virtualized interfaces such as libvirt<fnref n="3" />, LXC and systemd-nspawn<fnref n="4" />. The control over host's resources is provided through Control Groups (cgroups) thus it limits the amount of resources used by a container such as memory, disk space and I/O [42]. Docker features a layered filesystem called AuFS (Advanced Multi-Layered Unification Filesystem) which allows to overlay one or more existing filesystems. When a process needs to create a copy, AuFS creates a copy of that file. This feature provides image versioning management and exposing base images to more specialized virtualized systems [29]. Docker

has emerged as a key player in the virtualization field since it has being widely adopted in the industry and academy because it leverages infrastructure consolidation and exhibits a low resource footprint. Docker has boosted the adoption of service oriented architectures (e.g. microservices [15]) because it ease the deployment of self-contained modules which are able to independently interact with third parties using well-known and widely adopted network protocols (e.g. web services). These service oriented architectures encourage the adoption of adaptable and extensible computational environments (e.g. workflows) accelerating the pace of scientific progress [16].

## II-C. Singularity

Singularity[23] is another container-based approach developed at Lawrence Berkeley National Laboratory (LBNL)[1]. It was created with the idea of compute mobility in mind. Although Singularity uses namespaces, it is used basically for application portability instead of host virtualization. In other words Singularity virtualizes only what is necessary to achieve run-time application container and portable environments. Singularity does not support user escalation or context changes therefore Singularity's container inherits permissions of the user who runs that container. Because it does not support context change then I/O operations flow directly between environments where those operations are happening reducing the operation overhead and execution times. Singularity seamlessly integrates with diverse HPC environments and tools, e.g. resource managers, HPC file systems, GPUs, etc. Singularity's design enables the utilization of vintage Container OS like RHEL 5 and also supports Docker-based images.

## II-D. Comparing LXC, Docker and Singularity

Three COS technologies have been discussed. Table I presents a summary of some features described above (versions compared are given in table II). The authors would like to note that this table, and the following discussion, is relevant to the time of writing of this paper. Docker has most of all

<footnote n="3">³https://libvirt.org/</footnote>

<footnote n="4">⁴https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html</footnote>
<table><caption>TABLE I
FEATURES OF LXC, DOCKER AND SINGULARITY</caption><thead><tr><th>COS<br>Feature</th><th>LXC</th><th>Docker</th><th>Singularity</th></tr></thead><tbody><tr><td>Support namespaces</td><td>Yes</td><td>Yes</td><td>Yes</td></tr><tr><td>Support cgroups</td><td>Yes</td><td>Yes</td><td>No</td></tr><tr><td>Support port mapping</td><td>Yes</td><td>Yes</td><td>No</td></tr><tr><td>User escalation</td><td>Yes</td><td>Yes</td><td>No</td></tr><tr><td>Unprivileged hardware access</td><td>No</td><td>No</td><td>Yes</td></tr><tr><td>API for applications and developers</td><td>Yes</td><td>Yes</td><td>No</td></tr><tr><td>Image Layering</td><td>No</td><td>Yes</td><td>Yes</td></tr><tr><td>Support snapshots</td><td>Yes</td><td>Yes</td><td>No <sup>(1)</sup></td></tr><tr><td>Network interface</td><td>Host or Bridge</td><td>Bridge</td><td>Host</td></tr><tr><td>Default filesystem</td><td>Host <sup>(2)</sup></td><td>AuFS</td><td>ext3</td></tr><tr><td>Access to host filesystem</td><td>Yes</td><td>Yes</td><td>Yes</td></tr><tr><td>Root daemon</td><td>Yes</td><td>Yes</td><td>No</td></tr><tr><td>Registry/Repository for the images</td><td>Yes</td><td>Yes</td><td>Yes</td></tr><tr><td>Build a container from a file</td><td>No</td><td>Yes</td><td>Yes</td></tr><tr><td>HPC accommodations</td><td>No</td><td>No</td><td>Yes</td></tr><tr><td>Keep modifications after restart</td><td>Yes</td><td>No</td><td>Yes</td></tr></tbody></table>

those characteristics (e.g. user escalation, API for developers, versioning management), then it is a very handy tool for leveraging the development of sophisticated enterprise and research tools. LXC is known as the predecessor of Docker. LXC has evolved in the meantime. One big evolution is LXD which is an API for LXC. For instance, it allows the live migration of containers between different LXD hosts. Therefore, users of LXC/LXD are not the same than the docker users (e.g. LXC is provided by the Proxmox virtualization server solution <fnref n="5" />). Indeed, state full containers and full environment (like singularity) could be quite interesting features. On the other hand, Singularity exhibits a limited number of properties because it is mostly conceived for code mobility and high availability of resources.

In next section, HPC benchmarks were run against these COS technologies and preliminary results exhibit that the absence of some features positively affects the performance of these benchmarks.

### III. METHODOLOGY AND BENCHMARKS

This section studies the computational performance of COS technologies vs bare metal. We performed several experiments with the current most popular COS implementations. Virtualization technologies and their versions are given in Table II.

The performance experiments were executed at two facilities Universidad del Valle Cluster and Montpellier Bioinformatics Biodiversity cluster computing platform.

Configuration of computational nodes used on this work are as follows: CPU model Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz(64-core node); Memory 164 GB DDR3-1,866 MHz, 72-bit wide bus at 14.9 GB/s on P244br and a HPE Dynamic Smart Array B140i Disk; OS Ubuntu 16.04 (64-bit) distribution was installed on the host machine.

We used the industry reference HPL-Lapack benchmark to test CPU performance, and microbenchmarks to individually measure memory, network, I/O and GPU overhead.

We know that results may vary significantly depending on the CPU architecture. versions of the kernel may introduce gains and losses of performance that would influence the results of experiments. Hence, we took care of compiling the same sha1sum binary for all benchmarks, using the host network for Singularity.

<table><caption>TABLE II
VIRTUALIZATION TECHNOLOGIES AND THEIR VERSIONS</caption><thead><tr><td>Virtualization technologies</td><td>Versions</td></tr></thead><tbody><tr><td>Singularity</td><td>2.2.1</td></tr><tr><td>Docker</td><td>17.03.0-ce, build 60ccb22</td></tr><tr><td>LXC</td><td>2.0.9</td></tr></tbody></table>

#### III-A. Time to execute a basic operation

The basic operation includes the start-up of the container and the execution of the very basic and very well known command `"/bin/echo Hello World"`. We used `"/usr/bin/time"` from the host to monitor it. From a native point of view, it always took 0.00 second. The three containers (LXC, Docker, Singularity) are minimalists and have been similarly built. The images are already present on the host.

We compared 6 operations fig 3: the native `"/bin/echo Hello World"`, and the same within `"singularity exec", "docker run", "docker exec", "lxc start + lxc exec"` and `"lxc exec"`. The `"docker run"` command includes the boot, the execution and the shutdown of the container, while `"lxc exec"` and `"docker exec"` need a running container. So, we decided to add `"lxc start + lxc exec + lxc stop"` to the chart. Considering this graph, all the shutdown operations of a container are the slowest.

Then, we analyzed the strace<fnref n="6" /> outputs of the previous commands, and we did not notice any specific bottleneck. However, we observed a significant amount of `"futex"` (143) and `"rt_sigprocmask"` (122) operations with `"docker run"`. These operations are usually dealing with synchronization mechanisms over threads when they are accessing shared resources e.g. shared memory regions. `"docker run"` is an

<footnote n="5">⁵https://www.proxmox.com</footnote>

<footnote n="1">¹However, a singularity image is only one file.</footnote>

<footnote n="2">²Tight integration with ZFS.</footnote>

<footnote n="6">⁶http://man7.org/linux/man-pages/man1/strace.1.html</footnote>
<figure><img src="image_4.png" /><figcaption>Fig. 3. Elapsed time (in seconds) for running `/bin/echo Hello Word`.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 4. Rate of execution for solving the linear system.</figcaption></figure>

operation to create a new container (a.k.a. new running process) which requires to access and modify shared resources and data structures at kernel level.

### III-B. CPU performance

In order to evaluate COS technologies for HPC we run the HPL-Benchmark [34] for a real vs virtual cluster as the ratio between the HPL benchmark performance of the cluster and the performance of a real environment formed with only one instance of same type, expressed as a percentage.

The benchmark were compiled using GNU C/C++ 5.4 and OpenMPI 2.0.2. We did not use any additional architecture- or instance-dependent optimizations. We used the SHA-1 hashes [12] with the `sha1sum` program and checked the libraries with the `ldd` utility to ensure the binaries integrity. For the HPL benchmark, the performance results depend on two main factors: the Basic Linear Algebra Subprogram (BLAS) [11] library, and the problem size. We used in our experiments the GotoBLAS library, which is one of the best portable solutions, freely available to scientists. Searching for the problem size that can deliver peak performance is extensive; instead, we used the same problem size 10 times (10 N, 115840 Ns) for performance analysis.

Figure 4 shows the performance of HPL-Benchmark. The Y axis is demonstrating the differences in technologies (that is why it doesn't goto zero). The LXC was not able to achieve native performance presenting an average overhead of 7.76%, Docker overhead was 2.89%, this could be probably caused by the default CPU use restrictions set on the daemon which by default each container is allowed to use a node's CPU for a predefined amount of time. Singularity was able to achieve a better performance than native with 5.42% because is not emulating a full hardware level virtualization (only the mount namespace) paradigm and as the image itself is only a single metadata lookup this can yield in very high performance benefits.

### III-C. Disk I/O performance

The disk performance was evaluated with the IOzone benchmark [32]. It generates and measures a variety of

<figure><img src="image_6.png" /><figcaption>Fig. 5. IOzone benchmark write and read.</figcaption></figure>

file operations and access patterns (such as Initial Write, Read, Re-Read and Rewrite). We ran the benchmark with a file size of 15GB and 64KB for the record size, under two(2) scenarios. The first scenario was a totally contained filesystem (without any bind or mount volume), and the second scenario was a NFS binding from the local cluster.

A closer inspection in COS shown in Figure 5 reveals that both LXC and Singularity had similar results for write operations. For read operations, where the Singularity slightly reach the native performance, and LXC had an overhead of 16.39% against native. On the other hand, with Docker, we observed a lost of performance of 37.28% on write and 65.25% on read. Figure 6 shows the performance of random read and random write. We noticed a similar behavior than the read and write standard operations. Docker introduces a greater overhead on random I/O processes. While LXC and Singularity filesystem implementations allows a better I/O performance, Docker advanced multi-layered unification filesystem (AUFS) has it drawbacks. When an application running in a container needs to write a single new value to a file on a AUFS, it must copy on write up the file from the underlying image. The AUFS storage driver searches each image layer for the file. The search order is from top to
<figure><img src="image_7.png" /><figcaption>Fig. 6. IOzone benchmark random write and read.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 8. OSU MPI bandwidth Test msgsize 4 MB.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 7. STREAM benchmark results.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 9. OSU MPI Latency Test msgsize 1 byte.</figcaption></figure>

bottom. When it is found, the entire file is copied up to the container's top writable layer. From there, it can be opened and modified.[4]

### III-D. Memory performance

The Memory performance on single node was evaluated with the STREAM application benchmark[27]. It is a simple synthetic benchmark program that measures sustainable memory bandwidth (in MB/s) and the corresponding computation rate for simple vector kernels [26]. The STREAM benchmark is specifically designed to work with datasets much larger than the available cache on any given system, so that the results are (presumably) more indicative of the performance of very large, vector style applications. Performance evaluation is tight to the memory bandwidth of the system. Performance is therefore gated by memory bandwidth and not latency. The benchmark has four components: COPY, SCALE, ADD and TRIAD.

Results are presented in fig 7.

Figure 7 presents different performance for COS and native systems, for vector operation. This is due to the fact that container-based systems have no resource constraints and can use as much of a given resource as the host's kernel scheduler will allow. The worst results were observed in Docker, which

presented an average overhead of approximately 36% when compared to the native throughput.

### III-E. Network bandwidth and latency performance

For the MPI-level network evaluation we used the MVA-PICH OSU Micro-Benchmarks 5.3.2 [30] using a direct 10 Gbps Ethernet link between the nodes. We run point to point tests for measuring bandwidth and network latency.

Docker and LXC attaches all containers on the host to a bridge and connects the bridge to the network via NAT. We did not set any special network configuration for any technology, more than their native networking documented on each project web page (creating bridges for LXC and using Docker Swarm and creating an overlay network for Docker). Singularity needed no additional network configurations.

Figure 8 shows the network bandwidth comparison for the COS. LXC has the best network scores with a great difference against the other two COS being evaluated. The singularity container showed a lower performance than the native implementation followed by Docker which presented the worst results. Its average bandwidth was 16.96% smaller than native. Figure 9 shows that LXC has less than 32% the network latency against native. The worst bandwidth and latency was observed with Docker. These results can be
explained due to different implementations of the network isolation of the virtualization systems. While Singularity container does not implement virtualized network devices, both Docker and LXC implement network namespace that provides an entire network subsystem. COS network performance degradation is caused by the extra complexity of transmit and receive packets (e.g. Daemon processes).

### III-F. GPU performance

The performance studies were executed on a Dell PowerEdge R720, with 2*Intel(R) Xeon(R) CPU E5-2603 @ 1.80GHz (8 cores) and a NVIDIA Tesla K20M.<fnref n="7" /> From a system point of view, we used Ubuntu 16.04.2 (64-bit), with NVIDIA cuda 8.0[21] and the NVIDIA driver version 375.26. The virtualization technologies and their versions are given in Table III.

<table><caption>TABLE III
VIRTUALIZATION TECHNOLOGIES AND THEIR VERSIONS</caption><thead><tr><td>Virtualization technologies</td><td>Versions</td></tr></thead><tbody><tr><td>Singularity</td><td>2.2.1</td></tr><tr><td>Docker</td><td>17.03.0-ce, build 60ccb22</td></tr><tr><td>LXC</td><td>2.0.9</td></tr></tbody></table>

In order to evaluate COS technologies for GPU-HPC, we used the NAMD (NAnoscale Molecular Dynamics) [20] program, as a benchmark tool. We ran those GPU benchmarks on a Tesla K20m with “NAMD x86_64 multicore CUDA version 2017-03-16” [on the stmv dataset (1066628 Atoms)], using the 8 cores and the GPU card, without any specific additional configuration, except the use of the “gpu4singularity”<fnref n="8" /> code for Singularity and the “nvidia-docker”<fnref n="9" /> tool for Docker. For a real vs virtual cluster, the ratio is printed in the log as “days/ns”(lower is better).

Figure 10 shows the performance of NAMD-Benchmark. The Y axis is in “days/ns” (the lower the better). LXC was not able to achieve native performance. Docker achieved a better performance than native, which can be explained on the work that Nvidia is doing to build cloud-native gpu applications. Nevertheless, Docker does not natively support NVIDIA GPUs with containers [5]. Singularity was able to achieve a better performance than native given that it provides native gpu support [23].

### III-G. Source Code

The scripts to run the experiments from this paper are available at https://github.com/ArangoGutierrez/containers-benchs

<figure><img src="image_11.png" /><figcaption>Fig. 10. Tesla K20m benchmarks on NAMD.</figcaption></figure>

## IV. RELATED WORK

Some papers have explored the overhead of container-based virtualization tools as presented by [13], [43], [22]. Mostly they compare the performance overhead of COS versus classic Virtual machine technologies (e.g KVM, LinuxV-server). They all agree that the current resource management implementation for LXC and Docker, lead to poor isolation and security.

Containers are proving to be an extremely valuable technology for scientific research, delivering benefits such as portability and reproducibility to scientific users. COS can emulate a single program and can be executed directly, with less overhead that with running a virtual machines. Indeed, some works already described COS technologies in a scientific use case [31], [7], [33], [10]. Despite the advantages offered by container technologies, the implications for scientific computing, including HPC, are still unclear, although there are already some initiatives like Singularity, Shifter[19] (See also benchmarks on Cray systems for Shifter [6]), Charliecloud[35], cHPC[41] or Docker Universe Applications in HTCondor<fnref n="10" />.

Many core technologies (a.k.a. GPU cards) are widely deployed on private data centers and more recently they have been exposed as services in the cloud computing landscape in order to attend the increasing growth for computational power in different fields of research, e.g. bioinformatics [24], storage processing [38] and multimedia [44], among others. Virtualization technologies and their impact on GPU technologies has not been broadly studied because the challenges exhibited to virtualize the functionality of GPUs and to support GPU passthrough. J.P. Walters et al.[40] run non-standardized benchmarks for KVM, Xen, VMWare and LXC. Using GPU Passthrough, these virtualization technologies are put under test running CUDA and OpenCL applications. Preliminary results exhibit a penalty over 10% on Xen and KVM. VMWare exhibited an irregular performance and LXC showed a closest performance to the native case.

To the best of our knowledge, there are no similar publications to this one. In particular, our work assesses three COS

<footnote n="7">⁷Kepler architecture[25]. GK110 Graphics processors, 2496 CORES, 208 TMUS, 40 Rops, 5120 MB Memory size, GDDR5 Memory type, 320 bit Bus width</footnote>

<footnote n="8">⁸https://github.com/NIH-HPC/gpu4singularity</footnote>

<footnote n="9">⁹https://github.com/NVIDIA/nvidia-docker</footnote>

<footnote n="10">¹⁰http://research.cs.wisc.edu/htcondor/manual/v8.4/2.12Docker_Universe.html</footnote>
technologies using standardized benchmarks. This approach gives a preliminary approach to characterize the assessment of COS and virtualization technologies in general.

There is a little research effort around container-based solutions for heavy HPC applications. Performance evaluations on literature usually does not put under test the HPL-LAPACK benchmark, instead a version of HPL-LINPACK, where matrix size could impact on the final result (e.g CPU cycles). Reports like [13] used a compiled version of LINPACK from INTEL, here we compiled the binary inside of each COS, to replicate a normal work flow, when running on a HPC cluster. Moreover, HPL results may vary significantly depending on the CPU architecture.

## V. CONCLUSIONS

This paper presented a performance comparison of container-based virtualization tools (Docker, LXC, Singularity) against bare-metal. According to our results, we observed that Singularity containers are usually more suitable for HPC implementation than Docker or LXC. From a network point of view, LXC is very efficient, however, not all namespaces are equal, and Singularity does swap out the user namespaces. Therefore, if the container have more efficient libraries than the host, the Singularity solution could yield a performance increase, while LXC and Docker control their resources by cgroups namespace, which results on a overhead for CPU intensive processes. Besides, Singularity optimizes HPC-specific libraries like CUDA or OpenMPI. For GPU applications, we recommend the implementation of Docker and Singularity to deploy on HPC clusters, or in the cloud. CUDA accelerated machine learning projects have already started offering Dockerfiles in order to run those applications over a container ready system. [5]

For I/O-intensive workloads, Container images can be much more optimal then running against shared storage (even when the container image exists on that remote storage). That is normal, as this is the same principle as cache or even the HPC scratch, that is to say a way to have data close to the process. Moreover, we would avoid the use of the standard Docker-based solutions (AUFS), due to overhead issues.

Concerning small tasks like “Hello World”, or even a more consistent memory job (see STREAM results), one more time, we would avoid the use of Docker, except if the container is already running on the host. That could lead to a real problem in a HPC system where the image needs to be downloaded everywhere and then, started, before being executed. Contrary to Singularity, where you have only one file, which can be shared through the network or can be stored on a distributed filesystem. Furthermore, a distributed filesystem can be significantly impacted by the metadata accesses, thus Singularity can limit that problem.

Singularity blocks privilege escalation within the container to avoid users of having root access. Docker instead must be isolated thus will preclude access to high performance networks (e.g. InfiniBand) and optimized storage platforms.

Considering our container-based results, COS and particularly Singularity are a good alternative to overcome the vir-

tualization overhead issues. COS environments can be used on mixed environments where HPC and HPSS are required. In order to take the HPC scenario, COS technologies must focus on: container overhead, container technology and architecture concerns (e.g. privilege escalation and network/file system access), and workflow compatibility.

## VI. ACKNOWLEDGMENTS

This work tests where heavily run on the Engineering faculty Cluster, University of Valle. This work also largely benefited from the Montpellier Bioinformatics Biodiversity cluster computing platform. We would also like to thank David Godlove from the NIH Biowulf Cluster for its code on GPU.

## REFERENCES

[1] A container for hpc. http://www.admin-magazine.com/HPC/Articles/Singularity-A-Container-for-HPC. Accessed: 2017-03-18.

[2] Google google trends. https://trends.google.com/trends/. Accessed: 2017-03-15.

[3] LinuxContainers lxc linux containers. https://linuxcontainers.org/. Accessed: 2017-03-15.

[4] Docker and aufs in practice, Apr 2017.

[5] Nvidia docker: Gpu server application deployment made easy, Feb 2017.

[6] Donald Bahls. Evaluating shifter for hpc applications. In *Cray User Group Conference Proceedings*, 2016.

[7] Peter Belmann, Johannes Dröge, Andreas Bremges, Alice C McHardy, Alexander Sczyrba, and Michael D Barton. Bioboxes: standardised containers for interchangeable bioinformatics software. *Gigascience*, 4(1):47, 2015.

[8] David Bernstein. Containers and cloud: From lxc to docker to kubernetes. *IEEE Cloud Computing*, 1(3):81–84, 2014.

[9] Rajkumar Buyya, Chee Shin Yeo, and Srikumar Venugopal. Market-oriented cloud computing: Vision, hype, and reality for delivering it services as computing utilities. In *High Performance Computing and Communications, 2008. HPCC'08. 10th IEEE International Conference on*, pages 5–13. IEEE, 2008.

[10] Paolo Di Tommaso, Maria Chatzou, Evan W Floden, Pablo Prieto Barja, Emilio Palumbo, and Cedric Notredame. Nextflow enables reproducible computational workflows. *Nature Biotechnology*, 35(4):316–319, 2017.

[11] Jack Dongarra. Preface: basic linear algebra subprograms technical (blast) forum standard. *International Journal of High Performance Computing Applications*, 16(2):115–115, 2002.

[12] D Eastlake 3rd and Paul Jones. Us secure hash algorithm 1 (sha1). Technical report, 2001.

[13] Wes Felter, Alexandre Ferreira, Ram Rajamony, and Juan Rubio. An updated performance comparison of virtual machines and linux containers. In *Performance Analysis of Systems and Software (ISPASS), 2015 IEEE International Symposium on*, pages 171–172. IEEE, 2015.

[14] Ian Foster, Yong Zhao, Ioan Raicu, and Shiyong Lu. Cloud computing and grid computing 360-degree compared. In *Grid Computing Environments Workshop, 2008. GCE'08*, pages 1–10. IEEE, 2008.

[15] Martin Fowler and James Lewis. Microservices. http://martinfowler.com/articles/microservices.html [last accessed on February 17, 2015], 2014.

[16] Yolanda Gil, Ewa Deelman, Mark Ellisman, Thomas Fahringer, Geoffrey Fox, Dennis Gannon, Carole Goble, Miron Livny, Luc Moreau, and Jim Myers. Examining the challenges of scientific workflows. Computer, 40(12), 2007.

[17] Matt Helsley. Lxc: Linux container tools. IBM devloperWorks Technical Library, page 11, 2009.

[18] Keith R. Jackson, Lavanya Ramakrishnan, Krishna Muriki, Shane Canon, Shreyas Cholia, John Shalf, Harvey J. Wasserman, and Nicholas J. Wright. Performance analysis of high performance computing applications on the amazon web services cloud. In *Proceedings of the 2010 IEEE Second International Conference on Cloud Computing Technology and Science, CLOUDCOM '10*, pages 159–168, Washington, DC, USA, 2010. IEEE Computer Society.
[19] Douglas M Jacobsen and Richard Shane Canon. Contain this, unleashing docker for hpc. *Proceedings of the Cray User Group*, 2015.

[20] Laxmikant V Kale, Abhinav Bhatele, Eric J Bohm, and James C Phillips. Namd (nanoscale molecular dynamics). In *Encyclopedia of Parallel Computing*, pages 1249–1254. Springer, 2011.

[21] David Kirk et al. Nvidia cuda software and gpu parallel computing architecture. In *ISMM*, volume 7, pages 103–104, 2007.

[22] Zhanibek Kozhirbayev and Richard O Sinnott. A performance comparison of container-based technologies for the cloud. *Future Generation Computer Systems*, 68:175–182, 2017.

[23] Gregory M. Kurtzer, Vanessa Sochat, and Michael W. Bauer. Singularity: Scientific containers for mobility of compute. *PLOS ONE*, 12(5):1–20, 05 2017.

[24] Sheng-Ta Lee, Chun-Yuan Lin, and Che Lun Hung. Gpu-based cloud service for smith-waterman algorithm using frequency distance filtration scheme. *BioMed research international*, 2013, 2013.

[25] Erik Lindholm, John Nickolls, Stuart Oberman, and John Montrym. Nvidia tesla: A unified graphics and computing architecture. *IEEE micro*, 28(2), 2008.

[26] John D. McCalpin. Stream: Sustainable memory bandwidth in high performance computers. Technical report, University of Virginia, Charlottesville, Virginia, 1991-2007. A continually updated technical report. http://www.cs.virginia.edu/stream/.

[27] John D McCalpin. Sustainable memory bandwidth in current high performance computers. *Silicon Graphics Inc*, 1995.

[28] Peter M. Mell and Timothy Grance. Sp 800-145. the nist definition of cloud computing. Technical report, Gaithersburg, MD, United States, 2011.

[29] Dirk Merkel. Docker: lightweight linux containers for consistent development and deployment. *Linux Journal*, 2014(239):2, 2014.

[30] OSU Micro-Benchmarks. Osu network-based computing laboratory. URL: http://mvapich.cse.ohio-state.edu/benchmarks.

[31] François Moreews, Olivier Sallou, Hervé Ménager, et al. Bioshadock: a community driven bioinformatics shared docker-based tools registry. *F1000Research*, 4, 2015.

[32] William D Norcott. Izone web page, 2012.

[33] Brian D O'Connor, Denis Yuen, Vincent Chung, Andrew G Duncan, Xiang Kun Liu, Janice Patricia, Benedict Paten, Lincoln Stein, and Vincent Ferretti. The dockstore: enabling modular, community-focused sharing of docker-based genomics tools and workflows. *F1000Research*, 6, 2017.

[34] Antoine Petitet. Hpl-a portable implementation of the high-performance linpack benchmark for distributed-memory computers. http://www.netlib-.org/~benchmark/hpl/, 2004.

[35] Reid Friedhorsky and Tim Randles. Charliecloud: Unprivileged containers for user-defined software stacks in hpc. Technical report, Los Alamos National Laboratory (LANL), 2016.

[36] Mendel Rosenblum and Tal Garfinkel. Virtual machine monitors: Current technology and future trends. *Computer*, 38(5):39–47, 2005.

[37] Cristian Ruiz, Emmanuel Jeanvoine, and Lucas Nussbaum. Performance evaluation of containers for HPC. In *VHPC - 10th Workshop on Virtualization in High-Performance Cloud Computing*, VHPC - 10th Workshop on Virtualization in High-Performance Cloud Computing, page 12, Vienna, Austria, August 2015.

[38] Weibin Sun, Robert Ricci, and Matthew L. Curry. Gpustore: Harnessing gpu computing for storage systems in the os kernel. In *Proceedings of the 5th Annual International Systems and Storage Conference, SYSTOR '12*, pages 9:1–9:12, New York, NY, USA, 2012. ACM.

[39] Rich Uhlig, Gil Neiger, Dion Rodgers, Amy L Santoni, Fernando CM Martins, Andrew V Anderson, Steven M Bennett, Alain Kagi, Felix H Leung, and Larry Smith. Intel virtualization technology. *Computer*, 38(5):48–56, 2005.

[40] J. P. Walters, A. J. Younge, D. I. Kang, K. T. Yao, M. Kang, S. P. Crago, and G. C. Fox. Gpu passthrough performance: A comparison of kvm, xen, vmware esxi, and lxc for cuda and opencl applications. In *2014 IEEE 7th International Conference on Cloud Computing*, pages 636–643, June 2014.

[41] Ole Weidner, Malcolm Atkinson, Adam Barker, and Rosa Filgueira Vicente. Rethinking hpc platforms: Challenges, opportunities and recommendations. *arXiv preprint arXiv:1702.05513*, 2017.

[42] Wikipedia. Docker (software) — wikipedia, the free encyclopedia, 2017. Online, accessed 18-March-2017.
<footnote>[43] Miguel G Xavier, Marcelo V Neves, Fabio D Rossi, Tiago C Ferreto, Timoteo Lange, and Cesar AF De Rose. Performance evaluation of container-based virtualization for high performance computing environments. In *Parallel, Distributed and Network-Based Processing (PDP), 2013 21st Euromicro International Conference on*, pages 233–240. IEEE, 2013.</footnote>

<footnote>[44] Wenwu Zhu, Chong Luo, Jianfeng Wang, and Shipeng Li. Multimedia cloud computing. *IEEE Signal Processing Magazine*, 28(3):59–69, 2011.</footnote>