Performance Evaluation of Container Runtimes
Lennart Espe, Anshul Jindala, Vladimir Podolskiyband Michael Gerndt

Chair of Computer Architecture and Parallel Systems, TU Munich, Garching, Germany

Keywords:

Container Runtime, OCI Runtime, Performance Evaluation, Benchmarking for Container Runtimes,
Containers, Resource Management, Cloud Computing.

Abstract:

The co-location of containers on the same host leads to significant performance concerns in the multi-tenant
environment such as the cloud. These concerns are raised due to an attempt to maximize host resource utilization by increasing the number of containers running on the same host. The selection of a container runtime
becomes critical in the case of strict performance requirements. In the scope of the study, two commonly usedruntimes were evaluated: containerd (industry standard) and CRI-O (reference implementation of the CRI)
on two different Open Container Initiative (OCI) runtimes: runc (default for most container runtimes) andgVisor (highly secure alternative to runc). Evaluation aspects of container runtimes include the performanceof running containers, the performance of container runtime operations, and scalability. A tool called TouchStone was developed to address these evaluation aspects. The tool uses the CRI standard and is pluggable
into any Kubernetes-compatible container runtime. Performance results demonstrate the better performanceof containerd in terms of CPU usage, memory latency and scalability aspects, whereas file system operations(in particular, write operations) are performed more efficiently by CRI-O.

1

INTRODUCTION

Cloud computing as an infrastructure model has become an industry standard. Container technology
plays a leading role in this change allowing to efficiently share host resources among multiple tenants.
Hence, the demand for the performance evaluationof container workloads has risen. The widely-usedcontainer orchestrator Kubernetes supports every container runtime that implements the standardized Container Runtime Interface (CRI) (Hong, 2019). Although the cluster administrator can choose any CRI
implementation, a guiding evaluation of runtime performance is missing.
Resource management is the most important task
in multi-tenant environments such as the cloud. It
aims at resource efficiency in dense deployment models like FaaS (Functions-as-a-Service, a service that
abstracts away any underlying infrastructure of an
application deployment) and other serverless architectures. As new deployment models are being introduced to the market, more requirements on specific facets of container runtime performance pop up.
For example, Function-as-a-Service infrastructure deabhttps://orcid.org/0000-0002-7773-5342
https://orcid.org/0000-0002-2775-3630

mands for low start-up times of the underlying container runtime. Past studies have shown that theperformance overhead of wide-spread container runtimes is low and they are more resource-efficient than
traditional virtual machines (Kozhirbayev and Sinnott, 2017). Combined with the ability to co-locate
containers, this resource-efficiency leads to container
runtimes being the main driver behind energy andcost reduction in the container-based computing environments. Still, though virtual machines are moreresource-intensive, they offer a higher degree of security and performance isolation (Bui, 2015).
In this study, two commonly used runtimes areevaluated: containerd and CRI-O on two different Open Container Initiative (OCI) runtimes: runc
(default for most container runtimes) and gVisor(highly secure alternative to runc). The evaluation
tool TouchStone is developed to evaluate various performance aspects of resource management in container runtimes in a reliable manner. TouchStone
measures how well the runtime’s performance scaleswith the increase in the number of containers, howresource overuse is coped with as well as general runtime performance.
Section 2 provides the basic background knowledge for the paper. Section 3 delves into the imple273

Espe, L., Jindal, A., Podolskiy, V. and Gerndt, M.
Performance Evaluation of Container Runtimes.
DOI: 10.5220/0009340402730281
In Proceedings of the 10th International Conference on Cloud Computing and Services Science (CLOSER 2020), pages 273-281
ISBN: 978-989-758-424-4
Copyright c 2020 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved

CLOSER 2020 - 10th International Conference on Cloud Computing and Services Sciencementation details of the container runtimes studies.
Section 4 discusses related works. Section 5 examines the architecture and implementation of TouchStone evaluation tool. Section 6 provides evaluation
results. Section 7 summarizes and discusses the results. Section 8 concludes the work.

2

BACKGROUND

2.1

Evaluated Container Runtimes

A container runtime is a software that runs the containers and manages the container images on a deployment node. Specific container runtimes are discussed in the following subsection. This subsection
explores two container runtimes: containerd and CRIO.
2.1.1

Containerd

Containerd is the default runtime used by the Docker
engine (Crosby, 2017). It is often being referred toas an industry standard because of its wide adoption.
Underneath, this runtime uses runc, the reference implementation of the OCI runtime specification. containerd stores and manages images and snapshots; itstarts and stops containers by delegating execution
tasks to the OCI runtime (The containerd authors, ).
To start a new containerized process, containerd has
to undertake the following actions: 1) create a new
Container from a given OCI image; 2) create a new
Task in the Container context; 3) start the Task (atthis point runc takes over and starts executing the OCI
bundle supplied by containerd).
Containerd provides CRI-compatible gRPC endpoint enabled by default. When using this endpoint,
abstractions specific to containerd are hidden from theclient so that the user can operate on CRI-level abstractions.
2.1.2

CRI-O

Container Runtime Interface (CRI) is a standardized interface for Kubernetes plugins that executeand watch over containers. It was created as an attempt to stabilize the communication interface between kubelet and the host container runtime. It isbased on gRPC, a cross-language library for remote
procedure calls using Protocol Buffers.
CRI-O is a container runtime built to provide
a bridge between OCI runtimes and the high-level
Kubernetes CRI. It is based on an older version of
the Docker architecture that was built around graph274

drivers (Walli, ). It is mainly developed by RedHat,
and it serves as a default runtime for OpenShift, a
popular Kubernetes distribution for enterprise (McCarty, ). The typical life cycle of the interaction with
CRI-O is similar to containerd over the CRI endpoint.
Major differences in internal container handling tocontainerd do not exist since runc is the default OCI
runtime when running CRI-O.

2.2 Evaluated OCI Runtimes
The Open Container Initiative is a Linux Foundation project to design open standards for operatingsystem-level virtualization, most importantly Linuxcontainers. The Open Containers Initiative defines
two standards – the image-spec for OCI images andthe runtime-spec for OCI runtimes1,2 . The typicaljob sequence would be that a container runtime (e.g.
containerd) downloads an OCI image, unpacks it andprepares an OCI bundle (a container specification including the root filesystem) on the local disk. After
that, a OCI runtime like runc is able to create a running instance from this container specification. OCI
images can be created using a number of tools, forexample docker build command. After the successfulbuild, these images are usually pushed and published
to a public or private container registry. This subsection explores two container runtimes: runc and gVisor(runsc).
2.2.1

Runc

Runc is the lowest runtime layer of containerd that
explicitly handles containers. It was used as the reference implementation when drafting the OCI runtime
specification (Open Container Initiative, 2019). Internally, runc uses libcontainer to interact with OS-levelcomponents.
2.2.2

Gvisor (Runsc)

Typical OCI runtimes like runc limit system accessusing different capability models integrated into theLinux kernel like AppArmor, SELinux and SECCOMP. If an adversary has access to a limited set ofsystem calls and manages to exploit a vulnerability
in the Linux kernel, then he may be able to escapethe sandbox and gain privileged access. This is a major risk when running untrusted code like a providerof FaaS infrastructure does. When running a container in a VMM-based runtime like Kata Containers,a hypervisor will be started and it will virtualize the
1 github.com/opencontainers/image-spec/
2 github.com/opencontainers/runtime-spec/

Performance Evaluation of Container Runtimes

underlying system hardware the container is runningon (kat, 2019); this approach has a significant performance overhead.
gVisor’s approach to container isolation is in between that of a typical container runtime and of afully virtualized runtime. The idea of a guest kernel of VMM-based runtimes served as an inspiration
for a user-space kernel Sentry (which uses ptrace
for syscall interception) and for container file-systemaccess manager Gofer. Both of them are instantiated once for every OCI container in a pod sandbox.
This approach is a middle ground between the default
control groups / namespaces approach and the VMM
model. To reduce overhead, it does not virtualize
system hardware and leaves scheduling and memory
management to the host kernel. This split of responsibilities negatively impacts performance for applications loaded with system calls and leads to potential
incompatibilities between some workloads and gVisor due to specific Linux kernel behaviour or unimplemented Linux system calls (LLC, 2019).
gVisor is implemented as an OCI-compatible runtime that can be easily plugged into well-known container tools like containerd, Docker and CRI-O. Using a containerd plugin it is also possible to switch
between the default runtime and gVisor using annotations (LLC, 2019).

3

RELATED WORK

The literature on container runtime evaluation is
scarce with the most of the related research only partially being devoted to the evaluation of container runtime overhead.
Avino et al. performed the research in the context
of resource-critical mobile edge computing, and havediscovered that the CPU usage of the Docker process is constant regardless of the CPU cycle consumption of the containerized process (Avino et al., 2017).
Casalicchio et al. have benchmarked Docker Engine
for CPU- and I/O-intensive workloads with the results reporting on the reduction in overhead of Docker
Engine from 10% down to 5% when the amountof requested CPU cycles by the container is over
80% (Casalicchio and Perciballi, 2017). Kozhirbayev
et al. compared Docker to rivaling Flockport containers (based on LXC) showing that I/O and system calls
were responsible for most performance overheads forboth container implementations with Docker memory
performance being slightly better than that of Flockport (Kozhirbayev and Sinnott, 2017). Morabito etal. also identify the disk performance as the majorbottleneck both for container-based and hypervisor-

based virtualization although no significant overheads
in other performance metrics were detected (Morabito
et al., 2015).
The study by Xavier et al. went further by discovering that the near-native performance overheadof containers (LXC, Linux-VServer, and Open VZ) is
acceptable for HPC applications (Xavier et al., 2013).
The same argument holds for Docker in data-intensive
HPC applications (Chung et al., 2016). Lee et al.
evaluated performance of containers under the hoodof production serverless computing environments by
AWS, Microsoft, Google and IBM concluding that
Amazon Lambda performs better in terms of CPU,
network bandwidth, and I/O throughput (Lee et al.,2018). K. Kushwaha’s study on the performance of
VM-based and OS-level container runtimes such as
runc, kata-containers, also containerd and CRI-O concluded that containerd performs better in comparison to CRI-O and Docker due to different file system driver interface design though CRI-O has very
low container start-up latency in comparison to containerd (Kushwaha, 2017).

4

TouchStone

4.1 Overview of the Tool
This section describes the evaluation tool TouchStone(the name refers to small stones that have beenused in the past to determine the quality of gold)
which was implemented to conduct tests for thisstudy. The tool is published under the MIT license
on GitHub. TouchStone is implemented in Go due
to available set of open-source tools that use CRI are
implemented in Go.
The main motivation for the development of thetool is to improve container runtime evaluation bycollecting meaningful evaluation data during multiple runs and exporting it. TouchStone allows to evaluate CPU, memory and block I/O performance aswell as container operations performance and scalability of the performance under varying load. TouchStone runs evaluations in a testing matrix that tests
each configuration pair consisting of container runtime (CRI runtime) and OCI runtime once. The
tests are identified by a hierarchical naming scheme
context.component.facet, for instance, performance.cpu.total in a CPU performance benchmark. All of the configuration listed above is doneusing a YAML file. After all benchmarks have beenrun, an index of the test results is generated and injected together with the results into an an easily readable HTML report.
275

CLOSER 2020 - 10th International Conference on Cloud Computing and Services Sciencepkg/benchmark

pkg/runtime

Benchmark interface+ NewClient(address): Client

runtime in benchmarks, the runtime has to expose thisendpoint either directly or via symbolic link.

CRI+ Run(client, handler): Report+ Name(): string+ Labels(): []string

Client struct...
RuntimeServiceClient
ImageServiceClient

Report interface+ Aggregate(report): Report+ Scale(scale): Report

pkg/visual+ Write(file, entries, index)

CRIs: []string
OCIs: []string
Items: []Benchmark
Runs: int

generates

Matrix struct+ Index(): Index+ Run(): []MatrixEntry

pkg/config+ Parse(file): Config

MatrixEntry struct

Config struct

CRI, OCI: string
Results: []MatrixResult+ Matrix(): Matrix

MatrixResult struct
Name: []string
Aggregated: Report
Reports: []Report
Index map[string]IndexEntry

pkg/benchmark/suites+ All(): []Benchmark
Performance: []Benchmark
Operations: []Benchmark

IndexEntry struct

Limits: []Benchmark
Scalability: []Benchmark

Labels: []string

4.2.3

pkg/benchmark

This package provides benchmarking framework anddefines types Matrix, Benchmark and Report as well
as Index and some helpers. Matrix specifies a list oftargets to benchmark and has a slice of Benchmarksthat are invoked for each combination of CRI runtime
and OCI runtime. For each CRI runtime a new runtime.Client is instantiated.
After Benchmark finishes, a report is generated.
All the reports are collected together with the reportaggregate and the benchmark name in a MatrixResult. When all CRI/OCI combinations benchmarks
are finished, the array of results is written to the destination file specified in the YAML configuration andthe Index of the Matrix is generated. Index stores
benchmark-report combinations and persists acrossdifferent configuration runs.

pkg/cmd+ Execute()

Figure 1: Touchstone Package Overview.

4.2

Architecture

TouchStone consists of multiple packages as shownin the Figure 1. Following subsections provide an
overview for each of them.
4.2.1

pkg/cmd

This package implements the command line interface(CLI). TouchStone provides four commands to theuser – version, benchmark, list and index. The
implementation of the user interface is based on the
Cobra framework used by e.g. Docker and Kubernetes. version command returns the version of thetool and all CRI implementations it is able to talk to.
benchmark command runs the specified benchmark.
list command lists all benchmarking suites while
the index command generates an index to preview
the results generated by the benchmark command.
4.2.2

pkg/config

pkg/config implements YAML-parsable configuration structure. Each configuration defines an output
file to write the benchmark results to, a list of filters
specifying which benchmarks to run, basic benchmark parameters like runs per benchmark and benchmark scale as well as which container (CRI) and OCI
runtime to use. In case the user wants to include a new
276

4.2.4

pkg/benchmark/suites

Benchmarks implemented in this package can begrouped into four categories:• Limits. Suite attempts to benchmark performanceof containers with capped resources. Behaviourof gVisor on such benchmark is especially interesting due to the hypervisor being included in thecontrol group accounting.
• Operations. Suite tests common container runtime operation performance including latencies
during creation and destruction as well as container launch time.
• Performance. Suite collects conventional performance metrics of containers running using a specific runtime with measurements provided by sysbench3 .
• Scalability. Suite quantifies how scalable is theperformance of a given runtime in relation to theamount container instances.
4.2.5

pkg/runtime

This package contains a wrapper for the default CRI
gRPC client. It includes helpers to simplify creationof containers and sandboxes as well as pulling images.

3 github.com/akopytov/sysbenchPerformance Evaluation of Container Runtimes

4.2.6

pkg/visual

It uses HTML templating from text/template,
Bootstrap and Chart.js to provide a visualization of
the collected results.

5

EXPERIMENTAL RESULTS

5.1

Test Setting

To run all the benchmarks on the same machine, acollection of container tools is used:• containerd (v1.2.0-621-g04e7747e);• CRI-O (1.15.1-dev);

with runc introducing higher performance (see Figure 2a).
However, when looking at the minimum and average latency, it can be observed that runsc introduces
an additional overhead of a factor between 1.48x and
1.72x when accessing memory. Additionally, it seems
that containerd performs on average better when accessing memory as seen in Figure 2a.
The maximum latency benchmark is in stark contrast to the average latency as seen in Figure 2c,
because containerd/runc has a latency spike incomparison to containerd|runsc, crio/runc andcrio/runsc. The benchmark was repeated with a reordered test execution and the same effect has been
observed. Apparently, this effect is rather a containerd/runsc runtime feature than an outlier.

• runc (1.0.0-rc8+dev);

5.2.3• runsc(release-20190529.1-194gf10862696c85);

All of these tools were compiled from scratch on
the testing machine to mitigate the incompatibility
risk. They were run on a linux kernel 4.19.0-5amd64 under Debian Buster. All benchmarks havebeen executed 5 to 20 times depending on the benchmark workload. Due to the type of benchmarks ran
and especially their vulnerability to strong outliers,
only the median of the generated dataset is shown inresulting graphs and tables. All tests have been executed in a virtual machine with 2 CPU cores (2,9 GHz
Intel Core i5-6267U CPU) and 8 GiB of RAM. These
conditions are similar to a typical cloud virtual machine though server CPUs tend to have more cache,additional instructions and other features.

Sysbench’s fileio read benchmarks reveals large
difference in performance between runc and runsc
as shown in Figure 4a. gVisor introduces massive
delays in workloads loaded with file operations likethis synthetic test. The observed performance drop isdue to an inefficient implementation of the virtual filesystem provided by Gofer, which has been improvedin a recent gVisor version4 . The difference between
containerd and crio when using runc is negligible,
even though containerd is slightly faster in almost
all the cases.
When writing to disk, the differences between
runc and runsc stand out far less pronounced thanin the read benchmarks as shown in Figure 4b. Therefore, the underlying hardware and operating systemare, up to a certain degree, the cause for smaller variation in write speed. In contrast to read benchmarks,
CRI-O performs slightly better than containerd in allcases.

5.2

General Performance

5.3 Container Operations

5.2.1

CPU

Figure 5 points at a slightly better performance ofcontainerd when creating a container (in comparison to CRI-O). This head start is diminished by thehigher run latency such that in total containerd performs worse than CRI-O. It is also visible that runsc
operates almost always faster than runc.

• CNI (v0.7.5);• crictl (1.15.0-2-g4f465cf).

The CPU benchmark measures the time needed tocompute all prime numbers up to a certain maximum,
thus it should not use a lot of system calls. Figure 3
shows that containerd has a lower computational overhead compared to CRI-O, and the same applies torunc in comparison to runsc.
5.2.2

MemoryWhen looking at the total time spent in the memory
benchmark, containerd is the clear winner. A subtle
difference between runc and runsc can also be seen

Disk I/O

5.4 Scalability of Runtime Performance
Scalability benchmarks test the performance of starting, running and destroying an increasing amount oflong-running containers.
4 github.com/google/gvisor/pull/447

277

CLOSER 2020 - 10th International Conference on Cloud Computing and Services Sciencecrio/runsc

crio/runsc

crio/runsc

crio/runc

crio/runc

crio/runc

containerd/runsc

containerd/runsc

containerd/runsc

containerd/runc

containerd/runc

containerd/runc

0

5

10

15

(a) Total Time and Measurements Are
in Seconds.

Avg

Min

0

0,05

0,1

0,15

0

0,2

(b) Minimum Avg. Latency and Measurements Are in Milliseconds.

1

2

3

4

(c) Maximum Latency and Measurements Are in Milliseconds.

Figure 2: performance.memory Benchmark for Different Aggregations with 10 Runs.
Table 1: Overview of the Evaluation Results.

Metric
CPUTime(s)HardLimit(s)ScalingLimit(s)MemTimeMemMinLatency(ms)
MemAvgLatency(ms)MemMaxLatency(ms)FileRndRead(s)FileSeqRead(s)FileRndWrite(s)FileSeqRewrite(s)FileSeqWrite(s)OpCreate(s)OpRun(s)OpCreateAndRun(s)OpDestroy(s)Scalability5(s)Scalability10(s)Scalability50(s)cd/runc
23.86
340.33
28.38
10.12
0.10
0.11
3.55
0.03
0.36
6.39
2.64
2.71
0.50
0.18
0.68
0.48
3.49
8.35
37.69

crio/runc
25.35
354.98
28.22
13.23
0.10
0.13
1.14
0.04
0.39
6.03
2.43
2.54
0.51
0.03
0.54
0.53
4.77
9.71
51.66

crio/runsc
26.82
403.40
14.51
0.15
0.20
0.95
0.31
3.26
12.03
6.57
6.54
0.48
0.05
0.54
0.40
7.03
16.81
145.77benchmark. crio/runsc demonstrates significant
slowdown on 50 containers, which might be due tothe larger runtime overhead.

crio/runsc
crio/runc
containerd/runsc

5.5 Resource Limits and Quotas

containerd/runc
0

5

10

15

20

25

30

Figure 3: performance.cpu.time, 20 Runs, Measurementin Seconds.

With 5 containers there is not much of a difference
to the section 5.3; containerd/runsc performs best
overall while crio/runsc falls behind.
The observation made in 5 containers test continues and becomes more pronounced. We can see thatthe required time for starting and stopping 10 containers is in a linear relation to the container count.
The linear trend discontinues with 50 containers and containerd/runc comes out ahead in this278

cd/runsc
26.29
10.6
0.14
0.19
1.98
0.29
3.32
12.74
6.74
6.80
0.42
0.10
0.53
0.55
2.96
5.99
43.36

This benchmark focused at testing containers with the
CPU Hard and Scaling limits using the same sysbench
CPU test from subsection 5.2.1, but with capped CPU
resource. Capping configuration was set to cfs quota us = 1000 and cfs period us = 10000.
Table 1 showing the CPU Hard Limit showsthat the overhead of runsc hits the performancemuch harder in this benchmark compared to performance.cpu.time. This behaviour can be explainedby the fact that gVisor adds both Sentry and Goferto the control group. Since these components require
CPU time too, and are run in user space instead of kernel space, they count towards the control group CPU

Performance Evaluation of Container Runtimes

crio/runsc

crio/runsc

crio/runc

crio/runc

containerd/runsc

containerd/runsc

containerd/runc

containerd/runc

SeqReadRndRead

0

1

2

3

4

SeqWrite
SeqRewrite
RndWrite(a) Read Throughput.

0

5

10

15

(b) Write Throughput.

Figure 4: performance.disk Benchmark for Read and Write with 10 Runs and All the Measurements Are in Seconds.

6
crio/runsc

crio/runc

containerd/runsc

containerd/runc

0
Run

Destroy0,20,4CreateAndRun

0,6

0,8

CreateFigure 5: operations.container.lifecycle, 20 Runs,
Measurement in Seconds.

account. Due to a runsc bug, the containerd/runsc
setup ignored the control group limits and hence was
excluded from the benchmark.
The scaling benchmark shines light into how
workloads behave that are scaled from half of the allocatable CPU time up to full allowance over the courseof 10 seconds. Since gVisor does not support updating resource limits during the execution of a container, runsc had to be excluded from these tests. It
can be seen that the performance difference is verysmall as shown in Table 1 Scaling Limit, most likely
due to the low logical overhead between CRI runtime
and OCI runtime

DISCUSSION

Table 1 shows the summary results of all the benchmarks run in the scope of the study with the highlighted ones being the best for that particular benchmark. To summarize, containerd performs better
when running a container without limits due to lower
runtime overhead, although CRI-O is faster when running a container that has been created beforehand.
The lead of containerd continues in memory performance. Disk performance gives a different picture:
although containerd takes an upper hand in read performance, its superiority is broken by CRI-O in file
read performance.
When comparing runc and runsc alias gVisor, runc
comes ahead as the clear leader. This is not surprising due to Gofer virtual file system being the bottleneck. It should be noted that runsc may perform on
the same level as runc when running workloads that
do not perform a lot of system calls, e.g. the performance.cpu.time benchmark. In every other benchmark runsc did worse than runc except the operations.container.lifecycle. It should be notedthat runsc performs worse than runc even in compute bound workloads when running in a rigourously
limited environment like limits.cpu.time. The
containerd/runc setup performs best for I/O-heavy
workloads like databases and web servers, while
crio/runc is a solid starter pack in almost any case.
The usage of gVisor should be limited to systems
where security is vitally important and has to beachieved with limited loss in raw computing power.
Running very small containers with strong resource
capping may be significantly slower in gVisor compared to runc.

279

CLOSER 2020 - 10th International Conference on Cloud Computing and Services Sciencecrio/runsc

crio/runsc

crio/runsc

crio/runc

crio/runc

crio/runc

containerd/runsc

containerd/runsc

containerd/runsc

containerd/runc

containerd/runc
0

2

4

6

8

(a) 5 Containers.

containerd/runc
0

5

10

(b) 10 Containers.

15

20

0

50

100

150

200

(c) 50 Containers.

Figure 6: scalability.runtime Benchmark, 10 Runs for Different Number of Containers, All the Measurements in Seconds.

7

CONCLUSION

The developed evaluation tool TouchStone has been
applied to wide-spread container and OCI runtimes
such as containerd, CRI-O and runc and gVisor todetermine performance implications of various evaluated aspects such as scaling the number of containers
started in the runtime. The evaluation results highlighted crio/runc setup as the best starting option
almost for any use-case, whereas containerd/runc
turned out to excel at I/O-heavy workloads. Runtimes
setups including runsc might provide better security,
but their use must be carefully considered when performance is important. Prospective directions of future work include automatic deployment of a Kubernetes cluster with the container runtime picked specifically for given types of workloads and the support
for cross-machine evaluations, i.e. running Touchstone on multiple machines and aggregating the results from such distributed testing environment.

ACKNOWLEDGEMENTS
This work was supported by the funding of German
Federal Ministry of Education and Research (BMBF)in the scope of Software Campus program.

REFERENCES(2019). Kata containers. https://katacontainers.io. Accessedon 19.07.2019 14:05.
Avino, G., Malinverno, M., Malandrino, F., Casetti, C., andChiasserini, C.-F. (2017). Characterizing docker overhead in mobile edge computing scenarios. In Proceedings of the Workshop on Hot Topics in Container Networking and Networked Systems, pages 30–35. ACM.
Bui, T. (2015). Analysis of docker security. arXiv preprint
arXiv:1501.02967.
Casalicchio, E. and Perciballi, V. (2017). Measuring docker
performance: What a mess!!! In Proceedings of280

the 8th ACM/SPEC on International Conference on
Performance Engineering Companion, pages 11–16.
ACM.
Chung, M. T., Quang-Hung, N., Nguyen, M., and Thoai, N.
(2016). Using docker in high performance computing
applications. In 2016 IEEE Sixth International Conference on Communications and Electronics (ICCE),
pages 52–57.
Crosby, M. (2017). What is containerd? https://blog.docker.
com/2017/08/what-is-containerd-runtime/. Accessedon 17.06.2019 18:57.
Hong, Y.-J. (2019). Introducing container runtime interface
in kubernetes. https://kubernetes.io/blog/2016/12/
container-runtime-interface-cri-in-kubernetes/. Accessed on 03.07.2019 17:59.
Kozhirbayev, Z. and Sinnott, R. O. (2017). A performancecomparison of container-based technologies for the
cloud. Future Generation Computer Systems, 68:175–
182.
Kushwaha, K. (2017). How container runtimes matter inkubernetes? https://events.linuxfoundation.org/wpcontent/uploads/2017/11/How-Container-RuntimeMatters-in-Kubernetes -OSS-Kunal-Kushwaha.pdf.
Accessed on 6.06.2019.
Lee, H., Satyam, K., and Fox, G. (2018). Evaluation of production serverless computing environments. In 2018
IEEE 11th International Conference on Cloud Computing (CLOUD), pages 442–450.
LLC, G. (2019).
gvisor architecture guide.
https://gvisor.dev/docs/architecture guide/. Accessed on
10.07.2019 20:07.
McCarty, S. Red hat openshift container platform 4
now defaults to cri-o as underlying container engine.
https://www.redhat.com/en/blog/red-hatopenshift-container-platform-4-now-defaults-crio-underlying-container-engine.
Accessed on
22.07.2019 16:14.
Morabito, R., Kjällman, J., and Komu, M. (2015). Hypervisors vs. lightweight virtualization: A performancecomparison. In 2015 IEEE International Conferenceon Cloud Engineering, pages 386–393.
Open Container Initiative (2019). Oci runtime specification.
https://github.com/opencontainers/runtime-spec/tree/
74b670efb921f9008dcdfc96145133e5b66cca5c/.
Accessed on 30.06.2019 12:54.
The containerd authors. containerd – getting started.

Performance Evaluation of Container Runtimes

https://containerd.io/docs/getting-started/. Accessedon 22.07.2019 19:00.
Walli, S. Demystifying the open container initiative(oci) specifications. https://blog.docker.com/2017/07/demystifying-open-container-initiative-ocispecifications/. Accessed on 30.06.2019 15:50.
Xavier, M. G., Neves, M. V., Rossi, F. D., Ferreto, T. C.,
Lange, T., and De Rose, C. A. F. (2013). Performance
evaluation of container-based virtualization for highperformance computing environments. In 2013 21stEuromicro International Conference on Parallel, Distributed, and Network-Based Processing, pages 233–240.

281