Which CRI should I use to replace docker for my kubernetes cluster
By: Ulrich Giraud
Tags: Containers, Cri, Kubernetes, Docker, Container Orchestration

So, the deprecation of docker by kubernetes as made some waves and worried a good part of the community, but, docker isn’t the only implementation of the CRI.

Alright then, what other CRI should I use? you may ask, and in fact this is the exact question I had, and the purpose of this story.

I’ve tested the following CRIs and tried to get an answer by benching them:

dockershim

containerd

crio

For cri-o, 2 backends have been tested : runc and crun, to test against cgroupsv2 and see the impact if any

Test environment

The test environment will be a kubernetes 1.19.4 cluster, created using ansible and the roles from https://gitlab.com/incubateur-pe ( https://galaxy.ansible.com/incubateurpe )

The servers will run on kvm and configured as follows :

master : Centos/7, 2vcpus/2Go ram

crio-crun node : Fedora-32, 2vcpus/4Go ram

other nodes : Centos/7, 2vcpus/2Go ram

The underlying hardware is an i7–9700K, 64Go of ram and a mp510 nvme drive

Cluster creation

Pretty straightforward, I used molecule to pop up a cluster, and set it to use a different cri on each worker node. The source of the cluster is here : https://gitlab.com/incubateur-pe/kubernetes-bare-metal/-/tree/dev/molecule/criBench

So let’s run a “molecule converge” and, 10 minutes later, we’ve got ourselves the following cluster:

Ok! time to bench it!

First bench : bucketbench

Bucketbench ( https://github.com/estesp/bucketbench ) is a tool for executing a sequence of actions against a container engine, which is perfect to get an idea of the performance of each of the previous nodes.

The scenario is simple :

3 threads

15 iterations

run/stop/delete

And the results are ( in ms ) :

Alright! we have quite a difference in performance!

But, wait a minute, what is this docker-shim? Why is there 5 instances tested but only 4 servers?

Ok, let’s dig down, docker, as we use it with the docker client is not what kubernetes uses, in fact, docker implements the CRI and provides a socket which is callable like any other cri socket.

So, the difference here is :

- docker-shim: bench via the cri socket

- docker-cli : bench via the docker client

As you can see, there is a tremendous difference in performance when using the cli, and in fact, docker isn’t as bad as I thought, he is in fact faster that cri-o in this test.

But, the clear winner here seems to be containerd

Second bench: kubernetes

While running the tests above, I had the feeling that this was not the whole story: what is their behaviour when used by kubernetes? is there more than just run/stop/delete to test? Does this difference in performance have any meaning on a real cluster?

And, in fact, I was right, those 3 verbs are far from the whole story!

Let’s dive in, with, this time, prometheus in the cluster, grafana to visualize the metrics, a custom dashboard to get a clear view ( https://gitlab.com/ulrich.giraud/bench-cri/-/blob/master/dashboard/dashboard_bench.json ) and something to deploy in the cluster

As the test is on the runtime only, and not the workload, what I’ve deployed in the cluster is a daemonset ( to get it on each node each time ) of a busybox launching a “sleep infinity”:

---

apiVersion: apps/v1

kind: DaemonSet

metadata:

name: benchds-replaceme

namespace: benchds

labels:

k8s-app: benchds

spec:

selector:

matchLabels:

name: benchds

template:

metadata:

labels:

name: benchds

spec:

containers:

- name: benchds

image: busybox:latest

command:

- sleep

- infinity

resources:

limits:

memory: 20Mi

requests:

cpu: 10m

memory: 20Mi

This daemonset will be deployed with unique name :

a hundred times with some delay between 2 creations

a hundred times in bulk

a thousand times in bulk

Grafana gave me the following results, which are great, but not really easy to read and compare :