Project: Kata Containers (Runtime / Container Runtime)
File: containerd-kata.md

This document covers the installation and configuration of and . The containerd provides not only the `ctr` command line tool, but also the interface for and other CRI clients. This document is primarily written for Kata Containers v1.5.0-rc2 or above, and containerd v1.2.0 or above. Previous versions are addressed here, but we suggest users upgrade to the newer versions for better support. is a Kubernetes feature first introduced in Kubernetes 1.12 as alpha. It is the feature for selecting the container runtime configuration to use to run a pods containers. This feature is supported in `containerd` since . Before the `RuntimeClass` was introduced, Kubernetes was not aware of the difference of runtimes on the node. `kubelet` creates Pod sandboxes and containers through CRI implementations, and treats all the Pods equally. However, there are requirements to run trusted Pods (i.e. Kubernetes plugin) in a native container like runc, and to run untrusted workloads with isolated sandboxes (i.e. Kata Containers). As a result, the CRI implementations extended their semantics for the requirements: At the beginning, checks the network configuration of a Pod, and treat Pod with `host` network as trusted, while others are treated as untrusted. The containerd introduced an annotation for untrusted Pods since : ```yaml annotations: io.kubernetes.cri.untrusted-workload: "true" ``` Similarly, CRI-O introduced the annotation `io.kubernetes.cri-o.TrustedSandbox` for untrusted Pods. To eliminate the complexity of user configuration introduced by the non-standardized annotations and provide extensibility, `RuntimeClass` was introduced. This gives users the ability to affect the runtime behavior through `RuntimeClass` without the knowledge of the CRI daemons. We suggest that users with multiple runtimes use `RuntimeClass` instead of the deprecated annotations. The implements the for Kata. With `shimv2`, Kubernetes can launch Pod and OCI-compatible containers with one shim per Pod. Prior to `shimv2`, `2N+1` shims (i.e. a `containerd-shim` and a `kata-shim` for each container and the Pod sandbox itself) and no standalone `kata-proxy` process were used, even with VSOCK not available. The shim v2 is introduced in containerd and Kata `shimv2` is implemented in Kata Containers v1.5.0. Follow the instructions to . Note: `cri` is a native plugin of containerd 1.1 and above. It is built into containerd and enabled by default. You do not need to install `cri` if you have containerd 1.1 or above. Just remove the `cri` plugin from the list of `disabled_plugins` in the containerd configuration file (`/etc/containerd/config.toml`). Follow the instructions from the . Then, check if `containerd` is now available: ```bash $ command -v containerd ``` If you have installed Kubernetes with `kubeadm`, you might have already installed the CNI

You can manually install CNI plugins as follows: ```bash $ git clone https://github.com/containernetworking/plugins.git $ pushd plugins $ ./build_linux.sh $ sudo mkdir /opt/cni $ sudo cp -r bin /opt/cni/ $ popd ``` Note: `cri-tools` is a set of tools for CRI used for development and testing. Users who only want to use containerd with Kubernetes can skip the `cri-tools`. You can install the `cri-tools` from source code: ```bash $ git clone https://github.com/kubernetes-sigs/cri-tools.git $ pushd cri-tools $ make $ sudo -E make install $ popd ``` By default, the configuration of containerd is located at `/etc/containerd/config.toml`, and the `cri` plugins are placed in the following section: ```toml [plugins] [plugins.cri] [plugins.cri.containerd] [plugins.cri.containerd.default_runtime] [plugins.cri.cni] conf_dir = "/etc/cni/net.d" ``` The following sections outline how to add Kata Containers to the configurations. For Kata Containers v1.5.0 or above (including `1.5.0-rc`) Containerd v1.2.0 or above Kubernetes v1.12.0 or above The `RuntimeClass` is suggested. The following configuration includes two runtime classes: `plugins.cri.containerd.runtimes.runc`: the runc, and it is the default runtime. `plugins.cri.containerd.runtimes.kata`: The function in containerd (reference ) where the dot-connected string `io.containerd.kata.v2` is translated to `containerd-shim-kata-v2` (i.e. the binary name of the Kata implementation of ). ```toml [plugins.cri.containerd] no_pivot = false [plugins.cri.containerd.runtimes] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] privilegedwithouthost_devices = false runtime_type = "io.containerd.runc.v2" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] BinaryName = "" CriuImagePath = "" CriuPath = "" CriuWorkPath = "" IoGid = 0 [plugins.cri.containerd.runtimes.kata] runtime_type = "io.containerd.kata.v2" privilegedwithouthost_devices = true pod_annotations = ["io.katacontainers.*"] container_annotations = ["io.katacontainers.*"] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata.options] ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration.toml" ``` `privilegedwithouthost_devices` tells containerd that a privileged Kata container should not have direct access to all host devices. If unset, containerd will pass all host devices to Kata container, which may cause security issues. `pod_annotations` is the list of pod annotations passed to both the pod sandbox as well as container through the OCI config. `container_annotations` is the list of container annotations passed through to the OCI config of the containers. This `ConfigPath` option is optional. If you do not specify it, shimv2 first tries to get the configuration file from the environment variable `KATACONFFILE`. If neither are set, shimv2 will use the default Kata configuration file paths (`/etc/kata-containers/configuration.toml` and `/usr/share/defaults/kata-containers/configuration.toml`). For cases without `RuntimeClass` support, we can use the legacy annotation method to support using Kata Containers for an untrusted workload. With the following configuration, you can run trusted workloads with a runtime such as `runc` and then, run an untrusted workload with Kata Containers: ```toml [plugins.cri.containerd] [plugins.cri.containerd.default_runtime] runtime_type = "io.containerd.runtime.v1.linux" [plugins.cri.containerd.untrustedworkloadruntime] runtime_type =

``` You can find more information on the If you want to set Kata Containers as the only runtime in the deployment, you can simply configure as follows: ```toml [plugins.cri.containerd] [plugins.cri.containerd.default_runtime] runtime_type = "io.containerd.kata.v2" ``` Note: If you skipped the section, you can skip this section too. First, add the CNI configuration in the containerd configuration. The following is the configuration if you installed CNI as the section outlined. Put the CNI configuration as `/etc/cni/net.d/10-mynet.conf`: ```json { "cniVersion": "0.2.0", "name": "mynet", "type": "bridge", "bridge": "cni0", "isGateway": true, "ipMasq": true, "ipam": { "type": "host-local", "subnet": "172.19.0.0/24", "routes": [ { "dst": "0.0.0.0/0" } ] } } ``` Next, reference the configuration directory through containerd `config.toml`: ```toml [plugins.cri.cni] conf_dir = "/etc/cni/net.d" ``` The configuration file of `crictl` command line tool in `cri-tools` locates at `/etc/crictl.yaml`: ```yaml runtime-endpoint: unix:///var/run/containerd/containerd.sock image-endpoint: unix:///var/run/containerd/containerd.sock timeout: 10 debug: true ``` To run a container with Kata Containers through the containerd command line, you can run the following: ```bash $ sudo ctr image pull docker.io/library/busybox:latest $ sudo ctr run --cni --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh ``` This launches a BusyBox container named `hello`, and it will be removed by `--rm` after it quits. The `--cni` flag enables CNI networking for the container. Without this flag, a container with just a loopback interface is created. Use the script to create rootfs ```bash ctr i pull quay.io/prometheus/busybox:latest ctr i export rootfs.tar quay.io/prometheus/busybox:latest rootfs_tar=rootfs.tar bundle_dir="./bundle" mkdir -p "${bundle_dir}" rootfsdir="${bundledir}/rootfs" mkdir -p "${rootfs_dir}" layers_dir="$(mktemp -d)" tar -C "${layersdir}" -pxf "${rootfstar}" for ((i=0;i<$(cat ${layers_dir}/manifest.json | jq -r ".[].Layers | length");i++)); do tar -C ${rootfsdir} -xf ${layersdir}/$(cat ${layers_dir}/manifest.json | jq -r ".[].Layers[${i}]") done ``` Use runc spec to generate `config.json` ```bash cd ./bundle/rootfs runc spec mv config.json ../ ``` Change the root `path` in `config.json` to the absolute path of rootfs ```JSON "root":{ "path":"/root/test/bundle/rootfs", "readonly": false }, ``` ```bash sudo ctr run -d --runtime io.containerd.run.kata.v2 --config bundle/config.json hello sudo ctr t exec --exec-id ${ID} -t hello sh ``` With the `crictl` command line of `cri-tools`, you can specify runtime class with `-r` or `--runtime` flag. Use the following to launch Pod with `kata` runtime class with the pod in of `cri-tools`: ```bash $ sudo crictl runp -r kata podsandbox-config.yaml 36e23521e8f89fabd9044924c9aeb34890c60e85e1748e8daca7e2e673f8653e ``` You can add container to the launched Pod with the following: ```bash $ sudo crictl create 36e23521e8f89 container-config.yaml podsandbox-config.yaml 1aab7585530e62c446734f12f6899f095ce53422dafcf5a80055ba11b95f2da7 ``` Now, start it with the following: ```bash $ sudo crictl start 1aab7585530e6 1aab7585530e6 ``` In Kubernetes, you need to create a `RuntimeClass` resource and add the `RuntimeClass` field in the Pod Spec (see this for more information). If `RuntimeClass` is not supported, you can use the following annotation in a Kubernetes pod to identify as an untrusted workload: ```yaml annotations: io.kubernetes.cri.untrusted-workload: "true" ```