Project: Kata Containers (Runtime / Container Runtime)
File: how-to-use-kata-containers-with-firecracker.md

This document provides an overview on how to run Kata Containers with the AWS Firecracker hypervisor. AWS Firecracker is an open source virtualization technology that is purpose-built for creating and managing secure, multi-tenant container and function-based services that provide serverless operational models. AWS Firecracker runs workloads in lightweight virtual machines, called `microVMs`, which combine the security and isolation properties provided by hardware virtualization technology with the speed and flexibility of Containers. Please refer to AWS Firecracker for more details. This document requires the presence of Kata Containers on your system. Install using the instructions available through the following links: Kata Containers Kata Containers manual installation: Automated installation does not seem to be supported for Clear Linux, so please use steps. Note: Create rootfs image and not initrd image. For information about the supported version of Firecracker, see the Kata Containers . To install Firecracker we need to get the `firecracker` and `jailer` binaries: ```bash $ release_url="https://github.com/firecracker-microvm/firecracker/releases" $ version=$(yq read <kata-repository>/versions.yaml assets.hypervisor.firecracker.version) $ arch=`uname -m` $ curl ${release_url}/download/${version}/firecracker-${version}-${arch} -o firecracker $ curl ${release_url}/download/${version}/jailer-${version}-${arch} -o jailer $ chmod +x jailer firecracker ``` To make the binaries available from the default system `PATH` it is recommended to move them to `/usr/local/bin` or add a symbolic link: ```bash $ sudo ln -s $(pwd)/firecracker /usr/local/bin $ sudo ln -s $(pwd)/jailer /usr/local/bin ``` More details can be found in In order to run Kata with AWS Firecracker a block device as the backing store for a VM is required. To interact with `containerd` and Kata we use the `devmapper` `snapshotter`. To check support for your `containerd` installation, you can run: ``` $ ctr plugins ls |grep devmapper ``` if the output of the above command is: ``` io.containerd.snapshotter.v1 devmapper linux/amd64 ok ``` then you can skip this section and move on to `Configure Kata Containers with AWS Firecracker` If the output of the above command is: ``` io.containerd.snapshotter.v1 devmapper linux/amd64 error ``` then we need to setup `devmapper` `snapshotter`. Based on a [very useful guide](https://docs.docker.com/storage/storagedriver/device-mapper-driver/) from docker, we can set it up using the following scripts: Note: The following scripts assume a 100G sparse file for storing container images, a 10G sparse file for the thin-provisioning pool and 10G base image files for any sandboxed container created. This means that we will need at least 10GB free space. ``` set -ex DATA_DIR=/var/lib/containerd/devmapper POOL_NAME=devpool mkdir -p ${DATA_DIR} sudo touch "${DATA_DIR}/data" sudo truncate -s 100G "${DATA_DIR}/data" sudo touch "${DATA_DIR}/meta" sudo truncate -s 10G "${DATA_DIR}/meta" DATADEV=$(sudo losetup --find --show "${DATADIR}/data") METADEV=$(sudo losetup --find --show "${DATADIR}/meta") SECTOR_SIZE=512 DATASIZE="$(sudo blockdev --getsize64 -q ${DATADEV})" LENGTHINSECTORS=$(bc <<< "${DATASIZE}/${SECTORSIZE}") DATABLOCKSIZE=128 LOWWATERMARK=32768 sudo dmsetup create "${POOL_NAME}" \ --table "0 ${LENGTHINSECTORS} thin-pool ${METADEV} ${DATADEV} ${DATABLOCKSIZE} ${LOWWATERMARK}" cat << EOF [plugins]

poolname = "${POOLNAME}" rootpath = "${DATADIR}" baseimagesize = "10GB" discard_blocks = true EOF ``` Make it executable and run it: ```bash $ sudo chmod +x ~/scripts/devmapper/create.sh $ cd ~/scripts/devmapper/ $ sudo ./create.sh ``` Now, we can add the `devmapper` configuration provided from the script to `/etc/containerd/config.toml`. Note: If you are using the default `containerd` configuration (`containerd config default >> /etc/containerd/config.toml`), you may need to edit the existing `[plugins."io.containerd.snapshotter.v1.devmapper"]`configuration. Save and restart `containerd`: ```bash $ sudo systemctl restart containerd ``` We can use `dmsetup` to verify that the thin-pool was created successfully. ```bash $ sudo dmsetup ls ``` We should also check that `devmapper` is registered and running: ```bash $ sudo ctr plugins ls | grep devmapper ``` This script needs to be run only once, while setting up the `devmapper` `snapshotter` for `containerd`. Afterwards, make sure that on each reboot, the thin-pool is initialized from the same data directory. Otherwise, all the fetched containers (or the ones that you have created) will be re-initialized. A simple script that re-creates the thin-pool from the same data directory is shown below: ``` set -ex DATA_DIR=/var/lib/containerd/devmapper POOL_NAME=devpool DATADEV=$(sudo losetup --find --show "${DATADIR}/data") METADEV=$(sudo losetup --find --show "${DATADIR}/meta") SECTOR_SIZE=512 DATASIZE="$(sudo blockdev --getsize64 -q ${DATADEV})" LENGTHINSECTORS=$(bc <<< "${DATASIZE}/${SECTORSIZE}") DATABLOCKSIZE=128 LOWWATERMARK=32768 sudo dmsetup create "${POOL_NAME}" \ --table "0 ${LENGTHINSECTORS} thin-pool ${METADEV} ${DATADEV} ${DATABLOCKSIZE} ${LOWWATERMARK}" ``` We can create a systemd service to run the above script on each reboot: ```bash $ sudo nano /lib/systemd/system/devmapper_reload.service ``` The service file: ``` [Unit] Description=Devmapper reload script [Service] ExecStart=/path/to/script/reload.sh [Install] WantedBy=multi-user.target ``` Enable the newly created service: ```bash $ sudo systemctl daemon-reload $ sudo systemctl enable devmapper_reload.service $ sudo systemctl start devmapper_reload.service ``` To configure Kata Containers with AWS Firecracker, copy the generated `configuration-fc.toml` file when building the `kata-runtime` to either `/etc/kata-containers/configuration-fc.toml` or `/usr/share/defaults/kata-containers/configuration-fc.toml`. The following command shows full paths to the `configuration.toml` files that the runtime loads. It will use the first path that exists. (Please make sure the kernel and image paths are set correctly in the `configuration.toml` file) ```bash $ sudo kata-runtime --show-default-config-paths ``` Next, we need to configure containerd. Add a file in your path (e.g. `/usr/local/bin/containerd-shim-kata-fc-v2`) with the following contents: ``` KATACONFFILE=/etc/kata-containers/configuration-fc.toml /usr/local/bin/containerd-shim-kata-v2 $@ ``` Note: You may need to edit the paths of the configuration file and the `containerd-shim-kata-v2` to correspond to your setup. Make it executable: ```bash $ sudo chmod +x /usr/local/bin/containerd-shim-kata-fc-v2 ``` Add the relevant section in `containerd`s `config.toml` file (`/etc/containerd/config.toml`): ``` [plugins.cri.containerd.runtimes] [plugins.cri.containerd.runtimes.kata-fc] runtime_type = "io.containerd.kata-fc.v2" ``` Note: If you are using the default `containerd` configuration (`containerd config default >> /etc/containerd/config.toml`), the configuration should change to : ``` [plugins."io.containerd.grpc.v1.cri".containerd.runtimes] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-fc] runtime_type = "io.containerd.kata-fc.v2" ``` Restart `containerd`: ```bash $ sudo systemctl restart containerd ``` We are now ready to launch a container using Kata with Firecracker to verify that everything worked: ```bash $ sudo ctr images pull --snapshotter devmapper docker.io/library/ubuntu:latest $ sudo ctr run --snapshotter devmapper --runtime io.containerd.run.kata-fc.v2 -t --rm docker.io/library/ubuntu ```