Project: Kata Containers (Runtime / Container Runtime)
File: using-Intel-QAT-and-kata.md

Intel QuickAssist Technology (QAT) provides hardware acceleration for security (cryptography) and compression. These instructions cover the steps for the latest which already include the QAT host driver. These instructions can be adapted to any Linux distribution. These instructions guide the user on how to download the kernel sources, compile kernel driver modules against those sources, and load them onto the host as well as preparing a specially built Kata Containers kernel and custom Kata Containers rootfs. Download kernel sources Compile Kata kernel Compile kernel driver modules against those sources Download rootfs Add driver modules to rootfs Build rootfs image There are some steps to complete only once, some steps to complete with every reboot, and some steps to complete when the host kernel changes. The following list of variables must be set before running through the scripts. These variables refer to locations to store modules and configuration files on the host and links to the drivers to use. Modify these as needed to point to updated drivers or different install locations. Make sure to check for the latest driver. ```bash $ export QATDRIVERVER=qat1.7.l.4.14.0-00031.tar.gz $ export QATDRIVERURL=https://downloadmirror.intel.com/30178/eng/${QATDRIVERVER} $ export QATCONFLOCATION=~/QAT_conf $ export QAT_DOCKERFILE=https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes/main/demo/openssl-qat-engine/Dockerfile $ export QAT_SRC=~/src/QAT $ export GOPATH=~/src/go $ export KATAKERNELLOCATION=~/kata $ export KATAROOTFSLOCATION=~/kata ``` The host could be a bare metal instance or a virtual machine. If using a virtual machine, make sure that KVM nesting is enabled. The following instructions reference an Intel C62X chipset. Some of the instructions must be modified if using a different Intel QAT device. The Intel QAT chipset can be identified by executing the following. ```bash $ for i in 0434 0435 37c8 1f18 1f19; do lspci -d 8086:$i; done ``` These packages are necessary to compile the Kata kernel, Intel QAT driver, and to prepare the rootfs for Kata. also needs to be installed to be able to build the rootfs. To test that everything works a Kubernetes pod is started requesting Intel QAT resources. For the pass through of the virtual functions the kernel boot parameter needs to have `INTEL_IOMMU=on`. ```bash $ sudo apt update $ sudo apt install -y golang-go build-essential python pkg-config zlib1g-dev libudev-dev bison libelf-dev flex libtool automake autotools-dev autoconf bc libpixman-1-dev coreutils libssl-dev $ sudo sed -i 's/GRUBCMDLINELINUXDEFAULT=""/GRUBCMDLINELINUXDEFAULT="intel_iommu=on"/' /etc/default/grub $ sudo update-grub $ sudo reboot ``` This will download the . Make sure to check the website for the latest version. ```bash $ mkdir -p $QAT_SRC $ cd $QAT_SRC $ curl -L $QATDRIVERURL | tar zx ``` Modify the instructions below as necessary if using a different Intel QAT hardware platform. You can learn more about customizing configuration files at the This section starts from a base config file and changes the `SSL` section to `SHIM` to support the OpenSSL engine. There are more tweaks that you can make depending on the use case and how many Intel QAT engines should be run. You can find more information about how to customize in the Note: This section assumes that a Intel QAT `c6xx` platform is used. ```bash $ mkdir -p $QATCONFLOCATION $ cp $QATSRC/quickassist/utilities/adfctl/conffiles/c6xxvfdev0.conf.vm $QATCONFLOCATION/c6xxvf_dev0.conf $ sed -i 's/\[SSL\]/\[SHIM\]/g' $QATCONFLOCATION/c6xxvf_dev0.conf ``` To enable virtual functions, the host OS should have IOMMU groups enabled. In the UEFI Firmware Intel Virtualization Technology for Directed I/O (Intel VT-d) must be enabled. Also, the kernel boot parameter should be `inteliommu=on` or `inteliommu=ifgx_off`. This should have been set from the instructions

Check the output of `/proc/cmdline` to confirm. The following commands assume you installed an Intel QAT card, IOMMU is on, and VT-d is enabled. The vendor and device ID add to the `VFIO-PCI` driver so that each exposed virtual function can be bound to the `VFIO-PCI` driver. Once complete, each virtual function passes into a Kata Containers container using the PCIe device passthrough feature. For Kubernetes, the for Kubernetes handles the binding of the driver, but the VFs still must be enabled. ```bash $ sudo modprobe vfio-pci $ QATPCIBUSPFNUMBERS=$((lspci -d :435 && lspci -d :37c8 && lspci -d :19e2 && lspci -d :6f54) | cut -d ' ' -f 1) $ QATPCIBUSPF1=$(echo $QATPCIBUSPFNUMBERS | cut -d ' ' -f 1) $ echo 16 | sudo tee /sys/bus/pci/devices/0000:$QATPCIBUSPF1/sriov_numvfs $ QATPCIIDVF=$(cat /sys/bus/pci/devices/0000:${QATPCIBUSPF1}/virtfn0/uevent | grep PCIID) $ QATVENDORANDIDVF=$(echo ${QATPCIIDVF/PCIID=} | sed 's/:/ /') $ echo $QATVENDORANDIDVF | sudo tee --append /sys/bus/pci/drivers/vfio-pci/new_id ``` Loop through all the virtual functions and bind to the VFIO driver ```bash $ for f in /sys/bus/pci/devices/0000:$QATPCIBUSPF1/virtfn* do QATPCIBUS_VF=$(basename $(readlink $f)) echo $QATPCIBUS_VF | sudo tee --append /sys/bus/pci/drivers/c6xxvf/unbind echo $QATPCIBUS_VF | sudo tee --append /sys/bus/pci/drivers/vfio-pci/bind done ``` If the following command returns empty, then the virtual functions are not properly enabled. This command checks the enumerated device IDs for just the virtual functions. Using the Intel QAT as an example, the physical device ID is `37c8` and virtual function device ID is `37c9`. The following command checks if VF's are enabled for any of the currently known Intel QAT device ID's. The following `ls` command should show the 16 VF's bound to `VFIO-PCI`. ```bash $ for i in 0442 0443 37c9 19e3; do lspci -d 8086:$i; done ``` Another way to check is to see what PCI devices that `VFIO-PCI` is mapped to. It should match the device ID's of the VF's. ```bash $ ls -la /sys/bus/pci/drivers/vfio-pci ``` This example automatically uses the latest Kata kernel supported by Kata. It follows the instructions from the and uses the latest Kata kernel . There are some patches that must be installed as well, which the `build-kernel.sh` script should automatically apply. If you are using a different kernel version, then you might need to manually apply them. Since the Kata Containers kernel has a minimal set of kernel flags set, you must create a Intel QAT kernel fragment with the necessary `CONFIGCRYPTO*` options set. Update the config to set some of the `CRYPTO` flags to enabled. This might change with different kernel versions. The following instructions were tested with kernel `v5.4.0-64-generic`. ```bash $ mkdir -p $GOPATH $ cd $GOPATH $ go get -v github.com/kata-containers/kata-containers $ cat << EOF > $GOPATH/src/github.com/kata-containers/kata-containers/tools/packaging/kernel/configs/fragments/common/qat.conf CONFIG_PCIEAER=y CONFIG_UIO=y CONFIGCRYPTOHW=y CONFIGCRYPTODEVQATC62XVF=m CONFIGCRYPTOCBC=y CONFIG_MODULES=y CONFIGMODULESIG=y CONFIGCRYPTOAUTHENC=y CONFIGCRYPTODH=y EOF $ $GOPATH/src/github.com/kata-containers/kata-containers/tools/packaging/kernel/build-kernel.sh setup ``` ```bash $ cd $GOPATH $ export LINUX_VER=$(ls -d kata-linux-*) $ sed -i 's/EXTRAVERSION =/EXTRAVERSION = .qat.container/' $LINUX_VER/Makefile $ $GOPATH/src/github.com/kata-containers/kata-containers/tools/packaging/kernel/build-kernel.sh build ``` ```bash $ export KATAKERNELNAME=vmlinux-${LINUXVER}qat $ mkdir -p $KATAKERNELLOCATION $ cp ${GOPATH}/${LINUXVER}/vmlinux ${KATAKERNELLOCATION}/${KATAKERNEL_NAME} ``` These instructions build upon the OS builder instructions located in the . At this point it is recommended that is installed first, and then is use to install Kata. This will make sure that the correct `agent` version is installed into the rootfs in the steps

The following instructions use Ubuntu as the root filesystem with systemd as the init and will add in the `kmod` binary, which is not a standard binary in a Kata rootfs image. The `kmod` binary is necessary to load the Intel QAT kernel modules when the virtual machine rootfs boots. ```bash $ export OSBUILDER=$GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder $ export ROOTFS_DIR=${OSBUILDER}/rootfs-builder/rootfs $ export EXTRA_PKGS='kmod' ``` Make sure that the `kata-agent` version matches the installed `kata-runtime` version. Also make sure the `kata-runtime` install location is in your `PATH` variable. The following `AGENT_VERSION` can be set manually to match the `kata-runtime` version if the following commands don't work. ```bash $ export PATH=$PATH:/opt/kata/bin $ cd $GOPATH $ export AGENT_VERSION=$(kata-runtime version | head -n 1 | grep -o "[0-9.]\+") $ cd ${OSBUILDER}/rootfs-builder $ sudo rm -rf ${ROOTFS_DIR} $ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true SECCOMP=no ./rootfs.sh ubuntu' ``` After the Kata Containers kernel builds with the proper configuration flags, you must build the Intel QAT drivers against that Kata Containers kernel version in a similar way they were previously built for the host OS. You must set the `KERNELSOURCEROOT` variable to the Kata Containers kernel source directory and build the Intel QAT drivers again. The `make` command will install the Intel QAT modules into the Kata rootfs. ```bash $ cd $GOPATH $ export LINUX_VER=$(ls -d kata*) $ export KERNELMAJORVERSION=$(awk '/^VERSION =/{print $NF}' $GOPATH/$LINUX_VER/Makefile) $ export KERNELPATHLEVEL=$(awk '/^PATCHLEVEL =/{print $NF}' $GOPATH/$LINUXVER/Makefile) $ export KERNELSUBLEVEL=$(awk '/^SUBLEVEL =/{print $NF}' $GOPATH/$LINUXVER/Makefile) $ export KERNELEXTRAVERSION=$(awk '/^EXTRAVERSION =/{print $NF}' $GOPATH/$LINUXVER/Makefile) $ export KERNELROOTFSDIR=${KERNELMAJORVERSION}.${KERNELPATHLEVEL}.${KERNELSUBLEVEL}${KERNEL_EXTRAVERSION} $ cd $QAT_SRC $ KERNELSOURCEROOT=$GOPATH/$LINUX_VER ./configure --enable-icp-sriov=guest $ sudo -E make all -j $($(nproc ${CI:+--ignore 1})) $ sudo -E make INSTALLMODPATH=$ROOTFS_DIR qat-driver-install -j $($(nproc ${CI:+--ignore 1})) ``` The `usdm_drv` module also needs to be copied into the rootfs modules path and `depmod` should be run. ```bash $ sudo cp $QATSRC/build/usdmdrv.ko $ROOTFSDIR/lib/modules/${KERNELROOTFS_DIR}/updates/drivers $ sudo depmod -a -b ${ROOTFSDIR} ${KERNELROOTFS_DIR} $ cd ${OSBUILDER}/image-builder $ script -fec 'sudo -E USEDOCKER=true ./imagebuilder.sh ${ROOTFS_DIR}' ``` Note: Ignore any errors on modules.builtin and modules.order when running `depmod`. ```bash $ mkdir -p $KATAROOTFSLOCATION $ cp ${OSBUILDER}/image-builder/kata-containers.img $KATAROOTFSLOCATION ``` The following instructions uses a OpenSSL Dockerfile that builds the Intel QAT engine to allow OpenSSL to offload crypto functions. It is a convenient way to test that VFIO device passthrough for the Intel QAT VFs are working properly with the Kata Containers VM. Use the OpenSSL Intel QAT to build a container image with an optimized OpenSSL engine for Intel QAT. Using `docker build` with the Kata Containers runtime can sometimes have issues. Therefore, make sure that `runc` is the default Docker container runtime. ```bash $ cd $QAT_SRC $ curl -O $QAT_DOCKERFILE $ sudo docker build -t openssl-qat-engine . ``` Note: The Intel QAT driver version in this container might not match the Intel QAT driver compiled and loaded on the host when compiling. The `ctr` tool can be used to interact with the containerd daemon. It may be more convenient to use this tool to verify the kernel and image instead of setting up a Kubernetes cluster. The correct Kata runtimes need to be added to the containerd `config.toml`. Below is a sample snippet that can be added to allow QEMU and Cloud Hypervisor (CLH) to work with `ctr`. ``` [plugins.cri.containerd.runtimes.kata-qemu] runtime_type = "io.containerd.kata-qemu.v2" privilegedwithouthost_devices = true pod_annotations = ["io.katacontainers.*"] [plugins.cri.containerd.runtimes.kata-qemu.options] ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml" [plugins.cri.containerd.runtimes.kata-clh] runtime_type = "io.containerd.kata-clh.v2" privilegedwithouthost_devices = true pod_annotations = ["io.katacontainers.*"] [plugins.cri.containerd.runtimes.kata-clh.options] ConfigPath =

``` In addition, containerd expects the binary to be in `/usr/local/bin` so add this small script so that it redirects to be able to use either QEMU or Cloud Hypervisor with Kata. ```bash $ echo '#!/usr/bin/env bash' | sudo tee /usr/local/bin/containerd-shim-kata-qemu-v2 $ echo 'KATACONFFILE=/opt/kata/share/defaults/kata-containers/configuration-qemu.toml /opt/kata/bin/containerd-shim-kata-v2 $@' | sudo tee -a /usr/local/bin/containerd-shim-kata-qemu-v2 $ sudo chmod +x /usr/local/bin/containerd-shim-kata-qemu-v2 $ echo '#!/usr/bin/env bash' | sudo tee /usr/local/bin/containerd-shim-kata-clh-v2 $ echo 'KATACONFFILE=/opt/kata/share/defaults/kata-containers/configuration-clh.toml /opt/kata/bin/containerd-shim-kata-v2 $@' | sudo tee -a /usr/local/bin/containerd-shim-kata-clh-v2 $ sudo chmod +x /usr/local/bin/containerd-shim-kata-clh-v2 ``` After the OpenSSL image is built and imported into containerd, a Intel QAT virtual function exposed in the step above can be added to the `ctr` command. Make sure to change the `/dev/vfio` number to one that actually exists on the host system. When using the `ctr` tool, the`configuration.toml` for Kata needs to point to the custom Kata kernel and rootfs built above and the Intel QAT modules in the Kata rootfs need to load at boot. The following steps assume that `kata-deploy` was used to install Kata and QEMU is being tested. If using a different hypervisor, different install method for Kata, or a different Intel QAT chipset then the command will need to be modified. Note: The following was tested with . ```bash $ config_file="/opt/kata/share/defaults/kata-containers/configuration-qemu.toml" $ sudo sed -i "/kernel =/c kernel = "\"${KATAROOTFSLOCATION}/${KATAKERNELNAME}\""" $config_file $ sudo sed -i "/image =/c image = "\"${KATAKERNELLOCATION}/kata-containers.img\""" $config_file $ sudo sed -i -e 's/^kernelparams = "\(.*\)"/kernelparams = "\1 modules-load=usdmdrv,qatc62xvf"/g' $config_file $ sudo docker save -o openssl-qat-engine.tar openssl-qat-engine:latest $ sudo ctr images import openssl-qat-engine.tar $ sudo ctr run --runtime io.containerd.run.kata-qemu.v2 --privileged -t --rm --device=/dev/vfio/180 --mount type=bind,src=/dev,dst=/dev,options=rbind:rw --mount type=bind,src=${QATCONFLOCATION}/c6xxvfdev0.conf,dst=/etc/c6xxvfdev0.conf,options=rbind:rw docker.io/library/openssl-qat-engine:latest bash ``` Below are some commands to run in the container image to verify Intel QAT is working ```sh root@67561dc2757a/ # cat /proc/modules qat_c62xvf 16384 - - Live 0xffffffffc00d9000 (OE) usdm_drv 86016 - - Live 0xffffffffc00e8000 (OE) intel_qat 249856 - - Live 0xffffffffc009b000 (OE) root@67561dc2757a/ # adf_ctl restart Restarting all devices. Processing /etc/c6xxvf_dev0.conf root@67561dc2757a/ # adf_ctl status Checking status of all devices. There is 1 QAT acceleration device(s) in the system: qatdev0 - type: c6xxvf, instid: 0, node_id: 0, bsf: 0000:01:01.0, #accel: 1 #engines: 1 state: up root@67561dc2757a/ # openssl engine -c -t qat-hw (qat-hw) Reference implementation of QAT crypto engine v0.6.1 [RSA, DSA, DH, AES-128-CBC-HMAC-SHA1, AES-128-CBC-HMAC-SHA256, AES-256-CBC-HMAC-SHA1, AES-256-CBC-HMAC-SHA256, TLS1-PRF, HKDF, X25519, X448] [ available ] ``` Start a Kubernetes cluster with containerd as the CRI. The host should already be setup with 16 virtual functions of the Intel QAT card bound to `VFIO-PCI`. Verify this by looking in `/dev/vfio` for a listing of devices. You might need to disable Docker before initializing Kubernetes. Be aware that the OpenSSL container image built above will need to be exported from Docker and imported into containerd. If Kata is installed through there will be multiple `configuration.toml` files associated with different hypervisors. Rather than add in the custom Kata kernel, Kata rootfs, and kernel modules to each `configuration.toml` as the default, instead use in the Kubernetes YAML file to tell Kata which kernel and rootfs to use. The easy way to do this is to use `kata-deploy` which will install the Kata binaries to `/opt` and properly configure the `/etc/containerd/config.toml` with annotation support. However, the `configuration.toml` needs to enable support for annotations as well. The following configures both QEMU and Cloud Hypervisor `configuration.toml` files that are currently available with Kata Container versions 2.0 and higher. ```bash $ sudo sed -i 's/enableannotations\s=\s\[\]/enableannotations = [".*"]/' /opt/kata/share/defaults/kata-containers/configuration-qemu.toml $ sudo sed -i 's/enableannotations\s=\s\[\]/enableannotations = [".*"]/' /opt/kata/share/defaults/kata-containers/configuration-clh.toml ``` Export the OpenSSL image from Docker and import into

```bash $ sudo docker save -o openssl-qat-engine.tar openssl-qat-engine:latest $ sudo ctr -n=k8s.io images import openssl-qat-engine.tar ``` The needs to be started so that the virtual functions can be discovered and used by Kubernetes. The following YAML file can be used to start a Kata container with Intel QAT support. If Kata is installed with `kata-deploy`, then the containerd `configuration.toml` should have all of the Kata runtime classes already populated and annotations supported. To use a Intel QAT virtual function, the Intel QAT plugin needs to be started after the VF's are bound to `VFIO-PCI` as described . Edit the following to point to the correct Kata kernel and rootfs location built with Intel QAT support. ```bash $ cat << EOF > kata-openssl-qat.yaml apiVersion: v1 kind: Pod metadata: name: kata-openssl-qat labels: app: kata-openssl-qat annotations: io.katacontainers.config.hypervisor.kernel: "$KATAKERNELLOCATION/$KATAKERNELNAME" io.katacontainers.config.hypervisor.image: "$KATAROOTFSLOCATION/kata-containers.img" io.katacontainers.config.hypervisor.kernelparams: "modules-load=usdmdrv,qat_c62xvf" spec: runtimeClassName: kata-qemu containers: name: kata-openssl-qat image: docker.io/library/openssl-qat-engine:latest imagePullPolicy: IfNotPresent resources: limits: qat.intel.com/generic: 1 cpu: 1 securityContext: capabilities: add: ["IPCLOCK", "SYSADMIN"] volumeMounts: mountPath: /etc/c6xxvf_dev0.conf name: etc-mount mountPath: /dev name: dev-mount volumes: name: dev-mount hostPath: path: /dev name: etc-mount hostPath: path: $QATCONFLOCATION/c6xxvf_dev0.conf EOF ``` Use `kubectl` to start the pod. Verify that Intel QAT card acceleration is working with the Intel QAT engine. ```bash $ kubectl apply -f kata-openssl-qat.yaml ``` ```sh $ kubectl exec -it kata-openssl-qat -- adf_ctl restart Restarting all devices. Processing /etc/c6xxvf_dev0.conf $ kubectl exec -it kata-openssl-qat -- adf_ctl status Checking status of all devices. There is 1 QAT acceleration device(s) in the system: qatdev0 - type: c6xxvf, instid: 0, node_id: 0, bsf: 0000:01:01.0, #accel: 1 #engines: 1 state: up $ kubectl exec -it kata-openssl-qat -- openssl engine -c -t qat-hw (qat-hw) Reference implementation of QAT crypto engine v0.6.1 [RSA, DSA, DH, AES-128-CBC-HMAC-SHA1, AES-128-CBC-HMAC-SHA256, AES-256-CBC-HMAC-SHA1, AES-256-CBC-HMAC-SHA256, TLS1-PRF, HKDF, X25519, X448] [ available ] ``` Check that `/dev/vfio` has VFs enabled. ```sh $ ls /dev/vfio 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 vfio ``` Check that the modules load when inside the Kata Container. ```sh bash-5.0# egrep "qat|usdm_drv" /proc/modules qat_c62xvf 16384 - - Live 0x0000000000000000 (O) usdm_drv 86016 - - Live 0x0000000000000000 (O) intel_qat 184320 - - Live 0x0000000000000000 (O) ``` Verify that at least the first `c6xxvf_dev0.conf` file mounts inside the container image in `/etc`. You will need one configuration file for each VF passed into the container. ```sh bash-5.0# ls /etc c6xxvfdev0.conf c6xxvfdev11.conf c6xxvfdev14.conf c6xxvfdev3.conf c6xxvfdev6.conf c6xxvfdev9.conf resolv.conf c6xxvfdev1.conf c6xxvfdev12.conf c6xxvfdev15.conf c6xxvfdev4.conf c6xxvf_dev7.conf hostname c6xxvfdev10.conf c6xxvfdev13.conf c6xxvfdev2.conf c6xxvfdev5.conf c6xxvf_dev8.conf hosts ``` Check `dmesg` inside the container to see if there are any issues with the Intel QAT driver. If there are issues building the OpenSSL Intel QAT container image, then check to make sure that runc is the default runtime for building container. ```sh $ cat /etc/systemd/system/docker.service.d/50-runtime.conf [Service] Environment="DOCKERDEFAULTRUNTIME=--default-runtime runc" ``` To check the built in firmware counters, the Intel QAT driver has to be compiled and installed to the host and can't rely on the built in host driver. The counters will increase when the accelerator is actively being used. To verify Intel QAT is actively accelerating the containerized application, use the following instructions to check if any of the counters increment. Make sure to change the PCI Device ID to match whats in the system. ```bash $ for i in 0434 0435 37c8 1f18 1f19; do lspci -d 8086:$i; done $ sudo watch cat /sys/kernel/debug/qatc6xx0000\:b1\:00.0/fw_counters $ sudo watch cat /sys/kernel/debug/qatc6xx0000\:b3\:00.0/fw_counters $ sudo watch cat /sys/kernel/debug/qatc6xx0000\:b5\:00.0/fw_counters ```