Project: Kata Containers (Runtime / Container Runtime)
File: how-to-pull-images-in-guest-with-kata.md

Kata Containers 3.3.0 introduces the guest image management feature, which enables the guest VM to directly pull images using `nydus snapshotter`. This feature is designed to protect the integrity of container images and guard against any tampering by the host, which is used for confidential containers. Please refer to for details. The k8s cluster with Kata 3.3.0+ is ready to use. `yq` is installed in the host and it's directory is included in the `PATH` environment variable. (optional, for DaemonSet only) To pull images in the guest, we need to do the following steps: Delete images used for pulling in the guest (optional, for containerd only) Install `nydus snapshotter`: Install `nydus snapshotter` by k8s DaemonSet (recommended) Install `nydus snapshotter` manually Though the `CRI Runtime Specific Snapshotter` is still an in containerd, which containerd is not supported to manage the same image in different `snapshotters`(The default `snapshotter` in containerd is `overlayfs`). To avoid errors caused by this, it is recommended to delete images (including the pause image) in containerd that needs to be pulled in guest later before configuring `nydus snapshotter` in containerd. To use DaemonSet to install `nydus snapshotter`, we need to ensure that `yq` exists in the host. Download `nydus snapshotter` repo ```bash $ nydussnapshotterinstall_dir="/tmp/nydus-snapshotter" $ nydussnapshotterurl=https://github.com/containerd/nydus-snapshotter $ nydussnapshotterversion="v0.13.11" $ git clone -b "${nydussnapshotterversion}" "${nydussnapshotterurl}" "${nydussnapshotterinstall_dir}" ``` Configure DaemonSet file ```bash $ pushd "$nydussnapshotterinstall_dir" $ yq write -i \ misc/snapshotter/base/nydus-snapshotter.yaml \ 'data.FS_DRIVER' \ "proxy" --style=double $ yq write -i \ misc/snapshotter/base/nydus-snapshotter.yaml \ 'data.ENABLECONFIGFROM_VOLUME' \ "false" --style=double $ yq write -i \ misc/snapshotter/base/nydus-snapshotter.yaml \ 'data.ENABLESYSTEMDSERVICE' \ "true" --style=double $ yq write -i \ misc/snapshotter/base/nydus-snapshotter.yaml \ 'data.ENABLERUNTIMESPECIFIC_SNAPSHOTTER' \ "true" --style=double ``` Install `nydus snapshotter` as a DaemonSet ```bash $ kubectl create -f "misc/snapshotter/nydus-snapshotter-rbac.yaml" $ kubectl apply -f "misc/snapshotter/base/nydus-snapshotter.yaml" ``` Wait 5 minutes until the DaemonSet is running ```bash $ kubectl rollout status DaemonSet nydus-snapshotter -n nydus-system --timeout 5m ``` Verify whether `nydus snapshotter` is running as a DaemonSet ```bash $ pods_name=$(kubectl get pods --selector=app=nydus-snapshotter -n nydus-system -o=jsonpath='{.items[*].metadata.name}') $ kubectl logs "${pods_name}" -n nydus-system deploying snapshotter install nydus snapshotter artifacts configuring snapshotter Not found nydus proxy plugin! running snapshotter as systemd service Created symlink /etc/systemd/system/multi-user.target.wants/nydus-snapshotter.service

``` Download `nydus snapshotter` binary from release ```bash $ ARCH=$(uname -m) $ golang_arch=$(case "$ARCH" in aarch64) echo "arm64" ;; ppc64le) echo "ppc64le" ;; x86_64) echo "amd64" ;; s390x) echo "s390x" ;; esac) $ releasetarball="nydus-snapshotter-${nydussnapshotterversion}-linux-${golangarch}.tar.gz" $ curl -OL ${nydussnapshotterurl}/releases/download/${nydussnapshotterversion}/${release_tarball} $ sudo tar -xfz ${release_tarball} -C /usr/local/bin --strip-components=1 ``` Download `nydus snapshotter` configuration file for pulling images in the guest ```bash $ curl -OL https://github.com/containerd/nydus-snapshotter/blob/main/misc/snapshotter/config-proxy.toml $ sudo install -D -m 644 config-proxy.toml /etc/nydus/config-proxy.toml ``` Run `nydus snapshotter` as a standalone process ```bash $ /usr/local/bin/containerd-nydus-grpc --config /etc/nydus/config-proxy.toml --log-to-stdout level=info msg="Start nydus-snapshotter. Version: v0.13.11-308-g106a6cb, PID: 1100169, FsDriver: proxy, DaemonMode: none" level=info msg="Run daemons monitor..." ``` Configure containerd for `nydus snapshotter` Configure `nydus snapshotter` to enable `CRI Runtime Specific Snapshotter` in containerd. This ensures run kata containers with `nydus snapshotter`. Below, the steps are illustrated using `kata-qemu` as an example. ```toml [plugins."io.containerd.grpc.v1.cri".containerd] disablesnapshotannotations = false discardunpackedlayers = false [proxy_plugins.nydus] type = "snapshot" address = "/run/containerd-nydus/containerd-nydus-grpc.sock" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu] snapshotter = "nydus" ``` Notes: The `CRI Runtime Specific Snapshotter` feature only works for containerd v1.7.0 and above. So for Containerd v1.7.0 below, in addition to the above settings, we need to set the global `snapshotter` to `nydus` in containerd config. For example: ```toml [plugins."io.containerd.grpc.v1.cri".containerd] snapshotter = "nydus" ``` Restart containerd service ```bash $ sudo systemctl restart containerd ``` To verify pulling images in a guest VM, please refer to the following commands: Run a kata container ```bash $ cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: busybox annotations: io.containerd.cri.runtime-handler: kata-qemu spec: runtimeClassName: kata-qemu containers: name: busybox image: quay.io/prometheus/busybox:latest imagePullPolicy: Always EOF pod/busybox created $ kubectl get pods NAME READY STATUS RESTARTS AGE busybox 1/1 Running 0 10s ``` Notes: The `CRI Runtime Specific Snapshotter` is still an experimental feature. To pull images in the guest under the specific kata runtime (such as `kata-qemu`), we need to add the following annotation in metadata to each pod yaml: `io.containerd.cri.runtime-handler: kata-qemu`. By adding the annotation, we can ensure that the feature works as expected. Verify that the pod's images have been successfully downloaded in the guest. If images intended for deployment are deleted prior to deploying with `nydus snapshotter`, the root filesystems required for the pod's images (including the pause image and the container image) should not be present on the host. ```bash $ sandbox_id=$(ps -ef| grep containerd-shim-kata-v2| grep -oP '(?<=-id\s)[a-f0-9]+'| tail -1) $ rootfscount=$(find /run/kata-containers/shared/sandboxes/$sandboxid -name rootfs -type d| grep -o "rootfs" | wc -l) $ echo $rootfs_count 0 ```