Chapter 6. Deploying OpenShift sandboxed containers on IBM Z and IBM LinuxONE You can deploy OpenShift sandboxed containers on IBM Z(R) and IBM(R) LinuxONE. OpenShift sandboxed containers deploys peer pods. The peer pod design circumvents the need for nested virtualization. For more information, see peer pod and Peer pods technical deep dive . Important OpenShift sandboxed containers on IBM Z(R) and IBM(R) LinuxONE is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . Cluster requirements You have installed Red Hat OpenShift Container Platform 4.14 or later on the cluster where you are installing the OpenShift sandboxed containers Operator. Your cluster has at least one worker node. 6.1. Peer pod resource requirements You must ensure that your cluster has sufficient resources. Peer pod virtual machines (VMs) require resources in two locations: The worker node. The worker node stores metadata, Kata shim resources ( containerd-shim-kata-v2 ), remote-hypervisor resources ( cloud-api-adaptor ), and the tunnel setup between the worker nodes and the peer pod VM. The libvirt virtual machine instance. This is the actual peer pod VM running in the LPAR (KVM host). The CPU and memory resources used in the Kubernetes worker node are handled by the pod overhead included in the RuntimeClass ( kata-remote ) definition used for creating peer pods. The total number of peer pod VMs running in the cloud is defined as Kubernetes Node extended resources. This limit is per node and is set by the limit attribute in the peerpodConfig custom resource (CR). The peerpodConfig CR, named peerpodconfig-openshift , is created when you create the kataConfig CR and enable peer pods, and is located in the openshift-sandboxed-containers-operator namespace. The following peerpodConfig CR example displays the default spec values: apiVersion: confidentialcontainers.org/v1alpha1 kind: PeerPodConfig metadata: name: peerpodconfig-openshift namespace: openshift-sandboxed-containers-operator spec: cloudSecretName: peer-pods-secret configMapName: peer-pods-cm limit: "10" 1 nodeSelector: node-role.kubernetes.io/kata-oc: "" 1 The default limit is 10 VMs per node. The extended resource is named kata.peerpods.io/vm , and enables the Kubernetes scheduler to handle capacity tracking and accounting. You can edit the limit per node based on the requirements for your environment after you install the OpenShift sandboxed containers Operator. A mutating webhook adds the extended resource kata.peerpods.io/vm to the pod specification. It also removes any resource-specific entries from the pod specification, if present. This enables the Kubernetes scheduler to account for these extended resources, ensuring the peer pod is only scheduled when resources are available. The mutating webhook modifies a Kubernetes pod as follows: The mutating webhook checks the pod for the expected RuntimeClassName value, specified in the TARGET_RUNTIME_CLASS environment variable. If the value in the pod specification does not match the value in the TARGET_RUNTIME_CLASS , the webhook exits without modifying the pod. If the RuntimeClassName values match, the webhook makes the following changes to the pod spec: The webhook removes every resource specification from the resources field of all containers and init containers in the pod. The webhook adds the extended resource ( kata.peerpods.io/vm ) to the spec by modifying the resources field of the first container in the pod. The extended resource kata.peerpods.io/vm is used by the Kubernetes scheduler for accounting purposes. Note The mutating webhook excludes specific system namespaces in OpenShift Container Platform from mutation. If a peer pod is created in those system namespaces, then resource accounting using Kubernetes extended resources does not work unless the pod spec includes the extended resource. As a best practice, define a cluster-wide policy to only allow peer pod creation in specific namespaces. 6.2. Deploying OpenShift sandboxed containers on IBM Z and IBM LinuxONE You can deploy OpenShift sandboxed containers on IBM Z(R) and IBM(R) LinuxONE by using the command line interface (CLI) to perform the following tasks: Install the OpenShift sandboxed containers Operator. Optional: Change the number of virtual machines running on each worker node. Configure the libvirt volume. Optional: Create a custom peer pod VM image. Create the peer pods secret. Create the peer pods config map. Create the peer pod VM image config map. Create the KVM host secret. Create the KataConfig custom resource. Configure the OpenShift sandboxed containers workload objects. 6.2.1. Installing the OpenShift sandboxed containers Operator You can install the OpenShift sandboxed containers Operator by using the CLI. Prerequisites You have installed the OpenShift CLI ( oc ). You have access to the cluster as a user with the cluster-admin role. Procedure Create an osc-namespace.yaml manifest file: apiVersion: v1 kind: Namespace metadata: name: openshift-sandboxed-containers-operator Create the namespace by running the following command: USD oc apply -f osc-namespace.yaml Create an osc-operatorgroup.yaml manifest file: apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: sandboxed-containers-operator-group namespace: openshift-sandboxed-containers-operator spec: targetNamespaces: - openshift-sandboxed-containers-operator Create the operator group by running the following command: USD oc apply -f osc-operatorgroup.yaml Create an osc-subscription.yaml manifest file: apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: sandboxed-containers-operator namespace: openshift-sandboxed-containers-operator spec: channel: stable installPlanApproval: Automatic name: sandboxed-containers-operator source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: sandboxed-containers-operator.v1.8.1 Create the subscription by running the following command: USD oc apply -f osc-subscription.yaml Verify that the Operator is correctly installed by running the following command: USD oc get csv -n openshift-sandboxed-containers-operator This command can take several minutes to complete. Watch the process by running the following command: USD watch oc get csv -n openshift-sandboxed-containers-operator Example output NAME DISPLAY VERSION REPLACES PHASE openshift-sandboxed-containers openshift-sandboxed-containers-operator 1.8.1 1.7.0 Succeeded Additional resources Using Operator Lifecycle Manager on restricted networks . Configuring proxy support in Operator Lifecycle Manager for disconnected environments. 6.2.2. Modifying the number of peer pod VMs per node You can change the limit of peer pod virtual machines (VMs) per node by editing the peerpodConfig custom resource (CR). Procedure Check the current limit by running the following command: USD oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \ -o jsonpath='{.spec.limit}{"\n"}' Modify the limit attribute of the peerpodConfig CR by running the following command: USD oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \ --type merge --patch '{"spec":{"limit":"<value>"}}' 1 1 Replace <value> with the limit you want to define. 6.2.3. Configuring the libvirt volume You must configure the libvirt volume on your KVM host. Peer pods use the libvirt provider of the Cloud API Adaptor to create and manage virtual machines. Prerequisites You have installed the OpenShift sandboxed containers Operator on your OpenShift Container Platform cluster by using the OpenShift Container Platform web console or the command line. You have administrator privileges for your KVM host. You have installed podman on your KVM host. You have installed virt-customize on your KVM host. Procedure Log in to the KVM host. Set the name of the libvirt pool by running the following command: USD export LIBVIRT_POOL=<libvirt_pool> You need the LIBVIRT_POOL value to create the secret for the libvirt provider. Set the name of the libvirt pool by running the following command: USD export LIBVIRT_VOL_NAME=<libvirt_volume> You need the LIBVIRT_VOL_NAME value to create the secret for the libvirt provider. Set the path of the default storage pool location, by running the following command: USD export LIBVIRT_POOL_DIRECTORY=<target_directory> 1 1 To ensure libvirt has read and write access permissions, use a subdirectory of the libvirt storage directory. The default is /var/lib/libvirt/images/ . Create a libvirt pool by running the following command: USD virsh pool-define-as USDLIBVIRT_POOL --type dir --target "USDLIBVIRT_POOL_DIRECTORY" Start the libvirt pool by running the following command: USD virsh pool-start USDLIBVIRT_POOL Create a libvirt volume for the pool by running the following command: USD virsh -c qemu:///system \ vol-create-as --pool USDLIBVIRT_POOL \ --name USDLIBVIRT_VOL_NAME \ --capacity 20G \ --allocation 2G \ --prealloc-metadata \ --format qcow2 6.2.4. Creating a custom peer pod VM image You can create a custom peer pod virtual machine (VM) image instead of using the default Operator-built image. You build an Open Container Initiative (OCI) container with the peer pod QCOW2 image. Later, you add the container registry URL and the image path to the peer pod VM image config map. Procedure Create a Dockerfile.podvm-oci file: FROM scratch ARG PODVM_IMAGE_SRC ENV PODVM_IMAGE_PATH="/image/podvm.qcow2" COPY USDPODVM_IMAGE_SRC USDPODVM_IMAGE_PATH Build a container with the pod VM QCOW2 image by running the following command: USD docker build -t podvm-libvirt \ --build-arg PODVM_IMAGE_SRC=<podvm_image_source> \ 1 --build-arg PODVM_IMAGE_PATH=<podvm_image_path> \ 2 -f Dockerfile.podvm-oci . 1 Specify the QCOW2 image source on the host. 2 Optional: Specify the path of the QCOW2 image if you do not use the default, /image/podvm.qcow2 . 6.2.5. Creating the peer pods secret You must create the peer pods secret for OpenShift sandboxed containers. The secret stores credentials for creating the pod virtual machine (VM) image and peer pod instances. By default, the OpenShift sandboxed containers Operator creates the secret based on the credentials used to create the cluster. However, you can manually create a secret that uses different credentials. Prerequisites LIBVIRT_POOL . Use the value you set when you configured libvirt on the KVM host. LIBVIRT_VOL_NAME . Use the value you set when you configured libvirt on the KVM host. LIBVIRT_URI . This value is the default gateway IP address of the libvirt network. Check your libvirt network setup to obtain this value. Note If libvirt uses the default bridge virtual network, you can obtain the LIBVIRT_URI by running the following commands: USD virtint=USD(bridge_line=USD(virsh net-info default | grep Bridge); echo "USD{bridge_line//Bridge:/}" | tr -d [:blank:]) USD LIBVIRT_URI=USD( ip -4 addr show USDvirtint | grep -oP '(?<=inet\s)\d+(\.\d+){3}') USD LIBVIRT_GATEWAY_URI="qemu+ssh://root@USD{LIBVIRT_URI}/system?no_verify=1" REDHAT_OFFLINE_TOKEN . You have generated this token to download the RHEL image at Red Hat API Tokens . Procedure Create a peer-pods-secret.yaml manifest file according to the following example: apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: CLOUD_PROVIDER: "libvirt" LIBVIRT_URI: "<libvirt_gateway_uri>" 1 LIBVIRT_POOL: "<libvirt_pool>" 2 LIBVIRT_VOL_NAME: "<libvirt_volume>" 3 REDHAT_OFFLINE_TOKEN: "<rh_offline_token>" 4 1 Specify the libvirt URI. 2 Specify the libvirt pool. 3 Specify the libvirt volume name. 4 Specify the Red Hat offline token, which is required for the Operator-built image. Create the secret by running the following command: USD oc apply -f peer-pods-secret.yaml 6.2.6. Creating the peer pods config map You must create the peer pods config map for OpenShift sandboxed containers. Procedure Create a peer-pods-cm.yaml manifest file according to the following example: apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "libvirt" DISABLECVM: "true" Create the config map by running the following command: USD oc apply -f peer-pods-cm.yaml 6.2.7. Creating the peer pod VM image config map You must create the config map for the peer pod VM image. Procedure Create a libvirt-podvm-image-cm.yaml manifest according to the following example: apiVersion: v1 kind: ConfigMap metadata: name: libvirt-podvm-image-cm namespace: openshift-sandboxed-containers-operator data: PODVM_DISTRO: "rhel" CAA_SRC: "https://github.com/confidential-containers/cloud-api-adaptor" CAA_REF: "<cloud_api_adaptor_version>" 1 DOWNLOAD_SOURCES: "no" CONFIDENTIAL_COMPUTE_ENABLED: "yes" UPDATE_PEERPODS_CM: "yes" ORG_ID: "<rhel_organization_id>" ACTIVATION_KEY: "<rhel_activation_key>" 2 IMAGE_NAME: "<podvm_libvirt_image>" PODVM_IMAGE_URI: "oci::<image_repo_url>:<image_tag>::<image_path>" 3 SE_BOOT: "true" 4 BASE_OS_VERSION: "<rhel_image_os_version>" 5 1 Specify the latest version of the Cloud API Adaptor source. 2 Specify your RHEL activation key. 3 Optional: Specify the following values if you created a container image: image_repo_url : Container registry URL. image_tag : Image tag. image_path : Image path. Default: /image/podvm.qcow2 . 4 SE_BOOT: "true" enables IBM Secure Execution for an Operator-built image. Set to false if you created a container image. 5 Specify the RHEL image operating system version. IBM Z(R) Secure Execution supports RHEL 9.5 and later versions. Create the config map by running the following command: USD oc apply -f libvirt-podvm-image-cm.yaml The libvirt pod VM image config map is created for your libvirt provider. 6.2.8. Creating the KVM host secret You must create the secret for your KVM host. Procedure Generate an SSH key pair by running the following command: USD ssh-keygen -f ./id_rsa -N "" Copy the public SSH key to your KVM host: USD ssh-copy-id -i ./id_rsa.pub <KVM_HOST_IP> Create the Secret object by running the following command: USD oc create secret generic ssh-key-secret \ -n openshift-sandboxed-containers-operator \ --from-file=id_rsa.pub=./id_rsa.pub \ --from-file=id_rsa=./id_rsa Delete the SSH keys you created: USD shred --remove id_rsa.pub id_rsa 6.2.9. Creating the KataConfig custom resource You must create the KataConfig custom resource (CR) to install kata-remote as a runtime class on your worker nodes. Creating the KataConfig CR triggers the OpenShift sandboxed containers Operator to do the following: Create a RuntimeClass CR named kata-remote with a default configuration. This enables users to configure workloads to use kata-remote as the runtime by referencing the CR in the RuntimeClassName field. This CR also specifies the resource overhead for the runtime. OpenShift sandboxed containers installs kata-remote as a secondary, optional runtime on the cluster and not as the primary runtime. Important Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows: A larger OpenShift Container Platform deployment with a greater number of worker nodes. Activation of the BIOS and Diagnostics utility. Deployment on a hard disk drive rather than an SSD. Deployment on physical nodes such as bare metal, rather than on virtual nodes. A slow CPU and network. Prerequisites You have access to the cluster as a user with the cluster-admin role. Procedure Create an example-kataconfig.yaml manifest file according to the following example: apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: example-kataconfig spec: enablePeerPods: true logLevel: info # kataConfigPoolSelector: # matchLabels: # <label_key>: '<label_value>' 1 1 Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, osc: 'true' . Create the KataConfig CR by running the following command: USD oc apply -f example-kataconfig.yaml The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes. Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation. Monitor the installation progress by running the following command: USD watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p" When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster. Verify that you have built the peer pod image and uploaded it to the libvirt volume by running the following command: USD oc describe configmap peer-pods-cm -n openshift-sandboxed-containers-operator Example output Name: peer-pods-cm Namespace: openshift-sandboxed-containers-operator Labels: <none> Annotations: <none> Data ==== CLOUD_PROVIDER: libvirt BinaryData ==== Events: <none> Monitor the kata-oc machine config pool progress to ensure that it is in the UPDATED state, when UPDATEDMACHINECOUNT equals MACHINECOUNT , by running the following command: USD watch oc get mcp/kata-oc Verify the daemon set by running the following command: USD oc get -n openshift-sandboxed-containers-operator ds/peerpodconfig-ctrl-caa-daemon Verify the runtime classes by running the following command: USD oc get runtimeclass Example output NAME HANDLER AGE kata kata 152m kata-remote kata-remote 152m 6.2.10. Configuring workload objects You must configure OpenShift sandboxed containers workload objects by setting kata-remote as the runtime class for the following pod-templated objects: Pod objects ReplicaSet objects ReplicationController objects StatefulSet objects Deployment objects DeploymentConfig objects Important Do not deploy workloads in an Operator namespace. Create a dedicated namespace for these resources. Prerequisites You have created the KataConfig custom resource (CR). Procedure Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example: apiVersion: v1 kind: <object> # ... spec: runtimeClassName: kata-remote # ... OpenShift Container Platform creates the workload object and begins scheduling it. Verification Inspect the spec.runtimeClassName field of a pod-templated object. If the value is kata-remote , then the workload is running on OpenShift sandboxed containers, using peer pods.

Commands:
apiVersion: confidentialcontainers.org/v1alpha1 kind: PeerPodConfig metadata: name: peerpodconfig-openshift namespace: openshift-sandboxed-containers-operator spec: cloudSecretName: peer-pods-secret configMapName: peer-pods-cm limit: "10" 1 nodeSelector: node-role.kubernetes.io/kata-oc: ""
apiVersion: v1 kind: Namespace metadata: name: openshift-sandboxed-containers-operator
oc apply -f osc-namespace.yaml
apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: sandboxed-containers-operator-group namespace: openshift-sandboxed-containers-operator spec: targetNamespaces: - openshift-sandboxed-containers-operator
oc apply -f osc-operatorgroup.yaml
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: sandboxed-containers-operator namespace: openshift-sandboxed-containers-operator spec: channel: stable installPlanApproval: Automatic name: sandboxed-containers-operator source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: sandboxed-containers-operator.v1.8.1
oc apply -f osc-subscription.yaml
oc get csv -n openshift-sandboxed-containers-operator
watch oc get csv -n openshift-sandboxed-containers-operator
NAME DISPLAY VERSION REPLACES PHASE openshift-sandboxed-containers openshift-sandboxed-containers-operator 1.8.1 1.7.0 Succeeded
oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator -o jsonpath='{.spec.limit}{"\n"}'
oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator --type merge --patch '{"spec":{"limit":"<value>"}}' 1
export LIBVIRT_POOL=<libvirt_pool>
export LIBVIRT_VOL_NAME=<libvirt_volume>
export LIBVIRT_POOL_DIRECTORY=<target_directory> 1
virsh pool-define-as USDLIBVIRT_POOL --type dir --target "USDLIBVIRT_POOL_DIRECTORY"
virsh pool-start USDLIBVIRT_POOL
virsh -c qemu:///system vol-create-as --pool USDLIBVIRT_POOL --name USDLIBVIRT_VOL_NAME --capacity 20G --allocation 2G --prealloc-metadata --format qcow2
FROM scratch ARG PODVM_IMAGE_SRC ENV PODVM_IMAGE_PATH="/image/podvm.qcow2" COPY USDPODVM_IMAGE_SRC USDPODVM_IMAGE_PATH
docker build -t podvm-libvirt --build-arg PODVM_IMAGE_SRC=<podvm_image_source> \ 1 --build-arg PODVM_IMAGE_PATH=<podvm_image_path> \ 2 -f Dockerfile.podvm-oci .
virtint=USD(bridge_line=USD(virsh net-info default | grep Bridge); echo "USD{bridge_line//Bridge:/}" | tr -d [:blank:]) LIBVIRT_URI=USD( ip -4 addr show USDvirtint | grep -oP '(?<=inet\s)\d+(\.\d+){3}') LIBVIRT_GATEWAY_URI="qemu+ssh://root@USD{LIBVIRT_URI}/system?no_verify=1"
apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: CLOUD_PROVIDER: "libvirt" LIBVIRT_URI: "<libvirt_gateway_uri>" 1 LIBVIRT_POOL: "<libvirt_pool>" 2 LIBVIRT_VOL_NAME: "<libvirt_volume>" 3 REDHAT_OFFLINE_TOKEN: "<rh_offline_token>" 4
oc apply -f peer-pods-secret.yaml
apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "libvirt" DISABLECVM: "true"
oc apply -f peer-pods-cm.yaml
apiVersion: v1 kind: ConfigMap metadata: name: libvirt-podvm-image-cm namespace: openshift-sandboxed-containers-operator data: PODVM_DISTRO: "rhel" CAA_SRC: "https://github.com/confidential-containers/cloud-api-adaptor" CAA_REF: "<cloud_api_adaptor_version>" 1 DOWNLOAD_SOURCES: "no" CONFIDENTIAL_COMPUTE_ENABLED: "yes" UPDATE_PEERPODS_CM: "yes" ORG_ID: "<rhel_organization_id>" ACTIVATION_KEY: "<rhel_activation_key>" 2 IMAGE_NAME: "<podvm_libvirt_image>" PODVM_IMAGE_URI: "oci::<image_repo_url>:<image_tag>::<image_path>" 3 SE_BOOT: "true" 4 BASE_OS_VERSION: "<rhel_image_os_version>" 5
oc apply -f libvirt-podvm-image-cm.yaml
ssh-keygen -f ./id_rsa -N ""
ssh-copy-id -i ./id_rsa.pub <KVM_HOST_IP>
oc create secret generic ssh-key-secret -n openshift-sandboxed-containers-operator --from-file=id_rsa.pub=./id_rsa.pub --from-file=id_rsa=./id_rsa
shred --remove id_rsa.pub id_rsa
apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: example-kataconfig spec: enablePeerPods: true logLevel: info kataConfigPoolSelector: matchLabels: <label_key>: '<label_value>' 1
oc apply -f example-kataconfig.yaml
watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
oc describe configmap peer-pods-cm -n openshift-sandboxed-containers-operator
Name: peer-pods-cm Namespace: openshift-sandboxed-containers-operator Labels: <none> Annotations: <none> Data ==== CLOUD_PROVIDER: libvirt BinaryData ==== Events: <none>
watch oc get mcp/kata-oc
oc get -n openshift-sandboxed-containers-operator ds/peerpodconfig-ctrl-caa-daemon
oc get runtimeclass
NAME HANDLER AGE kata kata 152m kata-remote kata-remote 152m
apiVersion: v1 kind: <object> spec: runtimeClassName: kata-remote