Chapter 3. Deploying OpenShift sandboxed containers workloads You can install the OpenShift sandboxed containers Operator using either the web console or OpenShift CLI ( oc ). Before installing the OpenShift sandboxed containers Operator, you must prepare your OpenShift Container Platform cluster. 3.1. Prerequisites Before you install OpenShift sandboxed containers, ensure that your OpenShift Container Platform cluster meets the following requirements: Your cluster must be installed on bare-metal infrastructure, on premise with Red Hat Enterprise Linux CoreOS (RHCOS) workers. Important OpenShift sandboxed containers only supports RHCOS worker nodes. RHEL nodes are not supported. Nested virtualization is not supported. 3.1.1. Resource requirements for OpenShift sandboxed containers OpenShift sandboxed containers lets users run workloads on their OpenShift Container Platform clusters inside a sandboxed runtime (Kata). Each pod is represented by a virtual machine (VM). Each VM runs in a QEMU process and hosts a kata-agent process that acts as a supervisor for managing container workloads, and the processes running in those containers. Two additional processes add more overhead: containerd-shim-kata-v2 is used to communicate with the pod. virtiofsd handles host file system access on behalf of the guest. Each VM is configured with a default amount of memory. Additional memory is hot-plugged into the VM for containers that explicitly request memory. A container running without a memory resource consumes free memory until the total memory used by the VM reaches the default allocation. The guest and its I/O buffers also consume memory. If a container is given a specific amount of memory, then that memory is hot-plugged into the VM before the container starts. When a memory limit is specified, the workload is terminated if it consumes more memory than the limit. If no memory limit is specified, the kernel running on the VM might run out of memory. If the kernel runs out of memory, it might terminate other processes on the VM. Default memory sizes The following table lists some the default values for resource allocation. Resource Value Memory allocated by default to a virtual machine 2Gi Guest Linux kernel memory usage at boot ~110Mi Memory used by the QEMU process (excluding VM memory) ~30Mi Memory used by the virtiofsd process (excluding VM I/O buffers) ~10Mi Memory used by the containerd-shim-kata-v2 process ~20Mi File buffer cache data after running dnf install on Fedora ~300Mi* [1] File buffers appear and are accounted for in multiple locations: In the guest where it appears as file buffer cache. In the virtiofsd daemon that maps allowed user-space file I/O operations. In the QEMU process as guest memory. Note Total memory usage is properly accounted for by the memory utilization metrics, which only count that memory once. Pod overhead describes the amount of system resources that a pod on a node uses. You can get the current pod overhead for the Kata runtime by using oc describe runtimeclass kata as shown below. Example USD oc describe runtimeclass kata Example output kind: RuntimeClass apiVersion: node.k8s.io/v1 metadata: name: kata overhead: podFixed: memory: "500Mi" cpu: "500m" You can change the pod overhead by changing the spec.overhead field for a RuntimeClass . For example, if the configuration that you run for your containers consumes more than 350Mi of memory for the QEMU process and guest kernel data, you can alter the RuntimeClass overhead to suit your needs. Note The specified default overhead values are supported by Red Hat. Changing default overhead values is not supported and can result in technical issues. When performing any kind of file system I/O in the guest, file buffers are allocated in the guest kernel. The file buffers are also mapped in the QEMU process on the host, as well as in the virtiofsd process. For example, if you use 300Mi of file buffer cache in the guest, both QEMU and virtiofsd appear to use 300Mi additional memory. However, the same memory is being used in all three cases. In other words, the total memory usage is only 300Mi, mapped in three different places. This is correctly accounted for when reporting the memory utilization metrics. Additional resources Installing a user-provisioned cluster on bare metal 3.2. Deploying OpenShift sandboxed containers workloads using the web console You can deploy OpenShift sandboxed containers workloads from the web console. First, you must install the OpenShift sandboxed containers Operator, then create the KataConfig custom resource (CR). Once you are ready to deploy a workload in a sandboxed container, you must manually add kata as the runtimeClassName to the workload YAML file. 3.2.1. Installing the OpenShift sandboxed containers Operator using the web console You can install the OpenShift sandboxed containers Operator from the OpenShift Container Platform web console. Prerequisites You have OpenShift Container Platform 4.9 installed. You have access to the cluster as a user with the cluster-admin role. Procedure From the Administrator perspective in the web console, navigate to Operators OperatorHub . In the Filter by keyword field, type OpenShift sandboxed containers . Select the OpenShift sandboxed containers tile. Read the information about the Operator and click Install . On the Install Operator page: Select preview-1.1 from the list of available Update Channel options. Verify that Operator recommended Namespace is selected for Installed Namespace . This installs the Operator in the mandatory openshift-sandboxed-containers-operator namespace. If this namespace does not yet exist, it is automatically created. Note Attempting to install the OpenShift sandboxed containers Operator in a namespace other than openshift-sandboxed-containers-operator causes the installation to fail. Verify that Automatic is selected for Approval Strategy . Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available. Click Install . The OpenShift sandboxed containers Operator is now installed on your cluster. Verification From the Administrator perspective in the web console, navigate to Operators Installed Operators . Verify that the OpenShift sandboxed containers Operator is listed in the in operators list. 3.2.2. Creating the KataConfig custom resource in the web console You must create one KataConfig custom resource (CR) to enable installing kata as a RuntimeClass on your cluster nodes. Prerequisites You have installed OpenShift Container Platform 4.9 on your cluster. You have access to the cluster as a user with the cluster-admin role. You have installed the OpenShift sandboxed containers Operator. Note Kata is installed on all worker nodes by default. If you want to install kata as a RuntimeClass only on specific nodes, you can add labels to those nodes, then define the label in the KataConfig CR when you create it. Procedure From the Administrator perspective in the web console, navigate to Operators Installed Operators . Select the OpenShift sandboxed containers Operator from the list of operators. In the KataConfig tab, click Create KataConfig . In the Create KataConfig page, select to configure the KataConfig CR via YAML view . Copy and paste the following manifest into the YAML view : apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig If you want to install kata as a RuntimeClass only on selected nodes, include the label in the manifest: apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig spec: kataConfigPoolSelector: matchLabels: <label_key>: '<label_value>' 1 1 Labels in kataConfigPoolSelector only support single values; nodeSelector syntax is not supported. Click Create . The new KataConfig CR is created and begins to install kata as a RuntimeClass on the worker nodes. Important OpenShift sandboxed containers installs Kata only as a secondary, optional runtime on the cluster and not as the primary runtime. Verification In the KataConfig tab, select the new KataConfig CR. In the KataConfig page, select the YAML tab. Monitor the installationStatus field in the status. A message appears each time there is an update. Click Reload to view the updated KataConfig CR. Once the value of Completed nodes equals the number of worker or labeled nodes, the installation is complete. The status also contains a list of nodes where the installation is completed. 3.2.3. Deploying a workload in a sandboxed container using the web console OpenShift sandboxed containers installs Kata as a secondary, optional runtime on your cluster, and not as the primary runtime. To deploy a pod-templated workload in a sandboxed container, you must manually add kata as the runtimeClassName to the workload YAML file. Prerequisites You have installed OpenShift Container Platform 4.9 on your cluster. You have access to the cluster as a user with the cluster-admin role. You have installed the OpenShift sandboxed containers Operator. You have created a KataConfig custom resource (CR). Procedure From the Administrator perspective in the web console, expand Workloads and select the type of workload you want to create. In the workload page, click to create the workload. In the YAML file for the workload, in the spec field where the container is listed, add runtimeClassName: kata . Example for Pod apiVersion: v1 kind: Pod metadata: name: example labels: app: httpd namespace: openshift-sandboxed-containers-operator spec: runtimeClassName: kata containers: - name: httpd image: 'image-registry.openshift-image-registry.svc:5000/openshift/httpd:latest' ports: - containerPort: 8080 Example for Deployment apiVersion: apps/v1 kind: Deployment metadata: name: example namespace: openshift-sandboxed-containers-operator spec: selector: matchLabels: app: httpd replicas: 3 template: metadata: labels: app: httpd spec: runtimeClassName: kata containers: - name: httpd image: >- image-registry.openshift-image-registry.svc:5000/openshift/httpd:latest ports: - containerPort: 8080 Click Save . OpenShift Container Platform creates the workload and begins scheduling it. 3.3. Deploying OpenShift sandboxed containers workloads using the CLI You can deploy OpenShift sandboxed containers workloads using the CLI. First, you must install the OpenShift sandboxed containers Operator, then create the KataConfig custom resource. Once you are ready to deploy a workload in a sandboxed container, you must add kata as the runtimeClassName to the workload YAML file. 3.3.1. Installing the OpenShift sandboxed containers Operator using the CLI You can install the OpenShift sandboxed containers Operator using the OpenShift Container Platform CLI. Prerequisites You have OpenShift Container Platform 4.9 installed on your cluster. You have installed the OpenShift CLI ( oc ). You have access to the cluster as a user with the cluster-admin role. You have subscribed to the OpenShift sandboxed containers catalog. Note Subscribing to the OpenShift sandboxed containers catalog provides openshift-sandboxed-containers-operator namespace access to the OpenShift sandboxed containers Operator. Procedure Create the Namespace object for the OpenShift sandboxed containers Operator. Create a Namespace object YAML file that contains the following manifest: apiVersion: v1 kind: Namespace metadata: name: openshift-sandboxed-containers-operator Create the Namespace object: USD oc create -f Namespace.yaml Create the OperatorGroup object for the OpenShift sandboxed containers Operator. Create an OperatorGroup object YAML file that contains the following manifest: apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-sandboxed-containers-operator namespace: openshift-sandboxed-containers-operator spec: targetNamespaces: - openshift-sandboxed-containers-operator Create the OperatorGroup object: USD oc create -f OperatorGroup.yaml Create the Subscription object to subscribe the Namespace to the OpenShift sandboxed containers Operator. Create a Subscription object YAML file that contains the following manifest: apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: openshift-sandboxed-containers-operator namespace: openshift-sandboxed-containers-operator spec: channel: "preview-1.1" installPlanApproval: Automatic name: sandboxed-containers-operator source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: sandboxed-containers-operator.v1.1.0 Create the Subscription object: USD oc create -f Subscription.yaml The OpenShift sandboxed containers Operator is now installed on your cluster. Note All the object file names listed above are suggestions. You can create the object YAML files using other names. Verification Ensure that the Operator is correctly installed: USD oc get csv -n openshift-sandboxed-containers-operator Example output NAME DISPLAY VERSION REPLACES PHASE openshift-sandboxed-containers openshift-sandboxed-containers-operator 1.1.0 1.0.2 Succeeded Additional resources Installing from OperatorHub using the CLI 3.3.2. Creating the KataConfig custom resource using the CLI You must create one KataConfig custom resource (CR) to install kata as a RuntimeClass on your nodes. Creating the KataConfig CR triggers the OpenShift sandboxed containers Operator to do the following: Install the needed RHCOS extensions, such as QEMU and kata-containers , on your RHCOS node. Ensure that the CRI-O runtime is configured with the correct kata runtime handlers. Create a RuntimeClass CR named kata with a default configuration. This enables users to configure workloads to use kata as the runtime by referencing the CR in the RuntimeClassName field. This CR also specifies the resource overhead for the runtime. Note Kata is installed on all worker nodes by default. If you want to install kata as a RuntimeClass only on specific nodes, you can add labels to those nodes, then define the label in the KataConfig CR when you create it. Prerequisites You have installed OpenShift Container Platform 4.9 on your cluster. You have installed the OpenShift CLI ( oc ). You have access to the cluster as a user with the cluster-admin role. You have installed the OpenShift sandboxed containers Operator. Procedure Create a YAML file with the following manifest: apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig (Optional) If you want to install kata as a RuntimeClass only on selected nodes, create a YAML file that includes the label in the manifest: apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig spec: kataConfigPoolSelector: matchLabels: <label_key>: '<label_value>' 1 1 Labels in kataConfigPoolSelector only support single values; nodeSelector syntax is not supported. Create the KataConfig resource: USD oc create -f <file name>.yaml The new KataConfig CR is created and begins to install kata as a RuntimeClass on the worker nodes. Important OpenShift sandboxed containers installs Kata only as a secondary, optional runtime on the cluster and not as the primary runtime. Verification Monitor the installation progress: USD watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p" Once the value of Is In Progress appears as false , the installation is complete. Additional resources Understanding how to update labels on nodes 3.3.3. Deploying a workload in a sandboxed container using the CLI OpenShift sandboxed containers installs Kata as a secondary, optional runtime on your cluster, and not as the primary runtime. To deploy a pod-templated workload in a sandboxed container, you must add kata as the runtimeClassName to the workload YAML file. Prerequisites You have installed OpenShift Container Platform 4.9 on your cluster. You have installed the OpenShift CLI ( oc ). You have access to the cluster as a user with the cluster-admin role. You have installed the OpenShift sandboxed containers Operator. You have created a KataConfig custom resource (CR). Procedure Add runtimeClassName: kata to any pod-templated object: Pod objects ReplicaSet objects ReplicationController objects StatefulSet objects Deployment objects DeploymentConfig objects Example for Pod objects apiVersion: v1 kind: Pod metadata: name: mypod spec: runtimeClassName: kata Example for Deployment objects apiVersion: apps/v1 kind: Deployment metadata: name: mypod labels: app: mypod spec: replicas: 3 selector: matchLabels: app: mypod template: metadata: labels: app: mypod spec: runtimeClassName: kata containers: - name: mypod image: myImage OpenShift Container Platform creates the workload and begins scheduling it. Verification Inspect the runtimeClassName field on a pod-templated object. If the runtimeClassName is kata , then the workload is running on a OpenShift sandboxed containers. 3.4. Additional resources The OpenShift sandboxed containers Operator is supported in a restricted network environment. For more information, Using Operator Lifecycle Manager on restricted networks . When using a disconnected cluster on a restricted network, you must configure proxy support in Operator Lifecycle Manager to access the OperatorHub. Using a proxy allows the cluster to fetch the OpenShift sandboxed containers Operator.

Commands:
oc describe runtimeclass kata
kind: RuntimeClass apiVersion: node.k8s.io/v1 metadata: name: kata overhead: podFixed: memory: "500Mi" cpu: "500m"
apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig
apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig spec: kataConfigPoolSelector: matchLabels: <label_key>: '<label_value>' 1
apiVersion: v1 kind: Pod metadata: name: example labels: app: httpd namespace: openshift-sandboxed-containers-operator spec: runtimeClassName: kata containers: - name: httpd image: 'image-registry.openshift-image-registry.svc:5000/openshift/httpd:latest' ports: - containerPort: 8080
apiVersion: apps/v1 kind: Deployment metadata: name: example namespace: openshift-sandboxed-containers-operator spec: selector: matchLabels: app: httpd replicas: 3 template: metadata: labels: app: httpd spec: runtimeClassName: kata containers: - name: httpd image: >- image-registry.openshift-image-registry.svc:5000/openshift/httpd:latest ports: - containerPort: 8080
apiVersion: v1 kind: Namespace metadata: name: openshift-sandboxed-containers-operator
oc create -f Namespace.yaml
apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-sandboxed-containers-operator namespace: openshift-sandboxed-containers-operator spec: targetNamespaces: - openshift-sandboxed-containers-operator
oc create -f OperatorGroup.yaml
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: openshift-sandboxed-containers-operator namespace: openshift-sandboxed-containers-operator spec: channel: "preview-1.1" installPlanApproval: Automatic name: sandboxed-containers-operator source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: sandboxed-containers-operator.v1.1.0
oc create -f Subscription.yaml
oc get csv -n openshift-sandboxed-containers-operator
NAME DISPLAY VERSION REPLACES PHASE openshift-sandboxed-containers openshift-sandboxed-containers-operator 1.1.0 1.0.2 Succeeded
apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig
apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig spec: kataConfigPoolSelector: matchLabels: <label_key>: '<label_value>' 1
oc create -f <file name>.yaml
watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
apiVersion: v1 kind: Pod metadata: name: mypod spec: runtimeClassName: kata
apiVersion: apps/v1 kind: Deployment metadata: name: mypod labels: app: mypod spec: replicas: 3 selector: matchLabels: app: mypod template: metadata: labels: app: mypod spec: runtimeClassName: kata containers: - name: mypod image: myImage